Big Data, Design, Scalability and Systems - Technology Performance Pulse

What is Greenplum Database? Intro to the Big Data Database

Scalegrid

MAY 13, 2020

It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. This feature-packed database provides powerful and rapid analytics on data that scales up to petabyte volumes.

Big Data

Big Data Database Artificial Intelligence Open Source

What is a Distributed Storage System

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. This guide delves into how these systems work, the challenges they solve, and their essential role in businesses and technology.

Storage

Storage Systems Big Data Azure

What Should You Know About Graph Database’s Scalability?

DZone

JANUARY 20, 2023

Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios. Do Not Be Misled Designing and implementing a scalable graph database system has never been a trivial task.

Scalability

Scalability Big Data Hardware Internet

In-Stream Big Data Processing

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. This system has been designed to supplement and succeed the existing Hadoop-based system that had too high latency of data processing and too high maintenance costs.

Big Data

Big Data Processing Lambda Database

Redis vs Memcached in 2024

Scalegrid

MARCH 28, 2024

In this comparison of Redis vs Memcached, we strip away the complexity, focusing on each in-memory data store’s performance, scalability, and unique features. Redis is better suited for complex data models, and Memcached is better suited for high-throughput, string-based caching scenarios.

Cache

Cache Storage Scalability Architecture

Mastering Hybrid Cloud Strategy

Scalegrid

MARCH 14, 2024

This approach allows companies to combine the security and control of private clouds with public clouds’ scalability and innovation potential. Understanding Hybrid Cloud Strategy A hybrid cloud merges the capabilities of public and private clouds into a singular, coherent system. A hybrid cloud strategy could be your answer.

Strategy

Strategy Cloud Artificial Intelligence Infrastructure

Driving down the cost of Big-Data analytics - All Things Distributed

All Things Distributed

AUGUST 18, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud.

Big Data

Big Data Analytics AWS Cloud

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io

Education

Education Software Engineering Scalability Engineering

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io

Education

Education Software Engineering Scalability Engineering

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

High Scalability

MARCH 30, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill. Who's Hiring? InterviewCamp.io

Education

Education Software Engineering Engineering Big Data

Post: Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc.

Education

Education Software Engineering Engineering Big Data

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

ACM Sigarch

MAY 31, 2023

Introduction Memory systems are evolving into heterogeneous and composable architectures. Heterogeneous and Composable Memory (HCM) offers a feasible solution for terabyte- or petabyte-scale systems, addressing the performance and efficiency demands of emerging big-data applications.

Latency

Latency Hardware Cache Architecture

Fast key-value stores: an idea whose time has come and gone

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. Why are developers using RInK systems as part of their design? Fetching too much data in a single query (i.e.,

Cache

Cache Latency Google Lambda

The AWS GovCloud (US) Region - All Things Distributed

All Things Distributed

AUGUST 16, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The Cloud First strategy is most visible with new Federal IT programs, which are all designed to be â??Cloud Government and Big Data. One particular early use case for AWS GovCloud (US) will be massive data processing and analytics.

AWS

AWS Government Big Data Cloud

Expanding the Cloud - Introducing Amazon ElastiCache - All Things.

All Things Distributed

AUGUST 22, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Systems that make extensive use of caching almost all report a significant reduction in the cost of their database tier. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. All Things Distributed. Comments ().

Cloud

Cloud Cache AWS Storage

No Server Required - Jekyll & Amazon S3 - All Things Distributed

All Things Distributed

AUGUST 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable. It is simple and elegant, as you would expect from someone who has won several design awards.

Servers

Servers Social Media AWS Website

Expanding the Cloud - AWS Import/Export Support for Amazon EBS.

All Things Distributed

JULY 7, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Amazon Import/Export is an important tool for customers to accelerate moving large amounts of data into the AWS storage systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Comments ().

AWS

AWS Cloud Storage Internet

Even more amazing papers at VLDB 2019 (that I didn’t have space to cover yet)

The Morning Paper

SEPTEMBER 19, 2019

” Crusher is a Google system for automatically discovering email templates (e.g. Hyper Dimension Shuffle describes how Microsoft improved the cost of data shuffling, one of the most costly operations, in their petabyte-scale internal big data analytics platform, SCOPE. for machine generated emails sent to humans).

Blockchain

Blockchain Hardware Google Analytics

NoSQL Data Modeling Techniques

Highly Scalable

MARCH 1, 2012

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. The main design theme is “ What answers do I have?” ” .

Database

Database Ecommerce Efficiency Engineering

Job Openings in AWS - Senior Leader in Database Services - All.

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The ideal candidate will be someone who has built and ran large scale distributed systems and/or databases. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Database Storage Scalability

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

All Things Distributed

AUGUST 20, 2012

Werner Vogels weblog on building scalable and robust distributed systems. It requires substantial upfront capital investments in cold data storage systems such as tape robots and tape libraries, then thereâ??s With Amazon Glacier any organization now has access to the same data archiving capabilities as the worldâ??s

Storage

Storage Cloud AWS Media

Introducing the AWS South America - All Things Distributed

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Expanding the Cloud â?? Introducing the AWS South America (Sao Paulo) Region.

AWS

AWS Latency Storage Big Data

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud - All.

All Things Distributed

JANUARY 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Flexibility is one of the key principles of Amazon Web Services - developers can select any programming language and software package, any operating system, any middleware and any database to build systems and applications that meet their requirements.

AWS

AWS Cloud Java Operating System

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. Naming is one of the fundamental concepts in Distributed Systems. By Werner Vogels on 05 December 2010 02:00 PM.

Cloud

Cloud Internet Internet AWS

40+ Best Web Development Blogs of 2018

KeyCDN

OCTOBER 2, 2018

Codrops Codrops features blogs with topics ranging from UI design and page animations to image formatting and general JavaScript practices. A List Apart A List Apart focuses on UX and branding from business and design-oriented perspectives. Topics include web design, security, web-based tools and workflows and more.

Development

Development Website Design Code

Hacking with AWS at The Next Web Hackaton - All Things Distributed

All Things Distributed

MARCH 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Up to 200 developers and designers will get together to hack up interesting applications using the Internets APIs and SDKs. If you want to be that team of 5-7 developer/designers you can sign up using this form. All Things Distributed. Comments ().

AWS

AWS Internet Internet Storage

Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo.

All Things Distributed

MARCH 2, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region.

AWS

AWS Cloud Games Latency

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Cluster Computer Instances for Amazon EC2 are a new instance type specifically designed for High Performance Computing applications. Not just for HPC but for mission critical enterprise systems such as OLTP. All Things Distributed. Comments ().

Cloud

Cloud AWS Automotive Latency

Driving Bandwidth Cost Down for AWS Customers. - All Things.

All Things Distributed

JUNE 29, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Driving Bandwidth Cost Down for AWS Customers. Comments ().

AWS

AWS Retail Innovation Strategy

Free at Last - A Fully Self-Sustained Blog Running in Amazon S3.

All Things Distributed

FEBRUARY 23, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Free at Last - A Fully Self-Sustained Blog Running in Amazon S3. Comments ().

AWS

AWS Storage Big Data Servers

From the Archives - Gapingvoid's Nobody Cares - All Things.

All Things Distributed

FEBRUARY 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. From the Archives - Gapingvoids Nobody Cares. Comments ().

AWS

AWS Storage Big Data Scalability

DROAM - Dreaming about Cheap Data Roaming - All Things.

All Things Distributed

JANUARY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. DROAM - Dreaming about Cheap Data Roaming. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications.

Wireless

Wireless AWS Internet Internet

New Route 53 and ELB features: IPv6, Zone Apex, WRR and more.

All Things Distributed

MAY 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. New Route 53 and ELB features: IPv6, Zone Apex, WRR and more. Comments ().

Internet

Internet Internet AWS Scalability

Reboot - All Things Distributed

All Things Distributed

SEPTEMBER 29, 2010

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. By Werner Vogels on 29 September 2010 07:50 AM. Comments ().

AWS

AWS Storage Big Data Scalability

World’s Top Web Performance Leaders To Watch

Rigor

SEPTEMBER 11, 2019

Rachel is also one of the people behind Perch , a PHP content management system (CMS). Vitaly specializes in front-end development, performance optimization, and responsive design, and he has made an epic contribution to the performance industry in addition to many other technology and design-related spaces. Patrick Hamann.

Performance

Performance Education Google Website

Spot Instances - Increased Control - All Things Distributed

All Things Distributed

JULY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Spot Instances - Increased Control. By Werner Vogels on 11 July 2011 07:22 AM.

AWS

AWS Storage Cloud Big Data

APAC Summer Tour - All Things Distributed

All Things Distributed

JULY 3, 2011

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. APAC Summer Tour. By Werner Vogels on 03 July 2011 03:57 PM. Comments ().

AWS

AWS Storage Big Data Cloud

I am looking for new application and platform services - All Things.

All Things Distributed

APRIL 23, 2010

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. I am looking for new application and platform services. Comments ().

AWS

AWS Storage Cloud Big Data

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

MARCH 25, 2019

Finally, imagine yourself in the role of a data platform reliability engineer tasked with providing advanced lead time to data pipeline (ETL) owners by proactively identifying issues upstream to their ETL jobs. Let’s review a few of these principles: Ensure data integrity ?—?Accurately Enable seamless integration?—?

Infrastructure

Infrastructure Big Data Transportation Architecture

Around the World in 28 Days - All Things Distributed

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Around the World in 28 Days. By Werner Vogels on 30 September 2010 04:27 AM.

AWS

AWS Storage Cloud Best Practices

Expanding the Cloud - New AWS Region: US-West (Northern.

All Things Distributed

DECEMBER 3, 2009

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics. All Things Distributed. Expanding the Cloud - New AWS Region: US-West (Northern California).

AWS

AWS Cloud Latency Storage

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Benchmarking Storage

What is Greenplum Database? Intro to the Big Data Database

What is a Distributed Storage System

Trending Sources

What Should You Know About Graph Database’s Scalability?

In-Stream Big Data Processing

Redis vs Memcached in 2024

Mastering Hybrid Cloud Strategy

Driving down the cost of Big-Data analytics - All Things Distributed

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Sponsored Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Sponsored Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Post: Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

Fast key-value stores: an idea whose time has come and gone

The AWS GovCloud (US) Region - All Things Distributed

Expanding the Cloud - Introducing Amazon ElastiCache - All Things.

No Server Required - Jekyll & Amazon S3 - All Things Distributed

Expanding the Cloud - AWS Import/Export Support for Amazon EBS.

Even more amazing papers at VLDB 2019 (that I didn’t have space to cover yet)

NoSQL Data Modeling Techniques

Job Openings in AWS - Senior Leader in Database Services - All.

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

Introducing the AWS South America - All Things Distributed

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud - All.

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

40+ Best Web Development Blogs of 2018

Hacking with AWS at The Next Web Hackaton - All Things Distributed

Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo.

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

Driving Bandwidth Cost Down for AWS Customers. - All Things.

Free at Last - A Fully Self-Sustained Blog Running in Amazon S3.

From the Archives - Gapingvoid's Nobody Cares - All Things.

DROAM - Dreaming about Cheap Data Roaming - All Things.

New Route 53 and ELB features: IPv6, Zone Apex, WRR and more.

Reboot - All Things Distributed

World’s Top Web Performance Leaders To Watch

Spot Instances - Increased Control - All Things Distributed

APAC Summer Tour - All Things Distributed

I am looking for new application and platform services - All Things.

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

Around the World in 28 Days - All Things Distributed

Expanding the Cloud - New AWS Region: US-West (Northern.

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

Stay Connected