Big Data, Blog, Design and Scalability - Technology Performance Pulse

What is Greenplum Database? Intro to the Big Data Database

Scalegrid

MAY 13, 2020

It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. This feature-packed database provides powerful and rapid analytics on data that scales up to petabyte volumes.

Big Data

Big Data Database Artificial Intelligence Open Source

Incremental Processing using Netflix Maestro and Apache Iceberg

The Netflix TechBlog

NOVEMBER 20, 2023

In this blog post, we talk about the landscape and the challenges in workflows at Netflix. Whether in analyzing A/B tests, optimizing studio production, training algorithms, investing in content acquisition, detecting security breaches, or optimizing payments, well structured and accurate data is foundational.

Processing

Processing Big Data Efficiency Engineering

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

MARCH 25, 2019

Finally, imagine yourself in the role of a data platform reliability engineer tasked with providing advanced lead time to data pipeline (ETL) owners by proactively identifying issues upstream to their ETL jobs. Design a flexible data model ? —?Represent Enable seamless integration?—? push or pull.

Infrastructure

Infrastructure Big Data Transportation Architecture

Driving down the cost of Big-Data analytics - All Things Distributed

All Things Distributed

AUGUST 18, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud.

Big Data

Big Data Analytics AWS Cloud

40+ Best Web Development Blogs of 2018

KeyCDN

OCTOBER 2, 2018

Reading and following the right web development blogs makes it much easier to get a solid education. That’s why in this post, we’ve compiled a list of the top web development blogs that will help you find your footing and acquire essential skills in modern web dev. Visit website 3. Visit website 5. Visit website 9.

Development

Development Website Design Code

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

The Netflix TechBlog

OCTOBER 18, 2022

As Big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have increasingly become more important for our data scientists and the company. Another dimension of scalability to consider is the size of the workflow.

Java

Java Scalability Traffic Architecture

Streaming SQL in Data Mesh

The Netflix TechBlog

NOVEMBER 3, 2023

Democratizing Stream Processing @ Netflix By Guil Pires , Mark Cho , Mingliang Liu , Sujay Jain Data powers much of what we do at Netflix. On the Data Platform team, we build the infrastructure used across the company to process data at scale. However, this design decision led to a different set of challenges.

Processing

Processing Engineering Infrastructure Latency

Free at Last - A Fully Self-Sustained Blog Running in Amazon S3.

All Things Distributed

FEBRUARY 23, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Free at Last - A Fully Self-Sustained Blog Running in Amazon S3. In a follow up to the last blog post I have removed the last two dependencies this blog had on running a server somewhere: comments are now served by Disqus and search is now handled by Bing.

AWS

AWS Storage Big Data Servers

Optimizing data warehouse storage

The Netflix TechBlog

DECEMBER 21, 2020

We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits. This article will list some of the use cases of AutoOptimize, discuss the design principles that help enhance efficiency, and present the high-level architecture.

Storage

Storage Latency Efficiency Data Engineering

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

ACM Sigarch

MAY 31, 2023

Heterogeneous and Composable Memory (HCM) offers a feasible solution for terabyte- or petabyte-scale systems, addressing the performance and efficiency demands of emerging big-data applications. The memory bandwidth will be a key player because the traditional method to add memory bandwidth by adding memory channels is not scalable.

Latency

Latency Hardware Cache Architecture

New AWS feature: Run your website from Amazon S3 - All Things.

All Things Distributed

FEBRUARY 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Because my blog is almost completely static content I wanted to run in this very simple configuration since the launch of Amazon S3. It would allow the blog to be powered by the incredible scale and reliability of Amazon S3 with a minimum of effort from my side.

AWS

AWS Website Storage Servers

The AWS GovCloud (US) Region - All Things Distributed

All Things Distributed

AUGUST 16, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The Cloud First strategy is most visible with new Federal IT programs, which are all designed to be â??Cloud Government and Big Data. One particular early use case for AWS GovCloud (US) will be massive data processing and analytics.

AWS

AWS Government Big Data Cloud

No Server Required - Jekyll & Amazon S3 - All Things Distributed

All Things Distributed

AUGUST 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable. But I really want a setup that allows me to thinker with the blog where ever I am (e.g. Comments ().

Servers

Servers Social Media AWS Website

The Amazon.com 2010 Shareholder Letter Focusses on Technology.

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Given that I have frequently written about many of these technologies on this blog I asked investor relations to be allowed to reprint it here. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter.

Technology

Technology Technology AWS Storage

Expanding the Cloud - Introducing Amazon ElastiCache - All Things.

All Things Distributed

AUGUST 22, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Caching has become a standard component in many applications to achieve a fast and predictable performance, but maintaining a collection of cache servers in a reliable and scalable manner is not a simple task. blog comments powered by Disqus. Comments ().

Cloud

Cloud Cache AWS Storage

Introducing the AWS South America - All Things Distributed

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Please also visit the AWS developer blog for more great stories from our South American customers. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. Driving down the cost of Big-Data analytics.

AWS

AWS Latency Storage Big Data

Music to my Ears - All Things Distributed

All Things Distributed

MARCH 28, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The scalability, reliability and durability requirements for Cloud Drive are very high which is why they decided to make use of the Amazon Simple Storage Service (S3) as the core component of their service. blog comments powered by Disqus. Comments ().

AWS

AWS Cloud Storage Internet

Job Openings in AWS - Senior Leader in Database Services - All.

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Database Storage Scalability

Expanding the Cloud - AWS Import/Export Support for Amazon EBS.

All Things Distributed

JULY 7, 2011

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Storage Internet

From the Archives - Gapingvoid's Nobody Cares - All Things.

All Things Distributed

FEBRUARY 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. While cleaning out the digital attic I ran into this drawing that Hugh MacLeod (aka "gapingvoid") made for me in reponse to a storm-in-a-teacup about blogging Amazon. blog comments powered by Disqus. Driving down the cost of Big-Data analytics.

AWS

AWS Storage Big Data Scalability

Reboot - All Things Distributed

All Things Distributed

SEPTEMBER 29, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Like many folks who started down the path of using real-time micro-blogging services (read: twitter ) the convenience of those platforms has made that sharing happens there instead of on the blogs we (used to) keep. blog comments powered by Disqus.

AWS

AWS Storage Big Data Scalability

Hacking with AWS at The Next Web Hackaton - All Things Distributed

All Things Distributed

MARCH 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Up to 200 developers and designers will get together to hack up interesting applications using the Internets APIs and SDKs. If you want to be that team of 5-7 developer/designers you can sign up using this form. blog comments powered by Disqus.

AWS

AWS Internet Internet Storage

Driving Bandwidth Cost Down for AWS Customers. - All Things.

All Things Distributed

JUNE 29, 2011

Werner Vogels weblog on building scalable and robust distributed systems. For more details see the announcement , the details pages of the services at [link] , and the posting on the AWS developer blog. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. Comments ().

AWS

AWS Retail Innovation Strategy

Simplifying IT - Create Your Application with AWS CloudFormation.

All Things Distributed

FEBRUARY 25, 2011

Werner Vogels weblog on building scalable and robust distributed systems. They had taken the approach that they would not only be offering their software as a scalable multi-tenant product but also as a single tenant environment for customers that want to have their own isolated environment. blog comments powered by Disqus.

AWS

AWS Cloud Scalability Storage

Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo.

All Things Distributed

MARCH 2, 2011

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Games Latency

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

All Things Distributed

AUGUST 20, 2012

Werner Vogels weblog on building scalable and robust distributed systems. The service redundantly stores data in multiple facilities and on multiple devices within each facility, as Amazon Glacier is designed to provide average annual durability of 99.999999999% for each item stored. designed for 11 ninesâ??) Comments ().

Storage

Storage Cloud AWS Media

5 Terabyte Object Support in Amazon S3 - All Things Distributed

All Things Distributed

DECEMBER 9, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3. Amazon S3 has always been a scalable, durable and available data repository for almost any customer workload. blog comments powered by Disqus. All Things Distributed. Comments ().

AWS

AWS Big Data Scalability Storage

New Route 53 and ELB features: IPv6, Zone Apex, WRR and more.

All Things Distributed

MAY 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. More information can found on the Elastic Load Balancing and the Route 53 detail pages and on the two blog posts on the AWS developer blog. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter.

Internet

Internet Internet AWS Scalability

World’s Top Web Performance Leaders To Watch

Rigor

SEPTEMBER 11, 2019

Jake is a developer advocate at Google working with the Chrome team to develop and promote web standards and developer tools, as well as a contributor to the Chromium blog. Find his writing on progressive web apps, responsiveness, and mobile-first design across the web , and follow him on Twitter at @ grigs. Jake Archibald.

Performance

Performance Education Google Website

DROAM - Dreaming about Cheap Data Roaming - All Things.

All Things Distributed

JANUARY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. DROAM - Dreaming about Cheap Data Roaming. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. blog comments powered by Disqus. Driving down the cost of Big-Data analytics.

Wireless

Wireless AWS Internet Internet

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Cluster Computer Instances for Amazon EC2 are a new instance type specifically designed for High Performance Computing applications. Cluster Compute Instances for Amazon EC2 change the HPC game for two important reasons: Access to scalable on-demand capacity.

Cloud

Cloud AWS Automotive Latency

Spot Instances - Increased Control - All Things Distributed

All Things Distributed

JULY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. To get started using Spot or for more details visit the Amazon EC2 Spot Instance web page, the AWS developer blog , and the EC2 Release Notes. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter.

AWS

AWS Storage Cloud Big Data

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud - All.

All Things Distributed

JANUARY 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Elastic Beanstalk makes it easy for developers to deploy and manage scalable and fault-tolerant applications on the AWS cloud. More information on the launch can be found on the AWS developer blog. blog comments powered by Disqus. Comments ().

AWS

AWS Cloud Java Scalability

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. blog comments powered by Disqus. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Benchmarking Storage

APAC Summer Tour - All Things Distributed

All Things Distributed

JULY 3, 2011

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Storage Big Data Cloud

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Often these namespaces are hierarchical in nature such that it becomes easier to manage them and to decentralize control, which makes the system more scalable. blog comments powered by Disqus. Driving down the cost of Big-Data analytics.

Cloud

Cloud Internet Internet AWS

Expanding the Cloud - Amazon S3 Reduced Redundancy Storage.

All Things Distributed

MAY 18, 2010

Werner Vogels weblog on building scalable and robust distributed systems. This new storage option enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy. The same goes for durability; core to the design of S3 is that we go to great lengths to never, ever lose a single bit.

Storage

Storage Cloud AWS Scalability

I am looking for new application and platform services - All Things.

All Things Distributed

APRIL 23, 2010

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Storage Cloud Big Data

NoSQL Data Modeling Techniques

Highly Scalable

MARCH 1, 2012

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. First, we should note that SQL and relational model in general were designed long time ago to interact with the end user. The main design theme is “ What answers do I have?” ” .

Database

Database Ecommerce Efficiency Engineering

Around the World in 28 Days - All Things Distributed

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Storage Cloud Best Practices

Amazon Cloudfront is Streaming Media 2010 Editor's pick - All.

All Things Distributed

APRIL 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. As Dan Rayburn wrote on his Business of Video blog, "Amazon will be in the drivers seat to own the market for small and medium sized content owners who need simple delivery at a great price.". blog comments powered by Disqus. All Things Distributed.

Media

Media AWS Storage Big Data

Expanding the Cloud - New AWS Region: US-West (Northern.

All Things Distributed

DECEMBER 3, 2009

Werner Vogels weblog on building scalable and robust distributed systems. blog comments powered by Disqus. he posts material that doesnt belong on this blog or on twitter. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Driving down the cost of Big-Data analytics.

AWS

AWS Cloud Latency Storage

Choosing Consistency - All Things Distributed

All Things Distributed

FEBRUARY 24, 2010

Werner Vogels weblog on building scalable and robust distributed systems. These new features will make it easier to transition those applications to SimpleDB that are designed with traditional database tools in mind. All Things Distributed. Choosing Consistency. By Werner Vogels on 24 February 2010 07:00 AM. Comments ().

AWS

AWS Latency Database Scalability

Amazon EC2 Cluster GPU Instances - All Things Distributed

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. These trade-offs have even impacted the way the lowest level building blocks in our computer architectures have been designed. Also more details can be found on the AWS Developer blog. blog comments powered by Disqus. All Things Distributed.

AWS

AWS Latency Programming Architecture

What is Greenplum Database? Intro to the Big Data Database

Incremental Processing using Netflix Maestro and Apache Iceberg

Trending Sources

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

Driving down the cost of Big-Data analytics - All Things Distributed

40+ Best Web Development Blogs of 2018

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

Streaming SQL in Data Mesh

Free at Last - A Fully Self-Sustained Blog Running in Amazon S3.

Optimizing data warehouse storage

Current status, needs, and challenges in Heterogeneous and Composable Memory from the HCM workshop (HPCA’23)

New AWS feature: Run your website from Amazon S3 - All Things.

The AWS GovCloud (US) Region - All Things Distributed

No Server Required - Jekyll & Amazon S3 - All Things Distributed

The Amazon.com 2010 Shareholder Letter Focusses on Technology.

Expanding the Cloud - Introducing Amazon ElastiCache - All Things.

Introducing the AWS South America - All Things Distributed

Music to my Ears - All Things Distributed

Job Openings in AWS - Senior Leader in Database Services - All.

Expanding the Cloud - AWS Import/Export Support for Amazon EBS.

From the Archives - Gapingvoid's Nobody Cares - All Things.

Reboot - All Things Distributed

Hacking with AWS at The Next Web Hackaton - All Things Distributed

Driving Bandwidth Cost Down for AWS Customers. - All Things.

Simplifying IT - Create Your Application with AWS CloudFormation.

Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo.

Expanding the Cloud ? Managing Cold Storage with Amazon Glacier

5 Terabyte Object Support in Amazon S3 - All Things Distributed

New Route 53 and ELB features: IPv6, Zone Apex, WRR and more.

World’s Top Web Performance Leaders To Watch

DROAM - Dreaming about Cheap Data Roaming - All Things.

Expanding the Cloud - Cluster Compute Instances for Amazon EC2.

Spot Instances - Increased Control - All Things Distributed

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud - All.

This week in review: GPUs, Zombies, Biomimicry and Tom Waits.

APAC Summer Tour - All Things Distributed

Expanding the Cloud with DNS - Introducing Amazon Route 53 - All.

Expanding the Cloud - Amazon S3 Reduced Redundancy Storage.

I am looking for new application and platform services - All Things.

NoSQL Data Modeling Techniques

Around the World in 28 Days - All Things Distributed

Amazon Cloudfront is Streaming Media 2010 Editor's pick - All.

Expanding the Cloud - New AWS Region: US-West (Northern.

Choosing Consistency - All Things Distributed

Amazon EC2 Cluster GPU Instances - All Things Distributed

Stay Connected