The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.

AWS 37

AWS Entertainment Open Source Benchmarking 37

The Netflix TechBlog

MARCH 5, 2019

All these micro-services are currently operated in AWS cloud infrastructure. As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. Give us a holler if you are interested in a thought exchange.

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

High Scalability

MAY 26, 2020

Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years. Advertise your job here! Cool Products and Services.

Education 100

Education AWS Software Engineering Scalability 100

O'Reilly

SEPTEMBER 15, 2021

Cloud certifications, specifically in AWS and Microsoft Azure, were most strongly associated with salary increases. As we’ll see later, cloud certifications (specifically in AWS and Microsoft Azure) were the most popular and appeared to have the largest effect on salaries. The top certification was for AWS (3.9% The Last Word.

Azure 145

Azure Social Media Programming AWS 145

All Things Distributed

AUGUST 24, 2012

The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile.

Storage 108

Storage Hardware AWS Data Engineering 108

The Symphonia

JUNE 19, 2018

I was fortunate to be both presenting a 2-day workshop (on AWS Serverless Architectures and Continuous Deployment) as well as hosting a full-day Serverless track of talks. This has proved especially true in the last couple of months, as we helped a company update it’s entire AWS infrastructure in a number of critical ways. Great stuff!

Serverless 40

Serverless AWS DevOps Open Source 40

InfoQ

DECEMBER 29, 2023

Zendesk reduced its data storage costs by over 80% by migrating from DynamoDB to a tiered storage solution using MySQL and S3. The company considered different storage technologies and decided to combine the relational database and the object store to strike a balance between querybility and scalability while keeping the costs down.

Storage 134

Storage Scalability Database Technology 134

O'Reilly

FEBRUARY 18, 2020

The results for data-related topics are both predictable and—there’s no other way to put it—confusing. Starting with data engineering, the backbone of all data work (the category includes titles covering data management, i.e., relational databases, Spark, Hadoop, SQL, NoSQL, etc.). This follows a 3% drop in 2018.

Software Architecture 134

Software Architecture DevOps Data Engineering Architecture 134

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

MAY 26, 2020

Service Segmentation: The ease of the cloud deployments has led to the organic growth of multiple AWS accounts, deployment practices, interconnection practices, etc. VPC Flow Logs VPC Flow Logs is an AWS feature that captures information about the IP traffic going to and from network interfaces in a VPC.

Network 150

Network Tuning AWS Big Data 150

All Things Distributed

OCTOBER 7, 2015

However, the data infrastructure to collect, store and process data is geared toward developers (e.g., In AWS’ quest to enable the best data storage options for engineers, we have built several innovative database solutions like Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift.

Cloud 137

Cloud Big Data AWS Analytics 137

Abhishek Tiwari

JULY 7, 2023

Airflow provides rich scheduling and execution semantics enabling data engineers to easily define complex pipelines, running at regular intervals. In reality, a DAG lacks the necessary workflow context, and relying solely on it can result in incomplete solutions and missed opportunities.

Logistics 52

Logistics Transportation Scalability Data Engineering 52

The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” We are loading the lineage data to a graph database to enable seamless integration with a REST data lineage service to address business use cases.

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

The Netflix TechBlog

OCTOBER 18, 2022

As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits. increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. Meson was based on a single leader architecture with high availability.

Java 202

Java Scalability Traffic Architecture 202

All Things Distributed

NOVEMBER 15, 2016

As I mentioned, we live in a world where massive volumes of data are being generated, every day, from connected devices, websites, mobile apps, and customer applications running on top of AWS infrastructure. Auto-discovery : One of the challenges with BI is discovering and accessing the data.

Analytics 152

Analytics Availability Media Social Media 152