The Netflix TechBlog

OCTOBER 18, 2022

We started seeing signs of scale issues, like: Slowness during peak traffic moments like 12 AM UTC, leading to increased operational burden. Meson was based on a single leader architecture with high availability. At Netflix, the peak traffic load can be a few orders of magnitude higher than the average load.

Java 202

Java Scalability Traffic Architecture 202

The Netflix TechBlog

MAY 26, 2020

VPC Flow Logs VPC Flow Logs is an AWS feature that captures information about the IP traffic going to and from network interfaces in a VPC. At Netflix we publish the Flow Log data to Amazon S3. We also adjusted the frequency in which Spark app instances are spun up such that any backlog would burn off during a trough in traffic.

Network 150

Network Tuning AWS Big Data 150

The Netflix TechBlog

MARCH 5, 2019

As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. In the Performance space, our data teams currently focus on the quality of experience on Netflix-enabled devices.

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

The Netflix TechBlog

JANUARY 11, 2022

To learn about Analytics and Viz Engineering, have a look at Analytics at Netflix: Who We Are and What We Do by Molly Jackman & Meghana Reddy and How Our Paths Brought Us to Data and Netflix by Julie Beckley & Chris Pham. Curious to learn about what it’s like to be a Data Engineer at Netflix?

Innovation 243

Innovation Metrics Engineering Testing 243

InfoQ

NOVEMBER 29, 2023

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic. By Rafal Gancarz

Processing 77

Processing Traffic Data Engineering Scalability 77

Adrian Cockcroft

FEBRUARY 13, 2023

STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions. SUS302 Optimizing architectures for sustainability — Katja Philipp AWS SA and Szymon Kochanski AWS SA. SUS209 — there was no talk with this code.

AWS 64

AWS Energy Architecture Programming 64

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 83

Cache Latency Traffic Database 83