The Netflix TechBlog

JANUARY 13, 2022

The data platform is built on top of several distributed systems, and due to the inherent nature of these systems, it is inevitable that these workloads run into failures periodically. We have been working on an auto-diagnosis and remediation system called Pensive in the data platform to address these concerns.

Big Data 238

Big Data Infrastructure Metrics Hardware 238

The Netflix TechBlog

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Traffic 339

Traffic Latency Tuning Systems 339

Scalegrid

MARCH 14, 2024

Understanding Hybrid Cloud Strategy A hybrid cloud merges the capabilities of public and private clouds into a singular, coherent system. This combination allows for the fluid movement of data and applications across different environments, facilitating shared workloads seamlessly. We will examine each of these elements in more detail.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Storage Infrastructure 109

Scalegrid

MARCH 28, 2024

Key Takeaways Redis offers complex data structures and additional features for versatile data handling, while Memcached excels in simplicity with a fast, multi-threaded architecture for basic caching needs. Memcached shines in scenarios where a simple, fast, and efficient caching solution is required without data persistence.

Cache 130

Cache Storage Scalability Architecture 130

Dynatrace

OCTOBER 4, 2022

Limited data availability constrains value creation. Modern IT environments — whether multicloud, on-premises, or hybrid-cloud architectures — generate exponentially increasing data volumes. Grail addresses today’s challenges of big data and cloud everywhere: Grail is highly scalable, cost-effective, and super-fast.

Analytics 235

Analytics Artificial Intelligence Storage Serverless 235

Dynatrace

JANUARY 16, 2023

As Kubernetes adoption increases and it continues to advance technologically, Kubernetes has emerged as the “operating system” of the cloud. Kubernetes is emerging as the “operating system” of the cloud. Kubernetes is emerging as the “operating system” of the cloud. Kubernetes moved to the cloud in 2022.

Open Source 343

Open Source Java Operating System Programming 343

Dynatrace

JUNE 29, 2022

To drive better outcomes using hybrid cloud architectures, it helps to understand their benefits—and how to orchestrate them seamlessly. What is hybrid cloud architecture? Hybrid cloud architecture is a computing environment that shares data and applications on a combination of public clouds and on-premises private clouds.

Infrastructure 182

Infrastructure Cloud Azure AWS 182

Dynatrace

JULY 6, 2022

As organizations continue to adopt multicloud strategies, the complexity of these environments grows, increasing the need to automate cloud engineering operations to ensure organizations can enforce their policies and architecture principles. Big data automation tools. How organizations benefit from automating IT practices.

Artificial Intelligence 188

Artificial Intelligence Tuning Strategy Big Data 188

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., Seer is an online system that observes the behaviour of cloud applications (using the DeathStarBench microservices for the evaluation) and predicts when QoS violations may be about to occur. ASPLOS’19.

Big Data 45

Big Data Cloud Performance Hardware 45

The Morning Paper

OCTOBER 26, 2020

As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, but there’s a lot more to it than that. These two narratives of reference architecture and ingestion/indexing system are interwoven throughout the paper.

Cloud 104

Cloud Big Data Latency Architecture 104

Dynatrace

DECEMBER 15, 2022

In fact, Gartner estimates that 80% of enterprises will shut down their on-premises data centers by 2025. This transition to public, private, and hybrid cloud is driving organizations to automate and virtualize IT operations to lower costs and optimize cloud processes and systems. So, what is ITOps? Why is IT operations important?

Artificial Intelligence 202

Artificial Intelligence DevOps Hardware Virtualization 202

Dynatrace

APRIL 11, 2023

Cloud application security remains challenging because organizations lack end-to-end visibility into cloud architecture. As organizations migrate applications to the cloud, they must balance the agility that microservices architecture brings with the complexity and lack of transparency that can also come with it.

Cloud 186

Cloud DevOps Open Source Retail 186

ACM Sigarch

APRIL 8, 2019

Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. big-data processing, machine learning, quantum computing, and so on). For those of us who pursued computer architecture as a career, this is well understood. Why is that? Should we be alarmed as a community?

Architecture 74

Architecture Open Source Hardware Software Engineering 74

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. For unclassified errors, the job may be retried multiple times with the default retry policy.

Tuning 210

Tuning Efficiency Big Data Engineering 210

ScaleOut Software

JULY 19, 2021

How are we managing the torrent of telemetry that flows into analytics systems from these devices? Today’s streaming analytics architectures are not equipped to make sense of this rapidly changing information and react to it as it arrives. The list goes on. The Limitations of Today’s Streaming Analytics.

IoT 78

IoT Analytics Big Data Architecture 78

Dynatrace

MARCH 24, 2023

Containers enable developers to package microservices or applications with the libraries, configuration files, and dependencies needed to run on any infrastructure, regardless of the target system environment. This means organizations are increasingly using Kubernetes not just for running applications, but also as an operating system.

Infrastructure 186

Infrastructure Open Source Operating System Cloud 186

Dynatrace

MARCH 14, 2022

Over the past decade, the industry moved from paper-based to electronic health records (EHRs)—digitizing the backbone of patient data. As patient care continues to evolve, IT teams have accelerated this shift from legacy, on-premises systems to cloud technology to more build, test, and deploy software, and fuel healthcare innovation.

Healthcare 227

Healthcare Artificial Intelligence Innovation Strategy 227

The Netflix TechBlog

JUNE 7, 2021

By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs , ELB Access Logs, eBPF flow logs on the instances, etc, we can provide network insight to users and central teams through multiple data visualization techniques like Lumen , Atlas , etc. What is BPF?

Network 325

Network Transportation AWS Cloud 325

Dynatrace

APRIL 25, 2023

Exploratory analytics with collaborative analytics capabilities can be a lifeline for CloudOps, ITOps, site reliability engineering, and other teams struggling to access, analyze, and conquer the never-ending deluge of big data. These analytics can help teams understand the stories hidden within the data and share valuable insights.

Analytics 200

Analytics Big Data Media Operating System 200

Dynatrace

OCTOBER 17, 2022

Artificial intelligence for IT operations, or AIOps, combines big data and machine learning to provide actionable insight for IT teams to shape and automate their operational strategy. The four stages of data processing. This process continues until the system identifies a root cause. Achieving autonomous operations.

DevOps 196

DevOps Artificial Intelligence Healthcare Innovation 196

ACM Sigarch

MAY 31, 2023

Introduction Memory systems are evolving into heterogeneous and composable architectures. Heterogeneous and Composable Memory (HCM) offers a feasible solution for terabyte- or petabyte-scale systems, addressing the performance and efficiency demands of emerging big-data applications.

Latency 52

Latency Hardware Cache Architecture 52

Dynatrace

MARCH 11, 2021

I took a big-data-analysis approach, which started with another problem visualization. This is what I wanted to optimize and avoid and many traditional (or homegrown) systems aren’t doing this. For this visualization I used the same backend architecture as for the real-time visualization I presented previously.

Tuning 263

Tuning Architecture Monitoring Big Data 263

The Netflix TechBlog

OCTOBER 15, 2019

Another issue exists for the capture of schema changes, where some systems, like MySQL, don’t support transactional schema changes [1][2]. By their nature, they can only rely on the lowest common denominator of participating systems. That system quickly grew very complex and became difficult to maintain.

Transportation 192

Transportation Architecture Processing Storage 192

Dynatrace

OCTOBER 14, 2021

Gartner defines AIOps as the combination of “big data and machine learning to automate IT operations processes, including event correlation, anomaly detection, and causality determination.” This contrasts stochastic AIOps approaches that use probability models to infer the state of systems. What is AIOps?

Artificial Intelligence 212

Artificial Intelligence DevOps Innovation Metrics 212

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform. Apply here.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform. Apply here.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform. Apply here.

Education 102

Education Software Engineering Scalability Engineering 102

ScaleOut Software

NOVEMBER 24, 2020

Real-Time Digital Twins Can Add Important New Capabilities to Telematics Systems and Eliminate Scalability Bottlenecks. However, telematics architectures face challenges in responding to telemetry in real time. Current Telematics Architecture. Challenges for Current Architectures.

Analytics 52

Analytics Architecture Scalability Software Architecture 52

High Scalability

MAY 26, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 100

Education AWS Software Engineering Scalability 100

All Things Distributed

JUNE 26, 2016

A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world – India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more.

AWS 90

AWS Cloud Healthcare Blockchain 90

ACM Sigarch

NOVEMBER 1, 2019

This blog post gives a glimpse of the computer systems research papers presented at the USENIX Annual Technical Conference (ATC) 2019, with an emphasis on systems that use new hardware architectures. USENIX ATC is a top-tier venue with a broad range of systems research papers from both industry and academia.

Architecture 12

Architecture Hardware Cache Storage 12

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. Why are developers using RInK systems as part of their design? Fetching too much data in a single query (i.e.,

Cache 79

Cache Latency Google Lambda 79

All Things Distributed

JULY 16, 2012

Werner Vogels weblog on building scalable and robust distributed systems. There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.

AWS 83

AWS Big Data Media Storage 83

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks. All Things Distributed. Comments ().

Technology 151

Technology Technology AWS Storage 151

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. These trade-offs have even impacted the way the lowest level building blocks in our computer architectures have been designed. Because of its focus on latency, the generic CPU yielded rather inefficient system for graphics processing. Comments ().

AWS 136

AWS Latency Programming Architecture 136

Brendan Gregg

APRIL 30, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. We both have had long careers supporting system administration, and LISA has always felt like a homecoming, reuniting with old friends while welcoming newcomers. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley.

DevOps 43

DevOps Network Best Practices Programming 43