Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

The Netflix TechBlog

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Traffic 339

Traffic Latency Tuning Systems 339

Dynatrace

OCTOBER 4, 2022

Limited data availability constrains value creation. Modern IT environments — whether multicloud, on-premises, or hybrid-cloud architectures — generate exponentially increasing data volumes. Grail addresses today’s challenges of big data and cloud everywhere: Grail is highly scalable, cost-effective, and super-fast.

Analytics 234

Analytics Artificial Intelligence Storage Serverless 234

Dynatrace

JANUARY 16, 2023

As Kubernetes adoption increases and it continues to advance technologically, Kubernetes has emerged as the “operating system” of the cloud. Kubernetes is emerging as the “operating system” of the cloud. Through effortless provisioning, a larger number of small hosts provide a cost-effective and scalable platform.

Open Source 342

Open Source Java Operating System Programming 342

Dynatrace

MARCH 24, 2023

Containers enable developers to package microservices or applications with the libraries, configuration files, and dependencies needed to run on any infrastructure, regardless of the target system environment. This means organizations are increasingly using Kubernetes not just for running applications, but also as an operating system.

Infrastructure 185

Infrastructure Open Source Operating System Cloud 185

The Netflix TechBlog

JUNE 7, 2021

By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs , ELB Access Logs, eBPF flow logs on the instances, etc, we can provide network insight to users and central teams through multiple data visualization techniques like Lumen , Atlas , etc. What is BPF?

Network 325

Network Transportation AWS Cloud 325

ScaleOut Software

JULY 19, 2021

How are we managing the torrent of telemetry that flows into analytics systems from these devices? Today’s streaming analytics architectures are not equipped to make sense of this rapidly changing information and react to it as it arrives. The list goes on. The Limitations of Today’s Streaming Analytics.

IoT 78

IoT Analytics Big Data Architecture 78

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MAY 26, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 100

Education AWS Software Engineering Scalability 100

ACM Sigarch

MAY 31, 2023

Introduction Memory systems are evolving into heterogeneous and composable architectures. Heterogeneous and Composable Memory (HCM) offers a feasible solution for terabyte- or petabyte-scale systems, addressing the performance and efficiency demands of emerging big-data applications.

Latency 52

Latency Hardware Cache Architecture 52

ScaleOut Software

NOVEMBER 24, 2020

Real-Time Digital Twins Can Add Important New Capabilities to Telematics Systems and Eliminate Scalability Bottlenecks. However, telematics architectures face challenges in responding to telemetry in real time. Current Telematics Architecture. Challenges for Current Architectures.

Analytics 52

Analytics Architecture Scalability Software Architecture 52

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. Why are developers using RInK systems as part of their design? Fetching too much data in a single query (i.e.,

Cache 79

Cache Latency Google Lambda 79

All Things Distributed

JULY 16, 2012

Werner Vogels weblog on building scalable and robust distributed systems. There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.

AWS 85

AWS Big Data Media Storage 85

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks. All Things Distributed. Comments ().

Technology 151

Technology Technology AWS Storage 151

ACM Sigarch

NOVEMBER 1, 2019

This blog post gives a glimpse of the computer systems research papers presented at the USENIX Annual Technical Conference (ATC) 2019, with an emphasis on systems that use new hardware architectures. USENIX ATC is a top-tier venue with a broad range of systems research papers from both industry and academia.

Architecture 12

Architecture Hardware Cache Storage 12

ScaleOut Software

APRIL 3, 2020

What’s missing is a flexible, fast, and easy-to-use software system that can be quickly adapted to track these assets in real time and provide immediate answers for logistics managers. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins.

Logistics 52

Logistics Analytics Scalability Cloud 52

ScaleOut Software

APRIL 3, 2020

What’s missing is a flexible, fast, and easy-to-use software system that can be quickly adapted to track these assets in real time and provide immediate answers for logistics managers. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins.

Logistics 52

Logistics Analytics Scalability Cloud 52

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. This team is constantly rethinking the assumptions behind how traditional databases were built and constantly working on building the right database architectures suited for the Cloud environment. Driving down the cost of Big-Data analytics.

AWS 77

AWS Database Storage Scalability 77

ScaleOut Software

JUNE 15, 2020

Instead, most applications just sift through the telemetry for patterns that might indicate exceptional conditions and forward the bulk of incoming messages to a data lake for offline scrubbing with a big data tool such as Spark. Maintain State Information for Each Data Source.

Analytics 52

Analytics IoT Lambda Big Data 52

ScaleOut Software

JUNE 15, 2020

Instead, most applications just sift through the telemetry for patterns that might indicate exceptional conditions and forward the bulk of incoming messages to a data lake for offline scrubbing with a big data tool such as Spark. Maintain State Information for Each Data Source.

Analytics 52

Analytics IoT Lambda Big Data 52

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. These trade-offs have even impacted the way the lowest level building blocks in our computer architectures have been designed. Because of its focus on latency, the generic CPU yielded rather inefficient system for graphics processing. Comments ().

AWS 136

AWS Latency Programming Architecture 136

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. Naming is one of the fundamental concepts in Distributed Systems. By Werner Vogels on 05 December 2010 02:00 PM.

Cloud 117

Cloud Internet Internet AWS 117

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Not just for HPC but for mission critical enterprise systems such as OLTP. Additionally, many high-end HPC applications take advantage of knowing their in-house hardware platforms to achieve major speedup by exploiting the specific processor architecture.

Cloud 119

Cloud AWS Automotive Latency 119

Brendan Gregg

APRIL 30, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. We both have had long careers supporting system administration, and LISA has always felt like a homecoming, reuniting with old friends while welcoming newcomers. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley.

DevOps 43

DevOps Network Best Practices Programming 43

All Things Distributed

MARCH 28, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Our smart phones and tablets are obvious examples, but many other devices are quickly gaining these capabilities; TV Sets and Hifi systems are internet enabled, and soon our treadmills and automobiles will be equally plugged into the digital world.

AWS 138

AWS Cloud Storage Internet 138

Brendan Gregg

APRIL 29, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. We both have had long careers supporting system administration, and LISA has always felt like a homecoming, reuniting with old friends while welcoming newcomers. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley.

DevOps 40

DevOps Network Best Practices Programming 40

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. Understanding Throughput-Oriented Architectures - background article in CACM on massively parallel and throughput vs latency oriented architectures.

AWS 68

AWS Cloud Benchmarking Storage 68

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup. Driving down the cost of Big-Data analytics.

AWS 77

AWS Storage Cloud Best Practices 77

The Netflix TechBlog

MARCH 25, 2019

Finally, imagine yourself in the role of a data platform reliability engineer tasked with providing advanced lead time to data pipeline (ETL) owners by proactively identifying issues upstream to their ETL jobs. Let’s review a few of these principles: Ensure data integrity ?—?Accurately Enable seamless integration?—?

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

The Netflix TechBlog

OCTOBER 18, 2022

As Big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have increasingly become more important for our data scientists and the company. Due to its popularity, the number of workflows managed by the system has grown exponentially.

Java 202

Java Scalability Traffic Architecture 202

Highly Scalable

MARCH 10, 2015

Most of this article represents an overview of the results published by retailers and researchers who built practical decision making and optimization systems combining abstract economic models with data mining methods. The goal is to suggest a personalized subset of incentives (e.g. Applications.

Retail 152

Retail C++ Analytics Metrics 152

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3. Moving data with Bulldozer at Netflix.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

DECEMBER 21, 2020

This article will list some of the use cases of AutoOptimize, discuss the design principles that help enhance efficiency, and present the high-level architecture. Use cases We found several use cases where a system like AutoOptimize can bring tons of value. Other Components Iceberg We use Apache Iceberg as the table format.

Storage 203

Storage Latency Efficiency Data Engineering 203