The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., Finally, we show that Seer can identify application level design bugs, and provide insights on how to better architect microservices to achieve predictable performance. ASPLOS’19.

Big Data 45

Big Data Cloud Performance Hardware 45

The Netflix TechBlog

MAY 26, 2020

It is easier to tune a large Spark job for a consistent volume of data. As you may know, S3 can emit messages when events (such as a file creation events) occur which can be directed into an AWS SQS queue. As with any sustainable engineering design, focusing on simplicity is very important.

Network 150

Network Tuning AWS Big Data 150

Dynatrace

MARCH 14, 2022

AIOps (or “AI for IT operations”) uses artificial intelligence so that big data can help IT teams work faster and more effectively. There are two main approaches to AIOps: Traditional AIOps: Machine learning models identify correlations between IT events. Gartner introduced the concept of AIOps in 2016.

Healthcare 218

Healthcare Artificial Intelligence Innovation Strategy 218

Dynatrace

AUGUST 10, 2021

And, they got the chance to do just that at our recent event, DynatraceGo, which I’ll tell you a bit more about in this blog. BPAY is in the midst of its digital transformation journey in which it is discovering the critical importance of developing “contemporary ways of designing, operating, and using” its software. DynatraceGo!

DevOps 246

DevOps Innovation Big Data Cloud 246

Dynatrace

DECEMBER 15, 2022

ITOps refers to the process of acquiring, designing, deploying, configuring, and maintaining equipment and services that support an organization’s desired business outcomes. ITOps is an IT discipline involving actions and decisions made by the operations team responsible for an organization’s IT infrastructure. ITOps vs. AIOps.

Artificial Intelligence 191

Artificial Intelligence DevOps Hardware Virtualization 191

The Netflix TechBlog

DECEMBER 21, 2020

We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits. This article will list some of the use cases of AutoOptimize, discuss the design principles that help enhance efficiency, and present the high-level architecture.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 202

Java Scalability Traffic Architecture 202

The Morning Paper

OCTOBER 26, 2020

As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, but there’s a lot more to it than that. it’s getting much more complex and expensive to process, store, and secure potentially sensitive data.

Cloud 104

Cloud Big Data Latency Architecture 104

The Netflix TechBlog

APRIL 29, 2019

Various software systems are needed to design, build, and operate this CDN infrastructure, and a significant number of them are written in Python. Orchestration The Big Data Orchestration team is responsible for providing all of the services and tooling to schedule and execute ETL and Adhoc pipelines.

Open Source 279

Open Source Network Infrastructure Big Data 279

Scalegrid

MARCH 28, 2024

Redis Data Types and Structures The design of Redis’s data structures emphasizes versatility. On the other hand, an append-only file ensures data safety by recording every write operation that modifies the dataset, allowing for complete data reconstruction in the event of a restart. Data transfer technology.

Cache 130

Cache Storage Scalability Architecture 130

Dynatrace

OCTOBER 14, 2021

Gartner defines AIOps as the combination of “big data and machine learning to automate IT operations processes, including event correlation, anomaly detection, and causality determination.” Typically, only the aggregated events will be accessible to ML and will often exclude additional details. What is AIOps?

Artificial Intelligence 201

Artificial Intelligence DevOps Innovation Metrics 201

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MAY 26, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Learn to balance architecture trade-offs and design scalable enterprise-level software. Who's Hiring? InterviewCamp.io Try out their platform.

Education 100

Education AWS Software Engineering Scalability 100

High Scalability

MARCH 30, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc.

Education 105

Education Software Engineering Engineering Big Data 105

Dynatrace

JUNE 1, 2020

Application discovery, tracing and diagnostics (ADTD): Application discovery, tracing and diagnosis is a set of processes designed to understand the relationships between application servers, map transactions across these nodes, and enable the deep inspection of methods using bytecode instrumentation (BCI) and/or distributed tracing.

Artificial Intelligence 180

Artificial Intelligence Social Media Monitoring IoT 180

Dynatrace

JUNE 1, 2020

Application discovery, tracing and diagnostics (ADTD): Application discovery, tracing and diagnosis is a set of processes designed to understand the relationships between application servers, map transactions across these nodes, and enable the deep inspection of methods using bytecode instrumentation (BCI) and/or distributed tracing.

Monitoring 131

Monitoring Performance Social Media Artificial Intelligence 131

The Netflix TechBlog

NOVEMBER 3, 2023

However, this design decision led to a different set of challenges. By keeping the logic of individual Processors simple, it allowed them to be reusable so we could centrally manage and operate them at scale. It also allowed them to be composable, so users could combine the different Processors to express the logic they needed.

Processing 265

Processing Engineering Infrastructure Latency 265

Abhishek Tiwari

DECEMBER 27, 2017

In 2018, we will see new data integration patterns those rely either on a shared high-performance distributed storage interface ( Alluxio ) or a common data format ( Apache Arrow ) sitting between compute and storage. Based on data access pattern i.e. hot, warm and cold Alluxio makes. Machine learning meets data integration.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

VoltDB

MARCH 13, 2024

In the age of big-data-turned-massive-data, maintaining high availability , aka ultra-reliability, aka ‘uptime’, has become “paramount”, to use a ChatGPT word. Sampling every thousandth event, for example, might give you a degree of visibility without bogging the system down.

Availability 52

Availability Latency DevOps Systems 52

Abhishek Tiwari

JULY 7, 2023

In the era of big data and complex data processing, data pipelines have emerged as a popular solution for managing and manipulating data. They provide a systematic approach to extract, transform, and load (ETL) data from various sources, enabling organizations to derive valuable insights.

Logistics 52

Logistics Transportation Scalability Data Engineering 52

Highly Scalable

MAY 1, 2012

Let us start with a simple example that illustrates capabilities of probabilistic data structures: Let us have a data set that is simply a heap of ten million random integer values and we know that it contains not more than one million distinct values (there are many duplicates). what is the cardinality of the data set)?

Analytics 191

Analytics Traffic Big Data Efficiency 191

Rigor

SEPTEMBER 11, 2019

Jake is a frequent speaker at many popular conferences and events, such as 100 Days of Google Dev , JAMstakConf , JSConf , SmashingConf , and dozens of others. Sergey is a principal engineer at Meetup and a well-known performance educator who regularly runs monthly hands-on Meet4SPEED events in New York City. Sergey Chernyshev.

Performance 89

Performance Education Google Website 89

The Morning Paper

JUNE 23, 2019

Why are developers using RInK systems as part of their design? Generally to cache data (including non-persistent data that never sees a backing store), to share non-persistent data across application services (e.g. A high CPU cost due to marshalling data to/from the RInK store formats to the application data format.

Cache 79

Cache Latency Google Lambda 79

O'Reilly

JANUARY 13, 2020

The biggest stories in Swift last year were the releases of SwiftUI , Apple’s newest framework for designing user interfaces across all Apple devices, and Swift for TensorFlow , a platform for deep learning and differentiable programming integrating Google’s TensorFlow framework with Swift. ” What lies ahead?

Programming 133

Programming Java Google C++ 133

Highly Scalable

MARCH 10, 2015

This article describes six major optimization problems related to marketing and pricing that can be solved leveraging data mining techniques. Although these problems are very different, we are trying to establish a common framework that helps to design optimization and data mining tasks required for solutions.

Retail 152

Retail C++ Analytics Metrics 152

All Things Distributed

JULY 3, 2011

Next to customer visits I will take part in a number of events organized by AWS and by our partners. This week in Japan there are three public events planned: July 4 - AWS HPC Night at Fuji Soft Hall in Akihabara. Next weekend I will travel to Australia where we will have two " Navigating the Cloud with AWS" Events.

AWS 67

AWS Storage Cloud Big Data 67

The Morning Paper

SEPTEMBER 19, 2019

Hyper Dimension Shuffle describes how Microsoft improved the cost of data shuffling, one of the most costly operations, in their petabyte-scale internal big data analytics platform, SCOPE. What if the network was no longer the bottleneck? Some cool algorithms: Pigeonring speeds up thresholded similarity searches.

Blockchain 46

Blockchain Hardware Google Analytics 46

SQL Server According to Bob

FEBRUARY 15, 2018

I have the blessed opportunity to speak at many customer events. Microsoft engineering is actually sending quite a few folks over the Atlantic to come talk about SQL Server 2017, SQL Server on Linux, GDPR, Performance, Security, Azure Data Lake, Azure SQL Database, Azure SQL Data Warehouse, and Azure CosmosDB.

Engineering 40

Engineering Azure Best Practices Servers 40

All Things Distributed

SEPTEMBER 30, 2010

The AWS Events team is organizing a number of events where I will present together with a number of AWS customers: AWS Cloud Computing Event in Berlin on October 7 with AWS customers moviepilot , Cellular , Schnee von morgen and Plinga. In India there will be 3 AWS Cloud Computing events. Countdown to What is Next in AWS.

AWS 75

AWS Storage Cloud Best Practices 75

Highly Scalable

MARCH 1, 2012

The following figure depicts imaginary “evolution” of the major NoSQL system families, namely, Key-Value stores, BigTable-style databases, Document databases, Full Text Search Engines, and Graph databases: NoSQL Data Models. The main design theme is “ What answers do I have?” ” .

Database 279

Database Ecommerce Efficiency Engineering 279

All Things Distributed

DECEMBER 13, 2016

With the launch of the AWS Europe (London) Region, AWS can enable many more UK enterprise, public sector and startup customers to reduce IT costs, address data locality needs, and embark on rapid transformations in critical new areas, such as big data analysis and Internet of Things. Fraud.net is a good example of this.

AWS 166

AWS Cloud Artificial Intelligence IoT 166

Smashing Magazine

JULY 29, 2019

I hope to raise the profile of difficulties faced by real people, which are avoidable if we design and develop in a way that is sympathetic to their needs. In Amazon’s case, there is room to make some big data savings on the desktop site and we shouldn’t get complacent just because the screen size suggests I’m not on a mobile device.

Cache 97

Cache Google Mobile Network 97