All Things Distributed

JANUARY 20, 2020

Back when Jeff Bezos filled orders in his garage and drove packages to the post office himself, crunching the numbers on costs, tracking inventory, and forecasting future demand was relatively simple.

Big Data 202

Big Data Logistics Metrics Retail 202

Uber Engineering

AUGUST 3, 2018

From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data.

Big Data 110

Big Data Transportation Engineering Storage 110

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Data Engineering 63

DZone

AUGUST 23, 2023

In the era of big data, efficient data management and query performance are critical for organizations that want to get the best operational performance from their data investments.

Big Data 130

Big Data Scalability Infrastructure Efficiency 130

The Morning Paper

DECEMBER 13, 2020

An overview of end-to-end entity resolution for big data , Christophides et al., It’s an important part of many modern data workflows, and an area I’ve been wrestling with in one of my own projects. ACM Computing Surveys, Dec. 2020, Article No.

Big Data 83

Big Data Open Source Processing Systems 83

Rigor

MAY 22, 2019

Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. As the big data era brings in multiple options for visualization, it has become apparent that not all solutions are created equal. Conclusion.

Big Data 76

Big Data Monitoring Performance Metrics 76

DZone

JUNE 13, 2023

The reason is straightforward, today, applications generate enormous amounts of data. As we embrace new technologies like cloud computing, big data analysis, and the Internet of Things (IoT), there is a noticeable spike in the amount of data generated from different applications.

Scalability 249

Scalability IoT Big Data Cloud 249

DZone

JULY 17, 2023

Apache Spark is a leading platform in the field of big data processing, known for its speed, versatility, and ease of use. Understanding Apache Spark Apache Spark is a unified computing engine designed for large-scale data processing. However, getting the most out of Spark often involves fine-tuning and optimization.

Performance 130

Performance Big Data Open Source Tuning 130

DZone

FEBRUARY 23, 2023

Introduction With big data streaming platform and event ingestion service Azure Event Hubs , millions of events can be received and processed in a single second. Any real-time analytics provider or batching/storage adaptor can transform and store data supplied to an event hub.

Azure 301

Azure Scalability Big Data Storage 301

The Morning Paper

SEPTEMBER 8, 2019

Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries. VLDB’19. For the larger more production-like query analysed in §4.2.1,

Big Data 57

Big Data Latency Analytics Azure 57

DZone

FEBRUARY 1, 2023

Built on Azure Blob Storage, Azure Data Lake Storage Gen2 is a suite of features for big data analytics. Azure Data Lake Storage Gen1 and Azure Blob Storage's capabilities are combined in Data Lake Storage Gen2. For instance, Data Lake Storage Gen2 offers scale, file-level security, and file system semantics.

Azure 247

Azure Storage Big Data Analytics 247

Dynatrace

MAY 1, 2023

Then, big data analytics technologies, such as Hadoop, NoSQL, Spark, or Grail, the Dynatrace data lakehouse technology, interpret this information. Here are the six steps of a typical ITOA process : Define the data infrastructure strategy. Why use a data lakehouse for causal AI? Why is ITOA important? Apache Spark.

Analytics 176

Analytics Big Data Cloud Infrastructure 176

DZone

JANUARY 19, 2023

Because with the advent of cloud providers, we are less worried about managing data centers. This leads to an increase in the size of data as well. Big data is generated and transported using various mediums in single requests. Everything is available within seconds on-demand.

Best Practices 259

Best Practices Latency Transportation Big Data 259

DZone

JANUARY 31, 2023

Big data is at the center of all business decisions these days. It refers to large volumes of data generated through different sources, and this data then provides the foundation for business decisions. There are different ways through which we can process data.

Processing 239

Processing Big Data Systems 239

DZone

JUNE 6, 2023

This can include the use of cloud computing, artificial intelligence, big data analytics, the Internet of Things (IoT), and other digital tools. The digital transformation of businesses involves the adoption of digital technologies to change the way companies operate and deliver value to their customers.

Testing 130

Testing Systems Artificial Intelligence IoT 130

The Netflix TechBlog

OCTOBER 28, 2021

Netflix’s unique work culture and petabyte-scale data problems are what drew me to Netflix. During earlier years of my career, I primarily worked as a backend software engineer, designing and building the backend systems that enable big data analytics.

Data Engineering 187

Data Engineering Engineering Software Engineering Big Data 187

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Driving down the cost of Big-Data analytics. Comments ().

Big Data 113

Big Data Analytics AWS Cloud 113

Dynatrace

JUNE 26, 2023

Software analytics offers the ability to gain and share insights from data emitted by software systems and related operational processes to develop higher-quality software faster while operating it efficiently and securely. This involves big data analytics and applying advanced AI and machine learning techniques, such as causal AI.

Software 176

Software Software DevOps Analytics 176

High Scalability

SEPTEMBER 14, 2020

This is a guest post by Limor Maayan-Wainstein , a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more. High performance computing (HPC) enables you to solve complex problems which cannot be solved by regular computing.

Cloud 262

Cloud Servers Big Data Virtualization 262

The Netflix TechBlog

JULY 15, 2021

I was later hired into my first purely data gig where I was able to deepen my knowledge of big data. After that, I joined MySpace back at its peak as a data engineer and got my first taste of data warehousing at internet-scale. In the data engineering space, very little of the same technology remains.

Data Engineering 204

Data Engineering Engineering Big Data Analytics 204

DZone

JULY 22, 2022

Honestly, these two terms have recently been doing rounds in the big data world. These technologies specialize in transmitting large amounts of data across different trading partners and companies.

Big Data 188

Big Data Performance Technology Technology 188

The Netflix TechBlog

JANUARY 13, 2022

This blog will explore these two systems and how they perform auto-diagnosis and remediation across our Big Data Platform and Real-time infrastructure. The streaming platform recently added Data Mesh , and we need to expand Streaming Pensive to cover that.

Big Data 244

Big Data Infrastructure Metrics Engineering 244

DZone

DECEMBER 21, 2020

Applications used in the field of Big Data process huge amounts of information, and this often happens in real time. Naturally, such applications must be highly reliable so that no error in the code can interfere with data processing. It is an open-source framework for distributed processing of large amounts of data.

Code 155

Code Java Big Data Open Source 155

Uber Engineering

JUNE 1, 2023

Uber runs more than 100K big data workloads per day using Apache Spark–at that scale it’s crucial to write optimized apps. The Delivery Data Solutions team built Spark Analysers, a real-time system to catch anti-patterns in the Spark application at Uber scale, helping Uber developers optimize their apps.

Big Data 82

Big Data Systems Development 82

DZone

JANUARY 20, 2023

Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios.

Scalability 207

Scalability Database Architecture Hardware 207

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Cache 194

Cache Big Data Engineering Data Engineering 194

DZone

JULY 29, 2019

In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on.

Performance 160

Performance Big Data Storage Open Source 160

DZone

DECEMBER 21, 2020

Setting up a data warehouse is the first step towards fully utilizing big data analysis. Still, it is one of many that need to be taken before you can generate value from the data you gather. An important step in that chain of the process is data modeling and transformation.

Scalability 193

Scalability Big Data Google Processing 193

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., ASPLOS’19.

Big Data 45

Big Data Cloud Performance Cache 45

DZone

DECEMBER 28, 2020

Nowadays, Big Data tests mainly include data testing, paving the way for the Internet of Things to become the center point. Digital transformation is yet another significant focus point for the sectors and the enterprises that are ranking top on cloud and business analytics. Besides, AI and ML seem to reach a new level.

Testing 190

Testing Software Software Big Data 190

Dynatrace

OCTOBER 25, 2022

“AIOps platforms address IT leaders’ need for operations support by combining big data and machine learning functionality to analyze the ever-increasing volume, variety and velocity of data generated by IT in response to digital transformation.” – Gartner Market Guide for AIOps platforms.

DevOps 204

DevOps Cloud Infrastructure Big Data 204

The Morning Paper

OCTOBER 26, 2020

Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built. What follows is a discussion of where big data systems might be heading, heavily inspired by the remarks in this paper, but with several of my own thoughts mixed in.

Cloud 104

Cloud Latency Big Data Architecture 104

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Infrastructure Storage 109

Dynatrace

JULY 6, 2022

This kind of automation can support key IT operations, such as infrastructure, digital processes, business processes, and big-data automation. Big data automation tools. These tools provide the means to collect, transfer, and process large volumes of data that are increasingly common in analytics applications.

DevOps 177

DevOps Monitoring Tuning Infrastructure 177

Dynatrace

OCTOBER 4, 2022

While data lakehouses combine the flexibility and cost-efficiency of data lakes with the querying capabilities of data warehouses, it’s important to understand how these storage environments differ. Data warehouses. Data warehouses were the original big data storage option.

Storage 195

Storage Analytics Architecture Artificial Intelligence 195

Dynatrace

JULY 6, 2020

Our customers have frequently requested support for this first new batch of services, which cover databases, big data, networks, and computing. See the health of your big data resources at a glance. Azure HDInsight supports a broad range of use cases including data warehousing, machine learning, and IoT analytics.

Azure 143

Azure Cloud Database Monitoring 143

DZone

MAY 21, 2019

The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. That’s because NVMe provides 6x higher bandwidth and IOPS advantage compared to SAS/SATA SSD.

Performance 100

Performance Big Data Storage Processing 100