Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109

Big Data Latency Transportation Traffic 109

DZone

FEBRUARY 23, 2023

Introduction With big data streaming platform and event ingestion service Azure Event Hubs , millions of events can be received and processed in a single second. Any real-time analytics provider or batching/storage adaptor can transform and store data supplied to an event hub.

Azure 306

Azure Big Data Analytics Storage 306

DZone

MAY 25, 2024

As NetEase expands its business horizons, the logs and time series data it receives explode, and problems like surging storage costs and declining stability come. As NetEase's pick among all big data components for platform upgrades, Apache Doris fits into both scenarios and brings much faster query performance.

Best Practices 130

Best Practices Big Data Games Analytics 130

Uber Engineering

AUGUST 3, 2018

From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data.

Big Data 110

Big Data Transportation Engineering Storage 110

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage Artificial Intelligence FinTech Performance 100

Dynatrace

OCTOBER 4, 2022

A data lakehouse features the flexibility and cost-efficiency of a data lake with the contextual and high-speed querying capabilities of a data warehouse. Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. How does a data lakehouse work?

Artificial Intelligence 210

Artificial Intelligence Storage Analytics Government 210

DZone

JUNE 3, 2019

Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data. For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB.

Storage 100

Storage Performance Network Scalability 100

The Netflix TechBlog

OCTOBER 28, 2021

Netflix’s unique work culture and petabyte-scale data problems are what drew me to Netflix. During earlier years of my career, I primarily worked as a backend software engineer, designing and building the backend systems that enable big data analytics. You can learn more about it from my talk at the Flink forward conference.

Data Engineering 178

Data Engineering Engineering Big Data Software Engineering 178

DZone

JANUARY 20, 2023

It has been a norm to perceive that distributed databases use the method of adding cheap PC(s) to achieve scalability (storage and computing) and attempt to store data once and for all on demand. Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios.

Scalability 210

Scalability Big Data Hardware Internet 210

Dynatrace

OCTOBER 4, 2022

Teams have introduced workarounds to reduce storage costs. Additionally, efforts such as lowered data retention times, two-tiered storage systems, shaky index management, sampled data, and data pipelines reduce the overall amount of stored data. Dynatrace discovers logs automatically at scale.

Analytics 235

Analytics Artificial Intelligence Storage Serverless 235

The Netflix TechBlog

JUNE 20, 2024

In this talk, Jessica Larson shares her takeaways from building a new data platform post-GDPR. To handle errors efficiently, Netflix developed a rule-based classifier for error classification called “Pensive.”

Data Engineering 204

Data Engineering Engineering Entertainment Software Engineering 204

DZone

MAY 30, 2019

With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. This has given rise to a completely new set of computing workloads for Machine Learning which drives Artificial Intelligence applications.

Artificial Intelligence 100

Artificial Intelligence Social Media FinTech Storage 100

DZone

JULY 29, 2019

These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.

Big Data 162

Big Data Government Open Source Storage 162

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Driving Storage Costs Down for AWS Customers. Comments (). At werner.ly

Big Data 108

Big Data Analytics AWS Cloud 108

Dynatrace

JANUARY 11, 2023

As cloud and big data complexity scales beyond the ability of traditional monitoring tools to handle, next-generation cloud monitoring and observability are becoming necessities for IT teams. Cloud storage monitoring. What is cloud monitoring? Virtual machine (VM) monitoring.

Cloud 223

Cloud Monitoring Best Practices Infrastructure 223

Scalegrid

MARCH 28, 2024

With these goals in mind, two in-memory data stores, Redis and Memcached, have emerged as the top contenders. This article will explore how they handle data storage and scalability, perform in different scenarios, and, most importantly, how these factors influence your choice. Data transfer technology. 3d render.

Cache 130

Cache Storage Architecture Scalability 130

Dynatrace

APRIL 20, 2023

“Logs magnify these issues by far due to their volatile structure, the massive storage needed to process them, and due to potential gold hidden in their content,” Pawlowski said, highlighting the importance of log analysis. “The weakness of a data lake is they fail when you need to access them fast,” Pawlowski said.

Analytics 188

Analytics Infrastructure Storage Efficiency 188

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. With Amazon Glacier any organization now has access to the same data archiving capabilities as the worldâ??s

Storage 121

Storage Cloud AWS Media 121

Scalegrid

JUNE 28, 2024

Can RabbitMQ handle the high-throughput needs of big data applications? For high-throughput big data applications, RabbitMQ may fall short of expectations. How does RabbitMQ support different programming languages?

IoT 130

IoT Healthcare Programming Open Source 130

Dynatrace

MARCH 24, 2023

Problems include provisioning and deployment; load balancing; securing interactions between containers; configuration and allocation of resources such as networking and storage; and deprovisioning containers that are no longer needed. How does container orchestration work?

Infrastructure 187

Infrastructure Open Source Operating System Cloud 187

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Storage Infrastructure 109

Percona

OCTOBER 18, 2023

release , we added support for physical backups and restores to significantly reduce Recovery Time Objective ( RTO ), especially for big data sets. However, the problem of losing data between backups – in other words, Recovery Point Objective (RPO) – for physical backups was not solved. spec: backup: enabled: true.

Best Practices 88

Best Practices Storage AWS Big Data 88

ScaleOut Software

JULY 19, 2021

Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention. The best they can usually do in real-time using general purpose tools is to filter and look for patterns of interest.

IoT 78

IoT Analytics Big Data Architecture 78

DZone

MAY 21, 2019

The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. That’s because NVMe provides 6x higher bandwidth and IOPS advantage compared to SAS/SATA SSD.

Performance 100

Performance Big Data Storage Processing 100

Dynatrace

JANUARY 16, 2023

Redis is an in-memory key-value store and cache that simplifies processing, storage, and interaction with data in Kubernetes environments. Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch. Note: The survey excluded all commercial observability offerings, including Dynatrace.

Open Source 343

Open Source Java Operating System Programming 343

Dynatrace

DECEMBER 15, 2022

Besides the traditional system hardware, storage, routers, and software, ITOps also includes virtual components of the network and cloud infrastructure. AIOps (artificial intelligence for IT operations) combines big data, AI algorithms, and machine learning for actionable, real-time insights that help ITOps continuously improve operations.

Artificial Intelligence 203

Artificial Intelligence DevOps Hardware Virtualization 203

The Morning Paper

OCTOBER 26, 2020

Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built. What follows is a discussion of where big data systems might be heading, heavily inspired by the remarks in this paper, but with several of my own thoughts mixed in.

Cloud 104

Cloud Big Data Latency Architecture 104

Scalegrid

MARCH 14, 2024

Public Cloud Infrastructure Third-party providers run public cloud services, delivering a broad array of offerings like computing power, storage solutions, and network capabilities that enhance the functionality of a hybrid cloud architecture. We will examine each of these elements in more detail.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 209

Storage Latency Efficiency Data Engineering 209

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022.

Analytics 187

Analytics Innovation Metrics Database 187

The Netflix TechBlog

MAY 4, 2023

Given the scale of the data being generated using replay traffic, we record the responses from the two sides to a cost-effective cold storage facility using technology like Apache Iceberg. This summary provides an excellent high-level view of the analysis and the overall match rate across the production and replay paths.

Traffic 345

Traffic Latency Tuning Systems 345

Percona

JANUARY 19, 2023

If CPU usage is not a bottleneck in your setup, you can leverage compression as it can improve performance which means that less data needs to be read from disk and written to memory, and indexes are compressed too. It can help us to save costs on storage and backup times. MyRocks is shipped in Percona Server for MySQL.

Open Source 83

Open Source Storage Database Big Data 83

All Things Distributed

JUNE 26, 2016

AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift. Advanced problem solving that connects big data with machine learning. warehouses to glean business insights for jobs, ad spend, or financials for mobile apps.

AWS 86

AWS Cloud Healthcare Blockchain 86

The Netflix TechBlog

OCTOBER 15, 2019

Thus, ensuring the atomicity of writes across different storage technologies remains a challenging problem for applications [3]. Delta Delta has been developed to address the limitations of existing solutions for data synchronization, and also allows to enrich data on the fly. clock drift on the partition leader).

Transportation 197

Transportation Architecture Processing Storage 197

All Things Distributed

DECEMBER 8, 2016

It adopted Amazon Redshift, Amazon EMR and AWS Lambda to power its data warehouse, big data, and data science applications, supporting the development of product features at a fraction of the cost of competing solutions. Kik Interactive is a Canadian chat platform with hundreds of millions of users around the globe.

AWS 150

AWS Cloud Lambda Innovation 150

Tech News Gather

OCTOBER 7, 2023

By optimizing your marketing and customer service based on call data, you can outperform competitors who rely solely on digital analytics. Data-Driven Decision Making In the age of big data, data-driven decision-making is paramount. It can lead to increased market share and business growth.

Strategy 52

Strategy Big Data Scalability Games 52

Abhishek Tiwari

DECEMBER 22, 2017

Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. If the majority of your data is unstructured such as text, images, documents, etc. Classic ETL. Late transformation.

Big Data 40

Big Data Retail Storage Google 40