Big Data Related Topics 
DZone
AUGUST 21, 2021
This article describes 3 different tricks that I used in dealing with big data sets (order of 10 million records) and that proved to enhance performance dramatically. tips and tricks big data performace big data sets
Scalegrid
MAY 13, 2020
It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. Polymorphic Data Storage.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
DZone
MARCH 7, 2023
Apache Spark is a powerful open-source distributed computing framework that provides a variety of APIs to support big data processing. Broadcast variables can be used to efficiently distribute large read-only data structures, such as lookup tables, to worker nodes.
Highly Scalable
AUGUST 20, 2013
The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. In recent years, this idea got a lot of traction and a whole bunch of solutions like Twitter’s Storm, Yahoo’s S4, Cloudera’s Impala, Apache Spark, and Apache Tez appeared and joined the army of Big Data and NoSQL systems. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.
All Things Distributed
JANUARY 20, 2020
Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars).
Scalegrid
NOVEMBER 25, 2019
ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data.
Uber Engineering
OCTOBER 17, 2018
To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.
Rigor
MAY 22, 2019
Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Many larger organizations leverage teams of data scientists to manage, configure and analyze the data pulled into BI and dashboarding solutions like Datadog and Grafana in an effort to get a single view into their performance. Automatic updates with the ability to manually refresh keeps the data featured on the dashboard accurate at all times.
Uber Engineering
AUGUST 3, 2018
From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.
The Morning Paper
DECEMBER 13, 2020
An overview of end-to-end entity resolution for big data , Christophides et al., It’s an important part of many modern data workflows, and an area I’ve been wrestling with in one of my own projects. Uncategorized Algorithms and data structures
DZone
AUGUST 4, 2022
big data sql apache spark spark sql metrics monitoring spark dataframe spark application sql optimization spark compute engineSpark-Radiant is Apache Spark Performance and Cost Optimizer.
The Morning Paper
SEPTEMBER 8, 2019
Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries. In total, the clusters store a few exabytes of data and are primarily responsible for all of the batch analytics at Microsoft. VLDB’19.
The Netflix TechBlog
DECEMBER 21, 2020
By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.
DZone
JUNE 29, 2020
Today data is an important factor for business success. In every business, it has been observed that data is playing a game-changing moment to improve business performance. Data is important and necessary in this increasingly competitive world.
DZone
APRIL 15, 2022
database big data performance connection pool connection management database connection pool dbcp jdbc connection poolRecently, I faced an issue related to a very high load on the database layer. The database was having too many connections in parallel.
The Netflix TechBlog
JANUARY 13, 2022
By Vikram Srivastava and Marcelo Mayworm Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. And we can’t discount the productivity impact it causes on data platform users.
The Morning Paper
MAY 14, 2019
Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., When trained on a weeks worth of trace data from a 20 server cluster, and then tested on traces collected from a different week (after servers had been patched, and the OS had been upgraded), Seer was correctly able to anticipate 93.45% of violations. ASPLOS’19.
The Netflix TechBlog
JUNE 7, 2021
At much less than 1% of CPU and memory on the instance, this highly performant sidecar provides flow data at scale for network insight. Flow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars.
The Netflix TechBlog
JULY 21, 2022
We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. Secondly, and more importantly, the sheer volume of the runtime data is a lot.
The Netflix TechBlog
OCTOBER 28, 2021
Data Engineers of Netflix?—?Interview Interview with Pallavi Phadnis This post is part of our “ Data Engineers of Netflix ” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Data Engineers of Netflix?—?Interview
The Netflix TechBlog
JULY 15, 2021
Data Engineers of Netflix?—?Interview Interview with Kevin Wylie This post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Kevin, what drew you to data engineering?
DZone
MAY 5, 2021
Anytime, every time or sometime you would have heard someone going around with data analysis and saying maybe this could have happened because of this, maybe users did not like the feature or maybe we were wrong all the time. Live Diagnostics.
All Things Distributed
AUGUST 18, 2011
Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. However, this cannot be done without efficient, scalable data analytics.
Uber Engineering
APRIL 5, 2018
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN
DZone
JANUARY 15, 2021
big data performance data science data security data integration etl data warehouse data breach eltIn the past, software systems ran in highly controlled environments on-premise and managed by an army of sysadmins. Today, migration to the cloud is relentless; the stage has completely shifted.
The Netflix TechBlog
MARCH 25, 2019
Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can Can I run a check myself to understand what data is behind this metric?” Let’s review a few of these principles: Ensure data integrity ?—?Accurately
DZone
JULY 29, 2019
In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.
Highly Scalable
MAY 1, 2012
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).
DZone
MARCH 16, 2020
This article compares different options for the in-memory maps and their performances in order for an application to move away from traditional RDBMS tables for frequently accessed data. java big data performance apache ignite in-memory data grid in-memory caching distributed cache
The Netflix TechBlog
JUNE 1, 2021
Data Engineers of Netflix?—?Interview Interview with Dhevi Rajendran Dhevi Rajendran This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix.
The Netflix TechBlog
JUNE 1, 2021
Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Data Engineers of Netflix?—?Interview
The Netflix TechBlog
MAY 26, 2020
By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, Custom Exporter Agents, etc, we can provide Network Insight to users through multiple data visualization techniques like Lumen , Atlas , etc.
ScaleOut Software
JULY 19, 2021
Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention.
Uber Engineering
DECEMBER 17, 2019
Architecture Uber Data Big Data Data Analytics Data Platform EoY Infrastructure Product Platform Uber Engineering
DZone
MAY 14, 2020
database big data performance benchmarking performance analysis redis alternative ec2 image ec2stack hardware news keydbWe've always been excited about Arm so when Amazon offered us early access to their new Arm-based instances we jumped at the chance to see what they could do.
The Netflix TechBlog
OCTOBER 15, 2019
Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), Beyond data synchronization, some applications also need to enrich their data by calling external services.
Uber Engineering
JANUARY 15, 2020
Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. .
Highly Scalable
MARCH 1, 2012
At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. To explore data modeling techniques, we have to start with a more or less systematic view of NoSQL data models that preferably reveals trends and interconnections.
Uber Engineering
DECEMBER 10, 2019
Architecture Uber Data Apache Spark Big Data Gradient Boosting Machine Learning Michelangelo ML Uber Engineering XGBoost
Uber Engineering
AUGUST 14, 2019
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Engineering Data Infrastructure data science Data Warehouse Engineering Efficiency
Highly Scalable
MARCH 10, 2015
Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods. Although there are many books on data mining in general and its applications to marketing and customer relationship management in particular [BE11, AS14, PR13 etc.],
DZone
JUNE 4, 2019
NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. Using a mix of historical data and financial modeling, one platform can provide the horsepower required for predicting future investment strategies for their financial customers. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.
Uber Engineering
OCTOBER 30, 2018
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster
Expert insights. Personalized for you.
246 
Let's personalize your content