Expert insights. Personalized for you.
database big data performance benchmarking performance analysis redis alternative ec2 image ec2stack hardware news keydbWe've always been excited about Arm so when Amazon offered us early access to their new Arm-based instances we jumped at the chance to see what they could do. MORE
Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. . MORE
Architecture Uber Data Apache Spark Big Data Gradient Boosting Machine Learning Michelangelo ML Uber Engineering XGBoost MORE
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. MORE
Beyond data synchronization, some applications also need to enrich their data by calling external services. Delta is an eventual consistent, event driven, data synchronization and enrichment platform. MORE
With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. MORE
Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. All Things Distributed. MORE
Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. big data performance data storage ssd nvme gpu ai ml MORE
Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions. MORE
Boris has unique expertise in that area – especially in Big Data applications. The CMG Impact conference (February 10-12, 2020 in Las Vegas) is coming. Looking at the program I have the same problem as I always had with CMG conferences – how could I attend all the sessions I want considering that we have multiple tracks? MORE
This article compares different options for the in-memory maps and their performances in order for an application to move away from traditional RDBMS tables for frequently accessed data. java big data performance apache ignite in-memory data grid in-memory caching distributed cache MORE
By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, Custom Exporter Agents, etc, we can provide Network Insight to users through multiple data visualization techniques like Lumen , Atlas , etc. MORE
Architecture Uber Data Big Data Data Analytics Data Platform EoY Infrastructure Product Platform Uber Engineering MORE
big-data processing, machine learning, quantum computing, and so on). Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. For those of us who pursued computer architecture as a career, this is well understood. MORE
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. MORE
Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. Design a flexible data model ? —?Represent MORE
I love data. I have spent virtually my entire career looking at data. Synthetic data, network data, system data, and the list goes on. As much as I love data, data is cold, it lacks emotion. Dynatrace news. MORE
Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries. MORE
Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., For the services under study, Seer has a sweet spot when trained with around 100GB of data and a 100ms sampling interval for measuring queue depths. MORE
NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases. MORE
Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Alternatively, a number of organizations have created their own internal home-grown systems for managing and distilling web performance and monitoring data. MORE
Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars). MORE
Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Obviously, containerized data workloads have their own challenges. MORE
The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. MORE
Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. Applications: ETL, Data Analysis. MORE
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN MORE
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture. Common in-memory data interfaces. MORE
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements). MORE
It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. Polymorphic Data Storage. MORE
From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog. MORE
At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. MORE
To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets. MORE
The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data. MORE
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. MORE
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. MORE
Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. MORE
ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data. MORE
In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch. MORE
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster MORE
These questions can be answered using the latest data as it streams in from the field. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins. MORE
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. MORE
Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods. MORE
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. MORE
After recreating the dataset, you can plot the raw numbers and perform custom analyses to understand the distribution of the data across test cells. However, the default reports only provide a summary view of the data with some powerful but limited filtering options. MORE
Big Data Related Topics 
Scalegrid
MAY 13, 2020
It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. Polymorphic Data Storage.
Highly Scalable
AUGUST 20, 2013
The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Scalegrid
NOVEMBER 25, 2019
ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data.
All Things Distributed
JANUARY 20, 2020
Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars).
Abhishek Tiwari
DECEMBER 27, 2017
Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Obviously, containerized data workloads have their own challenges.
Rigor
MAY 22, 2019
Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Alternatively, a number of organizations have created their own internal home-grown systems for managing and distilling web performance and monitoring data.
Uber Engineering
OCTOBER 17, 2018
To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.
Uber Engineering
AUGUST 3, 2018
From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.
The Morning Paper
SEPTEMBER 8, 2019
Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries.
The Netflix TechBlog
MAY 26, 2020
By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, Custom Exporter Agents, etc, we can provide Network Insight to users through multiple data visualization techniques like Lumen , Atlas , etc.
DZone
MAY 14, 2020
database big data performance benchmarking performance analysis redis alternative ec2 image ec2stack hardware news keydbWe've always been excited about Arm so when Amazon offered us early access to their new Arm-based instances we jumped at the chance to see what they could do.
DZone
MARCH 16, 2020
This article compares different options for the in-memory maps and their performances in order for an application to move away from traditional RDBMS tables for frequently accessed data. java big data performance apache ignite in-memory data grid in-memory caching distributed cache
DZone
JULY 29, 2019
In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.
The Netflix TechBlog
OCTOBER 15, 2019
Beyond data synchronization, some applications also need to enrich their data by calling external services. Delta is an eventual consistent, event driven, data synchronization and enrichment platform.
The Netflix TechBlog
MARCH 25, 2019
Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. Design a flexible data model ? —?Represent
All Things Distributed
AUGUST 18, 2011
Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. All Things Distributed.
Highly Scalable
MAY 1, 2012
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).
DZone
JUNE 4, 2019
NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.
Uber Engineering
JANUARY 15, 2020
Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. .
DZone
JUNE 3, 2019
Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. big data performance data storage ssd nvme gpu ai ml
Uber Engineering
APRIL 5, 2018
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN
Uber Engineering
DECEMBER 17, 2019
Architecture Uber Data Big Data Data Analytics Data Platform EoY Infrastructure Product Platform Uber Engineering
Highly Scalable
MARCH 1, 2012
At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques.
Uber Engineering
AUGUST 14, 2019
Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.
Highly Scalable
MARCH 10, 2015
Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods.
Uber Engineering
DECEMBER 10, 2019
Architecture Uber Data Apache Spark Big Data Gradient Boosting Machine Learning Michelangelo ML Uber Engineering XGBoost
Highly Scalable
JANUARY 31, 2012
Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. Applications: ETL, Data Analysis.
Uber Engineering
OCTOBER 30, 2018
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster
Highly Scalable
JUNE 5, 2012
Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures.
Uber Engineering
OCTOBER 30, 2018
Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware.
Abhishek Tiwari
DECEMBER 27, 2017
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture. Common in-memory data interfaces.
DZone
MAY 21, 2019
The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations.
Dynatrace
SEPTEMBER 9, 2019
I love data. I have spent virtually my entire career looking at data. Synthetic data, network data, system data, and the list goes on. As much as I love data, data is cold, it lacks emotion. Dynatrace news.
DZone
MAY 30, 2019
With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous.
Alex Podelko
DECEMBER 19, 2019
Boris has unique expertise in that area – especially in Big Data applications. The CMG Impact conference (February 10-12, 2020 in Las Vegas) is coming. Looking at the program I have the same problem as I always had with CMG conferences – how could I attend all the sessions I want considering that we have multiple tracks?
The Netflix TechBlog
APRIL 29, 2019
Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions.
High Scalability
MAY 26, 2020
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.
ACM Sigarch
APRIL 8, 2019
big-data processing, machine learning, quantum computing, and so on). Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. For those of us who pursued computer architecture as a career, this is well understood.
High Scalability
MAY 12, 2020
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.
High Scalability
APRIL 28, 2020
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.
High Scalability
APRIL 14, 2020
Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.
ScaleOut Software
APRIL 3, 2020
These questions can be answered using the latest data as it streams in from the field. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins.
The Netflix TechBlog
SEPTEMBER 10, 2019
After recreating the dataset, you can plot the raw numbers and perform custom analyses to understand the distribution of the data across test cells. However, the default reports only provide a summary view of the data with some powerful but limited filtering options.
130 
Let's personalize your content