Scalegrid

MAY 13, 2020

It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. Polymorphic Data Storage.

Big Data 197

Big Data Database Artificial Intelligence Open Source 197

Scalegrid

NOVEMBER 25, 2019

ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data.

Big Data 151

Big Data Database Open Source Google 151

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. In recent years, this idea got a lot of traction and a whole bunch of solutions like Twitter’s Storm, Yahoo’s S4, Cloudera’s Impala, Apache Spark, and Apache Tez appeared and joined the army of Big Data and NoSQL systems. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.

Big Data 136

Big Data Processing Lambda Systems 136

All Things Distributed

JANUARY 20, 2020

Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars). The team is constantly looking for ways to get more accurate data, faster. That's why, in 2019, they had an idea: Build a data lake that can support one of the largest logistics networks on the planet.

Big Data 129

Big Data Logistics Retail Government 129

The Morning Paper

DECEMBER 13, 2020

An overview of end-to-end entity resolution for big data , Christophides et al., It’s an important part of many modern data workflows, and an area I’ve been wrestling with in one of my own projects. Uncategorized Algorithms and data structures

Big Data 74

Big Data Open Source Processing Analytics 74

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Containerized data workloads running on Kubernetes offer several advantages over traditional virtual machine/bare metal based data workloads including but not limited to.

Big Data 55

Big Data Storage Benchmarking Hardware 55

Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.

Big Data 70

Big Data Latency Transportation Traffic 70

Uber Engineering

AUGUST 3, 2018

From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Big Data 70

Big Data Transportation Storage Infrastructure 70

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 204

Storage Big Data Infrastructure Efficiency 204

The Morning Paper

SEPTEMBER 8, 2019

Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries. In total, the clusters store a few exabytes of data and are primarily responsible for all of the batch analytics at Microsoft. VLDB’19.

Big Data 52

Big Data Analytics Latency Azure 52

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., When trained on a weeks worth of trace data from a 20 server cluster, and then tested on traces collected from a different week (after servers had been patched, and the OS had been upgraded), Seer was correctly able to anticipate 93.45% of violations. ASPLOS’19.

Big Data 55

Big Data Hardware Cache Latency 55

DZone

JUNE 29, 2020

Today data is an important factor for business success. In every business, it has been observed that data is playing a game-changing moment to improve business performance. Data is important and necessary in this increasingly competitive world. Data comes in large volumes everywhere and in complex structures. Being able to understand data is the preserve of longtime, highly paid data scientists and analysts.

Big Data 219

Big Data Games Analytics Performance 219

DZone

JANUARY 15, 2021

big data performance data science data security data integration etl data warehouse data breach eltIn the past, software systems ran in highly controlled environments on-premise and managed by an army of sysadmins. Today, migration to the cloud is relentless; the stage has completely shifted.

Engineering 122

Engineering Big Data Systems Infrastructure 122

The Netflix TechBlog

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can Can I run a check myself to understand what data is behind this metric?” Let’s review a few of these principles: Ensure data integrity ?—?Accurately

Infrastructure 204

Infrastructure Big Data Transportation Efficiency 204

The Netflix TechBlog

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), Beyond data synchronization, some applications also need to enrich their data by calling external services.

Big Data 189

Big Data Transportation Storage Architecture 189

DZone

JULY 29, 2019

In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.

Storage 140

Storage Government Open Source Big Data 140

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. However, this cannot be done without efficient, scalable data analytics.

Big Data 71

Big Data Analytics AWS Scalability 71

The Netflix TechBlog

MAY 26, 2020

While we strive to keep the ecosystem simple, the inherent nature of leveraging a variety of technologies will lead us to complications and challenges such as: App Dependencies and Data Flow Mappings: Without understanding and having visibility into an application’s dependencies and data flows, it is difficult for both service owners and centralized teams to identify systemic issues. At Netflix we publish the Flow Log data to Amazon S3.

Network 128

Network Big Data Tuning AWS 128

DZone

MARCH 16, 2020

This article compares different options for the in-memory maps and their performances in order for an application to move away from traditional RDBMS tables for frequently accessed data. java big data performance apache ignite in-memory data grid in-memory caching distributed cacheOverview.

Cache 109

Cache Java Database Big Data 109

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN

Systems 70

Systems Storage Big Data Infrastructure 70

DZone

MAY 14, 2020

database big data performance benchmarking performance analysis redis alternative ec2 image ec2stack hardware news keydbWe've always been excited about Arm so when Amazon offered us early access to their new Arm-based instances we jumped at the chance to see what they could do. We are, of course, referring to the Amazon EC2 M6g instances powered by AWS Graviton2 processors.

Benchmarking 100

Benchmarking AWS Big Data Hardware 100

Highly Scalable

MAY 1, 2012

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).

Analytics 139

Analytics Big Data Traffic Efficiency 139

DZone

JUNE 4, 2019

NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. Using a mix of historical data and financial modeling, one platform can provide the horsepower required for predicting future investment strategies for their financial customers. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.

Storage 100

Storage FinTech Artificial Intelligence Analytics 100

DZone

JUNE 3, 2019

Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data. big data performance data storage ssd nvme gpu ai ml

Storage 100

Storage Network Scalability Big Data 100

Highly Scalable

MARCH 1, 2012

At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. To explore data modeling techniques, we have to start with a more or less systematic view of NoSQL data models that preferably reveals trends and interconnections.

Big Data 219

Big Data Database Ecommerce Efficiency 219

Uber Engineering

JANUARY 15, 2020

Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. . Architecture Uber Data Apache Pinot Big Data Data Analytics Database Datastore Hadoop HDFS Presto SQL Uber Engineering

Engineering 65

Engineering Big Data Analytics Database 65

Uber Engineering

DECEMBER 17, 2019

Uber’s busy 2019 included our billionth delivery of an Uber Eats order, 24 million miles covered by bike and scooter riders on our platform, and trips to top destinations such as the Empire State Building, the Eiffel Tower, and the … The post Uber’s Data Platform in 2019: Transforming Information to Intelligence appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Analytics Data Platform EoY Infrastructure Product Platform Uber Engineering

Big Data 64

Big Data Infrastructure Analytics Engineering 64

Highly Scalable

MARCH 10, 2015

Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods. Although there are many books on data mining in general and its applications to marketing and customer relationship management in particular [BE11, AS14, PR13 etc.],

Retail 135

Retail Big Data Analytics C++ 135

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Engineering Data Infrastructure data science Data Warehouse Engineering Efficiency

Efficiency 58

Efficiency Engineering Storage Design 58

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster

Infrastructure 66

Infrastructure Technology Technology Big Data 66

Uber Engineering

DECEMBER 10, 2019

Michelangelo , Uber’s machine learning (ML) platform, powers machine learning model training across various use cases at Uber, such as forecasting rider demand , fraud detection , food discovery and recommendation for Uber Eats , and improving the accuracy of … The post Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber appeared first on Uber Engineering Blog.

Big Data 48

Big Data Engineering Architecture 48

Highly Scalable

JANUARY 31, 2012

Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. There is a software simulator of a digital communication system like WiMAX that passes some volume of random data through the system model and computes error probability of throughput. Applications: ETL, Data Analysis.

Big Data 129

Big Data C++ Ecommerce Network 129

Highly Scalable

JUNE 5, 2012

Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. At GridDynamics, we recently worked on a custom database for realtime web analytics where fast intersection of very large lists of IDs was a must for good performance.

C++ 101

C++ Big Data Java Efficiency 101

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster

Infrastructure 40

Infrastructure Technology Technology Software 40

Abhishek Tiwari

DECEMBER 27, 2017

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures.

Artificial Intelligence 46

Artificial Intelligence Storage Architecture Hardware 46

High Scalability

SEPTEMBER 14, 2020

This is a guest post by Limor Maayan-Wainstein , a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more.

Cloud 182

Cloud Virtualization Big Data Efficiency 182

DZone

DECEMBER 21, 2020

Applications used in the field of Big Data process huge amounts of information, and this often happens in real time. Naturally, such applications must be highly reliable so that no error in the code can interfere with data processing. Today, the Apache Flink project developed by the Apache Software Foundation, one of the leaders in the Big Data software market, was chosen as a test subject for the analyzer.

Code 139

Code Big Data Open Source Processing 139

DZone

DECEMBER 21, 2020

Setting up a data warehouse is the first step towards fully utilizing big data analysis. Still, it is one of many that need to be taken before you can generate value from the data you gather. An important step in that chain of the process is data modeling and transformation. This is where data is extracted, transformed, and loaded (ETL) or extracted, loaded, and transformed (ELT).

Google 125

Google Big Data Scalability Processing 125

The Morning Paper

OCTOBER 25, 2020

On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built.

Cloud 71

Cloud Big Data Latency Systems 71

DZone

DECEMBER 28, 2020

Nowadays, Big Data tests mainly include data testing, paving the way for the Internet of Things to become the center point. There has been a great advancement and growth in the sector of QA testing with the latest trends introducing into IT field services. The introduction of innovative technologies has brought the newest updates in software testing, development, design, and delivery. The high priority of enterprises around the world is cost optimization.

Software 152

Software Software Internet Internet 152

DZone

MAY 21, 2019

The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. Many businesses select non-volatile memory express (NVMe) storage when their data-intensive applications demand fast access to data.

Big Data 100

Big Data Storage Performance Processing 100

DZone

MAY 30, 2019

With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. This has given rise to a completely new set of computing workloads for Machine Learning which drives Artificial Intelligence applications. AI/ML can be applied across a broad spectrum of applications and industries.

Storage 100

Storage FinTech Social Media Artificial Intelligence 100

Dynatrace

SEPTEMBER 9, 2019

I love data. I have spent virtually my entire career looking at data. Synthetic data, network data, system data, and the list goes on. In recent years, the amount of data we analyze has exploded as we look at the data collected by Real User Monitoring (RUM), meaning every session, every action, in every region and so on. As much as I love data, data is cold, it lacks emotion. Dynatrace news.

Big Data 161

Big Data Virtualization Metrics Network 161