Scalegrid

MAY 13, 2020

It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data. It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. Polymorphic Data Storage.

Big Data 186

Big Data Database Artificial Intelligence Open Source 186

Scalegrid

NOVEMBER 25, 2019

ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data.

Big Data 144

Big Data Database Open Source Cloud 144

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. In recent years, this idea got a lot of traction and a whole bunch of solutions like Twitter’s Storm, Yahoo’s S4, Cloudera’s Impala, Apache Spark, and Apache Tez appeared and joined the army of Big Data and NoSQL systems. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.

Big Data 136

Big Data Processing Lambda Systems 136

All Things Distributed

JANUARY 20, 2020

Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars). The team is constantly looking for ways to get more accurate data, faster. That's why, in 2019, they had an idea: Build a data lake that can support one of the largest logistics networks on the planet.

Big Data 130

Big Data Logistics Retail Government 130

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Containerized data workloads running on Kubernetes offer several advantages over traditional virtual machine/bare metal based data workloads including but not limited to.

Big Data 55

Big Data Storage Benchmarking Hardware 55

Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.

Big Data 69

Big Data Latency Transportation Traffic 69

Rigor

MAY 22, 2019

Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Many larger organizations leverage teams of data scientists to manage, configure and analyze the data pulled into BI and dashboarding solutions like Datadog and Grafana in an effort to get a single view into their performance. Automatic updates with the ability to manually refresh keeps the data featured on the dashboard accurate at all times.

Big Data 64

Big Data Education Monitoring Best Practices 64

The Morning Paper

SEPTEMBER 8, 2019

Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries. In total, the clusters store a few exabytes of data and are primarily responsible for all of the batch analytics at Microsoft. VLDB’19.

Big Data 52

Big Data Analytics Latency Azure 52

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., When trained on a weeks worth of trace data from a 20 server cluster, and then tested on traces collected from a different week (after servers had been patched, and the OS had been upgraded), Seer was correctly able to anticipate 93.45% of violations. ASPLOS’19.

Big Data 52

Big Data Hardware Cache Latency 52

DZone

JUNE 29, 2020

Today data is an important factor for business success. In every business, it has been observed that data is playing a game-changing moment to improve business performance. Data is important and necessary in this increasingly competitive world. Data comes in large volumes everywhere and in complex structures. Being able to understand data is the preserve of longtime, highly paid data scientists and analysts.

Big Data 217

Big Data Games Analytics Performance 217

DZone

JULY 29, 2019

In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.

Government 138

Government Storage Open Source Big Data 138

The Netflix TechBlog

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), Beyond data synchronization, some applications also need to enrich their data by calling external services.

Big Data 173

Big Data Transportation Storage Architecture 173

The Netflix TechBlog

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can Can I run a check myself to understand what data is behind this metric?” Let’s review a few of these principles: Ensure data integrity ?—?Accurately

Infrastructure 188

Infrastructure Big Data Transportation Efficiency 188

DZone

MARCH 16, 2020

This article compares different options for the in-memory maps and their performances in order for an application to move away from traditional RDBMS tables for frequently accessed data. java big data performance apache ignite in-memory data grid in-memory caching distributed cacheOverview.

Cache 109

Cache Java Database Big Data 109

The Netflix TechBlog

MAY 26, 2020

While we strive to keep the ecosystem simple, the inherent nature of leveraging a variety of technologies will lead us to complications and challenges such as: App Dependencies and Data Flow Mappings: Without understanding and having visibility into an application’s dependencies and data flows, it is difficult for both service owners and centralized teams to identify systemic issues. At Netflix we publish the Flow Log data to Amazon S3.

Network 115

Network Tuning Big Data AWS 115

DZone

MAY 14, 2020

database big data performance benchmarking performance analysis redis alternative ec2 image ec2stack hardware news keydbWe've always been excited about Arm so when Amazon offered us early access to their new Arm-based instances we jumped at the chance to see what they could do. We are, of course, referring to the Amazon EC2 M6g instances powered by AWS Graviton2 processors.

Benchmarking 100

Benchmarking AWS Big Data Hardware 100

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. However, this cannot be done without efficient, scalable data analytics.

Big Data 63

Big Data Analytics AWS Scalability 63

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN

Systems 69

Systems Storage Big Data Infrastructure 69

Highly Scalable

MAY 1, 2012

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).

Analytics 139

Analytics Big Data Traffic Efficiency 139

DZone

JUNE 4, 2019

NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. Using a mix of historical data and financial modeling, one platform can provide the horsepower required for predicting future investment strategies for their financial customers. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.

Storage 100

Storage FinTech Artificial Intelligence Analytics 100

DZone

JUNE 3, 2019

Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data. big data performance data storage ssd nvme gpu ai ml

Storage 100

Storage Network Scalability Big Data 100

Highly Scalable

MARCH 1, 2012

At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. To explore data modeling techniques, we have to start with a more or less systematic view of NoSQL data models that preferably reveals trends and interconnections.

Big Data 219

Big Data Database Ecommerce Efficiency 219

Uber Engineering

JANUARY 15, 2020

Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. . Architecture Uber Data Apache Pinot Big Data Data Analytics Database Datastore Hadoop HDFS Presto SQL Uber Engineering

Engineering 65

Engineering Big Data Analytics Database 65

Uber Engineering

DECEMBER 17, 2019

Uber’s busy 2019 included our billionth delivery of an Uber Eats order, 24 million miles covered by bike and scooter riders on our platform, and trips to top destinations such as the Empire State Building, the Eiffel Tower, and the … The post Uber’s Data Platform in 2019: Transforming Information to Intelligence appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Analytics Data Platform EoY Infrastructure Product Platform Uber Engineering

Big Data 64

Big Data Infrastructure Analytics Engineering 64

Highly Scalable

MARCH 10, 2015

Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods. Although there are many books on data mining in general and its applications to marketing and customer relationship management in particular [BE11, AS14, PR13 etc.],

Retail 135

Retail Big Data Analytics C++ 135

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. Architecture Uber Data Big Data Data Engineering Data Infrastructure data science Data Warehouse Engineering Efficiency

Efficiency 58

Efficiency Engineering Storage Design 58

Uber Engineering

DECEMBER 10, 2019

Michelangelo , Uber’s machine learning (ML) platform, powers machine learning model training across various use cases at Uber, such as forecasting rider demand , fraud detection , food discovery and recommendation for Uber Eats , and improving the accuracy of … The post Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber appeared first on Uber Engineering Blog.

Big Data 48

Big Data Engineering Architecture 48

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster

Infrastructure 66

Infrastructure Technology Technology Big Data 66

Highly Scalable

JANUARY 31, 2012

Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. There is a software simulator of a digital communication system like WiMAX that passes some volume of random data through the system model and computes error probability of throughput. Applications: ETL, Data Analysis.

Big Data 129

Big Data C++ Ecommerce Network 129

Highly Scalable

JUNE 5, 2012

Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. At GridDynamics, we recently worked on a custom database for realtime web analytics where fast intersection of very large lists of IDs was a must for good performance.

C++ 101

C++ Big Data Java Efficiency 101

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster

Infrastructure 40

Infrastructure Technology Technology Software 40

Abhishek Tiwari

DECEMBER 27, 2017

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures.

Artificial Intelligence 46

Artificial Intelligence Storage Architecture Hardware 46

High Scalability

SEPTEMBER 14, 2020

This is a guest post by Limor Maayan-Wainstein , a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more.

Cloud 174

Cloud Virtualization Big Data Efficiency 174

The Morning Paper

OCTOBER 25, 2020

On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built.

Cloud 72

Cloud Big Data Latency Systems 72

DZone

MAY 21, 2019

The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations. Many businesses select non-volatile memory express (NVMe) storage when their data-intensive applications demand fast access to data.

Big Data 100

Big Data Storage Performance Processing 100

DZone

MAY 30, 2019

With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous. This has given rise to a completely new set of computing workloads for Machine Learning which drives Artificial Intelligence applications. AI/ML can be applied across a broad spectrum of applications and industries.

Storage 100

Storage FinTech Social Media Artificial Intelligence 100

Dynatrace

SEPTEMBER 9, 2019

I love data. I have spent virtually my entire career looking at data. Synthetic data, network data, system data, and the list goes on. In recent years, the amount of data we analyze has exploded as we look at the data collected by Real User Monitoring (RUM), meaning every session, every action, in every region and so on. As much as I love data, data is cold, it lacks emotion. Dynatrace news.

Big Data 167

Big Data Virtualization Metrics Network 167

The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. The processed data is typically stored as data warehouse tables in AWS S3. Figure 1 shows how we use Bulldozer to move data at Netflix.

Storage 141

Storage Latency Tuning Efficiency 141

Alex Podelko

DECEMBER 19, 2019

Boris has unique expertise in that area – especially in Big Data applications. The CMG Impact conference (February 10-12, 2020 in Las Vegas) is coming. Looking at the program I have the same problem as I always had with CMG conferences – how could I attend all the sessions I want considering that we have multiple tracks?

Efficiency 100

Efficiency Scalability Artificial Intelligence Blockchain 100

Dynatrace

JULY 6, 2020

Our customers have frequently requested support for this first new batch of services, which cover databases, big data, networks, and computing. All this comes with the Dynatrace zero-configuration approach, automatic service detection, continuous data capture in context, and answers, not just data, from the Dynatrace Davis AI engine, making you ready for large-scale Azure deployments. See the health of your big data resources at a glance. Dynatrace news.

Azure 124

Azure Cloud Database Virtualization 124

The Netflix TechBlog

SEPTEMBER 18, 2020

and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley.

Education 168

Education Data Engineering Innovation Efficiency 168

The Netflix TechBlog

APRIL 29, 2019

Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions. Python is also a tool we typically use for automation tasks, data exploration and cleaning, and as a convenient source for visualization work. We also use Python to detect sensitive data using Lanius.

Open Source 220

Open Source Infrastructure Network Media 220

Dynatrace

SEPTEMBER 15, 2020

After every experiment run Akamas changes application, runtime, database or cloud configuration based on monitoring data it captured during the previous experiment run. Supported technologies include cloud services, big data, databases, OS, containers, and application runtimes like the JVM.

Performance 170

Performance Metrics Technology Technology 170