Scalegrid

NOVEMBER 25, 2019

ScyllaDB is an open-source distributed NoSQL data store, reimplemented from the popular Apache Cassandra database. ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. There are dozens of quality articles on ScyllaDB vs. Cassandra, so we’ll stop short here so we can get to the real purpose of this article, breaking down the ScyllaDB user data.

Big Data 100

Big Data Database Cloud Benchmarking 100

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. The engine can also involve relatively static data (admixtures) loaded from the stores of Aggregated Data.

Big Data 176

Big Data Processing Analytics Lambda 176

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Obviously, containerized data workloads have their own challenges.

Big Data 71

Big Data Storage Benchmarking Hardware 71

Rigor

MAY 22, 2019

Big data is like the pollution of the information age. The Big Data Struggle and Performance Reporting. Alternatively, a number of organizations have created their own internal home-grown systems for managing and distilling web performance and monitoring data.

Big Data 79

Big Data Monitoring Performance Metrics 79

Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog. Uber is committed to delivering safer and more reliable transportation across our global markets.

Big Data 72

Big Data Latency Transportation Traffic 72

The Morning Paper

SEPTEMBER 8, 2019

Experiences with approximating queries in Microsoft’s production big-data clusters Kandula et al., Microsoft’s big data clusters have 10s of thousands of machines, and are used by thousands of users to run some pretty complex queries.

Big Data 91

Big Data Analytics Azure Latency 91

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., For the services under study, Seer has a sweet spot when trained with around 100GB of data and a 100ms sampling interval for measuring queue depths.

Big Data 90

Big Data Cloud Hardware Latency 90

The Netflix TechBlog

MARCH 25, 2019

Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. Design a flexible data model ? —?Represent

Infrastructure 283

Infrastructure Big Data Transportation Efficiency 283

DZone

JULY 29, 2019

In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.

Big Data 182

Big Data Government Storage Open Source 182

The Netflix TechBlog

OCTOBER 15, 2019

Beyond data synchronization, some applications also need to enrich their data by calling external services. Delta is an eventual consistent, event driven, data synchronization and enrichment platform.

Big Data 197

Big Data Transportation Processing Architecture 197

DZone

JUNE 4, 2019

NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. big data ai data storage ml nvme peperformanceNVMe Storage Use Cases.

Storage 130

Storage FinTech Artificial Intelligence Analytics 130

DZone

JUNE 3, 2019

Using local SSDs inside of the GPU node delivers fast access to data during training, but introduces challenges that impact the overall solution in terms of scalability, data access, and data protection. big data performance data storage ssd nvme gpu ai ml

Storage 130

Storage Big Data Network Scalability 130

Highly Scalable

MAY 1, 2012

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. The picture above depicts the fact that this data set basically occupies 40MB of memory (10 million of 4-byte elements).

Analytics 181

Analytics Big Data Traffic Efficiency 181

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. Architecture Apache Hadoop Big Data Big Data Analysis Garbage Collection Hadoop Hadoop Distributed File System Hadoop Infra & Analytics Team HBase HDFS HDFS Federation HIVE Hoodie Infrastructure JAR Jenkins NameNode PRC Puppet Router-based Federation Uber Uber Data Foundation Team Uber Engineering View File System WebHDFS YARN

Systems 61

Systems Storage Big Data Analytics 61

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Efficiency 69

Efficiency Engineering Storage Design 69

Highly Scalable

MARCH 1, 2012

At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques.

Big Data 254

Big Data Database Ecommerce Efficiency 254

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. All Things Distributed.

Big Data 52

Big Data Analytics AWS Scalability 52

Highly Scalable

MARCH 10, 2015

Retail is one of the most important business domains for data science and data mining applications because of its prolific data and numerous optimization problems such as optimal prices, discounts, recommendations, and stock levels that can be solved using data analysis methods.

Retail 175

Retail Big Data Analytics C++ 175

Highly Scalable

JANUARY 31, 2012

Applications: Log Analysis, Data Querying. Applications: Log Analysis, Data Querying, ETL, Data Validation. Solution: Problem description is split in a set of specifications and specifications are stored as input data for Mappers. Applications: ETL, Data Analysis.

Big Data 168

Big Data C++ Ecommerce Network 168

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. Architecture Apache Hadoop Apache Spark Big Data Capacity Planning Cassandra Cluster Management Data Center Hardware MySQL Peloton Redis Uber Uber Engineering Unified Resource Scheduler Workload Cluster

Infrastructure 64

Infrastructure Technology Technology Software 64

Highly Scalable

JUNE 5, 2012

Big Data Fundamentals Lucene algorithm index information retrieval lucene simd sseIntersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures.

Big Data 131

Big Data C++ Java Efficiency 131

Abhishek Tiwari

DECEMBER 27, 2017

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture. Common in-memory data interfaces.

Artificial Intelligence 60

Artificial Intelligence Storage Architecture Hardware 60

DZone

MAY 21, 2019

The demand for more IT resource-intensive applications has significantly increased today, whether it is to process quicker transactions, gain real-time insight, crunch big data sets, or to meet customer expectations.

Performance 130

Performance Big Data Storage Processing 130

Dynatrace

SEPTEMBER 9, 2019

I love data. I have spent virtually my entire career looking at data. Synthetic data, network data, system data, and the list goes on. As much as I love data, data is cold, it lacks emotion. Dynatrace news.

Big Data 224

Big Data Metrics Virtualization Network 224

DZone

MAY 30, 2019

With big data on the rise and data algorithms advancing, the ways in which technology has been applied to real-world challenges have grown more automated and autonomous.

Storage 130

Storage FinTech Artificial Intelligence Social Media 130

The Netflix TechBlog

APRIL 29, 2019

Device interaction for the collection of health and other operational data is yet another Python application. We are heavy users of Jupyter Notebooks and nteract to analyze operational data and prototype visualization tools that help us detect capacity regressions.

Open Source 286

Open Source Programming Infrastructure Network 286

The Netflix TechBlog

SEPTEMBER 10, 2019

After recreating the dataset, you can plot the raw numbers and perform custom analyses to understand the distribution of the data across test cells. However, the default reports only provide a summary view of the data with some powerful but limited filtering options.

Metrics 277

Metrics Infrastructure Architecture Testing 277

ACM Sigarch

APRIL 8, 2019

big-data processing, machine learning, quantum computing, and so on). Computer architecture is an important and exciting field of computer science, which enables many other fields (eg. For those of us who pursued computer architecture as a career, this is well understood.

Architecture 82

Architecture Education Open Source Google 82

The Morning Paper

SEPTEMBER 19, 2019

Their dataset has about 7B edges… Meanwhile, AnalyticDB is Alibaba’s real-time OLAP RDBMS handling 10PB of data (in excess of 100 trillion rows!). Hyper Dimension Shuffle describes how Microsoft improved the cost of data shuffling, one of the most costly operations, in their petabyte-scale internal big data analytics platform, SCOPE. We’ve been covering papers from VLDB 2019 for the last three weeks, and next week it will be time to mix things up again.

Blockchain 66

Blockchain Azure Database Hardware 66

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. We’ve seen similar high marshalling overheads in big data systems too.)

Cache 112

Cache Latency Lambda Servers 112

Abhishek Tiwari

DECEMBER 22, 2017

Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse.

Storage 52

Storage Retail Cloud Big Data 52

Rigor

SEPTEMBER 11, 2019

His most recent book, Hacking Web Performance , walks the reader through improving performance from initial load and data transfer to resource loading and the overall user experience. Would you like to be a part of Big Data and this incredible project?

Performance 95

Performance Education Google Website 95

All Things Distributed

NOVEMBER 5, 2015

This region will provide even lower latency and strong data sovereignty to local users. Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific.

Retail 89

Retail Entertainment IoT AWS 89

All Things Distributed

DECEMBER 13, 2016

The council has deployed IoT Weather Stations in Schools across the City and is using the sensor information collated in a Data Lake to gain insights on whether the weather or pollution plays a part in learning outcomes.

AWS 85

AWS Artificial Intelligence IoT Cloud 85

All Things Distributed

SEPTEMBER 5, 2013

Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data.

Mobile 69

Mobile Database Big Data Latency 69

All Things Distributed

JUNE 26, 2016

Seamless ingestion of large volumes of sensed data. AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift. Advanced problem solving that connects big data with machine learning.

Healthcare 83

Healthcare Blockchain AWS Cloud 83

Smashing Magazine

JULY 29, 2019

Many of us are lucky enough to be on mobile plans which allow several gigabytes of data transfer per month. Failing that, we are usually able to connect to home or public WiFi networks that are on fast broadband connections and have effectively unlimited data. The Cost Of Mobile Data.

Cache 100

Cache Mobile Google Website 100

Brendan Gregg

APRIL 30, 2018

Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley. USENIX’s LISA conference is the premier event for topics in production system engineering.

DevOps 53

DevOps Network Best Practices Programming 53

All Things Distributed

OCTOBER 7, 2015

We live in a world where massive volumes of data are generated from websites, connected devices and mobile apps. However, the data infrastructure to collect, store and process data is geared toward developers (e.g., Big data challenges.

Cloud 84

Cloud Analytics AWS Logistics 84

KeyCDN

OCTOBER 2, 2018

It’s awesome for discovering how grid systems, CSS animation, Big Data, etc all play roles in real-world web design. It includes tutorials, links to data-visualization tools, design resources and articles that cite real-world business experiments.

Development 83

Development Website Design Education 83

All Things Distributed

DECEMBER 12, 2018

WirelessCar is also using a host of AWS security services to ensure that the data it collects for its customers is secure, and also that the services the company offers are reliable as well.

AWS 109

AWS Games Cloud Serverless 109

All Things Distributed

JANUARY 6, 2016

We believe that with the launch of the Seoul Region, AWS will enable many more enterprise customers in Korea to reduce the cost of their IT operations and innovate faster in critical new areas such as big data analysis, Internet of Things, and more.

Games 83

Games AWS Cloud Infrastructure 83

All Things Distributed

NOVEMBER 15, 2016

Previously, I wrote about Amazon QuickSight , a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost.

Analytics 84

Analytics Availability Media Social Media 84