Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Infrastructure Storage 109

Dynatrace

OCTOBER 4, 2022

Teams have introduced workarounds to reduce storage costs. Additionally, efforts such as lowered data retention times, two-tiered storage systems, shaky index management, sampled data, and data pipelines reduce the overall amount of stored data. Dynatrace discovers logs automatically at scale.

Analytics 235

Analytics Artificial Intelligence Storage Serverless 235

Scalegrid

MARCH 28, 2024

With these goals in mind, two in-memory data stores, Redis and Memcached, have emerged as the top contenders. This article will explore how they handle data storage and scalability, perform in different scenarios, and, most importantly, how these factors influence your choice. Data transfer technology. 3d render.

Cache 130

Cache Storage Architecture Scalability 130

The Netflix TechBlog

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This technique facilitates validation on multiple fronts.

Traffic 339

Traffic Latency Tuning Systems 339

Dynatrace

JANUARY 16, 2023

As Kubernetes adoption increases and it continues to advance technologically, Kubernetes has emerged as the “operating system” of the cloud. Kubernetes is emerging as the “operating system” of the cloud. Kubernetes is emerging as the “operating system” of the cloud. Kubernetes moved to the cloud in 2022.

Open Source 343

Open Source Java Operating System Programming 343

Dynatrace

MARCH 24, 2023

Containers enable developers to package microservices or applications with the libraries, configuration files, and dependencies needed to run on any infrastructure, regardless of the target system environment. This means organizations are increasingly using Kubernetes not just for running applications, but also as an operating system.

Infrastructure 186

Infrastructure Open Source Operating System Cloud 186

Scalegrid

MARCH 14, 2024

Understanding Hybrid Cloud Strategy A hybrid cloud merges the capabilities of public and private clouds into a singular, coherent system. This combination allows for the fluid movement of data and applications across different environments, facilitating shared workloads seamlessly. The tool must be compatible with your current systems.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

All Things Distributed

AUGUST 18, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud.

Big Data 112

Big Data Analytics AWS Cloud 112

Dynatrace

DECEMBER 15, 2022

In fact, Gartner estimates that 80% of enterprises will shut down their on-premises data centers by 2025. This transition to public, private, and hybrid cloud is driving organizations to automate and virtualize IT operations to lower costs and optimize cloud processes and systems. So, what is ITOps? Why is IT operations important?

Artificial Intelligence 203

Artificial Intelligence DevOps Hardware Virtualization 203

All Things Distributed

AUGUST 20, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. All Things Distributed. Comments ().

Storage 122

Storage Cloud AWS Media 122

ScaleOut Software

JULY 19, 2021

How are we managing the torrent of telemetry that flows into analytics systems from these devices? Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention. The list goes on.

IoT 78

IoT Analytics Big Data Architecture 78

The Morning Paper

OCTOBER 26, 2020

As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, but there’s a lot more to it than that. These two narratives of reference architecture and ingestion/indexing system are interwoven throughout the paper.

Cloud 104

Cloud Big Data Latency Architecture 104

The Netflix TechBlog

OCTOBER 15, 2019

Another issue exists for the capture of schema changes, where some systems, like MySQL, don’t support transactional schema changes [1][2]. By their nature, they can only rely on the lowest common denominator of participating systems. That system quickly grew very complex and became difficult to maintain.

Transportation 192

Transportation Architecture Processing Storage 192

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

Tech News Gather

OCTOBER 7, 2023

With this data, you can implement intelligent call routing systems that ensure calls are directed to the right department or agent, reducing wait times and improving the overall customer experience. By optimizing your marketing and customer service based on call data, you can outperform competitors who rely solely on digital analytics.

Strategy 52

Strategy Big Data Scalability Games 52

All Things Distributed

JUNE 26, 2016

A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world – India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more.

AWS 90

AWS Cloud Healthcare Blockchain 90

Testsigma

MAY 7, 2020

IBM Big Data and Analytics Hub website cited a case study, where a US insurance company was estimating 15% of their testing efforts to be just test data collection for the backend system and the frontend system. The test data management for the company had become a big problem and had to be solved.

Testing 60

Testing Storage Database Processing 60

All Things Distributed

JULY 7, 2011

Werner Vogels weblog on building scalable and robust distributed systems. AWS Import/Export transfers data off of storage devices using Amazons high-speed internal network and bypassing the Internet. Amazon Import/Export is an important tool for customers to accelerate moving large amounts of data into the AWS storage systems.

AWS 98

AWS Cloud Storage Internet 98

ScaleOut Software

APRIL 3, 2020

What’s missing is a flexible, fast, and easy-to-use software system that can be quickly adapted to track these assets in real time and provide immediate answers for logistics managers. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins.

Logistics 52

Logistics Analytics Scalability Cloud 52

ScaleOut Software

APRIL 3, 2020

What’s missing is a flexible, fast, and easy-to-use software system that can be quickly adapted to track these assets in real time and provide immediate answers for logistics managers. Within seconds, the software performs aggregate analysis of this data for all real-time digital twins.

Logistics 52

Logistics Analytics Scalability Storage 52

All Things Distributed

AUGUST 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. Driving Storage Costs Down for AWS Customers. All Things Distributed.

Servers 126

Servers Social Media AWS Storage 126

All Things Distributed

JULY 16, 2012

Werner Vogels weblog on building scalable and robust distributed systems. There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.

AWS 85

AWS Media Big Data Storage 85

All Things Distributed

AUGUST 22, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Systems that make extensive use of caching almost all report a significant reduction in the cost of their database tier. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. All Things Distributed.

Cloud 117

Cloud Cache AWS Storage 117

All Things Distributed

AUGUST 16, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Government and Big Data. One particular early use case for AWS GovCloud (US) will be massive data processing and analytics. AWS GovCloud (US) will be used by several of these agencies to help them with their Bigger-than-Big-Data needs.

AWS 113

AWS Government Big Data Cloud 113

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers. The ideal candidate will be someone who has built and ran large scale distributed systems and/or databases.

AWS 77

AWS Database Storage Scalability 77

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics. All Things Distributed. Expanding the Cloud â?? Introducing the AWS South America (Sao Paulo) Region.

AWS 115

AWS Latency Storage Big Data 115

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. Why are developers using RInK systems as part of their design? Fetching too much data in a single query (i.e.,

Cache 79

Cache Latency Google Lambda 79

The Morning Paper

SEPTEMBER 19, 2019

Autoscaling tiered cloud storage in Anna. ” Crusher is a Google system for automatically discovering email templates (e.g. Hyper Dimension Shuffle describes how Microsoft improved the cost of data shuffling, one of the most costly operations, in their petabyte-scale internal big data analytics platform, SCOPE.

Blockchain 46

Blockchain Hardware Google Analytics 46

Testsigma

FEBRUARY 12, 2021

The Internet of Things is generally referred to as IoT which encompasses computers, cars, houses or some other technological system related. of companies invest over US$ 50 million in initiatives such as Artificial Intelligence (AI) and Big Data in 2020, up from 39.7% IoT Test Automation. billion in 2016. billion by 2025, up 32.6

Artificial Intelligence 92

Artificial Intelligence Software Software IoT 92

All Things Distributed

MARCH 2, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics. All Things Distributed. Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region.

AWS 114

AWS Cloud Games Latency 114

All Things Distributed

MARCH 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. It is likely that the Amazon Web Services will be used by many of the participants for their compute, storage, database and other cloud resource needs. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway.

AWS 96

AWS Internet Internet Storage 96

All Things Distributed

JANUARY 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Flexibility is one of the key principles of Amazon Web Services - developers can select any programming language and software package, any operating system, any middleware and any database to build systems and applications that meet their requirements.

AWS 115

AWS Cloud Java Operating System 115

All Things Distributed

JUNE 29, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics. All Things Distributed. Driving Bandwidth Cost Down for AWS Customers. Comments (). At werner.ly

AWS 106

AWS Retail Innovation Storage 106

All Things Distributed

FEBRUARY 23, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics. All Things Distributed. Free at Last - A Fully Self-Sustained Blog Running in Amazon S3. Comments ().

AWS 79

AWS Storage Big Data Servers 79

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. Naming is one of the fundamental concepts in Distributed Systems. By Werner Vogels on 05 December 2010 02:00 PM.

Cloud 117

Cloud Internet Internet AWS 117

Brendan Gregg

APRIL 30, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. We both have had long careers supporting system administration, and LISA has always felt like a homecoming, reuniting with old friends while welcoming newcomers. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley.

DevOps 43

DevOps Network Best Practices Programming 43

All Things Distributed

JANUARY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. DROAM - Dreaming about Cheap Data Roaming. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. Driving Storage Costs Down for AWS Customers. Driving down the cost of Big-Data analytics.

Wireless 109

Wireless AWS Internet Internet 109