All Things Distributed

AUGUST 18, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud.

Big Data 111

Big Data Analytics AWS Cloud 111

ScaleOut Software

JULY 19, 2021

Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention. The best they can usually do in real-time using general purpose tools is to filter and look for patterns of interest.

IoT 78

IoT Analytics Big Data Architecture 78

ScaleOut Software

NOVEMBER 24, 2020

Real-Time Digital Twins Can Add Important New Capabilities to Telematics Systems and Eliminate Scalability Bottlenecks. Every few seconds, the application servers collect batches of snapshots and write them to the database where they can be queried by dispatchers managing the fleet.

Analytics 52

Analytics Architecture Scalability Software Architecture 52

Percona

JANUARY 14, 2019

What if we use ClickHouse (which is a columnar analytical database) as our main datastore? Well, typically, an analytical database is not a replacement for a transactional or key/value datastore. However, ClickHouse is super efficient for timeseries and provides “sharding” out of the box (scalability beyond one node).

Database 40

Database Analytics Blockchain Healthcare 40

Highly Scalable

MARCH 1, 2012

NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. At the same time, NoSQL data modeling is not so well studied and lacks the systematic theory found in relational databases. The second one is database-managed indexes, at least in some implementations.

Database 279

Database Ecommerce Efficiency Engineering 279

All Things Distributed

JUNE 20, 2017

After the launch of the AWS APAC (Hong Kong) Region, there will be 19 Availability Zones in Asia Pacific for customers to build flexible, scalable, secure, and highly available applications. 9GAG has a small team of nine people, including three engineers to support the business, and uses AWS to service their global visitors.

AWS 146

AWS Logistics Cloud Social Media 146

The Morning Paper

SEPTEMBER 19, 2019

MongoDB is an important database, and this paper explains the tunable (per-operation) consistency models that MongoDB provides and how they are implemented under the covers. AliGraph covers Alibaba’s distributed graph engine supporting the development of new GNN applications. Industry papers. Tunable consistency in MongoDB.

Blockchain 46

Blockchain Hardware Google Analytics 46

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 83

Cache Latency Traffic Database 83

All Things Distributed

AUGUST 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications.

Servers 126

Servers Social Media AWS Website 126

All Things Distributed

JANUARY 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Flexibility is one of the key principles of Amazon Web Services - developers can select any programming language and software package, any operating system, any middleware and any database to build systems and applications that meet their requirements.

AWS 115

AWS Cloud Java Operating System 115

Brendan Gregg

APRIL 30, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley. LISA is a vendor-neutral event known for technical depth and rigor, and continues to attract an audience of seasoned professionals.

DevOps 43

DevOps Network Best Practices Programming 43

All Things Distributed

AUGUST 20, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Data is retrieved by scheduling a job, which typically completes within 3 to 5 hours. Amazon Glacier integrates seamlessly with other AWS services such as Amazon S3 and the different AWS Database services. All Things Distributed. Expanding the Cloud â??

Storage 122

Storage Cloud AWS Media 122

Brendan Gregg

APRIL 29, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. Join us for 3 days in Nashville at LISA'18. Post by Brendan Gregg and Rikki Endsley. LISA is a vendor-neutral event known for technical depth and rigor, and continues to attract an audience of seasoned professionals.

DevOps 40

DevOps Network Best Practices Programming 40

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. Science & Engineering. an engineering adventure to break the 1,000 mph barrier in a car. Job Openings in AWS - Senior Leader in Database Services.

AWS 67

AWS Cloud Benchmarking Storage 67

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. There is huge variety in exiting architectures and I am often impressed about the ingenuity of the engineers in how to best transform the application if "Lift & Shift" is not an option. Job Openings in AWS - Senior Leader in Database Services. Comments ().

AWS 75

AWS Storage Cloud Best Practices 75

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Cluster Computer Instances are similar to other Amazon EC2 instances but have been specifically engineered to provide high performance compute and networking. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications.

Cloud 118

Cloud AWS Automotive Latency 118

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. There is more than one Werner Vogels in this world and although I never get emails, snail mail or phones calls for any of my peers, I am sure they are somewhat frustrated if they type in our name in a search engine :-). All Things Distributed. Comments ().

Cloud 117

Cloud Internet Internet AWS 117

The Netflix TechBlog

FEBRUARY 16, 2021

Membership Engineering at Netflix is responsible for the plan and pricing configurations for every market worldwide. To solve the challenges mentioned above and meet our rapidly evolving business needs, we re-architected the legacy SKU catalog from the ground up and partnered with the Growth Engineering team to build a scalable SKU platform.

Mobile 209

Mobile Engineering Infrastructure Scalability 209

KeyCDN

OCTOBER 2, 2018

It’s awesome for discovering how grid systems, CSS animation, Big Data, etc all play roles in real-world web design. Subjects like version control, crowdfunding, database selection and code editor choices are essential to efficient modern workflows, and this is a good place to start learning about them. Visit website 12.

Development 75

Development Website Design Code 75

The Netflix TechBlog

MARCH 25, 2019

Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. You are about to make structural changes to the data and want to know who and what downstream to your service will be impacted.

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

All Things Distributed

APRIL 28, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Currently, each AWS Region contains multiple Availability Zones, which are distinct locations that are engineered to be insulated from failures in other Availability Zones. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications.

AWS 61

AWS Cloud Latency Storage 61

The Netflix TechBlog

OCTOBER 18, 2022

As Big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have increasingly become more important for our data scientists and the company. Another dimension of scalability to consider is the size of the workflow.

Java 202

Java Scalability Traffic Architecture 202

The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Latency 243

Latency Storage Big Data Tuning 243

All Things Distributed

OCTOBER 7, 2015

However, the data infrastructure to collect, store and process data is geared toward developers (e.g., In AWS’ quest to enable the best data storage options for engineers, we have built several innovative database solutions like Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift.

Cloud 137

Cloud Big Data AWS Analytics 137

The Netflix TechBlog

DECEMBER 21, 2020

Some of the optimizations are prerequisites for a high-performance data warehouse. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster. Both automatic (event-driven) as well as manual (ad-hoc) optimization.

Storage 203

Storage Latency Efficiency Data Engineering 203

All Things Distributed

APRIL 4, 2017

After the launch of the AWS EU (Stockholm) Region, there will be 13 Availability Zones in Europe for customers to build flexible, scalable, secure, and highly available applications. It will also give customers another region where they can store their data with the knowledge that it will not leave the EU unless they move it.

AWS 116

AWS Airlines Latency Games 116

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. And while many of our systems are based on the latest in computer science research, this often hasnt been sufficient: our architects and engineers have had to advance research in directions that no academic had yet taken. All Things Distributed. Comments ().

Technology 151

Technology Technology AWS Storage 151

Highly Scalable

MARCH 10, 2015

To a certain extent, such a high diversity of recommendation techniques is attributed to several implementation challenges like a sparsity of customer ratings, computational scalability, and lack of information on new items and customers. This approach is also known as the “poor man’s recommendation engine” due to its simplicity [RE03].

Retail 152

Retail C++ Analytics Metrics 152

All Things Distributed

MARCH 28, 2011

Werner Vogels weblog on building scalable and robust distributed systems. The scalability, reliability and durability requirements for Cloud Drive are very high which is why they decided to make use of the Amazon Simple Storage Service (S3) as the core component of their service. Driving down the cost of Big-Data analytics.

AWS 138

AWS Cloud Storage Internet 138

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. A third generation of APIs, however, left the graphics specifics interfaces behind and instead focused on exposing the pipeline as a generic highly parallel engine supporting task and data parallelism. Driving down the cost of Big-Data analytics.

AWS 136

AWS Latency Programming Architecture 136