The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

Dynatrace

SEPTEMBER 17, 2019

PostgreSQL & Elastic for data storage. AWS EKS for Integration and Production. When focusing on the LanguageController service we learn that it’s currently deployed in three pods across three EKS nodes across two AWS Availability Zones (AZ). 4 AWS EFS monitoring. NGINX as an API Gateway. REDIS for caching.

AWS 127

AWS Monitoring Ecommerce Lambda 127

Adrian Cockcroft

NOVEMBER 26, 2023

The Pantheon in Rome — Extremely sustainable architecture — photo by Adrian I wrote a medium post after AWS re:Invent 2022 summarizing the (lack of) news and all the talks related to Sustainability. This includes providing the efficient, resilient services AWS customers expect, while minimizing their environmental footprint.

AWS 52

AWS Energy IoT Best Practices 52

All Things Distributed

DECEMBER 8, 2016

Earlier this year, Amazon Web Services (AWS) announced it would launch a new AWS infrastructure region in Montreal, Quebec. The AWS Cloud now operates in 40 Availability Zones within 15 geographic regions around the world, with seven more Availability Zones and three more regions coming online in China, France, and the U.K.

AWS 155

AWS Cloud Lambda Innovation 155

Dynatrace

MAY 26, 2021

From May 17 to May 18, 2021, the Open-Source Engineering team at Dynatrace attended the virtual observability conference, o11yfest. The second focused on the OTel community, with more technical talks by representatives from companies like AWS, Elastic, and more. A recurring theme was Site Reliability Engineering. Dynatrace news.

AWS 205

AWS Strategy Storage Metrics 205

Percona

APRIL 20, 2023

In this blog post, you will learn: how to leverage FDE on Kubernetes with Percona Operator for PostgreSQL how to start using encrypted storage for already running cluster Prepare In most public clouds, block storage is not encrypted by default. The configuration of the storage class depends on your storage plugin.

Storage 119

Storage AWS Cloud Database 119

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

MAY 26, 2021

From May 17 to May 18, 2021, the Open-Source Engineering team at Dynatrace attended the virtual observability conference, o11yfest. The second focused on the OTel community, with more technical talks by representatives from companies like AWS, Elastic, and more. A recurring theme was Site Reliability Engineering. Dynatrace news.

AWS 147

AWS Strategy Storage Metrics 147

Percona

MARCH 7, 2023

One approach is to separate compute and storage to allow for independent scaling. It’s an open source alternative to AWS Aurora Postgres that utilizes a serverless architecture. It’s an open source alternative to AWS Aurora Postgres that utilizes a serverless architecture. So, what is Neon?

Serverless 110

Serverless Storage Open Source Architecture 110

Dotcom-Montior

OCTOBER 6, 2021

A site reliability engineer, or SRE, is a role that that encompasses aspects of both software engineering and operations/infrastructure. The term site reliability engineering first came into existence at Google in 2003 when a site reliability team was created. At that time, the team was made up of software engineers.

Engineering 67

Engineering DevOps Monitoring Google 67

Dynatrace

DECEMBER 15, 2022

3: Ingest AWS Fargate logs with Fluent Bit. In the case of AWS serverless container compute engine Fargate, for example, where OneAgent log collection is not supported, we recommend using Fluent Bit log forwarder. Let’s take this example of AWS Fargate. Ingest AWS Fargate logs with Fluent Bit. Format: json.

Strategy 174

Strategy AWS Open Source Transportation 174

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. Amazon Glacier integrates seamlessly with other AWS services such as Amazon S3 and the different AWS Database services.

Storage 122

Storage Cloud AWS Media 122

All Things Distributed

JUNE 26, 2016

In June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Examples of continuous sensing are found in the managed cloud platform built by Rachio on AWS IoT to enable the secure interaction of its connected devices with cloud applications/other devices. The opportunity to revolutionize.

AWS 90

AWS Cloud Healthcare Blockchain 90

Percona

DECEMBER 23, 2022

I wrote this post on MyRocks because I believe it is the most interesting new MySQL storage engine to have appeared over the last few years. The use case is the TPC-C benchmark but executed not on a high-end server but on a lower-spec virtual machine that is I/O limited like for example, with AWS EBS volumes. Conclusion.

Storage 55

Storage Cache C++ Virtualization 55

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on.

Performance 140

Performance Latency Hardware Storage 140

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 37

AWS Entertainment Open Source Benchmarking 37

Brendan Gregg

DECEMBER 31, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. Storage I/O. Dave Hahn (CORE team) gave an updated A Day in the Life of a Netflix Engineer III. Alex Maestretti (security) co-presented SecOps 2021 Today: Using AWS Services to Deliver SecOps.

Tuning 61

Tuning AWS Best Practices Network 61

Brendan Gregg

DECEMBER 30, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. Storage I/O. Dave Hahn (CORE team) gave an updated A Day in the Life of a Netflix Engineer III. Alex Maestretti (security) co-presented SecOps 2021 Today: Using AWS Services to Deliver SecOps.

Tuning 52

Tuning AWS Best Practices Network 52

Dynatrace

JANUARY 16, 2023

Most Kubernetes clusters in the cloud (73%) are built on top of managed distributions from the hyperscalers like AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). Cloud-hosted Kubernetes clusters are on par to overtake on-premises deployments in 2023.

Open Source 337

Open Source Java Operating System Programming 337

InfoQ

DECEMBER 29, 2023

Zendesk reduced its data storage costs by over 80% by migrating from DynamoDB to a tiered storage solution using MySQL and S3. The company considered different storage technologies and decided to combine the relational database and the object store to strike a balance between querybility and scalability while keeping the costs down.

Storage 133

Storage Scalability Database Technology 133

All Things Distributed

MAY 19, 2017

Soon after the launch of Amazon Relation Dabase Service (RDS) AWS customers were giving us feedback that they would love to migrate to RDS but what they would love even more was if we could also unshackle them from the high-cost, punitive licensing schemes that came with the proprietary databases.

Database 117

Database Design Cloud Storage 117

Dynatrace

OCTOBER 7, 2021

Amazon Web Services (AWS), offers a wide range of serverless solutions. To get a better understanding of AWS serverless, we’ll first explore the basics of serverless architectures, review AWS serverless offerings, and explore common use cases. AWS serverless offerings. Reliability.

Serverless 273

Serverless AWS Lambda Storage 273

Brendan Gregg

APRIL 14, 2022

I've worked with awesome colleagues not just in cloud engineering, but also in open connect, studio, DVD, NTech, talent, immigration, HR, PR/comms, legal, and most recently ANZ content. It was difficult and fascinating work. I've also created internal training materials and classes, apart from my books.

Java 145

Java Cloud Cache AWS 145

The Netflix TechBlog

JULY 30, 2019

and beyond By Anoop Panicker and Kishore Banala Conductor is a workflow orchestration engine developed and open-sourced by Netflix. Many of the Netflix Content and Studio Engineering services rely on Conductor for efficient processing of their business flows. The Netflix Media Database (NMDB) is one such example.

Lambda 189

Lambda Media Open Source Metrics 189

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

All Things Distributed

SEPTEMBER 25, 2012

Following the huge success of being able to provision a consistent, user-requested I/O rate for DynamoDB and Elastic Block Store (EBS), the AWS Database Services team has now released Provisioned IOPS, a new high performance storage option for the Amazon Relational Database Service (Amazon RDS). Provisioned IOPS storage in RDS.

Cloud 117

Cloud AWS Storage Database 117

All Things Distributed

JANUARY 19, 2011

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud. Additionally customers are not restricted to AWS services; they can mix-and-match services from other providers to best meet their needs. There are some excellent platforms running on AWS that do precisely this; Ruby on Rails developers have Heroku. Comments ().

AWS 115

AWS Cloud Java Operating System 115

Abhishek Tiwari

JULY 22, 2023

For instance, Sentinel policies can enforce rules such as "All AWS S3 buckets must have server-side encryption enabled" or "Only specific IAM roles can access certain resources." " Open Policy Agent (OPA) : OPA is an open-source policy as code framework that provides a general-purpose policy engine.

AWS 52

AWS Best Practices Infrastructure Traffic 52

Dynatrace

NOVEMBER 23, 2020

Nevertheless, there are related components and processes, for example, virtualization infrastructure and storage systems (see image below), that can lead to problems in your Kubernetes infrastructure. Configuring storage in Kubernetes is more complex than using a file system on your host. The Kubernetes experience. Conclusion.

Virtualization 159

Virtualization Infrastructure Monitoring Cloud 159

HammerDB

APRIL 19, 2023

As database performance is heavily influenced by the performance of storage, network, memory, and processors, we must understand the upper limit of these key components. For example, if you are buying the latest Amazon memory-optimized EC2 instance (R7iz), the AWS page ( [link] ) tells us the following: Up to 3.9 4.22 %usr 38.40

Performance 54

Performance Benchmarking Cache Storage 54

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on.

Performance 52

Performance Latency Hardware Storage 52

All Things Distributed

APRIL 28, 2010

Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region. Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. The new Singapore Region offers customers in APAC lower-latency access to AWS services. Comments ().

AWS 61

AWS Cloud Latency Storage 61

Percona

DECEMBER 11, 2023

A quick configuration change may do the trick in improving the performance of your AWS RDS for MySQL instance. A Dedicated Log Volume (DLV) is a specialized storage volume designed to house database transaction logs separately from the volume containing the database tables. Who can benefit from DLV? and later v13 versions, 14.7

AWS 96

AWS Benchmarking Performance Traffic 96

Percona Community

MAY 6, 2020

(Wed) Level: Intermediate Abstract At Yelp, Cassandra, our NoSQL database of choice, has been deployed on AWS compute (EC2) and AutoScaling Groups (ASG), backed by Block Storage (EBS). This deployment model has been quite robust over the years while presenting its own set of challenges.

AWS 40

AWS Storage Infrastructure Database 40

All Things Distributed

AUGUST 17, 2011

As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. Jekyll in written in Ruby and uses YAML for metadata management and uses the Liquid template engine to manipulate the content. Countdown to What is Next in AWS.

Servers 126

Servers Social Media AWS Website 126

The Netflix TechBlog

MAY 21, 2019

Netflix engineers run a series of tests and benchmarks to validate the device across multiple dimensions including compatibility of the device with the Netflix SDK, device performance, audio-video playback quality, license handling, encryption and security. We use Spinnaker to deploy instances and host the API containers on Titus ?—?which

Testing 163

Testing AWS Lambda Network 163

All Things Distributed

AUGUST 24, 2012

The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile.

Storage 108

Storage Hardware AWS Data Engineering 108