Brendan Gregg

DECEMBER 30, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. Fortunately, it was videoed for those who missed it. What I was most excited for was the launch of a new EC2 hypervisor, which I referred to in the video as the "c5 hypervisor".

Tuning 52

Tuning AWS Best Practices Network 52

High Scalability

MAY 26, 2020

Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years. Essilen Research Free Video Series on Hiring. Advertise your job here!

Education 100

Education AWS Software Engineering Scalability 100

Dynatrace

OCTOBER 28, 2021

In 2009, the purveyor of online videos migrated to AWS cloud infrastructure to deliver its entertainment to a growing audience. Greater innovation and collaboration from development and SRE teams means better software that meets new customer demands quickly with efficiency and high performance. The history of chaos engineering.

Engineering 221

Engineering Entertainment Cloud Infrastructure 221

The Netflix TechBlog

OCTOBER 19, 2020

Investigating a video streaming failure consists of inspecting all aspects of a member account. We earned the trust of our engineers by developing empathy for their operational burden and by focusing on providing efficient tracer library integrations in runtime environments. which is difficult when troubleshooting distributed systems.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

The Netflix TechBlog

MAY 21, 2019

Netflix engineers run a series of tests and benchmarks to validate the device across multiple dimensions including compatibility of the device with the Netflix SDK, device performance, audio-video playback quality, license handling, encryption and security. Detect a regression in a test case. It can be used in Python notebooks ?

Testing 163

Testing AWS Lambda Network 163

All Things Distributed

MARCH 11, 2015

AWS is enabling innovations in areas such as healthcare, automotive, life sciences, retail, media, energy, robotics that it is mind boggling and humbling. For example many of the Internet of Things innovations that we have seen come to life in the past years on AWS all have a significant analytics components to it.

Analytics 135

Analytics Cloud Healthcare Retail 135

All Things Distributed

FEBRUARY 4, 2014

AWS has built a reputation over the years for the breadth and depth of our services and the pace of our innovation with 280 features released in 2013. These are the people working behind the scenes helping customers fully leverage all of AWS’s capabilities when running their infrastructure on AWS.

AWS 132

AWS Energy Innovation Best Practices 132

Adrian Cockcroft

MAY 6, 2023

Then they tried to scale it to cope with high traffic and discovered that some of the state transitions in their step functions were too frequent, and they had some overly chatty calls between AWS lambda functions and S3. This is only one of many microservices that make up the Prime Video application. Finally, what were they building?

Serverless 145

Serverless Lambda Best Practices Traffic 145

Dynatrace

JANUARY 26, 2021

This allows teams to sidestep much of the cost and time associated with managing hardware, platforms, and operating systems on-premises, while also gaining the flexibility to scale rapidly and efficiently. REST APIs, authentication, databases, email, and video processing all have a home on serverless platforms. Pay Per Use.

Serverless 242

Serverless Efficiency Lambda Azure 242

Brendan Gregg

FEBRUARY 28, 2023

The video is now on [YouTube]: The slides are [online] and as a [PDF]: first prev next last / permalink/zoom In Q&A I was asked about CXL (compute express link) which was fortunate as I had planned to cover it and then forgot, so the question let me talk about it (although Q&A is missing from the video).

Performance 110

Performance Latency Cache Virtualization 110

The Netflix TechBlog

OCTOBER 18, 2019

In this talk, Kinjal used the example of the LinkedIn Feed, to demonstrate how they use bandit algorithms to solve for the optimal parameter selection problem efficiently. He concluded by stressing the efficiency their teams had achieved by doing online parameter exploration instead of the much slower human-in-the-loop manual explorations.

Infrastructure 126

Infrastructure Metrics Architecture Efficiency 126

The Netflix TechBlog

OCTOBER 18, 2019

In this talk, Kinjal used the example of the LinkedIn Feed, to demonstrate how they use bandit algorithms to solve for the optimal parameter selection problem efficiently. He concluded by stressing the efficiency their teams had achieved by doing online parameter exploration instead of the much slower human-in-the-loop manual explorations.

Infrastructure 100

Infrastructure Metrics Architecture Efficiency 100

All Things Distributed

AUGUST 18, 2011

Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. However, this cannot be done without efficient, scalable data analytics. The posting on the AWS developer blog also has some more background. Driving Storage Costs Down for AWS Customers.

Big Data 111

Big Data Analytics AWS Cloud 111

Brendan Gregg

FEBRUARY 28, 2023

The video is now on YouTube : The slides are online and as a PDF : first prev next last / permalink/zoom In Q&A I was asked about CXL (compute express link) which was fortunate as I had planned to cover it and then forgot, so the question let me talk about it (although Q&A is missing from the video).

Performance 52

Performance Latency Cache Virtualization 52

VoltDB

JUNE 1, 2017

Some of these will serve large urban areas and will be sensitive to efficiency. They want to compete with AWS or Azure, and want to use OpenStack as a foundation for a commercial offering. Watch the Video. See David’s full commentary, from the show, in this video. The “Public Private Cloud” folks. Where VoltDB fits.

Hardware 56

Hardware Latency Virtualization Cloud 56

Brendan Gregg

NOVEMBER 29, 2017

At Netflix, we've been using these technologies as they've been made available for instance types in the AWS EC2 cloud. The latest AWS hypervisor, Nitro, uses everything to provide a new hardware-assisted hypervisor that is easy to use and has near bare-metal performance. AWS called this [enhanced networking]. The first was c3.

Virtualization 111

Virtualization AWS Hardware Storage 111

The Netflix TechBlog

SEPTEMBER 24, 2021

After content ingestion, inspection and encoding, the packaging step encapsulates encoded video and audio in codec agnostic container formats and provides features such as audio video synchronization, random access and DRM protection. Packaging has always been an important step in media processing.

Cloud 237

Cloud Media Storage Cache 237

The Netflix TechBlog

APRIL 29, 2019

We use Python through the full content lifecycle, from deciding which content to fund all the way to operating the CDN that serves the final video to 148 million members. takes place in Amazon Web Services (AWS), whereas everything that happens afterwards (i.e., video streaming) takes place in the Open Connect network.

Open Source 279

Open Source Network Infrastructure Big Data 279

Brendan Gregg

NOVEMBER 28, 2017

At Netflix, we've been using these technologies as they've been made available for instance types in the AWS EC2 cloud. The latest AWS hypervisor, Nitro, uses everything to provide a new hardware-assisted hypervisor that is easy to use and has near bare-metal performance. AWS called this [enhanced networking]. The first was c3.

Virtualization 40

Virtualization AWS Hardware Storage 40

Smashing Magazine

SEPTEMBER 3, 2021

The good news is that, as developers, we can do an awful lot about it. Text, we are repeatedly told, is going out of fashion; all people are interested in now are videos and interactive experiences. According to Android Authority in “ How Much Data Does YouTube Actually Use? ”, a 360p YouTube video weighs about 5 to 7.5

Website 135

Website Energy Internet Internet 135

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3. Figure 1 shows how we use Bulldozer to move data at Netflix.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

APRIL 26, 2022

Figure 1: Netflix ML Architecture Fact: A fact is data about our members or videos. An example of data about members is the video they had watched or added to their My List. An example of video data is video metadata, like the length of a video. Time is a critical component of Axion?—?When

Storage 187

Storage Design Scalability Latency 187

High Scalability

JUNE 29, 2018

kellabyte : LOL at racking up an AWS bill of $140,000 in 4 hours of compute time. kellabyte : Recently I got to work on a project that really stressed Amazon AWS scalability. We spun up a cluster of 100,000 AWS instances multiple times. Google applies bulk discounts after the fact, AWS makes you ask for them ahead of time.

Scalability 131

Scalability Internet Internet Wireless 131

The Netflix TechBlog

MARCH 10, 2023

By Burak Bacioglu , Meenakshi Jindal Asset Management at Netflix At Netflix, all of our digital media assets (images, videos, text, etc.) Most applications in content and studio engineering at Netflix deal with assets; such as videos, images, text, etc. are stored in secure storage layers.

Strategy 258

Strategy Cache Analytics Storage 258

The Morning Paper

MARCH 1, 2020

Firecracker is the virtual machine monitor (VMM) that powers AWS Lambda and AWS Fargate, and has been used in production at AWS since 2018. The first version of AWS Lambda was built using Linux containers. All of the existing approaches we just examined involve trade-offs that AWS didn’t want to make.

Serverless 101

Serverless Virtualization Lambda AWS 101

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. AWS helps its customers do this too. Build security in from the ground up.

AWS 144

AWS Hardware Retail Virtualization 144

O'Reilly

MAY 19, 2020

Live online training, videos, books, certification prep, and more, from O’Reilly and our partner publishers. AWS is far and away the cloud leader, followed by Azure (at more than half of share) and Google Cloud. But most Azure and GCP users also use AWS; the reverse isn’t necessarily true. Amazon and AWS Ascendant.

Cloud 141

Cloud Serverless AWS Architecture 141

O'Reilly

NOVEMBER 28, 2023

How will AI adopters react when the cost of renting infrastructure from AWS, Microsoft, or Google rises? Several respondents also mentioned working with video: analyzing video data streams, video analytics, and generating or editing videos. But they may back off on AI development.

Open Source 108

Open Source Artificial Intelligence Google Infrastructure 108

Smashing Magazine

AUGUST 20, 2021

Fast-forward 30 years, and website technology has changed significantly — we have images, stylesheets, JavaScript, streaming video, AJAX, animation, WebSockets, WebGL, rounded corners in CSS — the list goes on. Twitch developer documentation hosted on AWS, edited on CloudCannon. The site will load in a snap. Large preview ).

Ecommerce 139

Ecommerce Website Government Internet 139

Smashing Magazine

AUGUST 3, 2023

This could be an image, a block of text, or even an embedded video. This is why the async and deferred attributes are crucial, as they ensure an efficient, seamless web browsing experience. Addressing Many Pages With Rare Visits Adopting the AWS CloudFront CDN significantly improved our website’s performance.

Performance 74

Performance Cache Traffic Metrics 74

Testsigma

JULY 22, 2021

Some of the first projects I tested on were video games and mobile navigation apps. And if I could just talk about these users and some of the barriers they face, or what it’s like to go through a digital experience as they do, we’ll learn an awful lot! How did you land in testing, Erin? I began testing as a hobby many years ago.

Software 97

Software Software Testing Energy 97

All Things Distributed

DECEMBER 12, 2013

items that share the same hash value in their primary key): e.g. if there is a DynamoDB table with PlayerName as the hash key and GameStartTime as the range key, you can use local secondary indexes to run efficient queries on other attributes like “Score.” Efficient Queries. What are Global Secondary Indexes?

Games 130

Games Scalability Database Retail 130

All Things Distributed

OCTOBER 12, 2016

Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency. ElastiCache for Redis Multi-AZ capability is built to handle any failover case for Redis Cluster with robustness and efficiency. Building upon Redis.

Scalability 130

Scalability Analytics Cache AWS 130

All Things Distributed

AUGUST 20, 2015

Developers need efficient methods to store, traverse, and query these relationships. Social media apps navigate relationships between friends, photos, videos, pages, and followers. Like many AWS innovations, the desire to build a solution for a scalable graph database came from Amazon’s retail business. Bringing it all together.

Database 134

Database Logistics Availability Social Media 134

Percona

APRIL 17, 2023

MongoDB security: Monitor MongoDB performance regularly As with any database, it’s essential to monitor MongoDB performance to ensure it’s running with optimal efficiency and effectiveness. This is because the setting has no use in modern PCIe/NVMe devices.

Best Practices 52

Best Practices Design Tuning Database 52

KeyCDN

JUNE 25, 2020

If you play video games that use the left-hand keys on the keyboard to move your character around, this concept is probably familiar to you. Install Vim using Git: According to Vim themselves, install Vim via Git is the simplest and most efficient method.

Games 125

Games Open Source Availability C++ 125

Brendan Gregg

SEPTEMBER 5, 2017

. ## Observability Here's the big picture of performance observability tools on Linux, from my [Linux performance] page, where I also have diagrams for other tool types, as well as videos and slides of prior Linux performance talks: I also have a USE Method: Linux Performance Checklist , as a different way to navigate and apply the tools.

Virtualization 76

Virtualization AWS Engineering Hardware 76