Brendan Gregg

DECEMBER 30, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. Our team looks after the BaseAMI, kernel tuning, OS performance tools and profilers, and self-service tools like Vector. Storage I/O. We help where we can. File System. Networking.

Tuning 52

Tuning AWS Best Practices Network 52

Percona

APRIL 14, 2023

Consider alternative tools, systems, and services: Many cloud providers offer long-term storage, serverless options, or component options for specific needs, with vastly different pricing models. Deactivate or eliminate elements you’re not using. With proper database management, it’s possible to cut your bills in half.

Database 81

Database Serverless Tuning Cloud 81

Percona

DECEMBER 27, 2023

The field of Platform Engineering has witnessed significant advancements, as evidenced by the publication of the CNCF platform whitepaper and the introduction of a dedicated Platform Engineering day at the upcoming KubeCon event. They also remove toil and allow engineers to focus on application development vs platform engineering work.

Cloud 77

Cloud Open Source Strategy Best Practices 77

HammerDB

APRIL 19, 2023

As database performance is heavily influenced by the performance of storage, network, memory, and processors, we must understand the upper limit of these key components. For storage, FIO is generally used. Storage: The system has a SATA drive for the operating system and one NVMe (Intel SSD D7-P5510 (3.84 Database: MySQL 8.0.31

Performance 54

Performance Benchmarking Cache Storage 54

Dynatrace

JUNE 8, 2021

Observability is a topic at the top of mind for all architects, Site Reliability Engineers (SREs), and more – each wanting to use observability to proactively detect issues and guarantee the best experience and availability to users. Stay tuned for more tips from our coach Jean Louis Lormeau. Ability to define automatic baselining.

Metrics 162

Metrics Tuning Programming Systems 162

Dynatrace

NOVEMBER 17, 2020

With these release candidate APIs available, instrumentation for web frameworks, storage clients, and much more can be built. We sat together with Armin Ruech and Daniel Dyla, software engineers at Dynatrace and leaders within the OpenTelemetry community, to hear about their involvement with the second most active CNCF project.

Open Source 151

Open Source Software Engineering Government Java 151

Percona

AUGUST 7, 2023

InnoDB Undo Log The undo log, also known as the rollback segment, is a crucial part of the InnoDB storage engine. Properly managing flushing and purging operations is crucial for system performance and tuning.

Database 95

Database Tuning Storage Open Source 95

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

Adrian Cockcroft

JUNE 21, 2022

So in 1988 I joined the Sun sales force and became a Systems Engineer (nowadays we would say Solutions Architect). I was mostly coding in C, tuning FORTRAN, and when I needed to do a lot of data analysis of benchmark results used the S-PLUS statistics language, that is the predecessor to R.

Tuning 52

Tuning Benchmarking Engineering C++ 52

Dynatrace

APRIL 26, 2022

EM gives visibility into the user device and performance from the endpoint to provide information on CPU, memory, operating systems, storage, security, networks, and whether software is up to date. Endpoint monitoring (EM). Endpoints can be physical (i.e., PC, smartphone, server) or virtual (virtual machines, cloud gateways).

Monitoring 196

Monitoring Social Media IoT Metrics 196

Percona

JANUARY 19, 2023

It can help us to save costs on storage and backup times. While MySQL can handle large data sets, it is always recommended to keep only the used data in the databases, as this will make data access more efficient, and also will help to save costs on storage and backups. It supports native sharding.

Open Source 85

Open Source Storage Database Big Data 85

Percona

DECEMBER 20, 2018

Linux OS Tuning for MySQL Database Performance. In this post we will review the most important Linux settings to adjust for performance tuning and optimization of a MySQL database server. We’ll note how some of the Linux parameter settings used OS tuning may vary according to different system types: physical, virtual or cloud.

Database 40

Database Tuning Performance AWS 40

Percona

AUGUST 17, 2023

In another survey, using multiple metrics for overall popularity, PostgreSQL ranks as the fourth most popular database management system ( DB-Engines, March 2023 ). The widespread use and popularity have come on like gangbusters in recent years, but PostgreSQL wasn’t always the cool kid. Free from vendor lock-in.

Open Source 77

Open Source Healthcare Social Media Database 77

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 85

Storage Engineering Database Hardware 85

HammerDB

MAY 5, 2019

Also it is interesting to note that the impact of this change was not observed in other databases or other MySQL storage engines such as MyRocks the only noticeable impact with HammerDB workloads occurred in MySQL with InnoDB where in the source code ut0ut.cc In the recent MySQL 8.0.16 The OS was: Ubuntu 18.04.1 23-generic x86_64).

Testing 48

Testing Tuning Latency Storage 48

SQL Server According to Bob

SEPTEMBER 9, 2019

I recall when we were tuning the sp_reset_connection (which releases the database lock and acquires it again) command we tested rates in excess of 250,000/sec to ensure the partitioned database lock scaled: [link]. Bob Dorr – Principal Software Engineer SQL Server.

Servers 103

Servers Database Scalability Tuning 103

The Morning Paper

MARCH 10, 2020

This benchmark can synthetically generate more precise key-value queries that represent the reads and writes of key-value stores to the underlying storage system. The paper examines three different uses of RocksDB at Facebook: UDB , the underlying storage engine for the MySQL databases storing the social graph data.

Benchmarking 59

Benchmarking Storage Cache Open Source 59

Adrian Cockcroft

NOVEMBER 26, 2023

Explore practical best practices for different customization scenarios including training models from scratch, fine-tuning with additional data using full or parameter-efficient techniques, retrieval augmented generation, and prompt engineering. AWS speakers: Wafae Bakkali, Isha Dua

AWS 52

AWS Energy IoT Best Practices 52

Brendan Gregg

APRIL 30, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. LISA originally stood for "Large Installation System Administration," where "large" meant systems with more than a gigabyte of storage, or with more than 100 users. Join us for 3 days in Nashville at LISA'18.

DevOps 43

DevOps Network Best Practices Programming 43

Brendan Gregg

APRIL 29, 2018

USENIX’s LISA conference is the premier event for topics in production system engineering. LISA originally stood for "Large Installation System Administration," where "large" meant systems with more than a gigabyte of storage, or with more than 100 users. Join us for 3 days in Nashville at LISA'18.

DevOps 40

DevOps Network Best Practices Programming 40

Percona

MARCH 23, 2023

Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Cloud Different cloud providers offer a range of instance types and sizes, each with varying amounts of CPU, memory, and storage. If you see concurrency issues, you can tune this variable.

Performance 124

Performance Hardware Tuning Storage 124

Dynatrace

APRIL 20, 2023

Causal AI—which brings AI-enabled actionable insights to IT operations—and a data lakehouse, such as Dynatrace Grail , can help break down silos among ITOps, DevSecOps, site reliability engineering, and business analytics teams. “It’s quite a big scale,” said an engineer at the financial services group.

Analytics 181

Analytics Infrastructure Storage Efficiency 181

Abhishek Tiwari

JULY 12, 2018

These data pipelines can process data at petabytes scale and to some extent, their success can be attributed to an army of engineers devoted to build and maintain internal data pipelines. Not everyone is operating at Netflix or Spotify scale data engineering function. In a nutshell, a data pipeline is a distributed system.

Latency 63

Latency Analytics Scalability Engineering 63

The Netflix TechBlog

NOVEMBER 6, 2023

Building in-video search To build such a visual search engine, we needed a machine learning system that can understand visual elements. To train these parameters as well as fine-tune the pretrained image-text model weights, we leverage in-house datasets that pair shots of varying durations with rich textual descriptions of their content.

Media 225

Media Social Media Tuning Efficiency 225

The Morning Paper

JUNE 13, 2019

Making queries to an inference engine has many of the same throughput, latency, and cost considerations as making queries to a datastore, and more and more applications are coming to depend on such queries. In “ A case for managed and model-less inference serving” Yadwadkar et al.

Hardware 58

Hardware Latency Serverless Energy 58

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. Then the KV DAL handles writing to the appropriate underlying storage engines depending on latency, availability, cost, and durability requirements.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

MARCH 14, 2023

User provides a sample image to find other similar images Prior engineering work Approach #1: on-demand batch processing Our first approach to surface these innovations was a tool to trigger these algorithms on-demand and on a per-show basis. Dawn Chenette , Design Lead This approach had several benefits for product engineering.

Media 291

Media Innovation Energy Architecture 291

The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 237

Cloud Media Storage Cache 237

Scalegrid

JULY 29, 2019

Managing a database is hard, as it needs continuous updating, tuning, and monitoring to ensure the performance of your website. Next, select the VM size, ranging from Micro at 10GB of storage up to X4XLarge at 700GB of storage, and then your MySQL version and storage engine. Stay tuned! Replication.

Database 166

Database Cloud Website Azure 166

Dynatrace

FEBRUARY 9, 2022

Cloud vendors such as Amazon Web Services (AWS), Microsoft, and Google provide a wide spectrum of serverless services for compute and event-driven workloads, databases, storage, messaging, and other purposes. Engineers often choose best-of-breed services from multiple sources to create a single application. Stay tuned for updates.

Serverless 181

Serverless Azure Lambda AWS 181

Percona

APRIL 7, 2023

If we were to select the most important MySQL setting, if we were given a freshly installed MySQL or Percona Server for MySQL and could only tune a single MySQL variable, which one would it be? To be fair, that is also true with PostgreSQL; it hasn’t been tuned either, and it, too, can also perform much better.

Tuning 135

Tuning Cache Servers Benchmarking 135

The Netflix TechBlog

MAY 14, 2021

Prodicle is one of the many applications that is at the exciting intersection of connecting the world of content productions to Netflix Studio Engineering. The watermarking functionality, at the start, was a simple offering with various Google Drive integrations for storage and links. Conductor and one of the new offerings Cosmos.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

JUNE 13, 2023

They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly. Evaluation of migration completeness: To verify the completeness of the records, cold storage services are used to take periodic data dumps from the two data stores and compared for completeness.

Traffic 279

Traffic Metrics Systems Strategy 279

Dynatrace

JULY 6, 2022

Organizations everywhere are adopting site reliability engineering (SRE) to cope with the growing complexity of hybrid and cloud-native environments. Virtualization has revolutionized system administration by making it possible for software to manage systems, storage, and networks. Design, implement, and tune effective SLOs.

Best Practices 181

Best Practices Open Source Tuning Infrastructure 181

Dynatrace

DECEMBER 9, 2020

To address potentially high numbers of requests during online shopping events like Singles Day or Black Friday, it’s crucial that this online shop have a memory storage strategy that allows for speed, scaling, and resilience of all microservices, especially the shopping cart service. What’s next?

Java 226

Java Traffic Strategy Architecture 226

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. systemctl stop tuned $ systemctl disable tuned Dirty ratio The dirty_ratio is the percentage of total system memory that can hold dirty pages.

Best Practices 52

Best Practices Design Tuning Database 52

Percona

OCTOBER 17, 2023

Many changes and new features are brought to the system, and as part of keeping in tune with the changes and how they can impact us, we go through the changes to better understand them. Dynamic WiredTiger tickets As a brief recap, WiredTiger tickets in MongoDB serve as the concurrency control mechanism within the WiredTiger storage engine.

Metrics 98

Metrics C++ Storage Database 98