Brendan Gregg

DECEMBER 30, 2017

My last talk for 2017 was at AWS re:Invent, on "How Netflix Tunes EC2 Instances for Performance," an updated version of my [2014] talk. Our team looks after the BaseAMI, kernel tuning, OS performance tools and profilers, and self-service tools like Vector. Storage I/O. We help where we can. File System. Networking.

Tuning 52

Tuning AWS Best Practices Network 52

The Netflix TechBlog

OCTOBER 19, 2020

Our distributed tracing infrastructure is grouped into three sections: tracer library instrumentation, stream processing, and storage. We earned the trust of our engineers by developing empathy for their operational burden and by focusing on providing efficient tracer library integrations in runtime environments.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Percona

APRIL 14, 2023

Below, we outline some proactive steps for achieving cost efficiency and maintaining performant database environments amid a turbulent economy: 1. Consider alternative tools, systems, and services: Many cloud providers offer long-term storage, serverless options, or component options for specific needs, with vastly different pricing models.

Database 85

Database Serverless Tuning Cloud 85

Scalegrid

JUNE 22, 2020

ScaleGrid provides 30% more storage on average vs. DigitalOcean for MySQL at the same affordable price. As you can see above, ScaleGrid and DigitalOcean offer the same plan configurations across this plan size, apart from SSD where ScaleGrid provides over 20% more storage for the same price. MySQL Configuration Management & Tuning.

Database 217

Database Benchmarking Latency Performance 217

Percona

DECEMBER 27, 2023

Consequently, they might miss out on the benefits of integrating security into the SDLC, such as enhanced efficiency, speed, and quality in software delivery. Customers will increasingly prioritize AI efficiency and education to tackle legal and ethical concerns. It’s worry-free and doesn’t require human intervention.

Cloud 82

Cloud Open Source Strategy Best Practices 82

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

Dynatrace

AUGUST 28, 2020

Azure Data Lake Storage Gen1. Azure Data Factory is a hybrid data integration service that enables you to quickly and efficiently create automated data pipelines—without writing any code. We’ll release additional monitoring support for new services soon, so stay tuned for further updates. Azure Logic Apps. Azure Data Factory v1.

Azure 154

Azure Cloud Tuning Monitoring 154

Dynatrace

APRIL 26, 2022

Digital experience monitoring enables companies to respond to issues more efficiently in real time, and, through enrichment with the right business data, understand how end-user experience of their digital products significantly affects business key performance indicators (KPIs). Endpoint monitoring (EM). Endpoints can be physical (i.e.,

Monitoring 202

Monitoring Social Media IoT Metrics 202

Percona

JANUARY 19, 2023

In this blog post, we will review key topics to consider for managing large datasets more efficiently in MySQL. Redundant indexes: It is known that accessing rows by fetching an index is more efficient than through a table scan in most cases. It can help us to save costs on storage and backup times.

Open Source 91

Open Source Storage Database Big Data 91

Dynatrace

JUNE 8, 2021

Continuous improvement of services is the most efficient process for all teams that are looking to improve the performance of their applications by considering all layers of their architecture. For example, an athlete must be in tune with their body to know when something isn’t operating right which could impact their future performance.

Metrics 165

Metrics Tuning Programming Systems 165

Dynatrace

DECEMBER 5, 2023

This growth was spurred by mobile ecosystems with Android and iOS operating systems, where ARM has a unique advantage in energy efficiency while offering high performance. Energy efficiency and carbon footprint outshine x86 architectures The first clear benefit of ARM in the enterprise IT landscape is energy efficiency.

Efficiency 264

Efficiency Architecture Energy Monitoring 264

Adrian Cockcroft

NOVEMBER 26, 2023

SUS101: Sustainability innovation in AWS Global Infrastructure AWS is determined to make the cloud the cleanest and most energy-efficient way to run customers’ infrastructure and business. This includes providing the efficient, resilient services AWS customers expect, while minimizing their environmental footprint.

AWS 52

AWS Energy IoT Best Practices 52

Percona

AUGUST 17, 2023

PostgreSQL has powerful and advanced features, including asynchronous replication, full-text searches of the database, and native support for JSON-style storage, key-value storage, and XML. It has evolved steadily, however, during 25 years as an open source project. And because you’re covered, you’re also free.

Open Source 82

Open Source Healthcare Social Media Database 82

SQL Performance

MARCH 17, 2020

Back in 2014, I wrote an article called Performance Tuning the Whole Query Plan. This is very efficient, as confirmed by the elapsed time of 13ms (all of which is CPU time). The structure of a b-tree index makes finding the next highest value very efficient — there is no need to scan through the duplicates. Test Environment.

Servers 86

Servers Tuning Database Efficiency 86

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 91

Storage Engineering Database Hardware 91

Percona

APRIL 24, 2023

Indexes are generally considered to be the panacea when it comes to SQL performance tuning, and PostgreSQL supports different types of indexes catering to different use cases. I keep seeing many articles and talks on “tuning” discussing how creating new indexes speeds up SQL but rarely ones discussing removing them.

Tuning 129

Tuning Cache Storage Database 129

The Morning Paper

JUNE 13, 2019

” If we want an increasing number of applications to use machine learning, we must automate issues that affect ease-of-use, performance, and cost efficiency for users and providers… Despite significant research, this is missing right now. We’d like to pack models as efficiently as possible on the underlying infrastructure.

Hardware 58

Hardware Latency Serverless Energy 58

Dynatrace

APRIL 20, 2023

With more automated approaches to log monitoring and log analysis, however, organizations can gain visibility into their applications and infrastructure efficiently and with greater precision—even as cloud environments grow. ” A data warehouse, on the other hand, is an efficient and fast option for querying data.

Analytics 186

Analytics Infrastructure Storage Efficiency 186

Dynatrace

AUGUST 1, 2022

You quickly realize that it will take ages to fill up the overprovisioned database storage. Two days later, your database runs out of storage in the middle of the night. Therefore, you don’t know your current growth rate and can’t estimate the required storage for keeping the database up and running for the next month.

Storage 186

Storage Database Network Metrics 186

The Netflix TechBlog

NOVEMBER 6, 2023

To train these parameters as well as fine-tune the pretrained image-text model weights, we leverage in-house datasets that pair shots of varying durations with rich textual descriptions of their content.

Media 225

Media Social Media Tuning Efficiency 225

The Netflix TechBlog

SEPTEMBER 24, 2021

Figure 1: A Simplified Video Processing Pipeline With this architecture, chunk encoding is very efficient and processed in distributed cloud computing instances. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 237

Cloud Media Storage Cache 237

The Netflix TechBlog

OCTOBER 27, 2020

Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store. What is Bulldozer Bulldozer is a self-serve data platform that moves data efficiently from data warehouse tables to key-value stores in batches. Figure 1 shows how we use Bulldozer to move data at Netflix.

Latency 243

Latency Storage Big Data Tuning 243

Percona

OCTOBER 18, 2023

Look closely at your current infrastructure (hardware, storage, networks, etc.) This is where you will fine-tune authentication mechanisms, storage paths, security policies, and memory allocation settings to optimize them for your specific use case(s). Should I be bringing in external experts to help out?

Open Source 95

Open Source Database Hardware Strategy 95

The Netflix TechBlog

JUNE 13, 2023

They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly. Evaluation of migration completeness: To verify the completeness of the records, cold storage services are used to take periodic data dumps from the two data stores and compared for completeness.

Traffic 279

Traffic Metrics Systems Strategy 279

Dynatrace

SEPTEMBER 28, 2020

Before we dive into the technical implementation, let me explain the visual concept of this “Global Status Page”: Another requirement for this status page was that it has to be lightweight, with no data storage at all. Getting the problem status of all environments has to be efficient. js framework.

Storage 205

Storage Architecture Tuning Efficiency 205

Dynatrace

FEBRUARY 16, 2023

Grail combines the big-data storage of a data warehouse with the analytical flexibility of a data lake. Learn more about the announcements at Perform 2023 in the Perform 2023 Guide: Organizations mine efficiencies with automation, causal AI. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022.

Analytics 184

Analytics Innovation Metrics Database 184

Abhishek Tiwari

JULY 12, 2018

These nodes and edges require a good amount of compute and storage which is typically distributed across a large number servers either running in the cloud or your own data center. If tuned for performance, there is a good change reliability is compromised - and vice versa. In a nutshell, a data pipeline is a distributed system.

Latency 63

Latency Analytics Scalability Engineering 63

Percona

AUGUST 27, 2018

Dangerous , because if you set this to a thread when using connection pooling OR ProxySQL and multiplexing, you may end up assigning a limitation to queries that instead, you wanted to run efficiently. Then we need to see IF implementing the tuning will work or not. Another cool useless feature??? Will this work? Good MySQL everyone.

Tuning 46

Tuning Virtualization Testing Servers 46

Dynatrace

NOVEMBER 15, 2021

Log analytics also help identify ways to make infrastructure environments more predictable, efficient, and resilient. Log analysis can reveal potential bottlenecks and inefficient configurations so teams can fine-tune system performance. Together, they provide continuous value to the business. Increased collaboration.

Analytics 207

Analytics Monitoring Artificial Intelligence DevOps 207

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. systemctl stop tuned $ systemctl disable tuned Dirty ratio The dirty_ratio is the percentage of total system memory that can hold dirty pages.

Best Practices 52

Best Practices Design Tuning Database 52

Percona

AUGUST 5, 2023

Otherwise, the storage engine does a scatter-gather and queries ALL partitions in a UNION that is not concurrent. Various partition types, like RANGE, LIST, HASH, and KEY, are used for specific needs, from range-based data to custom criteria, to ensure efficient data handling and the optimization of queries.

Strategy 40

Strategy Storage Database Best Practices 40

O'Reilly

SEPTEMBER 19, 2023

It progressed from “raw compute and storage” to “reimplementing key services in push-button fashion” to “becoming the backbone of AI work”—all under the umbrella of “renting time and storage on someone else’s computers.” ” (It will be easier to fit in the overhead storage.)

Hardware 99

Hardware Storage Big Data Blockchain 99

Percona

JULY 2, 2023

Having a backup strategy in place that takes regular backups and has secure storage is essential to protect the database in an enterprise-grade environment to ensure its availability in the event of failures or disasters. WALs in PostgreSQL are similar to transaction log files in the InnoDB storage engine for MySQL.

Strategy 52

Strategy Database Storage Open Source 52

The Netflix TechBlog

NOVEMBER 2, 2021

Video quality has matured in Cosmos and we are invested in making VQS more flexible and efficient. Stay tuned for more details on these algorithmic innovations. VQS currently represents the largest workload fueled by the Cosmos platform. Acknowledgments This work was made possible with the help of many stunning Netflix colleagues.

Media 171

Media Innovation Metrics Latency 171

All Things Distributed

NOVEMBER 28, 2012

Also, because similar data are stored sequentially, Amazon Redshift can compress data efficiently, which further reduces the amount of IO it needs to perform to return results. They contain large amounts of locally attached storage on multiple spindles and are connected by a minimally oversubscribed 10 Gigabit Ethernet network.

Cloud 164

Cloud Storage Architecture Scalability 164

Scalegrid

JANUARY 8, 2019

Simply put, in a MySQL semisynchronous replication configuration, the master commits transactions to the storage engine only after receiving acknowledgement from at least one of the slaves. Another configuration we can use to increase the efficiency of parallel execution on the slaves is to tune binlog_group_commit_sync_delay on the master.

Availability 113

Availability Tuning Speed Network 113

All Things Distributed

MARCH 7, 2013

Indexed Storage costs : We are lowering the price of indexed storage by 75%. Virginia) Region, the price of data storage will drop from $1 per GB per month to $0.25. DynamoDB runs on a fleet of SSD-backed storage servers that are specifically designed to support DynamoDB. s prices by 70%. For example, in our US East (N.

Ecommerce 139

Ecommerce Storage Scalability Hardware 139