Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

The Netflix TechBlog

NOVEMBER 9, 2022

While we understand it’s virtually impossible to achieve a linear increase in throughput as the number of vCPUs grow, a near-linear increase is attainable. What’s worse, average latency degraded by more than 50%, with both CPU and latency patterns becoming more “choppy.”

Hardware 363

Hardware Cache Performance Latency 363

Abhishek Tiwari

JULY 12, 2018

Building data pipelines can offer strategic advantages to the business. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects. In this post, we will discuss why you should avoid building data pipelines in first place.

Latency 63

Latency Analytics Scalability Engineering 63

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. better cluster resource utilization.

Big Data 63

Big Data Storage Benchmarking Hardware 63

HammerDB

OCTOBER 23, 2023

The data shows a scripted automated workload running a number of back to back tests each time with an increasing number of virtual users. Gbits/sec We also test both local port and socket connections and found that a local port provided the best comparative data for both MySQL and MariaDB. On MySQL, we saw a 1.5X

Benchmarking 62

Benchmarking Database Network C++ 62

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. I also wrote about these topics in detail for my recent [Systems Performance 2nd Edition] book.

Performance 52

Performance Latency Hardware Storage 52

VoltDB

JUNE 1, 2017

It was – like the hypothetical movie I describe above – more than a little bit odd, as you could leave a session discussing ever more abstract layers of virtualization and walk into one where they emphasized the critical importance of pinning a network interface to a specific VM for optimal performance.

Hardware 56

Hardware Latency Virtualization Cloud 56

Brendan Gregg

AUGUST 18, 2019

For example, iostat(1), or a monitoring agent, may tell you your average disk latency, but not the distribution of this latency. For smaller environments, it can be of more use helping eliminate latency outliers. bpftrace uses BPF (Berkeley Packet Filter), an in-kernel execution engine that processes a virtual instruction set.

Latency 68

Latency C++ Cache Programming 68

VoltDB

JUNE 1, 2017

It was – like the hypothetical movie I describe above – more than a little bit odd, as you could leave a session discussing ever more abstract layers of virtualization and walk into one where they emphasized the critical importance of pinning a network interface to a specific VM for optimal performance.

Hardware 40

Hardware Latency Virtualization Cloud 40

All Things Distributed

JANUARY 23, 2012

The Amazon Virtual Private Cloud extends on-premises compute with all the power of AWS, making it elastic, scalable and highly reliable. Data written to these volumes is maintained on your on-premises storage hardware while being asynchronously backed up to AWS, where it is stored in Amazon S3 in the form of Amazon EBS snapshots.

Storage 115

Storage AWS Virtualization Cloud 115

HammerDB

DECEMBER 5, 2020

A full understanding of why this is important requires some knowledge of the evolution of database hardware and software. For TPC-C this meant enough available spindles to reduce I/O latency and for TPC-H enough bandwidth for data throughput. This was both expensive and time consuming to configure.

C++ 40

C++ Benchmarking Virtualization Open Source 40

Dynatrace

JANUARY 26, 2021

Traditional computing models rely on virtual or physical machines, where each instance includes a complete operating system, CPU cycles, and memory. VMware commercialized the idea of virtual machines, and cloud providers embraced the same concept with services like Amazon EC2, Google Compute, and Azure virtual machines.

Serverless 243

Serverless Efficiency Lambda Azure 243

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Percona

APRIL 17, 2023

We’ll also go over some best practices for MongoDB security as well as MongoDB data modeling. Operating System (OS) settings Swappiness Swappiness is a Linux kernel setting that influences the behavior of the Virtual Memory manager when it needs to allocate a swap, ranging from 0-100. Two other schedulers are deadline and noop.

Best Practices 52

Best Practices Design Tuning Database 52

All Things Distributed

MARCH 11, 2016

This is a given, whether you are using the highest quality hardware or lowest cost components. When customers left the constraining, old world of IT hardware and datacenters behind, they started to develop systems with new and interesting usage patterns that no one had ever seen before. Primitives not frameworks. your resource usage.

AWS 144

AWS Hardware Retail Virtualization 144

The Morning Paper

DECEMBER 10, 2019

This is a companion paper to the " persistent problem " piece that we looked at earlier this week, going a little deeper into the object pointer representation choices and the mapping of a virtual object space into physical address spaces. Both abstractions must be implemented in a way that is efficient using existing hardware.

Hardware 52

Hardware Virtualization Operating System Programming 52

The Morning Paper

DECEMBER 8, 2019

Byte-addressable non-volatile memory,) NVM will fundamentally change the way hardware interacts, the way operating systems are designed, and the way applications operate on data. Therefore any programming abstraction must be low latency and the kernel needs to be kept off the path of persistent data access as much as possible.

Hardware 56

Hardware Programming Media Storage 56

SQL Performance

APRIL 27, 2020

Many of the newer features we have in SQL Server were initially launched in Azure SQL Database, including (but not limited to) Always Encrypted, Dynamic Data Masking, Row Level Security, and Query Store. Gen 5 is the primary hardware option now for most regions since Gen 4 is aging out. DTU Pricing Tier. GB per vCore. GB per vCore.

Azure 55

Azure Database Serverless Hardware 55

Dotcom-Montior

NOVEMBER 24, 2021

These systems can include physical servers, containers, virtual machines, or even a device, or node, that connects and communicates with the network. A three-tier system is a software application architecture that consists of a presentation layer, application layer, and data, or core, layer. Three-Tier. Multi-Tier. Concurrency.

Systems 74

Systems Monitoring Hardware Network 74

The Morning Paper

APRIL 14, 2019

The paper sets out what we can do in software given today’s hardware, and along the way also highlights areas where cooperation from hardware will be needed in the future. Microarchitectural channels. There are also stateless interconnects including buses and on-chip networks. Threat scenarios.

Hardware 44

Hardware Cache Latency Speed 44

HammerDB

OCTOBER 18, 2019

Nevertheless in this blog sometimes we do publish performance data to highlight best practices or potential configuration pitfalls and although we’ve mentioned this one before it is worth dedicating an entire post to it as this issue seems to appear numerous times running database workloads on Linux. (Yes

Benchmarking 63

Benchmarking Performance Hardware C++ 63

Alex Russell

APRIL 29, 2021

This post mines publicly available data on the pace of compatibility fixes and feature additions to assess the claim. This data set is derived by walking a subset of web platform features exposed to JavaScript and, as such, may not capture the full richness of a particular browser. Steve & Tim's Close-up Magic. Converging Views.

Media 145

Media Games Education Engineering 145

HammerDB

OCTOBER 12, 2018

As is also the case this limitation is at the database level (especially the storage engine) rather than the hardware level. driver: intel_pstate CPUs which run at the same hardware frequency: 0 . maximum transition latency: Cannot determine or is not supported. . hardware limits: 1000 MHz - 3.80 large-pages.

Best Practices 40

Best Practices Scalability Performance C++ 40

All Things Distributed

JANUARY 18, 2012

s web-based applications often encounter database scaling challenges when faced with growth in users, traffic, and data. Their tables can also grow without limits as their users store increasing amounts of data. Amazon DynamoDB offers low, predictable latencies at any scale. History of NoSQL at Amazon â?? The growth of Amazonâ??s

Scalability 167

Scalability Database Ecommerce Latency 167

SQL Server According to Bob

JANUARY 11, 2020

To ensure data integrity for our customers who are running SQL Server, it is important that I/O environments provide appropriate functionality. Atomicity A transaction must be an atomic unit of work; either all of its data modifications are performed or none of them are performed.

Servers 40

Servers Cache Media Hardware 40

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. What happens if processes really do try to populate all that virtual memory? Many new tools can now be written, and the main toolkit we're working on is [bcc]. Hit Ctrl-C to end. ^C

Virtualization 76

Virtualization AWS Engineering Hardware 76

Dynatrace

AUGUST 5, 2019

Before we talk about migrations, we must talk about how we gather the data to make better migration decisions – this is where our OneAgent differentiates itself from other approaches! There is no code or configuration change necessary to capture data and detect existing services. This is LIVE data queryable through an API!

Cloud 31

Cloud Traffic Database Network 31

SQL Server According to Bob

JANUARY 11, 2020

If ​​ it is ​​ not, ​​ the ​​ data ​​ is considered ​​ to be ​​ changed ​​ or ​​ damaged. In ​​ other words, ​​ a formula is applied ​​ to a given set of bytes.

Servers 40

Servers Cache Database Media 40

Smashing Magazine

JANUARY 11, 2021

It will help you build up a company-tailored case study with real data. Furthermore, using data from case studies and experiments published on WPO Stats will help increase sensitivity for business about why performance matters, and what impact it has on user experience and business metrics. Large preview ). Large preview ).

Performance 145

Performance Cache Media Metrics 145