Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Ford, et al., “TCP

Performance 140

Performance Latency Hardware Storage 140

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

The Netflix TechBlog

MAY 5, 2021

Netflix Drive relies on a data store that will be the persistent storage layer for assets, and a metadata store which will provide a relevant mapping from the file system hierarchy to the data store entities. Finally, once the encoded copy is prepared, this copy can be persisted by Netflix Drive to a persistent storage tier in the cloud.

Media 174

Media Storage Architecture Cloud 174

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). PC, smartphone, server) or virtual (virtual machines, cloud gateways). Endpoints can be physical (i.e.,

Monitoring 209

Monitoring Social Media IoT Metrics 209

All Things Distributed

AUGUST 20, 2015

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. Enter graph databases.

Database 134

Database Logistics Availability Social Media 134

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

IO River

NOVEMBER 2, 2023

Yes, these might change based on localization settings, but in terms of storage, it’s still a completely baked cookie that the server can choose to serve or not serve.Whenever a user clicks on an episode, these bits remain unchanged. Developed by none other than Apple, HLS is vital in minimizing delays, known in the tech world as latency!When

Architecture 52

Architecture Performance Internet Internet 52

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

ScaleOut Software

JULY 14, 2021

The Microsoft Azure IoT ecosystem offers a rich set of capabilities for processing IoT telemetry, from its arrival in the cloud through its storage in databases and data lakes. Acting as a switchboard for incoming and outgoing messages, Azure IoT Hub forms the core of these capabilities.

IoT 52

IoT Azure Analytics Serverless 52

IO River

NOVEMBER 2, 2023

Yes, these might change based on localization settings, but in terms of storage, it’s still a completely baked cookie that the server can choose to serve or not serve.Whenever a user clicks on an episode, these bits remain unchanged. Developed by none other than Apple, HLS is vital in minimizing delays, known in the tech world as latency!When

Architecture 52

Architecture Performance Internet Internet 52

All Things Distributed

MARCH 11, 2016

Marvin Theimer, Amazon Distinguished Engineer, once jokingly said that the evolution of Amazon S3 could best be described as starting off as a single engine Cessna plane, but over time the plane was upgraded to a 737, then a group of 747s, all the way to the large fleet of Airbus 380s that it is now. No gatekeepers.

AWS 144

AWS Hardware Retail Virtualization 144

ScaleOut Software

JULY 19, 2021

Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention. The best they can usually do in real-time using general purpose tools is to filter and look for patterns of interest.

IoT 78

IoT Analytics Big Data Architecture 78

HammerDB

MAY 5, 2019

However in the Skylake microarchitecture (you can see a list of CPUs here ) the PAUSE instruction changed and in the documentation it says “the latency of the PAUSE instruction in prior generation microarchitectures is about 10 cycles, whereas in Skylake microarchitecture it has been extended to as many as 140 cycles.”

Testing 48

Testing Tuning Latency Storage 48

Percona

APRIL 17, 2023

Operating System (OS) settings Swappiness Swappiness is a Linux kernel setting that influences the behavior of the Virtual Memory manager when it needs to allocate a swap, ranging from 0-100. The CFQ works well for many general use cases but lacks latency guarantees. Without further ado, let’s start with the OS settings.

Best Practices 52

Best Practices Design Tuning Database 52

Percona

APRIL 7, 2023

Here’s how the same test performed when running Percona Distribution for PostgreSQL 14 on these same servers: Queries: reads Queries: writes Queries: other Queries: total Transactions Latency (95th) MySQL (A) 1584986 1645000 245322 3475308 122277 20137.61 We have long been surfing the virtualization wave (to keep it broad).

Tuning 134

Tuning Cache Servers Benchmarking 134

All Things Distributed

JANUARY 18, 2012

Amazon DynamoDB offers low, predictable latencies at any scale. In response, we began to develop a collection of storage and database technologies to address the demanding scalability and reliability requirements of the Amazon.com ecommerce platform. Customers can typically achieve average service-side in the single-digit milliseconds.

Scalability 167

Scalability Database Ecommerce Latency 167

All Things Distributed

OCTOBER 2, 2017

VPC Endpoints give you the ability to control whether network traffic between your application and DynamoDB traverses the public Internet or stays within your virtual private cloud. Performant – DynamoDB consistently delivers single-digit millisecond latencies even as your traffic volume increases.

Internet 128

Internet Internet AWS Performance 128

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. Containerized data workloads running on Kubernetes offer several advantages over traditional virtual machine/bare metal based data workloads including but not limited to. faster access to external storage and data locality (I/O, bandwidth).

Big Data 63

Big Data Storage Benchmarking Hardware 63

Abhishek Tiwari

JULY 12, 2018

These data pipelines can process data at petabytes scale and to some extent, their success can be attributed to an army of engineers devoted to build and maintain internal data pipelines. Not everyone is operating at Netflix or Spotify scale data engineering function. In a nutshell, a data pipeline is a distributed system.

Latency 63

Latency Analytics Scalability Engineering 63

VoltDB

JULY 10, 2018

In these use cases, data processing usually has less than a 5 milliseconds latency budget. With the stored procedures framework, and the in-memory data storage engine, VoltDB drives most complex business logic in the lowest latency in a scalable manner, even in a virtualized environment like VMs and containers.

Architecture 40

Architecture Scalability Ecommerce Latency 40

VoltDB

JULY 10, 2018

In these use cases, data processing usually has less than a 5 milliseconds latency budget. With the stored procedures framework, and the in-memory data storage engine, VoltDB drives most complex business logic in the lowest latency in a scalable manner, even in a virtualized environment like VMs and containers.

Architecture 40

Architecture Scalability Ecommerce Latency 40

HammerDB

DECEMBER 5, 2020

From an engineering perspective this may be all that you need to know. For TPC-C this meant enough available spindles to reduce I/O latency and for TPC-H enough bandwidth for data throughput. As the home warehouse is chosen at random however you want enough to ensure an even spread of Virtual Users across the warehouses.

C++ 40

C++ Benchmarking Virtualization Open Source 40

HammerDB

OCTOBER 12, 2018

As is also the case this limitation is at the database level (especially the storage engine) rather than the hardware level. InnoDB is the storage engine that will deliver the best OLTP throughput and should be chosen for this test. . maximum transition latency: Cannot determine or is not supported. .

Best Practices 40

Best Practices Scalability Performance C++ 40

Alex Russell

APRIL 29, 2021

Apple forces developers of competing browsers to use their engine for all browsers on iOS , restricting their ability to deliver a better version of the web platform. They are, pound for pound, some of the best engine developers globally and genuinely want good things for the web. So is speedy resolution and agreement.

Media 145

Media Games Education Engineering 145

HammerDB

NOVEMBER 5, 2018

Additionally for the log disk component it is latency for an individual write that is crucial rather than the total I/O bandwidth. The first example shows a data load, the second a TPC-C based workload with 5 virtual users and the 2nd example with 10 virtual users. innodb_flush_log_at_trx_commit=0. PostgreSQL.

Benchmarking 40

Benchmarking Database C++ Virtualization 40

Brendan Gregg

SEPTEMBER 5, 2017

This includes many great engineers who I'm sure will excel in whatever they choose to work on next. Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. Many new tools can now be written, and the main toolkit we're working on is [bcc].

Virtualization 76

Virtualization AWS Engineering Hardware 76

SQL Server According to Bob

JANUARY 11, 2020

​​ The storage ​​ space that is required ​​ for the sparse file is only that of the actual bytes written to the file and not the ​​ maximum ​​ file ​​ size.

Servers 40

Servers Cache Database Media 40

Smashing Magazine

JANUARY 11, 2021

Performance isn’t just a technical concern: it affects everything from accessibility to usability to search engine optimization, and when baking it into the workflow, design decisions have to be informed by their performance implications. Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms.

Performance 145

Performance Cache Media Metrics 145