The Morning Paper

NOVEMBER 15, 2020

So before matching, the IDS/IPS has to reconstruct a TCP bytestream in the face of packet fragmentation, loss, and out-of-order delivery – a process known as reassembly. This makes the whole system latency sensitive. The Full Matcher runs on the CPU, polling a ring buffer populated by the DMA Engine.

Servers 128

Servers Hardware Latency Design 128

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. It became clear that real-time query processing and in-stream processing is the immediate need in many practical applications. Strict fault-tolerance is a principal requirement for the engine.

Big Data 154

Big Data Processing Lambda Database 154

Dynatrace

SEPTEMBER 30, 2021

Cloud migration is the process of transferring some or all your data, software, and operations to a cloud-based computing environment that offers unlimited scale and high availability. A key step in digital transformation is migrating from traditional on-prem IT processes to adopting cloud services. What is cloud migration?

Cloud 169

Cloud Traffic Best Practices Strategy 169

Percona

MARCH 23, 2023

Hardware Memory The amount of RAM to be provisioned for database servers can vary greatly depending on the size of the database and the specific requirements of the company. As datasets continue to grow in size, the amount of RAM required to store and process these datasets also increases. Refer to innodb_redo_log_capacity below.

Performance 123

Performance Hardware Tuning Storage 123

The Morning Paper

NOVEMBER 10, 2019

You need a lot of software engineers and the willingness to rewrite a lot of software to entertain that idea. Here are the bombshell paragraphs: Our datacenter applications seek ever more CPU-efficient and lower-latency communication, which Pony Express delivers. The desire for CPU efficiency and lower latencies is easy to understand.

Network 92

Network Transportation Latency Entertainment 92

CSS - Tricks

JULY 25, 2019

Estimated Input Latency. This process works by obtaining mobile 75th and 95th percentiles from HTTP Archive , then applying a log normal function. Estimated Input Latency. It’s time to come to terms that your customers aren’t using the same powerful hardware as you. They are: Time to Interactive ( TTI ). Speed Index.

Google 71

Google Engineering Speed Mobile 71

Dotcom-Montior

NOVEMBER 24, 2021

Within an organization, the responsibility of monitoring these large distributed systems typically falls on site reliability engineering (SRE) teams. Each process has its own independent memory that it works with. Software and hardware components are autonomous and execute tasks concurrently. Types of Distributed Systems.

Systems 74

Systems Monitoring Hardware Network 74

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

Percona

SEPTEMBER 1, 2023

This results in expedited query execution, reduced resource utilization, and more efficient exploitation of the available hardware resources. A finely tuned database processes queries more efficiently, leading to swifter results. MySQL relies heavily on the availability of hardware resources to perform at its best.

Tuning 52

Tuning Database Performance Hardware 52

The Morning Paper

JANUARY 30, 2020

Edge servers are the middle ground – more compute power than a mobile device, but with latency of just a few ms. physics engine that simulates 3D cubes falling from the air. These use their regression models to estimate processing time (which will depend on the hardware available, current load, etc.).

Mobile 104

Mobile Cloud Latency Games 104

John McCalpin

APRIL 25, 2023

This metric is interesting because we don’t always have the luxury of parallelizing every application we run, and our operating systems almost always process each call (e.g., The NEC Vector Engine processors provide a demonstration of very high single-core bandwidth. buffer copies for filesystem access) with a single thread.

Benchmarking 97

Benchmarking Cache Latency Tuning 97

Percona

MAY 2, 2023

At the same time that I see database engineers relying on the tool, sites such as StackOverflow are banning ChatGPT. Let’s look at some questions I did that a MySQL DBA usually needs to answer in an interview process. As the question comes from time to time, is my profession at risk of being extinct due to AI?

Social Media 119

Social Media Database Servers Cache 119

All Things Distributed

MARCH 11, 2016

Marvin Theimer, Amazon Distinguished Engineer, once jokingly said that the evolution of Amazon S3 could best be described as starting off as a single engine Cessna plane, but over time the plane was upgraded to a 737, then a group of 747s, all the way to the large fleet of Airbus 380s that it is now. Primitives not frameworks.

AWS 144

AWS Hardware Retail Virtualization 144

Dotcom-Montior

DECEMBER 8, 2021

In the world of a site reliability engineer (SRE) , failure is not only an option, but also expected. In a different article, we talked about chaos engineering and how SRE teams proactively seek out and test for failures to prevent the worst from happening. Systems, web applications, servers, devices, etc., It is an unavoidable fact.

Social Media 90

Social Media Monitoring Latency DevOps 90

Brendan Gregg

AUGUST 18, 2019

For example, iostat(1), or a monitoring agent, may tell you your average disk latency, but not the distribution of this latency. For smaller environments, it can be of more use helping eliminate latency outliers. bpftrace uses BPF (Berkeley Packet Filter), an in-kernel execution engine that processes a virtual instruction set.

Latency 68

Latency C++ Cache Programming 68

CSS Wizardry

OCTOBER 27, 2018

Simply knowing the different forms of performance testing that we have available to us, and where they sit in the product development process, makes it much easier for businesses to adopt a performance strategy and keep on top of things. Who: Engineers. Who: Engineers, Product Owners. When: During development.

Performance Testing 172

Performance Testing Testing Performance Strategy 172

The Morning Paper

JULY 4, 2019

This is a fascinating paper from members of Netflix’s Resilience Engineering team describing their chaos engineering initiatives: automated controlled experiments designed to verify hypotheses about how the system should behave under gray failure conditions, and to probe for and flush out any weaknesses. The Chaos automation platform.

Latency 77

Latency Engineering Metrics Traffic 77

Percona

JUNE 22, 2023

Another related variable, innodb_buffer_pool_instances, determines the number of buffer pool instances for the InnoDB storage engine, which can improve the performance of multi-core systems by reducing contention on the buffer pool latch.

Performance 102

Performance Monitoring Traffic Database 102

Abhishek Tiwari

JULY 12, 2018

These data pipelines can process data at petabytes scale and to some extent, their success can be attributed to an army of engineers devoted to build and maintain internal data pipelines. Not everyone is operating at Netflix or Spotify scale data engineering function. Depending on frameworks, data processing units (a.k.a

Latency 63

Latency Analytics Scalability Engineering 63

The Morning Paper

MARCH 3, 2020

It takes you through the thinking processes and engineering practices behind the design of a key part of the control plane for AWS Elastic Block Storage (EBS): the Physalia database that stores configuration information. Engineering decisions involve making lots of trade-offs. Millions of tiny databases , Brooker et al.,

Database 106

Database AWS Network Design 106

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. What about short-lived processes, like a service restarting in a loop? As a Xen guest, this profile was gathered using perf(1) and the kernel's software cpu-clock soft interrupts, not the hardware NMI. in total.

Speed 126

Speed Java AWS Virtualization 126

Alex Russell

APRIL 29, 2021

Apple forces developers of competing browsers to use their engine for all browsers on iOS , restricting their ability to deliver a better version of the web platform. They are, pound for pound, some of the best engine developers globally and genuinely want good things for the web. So is speedy resolution and agreement.

Media 145

Media Games Education Engineering 145

Smashing Magazine

AUGUST 3, 2021

Fortunately, we as engineers can avoid, or at least mitigate the impact of breakages in the web apps we build. Engineers need visibility on the root cause behind a degraded experience. Even errors not surfaced to end-users (secondary errors) must propagate to engineers. JavaScript is less forgiving. Resilience is extrinsic.

Network 145

Network Engineering Monitoring Availability 145

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Percona

JULY 1, 2023

Understanding DBaaS DBaaS cloud services allow users to use databases without configuring physical hardware and infrastructure or installing software. What we should really compare is the MySQL and Aurora database engines provided by Amazon RDS. Considering a Fully Managed DBaaS Offering For Your Business?

AWS 52

AWS Database Serverless Storage 52

The Morning Paper

MAY 21, 2019

This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions. Thus we’re fundamentally trading (de)-compression latency at access time for the ability to pack more data in memory.

Software 78

Software Software Google Hardware 78

Alex Russell

JANUARY 30, 2024

HTML, CSS, images, and fonts can all be parsed and run at near wire speeds on low-end hardware, but JavaScript is at least three times more expensive, byte-for-byte. Engineering is the discipline of designing solutions under specific constraints. That's where the good news ends.

Performance 90

Performance Network Mobile Speed 90

The Morning Paper

MAY 28, 2019

The goal is to produce a low-energy hardware classifier for embedded applications doing local processing of sensor data. The resulting system can integrate seamlessly into a scikit-learn based development process, and dramatically reduces the total energy usage required for classification with very low latency.

Energy 52

Energy Hardware Efficiency Architecture 52

Abhishek Tiwari

DECEMBER 27, 2017

In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures. It offers reliability and performance of a data warehouse, real-time and low-latency characteristics of a streaming system, and scale and cost-efficiency of a data lake.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. For instance, scatter/gather pattern can be used to implement a MapReduce like batch processing architecture on top of Kubernetes. Custom schedulers.

Big Data 63

Big Data Storage Benchmarking Hardware 63

All Things Distributed

NOVEMBER 15, 2010

From financial processing and traditional oil & gas exploration HPC applications to integrating complex 3D graphics into online and mobile applications, the applications of GPU processing appear to be limitless. For example, the most fundamental abstraction trade-off has always been latency versus throughput.

AWS 136

AWS Latency Programming Architecture 136

Adrian Cockcroft

NOVEMBER 11, 2019

In addition to the common financial calculation of risk as the product of probability and severity, engineering risk includes detectability. Staff should be familiar with recovery processes and the behavior of the system when it’s working hard to mitigate failures. Book: Engineering a Safer World by Nancy G.

Latency 52

Latency Engineering Systems Hardware 52

O'Reilly Software

APRIL 4, 2018

We live in the era of the connected experience, where our daily interactions with the world can be digitized, collected, processed, and analyzed to generate valuable insights. Accumulating all this data to process overnight is not an option anymore. Do we need to process each record individually? Process tolerance.

Architecture 80

Architecture Scalability Google Operating System 80

Smashing Magazine

AUGUST 22, 2021

Because we are dealing with network protocols here, we will mainly look at network aspects, of which two are most important: latency and bandwidth. Latency can be roughly defined as the time it takes to send a packet from point A (say, the client) to point B (the server). Two-way latency is often called round-trip time (RTT).

Performance 106

Performance Network Latency Servers 106

Alex Russell

MARCH 6, 2021

Advances in browser content processing. 3] As we'll see below, CPUs are not improving fast enough to cope with frontend engineers' rosy resource assumptions. Hardware Past As Performance Prologue. To update our global baseline from 2017, we want to update our priors on a few dimensions: The evolved device landscape.

Performance 124

Performance Network Cache Metrics 124

Percona

APRIL 17, 2023

The CFQ works well for many general use cases but lacks latency guarantees. The deadline excels at latency-sensitive use cases ( like databases ), and noop is closer to no schedule at all. 25.84 - Total 21.04 zone_reclaim_mode In some OS versions, the vm.zone_reclaim_mode is enabled. 25.84 - Total 21.04

Best Practices 52

Best Practices Design Tuning Database 52

Adrian Cockcroft

NOVEMBER 11, 2019

In addition to the common financial calculation of risk as the product of probability and severity, engineering risk includes detectability. Staff should be familiar with recovery processes and the behavior of the system when it’s working hard to mitigate failures. Book: Engineering a Safer World by Nancy G.

Latency 53

Latency Engineering Systems Hardware 53