Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. As (C) looked like a kernel rebuild, I started with (D) and (E). ## 5. I also rewrote this in C and called gettimeofday(2) directly: $ cat gettimeofdaybench.c. What on Earth is Ubuntu doing that results in 30% higher CPU time!?

Speed 126

Speed Java AWS Virtualization 126

Brendan Gregg

JULY 2, 2021

biolatency Disk I/O latency histogram heat map. runqlat CPU scheduler latency heat map. Then, having discovered everything is C or Python, some rewrite it all in a different language. execsnoop New processes (via exec(2)) table. opensnoop Files opened table. ext4slower Slow filesystem I/O table. BPF up and running!

Latency 145

Latency Cache Energy Systems 145

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. As (C) looked like a kernel rebuild, I started with (D) and (E). ## 6. I also rewrote this in C and called gettimeofday(2) directly: $ cat gettimeofdaybench.c What on Earth is Ubuntu doing that results in 30% higher CPU time!?

Speed 40

Speed Java AWS Virtualization 40

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. As (C) looked like a kernel rebuild, I started with (D) and (E). I also rewrote this in C and called gettimeofday(2) directly: $ cat gettimeofdaybench.c What on Earth is Ubuntu doing that results in 30% higher CPU time!?

Speed 52

Speed Java AWS Virtualization 52

Brendan Gregg

AUGUST 8, 2017

TABLE OF EXP(-T/C) FOR T = 5 SEC. EXPFF: EXP 0.920043902 ;C = 1 MIN EXP 0.983471344 ;C = 5 MIN EXP 0.994459811 ;C = 15 MIN. Some people have found values that seem to work for their systems and workloads: they know that when load goes over X, application latency is high and customers start complaining.

Latency 111

Latency Metrics C++ Systems 111

Brendan Gregg

MAY 8, 2021

Update: Some modern drives after 2015 are sealed with [helium].) From these outputs I try to determine if the problem is: - **The workload**: High-latency disk I/O is commonly caused by the workload applied. Rotational disks have extra latency from head seeks for random I/O, and spin ups from the idle state. Hit Ctrl-C to end.

Performance 40

Performance Latency Speed Systems 40

Brendan Gregg

JULY 2, 2021

biolatency Disk I/O latency histogram heat map 5. runqlat CPU scheduler latency heat map 10. Then, having discovered everything is C or Python, some rewrite it all in a different language. execsnoop New processes (via exec(2)) table 2. opensnoop Files opened table 3. ext4slower Slow filesystem I/O table 4. BPF up and running!

Open Source 40

Open Source Latency Cache Energy 40