Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 126

Speed Java AWS Virtualization 126

Brendan Gregg

JULY 2, 2021

biolatency Disk I/O latency histogram heat map. runqlat CPU scheduler latency heat map. Then, having discovered everything is C or Python, some rewrite it all in a different language. For a more recent example, I wrote cachestat(8) while on vacation in 2014 for use on the Netflix cloud, which was a mix of Linux 3.2

Latency 145

Latency Cache Energy Systems 145

Brendan Gregg

AUGUST 29, 2021

biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. Hit Ctrl-C to end. ^C C msecs : count distribution 0 -> 1 : 83 | | 2 -> 3 : 20 | | 4 -> 7 : 0 | | 8 -> 15 : 41 | | 16 -> 31 : 1620 | * | 32 -> 63 : 8139 | *| 64 -> 127 : 176 | | 128 -> 255 : 95 | | 256 -> 511 : 61 | | 512 -> 1023 : 93 | |. 41 33681 ^C.

Cache 105

Cache C++ AWS Latency 105

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you time time?

Speed 52

Speed Java AWS Virtualization 52

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 40

Speed Java AWS Virtualization 40

Brendan Gregg

AUGUST 29, 2021

100.00 ^C I'm looking at the r_await column in particular: the average wait time for reads in milliseconds. biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. Hit Ctrl-C to end. ^C Hit Ctrl-C to end. ^C avg-cpu: %user %nice %system %iowait %steal %idle 14.81 I think it's the workload. ## 4.

Cache 40

Cache C++ AWS Systems 40

Brendan Gregg

JULY 2, 2021

biolatency Disk I/O latency histogram heat map 5. runqlat CPU scheduler latency heat map 10. As an extreme example, I wrote cachestat(8) while on vacation in 2014 for use on the Netflix cloud, which was a mix of Linux 3.2 BPF didn't exist on those versions, so I used basic Ftrace capabilities that were available on Linux 3.2.

Open Source 40

Open Source Latency Cache Energy 40