Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 126

Speed Java AWS Virtualization 126

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. Hit Ctrl-C to end. ^C Hit Ctrl-C to end. ^C

Cache 105

Cache C++ AWS Latency 105

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you time time?

Speed 52

Speed Java AWS Virtualization 52

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 40

Speed Java AWS Virtualization 40

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 100.00 ^C I'm looking at the r_await column in particular: the average wait time for reads in milliseconds. . ## 2. 1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03

Cache 40

Cache C++ AWS Systems 40

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. Hit Ctrl-C to end. ^C C operation = 'read' usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 4479 | *| 8 -> 15 : 1028 | | 16 -> 31 : 14 | | 32 -> 63 : 1 | | [.].

Virtualization 76

Virtualization AWS Engineering Hardware 76