The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. We chose Open-Zipkin because it had better integrations with our Spring Boot based Java runtime environment.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. 1072-aws (xxx) 12/19/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 12.25 avg-cpu: %user %nice %system %iowait %steal %idle 14.81

Cache 105

Cache C++ AWS Latency 105

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. 1072-aws (xxx) 12/19/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 12.25 avg-cpu: %user %nice %system %iowait %steal %idle 14.81

Cache 40

Cache C++ AWS Systems 40

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. 30.14% in the middle of the flame graph.

Speed 126

Speed Java AWS Virtualization 126

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. This will slow this test a little.)

Speed 52

Speed Java AWS Virtualization 52

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. 30.14% in the middle of the flame graph.

Speed 40

Speed Java AWS Virtualization 40

The Netflix TechBlog

DECEMBER 17, 2019

There are services at Netflix that use RDBMS kind of databases such as MySQL or PostgreSQL via AWS RDS. Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. The destination may be a datastore or an external API.

Database 197

Database Traffic Transportation Open Source 197