Brendan Gregg

JULY 4, 2021

## References I've reproduced the talk references below, so you can click on links: - [Gregg 08] Brendan Gregg, “ZFS L2ARC,” [link] Jul 2008 - [Gregg 10] Brendan Gregg, “Visualizations for Performance Analysis (and More),” [link] 2010 - [Greenberg 11] Marc Greenberg, “DDR4: Double the speed, double the latency? Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

MachMetrics

JUNE 16, 2020

In a vacuum, an SSL certificate does add some additional latency, as it requires 2 extra round trips to establish a secure connection before sending any data to the browser. Secondly, SSL/HTTPS unlocks additional web performance benefits that more than make up for the added latency. To be completely honest, you almost have to use it.

Speed 86

Speed Latency Google Website 86

ACM Sigarch

MARCH 18, 2019

How many buffers are needed to track pending requests as a function of needed bandwidth and expected latency? Can one both minimize latency and maximize throughput for unscheduled work? These models are useful for insight regarding the basic computer system performance metrics of latency and throughput (bandwidth). Little’s Law.

Systems 60

Systems Latency Performance Analytics 60

SQL Performance

FEBRUARY 22, 2021

Recently one of our teams was investigating a log reader latency issue. We pay a lot of attention to latency here, along with any long-running transactions, because of downstream impact to technologies that use the log reader – like Availability Groups and transactional replication. sysprocesses.

Latency 57

Latency Servers Monitoring Systems 57

ACM Sigarch

MARCH 25, 2019

How many buffers are needed to track pending requests as a function of needed bandwidth and expected latency? Can one both minimize latency and maximize throughput for unscheduled work? The M/M/1 queue will show us a required trade-off among (a) allowing unscheduled task arrivals, (b) minimizing latency, and (c) maximizing throughput.

Systems 53

Systems Latency Performance C++ 53

Brendan Gregg

AUGUST 29, 2021

biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. cachestat]: /blog/2014-12-31/linux-page-cache-hit-ratio.html [60-second performance checklist]: Articles/Netflix_Linux_Perf_Analysis_60s.pdf [bcc]: [link] An r_wait of 33 ms is kinda high, and likely due to the queueing (avgqu-sz). Tracing block device I/O.

Cache 105

Cache C++ AWS Latency 105

Speed Curve

MARCH 31, 2014

For Q2 2014 we're switching to IE11 which is now the most popular version of IE according to Akamai and StatsCounter and the average connection speed has been updated to 9.8Mbps download, 2.5Mbps upload with a 10ms latency. The 9.8Mbps download speed is based on the latest State of the Internet report from Akamai.

Speed 40

Speed Latency Internet Internet 40

Tim Kadlec

SEPTEMBER 25, 2014

Macbook Air (2014). Macbook Air (2014). Macbook Air (2014). Macbook Air (2014). On powerful devices, like my Macbook Air (2014), parse and execution time was negligible. Google had a great post a few years back about how they reduced startup latency for Gmail. Firefox 31. iPad (4th Gen).

Latency 41

Latency Google Hardware Servers 41

Brendan Gregg

JULY 2, 2021

biolatency Disk I/O latency histogram heat map 5. runqlat CPU scheduler latency heat map 10. As an extreme example, I wrote cachestat(8) while on vacation in 2014 for use on the Netflix cloud, which was a mix of Linux 3.2 execsnoop New processes (via exec(2)) table 2. opensnoop Files opened table 3. It depends on the tool.

Open Source 40

Open Source Latency Cache Energy 40

Brendan Gregg

AUGUST 29, 2021

biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. cachestat]: /blog/2014-12-31/linux-page-cache-hit-ratio.html [60-second performance checklist]: /Articles/Netflix_Linux_Perf_Analysis_60s.pdf [bcc]: [link] An r_wait of 33 ms is kinda high, and likely due to the queueing (avgqu-sz).

Cache 40

Cache C++ AWS Systems 40

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 229

Processing Media Latency Innovation 229

Dynatrace

SEPTEMBER 29, 2020

Since we moved to AWS in May 2014 we have had an availability of 99.95%! Sydney, we have a disk write latency problem! It was on August 25 th at 14:00 when Davis initially alerted on a disk write latency issues to Elastic File System (EFS) on one of our EC2 instances in AWS’s Sydney Data Center.

Infrastructure 178

Infrastructure Cloud Monitoring AWS 178

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 126

Speed Java AWS Virtualization 126

The Netflix TechBlog

FEBRUARY 20, 2019

Today we are excited to announce latency heatmaps and improved container support for our on-host monitoring solution?—?Vector?—?to Remotely view real-time process scheduler latency and tcp throughput with Vector and eBPF What is Vector? to the broader community. Vector is open source and in use by multiple companies.

Performance 211

Performance Latency Open Source Metrics 211

CSS Wizardry

MARCH 3, 2019

If, however, there wasn’t a new file on the server, we’ll bring back a 304 header, no new file, but an entire roundtrip of latency. We can completely cut out the overhead of a roundtrip of latency. On high latency connections, this saving could be tangible. Steve Souders , 2014. Caching – 2014. Decorative Image.

Cache 264

Cache Latency Strategy Servers 264

All Things Distributed

APRIL 4, 2017

This enables customers to serve content to their end users with low latency, giving them the best application experience. In 2011, AWS opened a Point of Presence (PoP) in Stockholm to enable customers to serve content to their end users with low latency. In 2014 and 2015 respectively, AWS opened offices in Stockholm and Espoo, Finland.

AWS 117

AWS Airlines Latency Games 117

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you time time?

Speed 52

Speed Java AWS Virtualization 52

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 40

Speed Java AWS Virtualization 40

All Things Distributed

OCTOBER 25, 2018

The new AWS Africa (Cape Town) Region will have three Availability Zones and provide lower latency to end users across Sub-Saharan Africa. Since it launched in 2014, more than 9 million people have saved or borrowed on the JUMO platform.

AWS 167

AWS Cloud Education Programming 167

All Things Distributed

JULY 20, 2015

This architecture affords Amazon ECS high availability, low latency, and high throughput because the data store is never pessimistically locked. As you can see, the latency remains relatively jitter-free despite large fluctuations in the cluster size. latency and stability) with Empire as well as security benefits.

Latency 166

Latency Architecture AWS Open Source 166

Brendan Gregg

MAY 16, 2017

When I joined Netflix in April 2014, we had over 40 million subscribers in 41 countries. A latency outlier issue that happened every 15 minutes. In the last three years I developed the ftrace-based [perf-tools] and used them to solve many problems, which I wrote about in [lwn.net] and spoke about at [LISA 2014].

Java 75

Java Entertainment Engineering Scalability 75

All Things Distributed

DECEMBER 12, 2018

They can run applications in Sweden, serve end users across the Nordics with lower latency, and leverage advanced technologies such as containers, serverless computing, and more. Starting today, developers, startups, and enterprises—as well as government, education, and non-profit organizations—can use the new AWS Europe (Stockholm) Region.

AWS 124

AWS Cloud Games Serverless 124

The Morning Paper

JANUARY 23, 2020

Today we’ll be digging into the analysis of an incident that took place at Etsy on December 4th, 2014. 1:18pm a key observation was made that an API call to populate the homepage sidebar saw a huge jump in latency. This is part 2 of our look at Allspaw’s 2015 master thesis (here’s part 1 ).

Internet 53

Internet Internet Cache Engineering 53

Brendan Gregg

AUGUST 8, 2017

And here's an excerpt from [Linux] today (include/linux/sched/loadavg.h): #define EXP_1 1884 /* 1/exp(5sec/1min) as fixed-point */ #define EXP_5 2014 /* 1/exp(5sec/5min) */ #define EXP_15 2037 /* 1/exp(5sec/15min) */. Latency was acceptable and no one complained. Linux is also hard coding the 1, 5, and 15 minute constants.

Latency 111

Latency Metrics C++ Systems 111

SQL Performance

APRIL 8, 2019

The older Intel Xeon E5-26xx v3 (Haswell) series which was introduced in Q3 of 2014, had a maximum memory bandwidth of 2133MHz. They feature low latency, local NVMe storage that can directly leverage the 128 PCIe 3.0 On the other hand, Azure VM CPU and storage performance has come a long way since I wrote about it back in 2014 !

Azure 42

Azure Virtualization Storage Benchmarking 42

SQL Server According to Bob

SEPTEMBER 26, 2016

As we moved towards SQL Server 2014, the pace of hardware accelerated. Our customers who deployed Availability Groups were now using servers for primary and secondary replicas with 12+ core sockets and flash storage SSD arrays providing microsecond to low millisecond latencies.

Availability 40

Availability Servers Hardware Benchmarking 40

Alex Russell

APRIL 29, 2021

Delayed three years ( Chrome 40, November 2014 vs. Safari 11.1, A subset ( element.animate() ) has enabled developers to more easily create high-performance visual effects with lower risk of visual stuttering in Chrome and Firefox since 2014. Critical in adapting web content to mobile, particularly regarding multi-touch gestures.

Media 145

Media Games Education Engineering 145

Abhishek Tiwari

NOVEMBER 3, 2018

Using CDN for the whole website, you can offload most of the website traffic to your CDN which will handle not only large traffic spikes but also reduce the latency of content delivery. They often get blindsided by vendor’s pitch and end-up making decision based on some fancy demos (see my post from 2014 on Adobe AEM ).

Systems 63

Systems Cache Website Network 63

Alex Russell

MARCH 6, 2021

Devices and networks have evolved too: Alex Russell @slightlylate An update on mobile CPUs and the Performance Inequality Gap: Mid-tier Android devices (~$300) now get the single-core performance of a 2014 iPhone and the multi-core perf of a 2015 iPhone. mid-priced Androids were slightly faster than 2014's iPhone 6.

Performance 124

Performance Network Cache Metrics 124

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. Many new tools can now be written, and the main toolkit we're working on is [bcc]. Hit Ctrl-C to end. ^C Or, from a different point of view, Solaris was less secure.)

Virtualization 76

Virtualization AWS Engineering Hardware 76

KeyCDN

JUNE 20, 2019

Online users are becoming less and less patient meaning you as an eCommerce store owner need to implement methods for reducing latency and speeding up your website. This reduces latency and speeds up your website. As of 2014, the number of mobile users surpassed that of desktop users and shows no signs of slowing down.

Ecommerce 41

Ecommerce Speed Website Mobile 41