Brendan Gregg

JULY 15, 2019

BPF (eBPF) tracing is a superpower that can analyze everything, and I'll show you how in my upcoming book BPF Performance Tools: Linux System and Application Observability , coming soon from Addison Wesley. Some other future BPF books may be good, and I'll recommend them as appropriate. I lost count at 30 !

Systems 111

Systems Performance Education Technology 111

Brendan Gregg

JUNE 14, 2021

For USENIX LISA2021 I gave a 40 minute deep dive talk on BPF internals for Linux, focusing on observability tracing tools. Since there are already BPF internals references online (listed in this post) I used the opportunity to create some new content, showing how bpftrace instrumentation works from user space down to machine code.

Website 52

Website Code Performance 52

Brendan Gregg

DECEMBER 21, 2019

For my AWS re:Invent talk on BPF Performance Analysis at Netflix, I began with a demo of "BPF superpowers" (aka eBPF). astype(numpy.int16) sound = pygame.sndarray.make_sound(note[0]) # BPF b = BPF(text=""" #include <uapi/linux/ptrace.h> My [BPF Performance Tools] book has plenty of examples.

C++ 52

C++ Education Java AWS 52

Brendan Gregg

DECEMBER 21, 2019

For my AWS re:Invent talk on BPF Performance Analysis at Netflix, I began with a demo of "BPF superpowers" (aka eBPF). from __future__ import print_function from bcc import BPF import pygame, numpy, pygame.sndarray # Sound setup sampleRate = 44100 pygame.mixer.pre_init(sampleRate, -16, 1). usr/bin/python # # iwlstrength.py

C++ 52

C++ Education Java AWS 52

Brendan Gregg

DECEMBER 1, 2019

At Netflix we have 15 BPF programs running on cloud servers by default; Facebook has 40. BPF originally stood for Berkeley Packet Filter, but has been extended in Linux to become a generic kernel execution engine, capable of running a new type of user-defined and kernel-mode applications.

Software 52

Software Software Programming Operating System 52

Brendan Gregg

NOVEMBER 3, 2020

Two new technologies, BTF and CO-RE, are paving the way for BPF to become a billion-dollar industry. BTF and CO-RE eliminate these dependencies at runtime, not only making BPF more practical for embedded Linux environments, but for adoption everywhere. The BCC project has a collection of these, called [libbpf tools].

C++ 40

C++ Programming Storage Code 40

Brendan Gregg

NOVEMBER 3, 2020

Two new technologies, BTF and CO-RE, are paving the way for BPF to become a billion-dollar industry. BTF and CO-RE eliminate these dependencies at runtime, not only making BPF more practical for embedded Linux environments, but for adoption everywhere. The BCC project has a collection of these, called [libbpf tools].

C++ 40

C++ Programming Storage Code 40

DZone

JANUARY 25, 2024

Unlike its predecessor, the Berkeley Packet Filter (BPF) , eBPF allows the execution of sandboxed programs in privileged contexts, such as the OS kernel, without the need to modify kernel source code or disrupt overall program execution.

Monitoring 161

Monitoring Programming Operating System Efficiency 161

The Netflix TechBlog

JUNE 7, 2021

What is BPF? Berkeley Packet Filter (BPF) is an in-kernel execution engine that processes a virtual instruction set, and has been extended as eBPF for providing a safe way to extend kernel functionality.

Network 325

Network Transportation AWS Cloud 325

Brendan Gregg

JULY 14, 2019

BPF (eBPF) tracing is a superpower that can analyze everything, and I'll show you how in my upcoming book BPF Performance Tools: Linux System and Application Observability , coming soon from Addison Wesley. Since BPF does so many things, it is becoming a technology name and no longer an acronym. I lost count at 30 !

Systems 40

Systems Performance Education Technology 40

Brendan Gregg

JULY 2, 2021

There's an arms race to add eBPF (BPF) to commercial observability products, and in this post I'll describe how to quickly do that. People like to show me their BPF observability products after they have prototyped or built them, but I often wish I had given them advice before they started. You might not even like BPF.

Latency 145

Latency Cache Energy Systems 145

Brendan Gregg

JULY 15, 2020

The second edition adds content on BPF, BCC, bpftrace, perf, and Ftrace, mostly removes Solaris, makes numerous updates to Linux and cloud computing, and includes general improvements and additions. A year ago I announced [BPF Performance Tools: Linux System and Application Observability].

Systems 145

Systems Cloud Performance Tuning 145

Brendan Gregg

JULY 4, 2021

It's an exciting time for developments in computer performance, not just for the BPF technology (which I often [write about]) but also for processors with 3D stacking and cloud vendor CPUs (e.g., In case you are interested, at the same conference I also gave a talk on [BPF Internals].

Performance 140

Performance Latency Hardware Storage 140

Brendan Gregg

MARCH 8, 2020

I've also been working on Systems Performance 2nd Edition, now that the [BPF book] is done. At LISA I also ran a BPF performance tools workshop with over 200 attendees. Systems Performance]: /sysperfbook.html [BPF book]: /bpf-performance-tools-book.html [SREcon]: [link].

Systems 144

Systems Performance Benchmarking Tuning 144

Brendan Gregg

APRIL 14, 2022

I hope my last two books, [Systems Performance 2nd Ed] and [BPF Performance Tools] serve Netflix well in my absence and everyone else who reads them. I'll still be posting here in my next job. More on that soon.

Java 145

Java Cloud Cache AWS 145

Brendan Gregg

APRIL 27, 2023

BPF for security monitoring was first explored by myself and a Netflix security engineer, Alex Maestretti, in a [2017 BSides talk] (some slides below). Ideally I'd link to an example in bcc for security monitoring (we could create a subdirectory for them) but that currently doesn't exist. someone with solid pen-testing experience).

Open Source 144

Open Source Monitoring Engineering Design 144

The Netflix TechBlog

NOVEMBER 22, 2019

4pm-5pm OPN 303-R BPF Performance Analysis Brendan Gregg , Senior Performance Engineer Abstract : Extended BPF (eBPF) is an open-source Linux technology that powers a whole new class of software: mini programs that run on events. We share everything attendees need to implement CloudTrail in their own organizations.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

4pm-5pm OPN 303-R BPF Performance Analysis Brendan Gregg , Senior Performance Engineer Abstract : Extended BPF (eBPF) is an open-source Linux technology that powers a whole new class of software: mini programs that run on events. We share everything attendees need to implement CloudTrail in their own organizations.

AWS 100

AWS Entertainment Open Source Benchmarking 100

Brendan Gregg

JUNE 3, 2021

Later on I wrote much better socket tools (in my [DTrace] and [BPF] books). I'm reminded of this first case since my BPF tools are now appearing in observability products, and will grow to a scale much bigger than my DTrace tools. . ## Other Cases Of all the tools I had published as open source, I still can't believe socketsnoop.d

Open Source 144

Open Source Code Engineering Technology 144

Brendan Gregg

MARCH 16, 2024

" I pictured how this is used to walk stack traces in my BPF book. Stack Frame with Base Pointer ( x86-64 ABI ) Figure 2-6: Frame Pointer-based Stack Walking ( BPF book ) This stack-walking technique is commonly used by external profilers and debuggers, including Linux perf and eBPF, and ultimately visualized by flame graphs.

Java 145

Java Cache Google Hardware 145

Brendan Gregg

JULY 14, 2020

The second edition adds content on BPF, BCC, bpftrace, perf, and Ftrace, mostly removes Solaris, makes numerous updates to Linux and cloud computing, and includes general improvements and additions. A year ago I announced [BPF Performance Tools: Linux System and Application Observability].

Systems 52

Systems Cloud Performance Tuning 52

Brendan Gregg

JULY 14, 2020

The second edition adds content on BPF, BCC, bpftrace, perf, and Ftrace, mostly removes Solaris, makes numerous updates to Linux and cloud computing, and includes general improvements and additions. A year ago I announced [BPF Performance Tools: Linux System and Application Observability].

Systems 52

Systems Cloud Performance Tuning 52

Brendan Gregg

JULY 4, 2021

It's an exciting time for developments in computer performance, not just for the BPF technology (which I often [write about]) but also for processors with 3D stacking and cloud vendor CPUs (e.g., In case you are interested, at the same conference I also gave a talk on [BPF Internals].

Performance 52

Performance Latency Hardware Storage 52

Brendan Gregg

JULY 2, 2021

There's an arms race to add [eBPF] (BPF) to commercial observability products, and in this post I'll describe how to quickly do that. People like to show me their BPF observability products after they have prototyped or built them, but I often wish I had given them advice before they started. You might not even like BPF.

Open Source 40

Open Source Latency Cache Energy 40

Brendan Gregg

AUGUST 18, 2019

I [previously] wrote about bpftrace vs other tracers, including [BCC] (BPF Compiler Collection). bpftrace uses BPF (Berkeley Packet Filter), an in-kernel execution engine that processes a virtual instruction set. BPF is in the Linux kernel, and bpftrace is the best way to get started using BPF for observability.

Latency 68

Latency C++ Cache Programming 68

Brendan Gregg

MARCH 7, 2020

I've been working on Systems Performance 2nd Edition, now that the [BPF book] is done. At LISA I also ran a BPF performance tools workshop with over 200 attendees. Systems Performance]: /sysperfbook.html [BPF book]: /bpf-performance-tools-book.html [SREcon]: [link]

Systems 52

Systems Performance Benchmarking Tuning 52

Brendan Gregg

MARCH 7, 2020

I've been working on Systems Performance 2nd Edition, now that the [BPF book] is done. At LISA I also ran a BPF performance tools workshop with over 200 attendees. Systems Performance]: /sysperfbook.html [BPF book]: /bpf-performance-tools-book.html [SREcon]: [link].

Systems 52

Systems Performance Benchmarking Tuning 52

Brendan Gregg

APRIL 27, 2023

BPF for security monitoring was first explored by myself and a Netflix security engineer, Alex Maestretti, in a 2017 BSides talk (some slides below). Ideally I'd link to an example in bcc for security monitoring (we could create a subdirectory for them) but that currently doesn't exist. someone with solid pen-testing experience).

Open Source 52

Open Source Monitoring Engineering Design 52

Brendan Gregg

MARCH 23, 2024

But that's not to say that one is better or faster than the other: They emit the same BPF bytecode and are equally fast once running. Some longer notes: [1] bcc and bpftrace have many overlapping tools: the bcc ones are more capable (e.g., CLI options), and the bpftrace ones can be edited on the fly.

Servers 145

Servers Traffic Network Systems 145

Brendan Gregg

OCTOBER 8, 2018

It is being developed for BSD, too, where BPF originated. That post covered the [bcc] front end (BPF Complier Collection), which has let me rewrite my DTraceToolkit tools. Internally, bpftrace uses a lex/yacc parser to convert programs to AST, then llvm IR actions, then BPF. It's shaping up to be a DTrace version 2.0:

C++ 110

C++ Virtualization Programming Latency 110

Brendan Gregg

FEBRUARY 28, 2023

Titus, the Netflix container management platform, is now open source,” [link] Apr 2018 - [Cutress 19] Dr. DDR6: Here's What to Expect in RAM Modules,” [link] Nov 2020 - [Salter 20] Jim Salter, “Western Digital releases new 18TB, 20TB EAMR drives,” [link] Jul 2020 - [Spier 20] Martin Spier, Brendan Gregg, et al.,

Performance 110

Performance Latency Cache Virtualization 110

Brendan Gregg

JANUARY 1, 2019

bcc tools are of two parts: the BPF code for the kernel, written in C, and the user-space tool written in Python (or lua, or C++). There's also the kernel eBPF (aka BPF) engine: if you browse bcc and bpftrace issues, you'll see some requests for enhancements there. bcc Reference Guide. There's also lots of examples under bcc/tools/*.py.

C++ 111

C++ Virtualization Programming Code 111

Brendan Gregg

MAY 28, 2021

Back in 2015 we'd have BPF (iovisor) meetups in Santa Clara and most contributors would be there in person, with some having travelled. But I'm now excited about my new adventure: Doing an advanced tech role remotely from Australia. I know others who have also left the Bay Area or are planning to.

Google 144

Google Engineering Open Source AWS 144

Brendan Gregg

JUNE 3, 2021

Later on I wrote much better socket tools (in my [DTrace] and [BPF] books). I'm reminded of this first case since my BPF tools are now appearing in observability products, and will grow to a scale much bigger than my DTrace tools. . ## Other Cases Of all the tools I had published as open source, I still can't believe socketsnoop.d

Open Source 40

Open Source Code Engineering Technology 40

Brendan Gregg

DECEMBER 31, 2018

In summary: - Beginner: run [bcc] tools - Intermediate: develop [bpftrace] tools - Advanced: develop [bcc] tools, contribute to bcc & bpftrace Update: I have a new book about eBPF tracing, published by Addison Wesley: [BPF Performance Tools: Linux System and Application Observability]. Beginner ## 1. bcc Reference Guide.

C++ 52

C++ Virtualization Programming Code 52

Brendan Gregg

OCTOBER 15, 2019

I'd also recommend watching: - BPF at Facebook by Alexei Starovoitov ( youtube ) - The ubiquity but also the necessity of eBPF as a technology to keep the kernel relevant by David S. I'll need to come back to a future Kernel Recipes! kdress]: [link] [Steven Rostedt]: [link] [youtube]: [link] [slideshare]: [link] [perf-tools]: [link]

C++ 77

C++ Engineering Technology Technology 77

The Netflix TechBlog

NOVEMBER 22, 2019

4pm-5pm OPN 303-R BPF Performance Analysis Brendan Gregg , Senior Performance Engineer Abstract : Extended BPF (eBPF) is an open-source Linux technology that powers a whole new class of software: mini programs that run on events. We share everything attendees need to implement CloudTrail in their own organizations.

AWS 37

AWS Entertainment Open Source Benchmarking 37