Nick Desaulniers

The enemy's gate is down

Jan 20, 2023 - 8 minute read - Comments - debugging

Debugging -Wframe-larger-than=

Unless work is done per architecture to implement HAVE_ARCH_VMAP_STACK / CONFIG_VMAP_STACK, the Linux kernel defaults to two pages worth of stack per thread.


Note: on many contemporary systems the page size is 4KiB, but this is actually configurable for many architectures. The trade offs probably require a separate post. If you see code that checks for alignment via bitwise tricks like addr & 4095 == 0 without checking sysconf(_SC_PAGESIZE) it is perhaps a red flag for code that might be to reused on different systems.


As a first line of defense against overflowing a thread’s kernel stack, we enable -Wframe-larger-than= with a value based on CONFIG_FRAME_WARN (commonly 1024). This doesn’t guarantee we won’t recurse enough at runtime to overflow the stack. Defending against that is akin to solving the Halting Problem unless you want to go to the extreme length of MISRA C’s guidance that “functions should not call themselves, either directly or indirectly.”

It does help us frequently find large structs that were stack allocated and probably should have been heap (or sometimes statically) allocated. But the ergonomics of this warning have room for improvement.

Let’s say one day the build is failing because someone has introduced a new instance of -Wframe-larger-than=. In the logs you see:

<source>:8:6: warning: stack frame size (4104) exceeds limit (1024) in 'foo' [-Wframe-larger-than]
void foo (void) {
     ^

So you go and look at source and see:

void foo (void) {
    bar();
    baz();
    quux();
}

See any large local variables there? Thus begins the goose chase to understand what inlining decisions led to foo having a large stack frame. Or how about this?

void baz (void) {
    struct widget;
    ...

is a struct widget too large to be putting on the stack? Let’s look at the definition:

struct widget {
    struct gadget gadget;
    struct trombone trombone;
    long data [42];
    ...

What’s the sizeof struct widget? Can you do that calculation in your head quickly? What if the definitions of those structs are in other headers? More goose chasing and perhaps an argument in favor of an IDE.

DWARF has this information (if/when it’s produced) but we don’t have really great ways to visualize this information. DWARF is a Jack of All Trades, it’s good at many things, but kind of great at none and so gets easily dunked on leading to distinct unwind formats and type formats being used in the kernel.

While commiserating about this conundrum with my colleague, Paul Kirth, I mentioned that I really wished there was tooling that would “just break out the crayons and draw me a picture of the stack [usage of a given function].” Well, Paul must have been more upset than I was, because he implemented some really nice optimization remarks to help.

Starting with clang-16, you should be able to use -Rpass-analysis=stack-frame-layout to get a drawing of your stack layout.

Let’s see if we can better debug an instance of -Wframe-larger-than= affecting the Linux kernel. Looking at logs from the CI from last night, I see a case:

Warning: /builds/linux/fs/jffs2/xattr.c:775:6: warning: stack frame size (1216) exceeds limit (1024) in 'jffs2_build_xattr_subsystem' [-Wframe-larger-than]
void jffs2_build_xattr_subsystem(struct jffs2_sb_info *c)
     ^

Looks like I can reproduce this locally:

$ wget https://src.fedoraproject.org/rpms/kernel/raw/rawhide/f/kernel-aarch64-fedora.config -O .config
$ make -s LLVM=1 ARCH=arm64 -j128 olddefconfig fs/jffs2/xattr.o
fs/jffs2/xattr.c:775:6: warning: stack frame size (1216) exceeds limit (1024) in 'jffs2_build_xattr_subsystem' [-Wframe-larger-than]
void jffs2_build_xattr_subsystem(struct jffs2_sb_info *c)
     ^
119/1216 (9.79%) spills, 1097/1216 (90.21%) variables
1 warning generated.

Paul also recently added that tidbit (that was cutoff from CI logs) about the relative number of spills vs variables. That can help you quickly get a sense how many stack slots are variables’ storage vs spills from excessive register pressure (too many live values).

If we simply add -Rpass-analysis=stack-frame-layout, we’re going to get a beautiful ASCII table for every function in a given TU. We can use the flag -mllvm -fiter-print-funcs= to reduce the number of optimization remarks emitted. For the kernel, that might look like:

make -s LLVM=1 ARCH=arm64 -j128 fs/jffs2/xattr.o KCFLAGS="-Rpass-analysis=stack-frame-layout -mllvm -filter-print-funcs=jffs2_build_xattr_subsystem" 
fs/jffs2/xattr.c:775:6: warning: stack frame size (1216) exceeds limit (1024) in 'jffs2_build_xattr_subsystem' [-Wframe-larger-than]
void jffs2_build_xattr_subsystem(struct jffs2_sb_info *c)
     ^
119/1216 (9.79%) spills, 1097/1216 (90.21%) variables
fs/jffs2/xattr.c:776:1: remark: 
Function: jffs2_build_xattr_subsystem
Offset: [SP-8], Type: Spill, Align: 8, Size: 8
Offset: [SP-16], Type: Spill, Align: 8, Size: 8
Offset: [SP-24], Type: Spill, Align: 8, Size: 8
Offset: [SP-32], Type: Spill, Align: 8, Size: 8
Offset: [SP-40], Type: Spill, Align: 8, Size: 8
Offset: [SP-48], Type: Spill, Align: 8, Size: 8
Offset: [SP-56], Type: Spill, Align: 8, Size: 8
Offset: [SP-64], Type: Spill, Align: 8, Size: 8
Offset: [SP-72], Type: Spill, Align: 8, Size: 8
Offset: [SP-80], Type: Spill, Align: 8, Size: 8
Offset: [SP-88], Type: Spill, Align: 8, Size: 8
Offset: [SP-96], Type: Spill, Align: 8, Size: 8
Offset: [SP-104], Type: Variable, Align: 8, Size: 8
Offset: [SP-136], Type: Variable, Align: 8, Size: 28
    rr @ fs/jffs2/xattr.c:448
Offset: [SP-144], Type: Variable, Align: 8, Size: 8
Offset: [SP-1168], Type: Variable, Align: 8, Size: 1024
    xref_tmphash @ fs/jffs2/xattr.c:778
Offset: [SP-1176], Type: Spill, Align: 8, Size: 8
Offset: [SP-1180], Type: Spill, Align: 4, Size: 4
Offset: [SP-1192], Type: Spill, Align: 8, Size: 8
Offset: [SP-1196], Type: Spill, Align: 4, Size: 4 [-Rpass-analysis=stack-frame-layout]
{
^
1 warning generated.

(This kernel config had debug info enabled. Without this, lines above printing variable and line number for rr and xref_tmphash would be omitted).

Spills can be occupied by different variables at different points for the program counter (this is how DWARF encodes DW_AT_location). Not sure about the Variable slots at Offsets [SP-104] and [SP-144] yet, maybe there’s more to fix in this nascent analysis, but the sizes show those aren’t the droids… err… stack slots that I’m looking for.

So right off the bat, if I’m setting -Wframe-larger-than=1024 and xref_tmphash is 1024B, that’s a problem. fs/jffs2/xattr.c:778 corresponds to this statement.

#define XREF_TMPHASH_SIZE	(128)
...
struct jffs2_xattr_ref *xref_tmphash[XREF_TMPHASH_SIZE];

It doesn’t matter what the sizeof struct jffs2_xattr_ref since xref_tmphash is a 128 element array of pointers to such struct. 128 times 8 (pointers are 64b on the kernel’s aarch64 target) is 1024.

Let’s see if anyone has tried to fix this. Yep, looking at lore I see:

The patch from 2017 LGTM, and perhaps has fallen through the cracks and just needs to be pinged/reviewed, which I’ve done. Hopefully it can get picked up.

-Wframe-larger-than= issues in the Linux kernel are numerous (at least with clang builds for now). They can be a bit of work to track down. One of our oldest open issues still on the TODO list to fix is -fsanitize=kernel-address (via CONFIG_KASAN=y) leading to excessive stack usage (at least when compared to -fsanitize=address aka ASAN). Another TODO seems to be related to passing structs by value. My hope is these optimization remarks will help us differentiate between compiler bugs vs kernel source bugs quicker.

Whether or not an optimization remark is the most ergonomic tooling is hopefully still up for debate, though the current implementation was an iteration based on compromise. Requiring debug info for better diagnostics trades compile time in clang, which is also unfortunate.

For now though, I’m happy to celebrate improved tooling in this regard. Cheers Paul!

That’s pretty much the end of the post. Some other tools I’ve used in this area if you’re still interested:


llvm-dwarfdump or GNU objdump --dwarf=info can print the DWARF stream. I had written a python script to attempt to decode this information. It’s manually tested, incomplete, and quite buggy. The library I depend on for decoding DWARF and ELF doesn’t support all of the architectures the Linux kernel does, which has led to issues being unable to debug these warnings for specific architectures. A tool failure when things are actively on fire is stressful.

$ frame_larger_than.py arch/x86/kernel/kvm.o kvm_send_ipi_mask_allbutself
kvm_send_ipi_mask_allbutself:
        1024    struct cpumask          new_mask
        4       unsigned int            this_cpu
        8       const struct cpumask*   local_mask
        4       int                     pscr_ret__
        4       int                     pfo_ret__
cpumask_copy:
bitmap_copy:
        4       unsigned int            len
        4       unsigned int            len
cpumask_clear_cpu:
clear_bit:
arch_clear_bit:

“Poke-a-hole” aka pahole can print the size of structs. Check out the LWN article.

# Make sure you've built with debug info enabled!
$ pahole fs/jffs2/xattr.o
...
struct jffs2_xattr_ref {
        void *                     always_null;          /*     0     8 */
        struct jffs2_raw_node_ref * node;                /*     8     8 */
        uint8_t                    class;                /*    16     1 */
        uint8_t                    flags;                /*    17     1 */
        u16                        unused;               /*    18     2 */
        uint32_t                   xseqno;               /*    20     4 */
        union {
                struct jffs2_inode_cache * ic;           /*    24     8 */
                uint32_t           ino;                  /*    24     4 */
        };                                               /*    24     8 */
        union {
                struct jffs2_inode_cache * ic;                   /*     0     8 */
                uint32_t                   ino;                  /*     0     4 */
        };

        union {
                struct jffs2_xattr_datum * xd;           /*    32     8 */
                uint32_t           xid;                  /*    32     4 */
        };                                               /*    32     8 */
        union {
                struct jffs2_xattr_datum * xd;                   /*     0     8 */
                uint32_t                   xid;                  /*     0     4 */
        };

        struct jffs2_xattr_ref *   next;                 /*    40     8 */

        /* size: 48, cachelines: 1, members: 9 */
        /* last cacheline: 48 bytes */
};

The kernel has a script (scripts/stackusage) that can print the estimated stack usage of each function in vmlinux to a file. Example:

$ ./scripts/stackusage LLVM=1 -j128 defconfig all
...
./scripts/stackusage: output written to /tmp/stackusage.3991505.3gRw
$ cat /tmp/stackusage.3991505.3gRw
arch/x86/entry/common.c:119:do_int80_syscall_32	24	dynamic
arch/x86/entry/common.c:138:__do_fast_syscall_32	40	dynamic
arch/x86/entry/common.c:186:do_fast_syscall_32	16	static
arch/x86/entry/common.c:238:do_SYSENTER_32	0	static
...

Finally, GCC currently supports (but clang currently does not) -fconserve-stack. From playing with it, it seems that this flag causes GCC to limit inlining if would increase the stack usage of a caller beyond what appears to be through experimentation an arch specific threshold.

Clang recently got -finline-max-stacksize=, which feels like a nuclear option to me. We haven’t deployed it yet in the kernel, but it might ultimately be necessary to use. We’ll see. I would hate to potentially cover up other issues that should perhaps be fixed first.

Off by Two Critical Edge Splitting

comments powered by Disqus