This blog was originally published in June of 2023 and was updated in February of 2024.

On occasion, DBAs come across segmentation fault issues while executing some queries. However, this is one of the least-explored topics to date. I tried to search for details related to segmentation fault on the internet and found many articles; however, it failed to quench my thirst as none of them had the answer I was looking for. So, I decided to gather information and write a detailed blog about this issue.

To understand “segmentation fault (aka segfault),” it is a must to know the basic idea of segmentation and its implementation in C programming. This blog will also cover a scenario that causes “segmentation fault.”

Understanding memory management methods

To understand segmentation fault, it is necessary to understand memory management methods for processes.

When we need to execute any program, it should be loaded into memory first. Inside the memory, they can be allocated any available space. When a program leaves the memory, space becomes available; however, the OS may or may not be able to allocate vacant memory space to another program or process as it has some issues. As the amount of space required by the new program may be higher than the space available in a fragment, the program should be broken into different chunks before it is loaded into memory, due to which memory management becomes challenging because it leads to fragmentation.

To overcome these issues, the concept of paging and segmentation was introduced, where physical and virtual address spaces were designed. A detailed description of these concepts is below.

Paging

This was designed to allow non-contiguous space allocation to processes. Here, memory is divided into equal sizes of partitions where the code of a program resides. The chunks in the main memory are called frames, while they are called pages in the secondary(or HDD). In order to handle memory management, a structure called memory management unit(MMU) is built, which divides memory blocks into major sections: logical address space and physical address space.

Logical address space: it comprises logical addresses that are generated by the CPU for the program.

Physical address space: it has physical addresses that are pointers to actual locations in memory.

In order to perform the actual translation of a logical address to the physical address, MMU needs to perform memory mapping operations, which can be accomplished by another structure called a page table. A page table has actual references to relevant physical addresses for logical addresses.

The figure below describes the same.

page table

Segmentation

This scheme was introduced to overcome the disadvantages of paging; it works similarly to paging, though. Instead of fixed-size pages, it creates different sizes of segments that are based on program code. In this case, we do not need physical address space. Here, a segment table manages everything.

Here, virtual(logical) to physical address translation is much easier as segment tables store adequate information.

I will not dive into this topic further as it requires a bit more technical understanding. The purpose of adding this section was to have a basic understanding of mapping from logical to physical addresses.

What is segmentation fault?

A segmentation fault is a specific type of error that occurs when a program tries to access a segment of memory that it doesn’t have the permissions to access or that doesn’t exist, leading to the program’s abrupt termination by the operating system. As explained above, the CPU first fetches a logical address, and by using a page table or a segment table, it finds/calculates the physical address of the desired memory location. That is how memory management works.

In an attempt to access the desired location, we sometimes come across some issues that are described below.

  • Occasionally, after calculating the physical address using a page/segment table, the program comes across the issue that required contents (piece of code, variables, or anything else) are unavailable in the physical memory location. This phenomenon is called “page fault.” This is not unusual and doesn’t affect the course of the execution as it just loads desirable items in memory.
  • Another one is a classical case of an inaccessible memory location. When the generated physical address points to a physical location that is not accessible by the program. This is called “segmentation fault,” which terminates the process execution. This happens when a program tries to access a read-only portion of memory or another program’s space.

Although the segmentation fault has been maligned as a showstopper, it is still mandatory as it is a mechanism to protect against any internal corruption.

Note: segmentation fault has nothing to do with the segmentation memory management method.

A reproducible scenario

While exploring at the code level, there are several scenarios that result in a segmentation fault, such as buffer overflows, stack overflows, and so on. However, this blog is written from the database perspective; hence, I would not prefer to dive into those scenarios as they are very high-level programming concepts.

In this section, I will focus on a scenario in the PostgreSQL database that causes segmentation fault.

This is the one that I came across once where the database gets restarted due to “segmentation fault.” Below is a line of code that results in a segmentation fault on PostgreSQL 13.4 and 12.8.

The above example is taken from here. By doing some further analysis, it came to light that it creates issues with LEFT JOIN only. In the case of an equi-join, it works as expected. This error was fixed in later versions of PostgreSQL.

What causes segmentation fault?

As described above, the actual cause of segmentation faults is trying to access a memory address that is not accessible by the program, and there are various reasons for the same to happen. However, sophisticated users have a limited understanding of such concepts, and due to that, I will try to explain them in the simplest possible terms.

The following are possible causes for segmentation fault:

Operating system issues

Misconfigurations, a lack of resources, or system library difficulties can all cause segmentation faults. These arise when the application’s operations conflict with the operating system’s memory management policies, resulting in unauthorized access attempts that the OS must terminate.

Buggy OS kernel

Bugs in the OS kernel, particularly those affecting the kernel’s handling of memory operations or system calls performed by applications, can cause a segmentation fault. Because the kernel is responsible for managing resources and execution permissions, any irregularities in its operation can result in segmentation faults in running applications.

Faulty hardware(specifically memory)

Hardware issues, particularly those involving the system’s memory (RAM or disk storage), are common causes of segmentation faults. Faulty memory might cause an application to read or write to incorrect locations, resulting in a crash. 

Bug in a product (e.g., PostgreSQL, MySQL)

Bugs in software products like PostgreSQL and MySQL can also trigger segmentation faults. If these applications attempt to execute an operation that accesses forbidden memory areas, due to issues in the program’s code, it can result in a segmentation fault. 

Database corruption

Database corruption can lead to segmentation faults when the database engine tries to access the corrupted data. This corruption might stem from hardware failures, bugs in the database software, or improper shutdown procedures, causing the database system to attempt operations based on invalid data structures.

Though the scope of this error is not limited to the above-mentioned reasons only, these are the most probable ones. In order to know the root cause of the issue, one needs to troubleshoot it with the help of programmers.

Learn more: The ultimate guide to open source databases

Troubleshooting and diagnosing a segmentation fault

To delve into the root cause of segmentation fault, it is imperative to install debug symbols and enable the creation of a core dump on failure. This helps analyze the issue and shows what function or part of the code causes the issue. If requirements are not met, it is not able to generate the core dump, and it becomes impossible to trace the issue.

Enable core dump generation

Every database has different methods to generate core dump files. To enable the generation of core dump, one needs to set some kernel settings as below.

Here, any other path can be used instead of /var/crash.

Enable debugging

Debug symbols enable code-level debugging. It shows details about the file being executed and the line of the code where the execution is happening. It is the responsibility of software developers to build debug symbols. In PostgreSQL, debug symbols can be enabled at the time of installation as below.

Also, there are certain packages available in PostgreSQL, such as postgresql-12-dbg. 

In the case of MySQL, the following command during the source code installation may turn on debugging.

Allow the database to generate core dumps

After enabling the core dump generation and debugging, databases must also collaborate with the host OS to generate a core dump. Hence, the database should be started with an option to create core files. To accomplish this, one should start the database with such an option.

In the case of PostgreSQL, the pg_ctl command should be started with the -c option, as shown below.

In MySQL, the following lines can be added in my.cnf or my.ini.

Note: In the event of a crash, the OS dumps all the contents from memory in the core file. So, before enabling, be sure you have sufficient space to accommodate the core dump.

Debugging core files

Core files are version-specific, and they can be read with the binary of a specific version of the database. Another version’s binary file cannot read the core file generated by the current version of the database. Like, the core file generated by MySQL 8 cannot be read by the MySQL binary from any other version. 

The core dump can be traced by Gnu debugger(gdb). Below is an example of reading the core dump.

Apart from that, Valgrind is also one of the tools that can be used to debug the issue. To learn more about Valgrind, check out Profiling MySQL Memory Usage With Valgrind Massif.

Best practices for preventing segmentation faults

Implementing best practices that minimize the likelihood of segmentation errors is critical. These faults can cause unexpected application behavior and crashes, often due to improper memory access or management. Developers can significantly decrease the incidence of segmentation faults in their applications by adhering to coding standards and techniques designed to prevent such issues.

1. Initialize pointers

Always initialize pointers before use. Uninitialized pointers can point to random memory locations, leading to segmentation faults when accessed.

2. Dynamic memory allocation checks 

After allocating dynamic memory using functions like malloc  or new , always check to ensure the allocation is successful. If the system cannot allocate the requested memory, it will return nullptr , and attempting to access this can lead to segmentation faults.

3. Array bounds checking 

Perform bounds checking on all array accesses because accessing an array beyond its allocated size is a common cause of segmentation faults

4. Use smart pointers (C++) 

In C++, prefer using smart pointers ( std::unique_ptr , std::shared_ptr ) over raw pointers. Smart pointers automatically manage memory, reducing the risk of memory leaks and segmentation faults.

5. Null pointer checks

Always check if a pointer is nullptr  before dereferencing it. Accessing a null pointer is undefined behavior and a frequent source of segmentation faults. 

6. Memory management 

Adopt strict memory management practices. Ensure every malloc  or new  operation has a corresponding free or delete. Proper memory management helps avoid leaks and double-free errors, which can cause segmentation faults.

7. Use memory analysis tools 

Memory analysis tools can detect memory mismanagement issues, such as use-after-free and memory leaks, helping identify potential causes of segmentation faults during the development phase.

8. Avoid undefined behavior 

Stay clear of undefined behaviors in your code. Undefined behaviors, such as accessing uninitialized variables, can lead to segmentation faults. Adhering to the language’s standards and best practices is key to avoiding such pitfalls.

9. String handling 

Ensure strings are properly null-terminated, and that buffer sizes are respected to avoid overflow, which can overwrite adjacent memory and cause segmentation faults.

10. Safe library functions 

When possible, use safe versions of library functions that include bounds checking, such as strncpy  instead of strcpy . These functions help prevent buffer overflows, a common source of segmentation faults.

11. Testing and debugging

Step through your code with debugging tools to find segmentation faults. Comprehensive testing, such as unit tests and stress tests, can identify scenarios that could lead to segmentation faults, allowing you to resolve any issues ahead of time.

Percona’s initiative

As described, a segmentation fault is caused by various issues that are sometimes not even in the control of programmers. But in many cases, programs themselves are culprits and trigger segmentation faults; however, users have the least knowledge of the same. Percona is committed to strengthening the open-source community and has acknowledged the issue. The Percona team strongly believes that users should have knowledge of the perils associated with some non-standard modules (or PostgreSQL extensions) that are identified as troublemakers.

These details are planned to be added in pg_gather reports. At present, this is in the development phase. The next version of the pg_gather will have these details available.

In summary: Take control of poor database performance

Indeed, segmentation fault is an issue that has not been widely explored yet. Having said that, it frequently revisits database systems due to a variety of reasons. Basically, it surfaces due to an attempt to access an unauthorized area or segment of memory where a normal DBA is least aware of the same. The issue can be troubleshot by enabling core dump generation and installation of debug symbols.

Dive into the critical aspects of database management with our eBook, “Six Common Causes of Poor Database Performance.” This comprehensive guide illuminates the frequent pitfalls and challenges that can hinder your database’s efficiency, offering insights into troubleshooting and optimizing for peak performance. Whether you’re a seasoned professional or new to database administration, this eBook is an essential tool for diagnosing and overcoming common obstacles. Download your copy today and take the first step towards ensuring your database systems run smoothly and efficiently, supporting your business’s growth and technological resilience.

DOWNLOAD eBOOK

Segmentation fault FAQs

1. How do I detect a segmentation fault in my application?

Identifying a segmentation fault usually means watching for abrupt terminations or irregular actions within your application. For an in-depth investigation, employing debugging tools such as GDB (GNU Debugger) is beneficial in pinpointing the precise code line causing the fault. Executing your application in a debugging environment pauses it at the segmentation fault, enabling you to analyze the call stack and variable states at the moment of the fault.

2. What are the common signs that a segmentation fault has occurred?

Common signs of a segmentation fault include your application terminating unexpectedly, often accompanied by an error message to the console such as “Segmentation fault (core dumped)” or simply “segfault.” These faults typically occur due to attempts to access memory that the application does not have permission to access, leading to an immediate crash.

3. How do I fix a segmentation fault once detected?

Fixing a segmentation fault involves identifying the line of code and the specific operation causing the access violation. After locating the problem with a debugger, examine the code for common mistakes like dereferencing null pointers, accessing out-of-bounds array elements,  or improper handling of memory allocation and deallocation. Correcting these errors usually involves ensuring proper initialization of pointers, bounds checking for arrays, and careful management of memory allocation.

4. Can segmentation faults pose security risks to my application?

Yes, segmentation faults can pose security risks, especially if they result from exploitable vulnerabilities in your application, such as buffer overflows. Attackers may leverage such faults to execute arbitrary code or cause a denial of service.

 

Subscribe
Notify of
guest

0 Comments
Inline Feedbacks
View all comments