Finding Memory Corruption
Vulnerability Discovery Case Studies · VR Analysis & Discovery track
8 min read
In this module
A Complete Walkthrough: Heap Buffer Overflow Discovery
This lesson walks through finding a heap buffer overflow in a hypothetical but realistic image processing library. Source auditing and fuzzing together — demonstrating how the techniques from Introduction to Vulnerability Research through Advanced Fuzzing Techniques interact in practice. The target is a C library that decodes a custom image format, structurally similar to libpng, libjpeg, or libtiff.
Step 1: Target Selection
Our target, libimgdec, decodes images in a custom format. We chose it because:
- It processes untrusted input — image files from the network or disk
- It's written in C — memory safety depends on the developer getting every check right
- It has a parsing component — file format parsers are historically abundant with bugs
- The code is open-source — we can combine source audit with fuzzing
Step 2: Understanding the Format
Before auditing, understand what the code processes. The image format uses a chunk-based structure (similar to PNG):
[File Header: 8 bytes magic][Chunk: type(4) | length(4) | data(length) | checksum(4)][Chunk: type(4) | length(4) | data(length) | checksum(4)]...[End Chunk]Each chunk has a 4-byte type, a 4-byte length field, variable-length data, and a 4-byte checksum. The length field specifies how many bytes of data follow - and it comes from the untrusted file.
Step 3: Source-Sink Analysis
We apply the source-sink analysis technique from Source Code Auditing. First, identify the sources:
// SOURCE: file data enters heresize_t imgdec_read_file(imgdec_ctx *ctx, const char *filename) { FILE *f = fopen(filename, "rb"); fseek(f, 0, SEEK_END); ctx->file_size = ftell(f); fseek(f, 0, SEEK_SET); ctx->file_data = malloc(ctx->file_size); fread(ctx->file_data, 1, ctx->file_size, f); // All file data is TAINTED fclose(f); return ctx->file_size;}Now trace the tainted data through chunk parsing:
int parse_chunk(imgdec_ctx *ctx, size_t offset) { // Read chunk header from tainted file data uint32_t chunk_type = read_u32(ctx->file_data + offset); // TAINTED uint32_t chunk_length = read_u32(ctx->file_data + offset + 4); // TAINTED
// Allocate buffer for chunk data uint8_t *chunk_data = malloc(chunk_length); // TAINTED size! if (!chunk_data) return -1;
// Copy chunk data memcpy(chunk_data, ctx->file_data + offset + 8, chunk_length); // SINK
// Process based on type switch (chunk_type) { case CHUNK_IMAGE_DATA: return process_image_data(ctx, chunk_data, chunk_length); case CHUNK_PALETTE: return process_palette(ctx, chunk_data, chunk_length); // ... } free(chunk_data); return 0;}Step 4: Identifying the Vulnerability
The taint analysis reveals a critical issue. chunk_length is read directly from the file (tainted) and used in two dangerous ways:
- As the size argument to malloc() - if
chunk_lengthis very large, malloc could fail (but the return is checked) - As the length argument to memcpy() - and here is the real problem
The memcpy copies chunk_length bytes starting from offset + 8 in the file buffer. But what if chunk_length exceeds the remaining bytes in the file? The code never checks:
// MISSING CHECK:// chunk_length could be larger than (ctx->file_size - offset - 8)// This means memcpy reads past the end of ctx->file_dataThis is a heap buffer over-read: the memcpy reads past the end of the file data buffer. Depending on what is in the heap after file_data, this could:
- Read sensitive data from adjacent heap allocations (information disclosure)
- Trigger a segfault if it reads into unmapped memory (crash/DoS)
But there is a worse issue. Let us look at process_image_data:
int process_image_data(imgdec_ctx *ctx, uint8_t *data, uint32_t length) { // Image dimensions from a previous chunk uint32_t width = ctx->width; // Set from earlier HEADER chunk uint32_t height = ctx->height; // Set from earlier HEADER chunk
// Calculate pixel buffer size size_t pixel_buf_size = width * height * 4; // RGBA, 4 bytes per pixel uint8_t *pixels = malloc(pixel_buf_size);
// Decode compressed data into pixel buffer decompress(data, length, pixels, pixel_buf_size);
ctx->pixels = pixels; return 0;}The width * height * 4 calculation can integer overflow. If width = 0x10000 and height = 0x10000, then width * height = 0x100000000 which wraps to 0 in 32-bit arithmetic. malloc(0) returns either NULL or a minimal allocation that can't hold a single byte of image data, and decompress writes a full image into it - a massive heap buffer overflow.
Try the lesson's dimensions below, then the 32769 × 32768 preset, which wraps to a perfectly plausible 128 KiB allocation that no sanity check on "zero" would catch. Switch the arithmetic type to 64-bit and watch the bug disappear.
Integer overflow calculator
Size a pixel buffer with width * height * bpp and see what malloc really gets.
True product (needed)
17,179,869,184
16 GiB, 0x400000000
C result as uint32_t
0 (0x00000000)
Wrapped modulo 2^32: only the low 32 bits survive.
malloc receives (size_t, 64-bit)
0
malloc(0) returns NULL or a tiny chunk; either way, no room for pixels.
Copy writes past the end
17,179,869,184 bytes
16 GiB of heap overwritten (the decoder stops when it hits unmapped memory).
The check that stops it
if (bpp == 0 || (width && height > SIZE_MAX / width / bpp))
return IMGDEC_ERROR_OVERFLOW;
size_t size = (size_t)width * height * bpp;
/* or let the compiler check each step */
size_t tmp, size;
if (__builtin_mul_overflow(width, height, &tmp) ||
__builtin_mul_overflow(tmp, bpp, &size))
return IMGDEC_ERROR_OVERFLOW;With these inputs the SIZE_MAX guard lets the image through. A guard against uint32_t's maximum (4,294,967,295) would reject it, which is the check the original code needed if it insisted on uint32_t maths. Note the cast to size_t before multiplying.
Real vulnerabilities often involve two or more bugs working together. Here, the integer overflow in the size calculation (bug 1) creates an undersized allocation, and the decompression writes more data than allocated (bug 2, a consequence of bug 1). Neither bug alone might be exploitable - the overflow alone just creates a small allocation, and the decompression alone would be fine with a correctly-sized buffer. Together, they create a heap buffer overflow.
Step 5: Building a Fuzz Harness
With the vulnerability hypothesis from source audit, we build a targeted fuzz harness:
#include <stdint.h>#include <stddef.h>#include "imgdec.h"
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { if (size < 16) return 0; // Need at least header + one chunk
imgdec_ctx *ctx = imgdec_create(); imgdec_decode_memory(ctx, data, size); imgdec_destroy(ctx); return 0;}Compile with ASan to catch the corruption:
clang -fsanitize=fuzzer,address -g -O1 \ harness.c -limgdec -o harness_fuzz
# Create seed corpus with valid imagesmkdir corpus/cp test_images/*.img corpus/
# Run the fuzzer./harness_fuzz corpus/ -dict=imgformat.dictStep 6: Confirming with ASan
Within minutes, the fuzzer produces crashes. ASan reports confirm our analysis:
==54321==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000011WRITE of size 131072 at 0x602000000011 thread T0 #0 0x4a5678 in decompress src/decompress.c:234 #1 0x4a3456 in process_image_data src/imgdec.c:189 #2 0x4a2345 in parse_chunk src/imgdec.c:142
0x602000000011 is located 0 bytes after 1-byte region[0x602000000010,0x602000000011)allocated by thread T0 here: #0 0x... in malloc #1 0x4a3400 in process_image_data src/imgdec.c:185ASan confirms: the allocation at imgdec.c:185 (the malloc(pixel_buf_size) where pixel_buf_size wrapped to 0) is overflowed by the decompression at decompress.c:234. ASan reports a malloc(0) as a 1-byte region, so don't be thrown by the size.
Step 7: Minimizing the PoC
The fuzzer's crash input is 2KB. Minimise it:
The harness was built for libFuzzer, so use libFuzzer's own minimiser (afl-tmin is the equivalent for an AFL++ build):
./harness_fuzz -minimize_crash=1 -runs=100000 \ -exact_artifact_path=minimal_crash crash_input# Result: 48 bytes - just the header, a dimension chunk with overflow values,# and a small image data chunkThe minimised input demonstrates the vulnerability with the smallest possible file: a file header, a header chunk specifying dimensions that cause integer overflow, and an image data chunk.
Step 8: Writing the Report
With confirmed vulnerability, minimised PoC, and root cause identified, write the vulnerability report following the practices from Responsible Disclosure & UK Law: Writing Security Advisories:
Summary: Integer overflow in pixel buffer size calculation in process_image_data() leads to heap buffer overflow when processing crafted image files.
Root cause: width * height * 4 uses 32-bit arithmetic and can overflow to a small value, causing malloc() to allocate an undersized buffer. The subsequent decompress() call writes the full image data into this buffer.
Fix suggestion: Check for overflow before the multiplication, or use safe integer arithmetic:
// Safe multiplication check: reject zero dimensions first// (they'd also divide by zero below), then do the maths in 64 bitsif (width == 0 || height == 0) return IMGDEC_ERROR_BAD_DIMENSIONS;if ((uint64_t)width * height > SIZE_MAX / 4) return IMGDEC_ERROR_OVERFLOW;size_t pixel_buf_size = (size_t)((uint64_t)width * height * 4);