Skip to main content
Be Bitwise

DLLs & Dynamic Loading

Windows API - GUI & System Programming · Windows Programming track

9 min read

In this module

DLLs are how Windows shares code. Every Win32 call you've ever made — MessageBox, CreateFile, the lot — punches through a DLL on its way to the kernel. Loaders, exports, imports, the search order that's caused a thousand CVEs: this lesson is the user-mode side of how a Windows process is actually wired together.

DLL Fundamentals

A DLL (Dynamic Link Library) is a PE — Portable Executable — file holding code and data that multiple programs can share. Static libraries get welded into each binary at link time and bloat every executable that uses them. DLLs sit on disk once, get loaded into memory once per file, and have their pages shared (read-only) across every process using them.

DllMain: The Entry Point

Every DLL can have an optional entry point called DllMain:

c
BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason, LPVOID lpReserved) {
switch (fdwReason) {
case DLL_PROCESS_ATTACH:
// DLL loaded into a process
// Called once per process
// hinstDLL = base address of the DLL
break;
case DLL_THREAD_ATTACH:
// A new thread was created in the process
break;
case DLL_THREAD_DETACH:
// A thread is exiting cleanly
break;
case DLL_PROCESS_DETACH:
// DLL is being unloaded
// lpReserved != NULL means process is terminating
// lpReserved == NULL means FreeLibrary was called
break;
}
return TRUE; // Return FALSE to fail DLL loading
}

Critical rules for DllMain:

  • Do NOT call LoadLibrary or LoadLibraryEx (can cause deadlock due to loader lock)
  • Do NOT call functions from other DLLs (they may not be loaded yet)
  • Do NOT create threads (they will block waiting for DllMain to return)
  • Do NOT use synchronisation primitives that wait on other threads
  • Keep it minimal: initialise per-process or per-thread state only
The Loader Lock

DllMain is called while the Windows loader holds a process-wide lock called the loader lock. Any operation that requires the loader (loading other DLLs, resolving imports) will deadlock. This is one of the most common pitfalls in Windows DLL development. Complex initialisation should be deferred to a separate initialisation function that the application calls after the DLL is loaded.

Exporting Functions

c
// Method 1: __declspec(dllexport) in source code
__declspec(dllexport) int Add(int a, int b) {
return a + b;
}
// Method 2: .def file (more control over names and ordinals)
// mylib.def:
// LIBRARY mylib
// EXPORTS
// Add @1
// Subtract @2
// _InternalFunc @3 NONAME ; Export by ordinal only

The PE export table maps function names — and ordinal numbers, an older mechanism still used by Windows itself — to addresses inside the DLL. Run dumpbin /EXPORTS mylib.dll to see what a DLL is offering the world.

Implicit vs Explicit Linking

Implicit Linking (Load-Time)

The default. Your linker records which DLLs you need; the Windows loader resolves the addresses at process startup and patches up your import table before main() runs:

c
// In your code, just call the function
#include "mylib.h"
int result = Add(3, 4);
// Compile: cl app.c /link mylib.lib
// mylib.lib is an import library (not the DLL itself)
// It contains stubs that redirect to the DLL at runtime

Explicit Linking (Run-Time / Dynamic Loading)

When you don't know at compile time which DLL you need — plugins, optional features, undocumented APIs that may or may not exist on this Windows version — LoadLibrary and GetProcAddress let you wire it up by hand at runtime:

c
// Load the DLL at runtime
HMODULE hMod = LoadLibrary(TEXT("mylib.dll"));
if (!hMod) {
DWORD err = GetLastError();
// DLL not found or failed to load
return;
}
// Get function address by name
typedef int (*AddFunc)(int, int);
AddFunc pAdd = (AddFunc)GetProcAddress(hMod, "Add");
if (pAdd) {
int result = pAdd(3, 4); // Call the function
}
// Get function by ordinal
typedef int (*SubFunc)(int, int);
SubFunc pSub = (SubFunc)GetProcAddress(hMod, MAKEINTRESOURCEA(2));
// Unload when done
FreeLibrary(hMod);

Plugin systems live or die on this pattern. So does any code that has to gracefully handle "this function might not exist on Windows 7."

DLL Search Order

Hand LoadLibrary a bare filename — no path — and Windows starts hunting through directories in a fixed order:

text
1. Directory containing the application executable
2. System directory (C:\Windows\System32)
3. 16-bit system directory (C:\Windows\System)
4. Windows directory (C:\Windows)
5. Current directory
6. Directories in the PATH environment variable

DLL Hijacking

Bare-filename loads are an attack surface. Drop a malicious DLL into a directory that gets searched earlier than the legitimate one and the application happily loads your code instead:

text
Scenario: app.exe loads "helper.dll" (expected in System32)
Attack:
1. Attacker places malicious "helper.dll" in app's directory
2. app.exe calls LoadLibrary("helper.dll")
3. Search order: app directory is checked FIRST
4. Malicious DLL is loaded instead of the real one
5. DllMain runs attacker's code with app's privileges

Mitigations:

  • Use full paths: LoadLibrary(L"C:\\Windows\\System32\\helper.dll")
  • Use SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32)
  • Use SetDllDirectory(L"") to remove current directory from search
  • The KnownDLLs registry key protects common system DLLs
Key Concept
DLL Search Order Hijacking

DLL hijacking exploits the predictable order in which Windows searches for DLLs. By placing a malicious DLL in a higher-priority directory (typically the application's own directory), an attacker can execute code in the context of the loading application. This is used for privilege escalation (hijacking a DLL loaded by a privileged service), persistence (placing DLLs in autostart application directories), and lateral movement.

COM: The Component Object Model

COM is the binary component-sharing framework that runs underneath half of Windows. An object written in C++ can be called from C# or VB or PowerShell; an object running in another process (or another machine, with DCOM) looks the same to your code as one in your own address space. The abstraction is the point.

The Three Pillars of COM

  1. Interfaces: COM objects expose functionality through interfaces (pure virtual method tables). Every interface inherits from IUnknown.
  2. Reference Counting: Objects are kept alive by AddRef/Release reference counting.
  3. Registration: COM classes are registered in the registry (HKCR\CLSID) with their implementing DLL or EXE.

IUnknown: The Root Interface

c
// Every COM interface inherits from IUnknown
interface IUnknown {
HRESULT QueryInterface(REFIID riid, void **ppvObject);
ULONG AddRef(void);
ULONG Release(void);
};
  • QueryInterface: Ask the object if it supports a specific interface
  • AddRef: Increment reference count
  • Release: Decrement reference count (destroy when it reaches 0)

Creating COM Objects

c
// Initialize the COM library
HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
// Create an instance of a COM class
IShellLink *psl = NULL;
hr = CoCreateInstance(
&CLSID_ShellLink, // Which class to create
NULL, // No aggregation
CLSCTX_INPROC_SERVER, // In-process DLL server
&IID_IShellLink, // Which interface we want
(void **)&psl // Output pointer
);
if (SUCCEEDED(hr)) {
// Use the interface
psl->lpVtbl->SetPath(psl, L"C:\\target.exe");
psl->lpVtbl->SetDescription(psl, L"My Shortcut");
// Query for another interface on the same object
IPersistFile *ppf = NULL;
hr = psl->lpVtbl->QueryInterface(psl, &IID_IPersistFile, (void **)&ppf);
if (SUCCEEDED(hr)) {
ppf->lpVtbl->Save(ppf, L"C:\\Users\\Public\\Desktop\\shortcut.lnk", TRUE);
ppf->lpVtbl->Release(ppf);
}
psl->lpVtbl->Release(psl);
}
CoUninitialize();

COM Registration

COM classes are registered in the registry:

text
HKCR\CLSID\{00021401-0000-0000-C000-000000000046}\
(Default) = "Shortcut"
InprocServer32\
(Default) = "C:\Windows\System32\shell32.dll"
ThreadingModel = "Apartment"

Call CoCreateInstance and the runtime walks the registry, finds the DLL path keyed by your CLSID, loads it, and asks it for the requested object via DllGetClassObject. That registry lookup is also another reason COM has historically been a happy hunting ground for attackers — registry hijacks, type-library tricks, and surrogate processes all start by abusing this lookup.

COM is everywhere in Windows. The Shell uses it. So does DirectX, WMI, Active Directory, the Task Scheduler, and most of the system services that talk to each other. Get comfortable with IUnknown, QueryInterface, and reference counting, and a huge chunk of Windows internals stops looking like dark magic.