Lesson 6.7Windows Processes and Callable Interfaces

What a DLL Function Promises Its Caller

Infer an exported function's parameters, calling convention, result, and responsibility for memory cleanup before wrapping it in a small typed API.

Lesson 6.7

Read or listen Print chapter or book

About 28 minutes

Cheatsheet for Lesson 6.7Key ideas, reference and questions

A one-page summary of “What a DLL Function Promises Its Caller” you can keep beside your notes. Plain Markdown.

Lesson 6.7 — What a DLL Function Promises Its Caller: cheatsheet

Remember: A callable address still needs a correct signature, memory ownership, and a lifetime agreement.
Also useful: Picture the ABI as labelled delivery slots: agreeing on the parcel means little if each side uses a different slot.

Key terms

interface reversing: Recovering that call contract is interface reversing.
unsafe: unsafe is honest: the compiler cannot prove that a raw address really points to a function with this exact contract.

Review

Q: Which evidence is most important before converting a GetProcAddress result into a Rust function pointer? A: Argument locations, calling convention, return use, ownership, and behavior seen at several call sites
Q: A library's export list shows a function called Fetch. What does that name alone tell you? A: Other code can look it up by name, but its arguments and ownership are still unknown

Download cheatsheet (.md)

The executable-analysis chapter showed how a DLL export name leads to a function address. Finding that address does not tell us how to call the function. Before our code becomes a library inside another process, we need the agreement between caller and callee: what values cross the boundary, where they arrive, and who owns any memory they refer to.

The earlier text lesson supplied one part of this agreement: a string pointer had to refer to the right encoding and remain valid for the call. A DLL export requires the same care even when its name is visible.

A function name is not a contract

A DLL export table may reveal a name such as gha_start, GetPlayer, or FormatMessage. That name only tells you that other code can locate the function. It does not fully explain:

which arguments the caller passes;
where those arguments arrive;
what the return value means;
which side owns pointed-to memory;
whether the call changes global state;
whether it is safe to call more than once.

Recovering that call contract is interface reversing. We will practice on this book’s own Windows DLL, where the source is available afterward for checking our conclusions.

Reconstruct four parts of every boundary

Treat a machine-level function boundary as four connected contracts:

Representation: How many bits does each value use, and is it an integer, float, pointer, flag set, or structure?
Transport: Which registers or stack locations carry arguments and results?
Ownership: Who allocates, may mutate, and eventually releases pointed-to data?
Lifetime: For how long do those addresses and returned views remain valid?

Calling conventions mainly answer transport. They do not answer ownership or lifetime. A function can correctly receive a pointer in rcx and still crash later because the caller freed the buffer too early. Likewise, a return value in rax might be a borrowed pointer, an owned allocation, a handle, or merely a status code.

Each part of the contract follows from something the machine code does. A callee that reads four bytes through the pointer and never stores it needs only a short read-only borrow. A callee that saves the pointer in a global needs the data to outlive the call.

A saved pointer can outlive the data it points toThe address in the saved pointer never changes. What is stored at that address does.

1 During the call: the pointer is valid
Saved pointer → its 4 bytes → read
0x0A401000saved pointer
64
00
00
00
100correct
The caller’s buffer (blue) holds 100 as 64 00 00 00.
2 After the call: freed and reused
Saved pointer → its 4 bytes → read
0x0A401000saved pointer
2C
01
00
00
not read
Other code (gray) now owns those bytes and stores 300 as 2C 01 00 00.
3 Later: the same pointer reads 300
Saved pointer → its 4 bytes → read
0x0A401000saved pointer
2C
01
00
00
300wrong data
The address is still mapped, so the read succeeds with the wrong value.

Begin with the caller

On 64-bit Windows, the first four integer or pointer arguments normally arrive in rcx, rdx, r8, and r9. A return value normally leaves in rax. On 32-bit Windows, several calling conventions exist, so also inspect stack cleanup and decorated export names.

Suppose a call site looks like this:

These instructions are the machine-level version of that path. Read them in order: clear ecx, call the export through its import slot, test the returned eax, and branch on a nonzero result.

xor ecx, ecx
call qword ptr [gha_start]
test eax, eax
jne start_failed

Read in plain English, before any types are guessed, the call site says:

The caller puts zero in the first argument register.
It makes an indirect call through an imported function slot.
It tests the low 32 bits of the result.
Zero takes the success path; nonzero takes the failure path.

Reading a call site: what the caller does with the result. This step-by-step trace needs JavaScript. Here is the code and where the run ends.

xor ecx, ecx
call qword ptr [gha_start]
test eax, eax
jne start_failed
; zero: the success path continues here

Execution falls through to the success path. From this one call site you can write the tentative contract: zero in the first argument, zero back for success.

Final values: ecx = 0 (first argument); eax = 0; ZF = 1.

Read every step of the lesson’s example

Step 1, line 1. xor ecx, ecx exclusive-ors a register with itself, which always gives zero. The caller is putting zero in the first argument register.
ecx = 0 (first argument).
Step 2, line 2. call goes through the import slot to gha_start and runs it. The function leaves its return value in eax; here it is 0.
ecx = 0 (first argument); eax = 0.
Step 3, line 3. test eax, eax combines eax with itself and records the outcome in the flags without changing eax. Zero gives ZF = 1.
ecx = 0 (first argument); eax = 0; ZF = 1.
Step 4, line 4. jne jumps if not equal, which reads ZF = 0. ZF is 1, so the jump is not taken.
ecx = 0 (first argument); eax = 0; ZF = 1.
Step 5, line 5. Execution falls through to the success path. From this one call site you can write the tentative contract: zero in the first argument, zero back for success.
ecx = 0 (first argument); eax = 0; ZF = 1.

That supports a tentative contract:

type GhaStart = unsafe extern "system" fn(argument: *mut core::ffi::c_void) -> u32;

extern "system" selects the platform’s Windows ABI—application binary interface, the complete machine-level agreement between compiled pieces: which registers or stack slots carry arguments, who cleans up the stack, and where the return value lands. It is the compiled counterpart to a source-level API. unsafe is honest: the compiler cannot prove that a raw address really points to a function with this exact contract.

Where each part of the contract shows up

A decompiler’s guessed types are only a starting point. Each question about the contract has a place in the machine code that settles it:

Where each part of the contract shows up

Question	Where the answer shows up
Argument count	Registers or stack slots prepared at several call sites
Argument type	How the callee reads it: dereference, arithmetic, comparison, or API pass-through
Result meaning	Branches and messages immediately after the call
Buffer length	Bounds checks, loop limits, and paired size arguments
Ownership	Allocation/free calls and whether the caller stores the pointer
Repeatability	Global flags, reference counts, or “already started” branches

One call with a null argument does not prove the argument must always be null. Find a second caller or change the controlled test program.

Separate lookup from calling

GetProcAddress returns an untyped address. Keep the dangerous conversion in one function:

The resolver below takes the loaded module handle, asks for the exact gha_start export, and converts the returned address to one typed function pointer. A missing export must return an error; a found address is still safe to call only if the contract inferred above is correct for this DLL build.

use core::ffi::c_void;
use windows::{
    Win32::System::LibraryLoader::{GetProcAddress, HMODULE},
    core::PCSTR,
};

type GhaStart = unsafe extern "system" fn(*mut c_void) -> u32;

unsafe fn resolve_gha_start(module: HMODULE) -> anyhow::Result<GhaStart> {
    // SAFETY: the byte string is NUL-terminated and lives for this call.
    let raw = unsafe { GetProcAddress(module, PCSTR(b"gha_start\0".as_ptr())) }
        .ok_or_else(|| anyhow::anyhow!("DLL does not export gha_start"))?;

    // SAFETY: our debugger evidence and matching test DLL establish this ABI.
    Ok(unsafe { core::mem::transmute::<unsafe extern "system" fn() -> isize, GhaStart>(raw) })
}

Why not scatter transmute through the program? Because every conversion is a promise about ABI and types. One narrow boundary gives reviewers one place to check that promise.

Wrap machine results in named variants

The course DLL returns:

0 when the worker started;
1 when it was already running;
2 when the worker thread could not start.

Give those numbers names:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum StartStatus {
    Started,
    AlreadyRunning,
    ThreadCreationFailed,
    Unknown(u32),
}

impl From<u32> for StartStatus {
    fn from(raw: u32) -> Self {
        match raw {
            0 => Self::Started,
            1 => Self::AlreadyRunning,
            2 => Self::ThreadCreationFailed,
            value => Self::Unknown(value),
        }
    }
}

Keep Unknown. A future DLL can add a status without making your wrapper silently call it success.

Prove the calling convention

Windows VM

A source-level API can say that a function accepts one address and returns a number. The ABI answers the machine-level question of where that address arrives. On a 32-bit build, one convention might put it in a register while another expects it on the stack. The functions can look identical at the source level, yet the callee reads the wrong location if caller and callee disagree.

A wrong ABI can appear to work in one small test and still corrupt registers or the stack. The call itself may return without crashing; the damage shows up later, when an unrelated function reads garbage from a register it expected the callee to preserve, or when a mismatched stack-cleanup convention leaves esp off by a few bytes and a later ret lands somewhere nonsensical. Check:

Which architecture is the caller and DLL?
Who removes stack arguments on 32-bit builds?
Does the callee return with a plain ret or ret N?
Does a decorated name encode argument bytes?
Do several calls return with the stack pointer unchanged?

For x64 Windows there is one main user-mode convention, but argument type and ownership can still be wrong.

Test with a contract matrix

Test against the DLL from windows-labs rather than an unknown third-party export. Each test has an expected result:

Test	Expected result
First call with null	Returns `Started`
Second call	Returns `AlreadyRunning`
Unknown export name	Resolver returns an error; no call occurs
Wrong architecture	Windows refuses to load the DLL
Stop after start	Export returns and worker observes the stop flag

The source in windows-labs/src/windows_impl/dll.rs then shows the real contract, and any difference from the recovered one points at the exact part of the machine code that was misread.

Working practices for an undocumented DLL

Never guess a function pointer type and “try it” in an important process.
Use the disposable course fixture; a contract recovered from it applies to that exact build.
Treat every pointer’s size, mutability, lifetime, and ownership as a separate question.
Preserve an Unknown result path.
Keep loading, resolving, converting, calling, and unloading as visible stages.

This same method works for undocumented game plug-ins, supported mod SDKs, and old middleware. The goal is not merely to make one call succeed. The goal is to explain the interface well enough that a small wrapper can enforce it.

The contract gives a function address meaning: it tells a caller which values are allowed and which obligations survive the call. In-Process DLLs and Loader Boundaries builds the small DLL used here. Its explicit start and stop functions turn the recovered agreement into an interface whose loading, work, and cleanup can be followed in order.

Which evidence is most important before converting a GetProcAddress result into a Rust function pointer?

Finished Lesson 6.7?

Mark it done and the lesson list shows a beside it. A chapter whose lessons are all done is marked done too. Progress is saved in this browser only.

Chapter 6: 11 lessons

This book grew out of the original Game Hacking Academy by attilathedud and has been rewritten as a beginner-first course. Like the original repository, its source is published under the Apache License 2.0.