Skip to content
Lesson 12.5Loaders, Kernel Services, and Trust Boundaries

From Win32 Calls to Kernel Services

A Windows API call crosses layers of exports, native APIs, and system calls before a kernel service acts on a request.

Lesson 12.5
Read or listenPrint chapter or book
About 28 minutes

Keyboard shortcuts

?
Open this guide
Alt + B
Show or hide the lesson list
Alt + O
Show or fold the lesson outline
Alt + N
Open or close notes
Alt + H
Toggle semantic colours
Alt + D
Toggle page brightness
Alt + C
Toggle code brightness
← →
Move through a focused code trace
Home End
First or final step in a focused trace
Escape
Close this guide or an open panel

Reader shortcuts pause while you type in a text field.

Cheatsheet for Lesson 12.5Key ideas, reference and questions

A one-page summary of “From Win32 Calls to Kernel Services” you can keep beside your notes. Plain Markdown.

Lesson 12.5 — From Win32 Calls to Kernel Services: cheatsheet

  • Remember: A documented Windows function can keep its contract while the DLLs and internal layers implementing it change.
  • Also useful: A handle's size does not tell you who owns it. Check the API that produced it and its documented release rule; this lab's borrowed module handle must not be released with FreeLibrary.

Key terms

  • software layers: The boxes are software layers, not necessarily four separate processes.
  • forwarded export: A forwarded export is not a jump instruction the loader inserts.
  • A: In paired string APIs, W accepts UTF-16 code units and A accepts 8-bit text in the API's ANSI encoding; A is not a promise of ASCII or UTF-8.

Review

  • Q: What happens conceptually when a user-mode API needs the Windows kernel to perform privileged work? A: A controlled system-call transition enters a kernel service, then returns to user mode
  • Q: Which three ideas describe how a documented API call reaches the kernel? A: A contract, a library path, and a privilege transition
Download cheatsheet (.md)

One function call can cross several layers

Section titled “One function call can cross several layers”

The course tool can now name a process, open and own a limited handle, inspect page metadata, and observe threads. Each operation was a call to a Windows API. The remaining question is where such a call goes when the tool asks the operating system to do something it cannot do directly.

A call to a Windows function such as VirtualQueryEx does not instantly jump into the deepest part of Windows. It usually passes through layers.

Application caller
Documented Win32 contract
User-mode DLL implementation
Native service stub
System-call privilege gate
Kernel validates the request
Result returns to the caller

Use three separate ideas: contract, implementation path, and privilege transition. Win32 documents the contract application code may rely on. User-mode DLLs implement or forward that contract. A system-call instruction performs the controlled CPU transition for work that requires kernel privilege.

The boxes are software layers, not necessarily four separate processes. kernel32.dll and ntdll.dll are mapped into the same user-mode process as our program. Microsoft may change the implementation path while preserving the Win32 contract, which is why “calling a deeper layer” is usually less stable rather than more capable.

A call passes through layers in one process before the kernelThe application, the Win32 DLL and the ntdll stub all run in the same user-mode process. Only a controlled gate reaches the kernel.
  1. Request Call a Win32 function

    Step 1: Application to Win32 DLL

    The application calls an ordinary documented function.

  2. Request Call the native stub

    Step 2: Win32 DLL to ntdll stub

    The DLL’s implementation calls a lower-level stub in ntdll. Still the same process, still user mode.

  3. Encrypted Controlled system-call gate

    Step 3: ntdll stub to Kernel

    The stub crosses into the kernel through a gate the CPU controls. This is the only privilege change.

  4. Alone Validate the request

    Step 4: Kernel (alone)

    The kernel checks the arguments and the caller’s rights before acting.

  5. Reply Result

    Step 5: Kernel to ntdll stub

    The kernel returns a result or an error code.

  6. Reply Return

    Step 6: ntdll stub to Win32 DLL

    The stub hands the result back up.

  7. Reply Return to the caller

    Step 7: Win32 DLL to Application

    The Win32 function turns it into the value the application expects.

The application, the Win32 DLL and the ntdll stub all run in the same user-mode process. Only a controlled gate reaches the kernel.

A Win32 call such as VirtualQueryEx passes down through user-mode DLLs and the system-call gate before the kernel validates it. Calling a deeper layer is usually less stable, not more capable. The interactive version needs JavaScript; the lesson text around it explains the same idea.

User mode and kernel mode are permission levels

Section titled “User mode and kernel mode are permission levels”

Most game code, application code, and Windows DLL code runs in user mode. A user-mode mistake should not be able to overwrite the kernel or another process merely by using a bad pointer.

The Windows kernel and drivers run in kernel mode, where code can manage page tables, devices, scheduling, and other system-wide resources. A bug there can affect the whole computer. That is why the course does not turn ordinary game experiments into kernel-driver experiments.

The CPU enforces this border. A system call changes privilege through a defined entry point; it is not a magic function name and it is not the same thing as calling a normal function.

This design is often described as limited direct execution. The game normally runs directly on the CPU for speed, but in user mode and within its address-space permissions. Timers and interrupts let Windows regain control for scheduling. A system call lets the game request a restricted operation through a validated entry path.

The model resolves a common confusion: Windows is not interpreting every ordinary game instruction. Most instructions execute natively. Hardware privilege, address translation, interrupts, and controlled entries provide the limits that let the OS remain in control.

Win32 is the documented application interface

Section titled “Win32 is the documented application interface”

The documented Win32 API is the part application developers are normally expected to call. Examples from this course include:

  • ReadProcessMemory for a read-only inspector with query-and-read rights;
  • VirtualQueryEx for page metadata;
  • CreateFileW for files and local named pipes;
  • CreateFileMappingW for a shared-memory object.

Microsoft can change how a supported function is implemented while keeping its documented behavior. On modern Windows, an export in kernel32.dll may forward to another DLL such as KernelBase.dll. Your source code still calls the stable Win32 name.

A forwarded export is not a jump instruction the loader inserts. The export table entry itself can hold a string such as KERNELBASE.CreateFileW in place of a code address. When GetProcAddress resolves that entry, it reads the string, locates the named DLL, and looks up the function there instead — following another forward if that entry also happens to be one. Whoever asked for kernel32!CreateFileW never sees this redirection; they get back a usable address in KernelBase.dll either way, which is exactly why Microsoft can move an implementation between DLLs across Windows versions without breaking the caller’s import:

GetProcAddress(kernel32,
'CreateFileW')
kernel32's export table entry:
the string 'KERNELBASE.CreateFileW',
not a code address
loader locates KernelBase.dll
and looks up CreateFileW there
returns a usable address
in KernelBase.dll

ntdll.dll exposes many functions whose names begin with Nt, such as NtQueryVirtualMemory. Some native functions are documented for limited uses and many are not stable application contracts.

For ordinary tools, prefer documented interfaces:

“Closer to the kernel” does not mean “better for normal application code.”

Calling an undocumented entry point or copying a system-call number from one Windows build can make a tool fragile. Numbers and internal structures can change. The course resolves one native export so you can see the layer, but it deliberately does not call it or build a direct-syscall bypass.

DLL exports are names offered to other code

Section titled “DLL exports are names offered to other code”

A DLL can publish an export table. Each exported name maps to an entry that the loader can resolve. GetProcAddress asks a loaded module for one of those exports.

The returned address is a virtual address in the current process. Address Space Layout Randomization, or ASLR, means the same DLL can be mapped at a different base address in another process or after a reboot. Store the name or an RVA when that is the stable identity; do not paste a displayed absolute address into tomorrow’s program.

The lab gets handles to DLLs already loaded in its own process, resolves three names, and prints their process-local addresses. It never opens a game process and never calls the native function:

GetModuleHandleW('kernel32.dll')
exported_address:
VirtualQueryEx, ReadFile
GetModuleHandleW('ntdll.dll')
exported_address:
NtQueryVirtualMemory
print layer, name,
address in this run
Complete lab source: api_layers.rs

api_layers.rs

windows-labs/src/bin/ · Rust, 58 lines

#[cfg(windows)]
fn main() -> anyhow::Result<()> {
use anyhow::Context;
use windows::{
Win32::{
Foundation::HMODULE,
System::LibraryLoader::{GetModuleHandleW, GetProcAddress},
},
core::{PCSTR, s, w},
};
fn exported_address(module: HMODULE, name: PCSTR) -> anyhow::Result<usize> {
// SAFETY: module is loaded in this process and name is a terminated,
// static ASCII string. We only print the returned address.
let function = unsafe { GetProcAddress(module, name) }
.context("the requested export was not present")?;
Ok(function as usize)
}
// SAFETY: these DLL names are terminated, static UTF-16 strings. Windows
// loads both modules before a normal console program reaches main.
let kernel32 = unsafe { GetModuleHandleW(w!("kernel32.dll")) }?;
// SAFETY: ntdll.dll is likewise a terminated, static UTF-16 name for a
// module that is already loaded in every normal Windows process.
let ntdll = unsafe { GetModuleHandleW(w!("ntdll.dll")) }?;
let rows = [
(
"Win32",
"kernel32!VirtualQueryEx",
exported_address(kernel32, s!("VirtualQueryEx"))?,
),
(
"Win32",
"kernel32!ReadFile",
exported_address(kernel32, s!("ReadFile"))?,
),
(
"Native",
"ntdll!NtQueryVirtualMemory",
exported_address(ntdll, s!("NtQueryVirtualMemory"))?,
),
];
println!("Layer Export Address in this run");
for (layer, name, address) in rows {
println!("{layer:<8} {name:<35} {address:#018x}");
}
println!();
println!("These addresses belong only to this process and this run.");
println!("The lab resolved names; it did not call a native API or direct syscall.");
Ok(())
}
#[cfg(not(windows))]
fn main() {
eprintln!("This Windows API-layer lab must run on Windows.");
}

Why GetModuleHandleW instead of LoadLibraryW?

Section titled “Why GetModuleHandleW instead of LoadLibraryW?”

Both DLLs are already present in a normal Windows console process. GetModuleHandleW borrows their existing module handles; it does not request another load or add a loader reference that we would later need to release.

Why does GetProcAddress use an ASCII name?

Section titled “Why does GetProcAddress use an ASCII name?”

PE export names are byte strings. The s! macro creates a terminated static PCSTR. DLL file names use Windows wide strings, so GetModuleHandleW receives values from the w! macro instead.

This original attempt mixes byte strings and wide strings. It is intentionally a non-compiling example: the Windows bindings reject the wrong parameter types before any call runs.

Before: rejected string widths
fn resolve_virtual_query() -> windows::core::Result<usize> {
// SAFETY: static terminated strings outlive lookup; widths must match the API.
let kernel32 = unsafe { GetModuleHandleW(s!("kernel32.dll")) }?;
// SAFETY: the borrowed module stays loaded during this export lookup.
let address = unsafe { GetProcAddress(kernel32, w!("VirtualQueryEx")) };
address.ok_or_else(windows::core::Error::from_win32).map(|function| function as usize)
}

Use w! for the wide DLL filename and s! for the byte-encoded export name. Both strings are static and terminated; the DLL handle is borrowed for the lookup.

In this version, FARPROC already means an optional generic function pointer. ok_or_else removes that Option wrapper; the helper returns its address as usize for display. It does not call that pointer as VirtualQueryEx. The version-pinned type and lookup signature explain this return-type choice.

Improved code

Optional: practise matching the full API names, module argument and address type to their jobs.

fn resolve_virtual_query() -> windows::core::Result<usize> {
// SAFETY: static terminated strings outlive lookup; widths must match the API.
let kernel32 = unsafe { GetModuleHandleW(w!("kernel32.dll")) }?;
// SAFETY: the borrowed module stays loaded during this export lookup.
let address = unsafe { GetProcAddress(kernel32, s!("VirtualQueryEx")) };
address.ok_or_else(windows::core::Error::from_win32).map(|function| function as usize)
}

Show the diff

fn resolve_virtual_query() -> windows::core::Result<usize> {
// SAFETY: static terminated strings outlive lookup; widths must match the API.
let kernel32 = unsafe { GetModuleHandleW(s!("kernel32.dll")) }?;
let kernel32 = unsafe { GetModuleHandleW(w!("kernel32.dll")) }?;
// SAFETY: the borrowed module stays loaded during this export lookup.
let address = unsafe { GetProcAddress(kernel32, w!("VirtualQueryEx")) };
let address = unsafe { GetProcAddress(kernel32, s!("VirtualQueryEx")) };
address.ok_or_else(windows::core::Error::from_win32).map(|function| function as usize)
}

Takeaway: the API’s string type determines the encoding, even when both arguments look like ordinary names.

The types make the two encodings visible instead of silently mixing them.

Read the Windows signature before translating it

Section titled “Read the Windows signature before translating it”

Microsoft’s C documentation uses names such as DWORD, BOOL, and HANDLE to describe the API’s binary contract. Match that contract rather than choosing a Rust type just because its name sounds similar:

Windows spelling What it means at the boundary Rust reading rule
DWORD Unsigned 32-bit integer, even in a 64-bit process Usually u32; it does not grow into usize.
BOOL Signed 32-bit integer Boolean value Raw i32, or the binding’s projected type or result; Rust bool has a different representation. For Boolean truth, test nonzero rather than equality to 1.
HANDLE Opaque, pointer-sized object handle Use the binding’s handle type; it is not a Rust reference that you may dereference.

These distinctions come from Windows coding conventions and Windows data types. Read each function’s return-value section too: its success and failure rules can be more specific than its return type.

The lab’s string-width mistake is one instance of a broader convention. In paired string APIs, W accepts UTF-16 code units and A accepts 8-bit text in the API’s ANSI encoding; A is not a promise of ASCII or UTF-8. C headers may select an unsuffixed name using the UNICODE macro. Calling the explicit W function keeps the choice visible in Rust. See Working with Strings.

The ABI, or application binary interface, also specifies how arguments and results pass between caller and callee. WINAPI and CALLBACK in C signatures mark a calling convention, not an access right. Generated bindings supply the matching declarations. For a callback declared with that convention, match its exact signature and Rust’s extern "system" ABI: its underlying convention depends on the target, including stdcall on 32-bit Windows for non-variadic functions.

The GetModuleHandleW contract is the ownership rule behind that note. Likewise, a string pointer must stay valid for as long as the receiving API uses it; a pointer wrapper alone does not own its storage.

The crate names describe different jobs in the Rust bindings. They do not change which Windows layer runs a call or what the caller may access.

Crate Role
windows Generated projected APIs for Win32, COM, and WinRT, with typed values and conversions. This is the lab’s API crate.
windows-sys Raw Windows ABI declarations; the caller manages pointer contracts, ownership, and error translation.
windows-core Shared COM and WinRT interface types, traits, and projection support.
windows-strings Windows string representations, including owning strings and borrowed pointer wrappers.
windows-result Windows status and error types, including HRESULT and Result.

The generation tools have separate jobs: windows-bindgen turns Windows metadata into Rust bindings; windows-metadata reads and writes that metadata; windows-clang extracts C/C++ header declarations using libclang. The lab consumes pre-generated bindings. Add support or generation crates only when your own task needs them; many support types are already re-exported by the API crate.

One similarly named crate, windows-reference, boxes a value for a WinRT signature that requires IReference<T>. That is a Windows interface object, distinct from ordinary Rust borrowing with &T. The address inspector needs no such boxed value.

Windows VM

The program prints exported function names beside their addresses in this process. Its first run establishes a local observation; the second run tests whether those addresses are safe to treat as permanent identifiers.

Terminal window
cd windows-labs
cargo run --release --target i686-pc-windows-msvc --bin api_layers

Run it again after restarting the virtual machine and compare the addresses. A difference is expected. The names and behavior are the meaningful pieces.

Then use module_inventory against wesnoth.exe, ac_client.exe, or Quake3-UrT.exe. You are looking at two sides of the same loader idea:

  • module_inventory shows which DLL images the game loaded;
  • api_layers shows how one process asks a loaded DLL for an exported name.

The complete program is api_layers.rs.

References: libraries and native-system-service entry points, using Nt and Zw routines, GetModuleHandleW, and GetProcAddress.

Knowledge check

Follow the abstraction layers

What happens conceptually when a user-mode API needs the Windows kernel to perform privileged work?

Finished Lesson 12.5?

Mark it done and the lesson list shows a beside it. A chapter whose lessons are all done is marked done too. Progress is saved in this browser only.

Chapter 12: 10 lessons

This book grew out of the original Game Hacking Academy by attilathedud and has been rewritten as a beginner-first course. Like the original repository, its source is published under the Apache License 2.0.