Build Identity and Versioned Evidence

Keyboard shortcuts

×

?

Open this guide

Alt + B

Show or hide the lesson list

Alt + O

Show or fold the lesson outline

Alt + N

Open or close notes

Alt + H

Toggle semantic colours

Alt + D

Toggle page brightness

Alt + C

Toggle code brightness

← →

Move through a focused code trace

Home End

First or final step in a focused trace

Escape

Close this guide or an open panel

Reader shortcuts pause while you type in a text field.

Cheatsheet for Lesson 10.9Key ideas, reference and questions

A one-page summary of “Build Identity and Versioned Evidence” you can keep beside your notes. Plain Markdown.

Lesson 10.9 — Build Identity and Versioned Evidence: cheatsheet

• Remember: A matching hash establishes matching bytes against the chosen baseline; it does not establish the baseline’s author or safety.
• Also useful: Hashing in chunks is like checking pages as they pass a scanner: the whole file contributes without all its pages staying in memory together.

Key terms

• streaming: The improvement is streaming: reuse one buffer, and feed only the bytes filled by each read into the hasher.
• time-of-check/time-of-use: This is called a time-of-check/time-of-use race.

Review

• Q: Why does a patcher keep an executable's SHA-256 digest next to the RVAs and byte patterns it uses? A: So the patch can refuse to apply when the target build's bytes differ
• Q: Two installs of a game carry the same version label. Why is that not enough to reuse an address offset between them? A: Their bytes can differ after a rebuild, a mod or one patched file

Download cheatsheet (.md)

Download cheatsheet (.md)

A version label is not enough

Two files can both call themselves “Wesnoth 1.14.9” and still contain different bytes. A distribution may rebuild the executable, a mod may replace data, or a repair tool may update one DLL without changing the folder name.

That matters because much of what this course builds depends on exact bytes:

• an RVA belongs to one module layout;
• a byte pattern belongs to one compiled instruction sequence;
• an original-byte check belongs to one supported patch site;
• a PK3 edit belongs to one archive and path layout.

A cryptographic hash turns all file bytes into a fixed-size fingerprint. This lab uses SHA-256. Change one byte and the resulting digest should change dramatically.

Two files with the same label but different bytesA label says what someone claims. A digest is computed from the bytes themselves.

1. 1 Same label, one different byte

   File A

   Label

   version 1.2

   Bytes

   01

   02

   03

   File B

   Label

   version 1.2

   Bytes

   01

   02

   04

   Both say “version 1.2”, but File A ends 03 and File B ends 04.

2. 2 So their SHA-256 digests differ

   File A

   SHA-256

   digest A

   File B

   SHA-256

   digest B

   Any change to the bytes changes the digest, whatever the label says: these are not the same file.

3. 3 Compare against a trusted baseline

   Trusted baseline

   Known good digest

   digest of the trusted file

   Unknown file

   ↓ compare

   Computed digest

   digest of this file

   Check the bytes or the digest against a known good copy instead of relying on the label.

A label says what someone claims. A digest is computed from the bytes themselves.

Several different facts identify a build, and each one answers a separate question:

Fact	Question it answers	Important limit
Version label	What release does the program claim?	Different builds can share it.
Live module path	Which file did this process map?	A path does not identify its bytes.
Architecture and PE fields	What layout family is this file?	Many builds share those fields.
SHA-256 digest	Do all bytes match the baseline?	The baseline must already be trusted.
Trusted manifest or signature	Who supplied the expected identity?	It does not prove the live file matches.

For a reproducible lesson, confirm the live path and architecture, then compare its SHA-256 digest with a trusted record. No single friendly version string does all of those jobs.

What a hash proves—and what it does not

If today’s digest equals a trusted earlier digest, the file bytes match that earlier file. The hash does not prove who created the file or whether the original was safe.

TL;DR: A matching hash establishes matching bytes against the chosen baseline; it does not establish the baseline’s author or safety.

The word trusted matters. If an attacker can replace both a file and its manifest, a perfect hash comparison proves only that the two replacements agree. Keep the baseline somewhere the experiment does not rewrite, such as a read-only lab snapshot or version-controlled record.

A manifest records more than one fingerprint

The course format stores one file per line:

# GHA SHA-256 manifest v1
digest<TAB>byte-count<TAB>absolute-path

The byte count is not a substitute for SHA-256. It is extra context that makes a mismatch easier to explain.

The tool rejects tabs and newlines in paths because those characters would make the small text format ambiguous. If this game-file manifest grows to store mod IDs, signatures, several hashes, or nested package information, move to a defined format such as JSON, CBOR, or a signed catalog. The lab stays text-only so you can see the separators, field meanings, and rejected input forms.

Hash in chunks, not one giant allocation

The loop reads a fixed-size buffer at a time instead of the whole file, and only the bytes actually filled go into the hash:

Alternative: Hashing in chunks is like checking pages as they pass a scanner: the whole file contributes without all its pages staying in memory together.

Optional: practise the named operations and their inputs with a few whole-token blanks.

fn sha256(path: &Path) -> anyhow::Result<(u64, String)> {
    let mut file = File::open(path)?;
    let mut hasher = Sha256::new();
    let mut total = 0_u64;
    let mut buffer = [0_u8; 64 * 1024];

    loop {
        let count = file.read(&mut buffer)?;
        if count == 0 {
            break;
        }
        hasher.update(&buffer[..count]);
        total = total.checked_add(u64::try_from(count)?)
            .context("file size overflowed")?;
    }
    Ok((total, format!("{:x}", hasher.finalize())))
}

read returns the number of buffer bytes filled. A return value of zero means end of file. Only &buffer[..count] goes into the hash; hashing the unused remainder would produce the wrong answer.

The original helper reads the entire file into one byte vector, then hashes it. A large archive therefore needs an equally large temporary allocation; Path, File, Sha256 and the hash traits are the same helpers used above.

Before: read the whole file

fn sha256(path: &Path) -> anyhow::Result<String> {
    let mut hasher = Sha256::new();
    let bytes = std::fs::read(path)?;
    hasher.update(bytes);
    Ok(format!("{:x}", hasher.finalize()))
}

The improvement is streaming: reuse one buffer, and feed only the bytes filled by each read into the hasher. The digest still describes the complete file.

Improved code

fn sha256(path: &Path) -> anyhow::Result<String> {
    let mut hasher = Sha256::new();
    let mut file = File::open(path)?;
    let mut buffer = [0_u8; 64 * 1024];
    loop {
        let count = file.read(&mut buffer)?;
        if count == 0 { break; }
        hasher.update(&buffer[..count]);
    }
    Ok(format!("{:x}", hasher.finalize()))
}

Show the diff

fn sha256(path: &Path) -> anyhow::Result<String> {
    let mut hasher = Sha256::new();
    let bytes = std::fs::read(path)?;
    hasher.update(bytes);
    let mut file = File::open(path)?;
    let mut buffer = [0_u8; 64 * 1024];
    loop {
        let count = file.read(&mut buffer)?;
        if count == 0 { break; }
        hasher.update(&buffer[..count]);
    }
    Ok(format!("{:x}", hasher.finalize()))
}

Takeaway: hash large inputs in bounded chunks instead of keeping a second copy of the file in memory.

Why this version?

Streaming keeps memory use roughly constant. A 4 GB game archive does not require a 4 GB Vec just to calculate its fingerprint.

Reading a whole file into memory needs memory as big as the file. The lesson’s loop reads one fixed 64 KiB buffer at a time, so a 4 GiB archive needs only 64 KiB of buffer, at the price of many more read calls. The interactive version needs JavaScript; the lesson text around it explains the same idea.

Create without silently replacing trust

The important file option is create_new(true):

let output = OpenOptions::new()
    .write(true)
    .create_new(true)
    .open(manifest)
    .with_context(|| format!("refusing to replace {}", manifest.display()))?;

If the manifest already exists, creation fails. Updating a trusted baseline should be a conscious review step, not something the verifier performs automatically after detecting a change.

Narration: Automatically replacing a baseline after a mismatch erases the evidence of change. Keep the earlier record until the new state has been reviewed.

Verify every recorded field

For each non-comment line, the verifier:

1. parses the wanted SHA-256 digest;
2. parses the wanted byte count;
3. opens the recorded path;
4. streams the current bytes through SHA-256;
5. compares size and digest;
6. reports OK or CHANGED;
7. returns a failing exit status if anything changed.

As a flow, one manifest line becomes one pass/fail decision:

That final failure is useful in scripts. A colored message is for a person; a non-zero exit status is for another program.

Run the complete tool

Windows VM

The complete implementation—including creation, parsing, verification, checked size arithmetic, and non-overwriting output—is integrity_manifest.rs.

integrity_manifest.rs

windows-labs/src/bin/ · Rust, 130 lines

View on GitHub ↗

Show the codeHide the code

use std::{
    fs::{self, File, OpenOptions},
    io::{BufRead, BufReader, BufWriter, Read, Write},
    path::{Path, PathBuf},
};

use anyhow::{Context, Result, ensure};
use sha2::{Digest, Sha256};

fn sha256(path: &Path) -> Result<(u64, String)> {
    let mut file =
        File::open(path).with_context(|| format!("could not open {}", path.display()))?;
    let mut hasher = Sha256::new();
    let mut total = 0_u64;
    let mut buffer = [0_u8; 64 * 1024];

    loop {
        let count = file.read(&mut buffer)?;
        if count == 0 {
            break;
        }
        hasher.update(&buffer[..count]);
        total = total
            .checked_add(u64::try_from(count)?)
            .context("file size overflowed")?;
    }
    Ok((total, format!("{:x}", hasher.finalize())))
}

fn checked_path(text: &str) -> Result<PathBuf> {
    ensure!(
        !text.contains(['\t', '\n', '\r']),
        "paths may not contain tabs or newlines"
    );
    Ok(PathBuf::from(text))
}

fn create(manifest: &Path, files: impl Iterator<Item = String>) -> Result<()> {
    let mut entries = Vec::new();
    for text in files {
        let path = checked_path(&text)?;
        let canonical = fs::canonicalize(&path)
            .with_context(|| format!("could not resolve {}", path.display()))?;
        let canonical_text = canonical.to_string_lossy();
        ensure!(
            !canonical_text.contains(['\t', '\n', '\r']),
            "path cannot be represented safely"
        );
        let (size, digest) = sha256(&canonical)?;
        println!("added  {digest}  {}", canonical.display());
        entries.push((digest, size, canonical_text.into_owned()));
    }
    ensure!(
        !entries.is_empty(),
        "create needs at least one game or mod file"
    );

    let output = OpenOptions::new()
        .write(true)
        .create_new(true)
        .open(manifest)
        .with_context(|| format!("refusing to replace {}", manifest.display()))?;
    let mut output = BufWriter::new(output);
    writeln!(output, "# GHA SHA-256 manifest v1")?;
    for (digest, size, path) in &entries {
        writeln!(output, "{digest}\t{size}\t{path}")?;
    }
    output.flush()?;
    println!(
        "created {} with {} file(s)",
        manifest.display(),
        entries.len()
    );
    Ok(())
}

fn verify(manifest: &Path) -> Result<()> {
    let input =
        File::open(manifest).with_context(|| format!("could not open {}", manifest.display()))?;
    let mut checked = 0_usize;
    let mut failed = 0_usize;

    for (index, line) in BufReader::new(input).lines().enumerate() {
        let line = line?;
        if line.is_empty() || line.starts_with('#') {
            continue;
        }
        let mut fields = line.splitn(3, '\t');
        let wanted_hash = fields.next().context("missing digest")?;
        let wanted_size: u64 = fields.next().context("missing size")?.parse()?;
        let path = checked_path(fields.next().context("missing path")?)?;
        let (actual_size, actual_hash) =
            sha256(&path).with_context(|| format!("manifest line {}", index + 1))?;
        checked += 1;

        if actual_size == wanted_size && actual_hash.eq_ignore_ascii_case(wanted_hash) {
            println!("OK       {}", path.display());
        } else {
            failed += 1;
            println!("CHANGED  {}", path.display());
            println!("  wanted {wanted_hash} ({wanted_size} bytes)");
            println!("  actual {actual_hash} ({actual_size} bytes)");
        }
    }

    ensure!(checked > 0, "the manifest contains no files");
    ensure!(failed == 0, "{failed} of {checked} file(s) changed");
    println!("verified {checked} file(s)");
    Ok(())
}

fn main() -> Result<()> {
    let mut arguments = std::env::args().skip(1);
    let command = arguments
        .next()
        .context("usage: integrity_manifest <create|verify> <manifest> [files...]")?;
    let manifest = PathBuf::from(arguments.next().context("missing manifest path")?);

    match command.as_str() {
        "create" => create(&manifest, arguments),
        "verify" => {
            ensure!(
                arguments.next().is_none(),
                "verify reads file paths from the manifest"
            );
            verify(&manifest)
        }
        _ => anyhow::bail!("command must be create or verify"),
    }
}

Build it:

cd windows-labs
cargo build --release --target i686-pc-windows-msvc `
  --bin integrity_manifest

Create a baseline for the exact executable and important DLLs:

.\target\i686-pc-windows-msvc\release\integrity_manifest.exe create `
  .\wesnoth-1.14.9.manifest `
  "C:\Games\Wesnoth 1.14.9\wesnoth.exe" `
  "C:\Games\Wesnoth 1.14.9\libstdc++-6.dll"

Verify it before using an RVA or installing a patch:

.\target\i686-pc-windows-msvc\release\integrity_manifest.exe verify `
  .\wesnoth-1.14.9.manifest

Create separate manifests for the exact course builds of AssaultCube and Urban Terror. Do not add save files, logs, or configuration files that are expected to change every run.

Make mods explicit instead of invisible

A supported mod changes files on purpose. Treat it as a separate known state:

clean-install.manifest
course-mod.manifest

Do not update clean-install.manifest until a changed file looks clean again. Keep the original baseline, and create a second, reviewed manifest after the intended change, so each known state has its own list of digests.

In the Urban Terror texture lab from Lesson 10.4, a tool replaces a texture inside the zUrT43_001.pk3 archive. Before the tool runs, the archive’s digest belongs to the clean state. The tool keeps the untouched original as zUrT43_001.pk3.gha-backup, so that backup must hash to the same clean digest; if it does not, the original was not preserved. The rebuilt zUrT43_001.pk3 has a new digest, which belongs to the modded state.

Watch for time-of-check/time-of-use

The program hashes a file and later another tool may open it. The file could change between those actions. This is called a time-of-check/time-of-use race.

For this lab, close the game and mod tools while verifying. A launcher that must verify a versioned mod immediately before starting the game needs a stronger design: it can keep the checked file handle open, require a signed package, or let one trusted updater own both verification and launch.

Hashes complement runtime checks

An on-disk hash confirms the file. A live patcher should still check:

• the live module name and range;
• the supported architecture;
• a version-specific RVA or unique pattern;
• the exact original bytes at the final address.

Each check covers a gap the others leave. The on-disk hash cannot tell which copy of the file the process actually mapped, or whether anything changed the bytes in memory after loading; the original-byte check at the final address can. A patch should go ahead only when every check passes.

The manifest has identified the file and build. Before reading through an old offset table, we still need to establish that the new build uses the same object layout. The next lesson recovers that layout from several independent clues; process access rights then determine which operations Windows permits.

Reference: Microsoft PE format and NIST Secure Hash Standard.

🧩

Knowledge check

Tie offsets to exact bytes

Why does a patcher keep an executable's SHA-256 digest next to the RVAs and byte patterns it uses?

1So the patch can refuse to apply when the target build's bytes differ2So the digest can lock the file against being replaced by a game update3So the hash can hide the patched address from ASLR when the module loads4So the digest can grant the tool the process rights it needs to write

Check answer

Finished Lesson 10.9?

Mark it done and the lesson list shows a ✓ beside it. A chapter whose lessons are all done is marked done too. Progress is saved in this browser only.

Mark lesson doneLesson done. Mark it not done

Chapter 10: 10 lessons

📝 Notes