Message Parsing and Serialization

Keyboard shortcuts

×

?

Open this guide

Alt + B

Show or hide the lesson list

Alt + O

Show or fold the lesson outline

Alt + N

Open or close notes

Alt + H

Toggle semantic colours

Alt + D

Toggle page brightness

Alt + C

Toggle code brightness

← →

Move through a focused code trace

Home End

First or final step in a focused trace

Escape

Close this guide or an open panel

Reader shortcuts pause while you type in a text field.

Cheatsheet for Lesson 9.4Key ideas, reference and questions

A one-page summary of “Message Parsing and Serialization” you can keep beside your notes. Plain Markdown.

Lesson 9.4 — Message Parsing and Serialization: cheatsheet

• Remember: Framing, compression, text encoding, syntax, and session meaning can each fail independently.
• Also useful: The cursor is a bookmark: every accepted field advances it only after its bytes fit.

Key terms

• take: take is the one place that advances parsing state.
• Vec::with_capacity: Vec::with_capacity is an optimization, not a safety requirement.

Formulas and patterns

• TCP byte stream; -> frame: one length plus exactly that many bytes; -> compressed payload: one gzip member; -> document: Simple WML text and tree structure; -> game meaning: a lobby chat message

Review

• Q: What should a safe message rewriter do before changing one field? A: Validate framing, lengths, message type, and field bounds first
• Q: Your encoder's output can be decoded by your decoder. Why is one real captured fixture still valuable? A: Both halves might share one wrong assumption and still agree

Download cheatsheet (.md)

Download cheatsheet (.md)

Parse the real Wesnoth frame

The controlled captures from Wesnoth 1.14.9 reveal this frame:

u32 big-endian compressed length
gzip-compressed Simple WML payload

The first four bytes do not include themselves in the length. If the header ends in 4E, the following gzip member is 78 bytes and the complete TCP payload is 4 + 78 = 82 bytes. The length is big endian, so 78, hexadecimal 0x0000004E, appears on the wire as 00 00 00 4E; u32::from_be_bytes states that order directly.

Lesson 9.1 showed why a receiver needs this prefix: it accumulates bytes until it has the four-byte header, then exactly the advertised number of payload bytes, however TCP splits them across reads.

The header is a claim from the sender, not permission to allocate whatever it asks for:

Predict first

This receiver refuses any length above its limit, MAX_FRAME = 1,048,576 bytes. Its buffer holds the header FF FF FF FF and then four payload bytes.

What should it do now?

Answer: Refuse the frame now. FF FF FF FF is 255 × 16,777,216 + 255 × 65,536 + 255 × 256 + 255 = 4,294,967,295, which is above 1,048,576. The header alone decides it, so the receiver refuses before allocating or waiting.

What a receiver does with a length-prefixed frameThe first four bytes are a big-endian length; the limit is MAX_FRAME, 1,048,576 bytes. Change the claim and how many payload bytes have arrived.

Header, payload, decision

Receive buffer

Header = 5

byte 0byte 3

00×256³

00×256²

00×256

05×1

Payload bytes

48'H'

65'e'

6C'l'

6C'l'

6F'o'

Receiver’s decision

Take 4 + 5 = 9 bytes: one complete framenothing is left over

The header says 5 and 5 payload bytes are here, so 4 + 5 = 9 bytes form one complete frame.

Code that allocates from that claim could request nearly four gigabytes. First reject any length above the configured maximum; only then wait for the advertised number of payload bytes, with a bounded buffer and a way to stop a stalled connection. Having only some payload bytes so far does not by itself prove a network frame is false, because TCP may deliver the rest later. After a complete frame is assembled, the parser checks its exact length before decoding it. This separates byte validation from the later question of whether the decoded chat message makes sense.

Keep five layers separate

A protocol message has several nested layers. Each layer decodes the representation produced by the one outside it:

TCP byte stream
  -> frame: one length plus exactly that many bytes
  -> compressed payload: one gzip member
  -> document: Simple WML text and tree structure
  -> game meaning: a lobby chat message

Lesson 9.1 introduced these layers; here each one becomes its own parsing step. Only the last, the game-level schema, says that sender, room, and message form a chat event.

TL;DR: Framing, compression, text encoding, syntax, and session meaning can each fail independently.

This separation is a debugging tool. If the first four bytes advertise an impossible size, inspect framing. If decompression fails, inspect the wrapper and payload. If WML is malformed, inspect serialization. If valid WML is ignored by the game, inspect the expected message fields and connection state. Calling every failure a “packet problem” hides the layer that actually broke.

Several controlled chat messages in the capture.

Read with a cursor

The four-byte length at the front of the captured frame must be read before the compressed body, and every read must stop at the captured buffer’s end. A cursor pairs the borrowed bytes with the next unread position. If a read fails, the position stays where it was, so an error cannot leave a parser halfway through a claimed field:

#[derive(Debug)]
enum ParseError {
    Truncated,
    TooLarge(usize),
    TrailingBytes,
    InvalidGzip(std::io::Error),
    InvalidUtf8,
}

struct Cursor<'a> {
    bytes: &'a [u8],
    position: usize,
}

impl<'a> Cursor<'a> {
    fn take(&mut self, count: usize) -> Result<&'a [u8], ParseError> {
        // 📏 Calculate and validate the complete range before changing position.
        // A failed read therefore cannot leave the parser half-advanced.
        let end = self.position.checked_add(count).ok_or(ParseError::Truncated)?;
        let slice = self.bytes.get(self.position..end).ok_or(ParseError::Truncated)?;
        self.position = end;
        Ok(slice)
    }

    fn u32_be(&mut self) -> Result<u32, ParseError> {
        // 📦 The protocol says network byte order (big-endian), regardless of
        // the little-endian x86 CPU running the game.
        Ok(u32::from_be_bytes(
            self.take(4)?.try_into().map_err(|_| ParseError::Truncated)?,
        ))
    }
}

Alternative: The cursor is a bookmark: every accepted field advances it only after its bytes fit.

This cursor replaces direct indexing and hopeful conversion:

The original parser takes four header bytes, trusts their length, and indexes the payload. This is a failure example: a short header or a length beyond the captured buffer can panic; extra bytes are silently accepted.

Before: index directly into an untrusted frame

fn parse_payload(bytes: &[u8]) -> Result<&[u8], ParseError> {
    let length = u32::from_be_bytes(bytes[0..4].try_into().unwrap()); // Short header panics.
    let payload = &bytes[4..4 + length as usize]; // Trusts the claimed boundary.
    Ok(payload)
}

The bounded cursor makes every read fallible and advances only after a valid range has been obtained. Requiring the cursor to finish at the end rejects trailing bytes in this one-frame format.

Improved code

fn parse_payload(bytes: &[u8]) -> Result<&[u8], ParseError> {
    let mut cursor = Cursor { bytes, position: 0 };
    let length = cursor.u32_be()? as usize;
    let payload = cursor.take(length)?;
    if cursor.position != bytes.len() {
        return Err(ParseError::TrailingBytes);
    }
    Ok(payload)
}

Show the diff

fn parse_payload(bytes: &[u8]) -> Result<&[u8], ParseError> {
    let length = u32::from_be_bytes(bytes[0..4].try_into().unwrap()); // Short header panics.
    let payload = &bytes[4..4 + length as usize]; // Trusts the claimed boundary.
    let mut cursor = Cursor { bytes, position: 0 };
    let length = cursor.u32_be()? as usize;
    let payload = cursor.take(length)?;
    if cursor.position != bytes.len() {
        return Err(ParseError::TrailingBytes);
    }
    Ok(payload)
}

Takeaway: treat malformed input as a parse error, and check the complete frame boundary.

Why this version?

Network input is allowed to be short, oversized, or malformed. Indexing panics when a boundary is wrong, and unwrap() erases the difference between bad input and a programmer bug. The cursor makes failure a normal Result, advances only after a successful bounded read, and lets the caller reject hidden trailing data after one complete frame.

The lifetime 'a says every slice returned by the cursor borrows from the original input. The cursor owns only a position number; it does not copy payloads or create references that outlive the frame.

take is the one place that advances parsing state. checked_add stops a hostile count from wrapping the end to a small position, and get returns None for a range past the buffer. Later field readers get a simple promise: success consumes exactly the requested bytes; failure consumes nothing and returns an error.

try_into() converts a four-byte slice into [u8; 4], the fixed-size array required by from_be_bytes. The conversion can only succeed after take(4), but keeping the error path explicit makes the invariant visible instead of using unwrap().

Every read checks its boundary. Now decompress only the advertised bytes and cap the output:

use std::io::Read;
use flate2::read::GzDecoder;

const MAX_FRAME: usize = 1_048_576;

fn decode_wesnoth_frame(bytes: &[u8]) -> Result<String, ParseError> {
    let mut cursor = Cursor { bytes, position: 0 };
    let compressed_len = cursor.u32_be()? as usize;
    // 🛡️ Reject the advertised allocation before slicing or decompressing.
    if compressed_len > MAX_FRAME {
        return Err(ParseError::TooLarge(compressed_len));
    }

    let compressed = cursor.take(compressed_len)?;
    if cursor.position != bytes.len() {
        return Err(ParseError::TrailingBytes);
    }

    let mut decoder = GzDecoder::new(compressed);
    let mut plain = Vec::new();
    // 📏 Read one byte beyond the limit. Reaching that extra byte proves the
    // decompressed message is oversized without allocating without bound.
    decoder.by_ref().take((MAX_FRAME + 1) as u64)
        .read_to_end(&mut plain)
        .map_err(ParseError::InvalidGzip)?;
    if plain.len() > MAX_FRAME {
        return Err(ParseError::TooLarge(plain.len()));
    }

    String::from_utf8(plain).map_err(|_| ParseError::InvalidUtf8)
}

There are two separate size limits here:

• compressed_len limits how many bytes the parser accepts from the frame;
• the take(MAX_FRAME + 1) wrapper limits how many bytes decompression may produce.

Both matter because a small compressed input can expand into a very large output. Reading one byte beyond the limit lets the parser distinguish “exactly at the limit” from “too large” without allocating an unbounded vector.

There is a third limit outside this pure function: the connection’s accumulated input buffer. A peer could advertise a legal one-megabyte frame and then send one byte every few seconds. The stream reader needs a maximum buffered size, read timeouts or cancellation, and a policy for a connection that never finishes its frame. Parsing bounds protect completed inputs; transport bounds protect the time and memory spent waiting for them.

The trailing-byte check makes this function parse one complete frame. Without it, a valid prefix could hide extra bytes that the caller mistakenly ignores. A streaming connection reader should split the stream into exact frame slices first and call this parser once per slice.

Decompressing a captured chat frame reveals readable Simple WML:

[message]
  message="a"
  room="lobby"
  sender="FFFAAAKKKEEE"
[/message]

That is the actual structure we will rebuild.

Simple WML nests these bracketed tags and quoted fields as a tree. This lesson handles the one chat-message shape needed by our local Wesnoth proxy. If you extend the tool to inspect maps, units, or arbitrary add-on messages, use a real tokenizer that handles escaping, nested nodes, repeated fields, and comments instead of adding more substring searches.

The receiver also has an expected schema, even when no formal schema file exists. That schema includes field names, allowed repetitions, value types, and which fields are required. The sender writes one version of that agreement; the receiver reads according to the version it understands.

When a game update adds a field, ask two different compatibility questions:

• Can the new reader still understand old messages?
• Can the old reader safely ignore or default the new field?

Those questions are not the same. A tolerant tree format may ignore an unknown optional field, but renaming a required field can break both directions. For a versioned research tool, keep captured fixtures from every supported game build and record which message shape each fixture uses instead of guessing that one parser is universal.

Compress and frame a real chat message

Use flate2:

[dependencies]
flate2 = "1"

With the gzip encoder available, the next function accepts a chat field as text and rejects quotes or line breaks that would change the simple WML structure. Only validated fields go into the message before compression and length framing:

use flate2::{Compression, write::GzEncoder};
use std::io::Write;

fn simple_field(value: &str) -> Result<&str, &'static str> {
    if value.chars().any(|ch| matches!(ch, '"' | '\n' | '\r')) {
        return Err("quotes and newlines are not allowed in this lab field");
    }
    Ok(value)
}

fn encode_wesnoth_frame(wml: &str) -> Result<Vec<u8>, &'static str> {
    let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
    encoder.write_all(wml.as_bytes()).map_err(|_| "gzip write failed")?;
    let compressed = encoder.finish().map_err(|_| "gzip finish failed")?;
    let length = u32::try_from(compressed.len()).map_err(|_| "frame too large")?;

    let mut frame = Vec::with_capacity(4 + compressed.len());
    frame.extend_from_slice(&length.to_be_bytes());
    frame.extend_from_slice(&compressed);
    Ok(frame)
}

fn encode_chat(sender: &str, message: &str) -> Result<Vec<u8>, &'static str> {
    let sender = simple_field(sender)?;
    let message = simple_field(message)?;
    encode_wesnoth_frame(&format!(
        "[message]\nmessage=\"{message}\"\nroom=\"lobby\"\nsender=\"{sender}\"\n[/message]"
    ))
}

Encoding runs the decoding pipeline in reverse:

1. validate values before placing them inside quoted WML fields;
2. serialize the typed message into UTF-8 text;
3. finish the gzip stream so its footer is present;
4. convert the compressed size to u32 without truncation;
5. reserve one vector for header plus payload;
6. append the big-endian length and compressed bytes.

Vec::with_capacity is an optimization, not a safety requirement. It avoids reallocating as the two pieces are appended. u32::try_from is a correctness requirement because as u32 could silently discard high bits for an oversized buffer.

GzEncoder produces the correct gzip wrapper; a plain zlib stream is not interchangeable here. The first two compressed bytes should be 1F 8B.

Gzip and zlib can both use the DEFLATE compression algorithm but wrap it in different headers and checksums. “Same compression algorithm” does not mean “same wire format.” Matching the observed wrapper is part of implementing the protocol.

Round-trip tests

Feed the serializer one sender and one message, then give its frame to the decoder. If the resulting WML still contains both values, the length, compression wrapper, UTF-8 text, and field quoting agree between these two functions for this case:

#[test]
fn chat_round_trip() {
    let frame = encode_chat("lab-user", "hello-17").unwrap();
    let decoded = decode_wesnoth_frame(&frame).unwrap();
    assert!(decoded.contains("message=\"hello-17\""));
    assert!(decoded.contains("sender=\"lab-user\""));
}

Also test an empty input, a truncated gzip member, an oversized length, invalid UTF-8 after decompression, extra trailing bytes, and rejected quote/newline characters.

Narration: Encoding then decoding your own message checks consistency; captured fixtures also check whether both directions share the same mistaken assumption.

A round trip proves that the encoder produces something this decoder accepts. It does not prove compatibility by itself, because two functions could share the same mistake. Keep at least one captured fixture from the real local session and verify that the decoder accepts it. Then send an encoded message to the controlled server and confirm that another capture has the expected four-byte prefix and gzip header.

Use a small failure matrix:

Test input	Expected failure
Fewer than four bytes	Truncated while reading the length
Length larger than remaining bytes	Truncated while taking the payload
Length above the cap	TooLarge before decompression
Valid frame plus one extra byte	TrailingBytes
Broken gzip checksum	InvalidGzip
Decompressed non-UTF-8 bytes	InvalidUtf8
Quote inside a simple field	Rejected before serialization

Comparing one changed message field.

The goal is not merely to make one capture parse. It is to define exactly which byte strings are accepted and how malformed data fails.

The buildable protocol module is wesnoth_protocol.rs. It contains bounded frame reads, gzip encode/decode, output-size caps, strict trailing-byte checks, WML field sanitation, send_wml, encode_chat, and round-trip/error tests. Both the real chatbot and proxy import this same implementation instead of copying a shortened lesson version.

wesnoth_protocol.rs

windows-labs/src/ · Rust, 107 lines

View on GitHub ↗

Show the codeHide the code

//! Bounds-checked Wesnoth 1.14.9 gzip/WML framing used by the local network labs.

use std::io::{self, Read, Write};

use anyhow::{Context, Result};
use flate2::{Compression, read::GzDecoder, write::GzEncoder};

pub const MAX_FRAME: usize = 1_048_576;

pub fn simple_field(value: &str) -> Result<&str> {
    anyhow::ensure!(
        !value
            .chars()
            .any(|character| matches!(character, '"' | '\n' | '\r')),
        "WML fields in this lab cannot contain quotes or newlines"
    );
    anyhow::ensure!(value.len() <= 512, "WML field is too long");
    Ok(value)
}

pub fn encode_frame(wml: &str) -> Result<Vec<u8>> {
    anyhow::ensure!(wml.len() <= MAX_FRAME, "plain WML frame is too large");
    let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
    encoder
        .write_all(wml.as_bytes())
        .context("could not gzip WML")?;
    let compressed = encoder.finish().context("could not finish gzip member")?;
    let length = u32::try_from(compressed.len()).context("compressed frame is too large")?;
    let mut frame = Vec::with_capacity(4 + compressed.len());
    frame.extend_from_slice(&length.to_be_bytes());
    frame.extend_from_slice(&compressed);
    Ok(frame)
}

pub fn decode_frame(frame: &[u8]) -> Result<String> {
    anyhow::ensure!(frame.len() >= 4, "frame has no four-byte length header");
    let length = u32::from_be_bytes(frame[..4].try_into().expect("slice is four bytes")) as usize;
    anyhow::ensure!(length <= MAX_FRAME, "compressed frame is too large");
    anyhow::ensure!(
        frame.len() == 4 + length,
        "frame length does not match header"
    );

    let mut decoder = GzDecoder::new(&frame[4..]);
    let mut plain = Vec::new();
    decoder
        .by_ref()
        .take((MAX_FRAME + 1) as u64)
        .read_to_end(&mut plain)
        .context("invalid gzip payload")?;
    anyhow::ensure!(plain.len() <= MAX_FRAME, "decompressed WML is too large");
    String::from_utf8(plain).context("WML payload is not UTF-8")
}

pub fn read_frame(reader: &mut impl Read) -> io::Result<Vec<u8>> {
    let mut header = [0_u8; 4];
    reader.read_exact(&mut header)?;
    let length = u32::from_be_bytes(header) as usize;
    if length > MAX_FRAME {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "frame is too large",
        ));
    }
    let mut frame = Vec::with_capacity(4 + length);
    frame.extend_from_slice(&header);
    frame.resize(4 + length, 0);
    reader.read_exact(&mut frame[4..])?;
    Ok(frame)
}

pub fn send_wml(writer: &mut impl Write, wml: &str) -> Result<()> {
    let frame = encode_frame(wml)?;
    writer
        .write_all(&frame)
        .context("could not send complete frame")?;
    writer.flush().context("could not flush frame")?;
    Ok(())
}

pub fn encode_chat(sender: &str, message: &str) -> Result<Vec<u8>> {
    let sender = simple_field(sender)?;
    let message = simple_field(message)?;
    encode_frame(&format!(
        "[message]\nmessage=\"{message}\"\nroom=\"lobby\"\nsender=\"{sender}\"\n[/message]"
    ))
}

#[cfg(test)]
mod tests {
    use super::{decode_frame, encode_chat, encode_frame};

    #[test]
    fn real_chat_shape_round_trips() {
        let frame = encode_chat("ChatBot", "Hello!").unwrap();
        assert_eq!(&frame[4..6], &[0x1F, 0x8B]);
        let plain = decode_frame(&frame).unwrap();
        assert!(plain.contains("message=\"Hello!\""));
    }

    #[test]
    fn rejects_trailing_bytes() {
        let mut frame = encode_frame("[version]\n[/version]").unwrap();
        frame.push(0);
        assert!(decode_frame(&frame).is_err());
    }
}

Read the module by responsibility

Start with the pure functions that accept and return byte vectors; they are easiest to test. Next read the exact stream functions that repeatedly call read_exact for a header and payload. Finally read send_wml, which owns the complete serialization-and-write transaction.

This order keeps three kinds of failure distinct:

• transport failure: the socket closed or timed out;
• framing failure: the advertised message length was impossible;
• content failure: gzip, UTF-8, or WML data was invalid.

Preserving those categories in error messages is part of the explanation. If everything becomes “chat failed,” a learner cannot tell whether to inspect Wireshark, the frame cursor, or the message serializer.

This interactive lab needs JavaScript. The lesson text around it still explains the same idea.

🧩

Knowledge check

Preserve message structure

What should a safe message rewriter do before changing one field?

1Validate framing, lengths, message type, and field bounds first2Overwrite the bytes at a remembered offset without parsing3Assume one read equals one message and patch that read4Drop unknown trailing bytes quietly so the frame looks valid

Check answer

Finished Lesson 9.4?

Mark it done and the lesson list shows a ✓ beside it. A chapter whose lessons are all done is marked done too. Progress is saved in this browser only.

Mark lesson doneLesson done. Mark it not done

Chapter 9: 10 lessons

📝 Notes