Chapter 8: Building the Core Pipeline: Routing and Output Generation

Welcome to Chapter 8! In the previous chapters, we laid the groundwork for our Rust-based Static Site Generator (SSG). We’ve learned how to parse content, extract frontmatter, convert Markdown to HTML, and render that HTML using a templating engine like Tera. We even introduced the concept of component support within Markdown, preparing our system for dynamic interactions. Now, it’s time to connect these pieces and bring our SSG to life by defining how content maps to URLs and generating the final static HTML files.

This chapter is crucial as it forms the “core pipeline” of any SSG: taking processed content and transforming it into a deployable website. We will design and implement the routing logic that determines the final URL structure for each piece of content, and then build the output generation mechanism responsible for writing these rendered HTML files to our public directory. This involves careful consideration of file paths, directory creation, and error handling to ensure a robust build process.

By the end of this chapter, you will have a functional SSG that can read content, process it, render it into full HTML pages, and write those pages to a structured output directory. This marks a significant milestone, allowing us to preview our generated site and verify the entire content-to-HTML transformation. We’ll focus on production-ready code, ensuring our routing is flexible and our file operations are efficient and safe.

Planning & Design

The routing and output generation process involves orchestrating several components we’ve already built, along with new logic to manage file paths. Our goal is to take a collection of Content objects (which now contain rendered HTML from Chapter 7) and write them to the filesystem in a predictable and SEO-friendly structure.

Routing Strategy

A common and highly effective routing strategy for static sites is to mirror the content directory structure in the output. For example:

• content/posts/my-first-post.md should generate public/posts/my-first-post/index.html.
• content/pages/about.md should generate public/pages/about/index.html.
• content/index.md (or _index.md) should generate public/index.html.

This index.html pattern within a directory (/posts/my-first-post/index.html) is preferred because it allows for “pretty URLs” (e.g., /posts/my-first-post/ instead of /posts/my-first-post.html) when served by a web server. We’ll also allow frontmatter to override the default slug or even provide a full permalink for maximum flexibility.

Output Directory Structure

We’ll use a public directory as our default output destination. The structure within public will directly reflect our routing decisions.

Component Architecture

The SiteBuilder will be the central orchestrator. It will take a Config, our TemplateEngine, and the processed Content items. Its primary responsibility will be to manage the build flow, including routing and writing files.

File Structure Updates

We’ll introduce a new src/builder.rs module to house our SiteBuilder logic. Our src/main.rs will be updated to instantiate and run this builder.

.
├── Cargo.toml
├── src
│   ├── main.rs
│   ├── config.rs           # Project configuration
│   ├── content.rs          # Content parsing & representation (Frontmatter, Markdown AST, HTML)
│   ├── parser.rs           # Frontmatter and Markdown parsing logic
│   ├── renderer.rs         # HTML rendering with Tera
│   ├── builder.rs          # NEW: Orchestrates the build process (routing & output)
│   └── utils.rs            # Utility functions (e.g., file operations)
├── content                 # Example content directory
│   ├── posts
│   │   ├── first-post.md
│   │   └── second-post.md
│   └── about.md
├── templates               # Tera templates
│   ├── base.html
│   └── post.html
└── public                  # NEW: Output directory for generated static files

Step-by-Step Implementation

a) Setup/Configuration

First, let’s refine our Config struct and add a new module for the SiteBuilder.

1. Update Cargo.toml: We’ll likely need path-clean for robust path handling, and rayon for parallel processing later. If not already added, include them:

# Cargo.toml
[dependencies]
# ... other dependencies ...
serde = { version = "1.0", features = ["derive"] }
serde_yaml = "0.9"
serde_json = "1.0"
tera = "1.19"
pulldown-cmark = "0.11"
chrono = { version = "0.4", features = ["serde"] }
log = "0.4"
env_logger = "0.11"
anyhow = "1.0"
path-clean = "0.1" # For sanitizing file paths
rayon = "1.8"      # For parallel processing

2. Create src/builder.rs:

// src/builder.rs
use std::path::{Path, PathBuf};
use std::fs;

use anyhow::{Result, Context};
use log::{info, error, debug};
use path_clean::PathClean;
use rayon::prelude::*;

use crate::config::Config;
use crate::content::{Content, ContentPath};
use crate::renderer::TemplateRenderer;

/// The main orchestrator for building the static site.
pub struct SiteBuilder {
    config: Config,
    renderer: TemplateRenderer,
    content_items: Vec<Content>,
}

impl SiteBuilder {
    /// Creates a new `SiteBuilder` instance.
    pub fn new(config: Config, renderer: TemplateRenderer, content_items: Vec<Content>) -> Self {
        SiteBuilder {
            config,
            renderer,
            content_items,
        }
    }

    /// Determines the final output path for a given content item.
    ///
    /// This function implements our routing logic:
    /// 1. Prioritize `permalink` from frontmatter if available.
    /// 2. If `slug` is available, use it to construct the path.
    /// 3. Otherwise, derive from the content's source path.
    /// 4. Ensure pretty URLs (e.g., `/posts/my-post/index.html`).
    fn determine_output_path(&self, content: &Content) -> Result<PathBuf> {
        let output_base = &self.config.output_dir;
        let mut output_path = PathBuf::from(output_base);

        // 1. Check for `permalink` override in frontmatter
        if let Some(permalink) = &content.frontmatter.permalink {
            let clean_permalink = PathBuf::from(permalink).clean();
            output_path.push(clean_permalink);
            if output_path.extension().is_none() {
                // If permalink doesn't have an extension, assume it's a directory
                output_path.push("index.html");
            }
            return Ok(output_path);
        }

        // 2. Determine path based on content source and frontmatter slug
        let relative_source_path = content.source_path.strip_prefix(&self.config.content_dir)
            .context(format!("Failed to strip content_dir prefix from {:?}", content.source_path))?;

        let mut path_segments: Vec<&str> = relative_source_path.iter()
            .filter_map(|s| s.to_str())
            .collect();

        // Remove filename extension
        if let Some(last) = path_segments.last_mut() {
            if let Some((name, _ext)) = last.rsplit_once('.') {
                *last = name;
            }
        }

        // Handle `index` files: `content/posts/index.md` -> `public/posts/index.html`
        // `content/index.md` -> `public/index.html`
        let is_index_file = path_segments.last().map_or(false, |s| *s == "index" || *s == "_index");

        // Use slug if provided, otherwise the last path segment (file name without extension)
        if let Some(slug) = &content.frontmatter.slug {
            if let Some(last_segment_idx) = path_segments.iter().rposition(|s| !s.is_empty()) {
                path_segments[last_segment_idx] = slug;
            } else {
                path_segments.push(slug);
            }
        }

        // Construct the path
        for segment in path_segments {
            output_path.push(segment);
        }

        // If it's not an `index` file, make it a directory with an `index.html` inside
        if !is_index_file || path_segments.is_empty() {
             output_path.push("index.html");
        } else {
            // For true index files (e.g., content/index.md), output directly as index.html
            // But ensure it's `index.html` and not `index/index.html`
            if output_path.file_name().map_or(false, |s| s == "index" || s == "_index") {
                output_path.set_file_name("index.html");
            } else {
                 output_path.push("index.html");
            }
        }

        Ok(output_path.clean())
    }

    /// Renders a single content item and writes it to the determined output path.
    fn render_and_write_content(&self, content: &Content) -> Result<()> {
        let output_file_path = self.determine_output_path(content)?;
        let output_dir = output_file_path.parent()
            .context(format!("Failed to get parent directory for {:?}", output_file_path))?;

        // Ensure the output directory exists
        fs::create_dir_all(output_dir)
            .context(format!("Failed to create output directory {:?}", output_dir))?;

        // Render the content using the template engine
        let rendered_html = self.renderer.render_content(content)
            .context(format!("Failed to render content for {:?}", content.source_path))?;

        // Write the rendered HTML to the file
        fs::write(&output_file_path, rendered_html)
            .context(format!("Failed to write rendered HTML to {:?}", output_file_path))?;

        info!("Generated: {}", output_file_path.display());
        Ok(())
    }

    /// Executes the full site build process.
    pub fn build(&self) -> Result<()> {
        info!("Starting site build to output directory: {}", self.config.output_dir.display());

        // Clear the output directory before building (optional, but good for clean builds)
        if self.config.output_dir.exists() {
            fs::remove_dir_all(&self.config.output_dir)
                .context(format!("Failed to clear output directory: {}", self.config.output_dir.display()))?;
            debug!("Cleared output directory: {}", self.config.output_dir.display());
        }
        fs::create_dir_all(&self.config.output_dir)
            .context(format!("Failed to create output directory: {}", self.config.output_dir.display()))?;


        // Process all content items in parallel
        self.content_items.par_iter()
            .map(|content| {
                self.render_and_write_content(content)
                    .with_context(|| format!("Error processing content from {:?}", content.source_path))
            })
            .collect::<Result<Vec<()>>>()?; // Collect results to propagate any errors

        info!("Site build completed successfully.");
        Ok(())
    }
}

Explanation of src/builder.rs:

• SiteBuilder struct: Holds the Config, TemplateRenderer, and the Vec<Content> items that have already been parsed and partially processed.
• determine_output_path: This is the core routing logic.
  • It first checks for a permalink in the frontmatter, which offers the most control.
  • If no permalink, it derives the path from the content’s source_path, stripping the content_dir prefix.
  • It handles slug overrides for the final path segment.
  • Crucially, it implements the “pretty URL” pattern: some/path/file.md becomes public/some/path/file/index.html.
  • Special handling for index.md or _index.md files: these should directly become public/index.html or public/some/path/index.html without an extra directory level.
  • path-clean is used to sanitize paths, resolving .. and ./ segments, providing a robust and secure path.
• render_and_write_content:
  • Calls determine_output_path to get the target file location.
  • Uses fs::create_dir_all to ensure all parent directories exist before writing the file. This is idempotent and safe.
  • Delegates rendering to the TemplateRenderer (from Chapter 7).
  • Writes the final HTML to the file system using fs::write.
  • Includes robust error handling with anyhow and context for better debugging.
  • Logs the generation of each file using info!.
• build: The main entry point.
  • Initializes logging.
  • Clears the output_dir: This ensures a clean build every time, preventing stale files. This is a common practice for SSGs.
  • Parallel Processing with rayon: This is a key performance optimization. Instead of processing content items sequentially, par_iter() allows rayon to distribute the render_and_write_content calls across available CPU cores. This can significantly speed up builds for sites with many content files.
  • collect::<Result<Vec<()>>>()? is used to collect the results of the parallel operations. If any render_and_write_content call returns an Err, the entire build function will return an error, propagating the issue.

b) Core Implementation - Update main.rs

Now, let’s update our main.rs to use the new SiteBuilder.

// src/main.rs
mod config;
mod content;
mod parser;
mod renderer;
mod builder; // NEW: Import the builder module
mod utils;

use std::fs;
use std::path::PathBuf;
use anyhow::{Result, Context};
use log::{info, error};
use env_logger::Env;

use crate::config::Config;
use crate::parser::parse_content_file;
use crate::renderer::TemplateRenderer;
use crate::builder::SiteBuilder; // NEW

fn main() -> Result<()> {
    // Initialize logging
    env_logger::Builder::from_env(Env::default().default_filter_or("info")).init();

    info!("Starting SSG build process...");

    // 1. Load configuration
    let config = Config::load_from_file("config.toml")
        .context("Failed to load configuration from config.toml")?;
    info!("Configuration loaded: {:?}", config);

    // 2. Initialize TemplateRenderer
    let renderer = TemplateRenderer::new(&config.template_dir)
        .context("Failed to initialize template renderer")?;
    info!("Template renderer initialized from: {}", config.template_dir.display());

    // 3. Scan content directory and parse all content files
    let mut content_items = Vec::new();
    let content_path = &config.content_dir;

    if !content_path.exists() {
        error!("Content directory not found: {}", content_path.display());
        anyhow::bail!("Content directory not found.");
    }

    for entry in walkdir::WalkDir::new(content_path) {
        let entry = entry?;
        let path = entry.path();

        if path.is_file() && path.extension().map_or(false, |ext| ext == "md") {
            info!("Parsing content file: {}", path.display());
            match parse_content_file(path, &renderer) { // Pass renderer for component processing
                Ok(content) => content_items.push(content),
                Err(e) => error!("Error parsing {}: {:?}", path.display(), e),
            }
        }
    }
    info!("Parsed {} content items.", content_items.len());

    // 4. Create and run the SiteBuilder
    let builder = SiteBuilder::new(config, renderer, content_items);
    builder.build()
        .context("Site build failed")?;

    info!("SSG build process finished successfully!");

    Ok(())
}

Explanation of src/main.rs updates:

• We import builder::SiteBuilder.
• After parsing all content files and initializing the TemplateRenderer, we create an instance of SiteBuilder with our config, renderer, and the collected content_items.
• We then call builder.build() to kick off the entire process.
• Error handling is wrapped with anyhow::Context for clear error messages.

c) Testing This Component

To test our routing and output generation, let’s ensure we have some dummy content and templates.

1. Example config.toml:

# config.toml
base_url = "http://localhost:8000"
title = "My Awesome Rust SSG Site"
content_dir = "content"
template_dir = "templates"
output_dir = "public"

2. Example Content (content/posts/first-post.md):

---
title: "My First Post"
date: 2026-03-01
description: "This is my very first blog post using the Rust SSG."
tags: ["rust", "ssg", "blog"]
---

# Hello from my First Post!

This is some **Markdown content**.

It's exciting to see this rendered into HTML.

3. Example Content (content/pages/about/index.md):

---
title: "About Us"
date: 2026-03-02
description: "Learn more about our project."
slug: "about-us" # Example slug override
---

# About Our Project

We are building a modern SSG with Rust!

4. Example Content (content/index.md):

---
title: "Homepage"
date: 2026-03-02
description: "Welcome to the homepage."
---

# Welcome to our Rust SSG Site!

Explore our [posts](/posts/first-post/) and [about page](/about-us/).

5. Example Templates (templates/base.html and templates/post.html):

<!-- templates/base.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{{ page.title }} | {{ config.title }}</title>
    <meta name="description" content="{{ page.description | default(value=config.description) }}">
    <style>
        body { font-family: sans-serif; margin: 2em; line-height: 1.6; }
        nav a { margin-right: 1em; }
    </style>
</head>
<body>
    <header>
        <h1>{{ config.title }}</h1>
        <nav>
            <a href="/">Home</a>
            <a href="/posts/first-post/">First Post</a>
            <a href="/about-us/">About</a>
        </nav>
    </header>
    <main>
        {% block content %}{% endblock content %}
    </main>
    <footer>
        <p>&copy; {{ now() | date(format="%Y") }} {{ config.title }}</p>
    </footer>
</body>
</html>

<!-- templates/post.html -->
{% extends "base.html" %}

{% block content %}
    <article>
        <h2>{{ page.title }}</h2>
        <p><em>Published: {{ page.date | date(format="%Y-%m-%d") }}</em></p>
        {{ page.content | safe }}
    </article>
{% endblock content %}

6. Run the build:

Navigate to your project root in the terminal and run:

cargo run

You should see log messages indicating the parsing and generation of each file:

INFO ssg_project::main - Starting SSG build process...
INFO ssg_project::main - Configuration loaded: Config { base_url: "http://localhost:8000", title: "My Awesome Rust SSG Site", description: None, content_dir: "content", template_dir: "templates", output_dir: "public" }
INFO ssg_project::main - Template renderer initialized from: templates
INFO ssg_project::main - Parsing content file: content/posts/first-post.md
INFO ssg_project::main - Parsing content file: content/pages/about/index.md
INFO ssg_project::main - Parsing content file: content/index.md
INFO ssg_project::main - Parsed 3 content items.
INFO ssg_project::builder - Starting site build to output directory: public
INFO ssg_project::builder - Generated: public/posts/first-post/index.html
INFO ssg_project::builder - Generated: public/about-us/index.html
INFO ssg_project::builder - Generated: public/index.html
INFO ssg_project::builder - Site build completed successfully.
INFO ssg_project::main - SSG build process finished successfully!

7. Verify Output:

Check your public directory. It should now contain:

public
├── index.html
├── posts
│   └── first-post
│       └── index.html
└── about-us
    └── index.html

Open these HTML files in your browser to verify they are correctly rendered and contain the content.

Production Considerations

Error Handling

• File I/O Errors: We’ve used anyhow::Result and .context() extensively. This is critical for production, providing clear error messages with context about what file operation failed and where.
• Path Manipulation: PathClean helps prevent issues with malformed paths. Always sanitize user-provided or dynamically generated paths.
• Template Rendering Failures: The TemplateRenderer should return Result types, and these errors are now propagated up through the SiteBuilder to main, ensuring that a broken template halts the build with a descriptive error.
• Parallel Processing Errors: rayon’s collect::<Result<Vec<()>>>()? ensures that if any single content item fails to build, the entire build process fails, rather than silently continuing with partial output.

Performance Optimization

• Parallel Content Processing: The use of rayon::par_iter() is the most significant performance gain for content-heavy sites. Rendering and writing are often CPU-bound tasks, and rayon effectively utilizes all available cores.
• Efficient File Operations: fs::create_dir_all is efficient as it only creates directories that don’t already exist.
• Clean Build: Clearing the public directory ensures that no stale files are left from previous builds, which can be important for consistency and avoiding unexpected behavior in production.

Security Considerations

• Path Sanitization: Using path-clean is a good practice to prevent directory traversal vulnerabilities, especially if slugs or permalinks could be controlled by untrusted input (e.g., in a CMS integration). While less critical for a purely static site generator where content is typically trusted, it’s a robust habit.
• Minimizing Dependencies: Keeping the dependency tree lean reduces the attack surface.

Logging and Monitoring

• Structured Logging: env_logger (or tracing for more advanced needs) provides clear output. Using info!, warn!, error!, and debug! macros allows for configurable log levels, essential for debugging in development and monitoring in CI/CD.
• Visibility: Logging each generated file provides real-time feedback during the build process, which is helpful for large sites.

Code Review Checkpoint

At this point, we have significantly extended our SSG:

• New Module: src/builder.rs contains the SiteBuilder struct and its associated logic.
• Core Logic:
  • SiteBuilder::determine_output_path: Implements the routing logic, mapping Content objects to PathBuf for output files. It handles permalink and slug overrides and ensures pretty URLs.
  • SiteBuilder::render_and_write_content: Orchestrates rendering a single content item and writing it to disk, including directory creation.
  • SiteBuilder::build: The main build method, which now clears the output directory, processes all content in parallel using rayon, and handles overall error propagation.
• Modified Files:
  • Cargo.toml: Added path-clean and rayon.
  • src/main.rs: Updated to instantiate and run the SiteBuilder.
  • src/content.rs, src/renderer.rs (implicitly): Their interfaces are used by the SiteBuilder.

This integration means our SSG can now take raw content, process it through the pipeline, and produce a fully functional static website ready for deployment.

Common Issues & Solutions

1. Issue: Permission denied error when writing files.

   • Cause: The user running cargo run does not have write permissions to the public directory or its parent.
   • Solution:
     • Ensure your public directory is not read-only.
     • Run cargo run from a directory where your user has write permissions.
     • On Linux/macOS, check permissions with ls -l and use chmod if necessary (e.g., chmod -R 777 public for testing, but be careful in production).
     • On Windows, ensure the directory isn’t locked by another process or restricted by UAC.

2. Issue: Incorrect output paths (e.g., public/content/posts/first-post/index.html instead of public/posts/first-post/index.html).

   • Cause: The strip_prefix(&self.config.content_dir) call in determine_output_path might not be working as expected, or content_dir might not be correctly configured.
   • Solution:
     • Double-check your config.toml content_dir value matches the actual content directory name.
     • Add debug! logs inside determine_output_path to print content.source_path, self.config.content_dir, and relative_source_path to understand how the path is being processed.
     • Ensure content.source_path is an absolute path or relative to the project root, consistent with how content_dir is interpreted.

3. Issue: Template not found or Error rendering template during build.

   • Cause:
     • Incorrect template_dir in config.toml.
     • Template file names (e.g., post.html) don’t match what’s expected in Content::template_name.
     • Syntax errors within your Tera templates.
   • Solution:
     • Verify config.toml’s template_dir is correct.
     • Ensure the template_name set in Content (e.g., “post.html”) exactly matches a file in your templates directory.
     • Check Tera template files for syntax errors. Tera often provides good error messages, which anyhow::Context will help surface.
     • Add debug! statements in TemplateRenderer::render_content to print the template name being used and the context data.

Testing & Verification

To fully test and verify the work done in this chapter:

1. Execute a Full Build:

   cargo run
   

   Observe the console output. Look for INFO messages indicating each file being generated and a final “Site build completed successfully.” message. Any ERROR messages should be investigated.

2. Inspect the public Directory:

   • Verify its existence and that it’s clean (no old files).
   • Check the directory structure matches the routing logic:
     • public/index.html
     • public/posts/first-post/index.html
     • public/about-us/index.html (due to the slug override)
   • Ensure no unexpected files or directories are present.

3. Open Generated HTML Files in a Browser:

   • Navigate to public/index.html, public/posts/first-post/index.html, and public/about-us/index.html using your browser’s “Open File” feature or by serving the public directory with a simple local HTTP server (e.g., python3 -m http.server in the public directory).
   • Verify:
     • Content from Markdown is correctly rendered.
     • Frontmatter data (title, description, date) is injected into the template.
     • The correct base template is used.
     • Internal links (like those in index.md) work correctly when navigating between the generated pages.

This comprehensive verification process confirms that our routing and output generation pipeline is working as intended, producing a coherent and navigable static website.

Summary & Next Steps

In this chapter, we achieved a significant milestone: we built the core pipeline for our Rust SSG. We implemented sophisticated routing logic to map content source paths and frontmatter overrides to clean, pretty URLs in our output directory. We then developed the SiteBuilder to orchestrate the entire build process, including clearing the output directory, creating necessary subdirectories, and efficiently writing rendered HTML files in parallel using rayon. This makes our SSG not just functional but also performant for larger sites.

We also discussed critical production considerations, including robust error handling, performance optimizations like parallel processing, path sanitization for security, and effective logging for build visibility.

With a fully generated static site in our public directory, we’ve completed the fundamental content processing and rendering loop. The next logical step is to enhance the navigability and discoverability of our site. In Chapter 9: Advanced Navigation and Linking, we will delve into generating internal links automatically, creating dynamic navigation menus, and implementing a table of contents for long-form content, further improving the user experience of our generated static sites.