Chapter 15: Debugging, Testing, and Observability in SpaceTimeDB

Introduction

Welcome to Chapter 15! As we’ve journeyed through the capabilities of SpaceTimeDB, building real-time, collaborative applications, you might have encountered situations where things didn’t quite work as expected. This is a natural part of software development, and it highlights the critical importance of debugging, testing, and observability.

In this chapter, we’ll equip you with the essential skills and tools to confidently diagnose problems, ensure the correctness of your SpaceTimeDB logic, and monitor your applications in production. We’ll explore strategies for both server-side (reducer) and client-side debugging, delve into writing robust unit and integration tests, and discuss how to establish comprehensive observability using logs, metrics, and tracing. By the end of this chapter, you’ll not only be able to build powerful SpaceTimeDB applications but also maintain and scale them with confidence.

To get the most out of this chapter, you should have a solid understanding of SpaceTimeDB’s core concepts, including schema definition, reducers, and client-side interaction, as covered in previous chapters. Let’s dive in and make your SpaceTimeDB applications rock-solid!

Core Concepts

Building reliable real-time systems with SpaceTimeDB requires a structured approach to identifying and preventing issues. This section lays out the fundamental concepts for debugging, testing, and observing your applications.

Debugging SpaceTimeDB Applications

Debugging is the process of finding and fixing errors or bugs in your code. With SpaceTimeDB, debugging involves both your server-side Rust reducers and your client-side application logic.

Server-Side Reducer Debugging

SpaceTimeDB’s reducers are written in Rust, which offers excellent debugging capabilities. When a reducer executes, it runs within the SpaceTimeDB server environment.

1. spacetime-cli logs: The most straightforward way to see what’s happening inside your reducers is through the spacetime-cli’s logging functionality. When you run spacetime-cli dev or spacetime-cli deploy, the server’s output, including any println! or dbg! macros from your Rust code, will be streamed to your terminal.
   • println!: This macro prints formatted text to standard output, similar to console.log() in JavaScript. It’s useful for inspecting variable values at specific points in your code.
   • dbg!: This macro is a powerful debugging tool in Rust. It prints the file name, line number, and expression of its argument, along with its value and a pretty-printed representation. It’s especially handy for quickly inspecting complex data structures.
2. Rust log Crate: For more structured and production-ready logging, you can integrate the log crate into your reducer modules. This allows you to log messages with different levels (info, debug, warn, error) and can be configured to integrate with external logging systems in a deployed environment.
3. Rust Debugger Integration: For deep introspection, you can attach a debugger (like rust-lldb or rust-gdb, often integrated into IDEs like VS Code via extensions like Rust Analyzer) to your SpaceTimeDB server process. This allows you to set breakpoints, step through code line by line, and inspect the call stack and variable states in real-time. This is typically done against a locally running SpaceTimeDB instance.

Client-Side Debugging

Your client application interacts with SpaceTimeDB through its client libraries (e.g., TypeScript/JavaScript). Standard frontend debugging techniques apply here.

1. Browser Developer Tools: For web clients, the browser’s developer tools (Console, Network, Sources tabs) are your best friends. You can set breakpoints in your JavaScript/TypeScript code, inspect network requests (including WebSocket messages to/from SpaceTimeDB), and observe the client-side state.
2. Client Library Logging: SpaceTimeDB client libraries often have configurable logging levels. Enabling verbose logging can provide insights into connection status, subscription updates, and any errors received from the server.

Testing SpaceTimeDB Logic

Testing is about verifying that your code behaves as expected under various conditions. For SpaceTimeDB, this means ensuring your reducers are deterministic and correct, and that your client-server interactions are robust.

Why Testing is Crucial for SpaceTimeDB

SpaceTimeDB’s core principle of deterministic reducers means that for a given initial state and a sequence of reducer calls, the final state will always be the same. This determinism makes testing highly effective and crucial:

• Reliability: Ensures your shared state remains consistent across all clients.
• Prevent Regressions: Catches unintended side effects when modifying existing code.
• Confidence: Allows you to refactor and evolve your schema and reducers without fear.

Types of Testing

1. Unit Testing Reducers: This focuses on testing individual reducer functions in isolation. You provide a mock initial state, call the reducer with specific arguments, and assert that the resulting state change (or error) is correct. Rust’s built-in testing framework (cargo test) is perfect for this.
2. Integration Testing: This involves testing the interaction between different components. For SpaceTimeDB, this could mean:
   • Testing a client connecting to a local SpaceTimeDB instance.
   • Calling reducers from the client and observing the resulting state changes propagated back to the client.
   • Verifying schema evolution and data migrations.
3. End-to-End (E2E) Testing: Simulating real user interactions across your entire application stack, from the UI to SpaceTimeDB and back. This often involves tools like Playwright or Cypress for web applications, interacting with your SpaceTimeDB client.

Observability in Production

Once your SpaceTimeDB application is deployed, debugging with println! isn’t enough. Observability is the ability to understand the internal state of a system by examining its external outputs. It’s crucial for diagnosing issues in production environments without needing to deploy new code. The three pillars of observability are logs, metrics, and traces.

1. Logging

Logs are discrete, time-stamped records of events that happen within your application.

• Structured Logging: Instead of plain text, log data as structured JSON. This makes it machine-readable and easier to query and analyze in centralized logging systems (e.g., ELK Stack, Splunk, DataDog).
• Contextual Information: Include relevant context in your logs, such as user_id, reducer_name, table_name, error_code, or request_id.
• SpaceTimeDB Specifics: Your reducers should emit logs for important events (e.g., data mutations, authorization failures, complex logic branches). Your client-side code should log connection events, subscription changes, and client-side errors.

2. Metrics

Metrics are numerical measurements collected over time, often aggregated. They provide a high-level view of your system’s health and performance.

• Key SpaceTimeDB Metrics:
  • Reducer Execution Time: Average, p95, p99 latency for each reducer.
  • Reducer Call Count: How often each reducer is invoked.
  • Client Connection Count: Number of active WebSocket connections.
  • Data Change Rate: How many rows are inserted/updated/deleted per second.
  • Subscription Latency: Time taken for state changes to propagate to clients.
  • Error Rates: Number of reducer errors or client-side errors.
• Monitoring Tools: Metrics are typically collected by agents and sent to monitoring dashboards (e.g., Prometheus/Grafana, DataDog, New Relic) for visualization and alerting.

3. Tracing (Distributed Tracing)

Traces represent the end-to-end journey of a request or operation through a distributed system. While SpaceTimeDB itself provides a unified backend, your overall application might involve other services (e.g., an authentication service, external APIs).

• Correlation IDs: Propagate a unique correlation ID from the initial client request through your SpaceTimeDB reducer calls and any subsequent external service calls. This allows you to link related log entries and metric events.
• OpenTelemetry: Consider using a standard like OpenTelemetry to instrument your client and reducer code (if SpaceTimeDB integrates with it directly or allows custom instrumentation) to generate and export traces.

Alerting

Alerting is the act of notifying a team when a metric crosses a predefined threshold or an important log event occurs.

• Critical Alerts: For production-impacting issues (e.g., high error rates, server down).
• Warning Alerts: For potential issues that need investigation (e.g., increasing latency, resource utilization spikes).

Observability Architecture

• Client Application: Your frontend, game client, or other application using the SpaceTimeDB client library.
• SpaceTimeDB Server: The core SpaceTimeDB instance hosting your schema and reducers.
• Reducer Logic: The Rust code you write that modifies the database state.
• Logging System: Collects structured logs from both client and server.
• Metrics System: Collects numerical performance and health metrics.
• Monitoring Dashboard: Visualizes logs and metrics, allowing for querying and analysis.
• On-Call Team: Receives alerts when critical issues arise.
• Debugging Tools: Used during development for interactive problem-solving.

This diagram illustrates how development debugging tools integrate with the system during development, and how logs and metrics flow into dedicated systems for production observability.

Step-by-Step Implementation

Let’s get practical and implement some debugging and testing techniques.

Setting up a Basic Project (if not already done)

For this chapter, we’ll assume you have a basic SpaceTimeDB project set up. If not, quickly create one:

# Ensure you have spacetime-cli installed (latest stable as of 2026-03-14, e.g., v2.1.0)
# Check official docs for installation: https://spacetimedb.com/docs/getting-started/installation
# For example, `curl -sL https://install.spacetimedb.com | bash`

spacetime-cli new my_debug_test_app
cd my_debug_test_app

Now, let’s define a simple schema and reducer. Open src/lib.rs and replace its content:

// src/lib.rs
use spacetimedb::{spacetimedb, table, ReducerContext};

// Define a simple Counter table
#[spacetimedb(table)]
pub struct Counter {
    #[primarykey]
    pub id: u384, // Using u384 for a unique identifier, common in SpaceTimeDB for primary keys
    pub value: u32,
}

// Define a reducer to increment the counter
#[spacetimedb(reducer)]
pub fn increment_counter(ctx: ReducerContext, id: u384, amount: u32) -> Result<(), String> {
    // Attempt to get the counter by ID
    let mut counter = Counter::filter_by_id(&id)
        .ok_or_else(|| format!("Counter with id {} not found.", id))?;

    // Increment the value
    counter.value = counter.value.checked_add(amount)
        .ok_or_else(|| "Counter overflowed!".to_string())?;

    // Update the counter in the database
    counter.update();

    Ok(())
}

// Define a reducer to create a new counter
#[spacetimedb(reducer)]
pub fn create_counter(ctx: ReducerContext, id: u384, initial_value: u32) -> Result<(), String> {
    if Counter::filter_by_id(&id).is_some() {
        return Err(format!("Counter with id {} already exists.", id));
    }
    Counter::insert(Counter { id, value: initial_value });
    Ok(())
}

Build and run your SpaceTimeDB instance:

spacetime-cli dev

You should see output indicating the server is running and your module is deployed.

Step 1: Server-Side Debugging with println! and dbg!

Let’s add some debugging statements to our increment_counter reducer.

Open src/lib.rs again and modify the increment_counter reducer:

// src/lib.rs (modifications within increment_counter reducer)
// ... existing code ...

#[spacetimedb(reducer)]
pub fn increment_counter(ctx: ReducerContext, id: u384, amount: u32) -> Result<(), String> {
    // Debugging step 1: Print the incoming arguments
    println!("Reducer `increment_counter` called with ID: {:?}, Amount: {}", id, amount);

    let mut counter = Counter::filter_by_id(&id)
        .ok_or_else(|| {
            let error_msg = format!("Counter with id {} not found.", id);
            // Debugging step 2: Log error before returning
            eprintln!("Error in `increment_counter`: {}", error_msg);
            error_msg
        })?;

    // Debugging step 3: Inspect the counter object before modification
    dbg!(&counter); // `dbg!` takes a reference and prints it

    counter.value = counter.value.checked_add(amount)
        .ok_or_else(|| "Counter overflowed!".to_string())?;

    // Debugging step 4: Inspect the counter object after modification
    dbg!(&counter);

    counter.update();

    // Debugging step 5: Confirm success
    println!("Counter with ID {:?} successfully incremented to {}", id, counter.value);

    Ok(())
}

// ... existing code ...

Explanation:

• println! is used for simple messages, showing the input parameters and success messages.
• eprintln! is used for error messages, which typically go to standard error, making them distinct in logs.
• dbg!(&counter) is used to print the counter struct’s content before and after modification. The & means we’re passing a reference, so dbg! doesn’t consume ownership of counter.

Now, rebuild and redeploy your module by saving src/lib.rs. The spacetime-cli dev command should automatically detect changes and recompile.

Let’s call these reducers from the spacetime-cli to see the logs. Open a new terminal window (keep spacetime-cli dev running in the first one).

# Create a new counter
spacetime-cli call create_counter --args '[{"id": 123, "initial_value": 0}]'

# Increment the counter
spacetime-cli call increment_counter --args '[{"id": 123, "amount": 5}]'

# Try to increment a non-existent counter (should trigger error log)
spacetime-cli call increment_counter --args '[{"id": 456, "amount": 10}]'

Go back to your first terminal running spacetime-cli dev. You should see the println! and dbg! outputs mixed in with the SpaceTimeDB server logs. Notice how dbg! includes file, line, and value, which is incredibly helpful!

Step 2: Unit Testing a Reducer

Now, let’s write a unit test for our increment_counter reducer. SpaceTimeDB reducers are just Rust functions, so we can test them using Rust’s standard testing framework.

Create a new file src/tests.rs in your project root, or add the tests directly to src/lib.rs inside a #[cfg(test)] module. For simplicity, let’s add it to src/lib.rs.

Add the following block to the end of your src/lib.rs file:

// src/lib.rs (add this at the very end of the file)
#[cfg(test)]
mod tests {
    use super::*;
    use spacetimedb::{
        test_client::{
            ClientDb,
            ReducerContext, // Use the test client's ReducerContext
        },
        table,
    };

    // Define a test-specific Counter table if needed, or reuse the main one
    // For unit tests, we often mock the tables or use the same table definition
    // but operate on a test-specific ClientDb.

    #[test]
    fn test_create_counter_success() {
        let mut client_db = ClientDb::new();
        let ctx = ReducerContext::new(
            "test_identity".to_string(),
            client_db.timestamp_millis()
        );

        let id = u384::from(100);
        let initial_value = 50;

        // Call the reducer
        let result = create_counter(ctx, id, initial_value);

        // Assert the result is Ok
        assert!(result.is_ok(), "create_counter should succeed");

        // Verify the counter was inserted
        let counter = client_db.get_one::<Counter>().expect("Counter should exist");
        assert_eq!(counter.id, id);
        assert_eq!(counter.value, initial_value);
    }

    #[test]
    fn test_create_counter_already_exists() {
        let mut client_db = ClientDb::new();
        let ctx = ReducerContext::new(
            "test_identity".to_string(),
            client_db.timestamp_millis()
        );

        let id = u384::from(101);
        let initial_value = 10;

        // First, create the counter
        let _ = create_counter(ctx.clone(), id, initial_value).unwrap(); // Use clone for ctx

        // Try to create it again
        let result = create_counter(ctx, id, 20);

        // Assert that it returns an error
        assert!(result.is_err(), "create_counter should fail if counter already exists");
        assert_eq!(result.unwrap_err(), format!("Counter with id {} already exists.", id));
    }

    #[test]
    fn test_increment_counter_success() {
        let mut client_db = ClientDb::new();
        let ctx = ReducerContext::new(
            "test_identity".to_string(),
            client_db.timestamp_millis()
        );

        let id = u384::from(1);
        let initial_value = 10;
        let increment_amount = 5;

        // First, create the counter for the test
        client_db.insert(Counter { id, value: initial_value });

        // Call the increment reducer
        let result = increment_counter(ctx, id, increment_amount);

        // Assert the result is Ok
        assert!(result.is_ok(), "increment_counter should succeed");

        // Verify the counter value was updated
        let counter = client_db.get_one::<Counter>().expect("Counter should exist after increment");
        assert_eq!(counter.id, id);
        assert_eq!(counter.value, initial_value + increment_amount);
    }

    #[test]
    fn test_increment_counter_not_found() {
        let client_db = ClientDb::new(); // No need for mut if we're not inserting/updating
        let ctx = ReducerContext::new(
            "test_identity".to_string(),
            client_db.timestamp_millis()
        );

        let non_existent_id = u384::from(999);
        let increment_amount = 1;

        // Call the increment reducer on a non-existent ID
        let result = increment_counter(ctx, non_existent_id, increment_amount);

        // Assert that it returns an error
        assert!(result.is_err(), "increment_counter should fail if counter not found");
        assert_eq!(result.unwrap_err(), format!("Counter with id {} not found.", non_existent_id));
    }

    #[test]
    fn test_increment_counter_overflow() {
        let mut client_db = ClientDb::new();
        let ctx = ReducerContext::new(
            "test_identity".to_string(),
            client_db.timestamp_millis()
        );

        let id = u384::from(2);
        let initial_value = u32::MAX - 1; // Close to max value
        let increment_amount = 2; // Will cause overflow

        // Create the counter with a value near max
        client_db.insert(Counter { id, value: initial_value });

        // Call the increment reducer
        let result = increment_counter(ctx, id, increment_amount);

        // Assert that it returns an error
        assert!(result.is_err(), "increment_counter should fail on overflow");
        assert_eq!(result.unwrap_err(), "Counter overflowed!".to_string());
    }
}

Explanation:

• #[cfg(test)] mod tests { ... }: This module will only be compiled when running tests (cargo test).
• use spacetimedb::test_client::{ClientDb, ReducerContext};: We import ClientDb which acts as an in-memory database instance for testing, and ReducerContext specifically designed for tests.
• #[test]: Marks a function as a test function.
• ClientDb::new(): Creates a fresh, empty in-memory database for each test, ensuring isolation.
• client_db.insert(Counter { ... }): Allows you to set up the initial state of your database for the test.
• client_db.get_one::<Counter>(): Retrieves a single record from the test database.
• assert!, assert_eq!: Standard Rust macros for asserting conditions.

Now, run your tests from the root of your project:

cargo test

You should see output indicating that all your tests passed! This demonstrates how you can thoroughly test your reducer logic in isolation, ensuring its determinism and correctness.

Step 3: Client-Side Logging (Conceptual)

While we can’t run a full client example here, let’s conceptually discuss how you’d enable logging for a TypeScript/JavaScript client.

When initializing your SpaceTimeDB client, you typically have options to configure logging:

// Example: src/client/index.ts (conceptual)
import { SpacetimeDBClient } from '@clockworklabs/spacetimedb-client';

// Assume SpaceTimeDB is running locally on default port 3000
const client = new SpacetimeDBClient('ws://localhost:3000');

// Set log level (e.g., 'debug', 'info', 'warn', 'error', 'none')
// The exact method might vary slightly with client library updates,
// but usually involves a config object or a dedicated method.
client.setLogLevel('debug'); // Or client.configure({ logLevel: 'debug' });

client.onConnect(() => {
    console.log('SpacetimeDB client connected!');
    // Example: subscribe to the Counter table
    client.subscribe(['Counter']);
});

client.onDisconnect(() => {
    console.warn('SpacetimeDB client disconnected!');
});

client.on('error', (error) => {
    console.error('SpacetimeDB client error:', error);
});

// Listen for table updates
client.on('Counter:insert', (counter) => {
    console.log('New Counter inserted:', counter);
});

client.on('Counter:update', (oldValue, newValue) => {
    console.log('Counter updated from', oldValue, 'to', newValue);
});

client.connect();

// Later, call a reducer
// client.call('create_counter', 123, 0);
// client.call('increment_counter', 123, 1);

By enabling debug level logging, your client’s console would show detailed messages about WebSocket frames, subscription states, and reducer call acknowledgements. This is invaluable for understanding client-side synchronization issues.

Mini-Challenge: Test a Conditional Reducer

It’s your turn! Let’s add a new reducer and write tests for it.

Challenge: Create a new reducer called decrement_counter that takes an id and an amount. This reducer should:

1. Find the Counter by id. If not found, return an error.
2. Decrement its value by amount.
3. Crucially: If decrementing would make the value go below zero, it should instead return an error (“Cannot decrement below zero!”).
4. Update the Counter in the database.

After implementing the reducer, write at least three unit tests for decrement_counter in your src/lib.rs’s #[cfg(test)] block:

• One test for a successful decrement.
• One test for decrementing a non-existent counter.
• One test for attempting to decrement below zero.

Hint:

• Use checked_sub for safe subtraction, similar to how checked_add was used.
• Remember to set up the ClientDb with appropriate initial state for each test.

What to observe/learn:

• How to handle conditional logic and error cases within reducers.
• How to write comprehensive unit tests that cover different scenarios, including edge cases.
• The importance of Result<(), String> for reducer return types to convey success or specific errors.

// Add this new reducer to src/lib.rs
#[spacetimedb(reducer)]
pub fn decrement_counter(ctx: ReducerContext, id: u384, amount: u32) -> Result<(), String> {
    // TODO: Implement the logic here
    unimplemented!("Decrement counter reducer not yet implemented for the challenge!")
}

// Add your tests within the existing #[cfg(test)] mod tests { ... } block
// For example:
/*
    #[test]
    fn test_decrement_counter_success() {
        // ... your test code ...
    }

    #[test]
    fn test_decrement_counter_not_found() {
        // ... your test code ...
    }

    #[test]
    fn test_decrement_counter_below_zero() {
        // ... your test code ...
    }
*/

Once you’ve implemented the reducer and tests, run cargo test to verify your solution.

Common Pitfalls & Troubleshooting

Even with good practices, issues can arise. Knowing common pitfalls helps in quicker diagnosis.

1. Reducer Non-Determinism: This is a critical error in SpaceTimeDB. A reducer is non-deterministic if, given the same initial state and input arguments, it can produce different outputs (different final states or different errors).
   • Cause: Using ctx.timestamp_millis() or ctx.caller() directly to generate unique IDs or random numbers, or relying on external non-deterministic factors.
   • Troubleshooting: SpaceTimeDB’s runtime is designed to detect non-determinism. If you encounter errors related to “non-deterministic reducer,” carefully review your reducer logic.
     • Solution: For unique IDs, use id: u384 as a reducer argument, generated client-side, or use the ctx.timestamp_millis() for ordering events, but not for generating primary keys directly if multiple clients could call it simultaneously. For randomness, seed your random number generator with a deterministic value (e.g., id or ctx.timestamp_millis()) and acknowledge that this is pseudo-randomness relative to the deterministic seed.
2. Stale Client State / Synchronization Issues: Your client isn’t reflecting the latest data from the server.
   • Cause:
     • Client not subscribed to the relevant tables.
     • Network connectivity issues (WebSocket drops).
     • Client-side caching logic interfering with SpaceTimeDB updates.
     • Reducer logic not actually updating the expected table.
   • Troubleshooting:
     • Client-side: Use browser dev tools (Network tab) to inspect WebSocket traffic. Look for table_update messages. Check if your client’s on listeners are correctly set up and processing updates.
     • Server-side: Verify that your reducer successfully updates the table using spacetime-cli logs or by querying the database directly after a reducer call.
3. Performance Bottlenecks: Reducers or queries taking too long, leading to slow updates or a sluggish UI.
   • Cause:
     • Inefficient reducer logic (e.g., iterating over large tables, complex calculations).
     • Schema design issues (missing indexes, overly complex relations).
     • Large number of active subscriptions overwhelming the server or client.
   • Troubleshooting:
     • Server-side: Use spacetime-cli logs to see reducer execution times (SpaceTimeDB often logs these by default). Add println!/dbg! to time critical sections of your reducer. Ensure appropriate primary keys and indexes are defined in your schema.
     • Client-side: Monitor client-side frame rates and network latency. Optimize how your client processes updates and renders data.
4. Reducer Errors Not Propagating: A reducer fails, but the client doesn’t get an error response.
   • Cause: The client code isn’t handling the Result of the reducer call correctly, or the error is being swallowed somewhere.
   • Troubleshooting: Ensure your client-side client.call() handler explicitly checks for and logs errors. Review the spacetime-cli logs to confirm the reducer actually returned an Err and wasn’t just silently failing.

Summary

Congratulations! You’ve reached the end of this crucial chapter on debugging, testing, and observability in SpaceTimeDB. We’ve covered a lot of ground, equipping you with the knowledge to build more robust and maintainable real-time applications.

Here are the key takeaways:

• Debugging is essential: Use println!, eprintln!, dbg!, and Rust debugger integration for server-side (reducer) issues, and browser dev tools with client-side logging for frontend problems.
• Testing ensures correctness: SpaceTimeDB’s deterministic nature makes it highly testable. Implement unit tests for individual reducers using cargo test and ClientDb to ensure their logic is sound and robust against edge cases like non-existent data or overflows.
• Observability is paramount for production: Beyond development debugging, establish a system for production monitoring using structured logs, comprehensive metrics (reducer latency, error rates, connection counts), and potentially distributed tracing.
• Common pitfalls: Be aware of reducer non-determinism, stale client state, performance bottlenecks, and unhandled reducer errors, and know how to troubleshoot them effectively.
• Version Focus: We’ve used SpaceTimeDB v2.x and current Rust practices, ensuring modern best practices.

By integrating these practices into your development workflow, you’ll gain confidence in deploying and operating your SpaceTimeDB applications, knowing you have the tools to understand, verify, and diagnose their behavior.

What’s Next?

In the next chapter, we’ll shift our focus to deployment strategies and scaling considerations for SpaceTimeDB applications, taking your projects from local development to production-ready systems.

References

• SpacetimeDB Official Documentation
• The Rust Programming Language Book
• Rust log crate documentation
• Rust dbg! macro documentation

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.