Running and Testing Substreams

This is an important part of the development process, and running and validating your Substreams modules locally is a core part of a Substreams developer’s workflow.

Running Your Substreams Locally

Running your Substreams locally allows you to test and validate your modules. The following command runs the Substreams using the GUI:

substreams gui substreams.yaml graph_out -e eth.substreams.pinax.network:443

Command Breakdown

• substreams: The command to run the Substreams CLI.
• gui: Launches the GUI for visualizing and debugging the Substreams data flow.
• substreams.yaml: The manifest file defining your Substreams configuration and modules.
• graph_out: Specifies the module to run and output results.
  • Although we have specified this module as the output, the GUI will display results for all modules that graph_out is dependent on.
• -e eth.substreams.pinax.network:443: Specifies the endpoint to connect to the Ethereum network.
  • This should be the endpoint we have authenticated with earlier. Refer back to the Chapter on Authentication for more information.

Running with a Specific Block Range

Sometimes we may only want to run our Substreams within a specific range of blocks. To do this, use the -s (start block) and -t (end block) parameters:

substreams gui substreams.yaml graph_out -e eth.substreams.pinax.network:443 -s 12400000 -t 12500000

This command will execute your Substreams from block 12,400,000 to block 12,500,000.

Validating Module Outputs in the GUI

Using the GUI, you can validate the output of each module against expected on-chain data. This is a critical step to ensure that your Substreams produce accurate and reliable data.

In the previous sections, we discussed tracking Uniswap pool creations. By running your Substreams locally and using the GUI, you can inspect the output of the map_pools_created module. For example, at block 12369739, the expected output from our Uniswap Substreams package is:

{
  "pools": [
    {
      "address": "1d42064fc4beb5f8aaf85f4617ae8b3b5b8bd801",
      "token0": {
        "address": "1f9840a85d5af5bf1d1762f925bdaddc4201f984",
        "name": "Uniswap",
        "symbol": "UNI",
        "decimals": "18"
      },
      "token1": {
        "address": "c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
        "name": "Wrapped Ether",
        "symbol": "WETH",
        "decimals": "18"
      },
      "createdAtTxHash": "37d8f4b1b371fde9e4b1942588d16a1cbf424b7c66e731ec915aca785ca2efcf",
      "createdAtBlockNumber": "12369739",
      "createdAtTimestamp": "1620157956",
      "logOrdinal": "835"
    }
  ]
}

This output matches the on-chain event logs for the given transaction, which you can verify on Etherscan. Verifying your outputs against on-chain data will be a common task whilst developing Substreams.

Adding Logs

Adding debug logs to your modules is essential for troubleshooting and verifying the correctness of your Substreams. The Substreams crate comes with a log module. Use the substreams::log::debug! macro to add debug statements within your Rust code:

substreams::log::debug!("Cannot find pool in store with address {}", &pool.address);

Debug logs provide valuable insights into the data flow and help identify issues by logging intermediate states and values.

Here’s an example of adding a debug log to a module that processes pool events:

#[substreams::handlers::map]
pub fn map_pool_events(
    blk: eth::Block,
    pool_store: StoreGetProto<Pool>,
) -> Result<PoolEvents, substreams::errors::Error> {
    let mut pool_events = PoolEvents::default();
 
    for trx in blk.transactions() {
        for (log, _) in trx.logs_with_calls() {
            let pool_address = Hex::encode(&log.address);
            if let Some(pool) = pool_store.get_last(format!("pool:{}", &pool_address)) {
                substreams::log::debug!("Found pool: {}", &pool_address);
                if let Some(swap) = abi::pool_contract::events::Swap::match_and_decode(&log) {
                    ...
                }
            } else {
                substreams::log::debug!("Cannot find pool in store with address {}", &pool_address);
            }
        }
    }
    Ok(pool_events)
}

The crate also comes with a substreams::log::info! macro that logs a message at the INFO level. This can be useful for providing extra context to your Substreams in production environments.