Project Structure & Components

In this section, we will explore the fundamental components of a Substreams package. These components include modules, Protobuf messages, manifests, and the Makefile.

Modules

Modules are the core building blocks of a Substreams package. Each module is a Rust function that takes some input(s), executes some developer defined logic, and then returns an output. The inputs and outputs of these modules can vary depending on the module type and the module definition, however they are always protobuf objects. There are two types of modules:

1. map: Used for stateless data extraction, filtering, and transformation.
2. store: Used for stateful transformations, allowing data persistence across blocks.

Modules can be chained, allowing the output of one module to be used as the input for another. Here’s a simple example of a map and store module:

#[substreams::handlers::map]
fn map_grt_transfers(blk: eth::Block) -> Result<contract::GrtTransfers, substreams::errors::Error> {
  let mut grt_transfers = contract::GrtTransfers::default();
 
  grt_transfers.append(
    &mut blk
      .events::<grt_contract::events::Transfer>(&[&GRT_CONTRACT_ADDRESS])
      .map(|(event, log)| {
        GrtTransfer {
          ...
        }
      })
      .collect(),
  );
  Ok(grt_transfers)
}
 
#[substreams::handlers::store]
pub fn store_transfer_count(grt_transfers: contract::GrtTransfers, store: StoreAddInt64) {
  for transfer in grt_transfers {
    store.add(transfer.log_ordinal, "transfer_count", 1);
  }
}

• The map module processes each block to extract and transform Transfer events into a GrtTransfers message. This processing is stateless as it does not retain any data between blocks.
• The store module receives the GrtTransfers message from the map module and updates a transfer count in a persistent store. This processing is stateful, as it maintains a count across all blocks processed.
• This chain of modules demonstrates how data can flow through a series of transformations and updates, enabling complex data processing workflows.

Protobuf Messages

Protobuf messages define the data structures used by modules as inputs and outputs.

Detailed information on utilizing protobuf messages to structure your date can be found in the Protobuf Programming Guide.

Built-in Types

Built in types provided by Firehose are available for use in Substreams modules, for example Block and Clock.

Built in types such a Block and Clock provided by Firehose for Ethereum based networks can be found in the sf.ethereum.type.v2 package.

Non-EVM based networks are also supported by Substreams, however these use different protobuf definitions, and have to be imported into your Substreams package if you are providing support for any of these networks. The non-EVM based networks and their relevant packages at the time of writing are:

• Near sf.near.type.v1
• Solana sf.solana.type.v1
• Bitcoin sf.bitcoin.type.v1
• Cosmos sf.cosmos.type.v1
• Arweave sf.arweave.type.v1

Extended vs Base An important aspect to note is the use of Extended vs Base blocks across different networks. The Block object from the mentioned packages can be an Extended or Base block. For Ethereum, the Block object is an Extended block. For up-to-date information on networks and their corresponding Block type, refer to the Streaming Fast Chains Documentation.

The Extended block variant includes all fields available in the Block object. In contrast, the Base block variant omits some fields. For instance, Extended block variants include transaction calls and balance changes within a block, while Base blocks do not. This distinction is crucial when building Substreams packages for chains that only support base blocks or when supporting both block variants in the same package.

A comprehensive list of the difference between the extended and base block can be found in the EVM Substreams Data Reference Guide.

Custom Types

Custom protobuf messages can also be defined by the developer. These custom messages are defined in the proto folder of the project, and converted to Rust types in the src folder utilising the protogen command we discussed in the previous chapter.

These custom protobuf messages allow you to build out a schema specific to the Substreams project you are building.

Here’s a basic example of a token transfer:

message Transfer {
    string evt_tx_hash = 1;
    uint32 evt_index = 2;
    google.protobuf.Timestamp evt_block_time = 3;
    uint64 evt_block_number = 4;
    bytes from = 5;
    bytes to = 6;
    string value = 7;
    string amount_usd = 8;
}

Manifest

The manifest file (substreams.yaml) is the central configuration file for a Substreams package. It defines the modules, their inputs, outputs, and other essential metadata required for the package to function.

Structure of a Manifest

A typical manifest file includes the following sections:

• Package Information: Metadata about the package, including its name, version, and description.
• Imports: External Protobuf definitions and other packages required by your Substreams package.
• Modules: Definitions of the modules, including their name, type (map or store), inputs, and outputs.

Example Manifest

Here’s an example of a simple manifest file pertaining to the previous example:

specVersion: v0.1.0
 
package:
  name: grt_token_substreams_powered_subgraph
  version: 0.1.0
 
network: mainnet
 
protobuf:
  files:
    - curve/v1/curve.proto
  importPaths:
    - ./proto
 
binaries:
  default:
    type: wasm/rust-v1
    file: ./target/wasm32-unknown-unknown/release/substreams.wasm
 
modules:
  - name: map_grt_transfers
    kind: map
    initialBlock: 11446769
    inputs:
      - source: sf.ethereum.type.v2.Block
    output:
      type: proto:contract.v1.GrtTransfers
 
  - name: store_transfer_count
    kind: store
    inputs:
      - map: map_grt_transfers
    updatePolicy: add
    valueType: int64
 
  - name: graph_out
    kind: map
    inputs:
      - map: map_grt_transfers
      - store: store_transfer_count

Key Components

• Package: This section contains metadata about the package such as name and version.
• Network: The network the Substreams package is executed on.
• Protobuf: This section defines Protobuf definitions that the Substreams package utilizes.
• Modules: This section defines the modules within the package, specifying their name, kind, initialBlock, inputs, output, and store specific fields such a updatePolicy and valueType (more on these in the Modules Section).

For comprehensive manifest reference documentation, refer to the Streaming Fast Manifests Reference Documentation.

Makefile

The Makefile generated when initializing a Substreams repository using the substreams init command automates common tasks, improving workflow efficiency by reducing the amount of text needed to run commands in the command line. Below is the generated Makefile.

CARGO_VERSION := $(shell cargo version 2>/dev/null)
 
.PHONY: build
build:
ifdef CARGO_VERSION
	cargo build --target wasm32-unknown-unknown --release
else
	@echo "Building substreams target using Docker. To speed up this step, install a Rust development environment."
	docker run --rm -ti --init -v ${PWD}:/usr/src --workdir /usr/src/ rust:bullseye cargo build --target wasm32-unknown-unknown --release
endif
 
.PHONY: run
run: build
	substreams run substreams.yaml $(if $(MODULE),$(MODULE),map_events) $(if $(START_BLOCK),-s $(START_BLOCK)) $(if $(STOP_BLOCK),-t $(STOP_BLOCK))
 
.PHONY: gui
gui: build
	substreams gui substreams.yaml $(if $(MODULE),$(MODULE),map_events) $(if $(START_BLOCK),-s $(START_BLOCK)) $(if $(STOP_BLOCK),-t $(STOP_BLOCK))
 
.PHONY: protogen
protogen:
	substreams protogen ./substreams.yaml --exclude-paths="sf/substreams,google"
 
.PHONY: pack
pack: build
	substreams pack substreams.yaml

Commands

1. Build

Command: make build Description: Compiles the Rust code to a WebAssembly target. If Cargo is installed, it uses Cargo; otherwise, it falls back to

2. Run

Command: make run Description: Builds the package and runs it using Substreams, with optional parameters for the module, start block, and stop block.

3. GUI

Command: make gui Description: Builds the package and runs the Substreams GUI, with optional parameters for the module, start block, and stop block.

4. Protogen

Command: make protogen Description: Generates Protobuf definitions for the project, excluding specified paths.

5. Pack

Command: make pack Description: Builds and packages the Substreams module.

Extensibility

The Makefile can be extended to include additional automation tasks, further streamlining the development process. For example, a command make test which tests the Substreams package can be added to the Makefile:

.PHONY: test
test:
    cargo test