DocumentationSubstreamsBasicsSubstreams CLI

Substreams CLI

This section provides a high level overview of the functionality of the Substreams CLI.

It provides several commands that will be instrumental on your Substreams development journey.

A complete list of the commands and their descriptions can be found by using the substreams --help command.

Available Commands:
  alpha       Group of commands that are currently being available for testing but could change at any time
  completion  Generate the autocompletion script for the specified shell
  graph       Generate mermaid-js graph document
  gui         Stream module outputs from a given package on a remote endpoint
  help        Help about any command
  info        Display package modules and docs
  init        Initialize a new, working Substreams project from scratch
  inspect     Display low-level package structure
  pack        Build an .spkg out of a .yaml manifest
  protogen    Generate Rust bindings from a package
  run         Stream module outputs from a given package on a remote endpoint
  tools       Developer tools related to substreams

Flags:
  -h, --help                    help for substreams
      --ipfs-timeout duration   IPFS timeout when resolving substreams-based subgraphs as manifest (default 10s)
      --ipfs-url string         IPFS endpoint to resolve substreams-based subgraphs as manifest (default "https://ipfs.network.thegraph.com")
  -v, --version                 version for substreams

These commands will become more intuitive as we progress through the series. For now, let’s focus on some of the core commands that will be essential to your Substreams development workflow.

init

The substreams init command scaffolds a Substreams repository from user provided parameters in the CLI.

For example, if you wanted to build a Substreams package which tracks GRT token transfers on the Ethereum blockchain, you could pass in the following parameters following the substreams init command:

$ substreams init  
Project name (lowercase, numbers, undescores): grt_token
Protocol: Ethereum
Ethereum chain: Mainnet
✔ Contract address to track (leave empty to use "Bored Ape Yacht Club"): 0xc944e90c64b2c07662a292be6244bdf05cda44a7
Would you like to track another contract? (Leave empty if not): 
Tracking 1 contract(s), let's define a short name for each contract
✔ Choose a short name for c944e90c64b2c07662a292be6244bdf05cda44a7 (lowercase and numbers only): grt
Retrieving Ethereum Mainnet contract information (ABI & creation block)
Fetched contract ABI for c944e90c64b2c07662a292be6244bdf05cda44a7
Fetched initial block 11446769 for c944e90c64b2c07662a292be6244bdf05cda44a7 (lowest 11446769)
Generating ABI Event models for grt
  Generating ABI Events for Approval (owner,spender,value)
  Generating ABI Events for MinterAdded (account)
  Generating ABI Events for MinterRemoved (account)
  Generating ABI Events for NewOwnership (from,to)
  Generating ABI Events for NewPendingOwnership (from,to)
  Generating ABI Events for Transfer (from,to,value)
Track a dynamic datasource: N
Writing project files
Generating Protobuf Rust code
Project "grt_token" initialized at "/substreams/grt"
 
Run 'make build' to build the wasm code.
 
The following substreams.yaml files have been created with different sink targets:
 * substreams.yaml: no sink target
 * substreams.sql.yaml: PostgreSQL sink
 * substreams.clickhouse.yaml: Clickhouse sink
 * substreams.subgraph.yaml: Sink into Substreams-based subgraph

This command generates all necessary files for a Substreams project that tracks and outputs events from the GRT token contract. Depending on your use case, you may need to remove, edit, or tweak the scaffolded functionality to fit your specific requirements.

Using the init command is a great starting point for any new Substreams project you work on.

pack

The substreams pack command builds a .spkg file from a .yaml manifest file defined in your Substreams project.

These packages are ready-to-consume binaries containing all the necessary artifacts to run a Substreams project. They can then be shared via the (Substreams Registry)[https://substreams.dev] allowing other developers to make use of your Substreams package! This also means you have the ability to import other Substreams packages into your project to make use of them in your Substreams project. As stated previously, Substreams promote modularity, composability, and reusability, enabling developers to build on each other’s work and create complex data processing pipelines more efficiently.

protogen

You may have noticed that Substreams uses protobuf messages for module inputs and outputs, however Substreams modules are written using Rust. The substreams protogen command allows you to generate Rust code that can be utilised in your modules for each protobuf message defined in your schema.

Each time you update your protobuf schema you will need to run the substreams protogen command to re-generate the Rust bindings associated with your package.

Let’s take a protobuf message on the Substreams repository that we just initialized using substreams init as an example.

message grt_Transfer {
    string evt_tx_hash = 1;
    uint32 evt_index = 2;
    google.protobuf.Timestamp evt_block_time = 3;
    uint64 evt_block_number = 4;
    bytes from = 5;
    bytes to = 6;
    string value = 7;
}

Let’s extend the grt_Transfer message to include additional fields, for example, a USD value of the transfer. To do this, add the relevant field to the protobuf message, and then run the substreams protogen command to generate the Rust bindings for the new message.

message grt_Transfer {
    string evt_tx_hash = 1;
    uint32 evt_index = 2;
    google.protobuf.Timestamp evt_block_time = 3;
    uint64 evt_block_number = 4;
    bytes from = 5;
    bytes to = 6;
    string value = 7;
    string amount_usd = 8;
}
substreams protogen ./substreams.yaml --exclude-paths="sf/ethereum,sf/substreams,google"

The newly generated code is then saved to src/pb/contract.v1.rs and looks like the following:

#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct GrtTransfer {
    #[prost(string, tag="1")]
    pub evt_tx_hash: ::prost::alloc::string::String,
    #[prost(uint32, tag="2")]
    pub evt_index: u32,
    #[prost(message, optional, tag="3")]
    pub evt_block_time: ::core::option::Option<::prost_types::Timestamp>,
    #[prost(uint64, tag="4")]
    pub evt_block_number: u64,
    #[prost(bytes="vec", tag="5")]
    pub from: ::prost::alloc::vec::Vec<u8>,
    #[prost(bytes="vec", tag="6")]
    pub to: ::prost::alloc::vec::Vec<u8>,
    #[prost(string, tag="7")]
    pub value: ::prost::alloc::string::String,
    #[prost(string, tag="8")]
    pub amount_usd: ::prost::alloc::string::String,
}

The make protogen command can also be used in replacement of the substreams protogen command. More on this in the next section.

info

Info is a helpful command that allows you to display all modules within a package and their relevant docs.

This command is particularly useful when importing an external Substreams package into your project. It provides information on the expected inputs and outputs of the modules, as well as their initial blocks, aiding the integration process.

Running the command on our previously initialised GRT token project outputs the following

$ substreams info
Package name: grt_token
Version: v0.1.0
Modules:
----
Name: map_events
Initial block: 11446769
Kind: map
Input: source: sf.ethereum.type.v2.Block
Output Type: proto:contract.v1.Events
Hash: 45c307063dbe4a06325b18a9f639f4badbfe6fc5
 
Name: db_out
Initial block: 11446769
Kind: map
Input: map: map_events
Output Type: proto:sf.substreams.sink.database.v1.DatabaseChanges
Hash: da0096c2107d805ae5365689497a3ae0e09e3133
 
Name: graph_out
Initial block: 11446769
Kind: map
Input: map: map_events
Output Type: proto:sf.substreams.entity.v1.EntityChanges
Hash: 4ca6b59421a5ed1b74516a361ae593d6000596fe
 
Network: mainnet

run

The substreams run command runs a Substreams project on a remote endpoint, allowing you to stream module outputs for your specified Substreams package.

Using the .spkg file that was output from the substreams pack command in the previous sections, we can start streaming some data.

Ensure you have authenticated with a Substreams provider before using substreams run. Information about how to authenticate can be found in the previous section.

The following command streams module outputs for the map_events module from our package on the Pinax endpoint that we previously authenticated with.

$ substreams run substreams.yaml map_events -e eth.substreams.pinax.network:443

The output should look like the following:

Connected (trace ID 1f81982fe49472298b0dca2c2dbd6118)
Progress messages received: 0 (0/sec)
Backprocessing history up to requested target block 11446769:
(hit 'm' to switch mode)





----------- BLOCK #11,446,769 (ec214b350fe1839500d8f10e6bb0a132a224d526c1e4fc5c939f3183bb6df8d0) ---------------
{
  "@module": "map_events",
  "@block": 11446769,
  "@type": "contract.v1.Events",
  "@data": {
    "grtMinterAddeds": [
      {
        "evtTxHash": "079625b9f58a40f1948b396b7007d09ff4aa193d7ec798923910fc179294cab8",
        "evtIndex": 42,
        "evtBlockTime": "2020-12-13T20:17:13Z",
        "evtBlockNumber": "11446769",
        "account": "BlkKZB3D60Pyzr5DVXY4nyCRFto="
      }
    ],
    "grtTransfers": [
      {
        "evtTxHash": "079625b9f58a40f1948b396b7007d09ff4aa193d7ec798923910fc179294cab8",
        "evtIndex": 41,
        "evtBlockTime": "2020-12-13T20:17:13Z",
        "evtBlockNumber": "11446769",
        "from": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
        "to": "BlkKZB3D60Pyzr5DVXY4nyCRFto=",
        "value": "10000000000000000000000000000"
      }
    ]
  }
}

----------- BLOCK #11,446,770 (0f1dfb864301b783c909bf4fc8b6e4224a19fac8087223ea1ccb25165883a94d) ---

...

You’ll be able to see the module output for each block as the stream progresses.

gui

The GUI command allows for the streaming of data in the same way as the substreams run command, but outputs data to a terminal based graphical user interface. This is a useful tool for navigating between blocks and modules, and allows you to visualise the data being streamed in a more user-friendly manner.

alt text

This will be a go-to tool for development, testing, and debugging issues in your Substreams projects.

A more comprehensive overview of the GUI can be found on the Streaming Fast GUI documentation.

graph

The substreams graph command generates a mermaid-js graph document

Although the example Substreams package built using the init command is fairly basic, as Substreams grow in size and complexity graph can be a useful tool to help visualize the data flow in your project.

AuthenticationProject Structure & Components