Introduction to Substreams
Welcome to Substreams Development Basics! This series of guides is designed to introduce you to Substreams, a powerful blockchain indexing technology for constructing scalable and incredibly efficient data processing pipelines tailored for blockchain data.
In this series, we will be delving into the core building blocks of Substreams, as well as providing you with practical knowledge of how to build and deploy Substreams Powered Subgraphs. Although this series will focus on Substreams Powered Subgraphs, there are other data sinks that are also compatible with Substreams.
Advisory and Prerequisites
Before diving into Substreams, it’s advisable to have a foundational understanding of Rust and Subgraphs. Subgraph knowledge is essential for understanding how to build Substreams-powered Subgraphs, and Rust is what we build our Substreams packages in. Here are some resources to get you started:
Rust Programming Language:
Subgraphs and The Graph Protocol:
- The Graph Documentation
- Creating a Subgraph
- BuildersDAO - Subgraph Development Basics
- BuildersDAO - Subgraph Data Reference Guide
What are Substreams?
Substreams is a cutting-edge blockchain indexing technology developed by Streaming Fast for The Graph Network. Substreams aim to streamline the process of extracting, transforming, and sinking blockchain data by allowing developers to build composable data streams.
Substreams build upon The Graph’s general purpose indexing technology by leveraging Firehose, a core compoment of Streaming Fast’s technology stack that optimizes the data extraction process. Utilizing parallelization, Substreams enables data processing pipelines to scale and process vast amounts of blockchain data in near real-time. This advancement in The Graph’s indexing technology has significantly improved the performance of Subgraphs, reducing sync times by up to 100x in some cases!
For a deeper dive into firehose and its capabilities, checkout the Firehose Documentation.
Key Features
Real-time processing: Substreams can process vast amounts of blockchain data in near real-time. High Scalability: Substreams harness parallel processing to efficiently manage vast volumes of blockchain data supporting scalable and robust data operations across multiple networks. Enhanced Developer Experience: Substreams CLI provides a GUI offering developers the ability to locally run and debug their Substreams in an efficient manner. Composable Substreams modules greatly reduce development time. Modular Architecture: The design of Substreams supports a modular approach, allowing developers to build, share, and reuse Substreams modules. This modularity enables faster development cycles by allowing you to import pre-built components such as pricing Substreams modules.
Modularity and Composability
The importance of the modular and composable nature of Substreams should not be understated. Substreams promote modularity, enabling developers to separate functionalities into distinct modules. This allows for a clear separation of concerns between different parts of the business logic, resulting in maintainable and scalable codebases.
With traditional Subgraphs, common functionality is often rebuilt repeatedly. Substreams, however, promote composability. Developers can leverage existing modules across various projects, reducing duplication of efforts, and encouraging an ecosystem of shared, common functionalities. This in turn accelerates the development environment within The Graph ecosystem and beyond, allowing teams to focus on innovation, rather than replication.
Repository Reference
Throughout this guide, we will reference a sample project to help you understand how to build and deploy Substreams-powered Subgraphs. You can find the complete code for the Uniswap V3 example discussed in this series in the uniswap-v3-demo-sps repository.
Next Steps
If this is all seeming a bit abstract right now, don’t worry! As we move forward in this series, you’ll gain deeper insights into the technical aspects and practical implementations of Substreams. This series is designed to equip you with the knowledge and skills necessary to leverage Substreams effectively in your blockchain development projects.