Core verification

Coreblocks is verified at several levels of abstraction. Beside of unit tests and module tests, we also synthesise the core to the ECP5 FPGA target, to check that it can work in reality. Performance is verified using synthesis results and a set of benchmarks simulated with cycle precision in cocotb. We also verify correctness of the core behaviour by running assembler tests from riscv-tests and riscv-arch-tests.

These three verification steps are automatically run by CI on every commit delivered to the master branch. Running the checks in CI allow us to collect historical data, which are available in the form of the graphs on a dedicated benchmark subpage.

In CI we use pre-built docker containers, which are publicly available on our github page. In the following subsections we provide the instructions on how to manually run verification steps using these containers. They can be recreated using standard docker build commands:

docker build --platform linux/amd64 -t "amaranth-synth:latest" -f ./docker/AmaranthSynthECP5.Dockerfile .

Synthesis

The basic step in verification is to see if it is possible to synthesise the Core circuit. This allows us to control the level of complexity of the core. Although Coreblocks is an educational core, we want it to be practical. It should have an acceptable maximum frequency and shouldn’t use too many resources, so it can be run on a FPGA. The synthesis step ensures that these requirements are met. In addition, it checks whether the code that is acceptable for Amaranth is also acceptable for the synthesis tools.

The main properties collected in the synthesis step:

  • Max clock frequency

  • Number of logic cells used

  • Number of carry cells used

  • Number of RAM cells used

  • Number of DFF cells used

The configuration of the docker container is described in the AmaranthSynthECP5.Dockerfile, which can be found in our repo.

Manual reproduction

sudo docker pull ghcr.io/kuznia-rdzeni/amaranth-synth:latest
sudo docker run -it --rm ghcr.io/kuznia-rdzeni/amaranth-synth:latest
git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
cd coreblocks
apt update
apt install python3.11-venv
python3 -m venv venv
. venv/bin/activate
python3 -m pip install --upgrade pip
pip3 install -r requirements-dev.txt
PYTHONHASHSEED=0 ./scripts/synthesize.py --verbose --config full
./scripts/parse_benchmark_info.py
cat benchmark.json

The main point of the above listing is the synthesize.py script, which creates an instance of the Core object with the configuration provided by the user and then passes it to Amaranth to generate a Verilog description from that instance. This description is then processed by Yosys and nextpnr-ecp5 to generate the ECP5 bitstream.

A strength of Coreblocks is its modularity, so we can provide different configurations with little effort. You can choose a configuration to synthesise using the --config argument to the synthesise.py script.

Dependencies

In order to perform synthesis we use:

  • yosys - to synthesise the Verilog generated by Amaranth up to gate level;

  • nextpnr-ecp5 - to perform the “Place and Route” step;

  • prjtrellis - provides the description of the ECP5 bitstream format.

Benchmarking

The maximum clock frequency determined by synthesis isn’t the only measure of performance. In theory, it is always possible to increase Fmax by increasing latency. To avoid the pitfall of too long latency affecting the core throughput, we monitor the number of instructions executed per clock cycle (IPC). We simulate the core with cycle accuracy and run benchmarks written in C inside the simulation. The benchmarks are taken from embench.

The benchmarking is done in two steps. First, we compile the C programs into binary format. Second, we run the binaries on the simulated core. To compile the code we use riscv-gnu-toolchain, with glibc configured for different architectural subsets of RISC-V extensions (you can check the exact configuration in riscv-toolchain.Dockerfile).

Once we have the binaries, we can run them in simulation. This is done using Cocotb and Verilator. We use Amaranth features to generate Verilog code describing Coreblocks instance, which is passed to Verilator for compilation. Cocotb controls the simulation and execution of the program by stubbing external interfaces. Pre-compiled Verilator in a compatible version is available in Verilator.Dockerfile.

Benchmarks manual execution

# ========== STEP 1: Compilation ==========
# Clone coreblocks into host file system
git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
cd coreblocks
git submodule update --init --recursive
cd ..
sudo docker pull ghcr.io/kuznia-rdzeni/riscv-toolchain:latest
# Run docker with the coreblocks directory mounted into it
sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/riscv-toolchain:latest
cd /coreblocks/test/external/embench
# Compilation will put binaries in the subdirectory of the /coreblocks directory, which is shared with the host
# so that binaries survive after the docker container is closed
make
exit

# ========== STEP 2: Execution ==========
sudo docker pull ghcr.io/kuznia-rdzeni/verilator:latest
# Run docker with the coreblocks directory mounted into it. This directory contains
# benchmark binaries after running the first step.
sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/verilator:latest
apt update
apt install python3.11-venv
python3 -m venv venv
. venv/bin/activate
python3 -m pip install --upgrade pip
cd coreblocks
pip3 install -r requirements-dev.txt
PYTHONHASHSEED=0 ./scripts/gen_verilog.py --verbose --config full
./scripts/run_benchmarks.py

Regression tests

Regression tests should ensure that Coreblocks is compliant with RISC-V specification requirements. Tests include assembler programs that tests entire RISC-V instruction set. We execute these programs in a similar way to benchmarks. So, as a first step, we compile the programs to the binary format and then we run them on core simulated by Verilator and Cocotb.

Regression tests manual execution

# ========== STEP 1: Compilation ==========
# Clone coreblocks into host file system
git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
cd coreblocks
git submodule update --init --recursive
cd ..
sudo docker pull ghcr.io/kuznia-rdzeni/riscv-toolchain:latest
# Run docker with the coreblocks directory mounted into it
sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/riscv-toolchain:latest
cd /coreblocks/test/external/riscv-tests
# Compilation will put binaries in the subdirectory of the /coreblocks directory, which is shared with the host
# so that binaries survive after the docker container is closed
make
exit

# ========== STEP 2: Execution ==========
sudo docker pull ghcr.io/kuznia-rdzeni/verilator:latest
# Run docker with the coreblocks directory mounted into it. This directory contains
# regression test binaries after running the first step.
sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/verilator:latest
apt update
apt install python3.11-venv
python3 -m venv venv
. venv/bin/activate
python3 -m pip install --upgrade pip
cd coreblocks
pip3 install -r requirements-dev.txt
PYTHONHASHSEED=0 ./scripts/gen_verilog.py --verbose --config full
./scripts/run_tests.py -a regression