Development environment

Setting up

In order to prepare the development environment, please follow the steps below:

  1. Install the Python 3.11 interpreter and pip package manager.

    • Optionally create a Python virtual environment with python3 -m venv venv in the project directory and activate it using generated script: . venv/bin/activate.

  2. Install all required libraries with pip3 install -r requirements-dev.txt.

  3. Install riscv64-unknown-elf binutils using your favourite package manager. On Debian-based distros the package is called binutils-riscv64-unknown-elf, on Arch-based - riscv64-unknown-elf-binutils.

  4. Optionally, install all precommit hooks with pre-commit install. This will automatically run the linter before commits.

Using scripts

The development environment contains a number of scripts which are run in CI, but are also intended for local use. They are:

run_tests.py

Runs the unit tests. By default, every available test is run (excluding the riscv-tests and riscv-arch-test test suites). Tests from a specific file can be run using the following call (test_alu is used as an example):

scripts/run_tests.py test_alu

One can even run a specific test class from a file:

scripts/run_tests.py test_alu.TestAluUnit

Or a specific test method:

scripts/run_tests.py test_alu.TestAluUnit.test_fu

The argument to run_tests.py is used to search within the full names of tests. The script runs all the tests which match the query. Thanks to this, if a given test class name is unique, just the class name can be used as an argument.

The run_tests.py script has the following options:

  • -l, --list – lists available tests. This option is helpful, e.g., to find a name of some specific instance of a parametrized test.

  • -t, --trace – generates waveforms in the vcd format, which can be used, among others, in gtkwave and surfer. Also, gtkw files (for gtkwave) are generated, which can help find commonly used signals. The files are saved in the test/__traces__/ directory. Useful for debugging and test-driven development.

  • -p, --profile – generates Transactron execution profile information, which can then be read by transactron-prof. The files are saved in the test/__profile__/ directory. Useful for analyzing performance.

  • -v, --verbose – makes the test runner more verbose. It will, for example, print the names of all the tests being run.

  • -j, --jobs – the number of concurrent testing jobs. By default, the number of jobs will be equal to the number of hardware threads available.

  • -s, --no-capture – test stdout/stderr output will be visible for the currently running test. This is useful to see the output of passing tests, including prints and logs. The number of jobs is automatically reduced to 1 to avoid confusion.

  • -a, --all – include the riscv-tests and riscv-arch-test test suites.

  • -b, --backend – select a backend for the test suites selected by -a. Two backends are available:

    • cocotb – utilizes cocotb and Verilator. Fast, but waveform traces are less useful than in pysim.

    • pysim – utilizes the pysim engine from Amaranth. The cocotb backend is selected by default.

  • -c, --count – number of matching tests to run. Useful for, e.g., quick checking of a test with a lot of parametrizations.

  • --log-level – sets the log level, which filters log messages generated by tests. This includes the messages from the Transactron logging library. Use with -s to see logs for non-failing tests.

  • --log-filter – filter log output. This allows to, e.g., see logs only from a chosen subsystem. Faster than grepping the script output.

The run_tests.py is actually a thin wrapper over pytest. For more advanced use, see pytest --help.

lint.sh

Checks the code formatting and typing. It should be run as follows:

scripts/lint.sh subcommand [filename...]

The following main subcommands are available:

  • format – reformats the code using black.

  • check_format – verifies code formatting using black and flake8.

  • check_types – verifies typing using pyright.

  • verify – runs all checks. The same set of checks is run in CI.

When confronted with would reformat [filename] message from black you may run:

black --diff [filename]

This way you may display the changes black would apply to [filename] if you chose the format option for lint.sh script. This may help you locate the formatting issues.

core_graph.py

Visualizes the core architecture as a graph. The script outputs a file in one of supported graph formats, which need to be passed to an appropriate tool to get a graph.

The core_graph.py script has the following options:

  • -p, --prune – removes disconnected nodes from the output graph.

  • -f FORMAT, --format FORMAT – selects the output format. Supported formats are elk (for Eclipse Layout Kernel), dot (for Graphviz), mermaid (for Mermaid).

build_docs.sh

Generates local documentation using Sphinx. The generated HTML files are located in build/html.

tprof.py

Processes Transactron profile files and presents them in a readable way. To generate a profile file, the run_tests.py script should be used with the --profile option. The tprof.py can then be run as follows:

scripts/tprof.py test/__profile__/profile_file.json

This displays the profile information about transactions by default. For method profiles, one should use the --mode=methods option.

The columns have the following meaning:

  • name – the name of the transaction or method in question. The method names are displayed together with the containing module name to differentiate between identically named methods in different modules.

  • source location – the file and line where the transaction or method was declared. Used to further disambiguate transaction/methods.

  • locked – for methods, shows the number of cycles the method was locked by the caller (called with a false condition). For transactions, shows the number of cycles the transaction could run, but was forced to wait by another, conflicting, transaction.

  • run – shows the number of cycles the given method/transaction was running.

To display information about method calls, one can use the --call-graph option. When displaying transaction profiles, this option produces a call graph. For each transaction, there is a tree of methods which are called by this transaction. Counters presented in the tree shows information about the calls from the transaction in the root of the tree: if a method is also called by a different transaction, these calls are not counted. When displaying method profiles, an inverted call graph is produced: the transactions are in the leaves, and the children nodes are the callers of the method in question. In this mode, the locked field in the tree shows how many cycles a given method or transaction was responsible for locking the method in the root.

Other options of tprof.py are:

  • --sort – selects which column is used for sorting rows.

  • --filter-name – filters rows by name. Regular expressions can be used.

  • --filter-loc – filters rows by source locations. Regular expressions can be used.