# Development environment

## Setting up

In order to prepare the development environment, please follow the steps below:

1. Install the Python 3.11 interpreter and pip package manager.
    * Optionally create a Python virtual environment with `python3 -m venv venv` in the project directory and activate it using generated script: `. venv/bin/activate`.
2. Install all required libraries with `pip3 install -r requirements-dev.txt`.
3. Install `riscv64-unknown-elf` binutils using your favourite package manager. On Debian-based distros the package is called `binutils-riscv64-unknown-elf`, on Arch-based - `riscv64-unknown-elf-binutils`.
4. Optionally, install all precommit hooks with `pre-commit install`. This will automatically run the linter before commits.

## Using scripts

The development environment contains a number of scripts which are run in CI, but are also intended for local use. They are:

### run\_tests.py

Runs the unit tests.
By default, every available test is run (excluding the `riscv-tests` and `riscv-arch-test` test suites).
Tests from a specific file can be run using the following call (`test_alu` is used as an example):

```
scripts/run_tests.py test_alu
```

One can even run a specific test class from a file:

```
scripts/run_tests.py test_alu.TestAluUnit
```

Or a specific test method:

```
scripts/run_tests.py test_alu.TestAluUnit.test_fu
```

The argument to `run_tests.py` is used to search within the full names of tests.
The script runs all the tests which match the query.
Thanks to this, if a given test class name is unique, just the class name can be used as an argument.

The `run_tests.py` script has the following options:

* `-l`, `--list` -- lists available tests. This option is helpful, e.g., to find a name of some specific instance of a parametrized test.
* `-t`, `--trace` -- generates waveforms in the `vcd` format, which can be used, among others, in [gtkwave](https://github.com/gtkwave/gtkwave) and [surfer](https://gitlab.com/surfer-project/surfer).
  Also, `gtkw` files (for `gtkwave`) are generated, which can help find commonly used signals.
  The files are saved in the `test/__traces__/` directory.
  Useful for debugging and test-driven development.
* `-p`, `--profile` -- generates Transactron execution profile information, which can then be read by `transactron-prof`.
  The files are saved in the `test/__profile__/` directory.
  Useful for analyzing performance.
* `-v`, `--verbose` -- makes the test runner more verbose. It will, for example, print the names of all the tests being run.
* `-j`, `--jobs` -- the number of concurrent testing jobs.
  By default, the number of jobs will be equal to the number of hardware threads available.
* `-s`, `--no-capture` -- test stdout/stderr output will be visible for the currently running test. 
  This is useful to see the output of passing tests, including prints and logs.
  The number of jobs is automatically reduced to 1 to avoid confusion.
* `-a`, `--all` -- include the `riscv-tests` and `riscv-arch-test` test suites.
* `-b`, `--backend` -- select a backend for the test suites selected by `-a`.
  Two backends are available:
  - `cocotb` -- utilizes cocotb and Verilator. Fast, but waveform traces are less useful than in pysim.
  - `pysim` -- utilizes the pysim engine from Amaranth.
  The `cocotb` backend is selected by default.
* `-c`, `--count` -- number of matching tests to run. Useful for, e.g., quick checking of a test with a lot of parametrizations.
* `--log-level` -- sets the log level, which filters log messages generated by tests.
  This includes the messages from the Transactron logging library.
  Use with `-s` to see logs for non-failing tests.
* `--log-filter` -- filter log output.
  This allows to, e.g., see logs only from a chosen subsystem.
  Faster than grepping the script output.

The `run_tests.py` is actually a thin wrapper over `pytest`.
For more advanced use, see `pytest --help`.

### lint.sh

Checks the code formatting and typing. It should be run as follows:

```
scripts/lint.sh subcommand [filename...]
```

The following main subcommands are available:

* `format` -- reformats the code using `black`.
* `check_format` -- verifies code formatting using `black` and `flake8`.
* `check_types` -- verifies typing using `pyright`.
* `verify` -- runs all checks. The same set of checks is run in CI.

When confronted with `would reformat [filename]` message from `black` you may run:

```
black --diff [filename]
```
This way you may display the changes `black` would apply to `[filename]` if you chose the `format` option for `lint.sh` script. This may help you locate the formatting issues.

### core\_graph.py

Visualizes the core architecture as a graph. The script outputs a file in one of supported graph formats, which need to be passed to an appropriate tool to get a graph.

The `core_graph.py` script has the following options:

* `-p`, `--prune` -- removes disconnected nodes from the output graph.
* `-f FORMAT`, `--format FORMAT` -- selects the output format. Supported formats are `elk` (for [Eclipse Layout Kernel](https://www.eclipse.org/elk/)), `dot` (for [Graphviz](https://graphviz.org/)), `mermaid` (for [Mermaid](https://mermaid.js.org/)).

### build\_docs.sh

Generates local documentation using [Sphinx](https://www.sphinx-doc.org/). The generated HTML files are located in `build/html`.

### tprof.py

Processes Transactron profile files and presents them in a readable way.
To generate a profile file, the `run_tests.py` script should be used with the `--profile` option.
The `tprof.py` can then be run as follows:

```
scripts/tprof.py test/__profile__/profile_file.json
```

This displays the profile information about transactions by default.
For method profiles, one should use the `--mode=methods` option.

The columns have the following meaning:

* `name` -- the name of the transaction or method in question. The method names are displayed together with the containing module name to differentiate between identically named methods in different modules.
* `source location` -- the file and line where the transaction or method was declared. Used to further disambiguate transaction/methods.
* `locked` -- for methods, shows the number of cycles the method was locked by the caller (called with a false condition). For transactions, shows the number of cycles the transaction could run, but was forced to wait by another, conflicting, transaction.
* `run` -- shows the number of cycles the given method/transaction was running.

To display information about method calls, one can use the `--call-graph` option.
When displaying transaction profiles, this option produces a call graph. For each transaction, there is a tree of methods which are called by this transaction.
Counters presented in the tree shows information about the calls from the transaction in the root of the tree: if a method is also called by a different transaction, these calls are not counted.
When displaying method profiles, an inverted call graph is produced: the transactions are in the leaves, and the children nodes are the callers of the method in question.
In this mode, the `locked` field in the tree shows how many cycles a given method or transaction was responsible for locking the method in the root.

Other options of `tprof.py` are:

* `--sort` -- selects which column is used for sorting rows.
* `--filter-name` -- filters rows by name. Regular expressions can be used.
* `--filter-loc` -- filters rows by source locations. Regular expressions can be used.