22. Verification¶

This chapter documents verification of the VeeR EL2 Core and coverage data collection, including RTL-level tests designed to exercise parts of the core’s logic, software execution tests, randomized code generator tests, as well as verification coverage.

Tests listed in this chapter are run in Continuous Integration pipelines of the cores-veer-el2 repository via GitHub Actions.

22.1. RTL-level tests¶

Verification of the VeeR EL2 Core includes an RTL test suite created to exercise details of the core’s internal architecture. These tests complement the software execution tests described in later section of this chapter.

RTL-level tests include block-level as well as top-level tests developed for the VeeR EL2 Core:

• Block-level tests are located in the verification/block directory of the cores-veer-el2 repository.

• Top-level tests are located in the verification/top directory of the cores-veer-el2 repository.

22.1.1. Cocotb tests¶

The main group of RTL tests were implemented using cocotb, a popular co-simulation testbench library for Python, allowing for re-use of the extensive Python testing ecosystem for design verification.

The test files are located in the verification/block/ directory of the Cores-VeeR-EL2 repository.

The verification environment is extended by a PyUVM (Universal Verification Methodology implemented in Python instead of SystemVerilog) test with a corresponding CI workflow. It implements a basic PyUVM structure to test the core’s behavior when interrupt pins are stimulated.

22.1.2. UVM verification¶

UVM tests are run as the Test-UVM job in the CI pipelines of the cores-veer-el2 repository. The test files are located in the testbench/uvm directory of the cores-veer-el2 repository.

22.2. Software execution tests¶

22.2.1. Regression tests¶

Regression tests are run as the Test-Regression job in the CI pipelines of the cores-veer-el2 repository.

Regression testing involves execution of rudimentary software that was executed correctly on previous runs. The regression test executes .hex files of the following pieces of software:

• A hello_world program

• A Dhrystone benchmark program

• A Coremark benchmark program

Regression tests include verification of privilege mode switching. The test files are located in the testbench/tests/modesw directory of the cores-veer-el2 repository and described in more detail in the README file.

22.2.2. Renode tests¶

Renode is a deterministic simulation framework used to verify proper software execution. The tests are defined in the testbench/tests directory of the cores-veer-el2 repository.

Renode uses Robot Framework, an open source automation framework for test automation and robotic process automation (RPA). A testing script to execute test binaries is defined in the veer.robot file.

For detailed information regarding verification of the VeeR EL2 core with Renode, refer to the VeeR EL2 Support in Renode README.

22.2.3. RISCOF Verification¶

RISCOF is a RISC-V core test framework. It uses official architectural assembly test programs where the core memory state is compared against a reference. The comparison happens between the simulated core under test and a reference ISS, similar to tests with RISCV-DV described below.

The RISCOF framework exercises the VeeR core with tests that use the Spike ISS and Verilator. Like with RISCV-DV, a mismatch in memory signature comparison is reported as a CI failure.

The compared memory region is known as the memory signature. Its boundaries are defined by special symbols defined in each test program. It is the responsibility of the simulator / ISS to dump the memory signature for comparison by RISCOF.

For more detailed information about verification of the VeeR EL2 core with RISCOF, refer to the README.

22.2.3.1. Adaptation of VeeR simulation and testbench to RISCOF¶

Adaptation of VeeR for RISCOF required implementing the memory signature dump. Thanks to the use of Verilator it was possible to automatically load and parse the symbol map extracted from the binary ELF file and inject signature boundary addresses to the RTL simulation. The simulation executable accepts the --symbols argument which specifies the symbol map file obtained using the nm utility. The addresses can also be provided manually via the --mem-signature argument as two hexadecimal numbers.

The memory dump itself is implemented as a SystemVerilog task called from the RTL code right before the simulation ends. The output file name is fixed to veer.signature. The task automatically assures access to the correct memory - data is loaded from DCCM by default. Otherwise, it is taken from the generic RAM present in the testbench. When no signature range is defined, the dump is not written.

22.2.3.2. RISCOF model plugin¶

RISCOF uses Python plugins to interface with simulated cores and ISSs. The task of a plugin is to build and link assembly test programs in a way suitable for core / ISS use and to run the simulation. The plugin may also provide dedicated code snippets used by test programs to communicate with the simulation (eg. signal program end). Currently plugins are available for the Spike ISS and the SAIL ISS. Since every RTL simulation framework for a core is different, it requires a dedicated plugin.

The VeeR plugin, located in the /tools/riscof/veer directory of this repository, performs the following tasks:

• Code compilation and linking

• Symbol dump (to get the memory signature address range)

• Simulation run

• Result collection (moving / renaming the output memory signature dump file)

The VeeR RTL needs to be “Verilated” upfront as the plugin assumes that the simulation binary is already present.

22.3. RISCV-DV verification¶

RISCV-DV is a verification framework (originally from Google, now also in CHIPS Alliance) designed to test and (co-)verify RISC-V CPU cores.

RISCV-DV tests are run as the Test-RISCV-DV job in the CI pipelines of the cores-veer-el2 repository.

Pre-generated RISCV-DV test programs for fallback in case of failure in generation in CI pipelines are stored in the .github/assets/riscv-dv directory.

The framework generates random instruction chains to exercise certain core features and relies on the Universal Verification Methodology (UVM) for code generation. These instructions are then simultaneously executed by the core (RTL simulation) and by a reference RISC-V ISS (instruction set simulator), for example Spike. The core states of both are then compared and an error is reported in case of a mismatch.

RISCV-DV tests are run as the Test-RISCV-DV job in the CI pipelines of the cores-veer-el2 repository. This job is responsible for running the RISCV-DV test suite on the VeeR core, downloading Renode and building the VeeR ISS.

A mismatch between an ISS trace and the RTL trace reported by RISCV-DV framework immediately triggers a CI error.

22.3.1. RISCV-DV Test flow¶

The RISCV-DV test flow for the VeeR EL2 Core looks as follows:

• Generate a program using a RISCV-DV generator

• Run the program in a RISC-V ISS, collect the trace

• Run the program inside a simulation of the VeeR EL2 core using Verilator and collect the trace

• Compare both execution trace files

• If no mismatches are found, the test is successful

Fig. 22.1 below illustrates this flow.

For more detailed information about verification of the VeeR EL2 core with RISCV-DV, refer to the README.

In a physical design, external stimulus, e.g. from interrupt source or debugging requests, causes relevant CSRs to be updated automatically, which is not always the case for tests relying solely on generated streams of instructions. In the future, tests to verify these features will be implemented using the RISCV-DV handshaking mechanism, a feature put in place specifically to update the core’s internal state properly.

22.3.2. Current implementation¶

Since RISCV-DV does not provide a generic way of simulating RISC-V cores at RTL level, this implementation runs the existing testbench for VeeR EL2 in Verilator. RISCV-DV requires execution traces in its own standardized format, so we developed a Python script which parses the VeeR EL2 execution log and converts it to the CSV format accepted by RISCV-DV.

The end-to-end flow is implemented by the Makefile in tools/riscv-dv. The RISCV-DV run.py script is used for random code generation, compilation and ISS execution. A set of Makefile rules implements building the verilated testbench, running it, converting trace logs and trace comparison. The comparison itself is done by instr_trace_compare.py in RISCV-DV.

The flow currently supports three ISSs:

• Spike

• Renode

The CI workflow for RISCV-DV builds or downloads the prerequisites (Verilator, Spike, Renode) and invokes the test flow. A failure of any of RISCV-DV tests is reported as CI failure.

22.3.3. Renode integration¶

Renode is Antmicro’s open source development framework which allows debugging and testing unmodified embedded software on your PC - from bare System-on-Chips, through complete devices, to multi-node systems.

As a part of the the project, we extended the RISCV-DV framework with Renode ISS support. The work also included defining a virtual Renode platform for VeeR EL2, executing RISCV-DV pseudo-random programs on it and collecting execution trace logs. Basic level VeeR EL2 support in Renode opens up the potential to simulate the Caliptra RoT as well as the RoT in the context of a larger SoC in conjunction with Renode’s support for e.g. ARM and RISC-V cores and peripherals or OpenTitan peripherals (some of which are used in Caliptra)

22.4. Verification coverage¶

For each of the tests described in this chapter, data is collected and processed to determine verification coverage.

22.4.1. Coverage analysis¶

To verify whether all parts of a design have been tested, you can get coverage reports from simulation runs which inform about a percentage of possible design states that were simulated. You can then use this information to prepare new testing scenarios that test previously untested design states.

Results for verification coverage of the entire design as well as its parts are available in the VeeR EL2 coverage dashboard. The results are updated with each Pull Request, as visible in the Active pull request list. Data is only available for open Pull Requests and is removed once a Pull Request is closed or merged.

22.4.2. Coverage analysis with open source tools¶

Verilator supports certain method of test coverage analysis. There are three ways in which Verilator collects coverage data:

• Line coverage

  Verilator automatically counts changes to the RTL code flow at all possible branch points.

• Toggle coverage

  Automatic toggle count for each signal. Does not apply to certain signal types as described in Verilator’s documentation.

• Functional coverage

  Verilator automatically counts events defined by the cover property, which are implemented in the source code.

To enable coverage collection with Verilator, simply add the following lines to the C++ testbench:

  // Write coverage data
#if VM_COVERAGE
  Verilated::threadContextp()->coveragep()->write("coverage.dat");
#endif

The genhtml utility from the lcov package is used to convert the data collected from the verification into .html files present it in the form of the VeeR EL2 coverage dashboard.

22.4.3. Identification of signals without coverage¶

The recommended way to identify signals with low coverage is to use the annotation mechanism. The annotation mechanism is provided by the verilator_coverage tool and is capable of writing source files with annotations next to each coverage point.

Coverage reports in the form of .dat files can be combined using the following command:

verilator_coverage coverage_test_*.dat --write combined.dat

In order to annotate the coverage results back to the source files, run the command:

verilator_coverage coverage_test.dat --annotate <output_dir>

The result of the command should be a copy of the source files used in the simulation, located in the output directory of choice. In the annotated source files, you will find that annotations are placed at the beginning of lines, e.g.:

   153724    input  logic en;
  %000000    input  logic scan_mode;

Interpretations of these results are placed inline:

   153724    input  logic en; // there were 153724 coverage hits
  %000000    input  logic scan_mode; // signal never toggled, so line starts with '%'

If a single line contains more than one signal definition (or a multi-bit signal), it may be useful to run with the annotate-points argument, so that the annotation resembles:

    153888    input logic SE, EN, CK,
    -000000  point: comment=SE
    +153888  point: comment=EN
    +2636790  point: comment=CK

In this mode, ‘+’ and ‘-’ are used to indicate whether a coverage point is above or below the threshold.

To read more about coverage in Verilator, see:

• Coverage analysis

• verilator_coverage executable documentation