The Regression Test Suite
*************************

Icarus Verilog development includes a regression test suite that is
included along with the source. The "ivtest" directory contains the
regression test suite, and this suite is used by the github actions as
continuous integration to make sure the code is always going forward.

NOTE: There are scripts written in perl to run the regression tests,
but they are being gradually replaced with a newer set of scripts. It
is the newer method that is described here.


Test Descriptions
=================

Regression tests are listed in the regress-vvp.list file. Each line
lists the name of the test and the path to the dest description. The
list file is therefore pretty simple, and all the description of the
test is in the description file:

   macro_str_esc  vvp_tests/macro_str_esc.json

The "name" is a simple name, and the test-description-file is the path
(relative the ivtest directory) to the description file. A simple test
description file is a JSON file, like this:

   {
     "type"   : "normal",
     "source" : "macro_str_esc.v",
     "gold"   : "macro_str_esc"
   }

This description file contains all the information that the vvp_reg.py
script needs to run the regression test. The sections below describe
the keys and values in the description file dictionary.


source (required)
-----------------

This specifies the name of the source file. The file is actually to be
found in the ivltests/ directory.


type (required)
---------------

This describes the kind of test to run. The valid values are:

* **normal** - Compile the source using the iverilog compiler vvp
  target, and if that succeeds execute it using the vvp command. If
  there is no gold file specified, then look for an output line with
  the "PASSED" string.

* **NI** - Mark the test as not implemented. The test will be skipped
  without running or reporting an error.

* **CE** - Compile, but expect the compiler to fail. This means the
  compiler command process must return an error exit.

* **EF** - Compile and run, but expect the run time to fail. This
  means the run time program must return an error exit.

* **TE** - This is specific to testing the vlog95 conversion and
  indicates the translated code failed to compile.


gold (optional)
---------------

If this is specified, it replaces the "Passed" condition with a
comparison of the output with a gold file. The argument is the name of
the gold file set, which will be found in the "gold/" directory. The
name here is actually the basename of the gold files, with separate
actual gold files for the iverilog and vvp stderr and stdout. For
example, if a "normal" test includes a gold file, then the program is
compiled and run, and the outputs are compared with the gold file to
make sure it ran properly.

The way the regression suite works, there are 4 log files created for
each test:

* foo-iverilog-stdout.log

* foo-iverilog-stderr.log

* foo-vvp-stdout.log

* foo-vvp-stderr.log

The "gold" value is the name of the gold file set. If the gold value
is "foo", Then the actual gold files are called:

* gold/foo-iverilog-stdout.gold

* gold/foo-iverilog-stderr.gold

* gold/foo-vvp-stdout.gold

* gold/foo/vvp-stderr.gold

If any of those files is empty, then the gold file doesn't need to be
present at all. The log files and the gold files are compared byte for
byte, so if the output you are getting is correct, then copy the log
to the corresponding gold, and you're done.

If the run type is "CE" or "RE", then the gold files still work, and
can be used to check that the error message is correct. If the gold
file setting is present, the error return is required, and also the
gold files must match.


iverilog-args (optional)
------------------------

If this is specified, it is a list of strings that are passed as
arguments to the iverilog command line.


vvp-args (optional)
-------------------

If this is specified, it is a list of strings that are passed as
arguments to the vvp command. These arguments go before the vvp input
file that is to be run.


vvp-args-extended (optional)
----------------------------

If this is specified, it is a lost of strings that are passed as
arguments to the vvp command. These are extended arguments, and are
placed after the vvp input file that is being run. This is where you
place things like plusargs.


strict, force-sv or vlog95 (optional)
-------------------------------------

Any of these can be used to create overrides for the type, gold or
iverilog-args when the given test type is run.
