e3.testsuite

Generic testsuite framework.

Submodules

• e3.testsuite._helpers
• e3.testsuite.control
• e3.testsuite.driver
• e3.testsuite.find_skipped_tests
• e3.testsuite.fragment
• e3.testsuite.main
• e3.testsuite.multiprocess_scheduler
• e3.testsuite.optfileparser
• e3.testsuite.process
• e3.testsuite.report
• e3.testsuite.result
• e3.testsuite.running_status
• e3.testsuite.testcase_finder
• e3.testsuite.utils

Attributes

logger

Exceptions

ProbingError	Exception raised in TestFinder.probe when a test is misformatted.
TestAbort	Raise this to abort silently the execution of a test fragment.

Classes

GAIAResultFiles	Filenames for a given result in a GAIA report.
ReportIndex	Lightweight index for test results.
Log	Object to hold long text or binary logs.
TestResult	Represent a result for a given test.
TestStatus	Testcase execution status.
ParsedTest	Basic information to instantiate a test driver.
TestFinder	Interface for objects that find testcases in the tests subdirectory.
YAMLTestFinder	Look for "test.yaml"-based tests.
CleanupMode	Mode for working space cleanups.
ColorConfig	Proxy for color management.
TestsuiteCore	Testsuite Core driver.
Testsuite	Testsuite class.

Functions

deprecated(→ Callable[[F], F])	Return a decorator to emit deprecation warnings.
dump_gaia_report(→ None)	Dump a GAIA-compatible testsuite report.
dump_result_logs_if_needed(→ Optional[GAIAResultFiles])	Shortcut to call dump_result_logs if a GAIA report is requested.
generate_report(→ None)	Generate a text report for testsuite results.
summary_line(→ str)	Format a summary line to describe the result test result.
dump_xunit_report(→ None)	Dump a testsuite report to filename in the standard XUnit XML format.
dump_environ(→ None)	Dump environment variables into a sourceable file.
enum_to_cmdline_args_map(→ Dict[str, EnumType])	Turn enum alternatives into command-line arguments.
isatty(→ bool)	Return whether stream is a TTY.

Package Contents

e3.testsuite.deprecated(stacklevel: int = 1) → Callable[[F], F]

Return a decorator to emit deprecation warnings.

The function that the decorator returns emits the deprecation warning only the first time it is called.

class e3.testsuite.GAIAResultFiles

Filenames for a given result in a GAIA report.

In a GAIA testsuite report, each result is described as one entry (line) in the “results” text file, and several optional files. This object contains the names for the files, when present, corresponding to a given result.

result: str | None

log: str | None

expected: str | None

out: str | None

diff: str | None

time: str | None

info: str | None

e3.testsuite.dump_gaia_report(report_index: e3.testsuite.report.index.ReportIndex, output_dir: str, discs: object = None, result_files: Dict[str, GAIAResultFiles] | None = None) → None

Dump a GAIA-compatible testsuite report.

Parameters:

• report_index – ReportIndex instance for all the test results to include in the report.

• output_dir – Directory in which to emit the report.

• discs – List of discriminants associated to the testsuite report, if any. See “dump_discriminants” for the expected format.

• result_files – If the log files for each result have already been generated, mapping from test names to result file names. None otherwise.

e3.testsuite.dump_result_logs_if_needed(env: e3.env.Env, result: e3.testsuite.result.TestResult, output_dir: str) → GAIAResultFiles | None

Shortcut to call dump_result_logs if a GAIA report is requested.

e3.testsuite.generate_report(output_file: IO[str], new_index: e3.testsuite.report.index.ReportIndex, old_index: e3.testsuite.report.index.ReportIndex | None, colors: e3.testsuite.utils.ColorConfig, show_all_logs: bool, show_xfail_logs: bool, show_error_output: bool, show_time_info: bool) → None

Generate a text report for testsuite results.

Parameters:

• output_file – Output file for the report.

• new_index – Testsuite results to display.

• old_index – Results from a previous testsuite run. If present, used to compute the new/already-detected/fixed regressions.

• colors – Color configuration for the output.

• show_all_logs – Whether to display logs for all testcases (successful tests are not displayed by default).

• show_xfail_logs – Whether to display logs for XFAIL results (hidden by default).

• show_error_output – Whether to display logs in test results.

• show_time_info – Whether to display time information for test results, if available.

e3.testsuite.summary_line(result: e3.testsuite.result.TestResultSummary, colors: e3.testsuite.utils.ColorConfig, show_time_info: bool) → str

Format a summary line to describe the result test result.

Parameters:

• colors – Proxy to introduce (or not) colors in the result.

• show_time_info – Whether to include timing information in the result.

class e3.testsuite.ReportIndex(results_dir: str)

Lightweight index for test results.

INDEX_FILENAME = '_index.json'

INDEX_MAGIC = 'e3.testsuite.report.index.ReportIndex:1'

results_dir

Directory that contain test results (YAML files).

entries: Dict[str, ReportIndexEntry]

Map test names to their ReportIndexEntry instances.

status_counters

Number of test result for each test status.

duration: float | None = None

Optional number of seconds for the total duration of the testsuite run.

save_and_add_result(result: e3.testsuite.result.TestResult) → None

Save a test result in the results directory and add it to the index.

Parameters:

result – Test result to save/add.

add_result(result: e3.testsuite.result.TestResultSummary, filename: str) → None

Add an entry to this index for the given test result.

Note that unlike save_and_add_result, this does not write the result data in the results dir: it is up to the caller to make sure of that.

Parameters:

• result – Result to add.

• filename – Name of the file that contains test result data.

classmethod read(results_dir: str) → ReportIndex

Read the index in the given results directory.

write() → None

Write the index on disk.

property has_failures: bool

Return whether there is at least one FAIL/ERROR test status.

e3.testsuite.dump_xunit_report(name: str, index: e3.testsuite.report.index.ReportIndex, filename: str) → None

Dump a testsuite report to filename in the standard XUnit XML format.

Parameters:

• name – Name for the teststuite report.

• index – Report index for the testsuite results to report.

• filename – Name of the text file to write.

• duration – Optional number of seconds for the total duration of the testsuite run.

class e3.testsuite.Log(content: AnyStr)

Bases: yaml.YAMLObject, Generic[AnyStr]

Object to hold long text or binary logs.

We ensure that when dump to yaml the result will be human readable.

yaml_loader

yaml_tag = '!e3.testsuite.result.Log'

log: AnyStr

property is_binary: bool

Return whether this log contains binary data.

property is_text: bool

Return whether this log contains text data.

__iadd__(content: AnyStr) → Log[AnyStr]

Add additional content to the log.

Parameters:

content – a message to log

__str__() → str

class e3.testsuite.TestResult(name: str, env: dict | None = None, status: TestStatus | None = None, msg: str = '')

Bases: yaml.YAMLObject

Represent a result for a given test.

yaml_loader

yaml_tag = '!e3.testsuite.result.TestResult'

test_name

env = None

status

msg: str | None = ''

log

processes: list = []

failure_reasons: Set[FailureReason]

expected: Log | None = None

out: Log | None = None

diff: Log | None = None

time: float | None = None

info: Dict[str, str]

set_status(status: TestStatus, msg: str | None = '') → None

Update the test status.

Parameters:

• status – New status. Note that only test results with status set to ERROR can be changed.

• msg – Optional short message to describe the result. Note that multiline strings are turned into single-line strings.

__str__() → str

save(results_dir: str) → str

Write this test results as a YAML file.

Parameters:

results_dir – Name of the directory in which to write the test result. When writing a testsuite report, this corresponds to the report’s results_dir (see e3.testsuite.report.index.ReportIndex).

Returns:

The base filename of the file written. It is generated from the testname.

property summary: TestResultSummary

class e3.testsuite.TestStatus(*args, **kwds)

Bases: enum.Enum

Testcase execution status.

PASS

FAIL

XFAIL

XPASS

VERIFY

SKIP

NOT_APPLICABLE

ERROR

color(colors: e3.testsuite.utils.ColorConfig) → str

Return the ANSI color code for this test status.

This returns an empty string if colors are disabled.

class e3.testsuite.ParsedTest

Basic information to instantiate a test driver.

test_name: str

Name for this testcase.

driver_cls: Type[e3.testsuite.driver.TestDriver] | None

Test driver class to instantiate, None to use the default one.

test_env: dict

Base test environment.

Driver instantiation will complete it with test directory, test name, etc.

test_dir: str

Directory that contains the testcase.

test_matcher: str | None = None

Textual text matcher.

If not None, string to match against the list of requested tests to run: in that case, the test is ignored if there is no match. This is needed to filter out tests in testsuites where tests don’t necessarily have dedicated directories.

exception e3.testsuite.ProbingError

Bases: Exception

Exception raised in TestFinder.probe when a test is misformatted.

class e3.testsuite.TestFinder

Interface for objects that find testcases in the tests subdirectory.

property test_dedicated_directory: bool

Return whether each test has a dedicated test directory.

Even though e3-testsuite is primarily designed for this to be true, some testsuites actually host multiple tests in the same directory. When this is the case, we need to probe all directories and only then filter which test to run using ParsedTest.test_matcher.

abstractmethod probe(testsuite: e3.testsuite.TestsuiteCore, dirpath: str, dirnames: List[str], filenames: List[str]) → TestFinderResult

Return a test if the “dirpath” directory contains a testcase.

Raise a ProbingError if anything is wrong.

Parameters:

• testsuite – Testsuite instance that is looking for testcases.

• dirpath – Directory to probe for a testcase.

• dirnames – List of directories that “dirpath” contains.

• filenames – List of files that “dirpath” contains.

class e3.testsuite.YAMLTestFinder

Bases: TestFinder

Look for “test.yaml”-based tests.

This considers that all directories that contain a “test.yaml” file are testcases. This file is parsed as YAML, the result is used as a test environment, and if it contains a “driver” key, it uses the testsuite driver whose name corresponds to the associated string value.

probe(testsuite: e3.testsuite.TestsuiteCore, dirpath: str, dirnames: List[str], filenames: List[str]) → TestFinderResult

Return a test if the “dirpath” directory contains a testcase.

Raise a ProbingError if anything is wrong.

Parameters:

• testsuite – Testsuite instance that is looking for testcases.

• dirpath – Directory to probe for a testcase.

• dirnames – List of directories that “dirpath” contains.

• filenames – List of files that “dirpath” contains.

class e3.testsuite.CleanupMode(*args, **kwds)

Bases: enum.Enum

Mode for working space cleanups.

NONE

PASSING

ALL

classmethod default() → CleanupMode

classmethod descriptions() → Dict[CleanupMode, str]

class e3.testsuite.ColorConfig(colors_enabled: bool | None = None)

Proxy for color management.

This embeds colorama’s Fore/Style, or DummyColors instances when colors are disabled.

Fore

Style

e3.testsuite.dump_environ(filename: str, env: e3.env.Env) → None

Dump environment variables into a sourceable file.

e3.testsuite.enum_to_cmdline_args_map(enum_cls: Type[EnumType]) → Dict[str, EnumType]

Turn enum alternatives into command-line arguments.

This helps exposing enums for options on the command-line. This turns alternative names into lower case and replaces underscores with dashes.

e3.testsuite.isatty(stream: IO[AnyStr]) → bool

Return whether stream is a TTY.

This is a safe predicate: it works if stream is None or if it does not even support TTY detection: in these cases, be conservative (consider it’s not a TTY).

e3.testsuite.logger

exception e3.testsuite.TestAbort

Bases: Exception

Raise this to abort silently the execution of a test fragment.

class e3.testsuite.TestsuiteCore(root_dir: str | None = None, testsuite_name: str = 'Untitled testsuite')

Testsuite Core driver.

This class is the base of Testsuite class and should not be instanciated. It’s not recommended to override any of the functions declared in it.

See documentation of Testsuite class for overridable methods and variables.

root_dir = b'.'

Root directory for the testsuite, i.e. directory from which the test directory (see self.test_dir) is looked up.

test_dir

Root directory for the tree in which testcases are searched.

consecutive_failures = 0

return_values: Dict[str, Any]

result_tracebacks: Dict[str, List[str]]

testsuite_name = 'Untitled testsuite'

aborted_too_many_failures = False

Whether the testsuite aborted because of too many consecutive test failures (see the –max-consecutive-failures command-line option).

use_multiprocessing = False

Whether to use multi-processing for tests parallelism.

Beyond a certain level of parallelism, Python’s GIL contention is too high to benefit from more processors. When we reach this level, it is more interesting to use multiple processes to cancel the GIL contention.

The actual value for this attribute is computed once the DAG is built, in the “compute_use_multiprocessing” method.

_test_counter() → int

_test_status_counters() → Dict[result.TestStatus, int]

_results() → Dict[str, result.TestStatus]

property test_counter: int

Return the number of test results in the report.

Warning: this method is obsolete and will be removed in the future.

property test_status_counters: Dict[result.TestStatus, int]

Return test result counts per test status.

Warning: this method is obsolete and will be removed in the future.

property results: Dict[str, result.TestStatus]

Return a mapping from test names to results.

Warning: this method is obsolete and will be removed in the future.

abstractmethod compute_use_multiprocessing() → bool

Return whether to use multi-processing for tests parallelism.

See docstring for the “use_multiprocessing” attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the “–force-multiprocessing” command line option.

testsuite_main(args: List[str] | None = None) → int

Main for the main testsuite script.

Parameters:

args – Command line arguments. If None, use sys.argv.

Returns:

The testsuite status code (0 for success, a positive for failure).

get_test_list(sublist: List[str]) → List[testcase_finder.ParsedTest]

Retrieve the list of tests.

Parameters:

sublist – A list of tests scenarios or patterns.

add_test(dag: e3.collection.dag.DAG, parsed_test: testcase_finder.ParsedTest) → None

Register a test to run.

Parameters:

• dag – The DAG of test fragments to execute for the testsuite.

• parsed_test – Test to instantiate.

dump_testsuite_result() → None

Log a summary of test results.

Subclasses are free to override this to do whatever is suitable for them.

collect_result(fragment: fragment.TestFragment) → None

Import test results from fragment into testsuite reports.

Parameters:

fragment – Test fragment (just completed) from which to import test results.

add_result(item: driver.ResultQueueItem) → None

Add a test result to the result index and log it.

Parameters:

item – Result queue item for the result to add.

add_test_error(test_name: str, message: str, tb: str | None = None) → None

Create and add an ERROR test status.

Parameters:

• test_name – Prefix for the test result to create. This adds a suffix to avoid clashes.

• message (str) – Error message.

• tb – Optional traceback for the error.

setup_result_dirs() → None

Create the output directory in which the results are stored.

run_standard_mainloop(dag: e3.collection.dag.DAG) → None

Run the main loop to execute test fragments in threads.

run_multiprocess_mainloop(dag: e3.collection.dag.DAG) → None

Run the main loop to execute test fragments in subprocesses.

property tests_subdir: str

Abstractmethod:

Return the subdirectory in which tests are looked for.

The returned directory name is considered relative to the root testsuite directory (self.root_dir).

property test_driver_map: Dict[str, Type[driver.TestDriver]]

Abstractmethod:

Return a map from test driver names to TestDriver subclasses.

Test finders will be able to use this map to fetch the test drivers referenced in testcases.

property default_driver: str | None

Abstractmethod:

Return the name of the default driver for testcases.

When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver.

abstractmethod test_name(test_dir: str) → str

Compute the test name given a testcase spec.

This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as :, …).

property test_finders: List[testcase_finder.TestFinder]

Abstractmethod:

Return test finders to probe tests directories.

abstractmethod add_options(parser: argparse.ArgumentParser) → None

Add testsuite specific switches.

Subclasses can override this method to add their own testsuite command-line options.

Parameters:

parser – Parser for command-line arguments. See <https://docs.python.org/3/library/argparse.html> for usage.

abstractmethod set_up() → None

Execute operations before running the testsuite.

Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases.

By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance:

• process testsuite-specific options;

• initialize environment variables;

• adjust self.env (object forwarded to test drivers).

abstractmethod tear_down() → None

Execute operation when finalizing the testsuite.

By default, this cleans the working (temporary) directory in which the tests were run.

abstractmethod write_comment_file(comment_file: IO[str]) → None

Write the comment file’s content.

Parameters:

comment_file – File descriptor for the comment file. Overriding methods should only call its “write” method (or print to it).

property default_max_consecutive_failures: int

Abstractmethod:

Return the default maximum number of consecutive failures.

In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite.

This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled.

property default_failure_exit_code: int

Abstractmethod:

Return the default exit code when at least one test fails.

property auto_generate_text_report: bool

Abstractmethod:

Return whether to automatically generate a text report.

This is disabled by default (and controlled by the –generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results.

abstractmethod adjust_dag_dependencies(dag: e3.collection.dag.DAG) → None

Adjust dependencies in the DAG of all test fragments.

Parameters:

• dag – DAG to adjust.

• fragments – Set of all fragments added so far to the DAG.

property multiprocessing_supported: bool

Abstractmethod:

Return whether running test fragments in subprocesses is supported.

When multiprocessing is enabled (see the “use_multiprocessing” attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData’s “previous_values” argument is always an empty dict).

This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don’t use the “previous_values” mechanism.

Testsuite authors can use the “–force-multiprocessing” testsuite option to check if this works.

class e3.testsuite.Testsuite(root_dir: str | None = None, testsuite_name: str = 'Untitled testsuite')

Bases: TestsuiteCore

Testsuite class.

When implementing a new testsuite you should create a class that inherit from this class.

property tests_subdir: str

Return the subdirectory in which tests are looked for.

The returned directory name is considered relative to the root testsuite directory (self.root_dir).

property test_driver_map: Dict[str, Type[driver.TestDriver]]

Abstractmethod:

Return a map from test driver names to TestDriver subclasses.

Test finders will be able to use this map to fetch the test drivers referenced in testcases.

property default_driver: str | None

Return the name of the default driver for testcases.

When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver.

test_name(test_dir: str) → str

Compute the test name given a testcase spec.

This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as :, …).

property test_finders: List[testcase_finder.TestFinder]

Return test finders to probe tests directories.

add_options(parser: argparse.ArgumentParser) → None

Add testsuite specific switches.

Subclasses can override this method to add their own testsuite command-line options.

Parameters:

parser – Parser for command-line arguments. See <https://docs.python.org/3/library/argparse.html> for usage.

set_up() → None

Execute operations before running the testsuite.

Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases.

By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance:

• process testsuite-specific options;

• initialize environment variables;

• adjust self.env (object forwarded to test drivers).

tear_down() → None

Execute operation when finalizing the testsuite.

By default, this cleans the working (temporary) directory in which the tests were run.

write_comment_file(comment_file: IO[str]) → None

Write the comment file’s content.

Parameters:

comment_file – File descriptor for the comment file. Overriding methods should only call its “write” method (or print to it).

property default_max_consecutive_failures: int

Return the default maximum number of consecutive failures.

In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite.

This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled.

property default_failure_exit_code: int

Return the default exit code when at least one test fails.

property auto_generate_text_report: bool

Return whether to automatically generate a text report.

This is disabled by default (and controlled by the –generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results.

adjust_dag_dependencies(dag: e3.collection.dag.DAG) → None

Adjust dependencies in the DAG of all test fragments.

Parameters:

• dag – DAG to adjust.

• fragments – Set of all fragments added so far to the DAG.

property multiprocessing_supported: bool

Return whether running test fragments in subprocesses is supported.

When multiprocessing is enabled (see the “use_multiprocessing” attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData’s “previous_values” argument is always an empty dict).

This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don’t use the “previous_values” mechanism.

Testsuite authors can use the “–force-multiprocessing” testsuite option to check if this works.

compute_use_multiprocessing() → bool

Return whether to use multi-processing for tests parallelism.

See docstring for the “use_multiprocessing” attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the “–force-multiprocessing” command line option.