e3.testsuite.driver.diff

Classes

`OutputRefiner`	Interface to refine a test output before baseline and actual comparison.
`RefiningChain`	Simple wrapper for a sequence of output refiners applied in chain.
`Substitute`	Replace substrings in outputs.
`CanonicalizeLineEndings`	Replace rn with n in outputs.
`ReplacePath`	Return an output refiner to replace the given path.
`PatternSubstitute`	Replace patterns in outputs.
`LineByLine`	Wrapper to apply an output refine line by line.
`DiffTestDriver`	Test driver to compute test output against a baseline.

Module Contents

class e3.testsuite.driver.diff.OutputRefiner

Bases: Generic[AnyStr]

Interface to refine a test output before baseline and actual comparison.

Sometimes, the way a library/tool works forces it to have outputs that depends on the environment (for instance: the location of the testsuite on the filesystem, the current date, etc.). Refiners makes it possible for a testsuite to “hide” these discrepancies during the diff computation.

Note that output refiners might get called bytes strings when the test drivers operate in binary mode.

abstractmethod refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.RefiningChain(refiners: List[OutputRefiner])

Bases: OutputRefiner[AnyStr]

Simple wrapper for a sequence of output refiners applied in chain.

refiners

refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.Substitute(substring: AnyStr, replacement: AnyStr | None = None)

Bases: OutputRefiner[AnyStr]

Replace substrings in outputs.

substring: AnyStr

replacement: AnyStr

refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.CanonicalizeLineEndings

Bases: OutputRefiner[AnyStr]

Replace rn with n in outputs.

refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.ReplacePath(path: str, replacement: str = '')

Bases: RefiningChain[str]

Return an output refiner to replace the given path.

class e3.testsuite.driver.diff.PatternSubstitute(pattern: AnyStr, replacement: AnyStr | None = None)

Bases: OutputRefiner, Generic[AnyStr]

Replace patterns in outputs.

regexp: Pattern[AnyStr]

replacement: AnyStr

refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.LineByLine(refiner: OutputRefiner)

Bases: OutputRefiner[AnyStr]

Wrapper to apply an output refine line by line.

refiner

refine(output: AnyStr) → AnyStr

Refine a test/baseline output.

Parameters:: output – Output to refine.

class e3.testsuite.driver.diff.DiffTestDriver(env: e3.env.Env, test_env: Dict[str, Any])

Bases: e3.testsuite.driver.classic.ClassicTestDriver

Test driver to compute test output against a baseline.

property baseline_file: Tuple[str, bool]

Return the test output baseline file.

Returns:: The name of the text file (relative to test directories) that contains the expected test output and whether the baseline is a regexp.

property baseline: Tuple[str | None, str | bytes, bool]

Return the test output baseline.

Subclasses can override this method if they want to provide a baseline that does not come from a file, short-circuiting the baseline_file property.

Returns:: The baseline absolute filename (if any), the baseline content, as a string or as a bytes string (depending on the default encoding), and whether the baseline is a regexp. The baseline filename is used to rewrite test output: leave it to None if rewriting does not make sense.

property output_refiners: List[OutputRefiner]

List of refiners for test baselines/outputs.

This just returns a refiner to canonicalize line endings unless the test environment contains a “strict_line_endings” key associated to true.

property refine_baseline: bool: Whether to apply output refiners to the output baseline.

property diff_ignore_white_chars: bool

Whether to ignore white characters in diff computations.

This returns whether the comparison between test output and baseline must ignore whitespaces (leading and trailing spaces, tabs and carriage returns on lines, and empty lines). Note that if we don’t ignore them, we still canonicalize line separators (CRLF are replaced by LF before the comparison).

Note that at some point, this mechanism should be unified with the output_refiners machinery. However, this relies on e3.diff’s ignore_white_chars feature, which is not trivial to reimplement.

property diff_context_size: int: Positive number of context lines to include in diff computations.

set_up() → None

Run initialization operations before a test runs.

Subclasses can override this to prepare testcase execution.

Having a callback separate from “run” is useful when dealing with inheritance: overriding the “set_up” method in subclasses allows to append setup actions before the testcase execution actually takes place (in the “run” method).

If everything happened in “run” method, that would not be possible unless re-implementing the “run” method in each subclass, with obvious code duplication issues.

compute_diff(baseline_file: str | None, baseline: AnyStr, actual: AnyStr, failure_message: str = 'unexpected output', ignore_white_chars: bool | None = None, context_size: int | None = None, truncate_logs_threshold: int | None = None) → List[str]

Compute the diff between expected and actual outputs.

Return an empty list if there is no diff, and return a list that contains an error message based on failure_message otherwise.

Parameters:

baseline_file – Absolute filename for the text file that contains the expected content (for baseline rewriting, if enabled), or None.
actual – Actual content to compare.
failure_message – Failure message to return if there is a difference.
ignore_white_chars – Whether to ignore whitespaces during the diff computation. If left to None, use self.diff_ignore_white_chars.
context_size – Positive number of context lines to include in diff computations. If left to None, use self.diff_context_size.
truncate_logs_threshold – Threshold to truncate the diff message in self.result.log. See e3.testsuite.result.truncated’s line_count argument. If left to None, use the testsuite’s --truncate-logs option.

compute_regexp_match(regexp: Pattern[AnyStr] | AnyStr, actual: AnyStr, failure_message: str = 'output does not match expected pattern', truncate_logs_threshold: int | None = None) → List[str]

Compute whether the actual output matches a regexp.

Return an empty list if the acutal content matches, and return a list that contains an error message based on failure_message otherwise.

Parameters:

regexp – Regular expression to use.
actual – Actual content to match.
failure_message – Failure message to return if there is a difference.
truncate_logs_threshold – Threshold to truncate the diff message in self.result.log. See e3.testsuite.result.truncated’s line_count argument. If left to None, use the testsuite’s --truncate-logs option.

compute_failures() → List[str]

Return a failure if self.output.log does not match the baseline.

This computes a diff with the content of the baseline file, unless there is a “baseline_regexp” entry in the test environment that evaluates to true.

Subclasses can override this if they need more involved analysis: for instance computing multiple diffs.