How to Write a Test¶
SoS, as of April 2021, uses the python-avocado testing framework to handle test cases in validating release and code changes. This allows for easy creation of new test cases, and facilitates higher confidence in downstream releases of SoS. Additionally, it provides an easy framework to allow those downstreams to push tests back into upstream.
As a general rule, all test cases defined in the test suite must pass for any PR to be merged.
If you are adding new functionality with your PR, it should bring with it enough test cases (even if it’s only 1) to vet that new functionality. If you are enhancing an existing plugin or creating a new plugin, it is highly recommended to include a test case, though not required.
Ideally, bug fix PRs should also include a regression test if it is feasible to do so in test VMs.
Testing environment¶
All tests are conducted on fresh VMs provisioned on GitHub runners for every PR opened. Any subsequent updates to the PR will trigger a new round of tests (and thus a new set of VMs provisioned for that test run). These VMs are provisioned by GitHub Actions.
Currently, tests are performed for Fedora, Ubuntu, Debian and CentOS Stream distributions.
Testing methodology¶
Tests are logically categorized into “stages” as follows:
stage 0 -> Unittests
stage 1 -> Basic function tests, no mocking allowed
stage 2 -> Mocked tests for specific scenarios/regressions
There are plans to potentially add “stage 3” tests which would allow for complex networking configurations and potentially multi-host arrangements, once the testing infrastructure allows for that.
Stage 0 tests are conducted by nose and are traditional unit tests.
Stage 1 tests involve executing actual sos commands, saving the
output and then inspecting the output and investigating test system
state (mostly to ensure that state was not changed by the sos
command execution). Stage 2 tests will allow for changes prior to the
sos command execution, such as dropping specific files on the
filesystem. Any changes made by a stage 2 test setup are removed after
the test has concluded.
For each PR, stage 0 and stage 1 tests are always run. Stage 1 tests are run against the distribution list above, for both the “current” release and the release immediate prior to that one. Stage 2 tests are run for current releases only.
To run the test suite manually, for stage 1, run the following from the git checkout:
# PYTHONPATH=tests/ avocado run -t stageone tests/
If you are on a system that can withstand interim changes during the
test suite, you can use -t stagetwo to run the stage 2 mocking
tests. It is not recommended to run both stage 1 and stage 2 tests in
the same invocation, as there is no guarantee of the order in which the
individual tests will run.
Writing a test for SoS¶
Most tests focus around sos report or specific plugins. These can
either be stage 1 or stage 2 tests. When determining which stage the
test belong to, consider the following:
Do I need to test that specific content within a file is properly removed via post processing?
Do I need to make sure a specific command/binary is present that is likely not a default installation?
Do I need to test that my iteration loop in a plugin over a config file is working as expected?
If you answered yes to any of these, you likely want a stage 2 test, and if not then a stage 1 test is fine.
Writing a Stage 1 test¶
Stage 1 tests are fairly straight forward. All these tests must subclass
StageOneReportTest and set the sos_cmd class attr. From there,
you only need to define test_ methods that test for your criteria.
Additionally, you’ll need to set the avocado tags in the class
docstring.
For example, consider this test:
from sos_tests import StageOneReportTest
class NormalSoSReport(StageOneReportTest):
"""
:avocado: tags=stageone
"""
sos_cmd = '-vvv --label thisismylabel'
def test_debug_in_logs_verbose(self):
self.assertSosLogContains('DEBUG')
def test_postproc_called(self):
self.assertSosLogContains('substituting scrpath')
def test_label_applied_to_archive(self):
self.assertTrue('thisismylabel' in self.archive)
def test_free_symlink_created(self):
self.assertFileCollected('free')
First we instantiate a new class which subclasses
StageOneReportTest. This allows us to ensure that avocado will run
the sos command built from sos_cmd, extract the resulting archive,
and make sure our assertions are relative to that extracted archive.
Avocado will execute all test_* methods defined within a class,
including the “base tests” defined in StageOneReportTest. That way,
you only need to write test methods for the exact criteria you’re
testing, rather than re-writing several tests such as our “no new kmods
were loaded” test.
The :avocado: tags=stageone line in the docstring ensures that
avocado identifies this test as stageone.
Here’s a breakdown of the remaining items in this test class:
sos_cmd- This class attr defines the options passed to ansos reportexecution. You only need to specify options, not the binary, path, or--batch.assertSosLogContains- Ensure that the given string appears insos_logs/sos.log. There is alsoassertSosLogNotContainswhich ensures the string does not appear there.assertFileCollected- Ensure the given file appears in the archive. There is alsoassertFileNotCollectedfor ensuring a file was not collected. Note that this is for single files only. If you need to match globs, useassertFileGlobInArchiveorassertFileGlobNotInArchive.
Writing a Stage 2 test¶
Stage 2 tests have all the bits of stage 1, with mocking functionality
strapped on top. For these, you’ll subclass StageTwoReportTest.
Let’s take a look at the following theoretical stage 2 test:
from sos_tests import StageTwoReportTest
class FooTest(StageTwoReportTest):
files = [('bar', '/etc/foo/bar')]
packages = {'rhel': 'foobar'}
install_plugins = ['foo']
sos_cmd = '-v '
def test_foo_plugin_enabled(self):
self.assertPluginIncluded('foo')
def test_foo_collected(self):
self.assertFileCollected('/etc/foo/bar')
There’s a lot going on here, mostly with mocking. As stated above, Stage 2 tests will perform mocking operations before an sos command execution, and clean up that mocking once it is complete. Let’s break this down:
files- This is a list of files that should be dropped on the test system’s filesystem. The values of this list should be tuples, in the format of(relative_src, absolute_dest). Meaning, files should be placed in the same (sub)dir as the test.pyfile. It is a good idea to create a subdirectory for each new stagetwo test, and place all mocked files therein.packages- This dict defines the list of packages to be installed, and is distro-dependent. These packages will be installed on the test system before the test, and uninstalled after the test. For RH family systems, you only need to definerhel,fedora, orcentoshere and it will apply to all 3. In the rare event the package name differs between sayrhelandfedora, simply define the right package name for each individually in this dict.install_plugins- A list of “fake” plugins to install for the local sos branch in order to run the plugin for the test. If your test requires the use of a plugin that does not exist in sos (e.g. one used specifically to test sos functionality, like timeouts) you can include that plugin here. These plugins should be written to the same directory as the test file - ideally, in a dedicated subdir within thetests/tree, and the name(s) in this list should be the filename without the.pyextention.
My test only applies to a specific distribution, how do I prevent it from failing on other distributions?¶
At current, you can limit specific test methods but not entire test
classes. You can do this via helper decorators provided in
sos_tests.
from sos_tests import StageOneReportTest, redhat_only, ubuntu_only
class Barfoo(StageOneReportTest):
sos_cmd = '--some-option'
@redhat_only
def test_redhat_criteria(self):
[...some assertion...]
@ubuntu_only
def test_ubuntu_criteria(self):
[...some assertion...]
The above test class will run the respectively decorated methods on test
systems matching those distributions, and skip the test on all
others. For both distributions, the test_* methods defined in the
base StageOneReportTest will run as normal.
Like the packages attr for mocking, @redhat_only applies to
RHEL, Fedora, and CentOS.
I need to do some non-mocking setup prior to a test executing¶
If you need to do some pre-test setup within a test class, use the
pre_sos_setup() method. This method will run prior to sos execution,
but note that it is not acceptable to do any mocking in this function.
As an example use case, look at one of our smoke tests:
class AllPluginSmokeTest(StageOneReportTest):
"""This test forcibly enables ALL plugins available to the local branch
and aims to make sure that there are no exceptions generated during the
execution
:avocado: tags=stageone
"""
def pre_sos_setup(self):
_cmd = '%s report --list-plugins' % SOS_BIN
out = process.run(_cmd, timeout=300).stdout.decode()
reg = DISABLED + '(.*?)' + OPTIONS
self.plugs = []
for result in re.findall(reg, out, re.S):
for line in result.splitlines():
try:
self.plugs.append(line.split()[0])
except Exception:
pass
self.sos_cmd = '-e %s' % ','.join(p for p in self.plugs)
In this test case, we need to generate a list of all locally available
plugins. So we run --list-plugins in pre_sos_setup() and then
grok the output, setting sos_cmd appropriately from that list.
I have a test case for a downstream bugzilla/launchpad/etc…¶
Great! We love getting new regression tests from downstreams. It is
preferred that tests like this be placed under
tests/vendor_tests/<vendor>/, and ideally identify which BZ/LP/etc
they are for. For example, tests/vendor_tests/redhat/bz12345.py
makes it easy at a glance to see what that test is addressing from that
vendor.