Package com.carrotsearch.randomizedtesting

Class RandomizedRunner

  • All Implemented Interfaces:
    org.junit.runner.Describable, org.junit.runner.manipulation.Filterable
    public final class RandomizedRunner
extends org.junit.runner.Runner
implements org.junit.runner.manipulation.Filterable
    A Runner implementation for running randomized test cases with predictable and repeatable randomness.

    Supports the following JUnit4 features:

    • BeforeClass-annotated methods (before all tests of a class/superclass),
    • Before-annotated methods (before each test),
    • Test-annotated methods,
    • After-annotated methods (after each test),
    • AfterClass-annotated methods (after all tests of a class/superclass),
    • Rule-annotated fields implementing MethodRule and TestRule.

    Contracts:

    • BeforeClass, Before methods declared in superclasses are called before methods declared in subclasses,
    • AfterClass, After methods declared in superclasses are called after methods declared in subclasses,
    • BeforeClass, Before, AfterClass, After methods declared within the same class are called in randomized order derived from the master seed (repeatable with the same seed),

    Deviations from "standard" JUnit:

    • test methods are allowed to return values (the return value is ignored),
    • hook methods need not be public; in fact, it is encouraged to make them private to avoid accidental shadowing which silently drops parent hooks from executing (applies to class hooks mostly, but also to instance hooks).
    • all exceptions raised during hooks or test case execution are reported to the notifier, there is no suppression or chaining of exceptions,
    • a test method must not leave behind any active threads; this is detected using ThreadGroup active counts and is sometimes problematic (many classes in the standard library leave active threads behind without waiting for them to terminate). One can use the ThreadLeakScope, ThreadLeakAction and other annotations to control how aggressive the detection strategy is and if it fails the test or not.
    • uncaught exceptions from any of children threads will cause the test to fail.
    See Also:
    RandomizedTest, ThreadLeakAction, ThreadLeakScope, ThreadLeakZombies, ThreadLeakGroup, ThreadLeakLingering, ThreadLeakFilters, Listeners, RandomizedContext, TestMethodProviders

    • Field Detail

      • AUGMENTED_SEED_PACKAGE

        public static final java.lang.String AUGMENTED_SEED_PACKAGE
        Fake package of a stack trace entry inserted into exceptions thrown by test methods. These stack entries contain additional information about seeds used during execution.
        See Also:
        Constant Field Values

      • DEFAULT_TIMEOUT

        public static final int DEFAULT_TIMEOUT
        Default timeout for a single test case. By default the timeout is disabled. Use global system property SysGlobals.SYSPROP_TIMEOUT or an annotation Timeout if you need to set timeouts or expect some test cases may hang. This will slightly slow down the tests because each test case is executed in a forked thread.
        See Also:
        SysGlobals.SYSPROP_TIMEOUT(), Constant Field Values

      • DEFAULT_KILLATTEMPTS

        public static final int DEFAULT_KILLATTEMPTS
        The default number of first interrupts, then Thread.stop attempts.
        See Also:
        Constant Field Values

      • DEFAULT_KILLWAIT

        public static final int DEFAULT_KILLWAIT
        Time in between interrupt retries or stop retries.
        See Also:
        Constant Field Values

      • DEFAULT_ITERATIONS

        public static final int DEFAULT_ITERATIONS
        The default number of test repeat iterations.
        See Also:
        Constant Field Values

      • logger

        static final java.util.logging.Logger logger
        Package scope logger.

      • sequencer

        private static final java.util.concurrent.atomic.AtomicLong sequencer
        A sequencer for affecting the initial seed in case of rapid succession of this class instance creations. Not likely, but can happen two could get the same seed.

      • DEFAULT_STACK_FILTERS

        private static final java.util.List<java.lang.String> DEFAULT_STACK_FILTERS

      • suiteClass

        private final java.lang.Class<?> suiteClass
        The class with test methods (suite).

      • runnerRandomness

        final Randomness runnerRandomness
        The runner's seed (master).

      • testCaseRandomnessOverride

        private Randomness testCaseRandomnessOverride
        If #SYSPROP_RANDOM_SEED property is used with two arguments (master:method) then this field contains method-level override.

      • iterationsOverride

        private final java.lang.Integer iterationsOverride
        The number of each test's randomized iterations.
        See Also:
        #SYSPROP_ITERATIONS

      • testCandidates

        private java.util.List<RandomizedRunner.TestCandidate> testCandidates
        All test candidates, processed (seeds assigned) and flattened.

      • suiteDescription

        private org.junit.runner.Description suiteDescription
        Class suite description.

      • runnerThreadGroup

        RunnerThreadGroup runnerThreadGroup
        All tests are executed under a specified thread group so that we can have some control over how many threads have been started/ stopped. System daemons shouldn't be under this group.

      • appendSeedParameter

        private boolean appendSeedParameter
        See Also:
        #SYSPROP_APPEND_SEED

      • traces

        private final TraceFormatting traces
        Stack trace filtering/ dumping.

      • containerRunner

        private RunnerContainer containerRunner
        The container we're running in.

      • classModel

        private ClassModel classModel
        Class model.

      • randomSupplier

        private final RandomSupplier randomSupplier
        Random class implementation supplier.

      • shuffledMethodsCache

        private java.util.Map<java.lang.Class<? extends java.lang.annotation.Annotation>,​java.util.List<java.lang.reflect.Method>> shuffledMethodsCache
        Methods cache.

      • zombieMarker

        static java.util.concurrent.atomic.AtomicBoolean zombieMarker
        A marker for flagging zombie threads (leaked threads that couldn't be killed).

      • mainThreadGroup

        static final java.lang.ThreadGroup mainThreadGroup
        The "main" thread group we will be tracking (including subgroups).

      • restoreProperties

        private final java.util.Map<java.lang.String,​java.lang.String> restoreProperties

    • Constructor Detail

      • RandomizedRunner

        public RandomizedRunner​(java.lang.Class<?> testClass)
                 throws org.junit.runners.model.InitializationError
        Creates a new runner for the given class.
        Throws:
        org.junit.runners.model.InitializationError

    • Method Detail

      • determineRandomSupplier

        private RandomSupplier determineRandomSupplier​(java.lang.Class<?> testClass)

      • detectContainer

        private static RunnerContainer detectContainer()
        Attempt to detect the container we're running under.

      • getDescription

        public org.junit.runner.Description getDescription()
        Return the current tree of test descriptions (filtered).
        Specified by:
        getDescription in interface org.junit.runner.Describable
        Specified by:
        getDescription in class org.junit.runner.Runner

      • filter

        public void filter​(org.junit.runner.manipulation.Filter filter)
            throws org.junit.runner.manipulation.NoTestsRemainException
        Implement Filterable because GUIs depend on it to run tests selectively.
        Specified by:
        filter in interface org.junit.runner.manipulation.Filterable
        Throws:
        org.junit.runner.manipulation.NoTestsRemainException

      • prune

        private static void prune​(org.junit.runner.Description suite,
                          java.util.Set<org.junit.runner.Description> permitted)

      • run

        public void run​(org.junit.runner.notification.RunNotifier notifier)
        Runs all tests and hooks.
        Specified by:
        run in class org.junit.runner.Runner

      • restoreSystemProperties

        private void restoreSystemProperties()

      • processSystemProperties

        private void processSystemProperties()

      • runSuite

        private void runSuite​(org.junit.runner.notification.RunNotifier notifier)
        Test execution logic for the entire suite.

      • runSuite

        private void runSuite​(RandomizedContext context,
                      org.junit.runner.notification.RunNotifier notifier)
        Test execution logic for the entire suite, executing under designated RunnerThreadGroup.

      • withCloseContextResources

        private static org.junit.runners.model.Statement withCloseContextResources​(org.junit.runners.model.Statement s,
                                                                           LifecycleScope scope)
        Wrap with a rule to close context resources.

      • fireTestFailure

        private void fireTestFailure​(org.junit.runner.notification.RunNotifier notifier,
                             org.junit.runner.Description description,
                             java.lang.Throwable t)

      • withClassBefores

        private org.junit.runners.model.Statement withClassBefores​(org.junit.runners.model.Statement s)
        Decorate a Statement with BeforeClass hooks.

      • withClassAfters

        private org.junit.runners.model.Statement withClassAfters​(org.junit.runners.model.Statement s)

      • withClassRules

        private org.junit.runners.model.Statement withClassRules​(org.junit.runners.model.Statement s)
        Wrap with ClassRules.

      • wrapBeforeAndAfters

        private org.junit.runners.model.Statement wrapBeforeAndAfters​(org.junit.runners.model.Statement s,
                                                              RandomizedRunner.TestCandidate c,
                                                              java.lang.Object instance)
        Wrap before and after hooks.

      • wrapExpectedExceptions

        private org.junit.runners.model.Statement wrapExpectedExceptions​(org.junit.runners.model.Statement s,
                                                                 RandomizedRunner.TestCandidate c)
        Wrap the given statement into another catching the expected exception, if declared.

      • wrapMethodRules

        private org.junit.runners.model.Statement wrapMethodRules​(org.junit.runners.model.Statement s,
                                                          RandomizedRunner.TestCandidate c,
                                                          java.lang.Object instance)
        Wrap the given statement in any declared MethodRules (old style rules).

      • getAnnotatedFieldValues

        private <T> java.util.List<T> getAnnotatedFieldValues​(java.lang.Object test,
                                                      java.lang.Class<? extends java.lang.annotation.Annotation> annotationClass,
                                                      java.lang.Class<T> valueClass)

      • createContext

        private RandomizedContext createContext​(java.lang.ThreadGroup tg)
        Create randomized context for the run. The context is shared by all threads in a given thread group (but the source of Randomness is assigned per-thread).

      • subscribeListeners

        private void subscribeListeners​(org.junit.runner.notification.RunNotifier notifier)
        Subscribe annotation listeners to the notifier.

      • unsubscribeListeners

        private void unsubscribeListeners​(org.junit.runner.notification.RunNotifier notifier)
        Unsubscribe listeners.

      • emptyToNull

        static java.lang.String emptyToNull​(java.lang.String value)
        Normalize empty strings to nulls.

      • hasIgnoreAnnotation

        private boolean hasIgnoreAnnotation​(RandomizedRunner.TestCandidate c)
        Returns true if we should ignore this test candidate.

      • getShuffledMethods

        private java.util.List<java.lang.reflect.Method> getShuffledMethods​(java.lang.Class<? extends java.lang.annotation.Annotation> ann)
        Construct a list of ordered framework methods. Minor tweaks are done depending on the annotation (reversing order, etc.).

      • collectTestCandidates

        private java.util.List<RandomizedRunner.TestCandidate> collectTestCandidates​(org.junit.runner.Description classDescription)
        Collect all test candidates, regardless if they will be executed or not. At this point individual test methods are also expanded into multiple executions corresponding to the number of iterations (#SYSPROP_ITERATIONS) and the initial method seed is preassigned.

        The order of test candidates is shuffled based on the runner's random.

        See Also:
        Rants.RANT_1

      • collectCandidatesForMethod

        private java.util.List<RandomizedRunner.TestCandidate> collectCandidatesForMethod​(java.util.Map<java.lang.String,​java.lang.Integer> descriptionRepetitions,
                                                                                  java.lang.reflect.Constructor<?> constructor,
                                                                                  RandomizedRunner.TestMethodExecution testCase)
        Collect test candidates for a single method and the given seed.

      • zeroForNull

        private static int zeroForNull​(java.lang.Integer v)
        Replace null with zero.

      • collectMethodExecutions

        public java.util.List<RandomizedRunner.TestMethodExecution> collectMethodExecutions​(java.lang.reflect.Constructor<?> constructor,
                                                                                    java.util.List<java.lang.reflect.Method> testMethods)
        Collect test method executions from list of test methods and potentially parameters from parameter factory methods.

      • getInstanceProvider

        private InstanceProvider getInstanceProvider​(java.lang.reflect.Constructor<?> constructor,
                                             java.lang.Object[] args)
        Determine instance provider.

      • createDefaultArgumentFormatting

        private static java.lang.String createDefaultArgumentFormatting​(java.lang.reflect.Constructor<?> constructor)
        Default formatting string for constructor arguments.

      • isConstantSeedForAllIterations

        private boolean isConstantSeedForAllIterations​(java.lang.reflect.Method method)
        Determine if a given method's iterations should run with a fixed seed or not.

      • determineMethodIterationCount

        private int determineMethodIterationCount​(java.lang.reflect.Method method)
        Determine method iteration count based on (first declaration order wins):
        • global property #SYSPROP_ITERATIONS.
        • method annotation Repeat.
        • class annotation Repeat.
        • The default (1).

        • determineMethodSeeds

          private long[] determineMethodSeeds​(java.lang.reflect.Method method)
          Determine a given method's initial random seed.
          See Also:
          Seed, Seeds

        • invoke

          void invoke​(java.lang.reflect.Method m,
            java.lang.Object instance,
            java.lang.Object... args)
     throws java.lang.Throwable
          Invoke a given method on a suiteClass instance (can be null for static methods).
          Throws:
          java.lang.Throwable

        • validateTestMethods

          private void validateTestMethods​(java.util.List<java.lang.reflect.Method> testMethods)
          Perform additional checks on methods returned from the providers.

        • validateTarget

          private void validateTarget()
          Validate methods and hooks in the suiteClass. Follows "standard" JUnit rules, with some exceptions on return values and more rigorous checking of shadowed methods and fields.

        • augmentStackTrace

          static <T extends java.lang.Throwable> T augmentStackTrace​(T e,
                                                           Randomness... seeds)
          Augment stack trace of the given exception with seed infos.

        • getAnnotationsFromClassHierarchy

          private static <T extends java.lang.annotation.Annotation> java.util.List<T> getAnnotationsFromClassHierarchy​(java.lang.Class<?> clazz,
                                                                                                              java.lang.Class<T> annotation)
          Collect all annotations from a clazz hierarchy. Superclass's annotations come first. Inherited annotations are removed (hopefully, the spec. isn't clear on this whether the same object is returned or not for inherited annotations).

        • seedFromAnnot

          private long[] seedFromAnnot​(java.lang.reflect.AnnotatedElement element,
                             long randomSeed)
          Get an annotated element's Seed annotation and determine if it's fixed or not. If it is fixed, return the seeds. Otherwise return randomSeed.

        • getTraceFormatting

          public TraceFormatting getTraceFormatting()
          Stack trace formatting utilities. These may be initialized to filter out certain packages.

        • seedFromThrowable

          public static java.lang.String seedFromThrowable​(java.lang.Throwable t)
          RandomizedRunner augments stack traces of test methods that ended in an exception and inserts a fake entry starting with AUGMENTED_SEED_PACKAGE.
          Returns:
          A string is returned with seeds combined, if any. Null is returned if no augmentation can be found.

        • methodName

          public static java.lang.String methodName​(org.junit.runner.Description description)
          Attempts to extract just the method name from parameterized notation.