• 
      

    Add formal support for writing both simulation and integration tests.

    Review Request #11555 — Created March 26, 2021 and submitted

    Information

    ReviewBot
    release-3.0.x

    Reviewers

    To ensure Review Bot works as expected, we need to write tests that
    simulate a call to an external tool, and tests that actually execute an
    external tool. A few of these have already been written, but maintenance
    was already becoming problematic, as we want to ensure consistency
    in both the data going into these tests and the data we assert.

    This adds a new BaseToolTestCase class for tool tests, along with an
    accompanying ToolTestCaseMetaclass and @integration_test and
    @simulation_test decorators.

    The decorators mark a test as being capable of integration testing and
    simulation testing, respectively. They're intended for
    BaseTool.execute() unit tests. The metaclass will scan for these and
    split any test function into separate simulation/integration tests (if
    the appropriate decorators are applied).

    This keeps things very easy to maintain, but with the expense of a quirk
    with test names. A test_foo() defined on the class will no longer be
    available to the test runner. Instead, separate
    test_simulation_foo() and test_integration_foo() test functions will
    be available. This isn't immediately obvious, but is a quirk we can
    learn to live with.

    Made use of this in upcoming changes. Tests all passed.

    Summary ID
    Add formal support for writing both simulation and integration tests.
    To ensure Review Bot works as expected, we need to write tests that simulate a call to an external tool, and tests that actually execute an external tool. A few of these have already been written, but maintenance was already becoming problematic, as we want to ensure consistency in both the data going into these tests and the data we assert. This adds a new `BaseToolTestCase` class for tool tests, along with an accompanying `ToolTestCaseMetaclass` and `@integration_test` ` and `@simulation_test` decorators. The decorators mark a test as being capable of integration testing and simulation testing, respectively. They're intended for `BaseTool.execute()` unit tests. The metaclass will scan for these and split any test function into separate simulation/integration tests (if the appropriate decorators are applied). This keeps things very easy to maintain, but with the expense of a quirk with test names. A `test_foo()` defined on the class will no longer be available to the test runner. Instead, separate `test_simulation_foo()` and `test_integration_foo()` test functions will be available. This isn't immediately obvious, but is a quirk we can learn to live with.
    89fa4180132a326036de570833cbffc3db4d2e29
    Description From Last Updated

    E127 continuation line over-indented for visual indent

    reviewbotreviewbot

    E127 continuation line over-indented for visual indent

    reviewbotreviewbot
    Checks run (1 failed, 1 succeeded)
    flake8 failed.
    JSHint passed.

    flake8

    chipx86
    Review request changed
    Change Summary:

    Fixed an indentation error.

    Commits:
    Summary ID
    Add formal support for writing both simulation and integration tests.
    To ensure Review Bot works as expected, we need to write tests that simulate a call to an external tool, and tests that actually execute an external tool. A few of these have already been written, but maintenance was already becoming problematic, as we want to ensure consistency in both the data going into these tests and the data we assert. This adds a new `BaseToolTestCase` class for tool tests, along with an accompanying `ToolTestCaseMetaclass` and `@integration_test` ` and `@simulation_test` decorators. The decorators mark a test as being capable of integration testing and simulation testing, respectively. They're intended for `BaseTool.execute()` unit tests. The metaclass will scan for these and split any test function into separate simulation/integration tests (if the appropriate decorators are applied). This keeps things very easy to maintain, but with the expense of a quirk with test names. A `test_foo()` defined on the class will no longer be available to the test runner. Instead, separate `test_simulation_foo()` and `test_integration_foo()` test functions will be available. This isn't immediately obvious, but is a quirk we can learn to live with.
    a4f84698589a4b7415b5b46015076bb492c11264
    Add formal support for writing both simulation and integration tests.
    To ensure Review Bot works as expected, we need to write tests that simulate a call to an external tool, and tests that actually execute an external tool. A few of these have already been written, but maintenance was already becoming problematic, as we want to ensure consistency in both the data going into these tests and the data we assert. This adds a new `BaseToolTestCase` class for tool tests, along with an accompanying `ToolTestCaseMetaclass` and `@integration_test` ` and `@simulation_test` decorators. The decorators mark a test as being capable of integration testing and simulation testing, respectively. They're intended for `BaseTool.execute()` unit tests. The metaclass will scan for these and split any test function into separate simulation/integration tests (if the appropriate decorators are applied). This keeps things very easy to maintain, but with the expense of a quirk with test names. A `test_foo()` defined on the class will no longer be available to the test runner. Instead, separate `test_simulation_foo()` and `test_integration_foo()` test functions will be available. This isn't immediately obvious, but is a quirk we can learn to live with.
    89fa4180132a326036de570833cbffc3db4d2e29

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    david
    1. Ship It!
    2. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-3.0.x (c8fcac0)