Generating and checking

About

In the hs-test library, tests are a set of test cases. One test case is an object of the TestCase class. The generate method should generate a list of objects of the TestCase class and return it. It's a low-level mechanism of writing tests.

In the previous sections, you saw tests with dynamically generated input, but it is possible to create tests with static predetermined input. If the program to test is rather simple you can use this kind of tests to check it.

Dynamic testing has some benefits over tests with static input:

1. You are able to test small parts of the output compared to checking the whole output. Parsing the whole output to determine if the program written correctly may be more error-prone rather than parsing small pieces of the output.
2. Some programs aren't testable with static input because the input may be dependent on the previous output (for example, Tic-Tac-Toe game).

This method of testing revealed serious weaknesses and limitations for testing complex programs. Use static input only when the program to be tested is very simple. In other cases, it's strongly recommended to use dynamic testing.

TestCase class

One test case is a single run of the user program. Before creating an object of the TestCase class, it is necessary to import it.

from hstest import TestCase

To parameterize a test case, uses named arguments.

TestCase(
    stdin=text,
    attach=object,
    files={src: content, src2: content2}
)

Input

The first and most frequently used parameter is stdin. It is intended to simulate input using the keyboard.

Let's look at this example:

TestCase(stdin="15\n23")

In this case, user's program below will run normally, count two numbers, and print their sum as if the user had entered these numbers using the keyboard. If you run this program directly, you will actually have to enter these two numbers from the keyboard. And you can use the stdin method to prepare this data in advance.

line = input()
first, second = line.split()
print(int(first) + int(second))

Arguments

The test case can also be parameterized by command line arguments. These arguments are passed to the program when starting the program from the console. For example:

$ python renamer.py --from old --to new --all

A Python program will get a list of six arguments: [renamer.py, --from, old, --to, new, --all] - the first parameter in Python is always a path to the launched file.

An example of how to set up a command-line arguments for test cases can be seen below:

TestCase(args=["--from", "old", "--to", "new", "--all"])

An example of using command line arguments in user code can be seen below:

import sys

print(len(sys.argv))
for arg in sys.argv:
    print(arg)

Files

You can also create external files for testing. Before starting the test case, these files are created on the file system so that the student's program can read them. After the user program finishes, these will be deleted from the file system.

An example of how to set up external files for test cases can be seen below:

TestCase(
    files={
        'file1.txt': 'File content',
        'file2.txt': 'Another content'
    }
)

Below you can see example of using these external files in user's code:

with open('file1.txt') as f:
    print(f.read())

Time limit

By default, the user program has 15 seconds to do all the necessary work. When the time runs out, the program shuts down and the user is shown a message that the time limit has been exceeded.

To use time limit, set the time_limit argument. Notice, that you need to specify the limit in milliseconds (so, by default it's 15000). If you want to disable time limit, make it zero or negative, for example -1.

TestCase(time_limit=10000)

Attach

You can also attach any object to the test case. This is an internal object, it will be available during checking inside the check method. You can put in the test input, feedback, anything that can help you to check this particular test case.

Below you can see how to attach the object to the test case.

from hstest import *


class SampleTest(StageTest):
    def generate(self) -> List[TestCase]:
        return [
            TestCase(attach=('Sample feedback', 1)),
            TestCase(attach=('Another feedback', 2))
        ]

    def check(self, reply: str, attach) -> CheckResult:
        feedback, count = attach
        if str(count) not in reply:
            return CheckResult.wrong(feedback)
        return CheckResult.correct()


if __name__ == '__main__':
    SampleTest().run_tests()

Generate method

You can generate these test cases within the generate method. It must return a list. You can see an example of using the method in the example of Attach (previous example).

Check method

The check method is used to check the user's solution. This method has 2 parameters - reply and attach. The first parameter is the output of the student program to the standard output. Below you can see examples of student programs that print data to the standard output.

print('Hello world!')
print('Second line also')

Everything that the user's program has outputted to the standard output is passed as the first argument to the check method. Therefore, in the following examples, the reply variable will be equal to "Hello world!\nSecond line also\n".

This method must return the CheckResult object. If the user's solution is incorrect, it is necessary to explain why - for example, in the example below in the check method it is checked that the user printed exactly 2 lines, and if the user not printed 2 lines, then inform the user what went wrong during checking.

from hstest import *


class SampleTest(StageTest):
    def generate(self) -> List[TestCase]:
        return [TestCase()]

    def check(self, reply: str, attach) -> CheckResult:
        if len(reply.strip().split()) == 2:
            return CheckResult.correct()
        return CheckResult.wrong("You should output exactly two lines")


if __name__ == '__main__':
    SampleTest().run_tests()

Instead of returning CheckResult object you may throw WrongAnswer or TestPassed error to indicate about the result of the test. It can be useful if you are under a lot of methods and don't want to continue checking everything else.

Since throwing any exceptions in the check method is prohibited (it indicates that something went wrong in tests and tests should be corrected), a lot of tests forced to be written like this:

... deep into function calls
    if some_parse_problem:
        raise Exception(feedback)
...

... check method
try:
    grids = Grid.parse(out)
except Exception ex:
    return CheckResult.wrong(ex.message)
...

But it is allowed to throw WrongAnswer and TestPassed errors. Now you can write this code in a more understandable way without many unnecessary try/catch constructions.

from hstest import WrongAnswer

... deep into function calls
    if some_parse_problem:
        raise WrongAnswer(feedback)
...

... check method
grids = Grid.parse(out)