Code testing¶

Writing automatic tests to test the code is a good practice that can save a lot of time and resources in the long run by helping to find bugs early on, i.e. as soon as the code is written. It also helps to avoid breaking a perfectly working code during code refactorization cycles.

The most basic type of an automatized test is a unit test. Unit test tests the functionality of a small isolated unit of code, typically a single function. There is always a trade-of between the complexity of a test and how large ratio of bugs can be found with it. Unit tests are usually very simple tests that help find trivial bugs (often just by running the code and checking the return value). Apart helping to uncover bugs, the process of writing unit tests itself forces one to write better code (structured into small independent functions) in the first place that can be testable by unit tests.

We use the pytest library (https://docs.pytest.org/) for the python code tests. The rest of this tutorial assumes that you are familiar with this package. Please refer to its documentation for more details.

Run existing tests¶

Make sure you have all required packages installed:

pip install -r requirements_device.txt
pip install -r requirements_server.txt
pip install pytest

Then you can run all tests using the following command when in the “plasmalab” repository directory:

pytest tests

Run in verbose mode (more information):
```
pytest -v tests
```
Run and jump into python debugger on first failed test:
```
pytest --pdb tests
```
Run tests for the device package only:
```
pytest tests/device
```

Run tests from a single test file:

pytest tests/device/test_source_meter_ki2400.py

Write tests: `device` package¶

Testing the device classes from the device package is tricky, because each class typically requires access to the actual physical device in order to run the code. Fortunately, in many cases the communication with the physical device, which is what we need to simulate, comes down to a simple sequence of text commands send to the device and text responses send back from the device.

Thus, we can simulate the whole device with one simple mock (fake) function that accepts the text commands and returns expected responses, which we know from the device manufacturer documentation. Refer to the subsections below to see how to use the existing mock functions/classes as tools for testing of the devices.

Note

Due to the simplicity of the approach, some functionality cannot be tested in this way and needs to be addressed in a different way:

Low level communication functionality
- How and where to the commands are sent, e.g. USB, VISA protocol, TCP socket
- This needs to be tested separately in the base classes that implement the communication.
Tricky hardware-specific things, e.g. time delays
- Time delays between commands are sometimes required due to slow processing capabilities of the device, slow communication speed.
- These things are entirely specific to given hardware and need to be tested directly in the lab.

VISA-based devices¶

A small number of devices, usually the most sophisticated ones, implement the standardized VISA protocol. The protocol serves as an abstraction above different connection options (RS-232 serial, GPIB, TCP/IP, etc.). We take advantage of this abstraction and focus only on sending of the so called SCPI commands.

In our case, the VISA things are hidden away in base class device.base.visabase.VisaHardwareBase and the device classes should always call its raw_scpi() method to send the SCPI commands and receive responses. Behavior of this method can be replaced with the mock helpers from tests/device/mock_visa.py as illustrated below.

Assuming we have a simple VISA-based device class:

from device import base

class MyDevice(base.VisaHardwareBase):
    def __init__(self, something):
        super().__init__("SOME-VISA-DEVICE-CODE")
        self._something = something

    @base.base_command
    def voltage(self):
        return float(self.raw_scpi(":VOLTAGE?"))

    @base.base_command
    def set_voltage(self, volts):
        if not (0 <= volts <= 120):
            raise base.CommandError("Voltage out of range!")
        self.raw_scpi(f":VOLTAGE {volts:.2f}")

We can then write tests like this:

# 1. Imports and some boilerplate

import pytest
from tests.device.mock_visa import MockVisa, set_responses, get_commands
from tests.device.utils import unwrap_hw_class

from device.base import CommandError
from device.my_device import MyDevice
unwrap_hw_class(MyDevice)  # Important! See note below for more info.

# 2. Create a fixture for convenience
#    (just creates an instance of the mock class)

@pytest.fixture
def mock_self():
    self = MockVisa()
    # For convenience, we can also "fake" here a constructor call,
    # i.e. assign some reasonable default values to the object's attributes
    self._something = "something"
    return self

# 3. The constructor test

def test_constructor():
    MyDevice("something")

# 4. The main tests

@pytest.mark.parametrize("volts", [1, 2.23, 119])
def test_voltage(mock_self, volts):
    """Test that voltage() calls the command and returns the value."""
    set_responses(mock_self, {
        ":VOLTAGE?": f"{volts:.2f}"
    })
    assert MyDevice.voltage(mock_self) == volts
    assert get_commands(mock_self) == [":VOLTAGE?"]

@pytest.mark.parametrize("volts", [1, 2.23, 119])
def test_set_voltage(mock_self, volts):
    """Test that set_voltage() calls the command."""
    MyDevice.set_voltage(mock_self, volts)
    assert get_commands(mock_self) == [":VOLTAGE {volts:.2f}"]

@pytest.mark.parametrize("volts", [-5, 120.1])
def test_set_voltage_error(mock_self, volts):
    """Test that set_voltage() checks if voltage is out of range."""
    with pytest.raises(CommandError):
        MyDevice.set_voltage(mock_self, volts)
    # Make
    assert get_commands(mock_self) == []

Note

When unit testing hardware device classes, it is important to always strip the class of all @base_command, @compound_command, etc. decorators. They add additional functionality, which we absolutely do not care for here. Thus, always call

from tests.device.utils import unwrap_hw_class
unwrap_hw_class(MyDevice)

after importing the device class.

You may have noticed that the way we use the tested class MyDevice is a bit unconventional. Apart from the constructor test, we do not create an instance. Instead we call the methods directly and provide our own fake instance of the class:

MyDevice.set_voltage(mock_self, volts)

In this expression, the mock_self object acts as a fake instance of MyDevice, while in fact it is an instance of MockVisa. We can configure it to give predefined responses to SCPI queries (using set_responses()) and we can inspect, which commands were send by the tested method (using get_commands()). In case the tested method accesses other attributes or methods of the MyDevice class apart from the raw_scpi() method, we need to mock them manually. For example, we would modify the example from above as follows:

# In the "my_device.py" file
...
class MyDevice(base.VisaHardwareBase)
    ...
    def _get_time(self):
        return time.time()
    @base.base_command
    def voltage(self):
        return [self._get_time(), float(self.raw_scpi(":VOLTAGE?"))]

# In the "test_my_device.py" file
...
@pytest.mark.parametrize("volts", [1, 2.23, 119])
def test_voltage(mock_self, volts)
    mock_self._get_time = lambda: 123.4  # Here! Fake the _get_time() method
    set_responses(mock_self, {
        ":VOLTAGE?": f"{volts:.2f}"
    })
    ...

plasmalab

Navigation

Related Topics

Code testing¶

Run existing tests¶

Write tests: `device` package¶

VISA-based devices¶

Code testing¶

Run existing tests¶

Write tests: device package¶

VISA-based devices¶

Write tests: `device` package¶