Testing | Haobin Tan

Testing

Fri, 02 Dec 2022 00:00:00 +0000

Getting Started with Python Testing

Fri, 02 Dec 2022 00:00:00 +0000

Testing Your Code

Automated vs. Manual Testing

Exploratory testing

A form of testing that is done without a plan. You’re just exploring the application.
To have a complete set of manual tests, all you need to do is make a list of all the features your application has, the different types of input it can accept, and the expected results.
Every time you make a change to your code, you need to go through every single item on that list and check it. 🤯

Automated testing

Execution of your test plan (the parts of your application you want to test, the order in which you want to test them, and the expected responses) by a script instead of a human

Unit Tests vs. Integration Tests

Integration testing: testing multiple components

Major challenge: When an integration test doesn’t give the right result, it’s very hard to diagnose the issue without being able to isolate which part of the system is failing.

Unit test: checks that a single component operates in the right way → helps you to isolate what is broken in your application and fix it faster.

Choosing a Test Runner

The three most popular test runners are:

unittest
nose or nose2
pytest

`unittest`

Contains both a testing framework and a test runner
Important requirements for writing and executing tests
- You put your tests into classes as methods
- You use a series of special assertion methods in the unittest.TestCase class instead of the built-in assert statement

nose

Compatible with any tests written using the unittest framework and can be used as a drop-in replacement for the unittest test runner
If you’re starting from scratch, it is recommended that you use nose2 instead of nose.

pytest

Supports execution of unittest test cases
The real advantage of pytest comes by writing pytest test cases. ( pytest test cases are a series of functions in a Python file starting with the name test_.)
Other great features
- Support for the built-in assert statement instead of using special self.assert*() methods
- Support for filtering for test cases
- Ability to rerun from the last failing test
- An ecosystem of hundreds of plugins to extend the functionality

Choosing the best test runner for your requirements and level of experience is important.

Here we will use unittest, the Python built-in standard library.

Writing Your First Test

Where to Write the Test

You can create a folder called tests/ and split the tests into multiple files
- It is convention to ensure each file starts with test_ so all test runners will assume that Python file contains tests to be executed.
- Some very large projects split tests into more subdirectories based on their purpose or usage.

How to Structure a Simple Test

Before you dive into writing tests, you’ll want to first make a couple of decisions:

What do you want to test?
Are you writing a unit test or an integration test?

Then the structure of a test should loosely follow this workflow:

Create your inputs
Execute the code being tested, capturing the output
Compare the output with an expected result

Example

Our project folder looks like this:

project/
│
├── my_sum/
│ └── __init__.py
|
└── test.py

In my_sum/__init__.py there is a function called sum():

def sum(arg):
 """takes an iterable (a list, tuple, or set) and adds the values together"""
 total = 0
 for val in arg:
 total += val
 return total

The most simple test would be a list of integers. So, in test.py with the following Python code:

import unittest

from my_sum import sum

class TestSum(unittest.TestCase):
 def test_list_int(self):
 """
 Test that it can sum a list of integers
 """
 data = [1, 2, 3]
 result = sum(data)
 self.assertEqual(result, 6)

if __name__ == '__main__':
 # Execute the test runner by discovering all classes in this file that inherit from `unittest.TestCase`.
 unittest.main()

How to Write Assertions

Assertion: validate the output against a known response.

Some general best practices around how to write assertions:

Make sure tests are repeatable and run your test multiple times to make sure it gives the same result every time
Try and assert results that relate to your input data, such as checking that the result is the actual sum of values in the sum() example

Some of the most commonly used methods in unittest:

Method	Equivalent to
`.assertEqual(a, b)`	`a == b`
`.assertTrue(x)`	`bool(x) is True`
`.assertFalse(x)`	`bool(x) is False`
`.assertIs(a, b)`	`a is b`
`.assertIsNone(x)`	`x is None`
`.assertIn(a, b)`	`a in b`
`.assertIsInstance(a, b)`	`isinstance(a, b)`

.assertIs(), .assertIsNone(), .assertIn(), and .assertIsInstance() all have opposite methods, named .assertIsNot(), and so forth.

Side Effects

Side effects means executing a piece of code will alter other things in the environment, such as the attribute of a class, a file on the filesystem, or a value in a database.

Side effects are quiet often and are an important part of testing. Decide if the side effect is being tested before including it in your list of assertions.

If you find that the unit of code you want to test has lots of side effects, you might be breaking the Single Responsibility Principle.

Breaking the Single Responsibility Principle means the piece of code is doing too many things and would be better off being refactored.
Following the Single Responsibility Principle is a great way to design code that it is easy to write repeatable and simple unit tests for, and ultimately, reliable applications.

Executing Test Runners

The Python application that executes your test code, checks the assertions, and gives you test results in your console is called the test runner.

There are many ways to execute the unittest test runner:

When you have a single test file named test.py, simply run python test.py

Another way is using the unittest command line

$ python -m unittest test

You can provide additional options to change the output. One of those is -v for verbose

$ python -m unittest -v test
test_list_int (test.TestSum) ... ok

----------------------------------------------------------------------
Ran 1 tests in 0.000s

Instead of providing the name of a module containing tests, you can request an auto-discovery
```
$ python -m unittest discover
```
This will search the current directory for any files named test*.py and attempt to test them.
Once you have multiple test files, as long as you follow the test*.py naming pattern, you can provide the name of the directory instead by using the -s flag and the name of the directory:
```
$ python -m unittest discover -s tests
```
unittest will run all tests in a single test plan and give you the results.
If your source code is not in the directory root and contained in a subdirectory, for example in a folder called src/, you can tell unittest where to execute the tests so that it can import the modules correctly with the -t flag:
```
$ python -m unittest discover -s tests -t src
```
unittest will change to the src/ directory, scan for all test*.py files inside the the tests directory, and execute them.

More Advanced Testing Scenarios

Remember the three basic steps of every test:

Create your inputs
Execute the code, capturing the output
Compare the output with an expected result

However, it’s not always as easy as creating a static value for the input like a string or a number. Sometimes, your application will require an instance of a class or a context.

The data that you create as an input is known as a fixture. It’s common practice to create fixtures and reuse them.

If you’re running the same test and passing different values each time and expecting the same result, this is known as parameterization.

Handling Expected Failures

To test an expected error without causing the test to fail, use .assertRaises() as a context-manager, then inside the with block execute the test steps.

E.g. we test the sum() method, providing it with a bad value, such as a single integer or a string.

import unittest

from my_sum import sum


class TestSum(unittest.TestCase):
 def test_list_int(self):
 """
 Test that it can sum a list of integers
 """
 data = [1, 2, 3]
 result = sum(data)
 self.assertEqual(result, 6)

 def test_bad_type(self):
 """
 This test case will now only pass if sum(data) raises a TypeError.
 You can replace TypeError with any exception type you choose.
 """
 data = "banana"
 with self.assertRaises(TypeError):
 result = sum(data)

if __name__ == '__main__':
 unittest.main()

Isolating Behaviors in Your Application

Some simple techniques you can use to test parts of your application that have many side effects:

Refactoring code to follow the Single Responsibility Principle
Mocking out any method or function calls to remove side effects
Using integration testing instead of unit testing for this piece of the application

Writing Integration Tests

Integration testing is the testing of multiple components of the application to check that they work together. Integration testing might require acting like a consumer or user of the application by:

Calling an HTTP REST API
Calling a Python API
Calling a web service
Running a command line

Each of these types of integration tests can be written in the same way as a unit test, following the Input, Execute, and Assert pattern.

Significant difference from unit tests:

Integration tests are checking more components at once and therefore will have more side effects than a unit test
Integration tests will require more fixtures to be in place, like a database, a network socket, or a configuration file.

A good practice is to separate your unit tests and your integration tests. A simple way to separate unit and integration tests is simply to put them in different folders:

project/
│
├── my_app/
│ └── __init__.py
│
└── tests/
 |
 ├── unit/
 | ├── __init__.py
 | └── test_sum.py
 |
 └── integration/
 ├── __init__.py
 └── test_integration.py

Automating Test Execution

There are some tools for executing tests automatically when you make changes and commit them to a source-control repository like Git. Automated testing tools are often known as CI/CD tools, which stands for “Continuous Integration/Continuous Deployment.” They can run your tests, compile and publish any applications, and even deploy them into production.

Reference

Getting Started With Testing in Python

Effective Python Testing With Pytest

Fri, 02 Dec 2022 00:00:00 +0000

What Makes `pytest` So Useful?

With pytest, common tasks require less code and advanced tasks can be achieved through a variety of time-saving commands and plugins. It’ll even run your existing tests out of the box, including those written with unittest.

Less Boilerplate

Most functional tests follow the Arrange-Act-Assert model:

Arrange, or set up, the conditions for the test
Act by calling some function or method
Assert that some end condition is true

pytest simplifies this workflow by allowing you to use normal functions and Python’s assert keyword directl. For example

# test_with_pytest.py

def test_always_passes():
 assert True

def test_always_fails():
 assert False

Advantages of pytest against the Python built-in unittest:

You don’t have to deal with any imports or classes. All you need to do is include a function with the test_ prefix.
You can use the assert keyword, you don’t need to learn or remember all the different self.assert* methods in unittest
pytest provides you with a much more detailed and easy-to-read output.

Nicer Output

Run test suite using the pytest command from the top-level folder of the project:

The report shows:

The system state, including which versions of Python, pytest, and any plugins you have installed
The rootdir, or the directory to search under for configuration and tests
The number of tests the runner discovered

The output indicates the status of each test using a syntax similar to unittest:

A dot (.) means that the test passed.
An F means that the test has failed.
An E means that the test raised an unexpected exception.

For tests that fail, the report gives a detailed breakdown of the failure. This extra output can come in extremely handy when debugging.

Less to Learn

Being able to use the assert keyword is powerful, which means that there’s nothing new to learn. Writing test with pytest looks very much like normal Python functions. All of this makes the learning curve for pytest shallower than it is for unittest.

Easier to Manage State and Dependencies

Your tests will often depend on types of data or test doubles that mock objects your code is likely to encounter, such as dictionaries or JSON files.

With unittest, you might extract these dependencies into .setUp() and .tearDown() methods so that each test in the class can make use of them. However, as your test classes get larger, you may inadvertently make the test’s dependence entirely implicit. In other words, by looking at one of the many tests in isolation, you may not immediately see that it depends on something else.

pytest leads you toward explicit dependency declarations that are still reusable thanks to the availability of fixtures.

pytest fixtures are functions that can create data, test doubles, or initialize system state for the test suite.

Any test that wants to use a fixture must explicitly use this fixture function as an argument to the test function, so dependencies are always stated up front

Example

# fixture_demo.py

import pytest

@pytest.fixture
def example_fixture():
 return 1

def test_with_fixture(example_fixture):
 # You can immediately tell that it depends on a fixture, 
 # without needing to check the whole file for fixture definitions.
 assert example_fixture == 1

Fixtures can also make use of other fixtures, again by declaring them explicitly as dependencies.

Easy to Filter Tests

Name-based filtering: You can limit pytest to running only those tests whose fully qualified names match a particular expression. You can do this with the -k parameter.
Directory scoping: By default, pytest will run only those tests that are in or under the current directory.
Test categorization: pytest can include or exclude tests from particular categories that you define. You can do this with the -m parameter.

Allows Test Parametrization

pytest offers its own solution in which each test can pass or fail independently. More see: Parametrization: Combining Tests.

Plugin-Based Architecture

One of the most beautiful features of pytest is its openness to customization and new features. Almost every piece of the program can be cracked open and changed. As a result, pytest users have developed a rich ecosystem of helpful plugins.

Fixtures: Managing State and Dependencies

pytest fixtures are a way of providing data, test doubles, or state setup to your tests.
Fixtures are functions that can return a wide range of values.
Each test that depends on a fixture must explicitly accept that fixture as an argument.

When to Create Fixtures

If you find yourself writing several tests that all make use of the same underlying test data, then a fixture may be in your future. You can pull the repeated data into a single function decorated with @pytest.fixture to indicate that the function is a pytest fixture.

Example

Let’s say we write functions to process the data returned by an API endpoint. The data represents a list of people, each with a given name, family name, and job title. One function format_data_for_display() should output a list of strings that include each person’s full name (their given_name followed by their family_name), a colon, and their title. Another function format_data_for_excel() should transform the data into comma-separated values for use in Excel.

Without using fixtures, the test script looks like:

# test_format_data.py

def test_format_data_for_display():
 people = [
 {
 "given_name": "Alfonsa",
 "family_name": "Ruiz",
 "title": "Senior Software Engineer",
 },
 {
 "given_name": "Sayid",
 "family_name": "Khan",
 "title": "Project Manager",
 },
 ]

 assert format_data_for_display(people) == [
 "Alfonsa Ruiz: Senior Software Engineer",
 "Sayid Khan: Project Manager",
 ]


def test_format_data_for_excel():
 people = [
 {
 "given_name": "Alfonsa",
 "family_name": "Ruiz",
 "title": "Senior Software Engineer",
 },
 {
 "given_name": "Sayid",
 "family_name": "Khan",
 "title": "Project Manager",
 },
 ]

 assert format_data_for_excel(people) == """given,family,title
Alfonsa,Ruiz,Senior Software Engineer
Sayid,Khan,Project Manager
"""

Notably, both the tests have to repeat the definition of the people variable, which is quite a few lines of code.

With fixture:

# test_format_data.py

import pytest

@pytest.fixture
def example_people_data():
 return [
 {
 "given_name": "Alfonsa",
 "family_name": "Ruiz",
 "title": "Senior Software Engineer",
 },
 {
 "given_name": "Sayid",
 "family_name": "Khan",
 "title": "Project Manager",
 },
 ]


def test_format_data_for_display(example_people_data):
 assert format_data_for_display(example_people_data) == [
 "Alfonsa Ruiz: Senior Software Engineer",
 "Sayid Khan: Project Manager",
 ]

def test_format_data_for_excel(example_people_data):
 assert format_data_for_excel(example_people_data) == """given,family,title
Alfonsa,Ruiz,Senior Software Engineer
Sayid,Khan,Project Manager
"""

Each test is now notably shorter but still has a clear path back to the data it depends on 👏. Be sure to name your fixture something specific. That way, you can quickly determine if you want to use it when writing new tests in the future!

When to Avoid Fixtures

Fixtures are great for extracting data or objects that you use across multiple tests. However, they aren’t always as good for tests that require slight variations in the data. Littering your test suite with fixtures is no better than littering it with plain data or objects. It might even be worse because of the added layer of indirection.

As with most abstractions, it takes some practice and thought to find the right level of fixture use.

Nevertheless, fixtures will likely be an integral part of your test suite. As your project grows in scope, the challenge of scale starts to come into the picture. One of the challenges facing any kind of tool is how it handles being used at scale, and luckily, pytest has a bunch of useful features that can help you manage the complexity that comes with growth.

How to Use Fixtures at Scale

In pytest, fixtures are modular. Being modular means that fixtures can be imported, can import other modules, and they can depend on and import other fixtures. → All this allows you to compose a suitable fixture abstraction for your use case.

For example, you may find that fixtures in two separate files, or modules, share a common dependency. In this case, you can move fixtures from test modules into more general fixture-related modules. That way, you can import them back into any test modules that need them. This is a good approach when you find yourself using a fixture repeatedly throughout your project.

If you want to make a fixture available for your whole project without having to import it, a special configuration module called conftest.py will allow you to do that.

pytest looks for a conftest.py module in each directory.
If you add your general-purpose fixtures to the conftest.py module, then you’ll be able to use that fixture throughout the module’s parent directory and in any subdirectories without having to import it. → This is a great place to put your most widely used fixtures.

Another interesting use case for fixtures and conftest.py is in guarding access to resources (e.g. test suite for code that deal with API calls). pytest provides a monkeypatch fixture to replace values and behaviors, which you can use to great effect. E.g.

# conftest.py

import pytest
import requests

@pytest.fixture(autouse=True)
def disable_network_calls(monkeypatch):
 def stunted_get():
 raise RuntimeError("Network access not allowed during testing!")
 monkeypatch.setattr(requests, "get", lambda *args, **kwargs: stunted_get())

Marks: Categorizing Tests

In any large test suite, it would be nice to avoid running all the tests when you’re trying to iterate quickly on a new feature. To not run all tests, you can take advantage of markers.

You can define categories for your tests and provides options for including or excluding categories when you run your suite.
You can mark a test with any number of categories.

Marking tests is useful for categorizing tests by subsystem or dependencies.

Sometimes it can be easy to mistype or misremember the name of a mark. pytest will warn you about marks that it doesn’t recognize in the test output.

You can use the --strict-markers flag to the pytest command to ensure that all marks in your tests are registered in your pytest configuration file, pytest.ini. It’ll prevent you from running your tests until you register any unknown marks.

More see: pytest documentation.

ytest provides a few marks out of the box:

skip skips a test unconditionally.
skipif skips a test if the expression passed to it evaluates to True.
xfail indicates that a test is expected to fail, so if the test does fail, the overall suite can still result in a passing status.
parametrize creates multiple variants of a test with different values as arguments. You’ll learn more about this mark shortly.

You can see a list of all the marks that pytest knows about by running pytest --markers.

Parametrization: Combining Tests

We have seen that fixtures can be used to reduce code duplication by extracting common dependencies. Nevertheless, fixtures aren’t quite as useful when you have several tests with slightly different inputs and expected outputs.

In these cases, you can parametrize a single test definition, and pytest will create variants of the test for you with the parameters you specify.

Let’s say you’ve written a function to tell if a string is a palindrome. An initial set of tests could look like this:

def test_is_palindrome_empty_string():
 assert is_palindrome("")

def test_is_palindrome_single_character():
 assert is_palindrome("a")

def test_is_palindrome_mixed_casing():
 assert is_palindrome("Bob")

def test_is_palindrome_with_spaces():
 assert is_palindrome("Never odd or even")

def test_is_palindrome_with_punctuation():
 assert is_palindrome("Do geese see God?")

def test_is_palindrome_not_palindrome():
 assert not is_palindrome("abc")

def test_is_palindrome_not_quite():
 assert not is_palindrome("abab")

All of these tests except the last two have the same shape:

def test_is_palindrome_<in some situation>():
 assert is_palindrome("<some string>")

To get rid of the boilerplate, you can use @pytest.mark.parametrize() to fill in this shape with different values, reducing your test code significantly:

@pytest.mark.parametrize("palindrome", [
 "",
 "a",
 "Bob",
 "Never odd or even",
 "Do geese see God?",
])
def test_is_palindrome(palindrome):
 assert is_palindrome(palindrome)


@pytest.mark.parametrize("non_palindrome", [
 "abc",
 "abab",
])
def test_is_palindrome_not_palindrome(non_palindrome):
 assert not is_palindrome(non_palindrome)

The first argument to parametrize() is a comma-delimited string of parameter names. You don’t have to provide more than one name
The second argument is a list of either tuples or single values that represent the parameter value(s).

You could take your parametrization a step further to combine all your tests into one:

@pytest.mark.parametrize("maybe_palindrome, expected_result", [
 ("", True),
 ("a", True),
 ("Bob", True),
 ("Never odd or even", True),
 ("Do geese see God?", True),
 ("abc", False),
 ("abab", False),
])
def test_is_palindrome(maybe_palindrome, expected_result):
 assert is_palindrome(maybe_palindrome) == expected_result

Keep in mind that make sure you’re not parametrizing your test suite into incomprehensibility.

You can use parametrization to separate the test data from the test behavior so that it’s clear what the test is testing, and also to make the different test cases easier to read and maintain.

Durations Reports: Fighting Slow Tests

If you want to improve the speed of your tests, then it’s useful to know which tests might offer the biggest improvements. pytest can automatically record test durations for you and report the top offenders.

Use the --durations option to the pytest command to include a duration report in your test results.

--durations expects an integer value n and will report the slowest n number of tests.

Example:

(venv) $ pytest --durations=5
...
============================= slowest 5 durations =============================
3.03s call test_code.py::test_request_read_timeout
1.07s call test_code.py::test_request_connection_timeout
0.57s call test_code.py::test_database_read

(2 durations < 0.005s hidden. Use -vv to show these durations.)
=========================== short test summary info ===========================
...

Short durations are hidden by default
Each test that shows up in the durations report is a good candidate to speed up because it takes an above-average amount of the total testing time.

Useful `pytest` Plugins

You can see which other plugins are available for pytest with this extensive list of third-party plugins.

`pytest-randomly`

pytest-randomly forces your tests to run in a random order. pytest always collects all the tests it can find before running them. pytest-randomly just shuffles that list of tests before execution.

The plugin will print a seed value in the configuration description. You can use that value to run the tests in the same order as you try to fix the issue.

`pytest-cov`

If you want to measure how well your tests cover your implementation code, then you can use the coverage package. pytest-cov integrates coverage, so you can run pytest --cov to see the test coverage report and boast about it on your project front page.

`pytest-django`

pytest-django provides a handful of useful fixtures and marks for dealing with Django tests.

`pytest-bdd`

pytest can be used to run tests that fall outside the traditional scope of unit testing. Behavior-driven development (BDD) encourages writing plain-language descriptions of likely user actions and expectations, which you can then use to determine whether to implement a given feature.

Reference

Effective Python Testing With Pytest

Testing | Haobin Tan

Testing

Getting Started with Python Testing

Testing Your Code

Automated vs. Manual Testing

Unit Tests vs. Integration Tests

Choosing a Test Runner

unittest

Writing Your First Test

Where to Write the Test

How to Structure a Simple Test

Example

How to Write Assertions

Side Effects

Executing Test Runners

More Advanced Testing Scenarios

Handling Expected Failures

Isolating Behaviors in Your Application

Writing Integration Tests

Automating Test Execution

Reference

Effective Python Testing With Pytest

What Makes pytest So Useful?

Less Boilerplate

Nicer Output

Less to Learn

Easier to Manage State and Dependencies

Easy to Filter Tests

Allows Test Parametrization

Plugin-Based Architecture

Fixtures: Managing State and Dependencies

When to Create Fixtures

Example

When to Avoid Fixtures

How to Use Fixtures at Scale

Marks: Categorizing Tests

Parametrization: Combining Tests

Durations Reports: Fighting Slow Tests

Useful pytest Plugins

pytest-randomly

pytest-cov

pytest-django

pytest-bdd

Reference

`unittest`

What Makes `pytest` So Useful?

Useful `pytest` Plugins

`pytest-randomly`

`pytest-cov`

`pytest-django`

`pytest-bdd`