webtest.runner

This module provides a high-level function called get_test_runner, which creates a WebtestRunner class for executing Visual Studio .webtest files found in one or more TestSets.

Test Sets and Test Runners

In order to execute a .webtest file, you must wrap it in a TestSet, then create a WebtestRunner class via the get_test_runner function. Here is a simple example:

from webtest.runner import TestSet, get_test_runner

my_tests = [TestSet('my_test.webtest')]
TestRunner = get_test_runner(my_tests)

Each TestSet in the list will be executed by a single WebtestRunner instance. If you have more than one .webtest file to execute, you can pass all the filenames to a single TestSet:

my_tests = [TestSet('test1.webtest', 'test2.webtest', 'test3.webtest')]
TestRunner = get_test_runner(my_tests)

The above example would run three tests in sequential order, in the same WebtestRunner instance. This is recommended if there are dependencies between the tests (either they must always run in a given sequence, or there are captured variables shared between them–more on this shortly).

Another way would be to create a separate TestSet instance for each test:

my_tests = [
    TestSet('test1.webtest'),
    TestSet('test2.webtest'),
    TestSet('test3.webtest'),
]
TestRunner = get_test_runner(my_tests)

Here, each .webtest could be run in a separate WebtestRunner instance, and not necessarily in sequential order. You might take this approach if all three tests are independent, and have no need of running in sequence or sharing variables.

The TestSet might also be used for logical grouping of related tests. For example, if you have some tests for invoicing, and others for billing, you might create your TestRunner like this:

my_tests = [
    TestSet('invoice_1.webtest', 'invoice_2.webtest'),
    TestSet('billing_1.webtest', 'billing_2.webtest'),
]
TestRunner = get_test_runner(my_tests)

Here again, the two invoice tests will be run in order, in the same WebtestRunner instance, and the two billing tests will also be run in order, in the same WebtestRunner instance.

This covers the essentials of using this module; the next sections deal with how to initialize and capture variable values, and how to control the execution sequence.

Parameterization and Capturing

There are two critically important things that are often needed in web application testing: parameterization of values, and capturing of responses. These are handled by the inclusion of variables. To use variables, you must first determine which parts of your .webtest file need to be parameterized. Typically, you will parameterize the Value attribute of some element, for instance:

<FormPostParameter Name="UID" Value="wapcaplet" />
<FormPostParameter Name="PWD" Value="W0nderPet$" />

To turn these into parameters, insert an ALL_CAPS name surrounded by curly braces in place of the Value attribute’s value:

<FormPostParameter Name="UID" Value="{USERNAME}" />
<FormPostParameter Name="PWD" Value="{PASSWORD}" />

Then, define values for these variables when you call get_test_runner, by passing a dictionary using the variables keyword:

my_vars = {
    'USERNAME': 'wapcaplet',
    'PASSWORD': 'W0nderPet$',
}
TestRunner = get_test_runner(my_tests, variables=my_vars)

Variables do not have a particular type; all variables are treated as strings, and must be defined as such in the variables dictionary.

This is just one way to set variable values, and would normally be used to initialize “global” variables that are used throughout all of your .webtest files. You may also initialize “local” variables in a single .webtest file by simply assigning the ALL_CAPS variable name a literal value when it is first referenced:

<FormPostParameter Name="INVOICE_ID" Value="{INVOICE_ID = 12345}" />

Here, INVOICE_ID is set to the value 12345; any later reference to INVOICE_ID in the same .webtest file (or in other .webtest files that come later in the same TestSet) will evaluate to 12345. See the WebtestRunner.eval_expressions method below for details.

Variables can also be set to the result of a “macro”; this is useful if you need to refer to the current date (when the script runs), or for generating random alphanumeric values:

<FormPostParameter Name="INVOICE_DATE" Value="{TODAY = today(%y%m%d)}"/>
<FormPostParameter Name="INVOICE_ID" Value="{INVOICE_ID = random_digits(10)}"/>

See the webtest.macro module for details on using macros and defining custom macros.

Finally, and perhaps most importantly, if you need to set a variable’s value from one of the HTTP responses in your .webtest, you can use a capture expression. For example, you may need to capture a session ID as it comes back from the login request, so that you can use it in subsequent requests. To do that, include a <Capture>...</Capture> element somewhere in the <Request...> tag. The Capture element can appear anywhere inside the Request element, though it makes the most chronological sense to put it at the end:

<Request Method="POST" Url="http://my.site/login" ...>
    ...
    <Capture>
        <![CDATA[{SESSION_ID = <sid>(.+)</sid>}]]>
    </Capture>
</Request>

This will look for <sid>...</sid> in the response body, and set the variable SESSION_ID equal to its contents. You capture an arbitrary number of variable values in this way, then refer to them later in the .webtest file (or in subsequent .webtest files in the same TestSet). See the WebtestRunner.eval_capture method below for additional details.

Sequencing

When calling get_test_runner, you can use the sequence keyword to control how the tests are executed. Using the same example as above:

my_tests = [
    TestSet('invoice_1.webtest', 'invoice_2.webtest'),
    TestSet('billing_1.webtest', 'billing_2.webtest'),
]

The default behavior is for each WebtestRunner instance (each Grinder worker thread, that is) to run all of the tests in sequential order. This is the same as passing sequence='sequential':

TestRunner = get_test_runner(my_tests, sequence='sequential')

and would give this runtime behavior:

  • Thread 0: invoice, billing
  • Thread 1: invoice, billing
  • ...

But perhaps you want the first thread to run the invoice tests, and the second thread to run the billing tests. To do this, pass sequence='thread' to get_test_runner:

TestRunner = get_test_runner(my_tests, sequence='thread')

Now, if you run two threads, then the first thread will run the first TestSet, and the second thread will run the second TestSet. If you have more than two threads, then the extra threads will cycle through the list of available TestSets again:

  • Thread 0: invoice
  • Thread 1: billing
  • Thread 2: invoice
  • Thread 3: billing
  • ...

Another option is to use random sequencing, so that each thread will choose a random TestSet to run:

TestRunner = get_test_runner(my_tests, sequence='random')

With this, you might end up with something like:

  • Thread 0: billing
  • Thread 1: billing
  • Thread 2: invoice
  • Thread 3: billing
  • Thread 4: invoice
  • ...

Finally, it’s possible to assign a “weight” to each TestSet; this is similar to random sequencing, except that it allows you to control how often each each TestSet is run in relation to the others. For example, if you would like to run the billing tests three times as often as the invoice tests:

my_tests = [
    TestSet('invoice_1.webtest', 'invoice_2.webtest', weight=1),
    TestSet('billing_1.webtest', 'billing_2.webtest', weight=3),
]
TestRunner = get_test_runner(my_tests, sequence='weighted')

The weight of each TestSet can be any numeric value; you might use integers to obtain a relative frequency, like the example above, or you might use floating-point values to define a percentage. Here’s the above example using percentages:

my_tests = [
    TestSet('invoice_1.webtest', 'invoice_2.webtest', weight=0.25),
    TestSet('billing_1.webtest', 'billing_2.webtest', weight=0.75),
]
TestRunner = get_test_runner(my_tests, sequence='weighted')

In other words, the invoice tests will be run 25% of the time, and the billing tests 75% of the time. In this case, you might end up with the following:

  • Thread 0: billing
  • Thread 1: billing
  • Thread 2: invoice
  • Thread 3: billing
  • ...

As with random sequencing, each thread will choose a TestSet at random, with the likelihood of choosing a particular TestSet being determined by the weight. This allows you to more closely mimic a real-world distribution of activity among your various test scenarios.

Before and After

If you have test steps that must be run at the beginning of testing (such as logging into the application), and/or steps that must be run at the end (such as logging out), you can encapsulate those in TestSets and pass them using the before_set and after_set keywords. For example:

login = TestSet('login.webtest')
logout = TestSet('logout.webtest')
TestRunner = get_test_runner(my_tests, before_set=login, after_set=logout)

The before_set will be run when a WebtestRunner instance is created (normally when the Grinder worker thread starts up), and the after_set will be run when the instance is destroyed (the thread finishes execution, or is interrupted).

Classes and functions

Below is the API documentation of the classes and functions defined in this module.

webtest.runner.get_test_runner(test_sets, variables={}, before_set=None, after_set=None, sequence='sequential', think_time=500, scenario_think_time=500, verbosity='quiet', macro_class=None)

Return a TestRunner base class that runs .webtest files in the given list of TestSets. This is the primary wrapper for executing your tests.

variables
Default variables for all TestRunner instances. Each TestRunner instance will get their own copy of these, but passing them here lets you define defaults for commonly-used variables like server name, username, or password.
test_sets
A list of TestSets, where each TestSet contains one or more webtests that must be run sequentially. Each TestSet will run in a single TestRunner instance, ensuring that they can share variable values.
before_set
A TestSet that should be run when the TestRunner is initialized. Use this if all webtests need to perform the same initial steps, such as logging in.
after_set
A TestSet that should be run when the TestRunner is destroyed. Use this if all webtests need to perform the same final steps, such as logging out.
sequence

How to run the given test sets. Allowed values:

‘sequential’
Each thread runs all TestSets in order.
‘random’
Each thread runs a random TestSet for each __call__.
‘weighted’
Each thread runs a random TestSet, with those having a larger weight being run more often.
‘thread’
Thread 0 runs the first TestSet, Thread 1 runs the next, and so on. If there are fewer threads than TestSets, some TestSets will not be run. If there are more threads than TestSets, the extra threads start over at 0 again.
think_time
Time in milliseconds to sleep between each request.
scenario_think_time
Time in milliseconds to sleep between each scenario.
verbosity

How chatty to be when logging. May be:

‘debug’
Everything, including response body
‘info’
Basic info, including request parameters and evaluated expressions
‘quiet’
Minimal info, including .webtest filename and test names
‘error’
Only log errors, nothing else
macro_class
The class (not the instance) where macro functions are defined. If None, the webtest.macro.Macro class is used; pass a derived class if you want to define your own macros. See webtest.macro for how to define your own macros.
class webtest.runner.TestSet(*webtest_filenames, **kwargs)

A collection of .webtest files that are executed sequentially, with an implied dependency between them.

webtest_filenames
One or more .webtest XML filenames of tests to run together in this set

Optional keyword arguments:

weight
A numeric indicator of how often to run this TestSet. Use this with the sequence='weighted' argument to get_test_runner.
class webtest.runner.WebtestRunner(**variables)

A base class for TestRunner instances that will run TestSets.

NOTE: This class is not meant to be instantiated or overridden directly–use the get_test_runner function instead.

classmethod set_class_attributes(test_sets, before_set=None, after_set=None, sequence='sequential', think_time=500, scenario_think_time=500, verbosity='quiet', macro_class=None)

Set attributes that affect all WebtestRunner instances.

See get_test_runner for what the parameters mean.

eval_expressions(value)

Parse the given string for variables or macros, and do any necessary variable assignment. Return the string with all expressions expanded.

Allowed expressions:

{MY_VAR}
Expand to the current value of MY_VAR
{MY_VAR = literal}
Assign MY_VAR a literal value, and expand to that value
{macro_name(args)}
Expand to the result of macro_name(args)
{MY_VAR = macro_name(args)}
Assign MY_VAR the result of calling macro_name(args), and also expand to the resulting value. See the webtest.macro module for more information.

Any given value that does not match any of these forms is simply returned as-is. If you need to use literal { or } characters in a string, precede them with a backslash, like \{ or \}.

The given value may contain multiple {...} expressions, and may have additional text before or after any expression. For example, if you have previously assigned two variables FOO and BAR:

>>> eval_expressions('{FOO = Hello}')
'Hello'
>>> eval_expressions('{BAR = world}')
'world'

you can combine them in an expression like this:

>>> eval_expressions('{FOO} {BAR}!')
'Hello world!'

The only caveat is that a {...} expression may not contain another {...} expression inside it.

eval_capture(request, response)

Evaluate any Capture expressions in the given request, and set variables to matching text in the given response. Return the number of capture expressions that were successfully evaluated.

In order for this to work, you should include a Capture element inside the Request element whose response you want to capture. Each expression inside Capture should follow this format:

{VAR_NAME = regexp}

Where regexp is a regular expression with parentheses around the part you want to capture (leave out the parentheses to capture the entire match). For example, if your response contains:

... <a href="http://python.org"> ...

And you want to store the URL into a variable called HREF, do:

{HREF = <a href="([^"]+)">}

If any capture expression is not found in the response, a CaptureFailed is raised. This makes them useful for verification too–if you want to ensure that a response contains expected text, just include a capture expression that looks for it. In this case, you can leave out the parentheses, since you don’t need to capture any particular part, and if you don’t need to keep the value for later, you can just assign it to a dummy variable.

For example, to verify that the response includes a form with a “submit” button:

{VERIFY = <form.*>.*<input type="submit".*>.*</form>}

You can include multiple {VAR_NAME = regexp} expressions in the same Capture element, to capture several variables from the same response, or to do several verifications.

Since regular expressions often contain characters that are precious to XML, such as < > & and so on, you can enclose your capture expressions in a CDATA block to prevent them from being interpreted as XML:

<Request ...>
    ...
    <Capture>
        <![CDATA[
            {VAR_A = <a href="([^"]+)">}
            {VAR_B = <b>([^<]+)</b>)}
        ]]>
    </Capture>
</Request>

You can include {VAR_NAME} references in the right-hand side of the expression; for example, if you have previously assigned a value to ORDER_NUMBER, and you want to capture the contents of a div having that order number as its id, you could do:

{ORDER_DIV = <div id="{ORDER_NUMBER}">(.*)</div>}

If ORDER_NUMBER was previously set to 12345, then the above will be expanded to:

{ORDER_DIV = <div id="12345">(.*)</div>}

before matching in the response body.

Table Of Contents

Previous topic

Grinder Webtest

Next topic

webtest.macro