niedziela, 24 listopada 2013

Finding test data

When you want to reproduce the issue reported by the customer or test new feature on any moderately complex system, you will find hard or tedious to find the right test data or test configuration. Particularly, you may ask:

  1. Is this misconfiguration common for all environments? After some changes made to the deployment script we have found some configuration parameters in database were incorrect. We started to investigate when it happened, since which release, whether this is the case for environments of type A or of type B.
  2. Where can I find environment with configuration A to reproduce this issue? One of the customers reported a defect, and we suspected it was related to his specific configuration. We didn't have time to build up a new environment with same configuration, so we started to search for a similar one among existing environments at hand.
  3. Are incoming XML documents processed same way for each configuration? We had a guess that the case that we wanted to reproduce was specific only for specific configurations but we needed to confirm that. So we searched for the output of XML processing in all available environments.
  4. What are realistic test data for test scenario X? I needed to find an example of certain flight type for which a system behave in  an expected way. It is often easier to start from finding the expected output of the system processing and then trace it back to the input (flight type) that caused that. But it was hard to say in which environment we will have enough historical data.
We usually find that our databases in regression and iteration test environments can be helpful in answering questions above. They contain historical data, specific system configurations and can be used immediately. However, when the number of environments grows, querying all databases becomes tedious. 

Solution

I came up with a simple solution. The Windows BAT script that queries multiple Oracle databases from the list for the same thing:

@echo off

set SQL=select value from parameter where name = 'SpecificParameterName';

set DB1="user1/password1@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = db.company.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = myservice)))"
set DB2="user2/password2@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = db.company.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = myservice)))"
set DB3="user3/password3@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = db.company.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = myservice)))"
set DB4="user4/password4@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = db.company.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = myservice)))"

echo %DB1%
@echo %SQL% | sqlplus -s %DB1%
echo %DB2%
@echo %SQL% | sqlplus -s %DB2%
echo %DB3%
@echo %SQL% | sqlplus -s %DB3%
echo %DB4%
@echo %SQL% | sqlplus -s %DB4%

I keep the script in my tester toolbox with a fixed list of database connections. Every time a question like the one of aboves comes, I just update SQL query and then run the script.

wtorek, 19 listopada 2013

Specification for test data

Recently, at work we've been struggling with making our tests easy to understand and independent to the configuration on which they run.

To make our discussion more focused let's assume we're testing a system processing flight bookings. The system reacts differently to oneway bookings than to roundtrips. A possible test case would create a roundtrip booking:

@BeforeMethod
public void setUp() {
  Booking b = create(booking()
    .with(itinerary()
      .with(flight().from("KRK").to("LHR").on(today().plusDays(5))
      .with(flight().from("LHR").to("KRK").on(today().plusDays(6)))
    .with(passenger("KOWALSKI/JAN)));
  ...
}

Presented way of defining test data has a number of shortcomings:

  1. Test data are hardcoded and tightly coupled with a specific environment configuration. It will run only in the configuration where flights from Cracow (KRK) to Heathrow (LHR) are available. However, it will fail when running on configuration that provides only flights in Asia. 
  2. Similarly, airlines tend to change flight schedules or offer certain flights only on selected week days. Consequently, your test may be valid today, but will start failing on Friday or in a month, when flight schedule changes.
  3. Finally, details clutters the intention of the test. It is not clear that we meant booking a roundtrip itinerary. When one will have to update the flights, for reasons given in previous two points, she may have no bloody idea what type of flights she must find.

Solution

One way to address this problem is to use Specification design pattern.

Instead of defining a specific itinerary, the booking should specify requirements for the itinerary. In our case will do this by using RountripSpecification.

@BeforeMethod
public void setUp() {
  Booking b = create(booking()
    .with(new RoundtripSpecification())
    .with(passenger("KOWALSKI/JAN)));
  ...
}

The specification declares which itinerary will satisfy it:

public interface Specification<T> {
   boolean isSatisfied(T candidate);
}

What is hidden here is the process of translating a specification into a specific itinerary that satisfies the specification. We will make ItineraryFinder class responsible for that. It will be called when creating a booking on a certain environment configuration.



Implementation details

There are two decisions to make when using Specification design pattern.

One is how to implement specification itself. Evans and Fowler, who have proposed this pattern for the first time (AFAIK), describe a number of possible strategies [1]. On one extreme we have easy to implement but not much flexible hardcoded specifications:

public class RoundtripSpecification implements Specification<Itinerary> {
   
  @Override
  public boolean isSatisfied(Itinerary candidate) {
    return candidate.size() == 2
      && candidate.getFlight(0).getOriginAirport()
         .equals(candidate.getFlight(1).getDestinationAirport());
  }
}

On a different extreme we have a whole framework of specifications that can be parametrized and combined together using logical operator. If you're interested in the latter then check the article I mention.

Another decision is how to implement ItineraryFinder class. In general this will be a class returning a candidate matching the specification.

In the simplest approach ItineraryFinder class can iterate over all candidastes and return first one matching the specification:

public Itinerary find(Specification<Itinerary> specification) {
  for(Itinerary candidate : this.getCandidates()) {
    if (specification.isSatisfiedBy(candidate)
      return candidate;
  }
  return null;
}

In this approach execution time scales badly with respect to the number of candidates and assumes that all candidates are in memory. In practice candidate itineraries are available in a database or provided by an external Web service.

Therefore, in another book [2], Evans propose combining Specification and Repository design patterns together (both are common in Domain Driver Design). The specification is responsible for retrieving a subset of candidates from the repository:

public class ItineraryFinderRepository {
  public Itinerary find(Specification<Itinerary> specification) {
    return specification.selectFrom(this);
  }
}

public class RoundtripSpecification implements Specification<Itinerary> {
   
  public Itinerary selectFrom(ItineraryFinderRepository repo) {
     Iterable<Itinerary> candidates = repository.getAllFor(currentDay());
     for(Itinerary candidate : candidates) {
       if (specification.isSatisfiedBy(candidate)
         return candidate;
     }
     return null;
  }
  ...
}


Take-away lesson

To sum up, we may observe that in tests specification enable:
  • decoupling test data requirements from finding those data
  • conveying intention of test data in clear and declarative way.

Read more

  1. Eric Evans and Martin Fowler, Specificationshttp://www.martinfowler.com/apsupp/spec.pdf 
  2. Chapter Nine. Making Implicit Concepts Explicit in Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software, 2003