How far to test, and how to estimate

Testing is not one thing; it is done in tiers matching the units you built. Small to large, four tiers.

• Unit test … is one function or module as specified
• Integration test … does the passing between joined modules match
• System test … as a whole, including performance and exceptions, is it as specified
• Operational test (acceptance test) … from the standpoint of the people who use it, does it meet the requirements

What is interesting is that this order is exactly the reverse of the order of design. Once you have built the program, unit test; against internal design, integration test; against external design, system test; against requirements definition, operational test. Down the left and up the right, so it is called the V model.

The substance of the pairing is which phase decisions the test is confirming.

Trying to test a single module runs into trouble, because that part does not run on its own. If it assumes being called from above, it needs somebody to call it; if it is written to call below, it needs somebody to be called. But the other side does not exist yet. So you provide the missing surroundings temporarily.

Here the name of what you provide changes with the direction of integration. This is the most confused point in this lesson.

Top-down testing starts from the upper modules. The upper ones call the lower ones, so you need a stand-in for the lower module that does not exist yet. That is a stub.

Bottom-up testing starts from the lower modules. A lower one does not run unless somebody calls it, so you need a stand-in for the upper module that does not exist yet. That is a driver.

The safest way to remember is from the meaning of the names. A driver drives: it sits above and makes the thing below move. A stub is a stump or a ticket stub, the lower side that only gets called and gives a short reply. Hold on to "start at the top, you need stubs; start at the bottom, you need drivers" — you need the name of the opposite direction from where you start.

There are broadly two standpoints for choosing test data. The first is white box testing, which looks at the internals (the internal structure) of the program and decides which paths to take.

With the internals visible, you can spot gaps such as "the inside of this if has never been taken once". This is almost always what a unit test uses.

The criteria for how much has to be covered are called coverage, and the names differ by strictness.

• Statement coverage … take every statement at least once
• Decision (branch) coverage … take both true and false at every decision
• Condition coverage … make true and false for each individual condition inside a decision
• Multiple condition coverage … make every combination of the individual conditions

The further down, the more test cases and the stricter it gets.

The coverage criteria have a containment relation, and the exam asks about it often.

Take that if (a > 0 && b === 1) again. Satisfying decision coverage needs both data that comes out true and data that comes out false. Going true takes the statements inside the if, and going false takes the statements outside. So if you have satisfied decision coverage, you have automatically satisfied statement coverage.

The reverse does not hold. Statement coverage only asks that every statement be taken once, so with an if that has no else, a single piece of data that comes out true achieves it. The false side is never taken, so it falls short of decision coverage.

That is why the relation is always one way: satisfy the strict one and the loose one follows, but not the other way round.

The other standpoint is black box testing, which looks only at the specification, not at the internals. You feed in an input and see whether the result matches the specification. The functional specification and the interface specification are the grounds for the test data.

Since the internals are not seen, you cannot notice that there is extra unused code. That weakness always shows up among the choices. On the other hand, not having to rewrite the tests when the internal structure changes is a strength.

The basic way of choosing data is equivalence partitioning: gather inputs that get treated the same into one group and pick just one representative from the group.

With a decision like if (age >= 18) there are only two treatments: the refused crowd and the admitted crowd. 10, 5 and 17 all give the same result, so picking one from the group is enough.

Picking representatives such as 10 and 30, values in the middle, misses a kind of bug. The easiest thing to get wrong is writing >= where you meant >, and when that slips, exactly one value at the boundary comes out different.

So you add boundary value analysis: aim at the joins between groups and at their immediate neighbours.

For if (age >= 18) the boundary is 18, and three values are worth aiming at.

• 17 … just before the boundary. Should be refused
• 18 … the boundary itself. Should be admitted
• 19 … just past the boundary. Should be admitted

If the writer had mistakenly written age > 18, it falls over the moment you try 18. Trying only middle values would never reveal it.

When counting, do not forget that one join has two values. For a range of 1 to 100, that is 0 and 1, 100 and 101: four values.

After fixing one bug, do not stop at trying the place you fixed. If you touched a shared function, other screens that were calling it might be affected too.

Confirming that is regression testing. Its purpose is single: to check that a change has not affected places that were supposed to be unaffected.

What marks it out is that it does not try the newly built function but whether what already worked still works. It runs at every round of maintenance, so doing it by hand every time is not realistic; it is a classic candidate for automation.

Questions usually ask for the name, so train yourself to react to the phrases "a change made for maintenance" and "places that should not be affected".

Running is not the only way of finding errors. There is also reading the deliverable with human eyes or with a tool, without running it, which is called static testing. The running kind is dynamic testing.

People reading it is a review, and the ways of doing it have names.

• Walkthrough … the author explains and the participants ask questions and comment. A meeting, informally
• Inspection … a moderator takes the chair, participants look with fixed roles along a checklist, and a formal record is kept
• Round robin … the participants take the chair in turn
• Pass-around … the deliverable is distributed or circulated and comments come back

Inspection is the most formal. Where the three words role, checklist and record appear, it is this. The tool-reading kind is static analysis, detecting errors by analysing the source code.

Hunting down the cause of a discovered error and fixing it is debugging. Keep them apart: testing is the tier that finds out whether there is one, debugging is the tier that pins down where it is and fixes it.

As tools there are tracers and debuggers that stop partway and let you look at variables, and tools that record the path taken at run time.

When a team builds something, the bugs found are recorded and counted one by one. Graphing the cumulative count gives a bug management chart, and the curve takes the shape called a reliability growth curve: found fast at first, levelling off as it goes, an S shape.

Reading it has a quirk. A curve levelling off does not mean "no more will come, so relax" — testing may simply have stalled. So you look at the number of test items got through alongside it. If the items are being got through and nothing turns up, things really are settling; but if items, bugs and unresolved counts are all flat, suspect that work itself is stuck on a hard bug.

Putting a number on "how big is this" before you build is estimation. Two ways are notable.

The LOC method (program step method) estimates the number of lines to write and multiplies by the effort per line. It is plain and easy to grasp, but the line count changes with the language and the style, which makes comparison hard.

The function point method measures not lines but the number of externally visible functions. You count external inputs, external outputs, external inquiries, internal logical files and external interface files, multiply each by a weight, add them up and apply a complexity adjustment factor.

Its advantage is that you can estimate from the specification alone, before writing the program. No predicting line counts, and the value does not wobble when the language changes.

There is also the analogy method, comparing with similar past work, and the standard task method, splitting the work finely and adding it up.

Last, the machinery for not mixing up what you fixed.

Version management records changes to the source code one at a time so you can go back at any point. Git is the well-known one.

Configuration management is wider. Not only the source code but the specification, the design documents and the test specifications too: it manages which versions form one set. Deciding the procedure for making a change is also configuration management work.

When it is not going well, problems like these arise.

• Somebody fixed something without following the procedure, and what used to work stops working
• The versions of the specification, the design document and the program drift apart, so you cannot tell what to fix
• A bug fix in the original is not reflected in the derived programs

What they share is that they all come from versions not being kept together. Conversely, "testing throws up so many bugs that we cannot progress" is not a version problem, so it is not about configuration management.