Use Case Levels of Test, Second Edition

Books, like software, get released with imperfections. The second edition of my book Use Case Levels of Test was just released -- in both print and kindle -- to correct defects in the first, make points I didn’t feel I properly made in the first edition, smooth out the rough edges, and add new material that didn’t make it into the first edition (scope control!).

Use Case Levels of Test, second edition

The focus of the second edition remains the same; a strategy for software test design based on the idea of use case levels of test combined with high bang for the buck ideas from software testing, quality function deployment (QFD), software reliability’s operational profiles, structured analysis and design’s C.R.U.D. matrix, and formal methods like model-based specification and discrete math for testers.

The goal of this “cross-pollination” is to provide testers with a test design strategy to

• Evaluate a set of use cases for test adequacy, determining if you are missing any essential for testing
• Budget test design time to maximize reliability and minimize testing cost
• Strike a balance between breadth of coverage of all use cases and depth of coverage for the most frequently used, critical use cases.
• Provide a step by step process for when to use the plethora of test techniques covered in so many testing books helping address the plea “Just tell me where to start!”
• Decompose the big problem of test design for a whole system or application into manageable chunks by using levels of test – not of units, modules or subsystems – but paths through the system
• Introduce innovative test design techniques not covered in other testing books; elaborate on key techniques covered only briefly in other books

Let me review here the disciplines that I see as having “cross-pollinated” to make this book, and in general touch on what I see as this book’s value added to the set of books we already have on testing.

 

Operational Profiles

In John Musa’s Amazon review of my first book (Succeeding with Use Cases) he commented: “I have always felt that there were many fruitful relationships between use cases and software reliability engineering”.

To me, operational profiles and use cases seem such a natural fit. Operational Profiles (from software reliability engineering; used in my book to help budget time in test design) have been discussed in a few other testing books but none that I’m aware discussing their integration with use case based testing as presented here, or to the depth discussed in my book. See for example QFD next.

 

QFD

Quality function deployment or QFD (from requirements engineering; used here for test prioritization) appeared in my first book and spurred a lot of interest. This book’s presentation of generating an operational profile from a  use case diagram via a QFD matrix is, as far as I’m aware, unique.

 

C.R.U.D.

The C.R.U.D. matrix (from structured analysis and design; used in my book to help with determining test adequacy of a set of use cases) has been covered in a few testing books and use case books. In Use Case Levels of Test I’ve tried to expand upon their use as described in these other books, as well as showing how to leverage an operational profile and C.R.U.D. matrix to help spot high risk data entities in the system or application.

 

Formal Methods, Discrete Math for Testers

Al Davis’ software development principle #28: “Know Formal Methods .. their use (even on the back of an envelope) can aid significantly in uncovering problems .. At least one person on every project should be comfortable with formal methods to ensure that opportunities for building quality into the product are not lost”.[1]

And why shouldn’t that “at least one person” be a tester?! [2] The formal methods community has long been concerned with test design, as indicated for example by panels such as the one I was on in 1996 asking the question “Formal methods and testing: why the state-of-the art is not the state-of-the practice”[3]

The approach discussed in Use Case Levels of Test of pairing light-weight, “back of an envelope” style, model-based specification and discrete math with use case scenario operations feels like a natural fit to me, and jives with Al Davis’ principle #28.

This topic was covered in my first book (Succeeding with Use Cases), and prompted some questions on how to expand the techniques while keeping it practical, so I’ve borrowed on and expanded on it in it in this book. I see the approach advocated in this book – selective use of these techniques on high risk operations of high risk use cases to augment use cases (but not replace! Al Davis’ principle #54[4]) - as a practical approach to helping close the gap between state-of-the art and practice in testing.

Discrete math for testers (sets, relations, Venn diagrams) has been covered by a number of testing books. They are powerful tools for testers, but getting across their practical application is in my opinion a shortcoming in many testing books. In this book I’ve tried to give them a very “Here’s how to use them” approach via lots of examples.

Prolog (Programming in Logic) came on the scene in the early 70s as a programming language popular for tackling problems in artificial intelligence like problem solving, natural language understanding (think syntax as in syntax testing), and rule-based expert systems. And the formal methods testing community recognized its potential as a tool to aid in test design. I’ve included one such example in this book, illustrating its use to sanity check that a syntax definition of an input to be used for testing actually says what we think it says, then to help write tests by acting like a “code coverage” tool, but for syntax rules.

 

A Deeper Dive on Some Commonly Discussed Testing Techniques

There are some topics covered in Use Case Levels of Test that have been covered in nearly every testing book written. For such areas I try to provide the 20/80 you need to know (so the book is fairly stand-alone) with pointers to other existing sources if you want to do further reading. But additionally, I try to provide some different angles on these topics.

For example, it’s probably the case that no other topic in testing has been written about as much as equivalence partitioning and boundary value analysis. But it’s also the case that most books use a simple numeric input to explain equivalence partitioning and boundary value analysis. So I’ve tried to add value on these well-discussed topics by taking on more complicated problems.

One example, syntax testing is a common problem in input testing, and recursive rule definitions a common way to describe many inputs. Yet few books tackle the problem of black-box test design from such recursive definitions of syntax. In this  book we’ll take on syntax testing of such inputs using the example of internet keyword search queries.

 

Use Case Levels of Test

Last but certainly not least is the idea of use case levels of test. Use cases as a basis for test design have been discussed by a number of books,  but at this writing the strategy presented here based on four levels of use case test is, as far as I’m aware, unique (for example generating an operational profile from a use case diagram; working with preconditions, postconditions and invariants at the operation level).

Use case levels of test provide the framework that hopefully helps address that plea so often uttered by testers and organizations starting to climb the testing learning curve: “Just tell me where to start!”. And augmented with operational profiles use case levels of test are key to budgeting time wisely in test design.

So, this book is the accumulation of thoughts, conference papers, white-papers, training classes and slide presentations I’ve done over the years explaining to others – as well as helping myself come to grips with – a framework built around use cases for leveraging a wealth of testing techniques, as well as techniques from other software disciplines, for innovation and a way of working smarter in software test design.

---

[1] 201 Principles of  Software Development by Alan Davis, McGraw-Hill, 1995

[2] Jorgensen, in Software Testing: A Craftsman’s Approach, argues “More than any other life cycle activity, testing lends itself to mathematical descriptions and analysis”.

[3] Formal Methods and Testing: Why the State-of-the Art is Not the State-of-the Practice, ACM SIGSOFT, Software Engineering Notes vol. 21 no 4, July 1996, p64.

[4] Principle #54: “Augment, never replace, natural language  ... In fact, one good idea is to keep the natural language and more formal specification side-by-side  ... do a manual check between the two to verify conformity..” 201 Principles of Software Development, Alan Davis.