agile acceptancetesting

Last modified: October 18th, 2013

Acceptance tests describe requirements in a testable format that the software must conform to. These acceptance tests are usually identified by a Product Owner or project stakeholder, but can also come from anyone with the appropriate level of product or domain knowledge.  Acceptance tests are defined at a story level and can address business rules, feature design, technical requirements, error conditions, or parafunctional testing needs such as usability and security.  These acceptance tests are used as a starting point for understanding when a unit of functionality or value is complete. Acceptance tests are also sometimes known as ‘Executable Specifications’ as they are requirements of the system that are testable.

Acceptance tests are defined before development work begins on the story. They are best defined in context and with examples, to give the team a common understanding of ‘done’.  During a sprint planning session, the team will work together to understand what tests are needed to complete a story.  It is important to note that acceptance tests do not replace the role of a testing group. A testing group will still perform more cross-cutting and in-depth testing, such as boundary conditions or error conditions. Acceptance tests simply answer the question, “What scenarios have to pass, for us as a team to feel comfortable that this work is complete?”  This set should include both happy-path and negative tests. The number of acceptance tests necessary will vary from story to story and team to team.  It is important that each team find the appropriate amount of testing needed that fits right for their situation.

While acceptance tests can be manual, there is a tremendous benefit in automating them.  There are numerous tools and frameworks that can be used for this type of testing, including Fit and FitNesse, Selenium, Cucumber, Robot Framework, Concordion, WATIR/WATIJ/WATIN, and EasyBee.  Using these frameworks, the development team implements hooks into their code that allow others to inject scenarios into the system under tests using common domain language.  Similar to the practice of Test-Driven Development (TDD), automating these tests first and implementing the software necessary to make them pass gives us a minimal code footprint, flexibility in changes and modification, and real view into the level of completeness of a deliverable. As an example, if an iteration consists of 20 points and there are a total of 100 acceptance tests defined, knowing that at the current point of our iteration we are passing 85 of those tests makes us very comfortable that we are closing in on completeness of these deliverables.  If these tests are automated, they naturally become part of an automated regression test suite after the iteration has been delivered.  The most value is recognized when these tests are run as part of the Continuous Integration build.

Automation of acceptance tests is powerful and can produce amazing time-savings and productivity gains over the long-term. However, when considering automation, do look at the area to be tested and how fragile it is. Sometimes automation is more costly (in terms of time, effort, or even false positives) than manual testing. Automating the verification of messaging around login attempts would be valuable, but investing heavy automation to make sure a complete UI is correct and usable would be very costly due to the high possibility of small changes to the location of elements in a UI.

Introduction to Acceptance Test Driven Development (Elisabeth Hendrickson)
Integrated Tests are a Scam
(JB Rainsberger)
Test Driven Development: By Example
(Kent Beck)
Automated Acceptance Tests - Theoretical or Practical
(Amr Elssamadisy for InfoQ)
Acceptance Testing: What, Why and How
Intro. to Acceptance Tests (Agile Modeling)
Acceptance Testing (Agile Alliance)