Notable changes​

Cucumber is a collection of different implementations. Each moves at the speed of its contributors and maintainers. So we start by highlight some changes from the individual projects.

Cucumber JVM​

In 2025, we've had 20 releases for Cucumber-JVM. These were mostly bug fixes and dependency updates. But I'd like to highlight a few notable changes.

• We had a contribution from Stefan Gasterstädt to make it possible to create locale sensitive parameter transformers. This looked quite complicated at the start, but the actual implementation turned out to be incredibly elegant.

• Julien Kronegg contributed several ^{1, 2, 3, 4} performance improvements.

• The Cucumber JUnit Platform Engine now supports rerun files. While the solution comes with a bunch of jankiness, it should smooth out the upgrade path from JUnit 4 to Junit 5+.

• With JUnit 4 entering maintenance mode cucumber-junit has been deprecated in favor of cucumber-junit-platform-engine.

Cucumber JS​

In 2025, we've made 8 releases for Cucumber-JS. These were mostly bug fixes, and small features, but we did make some architectural moves to enable bigger things. Some notable changes:

• Alexandru Gologan contributed support for execution sharding.
• Configuration files can be written in TypeScript.
• You can now write plugins to augment built-in behaviour

We're also working on @cucumber/node built around the Node.js test runner. It's still in the pre-1.0.0 phase, so APIs and behaviour might change. The stable canonical implementation of Cucumber for JavaScript continues to be @cucumber/cucumber for now.

Cucumber Ruby​

In 2025 Cucumber-Ruby turned Seventeen! Which means it is now old enough to drive in some parts of the world. It also saw 5 releases including v10.0.0 which lays some of the groundwork for upcoming architectural changes.

More visibly, support for Ruby 2.7 and 3.0 was removed and support for Ruby 4.0+ was added. The minimum supported Ruby version is now 3.1.

Thanks to Jérôme Lipowicz dependencies on default gems in stdlib are now explicitly declared. Backtrace filtering is now much more robust to handle a lot of the internal ruby changes, but ideally you won't see any of that.

Cucumber Messages​

Cucumber Messages is a schema to describe test results and other information from Cucumber in a new-line delimited JSON format. The first commit was done in 2018, and we are now at the point that we can dogfood our own implementation and build up the documentation and tooling through that.

For users the first results can be seen in the pretty formatter and HTML formatter which now create reports with more fidelity.

Cucumber JSON Report in maintenance-mode​

With the internal adoption of messages now in progress we have now also officially put the Cucumber JSON report in maintenance mode. The format is still supported but will not receive any changes. We don't expect this will have any significant impact, in practice the format had already been ossified.

Putting the format in maintenance mode meant that we had to pin down the exact format of the report in the form of the cucumber-json-schema. Unfortunately there been a significant drift between implementations and each flavor will need its own schema. So this is very much a work in progress, but for Cucumber-JVM at least that work is now completed.

Reqnroll​

While Reqnroll isn't under Cucumber umbrella, I would be amiss if I didn't find a way to highlight Chris Rudolphi's contributions.

Chris has been driving the adoption of Cucumber messages in Reqnroll and as part of that made numerous contributions to Cucumber. Including .NET implementations of tag-expressions, html-formatter and compatibility-kit.

Organizational stuff​

We created a governance repository to describe who we are and how we are organized and operate. This was rather boring to do but also essential to acquire funding and sponsorships.

Additionally, we have set up a security policy to let people know how to contact us in case of security issues.

Newsletter​

We're finally reviving our mailing list. If you're reading this in your email client, that means we've got it all working again.

If not, you can subscribe here.

Finances​

Cucumber is currently funded through donations from Open Collective, GitHub Sponsors, and Thanks.dev. We have also partnered with Tidelift and collect some ad revenue through Ethical Ads. In total, we received $65,545.63 with the majority coming from Tidelift.

On the expense side we spent $69.960,28 resulting in a small loss of $4,414.65 Fortunately we already built up some buffer in 2024 to absorb the loss.

The biggest expense was the maintenance and development of Cucumber. While number looks big in absolute terms, we try to compensate our maintainers fairly. So at market rates this comes down to about a cumulative 13-16 hours a week.

Sponsor Cucumber!​

The development of Cucumber is made possible by you - our users. We try to handle pull- and feature-requests, keep up to date with the latest versions, and respond to bugs and issues. And all of that in a timely manner.

Your support would enable us to do this with more focus and consistency rather than weekends and our spare time.

You can sponsor us through Open Collective or GitHub Sponsors.

Get involved​

We like to think Cucumber is a warm and friendly community where you can make friends and learn, building software that thousands of people rely on every day.

You can find us on Discord, GitHub Discussions or feed the robots on Stack Overflow.

We have a regular meeting on Thursdays at 4pm London time.

Thanks​

One year later on I'm elated to see the Cucumber project is still alive, active and developing. Thanks for being part of this community. It's an amazing privilege to be part of such a vibrant, popular open source project.

6099 days since the project was first started, 4158 days since we formed Cucumber Ltd, and 2004 days since SmartBear acquired Cucumber Ltd we have entered a new and exciting phase: Cucumber is back in community ownership.

What is happening today?​

What this means technically is that the intellectual property assets (the cucumber.io domain where this blog is hosted, the name trademarks and brand etc.) are transitioning from the ownership of SmartBear Software, into the Open Source Collective (OSC).

OSC is a 501(c)(6) non-profit registered in the USA, providing a similar service to open source projects as other foundations like the Apache Software Foundation, Linux Foundation or Software Freedom Conservancy. With control of the name and domain back in the hands of the community, we will be able to move the project forward independently, free from the commercial priorities of any single organisation.

What does this mean for me?​

Since February 2023, when SmartBear and I began exploring new paths for the project, the process introduced a period of uncertainty for those of us closest to it. We worked closely with SmartBear to navigate the legal details surrounding the name, and while that process took time, it also reflected a shared commitment to finding the best outcome for the project and its community.

If you're using one of the many open-source flavours of Cucumber like Cucumber-JVM, Cucumber-JS or Cucumber-Ruby you can now be confident knowing that the core team can focus 100% of their time on evolving those products to meet your needs.

This also means it may be time for you to take a more active role in ensuring the long-term health of the project. Yes, you! Folks like David Goss, Rien Korstanje, and Luke Hill work hard to keep Cucumber going. Here's how you can help them.

• Get involved in the community: Join our Discord to chat with other users and contributors. Even just supporting other users with their questions is a great way to contribute; it’s how I got started!

• Sponsor the project: Maintaining a thriving open-source project takes time, effort, and resources. If Cucumber has made a difference for you or your team, consider making a financial contribution through our sponsorship program. Your support will ensure we can keep Cucumber sustainable, growing, and improving for years to come.

Our financials are completely transparent. We try to pay our contributors fairly for their time, and we also need resources for hosting and other operational costs.

A note of thanks​

I want to acknowledge the support that SmartBear has provided over the years, helping Cucumber grow and reach new audiences. It was brave of them to take a chance on us back in 2019 and I'm grateful that their folks have done the work to see this through and transition stewardship back to the community. I think it demonstrates their integrity and appreciation of open source culture.

What is BDD?​

BDD is an agile development approach in which three practices are applied to each story/backlog item: Discovery, Formulation, and Automation. Much has been written about BDD and there are many good introductory articles available, but here I’d like to stress that these practices must be applied in the correct order. Discovery, then Formulation, then Automation.

https://cucumber.io/docs/bdd

In the context of BDD, Automation means the writing of test code before the code under test has been written. Just like Test-Driven Development (TDD), the failing automated test drives the development of the production code. There are two implications of this approach:

• Every test should be seen to fail when it is written and seen to pass when the correct production code is implemented. Never trust a test that you haven’t seen fail.
• The automation code must be written either by the people who will write the production code or by someone who is collaborating very closely with them.

Should automation be end-to-end?​

There’s a common misconception that all BDD scenarios will be automated end to end, exercising the entire application stack. Since each BDD scenario is only intended to specify a single business behavior, using an end-to-end approach for each one would be incredibly wasteful:

• Runtime – end-to-end tests take longer to run than tests that exercise specific areas of the code.
• Noise – the more of the code each test exercises, the more likely it is that many tests will all hit the same code paths. So, an error in that code path will cause all the tests that use it to fail, even if that part of the code has nothing to do with the business behavior the scenario was created to specify. In the face of multiple failing scenarios, it’s hard to diagnose which behavior has deviated from specification.

https://cucumber.io/blog/bdd/eviscerating-the-test-automation-pyramid/

The Test Automation Pyramid is a common metaphor that suggests most tests should not be end to end. Applying this approach to BDD automation means that we should consider the most appropriate automation to ensure that a specific behavior has been implemented as specified.

How should the UI be tested?​

BDD scenarios that describe how the UI should behave are usually written using tools such as Selenium. These can be slow and brittle because the UI is often tightly coupled with the rest of the application. However, conceptually, the UI is a component that interacts with other application components. It should therefore be possible to test the UI in isolation, near the bottom of the Test Automation Pyramid, by providing test doubles for the application components that it depends on.

Many applications are architected in such a way that the UI can only be exercised as part of an end-to-end test. Whenever possible, a more flexible architecture (that allows the UI to be exercised with the rest of the application “stubbed out”) should be preferred.

Is that all?​

Even with a flexible architecture and automation that conforms to the Test Automation Pyramid, there are challenges. Most test frameworks come with simple assertion libraries that verify if text or numerical values are set as expected. If you need to validate all the fields in a report, you will need an assertion for each of them. This approach leads to verbose automation code that is time consuming to write and difficult to maintain. Additionally, as soon as one assertion fails, the whole test fails without checking any subsequent assertions.

For many years, a technique called Approval Testing has been used in these situations, and several tools have been developed to help teams incorporate approval testing into their software development processes. The mechanics of how the tools work vary, but their approach is the same:

1. The first time the test is run, the output is checked manually. If correct, this output is stored as the “approved” output. If not, fix the software and repeat until the output produced is correct.
2. On subsequent test runs, the tool will compare the output produced to the “approved” output previously recorded. If they are found to be the same, then the approval test has passed. If not, the approval test has failed.

https://llewellynfalco.blogspot.com/2008/10/approval-tests.html

Naturally, it’s not quite as simple as that. For example, if the complex output that we’re comparing includes timestamps, these will likely be different each time the test is run. Therefore, approval testing tools typically include mechanisms to specify parts of the output that should not be compared. These exclusions will be specified when the approval test is first created and stored alongside the test.

Does approval testing work for UIs?​

Simply specifying areas of the output that should not be compared is insufficient if we’re trying to automatically verify the correctness of a visual component. Perhaps a text field is now too close to the border of a control or one visual element is overlaying/obscuring another one.

In these situations, machine learning (ML) and artificial intelligence (AI) can deliver huge benefits. Our tests can leverage these technologies to identify these hard-to-spot issues to a precision that the human eye cannot. But they take time – and slow feedback from a build is the enemy of automated testing.

Instead, AI/ML powered visual tests should be run in a separate stage in the build pipeline, after the faster automated checks have already passed. This ensures that developers get the fast feedback they require while also delivering confidence that the UI is free of visual defects.

If the visual tests pass, then all is well. If there’s a failure during the visual tests, then manual investigation is required – because not all failures indicate a defect in the code.

When is a failure not a fail?​

We normally think of a test as having a binary outcome. It either passes or fails. Life in software development is rarely that simple. To ensure that the software we ship satisfies our customers’ needs we want to minimize false positives. So, when a test passes, we need to be confident that the behavior being verified is implemented correctly.

When a test fails, it doesn’t necessarily mean that the behavior has been implemented incorrectly. In my experience there are three common situations that cause a test to fail:

1. Incorrect implementation: this could be caused by a misunderstanding of the specification or an error in the implementation.
2. Incorrect specification: the test is performing the wrong check(s) or the check(s) are being carried out incorrectly.
3. BDD/TDD: the test has been written before the behavior it’s designed to check has been implemented.

When any sort of failure happens in a build, investigation is required. If you find that Situation 1 or 2 has occurred, fix the defect (either in the implementation or the specification) and run the build again.

Situation 3) is a signal to the development team that the work is incomplete. Seeing an automated test fail is an important part of all BDD/TDD workflows. Usually, we would like the failure to be seen in the developers’ environment and made to pass before being pushed to CI. However, some workflows may see the tests committed and pushed before the behavior being verified is implemented.

VisualTest – an AI/ML powered visual approval testing tool​

There are several popular approval testing tools available (TextTest, ApprovalTests), but they lack the powerful AI/ML techniques that are needed for effective assurance of UIs and other graphical outputs. SmartBear provides a new tool that fills a gap that has long been overlooked in the full stack of UI testing - VisualTest.

https://smartbear.com/product/visualtest/

Conclusion​

The development team need regular, fast feedback to give them visibility of important aspects of their software’s quality. They need confidence that they’re developing the right thing in the right way.

BDD and TDD are techniques that give developers that confidence (among other benefits). Currently, most organisations that adopt these approaches use popular assertion-based tools to verify that the code being developed satisfies the specifications. This focus on assertion-based testing is unsuitable for some of the subtle and complex issues that occur when developing today’s applications.

Approval testing in all its flavours can help bridge the gap between automated assertion checking and time-consuming manual testing. Existing approval testing libraries are excellent when dealing with complicated textual outputs and simple graphical components. VisualTest leverages AI/ML to fill that gap and bring approval testing for modern UIs within reach of reliable automated testing.

The original version of this post was published on Medium by The BDD Advocate.

The full spectrum of Behaviour Driven Development involves collaborative activities at both a high level, e.g. discovery workshops, and at a lower level — in terms of automating the discovered use-cases.

The discovery however, is sometimes the hardest part, making sure all the use-cases of the system are known and understood by all concerned: from the product owners, who know how the product should work, to the programmers who need to build the system to these precise specifications.

There are several collaborative techniques for facilitating group activities that lead to a clearer understanding of the system, but knowing where to start is always challenging. I have found that, by collaboratively building a timeline, one is forced to gradually cover the whole scope of the system’s behaviour, and through which it becomes clear to everyone what the essential and/or most complex features of the system are.

I’ve decided to call this particular technique “Event Mapping”, in reference to two popular methods of collaboratively understanding software requirements. In both of these approaches, technical people should work alongside product people, with the aim of communicating as clearly as possible and developing a shared understanding.

The first is Event Storming, a highly collaborative technique for sketching out the timeline of a system (for either existing or future requirements) in terms of system-commands and system-events (which makes this technique very well suited to event-driven systems). Event Storming, a term coined by Alberto Brandolini, requires participants to write down system events that are of interest to the business on post-it notes and place them on a long sheet of paper that represents a timeline. This activity eventually delineates the system in terms of significant events that happen throughout its lifetime (e.g. “Order Was Dispatched”) and the commands which cause them (e.g. “Place Order for Items”).

In this way, the whole group effectively visualises what happens when and why. In this way it becomes easy to see and discuss both the logic and the order in which things are supposed to happen.

Example Mapping, another collaborative technique invented by Matt Wynne, focuses on deriving use-cases or scenarios from business rules. The output of an Example Mapping session is a set of clear-cut examples of how of a user would use the required system; the examples must be concrete enough to be translated into executable specifications (e.g. gherkin). For example, each “feature”, or “capability” of the system is focused on in turn. The business rules are clearly spelt out (for example “members are fined £5 if a book is returned late”) and these examples are teased-out collectively by the product owner and the team.

Each example is written down on sticky note and placed below the rule they illustrate. The examples should be as concrete as necessary to remove ambiguity from the rule that they are illustrating. Imagine we had a rule that says books can only be taken out for 2 days, otherwise a fine is issued. A relevant example used to illustrate this rule might be: “Jon attempts to return a book he took out 3 days ago”

From these, gherkin scenarios can then be easily written down or scenarios picked as candidates for automation.

A Gherkin scenario is basically a micro-timeline​

To recap: Using the Gherkin format of Given/When/Then for describing system use-cases is simply a way of making the three stages of a test explicit. Every test for a dynamic system has three phases:

1. Set-up the system in a known state, which it needs to be in for the use-case to be tested — i.e what needs to have happened before (2)
2. Take the action to test / Carry out the use-case on the system
3. Check that the expected outcome happened — as a result of playing out the use-case.

As a colleague of mine pointed out, after he’d taken part in a few Event Storming workshops: When you’ve got all the events and commands lined up in temporal order — it becomes trivial to write Given/When/Then scenarios.

The reason for this is of course that the sequence Event-Command-Event maps directly to Given/When/Then. Indeed, when modelling an event-driven system, things become even simpler: The implementation of the Given step are the relevant events that happen directly before the command we want to test. The When step is simply the command we give the system, and the Then step checks that the expected events were emitted as a result of executing the command. Considered in this way, scenarios are essentially micro-timelines.

In order to extract use-cases, we only need to look for the commands — plus the events that directly precede and follow it. The process for this is no different for an event-driven system than it is for a synchronous API based system.

Example use case: Library Lending System​

The “business rules” are simply the rules by which the business operates. For each rule, we need to come up with an example (how the rule is used). They can be written down at the same time as coming up with the events in the timeline. Depending on the rule, an alternate timeline might split off from the main one.

Event Mapping​

By sketching the events that happen in the system on to a downward-facing timeline, we can build up a mapping from events and commands directly to Gherkin scenarios. I’d encourage not thinking about the distinction between events and commands but rather just sketch-out “significant things” that happen over time. It will become quickly clear which are events and which are commands. The latter are, in-effect, how the user interacts with the system. For example, if the user is a library member, a command they might issue to the system would be “Return Book X”

It is helpful to then use the two colours traditionally assigned to events and commands in order to separate them visually (orange and blue, respectively). This assists greatly in identifying scenarios and extracting them into a Given/When/Then formulation. I’ve used Miro boards, which is a great tool for remote collaboration, and it’s nice for anchoring the event/command/event arrows to given/when/then (they “stick” together when moved around).

As with example mapping, the resulting scenarios each illustrate one of the rules shown above: For example, we might have

• An orange event sticky saying “Jon borrowed the book on the 21st Oct” (this could be event is dispatched on the 21st Oct.)
• A blue command sticky saying “Jon returns the book on the 28th of Oct” (the command is sent to the system on this date)
• A second orange event stick follows it saying “Jon is issued a £10 fine” (date here is irrelevant to this scenario)

This triad would together form the scenario: Returning a book late results in a £10 fine

Given: “Jon” borrowed the book at 12pm on the 21st Oct
When “Jon” returns the book at 12pm on the 28th of Oct
Then “Jon” should be issued a £10 fine

The scenarios are then excellent candidates for automation using tools such as Cucumber or Behat.

Alternately, we could consider automating the actual timeline — rather than use Gherkin. Given the Advanced API Miro-Boards provide, this is technically feasible. (Worth noting here that executable specifications don’t have to be written in Gherkin. The behavioural expectations of the system don’t have to be expressed in words, they can also be expressed in graphs or even pictorial examples — see Approval Testing tools)

As the number of scenarios grows, the ubiquitous language of the domain (see DDD) will gradually come into sharper focus. Which book was loaned? At what time? So you say that a member “reserves a book” but that the Library “puts a book on hold”?

As scenarios of a temporal nature are added, such as “What happens when a book is returned late?”, the times are added to the scenarios. It’s very useful, in such cases, to have a timeline right next to the scenarios themselves.
The timeline should be detailed enough to contain the precise times of borrowing and returning, and the scenarios should merely have to copy details of those specific interactions.

For alternate timelines (Jane returns the book late) I draw a parallel timeline which splits off from the main timeline. Events that happen on this timeline effectively become separate scenarios (e.g. the one where Jane is fined).

Finally, we can use different ways to mark or identify different scenarios. Here’s a screenshot from a session I ran at the last Cukenfest. I later identified the use-cases that can be mapped to scenarios and marked them using different coloured rectangles.

Feature flags are transforming the way we develop software. These sections of code govern the execution of a specific software feature, allowing that feature to be toggled on and off without a new deployment. Every tool we use to develop, test, and measure product changes must adapt.

Because of recent advances, feature flags allow releases to be more stable than ever. More teams are testing in production, as they are now able to roll back feature changes with the flip of a switch. Even with these advances, however, we still need to deploy best practices when testing our production changes, to protect users from a poor experience.

We need to bring testing to the forefront. By testing the multiple states of a feature flag directly, we can ensure the application will function properly as flags are toggled. In 2017, we announced an enhancement to our Java client which allows you to programmatically change the treatments returned by the Split client. As a result, you can start a test by setting the specific feature flag and treatments you wish to test.

Today, we’re partnering with SmartBear to enhance those capabilities to work directly with Cucumber.

How it works​

Our focus is to let teams toggle flags as part of their automated acceptance testing process. With Cucumber-Split, you can toggle feature flags declaratively with tags, then verify the software works as expected.

Imagine we're building firmware for a coffee machine, and can deploy the new firmware over the internet with continuous deployment. The coffee machine is currently able to serve espresso and caffè lattes. The team is working on an experimental new feature to serve flat whites as well, available only to a small set of customers.

We introduce a feature flag to control whether flat whites are available. Only a few customers will have flat white functionality, but we want to test to make sure it works as expected.

Two scenarios verify options in the drink selection menu: one where flat white is not available (the feature flag is off), and another where it is (the feature flag is on).

Feature: Make Coffee

  @split[flat-white:off]
  Scenario: Display available drinks
    Given the machine is not empty
    Then the following drinks should be available:
      | name          | price |
      | espresso      |  1.90 |
      | caffe latte   |  2.30 |

  @split[flat-white:on]
  Scenario: Display available drinks (including the new experimental flat white)
    Given the machine is not empty
    Then the following drinks should be available:
      | name          | price |
      | espresso      |  1.90 |
      | flat white    |  2.10 |
      | caffe latte   |  2.30 |

When we run these two scenarios with Cucumber, we gain confidence that the flat white feature flag works as expected, both when it’s on and when it’s off.

To learn more, and read a step-by-step installation guide, follow the detailed instructions in the Cucumber-Split GitHub repo.

Beyond testing solutions, the tools we use need to continue to evolve to be feature aware​

New, more nuanced solutions allow us to keep work moving with high quality.

• Work management solutions: Need to associate work to be done to the feature flag gating the work.
• Deployment solutions: Need to automate the configuration of feature flags as part of your deployment process.
• Application-monitoring solutions: Need to see feature flag context to better understand transaction level data to answer the simple question: “Do users seeing the new feature have higher page load times?”
• Error-monitoring solutions: Need to see feature flag context to better understand error data to answer the simple question: “Do users seeing the new feature have an increased number of errors?”
• Product-analytics solutions: Require feature flag context to understand the impact of new features on the product experience and the business. “Do users seeing the new feature engage at higher rates?”

Looking forward​

As feature flags increase in adoption, our tools need to adapt and accommodate our evolving development practices.

The feature delivery lifecycle is like a flywheel for being intentional, moving fast in a safe way, and being keenly aware of the impact your work has on end users. Trunk-based development and the demands of the mobile-app ecosystem mean teams rely on feature flags to maintain speed, safety, and agility.

Make sure you stay feature flag aware.

We’ve merged similar questions together, so what you see below is a summary of the outstanding questions.

Where can I learn more?​

We have published two BDD books (Discovery & Formulation) that are available from Leanpub (multiple ebook formats) and Amazon (print & Kindle). Visit http://bddbooks.com for full details.

I have previously blogged an excerpt from the Formulation book, describing the BRIEF acronym. https://cucumber.io/blog/bdd/keep-your-scenarios-brief/

The Cucumber open source community maintains documentation online. A good place to start is https://cucumber.io/docs/bdd

SmartBear provides free training videos that focus on using Cucumber (and Specflow) for C#, Java, JavaScript, and Ruby developers. https://school.cucumber.io

If you want to play around with Cucumber or Specflow, but don’t want to install the software on your machine, then head over to Cyber-Dojo, where you can experiment with whichever version of Cucumber that interests.

I have a question about the scenario you presented…​

In the webinar we formulated a scenario from sample software system being developed for a public library. You can watch the webinar online, but here’s the final scenario for reference:

Feature:
  Rule: Members pay reservation $1 per item
    Scenario: Library member reserves a book
      Given Andrew is a library member
      When he reserves one book
      Given he should be charged $1

There were a couple of questions specifically about the structure of this scenario.

“Are Given, When and Then the only keywords that introduce steps in a scenario?”​

• Given is used to set the context of a scenario
• When is used to describe the action that should cause the behaviour being illustrated to take place
• Then is used to describe the expected outcome.

Gherkin provides two other keywords that can be used to introduce steps: And and But. If you use one of these conjunctions, the implication is that the step is in the scope of the preceding Given / When / Then. So, if a Given is followed by an And, it means that the And step is intended to contribute to the context of the scenario.

“Can we pass the total quantity to be reserved?”​

Passing parameters is specifically an automation question, and the answer is “YES!”. You can find more details on parameter passing in the Cucumber Expression documentation.

Why doesn’t your scenario contain more detail?​

The ‘E’ in our BRIEF acronym stands for essential. Every piece of information in a scenario should be essential to understanding the system’s behaviour with respect to the Rule being illustrated and the Scenario being described. The running system will require plenty of other details to actually run, but the scenario should contain only information that is essential to understanding the specific behaviour being illustrated.

Details that aren’t essential for one scenario will be essential for others. It’s hard to see when we’re only looking at a single scenario in isolation, but becomes clearer if you imagine what a complete set of scenarios will look like.

That still leaves the question of how you can illustrate a behaviour in the middle of a customer journey without including Given steps that cover all the actions that a user would have to go through to get the system into the necessary state. The answer is to find a way of describing the journey to that state in a single intention revealing way.This is what the statement, the statement “Given Andrew is a library member” does in the sample scenario. It could be expanded with all the steps that Andrew had to go through to join the library, but that would be confusing. Instead, we’ve found a way that describes the intent of all those inessential steps that we’ve omitted.

An additional benefit of this approach to omitting incidental detail is that it gives our developers freedom to choose how to automate the scenario. They could automate this step by manipulating the system’s UI and going through the full sign up process. Or they could search the member database to find a suitable user to use in the scenario. Or the automation could directly insert a new library member into the database at the beginning of the scenario and delete it at the end. The choice is not important from a behavioural viewpoint – all of them would illustrate that reserving a book will cost Andrew $1.00 – but it does mean that we can build an automation suite that conforms to the test automation pyramid, robustly giving us fast feedback and confidence in our application.

Shouldn’t scenarios cover a full user journey?​

The scenarios that we.ve been looking at so far focus on illustrating a single behaviour. This is what we call an illustrative scenario. There’s also a need for scenarios that cover complete user interactions, which are called journey scenarios. We didn’t speak about these in the webinar, but they are covered in our Formulation book.

How can I use Gherkin to describe asynchronous behaviours?​

Asynchrony is always hard to reason about. Nonetheless, Gherkin scenarios that follow the BRIEF principles are perfectly suitable for illustrating single aspects of asynchronous behaviours. There’s no need to always document each step of the interactions in every scenario. As described in the previous answer, try to describe the context using a single, intention revealing statement.

Shouldn’t steps be atomic?​

It’s really important that each scenario should be isolated from every other scenario. This means that scenarios should be able to run in any order and still get the same result. All flavours of Cucumber go to great lengths to help enforce this, but automation engineers need to be aware of this necessity, because there are numerous ways that they can inadvertently break this important rule.

On the other hand, it’s frequently necessary for data to flow between steps within a scenario. Again, all flavours Cucumber provide mechanisms to facilitate this sharing of information – although these mechanisms vary between programming languages. You can find more details in the open source documentation.

Is this a “good” scenario?​

Given the url is input in the browser
When the user presses “enter”
Then the Login page opens

Without knowing what rule this scenario is supposed to illustrate, it’s hard to be certain whether it has been written in an appropriate way. On first inspection however, I suspect that it goes against the recommendation for scenarios to be intention revealing. The scenario as written seems to be very mechanical, talking about browsers, keypresses (“enter”), and pages.

What is the user trying to do? What behaviour are we trying to illustrate?

Even if you can convince yourself that this isn’t a problem, this scenario doesn’t include appropriate real data. What URL is the user navigating to?

So, its more than likely that this will not be a useful scenario that helps to define valuable behaviour from a business perspective.

Isn’t BDD just an expensive way of writing automated tests?​

As we mentioned during the webinar, we believe that there is little value to be gained from using Cucumber (and Gherkin) to write automated test scripts. You can use them in this way, but the resulting scripts are often brittle, deliver little documentation value to the team as a whole, and thus are costly to maintain.

BDD is a collaborative approach to developing software that leverages the different perspectives of team members to fully understand the requirements underpinning a story before any code or tests are written. This shared understanding is captured using business readable terminology that will then act as an unambiguous specification of the system’s expected behaviour.

Additionally, once automated, a failing scenario will guide the developers to write just enough code to get the scenario to pass (and by inference, exactly the behaviour that was specified). Once passing, the scenarios are described as living documentation, which can be relied upon to be correct, because they are validated against the actual system every time it is built.

It is within the context of these multiple benefits that the additional investment of writing Gherkin scenarios contributes a significant return on investment (ROI). If you’re simply using Cucumber/Gherkin to turn a manual test script into an automated test script, then your ROI will be minimal.

I’m already busy. How can I fit BDD in on top of everything else?​

Adopting BDD requires up front investment for long term savings. Stories should be refined (using example mapping) before they are pulled into a sprint. That’s because, before you have a detailed, shared understanding of what the story really entails, it’s impossible to know how large it actually is.

Using the automated scenarios to guide the development means that developers know exactly what they need to deliver, so unnecessary rework is minimised. The growing suite of automated scenarios act become a valuable regression suite, allowing faster releases and reduced customer defects. The trustworthy living documentation that this produces means that, when the team is asked to modify or enhance the product, they don’t need to waste time reverse engineering the system’s capabilities.

Adopting BDD takes time, planning, and organisational support. It requires a change of working practices to succeed. But the payoff is massive – in development costs, software quality, employee morale, and customer satisfaction.

Can BDD be used for APIs?​

In a word, “Yes”. There are different considerations to API design and documentation – the customer is much more likely to be another developer rather than a business person – but the principles of discovery, formulation, and automation still apply.

You can see an example of API design and implementation using BDD, Cucumber, and SwaggerHub in this conference session from Connect 2020.

Can I use Selenium when developing using a BDD approach?​

BDD and Cucumber don’t force you to use specific automation tools or libraries, so you can certainly use Selnium. However, remember the advice of the test automation pyramid. To paraphrase – build your test strategy on a foundation of small, fast, reliable automated tests. Using Selenium tends to produce end to end tests, at the top of the pyramid – and we want as few of these as we can get away with, because they are typically big, slow, and flaky.

Just because we write the scenarios using business readable language doesn’t mean that the automation can’t be implemented lower down the pyramid. I blogged about this some time ago.

Should I use Cucumber or TestNG?​

If you’re wanting to write automated tests, then a testing tool is what you need. In that case TestNG (or similar) is exactly what you need.

If you want to adopt BDD as a way of increasing collaboration, increasing quality, and reducing misunderstandings, then you need a collaboration tool that enables your business colleagues to participate directly. This is the use case that Cucumber and Gherkin have been designed for. Teams have used Cucumber simply as a test tool, but we do not recommend this, because we see little return on investment from using in this way.

Why does this matter?​

In 2017, GitHub ran a survey of 5,500 of its users. The data shows that people of colour and women are disproportionately not contributing to open source:

Only three percent identified as female and one percent as non-binary. According to Bureau of Labor Statistics, about 22.6 percent of professional computer programmers are female.

About 16 percent of respondents said they belonged to ethnic or national groups that are in the minority in the country they live in. Black, Asian, and Latino programmers account for a total of about 34 percent of programmers in the US, according to the bureau.

(Source: Wired.com)

That’s a massive number of people who are missing from our open-source communities.

In purely objective terms, this presents a huge opportunity for open source projects that are willing to put in the work to make their projects more welcoming. There’s an untapped pool of talent here, and projects that seek and respond to feedback from under-represented minorities will benefit from having more people making more contributions.

I also think there’s a moral argument. Reni Eddo-Lodge describes structural racism as:

Structural racism is dozens, or hundreds, or thousands of people with the same biases joining together to make up one organisation, and acting accordingly. Structural racism is an impenetrably white workplace culture set by those people, where anyone who falls outside the culture must conform or face failure. “Structural” is often the only way to describe what goes unnoticed – the silently raised eyebrows, the implicit biases, snap judgments made on assumptions of competency.

On that basis, it’s fair to say that open-source software has a problem with structural racism and, if you swap “white” for “male” in that quote, structural sexism too. The reason we have so few women and people of colour in open source is because of things that we white men are doing, deliberately or not, that makes this an unwelcoming environment for those people.

Structural racism isn’t the responsibility of people of colour to fix. As the people who have enjoyed the privilege of being white and male all our lives, we’re the ones with the power to change the structure. It’s up to us to make it work for everyone, not just ourselves.

Open Source is intimidating​

After this initial tweet, I embarked on a listening exercise, going to diversity meetups and having virtual coffees with people to hear about their experiences contributing to open source, about what had put them off. I took some notes in this Miro board.

A common theme was the feeling of intimidation. The research showing that women's contributions are more likely to be scrutinised than men's. The knowledge from experience that the expectations for minorities are higher. The risk that my livelihood might even be in jeopardy if I am criticized in public.

I learned about gatekeeping, the way that even well-meaning white folks will instinctively make all kinds of passive-aggressive moves to keep people who aren't like them out of their spaces, so they don't have to feel uncomfortable.

Then there is the feeling like an imposter, that everyone else is so much more experienced that you, so competent. I remember this feeling too from when I took my baby steps in open source many years ago. I can only imagine how much this is amplified by being from a less privileged background, with years of bitter experience dealing with microagressions. Finding the right community where you fit in and will be welcomed is super important.

There's not knowing where to start: Cloning an open-source codebase and wandering around it can be like turning up in a new city. Hopefully there are at least some signposts, but you're going to be lucky to find a decent map, let alone a tour guide.

Very often (and I know this has been the case for Cucumber) so called "good first issues" are anything but "good". They still require a lot of context to be able to figure out and hack on. Even if the maintainers would be happy to answer questions if people asked, that might not be obvious.

The value of time also came up. A kid in college whose parents are wealthy can afford to spend their evenings contributing to open source for free. Someone less privileged might be busy flipping burgers to cover their tuition fees.

The impasse​

All in all, I realised we're at an impasse: As open-source maintainers, we're never going to get feedback about how to be more welcoming, because the very nature of the problem is that people are too intimidated to show up, let alone give us that feedback.

From the inside of a project, with the curse of knowledge it's hard to know what needs to be explained or documented, what context is missing from a ticket.

I know that these feelings of intimidation or imposter syndrome are not limited to people from minority backgrounds. Opening up this feedback loop and starting to resolve these issues is going to benefit everyone.

So where do we start?

New contributors ensemble​

It’s early days for this initiative. Our fledgeling strategy is to focus on the experience of new contributors, to open channels for feedback as well as providing coaching and mentoring for anyone who is curious about taking their first steps in open source.

One of the things we’re trying that feels right is the new contributors ensemble.

A group of willing volunteers who are keen to take their first steps in open source, and don’t look like me, are meeting on Fridays for an ensemble/mob programming session. We’ll be working together to fix one of Cucumber-JS’s good first issues, streaming live on https://www.twitch.tv/cucumberbdd from 16:30 UTC.

Like a usability test, this is a great way for me to see, in real-time, what gets in the way of a newcomer becoming a contributor. We had one a couple of weeks ago which everyone thoroughly enjoyed, but I forgot to hit the record button so you'll have to take my word for it! I'll try and be better set up this time.

We’d like to make this a regular event, with lots of people showing up to learn and teach each other about contributing to open source projects, not necessarily Cucumber!

Watch the stream to pick up some tips for yourself about how to get started in open source or, if you’re interested in taking part, get in touch. The easiest way for us would be if you join the Cucumber Community Slack and find us in the #new-contributors channel, but you can also ping me on Twitter, my DMs are open.

If you have any other ideas or experiences to share about this topic, I'd love to hear from you.

1. Why should automation be done by the dev team?
2. Isn’t the business-readable documentation just extra overhead?
3. What’s wrong with changing the scenarios to enable automation?
4. Can all testing be automated?
5. How can Cucumber help us understand the root causes of failure?

The question​

Specific about Cucumber reporting. I find that sometimes a single bug can break several Given-When-Then scenarios, which is great to measure business impact, but not great to understand root-causes/bugs. Any ideas on this? We ended up creating a complementary root-cause report...?

This is a very interesting question that requires us to think about the purpose of our tests. You’ll probably be familiar with the popular test characterisations of unit, integration, and end-to-end – as well as many more.

Do we need unit tests AND end-to-end tests? If so, what different concerns do they address? If not, then which should we prefer? In answering this question, I’ll refer to two metaphors (one well known, the other less so) and an acronym.

Test automation pyramid​

The test automation pyramid has been widely discussed in the industry for many years.

I wrote a lengthy blog post on the subject a few months ago, which I will not repeat here in its entirety, but there is a section that directly addresses the question:

[Tests at the bottom of the pyramid] should exercise as little of the application as they can. These tests should document the behaviour and validate the correctness of small amounts of code. As we move up the [pyramid], we will write fewer tests, but they will exercise progressively more of the application. At the same time, the intent of the tests changes to documenting the interaction between components, the messages they pass between each other, and the protocols that they use. This includes behaviours such as validation and error handling.

When we reach the top of the [pyramid] we are exercising the whole application. We should already have confidence that each component behaves as intended. We should also have confidence that each interaction between components has been exercised to verify that both ends of the dependency have the same understanding of the protocol that they share. All that remains is to verify that “the system hangs together.” These are sometimes characterised as smoke tests, and they check that the application has been deployed successfully and correctly configured for that specific environment.

The foundation of your automated testing strategy is the small, fast, reliable tests at the bottom of the pyramid. If there is a defect in the code, then it should be caught by one of these tests. Since each test at the bottom of the pyramid is validating the “correctness of a small amount of code”, we expect only a single test to fail for any given defect.

Many high-level tests might depend on the code that contains the defect, but we would expect the build to fail and terminate before running these high-level tests. The foundation has been found to be defective – so there is no point running tests that depend on the foundation being sound.

Test automation iceberg​

If the low-level, programmer tests fail, then it is almost never worth invoking Cucumber, but this doesn’t necessarily fix the problem encountered by our questioner. In another blog post, I document an insight that my colleague Matt Wynne shared with me some years ago, which he called the test automation iceberg. The one-line summary of that post is that:

Not all Business-readable tests need to be End-to-end and not all End-to-end tests need to be business readable

So, Cucumber scenarios may be implemented to run anywhere in the test automation pyramid. The use of business-readable scenarios tells us only that the behaviour is of interest to the business – it says nothing about how the test itself is automated.

BRIEF​

In the second volume of the BDD Books series, Formulation, Gáspár Nagy and I introduce the BRIEF acronym. The last letter in the acronym, “F”, stands for “focused” which we describe in this way:

Focused: Most scenarios should be focused on illustrating a single rule.

What this means is that we should attempt to implement our scenario automation as far down the pyramid as possible, to ensure that we are validating a single behaviour. If there’s a defect in the implementation of that behaviour then only one (or a few related) scenarios should fail. This gets to the heart of the original question.

In Formulation, we differentiate between illustrative scenarios (that are focused on a single rule) and journey scenarios (that document a flow through several/many behaviours of the system). If illustrative and journey scenarios are run at the same time, then a defect in the implementation of a single behaviour may cause the failure of many journey scenarios. Cucumber provides several mechanisms for separating your scenarios into discrete suites that can be executed independently (see the free training video BDD with Cucumber, chapter 6 for some suggestions). Aslak Hellesøy has also described how a hexagonal architecture can be leveraged to implement illustrative scenarios at several locations within the pyramid.

Conclusion​

The foundation of an automated testing strategy should be built from the small, fast, reliable programmer tests that are popularly called unit tests. When there’s a defect in the code, you should expect one (or perhaps a few) of these tests to fail – not many. If any of these tests fail, then the foundation is unsound, and it is not worth running the higher level tests that are built on top of this broken foundation.

If a single defect is causing many unrelated tests to fail, then it is likely that you are either:

• running tests higher up the pyramid, even though lower-level tests have already failed, or
• you are missing the necessary lower-level tests that validate the behaviour in which the defect is located.

In summary, Cucumber is a tool designed to support BDD. It will parse business-readable specifications and, in response to each statement, execute relevant automation code that your team has written. To minimise the need for costly root-cause analysis when a defect is introduced, follow the test automation pyramid metaphor and the BRIEF acronym.

Post-script​

I salute the original questioner for their implementation of a root-cause report but suggest that it might have been better to have instead written sufficient lower-level checks to catch these defects. Whether the checks should be implemented directly in the programming language of the application (programmer tests) or using business domain terminology (Cucumber scenarios) should depend on whether the business is sufficiently interested in the behaviour that the check documents.

Further, I compliment the original questioner for the correct usage of the often-misspelled word complementary.

Seb Rose and Gáspár Nagy are the authors of The BDD Books. Cucumber Community Manager Tracey S. Rosenberg spoke to them about their books, the writing process, and what comes next.

Tracey: How did you decide, “Let us write books together about BDD?” Where did this all come from? What’s the origin story?​

Seb: At some point, maybe five years ago, maybe longer, Gáspár was thinking about a SpecFlow book and got in touch with Cucumber. The rest of the team said, “Seb, you could work with Gáspár on the SpecFlow book!” And I said, “That'd be wonderful.”

Gáspár: It was at least six years ago, definitely. When we were less busy ⁠–⁠ or we thought we were less busy! ⁠–⁠ we started to shape the form of the SpecFlow book. We wanted to make a new structure emphasizing all the goodness experienced by the BDD community regarding the BDD concept, and of course also how SpecFlow works. Discovery and formulation are not tool-specific, so we knew they shouldn’t go into a SpecFlow book. Also, discovery and formulation are separate activities. Because of this, we decided to write a separate book for each BDD practice.

Seb: We also realized that one big book was going to take one hell of a long time. So if we wrote a small book, we’d be able to get it done more quickly.

Tracey: Did you collaborate on all the different parts of every book, or did one of you say, you put an outline together and I'll do the full write-up?​

Seb: My time was in short supply when we were doing the first book, so I said, Gáspár, you write it and I’ll review it. We reversed that for the second book. Gáspár will write the third book, which is about SpecFlow. We have a fourth planned as well ⁠–⁠ in the style of 97 Things Every Architect Should Know, we’ll seek contributions from the community and edit them into a book of styles/patterns.

Tracey: And how does your process work?​

Gáspár: Almost every week for the last few years, we’ve spent at least an hour together and discussed what was happening. Then we try to make progress on the book, so it’s good to have allocated time to work together. We realized it didn’t make sense to write the book together, so that's why we use this “one writes, the other reviews” scheme. But we also need to discuss these things. Even more importantly, we need the inspiration to move on. Book writing is not easily motivational, because it takes a long time.

Tracey: Oh yes it does.

Gáspár: And even after that, you don't really see your readers, or maybe you see a few of them. So one of the important bits was that during these sessions, we were asking each other, how much have you finished? If you do that a couple of times, you feel that now, this week, I should really do something.

Seb: There's definitely guilt. You go to a meeting on a Friday and you think, “I haven't done what I said I was going to do last Friday.” But that’s life.

Gáspár: We're all equal from this perspective.

Seb: In terms of logistics, Gáspár does all the technical work of putting the books together and getting them onto Kindle, after I’ve turned all the text into something close to English. I do the marketing.

Gáspár: For the cover design, I put a draft together, and we were thinking about replacing it with something more professional, but then we decided it wasn’t necessary. For the second book, we used the same concept as the first ⁠–⁠ basically, we have geometrical colours in the background, and an antique which shows the beauty of engineering. On the first one we had a compass, and then for this one, we have a very nice pen. I don't know what will happen with the third book. I'll figure it out.

Seb: I think we can get one of those, you know, where they used to make solar systems that you wound up and turned round. That's a few years off, though!

Tracey: How has the reception been? Not just raw sales, but comments, or praise, or “why didn't you do this?”​

Gáspár: For the second book, it was easier, maybe because we had a little bit of history, or maybe we were handling the review process better. Generally the review comments were positive, and in general, we got nice feedback from many people. I think our praise quote list is two pages long! It was clear these people were really checking out the book, not just writing a few sentences because we asked them. We’ve also had good feedback from the readers.

Seb: When we wrote Discovery, we pinged Daniel Terhorst-North and asked him to either review the book or give us feedback. His response was, “Sure, I’ll help you promote it on Twitter.” So it's like, okay, that's nice. This time we asked him for a praise quote, a sentence or two, and he wrote back and sent us a foreword! So if Dan feels like it's worth him writing a foreword, when he hasn't even been asked for one, that's good feedback. I was well happy with that.

Tracey: High praise indeed. So what makes this book worth reading, if you’re trying to improve your BDD skills?​

Gáspár: We didn't want a book which says, “This is the way you should do it, and if you don't do it that way, you’re doing it wrong.” That kind of approach is far from how Seb and I work. But it’s not easy to show the real trade-offs that teams have to choose between.

In both books, we have a team ⁠–⁠ they’re working on an imaginary product called Where is my Pizza? ⁠–⁠ and this team is trying to solve concrete problems. They have options, they debate, and sometimes they say, well, we can't even make a decision about that.

The imaginary conversations crystallised our understanding of the doubts that teams experience. Sometimes we include extra material to cover some areas in more detail. If we can imagine a real team having these sorts of discussions, it probably makes sense to the readers, or at least gets a little bit closer to the kind of discussions teams have.

It was funny, because over the six years we were writing these books, the characters began living their own lives. Sometimes I imagine what Dave would say, or what Tracey requires ⁠–⁠ we have a Tracey as well, by the way.

Tracey: Fantastic!

Gáspár: Maybe at some point we’ll do a dramatization of the book.

Seb: There’s an audiobook of Discovery.

Gáspár: It’s also worth reading because formulation is a very practical topic. Everyone who deals with BDD needs to sit down and write the “given, when, then” sentences. I remember my own feelings, ten or eleven years ago, where we were not prepared for that ⁠–⁠ we just came together in a meeting and wrote the first scenario. We started with "given” and then said, “uh oh, wait a second, ah, how should this go?” I think that's a situation many people face. Obviously the discovery part is important, but to make the connection to discovery, formulation is the immediate need. People have to understand how to write down and create these scenarios. Hopefully the book gives enough guidance for that. Also, it shows that when you put more of the discovery practice into your process, it becomes even easier.

Seb: There's been quite a lot of development in the BDD field over the time that we’ve been writing this book. This idea of discovery, formulation, and automation comes out of writing these books. It was inspired by the work of Liz Keogh, but Gáspár came up with the term “formulation”, and I suggested we divide the SpecFlow book into three books, aligned with discovery, formulation, and automation. That fed into the diagram Matt Wynne eventually created for the BDD documentation on the cucumber.io site.

Discovery, Formulation, and Automation

Seb: These books don't simply document what we've done, or what other people have done ⁠–⁠ they’re part of the evolution of BDD. Gáspár came up with the term “formulation” around six years ago, and no one had used it in this way before. In fact, when we called the great and the good of the BDD world together, there was a lot of pushback against the term. People wanted to call it specification or whatever. We said, fine, everyone do what they want, but we carried on calling it formulation, and that's the word that stuck. So yeah, go us! Go Gáspár!

Gáspár: To be precise, I didn’t come up with the word. I don't know who deserves credit for it. I did bring it into this community, certainly. I think it’s important that we have names for things, and that some names are unique. English words, obviously any kind of words, can be used in multiple different forms, so there is no one single meaning of a word. You could use specification, or you could use documentation ⁠–⁠ there are many words that would be fitting ⁠–⁠ but when you use a word that’s a little bit special, that reduces the confusion. If I say “specification” or “documentation,” those words are overused. “Formulation” is special, and in the BDD context, everyone knows what we’re talking about.

Discovery and Formulation are available on:

• Leanpub - where you can get both bundled together!
• Amazon - Discovery and Formulation, with Discovery also available as an audiobook at Audible.
• For more from Seb and Gáspár, check out their Test Guild talk, “BDD is not Automation Testing” and follow them on Twiter at @sebrose and @gasparnagy.

1. Why should automation be done by the dev team?
2. Isn’t the business-readable documentation just extra overhead?
3. What’s wrong with changing the scenarios to enable automation?
4. Can all testing be automated?
5. How can Cucumber help us understand the root causes of failure?

The question​

Isn't the goal of repeatability for refactoring confidence, not test confidence?

In the presentation I gave at the Test Automation Guild conference, I compared traditional test automation to Behaviour-Driven Development. I used a scene from one of my favourite movies, Kelly’s Heroes, to describe the inherent risk that comes from excessive reliance on our automated tests. More on that later.

This question hints at the tension between automated tests and confidence in our software. How many tests is enough? What sort of tests give us the most useful information? Where is the sweet spot that maximises the return on automated test investment?

Repeatability​

I’ve written at length about the test automation pyramid, so I’ll not repeat most of that here. There is an important conclusion that I do want to summarise, and it’s this:

We invest in small, fast, repeatable tests (at the bottom of the pyramid) because they only fail when there’s a problem with the thing that they were created to test

They shouldn’t fail because some other part of the software has a problem. They shouldn’t fail because some external service is having a temporary outage.

This means that we will avoid the flickering test - the scourge of automated tests. When one of these tests fails, we can be confident that there really is a problem in that specific part of the software. And finally, the amount of debugging detective work that we’ll need to do is minimised.

Minefields​

In Kelly’s Heroes the platoon wander into a minefield. Everything seems fine until one unlucky soldier steps on a mine. To get out safely, the person that is nearest the edge of the field carefully works their way forward, probing for mines as they go. Everyone else works towards their nearest colleague. The end result is the minimal number of paths that are guaranteed to be safe.

If another platoon needs to cross that field, they could follow the paths marked out by Kelly’s platoon and be guaranteed to get across safely. That doesn’t mean that there are no mines in the field – just that, by following the safe paths, they’re guaranteed not to step on any.

Automated tests are like paths through the minefield. If a test does fail, then you fix the defect, and the test will now pass – every time it runs. Any defects not already detected, will never be detected by the automated tests. That’s not to suggest there is no value in the automated tests. As suggested by the original question, the continuing value of a passing automated test is to prevent a regression from creeping in (either during refactoring or during development of a new feature).

My friend Gojko Adzic has spoken about the decreasing value of an automated test over time. He interviewed a team at uSwitch that only ran a subset of their tests regularly. I've even come across people that suggest deleting tests if they haven’t failed for a certain period of time. Personally, I encourage teams to think of automated tests as documentation of expected system behaviour. Only delete a test if it doesn’t contribute to an overall understanding of how the system is intended to behave.

Explore!​

To be confident that your system will behave correctly in all possible situations, perhaps you might consider exhaustively testing every possible path through the software, using every possible data value. It doesn’t take long to see that this is impractical. Instead, testers frequently reduce the test space by using partitioning techniques that can dramatically reduce the number of tests needed, without massively reducing your confidence in the outcomes.

Even so, given the inherent complexity of modern software systems, there are many modes of failure that are hard (or impossible) to predict. Therefore, test plans and automated tests will naturally not catch many categories of defects. One widely accepted approach to rooting out these sorts of problems is called Exploratory Testing – as described in the excellent book Explore It! by Elisabeth Hendrickson.

From my perspective, the key aspects of Exploratory Testing that should be kept in mind are:

• It’s a job for an experienced tester
  • Testers with relevant domain knowledge are ideal
• It’s not a checkbox activity
  • Ongoing exploration as the product evolves is essential
• It complements automated testing
  • Refer to the Agile Testing Quadrant from Janet Gregory and Lisa Crispin’s Agile Testing book

Alternatives​

As well as exploratory testing, there are many other approaches to reducing the likelihood of defects in your software. A few of them (in no particular order) are:

• Mutation testing – tooling that injects defects into the codebase, one decision point at a time (to create a mutant). The automated tests are run against each mutant – if no tests fail then there’s something missing from the test suite.
• Chaos engineering – interfering with running systems to ensure that they are resilient in the face of degradation and failure.
• Blue/Green deployments – deploying the latest version of the system in parallel with the current version. A proportion of users are then directed to the latest version and are monitored to ensure that users are still able to achieve expected outcomes.
• Stack-trace driven development – ship the software without testing it and let your users report defects to you.

Conclusion​

Repeatability of automated test is needed for confidence in the quality of the software and the quality of the tests. The first thing that happens when an automated test begins to fail intermittently is that confidence in the test suite as a whole begins to suffer (see the Broken Windows theory).

The challenge with a finite suite of automated tests is that they continually check exactly the same paths through the software. By definition, therefore, they can’t detect defects that lie off those paths. There are a host of approaches that you might adopt, based upon your organisation’s attitude to risk, but the most effective is often to supplement automated testing with exploratory testing.

1. Why should automation be done by the dev team?
2. Isn’t the business-readable documentation just extra overhead?
3. What’s wrong with changing the scenarios to enable automation?
4. Can all testing be automated?
5. How can Cucumber help us understand the root causes of failure?

The question​

Why many times I find myself changing the BDD statements once I start automation, as there is certain information needed between steps? BDD framework is supposed to involve many stakeholders and yet I feel the statements are to be written in a very stringent fashion.

This is a question that gets right to the heart of the tension between specification and automation. Gáspár Nagy and I wrote Formulation to focus on the art of writing good executable specifications. One of the core messages, which warrants its own section, is that “readability trumps ease of automation”.

I’ll be describing some automation techniques while answering this question. There’s a wide variety of languages that are available for automating specifications, but I’ve chosen to write the examples using Java. The techniques are applicable no matter what language you use for automation.

Readability​

The goal of business-readable specifications is to facilitate and encourage collaboration between the business team and the delivery team. Over many years, we have identified six properties that we believe most scenarios should conform to – and captured them in a five-letter acronym: BRIEF.

The benefits of business-readable specifications are numerous (and covered in a previous post), but two of the BRIEF properties are worth expanding on.

Focused​

The majority of scenarios exist to clarify and illustrate how one business rule is expected to behave when implemented. Every application is made up of many thousands of interacting business rules, so we decompose specifications into the individual rules and illustrate each rule with one or more scenarios. These are called illustrative scenarios.

There is also value in scenarios that cover the interactions of multiple rules, or even a complete user journey. These are called journey scenarios and will not be covered in this article.

Essential​

When we write an illustrative scenario, we need to constrain ourselves to presenting the reader with information that is pertinent to the rule being illustrated. Any information that doesn’t contribute to a reader’s understanding of how the rule should behave is incidental and inessential.

It is very tempting to include information that is necessary for the application to work even if that information does nothing to clarify the specific behaviour of the rule being illustrated. If we give in to this temptation, there are a couple of bad side effects:

• It’s harder to read the specification – more data is being presented and it’s not clear which data is essential to the rule’s behaviour
• The specification becomes more brittle – the scenario may need to be changed when some part of the application unconnected to the rule that it illustrates changes

Automation​

Once the team has formulated a scenario to conform to the BRIEF properties, it should be our aim to automate it without having to introduce inessential information. There are many approaches and techniques to automate any scenario without having to change the text in any way. Here I present a high level overview of a few of them.

Passing data between steps​

Every implementation of Cucumber has a mechanism for passing data between steps.

• Cucumber Ruby and JavaScript use a World object which each step can interact with to attach or access information. Many other flavours of Cucumber use a similar mechanism.
• Cucumber Java uses dependency injection (DI) to share a single instance of an object between different step definition classes.
• SpecFlow supports multiple mechanisms for sharing data.

What all implementations have in common is that the shared data is deleted after each scenario. This is to ensure that scenarios run in isolation and can’t affect each other. The obvious benefit is that you can run any scenario on its own, or any set of scenarios in any order, and the results should always be the same.

If you use mechanisms for sharing data or make state changes that persist after the end of a scenario (such as not rolling back database changes), then you can get unexpected test results. Cucumber has mechanisms that allow you to initiate setup and teardown activities at scenario boundaries, but it is the developer’s responsibility to remember to do this.

Behaviour midway through a user journey​

Most scenarios should be illustrative scenarios (as described above), and the rule they illustrate will often only be relevant at specific points on a user journey.

It may seem reasonable to preface the scenario with all the steps required to get to the checkout page, but none of that information would help clarify the expected behaviour of the rule. Instead, there are at least three ways that the Given step of this scenario could be automated as it’s written (I’m assuming we’re dealing with a web application):

• End-to-end
  • Use Selenium (or similar) to add products to the basket, and navigate to the checkout page
  • A test double of the payment provider service or pre-allocated test card details would typically be used
  • The DOM would be inspected to verify that the expected message was shown
• Service/API level
  • Send HTTP requests to add products to the basket
  • A test double of the payment provider service or pre-allocated test card details would typically be used
  • The response payload would be inspected to verify that the expected message was returned
• Component/class level
  • Call method(s) on the relevant class(es) to add products to the basket
  • A test double of the payment component would return indicating success when called to authorise payment
  • The return value of the relevant method call would be compared to the expected value

Each of these approaches avoids the need for the scenario to contain inessential information, while allowing the team to decide the appropriate automation technique to get the desired speed of feedback and level of confidence.

Aslak Hellesøy and the Cucumber team regularly structure their automation code so that a scenario can be executed at any of these levels without altering the text of the scenario.

Automating at the bottom of the pyramid​

The test automation pyramid is a popular metaphor documenting a popular and successful strategy. I recommend you read this article even if you are fully aware of the metaphor already.

In the previous section, the three approaches described live at different levels of the pyramid:

• Component/class – near the bottom
  • the detail of our code is verified using plentiful, small, fast tests
  • mostly come for free with test driven development (TDD)
• Service/API – somewhere in the middle
  • interaction and communication between dependent components is verified using test doubles, virtualisation, and contract tests
  • these tests should be less interested in the actual data than in the categories of data that are being passed e.g., null, empty, zero/positive/negative, errors & exceptions
• End-to-end – near the top
  • some user journeys automated to verify that functionality is available as expected
  • also useful as smoke tests to ensure that deployment to an environment has been successful

Beth Skurrie has noted another variation as you move up the pyramid:

"... the more code the test covers, the lighter the touch of the test should be ... it's easy to fall into the trap of asserting more the more code a test covers, when it's much more maintainable to be asserting less, or at least, making more flexible assertions."

You’ll notice that I haven’t mentioned Behaviour-Driven Development (BDD) at all in this section. That’s because choosing whether to use developer test tools (e.g., JUnit) or natural language automation tools (e.g., Cucumber) depends on who will get value from reading the specification. So, choosing where in the pyramid you should automate should not be influenced by whether you’re using Cucumber or not. I go into this in more details in this article.

Invisible initialisation​

Many of your scenarios will require some level of setup, no matter where in the pyramid they are automated. Even if that setup is not essential information for someone reading the specification, you’ll still need to find a mechanism to make sure it happens.

Examples of initialisation include:

• spinning up a local webserver or in-memory database
• restoring persistent test data to a known state
• creating and injecting test doubles/mocks/stubs

It’s rare that these activities should be included in a scenario. I’d also recommend that they don’t belong in a feature file at all, so should not be part of a Background.

Techniques that I’ve found useful are:

• Cucumber (conditional) hooks
  • all Cucumber implementations have Before and After hooks that are called before or after a scenario runs
  • some implementations have additional hooks, such as BeforeStep/AfterStep and BeforeAll/AfterAll
  • hooks can be made conditional on the tags applied to the currently running scenario
  • this is the default approach that should be considered first
    • tags are part of the specification, so should always make sense to business colleagues
• Proxy/surrogate design pattern
  • access the resource through a proxy, that will create it when first needed
  • this approach should be considered when
    • there is occasional need for an object to be created
    • the creation is expensive, and
    • it is not otherwise possible to localise the object’s use
• Instance variable (for Cucumber’s that support DI)
  • access the resource through an instance variable on an injected shared object
    • the injected shared object can also defer resource creation until first access
  • Cucumber will only create the shared object when execution requires a step definition that requires it

An entity may be essential for correct running of an automated test, without being an essential ingredient of a specification whose purpose is to illustrate and clarify a business rule. These three initialisation techniques will enable you to preserve the business value of the specification, without causing an excessive burden on the automation.

Data builders​

Not all objects are simple to build. They may have many properties – some of which need to be manipulated to illustrate a specific rule, such as a customer that may require name, address, gender, date of birth. For scenarios where the value of these properties is unimportant, we should not need to specify them. For scenarios where it is necessary to set a specific value for only one of these properties, we should be able to do it in a way that preserves the need to only include essential information in the scenario.

The Builder pattern supports easy creation of complex objects, typically by offering:

• A default object which can be used when specific values are not required
• Overrides to set specific values for selected properties
• Named objects (personas) that represent well defined sets of values (e.g., overdrawn account, domiciled overseas etc.)

This approach will also work for objects that are the root of a network of connected objects, such as a customer that has multiple accounts.

Conclusion​

Once the formulated wording of a scenario has been agreed between business and delivery, it should not be changed unless either the business requirement it illustrates changes, or an essential omission or error is found. It should certainly not be modified to make the process of automation easier.

In this article, I have given an overview of five core techniques that can help simplify automation. Some of them are foundational techniques applied across all types of test automation, while others are more specific to the automation of Cucumber/SpecFlow step definitions.

1. Why should automation be done by the dev team?
2. Isn’t the business-readable documentation just extra overhead?
3. What’s wrong with changing the scenarios to enable automation?
4. Can all testing be automated?
5. How can Cucumber help us understand the root causes of failure?

The question​

Doesn't adding something like SpecFlow to interface with something like Selenium just add an extra code base that must be maintained? Does the benefit justify the extra work?

This question directly addresses the costs and benefits of a BDD approach. Although it directly references Selenium, tools like SpecFlow always add a layer that needs to be maintained, so this post won’t mention Selenium specifically.

Adopting BDD does lead to more collaboration between business and delivery teams, development of a shared vocabulary rooted in the business domain, and tooling required to automatically validate the resulting business-readable specifications. What possible benefit can there be that justifies the cost of adding more interactions and more tooling?

The cost of software​

Software takes time (and hence money) to create. We always seem to take longer than we expected to create the software we think our customers want, so there’s a desire to go faster. Common approaches to going faster forget that software creation is a creative process, rather than a manufacturing problem. Simply cranking the handle faster introduces more bugs into the software, which leads to slower delivery.

Here are my top reasons for why trying to go faster results in bad outcomes:

• Misunderstood requirements – human communication is notoriously error prone. If the customer, the developers, and the testers don’t have a shared understanding of what is being asked for (and why), then we will not deliver the functionality that is needed.
• Unarticulated assumptions – it is hard for experts to enumerate all of the assumptions that are foundational to their understanding of the business domain. Where assumptions remain unarticulated, misunderstandings will go unchecked.
• Unnecessary code – in the absence of stable, reliable requirements, the team become used to regular context switches and rework. This frequently leads them to write overly generic code, intended to be flexible enough to handle whatever changes in behaviour the customer asks for next.
• Defects & regressions – misunderstandings, missed assumptions, and superfluous code are direct causes of defects and regressions. Additionally, when teams feel pressure to go faster, it’s often practices that seem to be peripheral to producing production code (such as unit testing and documentation) that are skipped, leading to even more defects. The limiting factor in software development is not a programmer’s typing speed.
• Status reporting – the person paying the bills wants to know that their money is being spent wisely, but getting that information usually requires the team to spend time collecting and presenting that information in a format that the customer can understand.
• Maintenance – software lasts for years or decades, and over that time has to evolve to fit the changing business landscape. Unreliable documentation, poorly structured code, and incomprehensible test scripts all contribute to inflating the total cost of ownership – and yet these are exactly the areas where short cuts are taken when a team is pressured to focus on delivery.

The benefits of BDD​

Behaviour driven development is an agile approach to developing software, consisting of three practices applied iteratively to each increment of functionality:

• Discovery – creates a shared understanding of the requirements through collaboration, typically achieved through a structured conversation centred on rules and examples.
• Formulation – examples of system behaviour are documented using business terminology.
• Automation – the documentation is automated, creating living documentation that verifies the system’s behaviour.

In the following sections, I’ll outline how adopting these practices can mitigate the costs described in the previous section.

Discovery​

Discovery (aka 3 Amigos and Example Mapping) is where the team really begins to understand how a piece of functionality should behave to deliver value to the customer. The product owner/business analyst prepare their understanding of the next small increment and the delivery team test their understanding of it using concrete examples. Through this process, detailed requirements are refined, while misunderstandings and assumptions are surfaced and resolved.

Discovery takes time, but the detailed shared understanding that is achieved needs to be reached no matter what approach the team takes. If the team doesn’t learn about this early, before writing code or test scripts, they will learn later and have to rework code and test scripts that they have already created. Quite apart from the time spent doing the rework, there is the huge cost of the associated context switching.

Another significant benefit of Discovery is gaining the ability to right-size stories because having clear, detailed requirements makes it simple to split large stories into fine-grained increments. This gives the product owner the opportunity to prioritise work at a granularity that enables cheap experiments and MVPs.

Formulation​

The team has already derived significant value by reaching a shared, detailed understanding of what needs to be delivered. They can build on that by formulating that understanding as a business-readable specification, providing a feedback loop that ensures misunderstandings don’t creep in as the product evolves.

It’s during Formulation that the domain language becomes a powerful communication tool. This takes time and effort, but without rigorous definitions of terms being used, both common (e.g., account, customer, date) and domain specific, defects and regressions are inevitable.

Automation​

The incremental value of Formulation is multiplied when the specification is automated, delivering four significant benefits.

When the next increment of the specification is automated before the relevant production code is written, it can be used by the development team to guide development. As soon as the automation demonstrates that the product behaves as described in the specification, the development team knows that they have delivered what is required. No more and no less.

The automated specification can be used in the organisation’s continuous integration and delivery (CI/CD) pipeline as part of its regression testing strategy. This can significantly reduce the cost of creating and maintaining automated tests.

At this point, the automated specification becomes reliable, authoritative living documentation. Every time the CI/CD pipeline runs the product will be validated against the specification, giving a high level of confidence that the documentation is accurate. This is valuable immediately but becomes even more valuable as the product matures and the teams that maintain it grow and change.

Because the living documentation is business-readable, this gives businesspeople a transparent view of product development status that would normally require the provided by people from the delivery team. As the functionality of the product grows, the relevant sections of the specifications will appear in the documentation, complete with evidence that the behaviour they describe has verifiably been implemented.

Conclusion​

The original question asks whether the benefits justify the extra work. I hope I have shown that the benefits are significant, but I disagree with the initial assertion that it requires extra work. Instead, I believe that this approach simply makes the necessary work visible.

By encouraging teams to collaboratively learn the details of a feature before writing code or tests, there are reduced misunderstandings, fewer defects, and less context switching. This learning has to happen for the team to be able to deliver the software, but by learning early we reduce the waste of unnecessary rework.

Formulation and Automation take this learning and turn it into an artefact that improves communication, guides developers to only create code that is necessary, contributes to automated testing, and delivers documentation that can be relied on for as long as the product is in use. The value of reliable documentation on its own is hard to overestimate.

The value described in this post derives from adopting a behaviour-driven approach to software development. Using SpecFlow or Cucumber as automation tools without first engaging in Discovery and Formulation will not deliver the same benefits.

Going deeper​

• Gáspár Nagy and I have written_ books about Discovery and Formulation.
• Matt Wynne will be presenting a webinar called "The Engineering Manager's Dilemma" on April 28th, 2021 (registration needed)
• Aslak Hellesøy has described techniques for making functional acceptance tests run in milliseconds (often reducing dependencies on frameworks like Selenium).

The conference organiser, Joe Colantonio, hosted a Q&A session after the talk, but there wasn’t enough time to answer all of the questions. Handily, he provided me with a list of all the questions asked, along with his estimation of their “sentiment” – either neutral or negative. In this series of posts, I am going to address the five unanswered questions that he marked as having a negative sentiment.

• Why should automation be done by the dev team?
• Isn’t the business-readable documentation just extra overhead?
• What’s wrong with changing the scenarios to enable automation?
• Can all testing be automated?
• How can Cucumber help us understand the root causes of failure?

The question​

I do not get, why you say the automation part #5 on the graph, should be done by the dev team and never the QA, because it's part of the design process. For example, if a team uses Serenity-BDD with screenplay, which makes it very easy to implement SBEs, shouldn't dev team focus on prod code instead?

The question relates to this diagram (which was published in Discovery and as a LinkedIn article).

In my talk I made it clear that I believed that developers should be involved in step #5 - Automate. The questioner asks “… shouldn't DEV team focus on prod code …?” This is a very common question, rooted in a confusion between BDD and Test Automation.

Test automation​

Test automation is a generic term that can be applied to any activity that results in the automation of tests. Programmer (or unit) tests are one aspect of test automation. So are integration tests and end-to-end (E2E) tests. Load, performance, and penetration tests can also be automated.

Typically, teams that are focused on test automation do that automation after the code has been written. Development and testing are separate activities, often undertaken by different teams with different goals.

Development goal: implement the feature

Testing goal: check the implementation achieves expected quality

This approach is widespread and valuable but has some drawbacks. If you’d like to dig into those drawbacks there are some links in the Going deeper section below.

BDD​

Behaviour-driven development (BDD) is an approach that grew out of test-driven development (TDD) and agile software development. The goals of BDD are:

• Encouraging collaboration across roles to build shared understanding of the problem to be solved
• Working in rapid, small iterations to increase feedback and the flow of value
• Producing system documentation that is automatically checked against the system’s behaviour

The diagram at the beginning of this article lays out and idealised behaviour-driven process flow. As you can see, step #5 – Automate comes before #6 – Implement, which can seem back-to-front from some perspectives. How can we automate the testing of software that doesn’t exist yet?

Take a minute to change perspective and things don’t seem quite so crazy. The Automate step isn’t about testing at all. Each scenario describes one behaviour of the system which will need be implemented in code. When we automate that scenario, we begin to imagine the code we wish we had. This is a detailed design activity – and as such needs the involvement of someone with development skills.

This approach is, in this respect, identical to TDD – except that TDD is generally a developer-only activity. BDD is collaborative, bringing together the 3 Amigos (business, developer, tester) to collaborate throughout. The additional benefits that BDD brings are a shared understanding of the business domain and business-readable documentation.

#5 – Automate transforms a scenario in the business-readable documentation into a failing automated test. This guides the development team as they design and implement the code that will deliver the specified functionality. Once the behaviour has been implemented, the automation ensures the continued correctness of the system’s behaviour, and the documentation is considered living documentation. In contrast, documentation that lives in textual documents that are external to the system itself, can be thought of as dead documentation, because it usually reflects how the system used to behave.

BDD does not replace traditional testing and test automation, but it does reduce a team’s reliance on them to some degree. Nor does BDD replace a team’s need for people with testing skills. They are needed more than ever – to help the team reach a shared understanding, to share with developers their specialised domain knowledge, and to ensure customer satisfaction using specific skills such as exploratory, load, or usability testing.

Confusion​

The confusion between test automation and BDD may have its roots in its predecessor, TDD. It has always been problematic that a design and development technique should have the word “test” so prominent in its name. Especially in an industry where development and test have been so siloed for so long.

The confusion has been exacerbated by the understandable desire to utilise testers that don’t have development skills to write automated tests. This desire has been encouraged by the existence of natural language automation formats using the language of Given/When/Then:

• Given/When/Then are the core keywords of Gherkin
• Gherkin is the structured syntax understood by automation tools such as Cucumber
• Cucumber is a widely downloaded, open-source tool, available on numerous platforms
• Cucumber was created to support BDD

These facts allow people to deduce an “obvious” but incorrect conclusion:

• I conform to Gherkin when I write my tests
• I use Cucumber to automate my Gherkin
• Cucumber was created to support BDD
• Therefore, I am "doing BDD"

The correct conclusion should be:

• Therefore, I am automating tests using Given/When/Then

The correct logic flows in the opposite direction:

• To benefit from a shared understanding, the team needs to collaborate on the detailed specifications
• To assure the value of that understanding, it must be captured using business-readable terms
• Gherkin’s use of natural language and Given/When/Then makes it an ideal choice
• Cucumber’s ability to understand Gherkin makes it the ideal automation tool
• Therefore, Cucumber and Gherkin are supporting the team to work in a behaviour-driven way

Conclusion​

If the goal is to automate a test, then you may not need developer skills (I think you still need developer skills, but that is another article).

If the goal is to reduce misunderstandings (and hence defects, rework, and costs), then you should look beyond test-after automation to BDD. In which case functional automation is an integral part of the design and implementation process and requires the development team to be leading participants.

Going deeper​

I’ve presented a session called “Contrasting test automation and BDD” on this topic at a number of conferences and webinars over the past year. For more extensive coverage, please watch the video and take a look at the slides.

Gherkin is probably the reason for this win, because:

• Given/When/Then are the core keywords of Gherkin
• Gherkin is the structured syntax understood by automation tools such as Cucumber
• Cucumber is a widely downloaded, open-source tool, available on numerous platforms

So, it might be fair to say that when it comes to documenting and automating examples, “Gherkin rules.” That is not what this article is about.

Gherkin had been incredibly stable for almost a decade, but in late 2018 a new keyword was introduced: Rule. I’m going to explain the background to that change and bring you up to date with a recent fix that makes the Rule keyword consistent with other Gherkin keywords.

Example Mapping​

Matt Wynne discovered example mapping while helping a client’s team. They were struggling to analyse the details of a user story and Matt used a deck of coloured index cards to help teach them a structured way of reaching a shared understanding of what behaviour the story should deliver.

If you’re not familiar with example mapping, then I suggest you spend a few minutes reading Matt’s excellent introductory article, Introducing Example Mapping. For the purposes of this article, the salient point is that example mapping concerns itself with four concepts, each represented by a coloured card: Story (yellow), Rule (blue), Example (green), and Question (red).

Once example mapping is complete, you should have a structure that looks something like this:

You can tell that the story is well understood, because there are no red question cards in the map. All of the team’s questions have been answered satisfactorily.

For the story to be considered done, the team will need to deliver code that implements each of the rules, laid out horizontally underneath the story. To dispel any ambiguity, each rule is illustrated by examples, laid out vertically beneath it.

The next step is to formulate the examples – translating each example into a business-readable, natural language Scenario that can be automated by Cucumber.

Formulation​

Gherkin scenarios are written in feature files. Each feature file describes some capability of the system being developed. The example map also describes some capability of the system being developed. Formulation is the process of documenting the shared understanding reached during example mapping in a business-readable format (for current and future teams) that can also be understood by automation tools.

Of the four card types used in example mapping, only two of them should be captured in feature files – the rules and the examples.

The story card should not be reflected in a feature file, because a user story is a transient artefact of prioritisation and planning. I’ve talked about this at length.

Question cards should not be reflected in a feature file, because they should be answered before the example map is formulated as scenarios. The answers to the question cards may well have caused the creation of rules and/or examples which need to be documented in the feature file, but the questions themselves are transient.

Each scenario in a feature file represents a single example card from the example map. We’ve had a lot of experience with successful formulation of examples as scenarios, but this is out of scope for this article. If you want to know more, we’ve written a short book about the subject called Formulation.

Gherkin before Rule​

Each example in an example map illustrates one (and only one) rule. The relationship between rule and example is critical. Without its examples, a rule may be ambiguous. Without a rule, an example lacks context. Together they fully specify an expected behaviour of the product and guide the development team’s efforts.

You would therefore expect Gherkin to provide a mechanism for documenting the rule that each scenario illustrates, but until Gherkin 6 this was not possible. The root cause is that Matt discovered example mapping after BDD, Cucumber, and Gherkin were created.

For over five years, the recommendation was to record the rules illustrated in a feature file as part of the feature file description – a free text portion following the feature’s name.

Feature: Library changes

  Rules implemented:
  - Members pay reservation of $1 per item
  - Maximum of 2 active reservations per member

The problem with this approach is that most feature files document several rules and there is no way to associate each scenario in the feature file with the feature file description that lists the rule that it was created to illustrate.

An alternative is to add a comment to each scenario, referencing the rule that it illustrates by name.

Feature: Library changes

  Scenario: Reserving a single book
    Given Andrew is a member
    When he reserves one book
    Then he is charged $1.00

While this does document the association, these comments are no different from any other comment as far as IDEs or automation tools are concerned, so the important relationship is only visible after a careful reading of the text.

With the creation of a Rule keyword in 2018 the relationship between rule and scenario became a first-class concern of Gherkin.

Feature: Library changes

  Rule: Members pay reservation of $1 per item
  
    Scenario: Reserving a single book
      Given Andrew is a member
      When he reserves one book
      Then he is charged $1.00

Modern IDEs are able to display scenarios grouped by rule – and results of automation are grouped by rule as well. The critical relationship of rule and example that forms the foundation of example mapping can now be truthfully represented in a Gherkin feature file.

Inconsistent tagging​

In the effort to add the Rule keyword to Gherkin, an important piece of functionality was overlooked. Gherkin supports the ability to tag features and scenarios with domain-specific text. The tags can be used for many purposes, including the creation of documentation subsets and the selective invocation of automation code.

Each Gherkin scenario can have a distinct set of tags applied to it, but all scenarios in a feature file also inherit any tags applied at the feature file level. Since scenarios notionally illustrate a rule, it would make sense for them to inherit any tags applied at the rule level as well.

Unfortunately, the original implementation of the Rule keyword did not allow tags to be added at the Rule level.

Consistency is coming​

Last week, a pull request enabling the tagging of Rules was merged into the Gherkin project. Tag inheritance was also implemented and behaves in exactly the same way as feature tag inheritance. A scenario will inherit any tags applied to the rule that it illustrates, as well as any tags applied to the feature.

@my_feature_tag
Feature: Library changes

  @my_rule_tag
  Rule: Members pay reservation of $1 per item
  
    @my_scenario_tag
    Scenario: Reserving a single book
    # this scenario has all three tags applied 

This change will automatically be incorporated into future releases of flavours of Cucumber managed by the Cucumber open-source project – as well as any ports of Cucumber that use Cucumber’s Gherkin parser.

Over the coming months, expect to see Rule tagging ship with your Gherkin automation tool of choice. Finally, the Discovery practice of example mapping, and the world’s favourite Formulation format will be consistent with each other.

Gojko Adzic has been running a series of challenges called #GivenWhenThenWithStyle. This is my solution to challenge #16. You should definitely check out the rest of the challenges and their solutions.

Gáspár Nagy and I have written in more detail about writing and organising feature files in our new book, Formulation.

The winner is…​

From the survey, splitting by functional area and grouping by capability was the runaway winner.

Regarding the “other”, Ken Pugh again posted a detailed response in which he suggests organising feature files in a functional hierarchy. I’ll dig into this in more detail later in this post.

Dhanunjay was one of three participants that chose the feature file per story solution, for ease of maintenance. Another participant chose split by system component because it gave “better visibility of test coverage”. I have very strong reservations about both of these rationales, which I explain below.

Matt Wynne was one of the participants that selected the winning option, with this succinct reason:

Because every one of the other options leaks solution domain; this is the only one that's pure problem domain.

This reason was echoed by other participants André, Marcel Kwakernaak, and Geert-Jan Thomas who all recognised the importance of feature files as business-facing documentation. I particularly like this comment, left by an anonymous participant:

The software that is developed is for business purpose, therefore ability to master business capability is prime over technical aspect. I like the idea of being able to quickly pin down system-components in question that generate issues, so a supporting structure that conveys which system-component is to "blame" is very welcome - typically I do this with tags and evaluate the test-reports based on tags…

More on tags later.

Specification or testing?​

In my experience, the Given/When/Then format has two major drivers in industry:

• Specification of system behaviour
• Test automation

It was originally used as a way to specify the behaviour of systems. To automatically verify that the software being developed met those specifications, the Given/When/Then examples needed to be written in a format that automation tools could understand. This led to the creation of the Gherkin language, which defined a structured way to capture Given/When/Then specifications as scenarios in feature files. Gherkin enabled the creation of automation tools (such as JBehave, Cucumber, SpecFlow) that could read the specifications and execute automation code in response.

With the increased demand for test automation, Gherkin’s similarity to natural language was recognised as a way to enable test automation tasks to be performed by people that did not have software development skills.

The winning option for this challenge implicitly assumes that Gherkin is being used to specify system behaviour. The responses expect feature files to be used as documentation, throughout the lifetime of the product. This has always been the intention of Behaviour-driven development (BDD) and Specification by example (SbE).

The remainder of this solution will assume that you intend to use Gherkin to document system behaviour.

Features are documentation​

Gherkin documentation is split into feature files, but there’s no standard definition of what a feature is. The Collins dictionary defines a feature as “an interesting or important part or characteristic,” and I don’t intend to try to be more specific.

To illustrate feature file organisation I’ll use diagrams below that relate to an imaginary online pizza ordering application (taken from our book, Formulation).

Navigating the documentation​

There's nothing worse than trying to find a specific piece of information in poorly organized documentation, so feature files should be organised in a logical way. A typical way to structure documentation is to use chapters, sections, sub-sections and so on.

Each scenario documents an example of the system's behaviour. Scenarios live in feature files, which in turn live in source control, which is typically organised hierarchically. The feature files collectively document the behaviour of the system, but are only useful when people can easily navigate to the part that they are interested in. A hierarchical tree of folders provides a simple way to structure our documentation that is identical to traditional documentation.

Figure 1 - Traditional documentation

Consider a system that allows customers to order pizzas online. A partial view of the documentation might look like this:

Figure 2 – Online ordering

Zoom-in/out​

A hierarchical structure also allows us to organise documentation by level of detail. The deeper the folder, the more fine-grained the behaviour described. Here's how the Customers hierarchy might actually look.

Figure 3 - Customers documentation

Even in a simple system, there are many levels of detail. Some readers will be looking for the big picture - they should not need to travel far from the root of the documentation. Others will be interested in specific details - they should be able to navigate to the relevant section easily.

This is similar to how we use online maps. We can get a good idea of where New York is relative to Washington without needing much detail. If we actually want to drive from one to the other, we will need to zoom in and get details about which route to take.

Specifying shared behaviours​

You may have noticed a send_email.feature inside email confirmation. There may well be other features that send emails and we wouldn’t want to create a copy of that feature file wherever that functionality is used.

When we find a feature that is used by several parts of our system, we should ensure that a single feature file describes the behaviour. Then, if it needs to change, we only have to change the documentation in one place.

Figure 4 – Extracting shared email functionality

It's easy to move shared features into a separate part of the tree, but at the moment, there's no special way to link from one feature file to another in Gherkin. For now, I recommend that you document the dependencies of a feature in the feature file description. The best information to include is the name and relative file path of the:

Feature: Send unique link
  This feature is dependent on:
  - Send Email - ../../../communication/email/send_email.feature

Targeted documentation​

There is another important aspect of documentation - different consumers require different information. To keep the documentation readable, we need to ensure that sales executives don't get forced to read API documentation, while integrators may not be interested in all the details of our application.

Depending on your situation, you may choose to create separate hierarchies for specific stakeholders or interleave the targeted documentation throughout.

Figure 5 - Separate hierarchy

Figure 6 - Interleaved

There are situations where it is impractical to represent cross-cutting interests hierarchically. In these cases what you need is something like an index that allows you to find all the relevant pieces of documentation. Gherkin has a feature called tags that can be used for this purpose.

Stories aren’t features​

It is common to see teams create a new feature file for each user story that they implement. I’ve written extensively about why this is a bad idea.

TL;DR – User stories are intended to be used to deliver small increments of functionality. If you create a feature file for each story, then you will expend unnecessary effort maintaining existing feature files. Additionally, feature files created this way will not be useful documentation, because each will only describe a thin slice of functionality.

Conclusion​

Of the options provided in this challenge, the winner is the best choice. Personally, I would have chosen Other, because there is more to organising your documentation than simply splitting by functional area.

I have a different take to Gojko, that is more in line with the community response. This article describes my approach – but you’d be advised to read Gojko’s original posts first (and subscribe to the challenge).

BRIEF – the acronym​

My colleague, Gáspár Nagy, and I created the BRIEF acronym to capture six principles that we find useful when writing scenarios. I’m going to critique Gojko’s solution using these principles.

We’ve written about BRIEF in the second book in our BDD series, Formulation, and there's also a blog post about it. Here is a high-level summary of the acronym to help you read this article:

Overall structure​

Gojko’s solution is a scenario outline with four example tables containing a total of 26 examples. This has a number of problems:

• B – although the scenario does use business language, my instinct is that there are very few customers or product owners that would be interested enough in the detail to actually read and review it in detail. Collaboration is the purpose of using business language and this sort of scenario can easily put people off.

• F – the rule that this scenario is trying to illustrate, as stated in the original post, is “banks will cancel an authorisation if it has not been charged for a month.” However, the majority of the examples are trying to illustrate the meaning of “month” rather than focusing on authorisation cancellation. These are two separate rules, but this scenario is attempting to illustrate both of them.

• Brief – the scenario outline (when combined with the example tables) is long. Most users won’t be able to read the entire thing on a single screen, so they will need to scroll around the page or rely on their memory. Even if they have a large screen or a small font, it’s still hard to parse all this information – which is borne out by the errors that have made it into both the February example tables.
  The second example in February (non leap) should read:

  | 27 January 2019 | 28 February 2019 | cancelled |

  The fourth example in February (non leap) should read:

  | 28 January 2019 | 28 February 2019 | authorised |

  The ninth example in February (leap year) should read:

  | 29 January 2020 | 29 February 2020 | authorised |

  The twelfth example in February (leap year) should read:

  | 30 January 2020 | 29 February 2020 | cancelled |

  Both tables for February are also missing any examples for authorisation dates of 31 January.

Dates​

Using actual dates does appear to follow the real data principle, but is all that information actually relevant to the rule?

• E – the only reason that the year is used in the examples is to determine whether it’s a leap year or not. It’s entirely unnecessary for the first two example tables. For the last two example tables, it’s only used to indicate whether it’s a leap year or not, so showing the actual year is unnecessary. Having extra information frequently leads to confusion, when a reader has to decide whether it’s significant or not.

What is meant by “match”?​

The original problem statement doesn’t use the word “match”, but it appears three times in the solution scenario outline. What does it mean? Is it explained by the comment in the end of month example table: “If matching date cannot be caculated (sic), use end of month to cancel”?

• I – the examples are trying to illustrate the mechanics of how a month is calculated. Having read the examples a number of times, the definition of the term “match” is not easy to understand. A cursory reading of this example table might even give a false sense of comfort.
• F – there is a rule about how a month should be calculated. Trying to illustrate this rule in a scenario that is supposed to illustrate authorisation cancellation is distracting.

It seems to me that the September/October examples shouldn’t even be in this example table. I presume that “matching date cannot be calculated” means that the day number of the authorising month is not present in the processing month (e.g. if authorised on 31 March there is no corresponding 31 April). However, the first example is for 30 September, and 30 October is entirely valid, so why is this example present in this table?

Quite apart from that, the rule itself seems inconsistent. When there is a “matching” date in the processing month, cancellation takes place the day after that date. The example for 31 March shows the authorisation being cancelled on 30 April, but it would appear more consistent to cancel the authorisation the day after the end of month i.e. 1 May. This is, of course, the decision of the product owner, but since the example is just one of many in this scenario outline it may well escape notice. As you will see in the scenarios below, this leads to some very strange inconsistencies (based upon my deduction of how the calculation is intended to take place).

Another solution​

I would suggest writing a much simpler scenario to illustrate the cancellation rule. If after reviewing the calculation itself, the product owner decides to stick with this outlandish definition of a “month” then I would encourage the team to discover a domain-specific name for it!

  Scenario Outline: cancelling uncharged authorisations
   
  To avoid locking up client funds indefinitely, we need to cancel an authorisation if it has not been charged for a month. The accuracy level of 1 day is enough.
   
    Given a transaction is authorised
    And the transaction was not charged
    When the cancellation job runs <processing period> later
    Then the authorisation should be <status>
   
    Examples:
    | processing period | status     |
    | a month           | authorised |
    | a month and a day | cancelled  |

Calculating a month​

The month calculation could then be illustrated as shown in the scenarios below. Certainly, this sort of analysis should be provided to challenge the product owner’s definition of a “month”, but whether Given/When/Then is the best way to do it is open to question. Since exhaustive scenarios like these are more like test cases than a business specification, I would not put these in the same feature file with higher level scenario above. I would either:

• Consign them to a separate feature file in a sub-folder, where they won’t distract from the narrative flow, or
• “Push them down the stack” into programmer tests

Notice that there is no mention of authorisation, processing, or cancellation in the follwing scenarios. They simply illustrate how a month should be calculated.

  Scenario Outline: month calculations, not including February
    Given something happened on <start date>
    When a calculation is made on <calc date>
    Then the resulting period should be <period>

    Examples: start and calculation month both have 31 days
    | start date | calc date   | period            |
    | 31 July    | 31 August   | a month           |
    | 31 July    | 1 September | a month and a day | 

    Examples: start month has 30 days and calculation month has 31 days
    | start date | calc date | period            |
    | 30 April   | 30 May    | a month           |
    | 30 April   | 31 May    | a month and a day |

    Examples: start month has 31 days and calculation month has 30 days

    This leads to CRAZY inconsistencies! 
    Have I misunderstood the matching calculation?

    | start date | calc date   | period            |
    | 30 October | 30 November | a month           |
    | 30 October | 1 December  | a month and a day |
    | 31 October | 29 November | a month           |
    | 31 October | 30 November | a month and a day |

  Scenario Outline: month calculation for February
    Given this is a <year type> year
    And something happened on <start date>
    When a calculation is made on <calc date>
    Then the resulting period should be <period>

    Examples: normal year
    | start date | calc date   | year type | period            |
    | 27 January | 27 February | normal    | a month           |
    | 27 January | 28 February | normal    | a month and a day |
    | 28 January | 27 February | normal    | a month           |
    | 28 January | 28 February | normal    | **a month**           |
    | 29 January | 27 February | normal    | a month           |
    | 29 January | 28 February | normal    | a month and a day |
    | 30 January | 27 February | normal    | a month           |
    | 30 January | 28 February | normal    | a month and a day |
    | 31 January | 27 February | normal    | a month           |
    | 31 January | 28 February | normal    | a month and a day |

    Examples: leap year
    | start date | calc date   | year type | period            |
    | 27 January | 27 February | leap      | a month           |
    | 27 January | 28 February | leap      | a month and a day |
    | 27 January | 29 February | leap      | a month and a day |
    | 28 January | 27 February | leap      | a month           |
    | 28 January | 28 February | leap      | a month           |
    | 28 January | 29 February | leap      | a month and a day |
    | 29 January | 27 February | leap      | a month           |
    | 29 January | 28 February | leap      | a month           |
    | 29 January | 29 February | leap      | **a month**           |
    | 30 January | 27 February | leap      | a month           |
    | 30 January | 28 February | leap      | a month           |
    | 30 January | 29 February | leap      | a month and a day |
    | 31 January | 27 February | leap      | a month           |
    | 31 January | 28 February | leap      | a month           |
    | 31 January | 29 February | leap      | a month and a day |

The periods underlined above are based upon my reverse-engineering of the “matching” rule and so are possibly incorrect. The first directly contradicts the example provided in Gojko’s solution. The second isn’t present in Gojko’s solution, but (by inference) I believe it would have contradicted Gojko's solution. Such is life.

Conclusion​

I have been enjoying the #GivenWhenThenWithStyle series and find it vary valuable. Hopefully I’ve laid out my differences of opinion with Gojko in an intelligible and constructive way.

Gojko is one of the most influential practitioners in the field and his book Bridging the communication gap is what got me interested in this approach in the first place. It was his encouragement at a conference workshop a few years ago that led me to create the BRIEF acronym.

I look forward to our next meeting, COVID permitting, somewhere on the conference circuit. Or, possibly, at a vegan restaurant.

We're thinking about running a … one-day class [about BDD] designed for business execs. Tell us the questions we must answer for it to be useful.

We received 60 responses, including some very good questions.

As well as providing valuable input into the design of our upcoming course, BDD for busy people, we felt that we should answer some of the questions in a public forum. This is the second in the series.

“Where does the sign-off of requirements happen – and how is it documented? Can we have full traceability?”

Traceability is a big deal. In regulated industries it’s a legal necessity to demonstrate that you have followed the approved process, but it’s important in unregulated businesses as well. If you can prove to yourself that a bug-fix your team has just made fixed the intended defect (and hasn’t inadvertently broken any other functionality), then you will have the confidence to release it. Traceability from the original bug report, to the automated test(s) that initially demonstrated the defect and now demonstrate that it has been fixed, will be essential. Further evidence that relevant regression tests (automated or manual) continue to pass will also be needed. The change may need to be signed-off for compliance or process reasons.

BDD isn’t a methodology​

Behaviour-Driven Development (BDD) is a set of practices (Discovery, Formulation, Automation) that help enable teams to deliver software in an iterative and incremental way. We have seen how BDD can be a game-changer for organisations that aspire to agility (which is most of them), but we realise that there is more to delivering valuable software than the BDD practices.

You can use BDD with any of the agile methodologies (Scrum, SAFe, XP, LeSS etc.), because BDD is about building the shared understanding that’s necessary to deliver and evolve great software products. You can use the BDD practices in any industry, on any platform, in any country, because BDD practices are agnostic about most of the specifics of your context.

The corollary to this is that BDD will not tell you how to implement traceability or capture sign-off. BDD is not a methodology and does not prevent you adopting your methodology of choice.

Where do requirements come from?​

Matt Wynne created a lovely diagram that captures software product development at a very high level.

In this diagram, requirements are represented by the light bulb at the left. The requirement may have originated from a customer enhancement request, an insight from the CEO, a suggestion from an intern, or any number of other sources. How those myriad ideas are validated and prioritised is a problem not addressed by BDD.

The first practice of BDD, Discovery, starts when the team collaboratively works together to reach a shared understanding of a requirement. During Discovery we refine, modify, and split requirements into smaller incremental work items. Any traceability or sign-off that occurs pre-Discovery will need to be revisited, because it is only during Discovery that we achieve something approaching a full understanding of the requirement.

Traceability is dynamic​

Over time requirements will change. Any approach we have that tracks the progress of requirements from inception to delivery needs to accommodate their dynamic nature. For this approach to be useful we need to be able to traverse and query the relationships to ensure, for example, that every requirement has a sponsor, or that every test has a successful execution.

Cucumber provides a subset of this functionality out-of-the-box. Feature files document the requirements and Cucumber automatically verifies them against them against the software that has been delivered. All divergences between requirements and implementation are reported immediately.

The limitation is that BDD and Cucumber concern themselves with only a subset of the full software lifecycle, leaving pre-Discovery and post-Automation traceability as an exercise for each organisation to solve. Additionally, while the Cucumber eco-system is relatively well-served with developer tools that can navigate from feature file to code to test result, there is currently no way to easily expose these relationships in a business-friendly way.

Sign-off is static​

As the requirements change, it becomes necessary to record not just the current relationships, but also previous relationships. Since Cucumber feature files are checked into source code management (SCM) alongside the code that they document, we can use the versioning capabilities of the SCM to record snapshots of the dynamic evolution of our software systems.

A sign-off can be thought of as an approved snapshot that contains all relevant artefacts. We can utilise the ability to create tags or branches within the SCM as a persistent mechanism for recording that a certain version of the software has been approved. Unfortunately, SCMs typically contain only a subset of the necessary artefacts – even if managers and executives were prepared to use them for sign-off purposes.

Integration or monolith?​

Recording relationships as cards on a board is valuable, but difficult to search, traverse, or persist. There is very little tooling that supports this sort of relationship traceability. Some monolithic tools do exist (c.f. DOORS), but they require the wholesale adoption of an integrated toolset.

Over the past decade a set of open standards has been developed, called Open Services for Lifecycle Collaboration (OSLC). This offers a vendor-independent approach to solving this problem by defining mechanisms that allow smaller, focussed tools to work together. Uptake has been slow.

Is requirements traceability something that interests you? We're looking to talk with people who have unmet traceability needs to understand what challenges they're most interested in overcoming.

If you're willing to help us understand your requirements traceability needs,
please fill out this very brief survey to participate.

Conclusion​

BDD (and Cucumber) provide traceability within the scope of their competence – from story to test execution. Cucumber already makes this information available to business sponsors and executives through HTML reports and the newly released Cucumber Reports. Work is ongoing to make this simpler and more wide-ranging.

Broader traceability that spans the full software lifecycle (outside the SCM) requires integration between multiple tools across the organisation. Expect to see tools appear in this space to ease the tensions between continuous delivery, traceability, and sign-off.

By the time we get to designing software systems, the complexity is such that requirements are a collaborative effort that need to be recorded in a structured way. During the 20th century, various formal and semi-formal approaches were integrated into a waterfall model – where each project had distinct, gated phases and each phase had defined outputs, such as “requirements specification”. These approaches to requirements management did not remove the risk of project failure, while adding considerable upfront costs that were often unjustified in the rapidly changing business environment.

The Road to Nowhere, Glasgow​

The evolution of agile ways of delivering software removed the phased approach to defining detailed requirements in favour of incremental, just-in-time requirements definition. A side effect was a perceptible decline in enthusiasm for documentation of any kind. Instead, user stories were misused as agile requirements, causing ambiguities that inevitably led to rework and stakeholder dissatisfaction.

What can we do to avoid ambiguous requirements without imposing excessive financial or time costs on our projects?

Crosswalks​

It is dangerous for pedestrians to cross roads, so someone came up with the idea of crosswalks and signals. The goal might have been: “let’s install signals that tell pedestrians when it’s safe to cross the road.”

One of the user stories for a safety signal at a crosswalk might have looked like this:

As a pedestrian
I want clear signalling at a crosswalk
So that I can be safe when crossing the road

Attached to the card is the following acceptance criteria (AC):

• There should be separate signals for “safe to cross”, “not safe to cross”, and “safe to finish crossing, but not safe to start”

Take a minute to think about the goal, the user story, and the acceptance criteria. Is there any scope for misunderstanding? If so, how would you confirm your understanding of what’s being asked for and surface any ambiguity?

Illustrating using concrete examples​

Concrete examples are used in all walks of life to clarify rules. We use them when teaching our children:

“When you come to a cross-walk, if the signal is red, then you shouldn’t cross the road.”
“When can I cross the road?”
“You can cross the road if the signal is green.”
“What if the red signal is flashing?”
“The flashing red signal means that you shouldn’t start to cross the road. If you’re already crossing the road, then keep going, because the signal will turn to solid red soon.”

This conversation includes a number of examples that were used to clarify how to behave at a crosswalk:

1. Arrive at a crosswalk
   Signal is green
   Cross the road
2. Arrive at a crosswalk
   Signal is solid red
   Don’t cross the road
3. Arrive at a crosswalk
   Signal is flashing red
   Don’t cross the road
4. Crossing the road on a crosswalk
   Signal starts flashing red
   Finish crossing the road

These four concrete examples exhibit the structure that’s common to all concrete examples – they consist of three parts: context, action, and outcome:

• Context: the initial state of the system. In this case, where you are in relation to the crosswalk
• Action: an input or event that causes a behaviour. In this case the state of the crosswalk signal
• Outcome: the resulting intended behaviour. In this case whether to start/continue crossing or not

Clarifying using concrete examples​

Concrete examples aren’t just useful for illustrating the intended behaviour of a system. They’re also useful for challenging and clarifying the underlying intent. Consider these examples:

1. Colour-blind pedestrian arrives at a crosswalk
   Signal is green
   ???
2. Blind pedestrian arrives at a crosswalk
   Signal is green
   ???
3. Crossing the road on a crosswalk
   Signal turns solid red
   ???

The question marks in these examples indicate that the user story is insufficient to achieve the goal. In fact, crosswalk signals typically use more than just colours to indicate whether to walk or wait and many crosswalks have audible signals as well.

And what should you do if the signal goes solid red while you’re on the crosswalk? Stop walking? Walk faster? Panic?

The concreteness of an example helps us consider how a system should behave in specific situations. This engages our brain at a level of detail that doesn’t happen when we consider things in the abstract, which is why concrete examples are so powerful when it comes to clarifying requirements.

So many examples​

As you can see, it’s easy to generate lots of concrete examples, but more is not necessarily better. Having too many examples can make it hard to see what’s important, as well as imposing a maintenance burden when the business needs change. There are two common ways of managing the volume of examples.

The first is to ensure that all examples are valuable. If two examples illustrate the same behaviour, then remove one of them. One way of demonstrating the value is by giving each example a name that explains why it exists. From the crosswalk examples, some names could be “Signal is green”, “Signal is solid red” and so on.

The second is to separate the examples according to the aspect of the requirement that the example illustrates. Even though all seven examples listed above illustrate the user story’s motivation of telling pedestrians when it’s safe to walk, that user story can be thought of as having several separate acceptance criteria:

• Sighted person at crosswalk should be informed if it is safe to cross
• Sighted person on crosswalk should be informed that it will soon become unsafe to be on the crosswalk
• Colour blind person at crosswalk should be informed if it is safe to cross
• Blind person at crosswalk should be informed if it is safe to cross

Acceptance criteria are a common component of user stories, but they are frequently misunderstood as test cases. Instead, you should think of acceptance criteria as rules that need to be satisfied for a user story to be considered delivered.

Spot the requirement​

Requirement itself is an ambiguous term. The goal of telling pedestrians when it’s safe to cross the road is a requirement. The user story describes a safety requirement from the perspective of a pedestrian. The acceptance criteria also describe requirements, but at a finer granularity.

They are all requirements, but as we have seen, it is harder to reason about large, abstract concepts than about detailed, concrete concepts. For the remainder of this post, when I use the word “requirement”, I will be using it in a general, hand-wavy way to mean “some need that the system must satisfy”. Most of the time I will be more specific and use the following terms:

• User story: a narrative description of a need that serves as a “placeholder for a conversation”
• Rule (a.k.a. acceptance criteria): a fine-grained description of a required behaviour of the system
• Concrete example: a single combination of context-action-outcome that clarifies how a rule should behave

Example Mapping​

The Cucumber team have been using a technique called Example Mapping for almost ten years to power structured, collaborative conversations in agile teams. It was discovered by Matt Wynne and through his blog post on the subject, and many consulting engagements, it has gone on to improve requirements for countless teams across the world.

I don’t propose to repeat the whole of Matt’s excellent blog post here. Instead, I’d like to summarise some of the major advantages that example mapping brings:

• Short, frequent example mapping workshops: short timespans allow people to focus; high frequency maintains momentum
• Full engagement of all participants: there are cards and pens for all
• Visual feedback: the example map will tell you if the story is ready to be worked on
• Actionable outcomes: question cards are allocated to participants; answers will feed into the next session

Originally, example mapping required teams to be collocated, with access to 4 coloured decks of index cards and buckets of Sharpies. Indeed, this may still be the most effective way of running an example mapping session, but with remote working, distributed teams, and electronic project management tools, this approach is no longer generally suitable.

Traceability​

One property that many organisations are looking for is requirements traceability – defined as "the ability to describe and follow the life of a requirement in both a forwards and backwards direction (i.e. from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases)"

Simply put, traceability should be able to answer questions like:

• Has this requirement been satisfied?
• What requirements does this test case verify?
• Which versions of the product is this requirement available in?
• What requirement(s) required this code to be written?

The most typical demand for traceability comes from regulated industries, where external auditors demand evidence of compliance. However, traceability has far broader applicability than that. With the continuing shift towards continuous deployment, there’s a growing need for low overhead mechanisms that enable internal processes to automatically assess the fitness of software for release.

Example mapping already provides a link from user story to concrete example. Teams that are using a behaviour-driven development (BDD) approach will then formulate the concrete examples into business-readable, executable specifications that will guide development and provide long-lived, reliable, living documentation. Automated execution of the specifications extends traceability to test executions and product versions.

This is a significant advance when compared to the prevalent forms of requirements traceability in the industry today, such as spreadsheets or links in ALM tools. Even better, there is no overhead incurred. Every activity in the BDD approach delivers intrinsic value. Traceability comes as a secondary outcome.

If you're willing to help us understand your requirements traceability needs,
please fill out this very brief survey to participate.

Conclusion​

The need for good requirements has existed since humankind first undertook projects that made use of more than a handful of people. Without good requirements, project outcomes are less likely to meet expectations, with negative consequences for all concerned. Manual quality processes are error-prone but do provide a level of confidence that we’re delivering what we have been asked for.

Modern software development organisations can no longer afford the rework that results from poor requirements, the delay that arises from manual quality processes, or the customer dissatisfaction caused by defective products. Example Mapping, automated tests, and requirements traceability combine to provide an effective solution that neither costs the earth nor taxes the brain.

I asked Ryan Marsh - co-author and trainer of the class to give an answer. Here's what he had to say.

There are three steps to scaling BDD across the organization:

1. Create awareness around the problem BDD solves
2. Use training to create a base level of skill and soften the impact of change
3. Follow up and measure

Create awareness​

BDD solves a common list of problems affecting nearly all software teams. You must make teams aware of the pain they are experiencing on a regular basis which they may have become used to.

1. Lack of adequate understanding of the scope and customer needs. This leads to missed edge cases and ultimately rework, defects, failed sprints, anxiety-inducing sprint demos, and variability in team throughput.
2. Lack of trustworthy requirements documentation, especially as an application ages. This leads to confusion, extra labor before and after the sprint, defects, knowledge in the heads of a few, and even re-writes or reverse engineering.
3. Lack of customer-focused test suite. In spite of QA’s best efforts, there is often a gap in test suites. This leads to regressions, low/no information to product owners on the business impact of test failures, or even testing and work on no-value non-requirements.

Use training​

Whole team training, such as our BDD Kickstart, can provide a shared baseline of skills on the team that will lead to success. Our training builds a whole team experience that emphasizes everyone’s role. We resolve concerns in a group setting and create excitement around the benefits of BDD.

Follow up and measure​

As with any large initiative, it is important to follow up with teams and plot their progress along leading measures. Some teams have more challenging domains than others. This is normal and we offer tailored coaching to help those teams overcome inertia and tough challenges.