Data-Driven Decision Making – Product Management with Hypotheses

Source:-infoq.com

Key Takeaways
Stating Product Conjectures or Hypotheses in plain English is a common practice in Product Management. In order to make data-driven decisions using Hypotheses, they need to be formulated in a way that lends itself to automated evaluation.
Hypotheses can be formulated semi-formally using Capability / Outcome / Measurable Signal triples, which allows for a good mix of qualitative and quantitative specifications.
Automated evaluation of Hypotheses requires development capacity to be planned for software developers as part of feature implementation.
Automated evaluation of Hypotheses also requires software developers to possess Operations skills and apply them during feature implementation.
Documentation of Hypotheses needs to be done directly in the tool where user stories are specified.
Data-Driven Decision Making Series

The Data-Driven Decision Making Series provides an overview of how the three main activities in the software delivery – Product Management, Development and Operations – can be supported by data-driven decision making.

It consists of several articles, each highlighting an area where data-driven decision making can be applied:

In Product Management, Hypotheses can be used to steer the effectiveness of product decisions.
In Development, Continuous Delivery Indicators can be used to steer the efficiency of the development process.
In Operations, SRE’s SLIs and SLOs can be used to steer the reliability of services in production.
In the Series we also show how applying Hypotheses, CD Indicators and SRE’s SLIs / SLOs at the same time enables the software delivery organization to optimize for effectiveness, efficiency and service reliability in parallel.

All the articles in the Series are listed on Vladyslav Ukis’s InfoQ profile.

Introduction
Software product delivery organizations deliver complex software systems on an evermore frequent basis. The main activities involved in the software delivery are Product Management, Development and Operations (by this we really mean activities as opposed to separate siloed departments that we do not recommend). In each of the activities many decisions have to be made fast to advance the delivery. In Product Management, the decisions are about feature prioritization. In Development, it is about the efficiency of the development process. And in Operations, it is about reliability.

RELATED VENDOR CONTENT
Mastering Remote Meetings – Download the InfoQ eMag
Free Product Owner Learning Path
Free Scrum Master Learning Path
Conversation-Driven Development: Introduction to ChatOps
Digital Transformation Game Plan – Download Now (By O’Reilly)
RELATED SPONSOR

LaunchDarkly Feature Management Platform. Dynamically control the availability of application features to your users. Start Free Trial.

The decisions can be made based on the experience of the team members. Additionally, the decisions can be made based on data. This should lead to a more objective and transparent decision making process. Especially with the increasing speed of the delivery and the growing number of delivery teams, an organization’s ability to be transparent is an important means for everyone’s continuous alignment without time-consuming synchronization meetings.

In this article, we explore how the activities of Product Management can be supported by data from Hypotheses and how the data can be used for rapid data-driven decision making. This, in turn, leads to increased transparency and decreased politicization of the product delivery organization, ultimately supporting better business results such as user engagement with the software and accrued revenue.

We report on the application of Hypotheses in Product Management at Siemens Healthineers in a large-scale distributed software delivery organization consisting of 16 software delivery teams located in thee countries.

Process Indicators, Not People KPIs
In order to steer Product Management in a data-driven way, we need to have a way of expressing the main activities in Product Management using data. That data needs to be treated as Process Indicators of what is going on, rather than as People Key Performance Indicators (KPIs) used for people evaluation. This is important because if used for people evaluation, the people may be inclined to tweak the data to be evaluated in favorable terms.

It is important that this approach to the data being treated as Process Indicators instead of people evaluation KPIs be set by the leadership of the product delivery organization in order to achieve unskewed data quality and data evaluation.

Hypotheses
One of the central questions in Product Management is “what to build?” This is very important to get right as building software features that, once ready, do not get used by the customers is a total waste.

In order to approach the question of “what to build?” product delivery teams run small experiments to explore the customer needs, ideally in Production. Each experiment needs associated measurements that are used to either confirm or disprove initial assumptions.

This process is the subject of the Hypothesis Driven Development (HDD). It is well-described in How to Implement Hypothesis-Driven Development. In essence, an experiment is called Hypothesis in HDD and is described using a <Capability> / <Outcome> / <Measurable Signal> notation:

Hypothesis:
We believe that this <Capability>
Will result in this customer <Outcome>
We will know we have succeeded when we see this <Measurable Signal> in production

The definition of Hypothesis for a feature is done before the feature implementation begins. A product delivery team declares which <Capability> they want to put into the product to achieve a specific customer <Outcome>. The customer <Outcome> becomes evident when a defined <Measurable Signal> becomes visible in production.

With that the product delivery teams define upfront how they envision the features being used by customers in production. During the feature development, the teams additionally implement <Measurable Signals> to create instrumentation necessary to see how the features are being used in production. Finally, once the features have been deployed to production, the teams evaluate the actual <Measurable Signals> to understand whether the Hypothesis turned out to be true of false (Build → Measure → Learn).

After that, the process is repeated; based on the results of the first Hypothesis, the second one is formulated, implemented and measured, and so forth.

In our experience, the entire process also moves a product delivery team from being “just a feature factory” towards becoming responsible for delivering software that is actually being used by customers in production.

That is, the focus of the product delivery teams is set on the value they provide to the customers, as opposed to just counting features delivered to production.

A team that consistently works with Hypotheses navigates the customer problem domain effectively by optimizing for customer usage of the software based on data from production (automated feedback loop; every release is a scientific experiment). A team that does not work with Hypotheses just produces features and does not measure the feature usage by customers in production.

From Projects to Products
Today, many organizations transition from projects to products. Introducing Hypotheses can support the transition. This is because in projects, the requirement engineering is scoped for Development, where the project often ends, thus excluding Release and Operations aspects.

Stating a Hypothesis with a <Measurable Signal> as part of the requirement engineering process necessarily includes Release and Operations aspects (Dev + Ops = DevOps, so to speak). This way, the requirement engineering includes the evidence of user behavior, which is a step towards products and away from projects.

Additional Hypotheses-related aspects that help transit from projects to products are:

As the Hypotheses cover business success criteria to be achieved, the development team is becoming more business-driven without the need of a project.
The development team can run with less management involvement under the guidance of <Outcome> being measured using <Measurable Signals>. The team can do many releases on their own, which used to require separate projects.
The current state of the team is transparent with the values of <Measurable Signals>, which also guide the team decision making on the way to the <Outcome>.
Enablement
With the Indicators Framework defined, it was clear to us that its introduction to the organization of 16 dev teams could only be effective if sufficient support could be provided to the teams.

For the definition of Hypotheses we expanded our Business Feature template to include the <Capability>, <Outcome> and <Measurable Signals> fields.

We ran small workshops of 30 – 60 minutes with the teams where we took one requirement the team was about to take up and turned it into a Hypothesis. Later, once the requirement was deployed to production, we met the team and evaluated the <Measurable Signals>.

Adoption
We introduced the suggested Indicators Framework to an organization of 16 dev teams working on “teamplay” – a global digital service from the healthcare domain (more about “teamplay” can be learned at Adopting Continuous Delivery at teamplay, Siemens Healthineers). The teams got quite interested in Hypotheses right from the start.

It was easy for the teams to grasp the Hypotheses definition process. The discussions during the feature Hypotheses definition proved to be very valuable for further requirement engineering (using BDD). It was possible to hammer out the scope of the features very early on. A Hypothesis definition at this point created a nicely defined boundary, in which detailed user stories can be created later on.

An example Hypothesis definition from one of teamplay User Administration is:

Capability Outcome Measurable Signal
Enable user invitations by hospital admins
Admin User: ability to onboard non-admin users with appropriate user rights

Non-Admin User: ability to skip the registration process and access applications by just accepting an invitation from the hospital admin

Average users per hospital in 2020 > Average users per hospital in 2019 + 30%
> 10% of all hospitals registered before 31.12.2019 used the user invitation feature at least once in 2020
> 50% of all hospitals registered after 01.01.2020 used the user invitation feature at least once in 2020
Time between the hospital admin sent the invitation and the user accepted the invitation is < 1 week
When implemented, the first Measurable Signal values showed that it will take time until the usage of the feature increases to the point that was specified in the Hypothesis. However, there was qualitative feedback from Sales that customer onboarding using user invitations became significantly easier (unexpected qualitative Measurable Signal). Based on that feedback, the team can see over time whether the Hypothesis definition should be changed to reflect the usefulness of the feature to the customers as reported by the customers.

Another example Hypothesis from teamplay Data Access Management is:

Capability Outcome Measurable Signal
Data access by hospital location and department
Admin User: ability to enable users to see data appropriate to their scope of work

Non-Admin User: default focus on data from my hospital department to streamline my work

C-Level User: compare KPIs of different hospital departments

> 50% of hospital networks enabled Data Access Management
# of non-admin users in hospital networks having access to all data > 0
The implementation of measurable signals varied by team. A few teams implemented the measurable signals in production. With the planned increase of the frequency of team releases, we think that the teams will increasingly adopt the implementation of the measurable signals.

One of the teams had an interesting experience with Hypotheses. After a Hypothesis definition for a feature, the team started implementation. During the implementation, they encountered significant limitations in an external framework used. The limitations were so severe that it became clear that the <Capability> could not be implemented as intended and, therefore, the <Outcome> could not be achieved. The team started looking for another external framework.

Soon, we will be introducing a “teamplay Service Standard” – a short list of topics important to us in a service. Two topics there, will concern Hypotheses: “Establish data-driven prioritization for a service” and “Define what success looks like and iterate towards it”. With that, we are going to give the Hypotheses an additional push in the organization.

Prioritization
Our teams need more experience with Hypotheses in order to consistently use the data at hand as an input for prioritization. The data comes in different forms:

Positively / negatively tested Hypotheses
Unexpected insights from Measurable Signals
Now that the data is available, it needs to be taken into account by the dev teams, and especially product owners, to make the best prioritization decisions. The prioritization trade-offs are:

Invest in features to increase product effectiveness and / or
Invest in development efficiency and / or
Invest in service reliability
Summary
In summary, if a team optimizes their Product Management Process using Hypotheses, then the team is able to gradually optimize their ways of working in a data-driven way so that over time the team can achieve a state where they build features evidently being used by the users.

Hypotheses help depoliticize and enable transparency in the decision making process of the software delivery organization. Finally, it supports the organization in driving better business results, such as user engagement with the software and revenue.

This article is part of the Data-Driven Decision Making for Software Product Delivery Organizations Series. The Series provides an overview of how the three main activities in the software delivery – Product Management, Development and Operations – can be supported by data-driven decision making. Future articles will shed light on data-driven decision making in Development, Operations and combinations of data-driven decision making in Product Management, Development and Operations.