How to quantify the success of online tools

This question comes from another client of ours who maintains a suite of online tools that help their constituents:

  1. Carry out relatively sophisticated calculations,
  2. Make significant business or operational decisions
  3. Encourage and facilitate multi-step processes

These tools represent (encapsulate) amazing value to their users, but measuring their impact is tricky; the question goes…

How do you know you have achieved what you wanted to achieve (for example, change user behaviour). I guess user stats derived from Google Analytics are the first and easiest step, but can you measure above and beyond that.

This is a great question – and one that troubles many groups. These types of tools can take a good level of investment to create, which makes measuring their impact additionally important. So what do we recommend…?

There are several ways that measurement can be implemented and monitored, we use a framework to describe these and decide which will be used. The framework breaks down the possible options into conceptual types, namely:

  1. Awareness
  2. Usage
  3. Estimated impact
  4. Calculated impact
  5. Qualitative research (internal and external)

As you might infer from the list, the approaches get more meaningful/valuable, and more expensive as you read down.

1. Awareness

Sometimes this is neglected – but I think awareness is an important measure in and of itself. A person’s awareness of an online tool demonstrates that the organisation behind it is actively delivering value, as well as providing options for use. Simply measuring views (of the tool pages and also supporting info pages) is one (simple) way of estimating awareness.

2. Usage

This is the most common measure and at least it demonstrates that a tool is being used, which in itself infers value. Within this category there can be varying levels of sophistication, for example we could simply be measuring number of activations, or we could be measuring number of completions. Note that these more sophisticated measurements will often require pre-planning and possibly implementation of limited, but specialist, functionality.

3. Estimated impact

This approach is one that is highly attainable, but not often implemented. The concept is that for each stage or tool output we can estimate a value in the real world. If for example, hypothetically, if we are promoting change – e.g. reduced carbon footprint, we might estimate that 30% of the people who took the trouble to step through the tool and work out what actions to take (or stop taking) actually did so. Because we also know what recommendations the tool made we can infer the impact by a relatively simple calculation. It’s not an exact science – but it is a very attainable approach to estimating impact, which can otherwise be a very expensive task.

4. Calculated impact

This approach builds on the estimated impact and is different in a couple of key ways. Firstly we attempt to verify what actions were taken (we might get this measurement as part of a process facilitation – we ask people to enter and track the impact of an action which proves that the actions did take place, e.g. a reduced power bill). Secondly we are more rigorous about the calculation process, for example we might measure and verify the steps, we could modify calculations based on feedback from the user, or we might verify a selected sample in the real world.

5. Qualitative research (internal & external)

This option is a little different and might be thought of as verification rather than measurement. By ‘internal’ I am referring to the process of polling actual users for feedback and more information via the website or tool, and by ‘external’ I mean via any other channel – e.g. over the phone or face to face (assuming the contact details are captured via the tool). In some ways this more qualitative approach can be taken at any level (1-4 above) and it is also has the widest range of possible associated costs and risks as it is not as leveragable.

Measuring Success of an Online Tool-01b forblog