Testing redesigned customer experiences

Test a redesigned experience against the customer’s complete task: from their starting point to a result they can recognise. Decide what should improve, trial the proposed route with relevant customers, and check the service behind it. A smoother screen or shorter interaction is useful evidence, but it does not prove the customer received the right outcome.

Decide what the test must answer

Describe the change and the decision it will inform. For a hypothetical appointment service, ask whether a revised rescheduling route helps people obtain a suitable new appointment and understand the change has taken effect. That is more precise than asking whether they like the new design.

Set the task boundary, intended customers and success condition. Include unsuccessful attempts: someone who cannot find an appointment or needs staff help still has a result worth recording. Note what the redesign could worsen, such as access to an alternative route or the accuracy of the final booking.

Choose a test that fits the decision

Use more than one check when the decision spans these questions. Observation can expose a confusing step, while authorised service records show whether a request was acted on. Feedback can describe the customer’s understanding.

For a whole service or an end-to-end journey with several parts, designing metrics that represent the complete experience can be difficult. Watching participants attempt a set of tasks then provides a direct check on whether they succeed across the journey.

Keep the conditions comparable

Record the version, dates, entry route, eligible group and task used in each test. Give participants a goal without directing them to the redesigned control.

Record whether they finished unaided, needed help, stopped or reached a result that could not yet be verified. In a live comparison, retain cases still awaiting an outcome.

Before interpreting a change in results, check what else changed. A new invitation point may bring different people into the feedback sample.

A policy change, seasonal demand or a different case mix may alter live results. If conditions differ, describe the findings separately or investigate the difference; a simple before-and-after movement is not proof of cause.

Include customers who may struggle with the usual route, such as people using different channels or access arrangements relevant to the service. A test that reaches only successful online customers cannot show whether those who left earlier can now complete the task.

When repeating a benchmark, do so periodically and compare results to see whether the service is becoming easier to use, provided the task and participant conditions remain clear.

If the service includes several tasks or transactions, observe people attempting a set of tasks rather than relying on a single performance measure. Keep the set focused: a small number of well-chosen tasks can provide clearer evidence than an overloaded session.

Key Considerations When Testing Redesigned Experiences

Check the service, not only the interaction

Follow a suitable test request beyond submission: did it reach the right team, could that team act on it, and did the customer receive an accurate status and the intended result? Use a safe test arrangement for consequential transactions and verify live outcomes only through authorised processes.

If a participant completes a redesigned form but the request enters the wrong queue, improving the form again will not fix that failure. If the service delivers the result but people cannot tell, the status or explanation may need work. Check these possible causes separately.

Select tasks that support a fair comparison

Choose tasks that are believable for participants and that most customers need to complete. Analytics can identify common tasks so testing reflects journeys that matter, not unusual edge cases.

A task should have a clear correct outcome and remain consistent when you repeat the test. Avoid requiring participants to log in or enter details they would not have access to; those requirements obstruct the test without helping answer its question.

Keep the task set manageable. As a rule of thumb, use no more than five tasks per participant, with up to 10 minutes for each task. Too many tasks can overwhelm participants and make their attempts harder to interpret.

Combine measures with observed task completion

Interpret observed task completion alongside performance metrics and other evidence relevant to the decision. For a multi-step service, also trace the request to its verified outcome.

Make a bounded release decision

Agree in advance who can decide to continue, revise, widen or stop the release. Give that person a short record of the customer outcome, observed difficulties, verified results, unknown cases and groups the test did not reach. Include any adverse effect and the action needed to protect affected customers.

A promising prototype may justify a limited pilot. A successful pilot may justify wider release with monitoring. Neither guarantees that the experience will work unchanged at scale. After release, revisit the same customer task and investigate new failures.