Comparing feedback before and after an experience change

Compare feedback once you confirm both sets cover the same customer task, question and invitation point. Show who could answer in each period. A score that shifts after a redesign may reflect a changed experience, changed respondents or a changed way of asking.

Freeze the comparison question

Suppose a delivery-query service changes the status message shown after a customer asks where an order is. In that hypothetical, a question about whether the message explained the next step could be asked at that point in both periods. A question about whether the order arrived belongs later and answers a different question.

Keep the exact wording, response options, scale direction and reference period with each result. If the question changes from “How clear was the next step?” to “How satisfied were you with delivery?”, do not join the scores into one trend. Keep both versions and explain the break.

Check who entered each set

Record the task, channels, eligible customers, invitation trigger, dates, invitations and valid responses where known. A post-completion survey will miss people who abandoned the earlier step. If the redesign changes who reaches the survey, post-change responses may describe a different group even when the invitation itself is unchanged.

Compare the mix of relevant case types. A period with mostly routine queries may differ from one with many delayed orders. Keep results for meaningful groups visible when the counts permit; do not present a thin subgroup as a stable population estimate.

Free-text comments can point to a failure worth checking. Their count is a count of submitted comments, not a rate for all customers.

Read the change beside other evidence

Set the two periods side by side using the same calculation and show response counts. Include “not sure”, missing or inapplicable answers according to a stated rule. Then check authorised service records for the outcome the redesign was meant to improve. A clearer status message could receive better feedback while orders still arrive late; an accurate delivery outcome could coexist with a confusing message.

If feedback worsens, read relevant comments and cases before naming a cause. If it improves, look for contrary cases and people the invitation failed to reach. Changes in volume, channel mix, fulfilment conditions or another service rule may also explain movement. A before-and-after comparison alone cannot isolate the redesign’s effect.

State a conclusion the data can bear

If the evidence warrants it, a report could say that respondents invited after the delivery-query message rated its clarity more favourably in the later period, while noting any difference in case mix or question placement. It should not extend that finding to every customer without evidence for the wider claim.

End with the next decision: retain the change while checking delivery outcomes, revise wording for a group still struggling, or collect more comparable feedback. Preserve both survey versions and their collection rules so a later reviewer can understand the comparison.