Reporting customer feedback with sample limitations

Report each feedback figure with its question, respondent group and collection route. Put the valid response count and main exclusions beside the score or percentage. Responses to a post-service invitation describe those respondents; without a suitable design and further checks, they do not establish the view of every customer.

State who could appear in the result

Start with the group the decision concerns, such as everyone who attempted an illustrative online address change during a defined period. Then show whom the feedback route could reach.

A survey offered only after submission excludes earlier exits. A support survey excludes people who never reached support. An open comment box may have no defined invitation population.

Use a label such as “Among valid responses to the post-submission question” and give the actual count. If the numbers eligible, invited or exposed are known, show them too. Do not calculate a response rate from page views or another count that does not match the invitation method.

Put a method note beside each figure

Include the exact question and answer options, task and dates covered, eligibility rule, invitation method, valid answer count and calculation. Explain how missing, “not sure” and inapplicable answers were handled. State the main groups the route missed.

For an effort result, say which end of the scale means easier. For a satisfaction percentage, identify which answers count as satisfied. A reader should be able to tell what the displayed value means without searching for a distant footnote.

Feedback Collection Method Comparison

Distinguish the limitations

Coverage: Some people had no opportunity to answer through the route. A completion-only invitation misses those who left earlier.

Nonresponse: Some invited people did not answer. Their experiences may differ from respondents’. A response rate is useful context but does not, by itself, establish whether nonresponse bias exists or how large it is.

Precision and interpretation: A small subgroup’s percentage can change substantially with only a few answers. Wording, scale, timing and collection mode can also affect a score. Do not attach a conventional probability-sample margin of sampling error to an opt-in feedback route merely because it reports a percentage.

Show the limitation beside the claim it qualifies. If an interval is proposed, the sampling design and analysis need appropriate statistical review. Otherwise, describe the observed respondents without false precision.

Pros and Cons of Opt-In Feedback Routes

Explain changes before comparing periods

Before calling a movement a trend, check whether the question, eligible group, invitation point, channel mix and calculation remained comparable. A new survey placement may hear from people previously missed while also breaking comparability with earlier results.

Mark such a change on a chart or report the periods separately. Explain which comparisons remain useful. For small groups, show counts with percentages so readers can judge how many answers underlie a movement.

Reporting Feedback with Sample Limitations – Checklist

Treat comments as accounts

Free-text feedback can describe a respondent’s experience or suggest a failure to investigate. Group comments using a clear theme definition, retain contrary accounts and protect personal details in shared reports. A count of comments about an issue is a count of submitted comments, not the proportion of all customers affected.

End with a bounded decision: what the evidence suggests checking, who will check it and what further result would change the decision. If the sample cannot answer the original question, state what additional evidence is needed.

Handling Free-Text Feedback: From Input to Action