Research question and scope
This protocol measures how often a bilingual reply needs a meaning correction before it is sent. The unit is one reviewed reply tied to one ticket. A meaning correction changes what the customer would understand about the amount, date, product, or next step. The measure describes the review step, not the writer's language ability.
Set the scope to one queue, one language pair, and a fixed period. Record the review trigger and the glossary version in force. If the trigger changes during the period, split the sample at the effective date. A different trigger creates a different denominator.
Keep the measure separate from a customer satisfaction score. A reply can be corrected and still please the customer, or pass review and still confuse them. The purpose is to test the review record. Report satisfaction separately.
Population and unit of analysis
Define the population as every customer-facing reply that went through the review step during the period. Include replies that were approved, corrected, and held for an approval. The unit is the reply, not the ticket or the word. A ticket with two customer replies is two units.
Include replies that were sent without review under a recorded exemption, and mark them as exempt. Exclude internal notes and template edits that never reached a customer. Report the excluded volume. A rate built only from corrected replies will overstate the error rate.
Exclude a reply that was rewritten from scratch after a product change, and record it separately. A full rewrite is not a correction of the original. State the exclusion count beside the denominator. Record the reason for each exclusion.
Definitions that make the measure reproducible
Define a meaning correction as a change to the amount, date, product, policy, or next step that the customer would act on. Define a tone correction as a change that affects courtesy but not the facts. Define a term correction as a glossary fix and a format correction as a layout change. Publish one example of each.
Define a reviewed reply as one with a reviewer name, a decision, and a time. A reply with a reviewer name but no decision is partial. A reply sent while the review was pending is unreviewed. The record should allow the three states to be separated.
Decide how to treat a correction made by the writer before the reviewer saw the reply. Count it as a self-correction and report it separately. It shows the writer caught the issue, but it does not test the review step. Record the rule so two reviewers agree.
Sampling and data collection
Draw the sample from the review log and the ticket system for the full period. Record the export date, the filters, and the row count. Include the exempt replies in a separate sample. A reviewed-only sample cannot show whether the trigger is right.
Capture the ticket identifier, the language pair, the writer, the reviewer, the review time, the decision, the correction category, and the send time. Add the glossary version where available. Store the export with the report. The correction category and the reviewer decision are the core evidence.
Check a small overlap sample with a second reviewer. If the reviewers disagree about whether a change is meaning or tone, the definition is too vague. Rewrite the rule and retest before the full sample. Agreement matters more than sample size.
Record the time between the review request and the send. A review that adds a day is a delivery risk even when the quality is good. Note the shifts where the reviewer was unavailable. The wait time belongs beside the correction rate.
Measuring meaning corrections
Report the share of reviewed replies with a meaning correction, a tone correction, a term correction, and no correction. Show the counts behind each share. The categories describe different problems and should not be collapsed. A single correction rate hides a glossary gap behind a tone edit.
Break the result down by language pair, queue, and writer tenure. A new writer may need more term corrections, while an experienced writer may show fewer but more serious meaning errors. Compare only groups with similar volume and product scope. Report the group sizes beside the rates.
Report the share of exempt replies that later needed a correction or a follow-up. The exempt path is where the trigger is tested. A high rate suggests the exemption is too broad. Include the reason for each exemption.
Report the median correction time and the share of replies sent after the promised response window. A review that protects meaning but misses the service promise is not yet a working control. Include the queue's stated response target. Record the target with the result so the gap is visible.
Limitations and next action
The measure cannot show whether a customer misunderstood a reply that passed review. It also cannot show a correction made verbally before the written reply. Treat the result as a measure of the review record. Note the language pairs and channels that were not included.
Give the result to the support lead and the quality owner with one change to test. The change might be a narrower exemption, a required glossary field, or a reviewer on the night shift. Record the effective date and repeat the measure with the same definitions. A controlled change is easier to judge.
Review the boundary cases before changing the trigger. A few product terms may explain a large share of meaning corrections. Fix the glossary first if that is the pattern. The goal is a reply the customer understands, not a lower correction count.
