Research question and scope
This protocol measures how often a delegated inbox reply follows the approved tone guide. The unit is one outbound reply tied to one customer case. A compliant reply follows the guide's opening, next-step, and promise rules and contains no banned phrase. The measure describes writing discipline, not whether the answer was correct.
Set the scope to one inbox, one client, and a fixed period. Record the tone guide version and the approved examples in force. If the guide changes during the period, split the sample at the effective date. A different guide creates a different denominator.
Keep the measure separate from a customer satisfaction score. A polite reply can still give the wrong answer, and a blunt reply can resolve the case. The purpose is to test whether the written voice is being applied. Report satisfaction and accuracy separately.
State the decision the result will support before collection begins. The client may use it to rewrite a template, add a saved reply, or change the review sample. A measure with no decision becomes a monthly number nobody uses. Write the intended action beside the scope.
Name the audience and the review date before sampling. A report for the client service lead may need different detail from a report for the writing team. Record the audience and the meeting. A result with no scheduled review is easy to postpone.
Population and unit of analysis
Define the population as every outbound reply sent from the inbox during the period. Include replies to new questions, follow-ups, and internal forwards that reach a customer. The unit is the reply, not the case or the thread. A case with three replies is three units when each reply carries its own tone.
Exclude automated acknowledgements and system notifications that the desk did not write. Mark them as excluded and report the count. Mixing machine text into a tone measure makes the result look better or worse for reasons the team cannot control. The report should show the exclusion so the reader can judge the mix.
Include replies sent from a shared mailbox by any team member. Record the sender role and the queue. Leaving out a shared mailbox hides a large part of the customer experience. State the inclusion rule before drawing the sample.
Decide how to treat a reply that was edited after sending. Score the version the customer received, not the later draft. Record the edit time and the change. A corrected reply is still the message the customer read.
Decide whether a reply with an attachment-only body counts. Include it when the body contains a written message, and exclude a bare file transfer with no prose. Record the rule and the example. The boundary should be stable across reviewers.
Definitions that make the measure reproducible
Define a compliant reply as one that meets all four checks in the guide: opening, clear next step, promise check, and no banned phrase. Define a partial reply as one that fails exactly one check. Define a noncompliant reply as one that fails two or more. Publish one example of each before sampling.
Define the opening rule by the guide's first-line pattern, not by a general impression of warmth. If the guide allows two openings, list both. Record the one used. A reviewer should not have to infer the rule from the reply.
Define the next step as a sentence that states what happens, who does it, and when. A closing such as we will be in touch does not meet the rule. Record the sentence and the missing element. The next step is the part customers most often say was unclear.
Define a promise as any statement about money, timing, delivery, or outcome. A promise check passes only when the fact was verified in the source system. Record the verification reference and the time. A warm promise without a check is a tone failure with a business consequence.
Agree how to score a reply that combines a correct answer with a banned phrase. Score it as partial or noncompliant according to the failure count, then note the banned phrase separately. Record the phrase and the replacement. The separate note helps the next training session.
Sampling and data collection
Draw the sample from the sent-folder export for the full period. Record the export date, the filter, and the row count. Do not build the sample from replies the client praised or complained about. A selected sample cannot show the everyday voice.
Capture the reply identifier, the case, the queue, the sender role, the send time, and the four check results. Add the guide version and any edit time. Store the export with the report. The reply text and the guide version are the core evidence.
Check a small overlap sample with a second reviewer. If the reviewers disagree about whether an opening is warm enough, the rule is too vague. Rewrite the rule and retest before the full sample. Agreement matters more than sample size.
Record the time spent scoring each reply. A report that requires reading every thread will not be repeated. Note whether the reply tool exposes the sent text and the guide version. The collection cost belongs beside the result.
Take a pilot of twenty replies across two queues before the full draw. Use the pilot to test the definitions and the export fields. Record what was missing and how long each reply took. A pilot that takes ten minutes per reply is a warning about the full sample.
Measuring tone compliance
Report the share of replies that are compliant, partial, and noncompliant. Show the counts behind each share. The three groups describe different levels of adherence and should not be collapsed. A single compliance rate hides the reply that missed only the next step.
Break the result down by queue, sender role, and reply type. A queue with a weak template may show a lower rate than a queue with a careful writer. Compare only groups with the same guide version and a similar volume. A rate without the group size is hard to interpret.
Report the most common failed check and the most common banned phrase. A missing next step may point to a template, while a banned phrase may point to a habit. Record one example for each. The pattern should lead to one specific fix.
Report the share of replies that were edited after sending. A high edit rate suggests the writer knew the rule but sent before applying it. Record the median time to edit. The edit is a recovery step, not compliance at send time.
Compare the tone result with the promise-check result as two separate columns. A team can sound consistent while making unverified promises, or sound rough while checking every fact. Report both together. The pair describes the risk better than either measure alone.
Limitations and next action
The measure cannot show whether the customer understood or liked the reply. It also cannot show a tone rule that is itself wrong for the audience. Treat the result as a measure of adherence to the approved guide. Note the queues and reply types that were not included.
Give the result to the service lead and the client contact with one change to test. The change might be a saved reply, a shorter check, or a weekly sample. Record the effective date and repeat the measure with the same definitions. A controlled change is easier to judge.
Review the boundary cases before tightening the guide. A few replies to angry customers may legitimately depart from the standard opening. Fix the exception list first if that is the pattern. The goal is a recognizable voice, not a lower failure count by definition.
Schedule the next measurement for the same week in the following month. Keep the queue and reply-type filters unchanged so the comparison is fair. Record any guide change in the note. A trend built from different rules is not a trend.
