Research question and scope
This protocol asks how consistently a distributed support team applies its priority labels to similar work. The unit is one labeled item, identified by the case, the label, the labeler, and the time of labeling. The study measures agreement between labelers and the share of labels that change before resolution. It does not measure whether the underlying work was important to the customer.
Set the scope to one queue, one label set, and one fixed window. Record the label definitions, the labeling interface, and the shifts in force at the start. If the definitions change during the window, keep the earlier version or analyze the change separately. A clearer label set can look like improved judgment when it is only a new rubric.
Population and unit of analysis
Define the population as every item labeled during the window, including items that were relabeled later. The unit is the labeling event, not the case, because a case can carry more than one label over its life. Record the first label and every change with its time and author. Link the events so the analysis can show a chain.
Exclude items that were never triaged and record the exclusion count. Include items labeled by a supervisor, because supervisory labels are part of the same system. Test the unit definition on several ordinary cases before collection begins. Keep the excluded items with the sample so the denominator is auditable.
Definitions that make the measure reproducible
Define agreement as two labelers assigning the same priority to the same item using the written definitions. Define a justified change as a relabel supported by a new fact or a written rule. Define drift as a change in the share of high-priority labels over time without a matching change in arrivals. Define an ambiguous item as one that fits two definitions at once.
Do not count a change that happens after the customer escalates as agreement, because the escalation is new information. Do not count a label as correct merely because it matches the final outcome; hindsight is not available to the labeler. Each definition needs one positive and one boundary example. Publish the definitions with the sample.
Evidence standard and data dictionary
For each event, capture the item, the label, the labeler, the time, the written definitions in force, and any note the labeler added. Add the first response time and whether the item was relabeled. Record the customer's original words when the label depends on them. Keep the packet with the case record so a later reviewer can replay the decision.
Also capture the labeler's shift and queue load at the time. A label applied at the end of a busy shift may reflect fatigue or time pressure rather than a different reading of the rule. If the label was applied by a rule engine, record that separately. A reviewer should be able to score the event without asking the labeler.
Sampling and collection
Draw the sample from the label history rather than from a list of disputed cases. Record the export date, the filters, the row count, and the selection rule. Include every labeler and every shift so the result is not dominated by one person or one busy period. A fixed interval is acceptable when the population is ordered by time.
Collect the label as it was applied, not as it looks after later changes. If an item was relabeled, record both events and the reason. Do not infer the original label from the final status. Note the collection date so the reader can see how much of the record has aged.
Analysis and reporting
Report the number of items, the share of items with more than one label, and the agreement rate for a sample labeled twice. Break the result down by labeler, shift, and label. Show the most common disagreements and the definitions that caused them. Use the same denominator for every rate.
Compare agreement on ordinary items with agreement on ambiguous items. A low overall rate may be driven by a small set of definitions that need repair. Report missing evidence separately. If the sample is small, show counts rather than percentages.
Limitations and sources of error
The record cannot show how a labeler would have classified an item under a different definition. It also cannot capture a verbal discussion that changed the label before it was recorded. Agreement between two people is not proof that the label is correct. The study measures consistency, not the quality of the priority model.
Label histories can be edited, and a relabel may overwrite the original value. If the same person labeled and reviewed the item, note the independence limitation. A small sample may miss rare disagreement patterns. Record these cases and their treatment rather than dropping them.
Review cadence and next action
Give the result to the queue owner and the labelers and ask for one change. The change might be a rewritten definition, a worked example for the ambiguous items, or a second-label check on high-priority cases. Record the owner, the effective date, and the measure that will show whether the change worked. Change one definition at a time.
Repeat the protocol after the change and compare the same fields. If agreement rose but response time did not, the labels may be more consistent without changing the work. If agreement rose only for one labeler, the training may be the cause. Close the review when the owner accepts the finding and the next sample is scheduled.
Label design and rubric testing
Priority labels fail when the definitions overlap. Test the rubric by asking two labelers to classify the same ambiguous cases before the live sample. If they cannot agree with the definitions in front of them, the rubric is the problem. Rewrite the overlapping definitions and test again.
Keep the number of priority levels small enough to remember. A five-level scale with no written boundaries produces more disagreement than a three-level scale with clear examples. The label set should match the decisions the queue actually makes. Extra levels that no one uses are noise.
Shift and workload effects
Label agreement can vary by shift and workload. Compare agreement during peak hours with agreement during quiet periods. If the difference is large, the queue may need a simpler rule at peak or a second check on high-priority items. Do not assume the labelers changed their standards.
Record the queue depth at the time of labeling. A labeler with forty open items may use a different threshold than one with five. That difference is a working condition, not necessarily a training gap. Report it with the result so the owner can decide whether to adjust staffing or the rule.
Reporting without individual scores
Report agreement by label and by case type rather than by person. The goal is to find the definitions that cause disagreement, not to rank labelers. If coaching is needed, use examples privately and include the workload context. A public score can make people hide uncertainty instead of recording it.
Show the share of items that were relabeled after new information. That measure separates genuine ambiguity from inconsistency. A high relabel rate with a clear reason may be healthy. A high relabel rate with no reason is a signal that the first label was rushed.
Replication and version control
Keep the label definitions, the interface version, and the sample query with the report. Repeat the measurement after the rubric change using the same fields. If the queue or the label set changes, publish a new baseline. A second reviewer should be able to reproduce the classification.
Set the next review date before closing the current report. Assign the queue owner and the labelers who will take part. Keep the earlier sample for comparison. Consistency is a trend, not a single reading.
Share the report with the labelers before the owner meeting. They often know which definition caused a disagreement and can suggest a better example. Record their explanation with the result rather than paraphrasing it into a score. The next sample should test whether the explanation changed the labeling in practice. Keep the earlier examples so the comparison is visible.
