Research question and scope

This protocol asks whether a pre-hire work sample predicts the quality and independence a new outsourced hire demonstrates in the first ninety days. The unit is one hired specialist with a scored sample and a complete ninety-day record. The study compares the sample dimensions with later evidence such as accuracy, rework, escalation behavior, and review turnaround. It does not evaluate the person's worth, and it does not assume that a single hiring method explains later performance.

Set the scope to one role family, one hiring process version, and one review rubric. Record the sample task, the time limit, the tools provided, and the reviewers who scored it. If the rubric changed during the window, keep the earlier definition for the full cohort or analyze the change as a separate period. A rubric change can look like a change in candidate quality when it is only a change in the standard.

Population and unit of analysis

Define the population as all specialists hired into the role family during the window who completed both the sample and the ninety-day review. The unit is the hired specialist, identified by an internal study code rather than a name. A specialist who changed roles during the window remains one unit with a role-change note. A specialist who left before ninety days is included with a reason category, not dropped.

Record the sample score, the reviewer, the hiring decision, and the start date. Add the ninety-day quality score, the rework rate, the escalation count, and the number of items completed without help. These strata matter because a specialist who handles simple volume and one who handles exceptions should not be compared on the same raw count. Test the unit definition on two ordinary hires before collection begins.

Definitions that make the measure reproducible

Define sample score as the rubric total recorded before the hiring decision, not a score reconstructed afterward. Define ninety-day quality as the share of reviewed items that passed the written standard on the first review. Define rework as an item returned for a correction that the specialist could have prevented with the information available. Define independence as the share of items completed without a follow-up question to the supervisor.

Do not treat a low volume as high quality, and do not treat a high volume as low quality without the review denominator. Do not count an escalation that the written rule required as a failure of independence. Separate a policy escalation from a confusion escalation, because the first is the desired behavior and the second is a training signal. Each definition should include one positive example and one boundary example.

Record the sample conditions as part of the definition, because a candidate who completed the task with a template is not comparable to one who started from a blank page. The same rubric applied to different conditions produces a score that looks precise but measures different things. Note the tools, the reference material, and the time limit for each hire. If the conditions changed, analyze the cohort in two groups rather than pooling them.

Evidence standard and data dictionary

For each hire, capture the sample task, the rubric version, the dimension scores, the reviewer, and the hiring decision. Add the start date, the role, the first review date, and the ninety-day review date. Record the quality denominator, the rework count, the escalation count by type, and the follow-up question count. Code each field as present, missing, or not applicable.

Also record whether the sample was paid, whether it used the real tools, and whether the candidate received the same instructions as the comparison group. These process fields help explain a weak correlation that comes from a poorly designed sample rather than from the candidate. If a ninety-day measure was not collected, mark it missing rather than substituting a manager's impression. A second coder should be able to reproduce the classification from the packet.

Sampling and collection

Include every eligible hire in the window rather than selecting the strongest or weakest cases. If the cohort is too large for manual review, draw a random sample and record the selection rule. Keep the sample scores in a separate file from the ninety-day scores until both are complete, so the reviewer is not anchored by the earlier result. Record the date the two files were merged.

Collect the ninety-day evidence from the queue and review systems rather than from a retrospective rating. If the team uses a manager rating, record the rating and the specific examples behind it. Do not let a manager revise the sample score after seeing the new hire's performance. Note the collection date separately from the start date so the reader can see how much of the record has aged.

Analysis and reporting

Report the correlation between each sample dimension and each ninety-day measure, along with the number of hires in the cohort. Show the scatter or a simple table of high and low sample scores against high and low performance. Present the results by role and by reviewer rather than a single company-wide number. Include the hires with missing data and explain how they were treated.

Look for a dimension that predicts one outcome but not another, such as accuracy predicting quality but not independence. That pattern is more useful than a single overall correlation because it tells the hiring team what the sample actually measures. Do not report a correlation as a cause, and do not use a small cohort to make a confident claim. State the uncertainty plainly and show the range of plausible results.

A useful check is to compare the sample score with the reviewer's independent ninety-day judgment as well as with the system measures. If the two later measures disagree, the system measure may be too narrow or the manager may be weighting different behavior. Report both and describe the disagreement rather than choosing the one that supports the hiring decision. That transparency makes the validation more credible.

Limitations and sources of error

The sample cannot measure motivation, reliability, or fit with a particular supervisor. It can measure only the work produced under the sample conditions. A new hire's performance is also shaped by onboarding, access, queue difficulty, and management, so a weak correlation does not prove the sample is useless. A strong correlation does not prove the sample caused the performance.

The cohort may be too small to support a stable estimate, and the reviewers may share a bias toward candidates who communicate in a familiar style. If the same reviewer scored the sample and the ninety-day review, note the independence limitation and analyze that subgroup separately. Hiring records also contain sensitive personal data, so report only aggregated results and study codes. Attrition is another source of bias, because hires who leave before ninety days may be the ones the sample failed to predict. The report should describe that group separately so the main estimate is not distorted by hires whose outcome is unknown.

Review cadence and next action

Give the result to the hiring owner and the quality lead and ask them to choose one change to the sample or the rubric. The change might be a new task, a clearer scoring anchor, or a requirement that two reviewers score independently. Record the owner, the effective date, and the measure that will show whether the change improved prediction. Avoid changing the task and the rubric at the same time.

Repeat the protocol with the next cohort and compare the same dimensions. If prediction improves, keep the change and document the evidence. If it does not, inspect whether the ninety-day measures are stable enough to predict. Close the review when the owner accepts the finding and the next cohort is scheduled.

Keep the original sample task and rubric in an archive so a later cohort can be compared with the same instrument. If the task is changed, treat the change as a new validation rather than assuming the old correlation still applies. Record the date of the change and the reason. The archive also protects the process when the hiring manager changes.