Two Evaluations. A Candidate Has to Clear Both.
Every developer on the marketplace sits a live coding evaluation, a separate review of how they actually work with AI tools, and reference checks — and doing well on one does not excuse doing badly on the other.
Three Checks, Kept Separate on Purpose
The first check is a live coding evaluation testing fundamental engineering judgement — debugging, design reasoning, honest code review — the same way any serious technical interview would run it, with no AI assistant open. The second is a distinct evaluation of AI-assisted work specifically: reviewing samples, watching how a candidate prompts and iterates, testing whether they catch confident-but-wrong output and whether they know when not to reach for the tool at all. The third is reference checks weighed by how much the reference actually knows.
None of the three substitutes for the others. A candidate who is excellent at prompting but shaky on fundamentals fails the first check regardless of how impressive the AI-assisted demo looked. A candidate who is a strong traditional engineer but has never learned to review AI output critically fails the second. Both have to clear, independently, before a profile reaches you.
Read our AI code review standardsWhat Each Check Actually Involves
Tailored to the role and stack rather than run identically for everyone. A backend candidate spends longer on a service under load; a frontend candidate spends longer on state and rendering bugs.
Live coding evaluation: fundamentals, no assistant in the room
The candidate debugs something genuinely broken, reasons through a design decision, and reviews a piece of code honestly rather than politely — all without an AI tool open. This is deliberate. The point is not that AI tools are off-limits in the job; it is that judging whether an AI tool’s output is correct requires the same underlying skill as writing correct code by hand, and that skill has to exist before it can be pointed at a machine’s suggestions.
We watch the reasoning, not just the result: whether they form a theory before changing anything, whether they read an error properly instead of skimming it, and what they do in the minutes after a first theory turns out to be wrong.
AI-assisted work review: prompting, iteration, and catching what is wrong
A separate session built entirely around how the candidate works with AI coding tools. We look at real work samples produced with assistance, and we watch them prompt and iterate live: how they scope a request, how they read what comes back, whether they accept a plausible-looking suggestion or interrogate it first.
Two things matter most here. Do they catch the kind of confident, well-formatted, subtly wrong output these tools produce — a broken auth check, an edge case quietly dropped, a hallucinated method on a real library? And do they know when the honest answer is to close the assistant and write the code themselves, rather than iterating on a prompt for something a human should just type?
Reference checks: weighed, not just collected
We verify work history and take references where they exist and are meaningful. A former manager or technical lead who genuinely reviewed the candidate’s work for months is worth reading carefully. A friendly peer supplied by the candidate is worth very little, and we weigh it accordingly rather than counting every reference the same. Where a claim about past work cannot reasonably be corroborated, that goes into the profile as a flag rather than being smoothed over.
Why We Do Not Average the Two Scores
Averaging is tempting because it produces one tidy number. It is also how you end up hiring the exact failure mode this whole process exists to catch.
A large strength hiding a real weakness
- A developer who prompts fluently but cannot tell correct code from confident-looking code, scored high enough overall to clear.
- A strong traditional engineer who never learned to review AI output critically, carried by a fundamentals score that has nothing to do with that gap.
- Speed treated as a proxy for skill, because a fast demo is more memorable than a slow, careful one.
No weak half to hide behind
- Fundamentals solid enough to judge AI output on their own merits, independent of how impressive the AI-assisted work looked.
- Demonstrated judgement about when AI assistance helps and when it gets in the way, not just fluency with a prompt box.
- Two independent clearances, so a client can trust either half of the story without having to take our word on the other.
How the Checks Narrow a Pool
The shape of the process, not a measured result. Every figure below is illustrative — we do not publish a real pass rate.
Applications and approaches
Illustrative figure only — everyone who applies or is approached
Screening call and work history
Illustrative figure only — stack and experience claims checked against what they have shipped
Live coding evaluation and AI-assisted work review
Illustrative figure only — both checks described above, cleared independently
Reference checks and shortlist sent to a client
Illustrative figure only — the profiles that reach your inbox
Illustrative diagram showing the shape of the process. Proportions are drawn for clarity and are not a record of any real pass rate, because a number without an audited denominator behind it is marketing rather than evidence.
What Vetting Cannot Catch
A process that claims to eliminate risk is describing something other than hiring. Here is what ours genuinely does not see — which is also why the two-week replacement guarantee exists.
Questions About Vetting
What exactly happens in the live coding evaluation?
A session run the way any serious technical interview would be run, with no AI assistant open. Debugging something that is genuinely broken, reasoning through a design decision out loud, reviewing a piece of code honestly rather than politely. It tests the fundamentals a developer would still need if every AI tool vanished tomorrow, because those fundamentals are what tells an AI tool whether its own output is any good.
What does the AI-assisted work review actually look at?
A separate session, entirely about how the candidate works with AI tools rather than whether they can code without one. We look at real work samples, watch how they prompt and iterate, and specifically test two things: whether they catch AI output that looks right and is not, and whether they know when to close the assistant and write the code themselves. Fast and uncritical is not a pass here.
Why isn’t the score just averaged across both checks?
Because an average lets a large strength in one hide a real weakness in the other, and the weakness is the one that costs you later. A developer who is fast with an AI assistant but weak on fundamentals will ship confident, plausible, subtly wrong code at speed. A developer with strong fundamentals who never learned to review AI output critically will eventually rubber-stamp something dangerous. Clearing both independently is the only structure that rules out either failure mode.
Are the reference checks meaningful, or a formality?
We weigh them by quality rather than treating every reference as equal. A former manager or technical lead who actually reviewed the candidate’s work for months is worth reading carefully. A friendly peer arranged by the candidate is worth very little, and we say so rather than pretending otherwise. Where a claim about past work cannot be corroborated at all, that is flagged in the profile instead of quietly smoothed over.
What can this vetting process not catch?
Whether someone keeps reviewing AI output critically once nobody is watching, rather than just during an assessment they know is being scored. Long-term fit, behaviour under a rushed deadline, chemistry with your specific team, and how someone performs on a stack or a codebase they have never touched. A vetting process that claims to catch all of that is overselling itself, which is also why the two-week replacement window exists.
Do you re-vet developers as AI tools change?
We are not testing mastery of one specific tool, because whichever tool is sharpest this year will not necessarily be sharpest next year. The AI-assisted evaluation is built around transferable judgement — prompting, iterating, reviewing critically, knowing when not to use the tool at all — which holds up as the tools underneath it change. For longer engagements we would rather a client tell us if that judgement seems to be slipping than rely on a one-time score.
See What Clearing Both Checks Looks Like
Tell us about your project and get matched with developers who cleared both evaluations, within days.