By Download the AI-Native Skills Assessment Template (PDF)
Download AI-Native Skills Assessment Template (PDF)
The PDF is a printable interviewer template for live screen-share sessions. It includes the full 60-90 minute session structure, example warm-up tasks, four applied build exercises, eval discussion questions, cost/tradeoff conversation prompts, the complete five-dimension rubric with level 1-4 criteria, a scorecard, placement thresholds, and an anti-cheating checklist. Free to use, no sign-up required.
TL;DR: AI-native skills assessment in 2026 has to test five dimensions: prompt iteration, tool fluency (Cursor, Claude Code, LangChain, Copilot), eval writing, AI-output verification, and cost / latency reasoning. Per Gartner’s 2026 AI Skills Framework, only 25-30% of senior engineers who list “AI tooling” on a resume demonstrate genuine fluency under live observation. Methodologies range from take-home portfolios (low signal, easily ghosted) to live IP-tagged screen-share builds (high signal, tamper-resistant).
Every IT staffing provider in 2026 claims AI-native engineers. The problem is that “AI-native” has no standard definition. Without an assessment methodology that produces direct, observable signals, the claim is unverifiable.
This article breaks down what an AI-native skills assessment should actually test, the methodologies that produce the strongest signals, scoring rubrics with concrete dimensions, and anti-cheating measures that stand up against AI-generated portfolios. For the broader vendor-evaluation context, see our checklist on how to evaluate IT staffing companies.
The Resume-to-Reality Gap
Per Gartner, 80% of senior software engineers in 2026 list at least one AI tool on their resume. Cursor, Claude Code, GitHub Copilot, LangChain, LangGraph, and LangSmith are the most common entries. Under live observation, roughly 25-30% of those engineers demonstrate fluency that goes past surface use. The gap is large enough that resume-based screening alone produces an unacceptable false-positive rate for any role where AI tooling is load-bearing.
The gap is widening rather than closing. Robert Half’s 2026 Hiring Quality Report finds that as AI tools became ubiquitous in 2025, candidates began listing them on resumes by default whether or not they had shipped a feature using them. The same report notes a parallel rise in AI-generated portfolio work, where the engineer can describe what was built but cannot reproduce the reasoning behind specific choices.
The implication for buyers is that AI tooling fluency cannot be inferred from artifacts. It has to be observed in real time, against a real task, with the engineer narrating their workflow. Anything else produces ambiguous signals at best and false confidence at worst.
The Five Dimensions of AI-Native Skill
Gartner’s 2026 AI Skills Framework defines AI-native engineering across five testable dimensions. Each dimension produces a Level 1 to Level 4 score, with Level 4 corresponding to senior-grade fluency. A senior AI engineer should test at Level 4 on all five.
Dimension one: prompt iteration. Can the engineer iterate on a prompt when the first answer is wrong? Level 1 candidates copy-paste a single prompt and accept whatever comes back. Level 4 candidates run a structured iteration loop: identify the failure mode, refine the prompt’s framing, add constraints or examples, and verify the new output against the original goal. Strong candidates can articulate why each iteration changed the output.
Dimension two: tool fluency. Can the engineer move fluently between two or three AI tools depending on the task? Level 1 candidates only use one tool (typically Copilot). Level 4 candidates show fluent use of Cursor or Claude Code for codebase-aware editing, a chat interface for exploratory reasoning, and at least one agent framework for multi-step automation. They have configured custom workflows, set up custom MCP servers or rules files, and can switch tools mid-task when the current one stalls.
Dimension three: eval writing. Can the engineer test an LLM-backed feature? Level 1 candidates do not write evals. Level 4 candidates write structured eval suites with reference inputs and expected outputs, score them with both deterministic checks and LLM-as-judge patterns, and track drift over time. The Gartner framework treats eval writing as the single strongest predictor of production-grade AI work.
Dimension four: AI-output verification. Does the engineer verify AI-generated code or accept it on plausibility? Level 1 candidates accept any output that compiles. Level 4 candidates run adversarial tests against AI output: edge cases, deliberate misuse, and security checks. They treat AI output as a draft, not a deliverable.
Dimension five: cost and latency reasoning. Does the engineer reason about provider costs and inference latency? Level 1 candidates do not consider it. Level 4 candidates model cost per request, latency budgets, and quality-cost tradeoffs across providers. They can articulate when to route to a smaller cheaper model versus a larger expensive one.
How to Test: Methodology by Signal Strength
Methodology choice determines signal strength. Forrester’s 2026 report ranks six common methodologies by signal strength (the predictive correlation between assessment score and 90-day placed-engineer output quality).
Resume claim. Signal strength: 10/100. The lowest reliable signal in the assessment stack. Useful only for filtering out candidates who have not encountered AI tools at all.
Take-home portfolio. Signal strength: 20/100. The candidate ships a finished project that uses AI tooling. The problem is reproducibility. Roughly 35% of take-home portfolios in 2026 contain AI-generated work the candidate cannot reproduce in live conversation per Robert Half.
Multiple-choice AI quiz. Signal strength: 30/100. Tests recognition memory, not applied skill. A candidate can score 85% on a LangChain knowledge quiz and still fail to ship a working LangChain feature in live observation.
Pair-programming with reviewer. Signal strength: 55/100. The candidate works through a problem with an assessor present. Stronger than take-home but generic across AI-native and non-AI roles. The assessor sees code but not the candidate’s AI workflow.
Live screen-share build. Signal strength: 80/100. The candidate shares their screen while building a small feature using AI tooling. The assessor sees the entire workflow: prompt iteration, tool switching, code review, debugging. This is the minimum bar for a credible AI-native assessment in 2026.
Live build with IP-tagged eval. Signal strength: 95/100. The strongest available methodology. The assessment is built around a custom problem that does not exist in public training data. The assessor watches the candidate’s full workflow and scores against pre-published rubric dimensions. Tamper-resistant because the candidate cannot pre-prepare an answer.
The Live Screen-Share Build: How It Works
A 60-90 minute live screen-share assessment is the operating standard at quality IT staffing providers in 2026. The session has four parts.
Part one: warm-up (10 minutes). The assessor opens with a low-stakes coding task using the candidate’s preferred AI tool. The goal is to verify the candidate’s normal workflow without performance anxiety distorting the signal. The assessor watches keyboard shortcuts, prompt patterns, and tool muscle memory.
Part two: applied build (35-45 minutes). The candidate is given a task that requires AI tooling to ship efficiently. Typical tasks: build a small RAG pipeline against a provided document set, add an LLM-backed feature to a partially-built repo, debug an existing agent that has a known failure mode. The candidate narrates their workflow throughout. The assessor scores against the five rubric dimensions.
Part three: eval discussion (15 minutes). The candidate is asked to describe how they would test the feature they just built. Strong candidates outline a structured eval suite with reference inputs, scoring rules, and drift detection. Weak candidates default to “I’d write some unit tests” without acknowledging the non-deterministic surface of LLM-backed features.
Part four: cost and tradeoff conversation (10 minutes). The assessor asks the candidate to estimate cost per request, latency budget, and where they would optimize. Strong candidates have intuition about provider pricing and can articulate quality-vs-cost tradeoffs. Weak candidates are unaware of cost considerations entirely.
The assessor produces a five-dimension scorecard at the end. Senior placement requires Level 4 on at least four of five dimensions; the fifth must be Level 3 or higher.
Anti-Cheating Measures Against AI-Generated Portfolios
The shift to AI-generated work products creates new spoofing surfaces. Quality providers run four anti-cheating measures.
Live observation as the default. Take-home portfolios are accepted as supporting evidence but never as primary assessment. The candidate has to reproduce the reasoning behind portfolio work in live conversation. Per SIA, the live-reproduction rate for AI-generated portfolio work is roughly 40%; for genuine work, it is above 90%.
IP-tagged custom problems. Assessment problems are custom-built and not published online. They do not appear in public training data for major LLMs. The candidate cannot solve them by pasting the problem into ChatGPT and copying the answer. Quality providers rotate the problem set every 60-90 days to prevent leakage.
Workflow scoring, not output scoring. The score is built from the candidate’s workflow (prompt iteration, tool fluency, eval discipline), not from the final code output. Even if the candidate’s AI tool produces correct code, weak workflow scoring blocks senior placement.
Identity verification. The candidate’s face is on camera throughout the assessment. The candidate’s resume photo and the live-video face are compared. The assessor verifies the candidate’s screen-share matches their declared editor and tool setup. Ghost-candidate substitution (where a recruiter or coach takes the assessment in place of the real engineer) is caught here.
For procurement teams running RFPs, asking providers to describe their anti-cheating measures is a useful filter. A provider that cannot describe specific measures is running a weaker assessment regardless of headline claims.
Tool-Specific Test Patterns
Each major AI tool has signature workflows worth observing. The strongest live assessments include at least one task that exercises each.
Cursor. Test the candidate’s use of Cmd-K (inline edit), agent mode for multi-file changes, and rules file configuration. Strong candidates have a custom rules file shaped to their team’s coding standards. Weak candidates use Cursor as a fancier autocomplete.
Claude Code. Test the candidate’s session management, plan-before-execute discipline, and use of subagents or tools. Strong candidates structure long sessions with explicit task planning. Weak candidates dump a vague prompt and hope.
GitHub Copilot. Test the candidate’s use of Copilot Chat versus tab-completion, and whether they distinguish autocomplete drafts from production-ready code. Strong candidates routinely overwrite Copilot suggestions; weak candidates accept them on plausibility.
LangChain or LangGraph. Test the candidate’s ability to build a small multi-step chain or graph, handle state, and add observability. Strong candidates configure LangSmith or an alternative trace tool. Weak candidates have copied tutorials but cannot reason about graph state transitions.
Custom MCP / agent frameworks. Test whether the candidate has built or configured an MCP server or a custom agent. This is the strongest senior-grade signal in 2026 because it requires both AI fluency and systems engineering instincts.
Scoring Rubric Discipline
Rubric discipline is the difference between a useful assessment and a vibes-based interview. Each dimension is scored Level 1-4 against pre-published criteria. The assessor records the moment in the session that earned the score (the specific prompt iteration, the specific tool switch, the specific eval pattern). A second assessor reviews the recording and the scorecard; inter-rater agreement above 80% is a quality signal per Gartner.
Common rubric failure modes: scoring the output instead of the workflow, scoring the candidate’s confidence instead of their competence, scoring AI familiarity instead of AI fluency, and drifting the bar to match the week’s applicant pool. All four are catchable in inter-rater review.
Quality providers publish their rubric (or a summary) on request. A provider that will not share rubric dimensions is running a less disciplined assessment than one that will.
Where AI-Native Assessment Is Going
Three shifts will play out in 2026-2027 per Gartner’s framework.
The agent-building bar rises. By 2027, Gartner expects senior placement to require demonstrated ability to build a small custom agent with tool use and memory, not just consume existing agent frameworks. Providers that have not added agent-building tasks to their assessment will fall behind.
Eval discipline becomes table stakes. Eval writing is currently a senior differentiator. By 2027 it will be a mid-level expectation. Providers should already be testing it at every seniority level.
Cost reasoning enters the rubric. Cost per request and latency budget reasoning, currently a senior-only test, will move into mid-level assessment as inference costs become a routine engineering concern.
For buyers, the implication is to ask providers how their assessment rubric is changing. Static rubrics that have not evolved since 2024 are a stale-funnel signal. For more context on the rest of the vetting funnel, see our companion article on how IT staffing vetting funnels work.
Start an Engagement With Live-Assessed AI-Native Engineers
Second Talent runs the AI tooling assessment as a 60-90 minute live screen-share session with IP-tagged custom problems and the five-dimension scoring rubric described above. Every engineer placed at the senior level has scored Level 4 on at least four of five dimensions. We re-run the assessment on a 12-month cycle to catch skill drift, and our rubric is published on request for any RFP participant.
Common starting points:
- How to evaluate IT staffing companies
- 15 questions to ask IT staffing providers
- AI Staffing & Augmentation Services overview
- Hire an LLM Engineer
- Hire an AI Automation Engineer
Matching in 24 hours. $0 upfront. Pay only when you make a hire.
