Build an AI competency framework

01What should an AI competency framework measure?#

Measure four things: judgment, direction, verification, and workflow design. Tool familiarity matters, but it expires quickly. A durable framework tests whether someone can choose an appropriate use case, give the system useful context, evaluate the result, and turn a successful experiment into repeatable work.

Do not make prompt cleverness a competency. Good prompting is one technique inside direction; it is not the outcome the organization needs.

02What do the four levels of AI fluency look like?#

Use four levels that describe increasing responsibility: aware, capable, integrated, and transformative. The progression moves from understanding approved use, to completing a task with review, to running a repeatable workflow, to redesigning work for others. Each level includes accountability, not just greater automation.

This adapts the useful shape of Zapier's public fluency rubric while changing the labels and adding explicit evidence. Zapier's current version also emphasizes accountability: defining good output before starting, evaluating it critically, and owning what ships.

03How should competency expectations differ by role?#

Keep the four competencies consistent and change the evidence by role. A recruiter, analyst, and executive should not be tested on the same workflow. Start with three recurring tasks in each role, then define what capable and integrated performance looks like for those tasks.

1. List the work

   Choose three recurring tasks where better speed, quality, or coverage would matter. Avoid hypothetical use cases.

2. Name the risk

   For each task, write what must not be exposed, assumed, or decided by the system alone.

3. Define evidence

   Describe an artifact someone can show: a tested workflow, a review checklist, a before-and-after measure, or a decision log.

4. Set the level

   Choose the level the role needs now. Not every job needs transformative fluency, and pretending otherwise makes the framework meaningless.

04How do you assess AI competency fairly?#

Use a short work sample and a structured conversation, not a confidence survey. Give the person a familiar task, let them use an approved AI tool, and ask them to explain their choices. Score the process and the review discipline alongside the final output.

• Did they define the outcome and quality bar before starting?
• Did they provide the minimum useful context without exposing restricted information?
• Did they notice what the system missed or invented?
• Could they explain where human judgment changed the result?
• Could another person repeat the workflow from their documentation?

Assess the slope as well as the snapshot. A person who can show what they tried, rejected, and improved may be a stronger bet than someone fluent in one fashionable tool.

05How do you turn the assessment into development?#

Move one level at a time through real work. Give each person one recurring task, one quality measure, one approved tool, and a review partner. Reassess the artifact after four weeks. The goal is not more AI activity; it is better work with visible accountability.

1. Week 1

   Baseline the task and agree on what good output means.

2. Week 2

   Run the task with AI, keeping the prompts, drafts, and corrections.

3. Week 3

   Repeat it, remove unnecessary steps, and document the review gate.

4. Week 4

   Compare the result with the baseline and decide whether to adopt, revise, or stop.

Questions people ask#

Should AI fluency be part of every job description?

Only if you can define what it means for that role. A generic requirement rewards confidence and tool vocabulary. A role-specific requirement names the workflows, judgment, and evidence expected.

Should employees be rated on how often they use AI?

No. Frequency encourages performative usage and ignores whether the work improved. Assess appropriate use, output quality, review discipline, and demonstrated impact on a real workflow.

Does everyone need to reach the highest level?

No. Most roles need capable or integrated fluency. Transformative fluency belongs to people redesigning work across a team or function and carrying the accountability that comes with it.

How often should the framework be updated?

Review the examples every six months and the underlying competencies annually. Tools change quickly; judgment, direction, verification, and workflow design should remain stable.

Sources#

1. 01
   How to measure your AI fluency-and move to the next level

   Zapier

   Zapier's public four-level rubric and its definition of fluency as effective, responsible, confident use in daily work.

2. 02
   Raising the AI fluency bar for every Zapier hire

   Zapier

   Zapier's updated emphasis on mindset, strategy, building, accountability, and evidence of a person's learning trajectory.

3. 03
   AI Risk Management Framework Core

   NIST

   Guidance on training, role clarity, operator proficiency, documented oversight, and defining the tasks an AI system supports.