Human Formation Is Becoming the Primary Constraint on Trustworthy AI Capability

Signal Intelligence Briefing 002

Answer first: Human formation is becoming the primary constraint on trustworthy AI capability because AI now makes fluent outputs easier to produce while making judgement, responsibility, and discernment harder to verify. Technical AI skill still matters, but skill without formation is fragile: people and institutions need the capacity to decide when, how, and whether AI-generated work should be trusted.

Machine intelligence is reducing the cost of cognitive output. It is not reducing the need for human judgement.

The emerging constraint is no longer access to information, content, or technical tools. It is the formation of the human being using them: judgement, discernment, restraint, responsibility, and readiness under uncertainty.

The evidence supports the core signal, with one important qualification: technical AI skill still matters. But technical skill without formation is increasingly fragile (Dell’Acqua et al., 2023; Microsoft, 2024; UNESCO, 2023; World Economic Forum, 2025; PwC, 2025).

What is the purpose of this briefing?

Primary goal

To examine whether human formation is becoming the rate-limiting factor in trustworthy AI capability.

Core strategic premise

As AI becomes more capable, the decisive question shifts from Can people use the tools? to Can people judge when, how, and whether to trust what the tools produce?

What this briefing is not

This is not an argument against AI skills, technical fluency, productivity, or adoption.

Desired reader outcome

Readers should leave with a sharper distinction between AI use, AI skill, and AI-era human capability.

What are the main signals?

Signal 1: AI raises the value of judgement rather than replacing it

International evidence strongly supports the claim that AI creates new demands for human discernment, especially where tasks are ambiguous or consequential (Dell’Acqua et al., 2023; OECD, 2025; UNESCO, 2023).

Signal 2: Technical AI skill remains valuable, but insufficient

Labour-market and workplace evidence still shows strong demand for AI skills and measurable productivity gains from AI use, so the stronger claim is not replacement but dependency: technical skill requires judgement to be trustworthy (Microsoft, 2024; OECD, 2025; World Economic Forum, 2025; PwC, 2025).

Signal 3: AI amplifies existing capability gaps

Experimental evidence suggests that AI performance gains depend on task fit, user expertise and the ability to recognise when a task sits outside the tool’s effective frontier (Dell’Acqua et al., 2023; Otis et al., 2023).

Signal 4: Assessment systems are exposed

If AI can produce visible outputs, then output alone becomes weaker evidence of human capability. NZQA’s current architecture already recognises judgement at higher levels, while current AI guidance focuses mainly on academic integrity, authenticity and moderation (NZQA, 2025a; NZQA, 2025b).

Signal 5: New Zealand has ingredients, not yet a coherent architecture

TEC contains formation-adjacent language around adaptability, transferable skills, learner success and confidence, while NZQA contains partial judgement architecture. These pieces have not yet cohered around AI-era formation (Ministry of Education, 2025; TEC, 2026a; TEC, 2026b; NZQA, 2025a; NZQA, 2025c; NZQA, 2026).

Why does human formation matter for trustworthy AI capability?

AI has changed the evidentiary environment.

Historically, many systems treated output as evidence of capability. A written response, assessment task, report, design, plan, analysis or recommendation could reasonably indicate what a person knew, understood, or could do.

That assumption is weakening.

When machine systems can generate fluent output, the deeper question becomes:

What evidence do we have that the human can judge, reason, adapt, and take responsibility?

This matters for education, workforce capability, assessment integrity, professional practice, organisational governance, AI safety, equity, and credential trust.

Poor judgement now scales faster. Misframing travels further. Incoherence becomes more productive-looking before it becomes visible as failure.

What is the current system reality?

Most AI adoption activity still concentrates on visible and trainable layers:

• tool access
• prompt skill
• productivity gains
• policy compliance
• AI literacy
• workflow automation

These are necessary. They are not sufficient.

The international evidence shows a more complex pattern: AI can improve performance where the user understands the task, domain, limits and context, but degrade performance where the user cannot locate the tool’s boundary (Dell’Acqua et al., 2023; Microsoft, 2024; Otis et al., 2023).

This creates a new capability problem: the better AI becomes at producing convincing outputs, the more important human judgement becomes in deciding what those outputs mean.

What may the system be misunderstanding?

The dominant misunderstanding is sequence.

Many systems assume:

Access -> Use -> Skill -> Capability -> Judgement

The emerging evidence suggests a different order:

Formation -> Judgement -> Responsible Use -> Capability -> Trustworthy Output

AI does not simply add capability. It amplifies what is already present.

• Where there is clarity, it accelerates clarity.
• Where there is confusion, it accelerates confusion.
• Where there is weak judgement, it may produce confident error.

This is why formation cannot sit downstream of adoption. It is an upstream condition.

What adaptation patterns are emerging?

Pattern 1: From output to process

Assessment and capability systems are beginning to move away from final product alone and toward process, reasoning, reflection, oral defence, situated judgement and evidence of decision-making. NZQA and Ako Aotearoa guidance both highlight this pressure clearly (NZQA, 2025b; Ako Aotearoa, 2025a; Ako Aotearoa, 2025b).

Pattern 2: From skill lists to capability ecology

TEC’s policy language already includes transferable skills, adaptability, confidence, learner success and holistic support, but these are not yet explicitly connected to AI-era judgement and formation (Ministry of Education, 2025; TEC, 2026a; TEC, 2026b).

Pattern 3: From AI literacy to AI judgement

Basic AI literacy asks: Can you use the tool? AI judgement asks: Can you evaluate what the tool gives you, recognise its limits, and decide responsibly?

Pattern 4: From governance as policy to governance as formation

Responsible AI governance depends on the quality of human judgement inside institutions, not merely the existence of policy documents (UNESCO, 2023; OECD, 2025a; OECD, 2025b).

What are the strategic implications?

For educators

AI-era learning design must make thinking visible. If learners can produce fluent work without developing judgement, education risks certifying performance without capability (Ako Aotearoa, 2025a; NZQA, 2025b). This connects directly with the shift toward visible thinking as evidence of reasoning, not merely as an instructional activity.

For assessment designers

Assessment must increasingly verify process, reasoning, judgement and responsible use, not just output (Ako Aotearoa, 2025a; Ako Aotearoa, 2025b; NZQA, 2025b). This is why AI creates a capability trust problem, not only an academic integrity problem.

For employers

Hiring and workforce development should distinguish between AI fluency and AI judgement. The valuable worker is not only the one who can use AI, but the one who can challenge it (Dell’Acqua et al., 2023; Microsoft, 2024; OECD, 2025).

For policy makers

Capability frameworks need language for judgement, readiness, discernment and responsible agency. Without naming these, systems struggle to fund, measure or require them (OECD, 2025a; OECD, 2025b; UNESCO, 2023).

For New Zealand

The evidence suggests a follow-up signal: New Zealand has many relevant components, especially through TEC’s learner success orientation, NZQA’s higher-level judgement descriptors, programme approval criteria and the iQAF reform window, but these have not yet cohered into an explicit AI-era capability architecture (Ministry of Education, 2025; TEC, 2026a; TEC, 2026b; NZQA, 2025a; NZQA, 2025b; NZQA, 2025c; NZQA, 2026).

What intervention options become visible?

• Define AI capability as judgement-mediated capability. Not tool use alone. Not policy compliance alone.
• Embed judgement into capability frameworks. Especially where AI-mediated work is likely.
• Redesign assessment around visible thinking. Require evidence of reasoning, process, reflection, context and decision-making.
• Treat AI as a mirror before treating it as an accelerator. Use AI to reveal assumptions, gaps and reasoning quality.
• Develop formation-aware micro-credentials. Micro-credentials should not only certify tool exposure; they should evidence responsible capability.
• Create sector-level AI readiness diagnostics. Especially for tertiary providers, tutors, assessors and workforce capability teams.

What principles should guide trustworthy AI capability?

• Formation before acceleration
• Judgement before automation
• Process before product
• Capability before credential
• Readiness before deployment
• Governance through people, not only policy
• AI as amplifier, not substitute
• Human responsibility remains non-delegable

What should leaders watch next?

• Evidence of AI-related deskilling or over-reliance
• Changes to NZQA assessment and moderation guidance
• TEC language on AI readiness, learner success and future capability
• Employer demand for judgement, adaptability and critical thinking
• Micro-credential credibility in AI-mediated environments
• Sector movement from AI literacy toward AI judgement
• International policy linking AI, education and human flourishing

Key terms

• Human formation: the development of judgement, discernment, responsibility, restraint and readiness in the person using AI.
• Trustworthy AI capability: the ability to use AI in ways that can be judged, explained, bounded and responsibly trusted.
• AI judgement: the capacity to evaluate AI output, recognise limits, situate it in context and decide what should happen next.
• Capability trust: confidence that a person or system can demonstrate real capability beneath fluent AI-assisted output.

Conclusion: the future bottleneck is formation

The signal is clear.

AI capability is not simply a technical matter. As machine intelligence becomes more capable, the human layer becomes more consequential (OECD, 2025a; OECD, 2025b; UNESCO, 2023; Dell’Acqua et al., 2023).

The strongest organisations, education systems and policy environments will not be those that merely provide access to AI. They will be those that form people capable of using it with judgement.

The future bottleneck is not intelligence.

It is formation.

FAQ

Is human formation more important than technical AI skill?

Not instead of technical skill. Technical AI skill still matters. The stronger claim is that technical skill becomes trustworthy only when it is governed by judgement, discernment and responsibility.

Why does AI make judgement more important?

AI can produce fluent, plausible outputs quickly. That increases the value of the human capacity to test meaning, recognise limits, assess context and decide whether an output should be trusted.

What does this mean for education and assessment?

It means assessment systems need stronger evidence of process, reasoning and judgement. Output alone is becoming a weaker signal of capability in AI-assisted environments.

How does this connect to AI governance?

AI governance depends on the quality of human judgement inside the system. Policies matter, but policy cannot substitute for people who can interpret, challenge and take responsibility for AI-mediated work.

References

Academic and institutional references

• Ako Aotearoa. (2025a). AI, assessment, and academic integrity. Ako Aotearoa.
• Ako Aotearoa. (2025b). AI and assessment integrity. Ako Aotearoa.
• Dell’Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the jagged technological frontier. Harvard Business School Working Paper No. 24-013.
• Jaffe, S., Shah, N. P., Butler, J., Farach, A., Cambon, A., Hecht, B., Schwarz, M., & Teevan, J. (Eds.). (2024). Generative AI in real-world workplaces. Microsoft Research.
• Otis, N. G., Clarke, R., Delecourt, S., Holtz, D., & Koning, R. (2023). The uneven impact of generative AI on entrepreneurial performance. Harvard Business School Working Paper No. 24-042.
• Ministry of Education. (2025). Tertiary Education Strategy 2025-2030. New Zealand Government.
• New Zealand Qualifications Authority. (2025a). Level descriptors for the NZQCF. NZQA.
• New Zealand Qualifications Authority. (2025b). Academic integrity and artificial intelligence. NZQA.
• New Zealand Qualifications Authority. (2025c). Guidelines for programme approval and accreditation of New Zealand Certificates Levels 1-6 and New Zealand Diplomas Levels 5-7. NZQA.
• New Zealand Qualifications Authority. (2026). New tertiary quality assurance framework goes live. NZQA.
• Organisation for Economic Co-operation and Development. (2025a). Introducing the OECD AI Capability Indicators. OECD Publishing.
• Organisation for Economic Co-operation and Development. (2025b). Education for human flourishing: A conceptual framework. OECD Publishing.
• PwC. (2025). 2025 Global AI Jobs Barometer. PwC.
• Tertiary Education Commission. (2026a). Oritetanga Learner Success Framework. TEC.
• Tertiary Education Commission. (2026b). Learner Success: 7 Capabilities Diagram. TEC.
• World Economic Forum. (2025). The Future of Jobs Report 2025. World Economic Forum.
• UNESCO. (2023). Guidance for generative AI in education and research. UNESCO.

Practitioner and sector references

• Smith, G. (2026). AI capability and judgement. thisisgraeme.me.
• Smith, G. (2026). The capability gap: Why AI adoption is not the same as AI readiness. thisisgraeme.me.
• Smith, G. (2026). The judgement layer: AI. thisisgraeme.me.
• Smith, G. (2026). After work: What comes next is not automation, it is formation. thisisgraeme.me.
• Smith, G. (2026). AI, assessment, and the growing capability trust problem. thisisgraeme.me.