Overview
Reading a university’s AI policy tells you what it says. It does not tell you
whether, in practice, a student, marker or integrity officer facing a specific
situation can actually work out what is allowed. This project tests that
directly: a fixed bank of realistic AI-use scenarios was applied to the full
public policy environment of 20 Australian universities, and each outcome was
classified according to whether the published documentation actually resolved
the case.
Problem
Australian universities have produced generative-AI rules at speed, but those
rules are frequently spread across several documents — a university-wide
policy, unit-level guidance, assessment instructions — that a student has to
reconcile themselves. Each document can look reasonable on its own while the
combination still leaves a specific, everyday question unanswered. Without
testing policies against realistic cases, neither students nor researchers can
tell how much of this ambiguity actually exists.
My role
Sole researcher: designed the scenario bank and classification scheme,
assembled and coded the policy environment for all 20 universities, and led
the reliability checking and analysis, under the supervision of Dr Jon Mason
at Charles Darwin University.
Research question
Does the combination of documents a university publishes reliably tell
someone whether a specific, realistic AI-assisted behaviour is acceptable,
prohibited or conditional — or does it leave the question open?
Data sources
The full public AI-governance environment for 20 Australian universities:
academic integrity policies and procedures, assessment policies, and
unit-level or student-facing generative-AI guidance, wherever each institution
chose to publish it.
Method
I built a fixed bank of realistic test scenarios spanning common forms of
AI-assisted student work — from fully transparent, verified use through to
undisclosed, substantially AI-produced submissions — and applied each scenario
to each university’s published policy environment using a structured
classification protocol. A meaningful portion of the classifications were
independently re-coded by a second coder, and disagreements were resolved by
protocol rather than by a single analyst’s judgement, so that the reliability
of the classification scheme itself could be checked and reported alongside
the substantive findings.
Findings
The classification is complete and reported in a manuscript currently under
peer review; I am holding back exact figures here until the published version
is out, rather than previewing results that are still in the review process.
In general terms: how clearly a behaviour was resolved varied enormously both
by university and by the type of behaviour — transparent, verified uses were
consistently the most clearly resolved, while more ambiguous or
partially-disclosed uses were frequently left undetermined by the published
rules. Independent double-coding showed strong agreement on whether a case was
resolved at all, which gives confidence in the classification scheme itself.
Full figures and the university-by-university breakdown will be added here
once the manuscript is published.
Outputs
- A manuscript reporting the full classification scheme and results —
currently under peer review. A link will be added once it is published;
the current draft is available on request.
- The classification protocol also underlies the case-outcome logic used in
the Case Portfolio published elsewhere on this site.
Impact
No externally verified impact to report yet, as the manuscript has not
completed peer review. The classification scheme is already informing the
design of the Academic Integrity Decision Framework published on this site.
Limitations
- The analysis covers publicly available documents only; it does not observe
actual enforcement practice or internal-only guidance.
- Policies change quickly; the corpus is a dated snapshot rather than a live
record.
- Classification involves interpretive judgement; the dual-coding step was
used to measure — not eliminate — that risk, and decision rules are
documented so the study can be replicated.
Lessons learned
- Institutions vary so widely in how they use terminology that a shared
classification vocabulary had to be built and tested before any cross-
institution comparison was meaningful.
- Measuring whether a case was resolved at all turned out to be at least as
informative as measuring what the resolution was — an unresolved case is a
governance gap regardless of which way a policy eventually leans.
Manuscript under peer review. I will link the published version here once it
is available — in the meantime, get in touch if you would like to see the
current draft.
Downloadable materials
Not yet published while the manuscript is under review. The full coding
matrix will be made available here after publication.