Overview
A technical pipeline that supports my policy research: it gathers publicly
available university AI-policy documents, uses the Gemini API to extract
structured information from them, and stores the results in a consistent
schema — with a mandatory human-verification step before any record is used
in research.
Problem
Manually locating, reading and tabulating AI-policy provisions across dozens
of institutional websites is slow, error-prone and hard to repeat when policies
change. The research needed a repeatable way to build and refresh a structured
policy corpus without sacrificing accuracy.
My role
Designed and built the system: source identification, extraction scripts,
prompt design for structured extraction, output schema, and the verification
workflow.
System purpose
Turn scattered, inconsistently formatted public policy pages into a structured,
dated, verifiable data set that the policy-analysis projects can query and
re-run.
Workflow
- Source list — maintain a curated list of institutional policy URLs.
- Collection — fetch and archive page content with retrieval dates,
respecting robots.txt and request-rate limits.
- Extraction — send archived text to the Gemini API with a fixed schema
prompt that extracts fields such as permission model, disclosure rules and
defined terms, returning structured JSON.
- Validation — machine-check the JSON against the schema; flag missing or
low-confidence fields.
- Human verification — every extracted record is checked against the
source document by a human before it enters the research data set; the
verification status is stored on the record.
- Export — verified records feed the coding matrix used in the
policy-analysis project.
Technology used
- Python — pipeline orchestration and data handling
- Web extraction — HTTP retrieval and HTML-to-text processing of public pages
- Gemini API — schema-guided extraction of policy fields from document text
- JSON schema validation — structural checks on all model output
- Spreadsheet / CSV export — for the human coding and verification layer
Structured outputs
Each record captures: institution, document title and URL, retrieval date,
policy scope, permission model, disclosure requirements, verification
expectations, consequence provisions, defined terms, and verification status
(unverified / verified / corrected).
Human-verification process
Model-extracted values are treated as claims, not data. A record only enters
the analysis after a human has compared each extracted field with the source
text. Corrections are logged, which also produces an error profile of the
extraction step itself.
Findings
Formal extraction-accuracy figures have not yet been measured and published.
The mandatory human-verification step was the design priority over raw
automation accuracy: no unverified, model-extracted value has ever been used
in the research itself, so the system’s reliability rests on that check, not
on an unmeasured accuracy claim.
Outputs
- Structured, dated policy corpus supporting the policy-analysis project
- Reusable extraction schema and prompt set
Impact
Its impact so far is internal: it is the data-collection backbone behind the
policy-analysis projects on this site.
Risks and limitations
- LLM extraction can misread, omit or hallucinate provisions; this is why
unverified records are never used in analysis.
- Public web pages may lag the authoritative internal policy version.
- Site structures change, so collectors require maintenance.
- Automated collection is limited to public documents and polite request
behaviour; no access-restricted material is gathered.
Lessons learned
- Constraining the model to a strict output schema, and validating it,
eliminates a large share of unusable output before human review.
- Logging human corrections turns verification effort into measurable evidence
about where LLM extraction fails.
Not published as a standalone paper — it is documented here as the data
infrastructure behind the policy-analysis projects.
Downloadable materials
Schema documentation is available on request while the pipeline is still
under active development.