Posted in

GitHub Copilot CLI Detects Documentation Drift

by ailona•

Drasi used GitHub Copilot CLI in Dev Containers to run synthetic-user agents that execute tutorials literally, detect documentation drift and environment failures, capture artifacts, and auto-file issues. Weekly automated runs surface breaking changes and keep tutorials aligned with dependencies.

Drasi converted documentation testing into a monitoring problem using GitHub Copilot agents. The change automates tutorial validation and finds silent documentation drift quickly.

Main feature/change and impact

Drasi introduced synthetic users powered by the GitHub Copilot CLI and Dev Containers. The agent runs in isolated Dev Containers that mirror user environments. It executes tutorial commands literally and verifies outputs. Failures produce preserved artifacts and a machine-readable report. This converts documentation maintenance from manual checks into continuous automated validation and reduces undetected breakages.

Practical implications

Teams must adopt reproducible Dev Container definitions and CI gating. Tests run weekly and on-demand inside sandboxed containers. Each run captures screenshots, terminal logs, and a final markdown report. Non-determinism is handled with model escalation and semantic screenshot comparison. Failed runs open actionable issues automatically, removing the need for heavy manual QA on small teams.
This agent has three critical characteristics: It is naïve: It has no prior knowledge of Drasi—it knows only what is explicitly written in the tutorial. It is literal: It executes every command exactly as written. If a step is missing, it fails. It is unforgiving: It verifies every expected output.
Drasi’s approach forces explicit commands and clear verification steps in docs. Projects should run synthetic-user checks in CI and store Dev Container definitions in the repository. Next steps include expanding coverage, tightening prompt constraints, and adding maintainer approval gates for external pull requests.

Key points from the article:

  • Synthetic agents run tutorials exactly as written to find failures.
  • Dev Containers ensure test environments match user experiences.
  • Agents capture screenshots, logs, and reports for debugging.
  • Prompt engineering yields machine-actionable pass/fail signals.
  • Weekly automated runs detect breaking changes early.

    • Related Coverage:

    From the Microsoft Azure Blog