RepoSage: AI-Assisted Repository Audit Tool

developer-toolingautomationrag-llm

Deterministic codebase audit tool: structured evidence, not vibes.

Zero runtime dependencies, zero API keys: 29 languages, 32 frameworks, 10 manifest parsers, all deterministic

Heuristics are signal, not proof: every check is file-presence or regex-based, with no semantic understanding unless you opt into the LLM layer. And LLM output is schema-constrained but not fact-checked against the deterministic report.

Situation

Teams want quick codebase audits, but 'chat with your repo' tools give fluent but ungrounded output: no deterministic evidence you can put in a CI gate, a PR review, or a ticket.

Task

A scanner that emits structured, evidence-backed audit reports (Markdown or JSON), reproducible run-to-run, useful with or without an LLM, and safe to drop into CI.

Action

A stdlib-only deterministic core: filesystem walk → language and framework detection (29 languages, 32 frameworks) → dependency manifest parsing (10 formats) → quality heuristics (tests, docs, CI, typing, lint) → architecture and risk analysis, all assembled into an immutable typed report rendered to Markdown or JSON.
Optional AI enrichment (Anthropic or OpenAI) plugs in at a single one-method Protocol seam, forced through a JSON tool-use schema. It can annotate the already-complete report but never restructure it. No API key means a clean exit with the full deterministic report still emitted.
Ships three ways: a PyPI package (v0.3.0), a published GitHub Actions marketplace action, and an MCP server. All three drive the same pipeline.

How it works

The true pipeline: a stdlib-only deterministic core (scan → detect → parse → score → risk) builds an immutable report; the LLM is a dashed amber bolt-on that plugs in at a single one-method Protocol seam, never in the core path. An MCP server is an alternate entry to the same pipeline.

Result

Shipped to PyPI (v0.3.0) with a published GitHub Action and an MCP server. The real use cases: auditing every repo in this portfolio and doing instant health checks on vibecoded projects from non-technical collaborators, where the structured Markdown report replaces an hour of code archaeology. The meta proof: RepoSage audits itself and scores 100/100, with 179 tests, strict mypy, Ruff, docs, and CI all detected. The tool works, or it would say so.

Six scored quality checks producing an X/100 score, plus five risk rules — The quality score is six file-presence checks (tests, CI, docs, packaging, lint config, typing) summed to X/100, alongside five risk rules (no tests, no CI, god modules ≥200 lines, no docs, ≥25 dependencies). Every finding points at the evidence that produced it.

The LLM plugs in at one Protocol seam and is forced through a JSON schema — The whole AI surface is one structural Protocol: enrich(report) -> EnrichmentResult. The LLM receives a report that is already complete and is forced through a JSON tool-use schema, so it can annotate but never break the structure. The core and the AI share ten lines of interface and nothing else.

Learning

Deterministic heuristics first, LLM as enrichment, not the other way around. The tool produces a complete, byte-identical report with no API key; the LLM only annotates it. Forcing that enrichment through a JSON-schema tool call meant the AI layer could never break the report schema: the deterministic core and the AI layer share a single ten-line Protocol and nothing else.

Tech Stack

PythonGitHub ActionsOpenAI APIAnthropic APIMCP

Services

AI AutomationDeveloper ToolingCodebase AuditSoftware QualityTechnical Documentation

Status

Active