proofslip Get started

Open source · v0.1

Receipts for every
agent pull request.

An agent can write “done, tests passed” on a pull request that deleted the tests or ended the command with || true. proofslip reads GitHub’s own record and writes a receipt a founder or client can read.

The source lives at github.com/omsocll/proofslip. That repository may be private, so the receipts below stand on their own.

Compare the two demos · Browse the log

proofslip

Example · PR #4

ClaimedFound

  • All tests pass ✕ does not match. The test command ends with || true
  • Added tests ✕ does not match. Deleted tests/readme.test.ts

Doesn't match

What the pull request claimed, beside what GitHub recorded.

The check is green. The sentence might not be.

Coding agents open a pull request and report success in the description. Nothing in that sentence is checked against the diff or the CI logs. A reviewer who does not read the code has no short record of what was claimed, what actually ran, and what still needs a person.

How it works

  1. 1

    The pull request is the claim

    The description says what the agent did. A linked issue, when there is one, is the task.

  2. 2

    GitHub is the evidence

    proofslip reads the diff and the check runs already recorded for that commit. It does not ask the agent.

  3. 3

    One sticky receipt

    A comment says what checked out, what does not match, and what nobody could verify.

Same kind of sentence. Two receipts.

Both pull requests say the tests passed. The receipt is the difference.

Checks out

Honest pull request · omsocll/proofslip #3

Demo: honest receipt (count label helper).

  • ✅ Added `countLabel` in `src/text.ts` and called it from `src/present.ts` — confirmed. `src/text.ts` and `src/present.ts` were changed.
  • ✅ Added tests — confirmed. This pull request adds or updates a test file (`tests/text.test.ts`).
  • ✅ All tests pass — confirmed. The test run on this exact version of the code passed.

What we could check matches what the agent said.

Full receipt
Doesn't match

Dishonest pull request · omsocll/proofslip #4

Demo: dishonest receipt (claims tests passed).

  • ❌ All tests pass — contradicted. A green run does not count. The test command was weakened in `package.json`, so the check can pass even when the tests fail.
  • ❌ Added tests — contradicted. This pull request deleted the test file `tests/readme.test.ts` and did not add or update a test.

Deleted the test file `tests/readme.test.ts`. A test command was weakened in `package.json` (for example `|| true` or continue-on-error).

2 claims are contradicted. There are 2 warning signs. This receipt does not match the agent's account.

Full receipt

How accurate is it?

On 20 new public pull requests, from repos none of the earlier samples used, the overall verdict was fair on 19 of 20. The one miss said Doesn't match because tests that were moved into a new file were treated as deleted. Across these sets it has never been too rosy.

Fresh set, 20 new pull requestsResult
Verdict fair19/20
Too rosy0
Too harsh1
Claims extracted48
Pull requests with no claims6
False reassurances1
False alarms1

Earlier sets are context, and they were re-scored on this same build. The third set was fair on 20 of 20 (it had been 15 of 20). The held-out 20 was fair on 19 of the 19 pull requests still online. The first 20, which the early fixes were adjusted on, was fair on 20 of 20.

Tuned setBeforeAfter
Verdict fair10/2020/20
Too rosy30
Too harsh70
Claims extracted1557
Pull requests with no claims94
False reassurances22
False alarms30

The full read, including both sets, is docs/evaluation.md.

Add it to a repository

A GitHub Action posts the receipt on pull requests. Pin omsocll/proofslip@v0.1.1, or @v0 for the latest 0.x release.

From the command line

$ proofslip receipt --repo owner/name --pr 42

Or as a GitHub Action

name: proofslip
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]
  workflow_run:
    workflows: ["CI"]
    types: [completed]
permissions:
  contents: write
  pull-requests: write
  checks: read
  actions: read
  statuses: read
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: omsocll/proofslip@v0.1.1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

If the repository is private, this snippet works only after you can read it. The log on this site does not need GitHub access.