Building software · Claude Code

Track down flaky tests overnight and review the fixes by day

Idea + starter prompt, not run by us yet

For: Engineering teams whose test pipeline fails at random and slows every release

The author let Claude read test pipeline results and had it make a tool that follows the 30-day pass rate. It chose the flakiest tests, looked into possible causes and tried to build small cases that reproduce each failure. Without a reliable repeat, it tried a fix and reran tests overnight. The pipeline got faster and the pass rate rose, though some fixes were thrown away.

Steps

1. Give the agent read-only access to pipeline results.
2. Have it build a simple pass rate tracker first.
3. Work on the worst test first and ask for a small repeatable case.
4. Review every pull request by day and throw away bad fixes.

Starter prompt

I have read-only access set up for our test pipeline results at [location].

1. Build a small script that shows the pass rate per test over the last 30 days.
2. List the [5] flakiest tests.
3. For the worst one, suggest likely causes and try to make a small case that repeats the failure.

Work on branch [branch name]. Show me the proposed fix, the test results and the pull request text first. Wait for my approval before pushing the branch or opening the pull request. Do not merge it. Explain how confident you are and what evidence you have.

Original post: Hacker News · found via agentusecases.ai. Written up in our own words. We haven't run it, so test it on made-up data first.