How Replay works
A bug recording goes in one end. A failing test, a patch, and proof come out the other. Every stage either passes a deterministic gate or the run ends with an honest status. Nothing is claimed that a test did not prove.
The pipeline
Ingest
ffmpeg
Watch
gate checked
Locate
gate checked
Reproduce
gate checked
Verify
gate checked
Fix
gate checked
Report
gate checked
One model per job
| stage | model | job |
|---|---|---|
| Watch | nvidia/Nemotron-3.5-Super-VL | Reads the frames, describes the user steps, finds the first frame where the defect is visible. |
| Locate | nvidia/Nemotron-3_5-Lightning | Reads the packed repository and ranks the source files most likely responsible. |
| Reproduce | nvidia/Nemotron-3-Ultra-550b-a55b | Writes a Playwright test that fails for the same reason the video shows. |
| Verify | nvidia/Nemotron-3.5-Super-VL | Compares the recording failure frame with the test screenshot: same defect or not. |
| Fix | nvidia/Nemotron-3-Ultra-550b-a55b | Patches the code, then reruns the reproduction test and the existing suite. |
| Report | nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B | Writes the bug report from recorded evidence only. |
Models run on NVIDIA Nemotron via Nebius Token Factory. A reproduction only counts when the test fails with the right assertion, and a fix only counts when the reproduction test and the existing suite both pass.
The stack
- Next.js frontend, Hono API and worker, one TypeScript monorepo.
- Playwright runs the tests inside a Docker sandbox, one container per run.
- FFmpeg extracts the frames, ripgrep searches the code, Zod validates every model output.
Source repository. The demo video is recorded when the benchmark run finishes.