DocDrift demo

DocDrift finds doc code snippets that no longer match the code, proposes fixes, and re-runs each fix. Only fixes that pass the re-run are shown as verified.

Scope: 3 synthetic seeded fixture repos, 5 snippets executed. Fixtures exist to demonstrate the pipeline end to end; scores on them are not a measure of real-world performance. A real open-source-repo benchmark is separate future work.

Live run. Model calls went through Nebius Token Factory (model nvidia/Nemotron-3_5-Lightning); token usage below is measured.

Record note: This page is rendered from a reconstructed record of the live run: the run's raw report was not retained in full, so every value shown comes from the run's surviving artifacts (recorded output, rendered page and public fixture code). The rejected fix's code and its rerun error are not in the record, which is why the stale_unfixed row shows no fix evidence.

calc_repo

precision 1.00 - recall 1.00 - verified fixes 1 of 1 stale

from fastcalc import add
print(add(1, 2))

from fastcalc import add_numbers
print(add_numbers(1, 2))

ver.py", line 11, in <module>
    exec(compile(s, "<snippet>", "exec"), g)
  File "<snippet>", line 1, in <module>
ImportError: cannot import name 'add_numbers' from 'fastcalc' (/repo/lib/fastcalc.py)

from fastcalc import add
print(add(1, 2))

greet_repo

precision 1.00 - recall 1.00 - verified fixes 0 of 1 stale

from greeter import greet
print(greet("Ada"))

st):
  File "/work/driver.py", line 11, in <module>
    exec(compile(s, "<snippet>", "exec"), g)
  File "<snippet>", line 2, in <module>
TypeError: greet() takes 0 positional arguments but 1 was given

from greeter import greet
print(greet(name="Ada"))

ok_repo

precision 1.00 - recall 1.00 - verified fixes 0 of 0 stale

from shapes import area
print(area(2, 3))

Source code - ran live on Nebius Token Factory with the NVIDIA Nemotron open model nvidia/Nemotron-3_5-Lightning.