Parallel mode: AssertionError in State.reload_meta after a sqlite cache write fails with 'database is locked' #22241

Crash Report

With parallel checking (-n), a worker crashes with AssertionError in State.reload_meta when a cache write was dropped because the sqlite cache was locked.

SqliteMetadataStore.write() returns False on sqlite3.OperationalError ("database is locked" after sqlite3's default 5 s busy timeout), and write_cache treats that as harmless ("The next run will simply find the cache entry out of date"). Sequential mypy indeed survives the same lock. In parallel mode, though, the worker that processed a module still reports it done, and another worker that needs it as a dependency calls reload_meta(), finds no meta, and hits assert new_meta_pair is not None. The worker exits, and the coordinator then fails with Worker N disconnected before sending data.

We first saw this when two mypy -n processes on overlapping file sets shared one --cache-dir right after a dependency upgrade (stale cache): 3 of 8 trials crashed on a private codebase, 0 of 6 ran sequentially and 0 of 6 used separate cache dirs. A minimal two-process repro with public packages did not crash in 81 trials, so the script below replaces the second mypy process with a small script that holds the sqlite write lock for 7 s. That makes the crash deterministic and shows the cause: an instrumented run logged collections/__init__.data.ff database is locked in one worker, then three other workers found no meta for collections.

Related: #21974 (concurrent invocations sharing a cache, different symptom). I know sharing a cache between concurrent runs is not supported (#13916). But a write mypy already tolerates in sequential mode should not become an assertion in parallel mode. Expected: a missing or unreadable meta in reload_meta is treated as a cache miss (the module gets loaded or rechecked), or at worst reported as an ordinary error.

Traceback

.venv/lib/python3.14/site-packages/mypy/typeshed/stdlib/time.pyi: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 2.4.0
.venv/lib/python3.14/site-packages/mypy/typeshed/stdlib/time.pyi: note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "<frozen runpy>", line 203, in _run_module_as_main
  File "mypy/build_worker/worker.py", line 144, in main
  File "mypy/build_worker/worker.py", line 246, in serve
    scc_result = process_stale_scc_interface(
  File "mypy/build.py", line 4966, in process_stale_scc_interface
  File "mypy/build.py", line 4790, in maybe_load_deps
    graph[mod_id].reload_meta()
  File "mypy/build.py", line 3028, in reload_meta
    assert new_meta_pair is not None
AssertionError

error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 2.4.0
note: use --pdb to drop into pdb
Traceback (most recent call last):
  File "mypy/build.py", line 1378, in receive_worker_message
    return receive(self.workers[idx].conn)
  File "mypy/ipc.py", line 496, in receive
OSError: No data received

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.14/site-packages/mypy/__main__.py", line 16, in console_entry
    main()
    ~~~~^^
  File "mypy/main.py", line 167, in main
  File "mypy/main.py", line 257, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "mypy/build.py", line 425, in build
  File "mypy/build.py", line 540, in build_inner
    graph = dispatch(sources, manager, stdout, connect_threads)
  File "mypy/build.py", line 4199, in dispatch
    process_graph(graph, manager)
  File "mypy/build.py", line 4667, in process_graph
    done, still_working, results = manager.wait_for_done(graph)
  File "mypy/build.py", line 1474, in wait_for_done
    return self.wait_for_done_workers(graph)
  File "mypy/build.py", line 1493, in wait_for_done_workers
    buf = self.receive_worker_message(idx)
  File "mypy/build.py", line 1386, in receive_worker_message
    raise OSError(
OSError: Worker 2 disconnected before sending data (exit code 2)

Master (2.5.0+dev.31f63e096, compiled wheel) fails the same way, at build.py line 3053.

To Reproduce

bash repro.sh (needs uv). It prints 10/10 runs crashed in reload_meta on 2.4.0 and on master. Controls: bash repro.sh 10 4 3 (lock held 3 s, under the busy timeout) and bash repro.sh 10 0 7 (no -n) both give 0/10. MYPY_SPEC='mypy @ git+https://github.com/python/mypy' selects another mypy.

#!/usr/bin/env bash
# Parallel mypy (-n) crashes with AssertionError in State.reload_meta when a cache write
# fails because another process holds the sqlite cache's write lock for longer than
# sqlite3's default 5 s busy timeout.
# usage: bash repro.sh [trials=10] [workers=4] [hold_seconds=7]
# Controls: hold_seconds below 5, or workers=0 (no -n, sequential mypy).
set -u
N=${1:-10}; W=${2:-4}; HOLD=${3:-7}
PAR=(-n "$W"); [ "$W" = 0 ] && PAR=()
cd "${WORKDIR:-$(mktemp -d)}"
[ -d .venv ] || uv venv -q --python 3.14
uv pip install -q "${MYPY_SPEC:-mypy==2.4.0}"
echo 'import asyncio, json, email.message' > m.py
cat > hold_lock.py <<'PY'
import glob, sqlite3, sys, time
# Stand-in for a second mypy process: once the run has written a few cache rows,
# hold the write lock on every cache shard for HOLD seconds, then release it.
d, hold = sys.argv[1], float(sys.argv[2])
while not glob.glob(f"{d}/cache*.db"):
    time.sleep(0.01)
time.sleep(0.2)
conns = [sqlite3.connect(p, isolation_level=None, timeout=60) for p in sorted(glob.glob(f"{d}/cache*.db"))]
rows = lambda: sum(c.execute("SELECT count(*) FROM files2").fetchone()[0] for c in conns)
start = rows()
while rows() < start + 20:
    time.sleep(0.01)
for c in conns:
    c.execute("BEGIN IMMEDIATE")
time.sleep(hold)
for c in conns:
    c.execute("ROLLBACK")
PY
crashes=0
for t in $(seq 1 "$N"); do
  rm -rf cache; mkdir cache
  V=$(.venv/bin/python -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')
  .venv/bin/python hold_lock.py "cache/$V" "$HOLD" & h=$!
  .venv/bin/mypy "${PAR[@]}" --show-traceback --cache-dir cache m.py > "run$t.log" 2>&1
  wait $h
  grep -q 'assert new_meta_pair is not None' "run$t.log" && crashes=$((crashes + 1))
done
.venv/bin/mypy --version
echo "$crashes/$N runs crashed in reload_meta; logs in $(pwd)"
grep -m1 -B12 -A3 'assert new_meta_pair' run*.log | head -20

Your Environment

• Mypy version used: 2.4.0 (compiled), and master 2.5.0+dev.31f63e096 (compiled wheel)
• Mypy command-line flags: -n 4 --cache-dir cache (default sqlite cache)
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.14.8
• Operating system and version: Ubuntu 26.04 (Linux 7.0)