Repository navigation
free-threaded: setiter_iternext releases the set once per racing thread #160573
Description
setiter_iternext reads the set before taking the critical section and then clears it outside the critical section, so every thread that reaches the exhausted branch decrements the same set.
Objects/setobject.c:1098readsPySetObject *so = si->si_set;unlocked.1100if (so == NULL) return NULL;lets every racing thread through, becausesi_setis still set at that point.1113-1124the critical section covers only the table scan.1126-1129clearssi->si_setand callsPy_DECREF(so)afterPy_END_CRITICAL_SECTION().
Eight threads that arrive together each hold a non-NULL so and each run one decrement, so the set loses seven references it should not.
Reproduction on an unmodified Py_DEBUG free-threading build (3.16.0a1+dev free-threading build, main at 53b0004):
import sys, threading
N = 300000
s = set(range(N))
for i in range(1, N):
s.discard(i) # one element, 299,999 dummy entries
keep = [s] * 1000 # extra decrefs must not free the set
base = sys.getrefcount(s)
it = iter(s); next(it)
barrier = threading.Barrier(8)
def worker():
barrier.wait()
try: next(it)
except StopIteration: pass
for r in range(3):
ts = [threading.Thread(target=worker) for _ in range(8)]
[t.start() for t in ts]; [t.join() for t in ts]
print(r, base, sys.getrefcount(s))Output:
0 1002 996
1 1002 996
2 1002 996
Python/gc_free_threading.c:1118: validate_gc_objects: Assertion "gc_get_refs(op) >= 0" failed: refcount is too small
object address : 0x20002375e10
object refcount : 995
object type name: set
object repr : {0}
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x00005f29df1515f0)
exit status 134.
This is a sample from one run. The final count lands at 995 or 996 depending on how many of the eight threads lose a reference (six or seven); the abort itself is stable across runs. Rounds 1 and 2 are flat because si_set is already NULL by then. The process aborts while finalizing, and the object it complains about is the set itself.
The large table is what makes the window wide enough to hit. The scan after the unlocked read takes long enough for all eight threads to pass the so == NULL check. With a one-element set the same eight-thread barrier loop ran 31,020 rounds on the same build with no hit.
This has the same shape as the dict iterator in gh-154130. #144356 covers the setiter_len length hint only, and I did not find a report for the iternext path.
Clearing si->si_set inside the critical section at 1113-1124 and decref'ing after Py_END_CRITICAL_SECTION() would make the release happen once. Treating si_set like di_dict, with _Py_XGetRef and an atomic exchange, is the other route.
I have the harness that produced this (repros/set_iter_double_release.py, with an instrumented build that counts second releases) if it is useful.
Activity
lumenyr commentedon Oct 24, 2026
ThreadSanitizer reports the race directly, so this no longer rests on refcount arithmetic.
Build the way the CI sanitizer job does (gcc 15.2 works as well as clang):
repro.pyis two threads callingnext()on oneiter(set)behind athreading.Barrier. Three races come back, all insetiter_iternext:si_posis read at 1114 and written at 1125 with no lock held on either side, andsi_setis cleared at 1127 while the other thread is still inside the function. That also explains the refcount loss in the original report: the thread that seeskey == NULLclearssi_setand drops the set while the other is mid-setiter_iternext.Taking the lock for the
si_posread and thesi_setclear, and releasing before returning, removes two of the three reports. The third survives because the clear is a plain store while the entry check at 1104 is a relaxed atomic load, so the two disagree about being atomic:Writing the clear through
FT_ATOMIC_STORE_PTR_RELAXEDas well makes the repro exit 0 with no reports,refssteady at 1002 across 200 rounds anddouble_release=0. Both parts are in one diff if that is the direction you want to take.The four other unlocked
si_setuses on main at dc28c3a stay open even withsetiter_iternextfixed, so the full fix has to reach them too:setiter_dealloc(1042) runsPy_XDECREF(si->si_set)with no lock.setiter_traverse(1050) runsPy_VISIT(si->si_set)while the clearing thread may already have released it, so the collector can be handed a pointer that is being freed.setiter_len(1059) readssi->si_set, then dereferencessi->si_set->used; already tracked in Data race in set iterator length_hint under no-gil #144356.setiter_reduce(1073-1077) doessetiterobject tmp = *si;and thenPy_XINCREF(tmp.si_set). The struct copy is not atomic, so the increment can land on a pointer the clearing thread has released, which is an increment on freed memory rather than a net refcount error.The complete shape is the one gh-154130 proposes for
di_dict: atomic load, atomic exchange to NULL, and exactly onePy_DECREFperformed by whichever thread saw the non-NULL exchange.