Repository navigation
Conversation
setiter_iternext() read si->si_set, checked it for NULL, walked the table under the set's critical section and then cleared si->si_set and decrefed the set after Py_END_CRITICAL_SECTION(). Threads that reach the exhausted branch together all pass the NULL check, so every one of them drops the iterator's single reference and the set is over-released. On a free-threaded debug build the damage shows up at finalization as "refcount is too small" in validate_gc_objects(); when the set is still alive it shows up earlier as a negative refcount the next time sys.getrefcount() reads it. Hold the iterator's own critical section across the size check, the walk, the clear and the counter updates, and store NULL with the same atomic store the lock-free fast path uses for its read. The iterator's reference is then dropped exactly once, after the lock has been released. Add a regression test in test_free_threading: eight threads race to exhaust one iterator over a set with a large dummy tail, and the set's refcount must stay unchanged. Without the fix it aborts with _Py_NegativeRefcount.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A set iterator shared between threads over-releases its set when it is exhausted.
setiter_iternext()readssi->si_setwith no lock held, returns early only when it is already NULL, and clears the field plusPy_DECREF(so)afterPy_END_CRITICAL_SECTION(). Threads that arrive on the exhausted branch together all pass the NULL check, so each of them drops the iterator's single reference.Measured on a free-threaded debug build (
--disable-gil --with-assertions, gcc 15.2), with eight threads exhausting one iterator over a 300,000-slot set:sys.getrefcount(s)goes 1002 to 995, seven extra decrefs, thenvalidate_gc_objects()aborts at finalization with "refcount is too small", exit 134.ThreadSanitizer reported the race on the unpatched tree and is clean with the change, on the same reproducer.
The change holds the iterator's own critical section across the size check, the table walk, the clear,
si_pos,si_usedandlen, and stores NULL withFT_ATOMIC_STORE_PTR_RELAXED, which is what the lock-free fast path reads. The iterator's reference is dropped once, after the lock has been released. On a build with the GIL the critical section compiles away, so behaviour there is unchanged.I kept the table walk inside the set's critical section as it is rather than moving to the
_Py_XGetRefplus exchange shape discussed in gh-154130, because gh-124397 asks for the minimal change that stops the crash and this keeps the single release easy to see. I am happy to rework it to the other shape if that is the direction you want for the iterator family.New test in
Lib/test/test_free_threading/test_set.py:TestSetIteratorExhaustion. It aborts with_Py_NegativeRefcounton unpatched main and passes here.test_set(644 tests) andtest_free_threading.test_set(8 tests) are green on the free-threaded debug build.NEWS entry added.