Repository navigation
gh-159963: Don't hold parent package import locks while a submodule executes - #159965
Merged
Merged
Conversation
…dule executes Since pythongh-83065, the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. Threads needing only the finished pkg were blocked for that long, and concurrent first imports could raise _DeadlockError or a spurious "partially initialized module" ImportError, or hang, with no import cycle involved. Parent packages are now imported, or waited for, before the submodule's lock is taken, and only that lock is held while it executes. The original pythongh-83065 deadlock stays fixed. importlib.import_module() now returns the partially initialized module in a cross-thread circular import, as the import statement already did, instead of raising _DeadlockError.
Documentation build overview
45 files changed ·
|
Address review: shorten the NEWS entry per the updated devguide guidance, reword the What's New entry, and trim the _resolve_parents() docstring and the parent fallback comment in _find_and_load_unlocked().
gpshead
commented
Oct 21, 2026
gpshead
commented
Oct 21, 2026
test_concurrent_first_imports_no_partial_module is the depth-one form of test_concurrent_first_imports_of_nested_packages, and test_waiters_recover_after_failed_package_init failed only where test_package_init_imports_submodule_deterministic already does.
gpshead
commented
Oct 22, 2026
gpshead
marked this pull request as ready for review
October 22, 2026 23:32
gpshead
requested review from
FFY00,
brettcannon,
ericsnowcurrently,
ncoghlan and
warsaw
as code owners
October 22, 2026 23:32
Member
Author
|
@emmatyping volunteered as review tribute at the sprint 😅 |
emmatyping
reviewed
Oct 23, 2026
After waiting for a module that another thread was executing, _find_and_load() looked in sys.modules again only after releasing the module's lock. If that import had failed and a thread had already begun retrying it, the waiter returned the new, still executing module. Look while still holding the lock. Reported by Emma Smith in review of pythonGH-159965.
|
Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
GH-160630 is a backport of this pull request to the 3.15 branch. |
gpshead
added a commit
that referenced
this pull request
Oct 25, 2026
…odule executes (GH-159965) (#160630) gh-159963: Don't hold parent package import locks while a submodule executes (GH-159965) Since gh-83065 (3.15.0), the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. Threads that needed only the finished pkg stayed blocked for that long, and concurrent first imports could raise _DeadlockError or a spurious "partially initialized module" ImportError, or hang, with no import cycle involved. Parent packages are now imported, or waited for if another thread is initializing them, before the submodule's module lock is taken, so only that lock is held while the submodule executes. The lock ordering deadlock fixed by gh-83065 stays fixed. importlib.import_module() now returns the partially initialized module in a circular import split across threads, as the import statement already did, instead of raising _DeadlockError. That was long-standing behavior rather than part of the 3.15.0 regression. (cherry picked from commit 7159b15) Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since gh-83065, the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. This shipped in 3.15.0. Threads needing only the finished pkg were blocked for that long, and concurrent first imports could raise _DeadlockError or a spurious "partially initialized module" ImportError, or hang, with no import cycle involved.
Parent packages are now imported, or waited for, before the submodule's lock is taken, and only that lock is held while it executes. The original gh-83065 deadlock stays fixed.
importlib.import_module() now returns the partially initialized module in a cross-thread circular import, as the import statement already did, instead of raising _DeadlockError. This part is not the 3.15.0 regression: import_module() has raised _DeadlockError there since at least 3.12 (3.12, 3.13, 3.14 and main all do), so it can be judged separately for the backport.
Two threads, step by step
The scenario from
test_concurrent_first_imports_of_sibling_packages: nothing is imported yet,a/leaf.pydoesfrom b.leaf import Xandb/__init__.pydoesimport a. The dependencies area.leaf→b.leaf→b→a, so there is no import cycle. Thread 2 starts while thread 1 is still insidea/__init__.py.sequenceDiagram participant T1 as Thread 1<br/>import a.leaf participant T2 as Thread 2<br/>import b.leaf Note over T1: 1. acquire locks a and a.leaf<br/>run a/__init__.py Note over T2: 2. acquire locks b and b.leaf<br/>run b/__init__.py (import a) T2-->>T1: 3. wait for lock a Note over T1: 4. a/__init__.py returns<br/>lock a is NOT released<br/>run a/leaf.py (from b.leaf import X) T1--xT2: 5. needs lock b.leaf, but T2 is waiting on T1 Note over T1: 6. _DeadlockErrorsequenceDiagram participant T1 as Thread 1<br/>import a.leaf participant T2 as Thread 2<br/>import b.leaf Note over T1: 1. acquire lock a<br/>run a/__init__.py Note over T2: 2. acquire lock b<br/>run b/__init__.py (import a) T2-->>T1: 3. wait for lock a Note over T1: 4. a/__init__.py returns<br/>release lock a Note over T1: 5. acquire lock a.leaf<br/>run a/leaf.py (from b.leaf import X) Note over T2: 6. a is ready, b/__init__.py returns, release lock b<br/>acquire lock b.leaf, run b/leaf.py T1-->>T2: 7. wait for lock b.leaf Note over T2: 8. release lock b.leaf<br/>import b.leaf done Note over T1: 9. X imported, release lock a.leaf<br/>import a.leaf doneBefore, thread 1 keeps lock
aaftera/__init__.pyhas returned, so thread 2, which needs only the finisheda, stays blocked while holding the locks thread 1 needs next. After, each thread holds one module lock at a time and both imports complete. Steps 5 to 8 of the right-hand side can interleave differently (thread 1 may loadb.leafitself); either way neither thread waits on a lock while holding one the other needs.authored using Claude Fable