Bug report

Bug description:

When a ZstdCompressor that uses worker threads (CompressionParameter.nb_workers) and a ZstdDict is freed by the garbage collector as part of a reference cycle, tp_clear releases the dictionary before the compression context is freed. A worker thread that is still compressing already submitted data then reads the freed dictionary content (as_prefix) or the freed ZSTD_CDict (as_digested_dict), and the interpreter crashes. ZstdCompressor_dealloc() already frees the context first, so the same objects freed by reference counting are fine.

Nobody needs to control the input beyond what the application already does: a program that abandons a multi-threaded compressor with a dictionary in the middle of a stream (for example after an error) while it is part of a reference cycle can crash at the next collection. The effect is a read of freed memory in a worker thread, seen as a segmentation fault.

import gc, random
from compression.zstd import CompressionParameter, ZstdCompressor, ZstdDict

class Cycle:
    pass

zd = ZstdDict(random.randbytes(1024) * 256, is_raw=True)
c = ZstdCompressor(options={CompressionParameter.compression_level: 10,
                            CompressionParameter.nb_workers: 1,
                            CompressionParameter.job_size: 1 << 20},
                   zstd_dict=zd.as_prefix)   # or zd.as_digested_dict
cycle = Cycle()
cycle.cycle = cycle
cycle.compressor = c
c.compress(random.randbytes(1024) * 1088)  # a worker thread starts on 1 MiB
del zd, c, cycle
gc.collect()
print("ok")

On a debug build of main (Linux x86-64, glibc, system libzstd 1.5.7) this dies with Fatal Python error: Segmentation fault in a libzstd worker thread during gc.collect(), 5 of 5 runs with as_prefix and 5 of 5 with as_digested_dict. Without the cycle.cycle = cycle line it prints ok. Whether the read of freed memory is noticed depends on the allocator; here the 256 KiB dictionary buffer is unmapped when freed. Only main was run; Modules/_zstd has the same tp_clear and tp_traverse code on the 3.14 and 3.15 branches (by inspection, not run). Free-threaded builds, other platforms and sanitizer builds were not run.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-160535: Keep the ZstdDict alive until the zstd context is freed #160536
• [3.15] gh-160535: Keep the ZstdDict alive until the zstd context is freed (GH-160536) #160635
• [3.14] gh-160535: Keep the ZstdDict alive until the zstd context is freed (GH-160536) #160636