profiling.sampling in --binary mode raises a UnicodeEncodeError while collecting the samples #160601

Activity

1. keewis

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 24, 2026
2. NekoAsakura

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   topic-profiling

   on Oct 24, 2026
3. patrick-5546 commented on Oct 24, 2026

   

   patrick-5546

   on Oct 24, 2026

   More actions

   A minimal way to trigger the same error, in case it helps. On Linux, Python decodes a file name that isn't valid UTF-8 with surrogateescape, so a script in such a directory has a co_filename containing a lone surrogate:

   cd "$(mktemp -d)"
   dir=$(printf 'caf\351')   # Latin-1 "café": not valid UTF-8
   mkdir "$dir"
   cat > "$dir/busy.py" <<'PY'
   import time
   
   def busy():
       deadline = time.monotonic() + 2
       while time.monotonic() < deadline:
           pass
   
   busy()
   PY
   python3.15 -m profiling.sampling run --binary -o out.bin "$dir/busy.py"

   This fails with the same traceback, ending in UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 23: surrogates not allowed, and leaves a 64-byte out.bin that replay rejects. --pstats profiles the same script without error. A function whose co_name contains a lone surrogate fails the same way. Reproduced with 3.15.1 and 3.16.0a1 on Linux.

   The cause appears to be (links at main, 94942ba):

   • The binary writer encodes file and function names with PyUnicode_AsUTF8AndSize, which rejects lone surrogates.
   • BinaryCollector.collect catches only OverflowError, and the sampling loop treats a UnicodeDecodeError, but not a UnicodeEncodeError, as a failed sample, so the error ends the profile before the file is finalized.

   The '\uddd0' in your traceback is outside U+DC80–U+DCFF, the only range surrogateescape produces, so in your run the surrogate came from somewhere other than a file name decoded that way. Either way, the writer would need to accept any str, for example by encoding with surrogatepass or backslashreplace; the reader already decodes with "replace".

   I used an AI assistant (GitHub Copilot CLI) to narrow this down and write the reproducer, and I've reviewed the above.
4. keewis commented on Oct 24, 2026

   

   keewis

   on Oct 24, 2026

   Last edited by keewis

   Author

   More actions

   looks like it is worse than that: with a debugger I can see a stack frame where the function name contains CJK characters as well as \xXX and \uXXXX sequences in a seemingly random order. And copy-pasting that function name into a separate python process and trying to encode as utf-8 results in the exact same error message.

   FrameInfo repr extracted using a debugger

   _remote_debugging.FrameInfo(filename='<unknown file>', location=None, funcname='틈䕛王\x00...', opcode=None)
   

   the exact name is truncated just in case this is some kind of memory dump, but it has a length of 1024 chars (not bytes)

   That's obviously not valid python and it definitely does not appear anywhere in my code, so I really wonder where this is coming from.

   I can't confirm this reliably, but it looks like this only happens if I run the command when there's no previous binary output file existing, if it does the weird stack frame does not appear and the output file is written successfully.