Repository navigation
gh-160309: Fix StreamReader.read() silently dropping an incomplete final character - #160642
Open
saiyam-jain-dev wants to merge 1 commit into
Open
saiyam-jain-dev wants to merge 1 commit into
saiyam-jain-dev wants to merge 1 commit into
Conversation
… final character When a byte stream ends with an incomplete multibyte sequence (e.g. a truncated UTF-8 character), StreamReader.read() would silently discard those bytes: the read loop exited on empty newdata while bytebuffer still held the fragment, and bytebuffer was never decoded. Fix by detecting leftover bytes in bytebuffer when the underlying stream is exhausted and decoding them immediately. The codec's stateless decode function (e.g. utf_8_decode with final=True) then applies the error handler: 'strict' raises UnicodeDecodeError, 'replace' emits the replacement character, and 'ignore' drops the bytes - matching the behaviour of bytes.decode() and io.TextIOWrapper.
|
The following commit authors need to sign the Contributor License Agreement: |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #160309
codecs.StreamReader.read() silently drops bytes when the stream ends with an incomplete multibyte sequence. The read loop exits on empty
ewdata while �ytebuffer still holds the trailing bytes, and those bytes are never decoded.
Bug demonstration (before fix)
`python
Every other decoding path raises or applies the error handler:
`python
Fix
When the underlying stream is exhausted and �ytebuffer is non-empty, decode the leftover bytes immediately before breaking out of the loop. The codec's stateless decode function applies inal=True semantics, so:
This matches the behaviour of �ytes.decode() and io.TextIOWrapper.
codecs.StreamReader.read()silently drops an incomplete character at the end of the stream #160309