Bug report

Bug description:

When an incremental encode() call of a stateful CJK codec (iso2022_jp*, iso2022_kr, hz) fails, the output of that call is discarded but the shift state it reached is kept. The next encode() then leaves out the escape sequence, so double-byte characters are written while the stream is still in ASCII mode. IncrementalEncoder.encode() (and so io.TextIOWrapper.write()) and StreamWriter.write() are affected. The encoder already restores its pending buffer on this path, just not the codec state.

import codecs

for enc, text in [("iso2022_jp", "世"), ("iso2022_kr", "가"),
                  ("hz", "聊")]:
    good = codecs.getincrementalencoder(enc)().encode(text)
    encoder = codecs.getincrementalencoder(enc)()
    try:
        encoder.encode(text + "\U0001F600")
    except UnicodeEncodeError:
        pass
    print(enc, encoder.encode(text), "expected:", good)

Output on main; every value should equal its "expected":

iso2022_jp b'@$' expected: b'\x1b$B@$'
iso2022_kr b'0!' expected: b'\x1b$)C\x0e0!'
hz b'AD' expected: b'~{AD'

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-160522: Restore CJK encoder state after a failed incremental encode #160523