Skip to content

gh-160306: Fix iso2022_jp_3 decoding of the JIS X 0213:2000 plane 2 mapping - #160702

Open
kinqsradio wants to merge 1 commit into
python:mainfrom
kinqsradio:fix/iso2022-jp3-roundtrip
Open

kinqsradio wants to merge 1 commit into
python:mainfrom
kinqsradio:fix/iso2022-jp3-roundtrip

Conversation

@kinqsradio

@kinqsradio kinqsradio commented Oct 26, 2026 •

Copy link
Copy Markdown

The JIS X 0213:2000 plane 2 emulation sets U+9B1D for cell 0x7D3B, but the following independent table lookup overwrites it with the 2004 mapping, U+9B1C. Make that lookup an else if, matching the adjacent plane 1 decoder and the EUC/Shift_JIS implementations.

The regression test checks the exact encoding and round trip, every two-chunk incremental decode split, and stream reading for both iso2022_jp_3 and iso2022_jp_2004. The latter retains U+9B1C.

Validation on a macOS arm64 debug build of current main (c436e8d83597):

  • The new regression fails before the C change and passes afterward.
  • ./python.exe -m test -j2 test_multibytecodec test_codecencodings_iso2022 test_codecencodings_jp test_codecencodings_cn test_codecencodings_kr test_codecencodings_tw test_codecencodings_hk test_codecs: passed (756 tests run, 17 skipped).
  • ./python.exe -m test -R 3:3 test_multibytecodec test_codecencodings_iso2022: passed.
  • Comparing all 35,344 plane 1 and plane 2 decode results across both ISO-2022 codec editions before and after the change found only the intended iso2022_jp_3 cell change, including unchanged invalid-cell results.
  • make patchcheck, Ruff, NEWS lint, and git diff --check: passed.

AI assistance was used to implement and validate this change.

Fixes #160306.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iso2022_jp_3 encodes U+9B1D but decodes the result as U+9B1C

1 participant