Skip to content

Use the external encoding for character boundaries in GzipReader - #163

Open
mame wants to merge 1 commit into
ruby:masterfrom
mame:zlib-gzreader-char-encoding
Open

mame wants to merge 1 commit into
ruby:masterfrom
mame:zlib-gzreader-char-encoding

Conversation

@mame

@mame mame commented Oct 24, 2026

Copy link
Copy Markdown
Member

GzipReader#getc and #gets with a limit split the data before it is converted to the internal encoding, but measured characters in the internal encoding.

  gz = Zlib.gzip("\x82\xa0\x82\xa2".b) # "あい" in Shift_JIS
  Zlib::GzipReader.new(StringIO.new(gz), external_encoding: "Shift_JIS",
                       internal_encoding: "UTF-8").each_char.to_a
  # expected: ["あ", "い"]
  # actual:   ["\x82", "\xA0", "\x82", "「"]

GzipReader#getc and #gets with a limit split the data before it is
converted to the internal encoding, but measured characters in the
internal encoding.

  gz = Zlib.gzip("\x82\xa0\x82\xa2".b) # "あい" in Shift_JIS
  Zlib::GzipReader.new(StringIO.new(gz), external_encoding: "Shift_JIS",
                       internal_encoding: "UTF-8").each_char.to_a
  # expected: ["あ", "い"]
  # actual:   ["\x82", "\xA0", "\x82", "「"]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mame
mame requested a review from nobu October 24, 2026 20:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant