[Nikhil Gupta (Denimworld12)]

Description

PuzzledConverter picks different words to mask, and prints different clues, depending on
whether the prompt arrives composed (NFC) or decomposed (NFD). The same sentence gives two
different puzzles.

from pyrit.converter.puzzled.keyword_masker import mask_prompt
import unicodedata

sent = "surprisingly café"
mask_prompt(unicodedata.normalize("NFC", sent), num_to_mask=1).masked_prompt
# 'surprisingly [WORD1]'      clue: '4-letter noun'

mask_prompt(unicodedata.normalize("NFD", sent), num_to_mask=1).masked_prompt
# '[WORD1] café'              clue: '12-letter word'

This only shows up when spaCy is installed, which is why it has not surfaced: spaCy is an
optional extra, and every Unicode test in the suite is decorated @_no_spacy.

A randomized sweep of 400 prompts over an accented vocabulary, with spaCy live, disagrees
between NFC and NFD on 294 of 400. With this change, 0 of 400.

Follow-up to #3155.

Cause

Both causes live in _pos_lookup, which is the one place in the module that had not been
made normalization-safe.

1. token.is_alpha discards tags spaCy already produced. It is token.text.isalpha(),
which is False for any text carrying a combining mark. So the gate drops marked words even
though the tagger labelled them:

word	is_alpha	spaCy pos_	kept today
café (NFC)	True	PROPN	yes
café (NFD)	False	NOUN	no
שָׁלוֹם	False	PROPN	no
عَرَبِيّ	False	NOUN	no
हिन्दी	False	NOUN	no

Pointed scripts are never tagged, in any normalization form — the same scripts #3155 made
maskable. They always fall back to the generic "word" clue.

2. The tagger itself is normalization-sensitive. NFC café tags NOUN; NFD café tags
ADJ. Context propagates, so neighbouring ASCII words shift too (documents came back
VERB in one form and NOUN in the other). No gate change can fix this one — the text has
to be normalized before it reaches the pipeline.

Because the part of speech decides a tier in _rank_candidates, a dropped or altered tag
does not just change the clue text, it changes which word gets masked.

Changes

Two lines in _pos_lookup:

-    for token in nlp(prompt):
-        if token.is_alpha and token.pos_ in _MASKABLE_POS:
+    for token in nlp(unicodedata.normalize("NFC", prompt)):
+        if _MASKABLE_WORD_PATTERN.fullmatch(token.text) and token.pos_ in _MASKABLE_POS:

_MASKABLE_WORD_PATTERN is the module's own \p{L}[\p{L}\p{M}]*, already used by
_iter_words to decide what a maskable word is. Using it here means the scanner and the
tagger finally agree on that definition — which the module docstring already says they do
("Scanning and validation use the same Unicode properties").

Tagging is normalized to NFC only for the pipeline's benefit; the keys are still built with
_word_key, so lookups against prompt words in any form keep matching.

Tests

Three tests in tests/unit/converter/test_puzzled_keyword_masker.py. On main the file is
3 failed, 107 passed; with this change it is 110 passed, so the fix changes nothing
the existing tests assert:

• test_pos_lookup_tags_words_with_combining_marks — a token whose is_alpha is False
  still contributes its tag.
• test_pos_lookup_normalizes_before_tagging — the pipeline receives NFC text.
• test_mask_prompt_selection_is_stable_across_normalization_forms — NFC and NFD of one
  prompt mask the same word with the same clue.

The existing _FakeToken gained no new default: the gate test passes is_alpha=False
explicitly, and the recording fake derives it with str.isalpha() the way spaCy does, so
the fakes now model the real behaviour instead of assuming it.

No new dependency: the tests use fakes, so they run whether or not spaCy is installed.

[Richard Lundeen (richlundeen)]

Looks good. Normalizing the tagger input and reusing the existing word pattern fixes the inconsistent keyword selection without changing the original prompt.

Reviewed with GitHub Copilot.