[Bernhard Merkle (bmerkle)]

Part of #346 (item 3). Items 1 and 2 are in #358; the two PRs are independent and can merge in either order. Together they fix #346.

The fuzzy related-terms index got different terms depending on how messages were ingested, so related-term expansion depended on the ingestion path:

	add_messages_with_indexing (before)	add_messages_streaming (before)
Entity types, facets, action subjects/objects/params	not embedded	embedded
Multi-word verbs	each verb separately ("looks", "up")	joined ("looks up"), like the term index
Types/facets of entities without a name	—	embedded, though the term index skips them
Terms from message metadata (speakers, recipients, …)	embedded	never embedded

Change

• New semantic_ref_index.collect_knowledge_terms and collect_related_terms_for_fuzzy_index return exactly the normalized, deduplicated terms a knowledge item adds to the semantic ref index (callers filter out nameless extracted entities, as the term index does).
• ConversationBase._update_related_terms_incremental and add_messages._collect_related_terms_for_fuzzy_index both use them.
• The streaming commit indexes metadata knowledge only at commit time, so those terms have no precomputed embedding. It now calls _update_related_terms_incremental(..., exclude=<precomputed terms>) before adding the precomputed ones, which embeds only the remaining (metadata) terms; if that embedding fails, the fuzzy index is still unchanged.

After this, on both paths the fuzzy index holds the same terms as the semantic ref index, which is also what build_related_terms_index produces when rebuilding from scratch.

Cost: add_messages_with_indexing embeds more terms per message, and add_messages_streaming embeds metadata terms at commit time. Terms already in the fuzzy index are skipped before embedding (memory and SQLite).

Tests (each fails on main):

• test_fuzzy_terms_match_semantic_ref_index_terms: both collectors return exactly the term-index terms for knowledge with multi-word verbs, a nameless entity, an inverse action, params and facets.
• test_add_messages_with_indexing_fuzzy_terms_match_term_index (with and without knowledge extraction) and test_streaming_fuzzy_terms_match_term_index (also with and without knowledge extraction): after ingesting on SQLite, fuzzy-index terms equal semantic-ref-index terms.
• test_memory_fuzzy_terms_match_term_index_across_batches (both ingestion paths): two overlapping batches on in-memory storage leave exactly one fuzzy entry per term-index term.
• test_streaming_commit_embeds_only_metadata_terms: commit-time embedding covers only metadata terms that have no precomputed embedding.
• test_fuzzy_index_skips_duplicate_terms, test_term_embedding_index_skips_held_terms_after_reload_and_clear, test_fuzzy_index_deserialize_drops_repeated_terms: each term is stored once on both backends, including after reload and when loading older data with repeated terms.

Fuzzy-index duplicates (review follow-up, 6f62c51): both the in-memory TermEmbeddingIndex and SqliteRelatedTermsFuzzy now skip terms they already hold and terms repeated within one call, so common terms such as person are stored once.

Not addressed (pre-existing): a failure after the fuzzy index is updated in a streaming commit rolls back the DB but not SqliteRelatedTermsFuzzy's in-memory state, so a retry on the same storage skips those terms (tracked in #362; TODO in conversation_base.py).

Testing: make locally: format, ruff and pyright (3.12 and 3.15) clean, build OK; 912 passed. 7 tests fail locally because of a broken jiter native module on my machine (plus an MCP sampling test that depends on it); they fail identically on main, so CI is the reference for those.

🤖 Generated with Claude Code

[Bernhard Merkle (bmerkle)]

Kevin Turcios (@KRRT7) could you take a look? This is the other half of #346: both ingestion paths now add the same terms to the fuzzy related-terms index, including metadata terms on the streaming path. Independent of #358.

[Copilot]

🟡 Changes recommended

Commit-time embedding failures can leave retries with missing persisted terms, and nameless metadata entities still produce inconsistent index contents.

2 open issues

• Embedding failure leaves cached terms missing after retry · New
• Nameless metadata entities are inconsistently indexed · New

What changed in this PR

Addresses #346’s ingestion-path inconsistency by sharing fuzzy-index term collection between incremental and streaming ingestion.

Changes:

• Centralizes normalized, deduplicated knowledge-term collection.
• Adds streaming metadata terms at commit time.
• Adds term-parity regression tests and documents the behavior change.

File	Description
tests/​test_add_messages_with_indexing.py	Tests incremental ingestion term parity.
tests/​test_add_messages_streaming.py	Tests streaming term parity, including metadata.
tests/​test_add_messages_pipeline.py	Tests shared collectors against semantic-index terms.
src/​typeagent/​storage/​memory/​semantic_ref_index.py	Adds shared knowledge-term collectors.
src/​typeagent/​knowpro/​conversation_base.py	Uses shared collection and embeds remaining metadata terms.
src/​typeagent/​knowpro/​add_messages.py	Delegates pipeline collection to the shared helper.
CHANGELOG.md	Documents ingestion behavior and embedding changes.

_{Review effort: Balanced}

[Kevin Turcios (KRRT7)]

taking a look

[Kevin Turcios (KRRT7)]

Both Copilot issues are fixed in 22b1d93, and the regression tests cover them. The new collectors return exactly the terms the term index adds. A few things, none blocking:

1. SQLite fuzzy index: in-memory duplicates within a streaming commit (made worse by this PR).
_commit_batch_from_chunk_results concatenates each chunk's related_terms without deduplicating (conversation_base.py:305). SqliteRelatedTermsFuzzy.add_terms_with_embeddings filters only against _added_terms from earlier calls, so repeats within one call all go into _vector_base and _terms_list. INSERT OR REPLACE collapses them in the DB. Repro: one TranscriptMessage with two chunks through _RichExtractor gives 10 rows in RelatedTermsFuzzy but 17 entries in _terms_list and _vector_base. This predates the PR (entity names repeated across chunks), but now types and facets like person show up in nearly every chunk. The fix is small: deduplicate inside add_terms_with_embeddings, or when building fuzzy_terms. test_streaming_fuzzy_terms_match_term_index doesn't catch it because get_terms() reads the DB.

2. In-memory TermEmbeddingIndex duplicates across batches.
You mention this in the description. Since every batch now re-adds common type terms, fuzzy lookups will fill top-k with repeated person hits. I'd like a tracking issue at least, or a fix here (skip already-held texts, as the SQLite index does).

3. The reorder narrows the rollback window but doesn't close it.
If the metadata embedding succeeds and something later in the transaction fails (e.g. update_conversation_timestamps), the DB rolls back, but _added_terms keeps the metadata terms, so a retry on the same storage skips them. The comment in _update_secondary_indexes_incremental_with_embeddings is correct for the case it describes. The underlying problem is that the SQLite fuzzy index's in-memory state isn't restored on rollback, which predates this PR. A note or follow-up issue would be enough.

[Bernhard Merkle (bmerkle)]

Kevin Turcios (@KRRT7) thanks for the review. All three points are addressed in 6f62c51:

1. SQLite duplicates within a commit: SqliteRelatedTermsFuzzy.add_terms_with_embeddings (and add_terms) now also skip repeats within the same call, keeping the first embedding, so _vector_base/_terms_list match the DB rows.
2. In-memory TermEmbeddingIndex duplicates across batches: fixed here rather than in a separate issue. It now tracks the texts it holds and skips held terms and in-call repeats, like the SQLite index.
   New test_fuzzy_index_skips_duplicate_terms covers both backends (in-call repeats, a cross-call repeat, and add_terms). It checks size() and lookup_term results, so the SQLite in-memory state is covered too, not just the DB. Without the fix it fails on both backends (7 entries instead of 3).
3. Rollback window: added a note and a TODO next to the reorder in _update_secondary_indexes_incremental_with_embeddings: a failure later in the transaction still leaves the fuzzy index's in-memory state ahead of the rolled-back DB. Fixing that properly (restoring in-memory state on rollback) predates this PR, so I've left it out of scope. Happy to open a tracking issue.

[Bernhard Merkle (bmerkle)]

Kevin Turcios (@KRRT7) opened #362 to track the rollback point (3); the TODO in conversation_base.py now references it.

[Bernhard Merkle (bmerkle)]

Guido van Rossum (@gvanrossum) can you please approve ?

[Guido van Rossum (gvanrossum)]

Happy to!