[Bernhard Merkle (bmerkle)]

Fixes #362 (follow-up from the #360 review).

Problem

SqliteRelatedTermsFuzzy keeps _added_terms, _terms_list and a VectorBase next to the RelatedTermsFuzzy table, and SqliteMessageTextIndex keeps a VectorBase next to MessageTextIndex. Both are updated together with the table writes. When the surrounding transaction rolled back, SqliteStorageProvider.__aexit__ restored only the tables:

• A retry of the same batch on the same provider treated the rolled-back terms as already present and skipped them, so they were never written to the persisted fuzzy index (reopening the database did not bring them back), and lookup_term could return terms that were not in the DB.
• The message text index kept vectors for rows that no longer existed; lookups hitting them were dropped silently.

Fix

• SqliteRelatedTermsFuzzy.reload_from_db() and SqliteMessageTextIndex.reload_from_db(): the loading code that lived in __init__ is now a method that first clears the in-memory state. __init__ calls it.
• SqliteStorageProvider.__aexit__ calls both after rollback(). This is the "reload from the table after a rollback" option from the issue: it is always correct, has no extra bookkeeping on the success path, and costs the same as opening the database, only on the failure path.
• SqliteStorageProvider.clear() now also clears the fuzzy index's in-memory state, and SqliteMessageTextIndex.clear() / deserialize() clear the VectorBase together with the table. All three previously left stale vectors behind, so a later lookup could hit a ghost vector that maps to no row.
• SqliteMessageTextIndex.reload_from_db() orders by index_position and raises ValueError if the positions are not contiguous from 0. VectorBase row i must hold the embedding at index_position i; the old unordered SELECT relied on storage order. A database whose positions have gaps can only have been produced by the clear()/deserialize() bug above, and loading it silently misaligned lookups.
• SqliteRelatedTermsIndex.fuzzy_index is typed as SqliteRelatedTermsFuzzy (it never returns None), so the provider can call reload_from_db() without a cast.
• The TODO in conversation_base.py is resolved and the comment updated.

Tests

• test_sqlite_rollback_reloads_in_memory_indexes: adds terms and messages inside a transaction that raises; checks the in-memory state matches the tables afterwards, and that a retry persists what the rollback dropped, with lookups confirming vectors line up with rows.
• test_sqlite_clear_resets_in_memory_indexes, test_sqlite_rollback_on_fresh_provider.
• test_add_messages_with_indexing_retry_after_rollback: end-to-end through add_messages_with_indexing, failing once in update_conversation_timestamps (after the indexes were updated), then retrying.
• test_streaming_retry_after_rollback_of_later_batch: same through add_messages_streaming with the first batch committed and the second rolled back.
• TestSqliteMessageTextIndexInMemoryState: clear() and deserialize() reset the VectorBase; non-contiguous index_position values are rejected.

All new tests fail on main and pass with this change. make is green (953 tests).

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The focused recovery changes align cached indexes with database state, with rollback-and-retry regression coverage and no identified blocking issues.

No open issues 🎉

What changed in this PR

Fixes #362 by restoring SQLite-backed in-memory indexes after transaction rollback, allowing retries on the same storage provider to persist and search terms correctly.

Changes:

• Reload fuzzy-term and message-text index state from the database after rollback.
• Clear cached index state alongside table data during clear and message-text deserialization operations.
• Add regression tests for rollback recovery and ingestion retries, and update documentation.

File	Description
tests/​test_sqlitestore.py	Tests index restoration and retry after rollback.
tests/​test_add_messages_with_indexing.py	Tests ingestion retry after a late transaction failure.
src/​typeagent/​storage/​sqlite/​related_terms_index.py	Extracts reusable state reloading and narrows the fuzzy-index property type.
src/​typeagent/​storage/​sqlite/​provider.py	Reloads both indexes after rollback and clears fuzzy-index state on reset.
src/​typeagent/​storage/​sqlite/​message_index.py	Adds ordered vector reloading and clears vectors during reset and deserialization.
src/​typeagent/​knowpro/​conversation_base.py	Updates the comment to describe rollback recovery and removes the resolved TODO.
CHANGELOG.md	Documents rollback recovery and index-clearing fixes.

_{Review effort: Balanced}