Skip to content

[BUG] Migration d2e3f4a5b6c7 is not restartable: a leftover search_index_rebuilt blocks every later start #1853

Open

Description

@SWSAmor

Bug Description

Migration d2e3f4a5b6c7 (add script n-grams to full-text search) is not restartable on SQLite.
rebuild_sqlite_search_index() runs CREATE VIRTUAL TABLE search_index_rebuilt USING fts5(…)
(alembic/versions/d2e3f4a5b6c7_add_script_ngrams_to_full_text_search.py:42) with no
IF NOT EXISTS and no prior DROP. If the run is interrupted after that CREATE, the table stays in
the database, alembic_version still says bcdbd5a942ca, and every later start fails with
table search_index_rebuilt already exists. The store cannot be opened until someone drops the table
by hand.

We hit this live. During the 0.24.1 upgrade of a store with 4 459 entities, our MCP gateway killed
the server mid-migration: it missed the gateway's startup window, which is our gateway's problem, not
BM's. The killed run had committed bcdbd5a942ca and the empty FTS5 table search_index_rebuilt,
while its INSERT … SELECT rolled back. The next start failed on CREATE VIRTUAL TABLE search_index_rebuilt (table already exists), and BM stayed down until we restored a backup. No
data was lost: search_index was intact. But any interruption at that point (crash, kill, power
loss) leaves a store that cannot start.

Steps To Reproduce

Throwaway home (BASIC_MEMORY_CONFIG_DIR pointing at an empty directory) with one project:

  1. bm tool write-note --title probe --folder notes --content x. The DB is now at head and has its
    runtime search_index.
  2. Downgrade with BM's own alembic to the revision before the migration:
    cfg = Config()
    cfg.set_main_option("script_location", str(Path(basic_memory.__file__).parent / "alembic"))
    cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
    command.downgrade(cfg, "bcdbd5a942ca")
  3. Recreate what an interruption leaves behind:
    sqlite3 memory.db "CREATE VIRTUAL TABLE search_index_rebuilt USING fts5(x);"
  4. Start again, the same write-note call.
  5. Control: sqlite3 memory.db "DROP TABLE IF EXISTS search_index_rebuilt;", then start again.

Expected Behavior

Step 4 completes the migration. A re-run should be idempotent.

Actual Behavior

Step 4 exits 1 with table search_index_rebuilt already exists, and alembic_version stays at
bcdbd5a942ca. After the manual DROP in step 5, the start succeeds and reaches head.

Environment

  • OS: macOS 27.0.1
  • Python version: 3.13.17
  • Basic Memory version: 0.24.1
  • Installation method: Homebrew
  • Claude Desktop version (if applicable): n/a

Additional Context

search_index_rebuilt is a scratch name: nothing outside this migration uses it, so dropping it
before the CREATE is safe. The same applies to any other copy-and-rename FTS migration.

Possible Solution

op.execute("DROP TABLE IF EXISTS search_index_rebuilt") right before the CREATE in
rebuild_sqlite_search_index().

Activity

  1. aaronspruit commented on Oct 24, 2026

    @aaronspruit

    I had a similar issue, but it was because the SQLITE_TMPDIR was not writeable.

    My server runs in Kubernetes with readOnlyRootFilesystem: true, and /tmp is not writable. On the first start, d2e3f4a5b6c7 failed with (sqlite3.OperationalError) disk I/O error on DROP TABLE search_index. The failed run left the empty search_index_rebuilt table, and alembic_version stayed at bcdbd5a942ca. Every later start then failed with table search_index_rebuilt already exists.

    A manual DROP TABLE search_index_rebuilt did not fix it. The next start failed again with disk I/O error, and it left the table behind again.

  2. MbappeWU commented on Oct 25, 2026

    @MbappeWU
    Contributor

    I'll take this: make migration d2e3f4a5b6c7 restartable by dropping a leftover search_index_rebuilt before CREATE, with a regression that simulates the interrupted-run leftover table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions