Persistence migration guide¶

Version 0.5 completes the persistence lifecycle introduced in 0.4. Creating a store, opening an archive, saving, and refreshing from disk now have separate, explicit operations. Applications that adopted the recommended 0.4 API need no further persistence changes.

The 0.4 compatibility paths are no longer present: constructor file_path=, instance load(), context-manager persistence, and the unversioned archive reader have been removed.

API at a glance¶

store = VectorStore(dimensions=1536, normalize=True)
store.add(vectors, metadata)
store.save("vectors.npz")

store.save()  # Update the bound archive.

loaded = VectorStore.open("vectors.npz")
loaded.reload()  # Deliberately discard memory and reread the archive.

The persistence changes from 0.4 to 0.5 are:

Store-owned state¶

Most code that reads configuration or uses NumPy operations for inspection and prefiltering remains unchanged. The familiar property names are still present:

store.dimensions
store.normalize
store.file_path
store.vectors
store.metadata

The difference is ownership. Configuration cannot be assigned directly, and the vector and metadata views reject normal row mutation. Their views describe the rows present when the property was requested, so code should request a fresh view after add(), clear(), or reload().

The views are a supported inspection boundary, not tamper-proof snapshots. Deliberately mutating their backing storage through .base, private attributes, ctypes, or similar escape hatches is unsupported and can corrupt the store. Call .copy() when code needs an independently mutable full-array snapshot. For metadata, this copies the outer row array while preserving the opaque payload objects by reference.

Row retrieval deliberately treats vectors and metadata differently:

row = store.get(0)
if row is not None:
    vector, payload = row
    vector[0] = 10.0           # Independent copy; the store is unchanged.
    payload["reviewed"] = True  # Shared application metadata object.

Vectors have one uniform NumPy representation, so copying a single row gives the caller clear ownership at bounded cost. Metadata can be any Python object, so the store preserves payload identity instead of imposing a potentially expensive or invalid deep-copy policy. Applications that need immutable metadata can use frozen application objects or copy payloads themselves.

Repeated additions¶

The add(vectors, metadata) signature and validation rules do not change. In 0.4, every noninitial call concatenated the new rows with the complete store. In 0.5, the store reserves private row capacity and grows it geometrically when needed. Repeated small additions therefore copy existing rows only when the current allocation is full.

This does not add a public capacity setting or change row indexes. len(store), inspection, search, retrieval, and persistence continue to use only active rows. clear() discards reserved storage as well as active rows, so it does not retain metadata payloads through unused capacity. A caller-held inspection view can still keep its previous NumPy buffer and payload references alive until the view itself is released.

Search result ordering¶

Version 0.5 makes exact metric ties deterministic. Cosine and dot-product searches still rank larger values first, and Euclidean search still ranks smaller distances first. When computed values are equal, the lower original store row index comes first.

This tie break also decides which rows are returned when equal values cross the top_k boundary. A shuffled within_rows input does not change the result: the original store indexes, rather than positions in the filtered input, break the tie. No call-site change is required, but code that depended on an incidental NumPy partition order should update its expectations.

Thread safety¶

Version 0.5 defines the existing synchronization boundary without adding internal locks. Search, get(), and inspection may run concurrently on one instance only while store state and shared metadata payloads remain unchanged.

Applications must externally synchronize all access to an instance whenever any thread may call add(), clear(), reload(), or save(), or mutate a metadata payload shared with the store. Saves must not overlap in-memory mutation because archive replacement protects the destination path, not the consistency of vectors and metadata read from a changing instance.

Atomic replacement also does not coordinate separate store instances writing the same path. Applications with multiple writers must serialize them; without that coordination, each complete save may replace another and the last successful replacement wins.

Creating and saving a new store¶

Previously, the destination was supplied while constructing the store:

store = VectorStore(dimensions=1536, file_path="vectors.npz")
store.add(vectors, metadata)
store.save()

Create the in-memory store first, then bind its destination with the first save:

store = VectorStore(dimensions=1536)
store.add(vectors, metadata)
store.save("vectors.npz")

Later save() calls update the bound archive. Supplying another path performs a Save As operation and binds the new destination after the write succeeds. Calling save() before a store is bound raises ValueError rather than silently leaving the data unsaved.

Opening an existing archive¶

The old API required callers to repeat configuration that should belong to the archive:

store = VectorStore(
    dimensions=1536,
    file_path="vectors.npz",
    normalize=True,
)
store.load()

Open a version 1 archive directly:

store = VectorStore.open("vectors.npz")

open() restores dimensions and normalize from the archive, validates its contents, loads its rows, and binds its path. Applications no longer need to keep archive configuration separately or risk loading the same vectors with different semantics.

Paths may be strings or string-valued os.PathLike objects such as pathlib.Path. Empty paths raise ValueError, while bytes and other non-path values raise TypeError. A None path is meaningful only for save(), where it reuses an existing binding. A relative path is anchored to the current working directory when save(path) or open(path) binds it. The public file_path is therefore absolute, and later working-directory changes do not redirect pathless saves or reloads. Symlinks and .. components are not canonicalized.

The generic parameter still describes application metadata. It can be kept when useful:

from dataclasses import dataclass


@dataclass(frozen=True)
class ChunkMetadata:
    source: str
    chunk_index: int


store = VectorStore[ChunkMetadata].open("vectors.npz")

ChunkMetadata is an example application type, not a class provided by this library.

Refreshing from disk¶

Use reload() when another process may have changed the bound archive and the current in-memory changes should be discarded:

store = VectorStore.open("vectors.npz")

# Later, after the file may have changed:
store.reload()

reload() always attempts to read. It raises if the store is unbound, the file is missing, or the archive is invalid. A failed reload leaves the current in-memory vectors and metadata unchanged.

Replacing context-manager persistence¶

The earlier context manager saved automatically on exit:

with VectorStore(dimensions=1536, file_path="vectors.npz") as store:
    store.add(vectors, metadata)

Use an explicit save after the work succeeds:

store = VectorStore(dimensions=1536)
store.add(vectors, metadata)
store.save("vectors.npz")

Normal Python control flow already prevents the final line from running if add() raises. The persistence boundary is visible, and readers do not need to remember an implicit exit side effect.

There is no replacement autosave context manager. An explicit save() keeps the persistence boundary visible and lets the application decide whether work completed successfully enough to persist.

Migrating an archive created before 0.4¶

Older archives contain only vectors and metadata. They do not record their dimensions or whether vectors use normalized or raw semantics, so open() cannot construct a correct store from them.

Before upgrading to 0.5, use the 0.4 compatibility API once with the archive's original configuration:

legacy = VectorStore(
    dimensions=1536,
    file_path="legacy-vectors.npz",
    normalize=True,
)
legacy.load()
legacy.save()

This code emits transition warnings in 0.4 by design. The final save() rewrites the archive as format version 1 with format_version, dimensions, normalize, vectors, and metadata. It can then be opened by 0.5:

store = VectorStore.open("legacy-vectors.npz")

Applications that can recreate archives from source vectors and metadata may do that instead. NumPy Vector Store 0.5 cannot perform this conversion because the unversioned file does not contain enough information to reconstruct its configuration safely.

Removal schedule¶

Persistence boundaries that do not change¶

Metadata is stored in a pickle-backed NumPy object array. Archives remain trusted input and must not be opened from untrusted or unverifiable sources.

Current releases retain forward-reading support for self-describing format-version-1 archives written by 0.4. The test suite keeps a binary fixture generated by the published 0.4.0 package on Python 3.11 and NumPy 1.23.2. This does not restore the unversioned reader or promise that an older release can read an arbitrary future format.

Read and write failures retain the exception type owned by their source: filesystem OSError subclasses, schema ValueError, and NumPy, pickle, or application metadata-loading errors are not collapsed into a package-specific exception.

Saves use same-directory temporary files and atomic replacement, but the library does not add file locking, coordinate concurrent writers, or promise power-loss durability across every operating system and filesystem.