Most of the time I reach for a vector database, the first version is slow in a way that is easy to miss. The queries return the right rows, the demo works, and then the table grows and every search starts scanning the whole dataset. Indexing is what fixes that, and LanceDB gives you more than one kind of index because "search" in a real application is rarely just vector similarity. You usually filter by price, by category, or by some text field at the same time.

This guide walks through the indexes LanceDB exposes in its Python SDK, with runnable code and a short note on when each one earns its place. I have spent a fair amount of time in LanceDB's index layer as a contributor, so the aim here is to be practical rather than exhaustive.

Setting up a table

Install the package and create a small table to work with. LanceDB stores data on disk and infers the schema from the first batch of rows, so you can start with plain dictionaries.

import lancedb db = lancedb.connect("./lancedb-data") rows = [ {"id": 1, "category": "audio", "price": 79.0, "description": "wireless over ear headphones with noise cancelling", "vector": [0.1, 0.2, 0.3, 0.4]}, {"id": 2, "category": "audio", "price": 19.0, "description": "wired earbuds with inline microphone", "vector": [0.2, 0.1, 0.4, 0.3]}, {"id": 3, "category": "camera", "price": 499.0, "description": "mirrorless camera body with fast autofocus", "vector": [0.9, 0.8, 0.1, 0.2]}, ] table = db.create_table("products", data=rows)

The vector column is the one we will search by similarity. The other columns are the ones we will filter on, and each of them can carry its own index.

Vector indexes

A vector index speeds up nearest neighbour search. Without one, LanceDB compares the query against every row, which is fine for a few thousand rows and painful past that.

The default type is IVF_PQ. IVF groups vectors into partitions so a query only looks inside the closest partitions, and PQ compresses each vector so the index stays small.

table.create_index( metric="l2", vector_column_name="vector", index_type="IVF_PQ", num_partitions=256, num_sub_vectors=16, )

Two things are worth knowing before you copy this. A vector index has to be trained, so it needs a reasonable number of rows to be useful. On a tiny table you can skip it and let LanceDB scan. And the metric you train with has to match the metric you search with, or the results will be wrong in a quiet way that is hard to debug.

If index size or recall is a concern, LanceDB also exposes scalar quantization through IVF_SQ. It compresses each vector with scalar quantization instead of product quantization, which keeps more of the original precision at the cost of a larger index. You switch to it by changing one argument.

table.create_index( metric="l2", vector_column_name="vector", index_type="IVF_SQ", num_partitions=256, )

Searching is the same regardless of which vector index you built.

results = table.search([0.1, 0.2, 0.3, 0.4]).limit(5).to_pandas()

Scalar indexes for filters

This is the part that people skip and then wonder why their filtered searches are slow. If you run a vector search and also filter by a column, LanceDB can use a scalar index on that column to narrow the candidates before it does the distance work. No scalar index means the filter is applied by scanning.

A BTree index fits columns you compare with ranges or equality, such as a price.

table.create_scalar_index("price", index_type="BTREE")

A Bitmap index fits columns with a small number of distinct values, such as a category.

table.create_scalar_index("category", index_type="BITMAP")

Now a search that combines similarity with a filter can use both indexes. The prefilter=True argument tells LanceDB to apply the filter before the vector search rather than after, which is what you want when the filter removes most of the rows.

results = ( table.search([0.1, 0.2, 0.3, 0.4]) .where("price < 100", prefilter=True) .limit(5) .to_pandas() )

A quick rule that has held up for me: reach for BTree when the column is ordered and you filter by ranges, and reach for Bitmap when the column has a handful of repeated values. If a column holds lists of tags rather than single values, there is a LABEL_LIST index for that case.

Full text search

Vector search finds rows that are semantically close. It does not help when a user types an exact term and expects to see it. For that you want a full text search index on the text column.

table.create_fts_index("description")

Then you search the text directly by setting the query type.

results = ( table.search("wireless headphones", query_type="fts") .limit(5) .to_list() )

Full text search and vector search answer different questions, and real applications often need both. Keeping an index on the text column means you can offer a keyword search without standing up a separate search engine next to your database.

Checking what you built

After creating indexes it is worth confirming they exist, especially since some index builds run in the background. list_indices reports the indexes on a table along with the columns they cover.

for index in table.list_indices(): print(index)

If an index you expected is missing, the usual causes are a table too small to train a vector index, or a build that has not finished yet.

Choosing an index

A short version of everything above, for when you are deciding what to add to a new table:

  • Searching by vector similarity on more than a few thousand rows: build a vector index. Start with IVF_PQ, and move to IVF_SQ if you want higher precision and can accept a larger index.
  • Filtering by a numeric or ordered column: BTree.
  • Filtering by a low cardinality column such as a status or category: Bitmap.
  • Filtering by a column that holds lists of values: LABEL_LIST.
  • Matching exact words in a text column: a full text search index.

None of these are expensive to add, and the cost of leaving them out only shows up once the table is large enough that a scan hurts. I would rather add the obvious ones early than debug a slow query under load later.

Closing

LanceDB's indexing is quieter than its vector search, but it is where a lot of the real performance lives. The pattern I keep coming back to is simple. Put a vector index on the column you search by similarity, put scalar indexes on the columns you filter by, and add a full text index wherever users type exact terms. Once those are in place, the database does the narrowing for you instead of scanning, and the queries stay fast as the data grows.

If you want to go deeper, the LanceDB documentation covers each index type and its parameters, and the source is open if you want to see how a given index is wired from the Rust core through to the Python SDK.