Python API Reference¶
This section contains the API reference for the Python API of LanceDB. Both synchronous and asynchronous APIs are available.
The general flow of using the API is:
- Use lancedb.connect or lancedb.connect_async to connect to a database.
- Use the returned lancedb.DBConnection or lancedb.AsyncConnection to create or open tables.
- Use the returned lancedb.table.Table or lancedb.AsyncTable to query or modify tables.
Installation¶
The following methods describe the synchronous API client. There is also an asynchronous API client.
Connections to db:// databases reject local-only options with
NotImplementedError. These include session and manifest_enabled=True on
connect, per-table storage and write options on create, and index_cache_size,
storage_options, or location on open (where the method accepts them).
Omit these options to use server-managed storage and caching. Connection-level
storage_options supported by an Enterprise deployment remain available.
The synchronous LanceDB Enterprise create_table uses the same positional order as the
LanceDB Embedded API: name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
embedding_functions. Prefer keyword arguments for optional parameters.
Connections (Synchronous)¶
lancedb.connect
¶
connect(uri: Optional [URI ] = None, *, api_key: Optional [str ] = None, region: str = 'us-east-1', host_override: Optional [str ] = None, sql_host_override: Optional [str ] = None, read_consistency_interval: Optional [timedelta ] = None, request_thread_pool: Optional [Union [int , ThreadPoolExecutor ]] = None, client_config: Union [ClientConfig , Dict [str , Any ], None] = None, storage_options: Optional [Dict [str , str ]] = None, session: Optional [Session ] = None, manifest_enabled: bool = False, namespace_client_impl: Optional [str ] = None, namespace_client_properties: Optional [Dict [str , str ]] = None, namespace_client_pushdown_operations: Optional [List [str ]] = None, **kwargs: Any ) -> DBConnection
Connect to a LanceDB database.
Parameters:
-
uri(, default:Optional [URI ]None) –The uri of the database. When
namespace_client_implis provided you may omituriand connect through a namespace client instead. -
api_key(, default:Optional [str ]None) –If presented, connect to LanceDB Enterprise or LanceDB Cloud (legacy). Otherwise, connect to a database on file system or cloud storage. Can be set via environment variable
LANCEDB_API_KEY. OAuth configuration is currently supported only byconnect_async; synchronousdb://connections require an API key. -
region(, default:str 'us-east-1') –The region to use for LanceDB Cloud (legacy).
-
host_override(, default:Optional [str ]None) –The URL of the LanceDB Enterprise deployment to connect to.
-
sql_host_override(, default:Optional [str ]None) –The LanceDB Enterprise SQL service endpoint override. The client connects lazily when SQL is first executed and retains that connection.
-
read_consistency_interval(, default:Optional [timedelta ]None) –The interval at which to check for updates to the table from other processes. If None, then consistency is not checked. For performance reasons, this is the default. For strong consistency, set this to zero seconds. Then every read will check for updates from other processes. As a compromise, you can set this to a non-zero timedelta for eventual consistency. If more than that interval has passed since the last check, then the table will be checked for updates. Note: this consistency only applies to read operations. Write operations are always consistent.
Stronger consistency is not free. The smaller the interval, the more often each read pays the cost of checking for updates against object storage, raising per-read latency and cost.
-
client_config(, default:Union [ClientConfig ,Dict [str ,Any ], None]None) –Configuration options for the HTTP client used by
db://connections. If a dict, then the keys are the attributes of the ClientConfig class. If None, then the default configuration is used. -
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. See available options at https://docs.lancedb.com/storage/
-
manifest_enabled(, default:bool False) –When true for LanceDB Embedded connections, use directory namespace manifests as the source of truth for table metadata. Existing directory-listed root tables are migrated into the manifest on access.
-
session(, default:Optional [Session ]None) –(For LanceDB Embedded only) A session to use for this connection. Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.
-
namespace_client_impl(, default:str None) –When provided along with
namespace_client_properties,connectreturns a namespace-backed connection by delegating to :func:connect_namespace. The value identifies which namespace implementation to load (e.g.,"dir"or"rest"). -
namespace_client_properties(, default:dict None) –Configuration to pass to the namespace client implementation. Required when
namespace_client_implis set. -
namespace_client_pushdown_operations(, default:list [str ]None) –Only used when
namespace_client_propertiesis provided. Forwards to :func:connect_namespaceto control which operations are executed on the namespace service (e.g.,["QueryTable", "CreateTable"]).
Examples:
For a local directory, provide a path for the database:
For object storage, use a URI prefix:
>>> db = lancedb.connect(
... "s3://my-bucket/lancedb",
... storage_options={
... "aws_access_key_id": "***",
... "aws_secret_access_key": "***",
... "aws_region": "us-east-1",
... },
... )
For Azure Blob Storage, credentials can be passed directly without setting environment variables:
>>> azure_storage_options = {
... "account_name": "some-account",
... "account_key": "some-key",
... }
>>> db = lancedb.connect(
... "az://my-container/my-database",
... storage_options=azure_storage_options,
... )
For tests and temporary data, use an in-memory database:
In-memory databases are not persisted. Tables are dropped when the last connection or table handle referencing them is closed.
Connect to LanceDB Enterprise or LanceDB Cloud (legacy):
>>> db = lancedb.connect("db://my_database", api_key="ldb_...",
... client_config={"retry_config": {"retries": 5}})
Connect to a namespace-backed database:
>>> db = lancedb.connect(namespace_client_impl="dir",
... namespace_client_properties={"root": "/tmp/ns"})
Returns:
-
conn() –DBConnection A connection to a LanceDB database.
lancedb.db.DBConnection
¶
Bases:
An active LanceDB connection interface.
Use close to release the connection's underlying resources, or use the connection as a context manager to close it automatically when leaving the block, including when an exception is raised.
Examples:
>>> import lancedb
>>> with lancedb.connect("memory://") as db:
... assert db.is_open()
>>> db.is_open()
False
close
¶
list_namespaces
¶
list_namespaces(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> NamespaceListing
List immediate child namespace names in the given namespace.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The parent namespace to list namespaces in. Empty list represents root namespace.
-
page_token(, default:Optional [str ]None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:Optional [int ]None) –The maximum number of results to return.
Returns:
-
–NamespaceListing Response containing namespace names and optional page_token for pagination.
create_namespace
¶
create_namespace(namespace_path: List [str ], mode: Optional [str ] = None, properties: Optional [Dict [str , str ]] = None) -> CreateNamespaceResponse
Create a new namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to create.
-
mode(, default:Optional [str ]None) –Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.
-
properties(, default:Optional [Dict [str ,str ]]None) –Properties to set on the namespace.
Returns:
-
–CreateNamespaceResponse Response containing the properties of the created namespace.
drop_namespace
¶
drop_namespace(namespace_path: List [str ], mode: Optional [str ] = None, behavior: Optional [str ] = None) -> DropNamespaceResponse
Drop a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to drop.
-
mode(, default:Optional [str ]None) –Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
-
behavior(, default:Optional [str ]None) –Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.
Returns:
-
–DropNamespaceResponse Response containing properties and transaction_id if applicable.
describe_namespace
¶
Describe a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to describe.
Returns:
-
–DescribeNamespaceResponse Response containing the namespace properties.
namespace_exists
¶
Check if a namespace exists.
Parameters:
-
namespace_id() –List [str ]The namespace identifier to check.
Returns:
-
–bool True if the namespace exists, False otherwise.
Raises:
-
–NotImplementedError If the connection type does not support namespace operations.
table_exists
¶
Check if a table exists.
Parameters:
-
table_id() –List [str ]The table identifier to check (full path including namespace segments and table name).
Returns:
-
–bool True if the table exists, False otherwise.
Raises:
-
–NotImplementedError If the connection type does not support namespace operations.
list_tables
¶
list_tables(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> TableListing
List tables in this database, one page at a time.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to list tables in. None or empty list represents root namespace.
-
page_token(, default:Optional [str ]None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:Optional [int ]None) –The maximum number of results to return, default 100. Zero returns an empty page without a continuation token.
Returns:
-
–TableListing Response containing table names and optional page_token for pagination.
table_names
¶
table_names(page_token: Optional [str ] = None, limit: Optional [int ] = None, *, namespace_path: Optional [List [str ]] = None) -> Iterable [str ]
List all tables in this database, in sorted order
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to list tables in. Empty list represents root namespace.
-
page_token(, default:Optional [str ]None) –The token to use for pagination. If not present, start from the beginning. Typically, this token is last table name from the previous page.
-
limit(, default:Optional [int ]None) –The maximum number of tables to return. None returns all tables; zero returns an empty list.
Returns:
-
Iterable of str–
create_storage_integration
¶
create_storage_integration(name: str , storage_prefix: str , *, aws_role_arn: Optional [str ] = None, aws_access_key_id_secret: Optional [Union [str , SecretReference ]] = None, aws_secret_access_key_secret: Optional [Union [str , SecretReference ]] = None, aws_region: Optional [str ] = None, storage_provider: Literal ['S3', 'AZURE', 'GCP', 'HUGGINGFACE'] = 'S3', azure_storage_account: Optional [str ] = None, azure_tenant_id_secret: Optional [Union [str , SecretReference ]] = None, azure_client_id_secret: Optional [Union [str , SecretReference ]] = None, azure_client_secret_secret: Optional [Union [str , SecretReference ]] = None, gcp_credentials_json_secret: Optional [Union [str , SecretReference ]] = None, huggingface_token_secret: Optional [Union [str , SecretReference ]] = None, namespace_path: Optional [List [str ]] = None) -> StorageIntegrationInfo
describe_storage_integration
¶
describe_storage_integration(name: str , *, namespace_path: Optional [List [str ]] = None) -> StorageIntegrationInfo
list_storage_integrations
¶
list_storage_integrations(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
drop_storage_integration
¶
create_table
¶
create_table(name: str , data: Optional [DATA ] = None, schema: Optional [Union [Schema , LanceModel ]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional [List [EmbeddingFunctionConfig ]] = None, *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, data_storage_version: Optional [str ] = None, enable_v2_manifest_paths: Optional [bool ] = None) -> Table
Create a Table in the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to create the table in. Empty list represents root namespace.
-
data(, default:Optional [DATA ]None) –User must provide at least one of
dataorschema. Acceptable types are:-
list-of-dict
-
pandas.DataFrame
-
pyarrow.Table or pyarrow.RecordBatch
-
-
schema(, default:Optional [Union [Schema ,LanceModel ]]None) –Acceptable types are:
-
pyarrow.Schema
Plain binary values are cast into blob columns declared with blob. To construct Arrow record batches explicitly, use blob_array.
-
-
mode(, default:str 'create') –The mode to use when creating the table. Can be "create", "overwrite", or "exist_ok". By default, if the table already exists, an exception is raised. If you want to overwrite the table, use mode="overwrite". To open an existing table without adding data, use mode="exist_ok".
-
exist_ok(, default:bool False) –If a table by the same name already exists, then raise an exception if exist_ok=False. If exist_ok=True, then open the existing table; it will not add the provided data but will validate against any schema that's specified.
-
on_bad_vectors(, default:str 'error') –How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".
-
fill_value(, default:float 0.0) –The value to use when filling vectors. Only used if on_bad_vectors="fill". NaN or null elements and missing trailing elements are replaced with this value. Valid elements are preserved, and vectors longer than the expected dimension are truncated.
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/
To enable stable row IDs (row IDs remain stable after compaction, update, delete, and merges), set
new_table_enable_stable_row_idsto"true"in storage_options when connecting to the database. -
data_storage_version(, default:Optional [str ]None) –Deprecated. Set
storage_optionswhen connecting to the database and setnew_table_data_storage_versionin the options. -
enable_v2_manifest_paths(, default:Optional [bool ]None) –Deprecated. Set
storage_optionswhen connecting to the database and setnew_table_enable_v2_manifest_pathsin the options.
Returns:
-
–LanceTable A reference to the newly created table.
-
!!! note–The vector index won't be created by default. To create the index, call the
create_indexmethod on the table.
Examples:
Can create with list of tuples or dictionaries:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
... {"vector": [0.2, 1.8], "lat": 40.1, "long": -74.1}]
>>> db.create_table("my_table", data)
LanceTable(name='my_table', ...)
>>> db["my_table"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
You can also pass a pandas DataFrame:
>>> import pandas as pd
>>> data = pd.DataFrame({
... "vector": [[1.1, 1.2], [0.2, 1.8]],
... "lat": [45.5, 40.1],
... "long": [-122.7, -74.1]
... })
>>> db.create_table("table2", data)
LanceTable(name='table2', ...)
>>> db["table2"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
>>> import pyarrow as pa
>>> custom_schema = pa.schema([
... pa.field("vector", pa.list_(pa.float32(), 2)),
... pa.field("lat", pa.float32()),
... pa.field("long", pa.float32())
... ])
>>> db.create_table("table3", data, schema = custom_schema)
LanceTable(name='table3', ...)
>>> db["table3"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: float
long: float
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
It is also possible to create an table from [Iterable[pa.RecordBatch]]:
>>> import pyarrow as pa
>>> def make_batches():
... for i in range(5):
... yield pa.RecordBatch.from_arrays(
... [
... pa.array([[3.1, 4.1], [5.9, 26.5]],
... pa.list_(pa.float32(), 2)),
... pa.array(["foo", "bar"]),
... pa.array([10.0, 20.0]),
... ],
... ["vector", "item", "price"],
... )
>>> schema=pa.schema([
... pa.field("vector", pa.list_(pa.float32(), 2)),
... pa.field("item", pa.utf8()),
... pa.field("price", pa.float32()),
... ])
>>> db.create_table("table4", make_batches(), schema=schema)
LanceTable(name='table4', ...)
open_table
¶
open_table(name: str , *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, index_cache_size: Optional [int ] = None, branch: Optional [str ] = None, version: Optional [int ] = None) -> Table
Open a Lance Table in the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to open the table from. None or empty list represents root namespace.
-
index_cache_size(, default:Optional [int ]None) –Deprecated: Use session-level cache configuration instead. Create a Session with custom cache sizes and pass it to lancedb.connect().
Set the size of the index cache, specified as a number of entries
The exact meaning of an "entry" will depend on the type of index: * IVF - there is one entry for each IVF partition * BTREE - there is one entry for the entire index
This cache applies to the entire opened table, across all indices. Setting this value higher will increase performance on larger datasets at the expense of more RAM
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/
-
branch(, default:Optional [str ]None) –If provided, open a handle scoped to this branch instead of the default branch. Reads and writes operate in the branch's context.
-
version(, default:Optional [int ]None) –If provided, open the table pinned to this version, producing a read-only handle. Composes with
branch: when both are given, opens that branch at the version; otherwise opensmainat the version. Callcheckout_latestto return to a writable state.
Returns:
-
A LanceTable object representing the table.–
create_materialized_view
¶
create_materialized_view(name: str , source: MaterializedViewSource , *, select: SelectArg = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> MaterializedView
Define a materialized view named name over the table source.
LanceDB Embedded creation and refresh require a source build with the
experimental-functions Cargo feature. Published packages leave it
disabled; LanceDB Enterprise operations remain available.
The view is populated before creation returns. Pass
with_no_data=True to create only its definition and empty backing
table. The view is a normal table: it can be queried, indexed and
searched, and it appears in table_names.
Parameters:
-
name() –str The name of the view.
-
source() –MaterializedViewSource The name of the source table.
-
select(, default:SelectArg None) –The view's columns: column names,
(alias, SQL expression)pairs, or a dict of the same. Omitting it selects every source column, expanded against the source schema at creation time. -
where(, default:Optional [str ]None) –SQL predicate; only matching source rows appear in the view.
-
limit(, default:Optional [int ]None) –Cap the view at this many rows, in materialization order.
-
with_no_data(, default:bool False) –Skip the initial refresh and leave the backing table empty.
-
auto_refresh_enabled(, default:Optional [bool ]None) –Enable or disable automatic refresh in LanceDB Enterprise. When omitted, the server determines whether automatic refresh is enabled. LanceDB Embedded does not support automatic refresh.
-
auto_refresh_interval_seconds(, default:Optional [float ]None) –Positive, finite refresh interval in seconds in LanceDB Enterprise. When omitted, the server determines when to refresh.
Returns:
-
–MaterializedView
create_materialized_view_async
¶
create_materialized_view_async(name: str , source: MaterializedViewSource , *, select: SelectArg = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> Job [None]
Submit materialized-view creation and return its job.
The job may already be complete in LanceDB Embedded. In LanceDB
Distributed, creation submits CREATE MATERIALIZED VIEW through the SQL
server and the returned id is the registry job ID. That ID can be
reopened with :meth:open_job, and the handle supports registry
progress, events, cancellation, and wait while the job runs.
Wait for the job before opening or querying the view.
Servers without early registry-handle support return a connection-scoped
query job instead. Its ID cannot be reopened with :meth:open_job.
open_materialized_view
¶
Open the materialized view named name.
Raises ValueError if the table exists but is not a materialized
view.
list_materialized_views
¶
list_materialized_views(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
The names of the materialized views in this database.
Local listings inspect table schemas one page at a time.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_materialized_view
¶
Drop a materialized view.
In LanceDB Enterprise, drops wait for the SQL command to remove the catalog binding. Physical cleanup continues in the background.
drop_materialized_view_async
¶
Start dropping a materialized view and return its job.
The job may already be complete in LanceDB Embedded. In LanceDB
Enterprise, jobs have
type query and use the SQL query ID as id. Waiting confirms
catalog removal; physical cleanup continues in the background.
drop_table
¶
Drop a table from the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to drop the table from. Empty list represents root namespace.
drop_table_async
¶
Start dropping a table and return its cleanup job.
rename_table
¶
rename_table(cur_name: str , new_name: str , cur_namespace_path: Optional [List [str ]] = None, new_namespace_path: Optional [List [str ]] = None)
Rename a table in the database.
Parameters:
-
cur_name() –str The current name of the table.
-
new_name() –str The new name of the table.
-
cur_namespace_path(, default:Optional [List [str ]]None) –The namespace of the current table. None or empty list represents root namespace.
-
new_namespace_path(, default:Optional [List [str ]]None) –The namespace to move the table to. If not specified, defaults to the same as cur_namespace.
drop_all_tables
¶
Drop all tables from the database
Lists the tables in the selected namespace and drops them one at a time.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to drop all tables from. None or empty list represents root namespace.
namespace_client
¶
Get the equivalent namespace client for this connection.
For native storage connections, this returns a DirectoryNamespace pointing to the same root with the same storage options.
For namespace connections, this returns the backing namespace client.
For LanceDB Enterprise connections, this returns a RestNamespace with the same URI and authentication headers.
Returns:
-
–LanceNamespace The namespace client for this connection.
serialize
¶
Serialize this connection for reconstruction.
The returned string can be passed to :func:lancedb.deserialize_conn
to recreate an equivalent connection, e.g. in a remote worker.
Returns:
-
–str Serialized representation of this connection.
create_function
¶
create_function(definition: FunctionDefinition , *, secrets: Optional [Sequence [EnvVarSecret ]] = None, namespace_path: Optional [List [str ]] = None) -> FunctionVersion
Build and register a scalar Python function, then return its version.
The server builds the OCI image and registers the completed artifact.
This is the blocking counterpart of :meth:create_function_async.
LanceDB Embedded connections raise NotImplementedError.
Parameters:
-
definition() –FunctionDefinition A callable decorated with function.
-
secrets(sequence of EnvVarSecret, default:None) –One EnvVarSecret per credential the Function needs, each naming a Secret and the environment variable its value arrives in. The Function's source is unchanged by this; it reads the variable the way it already did.
-
namespace_path(list of str, default:None) –The namespace holding the Function. None or an empty list is the root namespace.
Examples:
create_function_async
¶
create_function_async(definition: FunctionDefinition , *, secrets: Optional [Sequence [EnvVarSecret ]] = None, namespace_path: Optional [List [str ]] = None) -> Job [FunctionVersion ]
Submit a scalar Python function for building and registration.
The server-side job builds the OCI image, then registers the completed
artifact. The returned job has type query and the SQL query UUID as
its ID. Waiting on it returns the immutable Function version.
LanceDB Embedded connections raise NotImplementedError.
get_function
¶
get_function(name: str , *, version: Optional [Union [int , str ]] = None, namespace_path: Optional [List [str ]] = None) -> FunctionVersion
Open an immutable Function version from the remote catalog.
version may be an integer or string. When omitted, the server
resolves the latest currently visible revision and returns that exact
immutable version.
list_functions
¶
list_functions(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [FunctionVersion ]
List every published immutable Function version in a namespace.
Functions in child namespaces are not included. Results are ordered by
Function name then numeric version. Remote connections use
SHOW FUNCTIONS HISTORY to include every revision.
LanceDB Embedded connections raise
NotImplementedError.
Examples:
List the identities available to use in Function-backed columns:
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_function
¶
Drop one published Function revision, preserving its siblings.
Returns True when the revision was removed and False for a repeated drop
or a missing name. A revision never published under an existing name is
an error. Dropping the last visible revision releases the name and
schedules physical cleanup in the background. When the remote server
reports dependent warnings, they are emitted as UserWarning before
this method returns. Use
:meth:drop_function_async to
access the query job, whose completion confirms catalog removal of the
revision. Local connections raise NotImplementedError.
drop_function_async
¶
drop_function_async(name: str , *, version: str , namespace_path: Optional [List [str ]] = None) -> Tuple [bool , Job ]
Drop one Function revision and return its SQL query job.
The boolean reports removal of the requested revision from the catalog;
sibling revisions and their grants remain available. The job has type
query and its ID is the SQL query UUID. Repeated waits confirm that
revision's removal. Only the last visible revision's drop releases the
name and schedules background object cleanup. LanceDB Embedded
connections raise NotImplementedError.
create_secret
¶
Create a named Secret in this database.
Fails if the name is taken, so a create never silently becomes a
rotation. Nothing reads the value back: it is bound to a Function by
name and resolved by the service when that Function runs. Local
connections raise NotImplementedError.
alter_secret
¶
Replace the credential behind an existing Secret.
Fails if it does not exist. Every Function bound to the Secret uses the
new value from its next job, and no new Function version is created --
which is how a rotation reaches columns pinned to a version registered
before it. LanceDB Embedded connections raise NotImplementedError.
list_secrets
¶
list_secrets(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
The names of every Secret in this database.
Names only. No method returns a stored credential, by construction
rather than by policy. LanceDB Embedded connections raise
NotImplementedError.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_secret
¶
Drop a Secret.
Functions bound to it fail at their next job, naming the Secret; that
is the revocation path. The name becomes free to reuse, and a new
Secret under it is picked up by everything still bound to that name.
LanceDB Embedded connections raise NotImplementedError.
describe_secret
¶
What this database records about a Secret: name and timestamps.
Never the value -- there is no code path that could return one. Local
connections raise NotImplementedError.
create_view
¶
create_view(name: str , query: str , *, namespace_path: Optional [List [str ]] = None) -> ViewDescription
Create a view: a named query the database plans on every read.
The query is planned once, at creation, so one that cannot be planned is refused now rather than at the first read. A view holds no rows, and its readers see its sources as they are at read time.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create. LanceDB Embedded connections raise
NotImplementedError.
import lancedb db = lancedb.connect("db://my_database") # doctest: +SKIP view = db.create_view( ... "adults", "SELECT name FROM people WHERE age >= 18" ... ) # doctest: +SKIP view.schema # doctest: +SKIP name: string
describe_view
¶
What this database records about a view: its defining query and the schema that query resolved to.
The schema is the one recorded at creation; a source altered since then
shows up when the view is read. LanceDB Embedded connections raise
NotImplementedError.
drop_view
¶
Drop a view and wait for catalog removal.
The tables it reads are untouched: a view holds no rows of its own. Use
:meth:drop_view_async to retain its SQL query job. Completion confirms
catalog removal; physical cleanup continues in the background.
LanceDB Embedded connections raise NotImplementedError.
drop_view_async
¶
Drop a view and return its SQL query job.
The job has type query and its ID is the SQL query UUID. Call
:meth:Job.wait to confirm catalog removal. Physical cleanup continues
in the background. Local connections raise NotImplementedError.
list_views
¶
list_views(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
The names of the views in one namespace.
Names only; a definition comes from
describe_view. Local
connections raise NotImplementedError.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
create_property_graph
¶
create_property_graph(name: str , nodes: Sequence [NodeTable ], edges: Sequence [EdgeTable ] = (), *, namespace_path: Optional [List [str ]] = None) -> PropertyGraph
Create a property graph over tables in its namespace.
Each NodeTable names the table holding one
label's nodes and its key column, and each
EdgeTable the table holding one label's
edges and the node keys its source and destination columns reference
-- the shape of SQL/PGQ's CREATE PROPERTY GRAPH. The graph stores
nothing else: a query reads its tables through SQL GRAPH_TABLE, as
they are when it runs.
The database checks the definition against the tables now, so one they
cannot satisfy is refused here rather than at the first query. A name
another property graph holds is an error. Local connections raise
NotImplementedError.
import lancedb from lancedb.graph import EdgeTable, Endpoint, NodeTable db = lancedb.connect("db://my_database") # doctest: +SKIP graph = db.create_property_graph( ... "social", ... nodes=[NodeTable("person", key="person_id", label="Person")], ... edges=[ ... EdgeTable( ... "knows", ... label="KNOWS", ... source=Endpoint("src_id", references=("person", "person_id")), ... destination=Endpoint( ... "dst_id", references=("person", "person_id") ... ), ... ) ... ], ... ) # doctest: +SKIP graph.describe().nodes[0].properties # doctest: +SKIP ['person_id', 'name']
open_property_graph
¶
Open the property graph named name.
The graph is described to check that it exists, so a name no property
graph holds raises HttpError with
status_code 404. Local connections raise NotImplementedError.
drop_property_graph
¶
Drop a property graph and wait for its definition to be deleted.
The tables it reads are untouched: a property graph holds no rows of
its own. Local connections raise NotImplementedError.
list_property_graphs
¶
list_property_graphs(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
The names of the property graphs in one namespace.
Names only; a definition comes from
PropertyGraph.describe. Local
connections raise NotImplementedError.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
open_job
¶
Open a server-side job by id, returning a handle with its record already populated.
The returned Job answers for its own state, specification, result, failure and event history, so there is no separate connection-level call for any of them.
Raises JobNotFoundError when the server has no such job, the way
open_table does for a missing table.
list_gpu_classes
¶
List the operator-configured Function GPU classes.
Each record names the class and reports its configured GPU model, memory, per-worker GPU limit, and CPU, memory, and ephemeral-storage limits. This describes configured hardware, not current free capacity.
LanceDB Enterprise only; the principal needs System Usage permission.
Raises NotImplementedError when the server predates
GET /v1/gpu/classes.
list_jobs
¶
list_jobs(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [JobInfo ]
List server-side jobs across the database's tables.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
cancel_job
¶
Request cancellation of a server-side job by id.
Returns True if the server accepted the cancellation, False if no such job exists. Cancelling an already-terminal job is a no-op success.
pause_job
¶
Pause a server-side job by id.
The job's workers drain and it stays parked until resumed. Returns "pausing", "already_paused", or "committing" -- a job finalizing its results cannot be parked; retry shortly.
resume_job
¶
Resume a paused server-side job by id.
Its workers pick their work back up from checkpoints. Returns "resumed", "still_pausing" -- the pause's worker drain is not confirmed yet; retry shortly -- or "not_paused".
execute_query
¶
execute_query(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> RecordBatchReader
Execute SQL and return a blocking Arrow reader.
This submits through :meth:execute_query_async and waits until the
initial result stream is readable. It does not wait for the full query
to finish. A statement that ran as a server-side SQL job exposes the
durable Job Registry id in the reader schema metadata under
x-lancedb-sql-job-id. Use :meth:execute_query_async when the
query handle itself is needed.
execute_query_async
¶
execute_query_async(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> Query
Start executing SQL and return its query handle.
parameters binds values to the statement's $1 / $name
placeholders. They travel as Arrow rather than as SQL text, so a
float keeps its exact value and type and a vector stays a compact
fixed-size list. A list binds by position, a dict binds by name, and
a one-row pyarrow.RecordBatch or pyarrow.Table binds both
ways. A 1-D numpy or pyarrow array becomes a fixed-size list of its
element type, the shape of a vector column:
.. code-block:: python
db.execute_query(
"SELECT id FROM docs ORDER BY distance(vector, $1) LIMIT $2",
parameters=[np.array([0.1, 0.2, 0.3], dtype=np.float32), 10],
)
Parameters are supported in queries, not in DDL or DML. With or
without parameters, the server chooses whether to stream rows on the
initial call or return a polling continuation. The reader follows
either path transparently. Dropping an unread handle or its reader
stops a directly streamed query; deferred execution requires explicit
cancellation. Open reader() directly: a streamed query needs its
reader to make progress and reports running until the stream ends.
Once a server-side SQL job exists, describe().job_id exposes its
durable Job Registry id while the query is still running. This is
separate from the connection-scoped query id.
LanceDB Embedded connections do not support SQL.
describe_query
¶
Describe a submitted SQL query by its connection-scoped id.
Resource listings return iterators. Use list(db.list_tables()) to collect all
table names. db.list_tables().tables is a list of the first page; limit
controls its page size and page_token resumes at a saved token. The same
patterns apply to list_namespaces(). Other resource listings use page_limit.
Resource listings (list_storage_integrations, list_secrets, list_views, list_jobs, list_functions, and
list_materialized_views) return lazy iterators. Use list(db.list_views()) to
collect synchronous results, or [name async for name in db.list_views()] with
an asynchronous connection. page_limit controls each request, and page_token
resumes a listing. Drain the iterator's cached results before saving its token.
lancedb.listing.Listing
¶
Bases: ,
A lazy synchronous iterator with inspectable pagination state.
For example::
for name in connection.list_views(page_limit=20):
print(name)
Use list(connection.list_views()) to collect all results.
num_page_results
¶
Return the number of cached items available without another request.
page_token
¶
Return the next request's token.
See AsyncListing.page_token for initial, cached, final-page, and error behavior.
lancedb.listing.TableListing
¶
Bases:
Table iterator retaining the current page as a list.
For example::
page = db.list_tables(limit=20)
print(page.tables, page.page_token)
for name in db.list_tables(limit=20):
print(name)
lancedb.listing.NamespaceListing
¶
Bases:
Namespace iterator retaining the current page as a list.
For example::
page = db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)
lancedb.Session
¶
A session for managing caches and object stores across LanceDB operations.
Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.
approx_num_items
¶
Get the approximate number of items cached in the session.
Returns:
-
–int The number of cached items across all caches
size_bytes
¶
Get the current size of the session caches in bytes.
Returns:
-
–int The total size of all caches in the session
default
¶
Create a session with default cache sizes.
This is equivalent to creating a session with 6GB index cache and 1GB metadata cache.
Returns:
-
–Session A new Session with default cache sizes
Catalogs (Synchronous)¶
Remote catalogs manage databases through a server's root namespace. Opened databases are ordinary connections. Dropping a database requires it to be empty.
lancedb.connect_catalog
¶
connect_catalog(endpoint: str , *, api_key: Optional [str ] = None, client_config: Optional [Union [ClientConfig , dict [str , Any ]]] = None, sql_host_override: Optional [str ] = None, read_consistency_interval: Optional [timedelta ] = None, oauth_config: Optional [OAuthConfig ] = None) -> Catalog
Connect synchronously to an HTTP(S) server's root catalog.
See connect_catalog_async for options. Local filesystem and object-store catalogs are not supported.
lancedb.catalog.Catalog
¶
A synchronous remote catalog returned by connect_catalog.
Examples:
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.create_database("analytics", exist_ok=True)
for name in catalog.list_databases():
print(name)
create_database
¶
Create a database, or open an existing one when exist_ok=True.
connect_database
¶
Connect to an existing database by its logical name.
list_databases
¶
list_databases(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> DatabaseNames
Iterate lazily over database names, fetching more results as needed.
page_token resumes from a saved token; None starts at the beginning.
page_limit sets the maximum names per REST response (not the total);
None uses the server default. Request errors are raised during iteration.
drop_database
¶
Drop a database. A nonempty database is an error unless cascade is set,
which drops everything in it first; the name stays taken until the server
finishes cleaning up their storage.
lancedb.catalog.DatabaseNames
¶
Bases:
A synchronous iterator of database names with pagination state.
Returned by Catalog.list_databases.
num_page_results
¶
Return the number of names available without another REST request.
page_token
¶
Return the token for the next REST request.
See page_token for initial, cached, final-page, and error behavior.
Authorization¶
Some catalogs support the authorization API. This is an API that can be used to bind access control lists (ACLs) to individual catalog objects.
Each ACL contains a subject, an object, and privilege.
Subjects¶
There are three kinds of subjects: principals, groups, and roles.
A principal uniquely identifies each user accessing the system. Each LanceDB Enterprise API key corresponds to a unique principal, and each OpenID user has a unique principal.
All principals have an immutable ID which never changes, and is safe to include
in log files. For example, the API key
sk_AAAQEAYEAUDAOCAJBIFQYDIOB4IBCEQTCQKRMFYYDENBWHA5DYPQ==== maps to the
principal ID pk_aaaqeayeaudaocajbifqydiob4. APIs such as list_acls will
return the principal ID, never the API key.
OpenID principals also typically have a human-readable name associated with
them. For example, a principal might be named bob@example.com and might have
a principal ID of 8a004400-9443-4fab-9997-b9372a7ca8fd. Whenever
possible, APIs will return the principal name for convenience. However,
principal names are mutable and can be changed by the identity provider.
Here is an example of getting information about an API key:
from lancedb.authz import Subject
catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
principal = catalog.authz.get_principal(Subject.principal_api_key(api_key))
print(f"Found principal {principal.principal_id}")
OpenID principals can be members of groups. Like principals, groups also have IDs and human-readable names. It is not possible to modify group membership via this API; it is managed by the OpenID identity provider.
Here is an example of getting information about a group based on its name:
from lancedb.authz import Subject
catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
group = catalog.authz.get_group(Subject.group_name("Engineering"))
print(f"Found group {group.group_id} with name {group.group_name}")
Roles can contain either principals or groups. Unlike groups, they are managed by the catalog, not the identity provider. Roles can also contain API key principals.
Here is an example of managing the reader role:
from lancedb.authz import Subject
catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
catalog.authz.add_role_binding(subject=Subject.principal_name("Alice"), role="reader")
More subject types may be added in the future.
Objects¶
An object is any resource in the system that we can attach authorization rules to.
The Python builders and Rust client share resource validation and wire encoding.
Methods also accept literal strings such as table:tenant/db:public$events;
malformed names for known resource types raise ValueError locally before a
request is sent. Unknown resource types are preserved for compatibility with
newer servers.
Object types include:
The System object, a singleton which represents the entire system. You can
refer to it as Object.system().
Database objects which represents specific databases. For example,
Object.database("analytics") represents the "analytics" database.
Table objects which represent specific tables. For example,
Object.table(database="analytics", name="events") represents the "events"
table inside the "analytics" database. Tables are in the default namespace (which
is named "public") unless a specific namespace is given.
Similarly, View objects represent specific views. For example,
Object.view(database="analytics", name="recent_events") represents the
"recent_events" view inside the "analytics" database. Views are in the default
namespace (which is named "public") unless a specific namespace is given.
Secret objects identify named credentials, for example,
Object.secret(database="analytics", name="api-key"). Function objects identify
registered functions, for example,
Object.function(database="analytics", name="caption"). Both default to the
public namespace. Pass namespace_path=["platform", "ml"] to select a nested
namespace. These objects identify resources for ACL operations; a secret object
contains the credential's name, not its value.
Storage integration objects identify database-owned storage configuration, for
example Object.storage_integration(database="default", name="raw"). Pass
namespace_path for nested namespaces. Their keyspace is separate from Secrets,
Functions, and relations; the wire name is storage_integration:default:public$raw.
More object types may be added in the future.
Access control lists¶
An access control list attaches a specific set of privileges to a specific
(subject, object) pair. So, for example, we might have an access control list
that gives bob@example.com the ability to read the foo table, that looks like
this:
from lancedb.authz import Object, Subject, Privilege
catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
catalog.authz.add_acl(
object=Object.table(database="analytics", name="foo"),
subject=Subject.principal_name("bob@example.com"),
privilege=Privilege.SELECT,
)
Reading a table generally requires USAGE on its database and namespace, plus SELECT on the table. Each object has at most one owner.
OWNERSHIP is a special privilege which grants all the other privileges. By default, the principal who created an object owns it. Only one subject can own an object; granting OWNERSHIP to a new subject removes it from the previous owner. OWNERSHIP cannot be deleted.
To delegate secret or function creation, grant CREATE_SECRET or
CREATE_FUNCTION on the containing namespace. This preserves namespace ownership:
for privilege in [Privilege.CREATE_SECRET, Privilege.CREATE_FUNCTION]:
catalog.authz.add_acl(
object=Object.namespace(database="analytics", namespace_path=["public"]),
subject=Subject.role("developer"),
privilege=privilege,
)
Both privileges also accept their string names in synchronous and asynchronous ACL methods.
Storage integration creation requires CREATE_STORAGE_INTEGRATION on its containing
namespace, together with database and namespace USAGE:
catalog.authz.add_acl(
object=Object.namespace(database="default"),
subject=Subject.role("developer"),
privilege=Privilege.CREATE_STORAGE_INTEGRATION,
)
catalog.authz.add_acl(
object=Object.storage_integration(database="default", name="raw"),
subject=Subject.role("reader"),
privilege=Privilege.USAGE,
)
Creation grants ownership to the creator. Describing an integration requires
USAGE on that integration; dropping it requires OWNERSHIP. Listing integrations
requires USAGE on the selected database and namespace. Describing and dropping
also require parent USAGE grants. Manage these grants through catalog.authz.
By default, the list_acls function lists all the access control entries in
the system. By adding a function parameter, the output is filtered to just ACLs
matching that parameter. For example, this code lists all the ACLs for the
"reader" role:
from lancedb.authz import Subject
catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
print(entry)
Or, when using the async API:
catalog = await lancedb.connect_catalog_async(endpoint, api_key=api_key)
async for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
print(entry)
Remote SQL¶
Submit SQL against a LanceDB Enterprise database through the connection.
The connected database and default_namespace_path=["public"] are used for
unqualified tables. Fully qualified references can still query other databases
and namespaces available to the same deployment. execute_query returns a
reader as soon as its initial result stream is available. execute_query_async
returns a query handle once submission is accepted; use it to inspect progress, open a reader,
or cancel the query. The SQL client is initialized by the first query and
retained for the lifetime of the connection. Query ids are random,
connection-scoped references rather than encoded SQL or durable resume tokens.
Both APIs start with one Flight call. The server can stream results on that call or return a polling continuation that the SDK follows automatically. This behavior is the same with and without parameters. Open the reader directly; a directly streamed query needs its reader to make progress. Dropping an unread handle or its reader stops direct streaming. Deferred execution continues until explicitly cancelled.
For example:
import lancedb
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
host_override="https://api.example.com",
sql_host_override="grpc+tls://sql.example.com:10026",
)
reader = db.execute_query(
"""
SELECT events.id, accounts.name
FROM analytics.public.events AS events
JOIN users.public.accounts AS accounts ON events.user_id = accounts.id
""",
default_namespace_path=["public"],
)
for batch in reader:
print(batch.num_rows)
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
print(query.describe().status)
for batch in query.reader():
print(batch.num_rows)
# Values bind to $1 / $name placeholders and travel as Arrow, so a float32
# stays a float32 and a vector stays a compact fixed-size list:
import numpy as np
reader = db.execute_query(
"SELECT id FROM docs ORDER BY distance(vector, $vector) LIMIT $k",
parameters={"vector": np.random.rand(768).astype(np.float32), "k": 10},
)
# The async connection exposes the same lifecycle without blocking:
# async_db = await lancedb.connect_async(
# "db://analytics",
# api_key="ldb_...",
# host_override="https://api.example.com",
# sql_host_override="grpc+tls://sql.example.com:10026",
# )
# reader = await async_db.execute_query("SELECT * FROM events")
# query = await async_db.execute_query_async("SELECT * FROM events")
# description = await async_db.describe_query(query.id)
# async for batch in await query.reader():
# print(batch.num_rows)
# await query.cancel()
Namespaces (Synchronous)¶
A namespace-backed connection resolves tables through a Lance namespace service instead of listing a storage directory.
lancedb.connect_namespace
¶
connect_namespace(namespace_client_impl: str , namespace_client_properties: Dict [str , str ], *, read_consistency_interval: Optional [timedelta ] = None, storage_options: Optional [Dict [str , str ]] = None, session: Optional [Session ] = None, namespace_client_pushdown_operations: Optional [List [str ]] = None) -> LanceNamespaceDBConnection
Connect to a LanceDB database through a namespace.
Parameters:
-
namespace_client_impl() –str The namespace client implementation to use. For examples:
- "dir" for DirectoryNamespace
- "rest" for REST-based namespace
- Full module path for custom implementations
-
namespace_client_properties() –Dict [str ,str ]Configuration properties for the namespace client implementation. Different namespace implementations have different config properties. For example, use DirectoryNamespace with {"root": "/path/to/directory"}
-
read_consistency_interval(, default:Optional [timedelta ]None) –The interval at which to check for updates to the table from other processes. If None, then consistency is not checked.
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend
-
session(, default:Optional [Session ]None) –A session to use for this connection
-
namespace_client_pushdown_operations(, default:Optional [List [str ]]None) –List of namespace operations to push down to the namespace server. Supported values:
- "QueryTable": Execute queries on the namespace server via namespace.query_table() instead of locally.
- "CreateTable": Execute table creation on the namespace server via namespace.create_table() instead of using declare_table + local write.
Default is None (no pushdown, all operations run locally).
Returns:
-
–LanceNamespaceDBConnection A namespace-based connection to LanceDB
lancedb.namespace.LanceNamespaceDBConnection
¶
Bases:
A LanceDB connection that uses a namespace for table management.
This connection delegates table URI resolution to a lance_namespace instance, while using the standard LanceTable for actual table operations.
table_names
¶
table_names(page_token: Optional [str ] = None, limit: Optional [int ] = None, *, namespace_path: Optional [List [str ]] = None) -> Iterable [str ]
List table names in the database.
If limit is None, return all table names; zero returns an empty list.
.. deprecated::
Use :meth:list_tables instead, which provides proper pagination support.
create_table
¶
create_table(name: str , data: Optional [DATA ] = None, schema: Optional [Union [Schema , LanceModel ]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional [List [EmbeddingFunctionConfig ]] = None, *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, data_storage_version: Optional [str ] = None, enable_v2_manifest_paths: Optional [bool ] = None) -> Table
open_table
¶
open_table(name: str , *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, index_cache_size: Optional [int ] = None, branch: Optional [str ] = None, version: Optional [int ] = None) -> Table
create_materialized_view
¶
create_materialized_view(name: str , source: MaterializedViewSource , *, select: 'SelectArg' = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> 'MaterializedView'
Define a materialized view over a table in the root namespace. See DBConnection.create_materialized_view.
create_materialized_view_async
¶
create_materialized_view_async(name: str , source: MaterializedViewSource , *, select: 'SelectArg' = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> Job [None]
open_materialized_view
¶
Open the materialized view named name.
list_materialized_views
¶
list_materialized_views(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> Listing [str ]
The names of the materialized views in the root namespace.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_materialized_view
¶
drop_materialized_view_async
¶
drop_table_async
¶
Start dropping a table and return its cleanup job.
rename_table
¶
rename_table(cur_name: str , new_name: str , cur_namespace_path: Optional [List [str ]] = None, new_namespace_path: Optional [List [str ]] = None)
list_namespaces
¶
list_namespaces(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> NamespaceListing
List child namespaces under the given namespace.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The parent namespace path to list children from. If None, lists root-level namespaces.
-
page_token(, default:Optional [str ]None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:int None) –Maximum number of namespaces to return.
Returns:
-
–NamespaceListing Response containing namespace names and optional page_token for pagination.
create_namespace
¶
create_namespace(namespace_path: List [str ], mode: Optional [str ] = None, properties: Optional [Dict [str , str ]] = None) -> CreateNamespaceResponse
Create a new namespace.
Parameters:
-
namespace_path() –List [str ]The namespace path to create.
-
mode(, default:str None) –Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.
-
properties(, default:Dict [str ,str ]None) –Properties to set on the namespace.
Returns:
-
–CreateNamespaceResponse Response containing the properties of the created namespace.
drop_namespace
¶
drop_namespace(namespace_path: List [str ], mode: Optional [str ] = None, behavior: Optional [str ] = None) -> DropNamespaceResponse
Drop a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace path to drop.
-
mode(, default:str None) –Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
-
behavior(, default:str None) –Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.
Returns:
-
–DropNamespaceResponse Response containing properties and transaction_id if applicable.
describe_namespace
¶
Describe a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to describe.
Returns:
-
–DescribeNamespaceResponse Response containing the namespace properties.
namespace_exists
¶
Check if a namespace exists.
Parameters:
-
namespace_id() –List [str ]The namespace identifier to check.
Returns:
-
–bool True if the namespace exists, False otherwise.
table_exists
¶
Check if a table exists.
Parameters:
-
table_id() –List [str ]The table identifier to check (full path including namespace segments and table name).
Returns:
-
–bool True if the table exists, False otherwise.
list_tables
¶
list_tables(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> TableListing
List all tables in this database with pagination support.
Parameters:
-
namespace_path(, default:List [str ]None) –The namespace to list tables in. None or empty list represents root namespace.
-
page_token(, default:str None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:int None) –The maximum number of results to return.
Returns:
-
–TableListing Response containing table names and optional page_token for pagination.
namespace_client
¶
Get the namespace client for this connection.
For namespace connections, this returns the backing namespace client that was provided during construction.
Returns:
-
–LanceNamespace The namespace client for this connection.
Tables (Synchronous)¶
LanceDB Embedded and LanceDB Enterprise tables use the same arguments for index creation, deletion,
index statistics, and search. For example, table.delete(where="id = 1"),
table.create_index("cosine", 2) (the deprecated vector API), and
table.search([1.0, 1.0], fast_search=True) work with either backend.
fast_search=True excludes rows that have not been indexed.
create_scalar_index defaults to replace=True on both backends. Both scalar
and full-text index creation accept wait_timeout. LanceDB Enterprise maintenance methods
accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops.
optimize submits a durable compaction job and returns once that submission is
accepted, not once compaction itself finishes. Pruning and index optimization
are not yet supported remotely, so it silently skips them and returns with
zeroed pruning statistics. Remote compaction_options support per-job
target_rows_per_fragment, max_source_rows, io_buffer_size, and batch_size
on compatible servers. Other non-default settings raise NotImplementedError
before submission; older-server errors are returned without dropping options.
The source-row budget selects whole tasks and can produce no work if the first
task exceeds it. An I/O buffer size of zero disables byte-based I/O backpressure.
Use a positive size to limit I/O read-ahead; this is not a total-memory limit.
Each field resolves as invocation option → table tuning → deployment/built-in
default. These options apply only to the submitted job and leave table policy
unchanged. Table enabled controls automatic scheduling; explicit manual runs
remain allowed.
table.optimize(compaction_options={
"target_rows_per_fragment": 100_000,
"max_source_rows": 1_000_000,
"io_buffer_size": 64 * 1024 * 1024,
"batch_size": 1024,
})
Removed Tantivy options produce the same explanatory errors on both backends.
The old LanceDB Enterprise keyword names predicate for delete, index_name for
drop_index, index_uuid for index_stats, and column for create_fts_index
remain available with deprecation warnings. Use where, name, index_name,
and field_names, respectively. For positional calls, use the argument order
documented by Table below; specify index options by keyword when migrating
code that used the former LanceDB Enterprise-only argument order.
lancedb.table.Table
¶
Bases:
A Table is a collection of Records in a LanceDB Database.
Examples:
Create using DBConnection.create_table (more examples in that method's documentation).
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data=[{"vector": [1.1, 1.2], "b": 2}])
>>> table.head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
b: int64
----
vector: [[[1.1,1.2]]]
b: [[2]]
Can append new data with Table.add().
Can query the table with Table.search.
>>> table.search([0.4, 0.4]).select(["b", "vector"]).to_pandas()
b vector _distance
0 4 [0.5, 1.3] 0.82
1 2 [1.1, 1.2] 1.13
Search queries are much faster when an index is created. See Table.create_index.
namespace
¶
Return the namespace path of the table (empty for the root namespace).
tags
¶
Tag management for the table.
Similar to Git, tags are a way to add metadata to a specific version of the table.
.. warning::
Tagged versions are exempted from the :py:meth:`cleanup_old_versions()`
process.
To remove a version that has been tagged, you must first
:py:meth:`~Tags.delete` the associated tag.
Examples:
.. code-block:: python
table = db.open_table("my_table")
table.tags.create("v2-prod-20250203", 10)
tags = table.tags.list()
branches
¶
Branch management for the table.
Branches are isolated, writable lines of history forked from another
branch (or version). Writes on a branch do not affect main.
embedding_functions
¶
Get a mapping from vector column name to it's configured embedding function.
current_branch
¶
The branch this table handle is scoped to, or None for main.
count_rows
¶
Count the number of rows in the table.
Parameters:
-
filter(, default:Optional [str ]None) –A SQL where clause to filter the rows to count.
to_pandas
¶
Return the table as a pandas DataFrame.
Parameters:
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned for backends that support Lance blob-aware pandas conversion.
-
**kwargs–Forwarded to PyArrow / Lance pandas conversion.
Returns:
-
–DataFrame
to_arrow
¶
Return the table as a pyarrow Table.
Parameters:
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.
Returns:
-
–Table
to_lance
¶
Return a LanceDB Embedded table as a lance.LanceDataset.
LanceDB Enterprise tables cannot be opened as Lance datasets by the
client. Use
table.search().to_arrow() to read them through the server.
Returns:
-
–LanceDataset
to_polars
¶
Return the table as a Polars LazyFrame.
In LanceDB Enterprise, use table.search().to_polars() to convert query
results to a Polars DataFrame.
Parameters:
-
batch_size–Passed to Polars as the maximum row count for scanned PyArrow record batches.
Note
The Polars streaming engine is not supported because it does not currently implement Python PyArrow dataset scans. Use the default engine when collecting this LazyFrame.
Returns:
-
–LazyFrame
create_index
¶
create_index(metric: str = 'l2', num_partitions: Optional [int ] = None, num_sub_vectors: Optional [int ] = None, vector_column_name: str = VECTOR_COLUMN_NAME, replace: bool = True, accelerator: Optional [str ] = None, index_cache_size: Optional [int ] = None, num_bits: int = 8, index_type: Literal ['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT'] = 'IVF_PQ', max_iterations: int = 50, sample_rate: int = 256, m: int = 20, ef_construction: int = 300, *, config: Optional [IndexConfigType ] = None, wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None, train: bool = True, target_partition_size: Optional [int ] = None)
Create an index on a column.
This method supports both the new unified API and the legacy API
for backwards compatibility. The new API takes the column name as the
first positional argument and an index configuration object via
config; the legacy API takes the distance metric as the first
argument plus separate vector_column_name / num_partitions /
etc. parameters, and emits a DeprecationWarning.
Parameters:
-
metric(, default:str 'l2') –For new API: the column name to index. For legacy API: the distance metric ("l2", "cosine", "dot", "hamming").
-
config(, default:IndexConfigType None) –The index configuration object. If provided, uses the new unified API. Can be one of: IvfFlat, IvfPq, IvfSq, IvfRq, HnswPq, HnswSq, BTree, Bitmap, LabelList, Fm, NGram, ZoneMap, FTS.
-
replace(, default:bool True) –Whether to replace an existing index on this column.
-
wait_timeout(, default:timedelta None) –LanceDB Enterprise only: how long to wait for the index build to finish. If not set, the call returns once the build is scheduled.
-
name(, default:str None) –Custom name for the index.
-
train(, default:bool True) –Whether to train the index with existing data.
Notes
On LanceDB Embedded tables the index is built before this method
returns. On LanceDB Enterprise tables it returns once the build is
scheduled. Until the build finishes, a vector index may be missing
from list_indices, a scalar or FTS
index may be listed with 0 indexed rows, and queries won't use the
index. To block until the index is ready, pass wait_timeout, call
wait_for_index, or use
create_index_async and wait
on its job when the server returned a job ID.
wait_timeout waits for the submitted build when the server returns
a job ID. On older servers that don't return one, it falls back to
polling the index by name. wait_for_index always waits by name.
When replacing an existing index, those name-based waits may return
before the new build finishes because the old index can still match.
create_index_async can track a specific build only when the server
returned a job ID; otherwise it returns an already-complete job and
callers must use wait_for_index for that build.
Examples:
New API (recommended):
>>> table.create_index(
... "vector", config=IvfPq(distance_type="l2")
... )
>>> table.create_index("category", config=BTree())
>>> table.create_index("content", config=FTS())
Legacy API (deprecated):
create_index_async
¶
create_index_async(column: str , *, config: IndexConfigType , replace: Optional [bool ] = None, wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None, train: bool = True) -> Job
Create an index, returning a handle to the indexing job.
Takes the same arguments as :meth:create_index. The job may already
be complete when returned; callers must not assume the index exists
until :meth:Job.wait returns.
In LanceDB Enterprise, the returned job tracks this submission only if
the server returns a job ID. If it does not, the job is already
complete and does not wait for the background build; use
wait_for_index and account for the replacement caveat described in
create_index.
get_table_overrides
¶
Read Enterprise table overrides for this table.
Table overrides configure Enterprise-managed background behavior such as automatic compaction, cleanup, and reindex execution. This method is supported only on remote Enterprise tables.
update_table_overrides
¶
Merge Enterprise table overrides into the current table overrides.
Only fields supplied in overrides are changed; unrelated existing
overrides are preserved by the remote service update flow.
reset_table_overrides
¶
Clear Enterprise table overrides for this table.
drop_index
¶
Drop an index from the table.
Parameters:
-
name() –str The name of the index to drop.
Notes
This does not delete the index from disk, it just removes it from the table. To delete the index, run optimize after dropping the index.
Use list_indices to find the names of the indices.
wait_for_index
¶
Wait for indexing to complete for the given index names. This will poll the table until all the indices are fully indexed, or raise a timeout exception if the timeout is reached.
Parameters:
-
index_names() –Iterable [str ]The name of the indices to poll
-
timeout(, default:timedelta ) –timedelta (seconds=300)Timeout to wait for asynchronous indexing. The default is 5 minutes.
create_scalar_index
¶
create_scalar_index(column: str , *, replace: bool = True, index_type: ScalarIndexType = 'BTREE', wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None)
Create a scalar index on a column.
Parameters:
-
column() –str The column to be indexed. Must be a boolean, integer, float, or string column.
-
replace(, default:bool True) –Replace the existing index if it exists.
-
index_type(, default:ScalarIndexType 'BTREE') –The type of index to create.
-
wait_timeout(, default:Optional [timedelta ]None) –The timeout to wait if indexing is asynchronous.
-
name(, default:Optional [str ]None) –The name of the index. If not provided, a default name will be generated.
Examples:
Scalar indices, like vector indices, can be used to speed up scans. A scalar
index can speed up scans that contain filter expressions on the indexed column.
For example, the following scan will be faster if the column my_col has
a scalar index:
>>> import lancedb
>>> db = lancedb.connect("/data/lance")
>>> img_table = db.open_table("images")
>>> my_df = img_table.search().where("my_col = 7",
... prefilter=True).to_pandas()
Scalar indices can also speed up scans containing a vector search and a prefilter:
>>> import lancedb
>>> db = lancedb.connect("/data/lance")
>>> img_table = db.open_table("images")
>>> img_table.search([1, 2, 3, 4], vector_column_name="vector")
... .where("my_col != 7", prefilter=True)
... .to_pandas()
Scalar indices can only speed up scans for basic filters using
equality, comparison, range (e.g. my_col BETWEEN 0 AND 100), and set
membership (e.g. my_col IN (0, 1, 2))
Scalar indices can be used if the filter contains multiple indexed columns and
the filter criteria are AND'd or OR'd together
(e.g. my_col < 0 AND other_col> 100)
Scalar indices may be used if the filter contains non-indexed columns but,
depending on the structure of the filter, they may not be usable. For example,
if the column not_indexed does not have a scalar index then the filter
my_col = 0 OR not_indexed = 1 will not be able to use any scalar index on
my_col.
create_fts_index
¶
create_fts_index(field_names: Union [str , List [str ]], *, ordering_field_names: Optional [Union [str , List [str ]]] = None, replace: bool = False, writer_heap_size: Optional [int ] = 1024 * 1024 * 1024, use_tantivy: bool = False, tokenizer_name: Optional [str ] = None, with_position: bool = False, base_tokenizer: BaseTokenizerType = 'simple', language: str = 'English', max_token_length: Optional [int ] = 40, lower_case: bool = True, stem: Optional [bool ] = None, remove_stop_words: Optional [bool ] = None, custom_stop_words: Optional [List [str ]] = None, ascii_folding: bool = True, ngram_min_length: int = 3, ngram_max_length: int = 3, prefix_only: bool = False, block_size: int = 128, document_granularity: DocumentGranularity = DocumentGranularity.ROW, wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None)
Create a full-text search index on the table.
Warning - this API is highly experimental and is highly likely to change in the future.
Parameters:
-
field_names() –Union [str ,List [str ]]The name of the field to index. Native FTS indexes can only be created on a single field at a time. To search over multiple text fields, create a separate FTS index for each field.
-
replace(, default:bool False) –If True, replace the existing index if it exists. Note that this is not yet an atomic operation; the index will be temporarily unavailable while the new index is being created.
-
writer_heap_size(, default:Optional [int ]1024 * 1024 * 1024) –Deprecated legacy Tantivy parameter. Any value other than the default raises an error.
-
ordering_field_names(, default:Optional [Union [str ,List [str ]]]None) –Deprecated legacy Tantivy parameter. Setting this raises an error.
-
tokenizer_name(, default:Optional [str ]None) –A compatibility alias for native tokenizer configs. Can be "raw", "default" or the 2 letter language code followed by "_stem". So for english it would be "en_stem". For new native FTS indexes, use
base_tokenizerdirectly;tokenizer_nameis a legacy compatibility alias and does not expose model-backed tokenizer names such asjieba/defaultorlindera/ipadic. -
use_tantivy(, default:bool False) –Deprecated legacy Tantivy parameter. Setting this to True raises an error.
-
with_position(, default:bool False) –If False, do not store the positions of the terms in the text. This can reduce the size of the index and improve indexing speed. But it will raise an exception for phrase queries.
-
base_tokenizer(, default:str "simple") –The base tokenizer to use for tokenization. Options are: - "simple": Splits text by whitespace and punctuation. - "whitespace": Split text by whitespace, but not punctuation. - "raw": No tokenization. The entire text is treated as a single token. - "code": Tokenizes source code and identifiers. - "ngram": N-Gram tokenizer. - "icu": ICU dictionary-based word segmentation. - "icu/split": ICU segmentation with simple-style delimiter splitting. - "jieba/": Jieba tokenizer loaded from Lance's language model home. - "lindera/": Lindera tokenizer loaded from Lance's language model home.
-
language(, default:str "English") –The language to use for stemming and stop-word removal. This is not the primary way to enable CJK tokenization.
-
max_token_length(, default:int 40) –The maximum token length to index. Tokens longer than this length will be ignored.
-
lower_case(, default:bool True) –Whether to convert the token to lower case. This makes queries case-insensitive.
-
stem(, default:bool None) –Whether to stem the token. Stemming reduces words to their root form. For example, in English "running" and "runs" would both be reduced to "run".
Noneuses the base tokenizer's default: False forcodeandngram, True otherwise. -
remove_stop_words(, default:bool None) –Whether to remove stop words. Stop words are common words that are often removed from text before indexing. For example, in English "the" and "and".
Noneuses the base tokenizer's default: False forcodeandngram, True otherwise. -
custom_stop_words(list of str, default:None) –Custom words that replace the built-in language stop words.
Noneuses the built-in list; an empty list explicitly uses no stop words. -
ascii_folding(, default:bool True) –Whether to fold ASCII characters. This converts accented characters to their ASCII equivalent. For example, "café" would be converted to "cafe".
-
ngram_min_length(, default:int 3) –The minimum length of an n-gram.
-
ngram_max_length(, default:int 3) –The maximum length of an n-gram.
-
prefix_only(, default:bool False) –Whether to only index the prefix of the token for ngram tokenizer.
-
block_size(, default:int 128) –The number of documents per compressed posting block. Must be 128 or 256. A value of 256 uses the experimental FTS V3 format and may introduce breaking changes.
-
document_granularity(, default:DocumentGranularity ) –ROW ROWtreats the selected text in one table row as one document.LIST_ELEMENTtreats each element of the deepest list on the field path as one document and returns its physical coordinates in_doc_indexfor matching queries. -
wait_timeout(, default:Optional [timedelta ]None) –The timeout to wait if indexing is asynchronous.
-
name(, default:Optional [str ]None) –The name of the index. If not provided, a default name will be generated.
Notes
Model-backed tokenizers such as jieba/default and lindera/ipadic
require tokenizer models in Lance's language model home. Set
LANCE_LANGUAGE_MODEL_HOME to override the default platform data
directory under lance/language_models.
add
¶
add(data: DATA , mode: AddMode = 'append', on_bad_vectors: OnBadVectorsType = 'error', fill_value: float = 0.0, progress: Optional [Union [bool , Callable , Any ]] = None, write_parallelism: Optional [int ] = None, allow_external_blob_outside_bases: bool = False) -> AddResult
Add more data to the Table.
Plain binary values are cast into blob columns using the table's schema. To construct Arrow record batches explicitly, use blob_array.
Parameters:
-
data() –DATA The data to insert into the table. Acceptable types are:
-
list-of-dict
-
pandas.DataFrame
-
pyarrow.Table or pyarrow.RecordBatch
-
-
mode(, default:AddMode 'append') –The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields.
-
on_bad_vectors(, default:OnBadVectorsType 'error') –How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".
-
fill_value(, default:float 0.0) –The value to use when filling vectors. Only used if on_bad_vectors="fill".
-
progress(, default:Optional [Union [bool ,Callable ,Any ]]None) –Progress reporting during the add operation. Can be:
-
Trueto automatically create and display a tqdm progress bar (requirestqdmto be installed)::table.add(data, progress=True)
-
A callable that receives a dict with keys
output_rows,output_bytes,total_rows,elapsed_seconds,active_tasks,total_tasks, anddone::def on_progress(p): print(f"{p['output_rows']}/{p['total_rows']} rows, " f"{p['active_tasks']}/{p['total_tasks']} workers") table.add(data, progress=on_progress)
-
A tqdm-compatible progress bar whose
totalandupdate()will be called automatically. The postfix shows write throughput (MB/s) and active worker count::with tqdm() as pbar: table.add(data, progress=pbar)
-
-
write_parallelism(, default:Optional [int ]None) –Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory.
-
allow_external_blob_outside_bases(, default:bool False) –Store blob URIs that sit outside registered blob bases. The row keeps a reference, so the object has to stay readable. Local tables only.
Returns:
-
–AddResult An object containing the new version number of the table after adding data.
merge_insert
¶
Returns a LanceMergeInsertBuilder
that can be used to create a "merge insert" operation
This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like "insert if not exists", "update or insert (i.e. upsert)", or even replace a portion of existing data with new data (e.g. replace all data where month="january")
The merge insert operation works by combining new data from a source table with existing data in a target table by using a join. There are three categories of records.
"Matched" records are records that exist in both the source table and the target table. "Not matched" records exist only in the source table (e.g. these are new data) "Not matched by source" records exist only in the target table (this is old data)
The builder returned by this method can be used to customize what should happen for each category of data.
Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values.
Parameters:
-
on() –Union [str ,Iterable [str ]]A column (or columns) to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column. Passing several columns matches on the composite key: a source row updates a target row only when it agrees on every one of them.
Examples:
>>> import lancedb
>>> data = pa.table({"a": [2, 1, 3], "b": ["a", "b", "c"]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> new_data = pa.table({"a": [2, 3, 4], "b": ["x", "y", "z"]})
>>> # Perform a "upsert" operation
>>> res = table.merge_insert("a") \
... .when_matched_update_all() \
... .when_not_matched_insert_all() \
... .execute(new_data)
>>> res
MergeResult(version=2, num_updated_rows=2, num_inserted_rows=1, num_deleted_rows=0, num_attempts=1, num_rows=3)
>>> # The order of new rows is non-deterministic since we use
>>> # a hash-join as part of this operation and so we sort here
>>> table.to_arrow().sort_by("a").to_pandas()
a b
0 1 b
1 2 x
2 3 y
3 4 z
search
¶
search(query: Optional [Union [VEC , str , 'PIL.Image.Image', Tuple , FullTextQuery ]] = None, vector_column_name: Optional [str ] = None, query_type: QueryType = 'auto', ordering_field_name: Optional [str ] = None, fts_columns: Optional [Union [str , List [str ]]] = None, fast_search: Optional [bool ] = None) -> LanceQueryBuilder
Create a search query to find the nearest neighbors of the given query vector. We currently support vector search and full-text search.
All query options are defined in LanceQueryBuilder.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> data = [
... {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]},
... {"original_width": 2000, "caption": "foo", "vector": [0.5, 3.4, 1.3]},
... {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]}
... ]
>>> table = db.create_table("my_table", data)
>>> query = [0.4, 1.4, 2.4]
>>> (table.search(query)
... .where("original_width > 1000", prefilter=True)
... .select(["caption", "original_width", "vector"])
... .limit(2)
... .to_pandas())
caption original_width vector _distance
0 foo 2000 [0.5, 3.4, 1.3] 5.220000
1 test 3000 [0.3, 6.2, 2.6] 23.089996
Parameters:
-
query(, default:Optional [Union [VEC ,str , 'PIL.Image.Image',Tuple ,FullTextQuery ]]None) –The targeted vector to search for.
-
default None. Acceptable types are: list, np.ndarray, PIL.Image.Image
-
If None then the select/where/limit clauses are applied to filter the table
-
-
vector_column_name(, default:Optional [str ]None) –The name of the vector column to search.
The vector column needs to be a pyarrow fixed size list type
-
If not specified then the vector column is inferred from the table schema
-
If the table has multiple vector columns then the vector_column_name needs to be specified. Otherwise, an error is raised.
-
-
query_type(, default:QueryType 'auto') –default "auto". Acceptable types are: "vector", "fts", "hybrid", or "auto"
-
If "auto" then the query type is inferred from the query;
-
If
queryis a list/np.ndarray then the query type is "vector"; -
If
queryis a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found.
-
-
If
queryis a string, then the query type is "vector" if the table has embedding functions else the query type is "fts"
-
Returns:
-
–LanceQueryBuilder A query builder object representing the query. Once executed, the query returns
-
selected columns
-
the vector
-
and also the "_distance" column which is the distance between the query vector and the returned vector.
-
take_offsets
¶
Take a list of offsets from the table.
Offsets are 0-indexed and relative to the current version of the table. Offsets are not stable. A row with an offset of N may have a different offset in a different version of the table (e.g. if an earlier row is deleted).
Offsets are mostly useful for sampling as the set of all valid offsets is easily known in advance to be [0, len(table)).
No guarantees are made regarding the order in which results are returned. Repeated offsets produce repeated rows, which makes this method suitable for sampling with replacement.
Parameters:
-
offsets() –list [int ]The offsets to take.
-
with_row_id(, default:bool False) –If True, include the "_rowid" column in the results.
Returns:
-
–LanceTakeQueryBuilder A query builder that can be executed to get the rows at the given offsets.
take_row_ids
¶
Take a list of row ids from the table.
Row ids are not stable and are relative to the current version of the table. They can change due to compaction and updates.
Results follow the requested row-id order. Repeated IDs produce repeated rows; missing or deleted IDs are omitted. This also applies to remote tables.
Unlike offsets, row ids are not 0-indexed and no assumptions should be made about the possible range of row ids. In order to use this method you must first obtain the row ids by scanning or searching the table.
Even so, row ids are more stable than offsets and can be useful in some situations.
There is an ongoing effort to make row ids stable which is tracked at https://github.com/lancedb/lancedb/issues/1120
Parameters:
-
row_ids() –list [int ]The row ids to take.
-
with_row_id(, default:bool False) –If True, include the "_rowid" column in the results.
Returns:
-
–LanceTakeQueryBuilder A query builder that can be executed to get the rows.
blob_columns
¶
Names of blob v1 and v2 columns that can be fetched by row id.
Struct-nested blobs use dotted paths. Blobs inside lists are omitted because a row id does not identify an individual list element.
fetch_blobs
¶
Materialize full blob bytes for column at the given rows.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
The result has the same length and order as row_ids. Null blobs
produce null slots; valid empty blobs produce b"".
row_ids may be a list, an integer Arrow array or chunked array,
or a query table with row ids. _rowid values stay valid after
compaction when the table has stable row ids.
One missing or deleted row ID rejects the entire batch; no partial
results are returned.
Blobs inside lists cannot be fetched by row id yet.
LanceDB Enterprise limits each request to 1024 row IDs and 64 MiB of blob
bytes. The client splits requests automatically and reads an individual
larger blob through the Range route. This method still materializes all
bytes in memory; for large values use :meth:fetch_blob_files.
The returned Arrow large_binary array contains payload bytes, not
SQL/query descriptor dictionaries. Use .to_pylist() to obtain
Python bytes or None values for copying into a blob column.
Select row IDs with search().with_row_id(True) first. Selection and
fetching are separate requests: keep the source unchanged between them.
fetch_blob_ranges
¶
Materialize row-specific byte ranges from a blob column.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
Each request is a three-item (row_id, offset, length) sequence.
Requests may be repeated or reordered, including multiple ranges for
the same blob.
The result has the same length and order as requests; null blobs
produce null slots and empty ranges on non-null blobs produce b"".
_rowid values stay valid after compaction when the table has stable
row ids.
One missing or deleted row ID rejects the entire batch; no partial
results are returned.
Row IDs can be obtained from a query with with_row_id(True).
LanceDB Enterprise reads the requested ranges through version-pinned blob
handles using HTTP byte-range requests.
fetch_blob_files
¶
fetch_blob_files(column: str , row_ids: Union [list [int ], Array , ChunkedArray , Table ]) -> 'list[Optional[BlobFile]]'
Open lazy, seekable :class:~lancedb._blob.BlobFile handles.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
Prefer this over :meth:fetch_blobs for large payloads. row_ids is
a list[int], an integer Arrow array or chunked array, or a query
pyarrow.Table carrying row identity via _rowid or a
_lance_row_id field on the blob descriptor. Null blob values return
None. LanceDB Enterprise requires server version 0.5.0 or newer.
_rowid values stay valid after compaction when the table has stable
row ids.
One missing or deleted row ID rejects the entire batch; no partial
handles are returned.
open_blob_file
¶
Open a seekable file from a SQL blob_descriptor result.
Accepts the descriptor dictionary or an Arrow struct scalar. Its source
pins the table version, branch, column and row identity, so opening does
not change this table's checkout. Null descriptors return None.
Payload bytes load only when the returned handle is read. An external
blob with an unspecified size may require a metadata request on opening.
open_blob_files
¶
Iterate over lazy files from descriptor dictionaries or Arrow scalars.
Accepts an Arrow array/chunked array or any iterable. Opens each handle
as iteration advances, preserving input order and nulls as None.
Close each non-null handle after use, normally with a with block.
delete
¶
Delete rows from the table.
This can be used to delete a single row, many rows, all rows, or sometimes no rows (if your predicate matches nothing).
Parameters:
-
where() –Union [str ,Expr ]The filter condition. Can be a SQL string or a type-safe :class:
~lancedb.expr.Exprbuilt with :func:~lancedb.expr.coland :func:~lancedb.expr.lit.The filter must not be empty, or it will error.
Returns:
-
–DeleteResult An object containing the new version number of the table after deletion.
Examples:
>>> import lancedb
>>> data = [
... {"x": 1, "vector": [1.0, 2]},
... {"x": 2, "vector": [3.0, 4]},
... {"x": 3, "vector": [5.0, 6]}
... ]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 2 [3.0, 4.0]
2 3 [5.0, 6.0]
>>> table.delete("x = 2")
DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 3 [5.0, 6.0]
If you have a list of values to delete, you can combine them into a
stringified list and use the IN operator:
update
¶
update(where: Optional [Union [str , Expr ]] = None, values: Optional [dict ] = None, *, values_sql: Optional [Dict [str , str ]] = None) -> UpdateResult
This can be used to update zero to all rows depending on how many rows match the where clause. If no where clause is provided, then all rows will be updated.
Either values or values_sql must be provided. You cannot provide
both.
Embeddings are not computed by this method. Updating an embedding source
column requires providing all associated vector columns in the same call;
otherwise a ValueError is raised. Use merge_insert to recompute embeddings
automatically.
Parameters:
-
where(, default:Optional [Union [str ,Expr ]]None) – -
values(, default:Optional [dict ]None) –The values to update. The keys are the column names and the values are the values to set.
-
values_sql(, default:Optional [Dict [str ,str ]]None) –The values to update, expressed as SQL expression strings. These can reference existing columns. For example, {"x": "x + 1"} will increment the x column by 1.
Returns:
-
–UpdateResult - rows_updated: The number of rows that were updated
- version: The new version number of the table after the update
Examples:
>>> import lancedb
>>> from lancedb.expr import col
>>> import pandas as pd
>>> data = pd.DataFrame({"x": [1, 2, 3], "vector": [[1.0, 2], [3, 4], [5, 6]]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 2 [3.0, 4.0]
2 3 [5.0, 6.0]
>>> table.update(where=col("x") == 2, values={"vector": [10.0, 10]})
UpdateResult(rows_updated=1, version=2)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 3 [5.0, 6.0]
2 2 [10.0, 10.0]
>>> table.update(values_sql={"x": "x + 1"})
UpdateResult(rows_updated=3, version=3)
>>> table.to_pandas()
x vector
0 2 [1.0, 2.0]
1 4 [5.0, 6.0]
2 3 [10.0, 10.0]
cleanup_old_versions
¶
cleanup_old_versions(older_than: Optional [timedelta ] = None, *, delete_unverified: bool = False) -> 'CleanupStats'
Clean up old versions of the table, freeing disk space.
Parameters:
-
older_than(, default:Optional [timedelta ]None) –The minimum age of the version to delete. If None, then this defaults to two weeks.
-
delete_unverified(, default:bool False) –Because they may be part of an in-progress transaction, files newer than 7 days old are not deleted by default. If you are sure that there are no in-progress transactions, then you can set this to True to delete all files older than
older_than.
Returns:
-
–CleanupStats The stats of the cleanup operation, including how many bytes were freed.
See Also
Table.optimize: A more comprehensive optimization operation that includes cleanup as well as other operations.
Notes
This function is not available in LanceDB Enterprise, which manages cleanup for you automatically.
compact_files
¶
Run the compaction process on the table. This can be run after making several small appends to optimize the table for faster reads.
Arguments are passed onto Lance's
lance.dataset.DatasetOptimizer.compact_files.
For most cases, the default should be fine.
See Also
Table.optimize: A more comprehensive optimization operation that includes cleanup as well as other operations.
Notes
This function is not available in LanceDB Enterprise, which manages compaction for you automatically.
optimize
¶
optimize(*, cleanup_older_than: Optional [timedelta ] = None, delete_unverified: bool = False, retrain: bool = False, compaction_options: Optional [CompactionOptions ] = None)
Optimize the on-disk data and indices for better performance.
.. note::
On a LanceDB Enterprise (remote) table, only compaction is
currently supported. Pruning old versions and index optimization
are silently skipped there instead of being attempted, so this
call returns successfully, with zeroed pruning statistics, once
the compaction job has been submitted; it does not wait for that
job to finish. Remote compaction supports per-job
``target_rows_per_fragment``, ``max_source_rows``, ``io_buffer_size``,
and ``batch_size``. Omitted fields inherit table tuning, then
deployment/built-in defaults. Other non-default compaction settings raise
``NotImplementedError`` before submission. The server must support
these SQL options; older-server errors are propagated without
dropping the requested settings.
Modeled after VACUUM in PostgreSQL.
Optimization covers three operations:
- Compaction: Merges small files into larger ones
- Prune: Removes old versions of the dataset
- Index: Optimizes the indices, adding new data to existing indices
Parameters:
-
cleanup_older_than(, default:Optional [timedelta ]None) –All files belonging to versions older than this will be removed. Set to 0 days to remove all versions except the latest. The latest version is never removed.
.. warning::
Setting this to 0 deletes the data files of every older version, so any other reader or writer still using an older version of the table will fail. Only set it to 0 if no other process is working on this dataset. -
delete_unverified(, default:bool False) –Files leftover from a failed transaction may appear to be part of an in-progress operation (e.g. appending new data) and these files will not be deleted unless they are at least 7 days old. If delete_unverified is True then these files will be deleted regardless of their age.
.. warning::
This should only be set to True if you can guarantee that no other process is currently working on this dataset. Otherwise the dataset could be put into a corrupted state. -
retrain(, default:bool False) –This parameter is no longer used and is deprecated.
-
compaction_options(, default:Optional [CompactionOptions ]None) –Options that control file compaction. For large rows, derive a lower
target_rows_per_fragmentfrom the expected row size to bound later passes. This does not cap the first pass over an existing oversized fragment; see CompactionOptions.
Notes
The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations.
list_indices
¶
List all indices that have been created with Table.create_index
On LanceDB Enterprise tables, an index whose build hasn't finished may
be missing (vector indices) or listed with 0 indexed rows (scalar and
FTS indices), and num_unindexed_rows may be None. Use
wait_for_index to wait until an
index is ready.
tokenize
¶
tokenize(query: str , *, column: Optional [str ] = None, index_name: Optional [str ] = None) -> Iterable [FtsToken ]
Tokenize a query using the tokenizer configured on an FTS index.
Specify exactly one of column or index_name.
Model-backed tokenizers such as jieba/* and lindera/* are
rebuilt in the client process from index metadata. In LanceDB Enterprise,
this means the same tokenizer model files must also exist locally
(under LANCE_LANGUAGE_MODEL_HOME) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
differ from what the server's index produces. If the model files are
missing, an error naming the tokenizer is raised.
index_stats
¶
Retrieve statistics about an index
Parameters:
-
index_name() –str The name of the index to retrieve statistics for
Returns:
-
–IndexStatistics or NoneThe statistics about the index. Returns None if the index does not exist.
add_columns
¶
add_columns(transforms: Mapping [str , str | FunctionApplication ] | FunctionApplication | Field | List [Field ] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None = None, *, computed: Dict [str , str ] | None = None, auto_refresh_enabled: bool | None = None, auto_refresh_interval_seconds: int | None = None, auto_refresh_resource_budget: Optional [Dict [str , int ]] = None)
Add new columns with defined values.
Parameters:
-
transforms(, default:Mapping [str ,str |FunctionApplication ] |FunctionApplication |Field |List [Field ] |Schema |DATA |RecordBatch |RecordBatchReader |DataFrame | NoneNone) –pa.Field, List[pa.Field], pa.Schema, pandas.DataFrame, polars.DataFrame, pa.Table, pa.RecordBatch, pa.RecordBatchReader A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, a pyarrow Field or Schema can be provided to add new columns with the specified data types. The new columns will be initialized with null values.
A mapping with one
FunctionApplicationvalue keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come fromrename(columns=...). Function columns are supported only in LanceDB Enterprise.On local tables, a pandas or Polars DataFrame, PyArrow Table, RecordBatch, or RecordBatchReader supplies precomputed new-column values. Supply exactly one row per current live table row, in storage scan order; pandas indexes are ignored. The input contains only new columns. Readers are consumed once, and their batch boundaries need not match table fragments. Keep the table unchanged between computing these values and calling add_columns. Data inputs cannot be combined with computed columns. Remote tables do not support data inputs.
-
computed(, default:Dict [str ,str ] | NoneNone) –LanceDB Embedded declaration and refresh require a source build with the
experimental-functionsCargo feature, which published packages do not enable. LanceDB Enterprise operations remain available.A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order.
Unlike
transforms, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them fromrefresh_column. Declaring one therefore costs the same on a large table as on an empty one.A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped.
In LanceDB Enterprise the expression is planned by the server, and the refresh runs as a server job -- see
refresh_column_async. Cannot be combined withtransforms. -
auto_refresh_enabled(, default:bool | NoneNone) –Enable automatic refresh for each declared computed output. Defaults to True. LanceDB Embedded stores this option; scheduling requires a server.
-
auto_refresh_interval_seconds(, default:int | NoneNone) –Positive minimum interval after a successful refresh, in seconds. When omitted, the server determines when to refresh. Applies to every computed output in this call. Scheduling options require computed or Function columns.
-
auto_refresh_resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate CPU, memory, and GPU budget for automatic refreshes of a registered Function declaration.
cpu_millisandmemory_bytesare required;gpu_countdefaults to zero. This option cannot be used with SQL-expression columns. LanceDB Enterprise only.
Returns:
-
–AddColumnsResult version: the new version number of the table after adding columns.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
>>> table.add_columns(computed={"doubled": "x * 2"})
AddColumnsResult(version=2)
>>> table.refresh_column("doubled")
RefreshColumnResult(rows_filled=2, version=4)
>>> table.to_arrow().sort_by("x").to_pandas()
x doubled
0 1 2
1 2 4
refresh_column
¶
refresh_column(column: str , *, where: Optional [str ] = None, batch_size: Optional [int ] = None, max_concurrency: Optional [int ] = None, publication_interval_rows: Optional [int ] = None, resource_budget: Optional [Dict [str , int ]] = None, gpu_class: Optional [str ] = None, gpu_count: Optional [int ] = None, retry_policy: Optional [FunctionRetryPolicy ] = None, error_policy: Optional [Literal ['fail', 'skip']] = None, max_failed_fraction: Optional [float ] = None, max_failed_rows: Optional [int ] = None, row_timeout_seconds: Optional [float ] = None, init_timeout_seconds: Optional [float ] = None, num_workers: Optional [int ] = None, max_executor_processes_per_worker: Optional [int ] = None) -> 'RefreshColumnResult'
Fill the rows of a computed column that hold no value yet.
Local refresh requires the experimental-functions Cargo feature
in a source build. Published packages leave it disabled.
Declared with add_columns(computed=...), a column starts empty and
gets its values here. Rows appended since the last refresh are filled
by the next one, and rows whose inputs changed since they were computed
are recomputed; everything else is left as it is.
Supported for local tables only. For remote tables, a refresh runs as
a server job, through
refresh_column_async.
Parameters:
-
column() –str The name of the computed column to fill.
-
where(, default:Optional [str ]None) –SQL filter selecting the rows to refresh, as in
REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option. -
batch_size(, default:Optional [int ]None) –Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Has no effect on local expression-backed columns.
-
max_concurrency(, default:Optional [int ]None) –Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.
-
publication_interval_rows(, default:Optional [int ]None) –Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.
-
resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate resources for one registered Function refresh. The
cpu_millisandmemory_byteskeys are required;gpu_countis optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-levelgpu_countshorthand. LanceDB Enterprise only; local refreshes reject it. -
gpu_class(, default:Optional [str ]None) –GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.
-
gpu_count(, default:Optional [int ]None) –Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use
resource_budgetinstead. LanceDB Enterprise only. -
retry_policy(, default:Optional [FunctionRetryPolicy ]None) –Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.
-
error_policy(, default:Optional [Literal ['fail', 'skip']]None) –Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.
-
max_failed_fraction(, default:Optional [float ]None) –Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.
-
max_failed_rows(, default:Optional [int ]None) –Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.
-
row_timeout_seconds(, default:Optional [float ]None) –Deadline for one Function call. LanceDB Enterprise only.
-
init_timeout_seconds(, default:Optional [float ]None) –Deadline for Function initialization. LanceDB Enterprise only.
-
num_workers(, default:Optional [int ]None) –Worker count for an expression column; registered Function columns reject it and use
resource_budgetinstead. LanceDB Enterprise only. -
max_executor_processes_per_worker(, default:Optional [int ]None) –Cap on executor processes per Function worker. LanceDB Enterprise only.
Returns:
-
–RefreshColumnResult rows_filled: the number of rows given a value. version: the new version number of the table.
refresh_column_async
¶
refresh_column_async(column: str , *, where: Optional [str ] = None, batch_size: Optional [int ] = None, max_concurrency: Optional [int ] = None, publication_interval_rows: Optional [int ] = None, resource_budget: Optional [Dict [str , int ]] = None, gpu_class: Optional [str ] = None, gpu_count: Optional [int ] = None, retry_policy: Optional [FunctionRetryPolicy ] = None, error_policy: Optional [Literal ['fail', 'skip']] = None, max_failed_fraction: Optional [float ] = None, max_failed_rows: Optional [int ] = None, row_timeout_seconds: Optional [float ] = None, init_timeout_seconds: Optional [float ] = None, num_workers: Optional [int ] = None, max_executor_processes_per_worker: Optional [int ] = None) -> Job [RefreshColumnResult ]
Like :meth:refresh_column, but returns a handle to the refresh job
instead of blocking until it completes.
Local refresh requires a source build with experimental-functions.
Remote jobs do not require this feature.
The job may already be complete when returned; callers must not assume
the column is filled until :meth:Job.wait returns. Invalid input --
an unknown column, or one that is not computed -- raises here rather
than failing the job. For local tables, the job runs in-process; for
remote tables, it is the server's backfill job.
Parameters:
-
column() –str The name of the computed column to fill.
-
where(, default:Optional [str ]None) –SQL filter selecting the rows to refresh, as in
REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option. -
batch_size(, default:Optional [int ]None) –Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Lower this for slow Functions to stay within the executor's per-call deadline. Has no effect on local expression-backed columns.
-
max_concurrency(, default:Optional [int ]None) –Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.
-
publication_interval_rows(, default:Optional [int ]None) –Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.
-
resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate resources for one registered Function refresh. The
cpu_millisandmemory_byteskeys are required;gpu_countis optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-levelgpu_countshorthand. LanceDB Enterprise only; local refreshes reject it. -
gpu_class(, default:Optional [str ]None) –GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.
-
gpu_count(, default:Optional [int ]None) –Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use
resource_budgetinstead. LanceDB Enterprise only. -
retry_policy(, default:Optional [FunctionRetryPolicy ]None) –Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.
-
error_policy(, default:Optional [Literal ['fail', 'skip']]None) –Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.
-
max_failed_fraction(, default:Optional [float ]None) –Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.
-
max_failed_rows(, default:Optional [int ]None) –Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.
-
row_timeout_seconds(, default:Optional [float ]None) –Deadline for one Function call. LanceDB Enterprise only.
-
init_timeout_seconds(, default:Optional [float ]None) –Deadline for Function initialization. LanceDB Enterprise only.
-
num_workers(, default:Optional [int ]None) –Worker count for an expression column; registered Function columns reject it and use
resource_budgetinstead. LanceDB Enterprise only. -
max_executor_processes_per_worker(, default:Optional [int ]None) –Cap on executor processes per Function worker. LanceDB Enterprise only.
Returns:
-
–Job [RefreshColumnResult ]A job whose successful
waitreturns row counts plus the source and published table versions.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("computed_job_demo", [{"x": 1}, {"x": 2}])
>>> table.add_columns(computed={"doubled": "x * 2"})
AddColumnsResult(version=2)
>>> job = table.refresh_column_async("doubled")
>>> result = job.wait()
>>> result.rows_assigned
2
>>> job.status()
'finished'
function_errors
¶
function_errors(job_id: Optional [str ] = None, column: Optional [str ] = None, limit: Optional [int ] = None) -> 'FunctionErrors'
The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped with the input that failed and the error. This lists those records, newest job first, plus a summary for any fragment whose per-row detail was capped. LanceDB Enterprise only; reading errors needs read access to the table, since a message carries the value that failed.
Parameters:
-
job_id(, default:Optional [str ]None) –Only errors recorded by this job.
-
column(, default:Optional [str ]None) –Only errors on this column.
-
limit(, default:Optional [int ]None) –At most this many records (server default 10000, cap 100000).
Returns:
-
–FunctionErrors records,fragmentsandtruncated, the last saying whether the listing stopped at its limit. -
Each record identifies its row with ``fragment_id``, ``row_offset``,– -
the physical ``row_address`` (``(fragment_id << 32) |– -
row_offset``), and, when the table uses stable row IDs, the stable– -
``row_id`` read from the refresh's pinned table version. Prefer– -
``row_id`` for retries: compaction and rewrite can change the– -
physical fields, while a stable row ID survives them.– -
To retry the recorded rows with stable row IDs:– -
.. code-block:: python–errors = table.function_errors(column="embedding") row_ids = sorted( record.row_id for record in errors.records if record.row_id is not None ) if row_ids: row_id_list = ", ".join(str(row_id) for row_id in row_ids) database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowid IN ({row_id_list})" )
-
For a table without stable row IDs, collect the non-null physical– -
addresses before the table is compacted or rewritten and run the– -
SQL refresh:– -
.. code-block:: python–errors = table.function_errors(column="embedding") addresses = sorted( record.row_address for record in errors.records if record.row_address is not None ) if addresses: address_list = ", ".join(str(address) for address in addresses) database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowaddr IN ({address_list})" )
-
Records whose per-row detail was capped have no address and appear– -
only in ``fragments``; they cannot be selected individually here.–
Examples:
column_statuses
¶
Return exact status for Function-backed output columns.
All statuses describe one server table version. column=None returns
every Function output. This operation is available on LanceDB
Enterprise tables. A table handle pinned to an exact version requests
that version; a latest handle retains the normal read-freshness
behavior.
column_status
¶
Return exact status for one Function-backed output column.
wait_fresh
¶
Poll until the column has no stale or never-computed rows.
Rows already failed remain visible in the returned status and do not keep this method waiting. The method never submits or cancels work. Transport errors and unrecoverable status-evidence errors propagate. A typed finalization-pending response is retried; arbitrary failure responses are not.
alter_columns
¶
Alter column names and nullability.
Parameters:
-
alterations(, default:Iterable [Dict [str ,Any ]]()) –A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable.
Each alteration must specify at least one of "rename", "data_type", or "nullable". The legacy key "name" is also accepted for renaming. Unknown keys raise ValueError before any alterations are applied.
Local tables reject renaming source or vector columns used by an embedding function with ValueError, preserving the embedding binding.
Returns:
-
–AlterColumnsResult version: the new version number of the table after the alteration.
update_field_metadata
¶
Update per-field (column) metadata.
Parameters:
-
updates(, default:dict ()) –One or more dicts, each with:
- "path": str — dot-path to the field (e.g. "embedding" or "a.b.c").
- "metadata": dict[str, str | None] — keys to set; a value of
Nonedeletes that key. - "replace": bool, optional — replace the field's whole metadata map instead of merging (default False).
The following keys are treated specially, by convention, and should be used when appropriate:
- "lancedb:description": for a human-readable description of a field.
"lancedb:tag:<name>"for a user-defined key-value tag, where the suffix names the tag category; e.g. "lancedb:tag:model": "clip".- "lancedb:status" for status options ("production", "candidate", "deprecated", "archived") to designate the current life cycle state of this column.
Returns:
-
–UpdateFieldMetadataResult version: the new table version after the update.
replace_field_metadata
¶
Replace all metadata of a field in the schema.
.. deprecated:: 0.33.1
Use update_field_metadata
with replace=True instead.
Parameters:
-
field_name() –str The name of the field whose metadata is replaced.
-
new_metadata() –Dict [str ,str ]The new metadata, replacing all existing keys.
drop_columns
¶
Drop columns from the table.
Local tables reject dropping source or vector columns used by an embedding function with ValueError, preserving the embedding binding.
Parameters:
-
columns() –Iterable [str ]The names of the columns to drop.
Returns:
-
–DropColumnsResult version: the new version number of the table dropping the columns.
checkout
¶
Checks out a specific version of the Table
Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set.
This is a read-only operation that turns the table into a sort of "view"
or "detached head". Other table instances will not be affected. To make the
change permanent you can use the [Self::restore] method.
Any operation that modifies the table will fail while the table is in a checked
out state. To return the table to a normal state use
[Self::checkout_latest].
Parameters:
-
version() –Union [int ,str ]The version to check out. A version number (
int) or a tag (str) can be provided.
checkout_latest
¶
Ensures the table is pointing at the latest version
This can be used to manually update a table when the read_consistency_interval
is None
It can also be used to undo a [Self::checkout] operation
restore
¶
Restore a version of the table. This is an in-place operation.
This creates a new version where the data is equivalent to the specified previous version. Data is not copied (as of python-v0.2.1).
Parameters:
-
version(, default:int orstr None) –The version number or version tag to restore. If unspecified then restores the currently checked out version. If the currently checked out version is the latest version then this is a no-op.
uses_v2_manifest_paths
¶
Check if the table is using the new v2 manifest paths.
Returns:
-
–bool True if the table is using the new v2 manifest paths, False otherwise.
migrate_v2_manifest_paths
¶
Migrate the manifest paths to the new format.
This will update the manifest to use the new v2 format for paths.
This function is idempotent, and can be run multiple times without changing the state of the object store.
Danger
This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations.
You can use Table.uses_v2_manifest_paths to check if the table is already using the new path style.
lancedb.table.CompactionOptions
¶
Bases:
Options that control file compaction during table optimization.
Unspecified options use Lance's defaults for local tables. Remote tables
forward target_rows_per_fragment, max_source_rows, io_buffer_size,
and batch_size as per-job SQL options; other non-default settings are
unsupported. Omitted remote options inherit table tuning, then deployment/built-in
defaults. The server must support these SQL options. These settings do not
change table policy.
Compaction planning is row based. Lowering target_rows_per_fragment
based on the expected row size can bound later compaction passes once
oversized fragments have been rewritten. It does not split an existing
fragment, so the first pass over an oversized fragment is not subject to
that bound. max_bytes_per_file limits output file size, not compaction
memory. Source budgets keep whole planned tasks; if the first task exceeds
a budget, that run performs no compaction work.
Examples:
Derive a steady-state row target from the expected row size:
>>> desired_fragment_bytes = 750 * 1024 * 1024
>>> average_row_bytes = 1_500_000
>>> options: CompactionOptions = {
... "target_rows_per_fragment": max(
... 1, desired_fragment_bytes // average_row_bytes
... ),
... }
>>> await table.optimize(compaction_options=options)
target_rows_per_fragment
¶
Target rows per fragment; existing oversized fragments are not split.
max_bytes_per_file
¶
Maximum output data-file size; this does not bound compaction memory.
materialize_deletions
¶
Whether to rewrite fragments containing deleted rows (default: True).
materialize_deletions_threshold
¶
Minimum deleted-row fraction that makes a fragment eligible (default: 0.1).
io_buffer_size
¶
Input scan I/O read-ahead budget in bytes.
Zero disables byte-based I/O backpressure.
defer_index_remap
¶
Whether to defer index remapping during compaction (default: False).
index_remap_mode
¶
How to construct the old-to-new row-address mapping.
compaction_mode
¶
Whether compaction re-encodes data or uses binary copying.
binary_copy_read_batch_bytes
¶
Number of bytes read per batch during binary-copy compaction.
max_source_fragments
¶
Maximum number of source fragments compacted in one run.
max_source_rows
¶
Maximum live source rows per run, applied to whole planned tasks.
max_source_bytes
¶
Maximum source bytes per run, applied to whole planned tasks.
excluded_fragment_ids
¶
Fragment IDs to leave unchanged and use as planning boundaries.
max_overlays_per_fragment
¶
Maximum overlays before a fragment is fully compacted.
data_storage_version
¶
Data-file version or release selector for compacted output.
lancedb.table.FragmentStatistics
¶
lancedb.table.FragmentSummaryStats
¶
lancedb.table.TableStatistics
¶
lancedb.table.Tags
¶
Table tag manager.
list
¶
List all table tags.
Returns:
-
–dict [str ,Tag ]A dictionary mapping tag names to version numbers.
get_version
¶
Get the version of a tag.
Parameters:
-
tag() –str The name of the tag to get the version for.
create
¶
Create a tag for a given table version.
Parameters:
-
tag() –str The name of the tag to create. This name must be unique among all tag names for the table.
-
version() –int The table version to tag.
delete
¶
Delete tag from the table.
Parameters:
-
tag() –str The name of the tag to delete.
update
¶
Update tag to a new version.
Parameters:
-
tag() –str The name of the tag to update.
-
version() –int The new table version to tag.
lancedb.table.Branches
¶
Table branch manager.
create
¶
Create a branch and return a handle scoped to it.
Parameters:
-
name() –str Name of the new branch.
-
from_ref(, default:Optional [str ]None) –Source branch to fork from. Defaults to
main. -
from_version(, default:Optional [int ]None) –A specific version on
from_refto fork from. Defaults to latest.
checkout
¶
Check out an existing branch and return a handle scoped to it.
Parameters:
-
name() –str Name of the branch to check out.
-
version(, default:Optional [int ]None) –A specific version on the branch to pin. When set, the returned handle is a read-only view of that version; when omitted it tracks the branch's latest and stays writable.
cherry_pick
¶
Cherry-pick a branch onto main, or dry-run.
Parameters:
-
from_branch() –str Branch to cherry-pick from.
-
dry_run(, default:bool False) –When True, only preview. When False, attempt the cherry-pick.
Notes
A failed cherry-pick returns status="failed" instead of raising.
lancedb.RTEWriteSpec
¶
Specification enabling the realtime engine (RTE), which routes
merge_insert through Lance's MemWAL LSM-style write path.
Constructed via the bucket(...), identity(...), or unsharded()
classmethods, then optionally chain with_maintained_indexes(...) and
with_writer_config_defaults(...). A fresh spec maintains every index the
table has, including ones created later.
column
¶
Bucket and identity variants: the sharding column. None for unsharded.
maintained_indexes
¶
maintained_indexes: Optional [List [str ]] = <attribute 'maintained_indexes' of 'lancedb._lancedb.RTEWriteSpec' objects>
Indexes the RTE keeps up to date, or None for every supported one.
num_buckets
¶
Bucket variant only: the number of buckets.
spec_type
¶
Discriminator string identifying the variant ("bucket", "identity", or "unsharded").
writer_config_defaults
¶
writer_config_defaults: Dict [str , str ] = <attribute 'writer_config_defaults' of 'lancedb._lancedb.RTEWriteSpec' objects>
Default ShardWriter configuration recorded by this spec.
bucket
¶
Hash-bucket sharding by the unenforced primary key column.
identity
¶
Identity sharding — shard by the raw value of column.
column must be a deterministic function of the unenforced primary
key: every row with a given primary key must always produce the same
column value, or upserts of that key can land in different shards
and a stale version can win. Typically column is the primary key
itself or a stable attribute of it.
unsharded
¶
No sharding — every merge_insert call writes to a single
RTE shard.
with_maintained_indexes
¶
Set which indexes the RTE maintains. None (the default) is
every index the table has, including ones created later; a list is
verbatim, and an empty list maintains nothing.
with_writer_config_defaults
¶
Replace the default ShardWriter configuration recorded in the
MemWAL index, so every writer starts from the same defaults.
Functions and Jobs¶
A @function definition is also a local callable. Call a decorated function
directly with row inputs, or call a decorated class with its initialization fields
and then call the returned instance; call close() if the class defines it. This
path uses the current Python environment and does not create a registered
FunctionVersion, build the declared remote runtime, resolve Secrets, or execute
the remote Arrow adapter. A registered FunctionVersion call instead binds
col(...) inputs and returns a remote FunctionApplication.
from lancedb import function
@function
def normalize(text: str) -> str:
return text.strip().lower()
@function
class Scale:
def __init__(self, factor: float):
self.factor = factor
def __call__(self, value: float) -> float:
return value * self.factor
def close(self) -> None:
self.factor = 0.0
assert normalize(" Hello ") == "hello"
scaler = Scale(factor=2.0)
assert scaler(3.0) == 6.0
scaler.close()
Annotate a blob input with lancedb.BlobReader for lazy, seekable access, or
with bytes to materialize its contents before the Function runs. Reader
inputs infer the Blob V2 type without an explicit input schema. Readers are
scoped to each invocation and support read, readinto, seek, tell,
size, and cursor-independent read_range. For a local test or another
engine, construct BlobReader(read_range, size) with a callback over an
in-memory bytes fixture or the engine's byte-range reader; read_range does
not advance the cursor.
from lancedb import BlobReader
payload = b"0123456789"
def read_range(offset: int, length: int) -> bytes:
return payload[offset : offset + length]
with BlobReader(read_range, len(payload)) as reader:
assert reader.read_range(6, 4) == b"6789"
assert reader.tell() == 0
lancedb.BlobReader
¶
Bases:
Read a blob incrementally inside a Function.
Annotating a Function input with BlobReader requests a lazy, seekable
reader. Annotating it with bytes requests materialized bytes instead.
read_range does not change the sequential cursor. Readers are scoped
to the invocation and must not be retained for a later call.
A packaged Function's source imports the worker's lance_blob.BlobReader
in place of this class.
For local calls, construct a reader from a range-reading callable and the blob's size. The callable accepts an offset and length and returns bytes. This is also the interface for a reader backed by another engine.
Examples:
payload = b"hello world"
def read_range(offset, length):
return payload[offset : offset + length]
with BlobReader(read_range, len(payload)) as reader:
assert reader.read_range(6, 5) == b"world"
assert reader.tell() == 0
assert reader.read(5) == b"hello"
assert reader.tell() == 5
read_range does not change the sequential cursor. Use size() for
the declared payload size and close the reader when done; a local reader
owns no server state.
read_range
¶
Read a byte range without changing the cursor; truncate at EOF.
lancedb.functions.FunctionArtifact
¶
lancedb.functions.FunctionParameter
¶
lancedb.functions.FunctionResultField
¶
lancedb.functions.FunctionOutput
¶
lancedb.functions.FunctionSignature
¶
Bases:
Ordered inputs, output, and initialization fields of a Function.
initialization lists the fields of the single row each Function
instance is created with. Values come from each binding, not from the
Function version; the tuple is empty when the Function takes none.
lancedb.functions.PythonEnvironmentSpec
¶
Bases:
One LanceDB Enterprise-managed Python environment source.
lancedb.functions.function
¶
function(target: Optional [Callable [..., Any ]] = None, *, name: Optional [str ] = None, input_schema: Optional [Schema ] = None, output_schema: Optional [DataType | Field | Schema ] = None, pip: tuple [str , ...] | list [str ] = (), env: Optional [Mapping [str , str ]] = None, python_version: Optional [str ] = None, batch_size: Optional [int ] = None, max_concurrency: Optional [int ] = None, row_timeout_seconds: Optional [float ] = None, retry_policy: Optional [FunctionRetryPolicy ] = None, gpu: bool = False, gpu_class: Optional [str ] = None, conda: tuple [str , ...] | list [str ] = (), conda_channels: tuple [str , ...] | list [str ] = (), apt: tuple [str , ...] | list [str ] = (), image: Optional [str ] = None, code: Sequence [ModuleType ] = (), models: Sequence [HuggingFaceModel ] = ())
Prepare a Python function or class for remote Function registration.
Input and output signatures are inferred from supported annotations. A
parameter annotated T is non-nullable and binds only a non-nullable
column; annotate Optional[T] to bind a nullable one, and it receives
None for a null value. PyArrow fields are nullable unless declared
otherwise, so most columns need Optional. For Arrow types annotations
cannot express precisely, pass input_schema, output_schema, or
both; the one omitted is inferred. Scalar outputs must be non-nullable.
Every named-struct field may be nullable; Enterprise preserves the
struct's validity when the result is expanded into sibling columns.
Large values. A bytes parameter receives the whole value, which
is held in memory with its row and counts against the server's per-row
input limit. To pass large binary values, such as video, store them in a
Blob V2 column and annotate the parameter BlobReader, which streams.
Functions and classes. A decorated function is called once per row,
or once per batch when its parameters are annotated pyarrow.Array or
it takes one pyarrow.RecordBatch. These batch annotations require both
input_schema and output_schema explicitly: the annotations do not
specify Arrow element types or fields. This also applies to a class's
__call__ method. A decorated class is the same callable with state:
each remote instance calls __init__ once, calls
__call__ for every row or batch of every input it processes, and calls
close(), if defined, once when it retires. Load models, open clients,
and build rate limiters in __init__; every batch of the instance then
reuses them. An instance serves many batches of one binding, so state is
shared across them, but not across instances or workers.
Initialization. The annotated parameters of a class's __init__
are the Function's initialization fields. Their values belong to each
binding, not to the Function: function(text=col("body"), model="small")
binds column body and initializes every instance with model="small",
and another column can bind the same Function version with other values.
Initialization fields take booleans, integers, floats, strings, and lists
or structs of them (Annotated[dict, pa.struct(...)]). A parameter with
a default may be omitted from a binding; a null value then takes the
default. Use Secrets, not initialization, for credentials.
What travels with the Function. The artifact is a snapshot of the decorated source plus exactly the module-level names it references:
- modules, and classes and functions importable from an installed
package, become imports; the package must be declared in
piporconda; - literals (
None,bool,int,str,bytes, finitefloat, and tuples containing only these literals) are inlined; - functions and classes defined in
__main__, such as notebook cells, are packaged by source, together with what they reference; - modules and packages listed in
codeare shipped as Python source and imported normally, so helpers shared by several Functions live in one place.
A module imported by the packaged source -- including a direct import
inside a function or method body -- draws a warning at registration when
the client cannot resolve its top-level module and declares no package
dependencies, or when it resolves to local source outside code and
no declared requirement matches the imported module name. The warning
does not assert that the remote environment lacks the module:
pip/conda distribution names need not match the modules they
install, and the client cannot inspect the contents of a base or custom
image. Warnings follow normal Python warning filters and may be promoted
to errors. Declare every third-party import in pip or conda even
when it happens to be installed on the registration host.
Other global values, including Arrow types and schemas, lists, dictionaries,
sets, and NumPy arrays, cannot be captured as literals. Define them inside
the function or class body instead.
Closures, lambdas, nested definitions, mutable module-level
objects, code that reaches the module namespace another way
(globals()/eval, sys.modules, builtins), and a
non-standard __builtins__ are rejected where they can be seen and
otherwise unsupported. Changing any packaged source, including a module in
code, produces a new Function version; the environment built from
pip or conda is reused.
Parameters:
-
function(function or class) –The synchronous callable, or the class whose instances are called. For a function, annotate every parameter as
pyarrow.Arrayto opt into vectorized execution; a matching return annotation documents that the callable returns an Arrow array. -
name(, default:str None) –The remote Function name. Defaults to the callable name. Must start with an ASCII letter or underscore and contain only ASCII letters, digits, and underscores.
-
input_schema(, default:Schema None) –Explicit input fields in the exact order of the callable parameters. Required for batch annotations. When omitted, inputs are inferred from the parameter annotations.
-
output_schema(pyarrow.DataType, pyarrow.Field, or pyarrow.Schema, default:None) –Explicit scalar or named-struct output. Scalar outputs must be non-nullable. Required for batch annotations. When omitted, the output is inferred from the return annotation.
-
pip(sequence of str, default:()) –Pip requirements for the remote environment.
-
conda(sequence of str, default:()) –Conda packages for the remote environment, instead of
pip. -
conda_channels(sequence of str, default:()) –Conda channels in priority order; requires
conda. -
apt(sequence of str, default:()) –Debian packages (
"name"or"name=version") added to the remote image before the Python environment, for shared libraries pip and conda cannot provide, such as FFmpeg's fortorchcodec. Packages are unpacked without running their maintainer scripts. Works withpiporconda; requires image execution. -
image(, default:str None) –A digest-pinned registry image (
"host/repository@sha256:<digest>") the remote environment is built on instead of the default base. Pip requirements install only what the image lacks. -
env(mapping of str to str, default:None) –Environment variables included in the Function definition. Not for credentials -- these are ordinary configuration, stored with the Function and visible wherever it is. Prefer initialization for values that differ between bindings.
-
python_version(, default:str None) –Remote Python major/minor version. Defaults to the client version.
-
batch_size(, default:int None) –Maximum number of input rows supplied in one execution batch. Must be between 1 and 4294967295. Remote execution may use smaller batches. When omitted, the service chooses the batch size adaptively.
-
max_concurrency(, default:int None) –Maximum user-code calls in flight for one Function within one logical execution. Must be between 1 and 4294967295. A refresh can override this default for that execution only. Omitting it preserves the existing behavior. This is not an account-wide quota.
-
row_timeout_seconds(, default:float None) –Positive, finite deadline in seconds for each Python call: one row for scalar Functions and one batch for vectorized Functions. This does not govern class initialization. When omitted, the service applies its default per-call budget.
-
retry_policy(, default:FunctionRetryPolicy None) –Retry selected failed calls inside the Function, with bounded attempts and backoff. A refresh may override this default. Retrying a partial call must be safe; completed calls are not repeated. For batch Functions, skip after exhaustion applies to the whole failed call.
-
gpu(, default:bool False) –Whether every remote execution requires a GPU. The execution platform selects one compatible GPU for each worker. The requirement is part of the immutable Function version.
-
gpu_class(, default:str None) –Operator-defined GPU class for every execution of this Function, such as
"a10g"or"h100". Requiresgpu=Trueand follows the lowercase 63-character DNS-label rule. The class is part of the immutable Function version; per-refresh aggregate GPU budgets are selected throughrefresh_columninstead of registration. -
code(sequence of module, default:()) –Top-level modules or packages whose importable
.pyfiles ship with the Function and are importable by their own names. Package data files are not copied. To use a package that includes required data resources, install it withpip=orconda=and remove that package fromcode=so the copied Python package cannot shadow the installed distribution.@functionwarns when it skips files from a package incode=. -
models(sequence of HuggingFaceModel, default:()) –Immutable Hugging Face repository snapshots prepared in the worker node cache before user code starts. A Function may declare a repository once. Different revisions of one repository use different Function versions. Model Functions run offline with a private
HF_HOME; bindHF_TOKENthrough Secrets when the repository requires authentication.
Returns:
-
–FunctionDefinition A callable definition accepted by :meth:
lancedb.db.DBConnection.create_function, :meth:lancedb.db.AsyncConnection.create_function_asyncand :meth:lancedb.db.DBConnection.create_function_async.
Examples:
>>> from lancedb import function
>>> @function(pip=["numpy==2.2.0"])
... def score(value: float) -> float:
... return value * 2
>>> score(1.5)
3.0
>>> import pyarrow as pa
>>> @function(
... input_schema=pa.schema([pa.field("value", pa.int64(), nullable=False)]),
... output_schema=pa.int64(),
... )
... def batch_identity(value: pa.Array) -> pa.Array:
... return value
>>> batch_identity(pa.array([1, 2, 3])).to_pylist()
[1, 2, 3]
>>> @function(pip=["cupy-cuda12x"], gpu=True, gpu_class="a10g")
... def gpu_score(value: int) -> int:
... return value * 2
>>> gpu_score.registration_request.runtime.gpu
True
>>> gpu_score.registration_request.runtime.gpu_class
'a10g'
>>> @function
... class scale:
... def __init__(self, factor: float, offset: float = 0.0):
... self.factor, self.offset = factor, offset
... def __call__(self, value: float) -> float:
... return value * self.factor + self.offset
... def close(self):
... self.factor = 0.0
>>> scale_fn = scale(factor=2.0)
>>> scale_fn(1.5)
3.0
>>> scale_fn(2.0)
4.0
>>> scale_fn.close()
>>> [field.name for field in scale.registration_request.signature.initialization]
['factor', 'offset']
>>> import lancedb
>>> @function(
... input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
... output_schema=lancedb.blob("copy", nullable=False),
... )
... def copy_blobs(image: pa.Array) -> pa.Array:
... return image
Local execution. A function definition is callable with its decorated
input signature and runs the original function in the current Python
environment. A class definition is called with its initialization
arguments and constructs one instance; call that instance for row inputs,
reuse it as needed, and call close if it defines that method. This path
does not create a registered
FunctionVersion, build the declared
remote environment, resolve Secrets, or run the remote Arrow adapter or
sandbox. A registered FunctionVersion call is different: it binds
col(...) inputs and returns a remote
FunctionApplication.
scaled = scale(factor=2.0)
try:
assert scaled(1.5) == 3.0
assert scaled(2.0) == 4.0
finally:
scaled.close()
Blob v2 input is exposed as pyarrow.LargeBinaryArray in vectorized
mode (and as bytes in row mode). Return a LargeBinaryArray with the
declared nullability. Use :func:lancedb.blob in the explicit schemas;
plain pyarrow.large_binary() does not carry Blob v2 semantics.
A Blob v2 field may sit at the top level or inside a struct in inputs and
outputs. Only outputs may nest it in a list or large list (for example
pa.list_(lancedb.blob("item")), returned as a list of bytes);
fixed-size lists and maps are not supported. List-nested Blob v2 values
are queryable as Blob descriptors, but the row-id blob fetch APIs and
blob_mode="bytes" cannot fetch payloads inside lists.
lancedb.functions.FunctionDefinition
¶
A Python callable prepared for remote Function registration.
Instances are created with :func:function. Calling an instance is ordinary
local Python: a decorated function is called directly, while a decorated
class is constructed using its initialization fields and the returned
instance is called with each input. Reuse the instance and call its
close method when it defines one. This public call does not create a
remote application or build the declared remote runtime, and it does not
need the private implementation attribute.
Remote execution adapts the callable to the internal Arrow batch ABI
described by the registration artifact. By default it invokes the callable
once per row. A callable whose parameters are all annotated as
pyarrow.Array receives whole Arrow arrays and returns a
pyarrow.Array instead. This authoring choice is independent of the
internal artifact adapter in the registration request.
registration_request
¶
The immutable request sent by create_function_async.
Carries no secret bindings. Binding is a registration-time decision,
so a Function bound to Secrets is registered through :meth:bind_secrets,
which is what create_function calls.
bind_secrets
¶
The registration request for this definition bound to secrets.
Binding does not change the Function's source: each EnvVarSecret names a Secret and the environment variable its value should arrive in, and the Function reads that variable the way it already did. Whether the named Secrets exist is the server's answer, not this one.
lancedb.secrets.EnvVarSecret
¶
A Secret bound to the environment variable a Function's library reads.
Pass these in the secrets sequence of
DBConnection.create_function. The
Function's source is unchanged by binding: it reads MODEL_API_KEY the
way it always did, and the binding is what puts a value there.
This is a local value. Constructing it contacts no server, so it always succeeds and says nothing about whether the Secret exists; that is checked at registration, where a mistyped Secret name surfaces as a clear "does not exist" naming both the Secret and the variable bound to it. A mistyped variable name cannot be caught anywhere -- nothing knows which variables a Function reads -- so it surfaces on the first rows instead.
The type exists so a credential cannot be passed by accident. A bare string in the same position is a plausible-looking mistake with the opposite meaning, and it reads identically in a diff.
Parameters:
-
secret_name() –str The Secret's database-scoped name.
-
env_variable() –str The environment variable the Function reads it from.
-
secret_namespace_path(list of str, default:None) –The namespace the Secret is addressed within.
Noneand[]both mean the root namespace. Carried beside the name rather than joined into it, so neither is ever parsed back out of the other.
Examples:
lancedb.secrets.SecretInfo
¶
What a database records about a Secret. Never its value.
Returned by DBConnection.describe_secret.
created_at_millis
¶
When the Secret was created, in milliseconds since the Unix epoch.
updated_at_millis
¶
When the Secret's value was last rotated, in epoch milliseconds.
The only observable that a rotation landed: no API returns a credential,
so a caller confirms alter_secret took effect by watching this move.
lancedb.functions.FunctionRegistrationRequest
¶
Bases:
Stable remote registration envelope produced by :func:function.
Credential values deliberately have no field here. The only secret-shaped
thing a client sends is secret_bindings: the name of a Secret the
database already holds, which the remote service resolves at execution.
lancedb.functions.FunctionRetryPolicy
¶
Bases:
Bound retries of selected errors inside one Enterprise Function call.
max_attempts includes the first call. Matchers are alternatives;
transient=True selects connection/timeouts, HTTP 429/503 and the
provider codes SlowDown and ServiceUnavailable. Exception names
may be qualified, and subclasses and wrapped causes also match.
Backoff doubles with full jitter, bounded by max_backoff_seconds.
Retry-After above that bound ends retries. A failed batch is retried as
one call: under skip, exhaustion nulls every row in that call.
Completed calls are never repeated. Only opt in when repeating partial
work is safe; this does not make external side effects exactly once.
policy = FunctionRetryPolicy(transient=True, max_attempts=3) policy.max_attempts 3 disabled = FunctionRetryPolicy(max_attempts=1)
lancedb.functions.FunctionArtifactRequest
¶
lancedb.functions.FunctionArtifactContent
¶
lancedb.functions.PythonAdapterSpec
¶
lancedb.functions.FunctionImage
¶
lancedb.functions.FunctionVersion
¶
Bases:
A pinned object revision, independent of its executable image digest.
Unknown top-level fields from newer servers are accepted for forward
decoding, but they are not exposed on this value and are omitted from
to_canonical_json(). Known fields retain their complete values, while
unknown fields inside the open image.descriptor mapping remain
available.
name and namespace_path are the Function's parts, as a
[SecretReference][lancedb.functions.SecretReference] carries a Secret's;
they are what get_function takes.
An empty namespace_path is the root namespace.
source_digest identifies the registered source artifact, independently
of the built image. Compare it with
definition.registration_request.artifact.digest to check whether a local
FunctionDefinition has the same source.
It is None when the server does not provide a source digest.
gpu_class is the operator-defined GPU pool fixed into this Function
version, or None when registration used the deployment default.
batch_size, max_concurrency, row_timeout_seconds, and
result_stability retain the execution controls reported by the
catalog. They are None for versions registered before the
corresponding declaration.
lancedb.functions.HuggingFaceModel
¶
Bases:
One immutable Hugging Face model snapshot required by a Function.
revision is a full 40-character lowercase commit SHA, not a branch or
tag. A Function may declare each repository once; separate versions of the
same repository use separate Function versions. The service prepares the
snapshot before initialization, so the model can load without a revision
argument or a Hugging Face Hub request::
@function(
pip=["sentence-transformers==3.3.1"],
models=[HuggingFaceModel(
repo="sentence-transformers/all-MiniLM-L6-v2",
revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
)],
)
class Embed:
def __init__(self):
from sentence_transformers import SentenceTransformer
self.model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2", device="cpu"
)
def __call__(self, text: str) -> list[float]:
return self.model.encode(text).tolist()
lancedb.functions.PythonRuntimeSpec
¶
Bases:
Remote runtime definition with environment values.
V1 supports kind="python". Newer runtime kinds remain readable, while
their unknown payload fields are intentionally not retained by the client.
kind="python_v2" requires gpu=True and may name an operator-defined
gpu_class.
lancedb.GpuClass
¶
Configured hardware and worker limits of a Function GPU class.
cpu_millis
¶
ephemeral_storage_bytes
¶
ephemeral_storage_bytes: Optional [int ] = <attribute 'ephemeral_storage_bytes' of 'lancedb._lancedb.GpuClass' objects>
gpu_memory_bytes
¶
gpu_memory_bytes: Optional [int ] = <attribute 'gpu_memory_bytes' of 'lancedb._lancedb.GpuClass' objects>
gpu_model
¶
max_gpu_count_per_worker
¶
max_gpu_count_per_worker: Optional [int ] = <attribute 'max_gpu_count_per_worker' of 'lancedb._lancedb.GpuClass' objects>
memory_bytes
¶
resource_name
¶
lancedb.functions.FunctionVersionRef
¶
lancedb.functions.ApplicationInput
¶
Bases:
One parameter value.
Slice 1 freezes integers, strings, booleans, nulls, arrays, and objects. Floating-point literal encoding is deferred until Python authoring is introduced with a language-neutral numeric representation.
lancedb.functions.FunctionApplication
¶
Bases:
Immutable pre-declaration application of an exact Function version.
A named-struct output remains one application through table declaration and execution. FunctionApplication.rename records the result-field to table-column mapping without splitting sibling outputs into separate UDF calls.
rename
¶
Return a copy with result-field to table-column aliases.
lancedb.functions.InputBinding
¶
lancedb.functions.OutputMapping
¶
lancedb.functions.AssignmentMapping
¶
lancedb.functions.FunctionBinding
¶
lancedb.functions.RefreshColumnResult
¶
Bases:
Terminal result of an expression-backed or Function-backed refresh Job.
LanceDB Embedded jobs produce this value in process. LanceDB Enterprise decodes the same value from the durable server-job terminal payload.
Row counters describe logical rows planned for this refresh, not user-code
calls or retry attempts. rows_assigned counts rows successfully written
with an output value, rows_failed counts rows whose computation failed,
and rows_remaining counts planned rows neither assigned nor terminally
skipped. Rows already counted in rows_failed are covered and are not
counted again in rows_remaining.
For example, a refresh over 2,001 logical rows that assigns 1,891 values and
fails 110 rows reports rows_assigned=1,891, rows_failed=110, and
rows_remaining=0. Its failure-rate denominator is those 2,001 logical
rows, not the number of execution attempts.
rows_assigned
¶
Number of logical rows successfully assigned an output value by this refresh. Each row counts once, regardless of retry attempts, and failed rows are not included.
rows_failed
¶
Number of logical rows whose computation failed. Rows deferred because an input was not ready are not failures.
rows_remaining
¶
Number of planned logical rows neither assigned nor terminally skipped.
Rows already counted in rows_failed are covered and are not counted
again.
published_version
¶
Table version made visible by the refresh, when one was published.
rows_filled
¶
Deprecated compatibility alias for :attr:rows_assigned.
The value is the same successful logical-row count; it does not count execution attempts.
lancedb.FunctionErrors
¶
lancedb.FunctionErrorRecord
¶
One row a Function refresh skipped, as the server recorded it.
attempts
¶
created_at_millis
¶
created_at_millis: int = <attribute 'created_at_millis' of 'lancedb._lancedb.FunctionErrorRecord' objects>
error_message
¶
error_type
¶
fragment_id
¶
function_version
¶
function_version: str = <attribute 'function_version' of 'lancedb._lancedb.FunctionErrorRecord' objects>
row_address
¶
row_address: Optional [int ] = <attribute 'row_address' of 'lancedb._lancedb.FunctionErrorRecord' objects>
row_id
¶
row_offset
¶
row_offset: Optional [int ] = <attribute 'row_offset' of 'lancedb._lancedb.FunctionErrorRecord' objects>
table_version
¶
traceback
¶
traceback: Optional [str ] = <attribute 'traceback' of 'lancedb._lancedb.FunctionErrorRecord' objects>
User-code traceback, possibly truncated by the server.
lancedb.FunctionErrorFragment
¶
lancedb.FunctionColumnStatus
¶
Exact four-way status of one Function output at one table version.
auto_refresh
¶
auto_refresh: FunctionColumnAutoRefreshStatus = <attribute 'auto_refresh' of 'lancedb._lancedb.FunctionColumnStatus' objects>
binding_id
¶
function
¶
function: FunctionColumnVersionStatus = <attribute 'function' of 'lancedb._lancedb.FunctionColumnStatus' objects>
last_failure_job
¶
last_failure_job: Optional [FunctionColumnLastRefreshJob ] = <attribute 'last_failure_job' of 'lancedb._lancedb.FunctionColumnStatus' objects>
last_refresh_job
¶
last_refresh_job: Optional [FunctionColumnLastRefreshJob ] = <attribute 'last_refresh_job' of 'lancedb._lancedb.FunctionColumnStatus' objects>
never_computed
¶
never_computed: int = <attribute 'never_computed' of 'lancedb._lancedb.FunctionColumnStatus' objects>
table_version
¶
lancedb.FunctionColumnVersionStatus
¶
Exact immutable Function revision attached to a table output.
lancedb.FunctionColumnLastRefreshJob
¶
Latest matching refresh Job projected for a Function output.
automatic
¶
automatic: Optional [bool ] = <attribute 'automatic' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>
completed_at_millis
¶
completed_at_millis: Optional [int ] = <attribute 'completed_at_millis' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>
failure
¶
failure: Optional [JobFailureInfo ] = <attribute 'failure' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>
job_id
¶
lancedb.FunctionColumnAutoRefreshStatus
¶
Automatic-refresh projection for a Function output.
active_job_id
¶
active_job_id: Optional [str ] = <attribute 'active_job_id' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
blocked_reason
¶
blocked_reason: Optional [str ] = <attribute 'blocked_reason' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
effective_cooldown_millis
¶
effective_cooldown_millis: Optional [int ] = <attribute 'effective_cooldown_millis' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
enabled
¶
failure_streak
¶
failure_streak: int = <attribute 'failure_streak' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
interval_seconds
¶
interval_seconds: Optional [int ] = <attribute 'interval_seconds' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
next_run_at_millis
¶
next_run_at_millis: Optional [int ] = <attribute 'next_run_at_millis' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
next_run_at_reason
¶
next_run_at_reason: Optional [str ] = <attribute 'next_run_at_reason' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>
state
¶
lancedb.job.Job
¶
Bases:
Synchronous counterpart of AsyncJob with the same result type.
id
¶
Identifies the operation on the server that is running it.
See :attr:AsyncJob.id.
status
¶
The operation's current lifecycle state: "running", "finished", "failed", or "cancelled".
See :meth:AsyncJob.status.
wait
¶
Block until the operation reaches a terminal state.
Returns the endpoint's typed result, or None for a unit-result
job.
Raises JobFailedError if the operation failed, JobCancelledError
if it was cancelled, and TimeoutError if timeout elapses first.
Transient connection and read errors are retried within that timeout;
if it expires while disconnected, raises ConnectionError with
instructions to reconnect. Timing out does not cancel the job.
The timeout error identifies the job and explains that it continues on
the server. Registry jobs include the db.open_job(id).wait()
reattach call; SQL query handles must be kept because their IDs cannot
be reopened through the registry jobs API.
See :meth:AsyncJob.wait.
progress
¶
Fetch a structured progress snapshot. See :meth:AsyncJob.progress.
refresh
¶
Ask the backend for this job's current state and record.
See :meth:AsyncJob.refresh.
export_timeline
¶
events
¶
This job's recorded lifecycle events.
See :meth:AsyncJob.events.
lancedb.job.AsyncJob
¶
Bases:
A handle to an operation that may still be running.
The operation may already be complete when the handle is created. T
is the endpoint's terminal result type; unit-result jobs resolve to
None.
id
¶
The operation's opaque ID, interpreted according to :attr:job_type.
Registry jobs expose a registry job ID; SQL-backed jobs expose a
connection-scoped query ID as a string. Local operations return None.
Query IDs cannot be used with registry job APIs or to resume after a
client restart.
state
¶
The last observed lifecycle state, without contacting the backend.
None until the handle has talked to it. See :meth:AsyncJob.refresh.
job_type
¶
The job's kind of work, without contacting the backend.
SQL-backed handles report "query" immediately. Registry handles
report the server's job type after :meth:refresh or :meth:status.
In-process handles return None.
result
¶
The job-type-specific terminal result, as reported data rather than
the typed model :meth:AsyncJob.wait returns.
None until the job succeeds, so a job that never terminates reports
its progress through :meth:AsyncJob.events instead.
failure
¶
Why the job failed, when it failed and the server reports a reason.
status
¶
The operation's current lifecycle state: "running", "finished", "failed", or "cancelled".
A point snapshot; unlike wait it does not block or raise on a
terminal failure state. States a newer server reports that this
client version does not know pass through as-is.
progress
¶
Fetch a structured progress snapshot.
Registry jobs always report lifecycle fields. Queue metadata and
counters are included when supplied by the server. Older servers may
omit counters, and in-process operations return None. A missing
counter is unknown rather than zero. This is additive to :meth:status,
whose string contract is unchanged.
wait
¶
Wait until the operation reaches a terminal state.
Returns the endpoint's typed result, or None for a unit-result
job.
progress=True renders a progress bar, progress=callable
receives each structured snapshot, and a tqdm-like object is updated in
place. Observations use the backend's existing wait poller and never
start a second job. The bar advances by remaining rows when available,
because completed computation can include repeated work after a plan
change. On each wait call, the first complete observation synchronizes
a caller-owned bar with the server's current completed rows and total;
later observations in a stable execution epoch use a high-water mark. A
new execution epoch or smaller plan resets the bar to that plan.
False preserves the original wait behavior.
Raises JobFailedError if the operation failed, JobCancelledError
if it was cancelled, and TimeoutError if timeout elapses first.
Transient connection and read errors are retried with capped exponential
backoff within the same timeout (indefinitely when no timeout is given).
If the timeout expires while disconnected, raises ConnectionError
instead, with the job ID and instructions to resume waiting. The job
may still be running; timing out does not cancel it.
The timeout error identifies the job and explains that it continues on
the server. Registry jobs include the db.open_job(id).wait()
reattach call; SQL query handles must be kept because their IDs cannot
be reopened through the registry jobs API.
refresh
¶
Ask the backend for this job's current state, and for a server-side job its full record, then cache it for the properties below.
Registry details are None until this runs, because submitting an
operation does not fetch its full registry record. Query-backed jobs
expose only lifecycle state through this method. For registry jobs,
status fetches the whole record too; wait records only the terminal
state it establishes.
export_timeline
¶
Write a fresh recorded Function job timeline as Chrome trace JSON.
Open the file in Perfetto or Chrome tracing. The recording contains retained split spans grouped by worker and timestamped job phases. Batch counts and latency statistics are included as details; individual batch timestamps are not recorded. Running jobs export a partial snapshot without waiting or cancelling. Download again to include later work.
Requires a server-side Function job and a server supporting timeline
export. Unsupported operations raise NotImplementedError or a server
error. The destination is overwritten only after the trace is fetched.
Filesystem errors propagate to the caller.
Examples:
events
¶
This job's recorded lifecycle events.
Where the properties above report a terminal result only once the job
reaches one, events are written as the job runs and outlive the workers
that produced them. A distributed job records a claim/claim_complete
pair per unit of work, each carrying rows_processed, so a job that
never finishes still accounts for what it did.
Parameters:
-
limit(, default:Optional [int ]None) –Maximum event rows to return. The server caps results at 1000 by default and 10,000 at most, and truncates without saying so, so pass this for a job that emits an event per fragment.
-
filter(, default:Optional [str ]None) –SQL-like expression over the
state,updated_by,emitted_from,emitted_by, andclaim_entitycolumns, such asstate = 'claim_complete'.
lancedb.job.JobInfo
¶
lancedb.job.JobDescription
¶
The server-side record behind a Job handle.
creation_ms
¶
failure
¶
failure: Optional [JobFailureInfo ] = <attribute 'failure' of 'lancedb._lancedb.JobDescription' objects>
progress
¶
progress: Optional [dict [str , Any ]] = <attribute 'progress' of 'lancedb._lancedb.JobDescription' objects>
Structured progress snapshot. Lifecycle and queue fields are present even when a job type or older server supplies no counters.
result
¶
The job-type-specific terminal result. None until the job succeeds.
spec
¶
The job-type-specific specification it was submitted with.
lancedb.job.JobFailureInfo
¶
lancedb.sql.Query
¶
Synchronous counterpart of :class:AsyncQuery.
job_id
¶
The durable Job Registry id once the server reports this query as a job.
A running query reports it after :meth:describe observes the latest
server status. This is separate from the connection-scoped id.
reader
¶
Wait for the initial result stream and return a blocking reader.
Results are single-consumer. Calling this method more than once on the same query raises an error. Later batches block only until they become available, without waiting for the full query to finish.
cancel
¶
Request cancellation of this query's Flight execution.
A durable server-side job may reject this operation because its Flight
waiter cannot cancel the registry job. Use connection.cancel_job()
with job_id to cancel that job.
lancedb.sql.AsyncQuery
¶
A handle to a submitted SQL query on an asynchronous connection.
job_id
¶
The durable Job Registry id once the server reports this query as a job.
A running query reports it after :meth:describe observes the latest
server status. This is separate from the connection-scoped id.
reader
¶
Wait for the initial result stream and return its Arrow reader.
Results are single-consumer. Calling this method more than once on the same query raises an error. Later batches are streamed as they become available without waiting for the full query to finish.
cancel
¶
Request cancellation of this query's Flight execution.
A durable server-side job may reject this operation because its Flight
waiter cannot cancel the registry job. Use connection.cancel_job()
with job_id to cancel that job.
lancedb.sql.QueryParameters
¶
Values for a statement's $1 / $name placeholders.
A sequence binds by position ($1, $2, ...), a mapping binds by name
($name), and a one-row pyarrow.RecordBatch or pyarrow.Table binds
both ways: column i is $<i + 1> and, when its name is not a number,
also $<name>.
Materialized Views (Synchronous)¶
In the 0.41 SDK release, LanceDB Embedded materialized-view creation and refresh require a
source build with the experimental-functions Cargo feature. Published Python
packages leave it disabled. Opening, reading, listing and dropping existing views
remain available, as do LanceDB Enterprise operations. The same feature
requirement applies to LanceDB Embedded SQL computed-column declaration and refresh; ordinary
eager add_columns transforms remain available.
In LanceDB Enterprise, creation accepts auto_refresh_enabled and
auto_refresh_interval_seconds:
view = db.create_materialized_view(
"recent_orders",
"orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
Omit auto_refresh_interval_seconds to let the server determine when to refresh.
These options also apply to asynchronous creation and creation jobs. Embedded
connections built with experimental-functions support manual refresh and reject
automatic scheduling requests.
lancedb.materialized_view.MaterializedView
¶
Synchronous variant of AsyncMaterializedView.
refresh
¶
refresh(*, full: bool = False, source_version: Optional [int ] = None, cpu_millis: Optional [int ] = None, memory_bytes: Optional [int ] = None, gpu_count: Optional [int ] = None, gpu_class: Optional [str ] = None) -> 'RefreshMaterializedViewResult'
Recompute the view from its source. See AsyncMaterializedView.refresh.
refresh_async
¶
refresh_async(*, full: bool = False, source_version: Optional [int ] = None, cpu_millis: Optional [int ] = None, memory_bytes: Optional [int ] = None, gpu_count: Optional [int ] = None, gpu_class: Optional [str ] = None) -> 'Job[RefreshMaterializedViewResult]'
Submit a refresh and return its job without waiting.
lancedb.materialized_view.MaterializedViewDefinition
¶
The query that defines a materialized view, as stored::
SELECT columns
FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias]
[WHERE predicate] [GROUP BY expr, ...] [LIMIT n]
A Function in FROM position yields one row per element it returns.
Views¶
lancedb.view.ViewDescription
¶
What a database records about one view.
default_database
¶
The database that unqualified table names in query resolve against.
namespace_path
¶
The namespace holding the view; empty is the root namespace.
default_namespace_path
¶
The namespace path those unqualified names resolve against.
Recorded with the view because it outlives the session that declared it: a reader resolving the query against its own default namespace could read a different table than the view was defined over.
Property Graphs¶
A property graph is defined over tables of its namespace: which tables hold
nodes and which hold edges, each node table's key and label, and the node keys
each edge table's source and destination columns reference -- the shape of
SQL/PGQ's CREATE PROPERTY GRAPH. It stores no rows of its own; a query reads
its tables as nodes and edges through SQL GRAPH_TABLE.
Create one with create_property_graph, and manage it with open_property_graph, list_property_graphs and drop_property_graph.
lancedb.graph.NodeTable
¶
A table whose rows are the nodes of one label.
properties
¶
The columns exposed as node properties; every column when None. A
created graph reports the list it resolved to, the key among them.
lancedb.graph.EdgeTable
¶
A table whose rows are the edges of one label.
properties
¶
The columns exposed as edge properties; every column when None. A
created graph reports the list it resolved to, both endpoint columns among
them.
lancedb.graph.Endpoint
¶
lancedb.graph.PropertyGraph
¶
Synchronous variant of AsyncPropertyGraph.
lancedb.graph.PropertyGraphDescription
¶
What a database records about one property graph.
namespace_path
¶
The namespace holding the graph; empty is the root namespace.
Expressions¶
Type-safe expression builder for filters and projections. Use these instead of raw SQL strings with where and select.
lancedb.expr.Expr
¶
A type-safe expression node.
Construct instances with :func:col and :func:lit, then combine them
using Python operators or the named methods below.
Examples:
>>> from lancedb.expr import col, lit
>>> filt = (col("age") > lit(18)) & (col("name").lower() == lit("alice"))
>>> proj = {"double": col("x") * lit(2)}
Notes
An Expr has no Python truth value: bool(expr) raises TypeError.
Combine expressions with &, | and ~ (wrapping each comparison in
parentheses) instead of and, or and not, and write a range as
(col("x") > 1) & (col("x") < 5) rather than 1 < col("x") < 5.
isin
¶
Return True where the value is one of values (SQL IN).
cast
¶
Cast values to data_type.
Parameters:
-
data_type() –Union [str , 'pa.DataType']A PyArrow
DataType(e.g.pa.int32()) or one of the type name strings:"bool","int8","int16","int32","int64","uint8"–"uint64","float32","float64","string","date32","date64".
lancedb.expr.col
¶
lancedb.expr.lit
¶
lancedb.expr.func
¶
Call an arbitrary SQL function by name.
Parameters:
-
name() –str The SQL function name (e.g.
"lower","upper"). -
*args(, default:ExprLike ()) –The function arguments as :class:
Expror plain Python literals.
Examples:
Querying (Synchronous)¶
lancedb.query.Query
¶
Bases:
A LanceDB Query
Queries are constructed by the Table.search method. This class is a
python representation of the query. Normally you will not need to interact
with this class directly. You can build up a query and execute it using
collection methods such as to_batches(), to_arrow(), to_pandas(),
etc.
However, you can use the to_query() method to get the underlying query object.
This can be useful for serializing a query or using it in a different context.
vector
¶
vector: Annotated [Optional [Union [List [float ], List [List [float ]], Array , List [Array ]]], ensure_vector_query ] = None
lancedb.query.LanceQueryBuilder
¶
Bases:
An abstract query builder. Subclasses are defined for vector search, full text search, hybrid, and plain SQL filtering.
create
¶
create(table: 'Table', query: Optional [Union [ndarray , str , 'PIL.Image.Image', Tuple ]], query_type: str , vector_column_name: str , ordering_field_name: Optional [str ] = None, fts_columns: Optional [Union [str , List [str ]]] = None, fast_search: bool = None) -> Self
Create a query builder based on the given query and query type.
Parameters:
-
table('Table') –The table to query.
-
query() –Optional [Union [ndarray ,str , 'PIL.Image.Image',Tuple ]]The query to use. If None, an empty query builder is returned which performs simple SQL filtering.
-
query_type() –str The type of query to perform. One of "vector", "fts", "hybrid", or "auto". If "auto", the query type is inferred based on the query.
-
vector_column_name() –str The name of the vector column to use for vector search.
-
ordering_field_name(, default:Optional [str ]None) –.. deprecated:: 0.27.0 Use
order_by()method instead. -
fts_columns(, default:Optional [Union [str ,List [str ]]]None) –The columns to search in for full text search.
-
fast_search(, default:bool None) –Skip flat search of unindexed data.
to_df
¶
Deprecated alias for to_pandas(). Please use to_pandas() instead.
Execute the query and return the results as a pandas DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, *, blob_mode: BlobMode = 'lazy', timeout: Optional [timedelta ] = None, **kwargs) -> 'pd.DataFrame'
Execute the query and return the results as a pandas DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_arrow
¶
Execute the query and return the results as an Apache Arrow Table.
In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vectors.
This collects all results in memory. To process a large result a batch at a time, use to_batches.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.
to_batches
¶
to_batches(batch_size: Optional [int ] = None, *, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> RecordBatchReader
Execute the query and return the results as a pyarrow RecordBatchReader
For blob v2 projections, the default blob_mode="lazy" keeps the
auto _rowid column visible so batch consumers can call
fetch_blobs. The "bytes" and "descriptions" modes hide
automatic row ids.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
batch_size(, default:Optional [int ]None) –The maximum number of selected records in a RecordBatch object.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Stream blob payloads with query.to_batches(blob_mode="bytes") and
convert each batch separately with batch.to_pandas().
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pydantic
¶
Return the table as a list of pydantic models.
Parameters:
-
model() –type [T ]The pydantic model to use.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–List [LanceModel ]
to_polars
¶
Execute the query and return the results as a Polars DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
limit
¶
Set the maximum number of results to return.
Parameters:
-
limit() –Union [int , None]The maximum number of results to return. The default query limit is 10 results. For ANN/KNN queries, you must specify a limit. For plain searches, all records are returned if limit not set. WARNING if you have a large dataset, setting the limit to a large number, e.g. the table size, can potentially result in reading a large amount of data into memory and cause out of memory issues.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
offset
¶
Set the offset for the results.
Parameters:
-
offset() –int The offset to start fetching results from.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
select
¶
Set the columns to return.
Parameters:
-
columns() –Union [list [str ],dict [str ,Union [str ,Expr ]]]List of column names to be fetched. Or a dictionary of column names to SQL expressions or :class:
~lancedb.expr.Exprobjects. All columns are fetched if None or unspecified.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
where
¶
Set the where clause.
Parameters:
-
where() –Union [str ,Expr ]The filter condition. Can be a SQL string or a type-safe :class:
~lancedb.expr.Exprbuilt with :func:~lancedb.expr.coland :func:~lancedb.expr.lit. -
prefilter(, default:bool True) –If True, apply the filter before vector search, otherwise the filter is applied on the result of vector search. This feature is EXPERIMENTAL and may be removed and modified without warning in the future.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
Notes
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
with_row_id
¶
Set whether to return row ids.
Parameters:
-
with_row_id() –bool If True, return _rowid column in the results.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
with_row_address
¶
Set whether to return row addresses.
Parameters:
-
with_row_address(, default:bool True) –If True, return the _rowaddr column in the results.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
with_fragments
¶
Set the Lance fragments to scan for plain scanner-backed queries.
fragment_ids
¶
Set the Lance fragment ids to scan for plain scanner-backed queries.
use_rte
¶
Control RTE LSM read routing for this query.
By default (unset), a query against a table with an RTE write spec is
routed through the LSM scanner so it also returns data written via the
merge_insert LSM path that has not yet been compacted into the base
table (active/frozen memtables + SSTables); a table without a
spec reads the base table.
Parameters:
-
enable() –bool Trueforces the LSM scanner and errors if the table has no RTE write spec.Falsebypasses the RTE and reads the base table only, even when a spec is present.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).explain_plan(True)
>>> print(plan)
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
order_by
¶
Set the ordering for the results.
Parameters:
-
ordering() –Optional [List [ColumnOrdering ]]The ordering to use for the results. If None, then the default ordering will be used.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
analyze_plan
¶
Run the query and return its execution plan with runtime metrics.
This returns detailed metrics for each step, such as elapsed time, rows processed, bytes read, and I/O stats. It is useful for debugging and performance tuning.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).analyze_plan()
>>> print(plan)
AnalyzeExec verbose=true, elapsed=..., metrics=...
TracedExec, elapsed=..., metrics=...
ProjectionExec: elapsed=..., expr=[...],
metrics=[output_rows=..., elapsed_compute=..., output_bytes=...]
GlobalLimitExec: elapsed=..., skip=0, fetch=10,
metrics=[output_rows=..., elapsed_compute=..., output_bytes=...]
FilterExec: elapsed=..., _distance@2 IS NOT NULL, metrics=[...]
SortExec: elapsed=..., TopK(fetch=1), expr=[...],
preserve_partitioning=[...], filter=[...],
metrics=[output_rows=..., elapsed_compute=...,
output_bytes=..., row_replacements=...]
KNNVectorDistance: elapsed=..., metric=l2,
metrics=[output_rows=..., elapsed_compute=...,
output_bytes=..., output_batches=...]
LanceRead: elapsed=..., uri=..., projection=[vector],
num_fragments=..., range_before=None, range_after=None,
row_id=true, row_addr=false,
full_filter=--, refine_filter=--,
metrics=[output_rows=..., elapsed_compute=..., output_bytes=...,
fragments_scanned=..., ranges_scanned=1, rows_scanned=1,
bytes_read=..., iops=..., requests=..., task_wait_time=...]
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans. "aggregate" preserves the legacy summary, "per_worker" shows each worker separately, and "full" includes both.
Returns:
-
plan() –str The physical query execution plan with runtime metrics.
vector
¶
Set the vector to search for.
Parameters:
-
vector() –Union [VEC ,Tuple ]The vector to search for.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
text
¶
Set the text to search for.
Parameters:
-
text() –str |FullTextQuery If a string, it is treated as a MatchQuery. If a FullTextQuery object, it is used directly.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
rerank
¶
Rerank the results using the specified reranker.
Parameters:
-
reranker() –Reranker The reranker to use.
Returns:
-
The LanceQueryBuilder object.–
to_query_object
¶
Return a serializable representation of the query
Returns:
-
–Query The serializable representation of the query
lancedb.query.LanceVectorQueryBuilder
¶
Bases:
Examples:
>>> import lancedb
>>> data = [{"vector": [1.1, 1.2], "b": 2},
... {"vector": [0.5, 1.3], "b": 4},
... {"vector": [0.4, 0.4], "b": 6},
... {"vector": [0.4, 0.4], "b": 10}]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data=data)
>>> (table.search([0.4, 0.4])
... .distance_type("cosine")
... .where("b < 10")
... .select(["b", "vector"])
... .limit(2)
... .to_pandas())
b vector _distance
0 6 [0.4, 0.4] 0.000000
1 2 [1.1, 1.2] 0.000944
metric
¶
Set the distance metric to use.
This is an alias for distance_type() and may be deprecated in the future. Please use distance_type() instead.
Parameters:
-
metric() –Literal ['l2', 'cosine', 'dot']The distance metric to use. By default "l2" is used.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
distance_type
¶
Set the distance metric to use.
When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use.
Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.
Parameters:
-
distance_type() –Literal ['l2', 'cosine', 'dot']The distance metric to use. By default "l2" is used.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
nprobes
¶
Set the number of probes to use.
Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.
See discussion in Querying an ANN Index for tuning advice.
The value is retained as nprobes through client and server request
construction. Lance sets both probe bounds to this value. Explicit minimum
or maximum settings can override their respective bounds.
Parameters:
-
nprobes() –int The number of probes to use. Must be greater than 0.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
search_effort
¶
Set the initial IVF search effort to a finite value in [0, 1].
Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.
None (the default) uses Auto, internally 0.5. Zero starts at the caller minimum; one searches all partitions within the caller maximum. Later probing may continue when too few candidates are found. Any explicit effort, including 0.5, overrides nprobes in either call order. Explicit minimum and maximum bounds still apply. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.
Examples:
minimum_nprobes
¶
Set the minimum number of probes to use.
See nprobes for more details.
These partitions will be searched on every vector query and will increase recall at the expense of latency.
maximum_nprobes
¶
Set the maximum number of probes to use.
See nprobes for more details.
If this value is greater than minimum_nprobes then the excess partitions
will be searched only if we have not found enough results.
This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.
If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.
distance_range
¶
distance_range(lower_bound: Optional [float ] = None, upper_bound: Optional [float ] = None) -> LanceVectorQueryBuilder
Set the distance range to use.
Only rows with distances within range [lower_bound, upper_bound) will be returned.
Parameters:
-
lower_bound(, default:Optional [float ]None) –The lower bound of the distance range.
-
upper_bound(, default:Optional [float ]None) –The upper bound of the distance range.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
ef
¶
Set the number of candidates to consider during search.
Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.
This only applies to the HNSW-related index. The default value is 1.5 * limit.
Parameters:
-
ef() –int The number of candidates to consider during search.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
approx_mode
¶
Set the speed / accuracy tradeoff to use for approximate search.
This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.
Examples:
>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table("approx_vectors", [{"vector": [0.1, 0.2]}])
>>> query = table.search([0.1, 0.2]).approx_mode("fast")
Parameters:
-
approx_mode() –Literal ['fast', 'normal', 'accurate']The tradeoff to use. By default "normal" is used.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
refine_factor
¶
Set the refine factor to use, increasing the number of vectors sampled.
As an example, a refine factor of 2 will sample 2x as many vectors as requested, re-ranks them, and returns the top half most relevant results.
See discussion in Querying an ANN Index for tuning advice.
Parameters:
-
refine_factor() –int The refine factor to use.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and return the results as an Apache Arrow Table.
In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vectors.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.
to_query_object
¶
Build a Query object
This can be used to serialize a query
to_batches
¶
to_batches(batch_size: Optional [int ] = None, *, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> RecordBatchReader
Execute the query and return the result as a RecordBatchReader object.
Parameters:
-
batch_size(, default:Optional [int ]None) –The maximum number of selected records in a RecordBatch object.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If None, wait indefinitely. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –Controls blob conversion as described in :meth:
LanceQueryBuilder.to_batches.
Returns:
-
–RecordBatchReader
where
¶
Set the where clause.
Parameters:
-
where() –Union [str ,Expr ]The filter condition. Can be a SQL string or a type-safe :class:
~lancedb.expr.Exprbuilt with :func:~lancedb.expr.coland :func:~lancedb.expr.lit. -
prefilter(, default:bool None) –If True, apply the filter before vector search, otherwise the filter is applied on the result of vector search.
Returns:
-
–LanceQueryBuilder The LanceQueryBuilder object.
Notes
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
rerank
¶
Rerank the results using the specified reranker.
Parameters:
-
reranker() –Reranker The reranker to use.
-
query_string(, default:Optional [str ]None) –The query to use for reranking. This needs to be specified explicitly here as the query used for vector search may already be vectorized and the reranker requires a string query. This is only required if the query used for vector search is not a string. Note: This doesn't yet support the case where the query is multimodal or a list of vectors.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
bypass_vector_index
¶
If this is called then any vector index is skipped
An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.
Returns:
-
–LanceVectorQueryBuilder The LanceVectorQueryBuilder object.
fast_search
¶
Skip a flat search of unindexed data. This will improve search performance but search results will not include unindexed data.
Returns:
-
–LanceVectorQueryBuilder The LanceVectorQueryBuilder object.
lancedb.query.LanceFtsQueryBuilder
¶
Bases:
A builder for full text search for LanceDB.
phrase_query
¶
Set whether to use phrase query.
Parameters:
-
phrase_query(, default:bool True) –If True, then an unquoted string query will be wrapped in quotes.
Returns:
-
–LanceFtsQueryBuilder The LanceFtsQueryBuilder object.
fast_search
¶
Skip a flat search of unindexed data. This will improve search performance but search results will not include unindexed data.
Returns:
-
–LanceFtsQueryBuilder The LanceFtsQueryBuilder object.
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_batches
¶
to_batches(batch_size: Optional [int ] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy')
rerank
¶
Rerank the results using the specified reranker.
Parameters:
-
reranker() –Reranker The reranker to use.
Returns:
-
–LanceFtsQueryBuilder The LanceQueryBuilder object.
lancedb.query.LanceHybridQueryBuilder
¶
Bases:
A query builder that performs hybrid vector and full text search. Results are combined and reranked based on the specified reranker. By default, the results are reranked using the RRFReranker, which uses reciprocal rank fusion score for reranking.
To make the vector and fts results comparable, the scores are normalized.
Instead of normalizing scores, the normalize parameter can be set to "rank"
in the rerank method to convert the scores to ranks and then normalize them.
with_row_id
¶
Set whether to return row ids.
Recorded separately from _with_row_id, which rerank(return_score=
"all") also sets for its own use — on an RTE table only a caller who
asked is refused, and the reranker still falls back to the primary key.
phrase_query
¶
Set whether to use phrase query.
Parameters:
-
phrase_query(, default:bool True) –If True, then an unquoted string query will be wrapped in quotes.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
to_batches
¶
to_batches(batch_size: Optional [int ] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy')
rerank
¶
Rerank the hybrid search results using the specified reranker. The reranker must be an instance of Reranker class.
Parameters:
-
reranker(, default:Reranker ) –RRFReranker ()The reranker to use. Must be an instance of Reranker class.
-
normalize(, default:str 'score') –The method to normalize the scores. Can be "rank" or "score". If "rank", the scores are converted to ranks and then normalized. If "score", the scores are normalized directly.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
nprobes
¶
Set the number of probes to use for vector search.
Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.
The value is retained as nprobes until Lance sets both probe bounds to it.
Explicit minimum or maximum settings can override their respective bounds.
Parameters:
-
nprobes() –int The number of probes to use. Must be greater than 0.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
search_effort
¶
Set the initial IVF search effort to a finite value in [0, 1].
Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.
None (the default) uses Auto, internally 0.5. Zero starts at the caller minimum; one searches all partitions within the caller maximum. Later probing may continue when too few candidates are found. Any explicit effort, including 0.5, overrides nprobes in either call order. Explicit minimum and maximum bounds still apply. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.
Examples:
minimum_nprobes
¶
Set the minimum number of probes to use.
See nprobes for more details.
maximum_nprobes
¶
Set the maximum number of probes to use.
See nprobes for more details.
distance_range
¶
distance_range(lower_bound: Optional [float ] = None, upper_bound: Optional [float ] = None) -> LanceHybridQueryBuilder
Set the distance range to use.
Only rows with distances within range [lower_bound, upper_bound) will be returned.
Parameters:
-
lower_bound(, default:Optional [float ]None) –The lower bound of the distance range.
-
upper_bound(, default:Optional [float ]None) –The upper bound of the distance range.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
ef
¶
Set the number of candidates to consider during search.
Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.
This only applies to the HNSW-related index. The default value is 1.5 * limit.
Parameters:
-
ef() –int The number of candidates to consider during search.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
approx_mode
¶
Set the speed / accuracy tradeoff to use for approximate search.
This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.
Examples:
>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table(
... "approx_hybrid", [{"vector": [0.1, 0.2], "text": "dog"}]
... )
>>> query = (
... table.search(query_type="hybrid")
... .vector([0.1, 0.2])
... .text("dog")
... .approx_mode("accurate")
... )
Parameters:
-
approx_mode() –Literal ['fast', 'normal', 'accurate']The tradeoff to use. By default "normal" is used.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
metric
¶
Set the distance metric to use.
This is an alias for distance_type() and may be deprecated in the future. Please use distance_type() instead.
Parameters:
-
metric() –Literal ['l2', 'cosine', 'dot']The distance metric to use. By default "l2" is used.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
distance_type
¶
Set the distance metric to use.
When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use.
Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.
Parameters:
-
distance_type() –Literal ['l2', 'cosine', 'dot']The distance metric to use. By default "l2" is used.
Returns:
-
–LanceVectorQueryBuilder The LanceQueryBuilder object.
refine_factor
¶
Refine the vector search results by reading extra elements and re-ranking them in memory.
Parameters:
-
refine_factor() –int The refine factor to use.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
vector
¶
Set the vector to search for.
Parameters:
-
vector() –Union [VEC ,Tuple ]The vector to search for.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
bypass_vector_index
¶
If this is called then any vector index is skipped
An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.
Returns:
-
–LanceHybridQueryBuilder The LanceHybridQueryBuilder object.
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).explain_plan(True)
>>> print(plan)
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and display with runtime metrics.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
lancedb.query.LanceEmptyQueryBuilder
¶
Bases:
to_batches
¶
to_batches(batch_size: Optional [int ] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy') -> RecordBatchReader
rerank
¶
Rerank the results using the specified reranker.
Parameters:
-
reranker() –Reranker The reranker to use.
Returns:
-
–LanceEmptyQueryBuilder The LanceQueryBuilder object.
lancedb.query.LanceTakeQueryBuilder
¶
Bases:
Builder for parameterizing and executing take queries.
Full text queries¶
Structured full text queries can be passed to Table.search or AsyncTable.search in place of a query string, and combined with BooleanQuery.
lancedb.query.FullTextQuery
¶
lancedb.query.MatchQuery
¶
Bases:
Match query for full-text search.
Parameters:
-
query() –str The query string to match against.
-
column() –str The column to match against. None leaves column selection to query planning.
-
boost(, default:float 1.0) –The boost factor for the query. The score of each matching document is multiplied by this value.
-
fuzziness(, default:int or None0) –The maximum edit distance for each term in the match query. Defaults to 0 (exact match). If None, fuzziness is applied automatically by the rules: - 0 for terms with length <= 2 - 1 for terms with length <= 5 - 2 for terms with length > 5
-
max_expansions() –int The maximum number of terms to consider for fuzzy matching. Defaults to 50.
-
operator(, default:FullTextOperator OR) –The operator to use for combining the query results. Can be either
ANDorOR. IfAND, all terms in the query must match. IfOR, at least one term in the query must match. -
prefix_length() –int The number of beginning characters being unchanged for fuzzy matching. This is useful to achieve prefix matching.
-
document_granularity() –DocumentGranularity Explicitly select row or deepest-list-element documents. If omitted, the indexed granularity is inferred. When both granularities are indexed for the field, this must be specified. With no index, row granularity is used.
document_granularity
¶
lancedb.query.PhraseQuery
¶
Bases:
Phrase query for full-text search.
Parameters:
-
query() –str The query string to match against.
-
column() –str The column to match against. None leaves column selection to query planning.
-
slop(, default:int 0) –The maximum number of intervening positions permitted in the phrase.
-
document_granularity() –DocumentGranularity Explicitly select row or deepest-list-element documents. If omitted, the indexed granularity is inferred. When both granularities are indexed for the field, this must be specified. With no index, row granularity is used.
lancedb.query.BoostQuery
¶
Bases:
Boost query for full-text search.
Parameters:
-
positive() –dict The positive query object.
-
negative() –dict The negative query object.
-
negative_boost(, default:float 0.5) –The boost factor for the negative query.
lancedb.query.MultiMatchQuery
¶
Bases:
Multi-match query for full-text search.
Parameters:
-
query() –str |list [Query ]If a string, the query string to match against.
-
columns() –list [str ]The list of columns to match against.
-
boosts() –list [float ]The list of boost factors for each column. If not provided, all columns will have the same boost factor.
-
operator(, default:FullTextOperator OR) –The operator to use for combining the query results. Can be either
ANDorOR. It would be applied to all columns individually. For example, if the operator isAND, then the query "hello world" is equal tomatch("hello AND world", column1) OR match("hello AND world", column2).
lancedb.query.CombinedFieldsQuery
¶
Bases:
Combined-fields (BM25F) query for full-text search.
Unlike MultiMatchQuery, which scores each column on its own and keeps
the best one, this query treats the columns as a single virtual field, so
term statistics are blended across them. A query term can then match in
any of the columns, e.g. a first name in one column and a last name in
another.
Parameters:
-
query() –str The query string to match against.
-
columns() –list [str ]The columns combined into the virtual field. Every column must have the same tokenizer configuration, and at least one must have an FTS index.
-
boosts() –list [float ]The weight of each column, aligned with
columns. Each weight must be at least 1. If not provided, every column has a weight of 1. -
operator(, default:FullTextOperator OR) –The operator to use for combining the query terms. If
AND, every term must appear in at least one of the columns. IfOR, at least one term must match.
lancedb.query.BooleanQuery
¶
Bases:
Boolean query for full-text search.
Parameters:
-
queries() –list [tuple (Occur ,FullTextQuery )]The list of queries with their occurrence requirements. Each tuple contains an Occur value (MUST, SHOULD, or MUST_NOT) and a FullTextQuery to apply.
lancedb.query.DocumentGranularity
¶
lancedb.query.Occur
¶
Embeddings¶
EmbeddingFunction
¶
Bases: ,
An ABC for embedding functions.
All concrete embedding functions must implement the following methods: 1. compute_query_embeddings() which takes a query and returns a list of embeddings 2. compute_source_embeddings() which returns a list of embeddings for the source column For text data, the two will be the same. For multi-modal data, the source column might be images and the vector column might be text. 3. ndims() which returns the number of dimensions of the vector column
sensitive_keys
¶
Return a list of keys that are sensitive and should not be allowed to be set to hardcoded values in the config. For example, API keys.
remote_code_model
¶
Return the model whose repository code this function runs when its
trust_remote_code field is enabled, as a (name, revision) pair.
Return None if the function never runs code from a model repository.
When the function is restored from table metadata, trust_remote_code
is only enabled for models allowed with
allow_remote_code.
endpoints
¶
Return the non-default URLs this function sends data or credentials to.
When the function is restored from table metadata, each of these must be allowed with allow_endpoint.
compute_query_embeddings
¶
Compute the embeddings for a given user query
Returns:
-
A list of embeddings for each input. The embedding of each input can be None– -
when the embedding is not valid.–
compute_source_embeddings
¶
Compute the embeddings for the source column in the database
Returns:
-
A list of embeddings for each input. The embedding of each input can be None– -
when the embedding is not valid.–
compute_query_embeddings_with_retry
¶
Compute the embeddings for a given user query with retries
Returns:
-
A list of embeddings for each input. The embedding of each input can be None– -
when the embedding is not valid.–
compute_source_embeddings_with_retry
¶
Compute the embeddings for the source column in the database with retries.
Returns:
-
A list of embeddings for each input. The embedding of each input can be None– -
when the embedding is not valid.–
sanitize_input
¶
Sanitize the input to the embedding function.
SourceField
¶
Creates a pydantic Field that can automatically annotate the source column for this embedding function
VectorField
¶
Creates a pydantic Field that can automatically annotate the target vector column for this embedding function.
The field can be annotated as list[float] for compatibility with
static type checkers. LanceDB will infer the fixed vector dimension from
this embedding function.
EmbeddingFunctionConfig
¶
TextEmbeddingFunction
¶
UntrustedEmbeddingConfigError
¶
Bases:
Raised when using an embedding function restored from table metadata whose settings this process has not approved.
See allow_remote_code and allow_endpoint.
BedRockText
¶
Bases:
Parameters:
-
name(, default:str "amazon.titan-embed-text-v1") –The model ID of the bedrock model to use. Supported models for are: - amazon.titan-embed-text-v1 - cohere.embed-english-v3 - cohere.embed-multilingual-v3
-
region(, default:str "us-east-1") –Optional name of the AWS Region in which the service should be called.
-
profile_name(, default:str None) –Optional name of the AWS profile to use for calling the Bedrock service. If not specified, the default profile will be used.
-
assumed_role(, default:str None) –Optional ARN of an AWS IAM role to assume for calling the Bedrock service. If not specified, the current active credentials will be used.
-
role_session_name(, default:str "lancedb-embeddings") –Optional name of the AWS IAM role session to use for calling the Bedrock service. If not specified, "lancedb-embeddings" name will be used.
Examples:
import lancedb import pandas as pd from lancedb.pydantic import LanceModel, Vector
model = get_registry().get("bedrock-text").create()
class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()
df = pd.DataFrame({"text": ["hello world", "goodbye world"]}) db = lancedb.connect("tmp_path") tbl = db.create_table("test", schema=TextModel, mode="overwrite")
tbl.add(df)
rs = tbl.search("hello").limit(1).to_pandas()
client
¶
Create a boto3 client for Amazon Bedrock service
Returns:
-
–client The boto3 client for Amazon Bedrock service
compute_query_embeddings
¶
compute_source_embeddings
¶
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
Returns:
-
–list [list [float ]]The embeddings for the given texts
CohereEmbeddingFunction
¶
Bases:
An embedding function that uses the Cohere API
https://docs.cohere.com/docs/multilingual-language-models
Parameters:
-
name(, default:str "embed-multilingual-v3.0") –The name of the model to use. List of acceptable models:
* embed-english-v3.0 * embed-multilingual-v3.0 * embed-english-light-v3.0 * embed-multilingual-light-v3.0The v2.0 models (embed-english-v2.0, embed-english-light-v2.0 and embed-multilingual-v2.0) were retired by Cohere on April 4, 2026. The previous default produced 768-dimensional vectors, while the new default produces 1024-dimensional vectors. Existing tables containing vectors from a v2.0 model must be re-embedded with an active model into a table whose vector column has the matching dimension.
-
source_input_type(, default:str "search_document") –The input type for the source column in the database
-
query_input_type(, default:str "search_query") –The input type for the query column in the database
Notes
Cohere supports following input types:
| Input Type | Description |
|---|---|
"search_document" |
Used for embeddings stored in a vector |
| database for search use-cases. | |
"search_query" |
Used for embeddings of search queries |
| run against a vector DB | |
"semantic_similarity" |
Specifies the given text will be used |
| for Semantic Textual Similarity (STS) | |
"classification" |
Used for embeddings passed through a |
| text classifier. | |
"clustering" |
Used for the embeddings run through a |
| clustering algorithm |
Examples:
import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import EmbeddingFunctionRegistry
cohere = EmbeddingFunctionRegistry .get_instance() .get("cohere") .create(name="embed-multilingual-v3.0")
class TextModel(LanceModel): text: str = cohere.SourceField() vector: Vector(cohere.ndims()) = cohere.VectorField()
data = [ { "text": "hello world" }, { "text": "goodbye world" }]
db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")
tbl.add(data)
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
GeminiText
¶
Bases:
An embedding function that uses Google's Gemini API. Requires GOOGLE_API_KEY to be set.
https://ai.google.dev/gemini-api/docs/embeddings
Supports various tasks types:
| Task Type | Description |
|-------------------------|--------------------------------------------------------|
| "retrieval_query" | Specifies the given text is a query in a |
| | search/retrieval setting. |
| "retrieval_document" | Specifies the given text is a document in a |
| | search/retrieval setting. Using this task type |
| | requires a title but is automatically provided by |
| | Embeddings API |
| "semantic_similarity" | Specifies the given text will be used for Semantic |
| | Textual Similarity (STS). |
| "classification" | Specifies that the embeddings will be used for |
| | classification. |
| "clustering" | Specifies that the embeddings will be used for |
| | clustering. |
Note: The supported task types might change in the Gemini API, but as long as a supported task type and its argument set is provided, those will be delegated to the API calls.
Parameters:
-
name(, default:str "gemini-embedding-001") –The name of the model to use. Supported models include: - "gemini-embedding-001" (768 dimensions)
Note: The legacy "models/embedding-001" format is also supported but "gemini-embedding-001" is recommended.
-
query_task_type(, default:str "retrieval_query") –Sets the task type for the queries.
-
source_task_type(, default:str "retrieval_document") –Sets the task type for ingestion.
Examples:
import lancedb import pandas as pd from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import get_registry
model = get_registry().get("gemini-text").create()
class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()
df = pd.DataFrame({"text": ["hello world", "goodbye world"]}) db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")
tbl.add(df) rs = tbl.search("hello").limit(1).to_pandas()
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
InstructorEmbeddingFunction
¶
Bases:
An embedding function that uses the InstructorEmbedding library. Instructor models
support multi-task learning, and can be used for a variety of tasks, including
text classification, sentence similarity, and document retrieval. If you want to
calculate customized embeddings for specific sentences, you may follow the unified
template to write instructions:
"Represent the domain text_type for task_objective":
* domain is optional, and it specifies the domain of the text, e.g., science,
finance, medicine, etc.
* text_type is required, and it specifies the encoding unit, e.g., sentence,
document, paragraph, etc.
* task_objective is optional, and it specifies the objective of embedding,
e.g., retrieve a document, classify the sentence, etc.
For example, if you want to calculate embeddings for a document, you may write the instruction as follows: "Represent the document for retrieval"
Parameters:
-
name() –str The name of the model to use. Available models are listed at https://github.com/xlang-ai/instructor-embedding#model-list; The default model is hkunlp/instructor-base
-
batch_size(, default:int 32) –The batch size to use when generating embeddings
-
device(, default:str "cpu") –The device to use when generating embeddings
-
show_progress_bar(, default:bool True) –Whether to show a progress bar when generating embeddings
-
normalize_embeddings(, default:bool True) –Whether to normalize the embeddings
-
quantize(, default:bool False) –Whether to quantize the model
-
source_instruction(, default:str "represent the document for retrieval") –The instruction for the source column
-
query_instruction(, default:str "represent the document for retrieving the most) –similar documents" The instruction for the query
Examples:
import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import get_registry, InstructorEmbeddingFunction
instructor = get_registry().get("instructor").create( source_instruction="represent the document for retrieval", query_instruction="represent the document for retrieving the most " "similar documents" )
class Schema(LanceModel): vector: Vector(instructor.ndims()) = instructor.VectorField() text: str = instructor.SourceField()
db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=Schema, mode="overwrite")
texts = [{"text": "Capitalism has been dominant in the Western world since the " "end of feudalism, but most feel[who?] that..."}, {"text": "The disparate impact theory is especially controversial under " "the Fair Housing Act because the Act..."}, {"text": "Disparate impact in United States labor law refers to practices " "in employment, housing, and other areas that.."}]
tbl.add(texts)
query_instruction
¶
OCIGenAIEmbeddings
¶
Bases:
An embedding function that uses the Oracle Cloud Infrastructure (OCI) Generative AI service.
https://docs.oracle.com/en-us/iaas/Content/generative-ai/embed-models.htm
Requires the OCI Python SDK (pip install oci). No credentials are stored
on this object: requests are signed by the SDK using an API-key or
session-token profile from the OCI config file, or the instance / resource
principal of the host, so there are no sensitive keys to protect with
$var:.
Cohere embedding models are asymmetric: the source column is embedded with
input_type="SEARCH_DOCUMENT" and queries with "SEARCH_QUERY".
Inputs are sent in batches of at most 96 texts, the service limit.
Parameters:
-
name(, default:str "cohere.embed-v4.0") –Model ID of an on-demand embedding model:
cohere.embed-v4.0(1536 dims),cohere.embed-english-v3.0/cohere.embed-multilingual-v3.0(1024 dims) or their-light-v3.0variants (384 dims). A dedicated AI cluster endpoint OCID (ocid1.generativeaiendpoint...) is also accepted. -
compartment_id() –str OCID of the compartment the requests are authorized in and billed to. The service requires it; when not passed, the
OCI_COMPARTMENT_IDenvironment variable is used. -
region(, default:str "us-chicago-1") –OCI region used to derive the inference endpoint when
service_endpointis not set. -
service_endpoint() –str Full inference endpoint, e.g.
https://inference.generativeai.us-chicago-1.oci.oraclecloud.com. Takes precedence overregion; needed for realms with a different endpoint template. -
auth_type(, default:str "API_KEY") –One of
"API_KEY","SECURITY_TOKEN","INSTANCE_PRINCIPAL"or"RESOURCE_PRINCIPAL". The first two read profileauth_profilefrom the OCI config fileauth_file_location;SECURITY_TOKENadditionally uses the profile'ssecurity_token_filewritten byoci session authenticate. The principal types use the identity of the host (Compute instance, OKE workload, Functions, ...) and ignore the config file. -
auth_profile(, default:str "DEFAULT") –Profile name in the OCI config file.
-
auth_file_location(, default:str "~/.oci/config") –Path of the OCI config file.
-
truncate(, default:str "END") –How inputs longer than the model context are handled:
"NONE"(fail),"START"or"END"(drop tokens from that side). -
source_input_type(, default:str "SEARCH_DOCUMENT") –input_typesent when embedding the source column. -
query_input_type(, default:str "SEARCH_QUERY") –input_typesent when embedding queries. -
output_dimensions() –int Embedding size for models that support it (
cohere.embed-v4.0: 256, 512, 1024 or 1536). When set,ndims()returns it directly; otherwise the size is discovered with one probe request and cached.
Examples:
import lancedb from lancedb.embeddings import get_registry from lancedb.pydantic import LanceModel, Vector
model = get_registry().get("oci-genai").create(
name="cohere.embed-v4.0",
compartment_id="ocid1.compartment.oc1..
class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()
db = lancedb.connect("~/.lancedb") tbl = db.create_table("docs", schema=TextModel, mode="overwrite") tbl.add([{"text": "hello world"}, {"text": "goodbye world"}]) rs = tbl.search("hello").limit(1).to_pandas()
When restored from table metadata in a new process, an explicit
service_endpoint or non-default region must be approved with
allow_endpoint.
The default region does not need approval.
auth_type
¶
auth_type: Literal ['API_KEY', 'SECURITY_TOKEN', 'INSTANCE_PRINCIPAL', 'RESOURCE_PRINCIPAL'] = 'API_KEY'
compute_query_embeddings
¶
compute_source_embeddings
¶
generate_embeddings
¶
generate_embeddings(texts: Union [List [str ], ndarray ], *args, **kwargs) -> List [Union [List [float ], None]]
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
Returns:
-
–list [list [float ] | None]One embedding per input, in order. Empty inputs are not sent to the service (it rejects them) and come back as
None.
OllamaEmbeddings
¶
Bases:
An embedding function that uses Ollama
https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings https://ollama.com/blog/embedding-models
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
OpenClipEmbeddings
¶
Bases:
An embedding function that uses the OpenClip API For multi-modal text-to-image search
https://github.com/mlfoundations/open_clip
compute_query_embeddings
¶
Compute the embeddings for a given user query
Parameters:
-
query() –Union [str ,Image ]The query to embed. A query can be either text or an image.
sanitize_input
¶
Sanitize the input to the embedding function.
compute_source_embeddings
¶
Get the embeddings for the given images
generate_image_embedding
¶
Generate the embedding for a single image
Parameters:
-
image() –Union [str ,bytes ,Image ]The image to embed. If the image is a str, it is treated as a uri. If the image is bytes, it is treated as the raw image bytes.
OpenAIEmbeddings
¶
Bases:
An embedding function that uses the OpenAI API
https://platform.openai.com/docs/guides/embeddings
This can also be used for open source models that are compatible with the OpenAI API.
Notes
If you're running an Ollama server locally,
you can just override the base_url parameter
and provide the Ollama embedding model you want
to use (https://ollama.com/library):
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
NvidiaEmbeddings
¶
Bases:
Text embeddings using NVIDIA Nemotron 3.5 Embed 1B or 8B.
Uses the optional openai package for NVIDIA-hosted and self-hosted NIM
/v1/embeddings endpoints. Documents use input_type="passage" and
queries use input_type="query". Null and empty strings get a null
embedding. Image inputs are not supported.
Parameters:
-
name() –str Hosted model ID, or the served model ID from a NIM
/v1/modelsresponse. -
dim() –int Requested output dimensions. Defaults to 2048 for the hosted 1B model and 4096 for the hosted 8B model. Required for other served model IDs; use a dimension supported by your deployment.
-
base_url() –str API base URL including
/v1. Defaults to NVIDIA's hosted API. -
api_key() –str Registry variable reference, such as
$var:nvidia-key. Otherwise the hosted API usesNVIDIA_API_KEY. Custom endpoints only receive an explicitapi_keyand can run without authentication. -
batch_size() –int Maximum inputs per request (default 32). Reduce for deployment limits.
-
truncate() –str END(default) truncates overlong text;NONErejects it. -
max_retries() –int Inherited LanceDB retry limit, applied to each request. The OpenAI client's own retries are disabled.
Examples:
Set NVIDIA_API_KEY before using the hosted API:
>>> from lancedb.embeddings import get_registry
>>> from lancedb.pydantic import LanceModel, Vector
>>> func = get_registry().get("nvidia").create()
>>> class Document(LanceModel):
... text: str = func.SourceField()
... vector: Vector(func.ndims()) = func.VectorField()
Embedding runs client-side for both local and remote tables. For a custom endpoint restored from table metadata in a new process, first call allow_endpoint.
EmbeddingFunctionRegistry
¶
This is a singleton class used to register embedding functions and fetch them by name. It also handles serializing and deserializing. You can implement your own embedding function by subclassing EmbeddingFunction or TextEmbeddingFunction and registering it with the registry.
NOTE: Here TEXT is a type alias for Union[str, List[str], pa.Array, pa.ChunkedArray, np.ndarray]
Examples:
>>> registry = EmbeddingFunctionRegistry.get_instance()
>>> @registry.register("my-embedding-function")
... class MyEmbeddingFunction(EmbeddingFunction):
... def ndims(self) -> int:
... return 128
...
... def compute_query_embeddings(self, query: str, *args, **kwargs):
... return self.compute_source_embeddings(query, *args, **kwargs)
...
... def compute_source_embeddings(self, texts, *args, **kwargs):
... return [np.random.rand(self.ndims()) for _ in range(len(texts))]
...
>>> registry.get("my-embedding-function")
<class 'lancedb.embeddings.registry.MyEmbeddingFunction'>
register
¶
This creates a decorator that can be used to register an EmbeddingFunction.
Parameters:
-
alias(, default:Optional [str ]None) –a human friendly name for the embedding function. If not provided, the class name will be used.
get
¶
Fetch an embedding function class by name
Parameters:
-
name() –str The name of the embedding function to fetch Either the alias or the class name if no alias was provided during registration
list_functions
¶
List the names of registered embedding functions.
Returns:
-
–list [str ]Registered aliases, or class names for functions registered without an alias, in registration order. Each name can be passed to :meth:
get. The returned list is a copy; changing it does not change the registry.
Examples:
parse_functions
¶
Parse the metadata from an arrow table and return a mapping of the vector column to the embedding function and source column
Parameters:
-
metadata() –Optional [Dict [bytes ,bytes ]]The metadata from an arrow table. Note that the keys and values are bytes (pyarrow api)
Returns:
-
functions() –dict A mapping of vector column name to embedding function. An empty dict is returned if input is None or does not contain b"embedding_functions".
allow_remote_code
¶
Trust the code in a model repository when an embedding function that loads it is restored from table metadata.
Table metadata cannot enable trust_remote_code by itself, since anyone
who can write to the table could then run code in this process. Embedding
functions created in this process with trust_remote_code=True are
allowed automatically.
Parameters:
-
model() –str The model name, such as
"nomic-ai/nomic-embed-text-v1.5". -
revision(, default:Optional [str ]None) –The revision the table's embedding function must pin. Pinning a commit hash is recommended, since the code at a branch or tag can change after you review it. If None, any revision is trusted, including pull request refs that anyone can push to the repository.
allow_endpoint
¶
Trust a custom endpoint, such as an OpenAI base_url or Ollama host,
when an embedding function that uses it is restored from table metadata.
Table metadata cannot pick an endpoint by itself, since it could send your data and API keys to a server you don't control. Embedding functions created in this process are allowed to use their endpoints automatically.
Parameters:
-
url() –str The endpoint URL. Trailing slashes are ignored.
function_to_metadata
¶
Convert the given embedding function and source / vector column configs into a config dictionary that can be serialized into arrow metadata
get_table_metadata
¶
Convert a list of embedding functions and source / vector configs into a config dictionary that can be serialized into arrow metadata
set_var
¶
Set a variable. These can be accessed in embedding configuration using
the syntax $var:variable_name. If they are not set, an error will be
thrown letting you know which variable is missing. If you want to supply
a default value, you can add an additional part in the configuration
like so: $var:variable_name:default_value. Default values can be
used for runtime configurations that are not sensitive, such as
whether to use a GPU for inference.
The name must not contain a colon. Default values can contain colons.
SentenceTransformerEmbeddings
¶
Bases:
An embedding function that uses the sentence-transformers library
https://huggingface.co/sentence-transformers
Parameters:
-
name(, default:str "all-MiniLM-L6-v2") –The name of the model to use.
-
device(, default:str "cpu") –The device to use for the model
-
normalize(, default:bool True) –Whether to normalize the embeddings
-
trust_remote_code(, default:bool False) –Whether to run code from the model's repository. Only enable this for repositories you trust. When restored from table metadata, this is only enabled for models allowed with allow_remote_code.
-
revision(, default:Optional [str ]None) –The model revision to load, such as a commit hash.
embedding_model
¶
Get the sentence-transformers embedding model specified by the name, device, and trust_remote_code. This is cached so that the model is only loaded once per process.
generate_embeddings
¶
Get the embeddings for the given texts
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
get_embedding_model
¶
Get the sentence-transformers embedding model specified by the name, device, and trust_remote_code. This is cached so that the model is only loaded once per process.
TODO: use lru_cache instead with a reasonable/configurable maxsize
GteEmbeddings
¶
Bases:
Deprecated: GTE embeddings should be used through sentence-transformers.
An embedding function that uses GTE-LARGE MLX format(for Apple silicon devices only) as well as the standard cpu/gpu version from: https://huggingface.co/thenlper/gte-large.
For Apple users, you will need the mlx package installed, which can be done with: pip install mlx
Parameters:
-
name(, default:str "thenlper/gte-large") –The name of the model to use.
-
device(, default:str "cpu") –Sets the device type for the model.
-
normalize(, default:str "True") –Controls normalize param in encode function for the transformer.
-
mlx(, default:bool False) –Controls which model to use. False for gte-large,True for the mlx version.
Examples:
import lancedb import lancedb.embeddings.gte from lancedb.embeddings import get_registry from lancedb.pydantic import LanceModel, Vector import pandas as pd
model = get_registry().get("gte-text").create() # mlx=True for Apple silicon class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()
df = pd.DataFrame({"text": ["hi hello sayonara", "goodbye world"]}) db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")
tbl.add(df) rs = tbl.search("hello").limit(1).to_pandas()
embedding_model
¶
Get the embedding model specified by the flag, name and device. This is cached so that the model is only loaded once per process.
generate_embeddings
¶
Get the embeddings for the given texts.
Parameters:
-
texts() –Union [List [str ],ndarray ]The texts to embed
get_embedding_model
¶
Get the embedding model specified by the flag, name and device. This is cached so that the model is only loaded once per process.
TransformersEmbeddingFunction
¶
Bases:
An embedding function that can use any model from the transformers library.
Parameters:
name : str The name of the model to use. This should be a model name that can be loaded by transformers.AutoModel.from_pretrained. For example, "bert-base-uncased". default: "colbert-ir/colbertv2.0"" device : str The device to use for the model. Default is "cpu". show_progress_bar : bool Whether to show a progress bar when loading the model. Default is True. trust_remote_code : bool Whether or not to allow for custom models defined on the HuggingFace Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. When restored from table metadata, this is only enabled for models allowed with allow_remote_code. revision : Optional[str] The model revision to load, such as a commit hash. Default is None.
to download package, run :
pip install transformers
you may need to install pytorch as well - https://pytorch.org/get-started/locally/
ColbertEmbeddings
¶
Bases:
An embedding function that uses the colbert model from the huggingface library.
Parameters:
name : str The name of the model to use. This should be a model name that can be loaded by transformers.AutoModel.from_pretrained. For example, "bert-base-uncased". default: "colbert-ir/colbertv2.0""
to download package, run :
pip install transformers
you may need to install pytorch as well - https://pytorch.org/get-started/locally/
ImageBindEmbeddings
¶
Bases:
An embedding function that uses the ImageBind API For generating multi-modal embeddings across six different modalities: images, text, audio, depth, thermal, and IMU data
to download package, run :
pip install imagebind-packaged==0.1.2
embedding_model
¶
Get the embedding model. This is cached so that the model is only loaded once per process.
compute_query_embeddings
¶
Compute the embeddings for a given user query
Parameters:
-
query() –Union [str ]The query to embed. A query can be either text, image paths or audio paths.
compute_source_embeddings
¶
Get the embeddings for the given sourcefield column in the pydantic model.
sanitize_input
¶
Sanitize the input to the embedding function.
JinaEmbeddings
¶
Bases:
An embedding function that uses the Jina API
https://jina.ai/embeddings/
Parameters:
-
name(, default:str "jina-clip-v1". Note that some models support both image) –and text embeddings and some just text embedding
-
api_key(, default:str None) –The api key to access Jina API. If you pass None, you can set JINA_API_KEY environment variable
sanitize_input
¶
Sanitize the input to the embedding function.
compute_query_embeddings
¶
Compute the embeddings for a given user query
Parameters:
-
query() –Union [str ,Image ]The query to embed. A query can be either text or an image.
compute_source_embeddings
¶
generate_image_embedding
¶
Generate the embedding for a single image
Parameters:
-
image() –Union [str ,bytes ,Image ]The image to embed. If the image is a str, it is treated as a uri. If the image is bytes, it is treated as the raw image bytes.
generate_text_embeddings
¶
WatsonxEmbeddings
¶
Bases:
An embedding function that uses the IBM watsonx.ai Embeddings API.
API Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-embeddings
Supported embedding models: https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx
Parameters:
-
name(, default:str "ibm/slate-125m-english-rtrvr") –The ID of the embedding model to use. For new tables,
"ibm/granite-embedding-278m-multilingual"is recommended. -
api_key() –str IBM Cloud API key. Falls back to the
WATSONX_API_KEYenvironment variable when not provided. -
project_id() –str watsonx.ai project ID. Explicit value takes precedence over the
WATSONX_PROJECT_IDenvironment variable. Mutually exclusive withspace_id— exactly one must be supplied. -
space_id() –str watsonx.ai deployment space ID. Explicit value takes precedence over the
WATSONX_SPACE_IDenvironment variable. Mutually exclusive withproject_id— exactly one must be supplied. -
url() –str watsonx.ai service URL. Defaults to
"https://us-south.ml.cloud.ibm.com". -
params() –dict Extra parameters forwarded verbatim to
Embeddings(e.g.{"truncate_input_tokens": 512}).
VoyageAIEmbeddingFunction
¶
Bases:
An embedding function that uses the VoyageAI API
https://docs.voyageai.com/docs/embeddings
Parameters:
-
name() –str The name of the model to use. List of acceptable models:
* voyage-4 (1024 dims, general-purpose and multilingual retrieval) * voyage-4-lite (1024 dims, optimized for latency and cost) * voyage-4-large (1024 dims, best retrieval quality) * voyage-context-3 * voyage-3.5 * voyage-3.5-lite * voyage-3 * voyage-3-lite * voyage-multimodal-3 * voyage-multimodal-3.5 * voyage-finance-2 * voyage-multilingual-2 * voyage-law-2 * voyage-code-2 -
output_dimension() –int The output dimension for models that support flexible dimensions. Currently only voyage-multimodal-3.5 supports this feature. Valid options: 256, 512, 1024 (default), 2048.
Examples:
import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import EmbeddingFunctionRegistry
voyageai = EmbeddingFunctionRegistry .get_instance() .get("voyageai") .create(name="voyage-3")
class TextModel(LanceModel): text: str = voyageai.SourceField() vector: Vector(voyageai.ndims()) = voyageai.VectorField()
data = [ { "text": "hello world" }, { "text": "goodbye world" }]
db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")
tbl.add(data)
text_embedding_models
¶
text_embedding_models: list = ['voyage-4', 'voyage-4-lite', 'voyage-4-large', 'voyage-3.5', 'voyage-3.5-lite', 'voyage-3', 'voyage-3-lite', 'voyage-finance-2', 'voyage-multilingual-2', 'voyage-law-2', 'voyage-code-2']
multimodal_embedding_models
¶
compute_query_embeddings
¶
Compute the embeddings for a given user query
Parameters:
-
query() –Union [str ,Image ]The query to embed. A query can be either text or an image.
Returns:
-
List[np.array]: the list of embeddings–
compute_source_embeddings
¶
Compute the embeddings for the inputs
Parameters:
-
inputs() –Union [TEXT ,IMAGES ]The inputs to embed. The input can be either str, bytes, Path (to an image), PIL.Image or list of these.
Returns:
-
List[np.array]: the list of embeddings–
ColPaliEmbeddings
¶
Bases:
An embedding function that uses the ColPali engine for multimodal multi-vector embeddings.
This embedding function supports ColPali models, producing multivector outputs for both text and image inputs.
Parameters:
-
model_name() –str The name of the model to use (e.g., "Metric-AI/ColQwen2.5-3b-multilingual-v1.0") Supports models based on these engines: - ColPali: "vidore/colpali-v1.3" and others - ColQwen2.5: "Metric-AI/ColQwen2.5-3b-multilingual-v1.0" and others - ColQwen2: "vidore/colqwen2-v1.0" and others - ColSmol: "vidore/colSmol-256M" and others
-
device() –str The device for inference (default "auto").
-
dtype() –str Data type for model weights (default "bfloat16").
-
use_token_pooling() –bool DEPRECATED. Whether to use token pooling. Use
pooling_strategyinstead. -
pooling_strategy() –str The token pooling strategy to use, by default "hierarchical". - "hierarchical": Progressively pools tokens to reduce sequence length. - "lambda": A simpler pooling that uses a custom
pooling_func. -
pooling_func() –Callable A function to use for pooling when
pooling_strategyis "lambda". -
pool_factor() –int Factor to reduce sequence length if token pooling is enabled (default 2).
-
quantization_config() –Optional [BitsAndBytesConfig ]Quantization configuration for the model. (default None, bitsandbytes needed)
-
batch_size() –int Batch size for processing inputs (default 2).
-
offload_folder() –str Folder to offload model weights if using CPU offloading (default None). This is useful for large models that do not fit in memory.
generate_text_embeddings
¶
Generate embeddings for text input.
generate_image_embeddings
¶
Generate embeddings for a batch of images.
compute_query_embeddings
¶
Compute embeddings for a single user query (text only).
compute_source_embeddings
¶
Compute embeddings for a batch of source images.
Parameters:
-
images() –Union [str ,bytes ,List ,Array ,ChunkedArray ,ndarray ]Batch of images (paths, URLs, bytes, or PIL Images).
get_registry
¶
Utility function to get the global instance of the registry
Returns:
-
–EmbeddingFunctionRegistry The global registry instance
Examples:
from lancedb.embeddings import get_registry
registry = get_registry() openai = registry.get("openai").create()
Remote configuration¶
HeaderProvider
¶
Bases:
Abstract base class for providing custom headers for each request.
Users can implement this interface to provide dynamic headers for various purposes such as authentication (OAuth tokens, API keys), request tracking (correlation IDs), custom metadata, or any other header-based requirements. The provider is called before each request to ensure fresh header values are always used.
Error Handling
If get_headers() raises an exception, the request will fail. Implementations should handle recoverable errors internally (e.g., retry token refresh) and only raise exceptions for unrecoverable errors.
get_headers
¶
Get the latest headers to be added to requests.
This method is called before each request to the remote LanceDB server. Implementations should return headers that will be merged with existing headers.
Returns:
-
–Dict [str ,str ]Dictionary of header names to values to add to the request.
Raises:
-
–Exception If unable to fetch headers, the exception will be propagated and the request will fail.
ClientAuthMethod
¶
Bases: ,
How the client authenticates to the OAuth token endpoint.
The method applies to every OAuth request that carries client authentication: client-credentials, authorization-code exchange, refresh-token, and device-authorization requests. The Azure managed identity flow ignores this option.
NONE
¶
No client authentication, for public clients using PKCE or the device
flow. Cannot be combined with client_secret.
CLIENT_SECRET_BASIC
¶
HTTP Basic authentication. This is the RFC 6749 recommended method and
the normal default for confidential clients, including default Okta
applications. Requires client_secret.
CLIENT_SECRET_POST
¶
Credentials in the request body, for providers configured to require
it. Requires client_secret.
OAuthConfig
¶
OAuth configuration for LanceDB authentication.
All token acquisition and refresh is handled in the Rust layer. This config is passed through to Rust via PyO3.
Parameters:
-
issuer_url() –str OIDC issuer URL or OAuth authority URL. For Azure:
https://login.microsoftonline.com/{tenant_id}/v2.0 -
client_id() –str Application / Client ID.
-
scopes() –List [str ]OAuth scopes to request. For Azure managed identity, exactly one scope or resource is required. For example:
["api://{app_id}/.default"] -
flow(, default:OAuthFlowType ) –CLIENT_CREDENTIALS Authentication flow to use. Default: CLIENT_CREDENTIALS.
-
client_secret(, default:Optional [str ]None) –Client secret (required for CLIENT_CREDENTIALS).
-
client_auth_method(, default:Optional [ClientAuthMethod ]None) –How the client authenticates to the token endpoint (default: auto). With a
client_secretthe default isClientAuthMethod.CLIENT_SECRET_BASIC, which matches the RFC 6749 recommendation and the default configuration of Okta confidential applications; without a secret the client is public and no client authentication is sent. -
redirect_uri(, default:Optional [str ]None) –Loopback redirect URI for AUTHORIZATION_CODE. The default is
http://127.0.0.1:{callback_port}/callback. -
callback_port(, default:Optional [int ]None) –Port for the AUTHORIZATION_CODE loopback callback server (default: 8400).
-
use_pkce(, default:bool True) –Protect AUTHORIZATION_CODE with S256 PKCE (default: True).
-
managed_identity_client_id(, default:Optional [str ]None) –Client ID for user-assigned managed identity (AZURE_MANAGED_IDENTITY).
-
resource(, default:Optional [str ]None) –Resource indicator (RFC 8707), forwarded verbatim to authorization and token endpoints, including refresh requests. Must be an absolute URI without a fragment. Not supported for Azure managed identity.
-
audience(, default:Optional [str ]None) –Provider-specific audience, forwarded to authorization and token endpoints, including refresh requests. Not supported for Azure managed identity.
-
refresh_buffer_secs(, default:Optional [int ]None) –Seconds before expiry to trigger proactive refresh (default: 300). Keep this well below the token TTL; if it is greater than or equal to the TTL, each request refreshes the token.
-
token_cache(, default:Optional [TokenCacheOptions ]None) –Opt in to the persistent token cache so short-lived processes reuse one session. Only supported by AUTHORIZATION_CODE and DEVICE_CODE; azure managed identity is rejected. Default: None (memory only).
Examples:
Client Credentials (service-to-service):
>>> config = OAuthConfig(
... issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
... client_id="app-id",
... client_secret="secret",
... scopes=["api://lancedb-api/.default"],
... )
Providers that require an explicit target can use resource and/or
audience (these are forwarded unchanged):
Azure Managed Identity:
>>> config = OAuthConfig(
... issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
... client_id="app-id",
... scopes=["api://lancedb-api/.default"],
... flow=OAuthFlowType.AZURE_MANAGED_IDENTITY,
... )
Authorization Code with PKCE:
The authorization URL is written to standard error before LanceDB tries to open a browser, so it can be copied in headless environments.
>>> config = OAuthConfig(
... issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
... client_id="app-id",
... scopes=["openid", "api://lancedb-api/access"],
... flow=OAuthFlowType.AUTHORIZATION_CODE,
... )
Device Authorization, with a persistent cache so later processes reuse the session without a new device prompt. The verification URL and user code are written to standard error before polling begins:
>>> config = OAuthConfig(
... issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
... client_id="app-id",
... scopes=["openid", "offline_access", "api://lancedb-api/access"],
... flow=OAuthFlowType.DEVICE_CODE,
... token_cache=TokenCacheOptions(),
... )
OAuthFlowType
¶
Bases: ,
OAuth authentication flow types.
OAuthSession
¶
Explicit OAuth session lifecycle for the persistent token cache.
Built from the same :class:OAuthConfig used for
:func:lancedb.connect_async (including its token_cache options).
A connection created with the same configuration shares the cache, so
logging in here prepares tokens for later processes without any database
request.
login always runs the configured interactive flow and replaces the
cached session (the most recent login wins). logout removes only the
local credential; it does not revoke anything with the provider and does
not sign out of a browser SSO session.
Examples:
>>> config = OAuthConfig(
... issuer_url="https://issuer.example.com",
... client_id="my-app",
... scopes=["openid", "offline_access"],
... flow=OAuthFlowType.DEVICE_CODE,
... token_cache=TokenCacheOptions(),
... )
>>> session = OAuthSession(config)
>>> status = await session.login()
>>> status.refreshable
True
login
¶
Eagerly run the configured flow and store the session.
Returns a :class:lancedb._lancedb.SessionStatus describing the
cached session. A successful login always replaces any prior cached
session for this identity; if the provider does not issue a refresh
token (for example without offline_access), the previous record
is removed and refreshable is False.
status
¶
Report whether a cached session exists, with safe metadata.
Never contacts the identity provider and never exposes token values.
logout
¶
Remove the matching local cached credential.
Returns a :class:lancedb._lancedb.SessionLogout whose removed
flag reports whether a credential existed. Logout is idempotent.
TokenCacheOptions
¶
Options for the persistent OAuth token cache.
The cache is opt-in: it is only used when set as token_cache on
:class:OAuthConfig. Only refresh tokens are persisted, in a private
directory with owner-only permissions, so short-lived processes can reuse
an authenticated session instead of re-prompting on every start.
Parameters:
-
cache_dir(, default:Optional [str ]None) –Directory that holds cached credentials. Defaults to
$XDG_CACHE_HOME/lancedb/oauth,$HOME/.cache/lancedb/oauthon Unix, or%LOCALAPPDATA%\lancedb\oauthon Windows. The directory is created with owner-only permissions (0700) when missing. -
lock_timeout_secs(, default:Optional [int ]None) –How long to wait for the cross-process refresh lock before failing (default: 30 seconds).
Examples:
Multiple identities (issuer, client, scopes, flow, client authentication) get separate cache entries. Within one identity the most recent login wins.
TimeoutConfig
¶
RetryConfig
¶
Retry configuration for the remote HTTP client.
TlsConfig
¶
ClientConfig
¶
Configuration for the HTTP client used to connect to LanceDB Enterprise.
Context¶
lancedb.context.contextualize
¶
Create a Contextualizer object for the given DataFrame.
Used to create context windows. Context windows are rolling subsets of text data.
The input text column should already be separated into rows that will be the unit of the window. So to create a context window over tokens, start with a DataFrame with one token per row. To create a context window over sentences, start with a DataFrame with one sentence per row.
Examples:
>>> from lancedb.context import contextualize
>>> import pandas as pd
>>> data = pd.DataFrame({
... 'token': ['The', 'quick', 'brown', 'fox', 'jumped', 'over',
... 'the', 'lazy', 'dog', 'I', 'love', 'sandwiches'],
... 'document_id': [1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2]
... })
window determines how many rows to include in each window. In our case
this how many tokens, but depending on the input data, it could be sentences,
paragraphs, messages, etc.
>>> contextualize(data).window(3).stride(1).text_col('token').to_pandas()
token document_id
0 The quick brown 1
1 quick brown fox 1
2 brown fox jumped 1
3 fox jumped over 1
4 jumped over the 1
5 over the lazy 1
6 the lazy dog 1
7 lazy dog I 1
8 dog I love 1
9 I love sandwiches 2
10 love sandwiches 2
>>> (contextualize(data).window(7).stride(1).min_window_size(7)
... .text_col('token').to_pandas())
token document_id
0 The quick brown fox jumped over the 1
1 quick brown fox jumped over the lazy 1
2 brown fox jumped over the lazy dog 1
3 fox jumped over the lazy dog I 1
4 jumped over the lazy dog I love 1
5 over the lazy dog I love sandwiches 1
stride determines how many rows to skip between each window start. This can
be used to reduce the total number of windows generated.
>>> contextualize(data).window(4).stride(2).text_col('token').to_pandas()
token document_id
0 The quick brown fox 1
2 brown fox jumped over 1
4 jumped over the lazy 1
6 the lazy dog I 1
8 dog I love sandwiches 1
10 love sandwiches 2
groupby determines how to group the rows. For example, we would like to have
context windows that don't cross document boundaries. In this case, we can
pass document_id as the group by.
>>> (contextualize(data)
... .window(4).stride(2).text_col('token').groupby('document_id')
... .to_pandas())
token document_id
0 The quick brown fox 1
2 brown fox jumped over 1
4 jumped over the lazy 1
6 the lazy dog 1
9 I love sandwiches 2
min_window_size determines the minimum size of the context windows
that are generated.This can be used to trim the last few context windows
which have size less than min_window_size.
By default context windows of size 1 are skipped.
lancedb.context.Contextualizer
¶
Create context windows from a DataFrame. See lancedb.context.contextualize.
window
¶
Set the window size. i.e., how many rows to include in each window.
Parameters:
-
window() –int The window size.
stride
¶
Set the stride. i.e., how many rows to skip between each window.
Parameters:
-
stride() –int The stride.
groupby
¶
Set the groupby column. i.e., how to group the rows. Windows don't cross groups
Parameters:
-
groupby() –str The groupby column.
text_col
¶
Set the text column used to make the context window.
Parameters:
-
text_col() –str The text column.
min_window_size
¶
Set the (optional) min_window_size size for the context window.
Parameters:
-
min_window_size() –int The min_window_size.
Full text search¶
Pass custom_stop_words to lancedb.index.FTS:
from lancedb.index import FTS
table.create_index(
"text",
config=FTS(remove_stop_words=True, custom_stop_words=["acme", "internal"]),
)
The list replaces the built-in stop words and is used only when
remove_stop_words=True:
custom_stop_words=Noneuses the built-in list forlanguage.custom_stop_words=[]removes no words.- Values are passed through without trimming, lowercasing, or other rewriting.
The same option is available on lancedb.tokenize(...) and the deprecated
lancedb.table.Table.create_fts_index compatibility helper:
import lancedb
tokens = list(
lancedb.tokenize("acme makes searchable data", custom_stop_words=["acme"])
)
lancedb.tokenize
¶
tokenize(query: str , *, base_tokenizer: BaseTokenizerType = 'simple', language: str = 'English', max_token_length: Optional [int ] = 40, lower_case: bool = True, stem: Optional [bool ] = None, remove_stop_words: Optional [bool ] = None, custom_stop_words: Optional [List [str ]] = None, ascii_folding: bool = True, ngram_min_length: int = 3, ngram_max_length: int = 3, prefix_only: bool = False, analyzer: Optional [Literal ['text', 'code']] = None, split_identifiers: Optional [bool ] = None, split_on_numerics: Optional [bool ] = None, preserve_original: Optional [bool ] = None, index_operators: Optional [bool ] = None) -> Iterable [FtsToken ]
Tokenize a full-text search query using an explicit tokenizer.
This does not require an FTS index. The tokenizer options match
:class:lancedb.index.FTS. custom_stop_words accepts a list of strings.
When stem or remove_stop_words is None, the base tokenizer's default
applies: False for code and ngram, True otherwise. analyzer and the
code flags (split_identifiers, split_on_numerics, preserve_original
and index_operators) behave as they do on :class:lancedb.index.FTS.
Examples:
Inspect the identifier and subwords produced by the code analyzer:
lancedb.FtsToken
¶
Blobs¶
Blob columns store large binary values out of line so they can be read lazily instead of being materialized with the rest of the row.
lancedb.BlobType is lance.blob.BlobType when pylance is installed. Without
pylance, LanceDB uses a matching lance.blob.v2 extension type so blob columns
still work. Native table queries return descriptors. Call
fetch_blob_files for lazy reads or
fetch_blobs for eager bytes.
LanceDB Embedded supports these fetch APIs for both legacy blob v1 columns
(lance-encoding:blob) and blob v2 columns. For legacy columns, obtain row IDs
with search().with_row_id(True).to_arrow() and pass the result to the fetch API.
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
not return partial results.
To construct inline Blob V2 arrays for ingestion in record batches, use
blob_array. It accepts bytes and None, preserving
the distinction between an empty blob (b"") and a null blob:
import pyarrow as pa
from lancedb.schema import blob, blob_array
schema = pa.schema([pa.field("video_id", pa.string()), blob("video")])
batch = pa.RecordBatch.from_arrays(
[pa.array(["a", "b", "c"]), blob_array([b"abc", None, b""])],
schema=schema,
)
table = db.create_table("videos", data=iter([batch]), schema=schema)
table.add(iter([batch]))
create_table(..., schema=...) also casts plain binary values into blob columns
declared in the schema. table.add(...) casts them using the table's schema.
These conversions apply to Python rows and plain binary Arrow columns.
PyArrow's RecordBatch.from_pylist(..., schema=...) cannot convert raw bytes
into the blob extension type; use blob_array and RecordBatch.from_arrays
when constructing batches explicitly.
For external blob references, construct the descriptor struct described by
blob, then wrap it with pa.ExtensionArray.from_storage.
Inline data and external uri representations are mutually exclusive.
LanceDB Enterprise SQL returns blob bytes by default. Select blob_descriptor(video) to
return an openable reference instead, then use
open_blob_file or the lazy iterable
open_blob_files:
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
This example uses a LanceDB Enterprise SQL connection that produces source-bearing
descriptors. The descriptor contains kind, position, size, blob_id,
blob_uri, and a source struct with uri, version, committed_at_micros,
branch, column, row_reference, and row_is_address. The opener verifies
the source table and commit and reads the captured version without changing
the table handle's checkout. Raw native storage descriptors omit source;
use fetch_blob_files(column, row_ids) for those query results.
The singular API accepts a dictionary or Arrow struct scalar. The iterable
accepts Arrow arrays/chunked arrays or another iterable, opens handles as
iteration advances, and preserves order, duplicates, and nulls as None.
Close each non-null handle after use. Async tables provide
await table.open_blob_file(...) and
async for file in table.open_blob_files(...).
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
For an async table, the iterable is consumed with async for:
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
Payload bytes load on reads. Opening an external blob with unspecified size can issue a metadata-only HEAD request. LanceDB Enterprise reads use authenticated range requests that validate the captured URI and commit timestamp.
Blobs inside lists can be written and queried as descriptors. They are omitted
from blob_columns() because row ids do not identify individual list elements,
so the fetch APIs cannot read their bytes yet.
lancedb.blob
¶
blob(name: str , nullable: bool = True, *, inline_size_threshold: Optional [int ] = None, dedicated_size_threshold: Optional [int ] = None, pack_file_size_threshold: Optional [int ] = None) -> Field
Create a Lance blob v2 column field.
When pylance is installed this is lance.blob.BlobType.
Parameters:
-
name() –str Field name.
-
nullable(, default:bool True) –Whether the column accepts null values.
-
inline_size_threshold(, default:Optional [int ]None) –Maximum payload size in bytes to keep inline in the data file before using packed blob storage. Zero is allowed.
-
dedicated_size_threshold(, default:Optional [int ]None) –Maximum payload size in bytes to store in packed blob storage before using dedicated blob storage. Must be positive.
-
pack_file_size_threshold(, default:Optional [int ]None) –Maximum size in bytes of a single packed blob sidecar file before a new one is started. Must be positive.
Notes
Thresholds are validated and stored as Arrow field metadata, with the
same format as lance.blob_field. They do not require pylance.
The extension's storage is a struct with nullable fields data
(large_binary), uri (string), position (uint64), and size
(uint64). Each non-null blob must set exactly one of data and uri:
- Inline bytes use
data, leaving all other fields null.b""is a valid empty blob; use a null struct for a null blob. - External blobs use
uri, leavingdatanull. Set bothpositionandsizeto select a byte range, or leave both null for the whole object. A range'ssizemust be greater than zero.
Use blob_array to build inline blob arrays
for Arrow record batches. For external descriptors, build the struct
array with field.type.storage_type and wrap it with
pa.ExtensionArray.from_storage(field.type, storage).
create_table(..., schema=...) casts plain binary values into blob
columns declared by this field, and table.add(...) does the same
using the table's schema. PyArrow's RecordBatch.from_pylist does not
perform this conversion; construct blob arrays explicitly instead.
lancedb.schema.blob_array
¶
Build a Lance blob v2 extension array from inline bytes.
Uses the same extension type as blob, with or without pylance installed.
Parameters:
-
payloads() –Iterable [Optional [bytes ]]An iterable of
bytesorNone.Nonecreates a null blob;b""creates a valid empty blob.
Returns:
-
–ExtensionArray Inline blob values ready to use in Arrow tables or record batches.
Raises:
-
–TypeError If a payload is neither
bytesnorNone.
Examples:
BlobFile
¶
Bases:
Seekable lazy handle from :meth:~lancedb.table.Table.fetch_blob_files.
Bytes load on read, read1, read_range, or read_ranges, not
when the handle is opened. Range reads leave the cursor unchanged. Use
:meth:aread from async code.
read_ranges
¶
Read (offset, length) pairs without moving the cursor.
Utilities¶
lancedb.schema.vector
¶
A help function to create a vector type.
Parameters:
-
dimension() –int -
value_type(, default:DataType ) –float32 ()The type of the value in the vector.
Returns:
-
A PyArrow DataType for vectors.–
Examples:
lancedb.merge.LanceMergeInsertBuilder
¶
Bases:
Builder for a LanceDB merge insert operation
See merge_insert for
more context
when_matched_update_all
¶
Rows that exist in both the source table (new data) and the target table (old data) will be updated, replacing the old row with the corresponding matching row.
If there are multiple matches then the behavior is undefined. Currently this causes multiple copies of the row to be created but that behavior is subject to change.
Parameters:
-
where(, default:Optional [str ]None) –An optional filter to limit which rows are updated. Column references in this expression must be prefixed with "target." to refer to the existing table data. For example, to only update rows where the existing color is red, use:
where="target.color = 'red'"
when_not_matched_insert_all
¶
Rows that exist only in the source table (new data) should be inserted into the target table.
when_not_matched_by_source_delete
¶
when_not_matched_by_source_delete(condition: Union [str , Expr , None] = None) -> LanceMergeInsertBuilder
Rows that exist only in the target table (old data) will be deleted. An optional condition can be provided to limit what data is deleted.
Parameters:
-
condition(, default:Union [str ,Expr , None]None) –If None then all such rows will be deleted. Otherwise the condition will be used as a filter to limit what rows are deleted. Can be a SQL string or a type-safe :class:
~lancedb.expr.Exprbuilt with :func:~lancedb.expr.coland :func:~lancedb.expr.lit.
use_index
¶
Controls whether to use indexes for the merge operation.
When set to True (the default), the operation will use an index if available
on the join key for improved performance. When set to False, it forces a full
table scan even if an index exists. This can be useful for benchmarking or when
the query optimizer chooses a suboptimal path.
Parameters:
-
use_index() –bool Whether to use indices for the merge operation. Defaults to
True.
use_rte
¶
Control RTE routing for this merge.
By default (unset), a merge_insert on a table with an RTE write spec is
routed through Lance's MemWAL shard writer, and a table without one uses
the standard path.
Parameters:
-
enable() –bool Trueforces RTE routing and errors if the table has no RTE write spec.Falseforces the standard write path even when a spec is set.
validate_single_shard
¶
Controls how an LSM merge checks that its input targets a single shard.
When a table has an RTE write spec, every row in a merge_insert call
must route to the same shard. When True (the default), every row is
inspected to verify this. When False, only the first row is inspected
and the shard it routes to is used for the whole input — a faster path
for callers that have already pre-sharded their input.
Has no effect on tables without an RTE write spec.
Parameters:
-
validate_single_shard() –bool Whether to check every row routes to one shard. Defaults to
True.
execute
¶
execute(new_data: DATA , on_bad_vectors: str = 'error', fill_value: float = 0.0, timeout: Optional [timedelta ] = None) -> MergeInsertResult
Executes the merge insert operation
Nothing is returned but the Table is updated
Parameters:
-
new_data() –DATA New records which will be matched against the existing records to potentially insert or update into the table. This parameter can be anything you use for
add -
on_bad_vectors(, default:str 'error') –What to do if any of the vectors are not the same size or contains NaNs. One of "error", "drop", "fill".
-
fill_value(, default:float 0.0) –The value to use when filling vectors. Only used if on_bad_vectors="fill".
-
timeout(, default:Optional [timedelta ]None) –Maximum time to run the operation before cancelling it.
By default, there is a 30-second timeout that is only enforced after the first attempt. This is to prevent spending too long retrying to resolve conflicts. For example, if a write attempt takes 20 seconds and fails, the second attempt will be cancelled after 10 seconds, hitting the 30-second timeout. However, a write that takes one hour and succeeds on the first attempt will not be cancelled.
When this is set, the timeout is enforced on all attempts, including the first.
Returns:
-
–MergeInsertResult version: the new version number of the table after doing merge insert.
lancedb.otel.instrument_lancedb_metrics
¶
Register LanceDB metrics as OpenTelemetry observable instruments.
Installs a process-global metrics recorder and creates one observable
instrument per LanceDB metric on the given (or global) MeterProvider. The
user's configured MetricReader then collects them on its own schedule.
Counters and gauges map directly to observable counters/gauges. Each
histogram is exported as cumulative le bucket counts (<name>_bucket,
with an le attribute) plus <name>_count and <name>_sum.
Parameters:
-
meter_provider(, default:MeterProvider None) –The provider to register instruments on. Defaults to the global provider from
opentelemetry.metrics.get_meter_provider().
Returns:
-
–bool Trueif the recorder is installed and instruments are registered.Falseif a differentmetricsrecorder is already installed in this process (metricspermits only one global recorder), in which case a warning is emitted and no instruments are created.
Notes
Requires the OpenTelemetry API (pip install lancedb[otel]) and, to
actually export, an OpenTelemetry SDK (pip install opentelemetry-sdk)
configured by the application. Calling this more than once is safe;
instruments are created only on the first successful call.
Exceptions¶
Remote HTTP responses with structured resource details raise a subclass of
HttpError. Older servers that omit those details generally raise plain
HttpError, so except HttpError remains a compatible catch-all:
from lancedb.remote.errors import (
AlreadyExistsError,
FunctionNotFoundError,
HttpError,
NamespaceNotFoundError,
SecretNotFoundError,
)
try:
db.describe_namespace(["analytics"])
except (SecretNotFoundError, FunctionNotFoundError, NamespaceNotFoundError) as error:
print(error.error_code, error.resource_kind, error.resource_name)
except AlreadyExistsError as error:
print("resource already exists:", error.resource_name)
except HttpError as error:
print("request failed:", error.request_id, error.status_code)
Remote branch deletion sends a single request. If the server rejects it with
HTTP 409, it raises ConflictError with the server's reason, including when an
older server omits structured resource details. A structured already_exists
response keeps its more specific AlreadyExistsError type. Both are subclasses
of HttpError.
A remote query whose streamed response stops partway raises
ResponseInterruptedError, also a subclass of HttpError. It carries the number
of batches and rows already returned and, when the server reports it, the table
version the response reflected, so the query can be run again against that
version. It is never retried automatically, because the batches already returned
would be returned again.
SQL response failures raise SqlError, retaining the numeric grpc_code and
request_id. Servers that report a recognized x-lancedb-error-code select a
specific input-error subclass. HTTP failures use the corresponding HttpError
subclass. Missing metadata stays absent; diagnostic text is never used to guess
the error type. An Internal or Unavailable response remains a server failure,
even if its message or metadata resembles an input error.
For errors that can arrive over either transport, catch the shared input type:
from lancedb.exceptions import ColumnNotFoundError, SchemaMismatchError
from lancedb.remote.errors import HttpError, SqlError
try:
db.execute_query("SELECT missing FROM events").read_all()
except (ColumnNotFoundError, SchemaMismatchError) as error:
print(error.request_id, error.job_id, error.job_request_id)
except (HttpError, SqlError) as error:
print(error.request_id)
ReferenceNotFoundError identifies a missing branch or table reference. Table
errors are TableNotFoundError for HTTP and SqlTableNotFoundError for SQL;
both remain compatible with lance_namespace.errors.TableNotFoundError.
A background SQL job still raises JobFailedError. Its __cause__ is the typed
SqlError when available, and both carry the current request_id and original
worker job_request_id. The outer job ID can differ from sql_job_id. These
fields are None when the server does not report them. Older servers produce
generic HTTP/SQL exceptions with their transport status and current request ID.
lancedb.remote.errors.HttpError
¶
Bases:
An error that occurred during an HTTP request.
lancedb.remote.errors.ColumnNotFoundError
¶
Bases: ,
A remote HTTP request named a missing table column.
lancedb.remote.errors.ReferenceNotFoundError
¶
Bases: ,
A remote HTTP request named a missing branch or table reference.
lancedb.remote.errors.SchemaMismatchError
¶
Bases: ,
Input to a remote HTTP request does not match the table schema.
lancedb.remote.errors.SqlError
¶
Bases:
A SQL response with its gRPC status and correlation fields.
grpc_code is the numeric gRPC code. error_code is an optional
machine-readable classification supplied by the server; message text is
never used to choose a subclass. job_id and job_request_id are absent
unless the server reports an isolated SQL job. request_id identifies the
current request. Older servers still produce SqlError without a category.
Examples:
lancedb.remote.errors.SqlColumnNotFoundError
¶
Bases: ,
A SQL request named a missing column.
lancedb.remote.errors.SqlTableNotFoundError
¶
Bases: , ,
A SQL request named a missing table.
lancedb.remote.errors.SqlReferenceNotFoundError
¶
Bases: ,
A SQL request named a missing branch or table reference.
lancedb.remote.errors.SqlSchemaMismatchError
¶
Bases: ,
SQL input does not match the table schema.
lancedb.exceptions.ColumnNotFoundError
¶
Bases:
A requested table column does not exist.
Remote HTTP and SQL errors inherit this type and carry request metadata. Locally rejected queries have no request or job identifiers.
lancedb.exceptions.ReferenceNotFoundError
¶
lancedb.exceptions.SchemaMismatchError
¶
lancedb.remote.errors.TableNotFoundError
¶
Bases: , ,
A remote table was not found.
This is also a lance_namespace.errors.TableNotFoundError and a
ValueError, so existing missing-table handlers continue to work.
Like HttpError, it carries the
request_id and status_code of the failed request.
Examples:
>>> error = TableNotFoundError("Table 'missing' was not found", "req-123", 404)
>>> isinstance(error, ValueError)
True
>>> error.request_id
'req-123'
lancedb.remote.errors.SecretNotFoundError
¶
Bases:
A remote secret was not found.
lancedb.remote.errors.FunctionNotFoundError
¶
Bases:
A remote function was not found.
lancedb.remote.errors.NamespaceNotFoundError
¶
Bases:
A remote namespace was not found.
lancedb.remote.errors.AlreadyExistsError
¶
Bases:
A remote resource already exists.
lancedb.remote.errors.ConflictError
¶
Bases:
A remote operation conflicts with the current state of a resource.
For example, deleting a branch whose files are still needed by another branch raises this error. The server's reason is included in the message.
Examples:
>>> error = HttpError("main depends on the branch's files", "req-1", 409)
>>> isinstance(error, ConflictError)
True
>>> error.status_code
409
lancedb.remote.errors.ResponseInterruptedError
¶
Bases:
A streamed query response stopped before the server finished sending it.
The batches already returned are valid, but the result is incomplete. The
query is not retried automatically, because a new attempt would return those
rows a second time. To read the same data in a new attempt, check out
table_version first.
Examples:
lancedb.exceptions.MissingValueError
¶
Bases:
Exception raised when a required value is missing.
lancedb.exceptions.MissingColumnError
¶
Bases:
Exception raised when a column name specified is not in the DataFrame object
lancedb.exceptions.JobNotFoundError
¶
Bases:
Exception raised when opening a job the server does not have.
lancedb.exceptions.JobFailedError
¶
Bases:
Exception raised when an asynchronous job reaches the failed state.
job_id identifies the failed job when available. sql_job_id preserves
the SQL execution identity reported by the server, which may differ from
the outer job's identity. request_id and job_request_id identify the
current request and isolated worker request. A typed SQL cause is available
through __cause__ when the server reports one.
lancedb.exceptions.JobCancelledError
¶
Bases:
Exception raised when an asynchronous job was cancelled.
Integrations¶
Pydantic¶
lancedb.pydantic.pydantic_to_schema
¶
Convert a Pydantic Model to a PyArrow Schema.
Parameters:
-
model() –Type [BaseModel ]The Pydantic BaseModel to convert to Arrow Schema.
Returns:
-
–Schema The Arrow Schema
Raises:
-
–TypeError If a field has a type with no Arrow equivalent. A
Tuplefield becomes an Arrow list, so all its elements must have the same type, as inTuple[int, int]orTuple[float, ...]; for a fixed set of fields with different types, such asTuple[int, str], use a nestedLanceModel(an Arrow struct) instead. Enum values must also all have the same type.
Examples:
>>> from typing import List, Optional
>>> import pydantic
>>> from lancedb.pydantic import pydantic_to_schema, Vector
>>> class FooModel(pydantic.BaseModel):
... id: int
... s: str
... vec: Vector(1536) # fixed_size_list<item: float32>[1536]
... li: List[int]
...
>>> schema = pydantic_to_schema(FooModel)
>>> assert schema == pa.schema([
... pa.field("id", pa.int64(), False),
... pa.field("s", pa.utf8(), False),
... pa.field("vec", pa.list_(pa.float32(), 1536)),
... pa.field("li", pa.list_(pa.int64()), False),
... ])
lancedb.pydantic.Vector
¶
Vector(dim: int , value_type: DataType = pa.float32(), nullable: bool = True) -> Type [FixedSizeListMixin ]
Pydantic Vector Type.
Warning
Experimental feature.
Parameters:
-
dim() –int The dimension of the vector.
-
value_type(, default:DataType ) –float32 ()The value type of the vector, by default pa.float32()
-
nullable(, default:bool True) –Whether the vector is nullable, by default it is True.
Examples:
>>> import pydantic
>>> from lancedb.pydantic import Vector
...
>>> class MyModel(pydantic.BaseModel):
... id: int
... url: str
... embeddings: Vector(768)
>>> schema = pydantic_to_schema(MyModel)
>>> assert schema == pa.schema([
... pa.field("id", pa.int64(), False),
... pa.field("url", pa.utf8(), False),
... pa.field("embeddings", pa.list_(pa.float32(), 768))
... ])
Notes
Vector creates a type dynamically, so calls such as Vector(768) are
not valid static type annotations. For an embedding field, use the standard
list[float] annotation when running mypy; VectorField supplies the
fixed dimension to LanceDB::
class MyModel(LanceModel):
text: str = embeddings.SourceField()
vector: list[float] = embeddings.VectorField()
lancedb.pydantic.MultiVector
¶
Pydantic MultiVector Type for multi-vector embeddings.
This type represents a list of vectors, each with the same dimension. Useful for models that produce multiple embeddings per input, like ColPali.
Parameters:
-
dim() –int The dimension of each vector in the multi-vector.
-
value_type(, default:DataType ) –float32 ()The value type of the vectors, by default pa.float32()
-
nullable(, default:bool True) –Whether the multi-vector is nullable, by default it is True.
Examples:
>>> import pydantic
>>> from lancedb.pydantic import MultiVector
...
>>> class MyModel(pydantic.BaseModel):
... id: int
... text: str
... embeddings: MultiVector(128) # List of 128-dimensional vectors
>>> schema = pydantic_to_schema(MyModel)
>>> assert schema == pa.schema([
... pa.field("id", pa.int64(), False),
... pa.field("text", pa.utf8(), False),
... pa.field("embeddings", pa.list_(pa.list_(pa.float32(), 128)))
... ])
lancedb.pydantic.LanceModel
¶
Bases:
A Pydantic Model base class that can be converted to a LanceDB Table.
Examples:
>>> import lancedb
>>> from lancedb.pydantic import LanceModel, Vector
>>>
>>> class TestModel(LanceModel):
... name: str
... vector: Vector(2)
...
>>> db = lancedb.connect("./example")
>>> table = db.create_table("test", schema=TestModel)
>>> table.add([
... TestModel(name="test", vector=[1.0, 2.0])
... ])
AddResult(version=2)
>>> table.search([0., 0.]).limit(1).to_pydantic(TestModel)
[TestModel(name='test', vector=FixedSizeList(dim=2))]
parse_embedding_functions
¶
Parse the embedding functions from this model.
PyTorch¶
lancedb.streaming.StreamingDataset
¶
Bases:
An elastic, resumable PyTorch IterableDataset backed by a LanceDB table.
The table is partitioned into num_splits fixed splits using a
deterministic random shuffle controlled by shuffle_seed and epoch.
Each rank is assigned a contiguous block of splits, and within a rank each
DataLoader worker is assigned a contiguous sub-block. Samples are yielded
by round-robining over the assigned splits, one sample per split per cycle.
Internally __iter__ runs a two-stage pipeline:
- Stage 1 (I/O): one thread pool with
num_splits * io_queue_depthworkers fetches rawRecordBatchobjects from LanceDB in parallel across all splits and places them in a per-split raw-batch queue. - Stage 2 (transform): a second thread pool with
transform_parallelismworkers picks up raw batches, applies the transform, and places the results in a per-split cooked-row queue. By default, the number of workers is determined byos.cpu_count().
The main thread round-robins over the cooked queues, yielding one row per split per cycle.
Parameters:
-
table–LanceDB table to stream from.
-
num_splits(, default:Optional [int ]None) –Number of fixed splits to partition the table into. Must be divisible by
world_size. When used with DataLoader workers it must also be divisible byworld_size * num_workers. Defaults toworld_size. If the row count (after anyfilter) is not evenly divisible bynum_splits, the surplus rows — at mostnum_splits - 1per epoch — are silently dropped to keep all splits the same length. -
shuffle(, default:bool True) –Whether to randomly assign rows to splits. When
True(the default) rows are shuffled usingshuffle_seedandepoch. WhenFalserows are divided into splits sequentially in storage order, which can be useful for deterministic debugging or evaluation. Ignored whensplit_strategyis provided. -
split_strategy(, default:Optional [Callable [[PermutationBuilder ],PermutationBuilder ]]None) –Escape hatch for any
PermutationBuildersplit strategy this class doesn't otherwise expose a dedicated parameter for --split_hash,split_calculated,bootstrap_sample(uniform sampling with replacement), or any strategy added later. A callable that receives a freshPermutationBuilderfortableand returns a configured one; it is called and.execute()'d directly, taking over permutation construction entirely. When provided,shuffle,shuffle_seed,epoch, andshuffle_clump_sizeare ignored -- the callable is solely responsible for how rows are split and ordered. For example,split_strategy=lambda b: b.bootstrap_sample(10_000, fixed=num_splits, seed=42)draws 10,000 samples with replacement, divided evenly acrossnum_splitssplits. -
shuffle_seed(, default:Optional [int ]0) –Base seed for the random permutation. Combined with
epochso each epoch produces a different ordering. PassNoneto generate a random seed at construction time. -
epoch(, default:int 0) –Current training epoch. Combined with
shuffle_seedso that each epoch produces a different sample ordering. -
rank(, default:int 0) –This process's rank in the distributed training group.
-
world_size(, default:int 1) –Total number of processes in the distributed training group.
-
read_batch_size(, default:int ) –DEFAULT_READ_BATCH_SIZE Number of rows fetched from each split in a single
take_offsetscall. Larger values amortise per-request overhead (critical on object storage) at the cost of higher memory usage per split buffer. Defaults toDEFAULT_READ_BATCH_SIZE(64). -
io_queue_depth(, default:int ) –DEFAULT_PREFETCH_BATCHES Number of fetched batches to keep ahead of the transform stage per split: this bounds the combined count of in-flight fetches and already-fetched batches still waiting for a transform slot. Higher values overlap storage latency with transform and training compute at the cost of more memory and threads, but I/O stops running ahead once this many batches are buffered, applying backpressure when the transform stage (or
transform_queue_depth) is the bottleneck. Must be greater than zero. Defaults toDEFAULT_PREFETCH_BATCHES(4). -
columns(, default:Optional [list [str ]]None) –Optional list of column names to read. When set, only those columns are fetched from storage; all others are omitted.
None(the default) reads every column. -
shuffle_clump_size(, default:Optional [int ]None) –When set, rows are shuffled in contiguous groups of this size rather than individually. Larger clumps improve I/O locality (important on object storage) at the cost of reduced randomness.
None(the default) shuffles rows individually. -
filter(, default:Optional [str ]None) –Optional SQL filter expression (e.g.
"label = 'dog'"). Only rows that satisfy the predicate are included in the permutation. The filter is applied during permutation construction so split sizes reflect the filtered row count. -
transform(, default:Optional [Callable ]None) –Optional callable applied to each
pyarrow.RecordBatchbefore rows are yielded. Receives one batch at a time and must return an iterable whose length equals the number of rows in the batch. WhenNone(the default) rows are returned as plain Python dicts. -
transform_parallelism(, default:Optional [int ]None) –Maximum number of transforms to run concurrently. Must be greater than zero. When
None(the default), usesos.cpu_count()or 1 when the CPU count is unavailable. -
pack_sequences(, default:Optional [int ]None) –Sequence-packing mode: token lists from consecutive documents are joined with
eos_idand sliced into blocks of this many tokens. Each item is then a dict of two(pack_sequences,)LongTensors —input_idsanddoc_ids(per-position document index within the block, for block-diagonal masks or position-id resets). * Packing happens independently per owned split and preserves per-split resume state. * When a split cannot fill a real block for a cycle but an owned sibling still can, or when only a short tail remains at epoch end, the short buffer is padded topack_sequenceswithpad_idso every local cycle emits one block per owned split. *eos_id,pad_id, andcolumnsnaming a single integer-list column are required; incompatible withtransform. -
eos_id(, default:Optional [int ]None) –Separator token id between packed documents. Required with
pack_sequences, ignored otherwise. -
pad_id(, default:Optional [int ]None) –Padding token id used to complete blocks when a split runs out of real tokens mid-cycle or at epoch end. Required with
pack_sequences, ignored otherwise. It must be reserved for padding: padding positions retain the preceding document'sdoc_id(or zero in an all-padding block), so callers must mask them separately usinginput_ids == pad_id. -
blocks_per_epoch(, default:Optional [Union [int ,Literal ['auto']]]None) –Total number of packed blocks emitted globally per epoch. Required with
pack_sequences. An integer must be divisible bynum_splits. Every logical split emits exactlyblocks_per_epoch / num_splitsblocks: exhausted splits emit padding, while tokens beyond the budget are left out of the epoch. This fixed per-split budget keeps packed iteration and checkpoints independent of rank topology. Pass"auto"to estimate a corpus-level budget from a bounded sample of token lists. The estimate may be inaccurate. -
on_transform_error(, default:Union [str ,Callable [[Exception ],bool ]]'raise') –What to do when the transform raises an exception:
"raise"(the default): the exception propagates and iteration aborts."skip": the failing rows are dropped and iteration continues."warn": like"skip", but a warning is logged for each failing batch."fill": replace each failing row withfill_valueor the result ofrepair(exc, row_batch). Exactly one of these must be provided. Filled rows retain their original positions, so split lengths, elastic determinism, and checkpoint offsets are preserved.- a callable
handler(exc) -> bool: called with the exception; returnTrueto skip the failing rows orFalseto re-raise. Useful to skip only expected error types (compatible withwebdataset.handlersstyle handlers).
When a batch fails, the transform is re-invoked on each single-row slice of the batch so that only the rows that actually fail are skipped or filled. Transforms should therefore be deterministic and accept batches of any size (including one row). Skipped and filled rows are counted in
rows_skippedandrows_filledrespectively.Skipping weakens the elastic-determinism guarantee at the end of the epoch: splits that lose more rows than others run dry earlier, and each rank's iterator ends at the last cycle where every split it owns still has a row. Because bad rows are not distributed evenly across splits, this means one rank's iterator can yield noticeably fewer or more steps than another rank's in the same run — there is no cross-rank coordination that stops every rank at the same global step. This is generally safe for asynchronous or single-rank use, but synchronous distributed training (e.g. ranks that call
all_reduceevery step) can hang or deadlock if one rank's iterator is exhausted while others are still stepping; callers doing synchronous multi-rank training withon_transform_error="skip"or"warn"(or a skipping callable) are responsible for their own cross-rank stopping mechanism (e.g. broadcasting a stop signal onStopIteration). The final few global steps can also differ across topologies (bounded by the skew in bad-row counts across splits). The sequence of samples yielded from each split remains deterministic. Mid-epoch checkpoints remain exact provided the transform fails deterministically; in multi-rank training each rank must save its ownstate_dictand the states must be combined withmerge_state_dictsbefore resuming on a different topology. Prefer thefilterparameter when bad rows can be expressed as a SQL predicate (e.g."col IS NOT NULL") — filtering happens before splits are built, so every guarantee is fully preserved. -
fill_value(, default:Any ) –_UNSET Constant output row used with
on_transform_error="fill". A copy is made for each failed row so mutable placeholders are independent. For example,fill_value={"vector": [0.0, 0.0]}. Pass exactly one offill_valueorrepairin fill mode. -
repair(, default:Optional [Callable [[Exception ,RecordBatch ],Any ]]None) –Optional function
repair(exc, row_batch) -> rowused withon_transform_error="fill".row_batchis a one-rowpyarrow.RecordBatch. The returned value replaces that row's transformed output. Replacements should have the same structure as successful transform outputs. The repair must be deterministic for reproducible checkpoints and elastic steps. -
transform_queue_depth(, default:Optional [int ]None) –Number of transform-result batches to buffer per split in the post-transform queue before backpressure is applied to the transform stage. When the combined count of in-flight transform futures and already-buffered rows for a split reaches
transform_queue_depth * read_batch_size, no new transforms are submitted for that split until the consumer catches up. Useful for capping peak memory when the consumer (e.g. a GPU training step) is slower than the transform stage. Must be greater than zero.None(the default) imposes no limit. -
connection_factory(, default:Optional [Callable [[str ],Any ]]None) –Optional picklable function that takes the table name and returns an open LanceDB table. When set, pickling the dataset (as DataLoader workers do under the
spawnandforkserverstart methods) replaces the table's reopen state with its name, and each worker calls the factory with that name to reopen the table. The factory itself is pickled too, so credentials stay out of the pickle only if the factory loads them in the worker, for example from the environment, instead of capturing them.None(the default) pickles the table's own reopen state instead. -
worker_info_override–If set, used in place of
torch.utils.data.get_worker_info()to determine the DataLoader worker assignment. Intended for unit tests that need to simulate multiple workers without spawning real processes. If both this and the real worker info are non-None a warning is logged and the override takes precedence.
bytes_loaded
¶
Cumulative bytes of raw Arrow buffer data fetched from storage.
Measured on the RecordBatch before any transform is applied, so
the value reflects actual I/O rather than the size of transformed
output. Accumulates across multiple iterations of the same dataset
instance and is never reset automatically.
fetch_time
¶
Cumulative seconds spent waiting for data from LanceDB.
Measured per batch in the Stage 1 I/O threads as the total elapsed
time of the take_offsets call. Accumulates across all splits and
all iterations.
transform_time
¶
Cumulative seconds spent applying the transform.
Measured per batch in the Stage 2 transform threads as the elapsed
time inside the transform callable (or the default arrow2python
conversion when no transform is set). Accumulates across all splits
and all iterations.
raw_queue_depth
¶
Number of raw rows waiting for a transform thread across all splits.
A persistently non-zero value means Stage 2 (transform) is the bottleneck: I/O is completing faster than transforms can consume batches. Returns 0 when not iterating.
prefetch_queue_depth
¶
Number of rows transformed and ready to yield across all splits.
Counts rows whose transform has completed and are sitting in memory waiting for the main thread — rows that can be handed off with no I/O or CPU wait. Returns 0 when not iterating.
unscanned_rows
¶
Number of rows not yet submitted to the I/O stage across all splits.
Decreases as the I/O stage submits fetch requests. When this reaches zero all data has been requested from storage (though it may not have arrived yet). Returns 0 when not iterating.
rows_skipped
¶
Number of rows dropped because their transform raised an exception.
Only ever non-zero when on_transform_error is set to "skip",
"warn", or a callable that returned True. Accumulates across
multiple iterations of the same dataset instance and is never reset
automatically.
rows_filled
¶
Number of failed transform rows replaced by fill mode.
Accumulates across iterations of the same dataset instance and is never reset automatically.
consumed_rows
¶
Number of rows already yielded to the caller across all splits.
Monotonically increases throughout iteration. Returns 0 when not iterating.
state_dict
¶
Snapshot the dataset's consumption state.
When using DataLoader workers, construct a
StreamingDataLoader. It
commits worker state only when a prefetched batch is returned to the
trainer. A standard multi-process DataLoader cannot expose that
boundary, so calling this method after one has started raises
RuntimeError instead of returning stale producer state.
In row mode, the returned dict is topology-independent at global step
boundaries. positions_consumed_per_split records how far each
split's permutation has advanced, which can differ from the sample
count when on_transform_error skips rows. StreamingDataLoader
combines worker state in its parent process. Combine state dicts from
every rank with
merge_state_dicts
before resuming on a different topology.
Packed state includes partial token buffers and emitted block counts
for every logical split. When packing is sharded, merge every rank
state with merge_state_dicts before loading it.
load_state_dict
¶
Resume from a previously snapshotted state.
Raises ValueError if num_splits or shuffle_seed differ
from the checkpoint, since a different split structure or shuffle order
makes mid-epoch resumption meaningless. Also raises if the checkpoint
was built with split_strategy and records a permutation fingerprint
that does not match the current dataset's permutation -- this catches a
changed or non-deterministically-reseeded split_strategy that
num_splits/shuffle_seed alone cannot detect, since shuffle_seed
is unused on that path. Checkpoints from split_random/
split_sequential never carry this fingerprint (row ids are not stable
across compaction, so it would reject an otherwise-valid resume), and
continue to rely on shuffle_seed/num_splits alone as before.
Packed checkpoints pin pack_sequences, eos_id, pad_id,
blocks_per_epoch, and epoch.
merge_state_dicts
¶
Merge state dicts saved by different ranks into one exact state.
In row mode, each rank records exact consumer-committed progress for its own splits and lower bounds for the rest, so elementwise maxima recover both sample counts and permutation positions. In packed mode, the state that emitted the most blocks for each logical split supplies that split's permutation position and partial token buffer. Packed states must cover every rank at the same global step.
Raises ValueError if the states are empty, were not produced by
the same run, or do not represent the same global step.
The merge is always all-to-all and topology-agnostic: collect the
state_dict() from every rank of the previous run into
one list, merge that whole list, and hand the identical merged result
to every rank of the next run — regardless of whether the
topology grew, shrank, or stayed the same. There is no pairwise or
subset merging step, because each split's exact state is only known to
whichever iterator owned that split.
For example, checkpointing 8 ranks and resuming on 4 (the same pattern applies when growing, e.g. 4 ranks resuming on 8)::
states = [ds.state_dict() for ds in previous_run_datasets] # 8
merged = StreamingDataset.merge_state_dicts(states)
for ds in resumed_datasets: # now only 4 ranks
ds.load_state_dict(merged) # same dict on every rank
The rank count on either side never affects the merge itself, since
merge_state_dicts only cares about the list of states it is
given. Each split's position is recovered by elementwise maximum;
here rank 0 owned split 0 (and skipped two rows there) while rank 1
owned split 1 (and skipped one row):
rank0 = { ... "shuffle_seed": 0, "num_splits": 2, "epoch": 0, ... "samples_consumed_per_split": [3, 3], ... "positions_consumed_per_split": [5, 3], ... } rank1 = { ... "shuffle_seed": 0, "num_splits": 2, "epoch": 0, ... "samples_consumed_per_split": [3, 3], ... "positions_consumed_per_split": [3, 4], ... } merged = StreamingDataset.merge_state_dicts([rank0, rank1]) merged["positions_consumed_per_split"] [5, 4]
When a transform fails for some rows, fill mode replaces each failed row in place:
dataset = StreamingDataset(
table,
num_splits=8,
transform=transform,
on_transform_error="fill",
fill_value={"id": -1, "vector": [0.0, 0.0]},
)
for row in dataset:
train(row)
print(dataset.rows_filled)
The placeholder must have the same structure as a successful transform output.
Pass repair(exc, row_batch) -> row instead of fill_value to construct a
replacement from the failed one-row Arrow batch. Keep the repair deterministic
when using elastic training or checkpoints. Fill mode preserves split lengths
and sample positions.
lancedb.streaming.StreamingDataLoader
¶
Bases:
A PyTorch DataLoader with consumer-committed dataset checkpoints.
PyTorch workers prefetch batches ahead of the trainer, so worker-local
producer progress is not a safe checkpoint. This loader carries a state
snapshot alongside every internal batch and applies it to the parent
StreamingDataset only when that batch
is returned by next().
The trainer receives the same collated batch it would receive from a
standard torch.utils.data.DataLoader.
With more than one worker, row-mode state_dict() is available only at
complete logical step boundaries, when every split assigned to the rank has
the same consumed-sample count. Packed checkpoints require equal emitted-block
counts across the rank's splits for any worker count. persistent_workers=True
is not supported because prefetched worker copies cannot be restored from
parent-committed state. If batch collation raises, checkpointing remains
invalid for that dataset instance; restore the last valid checkpoint into a
fresh dataset before continuing.
Only one active iterator may own a dataset at a time, including when worker
processes are used. Exhausting or explicitly shutting down the iterator
releases that ownership. drop_last=True is not supported because worker
replicas discard incomplete tails independently, which cannot produce a
topology-independent checkpoint.
Parameters are the same as torch.utils.data.DataLoader except that
dataset must be a
StreamingDataset.
Subclasses that override StreamingDataset.__iter__ are not supported
because the custom iterator cannot provide the exact per-yield checkpoint
snapshots required by this loader.
Examples:
>>> # dataset = StreamingDataset(table, num_splits=2)
>>> # loader = StreamingDataLoader(dataset, batch_size=8, num_workers=2)
>>> # batch = next(iter(loader))
>>> # checkpoint = dataset.state_dict()
lancedb.permutation.PermutationBuilder
¶
A utility for creating a "permutation table" which is a table that defines an ordering on a base table.
The permutation table does not store the actual data. It only stores row ids and split ids to define the ordering. The [Permutation] class can be used to read the data from the base table in the order defined by the permutation table.
Permutations can split, shuffle, and filter the data in the base table.
A filter limits the rows that are included in the permutation. Splits divide the data into subsets (for example, a test/train split, or K different splits for cross-validation). Shuffling randomizes the order of the rows in the permutation.
Splits can optionally be named. If names are provided it will enable them to be referenced by name in the future. If names are not provided then they can only be referenced by their ordinal index. There is no requirement to name every split.
The permutation is stored in memory and will be lost when the program exits.
split_random
¶
split_random(*, ratios: Optional [list [float ]] = None, counts: Optional [list [int ]] = None, fixed: Optional [int ] = None, seed: Optional [int ] = None, clump_size: Optional [int ] = None, split_names: Optional [list [str ]] = None) -> PermutationBuilder
Configure random splits for the permutation.
One of ratios, counts, or fixed must be provided.
If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows.
If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows.
If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits.
Rows will be randomly assigned to splits. The optional seed can be provided to make the assignment deterministic.
If clump_size is provided, rows are shuffled as contiguous groups of that size, preserving I/O locality while still randomising the split assignment.
The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.
split_hash
¶
split_hash(columns: list [str ], split_weights: list [int ], *, discard_weight: Optional [int ] = None, split_names: Optional [list [str ]] = None) -> PermutationBuilder
Configure hash-based splits for the permutation.
First, a hash will be calculated over the specified columns. The splits weights are then used to determine how many rows to assign to each split. For example, if split weights are [1, 2] then the first split will contain 1/3 of the rows and the second split will contain 2/3 of the rows.
The optional discard weight can be provided to determine what percentage of rows should be discarded. For example, if split weights are [1, 2] and discard weight is 1 then 25% of the rows will be discarded.
Hash-based splits are useful if you want the split to be more or less random but you don't want the split assignments to change if rows are added or removed from the table.
The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.
split_sequential
¶
split_sequential(*, ratios: Optional [list [float ]] = None, counts: Optional [list [int ]] = None, fixed: Optional [int ] = None, split_names: Optional [list [str ]] = None) -> PermutationBuilder
Configure sequential splits for the permutation.
One of ratios, counts, or fixed must be provided.
If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows.
If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows.
If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits.
Rows will be assigned to splits sequentially. The first N1 rows are assigned to split 1, the next N2 rows are assigned to split 2, etc.
The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.
bootstrap_sample
¶
bootstrap_sample(num_samples: int , *, ratios: Optional [list [float ]] = None, counts: Optional [list [int ]] = None, fixed: Optional [int ] = None, seed: Optional [int ] = None, split_names: Optional [list [str ]] = None) -> PermutationBuilder
Sample rows uniformly with replacement, independently per split.
Unlike the other split strategies, the number of output rows is
num_samples, not the number of (optionally filtered) input rows: the
same row can appear zero, one, or many times, and num_samples may be
larger or smaller than the input row count. Useful for bootstrap
resampling, oversampling rare classes, or any training workload that
wants a fixed-size stream of random draws rather than a single
deterministic pass over every row. Combine with :meth:filter to
restrict the eligible population.
One of ratios, counts, or fixed must be provided, with the same meaning
as in :meth:split_random, except they divide num_samples across
splits rather than the input row count: if fixed is 4, each split gets
num_samples // 4 samples.
Each split draws its own sample using a seed derived independently from
(seed, split_index), so a split's content does not change if the samples are
later redistributed across a different number of splits.
The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.
split_calculated
¶
Use pre-calculated splits for the permutation.
The calculation should be an SQL statement that returns an integer value between 0 and the number of splits - 1. For example, if you have 3 splits then the calculation should return 0 for the first split, 1 for the second split, and 2 for the third split.
This can be used to implement any kind of user-defined split strategy.
The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.
shuffle
¶
Randomly shuffle the rows in the permutation.
An optional seed can be provided to make the shuffle deterministic.
If a clump size is provided, then data will be shuffled as small "clumps" of contiguous rows. This allows for a balance between randomization and I/O performance. It can be useful when reading from cloud storage.
filter
¶
Configure a filter for the permutation.
The filter should be an SQL statement that returns a boolean value for each row. Only rows where the filter is true will be included in the permutation.
lancedb.permutation.Permutation
¶
A Permutation is a view of a dataset that can be used as input to model training and evaluation.
A Permutation fulfills the pytorch Dataset contract and is loosely modeled after the huggingface Dataset so it should be easy to use with existing code.
A permutation is not a "materialized view" or copy of the underlying data. It is calculated on the fly from the base table. As a result, it is truly "lazy" and does not require materializing the entire dataset in memory.
shape
¶
The shape of the permutation
This will return self.num_rows, self.num_columns
with_batch_size
¶
Creates a new permutation with the given batch size
with_connection_factory
¶
Creates a new permutation that will use connection_factory to reopen
the base table when this permutation is unpickled in a worker process.
The factory is a callable that takes a single argument, the base table
name, and returns a LanceDB table. It must be picklable; the worker
will pickle it via standard pickle and call it to recover the base
table. Picklable callables in practice means top-level (module-level)
functions, functools.partial of such functions, or instances of
picklable classes implementing __call__. Lambdas and closures over
local variables don't pickle with the default protocol.
A factory is optional for normal LanceDB Embedded and Enterprise connections:
if not set, __getstate__ captures the table's own picklable reopen
state. Use a factory when that default state is not enough, for example
when credentials should be loaded from the worker environment instead
of being embedded in the pickle.
Examples:
Basic native (file-system path), parameterized via functools.partial::
import functools, lancedb
from lancedb.permutation import Permutation
def open_native_table(uri: str, table_name: str):
return lancedb.connect(uri).open_table(table_name)
factory = functools.partial(open_native_table, "/data/lance_db")
permutation = Permutation.identity(
factory("training")
).with_connection_factory(factory)
Native via :func:lancedb.connect_namespace (e.g. a directory- or
REST-backed namespace client). The factory takes the
implementation name and properties dict as partial-bound args so
the worker can rebuild the same namespace connection::
def open_via_namespace(
impl: str, properties: dict[str, str], table_name: str,
):
return lancedb.connect_namespace(impl, properties).open_table(
table_name,
)
factory = functools.partial(
open_via_namespace,
"dir",
{"root": "/data/lance_db"},
)
LanceDB Enterprise, reading credentials from env vars at worker startup so secrets aren't pickled into the dataset::
import os, lancedb
def open_remote_table(table_name: str):
db = lancedb.connect(
"db://my-database",
api_key=os.environ["LANCEDB_API_KEY"],
region=os.environ.get("LANCEDB_REGION", "us-east-1"),
)
return db.open_table(table_name)
permutation = Permutation.identity(
open_remote_table("training")
).with_connection_factory(open_remote_table)
identity
¶
Creates an identity permutation for the given table.
from_tables
¶
from_tables(base_table: Table , permutation_table: Optional [Table ] = None, split: Optional [Union [str , int ]] = None) -> Permutation
Creates a permutation from the given base table and permutation table.
A permutation table identifies which rows, and in what order, the data should be read from the base table. For more details see the [PermutationBuilder] class.
If no permutation table is provided, then the identity permutation will be created. An identity permutation is a permutation that reads all rows in the base table in the order they are stored.
The split parameter identifies which split to use. If no split is provided then the first split will be used.
flatten
¶
Flatten the permutation
Each column with a struct type will be flattened into multiple columns.
This flattening operation happens at read time as a post-processing step so this call is cheap and no data is copied or modified in the underlying dataset.
remove_columns
¶
Remove the given columns from the permutation
Note: this does not actually modify the underlying dataset. It only changes which columns are visible from this permutation. Also, this does not introduce a post-processing step. Instead, we simply do not read those columns in the first place.
If any of the provided columns does not exist in the current permutation then it will be ignored (no error is raised for missing columns)
Returns a new permutation with the given columns removed. This does not modify self.
rename_column
¶
Rename a column in the permutation
If there is no column named old_name then an error will be raised If there is already a column named new_name then an error will be raised
Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied.
rename_columns
¶
Rename the given columns in the permutation
If any of the columns do not exist then an error will be raised If any of the new names already exist then an error will be raised
Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied.
select_columns
¶
Select the given columns from the permutation
This method refines the current selection, potentially removing columns. It will not add back columns that were previously removed.
If any of the columns do not exist then an error will be raised.
This does not introduce a post-processing step. It simply reduces the amount of data we read.
iter
¶
Iterate over the permutation in batches
If skip_last_batch is True, the last batch will be skipped if it is not a multiple of batch_size.
with_format
¶
with_format(format: Literal ['numpy', 'python', 'python_col', 'pandas', 'arrow', 'torch', 'torch_row', 'torch_col', 'polars']) -> Permutation
Set the format for batches
If this method is not called, the "python" format will be used.
The format can be one of:
- "numpy" - the batch will be a dict of numpy arrays (one per column)
- "python" - the batch will be a list of dicts (one per row)
- "python_col" - the batch will be a dict of lists (one entry per column)
- "pandas" - the batch will be a pandas DataFrame
- "arrow" - the batch will be a pyarrow RecordBatch
- "torch" - the batch will be a list of per-row dicts mapping column
name to a 0-D torch tensor. Works with the default
torch.utils.data.DataLoader collate, which stacks the per-row
dicts back into a dict of batched tensors.
- "torch_row" - the batch will be a list of tensors, one per row
- "torch_col" - the batch will be a 2D torch tensor (first dim indexes columns)
- "polars" - the batch will be a polars DataFrame
Conversion may or may not involve a data copy. Lance uses Arrow internally and so it is able to zero-copy to the arrow and polars formats.
Conversion to torch and torch_col will be zero-copy but will only support a subset of data types (numeric types).
Conversion to numpy and/or pandas will typically be zero-copy for numeric types. Conversion of strings, lists, and structs will require creating python objects and this is not zero-copy.
For custom formatting, use with_transform which overrides this method.
with_transform
¶
Set a custom transform for the permutation
The transform is a callable that will be invoked with each record batch. The return value will be used as the batch for iteration.
Note: transforms are not invoked in parallel. This method is not a good place for expensive operations such as image decoding.
take_offsets
¶
Take rows from the permutation by offset
The returned value is passed through the permutation's current transform,
so with_format and with_transform affect this method in the same way
they affect iteration.
skip
¶
Skip the first skip rows of the permutation
Note: this method returns a new permutation and does not modify self
It is provided for compatibility with the huggingface Dataset API.
Use with_skip instead to avoid confusion.
take
¶
Limit the permutation to limit rows (following any skip)
Note: this method returns a new permutation and does not modify self
It is provided for compatibility with the huggingface Dataset API.
Use with_take instead to avoid confusion.
with_take
¶
Limit the permutation to limit rows (following any skip)
repeat
¶
Repeat the permutation times times
Note: this method returns a new permutation and does not modify self
It is provided for compatibility with the huggingface Dataset API.
Use with_repeat instead to avoid confusion.
lancedb.permutation.Transforms
¶
Namespace for common transformation functions
Reranking¶
TypeSafeReranker supports opt-in request batching:
from lancedb.rerankers import TypeSafeReranker
reranker = TypeSafeReranker(batch_size=40, max_concurrency=8)
The default batch_size=1 keeps the query and document in request state and
sends one request per non-null candidate. With batch_size=40, 80 non-null
candidates require two requests. max_concurrency still limits simultaneous
requests, and the SDK handles retries.
In batched mode, state contains only {"query": query}. Each independent
question contains {"question": instructions, "document": document} in its
structured instructions, so it sees only its own document and the shared query.
Custom instructions and criteria are kept verbatim: adapt prompts that explicitly
reference request-state fields such as state.document before enabling batching.
Null documents retain a zero score without an API call; empty strings are scored.
API failures, mismatched answer IDs, and invalid probabilities raise errors.
Batching changes the payload and can affect model scores; compare quality and
latency on your workload before opting in.
Reranker
¶
Bases:
rerank_vector
¶
Rerank function receives the result from the vector search. This isn't mandatory to implement
Parameters:
-
query() –str The input query
-
vector_results() –Table The results from the vector search
Returns:
-
–Table The reranked results
rerank_fts
¶
Rerank function receives the result from the FTS search. This isn't mandatory to implement
Parameters:
-
query() –str The input query
-
fts_results() –Table The results from the FTS search
Returns:
-
–Table The reranked results
rerank_hybrid
¶
Rerank function receives the individual results from the vector and FTS search results. You can choose to use any of the results to generate the final results, allowing maximum flexibility. This is mandatory to implement
Parameters:
-
query() –str The input query
-
vector_results() –Table The results from the vector search
-
fts_results() –Table The results from the FTS search
Returns:
-
–Table The reranked results
merge_results
¶
Merge the results from the vector and FTS search. This is a vanilla merging function that just concatenates the results and removes the duplicates.
NOTE: This doesn't take score into account. It'll keep the instance that was
encountered first. This is designed for rerankers that don't use the score.
In case you want to use the score, or support return_scores="all" you'll
have to implement your own merging function.
Parameters:
-
vector_results() –Table The results from the vector search
-
fts_results() –Table The results from the FTS search
rerank_multivector
¶
rerank_multivector(vector_results: Union [List [Table ], List [LanceVectorQueryBuilder ]], query: Union [str , None], deduplicate: bool = False)
This is a rerank function that receives the results from multiple vector searches. For example, this can be used to combine the results of two vector searches with different embeddings.
Parameters:
-
vector_results() –List [Table ] orList [LanceVectorQueryBuilder ]The results from the vector search. Either accepts the query builder if the results haven't been executed yet or the results in arrow format.
-
query(() –str or None)The input query. Some rerankers might not need the query to rerank. In that case, it can be set to None explicitly. This is intended to be handled by the reranker implementations.
-
deduplicate(, default:bool False) –Whether to deduplicate the results based on the
_rowidcolumn, by default False. Requires_rowidto be present in the results.
Returns:
-
–Table The reranked results
CohereReranker
¶
Bases:
Reranks the results using the Cohere Rerank API. https://docs.cohere.com/docs/rerank-guide
Parameters:
-
model_name(, default:str "rerank-english-v2.0") –The name of the cross encoder model to use. Available cohere models are: - rerank-english-v2.0 - rerank-multilingual-v2.0
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
top_n(, default:str None) –The number of results to return. If None, will return all results.
ColbertReranker
¶
Bases:
Reranks the results using the ColBERT model.
Parameters:
-
model_name(, default:str "colbert" (colbert-ir/colbert-v2.0)) –The name of the cross encoder model to use.
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
return_score(, default:str "relevance") –options are "relevance" or "all".
-
**kwargs–Additional keyword arguments to pass to the model, for example, 'device'. See AnswerDotAI/rerankers for more information.
CrossEncoderReranker
¶
Bases:
Reranks the results using a cross encoder model. The cross encoder model is used to score the query and each result. The results are then sorted by the score.
Parameters:
-
model_name(, default:str "cross-encoder/ms-marco-TinyBERT-L-6") –The name of the cross encoder model to use. See the sentence transformers documentation for a list of available models.
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
device(, default:str None) –The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu".
-
return_score(, default:str "relevance") –options are "relevance" or "all". Only "relevance" is supported for now.
-
trust_remote_code(, default:bool True) –If True, will trust the remote code to be safe. If False, will not trust the remote code and will not run it
-
batch_size(, default:int 32) –The number of query-passage pairs scored per forward pass.
-
**kwargs–Additional keyword arguments passed to
sentence_transformers.CrossEncoder, for examplemax_lengthormodel_kwargs={"torch_dtype": "bfloat16"}.
LinearCombinationReranker
¶
Bases:
Reranks the results using a linear combination of the scores from the
vector and FTS search. For missing scores, fill with fill value.
Parameters:
-
weight(, default:float 0.7) –The weight to give to the vector score. Must be between 0 and 1.
-
fill(, default:float 1.0) –The score to give to results that are only in one of the two result sets. This is treated as penalty, so a higher value means a lower score. TODO: We should just hardcode this-- its pretty confusing as we invert scores to calculate final score
-
return_score(, default:str "relevance") –opntions are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.
OpenaiReranker
¶
Bases:
Reranks the results using the OpenAI API. WARNING: This is a prompt based reranker that uses chat model that is not a dedicated reranker API. This should be treated as experimental.
Parameters:
-
model_name(, default:str "gpt-5.6-sol") –The name of the cross encoder model to use.
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
return_score(, default:str "relevance") –options are "relevance" or "all". Only "relevance" is supported for now.
-
api_key(, default:str None) –The API key to use. If None, will use the OPENAI_API_KEY environment variable.
JinaReranker
¶
Bases:
Reranks the results using the Jina Rerank API. https://jina.ai/rerank
Parameters:
-
model_name(, default:str "jina-reranker-v2-base-multilingual") –The name of the cross reanker model to use
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
top_n(, default:str None) –The number of results to return. If None, will return all results.
-
api_key(, default:str None) –The api key to access Jina API. If you pass None, you can set JINA_API_KEY environment variable
RRFReranker
¶
Bases:
Reranks the results using Reciprocal Rank Fusion(RRF) algorithm based on the scores of vector and FTS search.
Parameters:
-
K(, default:int 60) –A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. See paper: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
-
return_score(, default:str "relevance") –opntions are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.
rerank_multivector
¶
rerank_multivector(vector_results: Union [List [Table ], List [LanceVectorQueryBuilder ]], query: str = None, deduplicate: bool = True)
Overridden method to rerank the results from multiple vector searches. This leverages the RRF hybrid reranking algorithm to combine the results from multiple vector searches as it doesn't support reranking vector results individually.
MRRReranker
¶
Bases:
Reranks the results using Mean Reciprocal Rank (MRR) algorithm based on the scores of vector and FTS search. Algorithm reference - https://en.wikipedia.org/wiki/Mean_reciprocal_rank
MRR calculates the average of reciprocal ranks across different search results. For each document, it computes the reciprocal of its rank in each system, then takes the mean of these reciprocal ranks as the final score.
Parameters:
-
weight_vector(, default:float 0.5) –Weight for vector search results (0.0 to 1.0)
-
weight_fts(, default:float 0.5) –Weight for FTS search results (0.0 to 1.0) Note: weight_vector + weight_fts should equal 1.0
-
return_score(, default:str "relevance") –Options are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.
rerank_multivector
¶
rerank_multivector(vector_results: Union [List [Table ], List [LanceVectorQueryBuilder ]], query: str = None, deduplicate: bool = True)
Reranks the results from multiple vector searches using MRR algorithm. Each vector search result is treated as a separate ranking system, and MRR calculates the mean of reciprocal ranks across all systems. This cannot reuse rerank_hybrid because MRR semantics require treating each vector result as a separate ranking system.
AnswerdotaiRerankers
¶
Bases:
Reranks the results using the Answerdotai Rerank API. All supported reranker model types can be found here: - https://github.com/AnswerDotAI/rerankers
Parameters:
-
model_type(, default:str "colbert") –The type of the model to use.
-
model_name(, default:str "rerank-english-v2.0") –The name of the model to use from the given model type.
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
return_score(, default:str "relevance") –options are "relevance" or "all".
-
**kwargs–Additional keyword arguments to pass to the model. For example, 'device'. See AnswerDotAI/rerankers for more information.
TypeSafeReranker
¶
Bases:
Reranks the results using the TypeSafe System One API. https://docs.typesafe.ai/cookbooks/rerank_typesafe
Each result is scored independently: TypeSafe reads the query together with
the result's column value and answers a yes/no ("noul") question about
whether the document is relevant. The probability of "yes", between 0 and 1,
becomes the _relevance_score. Because every score is an absolute
probability rather than a position in the list, scores are comparable
across queries and can be thresholded. They are model estimates, though,
and can vary slightly between identical calls, so results with close scores
may swap places when the same search is repeated.
By default, one request is sent per non-null result. Set batch_size=40
to score up to 40 documents per request, with at most max_concurrency
requests in flight. Each question sees only its own document and the shared
query. Null documents keep a score of zero without an API request; empty
strings are scored normally. API and response-validation errors propagate.
Parameters:
-
model_name(, default:str "jev-latest") –The TypeSafe model to use.
-
column(, default:str "text") –The name of the column holding the document text to score.
-
instructions(, default:str None) –The yes/no question asked about each query and document pair. The state TypeSafe reads with
batch_size=1is{"query": <query>, "document": <column value>}. Defaults to a generic relevance question. Seebatch_sizefor the different location of the document in batched requests. -
criteria(, default:Mapping [str ,str ]None) –What a "yes" and a "no" mean, as a mapping with the keys
"true"and"false". Domain-specific criteria usually rank better than the generic default. Pass an empty mapping to send no criteria. -
return_score(, default:str "relevance") –Options are "relevance" or "all". If "all", keeps the vector and FTS scores alongside the relevance score.
-
api_key(, default:str None) –The API key to use. If None, the TypeSafe SDK reads the
TYPESAFE_API_KEYenvironment variable. -
max_concurrency(, default:int 8) –The maximum number of TypeSafe requests in flight for one rerank call.
-
batch_size(, default:int 1) –Positive integer limiting non-null documents per request. The default preserves the original request format. With a value greater than one, state is
{"query": <query>}and each question's instructions are{"question": <instructions>, "document": <column value>}. Instructions and criteria are preserved verbatim. Custom prompts that explicitly reference the document in request state (for example,state.document) must be adapted before opting into batching. The TypeSafe SDK handles retries in both modes.
Examples:
VoyageAIReranker
¶
Bases:
Reranks the results using the VoyageAI Rerank API. https://docs.voyageai.com/docs/reranker
Parameters:
-
model_name(, default:str "rerank-english-v2.0") –The name of the cross encoder model to use. Available voyageai models are: - rerank-2.5 - rerank-2.5-lite - rerank-2 - rerank-2-lite
-
column(, default:str "text") –The name of the column to use as input to the cross encoder model.
-
top_n(, default:int None) –The number of results to return. If None, will return all results.
-
return_score(, default:str "relevance") –options are "relevance" or "all". Only "relevance" is supported for now.
-
api_key(, default:str None) –The API key to use. If None, will use the VOYAGE_API_KEY environment variable.
-
truncation(, default:Optional [bool ]None) –
WatsonxReranker
¶
Bases:
Reranks the results using the IBM watsonx.ai Rerank API.
Uses the ibm_watsonx_ai SDK (Rerank.generate) under the hood.
API Docs: https://cloud.ibm.com/docs/apis/watsonx-ai#text-rerank
Supported rerank models: https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx#rerank
Parameters:
-
model_name(, default:str "cross-encoder/ms-marco-minilm-l-12-v2") –The ID of the rerank model to use.
-
column(, default:str "text") –The name of the column to use as input to the reranker.
-
top_n(, default:int None) –Return only the top-n results. If
None, all results are returned. -
return_score(, default:str "relevance") –Options are
"relevance"or"all". -
api_key(, default:str None) –IBM Cloud API key. Falls back to the
WATSONX_API_KEYenvironment variable when not provided. -
project_id(, default:str None) –watsonx.ai project ID. Explicit value takes precedence over the
WATSONX_PROJECT_IDenvironment variable. Mutually exclusive withspace_id— exactly one must be supplied. -
space_id(, default:str None) –watsonx.ai deployment space ID. Explicit value takes precedence over the
WATSONX_SPACE_IDenvironment variable. Mutually exclusive withproject_id— exactly one must be supplied. -
url(, default:str None) –watsonx.ai service URL. Defaults to
"https://us-south.ml.cloud.ibm.com". -
truncate_input_tokens(, default:int None) –Truncate each input to this many tokens before scoring. Passed directly to the
parametersdict ofRerank.generate.
Connections (Asynchronous)¶
lancedb.connect_catalog_async
¶
connect_catalog_async(endpoint: str , *, api_key: Optional [str ] = None, client_config: Optional [Union [ClientConfig , dict [str , Any ]]] = None, sql_host_override: Optional [str ] = None, read_consistency_interval: Optional [timedelta ] = None, oauth_config: Optional [OAuthConfig ] = None) -> AsyncCatalog
Connect to an HTTP(S) server's root catalog.
Root requests omit database-selection headers. API key, client configuration,
OAuth, and table read consistency settings are inherited by opened databases.
Database names containing slashes remain single logical names.
Set sql_host_override to the SQL service endpoint to execute SQL through
returned connections when the catalog endpoint uses HTTPS.
lancedb.catalog.AsyncCatalog
¶
An asynchronous remote catalog returned by connect_catalog_async.
Create/connect return ordinary AsyncConnection instances. Drop uses restricted behavior: remove the database's tables first.
create_database
¶
Create a database, or open an existing one when exist_ok=True.
connect_database
¶
Connect to an existing database by its logical name.
list_databases
¶
list_databases(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncDatabaseNames
Iterate lazily over all database names using async for.
page_token resumes from a saved token; None starts at the beginning.
page_limit sets the maximum names per REST response (not the total);
None uses the server default. Request errors are raised during iteration.
Examples:
drop_database
¶
Drop a database. A nonempty database is an error unless cascade is set,
which drops everything in it first; the name stays taken until the server
finishes cleaning up their storage.
lancedb.catalog.AsyncDatabaseNames
¶
Bases:
An async iterator of database names with pagination state.
Returned by list_databases.
num_page_results
¶
Return the number of names available without another REST request.
page_token
¶
Return the token for the next REST request.
Before the first request, this is the supplied starting token (None
starts at the beginning). After the final page is fetched, it is None,
even if names remain cached. Drain the cache before saving a token to
avoid skipping those names when resuming. A failed request terminates
iteration and leaves its token available for resuming a new iterator.
Connections represent a connection to a LanceDb database and can be used to create, list, or open tables.
lancedb.connect_async
¶
connect_async(uri: URI , *, api_key: Optional [str ] = None, region: str = 'us-east-1', host_override: Optional [str ] = None, sql_host_override: Optional [str ] = None, read_consistency_interval: Optional [timedelta ] = None, client_config: Optional [Union [ClientConfig , Dict [str , Any ]]] = None, storage_options: Optional [Dict [str , str ]] = None, session: Optional [Session ] = None, manifest_enabled: bool = False, namespace_client_properties: Optional [Dict [str , str ]] = None, oauth_config=None) -> AsyncConnection
Connect to a LanceDB database.
Parameters:
-
uri() –URI The uri of the database.
-
api_key(, default:Optional [str ]None) –If present, connect to LanceDB Enterprise or LanceDB Cloud (legacy). Otherwise, connect to a database on file system or cloud storage. Can be set via environment variable
LANCEDB_API_KEY. -
region(, default:str 'us-east-1') –The region to use for LanceDB Cloud (legacy).
-
host_override(, default:Optional [str ]None) –The URL of the LanceDB Enterprise deployment to connect to.
-
sql_host_override(, default:Optional [str ]None) –The LanceDB Enterprise SQL service endpoint override. The client connects lazily when SQL is first executed and retains that connection.
-
read_consistency_interval(, default:Optional [timedelta ]None) –The interval at which to check for updates to the table from other processes. If None, then consistency is not checked. For performance reasons, this is the default. For strong consistency, set this to zero seconds. Then every read will check for updates from other processes. As a compromise, you can set this to a non-zero timedelta for eventual consistency. If more than that interval has passed since the last check, then the table will be checked for updates. Note: this consistency only applies to read operations. Write operations are always consistent.
Stronger consistency is not free. The smaller the interval, the more often each read pays the cost of checking for updates against object storage, raising per-read latency and cost.
-
client_config(, default:Optional [Union [ClientConfig ,Dict [str ,Any ]]]None) –Configuration options for the HTTP client used by
db://connections. If a dict, then the keys are the attributes of the ClientConfig class. If None, then the default configuration is used. -
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. See available options at https://docs.lancedb.com/storage/
-
session(, default:Optional [Session ]None) –(For LanceDB Embedded only) A session to use for this connection. Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.
-
manifest_enabled(, default:bool False) –When true for LanceDB Embedded connections, use directory namespace manifests as the source of truth for table metadata. Existing directory-listed root tables are migrated into the manifest on access.
-
namespace_client_properties(, default:dict None) –Additional directory namespace client properties to use with
manifest_enabled=True. -
oauth_config(, default:OAuthConfig None) –OAuth configuration for LanceDB Enterprise. This is supported by
connect_asynconly; synchronousconnectuses API key authentication fordb://URIs.
Examples:
>>> import lancedb
>>> azure_storage_options = {
... "account_name": "some-account",
... "account_key": "some-key",
... }
>>> async def doctest_example():
... # For a local directory, provide a path to the database
... db = await lancedb.connect_async("~/.lancedb")
... # For object storage, use a URI prefix
... db = await lancedb.connect_async("s3://my-bucket/lancedb",
... storage_options={
... "aws_access_key_id": "***"})
... # Azure credentials can also be passed directly
... db = await lancedb.connect_async(
... "az://my-container/my-database",
... storage_options=azure_storage_options,
... )
... # For tests and temporary data, use an in-memory database
... db = await lancedb.connect_async("memory://")
... # Connect to LanceDB Enterprise or LanceDB Cloud (legacy)
... db = await lancedb.connect_async("db://my_database", api_key="ldb_...",
... client_config={
... "retry_config": {"retries": 5}})
Returns:
-
conn() –AsyncConnection A connection to a LanceDB database.
lancedb.db.AsyncConnection
¶
Bases:
An active LanceDB connection
To obtain a connection you can use the connect_async function.
This could be a LanceDB Embedded connection (using Lance) or a connection to LanceDB Enterprise.
Embedded connections do not currently hold any open resources but they may do so in the future (for example, for shared cache or connections to catalog services) Enterprise connections represent an open connection to the server. The close method can be used to release any underlying resources eagerly. The connection can also be used as a context manager.
Connections can be shared on multiple threads and are expected to be long lived. Connections can also be used as a context manager, however, in many cases a single connection can be used for the lifetime of the application and so this is often not needed. Closing a connection is optional. If it is not closed then it will be automatically closed when the connection object is deleted.
Examples:
>>> import lancedb
>>> async def doctest_example():
... with await lancedb.connect_async("/tmp/my_dataset") as conn:
... # do something with the connection
... pass
... # conn is closed here
close
¶
Close the connection, releasing any underlying resources.
It is safe to call this method multiple times.
Any attempt to use the connection after it is closed will result in an error.
list_namespaces
¶
list_namespaces(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> AsyncNamespaceListing
List immediate child namespace names in the given namespace.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The parent namespace to list namespaces in. None or empty list represents root namespace.
-
page_token(, default:Optional [str ]None) –The token to use for pagination. If not present, start from the beginning.
-
limit(, default:Optional [int ]None) –The maximum number of results to return.
Returns:
-
–AsyncNamespaceListing Response containing namespace names and optional pagination token
create_namespace
¶
create_namespace(namespace_path: List [str ], mode: Optional [str ] = None, properties: Optional [Dict [str , str ]] = None) -> CreateNamespaceResponse
Create a new namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to create.
-
mode(, default:Optional [str ]None) –Creation mode - "create", "exist_ok", or "overwrite". Case insensitive.
-
properties(, default:Optional [Dict [str ,str ]]None) –Properties to associate with the namespace
Returns:
-
–CreateNamespaceResponse Response containing namespace properties
drop_namespace
¶
drop_namespace(namespace_path: List [str ], mode: Optional [str ] = None, behavior: Optional [str ] = None) -> DropNamespaceResponse
Drop a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to drop.
-
mode(, default:Optional [str ]None) –Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
-
behavior(, default:Optional [str ]None) –Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.
Returns:
-
–DropNamespaceResponse Response containing properties and transaction_id if applicable.
describe_namespace
¶
Describe a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to describe.
Returns:
-
–DescribeNamespaceResponse Response containing the namespace properties.
namespace_exists
¶
Check if a namespace exists.
Parameters:
-
namespace_id() –List [str ]The namespace identifier to check. An empty list represents root.
Returns:
-
–bool True if the namespace exists, False if it is not found. Other errors, including permission errors, are propagated.
Examples:
table_exists
¶
Check if a table exists.
Parameters:
-
table_id() –List [str ]The full identifier, including namespace segments and table name. Must contain at least the table name.
Returns:
-
–bool True if the table exists, False if it is not found. Other errors, including permission errors, are propagated.
Examples:
list_tables
¶
list_tables(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> AsyncTableListing
List tables in this database, one page at a time.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to list tables in. None or empty list represents root namespace.
-
page_token(, default:Optional [str ]None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:Optional [int ]None) –The maximum number of results to return, default 100. Zero returns an empty page without a continuation token.
Returns:
-
–AsyncTableListing Response containing table names and optional page_token for pagination.
table_names
¶
table_names(*, namespace_path: Optional [List [str ]] = None, start_after: Optional [str ] = None, limit: Optional [int ] = None) -> Iterable [str ]
List all tables in this database, in sorted order
.. deprecated::
Use :meth:list_tables instead, which provides proper pagination support.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to list tables in. None or empty list represents root namespace.
-
start_after(, default:Optional [str ]None) –If present, only return names that come lexicographically after the supplied value.
This can be combined with limit to implement pagination by setting this to the last table name from the previous page.
-
limit(, default:Optional [int ]None) –The maximum number of tables to return. None returns all tables; zero returns an empty list.
Returns:
-
Iterable of str–
create_table
¶
create_table(name: str , data: Optional [DATA ] = None, schema: Optional [Union [Schema , LanceModel ]] = None, mode: Optional [Literal ['create', 'overwrite']] = None, exist_ok: Optional [bool ] = None, on_bad_vectors: Optional [str ] = None, fill_value: Optional [float ] = None, storage_options: Optional [Dict [str , str ]] = None, *, namespace_path: Optional [List [str ]] = None, embedding_functions: Optional [List [EmbeddingFunctionConfig ]] = None, location: Optional [str ] = None, namespace_client: Optional [Any ] = None) -> AsyncTable
Create an AsyncTable in the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to create the table in. Empty list represents root namespace.
-
data(, default:Optional [DATA ]None) –User must provide at least one of
dataorschema. Acceptable types are:-
list-of-dict
-
pandas.DataFrame
-
pyarrow.Table or pyarrow.RecordBatch
-
-
schema(, default:Optional [Union [Schema ,LanceModel ]]None) –Acceptable types are:
-
pyarrow.Schema
Plain binary values are cast into blob columns declared with blob. To construct Arrow record batches explicitly, use blob_array.
-
-
mode(, default:Optional [Literal ['create', 'overwrite']]None) –The mode to use when creating the table. Can be "create", "overwrite", or "exist_ok". By default, if the table already exists, an exception is raised. If you want to overwrite the table, use mode="overwrite". To open an existing table without adding data, use mode="exist_ok".
-
exist_ok(, default:Optional [bool ]None) –If a table by the same name already exists, then raise an exception if exist_ok=False. If exist_ok=True, then open the existing table; it will not add the provided data but will validate against any schema that's specified.
-
on_bad_vectors(, default:Optional [str ]None) –How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".
-
fill_value(, default:Optional [float ]None) –The value to use when filling vectors. Only used if on_bad_vectors="fill". NaN or null elements and missing trailing elements are replaced with this value. Valid elements are preserved, and vectors longer than the expected dimension are truncated.
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/
To enable stable row IDs (row IDs remain stable after compaction, update, delete, and merges), set
new_table_enable_stable_row_idsto"true"in storage_options when connecting to the database.
Returns:
-
–AsyncTable A reference to the newly created table.
-
!!! note–The vector index won't be created by default. To create the index, call the
create_indexmethod on the table.
Examples:
Can create with list of tuples or dictionaries:
>>> import lancedb
>>> async def doctest_example():
... db = await lancedb.connect_async("./.lancedb")
... data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
... {"vector": [0.2, 1.8], "lat": 40.1, "long": -74.1}]
... my_table = await db.create_table("my_table", data)
... print(await my_table.query().limit(5).to_arrow())
>>> import asyncio
>>> asyncio.run(doctest_example())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
You can also pass a pandas DataFrame:
>>> import pandas as pd
>>> data = pd.DataFrame({
... "vector": [[1.1, 1.2], [0.2, 1.8]],
... "lat": [45.5, 40.1],
... "long": [-122.7, -74.1]
... })
>>> async def pandas_example():
... db = await lancedb.connect_async("./.lancedb")
... my_table = await db.create_table("table2", data)
... print(await my_table.query().limit(5).to_arrow())
>>> asyncio.run(pandas_example())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
>>> import pyarrow as pa
>>> custom_schema = pa.schema([
... pa.field("vector", pa.list_(pa.float32(), 2)),
... pa.field("lat", pa.float32()),
... pa.field("long", pa.float32())
... ])
>>> async def with_schema():
... db = await lancedb.connect_async("./.lancedb")
... my_table = await db.create_table("table3", data, schema = custom_schema)
... print(await my_table.query().limit(5).to_arrow())
>>> asyncio.run(with_schema())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
lat: float
long: float
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]
It is also possible to create an table from [Iterable[pa.RecordBatch]]:
>>> import pyarrow as pa
>>> def make_batches():
... for i in range(5):
... yield pa.RecordBatch.from_arrays(
... [
... pa.array([[3.1, 4.1], [5.9, 26.5]],
... pa.list_(pa.float32(), 2)),
... pa.array(["foo", "bar"]),
... pa.array([10.0, 20.0]),
... ],
... ["vector", "item", "price"],
... )
>>> schema=pa.schema([
... pa.field("vector", pa.list_(pa.float32(), 2)),
... pa.field("item", pa.utf8()),
... pa.field("price", pa.float32()),
... ])
>>> async def iterable_example():
... db = await lancedb.connect_async("./.lancedb")
... await db.create_table("table4", make_batches(), schema=schema)
>>> asyncio.run(iterable_example())
open_table
¶
open_table(name: str , *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, index_cache_size: Optional [int ] = None, location: Optional [str ] = None, namespace_client: Optional [Any ] = None, managed_versioning: Optional [bool ] = None, branch: Optional [str ] = None, version: Optional [int ] = None) -> AsyncTable
Open a Lance Table in the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to open the table from. None or empty list represents root namespace.
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/
-
index_cache_size(, default:Optional [int ]None) –Deprecated: Use session-level cache configuration instead. Create a Session with custom cache sizes and pass it to lancedb.connect().
Set the size of the index cache, specified as a number of entries
The exact meaning of an "entry" will depend on the type of index: * IVF - there is one entry for each IVF partition * BTREE - there is one entry for the entire index
This cache applies to the entire opened table, across all indices. Setting this value higher will increase performance on larger datasets at the expense of more RAM
-
location(, default:Optional [str ]None) –The explicit location (URI) of the table. If provided, the table will be opened from this location instead of deriving it from the database URI and table name.
-
managed_versioning(, default:Optional [bool ]None) –Whether managed versioning is enabled for this table. If provided, avoids a redundant describe_table call when namespace_client is set.
-
branch(, default:Optional [str ]None) –If provided, open a handle scoped to this branch instead of the default branch. Reads and writes operate in the branch's context.
-
version(, default:Optional [int ]None) –If provided, open the table pinned to this version, producing a read-only handle. Composes with
branch: when both are given, opens that branch at the version; otherwise opensmainat the version. Callcheckout_latestto return to a writable state.
Returns:
-
A LanceTable object representing the table.–
create_materialized_view
¶
create_materialized_view(name: str , source: MaterializedViewSource , *, select: SelectArg = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> AsyncMaterializedView
Define a materialized view named name over the table source.
LanceDB Embedded creation and refresh require a source build with the
experimental-functions Cargo feature. Published packages leave it
disabled; LanceDB Enterprise operations remain available.
See
DBConnection.create_materialized_view.
create_materialized_view_async
¶
create_materialized_view_async(name: str , source: MaterializedViewSource , *, select: SelectArg = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> AsyncJob [None]
Submit materialized-view creation and return its job.
Wait for the returned job before opening or querying the view. In LanceDB
Enterprise, creation uses SQL; job_type is "query" and id
identifies that query on this connection.
open_materialized_view
¶
Open the materialized view named name.
Raises ValueError if the table exists but is not a materialized
view.
list_materialized_views
¶
list_materialized_views(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
The names of the materialized views in this database.
Local listings inspect table schemas one page at a time.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_materialized_view
¶
Drop a materialized view.
In LanceDB Enterprise, drops wait for the SQL command to remove the catalog binding. Physical cleanup continues in the background.
drop_materialized_view_async
¶
drop_materialized_view_async(name: str , *, namespace_path: Optional [List [str ]] = None) -> AsyncJob [None]
Start dropping a materialized view and return its job.
In LanceDB Enterprise, the job has type query and its ID identifies
the SQL query. Await :meth:AsyncJob.wait for catalog removal; physical
cleanup continues in the background.
clone_table
¶
clone_table(target_table_name: str , source_uri: str , *, target_namespace_path: Optional [List [str ]] = None, source_version: Optional [int ] = None, source_tag: Optional [str ] = None, is_shallow: bool = True) -> AsyncTable
Clone a table from a source table.
A shallow clone creates a new table that shares the underlying data files with the source table but has its own independent manifest. This allows both the source and cloned tables to evolve independently while initially sharing the same data, deletion, and index files.
Parameters:
-
target_table_name() –str The name of the target table to create.
-
source_uri() –str The URI of the source table to clone from.
-
target_namespace_path(, default:Optional [List [str ]]None) –The namespace for the target table. None or empty list represents root namespace.
-
source_version(, default:Optional [int ]None) –The version of the source table to clone.
-
source_tag(, default:Optional [str ]None) –The tag of the source table to clone.
-
is_shallow(, default:bool True) –Whether to perform a shallow clone (True) or deep clone (False). Currently only shallow clone is supported.
Returns:
-
An AsyncTable object representing the cloned table.–
rename_table
¶
rename_table(cur_name: str , new_name: str , cur_namespace_path: Optional [List [str ]] = None, new_namespace_path: Optional [List [str ]] = None)
Rename a table in the database.
Parameters:
-
cur_name() –str The current name of the table.
-
new_name() –str The new name of the table.
-
cur_namespace_path(, default:Optional [List [str ]]None) –The namespace of the current table. None or empty list represents root namespace.
-
new_namespace_path(, default:Optional [List [str ]]None) –The namespace to move the table to. If not specified, defaults to the same as cur_namespace.
drop_table
¶
Drop a table from the database.
Parameters:
-
name() –str The name of the table.
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to drop the table from. Empty list represents root namespace.
-
ignore_missing(, default:bool False) –If True, ignore if the table does not exist.
drop_table_async
¶
Start dropping a table and return its cleanup job.
The table may become unavailable before its data files are removed.
Await :meth:AsyncJob.wait to wait for cleanup to finish.
drop_all_tables
¶
Drop all tables from the database.
Lists the tables in the selected namespace and drops them one at a time.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The namespace to drop all tables from. None or empty list represents root namespace.
open_job
¶
Open a server-side job by id. See DBConnection.open_job.
list_gpu_classes
¶
List configured Function GPU classes. See DBConnection.list_gpu_classes.
create_function_async
¶
create_function_async(definition: FunctionDefinition , *, secrets: Optional [Sequence [EnvVarSecret ]] = None, namespace_path: Optional [List [str ]] = None) -> AsyncJob [FunctionVersion ]
Submit a scalar Python function for building and registration.
The server-side job builds the OCI image, then registers the completed
artifact. The returned job has type query and the SQL query UUID as
its ID. Waiting on it returns the immutable Function version.
secrets is a sequence of
EnvVarSecret, each naming a Secret and
the environment variable its value arrives in. namespace_path is
the namespace the Function is registered in; None or an empty list is
the root. LanceDB Embedded connections raise NotImplementedError.
get_function
¶
get_function(name: str , *, version: Optional [Union [int , str ]] = None, namespace_path: Optional [List [str ]] = None) -> FunctionVersion
Open an immutable Function version from the remote catalog.
version may be an integer or string. When omitted, the server
resolves the latest currently visible revision and returns that exact
immutable version.
list_functions
¶
list_functions(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [FunctionVersion ]
List every published immutable Function version in a namespace.
Functions in child namespaces are not included. Results are ordered by
Function name then version. LanceDB Embedded connections raise
NotImplementedError.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_function
¶
Drop one published Function revision, preserving its siblings.
Returns True when the requested revision was removed and False for a
repeated drop or a missing name. A revision never published under an
existing name is an error. Only the last visible revision's removal
releases the name and schedules background object cleanup. When the
remote server reports a warning, it is emitted as UserWarning
before this method returns, subject to the standard warnings
filters. A warnings-as-errors setting can therefore raise after the
catalog removal; it does not roll back the drop. Older servers that
omit the warning field remain silent.
drop_function_async
¶
drop_function_async(name: str , *, version: str , namespace_path: Optional [List [str ]] = None) -> Tuple [bool , AsyncJob ]
Drop one Function revision and return its SQL query job.
The boolean reports removal of the requested revision from the catalog;
sibling revisions and their grants remain available. The job has type
query and its ID is the SQL query UUID. Repeated waits confirm that
revision's removal. Only the last visible revision's drop releases the
name and schedules background object cleanup.
create_storage_integration
¶
create_storage_integration(name: str , storage_prefix: str , *, aws_role_arn: Optional [str ] = None, aws_access_key_id_secret: Optional [Union [str , SecretReference ]] = None, aws_secret_access_key_secret: Optional [Union [str , SecretReference ]] = None, aws_region: Optional [str ] = None, storage_provider: Literal ['S3', 'AZURE', 'GCP', 'HUGGINGFACE'] = 'S3', azure_storage_account: Optional [str ] = None, azure_tenant_id_secret: Optional [Union [str , SecretReference ]] = None, azure_client_id_secret: Optional [Union [str , SecretReference ]] = None, azure_client_secret_secret: Optional [Union [str , SecretReference ]] = None, gcp_credentials_json_secret: Optional [Union [str , SecretReference ]] = None, huggingface_token_secret: Optional [Union [str , SecretReference ]] = None, namespace_path: Optional [List [str ]] = None) -> StorageIntegrationInfo
Register cloud storage access using database-owned credential Secrets.
namespace_path follows Secret naming rules. A SecretReference can name
another database; a string names a root Secret in this database.
For Azure, set storage_provider="AZURE", azure_storage_account,
and Secret references for tenant ID, client ID and client secret.
For GCP, set storage_provider="GCP" and gcp_credentials_json_secret
to a Secret containing the application credentials JSON file content.
For Hugging Face, set storage_provider="HUGGINGFACE" and
optionally set huggingface_token_secret to a Secret containing the
token; omitting it selects anonymous access at the default rate limits.
Registration returns setup instructions without verifying storage access.
describe_storage_integration
¶
describe_storage_integration(name: str , *, namespace_path: Optional [List [str ]] = None) -> StorageIntegrationInfo
Return configuration and Secret references without credential values.
list_storage_integrations
¶
list_storage_integrations(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
Lazily list one namespace of this database's storage integrations.
drop_storage_integration
¶
Drop a storage integration with ownership permission.
create_secret
¶
Create a named Secret in this database.
Fails if the name is taken, so a create never silently becomes a rotation. Nothing reads the value back.
alter_secret
¶
Replace the credential behind an existing Secret.
Fails if it does not exist. Bound Functions use the new value from their next job, with no new Function version.
list_secrets
¶
list_secrets(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
The names of every Secret in this database. Names only.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_secret
¶
Drop a Secret. Bound Functions fail at their next job.
describe_secret
¶
What this database records about a Secret. Never the value.
create_view
¶
create_view(name: str , query: str , *, namespace_path: Optional [List [str ]] = None) -> ViewDescription
Create a view: a named query the database plans on every read.
describe_view
¶
What this database records about a view: query and schema.
drop_view
¶
Drop a view and wait for catalog removal.
The tables it reads are untouched. Use :meth:drop_view_async to get
its SQL query job. Completion confirms catalog removal; physical cleanup
continues in the background.
drop_view_async
¶
Drop a view and return its SQL query job.
The job has type query and its ID is the SQL query UUID. Await
:meth:AsyncJob.wait to confirm catalog removal. Physical cleanup
continues in the background.
list_views
¶
list_views(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
The names of the views in one namespace.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
create_property_graph
¶
create_property_graph(name: str , nodes: Sequence [NodeTable ], edges: Sequence [EdgeTable ] = (), *, namespace_path: Optional [List [str ]] = None) -> AsyncPropertyGraph
Create a property graph over tables in its namespace.
open_property_graph
¶
Open the property graph named name, describing it to check that
it exists.
drop_property_graph
¶
Drop a property graph and wait for its definition to be deleted.
The tables it reads are untouched.
list_property_graphs
¶
list_property_graphs(*, namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
The names of the property graphs in one namespace.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
list_jobs
¶
list_jobs(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [JobInfo ]
List server-side jobs across the database's tables.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
cancel_job
¶
Request cancellation of a server-side job by id.
Returns True if the server accepted the cancellation, False if no such job exists. Cancelling an already-terminal job is a no-op success.
pause_job
¶
Pause a server-side job by id.
The job's workers drain and it stays parked until resumed. Returns "pausing", "already_paused", or "committing" -- a job finalizing its results cannot be parked; retry shortly.
resume_job
¶
Resume a paused server-side job by id.
Its workers pick their work back up from checkpoints. Returns "resumed", "still_pausing" -- retry shortly -- or "not_paused".
execute_query
¶
execute_query(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> AsyncRecordBatchReader
Execute SQL and return an asynchronous Arrow reader.
This submits through :meth:execute_query_async and waits until the
initial result stream is readable. It does not wait for the full query
to finish. A statement that ran as a server-side SQL job exposes the
durable Job Registry id in the reader schema metadata under
x-lancedb-sql-job-id. Use :meth:execute_query_async when the
query handle itself is needed.
execute_query_async
¶
execute_query_async(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> AsyncQuery
Start executing SQL and return its query handle.
The database from connect_async is used for unqualified database
references. The namespace defaults to ["public"]. parameters
binds values to $1 / $name placeholders; see
DBConnection.execute_query_async.
LanceDB Embedded connections raise NotImplementedError.
describe_query
¶
Describe a submitted SQL query by its UUID or string ID.
For a handle with job_type == "query", pass its id on the
connection that submitted it. Registry job IDs are not query IDs.
namespace_client
¶
Get the equivalent namespace client for this connection.
For native storage connections, this returns a DirectoryNamespace pointing to the same root with the same storage options.
For namespace connections, this returns the backing namespace client.
For LanceDB Enterprise connections, this returns a RestNamespace with the same URI and authentication headers.
Returns:
-
–LanceNamespace The namespace client for this connection.
lancedb.listing.AsyncListing
¶
Bases: ,
A lazy async iterator with inspectable pagination state.
Returned by connection resource-listing methods. For example::
async for name in connection.list_views(page_limit=20):
print(name)
Requests happen during iteration; request errors terminate the iterator.
num_page_results
¶
Return the number of cached items available without another request.
page_token
¶
Return the token for the next request.
Initially this is the supplied starting token (None starts at the
beginning). After fetching the final page it is None, even if items
remain cached. Drain the cache before saving a token to avoid skipping
items when resuming. A failed request terminates iteration and retains
its token for retrying with a new iterator.
async for name in db.list_tables() loads pages as needed. To inspect the
first page, use page = await db.list_tables() and then page.tables. Reading
the page list does not consume the iterator. page.page_token is the token for
the next request; drain cached results before using it to resume in a new
listing. The same behavior applies to namespaces.
lancedb.listing.AsyncTableListing
¶
Bases:
Awaitable table iterator; awaiting loads one page.
For example::
page = await db.list_tables(limit=20)
print(page.tables, page.page_token)
async for name in db.list_tables(limit=20):
print(name)
lancedb.listing.AsyncNamespaceListing
¶
Bases:
Awaitable namespace iterator; awaiting loads one page.
For example::
page = await db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)
Namespaces (Asynchronous)¶
lancedb.connect_namespace_async
¶
connect_namespace_async(namespace_client_impl: str , namespace_client_properties: Dict [str , str ], *, read_consistency_interval: Optional [timedelta ] = None, storage_options: Optional [Dict [str , str ]] = None, session: Optional [Session ] = None, namespace_client_pushdown_operations: Optional [List [str ]] = None) -> AsyncLanceNamespaceDBConnection
Connect to a LanceDB database through a namespace (returns async connection).
This function is synchronous but returns an AsyncLanceNamespaceDBConnection that provides async methods for all database operations.
Parameters:
-
namespace_client_impl() –str The namespace client implementation to use. For examples:
- "dir" for DirectoryNamespace
- "rest" for REST-based namespace
- Full module path for custom implementations
-
namespace_client_properties() –Dict [str ,str ]Configuration properties for the namespace client implementation. Different namespace implementations have different config properties. For example, use DirectoryNamespace with {"root": "/path/to/directory"}
-
read_consistency_interval(, default:Optional [timedelta ]None) –The interval at which to check for updates to the table from other processes. If None, then consistency is not checked.
-
storage_options(, default:Optional [Dict [str ,str ]]None) –Additional options for the storage backend
-
session(, default:Optional [Session ]None) –A session to use for this connection
-
namespace_client_pushdown_operations(, default:Optional [List [str ]]None) –List of namespace operations to push down to the namespace server. Supported values:
- "QueryTable": Execute queries on the namespace server via namespace.query_table() instead of locally.
- "CreateTable": Execute table creation on the namespace server via namespace.create_table() instead of using declare_table + local write.
Default is None (no pushdown, all operations run locally).
Returns:
-
–AsyncLanceNamespaceDBConnection An async namespace-based connection to LanceDB
Examples:
>>> import lancedb
>>> # This function is sync, but returns an async connection
>>> db = lancedb.connect_namespace_async("dir", {"root": "/path/to/db"})
>>> # Use async methods on the connection
>>> async def use_db():
... tables = await db.table_names()
... table = await db.create_table("my_table", schema=schema)
lancedb.namespace.AsyncLanceNamespaceDBConnection
¶
An async LanceDB connection that uses a namespace for table management.
This connection delegates table URI resolution to a lance_namespace instance, while providing async methods for all operations.
table_names
¶
table_names(page_token: Optional [str ] = None, limit: Optional [int ] = None, *, namespace_path: Optional [List [str ]] = None) -> Iterable [str ]
List table names in the namespace.
If limit is None, return all table names; zero returns an empty list.
.. deprecated::
Use :meth:list_tables instead, which provides proper pagination support.
create_table
¶
create_table(name: str , data: Optional [DATA ] = None, schema: Optional [Union [Schema , LanceModel ]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional [List [EmbeddingFunctionConfig ]] = None, *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, data_storage_version: Optional [str ] = None, enable_v2_manifest_paths: Optional [bool ] = None) -> AsyncTable
Create a new table in the namespace.
open_table
¶
open_table(name: str , *, namespace_path: Optional [List [str ]] = None, storage_options: Optional [Dict [str , str ]] = None, index_cache_size: Optional [int ] = None, branch: Optional [str ] = None, version: Optional [int ] = None) -> AsyncTable
Open an existing table from the namespace.
create_materialized_view
¶
create_materialized_view(name: str , source: MaterializedViewSource , *, select: 'SelectArg' = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> 'AsyncMaterializedView'
Define a materialized view over a table in the root namespace.
create_materialized_view_async
¶
create_materialized_view_async(name: str , source: MaterializedViewSource , *, select: 'SelectArg' = None, where: Optional [str ] = None, limit: Optional [int ] = None, with_no_data: bool = False, auto_refresh_enabled: Optional [bool ] = None, auto_refresh_interval_seconds: Optional [float ] = None) -> AsyncJob [None]
Submit materialized-view creation and return its job.
open_materialized_view
¶
Open the materialized view named name.
list_materialized_views
¶
list_materialized_views(*, page_token: Optional [str ] = None, page_limit: Optional [int ] = None) -> AsyncListing [str ]
The names of the materialized views in the root namespace.
Returns a lazy iterator. page_limit limits each request, not the total;
page_token resumes from a saved token. Requests and errors occur during
iteration. See pagination state.
drop_materialized_view
¶
Drop a materialized view from the namespace.
drop_materialized_view_async
¶
drop_materialized_view_async(name: str , namespace_path: Optional [List [str ]] = None) -> AsyncJob [None]
Start dropping a materialized view and return its cleanup job.
drop_table
¶
Drop a table from the namespace.
drop_table_async
¶
Start dropping a table and return its cleanup job.
rename_table
¶
rename_table(cur_name: str , new_name: str , cur_namespace_path: Optional [List [str ]] = None, new_namespace_path: Optional [List [str ]] = None)
Rename a table in the namespace.
drop_all_tables
¶
Drop all tables in the namespace.
list_namespaces
¶
list_namespaces(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> AsyncNamespaceListing
List child namespaces under the given namespace.
Parameters:
-
namespace_path(, default:Optional [List [str ]]None) –The parent namespace path to list children from. If None, lists root-level namespaces.
-
page_token(, default:Optional [str ]None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:int None) –Maximum number of namespaces to return.
Returns:
-
–NamespaceListing Response containing namespace names and optional page_token for pagination.
create_namespace
¶
create_namespace(namespace_path: List [str ], mode: Optional [str ] = None, properties: Optional [Dict [str , str ]] = None) -> CreateNamespaceResponse
Create a new namespace.
Parameters:
-
namespace_path() –List [str ]The namespace path to create.
-
mode(, default:str None) –Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.
-
properties(, default:Dict [str ,str ]None) –Properties to set on the namespace.
Returns:
-
–CreateNamespaceResponse Response containing the properties of the created namespace.
drop_namespace
¶
drop_namespace(namespace_path: List [str ], mode: Optional [str ] = None, behavior: Optional [str ] = None) -> DropNamespaceResponse
Drop a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace path to drop.
-
mode(, default:str None) –Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
-
behavior(, default:str None) –Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.
Returns:
-
–DropNamespaceResponse Response containing properties and transaction_id if applicable.
describe_namespace
¶
Describe a namespace.
Parameters:
-
namespace_path() –List [str ]The namespace identifier to describe.
Returns:
-
–DescribeNamespaceResponse Response containing the namespace properties.
namespace_exists
¶
Check if a namespace exists.
Parameters:
-
namespace_id() –List [str ]The namespace identifier to check.
Returns:
-
–bool True if the namespace exists, False otherwise.
table_exists
¶
Check if a table exists.
Parameters:
-
table_id() –List [str ]The table identifier to check (full path including namespace segments and table name).
Returns:
-
–bool True if the table exists, False otherwise.
list_tables
¶
list_tables(namespace_path: Optional [List [str ]] = None, page_token: Optional [str ] = None, limit: Optional [int ] = None) -> AsyncTableListing
List all tables in this database with pagination support.
Parameters:
-
namespace_path(, default:List [str ]None) –The namespace to list tables in. None or empty list represents root namespace.
-
page_token(, default:str None) –Token for pagination. Use the token from a previous response to get the next page of results.
-
limit(, default:int None) –The maximum number of results to return.
Returns:
-
–TableListing Response containing table names and optional page_token for pagination.
execute_query
¶
execute_query(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> AsyncRecordBatchReader
Execute SQL when supported by the underlying connection.
execute_query_async
¶
execute_query_async(query: str , *, default_namespace_path: Optional [List [str ]] = None, parameters: Optional [QueryParameters ] = None) -> AsyncQuery
Start executing SQL when supported by the underlying connection.
Namespace-backed LanceDB Embedded connections do not support SQL.
describe_query
¶
Describe a submitted SQL query when supported.
namespace_client
¶
Get the namespace client for this connection.
For namespace connections, this returns the backing namespace client that was provided during construction.
Returns:
-
–LanceNamespace The namespace client for this connection.
Tables (Asynchronous)¶
Table hold your actual data as a collection of records / rows.
lancedb.table.AsyncTable
¶
An AsyncTable is a collection of Records in a LanceDB Database.
An AsyncTable can be obtained from the AsyncConnection.create_table and AsyncConnection.open_table methods.
An AsyncTable object is expected to be long lived and reused for multiple operations. AsyncTable objects will cache a certain amount of index data in memory. This cache will be freed when the Table is garbage collected. To eagerly free the cache you can call the close method. Once the AsyncTable is closed, it cannot be used for any further operations.
An AsyncTable can also be used as a context manager, and will automatically close when the context is exited. Closing a table is optional. If you do not close the table, it will be closed when the AsyncTable object is garbage collected.
Examples:
Create using AsyncConnection.create_table (more examples in that method's documentation).
>>> import lancedb
>>> async def create_a_table():
... db = await lancedb.connect_async("./.lancedb")
... data = [{"vector": [1.1, 1.2], "b": 2}]
... table = await db.create_table("my_table", data=data)
... print(await table.query().limit(5).to_arrow())
>>> import asyncio
>>> asyncio.run(create_a_table())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
child 0, item: float
b: int64
----
vector: [[[1.1,1.2]]]
b: [[2]]
Can append new data with AsyncTable.add().
>>> async def add_to_table():
... db = await lancedb.connect_async("./.lancedb")
... table = await db.open_table("my_table")
... await table.add([{"vector": [0.5, 1.3], "b": 4}])
>>> asyncio.run(add_to_table())
Can query the table with AsyncTable.vector_search.
>>> async def search_table_for_vector():
... db = await lancedb.connect_async("./.lancedb")
... table = await db.open_table("my_table")
... results = (
... await table.vector_search([0.4, 0.4]).select(["b", "vector"]).to_pandas()
... )
... print(results)
>>> asyncio.run(search_table_for_vector())
b vector _distance
0 4 [0.5, 1.3] 0.82
1 2 [1.1, 1.2] 1.13
Search queries are much faster when an index is created. See AsyncTable.create_index.
tags
¶
Tag management for the dataset.
Similar to Git, tags are a way to add metadata to a specific version of the dataset.
.. warning::
Tagged versions are exempted from the
:py:meth:`optimize(cleanup_older_than)` process.
To remove a version that has been tagged, you must first
:py:meth:`~Tags.delete` the associated tag.
branches
¶
Branch management for the table.
Branches are isolated, writable lines of history forked from another
branch (or version). Writes on a branch do not affect main.
close
¶
Close the table and free any resources associated with it.
It is safe to call this method multiple times.
Any attempt to use the table after it has been closed will raise an error.
set_unenforced_primary_key
¶
Set the unenforced primary key for this table to the given ordered list of columns.
"Unenforced" means LanceDB does not check uniqueness on writes; the
columns are recorded in the schema as the primary key so that
features such as merge_insert can use them. Calling this again
replaces any previously-set primary key.
Parameters:
-
columns() –str orIterable [str ]Either a single column name (single-column key) or an ordered iterable of column names (composite key). Each column dtype must be one of: int32, int64, utf8, large_utf8, binary, large_binary, fixed_size_binary.
enable_rte
¶
Install an RTEWriteSpec on this table.
The spec selects Lance's MemWAL LSM-style write path for future
merge_insert calls. RTEWriteSpec chooses one of three sharding
strategies:
RTEWriteSpec.bucket(column, num_buckets)— hash-bucket writes by the single-column unenforced primary key.RTEWriteSpec.identity(column)— shard by the raw value of a scalar column.RTEWriteSpec.unsharded()— route every write to a single shard.
All variants require the table to have an unenforced primary key set
via [set_unenforced_primary_key]; bucket sharding additionally
requires it to be the single column being bucketed.
By default the RTE maintains every index on the table, re-read as the
table changes, so an index created later is maintained too; one of a
kind the RTE cannot mirror is skipped. Name the set with
with_maintained_indexes to pin it instead: each name must already
exist and be maintainable (a still-building index is rejected, not
omitted); [] maintains none.
Parameters:
-
spec() –RTEWriteSpec The sharding spec to install.
Examples:
disable_rte
¶
Remove the RTEWriteSpec from this table.
Reverts to the standard merge_insert write path. Errors if no spec
is currently set, or if the RTE still holds rows that have not been
merged into the base table, since removing it would drop them.
get_rte_spec
¶
Read the RTEWriteSpec currently installed on this table.
Returns None when the RTE write path is not enabled at all — no
spec has been set, or one was removed with disable_rte.
That is a different answer from a spec whose maintained_indexes
is None, which is an installed spec selecting indexes
automatically.
The spec read back is the one that was installed, selection
included: None maintains every supported index the table has now
or gains later, [] maintains none, and a non-empty list maintains
exactly those. All three round-trip.
get_table_overrides
¶
Read Enterprise table overrides for this table.
Table overrides configure Enterprise-managed background behavior such as automatic compaction, cleanup, and reindex execution. This method is supported only on remote Enterprise tables.
update_table_overrides
¶
Merge Enterprise table overrides into the current table overrides.
Only fields supplied in overrides are changed; unrelated overrides
from the current read snapshot are preserved.
Examples:
reset_table_overrides
¶
Clear Enterprise table overrides for this table.
checkpoint_rte
¶
Converge this table's RTE write path into its base table.
One flush, freezing every memtable into an SSTable, then compaction triggers
until every SSTable that existed at that moment has reached base.
The loop runs client-side, reading progress from get_rte_stats.
Best-effort: SSTables created while it runs are deliberately not waited on, which is what lets it terminate on a table taking writes. Idempotent and safe on a cadence.
There is no deadline, and the caller owns that. It returns when the
target SSTables are gone, raises on a terminal server fault, and
otherwise waits however long the server takes. A slow table and a
stuck one are the same picture from the client: the compactor pool is
shared across every table on the node, so a checkpoint queued behind
unrelated work looks exactly like one that is merging. Wrap this in
asyncio.wait_for for a wall-clock bound; abandoning it partway
costs nothing.
flush_rte
¶
Freeze every table shard's active memtable into an SSTable.
Does not touch the base table — compacting SSTables into base is
compact_rte. On a node that has not claimed this table, this claims
it and replays its WAL log first.
compact_rte
¶
Trigger a background SSTable compaction pass per table shard.
Returns once the passes are dispatched, not once they finish: watch
get_rte_stats for progress, or use checkpoint_rte to loop
until the current SSTables have reached base.
get_rte_stats
¶
Read live per-table-shard RTE state.
Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my fresh-tier vector search brute-force". Mutates no table state, though on a node that has not claimed this table it claims it, exactly as a read would.
Returns None only when the RTE write path is not enabled.
Parameters:
-
include_sstable_rows(, default:bool False) –Report a row count per SSTable. Off by default: each count opens an uncached Lance dataset, and
checkpoint_rtepolls this needing only generation numbers.
close_rte_writers
¶
Drain and close any cached RTE shard writers for this table.
When an RTE write spec is installed, merge_insert opens RTE shard
writers and caches them for reuse across calls. This closes them,
flushing pending data; writers reopen lazily on the next
merge_insert. It is a no-op when no writers are cached.
schema
¶
The Arrow Schema of this Table. Blob v2 fields retain their Arrow extension type.
embedding_functions
¶
Get the embedding functions for the table
Returns:
-
funcs() –Dict [str ,EmbeddingFunctionConfig ]A mapping of the vector column to the embedding function or empty dict if not configured.
count_rows
¶
Count the number of rows in the table.
Parameters:
-
filter(, default:Optional [str ]None) –A SQL where clause to filter the rows to count.
head
¶
Return the first n rows of the table.
Parameters:
-
n–The number of rows to return.
query
¶
Returns an AsyncQuery that can be used to search the table.
Use methods on the returned query to control query behavior. The query can be executed with methods like to_arrow, to_pandas and more.
to_lance
¶
Return the Lance dataset backing a LanceDB Embedded table.
LanceDB Enterprise tables cannot be opened as Lance datasets by the client. Use to_arrow or query to read them through the server.
Parameters:
-
**kwargs–Forwarded to
lance.dataset.
Returns:
-
–LanceDataset The Lance dataset at this table handle's version and branch.
Examples:
to_pandas
¶
Return the table as a pandas DataFrame.
Parameters:
-
blob_mode(, default:BlobMode 'lazy') –Controls how Lance blob columns are returned. LanceDB Enterprise supports "descriptions"; "bytes" and "lazy" are not yet supported.
-
**kwargs–Forwarded to PyArrow / Lance pandas conversion.
Returns:
-
–DataFrame
to_arrow
¶
Return the table as a pyarrow Table.
Parameters:
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.
Returns:
-
–Table
create_index
¶
create_index(column: str , *, replace: Optional [bool ] = None, config: Optional [Union [IvfFlat , IvfPq , IvfRq , HnswPq , HnswSq , HnswFlat , BTree , Bitmap , LabelList , Fm , NGram , ZoneMap , FTS ]] = None, wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None, train: bool = True)
Create an index to speed up queries
Indices can be created on vector columns or scalar columns. Indices on vector columns will speed up vector searches. Indices on scalar columns will speed up filtering (in both vector and non-vector searches)
Parameters:
-
column() –str The column to index.
-
replace(, default:Optional [bool ]None) –Whether to replace the existing index
If this is false, and another index already exists on the same columns and the same name, then an error will be returned. This is true even if that index is out of date.
The default is True
-
config(, default:Optional [Union [IvfFlat ,IvfPq ,IvfRq ,HnswPq ,HnswSq ,HnswFlat ,BTree ,Bitmap ,LabelList ,Fm ,NGram ,ZoneMap ,FTS ]]None) –For advanced configuration you can specify the type of index you would like to create. You can also specify index-specific parameters when creating an index object.
-
wait_timeout(, default:Optional [timedelta ]None) –LanceDB Enterprise only: how long to wait for the index build to finish. If not set, the call returns once the build is scheduled.
-
name(, default:Optional [str ]None) –The name of the index. If not provided, a default name will be generated.
-
train(, default:bool True) –Whether to train the index with existing data. Vector indices always train with existing data.
Notes
On LanceDB Embedded tables the index is built before this method
returns. On LanceDB Enterprise tables it returns once the build is
scheduled. Until the build finishes, a vector index may be missing
from list_indices, a scalar or
FTS index may be listed with 0 indexed rows, and queries won't use the
index. To block until the index is ready, pass wait_timeout, call
wait_for_index, or use
create_index_async and
wait on its job when the server returned a job ID.
wait_timeout waits for the submitted build when the server returns
a job ID. On older servers that don't return one, it falls back to
polling the index by name. wait_for_index always waits by name.
When replacing an existing index, those name-based waits may return
before the new build finishes because the old index can still match.
create_index_async can track a specific build only when the server
returned a job ID; otherwise it returns an already-complete job and
callers must use wait_for_index for that build.
create_index_async
¶
create_index_async(column: str , *, replace: Optional [bool ] = None, config: Optional [Union [IvfFlat , IvfPq , IvfRq , HnswPq , HnswSq , HnswFlat , BTree , Bitmap , LabelList , Fm , NGram , ZoneMap , FTS ]] = None, wait_timeout: Optional [timedelta ] = None, name: Optional [str ] = None, train: bool = True) -> AsyncJob
Create an index, returning a handle to the indexing job.
Takes the same arguments as :meth:create_index. The job may already
be complete when returned; callers must not assume the index exists
until :meth:AsyncJob.wait resolves.
In LanceDB Enterprise, the returned job tracks this submission only if
the server returns a job ID. If it does not, the job is already
complete and does not wait for the background build; use
wait_for_index and account for the replacement caveat described in
create_index.
drop_index
¶
Drop an index from the table.
Parameters:
-
name() –str The name of the index to drop.
Notes
This does not delete the index from disk, it just removes it from the table. To delete the index, run optimize after dropping the index.
Use list_indices to find the names of the indices.
prewarm_index
¶
Prewarm an index in the table.
This is a hint to the database that the index will be accessed in the future and should be loaded into memory if possible. This can reduce cold-start latency for subsequent queries.
This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently.
It is generally wasteful to call this if the index does not fit into the available cache. Not all index types support prewarming; unsupported indices will silently ignore the request.
Parameters:
-
name() –str The name of the index to prewarm
prewarm_data
¶
Prewarm data for the table.
This is a hint to the database that the given columns will be accessed in the future and the database should prefetch the data if possible. Currently only supported in LanceDB Enterprise.
This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently.
This operation has a large upfront cost but can speed up future queries that need to fetch the given columns. Large columns such as embeddings or binary data may not be practical to prewarm. This feature is intended for workloads that issue many queries against the same columns.
Parameters:
-
columns(, default:Optional [List [str ]]None) –The columns to prewarm. If None, all columns are prewarmed.
wait_for_index
¶
Wait for indexing to complete for the given index names. This will poll the table until all the indices are fully indexed, or raise a timeout exception if the timeout is reached.
Parameters:
-
index_names() –Iterable [str ]The name of the indices to poll
-
timeout(, default:timedelta ) –timedelta (seconds=300)Timeout to wait for asynchronous indexing. The default is 5 minutes.
uri
¶
Get the table URI (storage location).
In LanceDB Enterprise, this fetches the location from the server via describe. In LanceDB Embedded, this returns the dataset URI.
Returns:
-
–str The full storage location of the table (e.g., S3/GCS path).
initial_storage_options
¶
Get the initial storage options that were passed in when opening this table.
For dynamically refreshed options (e.g., credential vending), use
:meth:latest_storage_options.
Warning: This is an internal API and the return value is subject to change.
Returns:
-
–Optional [Dict [str ,str ]]The storage options, or None if no storage options were configured.
latest_storage_options
¶
Get the latest storage options, refreshing from provider if configured.
This method is useful for credential vending scenarios where storage options may be refreshed dynamically. If no dynamic provider is configured, this returns the initial static options.
Warning: This is an internal API and the return value is subject to change.
Returns:
-
–Optional [Dict [str ,str ]]The storage options, or None if no storage options were configured.
add
¶
add(data: DATA , *, mode: Optional [Literal ['append', 'overwrite']] = 'append', on_bad_vectors: Optional [OnBadVectorsType ] = None, fill_value: Optional [float ] = None, progress: Optional [Union [bool , Callable , Any ]] = None, write_parallelism: Optional [int ] = None, allow_external_blob_outside_bases: bool = False) -> AddResult
Add more data to the AsyncTable.
Plain binary values are cast into blob columns using the table's schema. To construct Arrow record batches explicitly, use blob_array.
Parameters:
-
data() –DATA The data to insert into the table. Acceptable types are:
-
list-of-dict
-
pandas.DataFrame
-
pyarrow.Table or pyarrow.RecordBatch
-
-
mode(, default:Optional [Literal ['append', 'overwrite']]'append') –The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields.
-
on_bad_vectors(, default:Optional [OnBadVectorsType ]None) –How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".
-
fill_value(, default:Optional [float ]None) –The value to use when filling vectors. Only used if on_bad_vectors="fill".
-
progress(, default:Optional [Union [bool ,Callable ,Any ]]None) –A callback or tqdm-compatible progress bar. See :meth:
Table.addfor details. -
write_parallelism(, default:Optional [int ]None) –Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory.
-
allow_external_blob_outside_bases(, default:bool False) –Allow blob URIs outside registered bases. See :meth:
Table.add. LanceDB Embedded only.
merge_insert
¶
Returns a LanceMergeInsertBuilder
that can be used to create a "merge insert" operation
This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like "insert if not exists", "update or insert (i.e. upsert)", or even replace a portion of existing data with new data (e.g. replace all data where month="january")
The merge insert operation works by combining new data from a source table with existing data in a target table by using a join. There are three categories of records.
"Matched" records are records that exist in both the source table and the target table. "Not matched" records exist only in the source table (e.g. these are new data) "Not matched by source" records exist only in the target table (this is old data)
The builder returned by this method can be used to customize what should happen for each category of data.
Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values.
Parameters:
-
on() –Union [str ,Iterable [str ]]A column (or columns) to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column. Passing several columns matches on the composite key: a source row updates a target row only when it agrees on every one of them.
Examples:
>>> import lancedb
>>> data = pa.table({"a": [2, 1, 3], "b": ["a", "b", "c"]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> new_data = pa.table({"a": [2, 3, 4], "b": ["x", "y", "z"]})
>>> # Perform a "upsert" operation
>>> res = table.merge_insert("a") \
... .when_matched_update_all() \
... .when_not_matched_insert_all() \
... .execute(new_data)
>>> res
MergeResult(version=2, num_updated_rows=2, num_inserted_rows=1, num_deleted_rows=0, num_attempts=1, num_rows=3)
>>> # The order of new rows is non-deterministic since we use
>>> # a hash-join as part of this operation and so we sort here
>>> table.to_arrow().sort_by("a").to_pandas()
a b
0 1 b
1 2 x
2 3 y
3 4 z
search
¶
search(query: Optional [Union [VEC , str , 'PIL.Image.Image', Tuple , FullTextQuery ]] = None, vector_column_name: Optional [str ] = None, query_type: QueryType = 'auto', ordering_field_name: Optional [str ] = None, fts_columns: Optional [Union [str , List [str ]]] = None) -> Union [AsyncQuery , AsyncHybridQuery , AsyncFTSQuery , AsyncVectorQuery ]
Create a search query to find the nearest neighbors of the given query vector. We currently support vector search and full-text search.
All query options are defined in AsyncQuery.
Parameters:
-
query(, default:Optional [Union [VEC ,str , 'PIL.Image.Image',Tuple ,FullTextQuery ]]None) –The targeted vector to search for.
-
default None. Acceptable types are: list, np.ndarray, PIL.Image.Image
-
If None then a plain AsyncQuery is returned, equivalent to calling query. The select/where/limit clauses are applied to filter the table.
-
-
vector_column_name(, default:Optional [str ]None) –The name of the vector column to search.
The vector column needs to be a pyarrow fixed size list type
-
If not specified then the vector column is inferred from the table schema
-
If the table has multiple vector columns then the vector_column_name needs to be specified. Otherwise, an error is raised.
-
-
query_type(, default:QueryType 'auto') –default "auto". Acceptable types are: "vector", "fts", "hybrid", or "auto"
-
If "auto" then the query type is inferred from the query;
-
If
queryis a list/np.ndarray then the query type is "vector"; -
If
queryis a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found.
-
-
If
queryis a string, then the query type is "vector" if the table has embedding functions else the query type is "fts"
-
Returns:
-
AsyncQuery, AsyncHybridQuery, AsyncFTSQuery, or AsyncVectorQuery–A query builder object representing the query.
vector_search
¶
Search the table with a given query vector.
This is a convenience method for preparing a vector query and
is the same thing as calling nearestTo on the builder returned
by query. Seer nearest_to for more
details.
delete
¶
Delete rows from the table.
This can be used to delete a single row, many rows, all rows, or sometimes no rows (if your predicate matches nothing).
Parameters:
-
where() –Union [str ,Expr ]The filter condition. Can be a SQL string or a type-safe :class:
~lancedb.expr.Exprbuilt with :func:~lancedb.expr.coland :func:~lancedb.expr.lit.The filter must not be empty, or it will error.
Examples:
>>> import lancedb
>>> data = [
... {"x": 1, "vector": [1.0, 2]},
... {"x": 2, "vector": [3.0, 4]},
... {"x": 3, "vector": [5.0, 6]}
... ]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 2 [3.0, 4.0]
2 3 [5.0, 6.0]
>>> table.delete("x = 2")
DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas()
x vector
0 1 [1.0, 2.0]
1 3 [5.0, 6.0]
If you have a list of values to delete, you can combine them into a
stringified list and use the IN operator:
update
¶
update(updates: Optional [Dict [str , Any ]] = None, *, where: Optional [Union [str , Expr ]] = None, updates_sql: Optional [Dict [str , str ]] = None) -> UpdateResult
This can be used to update zero to all rows in the table.
If a filter is provided with where then only rows matching the
filter will be updated. Otherwise all rows will be updated.
Embeddings are not computed by this method. Updating an embedding source
column requires providing all associated vector columns in the same call;
otherwise a ValueError is raised. Use merge_insert to recompute embeddings
automatically.
Parameters:
-
updates(, default:Optional [Dict [str ,Any ]]None) –The updates to apply. The keys should be the name of the column to update. The values should be the new values to assign. This is required unless updates_sql is supplied.
-
where(, default:Optional [Union [str ,Expr ]]None) – -
updates_sql(, default:Optional [Dict [str ,str ]]None) –The updates to apply, expressed as SQL expression strings. The keys should be column names. The values should be SQL expressions. These can be SQL literals (e.g. "7" or "'foo'") or they can be expressions based on the previous value of the row (e.g. "x + 1" to increment the x column by 1)
Returns:
-
–UpdateResult An object containing: - rows_updated: The number of rows that were updated - version: The new version number of the table after the update
Examples:
>>> import asyncio
>>> import lancedb
>>> from lancedb.expr import col
>>> import pandas as pd
>>> async def demo_update():
... data = pd.DataFrame({"x": [1, 2], "vector": [[1, 2], [3, 4]]})
... db = await lancedb.connect_async("./.lancedb")
... table = await db.create_table("my_table", data)
... # x is [1, 2], vector is [[1, 2], [3, 4]]
... await table.update({"vector": [10, 10]}, where=col("x") == 2)
... # x is [1, 2], vector is [[1, 2], [10, 10]]
... await table.update(updates_sql={"x": "x + 1"})
... # x is [2, 3], vector is [[1, 2], [10, 10]]
>>> asyncio.run(demo_update())
add_columns
¶
add_columns(transforms: Mapping [str , str | FunctionApplication ] | FunctionApplication | Field | List [Field ] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None = None, *, computed: dict [str , str ] | None = None, auto_refresh_enabled: bool | None = None, auto_refresh_interval_seconds: int | None = None, auto_refresh_resource_budget: Optional [Dict [str , int ]] = None) -> AddColumnsResult
Add new columns with defined values.
Parameters:
-
transforms(, default:Mapping [str ,str |FunctionApplication ] |FunctionApplication |Field |List [Field ] |Schema |DATA |RecordBatch |RecordBatchReader |DataFrame | NoneNone) –pa.Field, List[pa.Field], pa.Schema, pandas.DataFrame, polars.DataFrame, pa.Table, pa.RecordBatch, pa.RecordBatchReader A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, you can pass a pyarrow field or schema to add new columns with NULLs.
A mapping with one
FunctionApplicationvalue keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come fromrename(columns=...). Function columns are supported only in LanceDB Enterprise.On local tables, a pandas or Polars DataFrame, PyArrow Table, RecordBatch, or RecordBatchReader supplies precomputed new-column values. Supply exactly one row per current live table row, in storage scan order; pandas indexes are ignored. The input contains only new columns. Readers are consumed once, and their batch boundaries need not match table fragments. Keep the table unchanged between computing these values and calling add_columns. Data inputs cannot be combined with computed columns. Remote tables do not support data inputs.
-
computed(, default:dict [str ,str ] | NoneNone) –LanceDB Embedded declaration and refresh require a source build with the
experimental-functionsCargo feature, which published packages do not enable. LanceDB Enterprise operations remain available.A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order.
Unlike
transforms, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them fromrefresh_column.A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped.
In LanceDB Enterprise the expression is planned by the server. Cannot be combined with
transforms. -
auto_refresh_enabled(, default:bool | NoneNone) –Enable automatic refresh for each computed output declared by this call. Defaults to True. LanceDB Embedded stores the setting; automatic scheduling requires LanceDB Enterprise.
-
auto_refresh_interval_seconds(, default:int | NoneNone) –Positive minimum seconds after a successful refresh before another automatic refresh is eligible. When omitted, the server determines when to refresh. Unchanged inputs do not force recomputation. Applies to every declared computed output. These options cannot be used with one-time transforms or ordinary all-null columns.
-
auto_refresh_resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate CPU, memory, and GPU budget for automatic refreshes of registered Function outputs.
cpu_millisandmemory_bytesare required;gpu_countdefaults to zero. SQL-expression columns do not use Function workers and reject this option. LanceDB Enterprise only.
Examples:
>>> await table.add_columns(
... computed={"total": "price * quantity"},
... auto_refresh_enabled=True,
... auto_refresh_interval_seconds=300,
... )
Returns:
-
–AddColumnsResult version: the new version number of the table after adding columns.
refresh_column
¶
refresh_column(column: str , *, where: Optional [str ] = None, batch_size: Optional [int ] = None, max_concurrency: Optional [int ] = None, publication_interval_rows: Optional [int ] = None, resource_budget: Optional [Dict [str , int ]] = None, gpu_class: Optional [str ] = None, gpu_count: Optional [int ] = None, retry_policy: Optional [FunctionRetryPolicy ] = None, error_policy: Optional [Literal ['fail', 'skip']] = None, max_failed_fraction: Optional [float ] = None, max_failed_rows: Optional [int ] = None, row_timeout_seconds: Optional [float ] = None, init_timeout_seconds: Optional [float ] = None, num_workers: Optional [int ] = None, max_executor_processes_per_worker: Optional [int ] = None) -> RefreshColumnResult
Fill the rows of a computed column that hold no value yet.
Local refresh requires the experimental-functions Cargo feature
in a source build. Published packages leave it disabled.
Declared with add_columns(computed=...), a column starts empty and
gets its values here. Rows appended since the last refresh are filled
by the next one, and rows whose inputs changed since they were computed
are recomputed; everything else is left as it is.
Supported for local tables only. For remote tables, a refresh runs as
a server job, through
refresh_column_async.
Parameters:
-
column() –str The name of the computed column to fill.
-
where(, default:Optional [str ]None) –SQL filter selecting the rows to refresh, as in
REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option. -
batch_size(, default:Optional [int ]None) –Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Has no effect on local expression-backed columns.
-
max_concurrency(, default:Optional [int ]None) –Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.
-
publication_interval_rows(, default:Optional [int ]None) –Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.
-
resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate resources for one registered Function refresh. The
cpu_millisandmemory_byteskeys are required;gpu_countis optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-levelgpu_countshorthand. LanceDB Enterprise only; local refreshes reject it. -
gpu_class(, default:Optional [str ]None) –GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.
-
gpu_count(, default:Optional [int ]None) –Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use
resource_budgetinstead. LanceDB Enterprise only. -
retry_policy(, default:Optional [FunctionRetryPolicy ]None) –Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.
-
error_policy(, default:Optional [Literal ['fail', 'skip']]None) –Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.
-
max_failed_fraction(, default:Optional [float ]None) –Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.
-
max_failed_rows(, default:Optional [int ]None) –Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.
-
row_timeout_seconds(, default:Optional [float ]None) –Deadline for one Function call. LanceDB Enterprise only.
-
init_timeout_seconds(, default:Optional [float ]None) –Deadline for Function initialization. LanceDB Enterprise only.
-
num_workers(, default:Optional [int ]None) –Worker count for an expression column; registered Function columns reject it and use
resource_budgetinstead. LanceDB Enterprise only. -
max_executor_processes_per_worker(, default:Optional [int ]None) –Cap on executor processes per Function worker. LanceDB Enterprise only.
Returns:
-
–RefreshColumnResult The number of rows filled and the new version of the table.
function_errors
¶
function_errors(job_id: Optional [str ] = None, column: Optional [str ] = None, limit: Optional [int ] = None) -> 'FunctionErrors'
The per-row errors Function refreshes recorded on this table.
A refresh running under a skip policy records each row it skipped with the input that failed and the error. This lists those records, newest job first, plus a summary for any fragment whose per-row detail was capped. LanceDB Enterprise only; reading errors needs read access to the table, since a message carries the value that failed.
Parameters:
-
job_id(, default:Optional [str ]None) –Only errors recorded by this job.
-
column(, default:Optional [str ]None) –Only errors on this column.
-
limit(, default:Optional [int ]None) –At most this many records (server default 10000, cap 100000).
Returns:
-
–FunctionErrors records,fragmentsandtruncated, the last saying whether the listing stopped at its limit. -
Each record identifies its row with ``fragment_id``, ``row_offset``,– -
the physical ``row_address`` (``(fragment_id << 32) |– -
row_offset``), and, when the table uses stable row IDs, the stable– -
``row_id`` read from the refresh's pinned table version. Prefer– -
``row_id`` for retries: compaction and rewrite can change the– -
physical fields, while a stable row ID survives them.– -
To retry the recorded rows with stable row IDs:– -
.. code-block:: python–errors = await table.function_errors(column="embedding") row_ids = sorted( record.row_id for record in errors.records if record.row_id is not None ) if row_ids: row_id_list = ", ".join(str(row_id) for row_id in row_ids) await database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowid IN ({row_id_list})" )
-
For a table without stable row IDs, collect the non-null physical– -
addresses before the table is compacted or rewritten and run the– -
SQL refresh:– -
.. code-block:: python–errors = await table.function_errors(column="embedding") addresses = sorted( record.row_address for record in errors.records if record.row_address is not None ) if addresses: address_list = ", ".join(str(address) for address in addresses) await database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowaddr IN ({address_list})" )
-
Records whose per-row detail was capped have no address and appear– -
only in ``fragments``; they cannot be selected individually here.–
Examples:
column_statuses
¶
Return exact status for Function-backed output columns.
All results describe one server table version. column=None returns
every Function output. A handle pinned to an exact version requests
that version.
column_status
¶
Return exact status for one Function-backed output column.
wait_fresh
¶
Poll until no stale or never-computed rows remain.
Failed rows do not block completion. Server and transport errors are not retried. Only the typed finalization-pending response is retried with bounded backoff; arbitrary 500 responses surface immediately.
refresh_column_async
¶
refresh_column_async(column: str , *, where: Optional [str ] = None, batch_size: Optional [int ] = None, max_concurrency: Optional [int ] = None, publication_interval_rows: Optional [int ] = None, resource_budget: Optional [Dict [str , int ]] = None, gpu_class: Optional [str ] = None, gpu_count: Optional [int ] = None, retry_policy: Optional [FunctionRetryPolicy ] = None, error_policy: Optional [Literal ['fail', 'skip']] = None, max_failed_fraction: Optional [float ] = None, max_failed_rows: Optional [int ] = None, row_timeout_seconds: Optional [float ] = None, init_timeout_seconds: Optional [float ] = None, num_workers: Optional [int ] = None, max_executor_processes_per_worker: Optional [int ] = None) -> AsyncJob [RefreshColumnResult ]
Like :meth:refresh_column, but returns a handle to the refresh job
instead of blocking until it completes.
Local refresh requires a source build with experimental-functions.
Remote jobs do not require this feature.
The job may already be complete when returned; callers must not assume
the column is filled until :meth:AsyncJob.wait resolves. Invalid
input -- an unknown column, or one that is not computed -- raises here
rather than failing the job. For local tables, the job runs in-process;
for remote tables, it is the server's backfill job.
Parameters:
-
column() –str The name of the computed column to fill.
-
where(, default:Optional [str ]None) –SQL filter selecting the rows to refresh, as in
REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option. -
batch_size(, default:Optional [int ]None) –Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Lower this for slow Functions to stay within the executor's per-call deadline. Has no effect on local expression-backed columns.
-
max_concurrency(, default:Optional [int ]None) –Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.
-
publication_interval_rows(, default:Optional [int ]None) –Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.
-
resource_budget(, default:Optional [Dict [str ,int ]]None) –Aggregate resources for one registered Function refresh. The
cpu_millisandmemory_byteskeys are required;gpu_countis optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-levelgpu_countshorthand. LanceDB Enterprise only; local refreshes reject it. -
gpu_class(, default:Optional [str ]None) –GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.
-
gpu_count(, default:Optional [int ]None) –Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use
resource_budgetinstead. LanceDB Enterprise only. -
retry_policy(, default:Optional [FunctionRetryPolicy ]None) –Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.
-
error_policy(, default:Optional [Literal ['fail', 'skip']]None) –Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.
-
max_failed_fraction(, default:Optional [float ]None) –Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.
-
max_failed_rows(, default:Optional [int ]None) –Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.
-
row_timeout_seconds(, default:Optional [float ]None) –Deadline for one Function call. LanceDB Enterprise only.
-
init_timeout_seconds(, default:Optional [float ]None) –Deadline for Function initialization. LanceDB Enterprise only.
-
num_workers(, default:Optional [int ]None) –Worker count for an expression column; registered Function columns reject it and use
resource_budgetinstead. LanceDB Enterprise only. -
max_executor_processes_per_worker(, default:Optional [int ]None) –Cap on executor processes per Function worker. LanceDB Enterprise only.
Returns:
-
–AsyncJob [RefreshColumnResult ]A job whose successful
waitreturns row counts plus the source and published table versions.
Examples:
>>> import asyncio
>>> import lancedb
>>> async def refresh_in_background():
... db = await lancedb.connect_async("./.lancedb")
... table = await db.create_table("computed_job_async_demo", [{"x": 1}])
... await table.add_columns(computed={"doubled": "x * 2"})
... job = await table.refresh_column_async("doubled")
... result = await job.wait()
... assert result.rows_assigned == 1
... return await job.status()
>>> asyncio.run(refresh_in_background())
'finished'
alter_columns
¶
Alter column names and nullability.
alterations : Iterable[Dict[str, Any]] A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable.
Each alteration must specify at least one of "rename", "data_type",
or "nullable". The legacy key "name" is also accepted for renaming.
Unknown keys raise ValueError before any alterations are applied.
Local tables reject renaming source or vector columns used by an
embedding function with ValueError, preserving the embedding binding.
Returns:
-
–AlterColumnsResult version: the new version number of the table after the alteration.
update_field_metadata
¶
Update per-field metadata. See
Table.update_field_metadata.
drop_columns
¶
Drop columns from the table.
Local tables reject dropping source or vector columns used by an embedding function with ValueError, preserving the embedding binding.
Parameters:
-
columns() –Iterable [str ]The names of the columns to drop.
version
¶
Retrieve the version of the table
LanceDb supports versioning. Every operation that modifies the table increases
version. As long as a version hasn't been deleted you can [Self::checkout]
that version to view the data at that point. In addition, you can
[Self::restore] the version to replace the current table with a previous
version.
checkout
¶
Checks out a specific version of the Table
Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set.
This is a read-only operation that turns the table into a sort of "view"
or "detached head". Other table instances will not be affected. To make the
change permanent you can use the [Self::restore] method.
Any operation that modifies the table will fail while the table is in a checked
out state. To return the table to a normal state use
[Self::checkout_latest].
Parameters:
-
version() –int |str The version to check out. A version number (
int) or a tag (str) can be provided.
checkout_latest
¶
Ensures the table is pointing at the latest version
This can be used to manually update a table when the read_consistency_interval
is None
It can also be used to undo a [Self::checkout] operation
restore
¶
Restore the table to the currently checked out version
This operation will fail if checkout has not been called previously
This operation will overwrite the latest version of the table with a previous version. Any changes made since the checked out version will no longer be visible.
Once the operation concludes the table will no longer be in a checked out state and the read_consistency_interval, if any, will apply.
take_offsets
¶
Take a list of offsets from the table.
Offsets are 0-indexed and relative to the current version of the table. Offsets are not stable. A row with an offset of N may have a different offset in a different version of the table (e.g. if an earlier row is deleted).
Offsets are mostly useful for sampling as the set of all valid offsets is easily known in advance to be [0, len(table)).
No guarantees are made regarding the order in which results are returned. Repeated offsets produce repeated rows.
Parameters:
-
offsets() –list [int ]The offsets to take.
-
with_row_id(, default:bool False) –If True, include the "_rowid" column in the results.
Returns:
-
–AsyncTakeQuery A query object that can be executed to get the rows at the given offsets.
take_row_ids
¶
Take a list of row ids from the table.
Results follow the requested row-id order. Repeated IDs produce repeated rows; missing or deleted IDs are omitted. This also applies to remote tables.
Row ids are not stable and are relative to the current version of the table. They can change due to compaction and updates.
Unlike offsets, row ids are not 0-indexed and no assumptions should be made about the possible range of row ids. In order to use this method you must first obtain the row ids by scanning or searching the table.
Even so, row ids are more stable than offsets and can be useful in some situations.
There is an ongoing effort to make row ids stable which is tracked at https://github.com/lancedb/lancedb/issues/1120
Parameters:
-
row_ids() –list [int ]The row ids to take.
-
with_row_id(, default:bool False) –If True, include the "_rowid" column in the results.
Returns:
-
–AsyncTakeQuery A query object that can be executed to get the rows.
fetch_blobs
¶
fetch_blob_ranges
¶
fetch_blob_files
¶
fetch_blob_files(column: str , row_ids: Union [list [int ], Array , ChunkedArray , Table ]) -> 'list[Optional[BlobFile]]'
open_blob_file
¶
Open a snapshot-bound descriptor as a seekable file.
See Table.open_blob_file.
open_blob_files
¶
current_branch
¶
The branch this table handle is scoped to, or None for main.
optimize
¶
optimize(*, cleanup_older_than: Optional [timedelta ] = None, delete_unverified: bool = False, retrain=False, compaction_options: Optional [CompactionOptions ] = None) -> OptimizeStats
Optimize the on-disk data and indices for better performance.
.. note::
On a LanceDB Enterprise (remote) table, only compaction is
currently supported. Pruning old versions and index optimization
are silently skipped there instead of being attempted, so this
call returns successfully, with zeroed pruning statistics, once
the compaction job has been submitted; it does not wait for that
job to finish. Remote compaction supports per-job
``target_rows_per_fragment``, ``max_source_rows``, ``io_buffer_size``,
and ``batch_size``. Omitted fields inherit table tuning, then
deployment/built-in defaults. Other non-default compaction settings raise
``NotImplementedError`` before submission. The server must support
these SQL options; older-server errors are propagated without
dropping the requested settings.
Modeled after VACUUM in PostgreSQL.
Optimization covers three operations:
- Compaction: Merges small files into larger ones
- Prune: Removes old versions of the dataset
- Index: Optimizes the indices, adding new data to existing indices
Parameters:
-
cleanup_older_than(, default:Optional [timedelta ]None) –All files belonging to versions older than this will be removed. Set to 0 days to remove all versions except the latest. The latest version is never removed.
.. warning::
Setting this to 0 deletes the data files of every older version, so any other reader or writer still using an older version of the table will fail. Only set it to 0 if no other process is working on this dataset. -
delete_unverified(, default:bool False) –Files leftover from a failed transaction may appear to be part of an in-progress operation (e.g. appending new data) and these files will not be deleted unless they are at least 7 days old. If delete_unverified is True then these files will be deleted regardless of their age.
.. warning::
This should only be set to True if you can guarantee that no other process is currently working on this dataset. Otherwise the dataset could be put into a corrupted state. -
retrain–This parameter is no longer used and is deprecated.
-
compaction_options(, default:Optional [CompactionOptions ]None) –Options that control file compaction. For large rows, derive a lower
target_rows_per_fragmentfrom the expected row size to bound later passes. This does not cap the first pass over an existing oversized fragment; see CompactionOptions.
Notes
The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations.
list_indices
¶
List all indices that have been created with Self::create_index
On LanceDB Enterprise tables, an index whose build hasn't finished may
be missing (vector indices) or listed with 0 indexed rows (scalar and
FTS indices), and num_unindexed_rows may be None. Use
wait_for_index to wait until
an index is ready.
tokenize
¶
tokenize(query: str , *, column: Optional [str ] = None, index_name: Optional [str ] = None) -> Iterable [FtsToken ]
Tokenize a query using the tokenizer configured on an FTS index.
Specify exactly one of column or index_name.
Model-backed tokenizers such as jieba/* and lindera/* are
rebuilt in the client process from index metadata. In LanceDB Enterprise,
this means the same tokenizer model files must also exist locally
(under LANCE_LANGUAGE_MODEL_HOME) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
differ from what the server's index produces. If the model files are
missing, an error naming the tokenizer is raised.
index_stats
¶
Retrieve statistics about an index
Parameters:
-
index_name() –str The name of the index to retrieve statistics for
Returns:
-
–IndexStatistics or NoneThe statistics about the index. Returns None if the index does not exist.
uses_v2_manifest_paths
¶
Check if the table is using the new v2 manifest paths.
Returns:
-
–bool True if the table is using the new v2 manifest paths, False otherwise.
migrate_manifest_paths_v2
¶
Migrate the manifest paths to the new format.
This will update the manifest to use the new v2 format for paths.
This function is idempotent, and can be run multiple times without changing the state of the object store.
Danger
This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations.
You can use AsyncTable.uses_v2_manifest_paths to check if the table is already using the new path style.
replace_field_metadata
¶
Replace the metadata of a field in the schema
.. deprecated:: 0.33.1
Use :func:update_field_metadata instead.
Parameters:
-
field_name() –str The name of the field to replace the metadata for
-
new_metadata() –dict [str ,str ]The new metadata to set
lancedb.table.AsyncTags
¶
Async table tag manager.
list
¶
List all table tags.
Returns:
-
–dict [str ,Tag ]A dictionary mapping tag names to version numbers.
get_version
¶
Get the version of a tag.
Parameters:
-
tag() –str The name of the tag to get the version for.
create
¶
Create a tag for a given table version.
Parameters:
-
tag() –str The name of the tag to create. This name must be unique among all tag names for the table.
-
version() –int The table version to tag.
delete
¶
Delete tag from the table.
Parameters:
-
tag() –str The name of the tag to delete.
update
¶
Update tag to a new version.
Parameters:
-
tag() –str The name of the tag to update.
-
version() –int The new table version to tag.
lancedb.table.AsyncBranches
¶
Async table branch manager.
create
¶
create(name: str , from_ref: Optional [str ] = None, from_version: Optional [int ] = None) -> 'AsyncTable'
Create a branch and return a handle scoped to it.
Parameters:
-
name() –str Name of the new branch.
-
from_ref(, default:Optional [str ]None) –Source branch to fork from. Defaults to
main. -
from_version(, default:Optional [int ]None) –A specific version on
from_refto fork from. Defaults to latest.
checkout
¶
Check out an existing branch and return a handle scoped to it.
Parameters:
-
name() –str Name of the branch to check out.
-
version(, default:Optional [int ]None) –A specific version on the branch to pin. When set, the returned handle is a read-only view of that version; when omitted it tracks the branch's latest and stays writable.
cherry_pick
¶
Cherry-pick a branch onto main, or dry-run.
A failed cherry-pick returns status="failed" instead of raising.
Materialized Views (Asynchronous)¶
lancedb.materialized_view.AsyncMaterializedView
¶
A handle on a materialized view: its table plus its definition.
Obtained from AsyncConnection.create_materialized_view or
AsyncConnection.open_materialized_view.
table
¶
The view, as the table it is. Queries, indexes and search all apply; writes are not blocked, but a rebuild replaces them.
refresh
¶
refresh(*, full: bool = False, source_version: Optional [int ] = None, cpu_millis: Optional [int ] = None, memory_bytes: Optional [int ] = None, gpu_count: Optional [int ] = None, gpu_class: Optional [str ] = None) -> 'RefreshMaterializedViewResult'
Recompute the view from its source.
In LanceDB Enterprise, full=True forces a rebuild; otherwise the
server chooses the refresh mode. LanceDB Embedded views always rebuild,
and source_version selects a source version instead of the latest.
LanceDB Enterprise SQL refreshes do not support source-version pinning.
cpu_millis (1000 = one core), memory_bytes, and gpu_count
cap aggregate resources for this execution, including its Function
processes. CPU or memory requires all three budget fields; GPU count
may be zero for CPU-only views. gpu_class selects an operator-defined
GPU class for this execution only. Omitted options preserve defaults.
These options require an Enterprise backend that enforces isolation;
Embedded refreshes reject them. Other servers may reject the new options.
LanceDB Enterprise results leave mode, rows_written, and
source_version as None because the SQL job adapter does not expose
these metrics.
refresh_async
¶
refresh_async(*, full: bool = False, source_version: Optional [int ] = None, cpu_millis: Optional [int ] = None, memory_bytes: Optional [int ] = None, gpu_count: Optional [int ] = None, gpu_class: Optional [str ] = None) -> 'AsyncJob[RefreshMaterializedViewResult]'
Submit a refresh and return its job without waiting.
The job may already be complete in LanceDB Embedded. In LanceDB
Distributed, refreshes are submitted through the SQL service and the
returned id is the registry job ID, usable with open_job for
progress, events, and cancellation while the refresh runs. Options and
result metrics follow
refresh.
Servers without early registry-handle support return a connection-scoped
query job instead. Its ID cannot be reopened with open_job.
Property Graphs (Asynchronous)¶
AsyncConnection returns this handle from
create_property_graph and
open_property_graph.
lancedb.graph.AsyncPropertyGraph
¶
A handle on one property graph: its name and namespace, and the connection it is described through.
Obtained from AsyncConnection.create_property_graph or
AsyncConnection.open_property_graph. The graph is queried through SQL
GRAPH_TABLE, not through this handle.
Indices (Asynchronous)¶
Indices can be created on a table to speed up queries. This section lists the indices that LanceDb supports.
IndexConfig
¶
A description of an index currently configured on a column
columns
¶
The columns in the index
Currently this is always a list of size 1. In the future there may be more columns to represent composite indices.
created_at
¶
When the index was created.
index_details
¶
index_details: Optional [Any ] = <attribute 'index_details' of 'lancedb._lancedb.IndexConfig' objects>
Index-type-specific details parsed as a Python object (dict, list, etc.).
Falls back to a raw string if JSON parsing fails. None when unavailable.
index_type
¶
The type of the index
index_uuid
¶
The UUID of the first segment of the index.
index_version
¶
index_version: Optional [int ] = <attribute 'index_version' of 'lancedb._lancedb.IndexConfig' objects>
The on-disk index format version.
num_indexed_rows
¶
num_indexed_rows: Optional [int ] = <attribute 'num_indexed_rows' of 'lancedb._lancedb.IndexConfig' objects>
The number of rows indexed, across all segments.
num_segments
¶
The number of segments that make up the index.
num_unindexed_rows
¶
num_unindexed_rows: Optional [int ] = <attribute 'num_unindexed_rows' of 'lancedb._lancedb.IndexConfig' objects>
The number of rows not yet covered by this index.
size_bytes
¶
The total size in bytes of all index files across all segments.
type_url
¶
The protobuf type URL, a precise type identifier for the index.
BTree
¶
Describes a btree index configuration
A btree index is an index on scalar columns. The index stores a copy of the column in sorted order. A header entry is created for each block of rows (currently the block size is fixed at 4096). These header entries are stored in a separate cacheable structure (a btree). To search for data the header is used to determine which blocks need to be read from disk.
For example, a btree index in a table with 1Bi rows requires sizeof(Scalar) * 256Ki bytes of memory and will generally need to read sizeof(Scalar) * 4096 bytes to find the correct row ids.
This index is good for scalar columns with mostly distinct values and does best when the query is highly selective. It works with numeric, temporal, and string columns.
The btree index does not currently have any parameters though parameters such as the block size may be added in the future.
Bitmap
¶
Describe a Bitmap index configuration.
A Bitmap index stores a bitmap for each distinct value in the column for
every row.
This index works best for low-cardinality numeric or string columns,
where the number of unique values is small (i.e., less than a few thousands).
Bitmap index can accelerate the following filters:
<,<=,=,>,>=IN (value1, value2, ...)between (value1, value2)is null
For example, a bitmap index with a table with 1Bi rows, and 128 distinct values, requires 128 / 8 * 1Bi bytes on disk.
LabelList
¶
Describe a LabelList index configuration.
LabelList is a scalar index that can be used on List<T> columns to
support queries with array_has_all and array_has_any
using an underlying bitmap index.
For example, it works with tags, categories, keywords, etc.
Fm
¶
Describe an FM-Index configuration.
Fm is a scalar index on string or binary columns that accelerates
substring search, i.e. contains(col, 'needle'). Unlike the tokenized
FTS index, it matches arbitrary substrings of the raw bytes.
For example, it works with url, path, content, etc.
NGram
¶
Describe an NGram index configuration.
An NGram index is a scalar index on a UTF-8 string column that
accelerates substring filters such as contains(col, 'needle'), along
with some LIKE and regular-expression predicates. It tokenizes values
into overlapping character n-grams (using Lance's default trigram
parameters) and narrows a scan to the rows that share the query's n-grams;
the exact predicate is then evaluated on those candidates, so the results
are the same with or without the index.
Unlike the Fm index, which matches arbitrary substrings of the raw
bytes, NGram is trigram based, so a query shorter than the n-gram length
cannot be served by the index.
For example, it works with text, description, comment, etc.
ZoneMap
¶
Describe a ZoneMap index configuration.
A ZoneMap index stores compact min and max summaries for contiguous
ranges (zones) of rows. During a scan it skips any zone whose summary
cannot satisfy the predicate, so it accelerates range and equality
filters such as col > 10, col BETWEEN a AND b, and col = x,
without changing the results.
It suits large numeric or temporal columns whose values are correlated
with row order (for example timestamp, price, id).
FTS
¶
Describe a FTS index configuration.
FTS supports string columns, lists of strings, and JSON columns created
from PyArrow pa.json_(). JSON columns use Lance's JSON-aware tokenizer
and are stored as LargeBinary with the lance.json logical type.
Arbitrary binary columns are not supported.
JSON queries use path,str,value triples. Paths are relative to the JSON
column and use dots for nested object keys. For example,
meta.tag,str,nature matches the string value at $.meta.tag;
it does not search other paths or JSON key names. The value is processed
with the configured text tokenizer. Use str, not string.
Create and query a JSON index through the modern API::
import pyarrow as pa
from lancedb.index import FTS
table = db.create_table(
"documents",
pa.table({"doc": pa.array(
['{"meta":{"tag":"nature"}}'], type=pa.json_()
)}),
)
table.create_index("doc", config=FTS())
table.search(
"meta.tag,str,nature", query_type="fts", fts_columns="doc"
).to_list()
Create the index on the JSON column itself, not a JSON subpath such as
doc.meta.tag. JSON-path scalar indexes do not provide FTS path scoping.
Plain text such as nature is not a valid JSON FTS query. The current
Lance tokenizer can panic on malformed triples; see
https://github.com/lance-format/lance/issues/8812.
LanceDB Enterprise uses the same FTS() configuration and requires a
server that supports JSON FTS. Wait for table.wait_for_index(["doc_idx"])
before searching a newly created index there with its default name.
Examples:
Create an index configuration that uses 256-document posting blocks:
Create an index that treats each deepest-list element as one document:
>>> from lancedb.query import DocumentGranularity
>>> config = FTS(document_granularity=DocumentGranularity.LIST_ELEMENT)
Create an index that splits code identifiers into searchable subwords:
Notes
Model-backed tokenizers such as jieba/default and lindera/ipadic
require tokenizer models in Lance's language model home. Set
LANCE_LANGUAGE_MODEL_HOME to override the default platform data
directory under lance/language_models.
HnswPq
¶
Describe a HNSW-PQ index configuration.
HNSW-PQ stands for Hierarchical Navigable Small World - Product Quantization. It is a variant of the HNSW algorithm that uses product quantization to compress the vectors. To create an HNSW-PQ index, you can specify the following parameters:
Parameters:
-
distance_type(, default:Literal ['l2', 'cosine', 'dot']'l2') –The distance metric used to train the index.
The following distance types are available:
"l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).
"cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].
"dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.
-
num_partitions(, default:Optional [int ]None) –The number of IVF partitions to create.
For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.
-
num_sub_vectors(, default:Optional [int ]None) –Number of sub-vectors of PQ.
This value controls how much the vector is compressed during the quantization step. The more sub vectors there are the less the vector is compressed. The default is the dimension of the vector divided by 16. If the dimension is not evenly divisible by 16 we use the dimension divided by 8.
The above two cases are highly preferred. Having 8 or 16 values per subvector allows us to use efficient SIMD instructions.
If the dimension is not visible by 8 then we use 1 subvector. This is not ideal and will likely result in poor performance.
-
num_bits(, default:int 8) –Number of bits to encode each sub-vector.
This value controls how much the sub-vectors are compressed. The more bits the more accurate the index but the slower search. Only 4 and 8 are supported.
-
max_iterations(, default:int 50) –Max iterations to train kmeans.
When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.
Increasing this might improve the quality of the index but in most cases the parameter is unused because kmeans will converge with fewer iterations. The parameter is only used in cases where kmeans does not appear to converge. In those cases it is unlikely that setting this larger will lead to the index converging anyways.
-
sample_rate(, default:int 256) –The rate used to calculate the number of training vectors for kmeans.
When an IVF index is trained, we need to calculate partitions. These are groups of vectors that are similar to each other. To do this we use an algorithm called kmeans.
Running kmeans on a large dataset can be slow. To speed this up we run kmeans on a random sample of the data. This parameter controls the size of the sample. The total number of vectors used to train the index is
sample_rate * num_partitions.Increasing this value might improve the quality of the index but in most cases the default should be sufficient.
-
m(, default:int 20) –The number of neighbors to select for each vector in the HNSW graph.
This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.
-
ef_construction(, default:int 300) –The number of candidates to evaluate during the construction of the HNSW graph.
This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than
efin the search phase. -
target_partition_size(, default:Optional [int ]None) –The target size of each partition.
This value controls the tradeoff between search performance and accuracy. faster search but less accurate results as higher value.
HnswSq
¶
Describe a HNSW-SQ index configuration.
HNSW-SQ stands for Hierarchical Navigable Small World - Scalar Quantization. It is a variant of the HNSW algorithm that uses scalar quantization to compress the vectors.
Parameters:
-
distance_type(, default:Literal ['l2', 'cosine', 'dot']'l2') –The distance metric used to train the index.
The following distance types are available:
"l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).
"cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].
"dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.
-
num_partitions(, default:Optional [int ]None) –The number of IVF partitions to create.
For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.
-
max_iterations(, default:int 50) –Max iterations to train kmeans.
When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.
Increasing this might improve the quality of the index but in most cases the parameter is unused because kmeans will converge with fewer iterations. The parameter is only used in cases where kmeans does not appear to converge. In those cases it is unlikely that setting this larger will lead to the index converging anyways.
-
sample_rate(, default:int 256) –The rate used to calculate the number of training vectors for kmeans.
When an IVF index is trained, we need to calculate partitions. These are groups of vectors that are similar to each other. To do this we use an algorithm called kmeans.
Running kmeans on a large dataset can be slow. To speed this up we run kmeans on a random sample of the data. This parameter controls the size of the sample. The total number of vectors used to train the index is
sample_rate * num_partitions.Increasing this value might improve the quality of the index but in most cases the default should be sufficient.
-
m(, default:int 20) –The number of neighbors to select for each vector in the HNSW graph.
This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.
-
ef_construction(, default:int 300) –The number of candidates to evaluate during the construction of the HNSW graph.
This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than
efin the search phase. -
target_partition_size(, default:Optional [int ]None) –The target size of each partition.
This value controls the tradeoff between search performance and accuracy. faster search but less accurate results as higher value.
HnswFlat
¶
Describe a HNSW-FLAT index configuration.
HNSW-FLAT stands for Hierarchical Navigable Small World without quantization.
It stores raw vectors in the HNSW graph, providing the highest recall among
the IVF_HNSW family at the cost of more memory and disk space compared to
:class:HnswSq or :class:HnswPq.
Parameters:
-
distance_type(, default:Literal ['l2', 'cosine', 'dot']'l2') –The distance metric used to train the index.
The following distance types are available:
"l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).
"cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].
"dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.
-
num_partitions(, default:Optional [int ]None) –The number of IVF partitions to create.
For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.
-
max_iterations(, default:int 50) –Max iterations to train kmeans.
When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.
-
sample_rate(, default:int 256) –The rate used to calculate the number of training vectors for kmeans.
-
m(, default:int 20) –The number of neighbors to select for each vector in the HNSW graph.
This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.
-
ef_construction(, default:int 300) –The number of candidates to evaluate during the construction of the HNSW graph.
This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than
efin the search phase. -
target_partition_size(, default:Optional [int ]None) –The target size of each partition.
IvfFlat
¶
Describes an IVF Flat Index
This index stores raw vectors. These vectors are grouped into partitions of similar vectors. Each partition keeps track of a centroid which is the average value of all vectors in the group.
IvfSq
¶
Describes an IVF Scalar Quantization (SQ) index.
This index applies scalar quantization to compress vectors and organizes the quantized vectors into IVF partitions. It offers a balance between search speed and storage efficiency while keeping good recall.
IvfPq
¶
Describes an IVF PQ Index
This index stores a compressed (quantized) copy of every vector. These vectors are grouped into partitions of similar vectors. Each partition keeps track of a centroid which is the average value of all vectors in the group.
During a query the centroids are compared with the query vector to find the closest partitions. The compressed vectors in these partitions are then searched to find the closest vectors.
The compression scheme is called product quantization. Each vector is divide
into subvectors and then each subvector is quantized into a small number of
bits. the parameters num_bits and num_subvectors control this process,
providing a tradeoff between index size (and thus search speed) and index
accuracy.
The partitioning process is called IVF and the num_partitions parameter
controls how many groups to create.
Note that training an IVF PQ index on a large dataset is a slow operation and currently is also a memory intensive operation.
IvfRq
¶
Describes an IVF RQ Index
IVF-RQ (RabitQ Quantization) compresses vectors using RabitQ quantization and organizes them into IVF partitions.
The compression scheme is called RabitQ quantization. Each dimension is
quantized into a small number of bits. The parameters num_bits and
num_partitions control this process, providing a tradeoff between
index size (and thus search speed) and index accuracy.
The partitioning process is called IVF and the num_partitions parameter
controls how many groups to create.
Note that training an IVF RQ index on a large dataset is a slow operation and currently is also a memory intensive operation.
lancedb.table.IndexStatistics
¶
Statistics about an index.
index_type
¶
index_type: Literal ['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT', 'FTS', 'BTREE', 'BITMAP', 'LABEL_LIST', 'NGRAM', 'ZONEMAP']
Querying (Asynchronous)¶
Queries allow you to return data from your database. Basic queries can be created with the AsyncTable.query method to return the entire (typically filtered) table. Vector searches return the rows nearest to a query vector and can be created with the AsyncTable.vector_search method.
lancedb.query.AsyncQuery
¶
Bases:
to_query_object
¶
Convert the query into a query object
This is currently experimental but can be useful as the query object is pure python and more easily serializable.
select
¶
Return only the specified columns.
By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your
existing columns. For example, you may not care about "a" or "b" but instead
simply want "a + b". This is often seen in the SELECT clause of an SQL query
(e.g. SELECT a+b FROM my_table).
To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.
For example, an SQL query might state SELECT a + b AS combined, c. The
equivalent input to this method would be {"combined": "a + b", "c": "c"}.
Columns will always be returned in the order given, even if that order is different than the order used when adding the data.
with_row_address
¶
Include the _rowaddr column in scanner-backed plain query results.
with_fragments
¶
Restrict scanner-backed plain query results to the given Lance fragments.
fragment_ids
¶
Restrict scanner-backed plain query results to the given Lance fragment ids.
to_batches
¶
to_batches(*, max_batch_length: Optional [int ] = None, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader
Execute the query and return the results as an Apache Arrow RecordBatchReader.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
max_batch_length(, default:Optional [int ]None) –The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Use async for batch in await query.to_batches(blob_mode="bytes")
to stream blob payloads and convert each batch with batch.to_pandas().
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and collect the results into an Apache Arrow Table.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse"bytes".
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'
Execute the query and collect the results into a pandas DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_polars
¶
Execute the query and collect the results into a Polars DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
Examples:
>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())
to_pydantic
¶
Convert results to a list of pydantic models.
Parameters:
-
model() –Type [LanceModel ]The pydantic model to use.
-
timeout(, default:timedelta None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–list [LanceModel ]
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
... plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
... print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and display with runtime metrics.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
where
¶
Only return rows matching the given predicate
The predicate can be a SQL string or a type-safe
:class:~lancedb.expr.Expr built with :func:~lancedb.expr.col
and :func:~lancedb.expr.lit.
Examples:
Filtering performance can often be improved by creating a scalar index on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
limit
¶
Set the maximum number of results to return.
By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.
offset
¶
Set the offset for the results.
Parameters:
-
offset() –int The offset to start fetching results from.
order_by
¶
Set the ordering for the results.
Parameters:
-
ordering() –Optional [List [ColumnOrdering ]]The ordering to use for the results. If None, then the default ordering will be used.
fast_search
¶
Skip searching un-indexed data.
This can make queries faster, but will miss any data that has not been indexed.
Tip
You can add new data into an existing index by calling AsyncTable.optimize.
use_rte
¶
Control RTE LSM read routing for this query.
By default (unset), a query against a table with an RTE write spec (see
AsyncTable.enable_rte)
is routed through the LSM scanner so it also returns data written via the
merge_insert LSM path that has not yet been compacted into the base
table (the active/frozen in-memory memtables and the SSTables),
deduplicated by primary key; a table without a spec reads the base table.
Parameters:
-
enable() –bool Trueforces the LSM scanner and errors if the table has no RTE write spec.Falsebypasses the RTE and reads the base table only, even when a spec is present.
postfilter
¶
If this is called then filtering will happen after the search instead of
before.
By default filtering will be performed before the search. This is how
filtering is typically understood to work. This prefilter step does add some
additional latency. Creating a scalar index on the filter column(s) can
often improve this latency. However, sometimes a filter is too complex or
scalar indices cannot be applied to the column. In these cases postfiltering
can be used instead of prefiltering to improve latency.
Post filtering applies the filter to the results of the search. This
means we only run the filter on a much smaller set of data. However, it can
cause the query to return fewer than limit results (or even no results) if
none of the nearest results match the filter.
Post filtering happens during the "refine stage" (described in more detail in
@see {@link VectorQuery#refineFactor}). This means that setting a higher refine
factor can often help restore some of the results lost by post filtering.
nearest_to
¶
Find the nearest vectors to the given query vector.
This converts the query from a plain query to a vector query.
This method will attempt to convert the input to the query vector expected by the embedding model. If the input cannot be converted then an error will be thrown.
By default, there is no embedding model, and the input should be something that can be converted to a pyarrow array of floats. This includes lists, numpy arrays, and tuples.
If there is only one vector column (a column whose data type is a fixed size list of floats) then the column does not need to be specified. If there is more than one vector column you must use AsyncVectorQuery.column to specify which column you would like to compare with.
If no index has been created on the vector column then a vector query will perform a distance comparison between the query vector and every vector in the database and then sort the results. This is sometimes called a "flat search"
For small databases, with tens of thousands of vectors or less, this can be reasonably fast. In larger databases you should create a vector index on the column. If there is a vector index then an "approximate" nearest neighbor search (frequently called an ANN search) will be performed. This search is much faster, but the results will be approximate.
The query can be further parameterized using the returned builder. There are various ANN search parameters that will let you fine tune your recall accuracy vs search latency.
Vector searches always have a
limit. If limit has not been
called then a default limit of 10 will be used.
Typically, a single vector is passed in as the query. However, you can also
pass in multiple vectors. When multiple vectors are passed in, if the vector
column is with multivector type, then the vectors will be treated as a single
query. Or the vectors will be treated as multiple queries, this can be useful
if you want to find the nearest vectors to multiple query vectors. Flat
searches share one table scan across the query vectors, avoiding the scan
and memory amplification of making multiple queries concurrently. If
multiple vectors are passed in then
an additional column query_index will be added to the results. This column
will contain the index of the query vector that the result is nearest to.
nearest_to_text
¶
nearest_to_text(query: str | FullTextQuery , columns: Union [str , List [str ], None] = None) -> AsyncFTSQuery
Find the documents that are most relevant to the given text query.
This method will perform a full text search on the table and return the most relevant documents. The relevance is determined by BM25.
The columns to search must be with native FTS index (Tantivy-based can't work with this method).
By default, all indexed columns are searched, now only one column can be searched at a time.
Parameters:
-
query() –str |FullTextQuery The text query to search for.
-
columns(, default:Union [str ,List [str ], None]None) –The columns to search in. If None, all indexed columns are searched. For now only one column can be searched at a time.
lancedb.query.AsyncVectorQuery
¶
Bases: ,
column
¶
Set the vector column to query
This controls which column is compared to the query vector supplied in the call to AsyncQuery.nearest_to.
This parameter must be specified if the table has more than one column whose data type is a fixed-size-list of floats.
nprobes
¶
Set the legacy IVF probe parameter
The number of probes must be greater than 0.
This argument is only used when the vector column has an IVF-based index. If there is no index then this value is ignored.
The IVF stage of IVF PQ divides the input into partitions (clusters) of related values.
The partition whose centroids are closest to the query vector will be exhaustiely searched to find matches. This parameter controls how many partitions will be searched.
Increasing this value will increase the recall of your query but will also increase the latency of your query. If this method is not called, Lance's adaptive probe defaults are used.
For best results we recommend tuning this parameter with a benchmark against your actual data to find the smallest possible value that will still give you the desired recall.
LanceDB retains this as nprobes through LanceDB Embedded and LanceDB
Enterprise request construction. Lance sets both probe bounds to this
value. Explicit minimum or maximum settings can override their respective
bounds.
search_effort
¶
Set the initial IVF search effort to a finite value in [0, 1].
Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.
None preserves Auto (internally 0.5) and fixed nprobes. Any explicit effort, including 0.5, overrides nprobes in either call order; explicit minimum and maximum bounds remain effective. Zero starts at the caller minimum and one searches all partitions within the maximum. Later probing may continue to obtain enough candidates. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.
Examples:
>>> import asyncio
>>> import lancedb
>>> async def build_query():
... db = await lancedb.connect_async("memory://")
... table = await db.create_table(
... "effort_async", [{"vector": [0.1, 0.2]}]
... )
... return table.vector_search([0.1, 0.2]).search_effort(0.75)
>>> query = asyncio.run(build_query())
minimum_nprobes
¶
Set the minimum number of probes to use.
See nprobes for more details.
These partitions will be searched on every indexed vector query and will increase recall at the expense of latency.
maximum_nprobes
¶
Set the maximum number of probes to use.
See nprobes for more details.
If this value is greater than minimum_nprobes then the excess partitions
will be searched only if we have not found enough results.
This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.
If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.
distance_range
¶
Set the distance range to use.
Only rows with distances within range [lower_bound, upper_bound) will be returned.
Parameters:
-
lower_bound(, default:Optional [float ]None) –The lower bound of the distance range.
-
upper_bound(, default:Optional [float ]None) –The upper bound of the distance range.
Returns:
-
–AsyncVectorQuery The AsyncVectorQuery object.
ef
¶
Set the number of candidates to consider during search
This argument is only used when the vector column has an HNSW index. If there is no index then this value is ignored.
Increasing this value will increase the recall of your query but will also increase the latency of your query. The default value is 1.5 * limit. This default is good for many cases but the best value to use will depend on your data and the recall that you need to achieve.
approx_mode
¶
Set the speed / accuracy tradeoff to use for approximate search
This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.
"fast" favors lower latency and may reduce recall, "normal" (the default) uses the index's default balance, and "accurate" favors higher recall and may increase latency.
Examples:
>>> import asyncio
>>> import lancedb
>>> async def build_query():
... db = await lancedb.connect_async("memory://")
... table = await db.create_table(
... "approx_async", [{"vector": [0.1, 0.2]}]
... )
... return table.vector_search([0.1, 0.2]).approx_mode("accurate")
>>> query = asyncio.run(build_query())
refine_factor
¶
A multiplier to control how many additional rows are taken during the refine step
This argument is only used when the vector column has an IVF PQ index. If there is no index then this value is ignored.
An IVF PQ index stores compressed (quantized) values. They query vector is compared against these values and, since they are compressed, the comparison is inaccurate.
This parameter can be used to refine the results. It can improve both improve recall and correct the ordering of the nearest results.
To refine results LanceDb will first perform an ANN search to find the nearest
limit * refine_factor results. In other words, if refine_factor is 3 and
limit is the default (10) then the first 30 results will be selected. LanceDb
then fetches the full, uncompressed, values for these 30 results. The results
are then reordered by the true distance and only the nearest 10 are kept.
Note: there is a difference between calling this method with a value of 1 and
never calling this method at all. Calling this method with any value will have
an impact on your search latency. When you call this method with a
refine_factor of 1 then LanceDb still needs to fetch the full, uncompressed,
values so that it can potentially reorder the results.
Note: if this method is NOT called then the distances returned in the _distance column will be approximate distances based on the comparison of the quantized query vector and the quantized result vectors. This can be considerably different than the true distance between the query vector and the actual uncompressed vector.
distance_type
¶
Set the distance metric to use
When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use. See @see {@link IvfPqOptions.distanceType} for more details on the different distance metrics available.
Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.
By default "l2" is used.
bypass_vector_index
¶
If this is called then any vector index is skipped
An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.
to_query_object
¶
Convert the query into a query object
This is currently experimental but can be useful as the query object is pure python and more easily serializable.
select
¶
Return only the specified columns.
By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your
existing columns. For example, you may not care about "a" or "b" but instead
simply want "a + b". This is often seen in the SELECT clause of an SQL query
(e.g. SELECT a+b FROM my_table).
To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.
For example, an SQL query might state SELECT a + b AS combined, c. The
equivalent input to this method would be {"combined": "a + b", "c": "c"}.
Columns will always be returned in the order given, even if that order is different than the order used when adding the data.
with_row_address
¶
Include the _rowaddr column in scanner-backed plain query results.
with_fragments
¶
Restrict scanner-backed plain query results to the given Lance fragments.
fragment_ids
¶
Restrict scanner-backed plain query results to the given Lance fragment ids.
to_batches
¶
to_batches(*, max_batch_length: Optional [int ] = None, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader
Execute the query and return the results as an Apache Arrow RecordBatchReader.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
max_batch_length(, default:Optional [int ]None) –The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Use async for batch in await query.to_batches(blob_mode="bytes")
to stream blob payloads and convert each batch with batch.to_pandas().
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and collect the results into an Apache Arrow Table.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse"bytes".
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'
Execute the query and collect the results into a pandas DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_polars
¶
Execute the query and collect the results into a Polars DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
Examples:
>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())
to_pydantic
¶
Convert results to a list of pydantic models.
Parameters:
-
model() –Type [LanceModel ]The pydantic model to use.
-
timeout(, default:timedelta None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–list [LanceModel ]
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
... plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
... print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and display with runtime metrics.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
where
¶
Only return rows matching the given predicate
The predicate can be a SQL string or a type-safe
:class:~lancedb.expr.Expr built with :func:~lancedb.expr.col
and :func:~lancedb.expr.lit.
Examples:
Filtering performance can often be improved by creating a scalar index on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
limit
¶
Set the maximum number of results to return.
By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.
offset
¶
Set the offset for the results.
Parameters:
-
offset() –int The offset to start fetching results from.
order_by
¶
Set the ordering for the results.
Parameters:
-
ordering() –Optional [List [ColumnOrdering ]]The ordering to use for the results. If None, then the default ordering will be used.
fast_search
¶
Skip searching un-indexed data.
This can make queries faster, but will miss any data that has not been indexed.
Tip
You can add new data into an existing index by calling AsyncTable.optimize.
use_rte
¶
Control RTE LSM read routing for this query.
By default (unset), a query against a table with an RTE write spec (see
AsyncTable.enable_rte)
is routed through the LSM scanner so it also returns data written via the
merge_insert LSM path that has not yet been compacted into the base
table (the active/frozen in-memory memtables and the SSTables),
deduplicated by primary key; a table without a spec reads the base table.
Parameters:
-
enable() –bool Trueforces the LSM scanner and errors if the table has no RTE write spec.Falsebypasses the RTE and reads the base table only, even when a spec is present.
postfilter
¶
If this is called then filtering will happen after the search instead of
before.
By default filtering will be performed before the search. This is how
filtering is typically understood to work. This prefilter step does add some
additional latency. Creating a scalar index on the filter column(s) can
often improve this latency. However, sometimes a filter is too complex or
scalar indices cannot be applied to the column. In these cases postfiltering
can be used instead of prefiltering to improve latency.
Post filtering applies the filter to the results of the search. This
means we only run the filter on a much smaller set of data. However, it can
cause the query to return fewer than limit results (or even no results) if
none of the nearest results match the filter.
Post filtering happens during the "refine stage" (described in more detail in
@see {@link VectorQuery#refineFactor}). This means that setting a higher refine
factor can often help restore some of the results lost by post filtering.
rerank
¶
nearest_to_text
¶
nearest_to_text(query: str | FullTextQuery , columns: Union [str , List [str ], None] = None) -> AsyncHybridQuery
Find the documents that are most relevant to the given text query, in addition to vector search.
This converts the vector query into a hybrid query.
This search will perform a full text search on the table and return the most relevant documents, combined with the vector query results. The text relevance is determined by BM25.
The columns to search must be with native FTS index (Tantivy-based can't work with this method).
By default, all indexed columns are searched, now only one column can be searched at a time.
Parameters:
-
query() –str |FullTextQuery The text query to search for.
-
columns(, default:Union [str ,List [str ], None]None) –The columns to search in. If None, all indexed columns are searched. For now only one column can be searched at a time.
lancedb.query.AsyncFTSQuery
¶
Bases:
A query for full text search for LanceDB.
to_query_object
¶
Convert the query into a query object
This is currently experimental but can be useful as the query object is pure python and more easily serializable.
select
¶
Return only the specified columns.
By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your
existing columns. For example, you may not care about "a" or "b" but instead
simply want "a + b". This is often seen in the SELECT clause of an SQL query
(e.g. SELECT a+b FROM my_table).
To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.
For example, an SQL query might state SELECT a + b AS combined, c. The
equivalent input to this method would be {"combined": "a + b", "c": "c"}.
Columns will always be returned in the order given, even if that order is different than the order used when adding the data.
with_row_address
¶
Include the _rowaddr column in scanner-backed plain query results.
with_fragments
¶
Restrict scanner-backed plain query results to the given Lance fragments.
fragment_ids
¶
Restrict scanner-backed plain query results to the given Lance fragment ids.
to_batches
¶
to_batches(*, max_batch_length: Optional [int ] = None, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader
Execute the query and return the results as an Apache Arrow RecordBatchReader.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
max_batch_length(, default:Optional [int ]None) –The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Use async for batch in await query.to_batches(blob_mode="bytes")
to stream blob payloads and convert each batch with batch.to_pandas().
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and collect the results into an Apache Arrow Table.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse"bytes".
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'
Execute the query and collect the results into a pandas DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_polars
¶
Execute the query and collect the results into a Polars DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
Examples:
>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())
to_pydantic
¶
Convert results to a list of pydantic models.
Parameters:
-
model() –Type [LanceModel ]The pydantic model to use.
-
timeout(, default:timedelta None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–list [LanceModel ]
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
... plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
... print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and display with runtime metrics.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
where
¶
Only return rows matching the given predicate
The predicate can be a SQL string or a type-safe
:class:~lancedb.expr.Expr built with :func:~lancedb.expr.col
and :func:~lancedb.expr.lit.
Examples:
Filtering performance can often be improved by creating a scalar index on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
limit
¶
Set the maximum number of results to return.
By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.
offset
¶
Set the offset for the results.
Parameters:
-
offset() –int The offset to start fetching results from.
order_by
¶
Set the ordering for the results.
Parameters:
-
ordering() –Optional [List [ColumnOrdering ]]The ordering to use for the results. If None, then the default ordering will be used.
fast_search
¶
Skip searching un-indexed data.
This can make queries faster, but will miss any data that has not been indexed.
Tip
You can add new data into an existing index by calling AsyncTable.optimize.
use_rte
¶
Control RTE LSM read routing for this query.
By default (unset), a query against a table with an RTE write spec (see
AsyncTable.enable_rte)
is routed through the LSM scanner so it also returns data written via the
merge_insert LSM path that has not yet been compacted into the base
table (the active/frozen in-memory memtables and the SSTables),
deduplicated by primary key; a table without a spec reads the base table.
Parameters:
-
enable() –bool Trueforces the LSM scanner and errors if the table has no RTE write spec.Falsebypasses the RTE and reads the base table only, even when a spec is present.
postfilter
¶
If this is called then filtering will happen after the search instead of
before.
By default filtering will be performed before the search. This is how
filtering is typically understood to work. This prefilter step does add some
additional latency. Creating a scalar index on the filter column(s) can
often improve this latency. However, sometimes a filter is too complex or
scalar indices cannot be applied to the column. In these cases postfiltering
can be used instead of prefiltering to improve latency.
Post filtering applies the filter to the results of the search. This
means we only run the filter on a much smaller set of data. However, it can
cause the query to return fewer than limit results (or even no results) if
none of the nearest results match the filter.
Post filtering happens during the "refine stage" (described in more detail in
@see {@link VectorQuery#refineFactor}). This means that setting a higher refine
factor can often help restore some of the results lost by post filtering.
nearest_to
¶
In addition doing text search on the LanceDB Table, also find the nearest vectors to the given query vector.
This converts the query from a FTS Query to a Hybrid query. Results from the vector search will be combined with results from the FTS query.
This method will attempt to convert the input to the query vector expected by the embedding model. If the input cannot be converted then an error will be thrown.
By default, there is no embedding model, and the input should be something that can be converted to a pyarrow array of floats. This includes lists, numpy arrays, and tuples.
If there is only one vector column (a column whose data type is a fixed size list of floats) then the column does not need to be specified. If there is more than one vector column you must use AsyncVectorQuery.column to specify which column you would like to compare with.
If no index has been created on the vector column then a vector query will perform a distance comparison between the query vector and every vector in the database and then sort the results. This is sometimes called a "flat search"
For small databases, with tens of thousands of vectors or less, this can be reasonably fast. In larger databases you should create a vector index on the column. If there is a vector index then an "approximate" nearest neighbor search (frequently called an ANN search) will be performed. This search is much faster, but the results will be approximate.
The query can be further parameterized using the returned builder. There are various ANN search parameters that will let you fine tune your recall accuracy vs search latency.
Hybrid searches always have a
limit. If limit has not been
called then a default limit of 10 will be used.
Typically, a single vector is passed in as the query. However, you can also
pass in multiple vectors. This can be useful if you want to find the nearest
vectors to multiple query vectors. Flat searches share one table scan across
the query vectors instead of issuing concurrent full scans.
If multiple vectors are passed in then an additional column query_index
will be added to the results. This column will contain the index of the
query vector that the result is nearest to.
lancedb.query.AsyncHybridQuery
¶
Bases: ,
A query builder that performs hybrid vector and full text search. Results are combined and reranked based on the specified reranker. By default, the results are reranked using the RRFReranker, which uses reciprocal rank fusion score for reranking.
To make the vector and fts results comparable, the scores are normalized.
Instead of normalizing scores, the normalize parameter can be set to "rank"
in the rerank method to convert the scores to ranks and then normalize them.
column
¶
Set the vector column to query
This controls which column is compared to the query vector supplied in the call to AsyncQuery.nearest_to.
This parameter must be specified if the table has more than one column whose data type is a fixed-size-list of floats.
nprobes
¶
Set the legacy IVF probe parameter
The number of probes must be greater than 0.
This argument is only used when the vector column has an IVF-based index. If there is no index then this value is ignored.
The IVF stage of IVF PQ divides the input into partitions (clusters) of related values.
The partition whose centroids are closest to the query vector will be exhaustiely searched to find matches. This parameter controls how many partitions will be searched.
Increasing this value will increase the recall of your query but will also increase the latency of your query. If this method is not called, Lance's adaptive probe defaults are used.
For best results we recommend tuning this parameter with a benchmark against your actual data to find the smallest possible value that will still give you the desired recall.
LanceDB retains this as nprobes through LanceDB Embedded and LanceDB
Enterprise request construction. Lance sets both probe bounds to this
value. Explicit minimum or maximum settings can override their respective
bounds.
search_effort
¶
Set the initial IVF search effort to a finite value in [0, 1].
Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.
None preserves Auto (internally 0.5) and fixed nprobes. Any explicit effort, including 0.5, overrides nprobes in either call order; explicit minimum and maximum bounds remain effective. Zero starts at the caller minimum and one searches all partitions within the maximum. Later probing may continue to obtain enough candidates. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.
Examples:
>>> import asyncio
>>> import lancedb
>>> async def build_query():
... db = await lancedb.connect_async("memory://")
... table = await db.create_table(
... "effort_async", [{"vector": [0.1, 0.2]}]
... )
... return table.vector_search([0.1, 0.2]).search_effort(0.75)
>>> query = asyncio.run(build_query())
minimum_nprobes
¶
Set the minimum number of probes to use.
See nprobes for more details.
These partitions will be searched on every indexed vector query and will increase recall at the expense of latency.
maximum_nprobes
¶
Set the maximum number of probes to use.
See nprobes for more details.
If this value is greater than minimum_nprobes then the excess partitions
will be searched only if we have not found enough results.
This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.
If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.
distance_range
¶
Set the distance range to use.
Only rows with distances within range [lower_bound, upper_bound) will be returned.
Parameters:
-
lower_bound(, default:Optional [float ]None) –The lower bound of the distance range.
-
upper_bound(, default:Optional [float ]None) –The upper bound of the distance range.
Returns:
-
–AsyncVectorQuery The AsyncVectorQuery object.
ef
¶
Set the number of candidates to consider during search
This argument is only used when the vector column has an HNSW index. If there is no index then this value is ignored.
Increasing this value will increase the recall of your query but will also increase the latency of your query. The default value is 1.5 * limit. This default is good for many cases but the best value to use will depend on your data and the recall that you need to achieve.
approx_mode
¶
Set the speed / accuracy tradeoff to use for approximate search
This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.
"fast" favors lower latency and may reduce recall, "normal" (the default) uses the index's default balance, and "accurate" favors higher recall and may increase latency.
Examples:
>>> import asyncio
>>> import lancedb
>>> async def build_query():
... db = await lancedb.connect_async("memory://")
... table = await db.create_table(
... "approx_async", [{"vector": [0.1, 0.2]}]
... )
... return table.vector_search([0.1, 0.2]).approx_mode("accurate")
>>> query = asyncio.run(build_query())
refine_factor
¶
A multiplier to control how many additional rows are taken during the refine step
This argument is only used when the vector column has an IVF PQ index. If there is no index then this value is ignored.
An IVF PQ index stores compressed (quantized) values. They query vector is compared against these values and, since they are compressed, the comparison is inaccurate.
This parameter can be used to refine the results. It can improve both improve recall and correct the ordering of the nearest results.
To refine results LanceDb will first perform an ANN search to find the nearest
limit * refine_factor results. In other words, if refine_factor is 3 and
limit is the default (10) then the first 30 results will be selected. LanceDb
then fetches the full, uncompressed, values for these 30 results. The results
are then reordered by the true distance and only the nearest 10 are kept.
Note: there is a difference between calling this method with a value of 1 and
never calling this method at all. Calling this method with any value will have
an impact on your search latency. When you call this method with a
refine_factor of 1 then LanceDb still needs to fetch the full, uncompressed,
values so that it can potentially reorder the results.
Note: if this method is NOT called then the distances returned in the _distance column will be approximate distances based on the comparison of the quantized query vector and the quantized result vectors. This can be considerably different than the true distance between the query vector and the actual uncompressed vector.
distance_type
¶
Set the distance metric to use
When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use. See @see {@link IvfPqOptions.distanceType} for more details on the different distance metrics available.
Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.
By default "l2" is used.
bypass_vector_index
¶
If this is called then any vector index is skipped
An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.
to_query_object
¶
Convert the query into a query object
This is currently experimental but can be useful as the query object is pure python and more easily serializable.
select
¶
Return only the specified columns.
By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your
existing columns. For example, you may not care about "a" or "b" but instead
simply want "a + b". This is often seen in the SELECT clause of an SQL query
(e.g. SELECT a+b FROM my_table).
To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.
For example, an SQL query might state SELECT a + b AS combined, c. The
equivalent input to this method would be {"combined": "a + b", "c": "c"}.
Columns will always be returned in the order given, even if that order is different than the order used when adding the data.
with_row_address
¶
Include the _rowaddr column in scanner-backed plain query results.
with_fragments
¶
Restrict scanner-backed plain query results to the given Lance fragments.
fragment_ids
¶
Restrict scanner-backed plain query results to the given Lance fragment ids.
to_batches
¶
to_batches(*, max_batch_length: Optional [int ] = None, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader
Execute the query and return the results as an Apache Arrow RecordBatchReader.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
max_batch_length(, default:Optional [int ]None) –The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Use async for batch in await query.to_batches(blob_mode="bytes")
to stream blob payloads and convert each batch with batch.to_pandas().
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and collect the results into an Apache Arrow Table.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse"bytes".
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'
Execute the query and collect the results into a pandas DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_polars
¶
Execute the query and collect the results into a Polars DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
Examples:
>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())
to_pydantic
¶
Convert results to a list of pydantic models.
Parameters:
-
model() –Type [LanceModel ]The pydantic model to use.
-
timeout(, default:timedelta None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–list [LanceModel ]
where
¶
Only return rows matching the given predicate
The predicate can be a SQL string or a type-safe
:class:~lancedb.expr.Expr built with :func:~lancedb.expr.col
and :func:~lancedb.expr.lit.
Examples:
Filtering performance can often be improved by creating a scalar index on the filter column(s).
Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.
limit
¶
Set the maximum number of results to return.
By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.
offset
¶
Set the offset for the results.
Parameters:
-
offset() –int The offset to start fetching results from.
order_by
¶
Set the ordering for the results.
Parameters:
-
ordering() –Optional [List [ColumnOrdering ]]The ordering to use for the results. If None, then the default ordering will be used.
fast_search
¶
Skip searching un-indexed data.
This can make queries faster, but will miss any data that has not been indexed.
Tip
You can add new data into an existing index by calling AsyncTable.optimize.
use_rte
¶
Control RTE LSM read routing for this query.
By default (unset), a query against a table with an RTE write spec (see
AsyncTable.enable_rte)
is routed through the LSM scanner so it also returns data written via the
merge_insert LSM path that has not yet been compacted into the base
table (the active/frozen in-memory memtables and the SSTables),
deduplicated by primary key; a table without a spec reads the base table.
Parameters:
-
enable() –bool Trueforces the LSM scanner and errors if the table has no RTE write spec.Falsebypasses the RTE and reads the base table only, even when a spec is present.
postfilter
¶
If this is called then filtering will happen after the search instead of
before.
By default filtering will be performed before the search. This is how
filtering is typically understood to work. This prefilter step does add some
additional latency. Creating a scalar index on the filter column(s) can
often improve this latency. However, sometimes a filter is too complex or
scalar indices cannot be applied to the column. In these cases postfiltering
can be used instead of prefiltering to improve latency.
Post filtering applies the filter to the results of the search. This
means we only run the filter on a much smaller set of data. However, it can
cause the query to return fewer than limit results (or even no results) if
none of the nearest results match the filter.
Post filtering happens during the "refine stage" (described in more detail in
@see {@link VectorQuery#refineFactor}). This means that setting a higher refine
factor can often help restore some of the results lost by post filtering.
rerank
¶
Rerank the hybrid search results using the specified reranker. The reranker must be an instance of Reranker class.
Parameters:
-
reranker(, default:Reranker ) –RRFReranker ()The reranker to use. Must be an instance of Reranker class.
-
normalize(, default:str 'score') –The method to normalize the scores. Can be "rank" or "score". If "rank", the scores are converted to ranks and then normalized. If "score", the scores are normalized directly.
Returns:
-
–AsyncHybridQuery The AsyncHybridQuery object.
explain_plan
¶
Return the execution plan for this query.
The output includes both the vector and FTS search plans.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> from lancedb.index import FTS
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", [{"vector": [99.0, 99.0], "text": "hello world"}])
... await table.create_index("text", config=FTS(with_position=False))
... plan = await table.query().nearest_to([1.0, 2.0]).nearest_to_text("hello").explain_plan(True)
... print(plan)
>>> asyncio.run(doctest_example())
RRFReranker(K=60)
ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance, _rowid@1 as _rowid]
LanceRead: uri=..., projection=[text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score, _rowid@0 as _rowid]
LanceRead: uri=..., projection=[vector, text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10
MatchQuery: column=text, query=[hello]
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and return the physical execution plan with runtime metrics.
This runs both the vector and FTS (full-text search) queries and returns detailed metrics for each step of execution—such as rows processed, elapsed time, I/O stats, and more. It’s useful for debugging and performance analysis.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
lancedb.query.AsyncTakeQuery
¶
Bases:
Builder for parameterizing and executing take queries.
to_query_object
¶
Convert the query into a query object
This is currently experimental but can be useful as the query object is pure python and more easily serializable.
select
¶
Return only the specified columns.
By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your
existing columns. For example, you may not care about "a" or "b" but instead
simply want "a + b". This is often seen in the SELECT clause of an SQL query
(e.g. SELECT a+b FROM my_table).
To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.
For example, an SQL query might state SELECT a + b AS combined, c. The
equivalent input to this method would be {"combined": "a + b", "c": "c"}.
Columns will always be returned in the order given, even if that order is different than the order used when adding the data.
with_row_address
¶
Include the _rowaddr column in scanner-backed plain query results.
with_fragments
¶
Restrict scanner-backed plain query results to the given Lance fragments.
fragment_ids
¶
Restrict scanner-backed plain query results to the given Lance fragment ids.
to_batches
¶
to_batches(*, max_batch_length: Optional [int ] = None, timeout: Optional [timedelta ] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader
Execute the query and return the results as an Apache Arrow RecordBatchReader.
A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.
Parameters:
-
max_batch_length(, default:Optional [int ]None) –The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"keeps blob v2 descriptors and auto row ids for manual fetches."descriptions"omits auto row ids."bytes"fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.
Examples:
Use async for batch in await query.to_batches(blob_mode="bytes")
to stream blob payloads and convert each batch with batch.to_pandas().
output_schema
¶
Return the output schema for the query
This does not execute the query.
to_arrow
¶
Execute the query and collect the results into an Apache Arrow Table.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –"lazy"returns blob v2 descriptors that carry the row idfetch_blobsreads."descriptions"returns them without it."bytes"returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse"bytes".
to_list
¶
Execute the query and return the results as a list of dictionaries.
Each list entry is a dictionary with the selected column names as keys,
or all table columns if select is not called. The vector and the "_distance"
fields are returned whether or not they're explicitly selected.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Passed through to :meth:
to_arrow.
to_pandas
¶
to_pandas(flatten: Optional [Union [int , bool ]] = None, timeout: Optional [timedelta ] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'
Execute the query and collect the results into a pandas DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())
Parameters:
-
flatten(, default:Optional [Union [int ,bool ]]None) –If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
-
blob_mode(, default:BlobMode 'lazy') –Controls how blob columns are returned. Local blob v2 queries support
"bytes"and"descriptions"without pylance. Eligible local plain scans also support"bytes"for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in"lazy"and"descriptions"modes, or payloads in"bytes"mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only"descriptions"until a stable table snapshot can bind the query and blob fetches. -
**kwargs–Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.
to_polars
¶
Execute the query and collect the results into a Polars DataFrame.
This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.
Parameters:
-
timeout(, default:Optional [timedelta ]None) –The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.
Examples:
>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
... async for batch in await table.query().to_batches():
... batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())
to_pydantic
¶
Convert results to a list of pydantic models.
Parameters:
-
model() –Type [LanceModel ]The pydantic model to use.
-
timeout(, default:timedelta None) –The maximum time to wait for the query to complete. If None, wait indefinitely.
Returns:
-
–list [LanceModel ]
explain_plan
¶
Return the execution plan for this query.
Examples:
>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
... conn = await connect_async("./.lancedb")
... table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
... plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
... print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
Parameters:
-
verbose(, default:bool False) –Use a verbose output format.
Returns:
-
plan() –str
analyze_plan
¶
Execute the query and display with runtime metrics.
Parameters:
-
distributed_metrics(, default:Literal ['aggregate', 'per_worker', 'full']'aggregate') –Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.
Returns:
-
plan() –str
use_rte
¶
Control RTE LSM read routing for this take query.
False bypasses the RTE and reads the base table only — the escape
hatch, since take-by-row-id/offset is not supported on the LSM scanner.
Storage integrations¶
lancedb.storage_integration.StorageIntegrationInfo
¶
Registered cloud configuration and setup instructions.
configuration_instructions is a single sentence of setup guidance for
display; its wording may change. Use the structured identity and policy fields
instead of parsing that text. Creating this object does not verify AWS access
or configure a storage operation to use it. Static-key authentication stores
only Secret references. No field contains AWS credentials.
lancedb.storage_integration.SecretReference
¶
Integrations belong to a database and follow the same namespace-path naming rules
as Secrets, with their own keyspace. Use a regular database under the cluster root for storage configuration needed
before opening an external database. default is a convention; no database with
that name is required or created automatically. The system database is read-only.
Client methods execute SQL through the existing SQL transport.
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
integration = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
)
print(integration.configuration_instructions)
integration = db.describe_storage_integration("raw")
for name in db.list_storage_integrations():
print(name)
db.drop_storage_integration("raw")
Static credentials use two existing, single-value Secrets instead of a role:
from lancedb.storage_integration import SecretReference
integration = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_access_key_id_secret=SecretReference("s3-key-id", database="default"),
aws_secret_access_key_secret=SecretReference("s3-secret-key", database="default"),
)
Pass namespace_path on the object APIs and Secret references to select a namespace.
A string Secret reference selects the root namespace of the integration's database.
Cross-database references require USAGE on the source database, namespace, and
Secret. Role and static-credential modes are mutually exclusive. Credential values
are never included in integration metadata.
configuration_instructions is one sentence of display guidance whose wording may
change. Use the structured identity and policy fields for programmatic access.
Registration does not verify AWS access. AsyncConnection provides the same methods;
await create, describe, and drop, and use async for for listing.
Azure storage integrations¶
Create three Secrets containing the Azure tenant ID, client ID and client secret in an ordinary database. Reference their names when registering Azure Blob access:
integration = db.create_storage_integration(
"customer_blobs",
"az://customer-data/raw/",
storage_provider="AZURE",
azure_storage_account="customeraccount",
azure_tenant_id_secret="azure_tenant",
azure_client_id_secret="azure_client",
azure_client_secret_secret="azure_credential",
)
print(integration.configuration_instructions)
Use SecretReference to qualify another database or namespace. Sync and async connections use the same options. Secret values are never part of integration metadata. Azure supports service principal client credentials; managed identity, account key and SAS authentication are not integration modes.
The equivalent SQL is:
CREATE STORAGE INTEGRATION customer_blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = public.azure_tenant
STORAGE_AZURE_CLIENT_ID_SECRET = public.azure_client
STORAGE_AZURE_CLIENT_SECRET_SECRET = public.azure_credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
Registration returns setup instructions and leaves access_verified false. It stores an account-qualified prefix such as az://customer-data@customeraccount.blob.core.windows.net/raw/.
GCP storage integrations¶
Create one Secret containing the complete GCP application credentials JSON file content, then reference its name when registering Google Cloud Storage access:
from pathlib import Path
db.create_secret("gcp_credentials", Path("credentials.json").read_text())
integration = db.create_storage_integration(
"customer_gcs",
"gs://customer-data/raw/",
storage_provider="GCP",
gcp_credentials_json_secret="gcp_credentials",
)
print(integration.configuration_instructions)
The equivalent SQL references the existing Secret:
CREATE STORAGE INTEGRATION customer_gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = public.gcp_credentials;
DESCRIBE STORAGE INTEGRATION customer_gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_gcs;
gcp_credentials_json_secret names a Secret; pass the file content to
create_secret, not to create_storage_integration. Use SecretReference to
qualify another database or namespace. Sync and async connections accept the same
options. The service interprets the stored JSON when accessing storage; registration
does not validate its credentials format or verify access, and leaves
access_verified false. Integration metadata contains the Secret reference, never
its JSON content. GCP options cannot be combined with AWS or Azure options.
Hugging Face storage integrations¶
Store a Hugging Face token as one Secret, then register the repository prefix with the existing integration methods. Only the Secret name is part of the integration configuration:
import os
from lancedb.storage_integration import SecretReference
db.create_secret("hf_token", os.environ["HF_TOKEN"])
info = db.create_storage_integration(
"private_hub",
"hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference("hf_token"),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
The equivalent SQL references the existing Secret:
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = public.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
Omit the token Secret for anonymous access at Hugging Face's default anonymous rate limits. A tokenless integration does not use a cluster's Hugging Face token:
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
When provided, the Secret must contain the token itself, with repository read
permission.
Cross-database and namespace references use the same SecretReference rules as
other integrations. Sync and async APIs accept the same options. Registration
returns one sentence of instructions and leaves access_verified false; it does
not contact Hugging Face. Metadata contains the Secret reference, never its value.
Hugging Face options cannot be combined with AWS, Azure or GCP options.
Repository prefixes use hf://datasets/<owner>/<repo>[/<path>] or
hf://models/<owner>/<repo>[/<path>]. An optional @revision follows the repository
name; omitting it allows matching reads of any revision. Prefixes must be concrete
paths without wildcards or traversal. This integration does not make Hugging Face
a writable database location for ATTACH DATABASE. Dataset SQL sources select a
matching integration automatically, with no integration argument in the query.