Skip to content

Python API Reference

This section contains the API reference for the Python API of LanceDB. Both synchronous and asynchronous APIs are available.

The general flow of using the API is:

  1. Use lancedb.connect or lancedb.connect_async to connect to a database.
  2. Use the returned lancedb.DBConnection or lancedb.AsyncConnection to create or open tables.
  3. Use the returned lancedb.table.Table or lancedb.AsyncTable to query or modify tables.

Installation

pip install lancedb

The following methods describe the synchronous API client. There is also an asynchronous API client.

Connections to db:// databases reject local-only options with NotImplementedError. These include session and manifest_enabled=True on connect, per-table storage and write options on create, and index_cache_size, storage_options, or location on open (where the method accepts them). Omit these options to use server-managed storage and caching. Connection-level storage_options supported by an Enterprise deployment remain available.

The synchronous LanceDB Enterprise create_table uses the same positional order as the LanceDB Embedded API: name, data, schema, mode, exist_ok, on_bad_vectors, fill_value, embedding_functions. Prefer keyword arguments for optional parameters.

Connections (Synchronous)

lancedb.connect

connect(uri: Optional[URI] = None, *, api_key: Optional[str] = None, region: str = 'us-east-1', host_override: Optional[str] = None, sql_host_override: Optional[str] = None, read_consistency_interval: Optional[timedelta] = None, request_thread_pool: Optional[Union[int, ThreadPoolExecutor]] = None, client_config: Union[ClientConfig, Dict[str, Any], None] = None, storage_options: Optional[Dict[str, str]] = None, session: Optional[Session] = None, manifest_enabled: bool = False, namespace_client_impl: Optional[str] = None, namespace_client_properties: Optional[Dict[str, str]] = None, namespace_client_pushdown_operations: Optional[List[str]] = None, **kwargs: Any) -> DBConnection

Connect to a LanceDB database.

Parameters:

  • uri (Optional[URI], default: None ) –

    The uri of the database. When namespace_client_impl is provided you may omit uri and connect through a namespace client instead.

  • api_key (Optional[str], default: None ) –

    If presented, connect to LanceDB Enterprise or LanceDB Cloud (legacy). Otherwise, connect to a database on file system or cloud storage. Can be set via environment variable LANCEDB_API_KEY. OAuth configuration is currently supported only by connect_async; synchronous db:// connections require an API key.

  • region (str, default: 'us-east-1' ) –

    The region to use for LanceDB Cloud (legacy).

  • host_override (Optional[str], default: None ) –

    The URL of the LanceDB Enterprise deployment to connect to.

  • sql_host_override (Optional[str], default: None ) –

    The LanceDB Enterprise SQL service endpoint override. The client connects lazily when SQL is first executed and retains that connection.

  • read_consistency_interval (Optional[timedelta], default: None ) –

    The interval at which to check for updates to the table from other processes. If None, then consistency is not checked. For performance reasons, this is the default. For strong consistency, set this to zero seconds. Then every read will check for updates from other processes. As a compromise, you can set this to a non-zero timedelta for eventual consistency. If more than that interval has passed since the last check, then the table will be checked for updates. Note: this consistency only applies to read operations. Write operations are always consistent.

    Stronger consistency is not free. The smaller the interval, the more often each read pays the cost of checking for updates against object storage, raising per-read latency and cost.

  • client_config (Union[ClientConfig, Dict[str, Any], None], default: None ) –

    Configuration options for the HTTP client used by db:// connections. If a dict, then the keys are the attributes of the ClientConfig class. If None, then the default configuration is used.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. See available options at https://docs.lancedb.com/storage/

  • manifest_enabled (bool, default: False ) –

    When true for LanceDB Embedded connections, use directory namespace manifests as the source of truth for table metadata. Existing directory-listed root tables are migrated into the manifest on access.

  • session (Optional[Session], default: None ) –

    (For LanceDB Embedded only) A session to use for this connection. Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.

  • namespace_client_impl (str, default: None ) –

    When provided along with namespace_client_properties, connect returns a namespace-backed connection by delegating to :func:connect_namespace. The value identifies which namespace implementation to load (e.g., "dir" or "rest").

  • namespace_client_properties (dict, default: None ) –

    Configuration to pass to the namespace client implementation. Required when namespace_client_impl is set.

  • namespace_client_pushdown_operations (list[str], default: None ) –

    Only used when namespace_client_properties is provided. Forwards to :func:connect_namespace to control which operations are executed on the namespace service (e.g., ["QueryTable", "CreateTable"]).

Examples:

For a local directory, provide a path for the database:

>>> import lancedb
>>> db = lancedb.connect("~/.lancedb")

For object storage, use a URI prefix:

>>> db = lancedb.connect(
...     "s3://my-bucket/lancedb",
...     storage_options={
...         "aws_access_key_id": "***",
...         "aws_secret_access_key": "***",
...         "aws_region": "us-east-1",
...     },
... )

For Azure Blob Storage, credentials can be passed directly without setting environment variables:

>>> azure_storage_options = {
...     "account_name": "some-account",
...     "account_key": "some-key",
... }
>>> db = lancedb.connect(
...     "az://my-container/my-database",
...     storage_options=azure_storage_options,
... )

For tests and temporary data, use an in-memory database:

>>> db = lancedb.connect("memory://")

In-memory databases are not persisted. Tables are dropped when the last connection or table handle referencing them is closed.

Connect to LanceDB Enterprise or LanceDB Cloud (legacy):

>>> db = lancedb.connect("db://my_database", api_key="ldb_...",
...                      client_config={"retry_config": {"retries": 5}})

Connect to a namespace-backed database:

>>> db = lancedb.connect(namespace_client_impl="dir",
...                      namespace_client_properties={"root": "/tmp/ns"})

Returns:

  • conn ( DBConnection ) –

    A connection to a LanceDB database.

lancedb.db.DBConnection

Bases: EnforceOverrides

An active LanceDB connection interface.

Use close to release the connection's underlying resources, or use the connection as a context manager to close it automatically when leaving the block, including when an exception is raised.

Examples:

>>> import lancedb
>>> with lancedb.connect("memory://") as db:
...     assert db.is_open()
>>> db.is_open()
False

uri

uri: str

The URI of this database.

is_open

is_open() -> bool

Return True if the connection is open.

close

close() -> None

Close the connection, releasing any underlying resources.

It is safe to call this method multiple times. Database operations on a closed connection raise RuntimeError: Connection is closed.

Examples:

>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> db.close()
>>> db.is_open()
False

list_namespaces

list_namespaces(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> NamespaceListing

List immediate child namespace names in the given namespace.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The parent namespace to list namespaces in. Empty list represents root namespace.

  • page_token (Optional[str], default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (Optional[int], default: None ) –

    The maximum number of results to return.

Returns:

  • NamespaceListing –

    Response containing namespace names and optional page_token for pagination.

create_namespace

create_namespace(namespace_path: List[str], mode: Optional[str] = None, properties: Optional[Dict[str, str]] = None) -> CreateNamespaceResponse

Create a new namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to create.

  • mode (Optional[str], default: None ) –

    Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.

  • properties (Optional[Dict[str, str]], default: None ) –

    Properties to set on the namespace.

Returns:

  • CreateNamespaceResponse –

    Response containing the properties of the created namespace.

drop_namespace

drop_namespace(namespace_path: List[str], mode: Optional[str] = None, behavior: Optional[str] = None) -> DropNamespaceResponse

Drop a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to drop.

  • mode (Optional[str], default: None ) –

    Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.

  • behavior (Optional[str], default: None ) –

    Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.

Returns:

  • DropNamespaceResponse –

    Response containing properties and transaction_id if applicable.

describe_namespace

describe_namespace(namespace_path: List[str]) -> DescribeNamespaceResponse

Describe a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to describe.

Returns:

  • DescribeNamespaceResponse –

    Response containing the namespace properties.

namespace_exists

namespace_exists(namespace_id: List[str]) -> bool

Check if a namespace exists.

Parameters:

  • namespace_id (List[str]) –

    The namespace identifier to check.

Returns:

  • bool –

    True if the namespace exists, False otherwise.

Raises:

  • NotImplementedError –

    If the connection type does not support namespace operations.

table_exists

table_exists(table_id: List[str]) -> bool

Check if a table exists.

Parameters:

  • table_id (List[str]) –

    The table identifier to check (full path including namespace segments and table name).

Returns:

  • bool –

    True if the table exists, False otherwise.

Raises:

  • NotImplementedError –

    If the connection type does not support namespace operations.

list_tables

list_tables(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> TableListing

List tables in this database, one page at a time.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to list tables in. None or empty list represents root namespace.

  • page_token (Optional[str], default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (Optional[int], default: None ) –

    The maximum number of results to return, default 100. Zero returns an empty page without a continuation token.

Returns:

  • TableListing –

    Response containing table names and optional page_token for pagination.

table_names

table_names(page_token: Optional[str] = None, limit: Optional[int] = None, *, namespace_path: Optional[List[str]] = None) -> Iterable[str]

List all tables in this database, in sorted order

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to list tables in. Empty list represents root namespace.

  • page_token (Optional[str], default: None ) –

    The token to use for pagination. If not present, start from the beginning. Typically, this token is last table name from the previous page.

  • limit (Optional[int], default: None ) –

    The maximum number of tables to return. None returns all tables; zero returns an empty list.

Returns:

  • Iterable of str –

create_storage_integration

create_storage_integration(name: str, storage_prefix: str, *, aws_role_arn: Optional[str] = None, aws_access_key_id_secret: Optional[Union[str, SecretReference]] = None, aws_secret_access_key_secret: Optional[Union[str, SecretReference]] = None, aws_region: Optional[str] = None, storage_provider: Literal['S3', 'AZURE', 'GCP', 'HUGGINGFACE'] = 'S3', azure_storage_account: Optional[str] = None, azure_tenant_id_secret: Optional[Union[str, SecretReference]] = None, azure_client_id_secret: Optional[Union[str, SecretReference]] = None, azure_client_secret_secret: Optional[Union[str, SecretReference]] = None, gcp_credentials_json_secret: Optional[Union[str, SecretReference]] = None, huggingface_token_secret: Optional[Union[str, SecretReference]] = None, namespace_path: Optional[List[str]] = None) -> StorageIntegrationInfo

describe_storage_integration

describe_storage_integration(name: str, *, namespace_path: Optional[List[str]] = None) -> StorageIntegrationInfo

list_storage_integrations

list_storage_integrations(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

drop_storage_integration

drop_storage_integration(name: str, *, namespace_path: Optional[List[str]] = None) -> None

create_table

create_table(name: str, data: Optional[DATA] = None, schema: Optional[Union[Schema, LanceModel]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, data_storage_version: Optional[str] = None, enable_v2_manifest_paths: Optional[bool] = None) -> Table

Create a Table in the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to create the table in. Empty list represents root namespace.

  • data (Optional[DATA], default: None ) –

    User must provide at least one of data or schema. Acceptable types are:

    • list-of-dict

    • pandas.DataFrame

    • pyarrow.Table or pyarrow.RecordBatch

  • schema (Optional[Union[Schema, LanceModel]], default: None ) –

    Acceptable types are:

    Plain binary values are cast into blob columns declared with blob. To construct Arrow record batches explicitly, use blob_array.

  • mode (str, default: 'create' ) –

    The mode to use when creating the table. Can be "create", "overwrite", or "exist_ok". By default, if the table already exists, an exception is raised. If you want to overwrite the table, use mode="overwrite". To open an existing table without adding data, use mode="exist_ok".

  • exist_ok (bool, default: False ) –

    If a table by the same name already exists, then raise an exception if exist_ok=False. If exist_ok=True, then open the existing table; it will not add the provided data but will validate against any schema that's specified.

  • on_bad_vectors (str, default: 'error' ) –

    How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".

  • fill_value (float, default: 0.0 ) –

    The value to use when filling vectors. Only used if on_bad_vectors="fill". NaN or null elements and missing trailing elements are replaced with this value. Valid elements are preserved, and vectors longer than the expected dimension are truncated.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/

    To enable stable row IDs (row IDs remain stable after compaction, update, delete, and merges), set new_table_enable_stable_row_ids to "true" in storage_options when connecting to the database.

  • data_storage_version (Optional[str], default: None ) –

    Deprecated. Set storage_options when connecting to the database and set new_table_data_storage_version in the options.

  • enable_v2_manifest_paths (Optional[bool], default: None ) –

    Deprecated. Set storage_options when connecting to the database and set new_table_enable_v2_manifest_paths in the options.

Returns:

  • LanceTable –

    A reference to the newly created table.

  • !!! note –

    The vector index won't be created by default. To create the index, call the create_index method on the table.

Examples:

Can create with list of tuples or dictionaries:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
...         {"vector": [0.2, 1.8], "lat": 40.1, "long":  -74.1}]
>>> db.create_table("my_table", data)
LanceTable(name='my_table', ...)
>>> db["my_table"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

You can also pass a pandas DataFrame:

>>> import pandas as pd
>>> data = pd.DataFrame({
...    "vector": [[1.1, 1.2], [0.2, 1.8]],
...    "lat": [45.5, 40.1],
...    "long": [-122.7, -74.1]
... })
>>> db.create_table("table2", data)
LanceTable(name='table2', ...)
>>> db["table2"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.

>>> import pyarrow as pa
>>> custom_schema = pa.schema([
...   pa.field("vector", pa.list_(pa.float32(), 2)),
...   pa.field("lat", pa.float32()),
...   pa.field("long", pa.float32())
... ])
>>> db.create_table("table3", data, schema = custom_schema)
LanceTable(name='table3', ...)
>>> db["table3"].head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: float
long: float
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

It is also possible to create an table from [Iterable[pa.RecordBatch]]:

>>> import pyarrow as pa
>>> def make_batches():
...     for i in range(5):
...         yield pa.RecordBatch.from_arrays(
...             [
...                 pa.array([[3.1, 4.1], [5.9, 26.5]],
...                     pa.list_(pa.float32(), 2)),
...                 pa.array(["foo", "bar"]),
...                 pa.array([10.0, 20.0]),
...             ],
...             ["vector", "item", "price"],
...         )
>>> schema=pa.schema([
...     pa.field("vector", pa.list_(pa.float32(), 2)),
...     pa.field("item", pa.utf8()),
...     pa.field("price", pa.float32()),
... ])
>>> db.create_table("table4", make_batches(), schema=schema)
LanceTable(name='table4', ...)

open_table

open_table(name: str, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, index_cache_size: Optional[int] = None, branch: Optional[str] = None, version: Optional[int] = None) -> Table

Open a Lance Table in the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to open the table from. None or empty list represents root namespace.

  • index_cache_size (Optional[int], default: None ) –

    Deprecated: Use session-level cache configuration instead. Create a Session with custom cache sizes and pass it to lancedb.connect().

    Set the size of the index cache, specified as a number of entries

    The exact meaning of an "entry" will depend on the type of index: * IVF - there is one entry for each IVF partition * BTREE - there is one entry for the entire index

    This cache applies to the entire opened table, across all indices. Setting this value higher will increase performance on larger datasets at the expense of more RAM

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/

  • branch (Optional[str], default: None ) –

    If provided, open a handle scoped to this branch instead of the default branch. Reads and writes operate in the branch's context.

  • version (Optional[int], default: None ) –

    If provided, open the table pinned to this version, producing a read-only handle. Composes with branch: when both are given, opens that branch at the version; otherwise opens main at the version. Call checkout_latest to return to a writable state.

Returns:

  • A LanceTable object representing the table. –

create_materialized_view

create_materialized_view(name: str, source: MaterializedViewSource, *, select: SelectArg = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> MaterializedView

Define a materialized view named name over the table source.

LanceDB Embedded creation and refresh require a source build with the experimental-functions Cargo feature. Published packages leave it disabled; LanceDB Enterprise operations remain available.

The view is populated before creation returns. Pass with_no_data=True to create only its definition and empty backing table. The view is a normal table: it can be queried, indexed and searched, and it appears in table_names.

Parameters:

  • name (str) –

    The name of the view.

  • source (MaterializedViewSource) –

    The name of the source table.

  • select (SelectArg, default: None ) –

    The view's columns: column names, (alias, SQL expression) pairs, or a dict of the same. Omitting it selects every source column, expanded against the source schema at creation time.

  • where (Optional[str], default: None ) –

    SQL predicate; only matching source rows appear in the view.

  • limit (Optional[int], default: None ) –

    Cap the view at this many rows, in materialization order.

  • with_no_data (bool, default: False ) –

    Skip the initial refresh and leave the backing table empty.

  • auto_refresh_enabled (Optional[bool], default: None ) –

    Enable or disable automatic refresh in LanceDB Enterprise. When omitted, the server determines whether automatic refresh is enabled. LanceDB Embedded does not support automatic refresh.

  • auto_refresh_interval_seconds (Optional[float], default: None ) –

    Positive, finite refresh interval in seconds in LanceDB Enterprise. When omitted, the server determines when to refresh.

Returns:

  • MaterializedView –

create_materialized_view_async

create_materialized_view_async(name: str, source: MaterializedViewSource, *, select: SelectArg = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> Job[None]

Submit materialized-view creation and return its job.

The job may already be complete in LanceDB Embedded. In LanceDB Distributed, creation submits CREATE MATERIALIZED VIEW through the SQL server and the returned id is the registry job ID. That ID can be reopened with :meth:open_job, and the handle supports registry progress, events, cancellation, and wait while the job runs. Wait for the job before opening or querying the view.

Servers without early registry-handle support return a connection-scoped query job instead. Its ID cannot be reopened with :meth:open_job.

open_materialized_view

open_materialized_view(name: str) -> MaterializedView

Open the materialized view named name.

Raises ValueError if the table exists but is not a materialized view.

list_materialized_views

list_materialized_views(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

The names of the materialized views in this database.

Local listings inspect table schemas one page at a time.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_materialized_view

drop_materialized_view(name: str, namespace_path: Optional[List[str]] = None) -> None

Drop a materialized view.

In LanceDB Enterprise, drops wait for the SQL command to remove the catalog binding. Physical cleanup continues in the background.

drop_materialized_view_async

drop_materialized_view_async(name: str, namespace_path: Optional[List[str]] = None) -> Job[None]

Start dropping a materialized view and return its job.

The job may already be complete in LanceDB Embedded. In LanceDB Enterprise, jobs have type query and use the SQL query ID as id. Waiting confirms catalog removal; physical cleanup continues in the background.

drop_table

drop_table(name: str, namespace_path: Optional[List[str]] = None)

Drop a table from the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to drop the table from. Empty list represents root namespace.

drop_table_async

drop_table_async(name: str, namespace_path: Optional[List[str]] = None) -> Job

Start dropping a table and return its cleanup job.

rename_table

rename_table(cur_name: str, new_name: str, cur_namespace_path: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None)

Rename a table in the database.

Parameters:

  • cur_name (str) –

    The current name of the table.

  • new_name (str) –

    The new name of the table.

  • cur_namespace_path (Optional[List[str]], default: None ) –

    The namespace of the current table. None or empty list represents root namespace.

  • new_namespace_path (Optional[List[str]], default: None ) –

    The namespace to move the table to. If not specified, defaults to the same as cur_namespace.

drop_database

drop_database()

Drop database This is the same thing as dropping all the tables

drop_all_tables

drop_all_tables(namespace_path: Optional[List[str]] = None)

Drop all tables from the database

Lists the tables in the selected namespace and drops them one at a time.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to drop all tables from. None or empty list represents root namespace.

namespace_client

namespace_client() -> LanceNamespace

Get the equivalent namespace client for this connection.

For native storage connections, this returns a DirectoryNamespace pointing to the same root with the same storage options.

For namespace connections, this returns the backing namespace client.

For LanceDB Enterprise connections, this returns a RestNamespace with the same URI and authentication headers.

Returns:

  • LanceNamespace –

    The namespace client for this connection.

serialize

serialize() -> str

Serialize this connection for reconstruction.

The returned string can be passed to :func:lancedb.deserialize_conn to recreate an equivalent connection, e.g. in a remote worker.

Returns:

  • str –

    Serialized representation of this connection.

create_function

create_function(definition: FunctionDefinition, *, secrets: Optional[Sequence[EnvVarSecret]] = None, namespace_path: Optional[List[str]] = None) -> FunctionVersion

Build and register a scalar Python function, then return its version.

The server builds the OCI image and registers the completed artifact. This is the blocking counterpart of :meth:create_function_async. LanceDB Embedded connections raise NotImplementedError.

Parameters:

  • definition (FunctionDefinition) –

    A callable decorated with function.

  • secrets (sequence of EnvVarSecret, default: None ) –

    One EnvVarSecret per credential the Function needs, each naming a Secret and the environment variable its value arrives in. The Function's source is unchanged by this; it reads the variable the way it already did.

  • namespace_path (list of str, default: None ) –

    The namespace holding the Function. None or an empty list is the root namespace.

Examples:

db.create_secret("model-api-key", os.environ["MODEL_API_KEY"])
db.create_function(
    analyze_caption,
    secrets=[
        EnvVarSecret(
            secret_name="model-api-key", env_variable="MODEL_API_KEY"
        )
    ],
)

create_function_async

create_function_async(definition: FunctionDefinition, *, secrets: Optional[Sequence[EnvVarSecret]] = None, namespace_path: Optional[List[str]] = None) -> Job[FunctionVersion]

Submit a scalar Python function for building and registration.

The server-side job builds the OCI image, then registers the completed artifact. The returned job has type query and the SQL query UUID as its ID. Waiting on it returns the immutable Function version. LanceDB Embedded connections raise NotImplementedError.

get_function

get_function(name: str, *, version: Optional[Union[int, str]] = None, namespace_path: Optional[List[str]] = None) -> FunctionVersion

Open an immutable Function version from the remote catalog.

version may be an integer or string. When omitted, the server resolves the latest currently visible revision and returns that exact immutable version.

list_functions

list_functions(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[FunctionVersion]

List every published immutable Function version in a namespace.

Functions in child namespaces are not included. Results are ordered by Function name then numeric version. Remote connections use SHOW FUNCTIONS HISTORY to include every revision. LanceDB Embedded connections raise NotImplementedError.

Examples:

List the identities available to use in Function-backed columns:

[(function.name, function.version) for function in db.list_functions()]

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_function

drop_function(name: str, *, version: str, namespace_path: Optional[List[str]] = None) -> bool

Drop one published Function revision, preserving its siblings.

Returns True when the revision was removed and False for a repeated drop or a missing name. A revision never published under an existing name is an error. Dropping the last visible revision releases the name and schedules physical cleanup in the background. When the remote server reports dependent warnings, they are emitted as UserWarning before this method returns. Use :meth:drop_function_async to access the query job, whose completion confirms catalog removal of the revision. Local connections raise NotImplementedError.

drop_function_async

drop_function_async(name: str, *, version: str, namespace_path: Optional[List[str]] = None) -> Tuple[bool, Job]

Drop one Function revision and return its SQL query job.

The boolean reports removal of the requested revision from the catalog; sibling revisions and their grants remain available. The job has type query and its ID is the SQL query UUID. Repeated waits confirm that revision's removal. Only the last visible revision's drop releases the name and schedules background object cleanup. LanceDB Embedded connections raise NotImplementedError.

create_secret

create_secret(name: str, value: str, *, namespace_path: Optional[List[str]] = None) -> None

Create a named Secret in this database.

Fails if the name is taken, so a create never silently becomes a rotation. Nothing reads the value back: it is bound to a Function by name and resolved by the service when that Function runs. Local connections raise NotImplementedError.

alter_secret

alter_secret(name: str, value: str, *, namespace_path: Optional[List[str]] = None) -> None

Replace the credential behind an existing Secret.

Fails if it does not exist. Every Function bound to the Secret uses the new value from its next job, and no new Function version is created -- which is how a rotation reaches columns pinned to a version registered before it. LanceDB Embedded connections raise NotImplementedError.

list_secrets

list_secrets(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

The names of every Secret in this database.

Names only. No method returns a stored credential, by construction rather than by policy. LanceDB Embedded connections raise NotImplementedError.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_secret

drop_secret(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a Secret.

Functions bound to it fail at their next job, naming the Secret; that is the revocation path. The name becomes free to reuse, and a new Secret under it is picked up by everything still bound to that name. LanceDB Embedded connections raise NotImplementedError.

describe_secret

describe_secret(name: str, *, namespace_path: Optional[List[str]] = None) -> SecretInfo

What this database records about a Secret: name and timestamps.

Never the value -- there is no code path that could return one. Local connections raise NotImplementedError.

create_view

create_view(name: str, query: str, *, namespace_path: Optional[List[str]] = None) -> ViewDescription

Create a view: a named query the database plans on every read.

The query is planned once, at creation, so one that cannot be planned is refused now rather than at the first read. A view holds no rows, and its readers see its sources as they are at read time.

There is no replace: a name already taken is an error, and changing a view is a drop followed by a create. LanceDB Embedded connections raise NotImplementedError.

import lancedb db = lancedb.connect("db://my_database") # doctest: +SKIP view = db.create_view( ... "adults", "SELECT name FROM people WHERE age >= 18" ... ) # doctest: +SKIP view.schema # doctest: +SKIP name: string

describe_view

describe_view(name: str, *, namespace_path: Optional[List[str]] = None) -> ViewDescription

What this database records about a view: its defining query and the schema that query resolved to.

The schema is the one recorded at creation; a source altered since then shows up when the view is read. LanceDB Embedded connections raise NotImplementedError.

drop_view

drop_view(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a view and wait for catalog removal.

The tables it reads are untouched: a view holds no rows of its own. Use :meth:drop_view_async to retain its SQL query job. Completion confirms catalog removal; physical cleanup continues in the background. LanceDB Embedded connections raise NotImplementedError.

drop_view_async

drop_view_async(name: str, *, namespace_path: Optional[List[str]] = None) -> 'Job[None]'

Drop a view and return its SQL query job.

The job has type query and its ID is the SQL query UUID. Call :meth:Job.wait to confirm catalog removal. Physical cleanup continues in the background. Local connections raise NotImplementedError.

list_views

list_views(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

The names of the views in one namespace.

Names only; a definition comes from describe_view. Local connections raise NotImplementedError.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

create_property_graph

create_property_graph(name: str, nodes: Sequence[NodeTable], edges: Sequence[EdgeTable] = (), *, namespace_path: Optional[List[str]] = None) -> PropertyGraph

Create a property graph over tables in its namespace.

Each NodeTable names the table holding one label's nodes and its key column, and each EdgeTable the table holding one label's edges and the node keys its source and destination columns reference -- the shape of SQL/PGQ's CREATE PROPERTY GRAPH. The graph stores nothing else: a query reads its tables through SQL GRAPH_TABLE, as they are when it runs.

The database checks the definition against the tables now, so one they cannot satisfy is refused here rather than at the first query. A name another property graph holds is an error. Local connections raise NotImplementedError.

import lancedb from lancedb.graph import EdgeTable, Endpoint, NodeTable db = lancedb.connect("db://my_database") # doctest: +SKIP graph = db.create_property_graph( ... "social", ... nodes=[NodeTable("person", key="person_id", label="Person")], ... edges=[ ... EdgeTable( ... "knows", ... label="KNOWS", ... source=Endpoint("src_id", references=("person", "person_id")), ... destination=Endpoint( ... "dst_id", references=("person", "person_id") ... ), ... ) ... ], ... ) # doctest: +SKIP graph.describe().nodes[0].properties # doctest: +SKIP ['person_id', 'name']

open_property_graph

open_property_graph(name: str, *, namespace_path: Optional[List[str]] = None) -> PropertyGraph

Open the property graph named name.

The graph is described to check that it exists, so a name no property graph holds raises HttpError with status_code 404. Local connections raise NotImplementedError.

drop_property_graph

drop_property_graph(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a property graph and wait for its definition to be deleted.

The tables it reads are untouched: a property graph holds no rows of its own. Local connections raise NotImplementedError.

list_property_graphs

list_property_graphs(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

The names of the property graphs in one namespace.

Names only; a definition comes from PropertyGraph.describe. Local connections raise NotImplementedError.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

open_job

open_job(job_id: str) -> Job

Open a server-side job by id, returning a handle with its record already populated.

The returned Job answers for its own state, specification, result, failure and event history, so there is no separate connection-level call for any of them.

Raises JobNotFoundError when the server has no such job, the way open_table does for a missing table.

list_gpu_classes

list_gpu_classes() -> List[GpuClass]

List the operator-configured Function GPU classes.

Each record names the class and reports its configured GPU model, memory, per-worker GPU limit, and CPU, memory, and ephemeral-storage limits. This describes configured hardware, not current free capacity.

LanceDB Enterprise only; the principal needs System Usage permission. Raises NotImplementedError when the server predates GET /v1/gpu/classes.

list_jobs

list_jobs(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[JobInfo]

List server-side jobs across the database's tables.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

cancel_job

cancel_job(job_id: str) -> bool

Request cancellation of a server-side job by id.

Returns True if the server accepted the cancellation, False if no such job exists. Cancelling an already-terminal job is a no-op success.

pause_job

pause_job(job_id: str) -> str

Pause a server-side job by id.

The job's workers drain and it stays parked until resumed. Returns "pausing", "already_paused", or "committing" -- a job finalizing its results cannot be parked; retry shortly.

resume_job

resume_job(job_id: str) -> str

Resume a paused server-side job by id.

Its workers pick their work back up from checkpoints. Returns "resumed", "still_pausing" -- the pause's worker drain is not confirmed yet; retry shortly -- or "not_paused".

execute_query

execute_query(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> RecordBatchReader

Execute SQL and return a blocking Arrow reader.

This submits through :meth:execute_query_async and waits until the initial result stream is readable. It does not wait for the full query to finish. A statement that ran as a server-side SQL job exposes the durable Job Registry id in the reader schema metadata under x-lancedb-sql-job-id. Use :meth:execute_query_async when the query handle itself is needed.

execute_query_async

execute_query_async(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> Query

Start executing SQL and return its query handle.

parameters binds values to the statement's $1 / $name placeholders. They travel as Arrow rather than as SQL text, so a float keeps its exact value and type and a vector stays a compact fixed-size list. A list binds by position, a dict binds by name, and a one-row pyarrow.RecordBatch or pyarrow.Table binds both ways. A 1-D numpy or pyarrow array becomes a fixed-size list of its element type, the shape of a vector column:

.. code-block:: python

db.execute_query(
    "SELECT id FROM docs ORDER BY distance(vector, $1) LIMIT $2",
    parameters=[np.array([0.1, 0.2, 0.3], dtype=np.float32), 10],
)

Parameters are supported in queries, not in DDL or DML. With or without parameters, the server chooses whether to stream rows on the initial call or return a polling continuation. The reader follows either path transparently. Dropping an unread handle or its reader stops a directly streamed query; deferred execution requires explicit cancellation. Open reader() directly: a streamed query needs its reader to make progress and reports running until the stream ends. Once a server-side SQL job exists, describe().job_id exposes its durable Job Registry id while the query is still running. This is separate from the connection-scoped query id.

LanceDB Embedded connections do not support SQL.

describe_query

describe_query(query_id: Union[UUID, str]) -> QueryDescription

Describe a submitted SQL query by its connection-scoped id.

Resource listings return iterators. Use list(db.list_tables()) to collect all table names. db.list_tables().tables is a list of the first page; limit controls its page size and page_token resumes at a saved token. The same patterns apply to list_namespaces(). Other resource listings use page_limit.

Resource listings (list_storage_integrations, list_secrets, list_views, list_jobs, list_functions, and list_materialized_views) return lazy iterators. Use list(db.list_views()) to collect synchronous results, or [name async for name in db.list_views()] with an asynchronous connection. page_limit controls each request, and page_token resumes a listing. Drain the iterator's cached results before saving its token.

lancedb.listing.Listing

Bases: Iterator[T], Generic[T]

A lazy synchronous iterator with inspectable pagination state.

For example::

for name in connection.list_views(page_limit=20):
    print(name)

Use list(connection.list_views()) to collect all results.

num_page_results

num_page_results() -> int

Return the number of cached items available without another request.

page_token

page_token() -> Optional[str]

Return the next request's token.

See AsyncListing.page_token for initial, cached, final-page, and error behavior.

lancedb.listing.TableListing

Bases: _PagedNameListing

Table iterator retaining the current page as a list.

For example::

page = db.list_tables(limit=20)
print(page.tables, page.page_token)
for name in db.list_tables(limit=20):
    print(name)

tables

tables: list[str]

lancedb.listing.NamespaceListing

Bases: _PagedNameListing

Namespace iterator retaining the current page as a list.

For example::

page = db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)

namespaces

namespaces: list[str]

lancedb.Session

A session for managing caches and object stores across LanceDB operations.

Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.

approx_num_items

approx_num_items: int = <attribute 'approx_num_items' of 'lancedb._lancedb.Session' objects>

Get the approximate number of items cached in the session.

Returns:

  • int –

    The number of cached items across all caches

size_bytes

size_bytes: int = <attribute 'size_bytes' of 'lancedb._lancedb.Session' objects>

Get the current size of the session caches in bytes.

Returns:

  • int –

    The total size of all caches in the session

default

default() -> Session

Create a session with default cache sizes.

This is equivalent to creating a session with 6GB index cache and 1GB metadata cache.

Returns:

  • Session –

    A new Session with default cache sizes

Catalogs (Synchronous)

Remote catalogs manage databases through a server's root namespace. Opened databases are ordinary connections. Dropping a database requires it to be empty.

lancedb.connect_catalog

connect_catalog(endpoint: str, *, api_key: Optional[str] = None, client_config: Optional[Union[ClientConfig, dict[str, Any]]] = None, sql_host_override: Optional[str] = None, read_consistency_interval: Optional[timedelta] = None, oauth_config: Optional[OAuthConfig] = None) -> Catalog

Connect synchronously to an HTTP(S) server's root catalog.

See connect_catalog_async for options. Local filesystem and object-store catalogs are not supported.

lancedb.catalog.Catalog

A synchronous remote catalog returned by connect_catalog.

Examples:

catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.create_database("analytics", exist_ok=True)
for name in catalog.list_databases():
    print(name)

authz

authz: Authorization

Authorization sharing this catalog's scope and credentials.

uri

uri: str

The catalog's root namespace endpoint.

create_database

create_database(name: str, *, exist_ok: bool = False) -> DBConnection

Create a database, or open an existing one when exist_ok=True.

connect_database

connect_database(name: str) -> DBConnection

Connect to an existing database by its logical name.

list_databases

list_databases(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> DatabaseNames

Iterate lazily over database names, fetching more results as needed.

page_token resumes from a saved token; None starts at the beginning. page_limit sets the maximum names per REST response (not the total); None uses the server default. Request errors are raised during iteration.

drop_database

drop_database(name: str, *, ignore_missing: bool = False, cascade: bool = False) -> None

Drop a database. A nonempty database is an error unless cascade is set, which drops everything in it first; the name stays taken until the server finishes cleaning up their storage.

lancedb.catalog.DatabaseNames

Bases: Iterator[str]

A synchronous iterator of database names with pagination state.

Returned by Catalog.list_databases.

num_page_results

num_page_results() -> int

Return the number of names available without another REST request.

page_token

page_token() -> Optional[str]

Return the token for the next REST request.

See page_token for initial, cached, final-page, and error behavior.

Authorization

Some catalogs support the authorization API. This is an API that can be used to bind access control lists (ACLs) to individual catalog objects.

Each ACL contains a subject, an object, and privilege.

Subjects

There are three kinds of subjects: principals, groups, and roles.

A principal uniquely identifies each user accessing the system. Each LanceDB Enterprise API key corresponds to a unique principal, and each OpenID user has a unique principal.

All principals have an immutable ID which never changes, and is safe to include in log files. For example, the API key sk_AAAQEAYEAUDAOCAJBIFQYDIOB4IBCEQTCQKRMFYYDENBWHA5DYPQ==== maps to the principal ID pk_aaaqeayeaudaocajbifqydiob4. APIs such as list_acls will return the principal ID, never the API key.

OpenID principals also typically have a human-readable name associated with them. For example, a principal might be named bob@example.com and might have a principal ID of 8a004400-9443-4fab-9997-b9372a7ca8fd. Whenever possible, APIs will return the principal name for convenience. However, principal names are mutable and can be changed by the identity provider.

Here is an example of getting information about an API key:

from lancedb.authz import Subject

catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
principal = catalog.authz.get_principal(Subject.principal_api_key(api_key))
print(f"Found principal {principal.principal_id}")

OpenID principals can be members of groups. Like principals, groups also have IDs and human-readable names. It is not possible to modify group membership via this API; it is managed by the OpenID identity provider.

Here is an example of getting information about a group based on its name:

from lancedb.authz import Subject

catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
group = catalog.authz.get_group(Subject.group_name("Engineering"))
print(f"Found group {group.group_id} with name {group.group_name}")

Roles can contain either principals or groups. Unlike groups, they are managed by the catalog, not the identity provider. Roles can also contain API key principals.

Here is an example of managing the reader role:

from lancedb.authz import Subject

catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
catalog.authz.add_role_binding(subject=Subject.principal_name("Alice"), role="reader")
It may take a few milliseconds for changes to role bindings to become visible to all clients.

More subject types may be added in the future.

Objects

An object is any resource in the system that we can attach authorization rules to. The Python builders and Rust client share resource validation and wire encoding. Methods also accept literal strings such as table:tenant/db:public$events; malformed names for known resource types raise ValueError locally before a request is sent. Unknown resource types are preserved for compatibility with newer servers.

Object types include:

The System object, a singleton which represents the entire system. You can refer to it as Object.system().

Database objects which represents specific databases. For example, Object.database("analytics") represents the "analytics" database.

Table objects which represent specific tables. For example, Object.table(database="analytics", name="events") represents the "events" table inside the "analytics" database. Tables are in the default namespace (which is named "public") unless a specific namespace is given.

Similarly, View objects represent specific views. For example, Object.view(database="analytics", name="recent_events") represents the "recent_events" view inside the "analytics" database. Views are in the default namespace (which is named "public") unless a specific namespace is given.

Secret objects identify named credentials, for example, Object.secret(database="analytics", name="api-key"). Function objects identify registered functions, for example, Object.function(database="analytics", name="caption"). Both default to the public namespace. Pass namespace_path=["platform", "ml"] to select a nested namespace. These objects identify resources for ACL operations; a secret object contains the credential's name, not its value.

Storage integration objects identify database-owned storage configuration, for example Object.storage_integration(database="default", name="raw"). Pass namespace_path for nested namespaces. Their keyspace is separate from Secrets, Functions, and relations; the wire name is storage_integration:default:public$raw.

More object types may be added in the future.

Access control lists

An access control list attaches a specific set of privileges to a specific (subject, object) pair. So, for example, we might have an access control list that gives bob@example.com the ability to read the foo table, that looks like this:

from lancedb.authz import Object, Subject, Privilege

catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
catalog.authz.add_acl(
    object=Object.table(database="analytics", name="foo"),
    subject=Subject.principal_name("bob@example.com"),
    privilege=Privilege.SELECT,
)

Reading a table generally requires USAGE on its database and namespace, plus SELECT on the table. Each object has at most one owner.

OWNERSHIP is a special privilege which grants all the other privileges. By default, the principal who created an object owns it. Only one subject can own an object; granting OWNERSHIP to a new subject removes it from the previous owner. OWNERSHIP cannot be deleted.

To delegate secret or function creation, grant CREATE_SECRET or CREATE_FUNCTION on the containing namespace. This preserves namespace ownership:

for privilege in [Privilege.CREATE_SECRET, Privilege.CREATE_FUNCTION]:
    catalog.authz.add_acl(
        object=Object.namespace(database="analytics", namespace_path=["public"]),
        subject=Subject.role("developer"),
        privilege=privilege,
    )

Both privileges also accept their string names in synchronous and asynchronous ACL methods.

Storage integration creation requires CREATE_STORAGE_INTEGRATION on its containing namespace, together with database and namespace USAGE:

catalog.authz.add_acl(
    object=Object.namespace(database="default"),
    subject=Subject.role("developer"),
    privilege=Privilege.CREATE_STORAGE_INTEGRATION,
)
catalog.authz.add_acl(
    object=Object.storage_integration(database="default", name="raw"),
    subject=Subject.role("reader"),
    privilege=Privilege.USAGE,
)

Creation grants ownership to the creator. Describing an integration requires USAGE on that integration; dropping it requires OWNERSHIP. Listing integrations requires USAGE on the selected database and namespace. Describing and dropping also require parent USAGE grants. Manage these grants through catalog.authz.

By default, the list_acls function lists all the access control entries in the system. By adding a function parameter, the output is filtered to just ACLs matching that parameter. For example, this code lists all the ACLs for the "reader" role:

from lancedb.authz import Subject

catalog = lancedb.connect_catalog("https://catalog.example.com", api_key=api_key)
for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
    print(entry)

Or, when using the async API:

catalog = await lancedb.connect_catalog_async(endpoint, api_key=api_key)
async for entry in catalog.authz.list_acls(subject=Subject.role("reader")):
    print(entry)

Remote SQL

Submit SQL against a LanceDB Enterprise database through the connection. The connected database and default_namespace_path=["public"] are used for unqualified tables. Fully qualified references can still query other databases and namespaces available to the same deployment. execute_query returns a reader as soon as its initial result stream is available. execute_query_async returns a query handle once submission is accepted; use it to inspect progress, open a reader, or cancel the query. The SQL client is initialized by the first query and retained for the lifetime of the connection. Query ids are random, connection-scoped references rather than encoded SQL or durable resume tokens.

Both APIs start with one Flight call. The server can stream results on that call or return a polling continuation that the SDK follows automatically. This behavior is the same with and without parameters. Open the reader directly; a directly streamed query needs its reader to make progress. Dropping an unread handle or its reader stops direct streaming. Deferred execution continues until explicitly cancelled.

For example:

import lancedb

db = lancedb.connect(
    "db://analytics",
    api_key="ldb_...",
    host_override="https://api.example.com",
    sql_host_override="grpc+tls://sql.example.com:10026",
)
reader = db.execute_query(
    """
    SELECT events.id, accounts.name
    FROM analytics.public.events AS events
    JOIN users.public.accounts AS accounts ON events.user_id = accounts.id
    """,
    default_namespace_path=["public"],
)
for batch in reader:
    print(batch.num_rows)

query = db.execute_query_async("SELECT * FROM events")
print(query.id)
print(query.describe().status)
for batch in query.reader():
    print(batch.num_rows)

# Values bind to $1 / $name placeholders and travel as Arrow, so a float32
# stays a float32 and a vector stays a compact fixed-size list:
import numpy as np

reader = db.execute_query(
    "SELECT id FROM docs ORDER BY distance(vector, $vector) LIMIT $k",
    parameters={"vector": np.random.rand(768).astype(np.float32), "k": 10},
)

# The async connection exposes the same lifecycle without blocking:
# async_db = await lancedb.connect_async(
#     "db://analytics",
#     api_key="ldb_...",
#     host_override="https://api.example.com",
#     sql_host_override="grpc+tls://sql.example.com:10026",
# )
# reader = await async_db.execute_query("SELECT * FROM events")
# query = await async_db.execute_query_async("SELECT * FROM events")
# description = await async_db.describe_query(query.id)
# async for batch in await query.reader():
#     print(batch.num_rows)
# await query.cancel()

Namespaces (Synchronous)

A namespace-backed connection resolves tables through a Lance namespace service instead of listing a storage directory.

lancedb.connect_namespace

connect_namespace(namespace_client_impl: str, namespace_client_properties: Dict[str, str], *, read_consistency_interval: Optional[timedelta] = None, storage_options: Optional[Dict[str, str]] = None, session: Optional[Session] = None, namespace_client_pushdown_operations: Optional[List[str]] = None) -> LanceNamespaceDBConnection

Connect to a LanceDB database through a namespace.

Parameters:

  • namespace_client_impl (str) –

    The namespace client implementation to use. For examples:

    • "dir" for DirectoryNamespace
    • "rest" for REST-based namespace
    • Full module path for custom implementations
  • namespace_client_properties (Dict[str, str]) –

    Configuration properties for the namespace client implementation. Different namespace implementations have different config properties. For example, use DirectoryNamespace with {"root": "/path/to/directory"}

  • read_consistency_interval (Optional[timedelta], default: None ) –

    The interval at which to check for updates to the table from other processes. If None, then consistency is not checked.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend

  • session (Optional[Session], default: None ) –

    A session to use for this connection

  • namespace_client_pushdown_operations (Optional[List[str]], default: None ) –

    List of namespace operations to push down to the namespace server. Supported values:

    • "QueryTable": Execute queries on the namespace server via namespace.query_table() instead of locally.
    • "CreateTable": Execute table creation on the namespace server via namespace.create_table() instead of using declare_table + local write.

    Default is None (no pushdown, all operations run locally).

Returns:

  • LanceNamespaceDBConnection –

    A namespace-based connection to LanceDB

lancedb.namespace.LanceNamespaceDBConnection

Bases: DBConnection

A LanceDB connection that uses a namespace for table management.

This connection delegates table URI resolution to a lance_namespace instance, while using the standard LanceTable for actual table operations.

read_consistency_interval

read_consistency_interval = read_consistency_interval

storage_options

storage_options = storage_options or {}

session

session = session

is_open

is_open() -> bool

close

close() -> None

serialize

serialize() -> str

table_names

table_names(page_token: Optional[str] = None, limit: Optional[int] = None, *, namespace_path: Optional[List[str]] = None) -> Iterable[str]

List table names in the database.

If limit is None, return all table names; zero returns an empty list.

.. deprecated:: Use :meth:list_tables instead, which provides proper pagination support.

create_table

create_table(name: str, data: Optional[DATA] = None, schema: Optional[Union[Schema, LanceModel]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, data_storage_version: Optional[str] = None, enable_v2_manifest_paths: Optional[bool] = None) -> Table

open_table

open_table(name: str, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, index_cache_size: Optional[int] = None, branch: Optional[str] = None, version: Optional[int] = None) -> Table

create_materialized_view

create_materialized_view(name: str, source: MaterializedViewSource, *, select: 'SelectArg' = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> 'MaterializedView'

Define a materialized view over a table in the root namespace. See DBConnection.create_materialized_view.

create_materialized_view_async

create_materialized_view_async(name: str, source: MaterializedViewSource, *, select: 'SelectArg' = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> Job[None]

open_materialized_view

open_materialized_view(name: str) -> 'MaterializedView'

Open the materialized view named name.

list_materialized_views

list_materialized_views(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> Listing[str]

The names of the materialized views in the root namespace.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_materialized_view

drop_materialized_view(name: str, namespace_path: Optional[List[str]] = None) -> None

drop_materialized_view_async

drop_materialized_view_async(name: str, namespace_path: Optional[List[str]] = None) -> Job[None]

drop_table

drop_table(name: str, namespace_path: Optional[List[str]] = None)

drop_table_async

drop_table_async(name: str, namespace_path: Optional[List[str]] = None) -> Job

Start dropping a table and return its cleanup job.

rename_table

rename_table(cur_name: str, new_name: str, cur_namespace_path: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None)

drop_database

drop_database()

drop_all_tables

drop_all_tables(namespace_path: Optional[List[str]] = None)

list_namespaces

list_namespaces(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> NamespaceListing

List child namespaces under the given namespace.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The parent namespace path to list children from. If None, lists root-level namespaces.

  • page_token (Optional[str], default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (int, default: None ) –

    Maximum number of namespaces to return.

Returns:

  • NamespaceListing –

    Response containing namespace names and optional page_token for pagination.

create_namespace

create_namespace(namespace_path: List[str], mode: Optional[str] = None, properties: Optional[Dict[str, str]] = None) -> CreateNamespaceResponse

Create a new namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace path to create.

  • mode (str, default: None ) –

    Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.

  • properties (Dict[str, str], default: None ) –

    Properties to set on the namespace.

Returns:

  • CreateNamespaceResponse –

    Response containing the properties of the created namespace.

drop_namespace

drop_namespace(namespace_path: List[str], mode: Optional[str] = None, behavior: Optional[str] = None) -> DropNamespaceResponse

Drop a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace path to drop.

  • mode (str, default: None ) –

    Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.

  • behavior (str, default: None ) –

    Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.

Returns:

  • DropNamespaceResponse –

    Response containing properties and transaction_id if applicable.

describe_namespace

describe_namespace(namespace_path: List[str]) -> DescribeNamespaceResponse

Describe a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to describe.

Returns:

  • DescribeNamespaceResponse –

    Response containing the namespace properties.

namespace_exists

namespace_exists(namespace_id: List[str]) -> bool

Check if a namespace exists.

Parameters:

  • namespace_id (List[str]) –

    The namespace identifier to check.

Returns:

  • bool –

    True if the namespace exists, False otherwise.

table_exists

table_exists(table_id: List[str]) -> bool

Check if a table exists.

Parameters:

  • table_id (List[str]) –

    The table identifier to check (full path including namespace segments and table name).

Returns:

  • bool –

    True if the table exists, False otherwise.

list_tables

list_tables(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> TableListing

List all tables in this database with pagination support.

Parameters:

  • namespace_path (List[str], default: None ) –

    The namespace to list tables in. None or empty list represents root namespace.

  • page_token (str, default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (int, default: None ) –

    The maximum number of results to return.

Returns:

  • TableListing –

    Response containing table names and optional page_token for pagination.

namespace_client

namespace_client() -> LanceNamespace

Get the namespace client for this connection.

For namespace connections, this returns the backing namespace client that was provided during construction.

Returns:

  • LanceNamespace –

    The namespace client for this connection.

Tables (Synchronous)

LanceDB Embedded and LanceDB Enterprise tables use the same arguments for index creation, deletion, index statistics, and search. For example, table.delete(where="id = 1"), table.create_index("cosine", 2) (the deprecated vector API), and table.search([1.0, 1.0], fast_search=True) work with either backend. fast_search=True excludes rows that have not been indexed.

create_scalar_index defaults to replace=True on both backends. Both scalar and full-text index creation accept wait_timeout. LanceDB Enterprise maintenance methods accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops. optimize submits a durable compaction job and returns once that submission is accepted, not once compaction itself finishes. Pruning and index optimization are not yet supported remotely, so it silently skips them and returns with zeroed pruning statistics. Remote compaction_options support per-job target_rows_per_fragment, max_source_rows, io_buffer_size, and batch_size on compatible servers. Other non-default settings raise NotImplementedError before submission; older-server errors are returned without dropping options. The source-row budget selects whole tasks and can produce no work if the first task exceeds it. An I/O buffer size of zero disables byte-based I/O backpressure. Use a positive size to limit I/O read-ahead; this is not a total-memory limit. Each field resolves as invocation option → table tuning → deployment/built-in default. These options apply only to the submitted job and leave table policy unchanged. Table enabled controls automatic scheduling; explicit manual runs remain allowed.

table.optimize(compaction_options={
    "target_rows_per_fragment": 100_000,
    "max_source_rows": 1_000_000,
    "io_buffer_size": 64 * 1024 * 1024,
    "batch_size": 1024,
})

Removed Tantivy options produce the same explanatory errors on both backends.

The old LanceDB Enterprise keyword names predicate for delete, index_name for drop_index, index_uuid for index_stats, and column for create_fts_index remain available with deprecation warnings. Use where, name, index_name, and field_names, respectively. For positional calls, use the argument order documented by Table below; specify index options by keyword when migrating code that used the former LanceDB Enterprise-only argument order.

lancedb.table.Table

Bases: ABC

A Table is a collection of Records in a LanceDB Database.

Examples:

Create using DBConnection.create_table (more examples in that method's documentation).

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data=[{"vector": [1.1, 1.2], "b": 2}])
>>> table.head()
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
b: int64
----
vector: [[[1.1,1.2]]]
b: [[2]]

Can append new data with Table.add().

>>> table.add([{"vector": [0.5, 1.3], "b": 4}])
AddResult(version=2)

Can query the table with Table.search.

>>> table.search([0.4, 0.4]).select(["b", "vector"]).to_pandas()
   b      vector  _distance
0  4  [0.5, 1.3]       0.82
1  2  [1.1, 1.2]       1.13

Search queries are much faster when an index is created. See Table.create_index.

name

name: str

The name of this Table

namespace

namespace: List[str]

Return the namespace path of the table (empty for the root namespace).

id

id: str

Return the full identifier of the table (namespace$name).

version

version: int

The version of this Table

schema

schema: Schema

The Arrow Schema of this Table

tags

tags: Tags

Tag management for the table.

Similar to Git, tags are a way to add metadata to a specific version of the table.

.. warning::

Tagged versions are exempted from the :py:meth:`cleanup_old_versions()`
process.

To remove a version that has been tagged, you must first
:py:meth:`~Tags.delete` the associated tag.

Examples:

.. code-block:: python

table = db.open_table("my_table")
table.tags.create("v2-prod-20250203", 10)

tags = table.tags.list()

branches

branches: 'Branches'

Branch management for the table.

Branches are isolated, writable lines of history forked from another branch (or version). Writes on a branch do not affect main.

embedding_functions

embedding_functions: Dict[str, EmbeddingFunctionConfig]

Get a mapping from vector column name to it's configured embedding function.

current_branch

current_branch() -> Optional[str]

The branch this table handle is scoped to, or None for main.

count_rows

count_rows(filter: Optional[str] = None) -> int

Count the number of rows in the table.

Parameters:

  • filter (Optional[str], default: None ) –

    A SQL where clause to filter the rows to count.

to_pandas

to_pandas(blob_mode: BlobMode = 'lazy', **kwargs) -> 'pandas.DataFrame'

Return the table as a pandas DataFrame.

Parameters:

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned for backends that support Lance blob-aware pandas conversion.

  • **kwargs –

    Forwarded to PyArrow / Lance pandas conversion.

Returns:

  • DataFrame –

to_arrow

to_arrow(*, blob_mode: BlobMode = 'lazy') -> Table

Return the table as a pyarrow Table.

Parameters:

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.

Returns:

  • Table –

to_lance

to_lance(**kwargs) -> LanceDataset

Return a LanceDB Embedded table as a lance.LanceDataset.

LanceDB Enterprise tables cannot be opened as Lance datasets by the client. Use table.search().to_arrow() to read them through the server.

Returns:

  • LanceDataset –

to_polars

to_polars(batch_size=None) -> 'pl.LazyFrame'

Return the table as a Polars LazyFrame.

In LanceDB Enterprise, use table.search().to_polars() to convert query results to a Polars DataFrame.

Parameters:

  • batch_size –

    Passed to Polars as the maximum row count for scanned PyArrow record batches.

Note

The Polars streaming engine is not supported because it does not currently implement Python PyArrow dataset scans. Use the default engine when collecting this LazyFrame.

Returns:

  • LazyFrame –

infer_tokenizer_configs

infer_tokenizer_configs(tokenizer_name: str) -> dict

create_index

create_index(metric: str = 'l2', num_partitions: Optional[int] = None, num_sub_vectors: Optional[int] = None, vector_column_name: str = VECTOR_COLUMN_NAME, replace: bool = True, accelerator: Optional[str] = None, index_cache_size: Optional[int] = None, num_bits: int = 8, index_type: Literal['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT'] = 'IVF_PQ', max_iterations: int = 50, sample_rate: int = 256, m: int = 20, ef_construction: int = 300, *, config: Optional[IndexConfigType] = None, wait_timeout: Optional[timedelta] = None, name: Optional[str] = None, train: bool = True, target_partition_size: Optional[int] = None)

Create an index on a column.

This method supports both the new unified API and the legacy API for backwards compatibility. The new API takes the column name as the first positional argument and an index configuration object via config; the legacy API takes the distance metric as the first argument plus separate vector_column_name / num_partitions / etc. parameters, and emits a DeprecationWarning.

Parameters:

  • metric (str, default: 'l2' ) –

    For new API: the column name to index. For legacy API: the distance metric ("l2", "cosine", "dot", "hamming").

  • config (IndexConfigType, default: None ) –

    The index configuration object. If provided, uses the new unified API. Can be one of: IvfFlat, IvfPq, IvfSq, IvfRq, HnswPq, HnswSq, BTree, Bitmap, LabelList, Fm, NGram, ZoneMap, FTS.

  • replace (bool, default: True ) –

    Whether to replace an existing index on this column.

  • wait_timeout (timedelta, default: None ) –

    LanceDB Enterprise only: how long to wait for the index build to finish. If not set, the call returns once the build is scheduled.

  • name (str, default: None ) –

    Custom name for the index.

  • train (bool, default: True ) –

    Whether to train the index with existing data.

Notes

On LanceDB Embedded tables the index is built before this method returns. On LanceDB Enterprise tables it returns once the build is scheduled. Until the build finishes, a vector index may be missing from list_indices, a scalar or FTS index may be listed with 0 indexed rows, and queries won't use the index. To block until the index is ready, pass wait_timeout, call wait_for_index, or use create_index_async and wait on its job when the server returned a job ID.

wait_timeout waits for the submitted build when the server returns a job ID. On older servers that don't return one, it falls back to polling the index by name. wait_for_index always waits by name. When replacing an existing index, those name-based waits may return before the new build finishes because the old index can still match. create_index_async can track a specific build only when the server returned a job ID; otherwise it returns an already-complete job and callers must use wait_for_index for that build.

Examples:

New API (recommended):

>>> table.create_index(
...     "vector", config=IvfPq(distance_type="l2")
... )
>>> table.create_index("category", config=BTree())
>>> table.create_index("content", config=FTS())

Legacy API (deprecated):

>>> table.create_index(
...     "l2", vector_column_name="vector"
... )

create_index_async

create_index_async(column: str, *, config: IndexConfigType, replace: Optional[bool] = None, wait_timeout: Optional[timedelta] = None, name: Optional[str] = None, train: bool = True) -> Job

Create an index, returning a handle to the indexing job.

Takes the same arguments as :meth:create_index. The job may already be complete when returned; callers must not assume the index exists until :meth:Job.wait returns.

In LanceDB Enterprise, the returned job tracks this submission only if the server returns a job ID. If it does not, the job is already complete and does not wait for the background build; use wait_for_index and account for the replacement caveat described in create_index.

get_table_overrides

get_table_overrides() -> dict[str, Any]

Read Enterprise table overrides for this table.

Table overrides configure Enterprise-managed background behavior such as automatic compaction, cleanup, and reindex execution. This method is supported only on remote Enterprise tables.

update_table_overrides

update_table_overrides(**overrides: Any) -> dict[str, Any]

Merge Enterprise table overrides into the current table overrides.

Only fields supplied in overrides are changed; unrelated existing overrides are preserved by the remote service update flow.

reset_table_overrides

reset_table_overrides() -> dict[str, Any]

Clear Enterprise table overrides for this table.

drop_index

drop_index(name: str) -> None

Drop an index from the table.

Parameters:

  • name (str) –

    The name of the index to drop.

Notes

This does not delete the index from disk, it just removes it from the table. To delete the index, run optimize after dropping the index.

Use list_indices to find the names of the indices.

wait_for_index

wait_for_index(index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300)) -> None

Wait for indexing to complete for the given index names. This will poll the table until all the indices are fully indexed, or raise a timeout exception if the timeout is reached.

Parameters:

  • index_names (Iterable[str]) –

    The name of the indices to poll

  • timeout (timedelta, default: timedelta(seconds=300) ) –

    Timeout to wait for asynchronous indexing. The default is 5 minutes.

stats

stats() -> TableStatistics

Retrieve table and fragment statistics.

create_scalar_index

create_scalar_index(column: str, *, replace: bool = True, index_type: ScalarIndexType = 'BTREE', wait_timeout: Optional[timedelta] = None, name: Optional[str] = None)

Create a scalar index on a column.

Parameters:

  • column (str) –

    The column to be indexed. Must be a boolean, integer, float, or string column.

  • replace (bool, default: True ) –

    Replace the existing index if it exists.

  • index_type (ScalarIndexType, default: 'BTREE' ) –

    The type of index to create.

  • wait_timeout (Optional[timedelta], default: None ) –

    The timeout to wait if indexing is asynchronous.

  • name (Optional[str], default: None ) –

    The name of the index. If not provided, a default name will be generated.

Examples:

Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column my_col has a scalar index:

>>> import lancedb
>>> db = lancedb.connect("/data/lance")
>>> img_table = db.open_table("images")
>>> my_df = img_table.search().where("my_col = 7",
...                                  prefilter=True).to_pandas()

Scalar indices can also speed up scans containing a vector search and a prefilter:

>>> import lancedb
>>> db = lancedb.connect("/data/lance")
>>> img_table = db.open_table("images")
>>> img_table.search([1, 2, 3, 4], vector_column_name="vector")
...     .where("my_col != 7", prefilter=True)
...     .to_pandas()

Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. my_col BETWEEN 0 AND 100), and set membership (e.g. my_col IN (0, 1, 2))

Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. my_col < 0 AND other_col> 100)

Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column not_indexed does not have a scalar index then the filter my_col = 0 OR not_indexed = 1 will not be able to use any scalar index on my_col.

create_fts_index

create_fts_index(field_names: Union[str, List[str]], *, ordering_field_names: Optional[Union[str, List[str]]] = None, replace: bool = False, writer_heap_size: Optional[int] = 1024 * 1024 * 1024, use_tantivy: bool = False, tokenizer_name: Optional[str] = None, with_position: bool = False, base_tokenizer: BaseTokenizerType = 'simple', language: str = 'English', max_token_length: Optional[int] = 40, lower_case: bool = True, stem: Optional[bool] = None, remove_stop_words: Optional[bool] = None, custom_stop_words: Optional[List[str]] = None, ascii_folding: bool = True, ngram_min_length: int = 3, ngram_max_length: int = 3, prefix_only: bool = False, block_size: int = 128, document_granularity: DocumentGranularity = DocumentGranularity.ROW, wait_timeout: Optional[timedelta] = None, name: Optional[str] = None)

Create a full-text search index on the table.

Warning - this API is highly experimental and is highly likely to change in the future.

Parameters:

  • field_names (Union[str, List[str]]) –

    The name of the field to index. Native FTS indexes can only be created on a single field at a time. To search over multiple text fields, create a separate FTS index for each field.

  • replace (bool, default: False ) –

    If True, replace the existing index if it exists. Note that this is not yet an atomic operation; the index will be temporarily unavailable while the new index is being created.

  • writer_heap_size (Optional[int], default: 1024 * 1024 * 1024 ) –

    Deprecated legacy Tantivy parameter. Any value other than the default raises an error.

  • ordering_field_names (Optional[Union[str, List[str]]], default: None ) –

    Deprecated legacy Tantivy parameter. Setting this raises an error.

  • tokenizer_name (Optional[str], default: None ) –

    A compatibility alias for native tokenizer configs. Can be "raw", "default" or the 2 letter language code followed by "_stem". So for english it would be "en_stem". For new native FTS indexes, use base_tokenizer directly; tokenizer_name is a legacy compatibility alias and does not expose model-backed tokenizer names such as jieba/default or lindera/ipadic.

  • use_tantivy (bool, default: False ) –

    Deprecated legacy Tantivy parameter. Setting this to True raises an error.

  • with_position (bool, default: False ) –

    If False, do not store the positions of the terms in the text. This can reduce the size of the index and improve indexing speed. But it will raise an exception for phrase queries.

  • base_tokenizer (str, default: "simple" ) –

    The base tokenizer to use for tokenization. Options are: - "simple": Splits text by whitespace and punctuation. - "whitespace": Split text by whitespace, but not punctuation. - "raw": No tokenization. The entire text is treated as a single token. - "code": Tokenizes source code and identifiers. - "ngram": N-Gram tokenizer. - "icu": ICU dictionary-based word segmentation. - "icu/split": ICU segmentation with simple-style delimiter splitting. - "jieba/": Jieba tokenizer loaded from Lance's language model home. - "lindera/": Lindera tokenizer loaded from Lance's language model home.

  • language (str, default: "English" ) –

    The language to use for stemming and stop-word removal. This is not the primary way to enable CJK tokenization.

  • max_token_length (int, default: 40 ) –

    The maximum token length to index. Tokens longer than this length will be ignored.

  • lower_case (bool, default: True ) –

    Whether to convert the token to lower case. This makes queries case-insensitive.

  • stem (bool, default: None ) –

    Whether to stem the token. Stemming reduces words to their root form. For example, in English "running" and "runs" would both be reduced to "run". None uses the base tokenizer's default: False for code and ngram, True otherwise.

  • remove_stop_words (bool, default: None ) –

    Whether to remove stop words. Stop words are common words that are often removed from text before indexing. For example, in English "the" and "and". None uses the base tokenizer's default: False for code and ngram, True otherwise.

  • custom_stop_words (list of str, default: None ) –

    Custom words that replace the built-in language stop words. None uses the built-in list; an empty list explicitly uses no stop words.

  • ascii_folding (bool, default: True ) –

    Whether to fold ASCII characters. This converts accented characters to their ASCII equivalent. For example, "café" would be converted to "cafe".

  • ngram_min_length (int, default: 3 ) –

    The minimum length of an n-gram.

  • ngram_max_length (int, default: 3 ) –

    The maximum length of an n-gram.

  • prefix_only (bool, default: False ) –

    Whether to only index the prefix of the token for ngram tokenizer.

  • block_size (int, default: 128 ) –

    The number of documents per compressed posting block. Must be 128 or 256. A value of 256 uses the experimental FTS V3 format and may introduce breaking changes.

  • document_granularity (DocumentGranularity, default: ROW ) –

    ROW treats the selected text in one table row as one document. LIST_ELEMENT treats each element of the deepest list on the field path as one document and returns its physical coordinates in _doc_index for matching queries.

  • wait_timeout (Optional[timedelta], default: None ) –

    The timeout to wait if indexing is asynchronous.

  • name (Optional[str], default: None ) –

    The name of the index. If not provided, a default name will be generated.

Notes

Model-backed tokenizers such as jieba/default and lindera/ipadic require tokenizer models in Lance's language model home. Set LANCE_LANGUAGE_MODEL_HOME to override the default platform data directory under lance/language_models.

add

add(data: DATA, mode: AddMode = 'append', on_bad_vectors: OnBadVectorsType = 'error', fill_value: float = 0.0, progress: Optional[Union[bool, Callable, Any]] = None, write_parallelism: Optional[int] = None, allow_external_blob_outside_bases: bool = False) -> AddResult

Add more data to the Table.

Plain binary values are cast into blob columns using the table's schema. To construct Arrow record batches explicitly, use blob_array.

Parameters:

  • data (DATA) –

    The data to insert into the table. Acceptable types are:

    • list-of-dict

    • pandas.DataFrame

    • pyarrow.Table or pyarrow.RecordBatch

  • mode (AddMode, default: 'append' ) –

    The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields.

  • on_bad_vectors (OnBadVectorsType, default: 'error' ) –

    How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".

  • fill_value (float, default: 0.0 ) –

    The value to use when filling vectors. Only used if on_bad_vectors="fill".

  • progress (Optional[Union[bool, Callable, Any]], default: None ) –

    Progress reporting during the add operation. Can be:

    • True to automatically create and display a tqdm progress bar (requires tqdm to be installed)::

      table.add(data, progress=True)

    • A callable that receives a dict with keys output_rows, output_bytes, total_rows, elapsed_seconds, active_tasks, total_tasks, and done::

      def on_progress(p): print(f"{p['output_rows']}/{p['total_rows']} rows, " f"{p['active_tasks']}/{p['total_tasks']} workers") table.add(data, progress=on_progress)

    • A tqdm-compatible progress bar whose total and update() will be called automatically. The postfix shows write throughput (MB/s) and active worker count::

      with tqdm() as pbar: table.add(data, progress=pbar)

  • write_parallelism (Optional[int], default: None ) –

    Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory.

  • allow_external_blob_outside_bases (bool, default: False ) –

    Store blob URIs that sit outside registered blob bases. The row keeps a reference, so the object has to stay readable. Local tables only.

Returns:

  • AddResult –

    An object containing the new version number of the table after adding data.

merge_insert

merge_insert(on: Union[str, Iterable[str]]) -> LanceMergeInsertBuilder

Returns a LanceMergeInsertBuilder that can be used to create a "merge insert" operation

This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like "insert if not exists", "update or insert (i.e. upsert)", or even replace a portion of existing data with new data (e.g. replace all data where month="january")

The merge insert operation works by combining new data from a source table with existing data in a target table by using a join. There are three categories of records.

"Matched" records are records that exist in both the source table and the target table. "Not matched" records exist only in the source table (e.g. these are new data) "Not matched by source" records exist only in the target table (this is old data)

The builder returned by this method can be used to customize what should happen for each category of data.

Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values.

Parameters:

  • on (Union[str, Iterable[str]]) –

    A column (or columns) to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column. Passing several columns matches on the composite key: a source row updates a target row only when it agrees on every one of them.

Examples:

>>> import lancedb
>>> data = pa.table({"a": [2, 1, 3], "b": ["a", "b", "c"]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> new_data = pa.table({"a": [2, 3, 4], "b": ["x", "y", "z"]})
>>> # Perform a "upsert" operation
>>> res = table.merge_insert("a")     \
...      .when_matched_update_all()     \
...      .when_not_matched_insert_all() \
...      .execute(new_data)
>>> res
MergeResult(version=2, num_updated_rows=2, num_inserted_rows=1, num_deleted_rows=0, num_attempts=1, num_rows=3)
>>> # The order of new rows is non-deterministic since we use
>>> # a hash-join as part of this operation and so we sort here
>>> table.to_arrow().sort_by("a").to_pandas()
   a  b
0  1  b
1  2  x
2  3  y
3  4  z

search

search(query: Optional[Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]] = None, vector_column_name: Optional[str] = None, query_type: QueryType = 'auto', ordering_field_name: Optional[str] = None, fts_columns: Optional[Union[str, List[str]]] = None, fast_search: Optional[bool] = None) -> LanceQueryBuilder

Create a search query to find the nearest neighbors of the given query vector. We currently support vector search and full-text search.

All query options are defined in LanceQueryBuilder.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> data = [
...    {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]},
...    {"original_width": 2000, "caption": "foo",  "vector": [0.5, 3.4, 1.3]},
...    {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]}
... ]
>>> table = db.create_table("my_table", data)
>>> query = [0.4, 1.4, 2.4]
>>> (table.search(query)
...     .where("original_width > 1000", prefilter=True)
...     .select(["caption", "original_width", "vector"])
...     .limit(2)
...     .to_pandas())
  caption  original_width           vector  _distance
0     foo            2000  [0.5, 3.4, 1.3]   5.220000
1    test            3000  [0.3, 6.2, 2.6]  23.089996

Parameters:

  • query (Optional[Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]], default: None ) –

    The targeted vector to search for.

    • default None. Acceptable types are: list, np.ndarray, PIL.Image.Image

    • If None then the select/where/limit clauses are applied to filter the table

  • vector_column_name (Optional[str], default: None ) –

    The name of the vector column to search.

    The vector column needs to be a pyarrow fixed size list type

    • If not specified then the vector column is inferred from the table schema

    • If the table has multiple vector columns then the vector_column_name needs to be specified. Otherwise, an error is raised.

  • query_type (QueryType, default: 'auto' ) –

    default "auto". Acceptable types are: "vector", "fts", "hybrid", or "auto"

    • If "auto" then the query type is inferred from the query;

      • If query is a list/np.ndarray then the query type is "vector";

      • If query is a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found.

    • If query is a string, then the query type is "vector" if the table has embedding functions else the query type is "fts"

Returns:

  • LanceQueryBuilder –

    A query builder object representing the query. Once executed, the query returns

    • selected columns

    • the vector

    • and also the "_distance" column which is the distance between the query vector and the returned vector.

take_offsets

take_offsets(offsets: list[int], *, with_row_id: bool = False) -> LanceTakeQueryBuilder

Take a list of offsets from the table.

Offsets are 0-indexed and relative to the current version of the table. Offsets are not stable. A row with an offset of N may have a different offset in a different version of the table (e.g. if an earlier row is deleted).

Offsets are mostly useful for sampling as the set of all valid offsets is easily known in advance to be [0, len(table)).

No guarantees are made regarding the order in which results are returned. Repeated offsets produce repeated rows, which makes this method suitable for sampling with replacement.

Parameters:

  • offsets (list[int]) –

    The offsets to take.

  • with_row_id (bool, default: False ) –

    If True, include the "_rowid" column in the results.

Returns:

  • LanceTakeQueryBuilder –

    A query builder that can be executed to get the rows at the given offsets.

take_row_ids

take_row_ids(row_ids: list[int], *, with_row_id: bool = False) -> LanceTakeQueryBuilder

Take a list of row ids from the table.

Row ids are not stable and are relative to the current version of the table. They can change due to compaction and updates.

Results follow the requested row-id order. Repeated IDs produce repeated rows; missing or deleted IDs are omitted. This also applies to remote tables.

Unlike offsets, row ids are not 0-indexed and no assumptions should be made about the possible range of row ids. In order to use this method you must first obtain the row ids by scanning or searching the table.

Even so, row ids are more stable than offsets and can be useful in some situations.

There is an ongoing effort to make row ids stable which is tracked at https://github.com/lancedb/lancedb/issues/1120

Parameters:

  • row_ids (list[int]) –

    The row ids to take.

  • with_row_id (bool, default: False ) –

    If True, include the "_rowid" column in the results.

Returns:

  • LanceTakeQueryBuilder –

    A query builder that can be executed to get the rows.

blob_columns

blob_columns() -> list[str]

Names of blob v1 and v2 columns that can be fetched by row id.

Struct-nested blobs use dotted paths. Blobs inside lists are omitted because a row id does not identify an individual list element.

fetch_blobs

fetch_blobs(column: str, row_ids: Union[list[int], Array, ChunkedArray, Table]) -> LargeBinaryArray

Materialize full blob bytes for column at the given rows.

LanceDB Embedded supports both legacy blob v1 and blob v2 columns. The result has the same length and order as row_ids. Null blobs produce null slots; valid empty blobs produce b"".

row_ids may be a list, an integer Arrow array or chunked array, or a query table with row ids. _rowid values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial results are returned.

Blobs inside lists cannot be fetched by row id yet.

LanceDB Enterprise limits each request to 1024 row IDs and 64 MiB of blob bytes. The client splits requests automatically and reads an individual larger blob through the Range route. This method still materializes all bytes in memory; for large values use :meth:fetch_blob_files.

The returned Arrow large_binary array contains payload bytes, not SQL/query descriptor dictionaries. Use .to_pylist() to obtain Python bytes or None values for copying into a blob column. Select row IDs with search().with_row_id(True) first. Selection and fetching are separate requests: keep the source unchanged between them.

fetch_blob_ranges

fetch_blob_ranges(column: str, requests: Sequence[Sequence[int]]) -> LargeBinaryArray

Materialize row-specific byte ranges from a blob column.

LanceDB Embedded supports both legacy blob v1 and blob v2 columns. Each request is a three-item (row_id, offset, length) sequence. Requests may be repeated or reordered, including multiple ranges for the same blob. The result has the same length and order as requests; null blobs produce null slots and empty ranges on non-null blobs produce b"".

_rowid values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial results are returned.

Row IDs can be obtained from a query with with_row_id(True). LanceDB Enterprise reads the requested ranges through version-pinned blob handles using HTTP byte-range requests.

fetch_blob_files

fetch_blob_files(column: str, row_ids: Union[list[int], Array, ChunkedArray, Table]) -> 'list[Optional[BlobFile]]'

Open lazy, seekable :class:~lancedb._blob.BlobFile handles.

LanceDB Embedded supports both legacy blob v1 and blob v2 columns. Prefer this over :meth:fetch_blobs for large payloads. row_ids is a list[int], an integer Arrow array or chunked array, or a query pyarrow.Table carrying row identity via _rowid or a _lance_row_id field on the blob descriptor. Null blob values return None. LanceDB Enterprise requires server version 0.5.0 or newer.

_rowid values stay valid after compaction when the table has stable row ids. One missing or deleted row ID rejects the entire batch; no partial handles are returned.

open_blob_file

open_blob_file(descriptor: Union[dict, StructScalar, None]) -> Optional[BlobFile]

Open a seekable file from a SQL blob_descriptor result.

Accepts the descriptor dictionary or an Arrow struct scalar. Its source pins the table version, branch, column and row identity, so opening does not change this table's checkout. Null descriptors return None. Payload bytes load only when the returned handle is read. An external blob with an unspecified size may require a metadata request on opening.

rows = db.execute_query(
    "SELECT blob_descriptor(video) AS video FROM videos"
).read_all()
with videos.open_blob_file(rows["video"][0]) as video:
    header = video.read(4096)

open_blob_files

open_blob_files(descriptors: Iterable) -> Iterator[Optional[BlobFile]]

Iterate over lazy files from descriptor dictionaries or Arrow scalars.

Accepts an Arrow array/chunked array or any iterable. Opens each handle as iteration advances, preserving input order and nulls as None. Close each non-null handle after use, normally with a with block.

for video in videos.open_blob_files(rows["video"]):
    if video is not None:
        with video:
            header = video.read(4096)

delete

delete(where: Union[str, Expr]) -> DeleteResult

Delete rows from the table.

This can be used to delete a single row, many rows, all rows, or sometimes no rows (if your predicate matches nothing).

Parameters:

  • where (Union[str, Expr]) –

    The filter condition. Can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

    The filter must not be empty, or it will error.

Returns:

  • DeleteResult –

    An object containing the new version number of the table after deletion.

Examples:

>>> import lancedb
>>> data = [
...    {"x": 1, "vector": [1.0, 2]},
...    {"x": 2, "vector": [3.0, 4]},
...    {"x": 3, "vector": [5.0, 6]}
... ]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
   x      vector
0  1  [1.0, 2.0]
1  2  [3.0, 4.0]
2  3  [5.0, 6.0]
>>> table.delete("x = 2")
DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas()
   x      vector
0  1  [1.0, 2.0]
1  3  [5.0, 6.0]

If you have a list of values to delete, you can combine them into a stringified list and use the IN operator:

>>> to_remove = [1, 5]
>>> to_remove = ", ".join([str(v) for v in to_remove])
>>> to_remove
'1, 5'
>>> table.delete(f"x IN ({to_remove})")
DeleteResult(num_deleted_rows=1, version=3)
>>> table.to_pandas()
   x      vector
0  3  [5.0, 6.0]

update

update(where: Optional[Union[str, Expr]] = None, values: Optional[dict] = None, *, values_sql: Optional[Dict[str, str]] = None) -> UpdateResult

This can be used to update zero to all rows depending on how many rows match the where clause. If no where clause is provided, then all rows will be updated.

Either values or values_sql must be provided. You cannot provide both.

Embeddings are not computed by this method. Updating an embedding source column requires providing all associated vector columns in the same call; otherwise a ValueError is raised. Use merge_insert to recompute embeddings automatically.

Parameters:

  • where (Optional[Union[str, Expr]], default: None ) –

    The filter condition. Can be a SQL string or a type-safe Expr built with col and lit. The filter must not be empty, or it will error.

  • values (Optional[dict], default: None ) –

    The values to update. The keys are the column names and the values are the values to set.

  • values_sql (Optional[Dict[str, str]], default: None ) –

    The values to update, expressed as SQL expression strings. These can reference existing columns. For example, {"x": "x + 1"} will increment the x column by 1.

Returns:

  • UpdateResult –
    • rows_updated: The number of rows that were updated
    • version: The new version number of the table after the update

Examples:

>>> import lancedb
>>> from lancedb.expr import col
>>> import pandas as pd
>>> data = pd.DataFrame({"x": [1, 2, 3], "vector": [[1.0, 2], [3, 4], [5, 6]]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
   x      vector
0  1  [1.0, 2.0]
1  2  [3.0, 4.0]
2  3  [5.0, 6.0]
>>> table.update(where=col("x") == 2, values={"vector": [10.0, 10]})
UpdateResult(rows_updated=1, version=2)
>>> table.to_pandas()
   x        vector
0  1    [1.0, 2.0]
1  3    [5.0, 6.0]
2  2  [10.0, 10.0]
>>> table.update(values_sql={"x": "x + 1"})
UpdateResult(rows_updated=3, version=3)
>>> table.to_pandas()
   x        vector
0  2    [1.0, 2.0]
1  4    [5.0, 6.0]
2  3  [10.0, 10.0]

cleanup_old_versions

cleanup_old_versions(older_than: Optional[timedelta] = None, *, delete_unverified: bool = False) -> 'CleanupStats'

Clean up old versions of the table, freeing disk space.

Parameters:

  • older_than (Optional[timedelta], default: None ) –

    The minimum age of the version to delete. If None, then this defaults to two weeks.

  • delete_unverified (bool, default: False ) –

    Because they may be part of an in-progress transaction, files newer than 7 days old are not deleted by default. If you are sure that there are no in-progress transactions, then you can set this to True to delete all files older than older_than.

Returns:

  • CleanupStats –

    The stats of the cleanup operation, including how many bytes were freed.

See Also

Table.optimize: A more comprehensive optimization operation that includes cleanup as well as other operations.

Notes

This function is not available in LanceDB Enterprise, which manages cleanup for you automatically.

compact_files

compact_files(*args, **kwargs)

Run the compaction process on the table. This can be run after making several small appends to optimize the table for faster reads.

Arguments are passed onto Lance's lance.dataset.DatasetOptimizer.compact_files. For most cases, the default should be fine.

See Also

Table.optimize: A more comprehensive optimization operation that includes cleanup as well as other operations.

Notes

This function is not available in LanceDB Enterprise, which manages compaction for you automatically.

optimize

optimize(*, cleanup_older_than: Optional[timedelta] = None, delete_unverified: bool = False, retrain: bool = False, compaction_options: Optional[CompactionOptions] = None)

Optimize the on-disk data and indices for better performance.

.. note::

On a LanceDB Enterprise (remote) table, only compaction is
currently supported. Pruning old versions and index optimization
are silently skipped there instead of being attempted, so this
call returns successfully, with zeroed pruning statistics, once
the compaction job has been submitted; it does not wait for that
job to finish. Remote compaction supports per-job
``target_rows_per_fragment``, ``max_source_rows``, ``io_buffer_size``,
and ``batch_size``. Omitted fields inherit table tuning, then
deployment/built-in defaults. Other non-default compaction settings raise
``NotImplementedError`` before submission. The server must support
these SQL options; older-server errors are propagated without
dropping the requested settings.

Modeled after VACUUM in PostgreSQL.

Optimization covers three operations:

  • Compaction: Merges small files into larger ones
  • Prune: Removes old versions of the dataset
  • Index: Optimizes the indices, adding new data to existing indices

Parameters:

  • cleanup_older_than (Optional[timedelta], default: None ) –

    All files belonging to versions older than this will be removed. Set to 0 days to remove all versions except the latest. The latest version is never removed.

    .. warning::

    Setting this to 0 deletes the data files of every older
    version, so any other reader or writer still using an older
    version of the table will fail. Only set it to 0 if no other
    process is working on this dataset.
    
  • delete_unverified (bool, default: False ) –

    Files leftover from a failed transaction may appear to be part of an in-progress operation (e.g. appending new data) and these files will not be deleted unless they are at least 7 days old. If delete_unverified is True then these files will be deleted regardless of their age.

    .. warning::

    This should only be set to True if you can guarantee that no other
    process is currently working on this dataset. Otherwise the dataset
    could be put into a corrupted state.
    
  • retrain (bool, default: False ) –

    This parameter is no longer used and is deprecated.

  • compaction_options (Optional[CompactionOptions], default: None ) –

    Options that control file compaction. For large rows, derive a lower target_rows_per_fragment from the expected row size to bound later passes. This does not cap the first pass over an existing oversized fragment; see CompactionOptions.

Notes

The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations.

list_indices

list_indices() -> Iterable[IndexConfig]

List all indices that have been created with Table.create_index

On LanceDB Enterprise tables, an index whose build hasn't finished may be missing (vector indices) or listed with 0 indexed rows (scalar and FTS indices), and num_unindexed_rows may be None. Use wait_for_index to wait until an index is ready.

tokenize

tokenize(query: str, *, column: Optional[str] = None, index_name: Optional[str] = None) -> Iterable[FtsToken]

Tokenize a query using the tokenizer configured on an FTS index.

Specify exactly one of column or index_name.

Model-backed tokenizers such as jieba/* and lindera/* are rebuilt in the client process from index metadata. In LanceDB Enterprise, this means the same tokenizer model files must also exist locally (under LANCE_LANGUAGE_MODEL_HOME) and must match the files the server uses. A mismatch cannot be detected: the returned tokens may differ from what the server's index produces. If the model files are missing, an error naming the tokenizer is raised.

index_stats

index_stats(index_name: str) -> Optional[IndexStatistics]

Retrieve statistics about an index

Parameters:

  • index_name (str) –

    The name of the index to retrieve statistics for

Returns:

  • IndexStatistics or None –

    The statistics about the index. Returns None if the index does not exist.

add_columns

add_columns(transforms: Mapping[str, str | FunctionApplication] | FunctionApplication | Field | List[Field] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None = None, *, computed: Dict[str, str] | None = None, auto_refresh_enabled: bool | None = None, auto_refresh_interval_seconds: int | None = None, auto_refresh_resource_budget: Optional[Dict[str, int]] = None)

Add new columns with defined values.

Parameters:

  • transforms (Mapping[str, str | FunctionApplication] | FunctionApplication | Field | List[Field] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None, default: None ) –

    pa.Field, List[pa.Field], pa.Schema, pandas.DataFrame, polars.DataFrame, pa.Table, pa.RecordBatch, pa.RecordBatchReader A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, a pyarrow Field or Schema can be provided to add new columns with the specified data types. The new columns will be initialized with null values.

    A mapping with one FunctionApplication value keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come from rename(columns=...). Function columns are supported only in LanceDB Enterprise.

    On local tables, a pandas or Polars DataFrame, PyArrow Table, RecordBatch, or RecordBatchReader supplies precomputed new-column values. Supply exactly one row per current live table row, in storage scan order; pandas indexes are ignored. The input contains only new columns. Readers are consumed once, and their batch boundaries need not match table fragments. Keep the table unchanged between computing these values and calling add_columns. Data inputs cannot be combined with computed columns. Remote tables do not support data inputs.

  • computed (Dict[str, str] | None, default: None ) –

    LanceDB Embedded declaration and refresh require a source build with the experimental-functions Cargo feature, which published packages do not enable. LanceDB Enterprise operations remain available.

    A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order.

    Unlike transforms, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them from refresh_column. Declaring one therefore costs the same on a large table as on an empty one.

    A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped.

    In LanceDB Enterprise the expression is planned by the server, and the refresh runs as a server job -- see refresh_column_async. Cannot be combined with transforms.

  • auto_refresh_enabled (bool | None, default: None ) –

    Enable automatic refresh for each declared computed output. Defaults to True. LanceDB Embedded stores this option; scheduling requires a server.

  • auto_refresh_interval_seconds (int | None, default: None ) –

    Positive minimum interval after a successful refresh, in seconds. When omitted, the server determines when to refresh. Applies to every computed output in this call. Scheduling options require computed or Function columns.

  • auto_refresh_resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate CPU, memory, and GPU budget for automatic refreshes of a registered Function declaration. cpu_millis and memory_bytes are required; gpu_count defaults to zero. This option cannot be used with SQL-expression columns. LanceDB Enterprise only.

Returns:

  • AddColumnsResult –

    version: the new version number of the table after adding columns.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
>>> table.add_columns(computed={"doubled": "x * 2"})
AddColumnsResult(version=2)
>>> table.refresh_column("doubled")
RefreshColumnResult(rows_filled=2, version=4)
>>> table.to_arrow().sort_by("x").to_pandas()
   x  doubled
0  1        2
1  2        4
>>> import pyarrow as pa
>>> result = table.add_columns(pa.table({"score": [0.25, 0.75]}).to_reader())
>>> table.to_arrow()["score"].to_pylist()
[0.25, 0.75]

refresh_column

refresh_column(column: str, *, where: Optional[str] = None, batch_size: Optional[int] = None, max_concurrency: Optional[int] = None, publication_interval_rows: Optional[int] = None, resource_budget: Optional[Dict[str, int]] = None, gpu_class: Optional[str] = None, gpu_count: Optional[int] = None, retry_policy: Optional[FunctionRetryPolicy] = None, error_policy: Optional[Literal['fail', 'skip']] = None, max_failed_fraction: Optional[float] = None, max_failed_rows: Optional[int] = None, row_timeout_seconds: Optional[float] = None, init_timeout_seconds: Optional[float] = None, num_workers: Optional[int] = None, max_executor_processes_per_worker: Optional[int] = None) -> 'RefreshColumnResult'

Fill the rows of a computed column that hold no value yet.

Local refresh requires the experimental-functions Cargo feature in a source build. Published packages leave it disabled.

Declared with add_columns(computed=...), a column starts empty and gets its values here. Rows appended since the last refresh are filled by the next one, and rows whose inputs changed since they were computed are recomputed; everything else is left as it is.

Supported for local tables only. For remote tables, a refresh runs as a server job, through refresh_column_async.

Parameters:

  • column (str) –

    The name of the computed column to fill.

  • where (Optional[str], default: None ) –

    SQL filter selecting the rows to refresh, as in REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option.

  • batch_size (Optional[int], default: None ) –

    Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Has no effect on local expression-backed columns.

  • max_concurrency (Optional[int], default: None ) –

    Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.

  • publication_interval_rows (Optional[int], default: None ) –

    Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.

  • resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate resources for one registered Function refresh. The cpu_millis and memory_bytes keys are required; gpu_count is optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-level gpu_count shorthand. LanceDB Enterprise only; local refreshes reject it.

  • gpu_class (Optional[str], default: None ) –

    GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.

  • gpu_count (Optional[int], default: None ) –

    Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use resource_budget instead. LanceDB Enterprise only.

  • retry_policy (Optional[FunctionRetryPolicy], default: None ) –

    Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.

  • error_policy (Optional[Literal['fail', 'skip']], default: None ) –

    Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.

  • max_failed_fraction (Optional[float], default: None ) –

    Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.

  • max_failed_rows (Optional[int], default: None ) –

    Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.

  • row_timeout_seconds (Optional[float], default: None ) –

    Deadline for one Function call. LanceDB Enterprise only.

  • init_timeout_seconds (Optional[float], default: None ) –

    Deadline for Function initialization. LanceDB Enterprise only.

  • num_workers (Optional[int], default: None ) –

    Worker count for an expression column; registered Function columns reject it and use resource_budget instead. LanceDB Enterprise only.

  • max_executor_processes_per_worker (Optional[int], default: None ) –

    Cap on executor processes per Function worker. LanceDB Enterprise only.

Returns:

  • RefreshColumnResult –

    rows_filled: the number of rows given a value. version: the new version number of the table.

refresh_column_async

refresh_column_async(column: str, *, where: Optional[str] = None, batch_size: Optional[int] = None, max_concurrency: Optional[int] = None, publication_interval_rows: Optional[int] = None, resource_budget: Optional[Dict[str, int]] = None, gpu_class: Optional[str] = None, gpu_count: Optional[int] = None, retry_policy: Optional[FunctionRetryPolicy] = None, error_policy: Optional[Literal['fail', 'skip']] = None, max_failed_fraction: Optional[float] = None, max_failed_rows: Optional[int] = None, row_timeout_seconds: Optional[float] = None, init_timeout_seconds: Optional[float] = None, num_workers: Optional[int] = None, max_executor_processes_per_worker: Optional[int] = None) -> Job[RefreshColumnResult]

Like :meth:refresh_column, but returns a handle to the refresh job instead of blocking until it completes.

Local refresh requires a source build with experimental-functions. Remote jobs do not require this feature.

The job may already be complete when returned; callers must not assume the column is filled until :meth:Job.wait returns. Invalid input -- an unknown column, or one that is not computed -- raises here rather than failing the job. For local tables, the job runs in-process; for remote tables, it is the server's backfill job.

Parameters:

  • column (str) –

    The name of the computed column to fill.

  • where (Optional[str], default: None ) –

    SQL filter selecting the rows to refresh, as in REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option.

  • batch_size (Optional[int], default: None ) –

    Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Lower this for slow Functions to stay within the executor's per-call deadline. Has no effect on local expression-backed columns.

  • max_concurrency (Optional[int], default: None ) –

    Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.

  • publication_interval_rows (Optional[int], default: None ) –

    Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.

  • resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate resources for one registered Function refresh. The cpu_millis and memory_bytes keys are required; gpu_count is optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-level gpu_count shorthand. LanceDB Enterprise only; local refreshes reject it.

  • gpu_class (Optional[str], default: None ) –

    GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.

  • gpu_count (Optional[int], default: None ) –

    Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use resource_budget instead. LanceDB Enterprise only.

  • retry_policy (Optional[FunctionRetryPolicy], default: None ) –

    Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.

  • error_policy (Optional[Literal['fail', 'skip']], default: None ) –

    Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.

  • max_failed_fraction (Optional[float], default: None ) –

    Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.

  • max_failed_rows (Optional[int], default: None ) –

    Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.

  • row_timeout_seconds (Optional[float], default: None ) –

    Deadline for one Function call. LanceDB Enterprise only.

  • init_timeout_seconds (Optional[float], default: None ) –

    Deadline for Function initialization. LanceDB Enterprise only.

  • num_workers (Optional[int], default: None ) –

    Worker count for an expression column; registered Function columns reject it and use resource_budget instead. LanceDB Enterprise only.

  • max_executor_processes_per_worker (Optional[int], default: None ) –

    Cap on executor processes per Function worker. LanceDB Enterprise only.

Returns:

  • Job[RefreshColumnResult] –

    A job whose successful wait returns row counts plus the source and published table versions.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("computed_job_demo", [{"x": 1}, {"x": 2}])
>>> table.add_columns(computed={"doubled": "x * 2"})
AddColumnsResult(version=2)
>>> job = table.refresh_column_async("doubled")
>>> result = job.wait()
>>> result.rows_assigned
2
>>> job.status()
'finished'

function_errors

function_errors(job_id: Optional[str] = None, column: Optional[str] = None, limit: Optional[int] = None) -> 'FunctionErrors'

The per-row errors Function refreshes recorded on this table.

A refresh running under a skip policy records each row it skipped with the input that failed and the error. This lists those records, newest job first, plus a summary for any fragment whose per-row detail was capped. LanceDB Enterprise only; reading errors needs read access to the table, since a message carries the value that failed.

Parameters:

  • job_id (Optional[str], default: None ) –

    Only errors recorded by this job.

  • column (Optional[str], default: None ) –

    Only errors on this column.

  • limit (Optional[int], default: None ) –

    At most this many records (server default 10000, cap 100000).

Returns:

  • FunctionErrors –

    records, fragments and truncated, the last saying whether the listing stopped at its limit.

  • Each record identifies its row with ``fragment_id``, ``row_offset``, –
  • the physical ``row_address`` (``(fragment_id << 32) | –
  • row_offset``), and, when the table uses stable row IDs, the stable –
  • ``row_id`` read from the refresh's pinned table version. Prefer –
  • ``row_id`` for retries: compaction and rewrite can change the –
  • physical fields, while a stable row ID survives them. –
  • To retry the recorded rows with stable row IDs: –
  • .. code-block:: python –

    errors = table.function_errors(column="embedding") row_ids = sorted( record.row_id for record in errors.records if record.row_id is not None ) if row_ids: row_id_list = ", ".join(str(row_id) for row_id in row_ids) database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowid IN ({row_id_list})" )

  • For a table without stable row IDs, collect the non-null physical –
  • addresses before the table is compacted or rewritten and run the –
  • SQL refresh: –
  • .. code-block:: python –

    errors = table.function_errors(column="embedding") addresses = sorted( record.row_address for record in errors.records if record.row_address is not None ) if addresses: address_list = ", ".join(str(address) for address in addresses) database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowaddr IN ({address_list})" )

  • Records whose per-row detail was capped have no address and appear –
  • only in ``fragments``; they cannot be selected individually here. –

Examples:

>>> errors = table.function_errors(column="embedding")
>>> for record in errors.records:
...     print(record.job_id, record.row_offset, record.error_message)
>>> if errors.truncated:
...     print("listing stopped at the limit")

column_statuses

column_statuses(column: Optional[str] = None) -> list[FunctionColumnStatus]

Return exact status for Function-backed output columns.

All statuses describe one server table version. column=None returns every Function output. This operation is available on LanceDB Enterprise tables. A table handle pinned to an exact version requests that version; a latest handle retains the normal read-freshness behavior.

column_status

column_status(column: str) -> FunctionColumnStatus

Return exact status for one Function-backed output column.

wait_fresh

wait_fresh(column: str, timeout: Optional[timedelta] = None) -> FunctionColumnStatus

Poll until the column has no stale or never-computed rows.

Rows already failed remain visible in the returned status and do not keep this method waiting. The method never submits or cancels work. Transport errors and unrecoverable status-evidence errors propagate. A typed finalization-pending response is retried; arbitrary failure responses are not.

alter_columns

alter_columns(*alterations: Iterable[Dict[str, str]])

Alter column names and nullability.

Parameters:

  • alterations (Iterable[Dict[str, Any]], default: () ) –

    A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable.

    Each alteration must specify at least one of "rename", "data_type", or "nullable". The legacy key "name" is also accepted for renaming. Unknown keys raise ValueError before any alterations are applied.

    Local tables reject renaming source or vector columns used by an embedding function with ValueError, preserving the embedding binding.

Returns:

  • AlterColumnsResult –

    version: the new version number of the table after the alteration.

update_field_metadata

update_field_metadata(*updates: dict[str, Any]) -> UpdateFieldMetadataResult

Update per-field (column) metadata.

Parameters:

  • updates (dict, default: () ) –

    One or more dicts, each with:

    • "path": str — dot-path to the field (e.g. "embedding" or "a.b.c").
    • "metadata": dict[str, str | None] — keys to set; a value of None deletes that key.
    • "replace": bool, optional — replace the field's whole metadata map instead of merging (default False).

    The following keys are treated specially, by convention, and should be used when appropriate:

    • "lancedb:description": for a human-readable description of a field.
    • "lancedb:tag:<name>" for a user-defined key-value tag, where the suffix names the tag category; e.g. "lancedb:tag:model": "clip".
    • "lancedb:status" for status options ("production", "candidate", "deprecated", "archived") to designate the current life cycle state of this column.

Returns:

  • UpdateFieldMetadataResult –

    version: the new table version after the update.

replace_field_metadata

replace_field_metadata(field_name: str, new_metadata: Dict[str, str])

Replace all metadata of a field in the schema.

.. deprecated:: 0.33.1 Use update_field_metadata with replace=True instead.

Parameters:

  • field_name (str) –

    The name of the field whose metadata is replaced.

  • new_metadata (Dict[str, str]) –

    The new metadata, replacing all existing keys.

drop_columns

drop_columns(columns: Iterable[str]) -> DropColumnsResult

Drop columns from the table.

Local tables reject dropping source or vector columns used by an embedding function with ValueError, preserving the embedding binding.

Parameters:

  • columns (Iterable[str]) –

    The names of the columns to drop.

Returns:

  • DropColumnsResult –

    version: the new version number of the table dropping the columns.

checkout

checkout(version: Union[int, str])

Checks out a specific version of the Table

Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set.

This is a read-only operation that turns the table into a sort of "view" or "detached head". Other table instances will not be affected. To make the change permanent you can use the [Self::restore] method.

Any operation that modifies the table will fail while the table is in a checked out state. To return the table to a normal state use [Self::checkout_latest].

Parameters:

  • version (Union[int, str]) –

    The version to check out. A version number (int) or a tag (str) can be provided.

checkout_latest

checkout_latest()

Ensures the table is pointing at the latest version

This can be used to manually update a table when the read_consistency_interval is None It can also be used to undo a [Self::checkout] operation

restore

restore(version: Optional[Union[int, str]] = None)

Restore a version of the table. This is an in-place operation.

This creates a new version where the data is equivalent to the specified previous version. Data is not copied (as of python-v0.2.1).

Parameters:

  • version (int or str, default: None ) –

    The version number or version tag to restore. If unspecified then restores the currently checked out version. If the currently checked out version is the latest version then this is a no-op.

list_versions

list_versions() -> List[Dict[str, Any]]

List all versions of the table

uses_v2_manifest_paths

uses_v2_manifest_paths() -> bool

Check if the table is using the new v2 manifest paths.

Returns:

  • bool –

    True if the table is using the new v2 manifest paths, False otherwise.

migrate_v2_manifest_paths

migrate_v2_manifest_paths()

Migrate the manifest paths to the new format.

This will update the manifest to use the new v2 format for paths.

This function is idempotent, and can be run multiple times without changing the state of the object store.

Danger

This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations.

You can use Table.uses_v2_manifest_paths to check if the table is already using the new path style.

lancedb.table.CompactionOptions

Bases: TypedDict

Options that control file compaction during table optimization.

Unspecified options use Lance's defaults for local tables. Remote tables forward target_rows_per_fragment, max_source_rows, io_buffer_size, and batch_size as per-job SQL options; other non-default settings are unsupported. Omitted remote options inherit table tuning, then deployment/built-in defaults. The server must support these SQL options. These settings do not change table policy.

Compaction planning is row based. Lowering target_rows_per_fragment based on the expected row size can bound later compaction passes once oversized fragments have been rewritten. It does not split an existing fragment, so the first pass over an oversized fragment is not subject to that bound. max_bytes_per_file limits output file size, not compaction memory. Source budgets keep whole planned tasks; if the first task exceeds a budget, that run performs no compaction work.

Examples:

Derive a steady-state row target from the expected row size:

>>> desired_fragment_bytes = 750 * 1024 * 1024
>>> average_row_bytes = 1_500_000
>>> options: CompactionOptions = {
...     "target_rows_per_fragment": max(
...         1, desired_fragment_bytes // average_row_bytes
...     ),
... }
>>> await table.optimize(compaction_options=options)

target_rows_per_fragment

target_rows_per_fragment: int

Target rows per fragment; existing oversized fragments are not split.

max_rows_per_group

max_rows_per_group: int

Maximum number of rows per row group (default: 1,024).

max_bytes_per_file

max_bytes_per_file: Optional[int]

Maximum output data-file size; this does not bound compaction memory.

materialize_deletions

materialize_deletions: bool

Whether to rewrite fragments containing deleted rows (default: True).

materialize_deletions_threshold

materialize_deletions_threshold: float

Minimum deleted-row fraction that makes a fragment eligible (default: 0.1).

num_threads

num_threads: Optional[int]

Number of compaction tasks to run in parallel.

batch_size

batch_size: Optional[int]

Number of rows per input scan batch.

io_buffer_size

io_buffer_size: Optional[int]

Input scan I/O read-ahead budget in bytes.

Zero disables byte-based I/O backpressure.

defer_index_remap

defer_index_remap: bool

Whether to defer index remapping during compaction (default: False).

index_remap_mode

index_remap_mode: Literal['direct', 'compact']

How to construct the old-to-new row-address mapping.

compaction_mode

compaction_mode: Optional[Literal['reencode', 'try_binary_copy', 'force_binary_copy']]

Whether compaction re-encodes data or uses binary copying.

binary_copy_read_batch_bytes

binary_copy_read_batch_bytes: Optional[int]

Number of bytes read per batch during binary-copy compaction.

max_source_fragments

max_source_fragments: Optional[int]

Maximum number of source fragments compacted in one run.

max_source_rows

max_source_rows: Optional[int]

Maximum live source rows per run, applied to whole planned tasks.

max_source_bytes

max_source_bytes: Optional[int]

Maximum source bytes per run, applied to whole planned tasks.

excluded_fragment_ids

excluded_fragment_ids: List[int]

Fragment IDs to leave unchanged and use as planning boundaries.

max_overlays_per_fragment

max_overlays_per_fragment: Optional[int]

Maximum overlays before a fragment is fully compacted.

data_storage_version

data_storage_version: str

Data-file version or release selector for compacted output.

lancedb.table.FragmentStatistics

Statistics about fragments.

num_fragments

num_fragments: int

num_small_fragments

num_small_fragments: int

lengths

lengths: FragmentSummaryStats

lancedb.table.FragmentSummaryStats

Statistics about fragments sizes

min

min: int

max

max: int

mean

mean: int

p25

p25: int

p50

p50: int

p75

p75: int

p99

p99: int

lancedb.table.TableStatistics

Statistics about a table and fragments.

total_bytes

total_bytes: int

num_rows

num_rows: int

num_deleted_rows

num_deleted_rows: Optional[int]

num_indices

num_indices: int

fragment_stats

fragment_stats: FragmentStatistics

lancedb.table.Tags

Table tag manager.

list

list() -> Dict[str, Tag]

List all table tags.

Returns:

  • dict[str, Tag] –

    A dictionary mapping tag names to version numbers.

get_version

get_version(tag: str) -> int

Get the version of a tag.

Parameters:

  • tag (str) –

    The name of the tag to get the version for.

create

create(tag: str, version: int) -> None

Create a tag for a given table version.

Parameters:

  • tag (str) –

    The name of the tag to create. This name must be unique among all tag names for the table.

  • version (int) –

    The table version to tag.

delete

delete(tag: str) -> None

Delete tag from the table.

Parameters:

  • tag (str) –

    The name of the tag to delete.

update

update(tag: str, version: int) -> None

Update tag to a new version.

Parameters:

  • tag (str) –

    The name of the tag to update.

  • version (int) –

    The new table version to tag.

lancedb.table.Branches

Table branch manager.

list

list() -> Dict[str, Any]

List all branches, mapping name to branch metadata.

create

create(name: str, from_ref: Optional[str] = None, from_version: Optional[int] = None) -> 'Table'

Create a branch and return a handle scoped to it.

Parameters:

  • name (str) –

    Name of the new branch.

  • from_ref (Optional[str], default: None ) –

    Source branch to fork from. Defaults to main.

  • from_version (Optional[int], default: None ) –

    A specific version on from_ref to fork from. Defaults to latest.

checkout

checkout(name: str, version: Optional[int] = None) -> 'Table'

Check out an existing branch and return a handle scoped to it.

Parameters:

  • name (str) –

    Name of the branch to check out.

  • version (Optional[int], default: None ) –

    A specific version on the branch to pin. When set, the returned handle is a read-only view of that version; when omitted it tracks the branch's latest and stays writable.

delete

delete(name: str) -> None

Delete a branch.

diff

diff(from_branch: str) -> Dict[str, Any]

Diff a branch against main.

cherry_pick

cherry_pick(from_branch: str, dry_run: bool = False) -> Dict[str, Any]

Cherry-pick a branch onto main, or dry-run.

Parameters:

  • from_branch (str) –

    Branch to cherry-pick from.

  • dry_run (bool, default: False ) –

    When True, only preview. When False, attempt the cherry-pick.

Notes

A failed cherry-pick returns status="failed" instead of raising.

lancedb.RTEWriteSpec

Specification enabling the realtime engine (RTE), which routes merge_insert through Lance's MemWAL LSM-style write path.

Constructed via the bucket(...), identity(...), or unsharded() classmethods, then optionally chain with_maintained_indexes(...) and with_writer_config_defaults(...). A fresh spec maintains every index the table has, including ones created later.

column

column: Optional[str] = <attribute 'column' of 'lancedb._lancedb.RTEWriteSpec' objects>

Bucket and identity variants: the sharding column. None for unsharded.

maintained_indexes

maintained_indexes: Optional[List[str]] = <attribute 'maintained_indexes' of 'lancedb._lancedb.RTEWriteSpec' objects>

Indexes the RTE keeps up to date, or None for every supported one.

num_buckets

num_buckets: Optional[int] = <attribute 'num_buckets' of 'lancedb._lancedb.RTEWriteSpec' objects>

Bucket variant only: the number of buckets.

spec_type

spec_type: str = <attribute 'spec_type' of 'lancedb._lancedb.RTEWriteSpec' objects>

Discriminator string identifying the variant ("bucket", "identity", or "unsharded").

writer_config_defaults

writer_config_defaults: Dict[str, str] = <attribute 'writer_config_defaults' of 'lancedb._lancedb.RTEWriteSpec' objects>

Default ShardWriter configuration recorded by this spec.

bucket

bucket(column: str, num_buckets: int) -> RTEWriteSpec

Hash-bucket sharding by the unenforced primary key column.

identity

identity(column: str) -> RTEWriteSpec

Identity sharding — shard by the raw value of column.

column must be a deterministic function of the unenforced primary key: every row with a given primary key must always produce the same column value, or upserts of that key can land in different shards and a stale version can win. Typically column is the primary key itself or a stable attribute of it.

unsharded

unsharded() -> RTEWriteSpec

No sharding — every merge_insert call writes to a single RTE shard.

with_maintained_indexes

with_maintained_indexes(indexes: Optional[List[str]]) -> RTEWriteSpec

Set which indexes the RTE maintains. None (the default) is every index the table has, including ones created later; a list is verbatim, and an empty list maintains nothing.

with_writer_config_defaults

with_writer_config_defaults(defaults: Dict[str, str]) -> RTEWriteSpec

Replace the default ShardWriter configuration recorded in the MemWAL index, so every writer starts from the same defaults.

Functions and Jobs

A @function definition is also a local callable. Call a decorated function directly with row inputs, or call a decorated class with its initialization fields and then call the returned instance; call close() if the class defines it. This path uses the current Python environment and does not create a registered FunctionVersion, build the declared remote runtime, resolve Secrets, or execute the remote Arrow adapter. A registered FunctionVersion call instead binds col(...) inputs and returns a remote FunctionApplication.

from lancedb import function


@function
def normalize(text: str) -> str:
    return text.strip().lower()


@function
class Scale:
    def __init__(self, factor: float):
        self.factor = factor

    def __call__(self, value: float) -> float:
        return value * self.factor

    def close(self) -> None:
        self.factor = 0.0


assert normalize(" Hello ") == "hello"
scaler = Scale(factor=2.0)
assert scaler(3.0) == 6.0
scaler.close()

Annotate a blob input with lancedb.BlobReader for lazy, seekable access, or with bytes to materialize its contents before the Function runs. Reader inputs infer the Blob V2 type without an explicit input schema. Readers are scoped to each invocation and support read, readinto, seek, tell, size, and cursor-independent read_range. For a local test or another engine, construct BlobReader(read_range, size) with a callback over an in-memory bytes fixture or the engine's byte-range reader; read_range does not advance the cursor.

from lancedb import BlobReader

payload = b"0123456789"


def read_range(offset: int, length: int) -> bytes:
    return payload[offset : offset + length]


with BlobReader(read_range, len(payload)) as reader:
    assert reader.read_range(6, 4) == b"6789"
    assert reader.tell() == 0

lancedb.BlobReader

Bases: RawIOBase

Read a blob incrementally inside a Function.

Annotating a Function input with BlobReader requests a lazy, seekable reader. Annotating it with bytes requests materialized bytes instead. read_range does not change the sequential cursor. Readers are scoped to the invocation and must not be retained for a later call.

A packaged Function's source imports the worker's lance_blob.BlobReader in place of this class.

For local calls, construct a reader from a range-reading callable and the blob's size. The callable accepts an offset and length and returns bytes. This is also the interface for a reader backed by another engine.

Examples:

payload = b"hello world"

def read_range(offset, length):
    return payload[offset : offset + length]

with BlobReader(read_range, len(payload)) as reader:
    assert reader.read_range(6, 5) == b"world"
    assert reader.tell() == 0
    assert reader.read(5) == b"hello"
    assert reader.tell() == 5

read_range does not change the sequential cursor. Use size() for the declared payload size and close the reader when done; a local reader owns no server state.

readable

readable() -> bool

seekable

seekable() -> bool

size

size() -> int

Return the blob size without reading its payload.

tell

tell() -> int

seek

seek(offset: int, whence: int = io.SEEK_SET) -> int

read_range

read_range(offset: int, length: int) -> bytes

Read a byte range without changing the cursor; truncate at EOF.

read

read(size: int = -1) -> bytes

readinto

readinto(buffer) -> int

lancedb.functions.FunctionArtifact

Bases: _RemoteValue

Content-addressed Python artifact identity.

kind

kind: str

digest

digest: str

entrypoint

entrypoint: str

lancedb.functions.FunctionParameter

Bases: _RemoteValue

name

name: str

arrow_type

arrow_type: str

nullable

nullable: bool

blob_access

blob_access: Optional[Literal['bytes', 'reader']] = None

lancedb.functions.FunctionResultField

Bases: _OpenRemoteValue

name

name: str

arrow_type

arrow_type: str

nullable

nullable: bool

lancedb.functions.FunctionOutput

Bases: _OpenRemoteValue

Scalar or ordered named-struct output; unknown kinds remain decodable.

kind

kind: str

arrow_type

arrow_type: Optional[str] = None

nullable

nullable: Optional[bool] = None

fields

fields: tuple[FunctionResultField, ...] = ()

lancedb.functions.FunctionSignature

Bases: _RemoteValue

Ordered inputs, output, and initialization fields of a Function.

initialization lists the fields of the single row each Function instance is created with. Values come from each binding, not from the Function version; the tuple is empty when the Function takes none.

inputs

inputs: tuple[FunctionParameter, ...]

output

output: FunctionOutput

initialization

initialization: tuple[FunctionParameter, ...] = ()

lancedb.functions.PythonEnvironmentSpec

Bases: _RemoteValue

One LanceDB Enterprise-managed Python environment source.

kind

kind: str

packages

packages: tuple[str, ...] = ()

channels

channels: tuple[str, ...] = ()

path

path: Optional[str] = None

modules

modules: tuple[str, ...] = ()

image

image: Optional[str] = None

system_packages

system_packages: tuple[str, ...] = ()

lancedb.functions.function

function(target: Optional[Callable[..., Any]] = None, *, name: Optional[str] = None, input_schema: Optional[Schema] = None, output_schema: Optional[DataType | Field | Schema] = None, pip: tuple[str, ...] | list[str] = (), env: Optional[Mapping[str, str]] = None, python_version: Optional[str] = None, batch_size: Optional[int] = None, max_concurrency: Optional[int] = None, row_timeout_seconds: Optional[float] = None, retry_policy: Optional[FunctionRetryPolicy] = None, gpu: bool = False, gpu_class: Optional[str] = None, conda: tuple[str, ...] | list[str] = (), conda_channels: tuple[str, ...] | list[str] = (), apt: tuple[str, ...] | list[str] = (), image: Optional[str] = None, code: Sequence[ModuleType] = (), models: Sequence[HuggingFaceModel] = ())

Prepare a Python function or class for remote Function registration.

Input and output signatures are inferred from supported annotations. A parameter annotated T is non-nullable and binds only a non-nullable column; annotate Optional[T] to bind a nullable one, and it receives None for a null value. PyArrow fields are nullable unless declared otherwise, so most columns need Optional. For Arrow types annotations cannot express precisely, pass input_schema, output_schema, or both; the one omitted is inferred. Scalar outputs must be non-nullable. Every named-struct field may be nullable; Enterprise preserves the struct's validity when the result is expanded into sibling columns.

Large values. A bytes parameter receives the whole value, which is held in memory with its row and counts against the server's per-row input limit. To pass large binary values, such as video, store them in a Blob V2 column and annotate the parameter BlobReader, which streams.

Functions and classes. A decorated function is called once per row, or once per batch when its parameters are annotated pyarrow.Array or it takes one pyarrow.RecordBatch. These batch annotations require both input_schema and output_schema explicitly: the annotations do not specify Arrow element types or fields. This also applies to a class's __call__ method. A decorated class is the same callable with state: each remote instance calls __init__ once, calls __call__ for every row or batch of every input it processes, and calls close(), if defined, once when it retires. Load models, open clients, and build rate limiters in __init__; every batch of the instance then reuses them. An instance serves many batches of one binding, so state is shared across them, but not across instances or workers.

Initialization. The annotated parameters of a class's __init__ are the Function's initialization fields. Their values belong to each binding, not to the Function: function(text=col("body"), model="small") binds column body and initializes every instance with model="small", and another column can bind the same Function version with other values. Initialization fields take booleans, integers, floats, strings, and lists or structs of them (Annotated[dict, pa.struct(...)]). A parameter with a default may be omitted from a binding; a null value then takes the default. Use Secrets, not initialization, for credentials.

What travels with the Function. The artifact is a snapshot of the decorated source plus exactly the module-level names it references:

  • modules, and classes and functions importable from an installed package, become imports; the package must be declared in pip or conda;
  • literals (None, bool, int, str, bytes, finite float, and tuples containing only these literals) are inlined;
  • functions and classes defined in __main__, such as notebook cells, are packaged by source, together with what they reference;
  • modules and packages listed in code are shipped as Python source and imported normally, so helpers shared by several Functions live in one place.

A module imported by the packaged source -- including a direct import inside a function or method body -- draws a warning at registration when the client cannot resolve its top-level module and declares no package dependencies, or when it resolves to local source outside code and no declared requirement matches the imported module name. The warning does not assert that the remote environment lacks the module: pip/conda distribution names need not match the modules they install, and the client cannot inspect the contents of a base or custom image. Warnings follow normal Python warning filters and may be promoted to errors. Declare every third-party import in pip or conda even when it happens to be installed on the registration host. Other global values, including Arrow types and schemas, lists, dictionaries, sets, and NumPy arrays, cannot be captured as literals. Define them inside the function or class body instead. Closures, lambdas, nested definitions, mutable module-level objects, code that reaches the module namespace another way (globals()/eval, sys.modules, builtins), and a non-standard __builtins__ are rejected where they can be seen and otherwise unsupported. Changing any packaged source, including a module in code, produces a new Function version; the environment built from pip or conda is reused.

Parameters:

  • function (function or class) –

    The synchronous callable, or the class whose instances are called. For a function, annotate every parameter as pyarrow.Array to opt into vectorized execution; a matching return annotation documents that the callable returns an Arrow array.

  • name (str, default: None ) –

    The remote Function name. Defaults to the callable name. Must start with an ASCII letter or underscore and contain only ASCII letters, digits, and underscores.

  • input_schema (Schema, default: None ) –

    Explicit input fields in the exact order of the callable parameters. Required for batch annotations. When omitted, inputs are inferred from the parameter annotations.

  • output_schema (pyarrow.DataType, pyarrow.Field, or pyarrow.Schema, default: None ) –

    Explicit scalar or named-struct output. Scalar outputs must be non-nullable. Required for batch annotations. When omitted, the output is inferred from the return annotation.

  • pip (sequence of str, default: () ) –

    Pip requirements for the remote environment.

  • conda (sequence of str, default: () ) –

    Conda packages for the remote environment, instead of pip.

  • conda_channels (sequence of str, default: () ) –

    Conda channels in priority order; requires conda.

  • apt (sequence of str, default: () ) –

    Debian packages ("name" or "name=version") added to the remote image before the Python environment, for shared libraries pip and conda cannot provide, such as FFmpeg's for torchcodec. Packages are unpacked without running their maintainer scripts. Works with pip or conda; requires image execution.

  • image (str, default: None ) –

    A digest-pinned registry image ("host/repository@sha256:<digest>") the remote environment is built on instead of the default base. Pip requirements install only what the image lacks.

  • env (mapping of str to str, default: None ) –

    Environment variables included in the Function definition. Not for credentials -- these are ordinary configuration, stored with the Function and visible wherever it is. Prefer initialization for values that differ between bindings.

  • python_version (str, default: None ) –

    Remote Python major/minor version. Defaults to the client version.

  • batch_size (int, default: None ) –

    Maximum number of input rows supplied in one execution batch. Must be between 1 and 4294967295. Remote execution may use smaller batches. When omitted, the service chooses the batch size adaptively.

  • max_concurrency (int, default: None ) –

    Maximum user-code calls in flight for one Function within one logical execution. Must be between 1 and 4294967295. A refresh can override this default for that execution only. Omitting it preserves the existing behavior. This is not an account-wide quota.

  • row_timeout_seconds (float, default: None ) –

    Positive, finite deadline in seconds for each Python call: one row for scalar Functions and one batch for vectorized Functions. This does not govern class initialization. When omitted, the service applies its default per-call budget.

  • retry_policy (FunctionRetryPolicy, default: None ) –

    Retry selected failed calls inside the Function, with bounded attempts and backoff. A refresh may override this default. Retrying a partial call must be safe; completed calls are not repeated. For batch Functions, skip after exhaustion applies to the whole failed call.

  • gpu (bool, default: False ) –

    Whether every remote execution requires a GPU. The execution platform selects one compatible GPU for each worker. The requirement is part of the immutable Function version.

  • gpu_class (str, default: None ) –

    Operator-defined GPU class for every execution of this Function, such as "a10g" or "h100". Requires gpu=True and follows the lowercase 63-character DNS-label rule. The class is part of the immutable Function version; per-refresh aggregate GPU budgets are selected through refresh_column instead of registration.

  • code (sequence of module, default: () ) –

    Top-level modules or packages whose importable .py files ship with the Function and are importable by their own names. Package data files are not copied. To use a package that includes required data resources, install it with pip= or conda= and remove that package from code= so the copied Python package cannot shadow the installed distribution. @function warns when it skips files from a package in code=.

  • models (sequence of HuggingFaceModel, default: () ) –

    Immutable Hugging Face repository snapshots prepared in the worker node cache before user code starts. A Function may declare a repository once. Different revisions of one repository use different Function versions. Model Functions run offline with a private HF_HOME; bind HF_TOKEN through Secrets when the repository requires authentication.

Returns:

  • FunctionDefinition –

    A callable definition accepted by :meth:lancedb.db.DBConnection.create_function, :meth:lancedb.db.AsyncConnection.create_function_async and :meth:lancedb.db.DBConnection.create_function_async.

Examples:

>>> from lancedb import function
>>> @function(pip=["numpy==2.2.0"])
... def score(value: float) -> float:
...     return value * 2
>>> score(1.5)
3.0
>>> import pyarrow as pa
>>> @function(
...     input_schema=pa.schema([pa.field("value", pa.int64(), nullable=False)]),
...     output_schema=pa.int64(),
... )
... def batch_identity(value: pa.Array) -> pa.Array:
...     return value
>>> batch_identity(pa.array([1, 2, 3])).to_pylist()
[1, 2, 3]
>>> @function(pip=["cupy-cuda12x"], gpu=True, gpu_class="a10g")
... def gpu_score(value: int) -> int:
...     return value * 2
>>> gpu_score.registration_request.runtime.gpu
True
>>> gpu_score.registration_request.runtime.gpu_class
'a10g'
>>> @function
... class scale:
...     def __init__(self, factor: float, offset: float = 0.0):
...         self.factor, self.offset = factor, offset
...     def __call__(self, value: float) -> float:
...         return value * self.factor + self.offset
...     def close(self):
...         self.factor = 0.0
>>> scale_fn = scale(factor=2.0)
>>> scale_fn(1.5)
3.0
>>> scale_fn(2.0)
4.0
>>> scale_fn.close()
>>> [field.name for field in scale.registration_request.signature.initialization]
['factor', 'offset']
>>> import lancedb
>>> @function(
...     input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
...     output_schema=lancedb.blob("copy", nullable=False),
... )
... def copy_blobs(image: pa.Array) -> pa.Array:
...     return image

Local execution. A function definition is callable with its decorated input signature and runs the original function in the current Python environment. A class definition is called with its initialization arguments and constructs one instance; call that instance for row inputs, reuse it as needed, and call close if it defines that method. This path does not create a registered FunctionVersion, build the declared remote environment, resolve Secrets, or run the remote Arrow adapter or sandbox. A registered FunctionVersion call is different: it binds col(...) inputs and returns a remote FunctionApplication.

scaled = scale(factor=2.0)
try:
    assert scaled(1.5) == 3.0
    assert scaled(2.0) == 4.0
finally:
    scaled.close()

Blob v2 input is exposed as pyarrow.LargeBinaryArray in vectorized mode (and as bytes in row mode). Return a LargeBinaryArray with the declared nullability. Use :func:lancedb.blob in the explicit schemas; plain pyarrow.large_binary() does not carry Blob v2 semantics.

A Blob v2 field may sit at the top level or inside a struct in inputs and outputs. Only outputs may nest it in a list or large list (for example pa.list_(lancedb.blob("item")), returned as a list of bytes); fixed-size lists and maps are not supported. List-nested Blob v2 values are queryable as Blob descriptors, but the row-id blob fetch APIs and blob_mode="bytes" cannot fetch payloads inside lists.

lancedb.functions.FunctionDefinition

A Python callable prepared for remote Function registration.

Instances are created with :func:function. Calling an instance is ordinary local Python: a decorated function is called directly, while a decorated class is constructed using its initialization fields and the returned instance is called with each input. Reuse the instance and call its close method when it defines one. This public call does not create a remote application or build the declared remote runtime, and it does not need the private implementation attribute.

Remote execution adapts the callable to the internal Arrow batch ABI described by the registration artifact. By default it invokes the callable once per row. A callable whose parameters are all annotated as pyarrow.Array receives whole Arrow arrays and returns a pyarrow.Array instead. This authoring choice is independent of the internal artifact adapter in the registration request.

registration_request

registration_request: FunctionRegistrationRequest

The immutable request sent by create_function_async.

Carries no secret bindings. Binding is a registration-time decision, so a Function bound to Secrets is registered through :meth:bind_secrets, which is what create_function calls.

bind_secrets

bind_secrets(secrets: Optional[Sequence[EnvVarSecret]]) -> FunctionRegistrationRequest

The registration request for this definition bound to secrets.

Binding does not change the Function's source: each EnvVarSecret names a Secret and the environment variable its value should arrive in, and the Function reads that variable the way it already did. Whether the named Secrets exist is the server's answer, not this one.

lancedb.secrets.EnvVarSecret

A Secret bound to the environment variable a Function's library reads.

Pass these in the secrets sequence of DBConnection.create_function. The Function's source is unchanged by binding: it reads MODEL_API_KEY the way it always did, and the binding is what puts a value there.

This is a local value. Constructing it contacts no server, so it always succeeds and says nothing about whether the Secret exists; that is checked at registration, where a mistyped Secret name surfaces as a clear "does not exist" naming both the Secret and the variable bound to it. A mistyped variable name cannot be caught anywhere -- nothing knows which variables a Function reads -- so it surfaces on the first rows instead.

The type exists so a credential cannot be passed by accident. A bare string in the same position is a plausible-looking mistake with the opposite meaning, and it reads identically in a diff.

Parameters:

  • secret_name (str) –

    The Secret's database-scoped name.

  • env_variable (str) –

    The environment variable the Function reads it from.

  • secret_namespace_path (list of str, default: None ) –

    The namespace the Secret is addressed within. None and [] both mean the root namespace. Carried beside the name rather than joined into it, so neither is ever parsed back out of the other.

Examples:

>>> from lancedb import EnvVarSecret
>>> binding = EnvVarSecret(
...     secret_name="model-api-key", env_variable="MODEL_API_KEY"
... )
>>> binding.secret_name, binding.env_variable
('model-api-key', 'MODEL_API_KEY')

secret_name

secret_name: str

The Secret's database-scoped name.

env_variable

env_variable: str

The environment variable the value is delivered in.

secret_namespace_path

secret_namespace_path

The namespace path the Secret is addressed within, root when empty.

lancedb.secrets.SecretInfo

What a database records about a Secret. Never its value.

Returned by DBConnection.describe_secret.

name

name: str

The Secret's database-scoped name.

created_at_millis

created_at_millis: int

When the Secret was created, in milliseconds since the Unix epoch.

updated_at_millis

updated_at_millis: int

When the Secret's value was last rotated, in epoch milliseconds.

The only observable that a rotation landed: no API returns a credential, so a caller confirms alter_secret took effect by watching this move.

from_json

from_json(value: dict) -> 'SecretInfo'

lancedb.functions.FunctionRegistrationRequest

Bases: _RemoteValue

Stable remote registration envelope produced by :func:function.

Credential values deliberately have no field here. The only secret-shaped thing a client sends is secret_bindings: the name of a Secret the database already holds, which the remote service resolves at execution.

name

name: str

artifact

artifact: FunctionArtifactRequest

signature

signature: FunctionSignature

runtime

runtime: PythonRuntimeSpec

batch_size

batch_size: Optional[int] = None

max_concurrency

max_concurrency: Optional[int] = None

row_timeout_seconds

row_timeout_seconds: Optional[float] = None

retry_policy

retry_policy: Optional[FunctionRetryPolicy] = None

secret_bindings

secret_bindings: tuple[SecretBinding, ...] = ()

models

models: tuple[HuggingFaceModel, ...] = ()

lancedb.functions.FunctionRetryPolicy

Bases: _RemoteValue

Bound retries of selected errors inside one Enterprise Function call.

max_attempts includes the first call. Matchers are alternatives; transient=True selects connection/timeouts, HTTP 429/503 and the provider codes SlowDown and ServiceUnavailable. Exception names may be qualified, and subclasses and wrapped causes also match.

Backoff doubles with full jitter, bounded by max_backoff_seconds. Retry-After above that bound ends retries. A failed batch is retried as one call: under skip, exhaustion nulls every row in that call. Completed calls are never repeated. Only opt in when repeating partial work is safe; this does not make external side effects exactly once.

policy = FunctionRetryPolicy(transient=True, max_attempts=3) policy.max_attempts 3 disabled = FunctionRetryPolicy(max_attempts=1)

model_config

model_config = ConfigDict(extra='forbid', frozen=True)

max_attempts

max_attempts: Annotated[int, Field(strict=True)] = 3

initial_backoff_seconds

initial_backoff_seconds: Annotated[float, Field(strict=True)] = 0.2

max_backoff_seconds

max_backoff_seconds: Annotated[float, Field(strict=True)] = 30.0

exception_types

exception_types: tuple[str, ...] = ()

message_pattern

message_pattern: Optional[str] = None

error_codes

error_codes: tuple[str, ...] = ()

transient

transient: Annotated[bool, Field(strict=True)] = False

honor_retry_after

honor_retry_after: Annotated[bool, Field(strict=True)] = True

lancedb.functions.FunctionArtifactRequest

Bases: _RemoteValue

Source artifact uploaded while registering a Function.

kind

kind: str

digest

digest: str

entrypoint

entrypoint: str

content

content: FunctionArtifactContent

adapter

adapter: PythonAdapterSpec

lancedb.functions.FunctionArtifactContent

Bases: _RemoteValue

Encoded artifact bytes uploaded during remote registration.

encoding

encoding: str

data

data: str

lancedb.functions.PythonAdapterSpec

Bases: _RemoteValue

Internal scalar-callable to Arrow-batch adapter selection.

kind

kind: str

version

version: _UInt32

lancedb.functions.FunctionImage

Bases: _RemoteValue

A complete OCI Function image identified by its exact manifest digest.

manifest_digest

manifest_digest: str

descriptor

descriptor: Mapping[str, Any]

source

source: bool

models

models: tuple[HuggingFaceModel, ...] = ()

lancedb.functions.FunctionVersion

Bases: _RemoteValue

A pinned object revision, independent of its executable image digest.

Unknown top-level fields from newer servers are accepted for forward decoding, but they are not exposed on this value and are omitted from to_canonical_json(). Known fields retain their complete values, while unknown fields inside the open image.descriptor mapping remain available.

name and namespace_path are the Function's parts, as a [SecretReference][lancedb.functions.SecretReference] carries a Secret's; they are what get_function takes. An empty namespace_path is the root namespace.

source_digest identifies the registered source artifact, independently of the built image. Compare it with definition.registration_request.artifact.digest to check whether a local FunctionDefinition has the same source. It is None when the server does not provide a source digest.

gpu_class is the operator-defined GPU pool fixed into this Function version, or None when registration used the deployment default.

batch_size, max_concurrency, row_timeout_seconds, and result_stability retain the execution controls reported by the catalog. They are None for versions registered before the corresponding declaration.

name

name: str

namespace_path

namespace_path: tuple[str, ...] = ()

object_id

object_id: str

location

location: str

version

version: _ObjectVersion

image

image: FunctionImage

source_digest

source_digest: Optional[str] = None

signature

signature: FunctionSignature

secret_bindings

secret_bindings: tuple[SecretBinding, ...] = ()

created_at_millis

created_at_millis: int

metadata

metadata: Mapping[str, str]

disabled

disabled: bool

gpu_class

gpu_class: Optional[str] = None

batch_size

batch_size: Optional[int] = None

max_concurrency

max_concurrency: Optional[int] = None

row_timeout_seconds

row_timeout_seconds: Optional[float] = None

result_stability

result_stability: Optional[str] = None

lancedb.functions.HuggingFaceModel

Bases: _RemoteValue

One immutable Hugging Face model snapshot required by a Function.

revision is a full 40-character lowercase commit SHA, not a branch or tag. A Function may declare each repository once; separate versions of the same repository use separate Function versions. The service prepares the snapshot before initialization, so the model can load without a revision argument or a Hugging Face Hub request::

@function(
    pip=["sentence-transformers==3.3.1"],
    models=[HuggingFaceModel(
        repo="sentence-transformers/all-MiniLM-L6-v2",
        revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
    )],
)
class Embed:
    def __init__(self):
        from sentence_transformers import SentenceTransformer

        self.model = SentenceTransformer(
            "sentence-transformers/all-MiniLM-L6-v2", device="cpu"
        )

    def __call__(self, text: str) -> list[float]:
        return self.model.encode(text).tolist()

repo

repo: str = Field(pattern='^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$')

revision

revision: str

lancedb.functions.PythonRuntimeSpec

Bases: _RemoteValue

Remote runtime definition with environment values.

V1 supports kind="python". Newer runtime kinds remain readable, while their unknown payload fields are intentionally not retained by the client. kind="python_v2" requires gpu=True and may name an operator-defined gpu_class.

kind

kind: str

python_version

python_version: Optional[str] = None

environment

environment: Optional[PythonEnvironmentSpec] = None

env

env: Optional[Mapping[str, str]] = None

gpu

gpu: Optional[bool] = None

gpu_class

gpu_class: Optional[str] = None

lancedb.GpuClass

Configured hardware and worker limits of a Function GPU class.

cpu_millis

cpu_millis: Optional[int] = <attribute 'cpu_millis' of 'lancedb._lancedb.GpuClass' objects>

ephemeral_storage_bytes

ephemeral_storage_bytes: Optional[int] = <attribute 'ephemeral_storage_bytes' of 'lancedb._lancedb.GpuClass' objects>

gpu_memory_bytes

gpu_memory_bytes: Optional[int] = <attribute 'gpu_memory_bytes' of 'lancedb._lancedb.GpuClass' objects>

gpu_model

gpu_model: Optional[str] = <attribute 'gpu_model' of 'lancedb._lancedb.GpuClass' objects>

max_gpu_count_per_worker

max_gpu_count_per_worker: Optional[int] = <attribute 'max_gpu_count_per_worker' of 'lancedb._lancedb.GpuClass' objects>

memory_bytes

memory_bytes: Optional[int] = <attribute 'memory_bytes' of 'lancedb._lancedb.GpuClass' objects>

name

name: str = <attribute 'name' of 'lancedb._lancedb.GpuClass' objects>

resource_name

resource_name: str = <attribute 'resource_name' of 'lancedb._lancedb.GpuClass' objects>

lancedb.functions.FunctionVersionRef

Bases: _OpenRemoteValue

name

name: str

namespace_path

namespace_path: tuple[str, ...] = ()

object_id

object_id: str

location

location: str

version

version: _ObjectVersion

manifest_digest

manifest_digest: str

lancedb.functions.ApplicationInput

Bases: _OpenRemoteValue

One parameter value.

Slice 1 freezes integers, strings, booleans, nulls, arrays, and objects. Floating-point literal encoding is deferred until Python authoring is introduced with a language-neutral numeric representation.

parameter

parameter: str

kind

kind: str

value

value: Any

lancedb.functions.FunctionApplication

Bases: _OpenRemoteValue

Immutable pre-declaration application of an exact Function version.

A named-struct output remains one application through table declaration and execution. FunctionApplication.rename records the result-field to table-column mapping without splitting sibling outputs into separate UDF calls.

function

function: FunctionVersionRef

inputs

inputs: tuple[ApplicationInput, ...]

output

output: FunctionOutput

columns

columns: Mapping[str, str] = Field(default_factory=dict)

initialization

initialization: Mapping[str, Any] = Field(default_factory=dict)

rename

rename(*, columns: Mapping[str, str]) -> FunctionApplication

Return a copy with result-field to table-column aliases.

lancedb.functions.InputBinding

Bases: _RemoteValue

parameter

parameter: str

field_id

field_id: _Int32

field_path

field_path: str

arrow_type

arrow_type: str

nullable

nullable: bool

lancedb.functions.OutputMapping

Bases: _RemoteValue

One stable result-field mapping.

result_field

result_field: str

output_name

output_name: str

output_field_id

output_field_id: _Int32

output_ordinal

output_ordinal: _UInt32

arrow_type

arrow_type: str

nullable

nullable: bool

lancedb.functions.AssignmentMapping

Bases: _RemoteValue

Internal physical column preserving flattened struct validity.

output_name

output_name: str

output_field_id

output_field_id: _Int32

lancedb.functions.FunctionBinding

Bases: _RemoteValue

Immutable Function binding persisted by the Enterprise table service.

binding_id

binding_id: str

function

function: FunctionVersionRef

inputs

inputs: tuple[InputBinding, ...]

outputs

outputs: tuple[OutputMapping, ...]

assignment

assignment: Optional[AssignmentMapping] = None

input_schema

input_schema: Optional[Mapping[str, Any]] = None

output_schema

output_schema: Optional[Mapping[str, Any]] = None

initialization

initialization: Optional[str] = None

Standard base64 of the validated one-row Arrow IPC initialization stream.

initialization_row

initialization_row() -> Optional[dict[str, Any]]

The initialization values this binding creates instances with.

lancedb.functions.RefreshColumnResult

Bases: _RemoteValue

Terminal result of an expression-backed or Function-backed refresh Job.

LanceDB Embedded jobs produce this value in process. LanceDB Enterprise decodes the same value from the durable server-job terminal payload.

Row counters describe logical rows planned for this refresh, not user-code calls or retry attempts. rows_assigned counts rows successfully written with an output value, rows_failed counts rows whose computation failed, and rows_remaining counts planned rows neither assigned nor terminally skipped. Rows already counted in rows_failed are covered and are not counted again in rows_remaining.

For example, a refresh over 2,001 logical rows that assigns 1,891 values and fails 110 rows reports rows_assigned=1,891, rows_failed=110, and rows_remaining=0. Its failure-rate denominator is those 2,001 logical rows, not the number of execution attempts.

rows_assigned

rows_assigned: _UInt64

Number of logical rows successfully assigned an output value by this refresh. Each row counts once, regardless of retry attempts, and failed rows are not included.

rows_failed

rows_failed: _UInt64

Number of logical rows whose computation failed. Rows deferred because an input was not ready are not failures.

rows_remaining

rows_remaining: _UInt64

Number of planned logical rows neither assigned nor terminally skipped. Rows already counted in rows_failed are covered and are not counted again.

source_version

source_version: _UInt64

Exact table version the refresh read.

published_version

published_version: Optional[_UInt64] = None

Table version made visible by the refresh, when one was published.

rows_filled

rows_filled: int

Deprecated compatibility alias for :attr:rows_assigned.

The value is the same successful logical-row count; it does not count execution attempts.

version

version: Optional[int]

Deprecated compatibility alias for :attr:published_version.

lancedb.FunctionErrors

A table's per-row Function errors.

fragments

fragments: list[FunctionErrorFragment] = <attribute 'fragments' of 'lancedb._lancedb.FunctionErrors' objects>

records

records: list[FunctionErrorRecord] = <attribute 'records' of 'lancedb._lancedb.FunctionErrors' objects>

truncated

truncated: bool = <attribute 'truncated' of 'lancedb._lancedb.FunctionErrors' objects>

lancedb.FunctionErrorRecord

One row a Function refresh skipped, as the server recorded it.

attempts

attempts: Optional[int] = <attribute 'attempts' of 'lancedb._lancedb.FunctionErrorRecord' objects>

column

column: str = <attribute 'column' of 'lancedb._lancedb.FunctionErrorRecord' objects>

created_at_millis

created_at_millis: int = <attribute 'created_at_millis' of 'lancedb._lancedb.FunctionErrorRecord' objects>

error_message

error_message: str = <attribute 'error_message' of 'lancedb._lancedb.FunctionErrorRecord' objects>

error_type

error_type: str = <attribute 'error_type' of 'lancedb._lancedb.FunctionErrorRecord' objects>

fragment_id

fragment_id: int = <attribute 'fragment_id' of 'lancedb._lancedb.FunctionErrorRecord' objects>

function

function: str = <attribute 'function' of 'lancedb._lancedb.FunctionErrorRecord' objects>

function_version

function_version: str = <attribute 'function_version' of 'lancedb._lancedb.FunctionErrorRecord' objects>

job_id

job_id: str = <attribute 'job_id' of 'lancedb._lancedb.FunctionErrorRecord' objects>

row_address

row_address: Optional[int] = <attribute 'row_address' of 'lancedb._lancedb.FunctionErrorRecord' objects>

row_id

row_id: Optional[int] = <attribute 'row_id' of 'lancedb._lancedb.FunctionErrorRecord' objects>

row_offset

row_offset: Optional[int] = <attribute 'row_offset' of 'lancedb._lancedb.FunctionErrorRecord' objects>

table_version

table_version: int = <attribute 'table_version' of 'lancedb._lancedb.FunctionErrorRecord' objects>

traceback

traceback: Optional[str] = <attribute 'traceback' of 'lancedb._lancedb.FunctionErrorRecord' objects>

User-code traceback, possibly truncated by the server.

lancedb.FunctionErrorFragment

A fragment whose per-row error detail was capped.

fragment_id

fragment_id: int = <attribute 'fragment_id' of 'lancedb._lancedb.FunctionErrorFragment' objects>

job_id

job_id: str = <attribute 'job_id' of 'lancedb._lancedb.FunctionErrorFragment' objects>

rows_recorded

rows_recorded: int = <attribute 'rows_recorded' of 'lancedb._lancedb.FunctionErrorFragment' objects>

rows_skipped

rows_skipped: int = <attribute 'rows_skipped' of 'lancedb._lancedb.FunctionErrorFragment' objects>

lancedb.FunctionColumnStatus

Exact four-way status of one Function output at one table version.

auto_refresh

auto_refresh: FunctionColumnAutoRefreshStatus = <attribute 'auto_refresh' of 'lancedb._lancedb.FunctionColumnStatus' objects>

binding_id

binding_id: str = <attribute 'binding_id' of 'lancedb._lancedb.FunctionColumnStatus' objects>

column

column: str = <attribute 'column' of 'lancedb._lancedb.FunctionColumnStatus' objects>

failed

failed: int = <attribute 'failed' of 'lancedb._lancedb.FunctionColumnStatus' objects>

fresh

fresh: int = <attribute 'fresh' of 'lancedb._lancedb.FunctionColumnStatus' objects>

function

function: FunctionColumnVersionStatus = <attribute 'function' of 'lancedb._lancedb.FunctionColumnStatus' objects>

last_failure_job

last_failure_job: Optional[FunctionColumnLastRefreshJob] = <attribute 'last_failure_job' of 'lancedb._lancedb.FunctionColumnStatus' objects>

last_refresh_job

last_refresh_job: Optional[FunctionColumnLastRefreshJob] = <attribute 'last_refresh_job' of 'lancedb._lancedb.FunctionColumnStatus' objects>

never_computed

never_computed: int = <attribute 'never_computed' of 'lancedb._lancedb.FunctionColumnStatus' objects>

rows

rows: int = <attribute 'rows' of 'lancedb._lancedb.FunctionColumnStatus' objects>

stale

stale: int = <attribute 'stale' of 'lancedb._lancedb.FunctionColumnStatus' objects>

table_version

table_version: int = <attribute 'table_version' of 'lancedb._lancedb.FunctionColumnStatus' objects>

lancedb.FunctionColumnVersionStatus

Exact immutable Function revision attached to a table output.

location

location: str = <attribute 'location' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

manifest_digest

manifest_digest: str = <attribute 'manifest_digest' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

name

name: str = <attribute 'name' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

namespace_path

namespace_path: list[str] = <attribute 'namespace_path' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

object_id

object_id: str = <attribute 'object_id' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

version

version: str = <attribute 'version' of 'lancedb._lancedb.FunctionColumnVersionStatus' objects>

lancedb.FunctionColumnLastRefreshJob

Latest matching refresh Job projected for a Function output.

automatic

automatic: Optional[bool] = <attribute 'automatic' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>

completed_at_millis

completed_at_millis: Optional[int] = <attribute 'completed_at_millis' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>

failure

failure: Optional[JobFailureInfo] = <attribute 'failure' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>

job_id

job_id: str = <attribute 'job_id' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>

state

state: str = <attribute 'state' of 'lancedb._lancedb.FunctionColumnLastRefreshJob' objects>

lancedb.FunctionColumnAutoRefreshStatus

Automatic-refresh projection for a Function output.

active_job_id

active_job_id: Optional[str] = <attribute 'active_job_id' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

blocked_reason

blocked_reason: Optional[str] = <attribute 'blocked_reason' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

effective_cooldown_millis

effective_cooldown_millis: Optional[int] = <attribute 'effective_cooldown_millis' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

enabled

enabled: bool = <attribute 'enabled' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

failure_streak

failure_streak: int = <attribute 'failure_streak' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

interval_seconds

interval_seconds: Optional[int] = <attribute 'interval_seconds' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

next_run_at_millis

next_run_at_millis: Optional[int] = <attribute 'next_run_at_millis' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

next_run_at_reason

next_run_at_reason: Optional[str] = <attribute 'next_run_at_reason' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

state

state: str = <attribute 'state' of 'lancedb._lancedb.FunctionColumnAutoRefreshStatus' objects>

lancedb.job.Job

Bases: Generic[T]

Synchronous counterpart of AsyncJob with the same result type.

id

id: Optional[str]

Identifies the operation on the server that is running it.

See :attr:AsyncJob.id.

state

state: Optional[str]

The last observed lifecycle state. See :attr:AsyncJob.state.

job_type

job_type: Optional[str]

The job's type. See :attr:AsyncJob.job_type.

creation_ms

creation_ms: Optional[int]

When the job was created. See :attr:AsyncJob.creation_ms.

spec

spec: Optional[Any]

The job's specification. See :attr:AsyncJob.spec.

result

result: Optional[Any]

The job's terminal result. See :attr:AsyncJob.result.

failure

failure: Optional[JobFailureInfo]

Why the job failed. See :attr:AsyncJob.failure.

status

status() -> str

The operation's current lifecycle state: "running", "finished", "failed", or "cancelled".

See :meth:AsyncJob.status.

wait

wait(timeout: Optional[timedelta] = None, progress: Any = False) -> T

Block until the operation reaches a terminal state.

Returns the endpoint's typed result, or None for a unit-result job.

Raises JobFailedError if the operation failed, JobCancelledError if it was cancelled, and TimeoutError if timeout elapses first. Transient connection and read errors are retried within that timeout; if it expires while disconnected, raises ConnectionError with instructions to reconnect. Timing out does not cancel the job.

The timeout error identifies the job and explains that it continues on the server. Registry jobs include the db.open_job(id).wait() reattach call; SQL query handles must be kept because their IDs cannot be reopened through the registry jobs API.

See :meth:AsyncJob.wait.

progress

progress() -> Optional[dict[str, Any]]

Fetch a structured progress snapshot. See :meth:AsyncJob.progress.

cancel

cancel()

Request cancellation. Cancelling a finished operation is a no-op.

refresh

refresh() -> None

Ask the backend for this job's current state and record.

See :meth:AsyncJob.refresh.

export_timeline

export_timeline(path: str | PathLike[str]) -> None

Write Chrome trace JSON for this Function job.

See :meth:AsyncJob.export_timeline. Open the file in Perfetto or Chrome tracing.

Examples:

>>> job.export_timeline("refresh.trace.json")

events

events(*, limit: Optional[int] = None, filter: Optional[str] = None) -> Table

This job's recorded lifecycle events.

See :meth:AsyncJob.events.

lancedb.job.AsyncJob

Bases: Generic[T]

A handle to an operation that may still be running.

The operation may already be complete when the handle is created. T is the endpoint's terminal result type; unit-result jobs resolve to None.

id

id: Optional[str]

The operation's opaque ID, interpreted according to :attr:job_type.

Registry jobs expose a registry job ID; SQL-backed jobs expose a connection-scoped query ID as a string. Local operations return None. Query IDs cannot be used with registry job APIs or to resume after a client restart.

state

state: Optional[str]

The last observed lifecycle state, without contacting the backend.

None until the handle has talked to it. See :meth:AsyncJob.refresh.

job_type

job_type: Optional[str]

The job's kind of work, without contacting the backend.

SQL-backed handles report "query" immediately. Registry handles report the server's job type after :meth:refresh or :meth:status. In-process handles return None.

creation_ms

creation_ms: Optional[int]

When the job was created, in milliseconds since the epoch.

spec

spec: Optional[Any]

The job-type-specific specification it was submitted with.

result

result: Optional[Any]

The job-type-specific terminal result, as reported data rather than the typed model :meth:AsyncJob.wait returns.

None until the job succeeds, so a job that never terminates reports its progress through :meth:AsyncJob.events instead.

failure

failure: Optional[JobFailureInfo]

Why the job failed, when it failed and the server reports a reason.

status

status() -> str

The operation's current lifecycle state: "running", "finished", "failed", or "cancelled".

A point snapshot; unlike wait it does not block or raise on a terminal failure state. States a newer server reports that this client version does not know pass through as-is.

progress

progress() -> Optional[dict[str, Any]]

Fetch a structured progress snapshot.

Registry jobs always report lifecycle fields. Queue metadata and counters are included when supplied by the server. Older servers may omit counters, and in-process operations return None. A missing counter is unknown rather than zero. This is additive to :meth:status, whose string contract is unchanged.

wait

wait(timeout: Optional[timedelta] = None, progress: Any = False) -> T

Wait until the operation reaches a terminal state.

Returns the endpoint's typed result, or None for a unit-result job.

progress=True renders a progress bar, progress=callable receives each structured snapshot, and a tqdm-like object is updated in place. Observations use the backend's existing wait poller and never start a second job. The bar advances by remaining rows when available, because completed computation can include repeated work after a plan change. On each wait call, the first complete observation synchronizes a caller-owned bar with the server's current completed rows and total; later observations in a stable execution epoch use a high-water mark. A new execution epoch or smaller plan resets the bar to that plan. False preserves the original wait behavior.

Raises JobFailedError if the operation failed, JobCancelledError if it was cancelled, and TimeoutError if timeout elapses first. Transient connection and read errors are retried with capped exponential backoff within the same timeout (indefinitely when no timeout is given). If the timeout expires while disconnected, raises ConnectionError instead, with the job ID and instructions to resume waiting. The job may still be running; timing out does not cancel it.

The timeout error identifies the job and explains that it continues on the server. Registry jobs include the db.open_job(id).wait() reattach call; SQL query handles must be kept because their IDs cannot be reopened through the registry jobs API.

cancel

cancel()

Request cancellation. Cancelling a finished operation is a no-op.

refresh

refresh() -> None

Ask the backend for this job's current state, and for a server-side job its full record, then cache it for the properties below.

Registry details are None until this runs, because submitting an operation does not fetch its full registry record. Query-backed jobs expose only lifecycle state through this method. For registry jobs, status fetches the whole record too; wait records only the terminal state it establishes.

export_timeline

export_timeline(path: str | PathLike[str]) -> None

Write a fresh recorded Function job timeline as Chrome trace JSON.

Open the file in Perfetto or Chrome tracing. The recording contains retained split spans grouped by worker and timestamped job phases. Batch counts and latency statistics are included as details; individual batch timestamps are not recorded. Running jobs export a partial snapshot without waiting or cancelling. Download again to include later work.

Requires a server-side Function job and a server supporting timeline export. Unsupported operations raise NotImplementedError or a server error. The destination is overwritten only after the trace is fetched. Filesystem errors propagate to the caller.

Examples:

>>> await job.export_timeline("refresh.trace.json")

events

events(*, limit: Optional[int] = None, filter: Optional[str] = None) -> Table

This job's recorded lifecycle events.

Where the properties above report a terminal result only once the job reaches one, events are written as the job runs and outlive the workers that produced them. A distributed job records a claim/claim_complete pair per unit of work, each carrying rows_processed, so a job that never finishes still accounts for what it did.

Parameters:

  • limit (Optional[int], default: None ) –

    Maximum event rows to return. The server caps results at 1000 by default and 10,000 at most, and truncates without saying so, so pass this for a job that emits an event per fragment.

  • filter (Optional[str], default: None ) –

    SQL-like expression over the state, updated_by, emitted_from, emitted_by, and claim_entity columns, such as state = 'claim_complete'.

lancedb.job.JobInfo

A row from Connection.list_jobs: one server-side job.

created_at_millis

created_at_millis: int = <attribute 'created_at_millis' of 'lancedb._lancedb.JobInfo' objects>

job_id

job_id: str = <attribute 'job_id' of 'lancedb._lancedb.JobInfo' objects>

job_type

job_type: str = <attribute 'job_type' of 'lancedb._lancedb.JobInfo' objects>

paused

paused: bool = <attribute 'paused' of 'lancedb._lancedb.JobInfo' objects>

Whether an operator has parked this job.

progress

progress: Optional[dict[str, Any]] = <attribute 'progress' of 'lancedb._lancedb.JobInfo' objects>

Structured progress for job types that report it. Missing fields are unknown, not zero.

state

state: str = <attribute 'state' of 'lancedb._lancedb.JobInfo' objects>

table

table: str = <attribute 'table' of 'lancedb._lancedb.JobInfo' objects>

lancedb.job.JobDescription

The server-side record behind a Job handle.

creation_ms

creation_ms: int = <attribute 'creation_ms' of 'lancedb._lancedb.JobDescription' objects>

failure

failure: Optional[JobFailureInfo] = <attribute 'failure' of 'lancedb._lancedb.JobDescription' objects>

job_id

job_id: str = <attribute 'job_id' of 'lancedb._lancedb.JobDescription' objects>

job_type

job_type: str = <attribute 'job_type' of 'lancedb._lancedb.JobDescription' objects>

progress

progress: Optional[dict[str, Any]] = <attribute 'progress' of 'lancedb._lancedb.JobDescription' objects>

Structured progress snapshot. Lifecycle and queue fields are present even when a job type or older server supplies no counters.

result

result: Optional[Any] = <attribute 'result' of 'lancedb._lancedb.JobDescription' objects>

The job-type-specific terminal result. None until the job succeeds.

spec

spec: Optional[Any] = <attribute 'spec' of 'lancedb._lancedb.JobDescription' objects>

The job-type-specific specification it was submitted with.

state

state: str = <attribute 'state' of 'lancedb._lancedb.JobDescription' objects>

lancedb.job.JobFailureInfo

The server's account of why a job failed.

message

message: Optional[str] = <attribute 'message' of 'lancedb._lancedb.JobFailureInfo' objects>

phase

phase: Optional[str] = <attribute 'phase' of 'lancedb._lancedb.JobFailureInfo' objects>

retryable

retryable: Optional[bool] = <attribute 'retryable' of 'lancedb._lancedb.JobFailureInfo' objects>

sql_job_id

sql_job_id: Optional[str] = <attribute 'sql_job_id' of 'lancedb._lancedb.JobFailureInfo' objects>

traceback

traceback: Optional[str] = <attribute 'traceback' of 'lancedb._lancedb.JobFailureInfo' objects>

User-code traceback, possibly truncated by the server.

lancedb.sql.Query

Synchronous counterpart of :class:AsyncQuery.

id

id: UUID

The stable identifier scoped to the connection that submitted it.

job_id

job_id: Optional[str]

The durable Job Registry id once the server reports this query as a job.

A running query reports it after :meth:describe observes the latest server status. This is separate from the connection-scoped id.

describe

describe() -> QueryDescription

Get a point-in-time description of the query.

reader

reader() -> RecordBatchReader

Wait for the initial result stream and return a blocking reader.

Results are single-consumer. Calling this method more than once on the same query raises an error. Later batches block only until they become available, without waiting for the full query to finish.

cancel

cancel() -> None

Request cancellation of this query's Flight execution.

A durable server-side job may reject this operation because its Flight waiter cannot cancel the registry job. Use connection.cancel_job() with job_id to cancel that job.

lancedb.sql.AsyncQuery

A handle to a submitted SQL query on an asynchronous connection.

id

id: UUID

The stable identifier scoped to the connection that submitted it.

job_id

job_id: Optional[str]

The durable Job Registry id once the server reports this query as a job.

A running query reports it after :meth:describe observes the latest server status. This is separate from the connection-scoped id.

describe

describe() -> QueryDescription

Get a point-in-time description of the query.

reader

reader() -> AsyncRecordBatchReader

Wait for the initial result stream and return its Arrow reader.

Results are single-consumer. Calling this method more than once on the same query raises an error. Later batches are streamed as they become available without waiting for the full query to finish.

cancel

cancel() -> None

Request cancellation of this query's Flight execution.

A durable server-side job may reject this operation because its Flight waiter cannot cancel the registry job. Use connection.cancel_job() with job_id to cancel that job.

lancedb.sql.QueryDescription

QueryDescription = QueryDescription

lancedb.sql.QueryParameters

QueryParameters = Union[RecordBatch, Table, Sequence[Any], Mapping[str, Any]]

Values for a statement's $1 / $name placeholders.

A sequence binds by position ($1, $2, ...), a mapping binds by name ($name), and a one-row pyarrow.RecordBatch or pyarrow.Table binds both ways: column i is $<i + 1> and, when its name is not a number, also $<name>.

Materialized Views (Synchronous)

In the 0.41 SDK release, LanceDB Embedded materialized-view creation and refresh require a source build with the experimental-functions Cargo feature. Published Python packages leave it disabled. Opening, reading, listing and dropping existing views remain available, as do LanceDB Enterprise operations. The same feature requirement applies to LanceDB Embedded SQL computed-column declaration and refresh; ordinary eager add_columns transforms remain available.

In LanceDB Enterprise, creation accepts auto_refresh_enabled and auto_refresh_interval_seconds:

view = db.create_materialized_view(
    "recent_orders",
    "orders",
    auto_refresh_enabled=True,
    auto_refresh_interval_seconds=300,
)

Omit auto_refresh_interval_seconds to let the server determine when to refresh. These options also apply to asynchronous creation and creation jobs. Embedded connections built with experimental-functions support manual refresh and reject automatic scheduling requests.

lancedb.materialized_view.MaterializedView

Synchronous variant of AsyncMaterializedView.

name

name: str

table

table: 'LanceTable'

The view, as the table it is.

definition

definition: MaterializedViewDefinition

The query that defines the view.

refresh

refresh(*, full: bool = False, source_version: Optional[int] = None, cpu_millis: Optional[int] = None, memory_bytes: Optional[int] = None, gpu_count: Optional[int] = None, gpu_class: Optional[str] = None) -> 'RefreshMaterializedViewResult'

Recompute the view from its source. See AsyncMaterializedView.refresh.

refresh_async

refresh_async(*, full: bool = False, source_version: Optional[int] = None, cpu_millis: Optional[int] = None, memory_bytes: Optional[int] = None, gpu_count: Optional[int] = None, gpu_class: Optional[str] = None) -> 'Job[RefreshMaterializedViewResult]'

Submit a refresh and return its job without waiting.

See AsyncMaterializedView.refresh_async.

lancedb.materialized_view.MaterializedViewDefinition

The query that defines a materialized view, as stored::

SELECT columns
FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias]
[WHERE predicate] [GROUP BY expr, ...] [LIMIT n]

A Function in FROM position yields one row per element it returns.

query

query: str

The defining query, in the canonical spelling the server stores.

Views

lancedb.view.ViewDescription

What a database records about one view.

name

name: str

The view's name within its namespace.

query

query: str

The defining query, as the database stores it.

default_database

default_database: str

The database that unqualified table names in query resolve against.

schema

schema: 'pa.Schema'

The schema the defining query resolved to when the view was created.

namespace_path

namespace_path: List[str] = field(default_factory=list)

The namespace holding the view; empty is the root namespace.

default_namespace_path

default_namespace_path: List[str] = field(default_factory=list)

The namespace path those unqualified names resolve against.

Recorded with the view because it outlives the session that declared it: a reader resolving the query against its own default namespace could read a different table than the view was defined over.

Property Graphs

A property graph is defined over tables of its namespace: which tables hold nodes and which hold edges, each node table's key and label, and the node keys each edge table's source and destination columns reference -- the shape of SQL/PGQ's CREATE PROPERTY GRAPH. It stores no rows of its own; a query reads its tables as nodes and edges through SQL GRAPH_TABLE.

Create one with create_property_graph, and manage it with open_property_graph, list_property_graphs and drop_property_graph.

lancedb.graph.NodeTable

A table whose rows are the nodes of one label.

table

table: str

The table, in the graph's namespace.

key

key: str

The column that identifies a node.

label

label: str

The label its nodes carry.

properties

properties: Optional[List[str]] = None

The columns exposed as node properties; every column when None. A created graph reports the list it resolved to, the key among them.

lancedb.graph.EdgeTable

A table whose rows are the edges of one label.

table

table: str

The table, in the graph's namespace.

label

label: str

The label its edges carry.

source

source: Endpoint

The column naming each edge's source node.

destination

destination: Endpoint

The column naming each edge's destination node.

properties

properties: Optional[List[str]] = None

The columns exposed as edge properties; every column when None. A created graph reports the list it resolved to, both endpoint columns among them.

lancedb.graph.Endpoint

One end of an edge: a column of the edge table and the node key it holds.

column

column: str

The edge table's column.

references

references: Tuple[str, str]

The node table and its key column, as (table, column).

lancedb.graph.PropertyGraph

Synchronous variant of AsyncPropertyGraph.

name

name: str

The graph's name within its namespace.

namespace_path

namespace_path: List[str]

The namespace holding the graph; empty is the root namespace.

describe

describe() -> PropertyGraphDescription

The graph's node and edge tables, each with the properties create resolved.

lancedb.graph.PropertyGraphDescription

What a database records about one property graph.

name

name: str

The graph's name within its namespace.

namespace_path

namespace_path: List[str]

The namespace holding the graph; empty is the root namespace.

nodes

nodes: List[NodeTable]

The node tables, each with the properties create resolved.

edges

edges: List[EdgeTable]

The edge tables, each with the properties create resolved.

Expressions

Type-safe expression builder for filters and projections. Use these instead of raw SQL strings with where and select.

lancedb.expr.Expr

A type-safe expression node.

Construct instances with :func:col and :func:lit, then combine them using Python operators or the named methods below.

Examples:

>>> from lancedb.expr import col, lit
>>> filt = (col("age") > lit(18)) & (col("name").lower() == lit("alice"))
>>> proj = {"double": col("x") * lit(2)}
Notes

An Expr has no Python truth value: bool(expr) raises TypeError. Combine expressions with &, | and ~ (wrapping each comparison in parentheses) instead of and, or and not, and write a range as (col("x") > 1) & (col("x") < 5) rather than 1 < col("x") < 5.

lower

lower() -> 'Expr'

Convert string column values to lowercase.

upper

upper() -> 'Expr'

Convert string column values to uppercase.

contains

contains(substr: 'ExprLike') -> 'Expr'

Return True where the string contains substr.

isin

isin(values: 'Iterable[ExprLike]') -> 'Expr'

Return True where the value is one of values (SQL IN).

cast

cast(data_type: Union[str, 'pa.DataType']) -> 'Expr'

Cast values to data_type.

Parameters:

  • data_type (Union[str, 'pa.DataType']) –

    A PyArrow DataType (e.g. pa.int32()) or one of the type name strings: "bool", "int8", "int16", "int32", "int64", "uint8"–"uint64", "float32", "float64", "string", "date32", "date64".

eq

eq(other: ExprLike) -> 'Expr'

Equal to.

ne

ne(other: ExprLike) -> 'Expr'

Not equal to.

lt

lt(other: ExprLike) -> 'Expr'

Less than.

lte

lte(other: ExprLike) -> 'Expr'

Less than or equal to.

gt

gt(other: ExprLike) -> 'Expr'

Greater than.

gte

gte(other: ExprLike) -> 'Expr'

Greater than or equal to.

and_

and_(other: 'Expr') -> 'Expr'

Logical AND.

or_

or_(other: 'Expr') -> 'Expr'

Logical OR.

to_sql

to_sql() -> str

Render the expression as a SQL string (useful for debugging).

lancedb.expr.col

col(name: str) -> Expr

Reference a table column by name.

Parameters:

  • name (str) –

    The column name.

Examples:

>>> from lancedb.expr import col, lit
>>> col("age") > lit(18)
Expr((age > 18))

lancedb.expr.lit

lit(value: Union[bool, int, float, str, bytes, date, datetime, Decimal]) -> Expr

Create a literal (constant) value expression.

Parameters:

  • value (Union[bool, int, float, str, bytes, date, datetime, Decimal]) –

    A Python bool, int, float, str, bytes, date, datetime, or Decimal.

Examples:

>>> from lancedb.expr import col, lit
>>> col("price") * lit(1.1)
Expr((price * 1.1))

lancedb.expr.func

func(name: str, *args: ExprLike) -> Expr

Call an arbitrary SQL function by name.

Parameters:

  • name (str) –

    The SQL function name (e.g. "lower", "upper").

  • *args (ExprLike, default: () ) –

    The function arguments as :class:Expr or plain Python literals.

Examples:

>>> from lancedb.expr import col, func
>>> func("lower", col("name"))
Expr(lower(`name`))

Querying (Synchronous)

lancedb.query.Query

Bases: BaseModel

A LanceDB Query

Queries are constructed by the Table.search method. This class is a python representation of the query. Normally you will not need to interact with this class directly. You can build up a query and execute it using collection methods such as to_batches(), to_arrow(), to_pandas(), etc.

However, you can use the to_query() method to get the underlying query object. This can be useful for serializing a query or using it in a different context.

vector_column

vector_column: Optional[str] = None

vector

vector: Annotated[Optional[Union[List[float], List[List[float]], Array, List[Array]]], ensure_vector_query] = None

filter

filter: Optional[Union[str, Expr]] = None

postfilter

postfilter: Optional[bool] = None

full_text_query

full_text_query: Optional[FullTextSearchQuery] = None

limit

limit: Optional[int] = None

distance_type

distance_type: Optional[str] = None

nprobes

nprobes: Optional[int] = None

columns

columns: QueryProjection = None

minimum_nprobes

minimum_nprobes: Optional[int] = None

maximum_nprobes

maximum_nprobes: Optional[int] = None

search_effort

search_effort: Optional[float] = None

lower_bound

lower_bound: Optional[float] = None

upper_bound

upper_bound: Optional[float] = None

refine_factor

refine_factor: Optional[int] = None

with_row_id

with_row_id: Optional[bool] = None

with_row_address

with_row_address: Optional[bool] = None

fragments

fragments: Optional[Any] = None

fragment_ids

fragment_ids: Optional[List[int]] = None

offset

offset: Optional[int] = None

take_offsets

take_offsets: Optional[List[int]] = None

take_row_ids

take_row_ids: Optional[List[int]] = None
fast_search: Optional[bool] = None

use_rte

use_rte: Optional[bool] = None

ef

ef: Optional[int] = None

approx_mode

approx_mode: Optional[str] = None

bypass_vector_index

bypass_vector_index: Optional[bool] = None

order_by

order_by: Optional[List[ColumnOrdering]] = None

model_config

model_config = ConfigDict(arbitrary_types_allowed=True)

from_inner

from_inner(req: PyQueryRequest) -> Self

lancedb.query.LanceQueryBuilder

Bases: ABC

An abstract query builder. Subclasses are defined for vector search, full text search, hybrid, and plain SQL filtering.

create

create(table: 'Table', query: Optional[Union[ndarray, str, 'PIL.Image.Image', Tuple]], query_type: str, vector_column_name: str, ordering_field_name: Optional[str] = None, fts_columns: Optional[Union[str, List[str]]] = None, fast_search: bool = None) -> Self

Create a query builder based on the given query and query type.

Parameters:

  • table ('Table') –

    The table to query.

  • query (Optional[Union[ndarray, str, 'PIL.Image.Image', Tuple]]) –

    The query to use. If None, an empty query builder is returned which performs simple SQL filtering.

  • query_type (str) –

    The type of query to perform. One of "vector", "fts", "hybrid", or "auto". If "auto", the query type is inferred based on the query.

  • vector_column_name (str) –

    The name of the vector column to use for vector search.

  • ordering_field_name (Optional[str], default: None ) –

    .. deprecated:: 0.27.0 Use order_by() method instead.

  • fts_columns (Optional[Union[str, List[str]]], default: None ) –

    The columns to search in for full text search.

  • fast_search (bool, default: None ) –

    Skip flat search of unindexed data.

to_df

to_df() -> 'pd.DataFrame'

Deprecated alias for to_pandas(). Please use to_pandas() instead.

Execute the query and return the results as a pandas DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, *, blob_mode: BlobMode = 'lazy', timeout: Optional[timedelta] = None, **kwargs) -> 'pd.DataFrame'

Execute the query and return the results as a pandas DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_arrow

to_arrow(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and return the results as an Apache Arrow Table.

In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vectors.

This collects all results in memory. To process a large result a batch at a time, use to_batches.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.

to_batches

to_batches(batch_size: Optional[int] = None, *, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> RecordBatchReader

Execute the query and return the results as a pyarrow RecordBatchReader

For blob v2 projections, the default blob_mode="lazy" keeps the auto _rowid column visible so batch consumers can call fetch_blobs. The "bytes" and "descriptions" modes hide automatic row ids.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • batch_size (Optional[int], default: None ) –

    The maximum number of selected records in a RecordBatch object.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Stream blob payloads with query.to_batches(blob_mode="bytes") and convert each batch separately with batch.to_pandas().

to_list

to_list(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pydantic

to_pydantic(model: type[T], *, timeout: Optional[timedelta] = None) -> list[T]

Return the table as a list of pydantic models.

Parameters:

  • model (type[T]) –

    The pydantic model to use.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • List[LanceModel] –

to_polars

to_polars(*, timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and return the results as a Polars DataFrame. In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vector.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

limit

limit(limit: Union[int, None]) -> Self

Set the maximum number of results to return.

Parameters:

  • limit (Union[int, None]) –

    The maximum number of results to return. The default query limit is 10 results. For ANN/KNN queries, you must specify a limit. For plain searches, all records are returned if limit not set. WARNING if you have a large dataset, setting the limit to a large number, e.g. the table size, can potentially result in reading a large amount of data into memory and cause out of memory issues.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

offset

offset(offset: int) -> Self

Set the offset for the results.

Parameters:

  • offset (int) –

    The offset to start fetching results from.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

select

select(columns: Union[list[str], dict[str, Union[str, Expr]]]) -> Self

Set the columns to return.

Parameters:

  • columns (Union[list[str], dict[str, Union[str, Expr]]]) –

    List of column names to be fetched. Or a dictionary of column names to SQL expressions or :class:~lancedb.expr.Expr objects. All columns are fetched if None or unspecified.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

where

where(where: Union[str, Expr], prefilter: bool = True) -> Self

Set the where clause.

Parameters:

  • where (Union[str, Expr]) –

    The filter condition. Can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

  • prefilter (bool, default: True ) –

    If True, apply the filter before vector search, otherwise the filter is applied on the result of vector search. This feature is EXPERIMENTAL and may be removed and modified without warning in the future.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

Notes

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

with_row_id

with_row_id(with_row_id: bool) -> Self

Set whether to return row ids.

Parameters:

  • with_row_id (bool) –

    If True, return _rowid column in the results.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Set whether to return row addresses.

Parameters:

  • with_row_address (bool, default: True ) –

    If True, return the _rowaddr column in the results.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

with_fragments

with_fragments(fragments: Any) -> Self

Set the Lance fragments to scan for plain scanner-backed queries.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Set the Lance fragment ids to scan for plain scanner-backed queries.

use_rte

use_rte(enable: bool) -> Self

Control RTE LSM read routing for this query.

By default (unset), a query against a table with an RTE write spec is routed through the LSM scanner so it also returns data written via the merge_insert LSM path that has not yet been compacted into the base table (active/frozen memtables + SSTables); a table without a spec reads the base table.

Parameters:

  • enable (bool) –

    True forces the LSM scanner and errors if the table has no RTE write spec. False bypasses the RTE and reads the base table only, even when a spec is present.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

explain_plan

explain_plan(verbose: Optional[bool] = False) -> str

Return the execution plan for this query.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).explain_plan(True)
>>> print(plan)
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

order_by

order_by(ordering: Optional[List[ColumnOrdering]]) -> Self

Set the ordering for the results.

Parameters:

  • ordering (Optional[List[ColumnOrdering]]) –

    The ordering to use for the results. If None, then the default ordering will be used.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Run the query and return its execution plan with runtime metrics.

This returns detailed metrics for each step, such as elapsed time, rows processed, bytes read, and I/O stats. It is useful for debugging and performance tuning.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).analyze_plan()
>>> print(plan)
AnalyzeExec verbose=true, elapsed=..., metrics=...
  TracedExec, elapsed=..., metrics=...
    ProjectionExec: elapsed=..., expr=[...],
    metrics=[output_rows=..., elapsed_compute=..., output_bytes=...]
      GlobalLimitExec: elapsed=..., skip=0, fetch=10,
      metrics=[output_rows=..., elapsed_compute=..., output_bytes=...]
        FilterExec: elapsed=..., _distance@2 IS NOT NULL, metrics=[...]
          SortExec: elapsed=..., TopK(fetch=1), expr=[...],
          preserve_partitioning=[...], filter=[...],
          metrics=[output_rows=..., elapsed_compute=...,
          output_bytes=..., row_replacements=...]
            KNNVectorDistance: elapsed=..., metric=l2,
            metrics=[output_rows=..., elapsed_compute=...,
            output_bytes=..., output_batches=...]
              LanceRead: elapsed=..., uri=..., projection=[vector],
              num_fragments=..., range_before=None, range_after=None,
              row_id=true, row_addr=false,
              full_filter=--, refine_filter=--,
              metrics=[output_rows=..., elapsed_compute=..., output_bytes=...,
              fragments_scanned=..., ranges_scanned=1, rows_scanned=1,
              bytes_read=..., iops=..., requests=..., task_wait_time=...]

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans. "aggregate" preserves the legacy summary, "per_worker" shows each worker separately, and "full" includes both.

Returns:

  • plan ( str ) –

    The physical query execution plan with runtime metrics.

vector

vector(vector: Union[VEC, Tuple]) -> Self

Set the vector to search for.

Parameters:

  • vector (Union[VEC, Tuple]) –

    The vector to search for.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

text

text(text: str | FullTextQuery) -> Self

Set the text to search for.

Parameters:

  • text (str | FullTextQuery) –

    If a string, it is treated as a MatchQuery. If a FullTextQuery object, it is used directly.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

rerank

rerank(reranker: Reranker) -> Self

Rerank the results using the specified reranker.

Parameters:

  • reranker (Reranker) –

    The reranker to use.

Returns:

  • The LanceQueryBuilder object. –

to_query_object

to_query_object() -> Query

Return a serializable representation of the query

Returns:

  • Query –

    The serializable representation of the query

lancedb.query.LanceVectorQueryBuilder

Bases: LanceQueryBuilder

Examples:

>>> import lancedb
>>> data = [{"vector": [1.1, 1.2], "b": 2},
...         {"vector": [0.5, 1.3], "b": 4},
...         {"vector": [0.4, 0.4], "b": 6},
...         {"vector": [0.4, 0.4], "b": 10}]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data=data)
>>> (table.search([0.4, 0.4])
...       .distance_type("cosine")
...       .where("b < 10")
...       .select(["b", "vector"])
...       .limit(2)
...       .to_pandas())
   b      vector  _distance
0  6  [0.4, 0.4]   0.000000
1  2  [1.1, 1.2]   0.000944

metric

metric(metric: Literal['l2', 'cosine', 'dot']) -> LanceVectorQueryBuilder

Set the distance metric to use.

This is an alias for distance_type() and may be deprecated in the future. Please use distance_type() instead.

Parameters:

  • metric (Literal['l2', 'cosine', 'dot']) –

    The distance metric to use. By default "l2" is used.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

distance_type

distance_type(distance_type: Literal['l2', 'cosine', 'dot']) -> 'LanceVectorQueryBuilder'

Set the distance metric to use.

When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use.

Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.

Parameters:

  • distance_type (Literal['l2', 'cosine', 'dot']) –

    The distance metric to use. By default "l2" is used.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

nprobes

nprobes(nprobes: int) -> LanceVectorQueryBuilder

Set the number of probes to use.

Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.

See discussion in Querying an ANN Index for tuning advice.

The value is retained as nprobes through client and server request construction. Lance sets both probe bounds to this value. Explicit minimum or maximum settings can override their respective bounds.

Parameters:

  • nprobes (int) –

    The number of probes to use. Must be greater than 0.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

search_effort

search_effort(search_effort: Optional[float]) -> LanceVectorQueryBuilder

Set the initial IVF search effort to a finite value in [0, 1].

Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.

None (the default) uses Auto, internally 0.5. Zero starts at the caller minimum; one searches all partitions within the caller maximum. Later probing may continue when too few candidates are found. Any explicit effort, including 0.5, overrides nprobes in either call order. Explicit minimum and maximum bounds still apply. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.

Examples:

>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table("effort_vectors", [{"vector": [0.1, 0.2]}])
>>> query = table.search([0.1, 0.2]).search_effort(0.75)

minimum_nprobes

minimum_nprobes(minimum_nprobes: int) -> LanceVectorQueryBuilder

Set the minimum number of probes to use.

See nprobes for more details.

These partitions will be searched on every vector query and will increase recall at the expense of latency.

maximum_nprobes

maximum_nprobes(maximum_nprobes: int) -> LanceVectorQueryBuilder

Set the maximum number of probes to use.

See nprobes for more details.

If this value is greater than minimum_nprobes then the excess partitions will be searched only if we have not found enough results.

This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.

If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.

distance_range

distance_range(lower_bound: Optional[float] = None, upper_bound: Optional[float] = None) -> LanceVectorQueryBuilder

Set the distance range to use.

Only rows with distances within range [lower_bound, upper_bound) will be returned.

Parameters:

  • lower_bound (Optional[float], default: None ) –

    The lower bound of the distance range.

  • upper_bound (Optional[float], default: None ) –

    The upper bound of the distance range.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

ef

ef(ef: int) -> LanceVectorQueryBuilder

Set the number of candidates to consider during search.

Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.

This only applies to the HNSW-related index. The default value is 1.5 * limit.

Parameters:

  • ef (int) –

    The number of candidates to consider during search.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

approx_mode

approx_mode(approx_mode: Literal['fast', 'normal', 'accurate']) -> LanceVectorQueryBuilder

Set the speed / accuracy tradeoff to use for approximate search.

This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.

Examples:

>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table("approx_vectors", [{"vector": [0.1, 0.2]}])
>>> query = table.search([0.1, 0.2]).approx_mode("fast")

Parameters:

  • approx_mode (Literal['fast', 'normal', 'accurate']) –

    The tradeoff to use. By default "normal" is used.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

refine_factor

refine_factor(refine_factor: int) -> LanceVectorQueryBuilder

Set the refine factor to use, increasing the number of vectors sampled.

As an example, a refine factor of 2 will sample 2x as many vectors as requested, re-ranks them, and returns the top half most relevant results.

See discussion in Querying an ANN Index for tuning advice.

Parameters:

  • refine_factor (int) –

    The refine factor to use.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and return the results as an Apache Arrow Table.

In addition to the selected columns, LanceDB also returns a vector and also the "_distance" column which is the distance between the query vector and the returned vectors.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.

to_query_object

to_query_object() -> Query

Build a Query object

This can be used to serialize a query

to_batches

to_batches(batch_size: Optional[int] = None, *, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> RecordBatchReader

Execute the query and return the result as a RecordBatchReader object.

Parameters:

  • batch_size (Optional[int], default: None ) –

    The maximum number of selected records in a RecordBatch object.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls blob conversion as described in :meth:LanceQueryBuilder.to_batches.

Returns:

  • RecordBatchReader –

where

where(where: Union[str, Expr], prefilter: bool = None) -> LanceVectorQueryBuilder

Set the where clause.

Parameters:

  • where (Union[str, Expr]) –

    The filter condition. Can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

  • prefilter (bool, default: None ) –

    If True, apply the filter before vector search, otherwise the filter is applied on the result of vector search.

Returns:

  • LanceQueryBuilder –

    The LanceQueryBuilder object.

Notes

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

rerank

rerank(reranker: Reranker, query_string: Optional[str] = None) -> LanceVectorQueryBuilder

Rerank the results using the specified reranker.

Parameters:

  • reranker (Reranker) –

    The reranker to use.

  • query_string (Optional[str], default: None ) –

    The query to use for reranking. This needs to be specified explicitly here as the query used for vector search may already be vectorized and the reranker requires a string query. This is only required if the query used for vector search is not a string. Note: This doesn't yet support the case where the query is multimodal or a list of vectors.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

bypass_vector_index

bypass_vector_index() -> LanceVectorQueryBuilder

If this is called then any vector index is skipped

An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.

Returns:

  • LanceVectorQueryBuilder –

    The LanceVectorQueryBuilder object.

fast_search() -> LanceVectorQueryBuilder

Skip a flat search of unindexed data. This will improve search performance but search results will not include unindexed data.

Returns:

  • LanceVectorQueryBuilder –

    The LanceVectorQueryBuilder object.

lancedb.query.LanceFtsQueryBuilder

Bases: LanceQueryBuilder

A builder for full text search for LanceDB.

phrase_query

phrase_query(phrase_query: bool = True) -> LanceFtsQueryBuilder

Set whether to use phrase query.

Parameters:

  • phrase_query (bool, default: True ) –

    If True, then an unquoted string query will be wrapped in quotes.

Returns:

  • LanceFtsQueryBuilder –

    The LanceFtsQueryBuilder object.

fast_search() -> LanceFtsQueryBuilder

Skip a flat search of unindexed data. This will improve search performance but search results will not include unindexed data.

Returns:

  • LanceFtsQueryBuilder –

    The LanceFtsQueryBuilder object.

to_query_object

to_query_object() -> Query

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> Table

to_batches

to_batches(batch_size: Optional[int] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy')

rerank

rerank(reranker: Reranker) -> LanceFtsQueryBuilder

Rerank the results using the specified reranker.

Parameters:

  • reranker (Reranker) –

    The reranker to use.

Returns:

  • LanceFtsQueryBuilder –

    The LanceQueryBuilder object.

lancedb.query.LanceHybridQueryBuilder

Bases: LanceQueryBuilder

A query builder that performs hybrid vector and full text search. Results are combined and reranked based on the specified reranker. By default, the results are reranked using the RRFReranker, which uses reciprocal rank fusion score for reranking.

To make the vector and fts results comparable, the scores are normalized. Instead of normalizing scores, the normalize parameter can be set to "rank" in the rerank method to convert the scores to ranks and then normalize them.

with_row_id

with_row_id(with_row_id: bool) -> Self

Set whether to return row ids.

Recorded separately from _with_row_id, which rerank(return_score= "all") also sets for its own use — on an RTE table only a caller who asked is refused, and the reranker still falls back to the primary key.

phrase_query

phrase_query(phrase_query: bool = True) -> LanceHybridQueryBuilder

Set whether to use phrase query.

Parameters:

  • phrase_query (bool, default: True ) –

    If True, then an unquoted string query will be wrapped in quotes.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

to_query_object

to_query_object() -> Query

to_arrow

to_arrow(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> Table

to_batches

to_batches(batch_size: Optional[int] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy')

rerank

rerank(reranker: Reranker = RRFReranker(), normalize: str = 'score') -> LanceHybridQueryBuilder

Rerank the hybrid search results using the specified reranker. The reranker must be an instance of Reranker class.

Parameters:

  • reranker (Reranker, default: RRFReranker() ) –

    The reranker to use. Must be an instance of Reranker class.

  • normalize (str, default: 'score' ) –

    The method to normalize the scores. Can be "rank" or "score". If "rank", the scores are converted to ranks and then normalized. If "score", the scores are normalized directly.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

nprobes

nprobes(nprobes: int) -> LanceHybridQueryBuilder

Set the number of probes to use for vector search.

Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.

The value is retained as nprobes until Lance sets both probe bounds to it. Explicit minimum or maximum settings can override their respective bounds.

Parameters:

  • nprobes (int) –

    The number of probes to use. Must be greater than 0.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

search_effort

search_effort(search_effort: Optional[float]) -> LanceHybridQueryBuilder

Set the initial IVF search effort to a finite value in [0, 1].

Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.

None (the default) uses Auto, internally 0.5. Zero starts at the caller minimum; one searches all partitions within the caller maximum. Later probing may continue when too few candidates are found. Any explicit effort, including 0.5, overrides nprobes in either call order. Explicit minimum and maximum bounds still apply. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.

Examples:

>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table(
...     "effort_hybrid", [{"vector": [0.1, 0.2], "text": "dog"}]
... )
>>> query = (
...     table.search(query_type="hybrid").vector([0.1, 0.2])
...     .text("dog").search_effort(0.75)
... )

minimum_nprobes

minimum_nprobes(minimum_nprobes: int) -> LanceHybridQueryBuilder

Set the minimum number of probes to use.

See nprobes for more details.

maximum_nprobes

maximum_nprobes(maximum_nprobes: int) -> LanceHybridQueryBuilder

Set the maximum number of probes to use.

See nprobes for more details.

distance_range

distance_range(lower_bound: Optional[float] = None, upper_bound: Optional[float] = None) -> LanceHybridQueryBuilder

Set the distance range to use.

Only rows with distances within range [lower_bound, upper_bound) will be returned.

Parameters:

  • lower_bound (Optional[float], default: None ) –

    The lower bound of the distance range.

  • upper_bound (Optional[float], default: None ) –

    The upper bound of the distance range.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

ef

ef(ef: int) -> LanceHybridQueryBuilder

Set the number of candidates to consider during search.

Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency.

This only applies to the HNSW-related index. The default value is 1.5 * limit.

Parameters:

  • ef (int) –

    The number of candidates to consider during search.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

approx_mode

approx_mode(approx_mode: Literal['fast', 'normal', 'accurate']) -> LanceHybridQueryBuilder

Set the speed / accuracy tradeoff to use for approximate search.

This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.

Examples:

>>> import lancedb
>>> db = lancedb.connect("memory://")
>>> table = db.create_table(
...     "approx_hybrid", [{"vector": [0.1, 0.2], "text": "dog"}]
... )
>>> query = (
...     table.search(query_type="hybrid")
...     .vector([0.1, 0.2])
...     .text("dog")
...     .approx_mode("accurate")
... )

Parameters:

  • approx_mode (Literal['fast', 'normal', 'accurate']) –

    The tradeoff to use. By default "normal" is used.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

metric

metric(metric: Literal['l2', 'cosine', 'dot']) -> LanceHybridQueryBuilder

Set the distance metric to use.

This is an alias for distance_type() and may be deprecated in the future. Please use distance_type() instead.

Parameters:

  • metric (Literal['l2', 'cosine', 'dot']) –

    The distance metric to use. By default "l2" is used.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

distance_type

distance_type(distance_type: Literal['l2', 'cosine', 'dot']) -> 'LanceHybridQueryBuilder'

Set the distance metric to use.

When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use.

Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.

Parameters:

  • distance_type (Literal['l2', 'cosine', 'dot']) –

    The distance metric to use. By default "l2" is used.

Returns:

  • LanceVectorQueryBuilder –

    The LanceQueryBuilder object.

refine_factor

refine_factor(refine_factor: int) -> LanceHybridQueryBuilder

Refine the vector search results by reading extra elements and re-ranking them in memory.

Parameters:

  • refine_factor (int) –

    The refine factor to use.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

vector

vector(vector: Union[VEC, Tuple]) -> LanceHybridQueryBuilder

Set the vector to search for.

Parameters:

  • vector (Union[VEC, Tuple]) –

    The vector to search for.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

text

text(text: str | FullTextQuery) -> LanceHybridQueryBuilder

bypass_vector_index

bypass_vector_index() -> LanceHybridQueryBuilder

If this is called then any vector index is skipped

An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.

Returns:

  • LanceHybridQueryBuilder –

    The LanceHybridQueryBuilder object.

explain_plan

explain_plan(verbose: Optional[bool] = False) -> str

Return the execution plan for this query.

Examples:

>>> import lancedb
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", [{"vector": [99.0, 99]}])
>>> query = [100, 100]
>>> plan = table.search(query).explain_plan(True)
>>> print(plan)
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and display with runtime metrics.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

lancedb.query.LanceEmptyQueryBuilder

Bases: LanceQueryBuilder

to_arrow

to_arrow(*, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> Table

to_query_object

to_query_object() -> Query

output_schema

output_schema() -> Schema

to_batches

to_batches(batch_size: Optional[int] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> RecordBatchReader

rerank

rerank(reranker: Reranker) -> LanceEmptyQueryBuilder

Rerank the results using the specified reranker.

Parameters:

  • reranker (Reranker) –

    The reranker to use.

Returns:

  • LanceEmptyQueryBuilder –

    The LanceQueryBuilder object.

lancedb.query.LanceTakeQueryBuilder

Bases: BaseQueryBuilder

Builder for parameterizing and executing take queries.

Full text queries

Structured full text queries can be passed to Table.search or AsyncTable.search in place of a query string, and combined with BooleanQuery.

lancedb.query.FullTextQuery

Bases: ABC

query_type

query_type() -> FullTextQueryType

Get the query type of the query.

Returns:

  • str –

    The type of the query.

to_json

to_json() -> str

Convert the query to a JSON string.

Returns:

  • str –

    A JSON string representation of the query.

Examples:

>>> from lancedb.query import MatchQuery
>>> query = MatchQuery("puppy", "text", fuzziness=2)
>>> query.to_json()
'{"match":{"column":"text","terms":"puppy","boost":1.0,"fuzziness":2,"max_expansions":50,"operator":"Or","prefix_length":0}}'

lancedb.query.MatchQuery

Bases: FullTextQuery

Match query for full-text search.

Parameters:

  • query (str) –

    The query string to match against.

  • column (str) –

    The column to match against. None leaves column selection to query planning.

  • boost (float, default: 1.0 ) –

    The boost factor for the query. The score of each matching document is multiplied by this value.

  • fuzziness (int or None, default: 0 ) –

    The maximum edit distance for each term in the match query. Defaults to 0 (exact match). If None, fuzziness is applied automatically by the rules: - 0 for terms with length <= 2 - 1 for terms with length <= 5 - 2 for terms with length > 5

  • max_expansions (int) –

    The maximum number of terms to consider for fuzzy matching. Defaults to 50.

  • operator (FullTextOperator, default: OR ) –

    The operator to use for combining the query results. Can be either AND or OR. If AND, all terms in the query must match. If OR, at least one term in the query must match.

  • prefix_length (int) –

    The number of beginning characters being unchanged for fuzzy matching. This is useful to achieve prefix matching.

  • document_granularity (DocumentGranularity) –

    Explicitly select row or deepest-list-element documents. If omitted, the indexed granularity is inferred. When both granularities are indexed for the field, this must be specified. With no index, row granularity is used.

query

query: str

column

column: Optional[str] = None

boost

boost: float = Field(1.0, kw_only=True)

fuzziness

fuzziness: Optional[int] = Field(0, kw_only=True)

max_expansions

max_expansions: int = Field(50, kw_only=True)

operator

operator: FullTextOperator = Field(OR, kw_only=True)

prefix_length

prefix_length: int = Field(0, kw_only=True)

document_granularity

document_granularity: Optional[DocumentGranularity] = Field(None, kw_only=True)

query_type

query_type() -> FullTextQueryType

lancedb.query.PhraseQuery

Bases: FullTextQuery

Phrase query for full-text search.

Parameters:

  • query (str) –

    The query string to match against.

  • column (str) –

    The column to match against. None leaves column selection to query planning.

  • slop (int, default: 0 ) –

    The maximum number of intervening positions permitted in the phrase.

  • document_granularity (DocumentGranularity) –

    Explicitly select row or deepest-list-element documents. If omitted, the indexed granularity is inferred. When both granularities are indexed for the field, this must be specified. With no index, row granularity is used.

query

query: str

column

column: Optional[str] = None

slop

slop: int = Field(0, kw_only=True)

document_granularity

document_granularity: Optional[DocumentGranularity] = Field(None, kw_only=True)

query_type

query_type() -> FullTextQueryType

lancedb.query.BoostQuery

Bases: FullTextQuery

Boost query for full-text search.

Parameters:

  • positive (dict) –

    The positive query object.

  • negative (dict) –

    The negative query object.

  • negative_boost (float, default: 0.5 ) –

    The boost factor for the negative query.

positive

positive: FullTextQuery

negative

negative: FullTextQuery

negative_boost

negative_boost: float = Field(0.5, kw_only=True)

query_type

query_type() -> FullTextQueryType

lancedb.query.MultiMatchQuery

Bases: FullTextQuery

Multi-match query for full-text search.

Parameters:

  • query (str | list[Query]) –

    If a string, the query string to match against.

  • columns (list[str]) –

    The list of columns to match against.

  • boosts (list[float]) –

    The list of boost factors for each column. If not provided, all columns will have the same boost factor.

  • operator (FullTextOperator, default: OR ) –

    The operator to use for combining the query results. Can be either AND or OR. It would be applied to all columns individually. For example, if the operator is AND, then the query "hello world" is equal to match("hello AND world", column1) OR match("hello AND world", column2).

query

query: str

columns

columns: list[str]

boosts

boosts: Optional[list[float]] = Field(None, kw_only=True)

operator

operator: FullTextOperator = Field(OR, kw_only=True)

query_type

query_type() -> FullTextQueryType

lancedb.query.CombinedFieldsQuery

Bases: FullTextQuery

Combined-fields (BM25F) query for full-text search.

Unlike MultiMatchQuery, which scores each column on its own and keeps the best one, this query treats the columns as a single virtual field, so term statistics are blended across them. A query term can then match in any of the columns, e.g. a first name in one column and a last name in another.

Parameters:

  • query (str) –

    The query string to match against.

  • columns (list[str]) –

    The columns combined into the virtual field. Every column must have the same tokenizer configuration, and at least one must have an FTS index.

  • boosts (list[float]) –

    The weight of each column, aligned with columns. Each weight must be at least 1. If not provided, every column has a weight of 1.

  • operator (FullTextOperator, default: OR ) –

    The operator to use for combining the query terms. If AND, every term must appear in at least one of the columns. If OR, at least one term must match.

query

query: str

columns

columns: list[str]

boosts

boosts: Optional[list[float]] = Field(None, kw_only=True)

operator

operator: FullTextOperator = Field(OR, kw_only=True)

query_type

query_type() -> FullTextQueryType

lancedb.query.BooleanQuery

Bases: FullTextQuery

Boolean query for full-text search.

Parameters:

  • queries (list[tuple(Occur, FullTextQuery)]) –

    The list of queries with their occurrence requirements. Each tuple contains an Occur value (MUST, SHOULD, or MUST_NOT) and a FullTextQuery to apply.

queries

queries: list[tuple[Occur, FullTextQuery]]

query_type

query_type() -> FullTextQueryType

lancedb.query.FullTextOperator

Bases: str, Enum

AND

AND = 'AND'

OR

OR = 'OR'

lancedb.query.DocumentGranularity

Bases: str, Enum

The unit treated as one full-text-search document.

ROW

ROW = 'row'

LIST_ELEMENT

LIST_ELEMENT = 'list_element'

lancedb.query.Occur

Bases: str, Enum

SHOULD

SHOULD = 'SHOULD'

MUST

MUST = 'MUST'

MUST_NOT

MUST_NOT = 'MUST_NOT'

Embeddings

EmbeddingFunction

Bases: BaseModel, ABC

An ABC for embedding functions.

All concrete embedding functions must implement the following methods: 1. compute_query_embeddings() which takes a query and returns a list of embeddings 2. compute_source_embeddings() which returns a list of embeddings for the source column For text data, the two will be the same. For multi-modal data, the source column might be images and the vector column might be text. 3. ndims() which returns the number of dimensions of the vector column

max_retries

max_retries: int = 7

create

create(**kwargs)

Create an instance of the embedding function

sensitive_keys

sensitive_keys() -> List[str]

Return a list of keys that are sensitive and should not be allowed to be set to hardcoded values in the config. For example, API keys.

remote_code_model

remote_code_model() -> Optional[Tuple[str, Optional[str]]]

Return the model whose repository code this function runs when its trust_remote_code field is enabled, as a (name, revision) pair.

Return None if the function never runs code from a model repository. When the function is restored from table metadata, trust_remote_code is only enabled for models allowed with allow_remote_code.

endpoints

endpoints() -> List[str]

Return the non-default URLs this function sends data or credentials to.

When the function is restored from table metadata, each of these must be allowed with allow_endpoint.

compute_query_embeddings

compute_query_embeddings(*args, **kwargs) -> list[Union[array, None]]

Compute the embeddings for a given user query

Returns:

  • A list of embeddings for each input. The embedding of each input can be None –
  • when the embedding is not valid. –

compute_source_embeddings

compute_source_embeddings(*args, **kwargs) -> list[Union[array, None]]

Compute the embeddings for the source column in the database

Returns:

  • A list of embeddings for each input. The embedding of each input can be None –
  • when the embedding is not valid. –

compute_query_embeddings_with_retry

compute_query_embeddings_with_retry(*args, **kwargs) -> list[Union[array, None]]

Compute the embeddings for a given user query with retries

Returns:

  • A list of embeddings for each input. The embedding of each input can be None –
  • when the embedding is not valid. –

compute_source_embeddings_with_retry

compute_source_embeddings_with_retry(*args, **kwargs) -> list[Union[array, None]]

Compute the embeddings for the source column in the database with retries.

Returns:

  • A list of embeddings for each input. The embedding of each input can be None –
  • when the embedding is not valid. –

sanitize_input

sanitize_input(texts: TEXT) -> Union[List[str], ndarray]

Sanitize the input to the embedding function.

safe_model_dump

safe_model_dump()

ndims

ndims() -> int

Return the dimensions of the vector column

SourceField

SourceField(**kwargs)

Creates a pydantic Field that can automatically annotate the source column for this embedding function

VectorField

VectorField(**kwargs)

Creates a pydantic Field that can automatically annotate the target vector column for this embedding function.

The field can be annotated as list[float] for compatibility with static type checkers. LanceDB will infer the fixed vector dimension from this embedding function.

EmbeddingFunctionConfig

Bases: BaseModel

This model encapsulates the configuration for a embedding function in a lancedb table. It holds the embedding function, the source column, and the vector column

vector_column

vector_column: str

source_column

source_column: str

function

function: EmbeddingFunction

TextEmbeddingFunction

Bases: EmbeddingFunction

A callable ABC for embedding functions that take text as input

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> list[Union[array, None]]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> list[Union[array, None]]

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> list[Union[array, None]]

Generate the embeddings for the given texts

UntrustedEmbeddingConfigError

Bases: PermissionError

Raised when using an embedding function restored from table metadata whose settings this process has not approved.

See allow_remote_code and allow_endpoint.

BedRockText

Bases: TextEmbeddingFunction

Parameters:

  • name (str, default: "amazon.titan-embed-text-v1" ) –

    The model ID of the bedrock model to use. Supported models for are: - amazon.titan-embed-text-v1 - cohere.embed-english-v3 - cohere.embed-multilingual-v3

  • region (str, default: "us-east-1" ) –

    Optional name of the AWS Region in which the service should be called.

  • profile_name (str, default: None ) –

    Optional name of the AWS profile to use for calling the Bedrock service. If not specified, the default profile will be used.

  • assumed_role (str, default: None ) –

    Optional ARN of an AWS IAM role to assume for calling the Bedrock service. If not specified, the current active credentials will be used.

  • role_session_name (str, default: "lancedb-embeddings" ) –

    Optional name of the AWS IAM role session to use for calling the Bedrock service. If not specified, "lancedb-embeddings" name will be used.

Examples:

import lancedb import pandas as pd from lancedb.pydantic import LanceModel, Vector

model = get_registry().get("bedrock-text").create()

class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()

df = pd.DataFrame({"text": ["hello world", "goodbye world"]}) db = lancedb.connect("tmp_path") tbl = db.create_table("test", schema=TextModel, mode="overwrite")

tbl.add(df)

rs = tbl.search("hello").limit(1).to_pandas()

name

name: str = 'amazon.titan-embed-text-v1'

region

region: str = 'us-east-1'

assumed_role

assumed_role: Union[str, None] = None

profile_name

profile_name: Union[str, None] = None

role_session_name

role_session_name: str = 'lancedb-embeddings'

source_input_type

source_input_type: str = 'search_document'

query_input_type

query_input_type: str = 'search_query'

model_config

model_config = ConfigDict(ignored_types=(cached_property))

client

client

Create a boto3 client for Amazon Bedrock service

Returns:

  • client –

    The boto3 client for Amazon Bedrock service

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[List[float]]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[List[float]]

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[List[float]]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

Returns:

  • list[list[float]] –

    The embeddings for the given texts

CohereEmbeddingFunction

Bases: TextEmbeddingFunction

An embedding function that uses the Cohere API

https://docs.cohere.com/docs/multilingual-language-models

Parameters:

  • name (str, default: "embed-multilingual-v3.0" ) –

    The name of the model to use. List of acceptable models:

    * embed-english-v3.0
    * embed-multilingual-v3.0
    * embed-english-light-v3.0
    * embed-multilingual-light-v3.0
    

    The v2.0 models (embed-english-v2.0, embed-english-light-v2.0 and embed-multilingual-v2.0) were retired by Cohere on April 4, 2026. The previous default produced 768-dimensional vectors, while the new default produces 1024-dimensional vectors. Existing tables containing vectors from a v2.0 model must be re-embedded with an active model into a table whose vector column has the matching dimension.

  • source_input_type (str, default: "search_document" ) –

    The input type for the source column in the database

  • query_input_type (str, default: "search_query" ) –

    The input type for the query column in the database

Notes

Cohere supports following input types:

Input Type Description
"search_document" Used for embeddings stored in a vector
database for search use-cases.
"search_query" Used for embeddings of search queries
run against a vector DB
"semantic_similarity" Specifies the given text will be used
for Semantic Textual Similarity (STS)
"classification" Used for embeddings passed through a
text classifier.
"clustering" Used for the embeddings run through a
clustering algorithm

Examples:

import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import EmbeddingFunctionRegistry

cohere = EmbeddingFunctionRegistry .get_instance() .get("cohere") .create(name="embed-multilingual-v3.0")

class TextModel(LanceModel): text: str = cohere.SourceField() vector: Vector(cohere.ndims()) = cohere.VectorField()

data = [ { "text": "hello world" }, { "text": "goodbye world" }]

db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")

tbl.add(data)

name

name: str = 'embed-multilingual-v3.0'

source_input_type

source_input_type: str = 'search_document'

query_input_type

query_input_type: str = 'search_query'

client

client: ClassVar = None

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[array]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[array]

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[array]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

GeminiText

Bases: TextEmbeddingFunction

An embedding function that uses Google's Gemini API. Requires GOOGLE_API_KEY to be set.

https://ai.google.dev/gemini-api/docs/embeddings

Supports various tasks types: | Task Type | Description | |-------------------------|--------------------------------------------------------| | "retrieval_query" | Specifies the given text is a query in a | | | search/retrieval setting. | | "retrieval_document" | Specifies the given text is a document in a | | | search/retrieval setting. Using this task type | | | requires a title but is automatically provided by | | | Embeddings API | | "semantic_similarity" | Specifies the given text will be used for Semantic | | | Textual Similarity (STS). | | "classification" | Specifies that the embeddings will be used for | | | classification. | | "clustering" | Specifies that the embeddings will be used for | | | clustering. |

Note: The supported task types might change in the Gemini API, but as long as a supported task type and its argument set is provided, those will be delegated to the API calls.

Parameters:

  • name (str, default: "gemini-embedding-001" ) –

    The name of the model to use. Supported models include: - "gemini-embedding-001" (768 dimensions)

    Note: The legacy "models/embedding-001" format is also supported but "gemini-embedding-001" is recommended.

  • query_task_type (str, default: "retrieval_query" ) –

    Sets the task type for the queries.

  • source_task_type (str, default: "retrieval_document" ) –

    Sets the task type for ingestion.

Examples:

import lancedb import pandas as pd from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import get_registry

model = get_registry().get("gemini-text").create()

class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()

df = pd.DataFrame({"text": ["hello world", "goodbye world"]}) db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")

tbl.add(df) rs = tbl.search("hello").limit(1).to_pandas()

name

name: str = 'gemini-embedding-001'

dim

dim: Optional[int] = None

query_task_type

query_task_type: str = 'retrieval_query'

source_task_type

source_task_type: str = 'retrieval_document'

model_config

model_config = ConfigDict(ignored_types=(cached_property))

client

client

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[array]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[array]

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[array]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

InstructorEmbeddingFunction

Bases: TextEmbeddingFunction

An embedding function that uses the InstructorEmbedding library. Instructor models support multi-task learning, and can be used for a variety of tasks, including text classification, sentence similarity, and document retrieval. If you want to calculate customized embeddings for specific sentences, you may follow the unified template to write instructions: "Represent the domain text_type for task_objective":

* domain is optional, and it specifies the domain of the text, e.g., science,
  finance, medicine, etc.
* text_type is required, and it specifies the encoding unit, e.g., sentence,
  document, paragraph, etc.
* task_objective is optional, and it specifies the objective of embedding,
  e.g., retrieve a document, classify the sentence, etc.

For example, if you want to calculate embeddings for a document, you may write the instruction as follows: "Represent the document for retrieval"

Parameters:

  • name (str) –

    The name of the model to use. Available models are listed at https://github.com/xlang-ai/instructor-embedding#model-list; The default model is hkunlp/instructor-base

  • batch_size (int, default: 32 ) –

    The batch size to use when generating embeddings

  • device (str, default: "cpu" ) –

    The device to use when generating embeddings

  • show_progress_bar (bool, default: True ) –

    Whether to show a progress bar when generating embeddings

  • normalize_embeddings (bool, default: True ) –

    Whether to normalize the embeddings

  • quantize (bool, default: False ) –

    Whether to quantize the model

  • source_instruction (str, default: "represent the document for retrieval" ) –

    The instruction for the source column

  • query_instruction (str, default: "represent the document for retrieving the most ) –

    similar documents" The instruction for the query

Examples:

import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import get_registry, InstructorEmbeddingFunction

instructor = get_registry().get("instructor").create( source_instruction="represent the document for retrieval", query_instruction="represent the document for retrieving the most " "similar documents" )

class Schema(LanceModel): vector: Vector(instructor.ndims()) = instructor.VectorField() text: str = instructor.SourceField()

db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=Schema, mode="overwrite")

texts = [{"text": "Capitalism has been dominant in the Western world since the " "end of feudalism, but most feel[who?] that..."}, {"text": "The disparate impact theory is especially controversial under " "the Fair Housing Act because the Act..."}, {"text": "Disparate impact in United States labor law refers to practices " "in employment, housing, and other areas that.."}]

tbl.add(texts)

name

name: str = 'hkunlp/instructor-base'

batch_size

batch_size: int = 32

device

device: str = 'cpu'

show_progress_bar

show_progress_bar: bool = True

normalize_embeddings

normalize_embeddings: bool = True

quantize

quantize: bool = False

source_instruction

source_instruction: str = 'represent the document for retrieval'

query_instruction

query_instruction: str = 'represent the document for retrieving the most similar documents'

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[array]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[array]

generate_embeddings

generate_embeddings(texts: List) -> List

get_model

get_model()

OCIGenAIEmbeddings

Bases: TextEmbeddingFunction

An embedding function that uses the Oracle Cloud Infrastructure (OCI) Generative AI service.

https://docs.oracle.com/en-us/iaas/Content/generative-ai/embed-models.htm

Requires the OCI Python SDK (pip install oci). No credentials are stored on this object: requests are signed by the SDK using an API-key or session-token profile from the OCI config file, or the instance / resource principal of the host, so there are no sensitive keys to protect with $var:.

Cohere embedding models are asymmetric: the source column is embedded with input_type="SEARCH_DOCUMENT" and queries with "SEARCH_QUERY". Inputs are sent in batches of at most 96 texts, the service limit.

Parameters:

  • name (str, default: "cohere.embed-v4.0" ) –

    Model ID of an on-demand embedding model: cohere.embed-v4.0 (1536 dims), cohere.embed-english-v3.0 / cohere.embed-multilingual-v3.0 (1024 dims) or their -light-v3.0 variants (384 dims). A dedicated AI cluster endpoint OCID (ocid1.generativeaiendpoint...) is also accepted.

  • compartment_id (str) –

    OCID of the compartment the requests are authorized in and billed to. The service requires it; when not passed, the OCI_COMPARTMENT_ID environment variable is used.

  • region (str, default: "us-chicago-1" ) –

    OCI region used to derive the inference endpoint when service_endpoint is not set.

  • service_endpoint (str) –

    Full inference endpoint, e.g. https://inference.generativeai.us-chicago-1.oci.oraclecloud.com. Takes precedence over region; needed for realms with a different endpoint template.

  • auth_type (str, default: "API_KEY" ) –

    One of "API_KEY", "SECURITY_TOKEN", "INSTANCE_PRINCIPAL" or "RESOURCE_PRINCIPAL". The first two read profile auth_profile from the OCI config file auth_file_location; SECURITY_TOKEN additionally uses the profile's security_token_file written by oci session authenticate. The principal types use the identity of the host (Compute instance, OKE workload, Functions, ...) and ignore the config file.

  • auth_profile (str, default: "DEFAULT" ) –

    Profile name in the OCI config file.

  • auth_file_location (str, default: "~/.oci/config" ) –

    Path of the OCI config file.

  • truncate (str, default: "END" ) –

    How inputs longer than the model context are handled: "NONE" (fail), "START" or "END" (drop tokens from that side).

  • source_input_type (str, default: "SEARCH_DOCUMENT" ) –

    input_type sent when embedding the source column.

  • query_input_type (str, default: "SEARCH_QUERY" ) –

    input_type sent when embedding queries.

  • output_dimensions (int) –

    Embedding size for models that support it (cohere.embed-v4.0: 256, 512, 1024 or 1536). When set, ndims() returns it directly; otherwise the size is discovered with one probe request and cached.

Examples:

import lancedb from lancedb.embeddings import get_registry from lancedb.pydantic import LanceModel, Vector

model = get_registry().get("oci-genai").create( name="cohere.embed-v4.0", compartment_id="ocid1.compartment.oc1..", auth_profile="DEFAULT", )

class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()

db = lancedb.connect("~/.lancedb") tbl = db.create_table("docs", schema=TextModel, mode="overwrite") tbl.add([{"text": "hello world"}, {"text": "goodbye world"}]) rs = tbl.search("hello").limit(1).to_pandas()

When restored from table metadata in a new process, an explicit service_endpoint or non-default region must be approved with allow_endpoint. The default region does not need approval.

name

name: str = 'cohere.embed-v4.0'

compartment_id

compartment_id: Optional[str] = None

region

region: str = _DEFAULT_REGION

service_endpoint

service_endpoint: Optional[str] = None

auth_type

auth_type: Literal['API_KEY', 'SECURITY_TOKEN', 'INSTANCE_PRINCIPAL', 'RESOURCE_PRINCIPAL'] = 'API_KEY'

auth_profile

auth_profile: str = 'DEFAULT'

auth_file_location

auth_file_location: str = '~/.oci/config'

truncate

truncate: Literal['NONE', 'START', 'END'] = 'END'

source_input_type

source_input_type: str = 'SEARCH_DOCUMENT'

query_input_type

query_input_type: str = 'SEARCH_QUERY'

output_dimensions

output_dimensions: Optional[int] = None

model_config

model_config = ConfigDict(ignored_types=(cached_property))

model_names

model_names() -> List[str]

On-demand text embedding models offered by the service.

endpoints

endpoints() -> List[str]

ndims

ndims() -> int

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[Union[List[float], None]]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[Union[List[float], None]]

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[Union[List[float], None]]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

Returns:

  • list[list[float] | None] –

    One embedding per input, in order. Empty inputs are not sent to the service (it rejects them) and come back as None.

OllamaEmbeddings

Bases: TextEmbeddingFunction

An embedding function that uses Ollama

https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings https://ollama.com/blog/embedding-models

name

name: str = 'nomic-embed-text'

host

host: str = DEFAULT_OLLAMA_HOST

options

options: Optional[dict] = None

keep_alive

keep_alive: Optional[Union[float, str]] = None

ollama_client_kwargs

ollama_client_kwargs: Optional[dict] = {}

endpoints

endpoints() -> List[str]

ndims

ndims() -> int

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray]) -> list[Union[array, None]]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

OpenClipEmbeddings

Bases: EmbeddingFunction

An embedding function that uses the OpenClip API For multi-modal text-to-image search

https://github.com/mlfoundations/open_clip

name

name: str = 'ViT-B-32'

pretrained

pretrained: str = 'laion2b_s34b_b79k'

device

device: str = 'cpu'

batch_size

batch_size: int = 64

normalize

normalize: bool = True

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: Union[str, Image], *args, **kwargs) -> List[ndarray]

Compute the embeddings for a given user query

Parameters:

  • query (Union[str, Image]) –

    The query to embed. A query can be either text or an image.

generate_text_embeddings

generate_text_embeddings(text: str) -> ndarray

sanitize_input

sanitize_input(images: IMAGES) -> Union[List[bytes], ndarray]

Sanitize the input to the embedding function.

compute_source_embeddings

compute_source_embeddings(images: IMAGES, *args, **kwargs) -> List[array]

Get the embeddings for the given images

generate_image_embedding

generate_image_embedding(image: Union[str, bytes, Image]) -> ndarray

Generate the embedding for a single image

Parameters:

  • image (Union[str, bytes, Image]) –

    The image to embed. If the image is a str, it is treated as a uri. If the image is bytes, it is treated as the raw image bytes.

OpenAIEmbeddings

Bases: TextEmbeddingFunction

An embedding function that uses the OpenAI API

https://platform.openai.com/docs/guides/embeddings

This can also be used for open source models that are compatible with the OpenAI API.

Notes

If you're running an Ollama server locally, you can just override the base_url parameter and provide the Ollama embedding model you want to use (https://ollama.com/library):

from lancedb.embeddings import get_registry
openai = get_registry().get("openai")
embedding_function = openai.create(
    name="<ollama-embedding-model-name>",
    base_url="http://localhost:11434",
    dim=768,  # the model's embedding dimension; required for unlisted names
    )

name

name: str = 'text-embedding-ada-002'

dim

dim: Optional[int] = None

base_url

base_url: Optional[str] = None

default_headers

default_headers: Optional[dict] = None

organization

organization: Optional[str] = None

api_key

api_key: Optional[str] = None

use_azure

use_azure: bool = False

ndims

ndims()

sensitive_keys

sensitive_keys()

endpoints

endpoints() -> List[str]

model_names

model_names()

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray]) -> List[array]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

NvidiaEmbeddings

Bases: TextEmbeddingFunction

Text embeddings using NVIDIA Nemotron 3.5 Embed 1B or 8B.

Uses the optional openai package for NVIDIA-hosted and self-hosted NIM /v1/embeddings endpoints. Documents use input_type="passage" and queries use input_type="query". Null and empty strings get a null embedding. Image inputs are not supported.

Parameters:

  • name (str) –

    Hosted model ID, or the served model ID from a NIM /v1/models response.

  • dim (int) –

    Requested output dimensions. Defaults to 2048 for the hosted 1B model and 4096 for the hosted 8B model. Required for other served model IDs; use a dimension supported by your deployment.

  • base_url (str) –

    API base URL including /v1. Defaults to NVIDIA's hosted API.

  • api_key (str) –

    Registry variable reference, such as $var:nvidia-key. Otherwise the hosted API uses NVIDIA_API_KEY. Custom endpoints only receive an explicit api_key and can run without authentication.

  • batch_size (int) –

    Maximum inputs per request (default 32). Reduce for deployment limits.

  • truncate (str) –

    END (default) truncates overlong text; NONE rejects it.

  • max_retries (int) –

    Inherited LanceDB retry limit, applied to each request. The OpenAI client's own retries are disabled.

Examples:

Set NVIDIA_API_KEY before using the hosted API:

>>> from lancedb.embeddings import get_registry
>>> from lancedb.pydantic import LanceModel, Vector
>>> func = get_registry().get("nvidia").create()
>>> class Document(LanceModel):
...     text: str = func.SourceField()
...     vector: Vector(func.ndims()) = func.VectorField()

Embedding runs client-side for both local and remote tables. For a custom endpoint restored from table metadata in a new process, first call allow_endpoint.

name

name: str = 'nvidia/nemotron-3.5-embed-1b'

dim

dim: Optional[int] = Field(default=None, gt=0)

base_url

base_url: str = NVIDIA_BASE_URL

api_key

api_key: Optional[str] = Field(default=None, repr=False)

batch_size

batch_size: int = Field(default=32, gt=0)

truncate

truncate: Literal['END', 'NONE'] = 'END'

sensitive_keys

sensitive_keys()

endpoints

endpoints() -> List[str]

ndims

ndims() -> int

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[Optional[List[float]]]

generate_embeddings

generate_embeddings(texts: TEXT) -> List[Optional[List[float]]]

compute_query_embeddings_with_retry

compute_query_embeddings_with_retry(query: str, *args, **kwargs) -> List[Optional[List[float]]]

compute_source_embeddings_with_retry

compute_source_embeddings_with_retry(texts: TEXT, *args, **kwargs) -> List[Optional[List[float]]]

EmbeddingFunctionRegistry

This is a singleton class used to register embedding functions and fetch them by name. It also handles serializing and deserializing. You can implement your own embedding function by subclassing EmbeddingFunction or TextEmbeddingFunction and registering it with the registry.

NOTE: Here TEXT is a type alias for Union[str, List[str], pa.Array, pa.ChunkedArray, np.ndarray]

Examples:

>>> registry = EmbeddingFunctionRegistry.get_instance()
>>> @registry.register("my-embedding-function")
... class MyEmbeddingFunction(EmbeddingFunction):
...     def ndims(self) -> int:
...         return 128
...
...     def compute_query_embeddings(self, query: str, *args, **kwargs):
...         return self.compute_source_embeddings(query, *args, **kwargs)
...
...     def compute_source_embeddings(self, texts, *args, **kwargs):
...         return [np.random.rand(self.ndims()) for _ in range(len(texts))]
...
>>> registry.get("my-embedding-function")
<class 'lancedb.embeddings.registry.MyEmbeddingFunction'>

get_instance

get_instance()

register

register(alias: Optional[str] = None)

This creates a decorator that can be used to register an EmbeddingFunction.

Parameters:

  • alias (Optional[str], default: None ) –

    a human friendly name for the embedding function. If not provided, the class name will be used.

reset

reset()

Reset the registry to its initial state

get

get(name: str) -> Type[EmbeddingFunction]

Fetch an embedding function class by name

Parameters:

  • name (str) –

    The name of the embedding function to fetch Either the alias or the class name if no alias was provided during registration

list_functions

list_functions() -> list[str]

List the names of registered embedding functions.

Returns:

  • list[str] –

    Registered aliases, or class names for functions registered without an alias, in registration order. Each name can be passed to :meth:get. The returned list is a copy; changing it does not change the registry.

Examples:

>>> registry = get_registry()
>>> "openai" in registry.list_functions()
True

parse_functions

parse_functions(metadata: Optional[Dict[bytes, bytes]]) -> Dict[str, EmbeddingFunctionConfig]

Parse the metadata from an arrow table and return a mapping of the vector column to the embedding function and source column

Parameters:

  • metadata (Optional[Dict[bytes, bytes]]) –

    The metadata from an arrow table. Note that the keys and values are bytes (pyarrow api)

Returns:

  • functions ( dict ) –

    A mapping of vector column name to embedding function. An empty dict is returned if input is None or does not contain b"embedding_functions".

allow_remote_code

allow_remote_code(model: str, revision: Optional[str] = None) -> None

Trust the code in a model repository when an embedding function that loads it is restored from table metadata.

Table metadata cannot enable trust_remote_code by itself, since anyone who can write to the table could then run code in this process. Embedding functions created in this process with trust_remote_code=True are allowed automatically.

Parameters:

  • model (str) –

    The model name, such as "nomic-ai/nomic-embed-text-v1.5".

  • revision (Optional[str], default: None ) –

    The revision the table's embedding function must pin. Pinning a commit hash is recommended, since the code at a branch or tag can change after you review it. If None, any revision is trusted, including pull request refs that anyone can push to the repository.

allow_endpoint

allow_endpoint(url: str) -> None

Trust a custom endpoint, such as an OpenAI base_url or Ollama host, when an embedding function that uses it is restored from table metadata.

Table metadata cannot pick an endpoint by itself, since it could send your data and API keys to a server you don't control. Embedding functions created in this process are allowed to use their endpoints automatically.

Parameters:

  • url (str) –

    The endpoint URL. Trailing slashes are ignored.

function_to_metadata

function_to_metadata(conf: EmbeddingFunctionConfig)

Convert the given embedding function and source / vector column configs into a config dictionary that can be serialized into arrow metadata

get_table_metadata

get_table_metadata(func_list)

Convert a list of embedding functions and source / vector configs into a config dictionary that can be serialized into arrow metadata

set_var

set_var(name: str, value: str) -> None

Set a variable. These can be accessed in embedding configuration using the syntax $var:variable_name. If they are not set, an error will be thrown letting you know which variable is missing. If you want to supply a default value, you can add an additional part in the configuration like so: $var:variable_name:default_value. Default values can be used for runtime configurations that are not sensitive, such as whether to use a GPU for inference.

The name must not contain a colon. Default values can contain colons.

get_var

get_var(name: str) -> str

Get a variable.

SentenceTransformerEmbeddings

Bases: TextEmbeddingFunction

An embedding function that uses the sentence-transformers library

https://huggingface.co/sentence-transformers

Parameters:

  • name (str, default: "all-MiniLM-L6-v2" ) –

    The name of the model to use.

  • device (str, default: "cpu" ) –

    The device to use for the model

  • normalize (bool, default: True ) –

    Whether to normalize the embeddings

  • trust_remote_code (bool, default: False ) –

    Whether to run code from the model's repository. Only enable this for repositories you trust. When restored from table metadata, this is only enabled for models allowed with allow_remote_code.

  • revision (Optional[str], default: None ) –

    The model revision to load, such as a commit hash.

name

name: str = 'all-MiniLM-L6-v2'

device

device: str = 'cpu'

normalize

normalize: bool = True

trust_remote_code

trust_remote_code: bool = False

revision

revision: Optional[str] = None

embedding_model

embedding_model

Get the sentence-transformers embedding model specified by the name, device, and trust_remote_code. This is cached so that the model is only loaded once per process.

remote_code_model

remote_code_model() -> Optional[Tuple[str, Optional[str]]]

ndims

ndims()

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray]) -> List[array]

Get the embeddings for the given texts

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

get_embedding_model

get_embedding_model()

Get the sentence-transformers embedding model specified by the name, device, and trust_remote_code. This is cached so that the model is only loaded once per process.

TODO: use lru_cache instead with a reasonable/configurable maxsize

GteEmbeddings

Bases: TextEmbeddingFunction

Deprecated: GTE embeddings should be used through sentence-transformers.

An embedding function that uses GTE-LARGE MLX format(for Apple silicon devices only) as well as the standard cpu/gpu version from: https://huggingface.co/thenlper/gte-large.

For Apple users, you will need the mlx package installed, which can be done with: pip install mlx

Parameters:

  • name (str, default: "thenlper/gte-large" ) –

    The name of the model to use.

  • device (str, default: "cpu" ) –

    Sets the device type for the model.

  • normalize (str, default: "True" ) –

    Controls normalize param in encode function for the transformer.

  • mlx (bool, default: False ) –

    Controls which model to use. False for gte-large,True for the mlx version.

Examples:

import lancedb import lancedb.embeddings.gte from lancedb.embeddings import get_registry from lancedb.pydantic import LanceModel, Vector import pandas as pd

model = get_registry().get("gte-text").create() # mlx=True for Apple silicon class TextModel(LanceModel): text: str = model.SourceField() vector: Vector(model.ndims()) = model.VectorField()

df = pd.DataFrame({"text": ["hi hello sayonara", "goodbye world"]}) db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")

tbl.add(df) rs = tbl.search("hello").limit(1).to_pandas()

name

name: str = 'thenlper/gte-large'

device

device: str = 'cpu'

normalize

normalize: bool = True

mlx

mlx: bool = False

embedding_model

embedding_model

Get the embedding model specified by the flag, name and device. This is cached so that the model is only loaded once per process.

ndims

ndims()

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray]) -> List[array]

Get the embeddings for the given texts.

Parameters:

  • texts (Union[List[str], ndarray]) –

    The texts to embed

get_embedding_model

get_embedding_model()

Get the embedding model specified by the flag, name and device. This is cached so that the model is only loaded once per process.

TransformersEmbeddingFunction

Bases: EmbeddingFunction

An embedding function that can use any model from the transformers library.

Parameters:

name : str The name of the model to use. This should be a model name that can be loaded by transformers.AutoModel.from_pretrained. For example, "bert-base-uncased". default: "colbert-ir/colbertv2.0"" device : str The device to use for the model. Default is "cpu". show_progress_bar : bool Whether to show a progress bar when loading the model. Default is True. trust_remote_code : bool Whether or not to allow for custom models defined on the HuggingFace Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. When restored from table metadata, this is only enabled for models allowed with allow_remote_code. revision : Optional[str] The model revision to load, such as a commit hash. Default is None.

to download package, run : pip install transformers you may need to install pytorch as well - https://pytorch.org/get-started/locally/

name

name: str = 'colbert-ir/colbertv2.0'

device

device: str = 'cpu'

trust_remote_code

trust_remote_code: bool = False

revision

revision: Optional[str] = None

model_config

model_config = ConfigDict(ignored_types=(cached_property))

remote_code_model

remote_code_model() -> Optional[Tuple[str, Optional[str]]]

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: str, *args, **kwargs) -> List[array]

compute_source_embeddings

compute_source_embeddings(texts: TEXT, *args, **kwargs) -> List[array]

ColbertEmbeddings

Bases: TransformersEmbeddingFunction

An embedding function that uses the colbert model from the huggingface library.

Parameters:

name : str The name of the model to use. This should be a model name that can be loaded by transformers.AutoModel.from_pretrained. For example, "bert-base-uncased". default: "colbert-ir/colbertv2.0""

to download package, run : pip install transformers you may need to install pytorch as well - https://pytorch.org/get-started/locally/

name

name: str = 'colbert-ir/colbertv2.0'

ImageBindEmbeddings

Bases: EmbeddingFunction

An embedding function that uses the ImageBind API For generating multi-modal embeddings across six different modalities: images, text, audio, depth, thermal, and IMU data

to download package, run : pip install imagebind-packaged==0.1.2

name

name: str = 'imagebind_huge'

device

device: str = 'cpu'

normalize

normalize: bool = False

model_config

model_config = ConfigDict(ignored_types=(cached_property))

embedding_model

embedding_model

Get the embedding model. This is cached so that the model is only loaded once per process.

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: Union[str], *args, **kwargs) -> List[ndarray]

Compute the embeddings for a given user query

Parameters:

  • query (Union[str]) –

    The query to embed. A query can be either text, image paths or audio paths.

generate_image_embeddings

generate_image_embeddings(image: IMAGES) -> ndarray

generate_audio_embeddings

generate_audio_embeddings(audio: AUDIO) -> ndarray

generate_text_embeddings

generate_text_embeddings(text: TEXT) -> ndarray

compute_source_embeddings

compute_source_embeddings(source: Union[IMAGES, AUDIO], *args, **kwargs) -> List[array]

Get the embeddings for the given sourcefield column in the pydantic model.

sanitize_input

sanitize_input(input: Union[IMAGES, AUDIO]) -> Union[List[bytes], ndarray]

Sanitize the input to the embedding function.

get_embedding_model

get_embedding_model()

fetches the imagebind embedding model

JinaEmbeddings

Bases: EmbeddingFunction

An embedding function that uses the Jina API

https://jina.ai/embeddings/

Parameters:

  • name (str, default: "jina-clip-v1". Note that some models support both image ) –

    and text embeddings and some just text embedding

  • api_key (str, default: None ) –

    The api key to access Jina API. If you pass None, you can set JINA_API_KEY environment variable

name

name: str = 'jina-clip-v1'

api_key

api_key: Optional[str] = None

ndims

ndims()

sensitive_keys

sensitive_keys() -> List[str]

sanitize_input

sanitize_input(inputs: Union[TEXT, IMAGES]) -> Union[List[Any], ndarray]

Sanitize the input to the embedding function.

compute_query_embeddings

compute_query_embeddings(query: Union[str, bytes, Path, Image], *args, **kwargs) -> List[ndarray]

Compute the embeddings for a given user query

Parameters:

  • query (Union[str, Image]) –

    The query to embed. A query can be either text or an image.

compute_source_embeddings

compute_source_embeddings(inputs: Union[TEXT, IMAGES], *args, **kwargs) -> List[array]

generate_image_embedding

generate_image_embedding(image: Union[str, bytes, Path, Image]) -> ndarray

Generate the embedding for a single image

Parameters:

  • image (Union[str, bytes, Image]) –

    The image to embed. If the image is a str, it is treated as a uri. If the image is bytes, it is treated as the raw image bytes.

generate_text_embeddings

generate_text_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[array]

WatsonxEmbeddings

Bases: TextEmbeddingFunction

An embedding function that uses the IBM watsonx.ai Embeddings API.

API Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-embeddings

Supported embedding models: https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx

Parameters:

  • name (str, default: "ibm/slate-125m-english-rtrvr" ) –

    The ID of the embedding model to use. For new tables, "ibm/granite-embedding-278m-multilingual" is recommended.

  • api_key (str) –

    IBM Cloud API key. Falls back to the WATSONX_API_KEY environment variable when not provided.

  • project_id (str) –

    watsonx.ai project ID. Explicit value takes precedence over the WATSONX_PROJECT_ID environment variable. Mutually exclusive with space_id — exactly one must be supplied.

  • space_id (str) –

    watsonx.ai deployment space ID. Explicit value takes precedence over the WATSONX_SPACE_ID environment variable. Mutually exclusive with project_id — exactly one must be supplied.

  • url (str) –

    watsonx.ai service URL. Defaults to "https://us-south.ml.cloud.ibm.com".

  • params (dict) –

    Extra parameters forwarded verbatim to Embeddings (e.g. {"truncate_input_tokens": 512}).

name

name: str = 'ibm/slate-125m-english-rtrvr'

api_key

api_key: Optional[str] = None

project_id

project_id: Optional[str] = None

space_id

space_id: Optional[str] = None

url

url: Optional[str] = None

params

params: Optional[Dict] = None

endpoints

endpoints() -> List[str]

sensitive_keys

sensitive_keys()

model_names

model_names()

Return the IDs of models currently available for new tables.

Legacy / deprecated IDs are intentionally excluded. They remain resolvable for dimension lookups on existing tables via MODELS_DIMS, but should not be used when creating new tables.

ndims

ndims()

generate_embeddings

generate_embeddings(texts: Union[List[str], ndarray], *args, **kwargs) -> List[List[float]]

VoyageAIEmbeddingFunction

Bases: EmbeddingFunction

An embedding function that uses the VoyageAI API

https://docs.voyageai.com/docs/embeddings

Parameters:

  • name (str) –

    The name of the model to use. List of acceptable models:

    * voyage-4 (1024 dims, general-purpose and multilingual retrieval)
    * voyage-4-lite (1024 dims, optimized for latency and cost)
    * voyage-4-large (1024 dims, best retrieval quality)
    * voyage-context-3
    * voyage-3.5
    * voyage-3.5-lite
    * voyage-3
    * voyage-3-lite
    * voyage-multimodal-3
    * voyage-multimodal-3.5
    * voyage-finance-2
    * voyage-multilingual-2
    * voyage-law-2
    * voyage-code-2
    
  • output_dimension (int) –

    The output dimension for models that support flexible dimensions. Currently only voyage-multimodal-3.5 supports this feature. Valid options: 256, 512, 1024 (default), 2048.

Examples:

import lancedb from lancedb.pydantic import LanceModel, Vector from lancedb.embeddings import EmbeddingFunctionRegistry

voyageai = EmbeddingFunctionRegistry .get_instance() .get("voyageai") .create(name="voyage-3")

class TextModel(LanceModel): text: str = voyageai.SourceField() vector: Vector(voyageai.ndims()) = voyageai.VectorField()

data = [ { "text": "hello world" }, { "text": "goodbye world" }]

db = lancedb.connect("~/.lancedb") tbl = db.create_table("test", schema=TextModel, mode="overwrite")

tbl.add(data)

name

name: str

output_dimension

output_dimension: Optional[int] = None

client

client: ClassVar = None

text_embedding_models

text_embedding_models: list = ['voyage-4', 'voyage-4-lite', 'voyage-4-large', 'voyage-3.5', 'voyage-3.5-lite', 'voyage-3', 'voyage-3-lite', 'voyage-finance-2', 'voyage-multilingual-2', 'voyage-law-2', 'voyage-code-2']

multimodal_embedding_models

multimodal_embedding_models: list = ['voyage-multimodal-3', 'voyage-multimodal-3.5']

contextual_embedding_models

contextual_embedding_models: list = ['voyage-context-3']

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: Union[str, Image], *args, **kwargs) -> List[ndarray]

Compute the embeddings for a given user query

Parameters:

  • query (Union[str, Image]) –

    The query to embed. A query can be either text or an image.

Returns:

  • List[np.array]: the list of embeddings –

compute_source_embeddings

compute_source_embeddings(inputs: Union[TEXT, IMAGES], *args, **kwargs) -> List[array]

Compute the embeddings for the inputs

Parameters:

  • inputs (Union[TEXT, IMAGES]) –

    The inputs to embed. The input can be either str, bytes, Path (to an image), PIL.Image or list of these.

Returns:

  • List[np.array]: the list of embeddings –

ColPaliEmbeddings

Bases: EmbeddingFunction

An embedding function that uses the ColPali engine for multimodal multi-vector embeddings.

This embedding function supports ColPali models, producing multivector outputs for both text and image inputs.

Parameters:

  • model_name (str) –

    The name of the model to use (e.g., "Metric-AI/ColQwen2.5-3b-multilingual-v1.0") Supports models based on these engines: - ColPali: "vidore/colpali-v1.3" and others - ColQwen2.5: "Metric-AI/ColQwen2.5-3b-multilingual-v1.0" and others - ColQwen2: "vidore/colqwen2-v1.0" and others - ColSmol: "vidore/colSmol-256M" and others

  • device (str) –

    The device for inference (default "auto").

  • dtype (str) –

    Data type for model weights (default "bfloat16").

  • use_token_pooling (bool) –

    DEPRECATED. Whether to use token pooling. Use pooling_strategy instead.

  • pooling_strategy (str) –

    The token pooling strategy to use, by default "hierarchical". - "hierarchical": Progressively pools tokens to reduce sequence length. - "lambda": A simpler pooling that uses a custom pooling_func.

  • pooling_func (Callable) –

    A function to use for pooling when pooling_strategy is "lambda".

  • pool_factor (int) –

    Factor to reduce sequence length if token pooling is enabled (default 2).

  • quantization_config (Optional[BitsAndBytesConfig]) –

    Quantization configuration for the model. (default None, bitsandbytes needed)

  • batch_size (int) –

    Batch size for processing inputs (default 2).

  • offload_folder (str) –

    Folder to offload model weights if using CPU offloading (default None). This is useful for large models that do not fit in memory.

model_name

model_name: str = 'Metric-AI/ColQwen2.5-3b-multilingual-v1.0'

device

device: str = 'auto'

dtype

dtype: str = 'bfloat16'

use_token_pooling

use_token_pooling: bool = True

pooling_strategy

pooling_strategy: Optional[str] = 'hierarchical'

pooling_func

pooling_func: Optional[Any] = None

pool_factor

pool_factor: int = 2

quantization_config

quantization_config: Optional[Any] = None

batch_size

batch_size: int = 2

offload_folder

offload_folder: Optional[str] = None

ndims

ndims()

Return the dimension of a vector in the multivector output (e.g., 128).

generate_text_embeddings

generate_text_embeddings(text: TEXT) -> List[List[List[float]]]

Generate embeddings for text input.

generate_image_embeddings

generate_image_embeddings(images: IMAGES) -> List[List[List[float]]]

Generate embeddings for a batch of images.

compute_query_embeddings

compute_query_embeddings(query: Union[str, IMAGES], *args, **kwargs) -> List[List[List[float]]]

Compute embeddings for a single user query (text only).

compute_source_embeddings

compute_source_embeddings(images: IMAGES, *args, **kwargs) -> List[List[List[float]]]

Compute embeddings for a batch of source images.

Parameters:

  • images (Union[str, bytes, List, Array, ChunkedArray, ndarray]) –

    Batch of images (paths, URLs, bytes, or PIL Images).

SigLipEmbeddings

Bases: EmbeddingFunction

model_name

model_name: str = 'google/siglip-base-patch16-224'

device

device: str = 'cpu'

batch_size

batch_size: int = 64

normalize

normalize: bool = True

ndims

ndims()

compute_query_embeddings

compute_query_embeddings(query: Union[str, Image], *args, **kwargs) -> List[ndarray]

generate_text_embeddings

generate_text_embeddings(text: str) -> ndarray

sanitize_input

sanitize_input(images: IMAGES) -> Union[List[bytes], ndarray]

compute_source_embeddings

compute_source_embeddings(images: IMAGES, *args, **kwargs) -> List[ndarray]

generate_image_embedding

generate_image_embedding(image: Union[str, bytes, Image]) -> ndarray

get_registry

get_registry() -> EmbeddingFunctionRegistry

Utility function to get the global instance of the registry

Returns:

  • EmbeddingFunctionRegistry –

    The global registry instance

Examples:

from lancedb.embeddings import get_registry

registry = get_registry() openai = registry.get("openai").create()

register

register(name)

Remote configuration

HeaderProvider

Bases: ABC

Abstract base class for providing custom headers for each request.

Users can implement this interface to provide dynamic headers for various purposes such as authentication (OAuth tokens, API keys), request tracking (correlation IDs), custom metadata, or any other header-based requirements. The provider is called before each request to ensure fresh header values are always used.

Error Handling

If get_headers() raises an exception, the request will fail. Implementations should handle recoverable errors internally (e.g., retry token refresh) and only raise exceptions for unrecoverable errors.

get_headers

get_headers() -> Dict[str, str]

Get the latest headers to be added to requests.

This method is called before each request to the remote LanceDB server. Implementations should return headers that will be merged with existing headers.

Returns:

  • Dict[str, str] –

    Dictionary of header names to values to add to the request.

Raises:

  • Exception –

    If unable to fetch headers, the exception will be propagated and the request will fail.

ClientAuthMethod

Bases: str, Enum

How the client authenticates to the OAuth token endpoint.

The method applies to every OAuth request that carries client authentication: client-credentials, authorization-code exchange, refresh-token, and device-authorization requests. The Azure managed identity flow ignores this option.

NONE

NONE = 'none'

No client authentication, for public clients using PKCE or the device flow. Cannot be combined with client_secret.

CLIENT_SECRET_BASIC

CLIENT_SECRET_BASIC = 'client_secret_basic'

HTTP Basic authentication. This is the RFC 6749 recommended method and the normal default for confidential clients, including default Okta applications. Requires client_secret.

CLIENT_SECRET_POST

CLIENT_SECRET_POST = 'client_secret_post'

Credentials in the request body, for providers configured to require it. Requires client_secret.

OAuthConfig

OAuth configuration for LanceDB authentication.

All token acquisition and refresh is handled in the Rust layer. This config is passed through to Rust via PyO3.

Parameters:

  • issuer_url (str) –

    OIDC issuer URL or OAuth authority URL. For Azure: https://login.microsoftonline.com/{tenant_id}/v2.0

  • client_id (str) –

    Application / Client ID.

  • scopes (List[str]) –

    OAuth scopes to request. For Azure managed identity, exactly one scope or resource is required. For example: ["api://{app_id}/.default"]

  • flow (OAuthFlowType, default: CLIENT_CREDENTIALS ) –

    Authentication flow to use. Default: CLIENT_CREDENTIALS.

  • client_secret (Optional[str], default: None ) –

    Client secret (required for CLIENT_CREDENTIALS).

  • client_auth_method (Optional[ClientAuthMethod], default: None ) –

    How the client authenticates to the token endpoint (default: auto). With a client_secret the default is ClientAuthMethod.CLIENT_SECRET_BASIC, which matches the RFC 6749 recommendation and the default configuration of Okta confidential applications; without a secret the client is public and no client authentication is sent.

  • redirect_uri (Optional[str], default: None ) –

    Loopback redirect URI for AUTHORIZATION_CODE. The default is http://127.0.0.1:{callback_port}/callback.

  • callback_port (Optional[int], default: None ) –

    Port for the AUTHORIZATION_CODE loopback callback server (default: 8400).

  • use_pkce (bool, default: True ) –

    Protect AUTHORIZATION_CODE with S256 PKCE (default: True).

  • managed_identity_client_id (Optional[str], default: None ) –

    Client ID for user-assigned managed identity (AZURE_MANAGED_IDENTITY).

  • resource (Optional[str], default: None ) –

    Resource indicator (RFC 8707), forwarded verbatim to authorization and token endpoints, including refresh requests. Must be an absolute URI without a fragment. Not supported for Azure managed identity.

  • audience (Optional[str], default: None ) –

    Provider-specific audience, forwarded to authorization and token endpoints, including refresh requests. Not supported for Azure managed identity.

  • refresh_buffer_secs (Optional[int], default: None ) –

    Seconds before expiry to trigger proactive refresh (default: 300). Keep this well below the token TTL; if it is greater than or equal to the TTL, each request refreshes the token.

  • token_cache (Optional[TokenCacheOptions], default: None ) –

    Opt in to the persistent token cache so short-lived processes reuse one session. Only supported by AUTHORIZATION_CODE and DEVICE_CODE; azure managed identity is rejected. Default: None (memory only).

Examples:

Client Credentials (service-to-service):

>>> config = OAuthConfig(
...     issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
...     client_id="app-id",
...     client_secret="secret",
...     scopes=["api://lancedb-api/.default"],
... )

Providers that require an explicit target can use resource and/or audience (these are forwarded unchanged):

>>> config.resource = "https://api.example.com"
>>> config.audience = "lancedb-api"

Azure Managed Identity:

>>> config = OAuthConfig(
...     issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
...     client_id="app-id",
...     scopes=["api://lancedb-api/.default"],
...     flow=OAuthFlowType.AZURE_MANAGED_IDENTITY,
... )

Authorization Code with PKCE:

The authorization URL is written to standard error before LanceDB tries to open a browser, so it can be copied in headless environments.

>>> config = OAuthConfig(
...     issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
...     client_id="app-id",
...     scopes=["openid", "api://lancedb-api/access"],
...     flow=OAuthFlowType.AUTHORIZATION_CODE,
... )

Device Authorization, with a persistent cache so later processes reuse the session without a new device prompt. The verification URL and user code are written to standard error before polling begins:

>>> config = OAuthConfig(
...     issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
...     client_id="app-id",
...     scopes=["openid", "offline_access", "api://lancedb-api/access"],
...     flow=OAuthFlowType.DEVICE_CODE,
...     token_cache=TokenCacheOptions(),
... )

issuer_url

issuer_url: str

client_id

client_id: str

scopes

scopes: List[str]

flow

flow: OAuthFlowType = CLIENT_CREDENTIALS

client_secret

client_secret: Optional[str] = field(default=None, repr=False)

client_auth_method

client_auth_method: Optional[ClientAuthMethod] = None

redirect_uri

redirect_uri: Optional[str] = None

callback_port

callback_port: Optional[int] = None

use_pkce

use_pkce: bool = True

managed_identity_client_id

managed_identity_client_id: Optional[str] = None

refresh_buffer_secs

refresh_buffer_secs: Optional[int] = None

token_cache

token_cache: Optional[TokenCacheOptions] = None

resource

resource: Optional[str] = None

audience

audience: Optional[str] = None

OAuthFlowType

Bases: str, Enum

OAuth authentication flow types.

CLIENT_CREDENTIALS

CLIENT_CREDENTIALS = 'client_credentials'

Client Credentials grant (service-to-service / M2M).

AUTHORIZATION_CODE

AUTHORIZATION_CODE = 'authorization_code'

Interactive Authorization Code grant, using PKCE by default.

DEVICE_CODE

DEVICE_CODE = 'device_code'

Device Authorization grant for CLI and headless environments.

AZURE_MANAGED_IDENTITY

AZURE_MANAGED_IDENTITY = 'azure_managed_identity'

Azure Managed Identity via IMDS.

OAuthSession

Explicit OAuth session lifecycle for the persistent token cache.

Built from the same :class:OAuthConfig used for :func:lancedb.connect_async (including its token_cache options). A connection created with the same configuration shares the cache, so logging in here prepares tokens for later processes without any database request.

login always runs the configured interactive flow and replaces the cached session (the most recent login wins). logout removes only the local credential; it does not revoke anything with the provider and does not sign out of a browser SSO session.

Examples:

>>> config = OAuthConfig(
...     issuer_url="https://issuer.example.com",
...     client_id="my-app",
...     scopes=["openid", "offline_access"],
...     flow=OAuthFlowType.DEVICE_CODE,
...     token_cache=TokenCacheOptions(),
... )
>>> session = OAuthSession(config)
>>> status = await session.login()
>>> status.refreshable
True

login

login()

Eagerly run the configured flow and store the session.

Returns a :class:lancedb._lancedb.SessionStatus describing the cached session. A successful login always replaces any prior cached session for this identity; if the provider does not issue a refresh token (for example without offline_access), the previous record is removed and refreshable is False.

status

status()

Report whether a cached session exists, with safe metadata.

Never contacts the identity provider and never exposes token values.

logout

logout()

Remove the matching local cached credential.

Returns a :class:lancedb._lancedb.SessionLogout whose removed flag reports whether a credential existed. Logout is idempotent.

TokenCacheOptions

Options for the persistent OAuth token cache.

The cache is opt-in: it is only used when set as token_cache on :class:OAuthConfig. Only refresh tokens are persisted, in a private directory with owner-only permissions, so short-lived processes can reuse an authenticated session instead of re-prompting on every start.

Parameters:

  • cache_dir (Optional[str], default: None ) –

    Directory that holds cached credentials. Defaults to $XDG_CACHE_HOME/lancedb/oauth, $HOME/.cache/lancedb/oauth on Unix, or %LOCALAPPDATA%\lancedb\oauth on Windows. The directory is created with owner-only permissions (0700) when missing.

  • lock_timeout_secs (Optional[int], default: None ) –

    How long to wait for the cross-process refresh lock before failing (default: 30 seconds).

Examples:

>>> opts = TokenCacheOptions(cache_dir="/tmp/my-app/oauth-cache")

Multiple identities (issuer, client, scopes, flow, client authentication) get separate cache entries. Within one identity the most recent login wins.

cache_dir

cache_dir: Optional[str] = None

lock_timeout_secs

lock_timeout_secs: Optional[int] = None

TimeoutConfig

Timeout configuration for remote HTTP client.

timeout

timeout: Optional[timedelta] = None

connect_timeout

connect_timeout: Optional[timedelta] = None

read_timeout

read_timeout: Optional[timedelta] = None

pool_idle_timeout

pool_idle_timeout: Optional[timedelta] = None

RetryConfig

Retry configuration for the remote HTTP client.

retries

retries: Optional[int] = None

connect_retries

connect_retries: Optional[int] = None

read_retries

read_retries: Optional[int] = None

backoff_factor

backoff_factor: Optional[float] = None

backoff_jitter

backoff_jitter: Optional[float] = None

statuses

statuses: Optional[List[int]] = None

TlsConfig

TLS/mTLS configuration for the remote HTTP client.

cert_file

cert_file: Optional[str] = None

key_file

key_file: Optional[str] = None

ssl_ca_cert

ssl_ca_cert: Optional[str] = None

assert_hostname

assert_hostname: bool = True

ClientConfig

Configuration for the HTTP client used to connect to LanceDB Enterprise.

user_agent

user_agent: str = f'LanceDB-Python-Client/{__version__}'

retry_config

retry_config: RetryConfig = field(default_factory=RetryConfig)

timeout_config

timeout_config: Optional[TimeoutConfig] = field(default_factory=TimeoutConfig)

blob_request_concurrency

blob_request_concurrency: Optional[int] = None

extra_headers

extra_headers: Optional[dict] = None

id_delimiter

id_delimiter: Optional[str] = None

tls_config

tls_config: Optional[TlsConfig] = None

header_provider

header_provider: Optional[HeaderProvider] = None

user_id

user_id: Optional[str] = None

Context

lancedb.context.contextualize

contextualize(raw_df: 'pd.DataFrame') -> Contextualizer

Create a Contextualizer object for the given DataFrame.

Used to create context windows. Context windows are rolling subsets of text data.

The input text column should already be separated into rows that will be the unit of the window. So to create a context window over tokens, start with a DataFrame with one token per row. To create a context window over sentences, start with a DataFrame with one sentence per row.

Examples:

>>> from lancedb.context import contextualize
>>> import pandas as pd
>>> data = pd.DataFrame({
...    'token': ['The', 'quick', 'brown', 'fox', 'jumped', 'over',
...              'the', 'lazy', 'dog', 'I', 'love', 'sandwiches'],
...    'document_id': [1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2]
... })

window determines how many rows to include in each window. In our case this how many tokens, but depending on the input data, it could be sentences, paragraphs, messages, etc.

>>> contextualize(data).window(3).stride(1).text_col('token').to_pandas()
                token  document_id
0     The quick brown            1
1     quick brown fox            1
2    brown fox jumped            1
3     fox jumped over            1
4     jumped over the            1
5       over the lazy            1
6        the lazy dog            1
7          lazy dog I            1
8          dog I love            1
9   I love sandwiches            2
10    love sandwiches            2
>>> (contextualize(data).window(7).stride(1).min_window_size(7)
...   .text_col('token').to_pandas())
                                  token  document_id
0   The quick brown fox jumped over the            1
1  quick brown fox jumped over the lazy            1
2    brown fox jumped over the lazy dog            1
3        fox jumped over the lazy dog I            1
4       jumped over the lazy dog I love            1
5   over the lazy dog I love sandwiches            1

stride determines how many rows to skip between each window start. This can be used to reduce the total number of windows generated.

>>> contextualize(data).window(4).stride(2).text_col('token').to_pandas()
                    token  document_id
0     The quick brown fox            1
2   brown fox jumped over            1
4    jumped over the lazy            1
6          the lazy dog I            1
8   dog I love sandwiches            1
10        love sandwiches            2

groupby determines how to group the rows. For example, we would like to have context windows that don't cross document boundaries. In this case, we can pass document_id as the group by.

>>> (contextualize(data)
...     .window(4).stride(2).text_col('token').groupby('document_id')
...     .to_pandas())
                   token  document_id
0    The quick brown fox            1
2  brown fox jumped over            1
4   jumped over the lazy            1
6           the lazy dog            1
9      I love sandwiches            2

min_window_size determines the minimum size of the context windows that are generated.This can be used to trim the last few context windows which have size less than min_window_size. By default context windows of size 1 are skipped.

>>> (contextualize(data)
...     .window(6).stride(3).text_col('token').groupby('document_id')
...     .to_pandas())
                             token  document_id
0  The quick brown fox jumped over            1
3     fox jumped over the lazy dog            1
6                     the lazy dog            1
9                I love sandwiches            2
>>> (contextualize(data)
...     .window(6).stride(3).min_window_size(4).text_col('token')
...     .groupby('document_id')
...     .to_pandas())
                             token  document_id
0  The quick brown fox jumped over            1
3     fox jumped over the lazy dog            1

lancedb.context.Contextualizer

Create context windows from a DataFrame. See lancedb.context.contextualize.

window

window(window: int) -> Contextualizer

Set the window size. i.e., how many rows to include in each window.

Parameters:

  • window (int) –

    The window size.

stride

stride(stride: int) -> Contextualizer

Set the stride. i.e., how many rows to skip between each window.

Parameters:

  • stride (int) –

    The stride.

groupby

groupby(groupby: str) -> Contextualizer

Set the groupby column. i.e., how to group the rows. Windows don't cross groups

Parameters:

  • groupby (str) –

    The groupby column.

text_col

text_col(text_col: str) -> Contextualizer

Set the text column used to make the context window.

Parameters:

  • text_col (str) –

    The text column.

min_window_size

min_window_size(min_window_size: int) -> Contextualizer

Set the (optional) min_window_size size for the context window.

Parameters:

  • min_window_size (int) –

    The min_window_size.

to_df

to_df() -> 'pd.DataFrame'

to_pandas

to_pandas() -> 'pd.DataFrame'

Create the context windows and return a DataFrame.

Pass custom_stop_words to lancedb.index.FTS:

from lancedb.index import FTS

table.create_index(
    "text",
    config=FTS(remove_stop_words=True, custom_stop_words=["acme", "internal"]),
)

The list replaces the built-in stop words and is used only when remove_stop_words=True:

  • custom_stop_words=None uses the built-in list for language.
  • custom_stop_words=[] removes no words.
  • Values are passed through without trimming, lowercasing, or other rewriting.

The same option is available on lancedb.tokenize(...) and the deprecated lancedb.table.Table.create_fts_index compatibility helper:

import lancedb

tokens = list(
    lancedb.tokenize("acme makes searchable data", custom_stop_words=["acme"])
)

lancedb.tokenize

tokenize(query: str, *, base_tokenizer: BaseTokenizerType = 'simple', language: str = 'English', max_token_length: Optional[int] = 40, lower_case: bool = True, stem: Optional[bool] = None, remove_stop_words: Optional[bool] = None, custom_stop_words: Optional[List[str]] = None, ascii_folding: bool = True, ngram_min_length: int = 3, ngram_max_length: int = 3, prefix_only: bool = False, analyzer: Optional[Literal['text', 'code']] = None, split_identifiers: Optional[bool] = None, split_on_numerics: Optional[bool] = None, preserve_original: Optional[bool] = None, index_operators: Optional[bool] = None) -> Iterable[FtsToken]

Tokenize a full-text search query using an explicit tokenizer.

This does not require an FTS index. The tokenizer options match :class:lancedb.index.FTS. custom_stop_words accepts a list of strings. When stem or remove_stop_words is None, the base tokenizer's default applies: False for code and ngram, True otherwise. analyzer and the code flags (split_identifiers, split_on_numerics, preserve_original and index_operators) behave as they do on :class:lancedb.index.FTS.

Examples:

Inspect the identifier and subwords produced by the code analyzer:

>>> import lancedb
>>> tokens = lancedb.tokenize(
...     "parseHTTPRequest", analyzer="code", split_identifiers=True
... )
>>> [token.text for token in tokens]
['parsehttprequest', 'parse', 'http', 'request']

lancedb.FtsToken

position

position: int = <attribute 'position' of 'lancedb._lancedb.FtsToken' objects>

text

text: str = <attribute 'text' of 'lancedb._lancedb.FtsToken' objects>

Blobs

Blob columns store large binary values out of line so they can be read lazily instead of being materialized with the rest of the row.

lancedb.BlobType is lance.blob.BlobType when pylance is installed. Without pylance, LanceDB uses a matching lance.blob.v2 extension type so blob columns still work. Native table queries return descriptors. Call fetch_blob_files for lazy reads or fetch_blobs for eager bytes. LanceDB Embedded supports these fetch APIs for both legacy blob v1 columns (lance-encoding:blob) and blob v2 columns. For legacy columns, obtain row IDs with search().with_row_id(True).to_arrow() and pass the result to the fetch API. One missing or deleted row ID rejects the entire blob-read batch; the APIs do not return partial results.

To construct inline Blob V2 arrays for ingestion in record batches, use blob_array. It accepts bytes and None, preserving the distinction between an empty blob (b"") and a null blob:

import pyarrow as pa
from lancedb.schema import blob, blob_array

schema = pa.schema([pa.field("video_id", pa.string()), blob("video")])
batch = pa.RecordBatch.from_arrays(
    [pa.array(["a", "b", "c"]), blob_array([b"abc", None, b""])],
    schema=schema,
)
table = db.create_table("videos", data=iter([batch]), schema=schema)
table.add(iter([batch]))

create_table(..., schema=...) also casts plain binary values into blob columns declared in the schema. table.add(...) casts them using the table's schema. These conversions apply to Python rows and plain binary Arrow columns. PyArrow's RecordBatch.from_pylist(..., schema=...) cannot convert raw bytes into the blob extension type; use blob_array and RecordBatch.from_arrays when constructing batches explicitly.

For external blob references, construct the descriptor struct described by blob, then wrap it with pa.ExtensionArray.from_storage. Inline data and external uri representations are mutually exclusive.

LanceDB Enterprise SQL returns blob bytes by default. Select blob_descriptor(video) to return an openable reference instead, then use open_blob_file or the lazy iterable open_blob_files:

rows = db.execute_query("""
    SELECT f.timestamp, blob_descriptor(v.video) AS video
    FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
    if row["video"] is not None:
        with videos.open_blob_file(row["video"]) as video:
            frame = extract_frame(video, row["timestamp"])

This example uses a LanceDB Enterprise SQL connection that produces source-bearing descriptors. The descriptor contains kind, position, size, blob_id, blob_uri, and a source struct with uri, version, committed_at_micros, branch, column, row_reference, and row_is_address. The opener verifies the source table and commit and reads the captured version without changing the table handle's checkout. Raw native storage descriptors omit source; use fetch_blob_files(column, row_ids) for those query results.

The singular API accepts a dictionary or Arrow struct scalar. The iterable accepts Arrow arrays/chunked arrays or another iterable, opens handles as iteration advances, and preserves order, duplicates, and nulls as None. Close each non-null handle after use. Async tables provide await table.open_blob_file(...) and async for file in table.open_blob_files(...).

for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
    if video is not None:
        with video:
            frame = extract_frame(video, row["timestamp"])

For an async table, the iterable is consumed with async for:

async for video in async_videos.open_blob_files(rows["video"]):
    if video is not None:
        with video:
            payload = await video.aread()

Payload bytes load on reads. Opening an external blob with unspecified size can issue a metadata-only HEAD request. LanceDB Enterprise reads use authenticated range requests that validate the captured URI and commit timestamp.

Blobs inside lists can be written and queried as descriptors. They are omitted from blob_columns() because row ids do not identify individual list elements, so the fetch APIs cannot read their bytes yet.

lancedb.blob

blob(name: str, nullable: bool = True, *, inline_size_threshold: Optional[int] = None, dedicated_size_threshold: Optional[int] = None, pack_file_size_threshold: Optional[int] = None) -> Field

Create a Lance blob v2 column field.

When pylance is installed this is lance.blob.BlobType.

Parameters:

  • name (str) –

    Field name.

  • nullable (bool, default: True ) –

    Whether the column accepts null values.

  • inline_size_threshold (Optional[int], default: None ) –

    Maximum payload size in bytes to keep inline in the data file before using packed blob storage. Zero is allowed.

  • dedicated_size_threshold (Optional[int], default: None ) –

    Maximum payload size in bytes to store in packed blob storage before using dedicated blob storage. Must be positive.

  • pack_file_size_threshold (Optional[int], default: None ) –

    Maximum size in bytes of a single packed blob sidecar file before a new one is started. Must be positive.

Notes

Thresholds are validated and stored as Arrow field metadata, with the same format as lance.blob_field. They do not require pylance.

The extension's storage is a struct with nullable fields data (large_binary), uri (string), position (uint64), and size (uint64). Each non-null blob must set exactly one of data and uri:

  • Inline bytes use data, leaving all other fields null. b"" is a valid empty blob; use a null struct for a null blob.
  • External blobs use uri, leaving data null. Set both position and size to select a byte range, or leave both null for the whole object. A range's size must be greater than zero.

Use blob_array to build inline blob arrays for Arrow record batches. For external descriptors, build the struct array with field.type.storage_type and wrap it with pa.ExtensionArray.from_storage(field.type, storage).

create_table(..., schema=...) casts plain binary values into blob columns declared by this field, and table.add(...) does the same using the table's schema. PyArrow's RecordBatch.from_pylist does not perform this conversion; construct blob arrays explicitly instead.

lancedb.schema.blob_array

blob_array(payloads: Iterable[Optional[bytes]]) -> ExtensionArray

Build a Lance blob v2 extension array from inline bytes.

Uses the same extension type as blob, with or without pylance installed.

Parameters:

  • payloads (Iterable[Optional[bytes]]) –

    An iterable of bytes or None. None creates a null blob; b"" creates a valid empty blob.

Returns:

  • ExtensionArray –

    Inline blob values ready to use in Arrow tables or record batches.

Raises:

  • TypeError –

    If a payload is neither bytes nor None.

Examples:

>>> import pyarrow as pa
>>> from lancedb.schema import blob, blob_array
>>> schema = pa.schema([blob("video")])
>>> videos = blob_array([b"abc", None, b""])
>>> batch = pa.RecordBatch.from_arrays([videos], schema=schema)
>>> batch.num_rows
3

BlobFile

Bases: RawIOBase

Seekable lazy handle from :meth:~lancedb.table.Table.fetch_blob_files.

Bytes load on read, read1, read_range, or read_ranges, not when the handle is opened. Range reads leave the cursor unchanged. Use :meth:aread from async code.

closed

closed: bool

aread

aread() -> bytes

close

close() -> None

readable

readable() -> bool

seekable

seekable() -> bool

seek

seek(offset: int, whence: int = io.SEEK_SET) -> int

tell

tell() -> int

size

size() -> int

readall

readall() -> bytes

read

read(size: Optional[int] = -1) -> bytes

read1

read1(size: Optional[int] = -1) -> bytes

Read up to size bytes with one underlying read.

read_range

read_range(offset: int, length: int) -> bytes

read_ranges

read_ranges(ranges: list[tuple[int, int]]) -> list[bytes]

Read (offset, length) pairs without moving the cursor.

readinto

readinto(b: WriteableBuffer) -> int

Utilities

lancedb.schema.vector

vector(dimension: int, value_type: DataType = pa.float32()) -> DataType

A help function to create a vector type.

Parameters:

  • dimension (int) –
  • value_type (DataType, default: float32() ) –

    The type of the value in the vector.

Returns:

  • A PyArrow DataType for vectors. –

Examples:

>>> import pyarrow as pa
>>> import lancedb
>>> schema = pa.schema([
...     pa.field("id", pa.int64()),
...     pa.field("vector", lancedb.vector(756)),
... ])

lancedb.merge.LanceMergeInsertBuilder

Bases: object

Builder for a LanceDB merge insert operation

See merge_insert for more context

when_matched_update_all

when_matched_update_all(*, where: Optional[str] = None) -> LanceMergeInsertBuilder

Rows that exist in both the source table (new data) and the target table (old data) will be updated, replacing the old row with the corresponding matching row.

If there are multiple matches then the behavior is undefined. Currently this causes multiple copies of the row to be created but that behavior is subject to change.

Parameters:

  • where (Optional[str], default: None ) –

    An optional filter to limit which rows are updated. Column references in this expression must be prefixed with "target." to refer to the existing table data. For example, to only update rows where the existing color is red, use: where="target.color = 'red'"

when_not_matched_insert_all

when_not_matched_insert_all() -> LanceMergeInsertBuilder

Rows that exist only in the source table (new data) should be inserted into the target table.

when_not_matched_by_source_delete

when_not_matched_by_source_delete(condition: Union[str, Expr, None] = None) -> LanceMergeInsertBuilder

Rows that exist only in the target table (old data) will be deleted. An optional condition can be provided to limit what data is deleted.

Parameters:

  • condition (Union[str, Expr, None], default: None ) –

    If None then all such rows will be deleted. Otherwise the condition will be used as a filter to limit what rows are deleted. Can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

use_index

use_index(use_index: bool) -> LanceMergeInsertBuilder

Controls whether to use indexes for the merge operation.

When set to True (the default), the operation will use an index if available on the join key for improved performance. When set to False, it forces a full table scan even if an index exists. This can be useful for benchmarking or when the query optimizer chooses a suboptimal path.

Parameters:

  • use_index (bool) –

    Whether to use indices for the merge operation. Defaults to True.

use_rte

use_rte(enable: bool) -> LanceMergeInsertBuilder

Control RTE routing for this merge.

By default (unset), a merge_insert on a table with an RTE write spec is routed through Lance's MemWAL shard writer, and a table without one uses the standard path.

Parameters:

  • enable (bool) –

    True forces RTE routing and errors if the table has no RTE write spec. False forces the standard write path even when a spec is set.

validate_single_shard

validate_single_shard(validate_single_shard: bool) -> LanceMergeInsertBuilder

Controls how an LSM merge checks that its input targets a single shard.

When a table has an RTE write spec, every row in a merge_insert call must route to the same shard. When True (the default), every row is inspected to verify this. When False, only the first row is inspected and the shard it routes to is used for the whole input — a faster path for callers that have already pre-sharded their input.

Has no effect on tables without an RTE write spec.

Parameters:

  • validate_single_shard (bool) –

    Whether to check every row routes to one shard. Defaults to True.

execute

execute(new_data: DATA, on_bad_vectors: str = 'error', fill_value: float = 0.0, timeout: Optional[timedelta] = None) -> MergeInsertResult

Executes the merge insert operation

Nothing is returned but the Table is updated

Parameters:

  • new_data (DATA) –

    New records which will be matched against the existing records to potentially insert or update into the table. This parameter can be anything you use for add

  • on_bad_vectors (str, default: 'error' ) –

    What to do if any of the vectors are not the same size or contains NaNs. One of "error", "drop", "fill".

  • fill_value (float, default: 0.0 ) –

    The value to use when filling vectors. Only used if on_bad_vectors="fill".

  • timeout (Optional[timedelta], default: None ) –

    Maximum time to run the operation before cancelling it.

    By default, there is a 30-second timeout that is only enforced after the first attempt. This is to prevent spending too long retrying to resolve conflicts. For example, if a write attempt takes 20 seconds and fails, the second attempt will be cancelled after 10 seconds, hitting the 30-second timeout. However, a write that takes one hour and succeeds on the first attempt will not be cancelled.

    When this is set, the timeout is enforced on all attempts, including the first.

Returns:

  • MergeInsertResult –

    version: the new version number of the table after doing merge insert.

lancedb.otel.instrument_lancedb_metrics

instrument_lancedb_metrics(meter_provider: Optional['MeterProvider'] = None) -> bool

Register LanceDB metrics as OpenTelemetry observable instruments.

Installs a process-global metrics recorder and creates one observable instrument per LanceDB metric on the given (or global) MeterProvider. The user's configured MetricReader then collects them on its own schedule.

Counters and gauges map directly to observable counters/gauges. Each histogram is exported as cumulative le bucket counts (<name>_bucket, with an le attribute) plus <name>_count and <name>_sum.

Parameters:

  • meter_provider (MeterProvider, default: None ) –

    The provider to register instruments on. Defaults to the global provider from opentelemetry.metrics.get_meter_provider().

Returns:

  • bool –

    True if the recorder is installed and instruments are registered. False if a different metrics recorder is already installed in this process (metrics permits only one global recorder), in which case a warning is emitted and no instruments are created.

Notes

Requires the OpenTelemetry API (pip install lancedb[otel]) and, to actually export, an OpenTelemetry SDK (pip install opentelemetry-sdk) configured by the application. Calling this more than once is safe; instruments are created only on the first successful call.

Exceptions

Remote HTTP responses with structured resource details raise a subclass of HttpError. Older servers that omit those details generally raise plain HttpError, so except HttpError remains a compatible catch-all:

from lancedb.remote.errors import (
    AlreadyExistsError,
    FunctionNotFoundError,
    HttpError,
    NamespaceNotFoundError,
    SecretNotFoundError,
)

try:
    db.describe_namespace(["analytics"])
except (SecretNotFoundError, FunctionNotFoundError, NamespaceNotFoundError) as error:
    print(error.error_code, error.resource_kind, error.resource_name)
except AlreadyExistsError as error:
    print("resource already exists:", error.resource_name)
except HttpError as error:
    print("request failed:", error.request_id, error.status_code)

Remote branch deletion sends a single request. If the server rejects it with HTTP 409, it raises ConflictError with the server's reason, including when an older server omits structured resource details. A structured already_exists response keeps its more specific AlreadyExistsError type. Both are subclasses of HttpError.

A remote query whose streamed response stops partway raises ResponseInterruptedError, also a subclass of HttpError. It carries the number of batches and rows already returned and, when the server reports it, the table version the response reflected, so the query can be run again against that version. It is never retried automatically, because the batches already returned would be returned again.

SQL response failures raise SqlError, retaining the numeric grpc_code and request_id. Servers that report a recognized x-lancedb-error-code select a specific input-error subclass. HTTP failures use the corresponding HttpError subclass. Missing metadata stays absent; diagnostic text is never used to guess the error type. An Internal or Unavailable response remains a server failure, even if its message or metadata resembles an input error.

For errors that can arrive over either transport, catch the shared input type:

from lancedb.exceptions import ColumnNotFoundError, SchemaMismatchError
from lancedb.remote.errors import HttpError, SqlError

try:
    db.execute_query("SELECT missing FROM events").read_all()
except (ColumnNotFoundError, SchemaMismatchError) as error:
    print(error.request_id, error.job_id, error.job_request_id)
except (HttpError, SqlError) as error:
    print(error.request_id)

ReferenceNotFoundError identifies a missing branch or table reference. Table errors are TableNotFoundError for HTTP and SqlTableNotFoundError for SQL; both remain compatible with lance_namespace.errors.TableNotFoundError.

A background SQL job still raises JobFailedError. Its __cause__ is the typed SqlError when available, and both carry the current request_id and original worker job_request_id. The outer job ID can differ from sql_job_id. These fields are None when the server does not report them. Older servers produce generic HTTP/SQL exceptions with their transport status and current request ID.

lancedb.remote.errors.HttpError

Bases: LanceDBClientError

An error that occurred during an HTTP request.

is_transport_error

is_transport_error = is_transport_error

job_id

job_id = job_id

job_request_id

job_request_id = job_request_id

error_code

error_code = error_code

resource_kind

resource_kind = resource_kind

resource_name

resource_name = resource_name

namespace_path

namespace_path = tuple(namespace_path) if namespace_path is not None else None

lancedb.remote.errors.ColumnNotFoundError

Bases: HttpError, ColumnNotFoundError

A remote HTTP request named a missing table column.

lancedb.remote.errors.ReferenceNotFoundError

Bases: HttpError, ReferenceNotFoundError

A remote HTTP request named a missing branch or table reference.

lancedb.remote.errors.SchemaMismatchError

Bases: HttpError, SchemaMismatchError

Input to a remote HTTP request does not match the table schema.

lancedb.remote.errors.SqlError

Bases: LanceDBClientError

A SQL response with its gRPC status and correlation fields.

grpc_code is the numeric gRPC code. error_code is an optional machine-readable classification supplied by the server; message text is never used to choose a subclass. job_id and job_request_id are absent unless the server reports an isolated SQL job. request_id identifies the current request. Older servers still produce SqlError without a category.

Examples:

>>> error = SqlError("Column 'missing' was not found", "req-1", 3,
...                  "column_not_found", "job-1", "worker-1")
>>> isinstance(error, SqlColumnNotFoundError)
True
>>> error.job_request_id
'worker-1'

grpc_code

grpc_code = grpc_code

error_code

error_code = error_code

job_id

job_id = job_id

job_request_id

job_request_id = job_request_id

lancedb.remote.errors.SqlColumnNotFoundError

Bases: SqlError, ColumnNotFoundError

A SQL request named a missing column.

lancedb.remote.errors.SqlTableNotFoundError

Bases: SqlError, ValueError, TableNotFoundError

A SQL request named a missing table.

lancedb.remote.errors.SqlReferenceNotFoundError

Bases: SqlError, ReferenceNotFoundError

A SQL request named a missing branch or table reference.

lancedb.remote.errors.SqlSchemaMismatchError

Bases: SqlError, SchemaMismatchError

SQL input does not match the table schema.

lancedb.exceptions.ColumnNotFoundError

Bases: ValueError

A requested table column does not exist.

Remote HTTP and SQL errors inherit this type and carry request metadata. Locally rejected queries have no request or job identifiers.

request_id

request_id: Optional[str] = None

job_id

job_id: Optional[str] = None

job_request_id

job_request_id: Optional[str] = None

lancedb.exceptions.ReferenceNotFoundError

Bases: ValueError

A requested branch or table reference does not exist.

request_id

request_id: Optional[str] = None

job_id

job_id: Optional[str] = None

job_request_id

job_request_id: Optional[str] = None

lancedb.exceptions.SchemaMismatchError

Bases: ValueError

Input data does not match the table's schema.

request_id

request_id: Optional[str] = None

job_id

job_id: Optional[str] = None

job_request_id

job_request_id: Optional[str] = None

lancedb.remote.errors.TableNotFoundError

Bases: HttpError, ValueError, TableNotFoundError

A remote table was not found.

This is also a lance_namespace.errors.TableNotFoundError and a ValueError, so existing missing-table handlers continue to work. Like HttpError, it carries the request_id and status_code of the failed request.

Examples:

>>> error = TableNotFoundError("Table 'missing' was not found", "req-123", 404)
>>> isinstance(error, ValueError)
True
>>> error.request_id
'req-123'

lancedb.remote.errors.SecretNotFoundError

Bases: HttpError

A remote secret was not found.

lancedb.remote.errors.FunctionNotFoundError

Bases: HttpError

A remote function was not found.

lancedb.remote.errors.NamespaceNotFoundError

Bases: HttpError

A remote namespace was not found.

lancedb.remote.errors.AlreadyExistsError

Bases: HttpError

A remote resource already exists.

lancedb.remote.errors.ConflictError

Bases: HttpError

A remote operation conflicts with the current state of a resource.

For example, deleting a branch whose files are still needed by another branch raises this error. The server's reason is included in the message.

Examples:

>>> error = HttpError("main depends on the branch's files", "req-1", 409)
>>> isinstance(error, ConflictError)
True
>>> error.status_code
409

lancedb.remote.errors.ResponseInterruptedError

Bases: HttpError

A streamed query response stopped before the server finished sending it.

The batches already returned are valid, but the result is incomplete. The query is not retried automatically, because a new attempt would return those rows a second time. To read the same data in a new attempt, check out table_version first.

Examples:

>>> error = ResponseInterruptedError("interrupted", "req-1", 2, 2048, 7)
>>> isinstance(error, HttpError)
True
>>> (error.rows, error.table_version)
(2048, 7)

batches

batches = batches

rows

rows = rows

table_version

table_version = table_version

lancedb.exceptions.MissingValueError

Bases: ValueError

Exception raised when a required value is missing.

lancedb.exceptions.MissingColumnError

Bases: KeyError

Exception raised when a column name specified is not in the DataFrame object

column_name

column_name = column_name

lancedb.exceptions.JobNotFoundError

Bases: ValueError

Exception raised when opening a job the server does not have.

lancedb.exceptions.JobFailedError

Bases: RuntimeError

Exception raised when an asynchronous job reaches the failed state.

job_id identifies the failed job when available. sql_job_id preserves the SQL execution identity reported by the server, which may differ from the outer job's identity. request_id and job_request_id identify the current request and isolated worker request. A typed SQL cause is available through __cause__ when the server reports one.

job_id

job_id = job_id

sql_job_id

sql_job_id = sql_job_id

request_id

request_id = request_id

job_request_id

job_request_id = job_request_id

lancedb.exceptions.JobCancelledError

Bases: RuntimeError

Exception raised when an asynchronous job was cancelled.

Integrations

Pydantic

lancedb.pydantic.pydantic_to_schema

pydantic_to_schema(model: Type[BaseModel]) -> Schema

Convert a Pydantic Model to a PyArrow Schema.

Parameters:

  • model (Type[BaseModel]) –

    The Pydantic BaseModel to convert to Arrow Schema.

Returns:

  • Schema –

    The Arrow Schema

Raises:

  • TypeError –

    If a field has a type with no Arrow equivalent. A Tuple field becomes an Arrow list, so all its elements must have the same type, as in Tuple[int, int] or Tuple[float, ...]; for a fixed set of fields with different types, such as Tuple[int, str], use a nested LanceModel (an Arrow struct) instead. Enum values must also all have the same type.

Examples:

>>> from typing import List, Optional
>>> import pydantic
>>> from lancedb.pydantic import pydantic_to_schema, Vector
>>> class FooModel(pydantic.BaseModel):
...     id: int
...     s: str
...     vec: Vector(1536)  # fixed_size_list<item: float32>[1536]
...     li: List[int]
...
>>> schema = pydantic_to_schema(FooModel)
>>> assert schema == pa.schema([
...     pa.field("id", pa.int64(), False),
...     pa.field("s", pa.utf8(), False),
...     pa.field("vec", pa.list_(pa.float32(), 1536)),
...     pa.field("li", pa.list_(pa.int64()), False),
... ])

lancedb.pydantic.vector

vector(dim: int, value_type: DataType = pa.float32())

lancedb.pydantic.Vector

Vector(dim: int, value_type: DataType = pa.float32(), nullable: bool = True) -> Type[FixedSizeListMixin]

Pydantic Vector Type.

Warning

Experimental feature.

Parameters:

  • dim (int) –

    The dimension of the vector.

  • value_type (DataType, default: float32() ) –

    The value type of the vector, by default pa.float32()

  • nullable (bool, default: True ) –

    Whether the vector is nullable, by default it is True.

Examples:

>>> import pydantic
>>> from lancedb.pydantic import Vector
...
>>> class MyModel(pydantic.BaseModel):
...     id: int
...     url: str
...     embeddings: Vector(768)
>>> schema = pydantic_to_schema(MyModel)
>>> assert schema == pa.schema([
...     pa.field("id", pa.int64(), False),
...     pa.field("url", pa.utf8(), False),
...     pa.field("embeddings", pa.list_(pa.float32(), 768))
... ])
Notes

Vector creates a type dynamically, so calls such as Vector(768) are not valid static type annotations. For an embedding field, use the standard list[float] annotation when running mypy; VectorField supplies the fixed dimension to LanceDB::

class MyModel(LanceModel):
    text: str = embeddings.SourceField()
    vector: list[float] = embeddings.VectorField()

lancedb.pydantic.MultiVector

MultiVector(dim: int, value_type: DataType = pa.float32(), nullable: bool = True) -> Type

Pydantic MultiVector Type for multi-vector embeddings.

This type represents a list of vectors, each with the same dimension. Useful for models that produce multiple embeddings per input, like ColPali.

Parameters:

  • dim (int) –

    The dimension of each vector in the multi-vector.

  • value_type (DataType, default: float32() ) –

    The value type of the vectors, by default pa.float32()

  • nullable (bool, default: True ) –

    Whether the multi-vector is nullable, by default it is True.

Examples:

>>> import pydantic
>>> from lancedb.pydantic import MultiVector
...
>>> class MyModel(pydantic.BaseModel):
...     id: int
...     text: str
...     embeddings: MultiVector(128)  # List of 128-dimensional vectors
>>> schema = pydantic_to_schema(MyModel)
>>> assert schema == pa.schema([
...     pa.field("id", pa.int64(), False),
...     pa.field("text", pa.utf8(), False),
...     pa.field("embeddings", pa.list_(pa.list_(pa.float32(), 128)))
... ])

lancedb.pydantic.LanceModel

Bases: BaseModel

A Pydantic Model base class that can be converted to a LanceDB Table.

Examples:

>>> import lancedb
>>> from lancedb.pydantic import LanceModel, Vector
>>>
>>> class TestModel(LanceModel):
...     name: str
...     vector: Vector(2)
...
>>> db = lancedb.connect("./example")
>>> table = db.create_table("test", schema=TestModel)
>>> table.add([
...     TestModel(name="test", vector=[1.0, 2.0])
... ])
AddResult(version=2)
>>> table.search([0., 0.]).limit(1).to_pydantic(TestModel)
[TestModel(name='test', vector=FixedSizeList(dim=2))]

to_arrow_schema

to_arrow_schema()

Get the Arrow Schema for this model.

field_names

field_names() -> List[str]

Get the field names of this model.

safe_get_fields

safe_get_fields()

parse_embedding_functions

parse_embedding_functions() -> List['EmbeddingFunctionConfig']

Parse the embedding functions from this model.

PyTorch

lancedb.streaming.StreamingDataset

Bases: IterableDataset

An elastic, resumable PyTorch IterableDataset backed by a LanceDB table.

The table is partitioned into num_splits fixed splits using a deterministic random shuffle controlled by shuffle_seed and epoch. Each rank is assigned a contiguous block of splits, and within a rank each DataLoader worker is assigned a contiguous sub-block. Samples are yielded by round-robining over the assigned splits, one sample per split per cycle.

Internally __iter__ runs a two-stage pipeline:

  • Stage 1 (I/O): one thread pool with num_splits * io_queue_depth workers fetches raw RecordBatch objects from LanceDB in parallel across all splits and places them in a per-split raw-batch queue.
  • Stage 2 (transform): a second thread pool with transform_parallelism workers picks up raw batches, applies the transform, and places the results in a per-split cooked-row queue. By default, the number of workers is determined by os.cpu_count().

The main thread round-robins over the cooked queues, yielding one row per split per cycle.

Parameters:

  • table –

    LanceDB table to stream from.

  • num_splits (Optional[int], default: None ) –

    Number of fixed splits to partition the table into. Must be divisible by world_size. When used with DataLoader workers it must also be divisible by world_size * num_workers. Defaults to world_size. If the row count (after any filter) is not evenly divisible by num_splits, the surplus rows — at most num_splits - 1 per epoch — are silently dropped to keep all splits the same length.

  • shuffle (bool, default: True ) –

    Whether to randomly assign rows to splits. When True (the default) rows are shuffled using shuffle_seed and epoch. When False rows are divided into splits sequentially in storage order, which can be useful for deterministic debugging or evaluation. Ignored when split_strategy is provided.

  • split_strategy (Optional[Callable[[PermutationBuilder], PermutationBuilder]], default: None ) –

    Escape hatch for any PermutationBuilder split strategy this class doesn't otherwise expose a dedicated parameter for -- split_hash, split_calculated, bootstrap_sample (uniform sampling with replacement), or any strategy added later. A callable that receives a fresh PermutationBuilder for table and returns a configured one; it is called and .execute()'d directly, taking over permutation construction entirely. When provided, shuffle, shuffle_seed, epoch, and shuffle_clump_size are ignored -- the callable is solely responsible for how rows are split and ordered. For example, split_strategy=lambda b: b.bootstrap_sample(10_000, fixed=num_splits, seed=42) draws 10,000 samples with replacement, divided evenly across num_splits splits.

  • shuffle_seed (Optional[int], default: 0 ) –

    Base seed for the random permutation. Combined with epoch so each epoch produces a different ordering. Pass None to generate a random seed at construction time.

  • epoch (int, default: 0 ) –

    Current training epoch. Combined with shuffle_seed so that each epoch produces a different sample ordering.

  • rank (int, default: 0 ) –

    This process's rank in the distributed training group.

  • world_size (int, default: 1 ) –

    Total number of processes in the distributed training group.

  • read_batch_size (int, default: DEFAULT_READ_BATCH_SIZE ) –

    Number of rows fetched from each split in a single take_offsets call. Larger values amortise per-request overhead (critical on object storage) at the cost of higher memory usage per split buffer. Defaults to DEFAULT_READ_BATCH_SIZE (64).

  • io_queue_depth (int, default: DEFAULT_PREFETCH_BATCHES ) –

    Number of fetched batches to keep ahead of the transform stage per split: this bounds the combined count of in-flight fetches and already-fetched batches still waiting for a transform slot. Higher values overlap storage latency with transform and training compute at the cost of more memory and threads, but I/O stops running ahead once this many batches are buffered, applying backpressure when the transform stage (or transform_queue_depth) is the bottleneck. Must be greater than zero. Defaults to DEFAULT_PREFETCH_BATCHES (4).

  • columns (Optional[list[str]], default: None ) –

    Optional list of column names to read. When set, only those columns are fetched from storage; all others are omitted. None (the default) reads every column.

  • shuffle_clump_size (Optional[int], default: None ) –

    When set, rows are shuffled in contiguous groups of this size rather than individually. Larger clumps improve I/O locality (important on object storage) at the cost of reduced randomness. None (the default) shuffles rows individually.

  • filter (Optional[str], default: None ) –

    Optional SQL filter expression (e.g. "label = 'dog'"). Only rows that satisfy the predicate are included in the permutation. The filter is applied during permutation construction so split sizes reflect the filtered row count.

  • transform (Optional[Callable], default: None ) –

    Optional callable applied to each pyarrow.RecordBatch before rows are yielded. Receives one batch at a time and must return an iterable whose length equals the number of rows in the batch. When None (the default) rows are returned as plain Python dicts.

  • transform_parallelism (Optional[int], default: None ) –

    Maximum number of transforms to run concurrently. Must be greater than zero. When None (the default), uses os.cpu_count() or 1 when the CPU count is unavailable.

  • pack_sequences (Optional[int], default: None ) –

    Sequence-packing mode: token lists from consecutive documents are joined with eos_id and sliced into blocks of this many tokens. Each item is then a dict of two (pack_sequences,) LongTensors — input_ids and doc_ids (per-position document index within the block, for block-diagonal masks or position-id resets). * Packing happens independently per owned split and preserves per-split resume state. * When a split cannot fill a real block for a cycle but an owned sibling still can, or when only a short tail remains at epoch end, the short buffer is padded to pack_sequences with pad_id so every local cycle emits one block per owned split. * eos_id, pad_id, and columns naming a single integer-list column are required; incompatible with transform.

  • eos_id (Optional[int], default: None ) –

    Separator token id between packed documents. Required with pack_sequences, ignored otherwise.

  • pad_id (Optional[int], default: None ) –

    Padding token id used to complete blocks when a split runs out of real tokens mid-cycle or at epoch end. Required with pack_sequences, ignored otherwise. It must be reserved for padding: padding positions retain the preceding document's doc_id (or zero in an all-padding block), so callers must mask them separately using input_ids == pad_id.

  • blocks_per_epoch (Optional[Union[int, Literal['auto']]], default: None ) –

    Total number of packed blocks emitted globally per epoch. Required with pack_sequences. An integer must be divisible by num_splits. Every logical split emits exactly blocks_per_epoch / num_splits blocks: exhausted splits emit padding, while tokens beyond the budget are left out of the epoch. This fixed per-split budget keeps packed iteration and checkpoints independent of rank topology. Pass "auto" to estimate a corpus-level budget from a bounded sample of token lists. The estimate may be inaccurate.

  • on_transform_error (Union[str, Callable[[Exception], bool]], default: 'raise' ) –

    What to do when the transform raises an exception:

    • "raise" (the default): the exception propagates and iteration aborts.
    • "skip": the failing rows are dropped and iteration continues.
    • "warn": like "skip", but a warning is logged for each failing batch.
    • "fill": replace each failing row with fill_value or the result of repair(exc, row_batch). Exactly one of these must be provided. Filled rows retain their original positions, so split lengths, elastic determinism, and checkpoint offsets are preserved.
    • a callable handler(exc) -> bool: called with the exception; return True to skip the failing rows or False to re-raise. Useful to skip only expected error types (compatible with webdataset.handlers style handlers).

    When a batch fails, the transform is re-invoked on each single-row slice of the batch so that only the rows that actually fail are skipped or filled. Transforms should therefore be deterministic and accept batches of any size (including one row). Skipped and filled rows are counted in rows_skipped and rows_filled respectively.

    Skipping weakens the elastic-determinism guarantee at the end of the epoch: splits that lose more rows than others run dry earlier, and each rank's iterator ends at the last cycle where every split it owns still has a row. Because bad rows are not distributed evenly across splits, this means one rank's iterator can yield noticeably fewer or more steps than another rank's in the same run — there is no cross-rank coordination that stops every rank at the same global step. This is generally safe for asynchronous or single-rank use, but synchronous distributed training (e.g. ranks that call all_reduce every step) can hang or deadlock if one rank's iterator is exhausted while others are still stepping; callers doing synchronous multi-rank training with on_transform_error="skip" or "warn" (or a skipping callable) are responsible for their own cross-rank stopping mechanism (e.g. broadcasting a stop signal on StopIteration). The final few global steps can also differ across topologies (bounded by the skew in bad-row counts across splits). The sequence of samples yielded from each split remains deterministic. Mid-epoch checkpoints remain exact provided the transform fails deterministically; in multi-rank training each rank must save its own state_dict and the states must be combined with merge_state_dicts before resuming on a different topology. Prefer the filter parameter when bad rows can be expressed as a SQL predicate (e.g. "col IS NOT NULL") — filtering happens before splits are built, so every guarantee is fully preserved.

  • fill_value (Any, default: _UNSET ) –

    Constant output row used with on_transform_error="fill". A copy is made for each failed row so mutable placeholders are independent. For example, fill_value={"vector": [0.0, 0.0]}. Pass exactly one of fill_value or repair in fill mode.

  • repair (Optional[Callable[[Exception, RecordBatch], Any]], default: None ) –

    Optional function repair(exc, row_batch) -> row used with on_transform_error="fill". row_batch is a one-row pyarrow.RecordBatch. The returned value replaces that row's transformed output. Replacements should have the same structure as successful transform outputs. The repair must be deterministic for reproducible checkpoints and elastic steps.

  • transform_queue_depth (Optional[int], default: None ) –

    Number of transform-result batches to buffer per split in the post-transform queue before backpressure is applied to the transform stage. When the combined count of in-flight transform futures and already-buffered rows for a split reaches transform_queue_depth * read_batch_size, no new transforms are submitted for that split until the consumer catches up. Useful for capping peak memory when the consumer (e.g. a GPU training step) is slower than the transform stage. Must be greater than zero. None (the default) imposes no limit.

  • connection_factory (Optional[Callable[[str], Any]], default: None ) –

    Optional picklable function that takes the table name and returns an open LanceDB table. When set, pickling the dataset (as DataLoader workers do under the spawn and forkserver start methods) replaces the table's reopen state with its name, and each worker calls the factory with that name to reopen the table. The factory itself is pickled too, so credentials stay out of the pickle only if the factory loads them in the worker, for example from the environment, instead of capturing them. None (the default) pickles the table's own reopen state instead.

  • worker_info_override –

    If set, used in place of torch.utils.data.get_worker_info() to determine the DataLoader worker assignment. Intended for unit tests that need to simulate multiple workers without spawning real processes. If both this and the real worker info are non-None a warning is logged and the override takes precedence.

bytes_loaded

bytes_loaded: int

Cumulative bytes of raw Arrow buffer data fetched from storage.

Measured on the RecordBatch before any transform is applied, so the value reflects actual I/O rather than the size of transformed output. Accumulates across multiple iterations of the same dataset instance and is never reset automatically.

fetch_time

fetch_time: float

Cumulative seconds spent waiting for data from LanceDB.

Measured per batch in the Stage 1 I/O threads as the total elapsed time of the take_offsets call. Accumulates across all splits and all iterations.

transform_time

transform_time: float

Cumulative seconds spent applying the transform.

Measured per batch in the Stage 2 transform threads as the elapsed time inside the transform callable (or the default arrow2python conversion when no transform is set). Accumulates across all splits and all iterations.

raw_queue_depth

raw_queue_depth: int

Number of raw rows waiting for a transform thread across all splits.

A persistently non-zero value means Stage 2 (transform) is the bottleneck: I/O is completing faster than transforms can consume batches. Returns 0 when not iterating.

prefetch_queue_depth

prefetch_queue_depth: int

Number of rows transformed and ready to yield across all splits.

Counts rows whose transform has completed and are sitting in memory waiting for the main thread — rows that can be handed off with no I/O or CPU wait. Returns 0 when not iterating.

unscanned_rows

unscanned_rows: int

Number of rows not yet submitted to the I/O stage across all splits.

Decreases as the I/O stage submits fetch requests. When this reaches zero all data has been requested from storage (though it may not have arrived yet). Returns 0 when not iterating.

rows_skipped

rows_skipped: int

Number of rows dropped because their transform raised an exception.

Only ever non-zero when on_transform_error is set to "skip", "warn", or a callable that returned True. Accumulates across multiple iterations of the same dataset instance and is never reset automatically.

rows_filled

rows_filled: int

Number of failed transform rows replaced by fill mode.

Accumulates across iterations of the same dataset instance and is never reset automatically.

consumed_rows

consumed_rows: int

Number of rows already yielded to the caller across all splits.

Monotonically increases throughout iteration. Returns 0 when not iterating.

state_dict

state_dict() -> dict

Snapshot the dataset's consumption state.

When using DataLoader workers, construct a StreamingDataLoader. It commits worker state only when a prefetched batch is returned to the trainer. A standard multi-process DataLoader cannot expose that boundary, so calling this method after one has started raises RuntimeError instead of returning stale producer state.

In row mode, the returned dict is topology-independent at global step boundaries. positions_consumed_per_split records how far each split's permutation has advanced, which can differ from the sample count when on_transform_error skips rows. StreamingDataLoader combines worker state in its parent process. Combine state dicts from every rank with merge_state_dicts before resuming on a different topology.

Packed state includes partial token buffers and emitted block counts for every logical split. When packing is sharded, merge every rank state with merge_state_dicts before loading it.

load_state_dict

load_state_dict(state: dict) -> None

Resume from a previously snapshotted state.

Raises ValueError if num_splits or shuffle_seed differ from the checkpoint, since a different split structure or shuffle order makes mid-epoch resumption meaningless. Also raises if the checkpoint was built with split_strategy and records a permutation fingerprint that does not match the current dataset's permutation -- this catches a changed or non-deterministically-reseeded split_strategy that num_splits/shuffle_seed alone cannot detect, since shuffle_seed is unused on that path. Checkpoints from split_random/ split_sequential never carry this fingerprint (row ids are not stable across compaction, so it would reject an otherwise-valid resume), and continue to rely on shuffle_seed/num_splits alone as before. Packed checkpoints pin pack_sequences, eos_id, pad_id, blocks_per_epoch, and epoch.

merge_state_dicts

merge_state_dicts(states: list[dict]) -> dict

Merge state dicts saved by different ranks into one exact state.

In row mode, each rank records exact consumer-committed progress for its own splits and lower bounds for the rest, so elementwise maxima recover both sample counts and permutation positions. In packed mode, the state that emitted the most blocks for each logical split supplies that split's permutation position and partial token buffer. Packed states must cover every rank at the same global step.

Raises ValueError if the states are empty, were not produced by the same run, or do not represent the same global step.

The merge is always all-to-all and topology-agnostic: collect the state_dict() from every rank of the previous run into one list, merge that whole list, and hand the identical merged result to every rank of the next run — regardless of whether the topology grew, shrank, or stayed the same. There is no pairwise or subset merging step, because each split's exact state is only known to whichever iterator owned that split.

For example, checkpointing 8 ranks and resuming on 4 (the same pattern applies when growing, e.g. 4 ranks resuming on 8)::

states = [ds.state_dict() for ds in previous_run_datasets]  # 8
merged = StreamingDataset.merge_state_dicts(states)
for ds in resumed_datasets:  # now only 4 ranks
    ds.load_state_dict(merged)  # same dict on every rank

The rank count on either side never affects the merge itself, since merge_state_dicts only cares about the list of states it is given. Each split's position is recovered by elementwise maximum; here rank 0 owned split 0 (and skipped two rows there) while rank 1 owned split 1 (and skipped one row):

rank0 = { ... "shuffle_seed": 0, "num_splits": 2, "epoch": 0, ... "samples_consumed_per_split": [3, 3], ... "positions_consumed_per_split": [5, 3], ... } rank1 = { ... "shuffle_seed": 0, "num_splits": 2, "epoch": 0, ... "samples_consumed_per_split": [3, 3], ... "positions_consumed_per_split": [3, 4], ... } merged = StreamingDataset.merge_state_dicts([rank0, rank1]) merged["positions_consumed_per_split"] [5, 4]

When a transform fails for some rows, fill mode replaces each failed row in place:

dataset = StreamingDataset(
    table,
    num_splits=8,
    transform=transform,
    on_transform_error="fill",
    fill_value={"id": -1, "vector": [0.0, 0.0]},
)
for row in dataset:
    train(row)
print(dataset.rows_filled)

The placeholder must have the same structure as a successful transform output. Pass repair(exc, row_batch) -> row instead of fill_value to construct a replacement from the failed one-row Arrow batch. Keep the repair deterministic when using elastic training or checkpoints. Fill mode preserves split lengths and sample positions.

lancedb.streaming.StreamingDataLoader

Bases: DataLoader

A PyTorch DataLoader with consumer-committed dataset checkpoints.

PyTorch workers prefetch batches ahead of the trainer, so worker-local producer progress is not a safe checkpoint. This loader carries a state snapshot alongside every internal batch and applies it to the parent StreamingDataset only when that batch is returned by next(). The trainer receives the same collated batch it would receive from a standard torch.utils.data.DataLoader.

With more than one worker, row-mode state_dict() is available only at complete logical step boundaries, when every split assigned to the rank has the same consumed-sample count. Packed checkpoints require equal emitted-block counts across the rank's splits for any worker count. persistent_workers=True is not supported because prefetched worker copies cannot be restored from parent-committed state. If batch collation raises, checkpointing remains invalid for that dataset instance; restore the last valid checkpoint into a fresh dataset before continuing. Only one active iterator may own a dataset at a time, including when worker processes are used. Exhausting or explicitly shutting down the iterator releases that ownership. drop_last=True is not supported because worker replicas discard incomplete tails independently, which cannot produce a topology-independent checkpoint.

Parameters are the same as torch.utils.data.DataLoader except that dataset must be a StreamingDataset. Subclasses that override StreamingDataset.__iter__ are not supported because the custom iterator cannot provide the exact per-yield checkpoint snapshots required by this loader.

Examples:

>>> # dataset = StreamingDataset(table, num_splits=2)
>>> # loader = StreamingDataLoader(dataset, batch_size=8, num_workers=2)
>>> # batch = next(iter(loader))
>>> # checkpoint = dataset.state_dict()

collate_fn

collate_fn = _CheckpointCollate(collate_fn)

lancedb.permutation.permutation_builder

permutation_builder(table: Table) -> PermutationBuilder

lancedb.permutation.PermutationBuilder

A utility for creating a "permutation table" which is a table that defines an ordering on a base table.

The permutation table does not store the actual data. It only stores row ids and split ids to define the ordering. The [Permutation] class can be used to read the data from the base table in the order defined by the permutation table.

Permutations can split, shuffle, and filter the data in the base table.

A filter limits the rows that are included in the permutation. Splits divide the data into subsets (for example, a test/train split, or K different splits for cross-validation). Shuffling randomizes the order of the rows in the permutation.

Splits can optionally be named. If names are provided it will enable them to be referenced by name in the future. If names are not provided then they can only be referenced by their ordinal index. There is no requirement to name every split.

The permutation is stored in memory and will be lost when the program exits.

split_random

split_random(*, ratios: Optional[list[float]] = None, counts: Optional[list[int]] = None, fixed: Optional[int] = None, seed: Optional[int] = None, clump_size: Optional[int] = None, split_names: Optional[list[str]] = None) -> PermutationBuilder

Configure random splits for the permutation.

One of ratios, counts, or fixed must be provided.

If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows.

If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows.

If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits.

Rows will be randomly assigned to splits. The optional seed can be provided to make the assignment deterministic.

If clump_size is provided, rows are shuffled as contiguous groups of that size, preserving I/O locality while still randomising the split assignment.

The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.

split_hash

split_hash(columns: list[str], split_weights: list[int], *, discard_weight: Optional[int] = None, split_names: Optional[list[str]] = None) -> PermutationBuilder

Configure hash-based splits for the permutation.

First, a hash will be calculated over the specified columns. The splits weights are then used to determine how many rows to assign to each split. For example, if split weights are [1, 2] then the first split will contain 1/3 of the rows and the second split will contain 2/3 of the rows.

The optional discard weight can be provided to determine what percentage of rows should be discarded. For example, if split weights are [1, 2] and discard weight is 1 then 25% of the rows will be discarded.

Hash-based splits are useful if you want the split to be more or less random but you don't want the split assignments to change if rows are added or removed from the table.

The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.

split_sequential

split_sequential(*, ratios: Optional[list[float]] = None, counts: Optional[list[int]] = None, fixed: Optional[int] = None, split_names: Optional[list[str]] = None) -> PermutationBuilder

Configure sequential splits for the permutation.

One of ratios, counts, or fixed must be provided.

If ratios are provided, they will be used to determine the relative size of each split. For example, if ratios are [0.3, 0.7] then the first split will contain 30% of the rows and the second split will contain 70% of the rows.

If counts are provided, they will be used to determine the absolute number of rows in each split. For example, if counts are [100, 200] then the first split will contain 100 rows and the second split will contain 200 rows.

If fixed is provided, it will be used to determine the number of splits. For example, if fixed is 3 then the permutation will be split evenly into 3 splits.

Rows will be assigned to splits sequentially. The first N1 rows are assigned to split 1, the next N2 rows are assigned to split 2, etc.

The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.

bootstrap_sample

bootstrap_sample(num_samples: int, *, ratios: Optional[list[float]] = None, counts: Optional[list[int]] = None, fixed: Optional[int] = None, seed: Optional[int] = None, split_names: Optional[list[str]] = None) -> PermutationBuilder

Sample rows uniformly with replacement, independently per split.

Unlike the other split strategies, the number of output rows is num_samples, not the number of (optionally filtered) input rows: the same row can appear zero, one, or many times, and num_samples may be larger or smaller than the input row count. Useful for bootstrap resampling, oversampling rare classes, or any training workload that wants a fixed-size stream of random draws rather than a single deterministic pass over every row. Combine with :meth:filter to restrict the eligible population.

One of ratios, counts, or fixed must be provided, with the same meaning as in :meth:split_random, except they divide num_samples across splits rather than the input row count: if fixed is 4, each split gets num_samples // 4 samples.

Each split draws its own sample using a seed derived independently from (seed, split_index), so a split's content does not change if the samples are later redistributed across a different number of splits.

The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.

split_calculated

split_calculated(calculation: str, split_names: Optional[list[str]] = None) -> PermutationBuilder

Use pre-calculated splits for the permutation.

The calculation should be an SQL statement that returns an integer value between 0 and the number of splits - 1. For example, if you have 3 splits then the calculation should return 0 for the first split, 1 for the second split, and 2 for the third split.

This can be used to implement any kind of user-defined split strategy.

The optional split_names can be provided to name the splits. If not provided, the splits can only be referenced by their index.

shuffle

shuffle(*, seed: Optional[int] = None, clump_size: Optional[int] = None) -> PermutationBuilder

Randomly shuffle the rows in the permutation.

An optional seed can be provided to make the shuffle deterministic.

If a clump size is provided, then data will be shuffled as small "clumps" of contiguous rows. This allows for a balance between randomization and I/O performance. It can be useful when reading from cloud storage.

filter

filter(filter: str) -> PermutationBuilder

Configure a filter for the permutation.

The filter should be an SQL statement that returns a boolean value for each row. Only rows where the filter is true will be included in the permutation.

execute

execute() -> LanceTable

Execute the configuration and create the permutation table.

lancedb.permutation.Permutation

A Permutation is a view of a dataset that can be used as input to model training and evaluation.

A Permutation fulfills the pytorch Dataset contract and is loosely modeled after the huggingface Dataset so it should be easy to use with existing code.

A permutation is not a "materialized view" or copy of the underlying data. It is calculated on the fly from the base table. As a result, it is truly "lazy" and does not require materializing the entire dataset in memory.

base_table

base_table = base_table

permutation_table

permutation_table = permutation_table

split

split = split

selection

selection = selection

transform_fn

transform_fn = transform_fn

batch_size

batch_size = batch_size

offset

offset = offset

limit

limit = limit

connection_factory

connection_factory = connection_factory

reader

reader: PermutationReader = _reader

schema

schema: Schema

num_columns

num_columns: int

The number of columns in the permutation

num_rows

num_rows: int

The number of rows in the permutation

column_names

column_names: list[str]

The names of the columns in the permutation

shape

shape: tuple[int, int]

The shape of the permutation

This will return self.num_rows, self.num_columns

with_batch_size

with_batch_size(batch_size: int) -> Permutation

Creates a new permutation with the given batch size

with_connection_factory

with_connection_factory(connection_factory: Callable[[str], Table]) -> Permutation

Creates a new permutation that will use connection_factory to reopen the base table when this permutation is unpickled in a worker process.

The factory is a callable that takes a single argument, the base table name, and returns a LanceDB table. It must be picklable; the worker will pickle it via standard pickle and call it to recover the base table. Picklable callables in practice means top-level (module-level) functions, functools.partial of such functions, or instances of picklable classes implementing __call__. Lambdas and closures over local variables don't pickle with the default protocol.

A factory is optional for normal LanceDB Embedded and Enterprise connections: if not set, __getstate__ captures the table's own picklable reopen state. Use a factory when that default state is not enough, for example when credentials should be loaded from the worker environment instead of being embedded in the pickle.

Examples:

Basic native (file-system path), parameterized via functools.partial::

import functools, lancedb
from lancedb.permutation import Permutation

def open_native_table(uri: str, table_name: str):
    return lancedb.connect(uri).open_table(table_name)

factory = functools.partial(open_native_table, "/data/lance_db")
permutation = Permutation.identity(
    factory("training")
).with_connection_factory(factory)

Native via :func:lancedb.connect_namespace (e.g. a directory- or REST-backed namespace client). The factory takes the implementation name and properties dict as partial-bound args so the worker can rebuild the same namespace connection::

def open_via_namespace(
    impl: str, properties: dict[str, str], table_name: str,
):
    return lancedb.connect_namespace(impl, properties).open_table(
        table_name,
    )

factory = functools.partial(
    open_via_namespace,
    "dir",
    {"root": "/data/lance_db"},
)

LanceDB Enterprise, reading credentials from env vars at worker startup so secrets aren't pickled into the dataset::

import os, lancedb

def open_remote_table(table_name: str):
    db = lancedb.connect(
        "db://my-database",
        api_key=os.environ["LANCEDB_API_KEY"],
        region=os.environ.get("LANCEDB_REGION", "us-east-1"),
    )
    return db.open_table(table_name)

permutation = Permutation.identity(
    open_remote_table("training")
).with_connection_factory(open_remote_table)

identity

identity(table: Table) -> Permutation

Creates an identity permutation for the given table.

from_tables

from_tables(base_table: Table, permutation_table: Optional[Table] = None, split: Optional[Union[str, int]] = None) -> Permutation

Creates a permutation from the given base table and permutation table.

A permutation table identifies which rows, and in what order, the data should be read from the base table. For more details see the [PermutationBuilder] class.

If no permutation table is provided, then the identity permutation will be created. An identity permutation is a permutation that reads all rows in the base table in the order they are stored.

The split parameter identifies which split to use. If no split is provided then the first split will be used.

unique

unique(_column: str) -> list[Any]

Get the unique values in the given column

flatten

flatten() -> Permutation

Flatten the permutation

Each column with a struct type will be flattened into multiple columns.

This flattening operation happens at read time as a post-processing step so this call is cheap and no data is copied or modified in the underlying dataset.

remove_columns

remove_columns(columns: list[str]) -> Permutation

Remove the given columns from the permutation

Note: this does not actually modify the underlying dataset. It only changes which columns are visible from this permutation. Also, this does not introduce a post-processing step. Instead, we simply do not read those columns in the first place.

If any of the provided columns does not exist in the current permutation then it will be ignored (no error is raised for missing columns)

Returns a new permutation with the given columns removed. This does not modify self.

rename_column

rename_column(old_name: str, new_name: str) -> Permutation

Rename a column in the permutation

If there is no column named old_name then an error will be raised If there is already a column named new_name then an error will be raised

Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied.

rename_columns

rename_columns(column_map: dict[str, str]) -> Permutation

Rename the given columns in the permutation

If any of the columns do not exist then an error will be raised If any of the new names already exist then an error will be raised

Note: this does not actually modify the underlying dataset. It only changes the name of the column that is visible from this permutation. This is a post-processing step but done at the batch level and so it is very cheap. No data will be copied.

select_columns

select_columns(columns: list[str]) -> Permutation

Select the given columns from the permutation

This method refines the current selection, potentially removing columns. It will not add back columns that were previously removed.

If any of the columns do not exist then an error will be raised.

This does not introduce a post-processing step. It simply reduces the amount of data we read.

iter

iter(batch_size: int, skip_last_batch: bool = False) -> Iterator[dict[str, Any]]

Iterate over the permutation in batches

If skip_last_batch is True, the last batch will be skipped if it is not a multiple of batch_size.

with_format

with_format(format: Literal['numpy', 'python', 'python_col', 'pandas', 'arrow', 'torch', 'torch_row', 'torch_col', 'polars']) -> Permutation

Set the format for batches

If this method is not called, the "python" format will be used.

The format can be one of: - "numpy" - the batch will be a dict of numpy arrays (one per column) - "python" - the batch will be a list of dicts (one per row) - "python_col" - the batch will be a dict of lists (one entry per column) - "pandas" - the batch will be a pandas DataFrame - "arrow" - the batch will be a pyarrow RecordBatch - "torch" - the batch will be a list of per-row dicts mapping column name to a 0-D torch tensor. Works with the default torch.utils.data.DataLoader collate, which stacks the per-row dicts back into a dict of batched tensors. - "torch_row" - the batch will be a list of tensors, one per row - "torch_col" - the batch will be a 2D torch tensor (first dim indexes columns) - "polars" - the batch will be a polars DataFrame

Conversion may or may not involve a data copy. Lance uses Arrow internally and so it is able to zero-copy to the arrow and polars formats.

Conversion to torch and torch_col will be zero-copy but will only support a subset of data types (numeric types).

Conversion to numpy and/or pandas will typically be zero-copy for numeric types. Conversion of strings, lists, and structs will require creating python objects and this is not zero-copy.

For custom formatting, use with_transform which overrides this method.

with_transform

with_transform(transform: Callable[RecordBatch, Any]) -> Permutation

Set a custom transform for the permutation

The transform is a callable that will be invoked with each record batch. The return value will be used as the batch for iteration.

Note: transforms are not invoked in parallel. This method is not a good place for expensive operations such as image decoding.

take_offsets

take_offsets(offsets: list[int]) -> Any

Take rows from the permutation by offset

The returned value is passed through the permutation's current transform, so with_format and with_transform affect this method in the same way they affect iteration.

skip

skip(skip: int) -> Permutation

Skip the first skip rows of the permutation

Note: this method returns a new permutation and does not modify self It is provided for compatibility with the huggingface Dataset API.

Use with_skip instead to avoid confusion.

with_skip

with_skip(skip: int) -> Permutation

Skip the first skip rows of the permutation

take

take(limit: int) -> Permutation

Limit the permutation to limit rows (following any skip)

Note: this method returns a new permutation and does not modify self It is provided for compatibility with the huggingface Dataset API.

Use with_take instead to avoid confusion.

with_take

with_take(limit: int) -> Permutation

Limit the permutation to limit rows (following any skip)

repeat

repeat(times: int) -> Permutation

Repeat the permutation times times

Note: this method returns a new permutation and does not modify self It is provided for compatibility with the huggingface Dataset API.

Use with_repeat instead to avoid confusion.

with_repeat

with_repeat(times: int) -> Permutation

Repeat the permutation times times

lancedb.permutation.Transforms

Namespace for common transformation functions

arrow2python

arrow2python(batch: RecordBatch) -> list[dict[str, Any]]

arrow2pythoncol

arrow2pythoncol(batch: RecordBatch) -> dict[str, list[Any]]

arrow2arrow

arrow2arrow(batch: RecordBatch) -> RecordBatch

arrow2numpy

arrow2numpy(batch: RecordBatch) -> ndarray

arrow2pandas

arrow2pandas(batch: RecordBatch) -> DataFrame

arrow2polars

arrow2polars() -> DataFrame

Reranking

TypeSafeReranker supports opt-in request batching:

from lancedb.rerankers import TypeSafeReranker

reranker = TypeSafeReranker(batch_size=40, max_concurrency=8)

The default batch_size=1 keeps the query and document in request state and sends one request per non-null candidate. With batch_size=40, 80 non-null candidates require two requests. max_concurrency still limits simultaneous requests, and the SDK handles retries.

In batched mode, state contains only {"query": query}. Each independent question contains {"question": instructions, "document": document} in its structured instructions, so it sees only its own document and the shared query. Custom instructions and criteria are kept verbatim: adapt prompts that explicitly reference request-state fields such as state.document before enabling batching. Null documents retain a zero score without an API call; empty strings are scored. API failures, mismatched answer IDs, and invalid probabilities raise errors. Batching changes the payload and can affect model scores; compare quality and latency on your workload before opting in.

Reranker

Bases: ABC

score

score = return_score

rerank_vector

rerank_vector(query: str, vector_results: Table)

Rerank function receives the result from the vector search. This isn't mandatory to implement

Parameters:

  • query (str) –

    The input query

  • vector_results (Table) –

    The results from the vector search

Returns:

  • Table –

    The reranked results

rerank_fts

rerank_fts(query: str, fts_results: Table)

Rerank function receives the result from the FTS search. This isn't mandatory to implement

Parameters:

  • query (str) –

    The input query

  • fts_results (Table) –

    The results from the FTS search

Returns:

  • Table –

    The reranked results

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table) -> Table

Rerank function receives the individual results from the vector and FTS search results. You can choose to use any of the results to generate the final results, allowing maximum flexibility. This is mandatory to implement

Parameters:

  • query (str) –

    The input query

  • vector_results (Table) –

    The results from the vector search

  • fts_results (Table) –

    The results from the FTS search

Returns:

  • Table –

    The reranked results

merge_results

merge_results(vector_results: Table, fts_results: Table)

Merge the results from the vector and FTS search. This is a vanilla merging function that just concatenates the results and removes the duplicates.

NOTE: This doesn't take score into account. It'll keep the instance that was encountered first. This is designed for rerankers that don't use the score. In case you want to use the score, or support return_scores="all" you'll have to implement your own merging function.

Parameters:

  • vector_results (Table) –

    The results from the vector search

  • fts_results (Table) –

    The results from the FTS search

rerank_multivector

rerank_multivector(vector_results: Union[List[Table], List[LanceVectorQueryBuilder]], query: Union[str, None], deduplicate: bool = False)

This is a rerank function that receives the results from multiple vector searches. For example, this can be used to combine the results of two vector searches with different embeddings.

Parameters:

  • vector_results (List[Table] or List[LanceVectorQueryBuilder]) –

    The results from the vector search. Either accepts the query builder if the results haven't been executed yet or the results in arrow format.

  • query ((str or None)) –

    The input query. Some rerankers might not need the query to rerank. In that case, it can be set to None explicitly. This is intended to be handled by the reranker implementations.

  • deduplicate (bool, default: False ) –

    Whether to deduplicate the results based on the _rowid column, by default False. Requires _rowid to be present in the results.

Returns:

  • Table –

    The reranked results

CohereReranker

Bases: Reranker

Reranks the results using the Cohere Rerank API. https://docs.cohere.com/docs/rerank-guide

Parameters:

  • model_name (str, default: "rerank-english-v2.0" ) –

    The name of the cross encoder model to use. Available cohere models are: - rerank-english-v2.0 - rerank-multilingual-v2.0

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • top_n (str, default: None ) –

    The number of results to return. If None, will return all results.

model_name

model_name = model_name

column

column = column

top_n

top_n = top_n

api_key

api_key = api_key

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

ColbertReranker

Bases: AnswerdotaiRerankers

Reranks the results using the ColBERT model.

Parameters:

  • model_name (str, default: "colbert" (colbert-ir/colbert-v2.0) ) –

    The name of the cross encoder model to use.

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • return_score (str, default: "relevance" ) –

    options are "relevance" or "all".

  • **kwargs –

    Additional keyword arguments to pass to the model, for example, 'device'. See AnswerDotAI/rerankers for more information.

CrossEncoderReranker

Bases: Reranker

Reranks the results using a cross encoder model. The cross encoder model is used to score the query and each result. The results are then sorted by the score.

Parameters:

  • model_name (str, default: "cross-encoder/ms-marco-TinyBERT-L-6" ) –

    The name of the cross encoder model to use. See the sentence transformers documentation for a list of available models.

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • device (str, default: None ) –

    The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu".

  • return_score (str, default: "relevance" ) –

    options are "relevance" or "all". Only "relevance" is supported for now.

  • trust_remote_code (bool, default: True ) –

    If True, will trust the remote code to be safe. If False, will not trust the remote code and will not run it

  • batch_size (int, default: 32 ) –

    The number of query-passage pairs scored per forward pass.

  • **kwargs –

    Additional keyword arguments passed to sentence_transformers.CrossEncoder, for example max_length or model_kwargs={"torch_dtype": "bfloat16"}.

model_name

model_name = model_name

column

column = column

device

device = device

trust_remote_code

trust_remote_code = trust_remote_code

batch_size

batch_size = batch_size

kwargs

kwargs = kwargs

model

model

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

LinearCombinationReranker

Bases: Reranker

Reranks the results using a linear combination of the scores from the vector and FTS search. For missing scores, fill with fill value.

Parameters:

  • weight (float, default: 0.7 ) –

    The weight to give to the vector score. Must be between 0 and 1.

  • fill (float, default: 1.0 ) –

    The score to give to results that are only in one of the two result sets. This is treated as penalty, so a higher value means a lower score. TODO: We should just hardcode this-- its pretty confusing as we invert scores to calculate final score

  • return_score (str, default: "relevance" ) –

    opntions are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.

weight

weight = weight

fill

fill = fill

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

merge_results

merge_results(vector_results: Table, fts_results: Table, fill: float)

OpenaiReranker

Bases: Reranker

Reranks the results using the OpenAI API. WARNING: This is a prompt based reranker that uses chat model that is not a dedicated reranker API. This should be treated as experimental.

Parameters:

  • model_name (str, default: "gpt-5.6-sol" ) –

    The name of the cross encoder model to use.

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • return_score (str, default: "relevance" ) –

    options are "relevance" or "all". Only "relevance" is supported for now.

  • api_key (str, default: None ) –

    The API key to use. If None, will use the OPENAI_API_KEY environment variable.

model_name

model_name = model_name

column

column = column

api_key

api_key = api_key

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

JinaReranker

Bases: Reranker

Reranks the results using the Jina Rerank API. https://jina.ai/rerank

Parameters:

  • model_name (str, default: "jina-reranker-v2-base-multilingual" ) –

    The name of the cross reanker model to use

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • top_n (str, default: None ) –

    The number of results to return. If None, will return all results.

  • api_key (str, default: None ) –

    The api key to access Jina API. If you pass None, you can set JINA_API_KEY environment variable

model_name

model_name = model_name

column

column = column

top_n

top_n = top_n

api_key

api_key = api_key

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

RRFReranker

Bases: Reranker

Reranks the results using Reciprocal Rank Fusion(RRF) algorithm based on the scores of vector and FTS search.

Parameters:

  • K (int, default: 60 ) –

    A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. See paper: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf

  • return_score (str, default: "relevance" ) –

    opntions are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.

K

K = K

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_multivector

rerank_multivector(vector_results: Union[List[Table], List[LanceVectorQueryBuilder]], query: str = None, deduplicate: bool = True)

Overridden method to rerank the results from multiple vector searches. This leverages the RRF hybrid reranking algorithm to combine the results from multiple vector searches as it doesn't support reranking vector results individually.

MRRReranker

Bases: Reranker

Reranks the results using Mean Reciprocal Rank (MRR) algorithm based on the scores of vector and FTS search. Algorithm reference - https://en.wikipedia.org/wiki/Mean_reciprocal_rank

MRR calculates the average of reciprocal ranks across different search results. For each document, it computes the reciprocal of its rank in each system, then takes the mean of these reciprocal ranks as the final score.

Parameters:

  • weight_vector (float, default: 0.5 ) –

    Weight for vector search results (0.0 to 1.0)

  • weight_fts (float, default: 0.5 ) –

    Weight for FTS search results (0.0 to 1.0) Note: weight_vector + weight_fts should equal 1.0

  • return_score (str, default: "relevance" ) –

    Options are "relevance" or "all" The type of score to return. If "relevance", will return only the relevance score. If "all", will return all scores from the vector and FTS search along with the relevance score.

weight_vector

weight_vector = weight_vector

weight_fts

weight_fts = weight_fts

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_multivector

rerank_multivector(vector_results: Union[List[Table], List[LanceVectorQueryBuilder]], query: str = None, deduplicate: bool = True)

Reranks the results from multiple vector searches using MRR algorithm. Each vector search result is treated as a separate ranking system, and MRR calculates the mean of reciprocal ranks across all systems. This cannot reuse rerank_hybrid because MRR semantics require treating each vector result as a separate ranking system.

AnswerdotaiRerankers

Bases: Reranker

Reranks the results using the Answerdotai Rerank API. All supported reranker model types can be found here: - https://github.com/AnswerDotAI/rerankers

Parameters:

  • model_type (str, default: "colbert" ) –

    The type of the model to use.

  • model_name (str, default: "rerank-english-v2.0" ) –

    The name of the model to use from the given model type.

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • return_score (str, default: "relevance" ) –

    options are "relevance" or "all".

  • **kwargs –

    Additional keyword arguments to pass to the model. For example, 'device'. See AnswerDotAI/rerankers for more information.

column

column = column

model_name

model_name = model_name

model_type

model_type = model_type

reranker

reranker = Reranker(model_name=model_name, model_type=model_type, **kwargs)

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

TypeSafeReranker

Bases: Reranker

Reranks the results using the TypeSafe System One API. https://docs.typesafe.ai/cookbooks/rerank_typesafe

Each result is scored independently: TypeSafe reads the query together with the result's column value and answers a yes/no ("noul") question about whether the document is relevant. The probability of "yes", between 0 and 1, becomes the _relevance_score. Because every score is an absolute probability rather than a position in the list, scores are comparable across queries and can be thresholded. They are model estimates, though, and can vary slightly between identical calls, so results with close scores may swap places when the same search is repeated.

By default, one request is sent per non-null result. Set batch_size=40 to score up to 40 documents per request, with at most max_concurrency requests in flight. Each question sees only its own document and the shared query. Null documents keep a score of zero without an API request; empty strings are scored normally. API and response-validation errors propagate.

Parameters:

  • model_name (str, default: "jev-latest" ) –

    The TypeSafe model to use.

  • column (str, default: "text" ) –

    The name of the column holding the document text to score.

  • instructions (str, default: None ) –

    The yes/no question asked about each query and document pair. The state TypeSafe reads with batch_size=1 is {"query": <query>, "document": <column value>}. Defaults to a generic relevance question. See batch_size for the different location of the document in batched requests.

  • criteria (Mapping[str, str], default: None ) –

    What a "yes" and a "no" mean, as a mapping with the keys "true" and "false". Domain-specific criteria usually rank better than the generic default. Pass an empty mapping to send no criteria.

  • return_score (str, default: "relevance" ) –

    Options are "relevance" or "all". If "all", keeps the vector and FTS scores alongside the relevance score.

  • api_key (str, default: None ) –

    The API key to use. If None, the TypeSafe SDK reads the TYPESAFE_API_KEY environment variable.

  • max_concurrency (int, default: 8 ) –

    The maximum number of TypeSafe requests in flight for one rerank call.

  • batch_size (int, default: 1 ) –

    Positive integer limiting non-null documents per request. The default preserves the original request format. With a value greater than one, state is {"query": <query>} and each question's instructions are {"question": <instructions>, "document": <column value>}. Instructions and criteria are preserved verbatim. Custom prompts that explicitly reference the document in request state (for example, state.document) must be adapted before opting into batching. The TypeSafe SDK handles retries in both modes.

Examples:

>>> reranker = TypeSafeReranker(batch_size=40, max_concurrency=8)

model_name

model_name = model_name

column

column = column

instructions

instructions = instructions or DEFAULT_INSTRUCTIONS

criteria

criteria = criteria

api_key

api_key = api_key

max_concurrency

max_concurrency = max_concurrency

batch_size

batch_size = batch_size

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

VoyageAIReranker

Bases: Reranker

Reranks the results using the VoyageAI Rerank API. https://docs.voyageai.com/docs/reranker

Parameters:

  • model_name (str, default: "rerank-english-v2.0" ) –

    The name of the cross encoder model to use. Available voyageai models are: - rerank-2.5 - rerank-2.5-lite - rerank-2 - rerank-2-lite

  • column (str, default: "text" ) –

    The name of the column to use as input to the cross encoder model.

  • top_n (int, default: None ) –

    The number of results to return. If None, will return all results.

  • return_score (str, default: "relevance" ) –

    options are "relevance" or "all". Only "relevance" is supported for now.

  • api_key (str, default: None ) –

    The API key to use. If None, will use the VOYAGE_API_KEY environment variable.

  • truncation (Optional[bool], default: None ) –

model_name

model_name = model_name

column

column = column

top_n

top_n = top_n

api_key

api_key = api_key

truncation

truncation = truncation

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table)

rerank_vector

rerank_vector(query: str, vector_results: Table)

rerank_fts

rerank_fts(query: str, fts_results: Table)

WatsonxReranker

Bases: Reranker

Reranks the results using the IBM watsonx.ai Rerank API.

Uses the ibm_watsonx_ai SDK (Rerank.generate) under the hood.

API Docs: https://cloud.ibm.com/docs/apis/watsonx-ai#text-rerank

Supported rerank models: https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx#rerank

Parameters:

  • model_name (str, default: "cross-encoder/ms-marco-minilm-l-12-v2" ) –

    The ID of the rerank model to use.

  • column (str, default: "text" ) –

    The name of the column to use as input to the reranker.

  • top_n (int, default: None ) –

    Return only the top-n results. If None, all results are returned.

  • return_score (str, default: "relevance" ) –

    Options are "relevance" or "all".

  • api_key (str, default: None ) –

    IBM Cloud API key. Falls back to the WATSONX_API_KEY environment variable when not provided.

  • project_id (str, default: None ) –

    watsonx.ai project ID. Explicit value takes precedence over the WATSONX_PROJECT_ID environment variable. Mutually exclusive with space_id — exactly one must be supplied.

  • space_id (str, default: None ) –

    watsonx.ai deployment space ID. Explicit value takes precedence over the WATSONX_SPACE_ID environment variable. Mutually exclusive with project_id — exactly one must be supplied.

  • url (str, default: None ) –

    watsonx.ai service URL. Defaults to "https://us-south.ml.cloud.ibm.com".

  • truncate_input_tokens (int, default: None ) –

    Truncate each input to this many tokens before scoring. Passed directly to the parameters dict of Rerank.generate.

model_name

model_name = model_name

column

column = column

top_n

top_n = top_n

api_key

api_key = api_key

project_id

project_id = project_id

space_id

space_id = space_id

url

url = url

truncate_input_tokens

truncate_input_tokens = truncate_input_tokens

rerank_hybrid

rerank_hybrid(query: str, vector_results: Table, fts_results: Table) -> Table

rerank_vector

rerank_vector(query: str, vector_results: Table) -> Table

rerank_fts

rerank_fts(query: str, fts_results: Table) -> Table

Connections (Asynchronous)

lancedb.connect_catalog_async

connect_catalog_async(endpoint: str, *, api_key: Optional[str] = None, client_config: Optional[Union[ClientConfig, dict[str, Any]]] = None, sql_host_override: Optional[str] = None, read_consistency_interval: Optional[timedelta] = None, oauth_config: Optional[OAuthConfig] = None) -> AsyncCatalog

Connect to an HTTP(S) server's root catalog.

Root requests omit database-selection headers. API key, client configuration, OAuth, and table read consistency settings are inherited by opened databases. Database names containing slashes remain single logical names. Set sql_host_override to the SQL service endpoint to execute SQL through returned connections when the catalog endpoint uses HTTPS.

lancedb.catalog.AsyncCatalog

An asynchronous remote catalog returned by connect_catalog_async.

Create/connect return ordinary AsyncConnection instances. Drop uses restricted behavior: remove the database's tables first.

authz

authz: AsyncAuthorization

Authorization sharing this catalog's scope and credentials.

uri

uri: str

The catalog's root namespace endpoint.

create_database

create_database(name: str, *, exist_ok: bool = False) -> AsyncConnection

Create a database, or open an existing one when exist_ok=True.

connect_database

connect_database(name: str) -> AsyncConnection

Connect to an existing database by its logical name.

list_databases

list_databases(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncDatabaseNames

Iterate lazily over all database names using async for.

page_token resumes from a saved token; None starts at the beginning. page_limit sets the maximum names per REST response (not the total); None uses the server default. Request errors are raised during iteration.

Examples:

async for name in catalog.list_databases():
    print(name)

drop_database

drop_database(name: str, *, ignore_missing: bool = False, cascade: bool = False) -> None

Drop a database. A nonempty database is an error unless cascade is set, which drops everything in it first; the name stays taken until the server finishes cleaning up their storage.

lancedb.catalog.AsyncDatabaseNames

Bases: AsyncIterator[str]

An async iterator of database names with pagination state.

Returned by list_databases.

num_page_results

num_page_results() -> int

Return the number of names available without another REST request.

page_token

page_token() -> Optional[str]

Return the token for the next REST request.

Before the first request, this is the supplied starting token (None starts at the beginning). After the final page is fetched, it is None, even if names remain cached. Drain the cache before saving a token to avoid skipping those names when resuming. A failed request terminates iteration and leaves its token available for resuming a new iterator.

Connections represent a connection to a LanceDb database and can be used to create, list, or open tables.

lancedb.connect_async

connect_async(uri: URI, *, api_key: Optional[str] = None, region: str = 'us-east-1', host_override: Optional[str] = None, sql_host_override: Optional[str] = None, read_consistency_interval: Optional[timedelta] = None, client_config: Optional[Union[ClientConfig, Dict[str, Any]]] = None, storage_options: Optional[Dict[str, str]] = None, session: Optional[Session] = None, manifest_enabled: bool = False, namespace_client_properties: Optional[Dict[str, str]] = None, oauth_config=None) -> AsyncConnection

Connect to a LanceDB database.

Parameters:

  • uri (URI) –

    The uri of the database.

  • api_key (Optional[str], default: None ) –

    If present, connect to LanceDB Enterprise or LanceDB Cloud (legacy). Otherwise, connect to a database on file system or cloud storage. Can be set via environment variable LANCEDB_API_KEY.

  • region (str, default: 'us-east-1' ) –

    The region to use for LanceDB Cloud (legacy).

  • host_override (Optional[str], default: None ) –

    The URL of the LanceDB Enterprise deployment to connect to.

  • sql_host_override (Optional[str], default: None ) –

    The LanceDB Enterprise SQL service endpoint override. The client connects lazily when SQL is first executed and retains that connection.

  • read_consistency_interval (Optional[timedelta], default: None ) –

    The interval at which to check for updates to the table from other processes. If None, then consistency is not checked. For performance reasons, this is the default. For strong consistency, set this to zero seconds. Then every read will check for updates from other processes. As a compromise, you can set this to a non-zero timedelta for eventual consistency. If more than that interval has passed since the last check, then the table will be checked for updates. Note: this consistency only applies to read operations. Write operations are always consistent.

    Stronger consistency is not free. The smaller the interval, the more often each read pays the cost of checking for updates against object storage, raising per-read latency and cost.

  • client_config (Optional[Union[ClientConfig, Dict[str, Any]]], default: None ) –

    Configuration options for the HTTP client used by db:// connections. If a dict, then the keys are the attributes of the ClientConfig class. If None, then the default configuration is used.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. See available options at https://docs.lancedb.com/storage/

  • session (Optional[Session], default: None ) –

    (For LanceDB Embedded only) A session to use for this connection. Sessions allow you to configure cache sizes for index and metadata caches, which can significantly impact memory use and performance. They can also be re-used across multiple connections to share the same cache state.

  • manifest_enabled (bool, default: False ) –

    When true for LanceDB Embedded connections, use directory namespace manifests as the source of truth for table metadata. Existing directory-listed root tables are migrated into the manifest on access.

  • namespace_client_properties (dict, default: None ) –

    Additional directory namespace client properties to use with manifest_enabled=True.

  • oauth_config (OAuthConfig, default: None ) –

    OAuth configuration for LanceDB Enterprise. This is supported by connect_async only; synchronous connect uses API key authentication for db:// URIs.

Examples:

>>> import lancedb
>>> azure_storage_options = {
...     "account_name": "some-account",
...     "account_key": "some-key",
... }
>>> async def doctest_example():
...     # For a local directory, provide a path to the database
...     db = await lancedb.connect_async("~/.lancedb")
...     # For object storage, use a URI prefix
...     db = await lancedb.connect_async("s3://my-bucket/lancedb",
...                                      storage_options={
...                                          "aws_access_key_id": "***"})
...     # Azure credentials can also be passed directly
...     db = await lancedb.connect_async(
...         "az://my-container/my-database",
...         storage_options=azure_storage_options,
...     )
...     # For tests and temporary data, use an in-memory database
...     db = await lancedb.connect_async("memory://")
...     # Connect to LanceDB Enterprise or LanceDB Cloud (legacy)
...     db = await lancedb.connect_async("db://my_database", api_key="ldb_...",
...                                      client_config={
...                                          "retry_config": {"retries": 5}})

Returns:

  • conn ( AsyncConnection ) –

    A connection to a LanceDB database.

lancedb.db.AsyncConnection

Bases: object

An active LanceDB connection

To obtain a connection you can use the connect_async function.

This could be a LanceDB Embedded connection (using Lance) or a connection to LanceDB Enterprise.

Embedded connections do not currently hold any open resources but they may do so in the future (for example, for shared cache or connections to catalog services) Enterprise connections represent an open connection to the server. The close method can be used to release any underlying resources eagerly. The connection can also be used as a context manager.

Connections can be shared on multiple threads and are expected to be long lived. Connections can also be used as a context manager, however, in many cases a single connection can be used for the lifetime of the application and so this is often not needed. Closing a connection is optional. If it is not closed then it will be automatically closed when the connection object is deleted.

Examples:

>>> import lancedb
>>> async def doctest_example():
...   with await lancedb.connect_async("/tmp/my_dataset") as conn:
...     # do something with the connection
...     pass
...   # conn is closed here

uri

uri: str

is_open

is_open()

Return True if the connection is open.

close

close()

Close the connection, releasing any underlying resources.

It is safe to call this method multiple times.

Any attempt to use the connection after it is closed will result in an error.

get_read_consistency_interval

get_read_consistency_interval() -> Optional[timedelta]

list_namespaces

list_namespaces(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> AsyncNamespaceListing

List immediate child namespace names in the given namespace.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The parent namespace to list namespaces in. None or empty list represents root namespace.

  • page_token (Optional[str], default: None ) –

    The token to use for pagination. If not present, start from the beginning.

  • limit (Optional[int], default: None ) –

    The maximum number of results to return.

Returns:

  • AsyncNamespaceListing –

    Response containing namespace names and optional pagination token

create_namespace

create_namespace(namespace_path: List[str], mode: Optional[str] = None, properties: Optional[Dict[str, str]] = None) -> CreateNamespaceResponse

Create a new namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to create.

  • mode (Optional[str], default: None ) –

    Creation mode - "create", "exist_ok", or "overwrite". Case insensitive.

  • properties (Optional[Dict[str, str]], default: None ) –

    Properties to associate with the namespace

Returns:

  • CreateNamespaceResponse –

    Response containing namespace properties

drop_namespace

drop_namespace(namespace_path: List[str], mode: Optional[str] = None, behavior: Optional[str] = None) -> DropNamespaceResponse

Drop a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to drop.

  • mode (Optional[str], default: None ) –

    Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.

  • behavior (Optional[str], default: None ) –

    Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.

Returns:

  • DropNamespaceResponse –

    Response containing properties and transaction_id if applicable.

describe_namespace

describe_namespace(namespace_path: List[str]) -> DescribeNamespaceResponse

Describe a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to describe.

Returns:

  • DescribeNamespaceResponse –

    Response containing the namespace properties.

namespace_exists

namespace_exists(namespace_id: List[str]) -> bool

Check if a namespace exists.

Parameters:

  • namespace_id (List[str]) –

    The namespace identifier to check. An empty list represents root.

Returns:

  • bool –

    True if the namespace exists, False if it is not found. Other errors, including permission errors, are propagated.

Examples:

>>> async def check_namespace(conn):
...     return await conn.namespace_exists(["analytics"])

table_exists

table_exists(table_id: List[str]) -> bool

Check if a table exists.

Parameters:

  • table_id (List[str]) –

    The full identifier, including namespace segments and table name. Must contain at least the table name.

Returns:

  • bool –

    True if the table exists, False if it is not found. Other errors, including permission errors, are propagated.

Examples:

>>> async def check_table(conn):
...     return await conn.table_exists(["analytics", "events"])

list_tables

list_tables(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> AsyncTableListing

List tables in this database, one page at a time.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to list tables in. None or empty list represents root namespace.

  • page_token (Optional[str], default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (Optional[int], default: None ) –

    The maximum number of results to return, default 100. Zero returns an empty page without a continuation token.

Returns:

  • AsyncTableListing –

    Response containing table names and optional page_token for pagination.

table_names

table_names(*, namespace_path: Optional[List[str]] = None, start_after: Optional[str] = None, limit: Optional[int] = None) -> Iterable[str]

List all tables in this database, in sorted order

.. deprecated:: Use :meth:list_tables instead, which provides proper pagination support.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to list tables in. None or empty list represents root namespace.

  • start_after (Optional[str], default: None ) –

    If present, only return names that come lexicographically after the supplied value.

    This can be combined with limit to implement pagination by setting this to the last table name from the previous page.

  • limit (Optional[int], default: None ) –

    The maximum number of tables to return. None returns all tables; zero returns an empty list.

Returns:

  • Iterable of str –

create_table

create_table(name: str, data: Optional[DATA] = None, schema: Optional[Union[Schema, LanceModel]] = None, mode: Optional[Literal['create', 'overwrite']] = None, exist_ok: Optional[bool] = None, on_bad_vectors: Optional[str] = None, fill_value: Optional[float] = None, storage_options: Optional[Dict[str, str]] = None, *, namespace_path: Optional[List[str]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, location: Optional[str] = None, namespace_client: Optional[Any] = None) -> AsyncTable

Create an AsyncTable in the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to create the table in. Empty list represents root namespace.

  • data (Optional[DATA], default: None ) –

    User must provide at least one of data or schema. Acceptable types are:

    • list-of-dict

    • pandas.DataFrame

    • pyarrow.Table or pyarrow.RecordBatch

  • schema (Optional[Union[Schema, LanceModel]], default: None ) –

    Acceptable types are:

    Plain binary values are cast into blob columns declared with blob. To construct Arrow record batches explicitly, use blob_array.

  • mode (Optional[Literal['create', 'overwrite']], default: None ) –

    The mode to use when creating the table. Can be "create", "overwrite", or "exist_ok". By default, if the table already exists, an exception is raised. If you want to overwrite the table, use mode="overwrite". To open an existing table without adding data, use mode="exist_ok".

  • exist_ok (Optional[bool], default: None ) –

    If a table by the same name already exists, then raise an exception if exist_ok=False. If exist_ok=True, then open the existing table; it will not add the provided data but will validate against any schema that's specified.

  • on_bad_vectors (Optional[str], default: None ) –

    How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".

  • fill_value (Optional[float], default: None ) –

    The value to use when filling vectors. Only used if on_bad_vectors="fill". NaN or null elements and missing trailing elements are replaced with this value. Valid elements are preserved, and vectors longer than the expected dimension are truncated.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/

    To enable stable row IDs (row IDs remain stable after compaction, update, delete, and merges), set new_table_enable_stable_row_ids to "true" in storage_options when connecting to the database.

Returns:

  • AsyncTable –

    A reference to the newly created table.

  • !!! note –

    The vector index won't be created by default. To create the index, call the create_index method on the table.

Examples:

Can create with list of tuples or dictionaries:

>>> import lancedb
>>> async def doctest_example():
...     db = await lancedb.connect_async("./.lancedb")
...     data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
...             {"vector": [0.2, 1.8], "lat": 40.1, "long":  -74.1}]
...     my_table = await db.create_table("my_table", data)
...     print(await my_table.query().limit(5).to_arrow())
>>> import asyncio
>>> asyncio.run(doctest_example())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

You can also pass a pandas DataFrame:

>>> import pandas as pd
>>> data = pd.DataFrame({
...    "vector": [[1.1, 1.2], [0.2, 1.8]],
...    "lat": [45.5, 40.1],
...    "long": [-122.7, -74.1]
... })
>>> async def pandas_example():
...     db = await lancedb.connect_async("./.lancedb")
...     my_table = await db.create_table("table2", data)
...     print(await my_table.query().limit(5).to_arrow())
>>> asyncio.run(pandas_example())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: double
long: double
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.

>>> import pyarrow as pa
>>> custom_schema = pa.schema([
...   pa.field("vector", pa.list_(pa.float32(), 2)),
...   pa.field("lat", pa.float32()),
...   pa.field("long", pa.float32())
... ])
>>> async def with_schema():
...     db = await lancedb.connect_async("./.lancedb")
...     my_table = await db.create_table("table3", data, schema = custom_schema)
...     print(await my_table.query().limit(5).to_arrow())
>>> asyncio.run(with_schema())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
lat: float
long: float
----
vector: [[[1.1,1.2],[0.2,1.8]]]
lat: [[45.5,40.1]]
long: [[-122.7,-74.1]]

It is also possible to create an table from [Iterable[pa.RecordBatch]]:

>>> import pyarrow as pa
>>> def make_batches():
...     for i in range(5):
...         yield pa.RecordBatch.from_arrays(
...             [
...                 pa.array([[3.1, 4.1], [5.9, 26.5]],
...                     pa.list_(pa.float32(), 2)),
...                 pa.array(["foo", "bar"]),
...                 pa.array([10.0, 20.0]),
...             ],
...             ["vector", "item", "price"],
...         )
>>> schema=pa.schema([
...     pa.field("vector", pa.list_(pa.float32(), 2)),
...     pa.field("item", pa.utf8()),
...     pa.field("price", pa.float32()),
... ])
>>> async def iterable_example():
...     db = await lancedb.connect_async("./.lancedb")
...     await db.create_table("table4", make_batches(), schema=schema)
>>> asyncio.run(iterable_example())

open_table

open_table(name: str, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, index_cache_size: Optional[int] = None, location: Optional[str] = None, namespace_client: Optional[Any] = None, managed_versioning: Optional[bool] = None, branch: Optional[str] = None, version: Optional[int] = None) -> AsyncTable

Open a Lance Table in the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to open the table from. None or empty list represents root namespace.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend. Options already set on the connection will be inherited by the table, but can be overridden here. See available options at https://docs.lancedb.com/storage/

  • index_cache_size (Optional[int], default: None ) –

    Deprecated: Use session-level cache configuration instead. Create a Session with custom cache sizes and pass it to lancedb.connect().

    Set the size of the index cache, specified as a number of entries

    The exact meaning of an "entry" will depend on the type of index: * IVF - there is one entry for each IVF partition * BTREE - there is one entry for the entire index

    This cache applies to the entire opened table, across all indices. Setting this value higher will increase performance on larger datasets at the expense of more RAM

  • location (Optional[str], default: None ) –

    The explicit location (URI) of the table. If provided, the table will be opened from this location instead of deriving it from the database URI and table name.

  • managed_versioning (Optional[bool], default: None ) –

    Whether managed versioning is enabled for this table. If provided, avoids a redundant describe_table call when namespace_client is set.

  • branch (Optional[str], default: None ) –

    If provided, open a handle scoped to this branch instead of the default branch. Reads and writes operate in the branch's context.

  • version (Optional[int], default: None ) –

    If provided, open the table pinned to this version, producing a read-only handle. Composes with branch: when both are given, opens that branch at the version; otherwise opens main at the version. Call checkout_latest to return to a writable state.

Returns:

  • A LanceTable object representing the table. –

create_materialized_view

create_materialized_view(name: str, source: MaterializedViewSource, *, select: SelectArg = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> AsyncMaterializedView

Define a materialized view named name over the table source.

LanceDB Embedded creation and refresh require a source build with the experimental-functions Cargo feature. Published packages leave it disabled; LanceDB Enterprise operations remain available. See DBConnection.create_materialized_view.

create_materialized_view_async

create_materialized_view_async(name: str, source: MaterializedViewSource, *, select: SelectArg = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> AsyncJob[None]

Submit materialized-view creation and return its job.

Wait for the returned job before opening or querying the view. In LanceDB Enterprise, creation uses SQL; job_type is "query" and id identifies that query on this connection.

open_materialized_view

open_materialized_view(name: str) -> AsyncMaterializedView

Open the materialized view named name.

Raises ValueError if the table exists but is not a materialized view.

list_materialized_views

list_materialized_views(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

The names of the materialized views in this database.

Local listings inspect table schemas one page at a time.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_materialized_view

drop_materialized_view(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a materialized view.

In LanceDB Enterprise, drops wait for the SQL command to remove the catalog binding. Physical cleanup continues in the background.

drop_materialized_view_async

drop_materialized_view_async(name: str, *, namespace_path: Optional[List[str]] = None) -> AsyncJob[None]

Start dropping a materialized view and return its job.

In LanceDB Enterprise, the job has type query and its ID identifies the SQL query. Await :meth:AsyncJob.wait for catalog removal; physical cleanup continues in the background.

clone_table

clone_table(target_table_name: str, source_uri: str, *, target_namespace_path: Optional[List[str]] = None, source_version: Optional[int] = None, source_tag: Optional[str] = None, is_shallow: bool = True) -> AsyncTable

Clone a table from a source table.

A shallow clone creates a new table that shares the underlying data files with the source table but has its own independent manifest. This allows both the source and cloned tables to evolve independently while initially sharing the same data, deletion, and index files.

Parameters:

  • target_table_name (str) –

    The name of the target table to create.

  • source_uri (str) –

    The URI of the source table to clone from.

  • target_namespace_path (Optional[List[str]], default: None ) –

    The namespace for the target table. None or empty list represents root namespace.

  • source_version (Optional[int], default: None ) –

    The version of the source table to clone.

  • source_tag (Optional[str], default: None ) –

    The tag of the source table to clone.

  • is_shallow (bool, default: True ) –

    Whether to perform a shallow clone (True) or deep clone (False). Currently only shallow clone is supported.

Returns:

  • An AsyncTable object representing the cloned table. –

rename_table

rename_table(cur_name: str, new_name: str, cur_namespace_path: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None)

Rename a table in the database.

Parameters:

  • cur_name (str) –

    The current name of the table.

  • new_name (str) –

    The new name of the table.

  • cur_namespace_path (Optional[List[str]], default: None ) –

    The namespace of the current table. None or empty list represents root namespace.

  • new_namespace_path (Optional[List[str]], default: None ) –

    The namespace to move the table to. If not specified, defaults to the same as cur_namespace.

drop_table

drop_table(name: str, *, namespace_path: Optional[List[str]] = None, ignore_missing: bool = False)

Drop a table from the database.

Parameters:

  • name (str) –

    The name of the table.

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to drop the table from. Empty list represents root namespace.

  • ignore_missing (bool, default: False ) –

    If True, ignore if the table does not exist.

drop_table_async

drop_table_async(name: str, *, namespace_path: Optional[List[str]] = None) -> AsyncJob

Start dropping a table and return its cleanup job.

The table may become unavailable before its data files are removed. Await :meth:AsyncJob.wait to wait for cleanup to finish.

drop_all_tables

drop_all_tables(namespace_path: Optional[List[str]] = None)

Drop all tables from the database.

Lists the tables in the selected namespace and drops them one at a time.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The namespace to drop all tables from. None or empty list represents root namespace.

open_job

open_job(job_id: str) -> AsyncJob

Open a server-side job by id. See DBConnection.open_job.

list_gpu_classes

list_gpu_classes() -> List[GpuClass]

List configured Function GPU classes. See DBConnection.list_gpu_classes.

create_function_async

create_function_async(definition: FunctionDefinition, *, secrets: Optional[Sequence[EnvVarSecret]] = None, namespace_path: Optional[List[str]] = None) -> AsyncJob[FunctionVersion]

Submit a scalar Python function for building and registration.

The server-side job builds the OCI image, then registers the completed artifact. The returned job has type query and the SQL query UUID as its ID. Waiting on it returns the immutable Function version. secrets is a sequence of EnvVarSecret, each naming a Secret and the environment variable its value arrives in. namespace_path is the namespace the Function is registered in; None or an empty list is the root. LanceDB Embedded connections raise NotImplementedError.

get_function

get_function(name: str, *, version: Optional[Union[int, str]] = None, namespace_path: Optional[List[str]] = None) -> FunctionVersion

Open an immutable Function version from the remote catalog.

version may be an integer or string. When omitted, the server resolves the latest currently visible revision and returns that exact immutable version.

list_functions

list_functions(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[FunctionVersion]

List every published immutable Function version in a namespace.

Functions in child namespaces are not included. Results are ordered by Function name then version. LanceDB Embedded connections raise NotImplementedError.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_function

drop_function(name: str, *, version: str, namespace_path: Optional[List[str]] = None) -> bool

Drop one published Function revision, preserving its siblings.

Returns True when the requested revision was removed and False for a repeated drop or a missing name. A revision never published under an existing name is an error. Only the last visible revision's removal releases the name and schedules background object cleanup. When the remote server reports a warning, it is emitted as UserWarning before this method returns, subject to the standard warnings filters. A warnings-as-errors setting can therefore raise after the catalog removal; it does not roll back the drop. Older servers that omit the warning field remain silent.

drop_function_async

drop_function_async(name: str, *, version: str, namespace_path: Optional[List[str]] = None) -> Tuple[bool, AsyncJob]

Drop one Function revision and return its SQL query job.

The boolean reports removal of the requested revision from the catalog; sibling revisions and their grants remain available. The job has type query and its ID is the SQL query UUID. Repeated waits confirm that revision's removal. Only the last visible revision's drop releases the name and schedules background object cleanup.

create_storage_integration

create_storage_integration(name: str, storage_prefix: str, *, aws_role_arn: Optional[str] = None, aws_access_key_id_secret: Optional[Union[str, SecretReference]] = None, aws_secret_access_key_secret: Optional[Union[str, SecretReference]] = None, aws_region: Optional[str] = None, storage_provider: Literal['S3', 'AZURE', 'GCP', 'HUGGINGFACE'] = 'S3', azure_storage_account: Optional[str] = None, azure_tenant_id_secret: Optional[Union[str, SecretReference]] = None, azure_client_id_secret: Optional[Union[str, SecretReference]] = None, azure_client_secret_secret: Optional[Union[str, SecretReference]] = None, gcp_credentials_json_secret: Optional[Union[str, SecretReference]] = None, huggingface_token_secret: Optional[Union[str, SecretReference]] = None, namespace_path: Optional[List[str]] = None) -> StorageIntegrationInfo

Register cloud storage access using database-owned credential Secrets.

namespace_path follows Secret naming rules. A SecretReference can name another database; a string names a root Secret in this database. For Azure, set storage_provider="AZURE", azure_storage_account, and Secret references for tenant ID, client ID and client secret. For GCP, set storage_provider="GCP" and gcp_credentials_json_secret to a Secret containing the application credentials JSON file content. For Hugging Face, set storage_provider="HUGGINGFACE" and optionally set huggingface_token_secret to a Secret containing the token; omitting it selects anonymous access at the default rate limits. Registration returns setup instructions without verifying storage access.

describe_storage_integration

describe_storage_integration(name: str, *, namespace_path: Optional[List[str]] = None) -> StorageIntegrationInfo

Return configuration and Secret references without credential values.

list_storage_integrations

list_storage_integrations(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

Lazily list one namespace of this database's storage integrations.

drop_storage_integration

drop_storage_integration(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a storage integration with ownership permission.

create_secret

create_secret(name: str, value: str, *, namespace_path: Optional[List[str]] = None) -> None

Create a named Secret in this database.

Fails if the name is taken, so a create never silently becomes a rotation. Nothing reads the value back.

alter_secret

alter_secret(name: str, value: str, *, namespace_path: Optional[List[str]] = None) -> None

Replace the credential behind an existing Secret.

Fails if it does not exist. Bound Functions use the new value from their next job, with no new Function version.

list_secrets

list_secrets(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

The names of every Secret in this database. Names only.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_secret

drop_secret(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a Secret. Bound Functions fail at their next job.

describe_secret

describe_secret(name: str, *, namespace_path: Optional[List[str]] = None) -> SecretInfo

What this database records about a Secret. Never the value.

create_view

create_view(name: str, query: str, *, namespace_path: Optional[List[str]] = None) -> ViewDescription

Create a view: a named query the database plans on every read.

See DBConnection.create_view.

describe_view

describe_view(name: str, *, namespace_path: Optional[List[str]] = None) -> ViewDescription

What this database records about a view: query and schema.

drop_view

drop_view(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a view and wait for catalog removal.

The tables it reads are untouched. Use :meth:drop_view_async to get its SQL query job. Completion confirms catalog removal; physical cleanup continues in the background.

drop_view_async

drop_view_async(name: str, *, namespace_path: Optional[List[str]] = None) -> AsyncJob[None]

Drop a view and return its SQL query job.

The job has type query and its ID is the SQL query UUID. Await :meth:AsyncJob.wait to confirm catalog removal. Physical cleanup continues in the background.

list_views

list_views(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

The names of the views in one namespace.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

create_property_graph

create_property_graph(name: str, nodes: Sequence[NodeTable], edges: Sequence[EdgeTable] = (), *, namespace_path: Optional[List[str]] = None) -> AsyncPropertyGraph

Create a property graph over tables in its namespace.

See DBConnection.create_property_graph.

open_property_graph

open_property_graph(name: str, *, namespace_path: Optional[List[str]] = None) -> AsyncPropertyGraph

Open the property graph named name, describing it to check that it exists.

See DBConnection.open_property_graph.

drop_property_graph

drop_property_graph(name: str, *, namespace_path: Optional[List[str]] = None) -> None

Drop a property graph and wait for its definition to be deleted.

The tables it reads are untouched.

list_property_graphs

list_property_graphs(*, namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

The names of the property graphs in one namespace.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

list_jobs

list_jobs(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[JobInfo]

List server-side jobs across the database's tables.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

cancel_job

cancel_job(job_id: str) -> bool

Request cancellation of a server-side job by id.

Returns True if the server accepted the cancellation, False if no such job exists. Cancelling an already-terminal job is a no-op success.

pause_job

pause_job(job_id: str) -> str

Pause a server-side job by id.

The job's workers drain and it stays parked until resumed. Returns "pausing", "already_paused", or "committing" -- a job finalizing its results cannot be parked; retry shortly.

resume_job

resume_job(job_id: str) -> str

Resume a paused server-side job by id.

Its workers pick their work back up from checkpoints. Returns "resumed", "still_pausing" -- retry shortly -- or "not_paused".

execute_query

execute_query(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> AsyncRecordBatchReader

Execute SQL and return an asynchronous Arrow reader.

This submits through :meth:execute_query_async and waits until the initial result stream is readable. It does not wait for the full query to finish. A statement that ran as a server-side SQL job exposes the durable Job Registry id in the reader schema metadata under x-lancedb-sql-job-id. Use :meth:execute_query_async when the query handle itself is needed.

execute_query_async

execute_query_async(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> AsyncQuery

Start executing SQL and return its query handle.

The database from connect_async is used for unqualified database references. The namespace defaults to ["public"]. parameters binds values to $1 / $name placeholders; see DBConnection.execute_query_async. LanceDB Embedded connections raise NotImplementedError.

describe_query

describe_query(query_id: Union[UUID, str]) -> QueryDescription

Describe a submitted SQL query by its UUID or string ID.

For a handle with job_type == "query", pass its id on the connection that submitted it. Registry job IDs are not query IDs.

namespace_client

namespace_client() -> LanceNamespace

Get the equivalent namespace client for this connection.

For native storage connections, this returns a DirectoryNamespace pointing to the same root with the same storage options.

For namespace connections, this returns the backing namespace client.

For LanceDB Enterprise connections, this returns a RestNamespace with the same URI and authentication headers.

Returns:

  • LanceNamespace –

    The namespace client for this connection.

drop_database

drop_database()

Drop database This is the same thing as dropping all the tables

lancedb.listing.AsyncListing

Bases: AsyncIterator[T], Generic[T]

A lazy async iterator with inspectable pagination state.

Returned by connection resource-listing methods. For example::

async for name in connection.list_views(page_limit=20):
    print(name)

Requests happen during iteration; request errors terminate the iterator.

num_page_results

num_page_results() -> int

Return the number of cached items available without another request.

page_token

page_token() -> Optional[str]

Return the token for the next request.

Initially this is the supplied starting token (None starts at the beginning). After fetching the final page it is None, even if items remain cached. Drain the cache before saving a token to avoid skipping items when resuming. A failed request terminates iteration and retains its token for retrying with a new iterator.

async for name in db.list_tables() loads pages as needed. To inspect the first page, use page = await db.list_tables() and then page.tables. Reading the page list does not consume the iterator. page.page_token is the token for the next request; drain cached results before using it to resume in a new listing. The same behavior applies to namespaces.

lancedb.listing.AsyncTableListing

Bases: _AsyncPagedNameListing

Awaitable table iterator; awaiting loads one page.

For example::

page = await db.list_tables(limit=20)
print(page.tables, page.page_token)
async for name in db.list_tables(limit=20):
    print(name)

tables

tables: list[str]

A copy of every table name in the most recently fetched page.

lancedb.listing.AsyncNamespaceListing

Bases: _AsyncPagedNameListing

Awaitable namespace iterator; awaiting loads one page.

For example::

page = await db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)

namespaces

namespaces: list[str]

A copy of every name in the most recently fetched page.

Namespaces (Asynchronous)

lancedb.connect_namespace_async

connect_namespace_async(namespace_client_impl: str, namespace_client_properties: Dict[str, str], *, read_consistency_interval: Optional[timedelta] = None, storage_options: Optional[Dict[str, str]] = None, session: Optional[Session] = None, namespace_client_pushdown_operations: Optional[List[str]] = None) -> AsyncLanceNamespaceDBConnection

Connect to a LanceDB database through a namespace (returns async connection).

This function is synchronous but returns an AsyncLanceNamespaceDBConnection that provides async methods for all database operations.

Parameters:

  • namespace_client_impl (str) –

    The namespace client implementation to use. For examples:

    • "dir" for DirectoryNamespace
    • "rest" for REST-based namespace
    • Full module path for custom implementations
  • namespace_client_properties (Dict[str, str]) –

    Configuration properties for the namespace client implementation. Different namespace implementations have different config properties. For example, use DirectoryNamespace with {"root": "/path/to/directory"}

  • read_consistency_interval (Optional[timedelta], default: None ) –

    The interval at which to check for updates to the table from other processes. If None, then consistency is not checked.

  • storage_options (Optional[Dict[str, str]], default: None ) –

    Additional options for the storage backend

  • session (Optional[Session], default: None ) –

    A session to use for this connection

  • namespace_client_pushdown_operations (Optional[List[str]], default: None ) –

    List of namespace operations to push down to the namespace server. Supported values:

    • "QueryTable": Execute queries on the namespace server via namespace.query_table() instead of locally.
    • "CreateTable": Execute table creation on the namespace server via namespace.create_table() instead of using declare_table + local write.

    Default is None (no pushdown, all operations run locally).

Returns:

  • AsyncLanceNamespaceDBConnection –

    An async namespace-based connection to LanceDB

Examples:

>>> import lancedb
>>> # This function is sync, but returns an async connection
>>> db = lancedb.connect_namespace_async("dir", {"root": "/path/to/db"})
>>> # Use async methods on the connection
>>> async def use_db():
...     tables = await db.table_names()
...     table = await db.create_table("my_table", schema=schema)

lancedb.namespace.AsyncLanceNamespaceDBConnection

An async LanceDB connection that uses a namespace for table management.

This connection delegates table URI resolution to a lance_namespace instance, while providing async methods for all operations.

read_consistency_interval

read_consistency_interval = read_consistency_interval

storage_options

storage_options = storage_options or {}

session

session = session

table_names

table_names(page_token: Optional[str] = None, limit: Optional[int] = None, *, namespace_path: Optional[List[str]] = None) -> Iterable[str]

List table names in the namespace.

If limit is None, return all table names; zero returns an empty list.

.. deprecated:: Use :meth:list_tables instead, which provides proper pagination support.

create_table

create_table(name: str, data: Optional[DATA] = None, schema: Optional[Union[Schema, LanceModel]] = None, mode: str = 'create', exist_ok: bool = False, on_bad_vectors: str = 'error', fill_value: float = 0.0, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, data_storage_version: Optional[str] = None, enable_v2_manifest_paths: Optional[bool] = None) -> AsyncTable

Create a new table in the namespace.

open_table

open_table(name: str, *, namespace_path: Optional[List[str]] = None, storage_options: Optional[Dict[str, str]] = None, index_cache_size: Optional[int] = None, branch: Optional[str] = None, version: Optional[int] = None) -> AsyncTable

Open an existing table from the namespace.

create_materialized_view

create_materialized_view(name: str, source: MaterializedViewSource, *, select: 'SelectArg' = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> 'AsyncMaterializedView'

Define a materialized view over a table in the root namespace.

create_materialized_view_async

create_materialized_view_async(name: str, source: MaterializedViewSource, *, select: 'SelectArg' = None, where: Optional[str] = None, limit: Optional[int] = None, with_no_data: bool = False, auto_refresh_enabled: Optional[bool] = None, auto_refresh_interval_seconds: Optional[float] = None) -> AsyncJob[None]

Submit materialized-view creation and return its job.

open_materialized_view

open_materialized_view(name: str) -> 'AsyncMaterializedView'

Open the materialized view named name.

list_materialized_views

list_materialized_views(*, page_token: Optional[str] = None, page_limit: Optional[int] = None) -> AsyncListing[str]

The names of the materialized views in the root namespace.

Returns a lazy iterator. page_limit limits each request, not the total; page_token resumes from a saved token. Requests and errors occur during iteration. See pagination state.

drop_materialized_view

drop_materialized_view(name: str, namespace_path: Optional[List[str]] = None) -> None

Drop a materialized view from the namespace.

drop_materialized_view_async

drop_materialized_view_async(name: str, namespace_path: Optional[List[str]] = None) -> AsyncJob[None]

Start dropping a materialized view and return its cleanup job.

drop_table

drop_table(name: str, namespace_path: Optional[List[str]] = None)

Drop a table from the namespace.

drop_table_async

drop_table_async(name: str, namespace_path: Optional[List[str]] = None) -> AsyncJob

Start dropping a table and return its cleanup job.

rename_table

rename_table(cur_name: str, new_name: str, cur_namespace_path: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None)

Rename a table in the namespace.

drop_database

drop_database()

Deprecated method.

drop_all_tables

drop_all_tables(namespace_path: Optional[List[str]] = None)

Drop all tables in the namespace.

list_namespaces

list_namespaces(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> AsyncNamespaceListing

List child namespaces under the given namespace.

Parameters:

  • namespace_path (Optional[List[str]], default: None ) –

    The parent namespace path to list children from. If None, lists root-level namespaces.

  • page_token (Optional[str], default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (int, default: None ) –

    Maximum number of namespaces to return.

Returns:

  • NamespaceListing –

    Response containing namespace names and optional page_token for pagination.

create_namespace

create_namespace(namespace_path: List[str], mode: Optional[str] = None, properties: Optional[Dict[str, str]] = None) -> CreateNamespaceResponse

Create a new namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace path to create.

  • mode (str, default: None ) –

    Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), or "overwrite" (replace if exists). Case insensitive.

  • properties (Dict[str, str], default: None ) –

    Properties to set on the namespace.

Returns:

  • CreateNamespaceResponse –

    Response containing the properties of the created namespace.

drop_namespace

drop_namespace(namespace_path: List[str], mode: Optional[str] = None, behavior: Optional[str] = None) -> DropNamespaceResponse

Drop a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace path to drop.

  • mode (str, default: None ) –

    Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.

  • behavior (str, default: None ) –

    Whether to restrict drop if not empty ("RESTRICT") or cascade ("CASCADE"). Case insensitive.

Returns:

  • DropNamespaceResponse –

    Response containing properties and transaction_id if applicable.

describe_namespace

describe_namespace(namespace_path: List[str]) -> DescribeNamespaceResponse

Describe a namespace.

Parameters:

  • namespace_path (List[str]) –

    The namespace identifier to describe.

Returns:

  • DescribeNamespaceResponse –

    Response containing the namespace properties.

namespace_exists

namespace_exists(namespace_id: List[str]) -> bool

Check if a namespace exists.

Parameters:

  • namespace_id (List[str]) –

    The namespace identifier to check.

Returns:

  • bool –

    True if the namespace exists, False otherwise.

table_exists

table_exists(table_id: List[str]) -> bool

Check if a table exists.

Parameters:

  • table_id (List[str]) –

    The table identifier to check (full path including namespace segments and table name).

Returns:

  • bool –

    True if the table exists, False otherwise.

list_tables

list_tables(namespace_path: Optional[List[str]] = None, page_token: Optional[str] = None, limit: Optional[int] = None) -> AsyncTableListing

List all tables in this database with pagination support.

Parameters:

  • namespace_path (List[str], default: None ) –

    The namespace to list tables in. None or empty list represents root namespace.

  • page_token (str, default: None ) –

    Token for pagination. Use the token from a previous response to get the next page of results.

  • limit (int, default: None ) –

    The maximum number of results to return.

Returns:

  • TableListing –

    Response containing table names and optional page_token for pagination.

execute_query

execute_query(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> AsyncRecordBatchReader

Execute SQL when supported by the underlying connection.

execute_query_async

execute_query_async(query: str, *, default_namespace_path: Optional[List[str]] = None, parameters: Optional[QueryParameters] = None) -> AsyncQuery

Start executing SQL when supported by the underlying connection.

Namespace-backed LanceDB Embedded connections do not support SQL.

describe_query

describe_query(query_id: Union[UUID, str]) -> QueryDescription

Describe a submitted SQL query when supported.

namespace_client

namespace_client() -> LanceNamespace

Get the namespace client for this connection.

For namespace connections, this returns the backing namespace client that was provided during construction.

Returns:

  • LanceNamespace –

    The namespace client for this connection.

Tables (Asynchronous)

Table hold your actual data as a collection of records / rows.

lancedb.table.AsyncTable

An AsyncTable is a collection of Records in a LanceDB Database.

An AsyncTable can be obtained from the AsyncConnection.create_table and AsyncConnection.open_table methods.

An AsyncTable object is expected to be long lived and reused for multiple operations. AsyncTable objects will cache a certain amount of index data in memory. This cache will be freed when the Table is garbage collected. To eagerly free the cache you can call the close method. Once the AsyncTable is closed, it cannot be used for any further operations.

An AsyncTable can also be used as a context manager, and will automatically close when the context is exited. Closing a table is optional. If you do not close the table, it will be closed when the AsyncTable object is garbage collected.

Examples:

Create using AsyncConnection.create_table (more examples in that method's documentation).

>>> import lancedb
>>> async def create_a_table():
...     db = await lancedb.connect_async("./.lancedb")
...     data = [{"vector": [1.1, 1.2], "b": 2}]
...     table = await db.create_table("my_table", data=data)
...     print(await table.query().limit(5).to_arrow())
>>> import asyncio
>>> asyncio.run(create_a_table())
pyarrow.Table
vector: fixed_size_list<item: float>[2]
  child 0, item: float
b: int64
----
vector: [[[1.1,1.2]]]
b: [[2]]

Can append new data with AsyncTable.add().

>>> async def add_to_table():
...     db = await lancedb.connect_async("./.lancedb")
...     table = await db.open_table("my_table")
...     await table.add([{"vector": [0.5, 1.3], "b": 4}])
>>> asyncio.run(add_to_table())

Can query the table with AsyncTable.vector_search.

>>> async def search_table_for_vector():
...     db = await lancedb.connect_async("./.lancedb")
...     table = await db.open_table("my_table")
...     results = (
...       await table.vector_search([0.4, 0.4]).select(["b", "vector"]).to_pandas()
...     )
...     print(results)
>>> asyncio.run(search_table_for_vector())
   b      vector  _distance
0  4  [0.5, 1.3]       0.82
1  2  [1.1, 1.2]       1.13

Search queries are much faster when an index is created. See AsyncTable.create_index.

name

name: str

The name of the table.

tags

tags: AsyncTags

Tag management for the dataset.

Similar to Git, tags are a way to add metadata to a specific version of the dataset.

.. warning::

Tagged versions are exempted from the
:py:meth:`optimize(cleanup_older_than)` process.

To remove a version that has been tagged, you must first
:py:meth:`~Tags.delete` the associated tag.

branches

branches: AsyncBranches

Branch management for the table.

Branches are isolated, writable lines of history forked from another branch (or version). Writes on a branch do not affect main.

is_open

is_open() -> bool

Return True if the table is open.

close

close()

Close the table and free any resources associated with it.

It is safe to call this method multiple times.

Any attempt to use the table after it has been closed will raise an error.

set_unenforced_primary_key

set_unenforced_primary_key(columns: Union[str, Iterable[str]]) -> None

Set the unenforced primary key for this table to the given ordered list of columns.

"Unenforced" means LanceDB does not check uniqueness on writes; the columns are recorded in the schema as the primary key so that features such as merge_insert can use them. Calling this again replaces any previously-set primary key.

Parameters:

  • columns (str or Iterable[str]) –

    Either a single column name (single-column key) or an ordered iterable of column names (composite key). Each column dtype must be one of: int32, int64, utf8, large_utf8, binary, large_binary, fixed_size_binary.

enable_rte

enable_rte(spec: 'RTEWriteSpec') -> None

Install an RTEWriteSpec on this table.

The spec selects Lance's MemWAL LSM-style write path for future merge_insert calls. RTEWriteSpec chooses one of three sharding strategies:

  • RTEWriteSpec.bucket(column, num_buckets) — hash-bucket writes by the single-column unenforced primary key.
  • RTEWriteSpec.identity(column) — shard by the raw value of a scalar column.
  • RTEWriteSpec.unsharded() — route every write to a single shard.

All variants require the table to have an unenforced primary key set via [set_unenforced_primary_key]; bucket sharding additionally requires it to be the single column being bucketed.

By default the RTE maintains every index on the table, re-read as the table changes, so an index created later is maintained too; one of a kind the RTE cannot mirror is skipped. Name the set with with_maintained_indexes to pin it instead: each name must already exist and be maintainable (a still-building index is rejected, not omitted); [] maintains none.

Parameters:

  • spec (RTEWriteSpec) –

    The sharding spec to install.

Examples:

>>> from lancedb import RTEWriteSpec
>>> # table.set_unenforced_primary_key("id")
>>> # table.enable_rte(RTEWriteSpec.bucket("id", 16))

disable_rte

disable_rte() -> None

Remove the RTEWriteSpec from this table.

Reverts to the standard merge_insert write path. Errors if no spec is currently set, or if the RTE still holds rows that have not been merged into the base table, since removing it would drop them.

get_rte_spec

get_rte_spec() -> Optional['RTEWriteSpec']

Read the RTEWriteSpec currently installed on this table.

Returns None when the RTE write path is not enabled at all — no spec has been set, or one was removed with disable_rte. That is a different answer from a spec whose maintained_indexes is None, which is an installed spec selecting indexes automatically.

The spec read back is the one that was installed, selection included: None maintains every supported index the table has now or gains later, [] maintains none, and a non-empty list maintains exactly those. All three round-trip.

get_table_overrides

get_table_overrides() -> dict[str, Any]

Read Enterprise table overrides for this table.

Table overrides configure Enterprise-managed background behavior such as automatic compaction, cleanup, and reindex execution. This method is supported only on remote Enterprise tables.

update_table_overrides

update_table_overrides(**overrides: Any) -> dict[str, Any]

Merge Enterprise table overrides into the current table overrides.

Only fields supplied in overrides are changed; unrelated overrides from the current read snapshot are preserved.

Examples:

>>> await table.update_table_overrides(
...     job_types={
...         "compaction": {"enabled": True},
...         "cleanup": {"enabled": False},
...     }
... )

reset_table_overrides

reset_table_overrides() -> dict[str, Any]

Clear Enterprise table overrides for this table.

checkpoint_rte

checkpoint_rte() -> None

Converge this table's RTE write path into its base table.

One flush, freezing every memtable into an SSTable, then compaction triggers until every SSTable that existed at that moment has reached base. The loop runs client-side, reading progress from get_rte_stats.

Best-effort: SSTables created while it runs are deliberately not waited on, which is what lets it terminate on a table taking writes. Idempotent and safe on a cadence.

There is no deadline, and the caller owns that. It returns when the target SSTables are gone, raises on a terminal server fault, and otherwise waits however long the server takes. A slow table and a stuck one are the same picture from the client: the compactor pool is shared across every table on the node, so a checkpoint queued behind unrelated work looks exactly like one that is merging. Wrap this in asyncio.wait_for for a wall-clock bound; abandoning it partway costs nothing.

flush_rte

flush_rte() -> None

Freeze every table shard's active memtable into an SSTable.

Does not touch the base table — compacting SSTables into base is compact_rte. On a node that has not claimed this table, this claims it and replays its WAL log first.

compact_rte

compact_rte() -> None

Trigger a background SSTable compaction pass per table shard.

Returns once the passes are dispatched, not once they finish: watch get_rte_stats for progress, or use checkpoint_rte to loop until the current SSTables have reached base.

get_rte_stats

get_rte_stats(*, include_sstable_rows: bool = False) -> Optional[dict]

Read live per-table-shard RTE state.

Answers "how far behind is my fresh tier", "which table shard is hot", and "why is my fresh-tier vector search brute-force". Mutates no table state, though on a node that has not claimed this table it claims it, exactly as a read would.

Returns None only when the RTE write path is not enabled.

Parameters:

  • include_sstable_rows (bool, default: False ) –

    Report a row count per SSTable. Off by default: each count opens an uncached Lance dataset, and checkpoint_rte polls this needing only generation numbers.

close_rte_writers

close_rte_writers() -> None

Drain and close any cached RTE shard writers for this table.

When an RTE write spec is installed, merge_insert opens RTE shard writers and caches them for reuse across calls. This closes them, flushing pending data; writers reopen lazily on the next merge_insert. It is a no-op when no writers are cached.

schema

schema() -> Schema

The Arrow Schema of this Table. Blob v2 fields retain their Arrow extension type.

embedding_functions

embedding_functions() -> Dict[str, EmbeddingFunctionConfig]

Get the embedding functions for the table

Returns:

  • funcs ( Dict[str, EmbeddingFunctionConfig] ) –

    A mapping of the vector column to the embedding function or empty dict if not configured.

count_rows

count_rows(filter: Optional[str] = None) -> int

Count the number of rows in the table.

Parameters:

  • filter (Optional[str], default: None ) –

    A SQL where clause to filter the rows to count.

head

head(n=5) -> Table

Return the first n rows of the table.

Parameters:

  • n –

    The number of rows to return.

query

query() -> AsyncQuery

Returns an AsyncQuery that can be used to search the table.

Use methods on the returned query to control query behavior. The query can be executed with methods like to_arrow, to_pandas and more.

to_lance

to_lance(**kwargs) -> LanceDataset

Return the Lance dataset backing a LanceDB Embedded table.

LanceDB Enterprise tables cannot be opened as Lance datasets by the client. Use to_arrow or query to read them through the server.

Parameters:

  • **kwargs –

    Forwarded to lance.dataset.

Returns:

  • LanceDataset –

    The Lance dataset at this table handle's version and branch.

Examples:

>>> async def get_lance_dataset(table):
...     return await table.to_lance()

to_pandas

to_pandas(blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Return the table as a pandas DataFrame.

Parameters:

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how Lance blob columns are returned. LanceDB Enterprise supports "descriptions"; "bytes" and "lazy" are not yet supported.

  • **kwargs –

    Forwarded to PyArrow / Lance pandas conversion.

Returns:

  • DataFrame –

to_arrow

to_arrow(*, blob_mode: BlobMode = 'lazy') -> Table

Return the table as a pyarrow Table.

Parameters:

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists.

Returns:

  • Table –

create_index

create_index(column: str, *, replace: Optional[bool] = None, config: Optional[Union[IvfFlat, IvfPq, IvfRq, HnswPq, HnswSq, HnswFlat, BTree, Bitmap, LabelList, Fm, NGram, ZoneMap, FTS]] = None, wait_timeout: Optional[timedelta] = None, name: Optional[str] = None, train: bool = True)

Create an index to speed up queries

Indices can be created on vector columns or scalar columns. Indices on vector columns will speed up vector searches. Indices on scalar columns will speed up filtering (in both vector and non-vector searches)

Parameters:

  • column (str) –

    The column to index.

  • replace (Optional[bool], default: None ) –

    Whether to replace the existing index

    If this is false, and another index already exists on the same columns and the same name, then an error will be returned. This is true even if that index is out of date.

    The default is True

  • config (Optional[Union[IvfFlat, IvfPq, IvfRq, HnswPq, HnswSq, HnswFlat, BTree, Bitmap, LabelList, Fm, NGram, ZoneMap, FTS]], default: None ) –

    For advanced configuration you can specify the type of index you would like to create. You can also specify index-specific parameters when creating an index object.

  • wait_timeout (Optional[timedelta], default: None ) –

    LanceDB Enterprise only: how long to wait for the index build to finish. If not set, the call returns once the build is scheduled.

  • name (Optional[str], default: None ) –

    The name of the index. If not provided, a default name will be generated.

  • train (bool, default: True ) –

    Whether to train the index with existing data. Vector indices always train with existing data.

Notes

On LanceDB Embedded tables the index is built before this method returns. On LanceDB Enterprise tables it returns once the build is scheduled. Until the build finishes, a vector index may be missing from list_indices, a scalar or FTS index may be listed with 0 indexed rows, and queries won't use the index. To block until the index is ready, pass wait_timeout, call wait_for_index, or use create_index_async and wait on its job when the server returned a job ID.

wait_timeout waits for the submitted build when the server returns a job ID. On older servers that don't return one, it falls back to polling the index by name. wait_for_index always waits by name. When replacing an existing index, those name-based waits may return before the new build finishes because the old index can still match. create_index_async can track a specific build only when the server returned a job ID; otherwise it returns an already-complete job and callers must use wait_for_index for that build.

create_index_async

create_index_async(column: str, *, replace: Optional[bool] = None, config: Optional[Union[IvfFlat, IvfPq, IvfRq, HnswPq, HnswSq, HnswFlat, BTree, Bitmap, LabelList, Fm, NGram, ZoneMap, FTS]] = None, wait_timeout: Optional[timedelta] = None, name: Optional[str] = None, train: bool = True) -> AsyncJob

Create an index, returning a handle to the indexing job.

Takes the same arguments as :meth:create_index. The job may already be complete when returned; callers must not assume the index exists until :meth:AsyncJob.wait resolves.

In LanceDB Enterprise, the returned job tracks this submission only if the server returns a job ID. If it does not, the job is already complete and does not wait for the background build; use wait_for_index and account for the replacement caveat described in create_index.

drop_index

drop_index(name: str) -> None

Drop an index from the table.

Parameters:

  • name (str) –

    The name of the index to drop.

Notes

This does not delete the index from disk, it just removes it from the table. To delete the index, run optimize after dropping the index.

Use list_indices to find the names of the indices.

prewarm_index

prewarm_index(name: str) -> None

Prewarm an index in the table.

This is a hint to the database that the index will be accessed in the future and should be loaded into memory if possible. This can reduce cold-start latency for subsequent queries.

This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently.

It is generally wasteful to call this if the index does not fit into the available cache. Not all index types support prewarming; unsupported indices will silently ignore the request.

Parameters:

  • name (str) –

    The name of the index to prewarm

prewarm_data

prewarm_data(columns: Optional[List[str]] = None) -> None

Prewarm data for the table.

This is a hint to the database that the given columns will be accessed in the future and the database should prefetch the data if possible. Currently only supported in LanceDB Enterprise.

This call initiates prewarming and returns once the request is accepted. It is idempotent and safe to call from multiple clients concurrently.

This operation has a large upfront cost but can speed up future queries that need to fetch the given columns. Large columns such as embeddings or binary data may not be practical to prewarm. This feature is intended for workloads that issue many queries against the same columns.

Parameters:

  • columns (Optional[List[str]], default: None ) –

    The columns to prewarm. If None, all columns are prewarmed.

wait_for_index

wait_for_index(index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300)) -> None

Wait for indexing to complete for the given index names. This will poll the table until all the indices are fully indexed, or raise a timeout exception if the timeout is reached.

Parameters:

  • index_names (Iterable[str]) –

    The name of the indices to poll

  • timeout (timedelta, default: timedelta(seconds=300) ) –

    Timeout to wait for asynchronous indexing. The default is 5 minutes.

stats

stats() -> TableStatistics

Retrieve table and fragment statistics.

uri

uri() -> str

Get the table URI (storage location).

In LanceDB Enterprise, this fetches the location from the server via describe. In LanceDB Embedded, this returns the dataset URI.

Returns:

  • str –

    The full storage location of the table (e.g., S3/GCS path).

initial_storage_options

initial_storage_options() -> Optional[Dict[str, str]]

Get the initial storage options that were passed in when opening this table.

For dynamically refreshed options (e.g., credential vending), use :meth:latest_storage_options.

Warning: This is an internal API and the return value is subject to change.

Returns:

  • Optional[Dict[str, str]] –

    The storage options, or None if no storage options were configured.

latest_storage_options

latest_storage_options() -> Optional[Dict[str, str]]

Get the latest storage options, refreshing from provider if configured.

This method is useful for credential vending scenarios where storage options may be refreshed dynamically. If no dynamic provider is configured, this returns the initial static options.

Warning: This is an internal API and the return value is subject to change.

Returns:

  • Optional[Dict[str, str]] –

    The storage options, or None if no storage options were configured.

add

add(data: DATA, *, mode: Optional[Literal['append', 'overwrite']] = 'append', on_bad_vectors: Optional[OnBadVectorsType] = None, fill_value: Optional[float] = None, progress: Optional[Union[bool, Callable, Any]] = None, write_parallelism: Optional[int] = None, allow_external_blob_outside_bases: bool = False) -> AddResult

Add more data to the AsyncTable.

Plain binary values are cast into blob columns using the table's schema. To construct Arrow record batches explicitly, use blob_array.

Parameters:

  • data (DATA) –

    The data to insert into the table. Acceptable types are:

    • list-of-dict

    • pandas.DataFrame

    • pyarrow.Table or pyarrow.RecordBatch

  • mode (Optional[Literal['append', 'overwrite']], default: 'append' ) –

    The mode to use when writing the data. Valid values are "append" and "overwrite". Overwrite adopts explicitly declared blob layouts, while inferred bytes retain the existing blob v2 fields.

  • on_bad_vectors (Optional[OnBadVectorsType], default: None ) –

    How to handle vectors with incorrect sizes, NaNs, or null elements. One of "error", "drop", "fill", "null".

  • fill_value (Optional[float], default: None ) –

    The value to use when filling vectors. Only used if on_bad_vectors="fill".

  • progress (Optional[Union[bool, Callable, Any]], default: None ) –

    A callback or tqdm-compatible progress bar. See :meth:Table.add for details.

  • write_parallelism (Optional[int], default: None ) –

    Number of partitions to write in parallel. Higher values increase throughput but also peak memory use, since each partition buffers data in flight. Defaults to an estimate based on the data size, capped at the number of CPU cores. Lower this if bulk ingestion is using too much memory.

  • allow_external_blob_outside_bases (bool, default: False ) –

    Allow blob URIs outside registered bases. See :meth:Table.add. LanceDB Embedded only.

merge_insert

merge_insert(on: Union[str, Iterable[str]]) -> LanceMergeInsertBuilder

Returns a LanceMergeInsertBuilder that can be used to create a "merge insert" operation

This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like "insert if not exists", "update or insert (i.e. upsert)", or even replace a portion of existing data with new data (e.g. replace all data where month="january")

The merge insert operation works by combining new data from a source table with existing data in a target table by using a join. There are three categories of records.

"Matched" records are records that exist in both the source table and the target table. "Not matched" records exist only in the source table (e.g. these are new data) "Not matched by source" records exist only in the target table (this is old data)

The builder returned by this method can be used to customize what should happen for each category of data.

Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values.

Parameters:

  • on (Union[str, Iterable[str]]) –

    A column (or columns) to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column. Passing several columns matches on the composite key: a source row updates a target row only when it agrees on every one of them.

Examples:

>>> import lancedb
>>> data = pa.table({"a": [2, 1, 3], "b": ["a", "b", "c"]})
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> new_data = pa.table({"a": [2, 3, 4], "b": ["x", "y", "z"]})
>>> # Perform a "upsert" operation
>>> res = table.merge_insert("a")     \
...      .when_matched_update_all()     \
...      .when_not_matched_insert_all() \
...      .execute(new_data)
>>> res
MergeResult(version=2, num_updated_rows=2, num_inserted_rows=1, num_deleted_rows=0, num_attempts=1, num_rows=3)
>>> # The order of new rows is non-deterministic since we use
>>> # a hash-join as part of this operation and so we sort here
>>> table.to_arrow().sort_by("a").to_pandas()
   a  b
0  1  b
1  2  x
2  3  y
3  4  z

search

search(query: Optional[Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]] = None, vector_column_name: Optional[str] = None, query_type: QueryType = 'auto', ordering_field_name: Optional[str] = None, fts_columns: Optional[Union[str, List[str]]] = None) -> Union[AsyncQuery, AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]

Create a search query to find the nearest neighbors of the given query vector. We currently support vector search and full-text search.

All query options are defined in AsyncQuery.

Parameters:

  • query (Optional[Union[VEC, str, 'PIL.Image.Image', Tuple, FullTextQuery]], default: None ) –

    The targeted vector to search for.

    • default None. Acceptable types are: list, np.ndarray, PIL.Image.Image

    • If None then a plain AsyncQuery is returned, equivalent to calling query. The select/where/limit clauses are applied to filter the table.

  • vector_column_name (Optional[str], default: None ) –

    The name of the vector column to search.

    The vector column needs to be a pyarrow fixed size list type

    • If not specified then the vector column is inferred from the table schema

    • If the table has multiple vector columns then the vector_column_name needs to be specified. Otherwise, an error is raised.

  • query_type (QueryType, default: 'auto' ) –

    default "auto". Acceptable types are: "vector", "fts", "hybrid", or "auto"

    • If "auto" then the query type is inferred from the query;

      • If query is a list/np.ndarray then the query type is "vector";

      • If query is a PIL.Image.Image then either do vector search, or raise an error if no corresponding embedding function is found.

    • If query is a string, then the query type is "vector" if the table has embedding functions else the query type is "fts"

Returns:

  • AsyncQuery, AsyncHybridQuery, AsyncFTSQuery, or AsyncVectorQuery –

    A query builder object representing the query.

vector_search(query_vector: Union[VEC, Tuple]) -> AsyncVectorQuery

Search the table with a given query vector. This is a convenience method for preparing a vector query and is the same thing as calling nearestTo on the builder returned by query. Seer nearest_to for more details.

delete

delete(where: Union[str, Expr]) -> DeleteResult

Delete rows from the table.

This can be used to delete a single row, many rows, all rows, or sometimes no rows (if your predicate matches nothing).

Parameters:

  • where (Union[str, Expr]) –

    The filter condition. Can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

    The filter must not be empty, or it will error.

Examples:

>>> import lancedb
>>> data = [
...    {"x": 1, "vector": [1.0, 2]},
...    {"x": 2, "vector": [3.0, 4]},
...    {"x": 3, "vector": [5.0, 6]}
... ]
>>> db = lancedb.connect("./.lancedb")
>>> table = db.create_table("my_table", data)
>>> table.to_pandas()
   x      vector
0  1  [1.0, 2.0]
1  2  [3.0, 4.0]
2  3  [5.0, 6.0]
>>> table.delete("x = 2")
DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas()
   x      vector
0  1  [1.0, 2.0]
1  3  [5.0, 6.0]

If you have a list of values to delete, you can combine them into a stringified list and use the IN operator:

>>> to_remove = [1, 5]
>>> to_remove = ", ".join([str(v) for v in to_remove])
>>> to_remove
'1, 5'
>>> table.delete(f"x IN ({to_remove})")
DeleteResult(num_deleted_rows=1, version=3)
>>> table.to_pandas()
   x      vector
0  3  [5.0, 6.0]

update

update(updates: Optional[Dict[str, Any]] = None, *, where: Optional[Union[str, Expr]] = None, updates_sql: Optional[Dict[str, str]] = None) -> UpdateResult

This can be used to update zero to all rows in the table.

If a filter is provided with where then only rows matching the filter will be updated. Otherwise all rows will be updated.

Embeddings are not computed by this method. Updating an embedding source column requires providing all associated vector columns in the same call; otherwise a ValueError is raised. Use merge_insert to recompute embeddings automatically.

Parameters:

  • updates (Optional[Dict[str, Any]], default: None ) –

    The updates to apply. The keys should be the name of the column to update. The values should be the new values to assign. This is required unless updates_sql is supplied.

  • where (Optional[Union[str, Expr]], default: None ) –

    The filter condition. Can be a SQL string or a type-safe Expr built with col and lit. Only rows that satisfy this filter will be updated.

  • updates_sql (Optional[Dict[str, str]], default: None ) –

    The updates to apply, expressed as SQL expression strings. The keys should be column names. The values should be SQL expressions. These can be SQL literals (e.g. "7" or "'foo'") or they can be expressions based on the previous value of the row (e.g. "x + 1" to increment the x column by 1)

Returns:

  • UpdateResult –

    An object containing: - rows_updated: The number of rows that were updated - version: The new version number of the table after the update

Examples:

>>> import asyncio
>>> import lancedb
>>> from lancedb.expr import col
>>> import pandas as pd
>>> async def demo_update():
...     data = pd.DataFrame({"x": [1, 2], "vector": [[1, 2], [3, 4]]})
...     db = await lancedb.connect_async("./.lancedb")
...     table = await db.create_table("my_table", data)
...     # x is [1, 2], vector is [[1, 2], [3, 4]]
...     await table.update({"vector": [10, 10]}, where=col("x") == 2)
...     # x is [1, 2], vector is [[1, 2], [10, 10]]
...     await table.update(updates_sql={"x": "x + 1"})
...     # x is [2, 3], vector is [[1, 2], [10, 10]]
>>> asyncio.run(demo_update())

add_columns

add_columns(transforms: Mapping[str, str | FunctionApplication] | FunctionApplication | Field | List[Field] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None = None, *, computed: dict[str, str] | None = None, auto_refresh_enabled: bool | None = None, auto_refresh_interval_seconds: int | None = None, auto_refresh_resource_budget: Optional[Dict[str, int]] = None) -> AddColumnsResult

Add new columns with defined values.

Parameters:

  • transforms (Mapping[str, str | FunctionApplication] | FunctionApplication | Field | List[Field] | Schema | DATA | RecordBatch | RecordBatchReader | DataFrame | None, default: None ) –

    pa.Field, List[pa.Field], pa.Schema, pandas.DataFrame, polars.DataFrame, pa.Table, pa.RecordBatch, pa.RecordBatchReader A map of column name to a SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns. Alternatively, you can pass a pyarrow field or schema to add new columns with NULLs.

    A mapping with one FunctionApplication value keeps its scalar or named-struct result in the named table column. A bare named-struct application expands its ordered result fields as one atomic binding; aliases come from rename(columns=...). Function columns are supported only in LanceDB Enterprise.

    On local tables, a pandas or Polars DataFrame, PyArrow Table, RecordBatch, or RecordBatchReader supplies precomputed new-column values. Supply exactly one row per current live table row, in storage scan order; pandas indexes are ignored. The input contains only new columns. Readers are consumed once, and their batch boundaries need not match table fragments. Keep the table unchanged between computing these values and calling add_columns. Data inputs cannot be combined with computed columns. Remote tables do not support data inputs.

  • computed (dict[str, str] | None, default: None ) –

    LanceDB Embedded declaration and refresh require a source build with the experimental-functions Cargo feature, which published packages do not enable. LanceDB Enterprise operations remain available.

    A mapping from output column names to SQL expressions derives each output field from its expression. A direct projection of a Blob v2 field inherits Blob v2 semantics; other expressions derive their ordinary Arrow type. Mapping order is declaration and dependency order.

    Unlike transforms, the expression is stored rather than evaluated now: the column is committed with no values, and rows get them from refresh_column.

    A refresh also recomputes the rows whose inputs changed since they were computed, so a mutated input is reflected by the next refresh. While a declaration reads a column, that column cannot be renamed, retyped or dropped.

    In LanceDB Enterprise the expression is planned by the server. Cannot be combined with transforms.

  • auto_refresh_enabled (bool | None, default: None ) –

    Enable automatic refresh for each computed output declared by this call. Defaults to True. LanceDB Embedded stores the setting; automatic scheduling requires LanceDB Enterprise.

  • auto_refresh_interval_seconds (int | None, default: None ) –

    Positive minimum seconds after a successful refresh before another automatic refresh is eligible. When omitted, the server determines when to refresh. Unchanged inputs do not force recomputation. Applies to every declared computed output. These options cannot be used with one-time transforms or ordinary all-null columns.

  • auto_refresh_resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate CPU, memory, and GPU budget for automatic refreshes of registered Function outputs. cpu_millis and memory_bytes are required; gpu_count defaults to zero. SQL-expression columns do not use Function workers and reject this option. LanceDB Enterprise only.

Examples:

>>> await table.add_columns(
...     computed={"total": "price * quantity"},
...     auto_refresh_enabled=True,
...     auto_refresh_interval_seconds=300,
... )

Returns:

  • AddColumnsResult –

    version: the new version number of the table after adding columns.

refresh_column

refresh_column(column: str, *, where: Optional[str] = None, batch_size: Optional[int] = None, max_concurrency: Optional[int] = None, publication_interval_rows: Optional[int] = None, resource_budget: Optional[Dict[str, int]] = None, gpu_class: Optional[str] = None, gpu_count: Optional[int] = None, retry_policy: Optional[FunctionRetryPolicy] = None, error_policy: Optional[Literal['fail', 'skip']] = None, max_failed_fraction: Optional[float] = None, max_failed_rows: Optional[int] = None, row_timeout_seconds: Optional[float] = None, init_timeout_seconds: Optional[float] = None, num_workers: Optional[int] = None, max_executor_processes_per_worker: Optional[int] = None) -> RefreshColumnResult

Fill the rows of a computed column that hold no value yet.

Local refresh requires the experimental-functions Cargo feature in a source build. Published packages leave it disabled.

Declared with add_columns(computed=...), a column starts empty and gets its values here. Rows appended since the last refresh are filled by the next one, and rows whose inputs changed since they were computed are recomputed; everything else is left as it is.

Supported for local tables only. For remote tables, a refresh runs as a server job, through refresh_column_async.

Parameters:

  • column (str) –

    The name of the computed column to fill.

  • where (Optional[str], default: None ) –

    SQL filter selecting the rows to refresh, as in REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option.

  • batch_size (Optional[int], default: None ) –

    Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Has no effect on local expression-backed columns.

  • max_concurrency (Optional[int], default: None ) –

    Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.

  • publication_interval_rows (Optional[int], default: None ) –

    Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.

  • resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate resources for one registered Function refresh. The cpu_millis and memory_bytes keys are required; gpu_count is optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-level gpu_count shorthand. LanceDB Enterprise only; local refreshes reject it.

  • gpu_class (Optional[str], default: None ) –

    GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.

  • gpu_count (Optional[int], default: None ) –

    Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use resource_budget instead. LanceDB Enterprise only.

  • retry_policy (Optional[FunctionRetryPolicy], default: None ) –

    Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.

  • error_policy (Optional[Literal['fail', 'skip']], default: None ) –

    Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.

  • max_failed_fraction (Optional[float], default: None ) –

    Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.

  • max_failed_rows (Optional[int], default: None ) –

    Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.

  • row_timeout_seconds (Optional[float], default: None ) –

    Deadline for one Function call. LanceDB Enterprise only.

  • init_timeout_seconds (Optional[float], default: None ) –

    Deadline for Function initialization. LanceDB Enterprise only.

  • num_workers (Optional[int], default: None ) –

    Worker count for an expression column; registered Function columns reject it and use resource_budget instead. LanceDB Enterprise only.

  • max_executor_processes_per_worker (Optional[int], default: None ) –

    Cap on executor processes per Function worker. LanceDB Enterprise only.

Returns:

  • RefreshColumnResult –

    The number of rows filled and the new version of the table.

function_errors

function_errors(job_id: Optional[str] = None, column: Optional[str] = None, limit: Optional[int] = None) -> 'FunctionErrors'

The per-row errors Function refreshes recorded on this table.

A refresh running under a skip policy records each row it skipped with the input that failed and the error. This lists those records, newest job first, plus a summary for any fragment whose per-row detail was capped. LanceDB Enterprise only; reading errors needs read access to the table, since a message carries the value that failed.

Parameters:

  • job_id (Optional[str], default: None ) –

    Only errors recorded by this job.

  • column (Optional[str], default: None ) –

    Only errors on this column.

  • limit (Optional[int], default: None ) –

    At most this many records (server default 10000, cap 100000).

Returns:

  • FunctionErrors –

    records, fragments and truncated, the last saying whether the listing stopped at its limit.

  • Each record identifies its row with ``fragment_id``, ``row_offset``, –
  • the physical ``row_address`` (``(fragment_id << 32) | –
  • row_offset``), and, when the table uses stable row IDs, the stable –
  • ``row_id`` read from the refresh's pinned table version. Prefer –
  • ``row_id`` for retries: compaction and rewrite can change the –
  • physical fields, while a stable row ID survives them. –
  • To retry the recorded rows with stable row IDs: –
  • .. code-block:: python –

    errors = await table.function_errors(column="embedding") row_ids = sorted( record.row_id for record in errors.records if record.row_id is not None ) if row_ids: row_id_list = ", ".join(str(row_id) for row_id in row_ids) await database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowid IN ({row_id_list})" )

  • For a table without stable row IDs, collect the non-null physical –
  • addresses before the table is compacted or rewritten and run the –
  • SQL refresh: –
  • .. code-block:: python –

    errors = await table.function_errors(column="embedding") addresses = sorted( record.row_address for record in errors.records if record.row_address is not None ) if addresses: address_list = ", ".join(str(address) for address in addresses) await database.execute_query( f"REFRESH COLUMN embedding ON {table.name} " f"WHERE _rowaddr IN ({address_list})" )

  • Records whose per-row detail was capped have no address and appear –
  • only in ``fragments``; they cannot be selected individually here. –

Examples:

>>> errors = await table.function_errors(column="embedding")
>>> for record in errors.records:
...     print(record.job_id, record.row_offset, record.error_message)
>>> if errors.truncated:
...     print("listing stopped at the limit")

column_statuses

column_statuses(column: Optional[str] = None) -> list[FunctionColumnStatus]

Return exact status for Function-backed output columns.

All results describe one server table version. column=None returns every Function output. A handle pinned to an exact version requests that version.

column_status

column_status(column: str) -> FunctionColumnStatus

Return exact status for one Function-backed output column.

wait_fresh

wait_fresh(column: str, timeout: Optional[timedelta] = None) -> FunctionColumnStatus

Poll until no stale or never-computed rows remain.

Failed rows do not block completion. Server and transport errors are not retried. Only the typed finalization-pending response is retried with bounded backoff; arbitrary 500 responses surface immediately.

refresh_column_async

refresh_column_async(column: str, *, where: Optional[str] = None, batch_size: Optional[int] = None, max_concurrency: Optional[int] = None, publication_interval_rows: Optional[int] = None, resource_budget: Optional[Dict[str, int]] = None, gpu_class: Optional[str] = None, gpu_count: Optional[int] = None, retry_policy: Optional[FunctionRetryPolicy] = None, error_policy: Optional[Literal['fail', 'skip']] = None, max_failed_fraction: Optional[float] = None, max_failed_rows: Optional[int] = None, row_timeout_seconds: Optional[float] = None, init_timeout_seconds: Optional[float] = None, num_workers: Optional[int] = None, max_executor_processes_per_worker: Optional[int] = None) -> AsyncJob[RefreshColumnResult]

Like :meth:refresh_column, but returns a handle to the refresh job instead of blocking until it completes.

Local refresh requires a source build with experimental-functions. Remote jobs do not require this feature.

The job may already be complete when returned; callers must not assume the column is filled until :meth:AsyncJob.wait resolves. Invalid input -- an unknown column, or one that is not computed -- raises here rather than failing the job. For local tables, the job runs in-process; for remote tables, it is the server's backfill job.

Parameters:

  • column (str) –

    The name of the computed column to fill.

  • where (Optional[str], default: None ) –

    SQL filter selecting the rows to refresh, as in REFRESH COLUMN ... WHERE. Unfilled rows outside it stay unfilled. LanceDB Enterprise only; local refreshes reject this option.

  • batch_size (Optional[int], default: None ) –

    Maximum rows per Function call on the remote server. Must be greater than zero. None uses the server default. Lower this for slow Functions to stay within the executor's per-call deadline. Has no effect on local expression-backed columns.

  • max_concurrency (Optional[int], default: None ) –

    Override the registered Function's maximum concurrent user-code calls for this logical execution. Must be between 1 and 4294967295. None uses the registration default. This is not an account-wide quota. LanceDB Enterprise only; local refreshes reject this option.

  • publication_interval_rows (Optional[int], default: None ) –

    Opt-in publication threshold for completed Function outputs. Must be greater than zero. None keeps whole-fragment publication. The actual publication boundary can be larger than this threshold because it also depends on completed work and Function batching. LanceDB Enterprise only; local refreshes reject this option.

  • resource_budget (Optional[Dict[str, int]], default: None ) –

    Aggregate resources for one registered Function refresh. The cpu_millis and memory_bytes keys are required; gpu_count is optional and defaults to zero, which requests a CPU-only budget. This budget cannot be combined with the top-level gpu_count shorthand. LanceDB Enterprise only; local refreshes reject it.

  • gpu_class (Optional[str], default: None ) –

    GPU class for Function workers. Must be a lowercase name of 1-63 letters, digits, or hyphens, without a leading or trailing hyphen. None keeps the Function's registered class or the deployment default. LanceDB Enterprise only.

  • gpu_count (Optional[int], default: None ) –

    Positive aggregate GPU budget for Function workers without changing the deployment's CPU and memory defaults. None leaves the server's configured budget unchanged. For a complete CPU and memory budget, use resource_budget instead. LanceDB Enterprise only.

  • retry_policy (Optional[FunctionRetryPolicy], default: None ) –

    Override retries of failed user-code calls. Only use when repeating partial work is safe. Set max_attempts=1 to disable retries. Enterprise Functions only; local refreshes reject this option.

  • error_policy (Optional[Literal['fail', 'skip']], default: None ) –

    Action after failure or retry exhaustion. Skip records the error and attempt count, subject to the server's failure limits.

  • max_failed_fraction (Optional[float], default: None ) –

    Under a skip policy, fail the job once more than this share of rows (0 to 1) has failed. LanceDB Enterprise only.

  • max_failed_rows (Optional[int], default: None ) –

    Under a skip policy, fail the job once more than this many rows have failed. LanceDB Enterprise only.

  • row_timeout_seconds (Optional[float], default: None ) –

    Deadline for one Function call. LanceDB Enterprise only.

  • init_timeout_seconds (Optional[float], default: None ) –

    Deadline for Function initialization. LanceDB Enterprise only.

  • num_workers (Optional[int], default: None ) –

    Worker count for an expression column; registered Function columns reject it and use resource_budget instead. LanceDB Enterprise only.

  • max_executor_processes_per_worker (Optional[int], default: None ) –

    Cap on executor processes per Function worker. LanceDB Enterprise only.

Returns:

  • AsyncJob[RefreshColumnResult] –

    A job whose successful wait returns row counts plus the source and published table versions.

Examples:

>>> import asyncio
>>> import lancedb
>>> async def refresh_in_background():
...     db = await lancedb.connect_async("./.lancedb")
...     table = await db.create_table("computed_job_async_demo", [{"x": 1}])
...     await table.add_columns(computed={"doubled": "x * 2"})
...     job = await table.refresh_column_async("doubled")
...     result = await job.wait()
...     assert result.rows_assigned == 1
...     return await job.status()
>>> asyncio.run(refresh_in_background())
'finished'

alter_columns

alter_columns(*alterations: Iterable[dict[str, Any]]) -> AlterColumnsResult

Alter column names and nullability.

alterations : Iterable[Dict[str, Any]] A sequence of dictionaries, each with the following keys: - "path": str The column path to alter. For a top-level column, this is the name. For a nested column, this is the dot-separated path, e.g. "a.b.c". - "rename": str, optional The new name of the column. If not specified, the column name is not changed. - "data_type": pyarrow.DataType, optional The new data type of the column. Existing values will be casted to this type. If not specified, the column data type is not changed. - "nullable": bool, optional Whether the column should be nullable. If not specified, the column nullability is not changed. Only non-nullable columns can be changed to nullable. Currently, you cannot change a nullable column to non-nullable.

Each alteration must specify at least one of "rename", "data_type",
or "nullable". The legacy key "name" is also accepted for renaming.
Unknown keys raise ValueError before any alterations are applied.

Local tables reject renaming source or vector columns used by an
embedding function with ValueError, preserving the embedding binding.

Returns:

  • AlterColumnsResult –

    version: the new version number of the table after the alteration.

update_field_metadata

update_field_metadata(*updates: dict[str, Any]) -> UpdateFieldMetadataResult

Update per-field metadata. See Table.update_field_metadata.

drop_columns

drop_columns(columns: Iterable[str])

Drop columns from the table.

Local tables reject dropping source or vector columns used by an embedding function with ValueError, preserving the embedding binding.

Parameters:

  • columns (Iterable[str]) –

    The names of the columns to drop.

version

version() -> int

Retrieve the version of the table

LanceDb supports versioning. Every operation that modifies the table increases version. As long as a version hasn't been deleted you can [Self::checkout] that version to view the data at that point. In addition, you can [Self::restore] the version to replace the current table with a previous version.

list_versions

list_versions()

List all versions of the table

checkout

checkout(version: int | str)

Checks out a specific version of the Table

Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set.

This is a read-only operation that turns the table into a sort of "view" or "detached head". Other table instances will not be affected. To make the change permanent you can use the [Self::restore] method.

Any operation that modifies the table will fail while the table is in a checked out state. To return the table to a normal state use [Self::checkout_latest].

Parameters:

  • version (int | str) –

    The version to check out. A version number (int) or a tag (str) can be provided.

checkout_latest

checkout_latest()

Ensures the table is pointing at the latest version

This can be used to manually update a table when the read_consistency_interval is None It can also be used to undo a [Self::checkout] operation

restore

restore(version: Optional[int | str] = None)

Restore the table to the currently checked out version

This operation will fail if checkout has not been called previously

This operation will overwrite the latest version of the table with a previous version. Any changes made since the checked out version will no longer be visible.

Once the operation concludes the table will no longer be in a checked out state and the read_consistency_interval, if any, will apply.

take_offsets

take_offsets(offsets: list[int], *, with_row_id: bool = False) -> AsyncTakeQuery

Take a list of offsets from the table.

Offsets are 0-indexed and relative to the current version of the table. Offsets are not stable. A row with an offset of N may have a different offset in a different version of the table (e.g. if an earlier row is deleted).

Offsets are mostly useful for sampling as the set of all valid offsets is easily known in advance to be [0, len(table)).

No guarantees are made regarding the order in which results are returned. Repeated offsets produce repeated rows.

Parameters:

  • offsets (list[int]) –

    The offsets to take.

  • with_row_id (bool, default: False ) –

    If True, include the "_rowid" column in the results.

Returns:

  • AsyncTakeQuery –

    A query object that can be executed to get the rows at the given offsets.

take_row_ids

take_row_ids(row_ids: list[int], *, with_row_id: bool = False) -> AsyncTakeQuery

Take a list of row ids from the table.

Results follow the requested row-id order. Repeated IDs produce repeated rows; missing or deleted IDs are omitted. This also applies to remote tables.

Row ids are not stable and are relative to the current version of the table. They can change due to compaction and updates.

Unlike offsets, row ids are not 0-indexed and no assumptions should be made about the possible range of row ids. In order to use this method you must first obtain the row ids by scanning or searching the table.

Even so, row ids are more stable than offsets and can be useful in some situations.

There is an ongoing effort to make row ids stable which is tracked at https://github.com/lancedb/lancedb/issues/1120

Parameters:

  • row_ids (list[int]) –

    The row ids to take.

  • with_row_id (bool, default: False ) –

    If True, include the "_rowid" column in the results.

Returns:

  • AsyncTakeQuery –

    A query object that can be executed to get the rows.

blob_columns

blob_columns() -> list[str]

fetch_blobs

fetch_blobs(column: str, row_ids: Union[list[int], Array, ChunkedArray, Table]) -> LargeBinaryArray

fetch_blob_ranges

fetch_blob_ranges(column: str, requests: Sequence[Sequence[int]]) -> LargeBinaryArray

fetch_blob_files

fetch_blob_files(column: str, row_ids: Union[list[int], Array, ChunkedArray, Table]) -> 'list[Optional[BlobFile]]'

open_blob_file

open_blob_file(descriptor: Union[dict, StructScalar, None]) -> Optional[BlobFile]

Open a snapshot-bound descriptor as a seekable file.

See Table.open_blob_file.

open_blob_files

open_blob_files(descriptors: Iterable) -> AsyncIterator[Optional[BlobFile]]

Asynchronously iterate over files in descriptor order, preserving nulls.

Use async for file in table.open_blob_files(descriptors). Each non-null file must be closed after use.

async for video in videos.open_blob_files(rows["video"]):
    if video is not None:
        with video:
            payload = await video.aread()

current_branch

current_branch() -> Optional[str]

The branch this table handle is scoped to, or None for main.

optimize

optimize(*, cleanup_older_than: Optional[timedelta] = None, delete_unverified: bool = False, retrain=False, compaction_options: Optional[CompactionOptions] = None) -> OptimizeStats

Optimize the on-disk data and indices for better performance.

.. note::

On a LanceDB Enterprise (remote) table, only compaction is
currently supported. Pruning old versions and index optimization
are silently skipped there instead of being attempted, so this
call returns successfully, with zeroed pruning statistics, once
the compaction job has been submitted; it does not wait for that
job to finish. Remote compaction supports per-job
``target_rows_per_fragment``, ``max_source_rows``, ``io_buffer_size``,
and ``batch_size``. Omitted fields inherit table tuning, then
deployment/built-in defaults. Other non-default compaction settings raise
``NotImplementedError`` before submission. The server must support
these SQL options; older-server errors are propagated without
dropping the requested settings.

Modeled after VACUUM in PostgreSQL.

Optimization covers three operations:

  • Compaction: Merges small files into larger ones
  • Prune: Removes old versions of the dataset
  • Index: Optimizes the indices, adding new data to existing indices

Parameters:

  • cleanup_older_than (Optional[timedelta], default: None ) –

    All files belonging to versions older than this will be removed. Set to 0 days to remove all versions except the latest. The latest version is never removed.

    .. warning::

    Setting this to 0 deletes the data files of every older
    version, so any other reader or writer still using an older
    version of the table will fail. Only set it to 0 if no other
    process is working on this dataset.
    
  • delete_unverified (bool, default: False ) –

    Files leftover from a failed transaction may appear to be part of an in-progress operation (e.g. appending new data) and these files will not be deleted unless they are at least 7 days old. If delete_unverified is True then these files will be deleted regardless of their age.

    .. warning::

    This should only be set to True if you can guarantee that no other
    process is currently working on this dataset. Otherwise the dataset
    could be put into a corrupted state.
    
  • retrain –

    This parameter is no longer used and is deprecated.

  • compaction_options (Optional[CompactionOptions], default: None ) –

    Options that control file compaction. For large rows, derive a lower target_rows_per_fragment from the expected row size to bound later passes. This does not cap the first pass over an existing oversized fragment; see CompactionOptions.

Notes

The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations.

list_indices

list_indices() -> Iterable[IndexConfig]

List all indices that have been created with Self::create_index

On LanceDB Enterprise tables, an index whose build hasn't finished may be missing (vector indices) or listed with 0 indexed rows (scalar and FTS indices), and num_unindexed_rows may be None. Use wait_for_index to wait until an index is ready.

tokenize

tokenize(query: str, *, column: Optional[str] = None, index_name: Optional[str] = None) -> Iterable[FtsToken]

Tokenize a query using the tokenizer configured on an FTS index.

Specify exactly one of column or index_name.

Model-backed tokenizers such as jieba/* and lindera/* are rebuilt in the client process from index metadata. In LanceDB Enterprise, this means the same tokenizer model files must also exist locally (under LANCE_LANGUAGE_MODEL_HOME) and must match the files the server uses. A mismatch cannot be detected: the returned tokens may differ from what the server's index produces. If the model files are missing, an error naming the tokenizer is raised.

index_stats

index_stats(index_name: str) -> Optional[IndexStatistics]

Retrieve statistics about an index

Parameters:

  • index_name (str) –

    The name of the index to retrieve statistics for

Returns:

  • IndexStatistics or None –

    The statistics about the index. Returns None if the index does not exist.

uses_v2_manifest_paths

uses_v2_manifest_paths() -> bool

Check if the table is using the new v2 manifest paths.

Returns:

  • bool –

    True if the table is using the new v2 manifest paths, False otherwise.

migrate_manifest_paths_v2

migrate_manifest_paths_v2()

Migrate the manifest paths to the new format.

This will update the manifest to use the new v2 format for paths.

This function is idempotent, and can be run multiple times without changing the state of the object store.

Danger

This should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations.

You can use AsyncTable.uses_v2_manifest_paths to check if the table is already using the new path style.

replace_field_metadata

replace_field_metadata(field_name: str, new_metadata: dict[str, str])

Replace the metadata of a field in the schema

.. deprecated:: 0.33.1 Use :func:update_field_metadata instead.

Parameters:

  • field_name (str) –

    The name of the field to replace the metadata for

  • new_metadata (dict[str, str]) –

    The new metadata to set

lancedb.table.AsyncTags

Async table tag manager.

list

list() -> Dict[str, Tag]

List all table tags.

Returns:

  • dict[str, Tag] –

    A dictionary mapping tag names to version numbers.

get_version

get_version(tag: str) -> int

Get the version of a tag.

Parameters:

  • tag (str) –

    The name of the tag to get the version for.

create

create(tag: str, version: int) -> None

Create a tag for a given table version.

Parameters:

  • tag (str) –

    The name of the tag to create. This name must be unique among all tag names for the table.

  • version (int) –

    The table version to tag.

delete

delete(tag: str) -> None

Delete tag from the table.

Parameters:

  • tag (str) –

    The name of the tag to delete.

update

update(tag: str, version: int) -> None

Update tag to a new version.

Parameters:

  • tag (str) –

    The name of the tag to update.

  • version (int) –

    The new table version to tag.

lancedb.table.AsyncBranches

Async table branch manager.

list

list() -> Dict[str, Any]

List all branches, mapping name to branch metadata.

create

create(name: str, from_ref: Optional[str] = None, from_version: Optional[int] = None) -> 'AsyncTable'

Create a branch and return a handle scoped to it.

Parameters:

  • name (str) –

    Name of the new branch.

  • from_ref (Optional[str], default: None ) –

    Source branch to fork from. Defaults to main.

  • from_version (Optional[int], default: None ) –

    A specific version on from_ref to fork from. Defaults to latest.

checkout

checkout(name: str, version: Optional[int] = None) -> 'AsyncTable'

Check out an existing branch and return a handle scoped to it.

Parameters:

  • name (str) –

    Name of the branch to check out.

  • version (Optional[int], default: None ) –

    A specific version on the branch to pin. When set, the returned handle is a read-only view of that version; when omitted it tracks the branch's latest and stays writable.

delete

delete(name: str) -> None

Delete a branch.

diff

diff(from_branch: str) -> Dict[str, Any]

Diff a branch against main.

cherry_pick

cherry_pick(from_branch: str, dry_run: bool = False) -> Dict[str, Any]

Cherry-pick a branch onto main, or dry-run.

A failed cherry-pick returns status="failed" instead of raising.

Materialized Views (Asynchronous)

lancedb.materialized_view.AsyncMaterializedView

A handle on a materialized view: its table plus its definition.

Obtained from AsyncConnection.create_materialized_view or AsyncConnection.open_materialized_view.

name

name: str

table

table: 'AsyncTable'

The view, as the table it is. Queries, indexes and search all apply; writes are not blocked, but a rebuild replaces them.

definition

definition() -> MaterializedViewDefinition

The query that defines the view.

refresh

refresh(*, full: bool = False, source_version: Optional[int] = None, cpu_millis: Optional[int] = None, memory_bytes: Optional[int] = None, gpu_count: Optional[int] = None, gpu_class: Optional[str] = None) -> 'RefreshMaterializedViewResult'

Recompute the view from its source.

In LanceDB Enterprise, full=True forces a rebuild; otherwise the server chooses the refresh mode. LanceDB Embedded views always rebuild, and source_version selects a source version instead of the latest. LanceDB Enterprise SQL refreshes do not support source-version pinning.

cpu_millis (1000 = one core), memory_bytes, and gpu_count cap aggregate resources for this execution, including its Function processes. CPU or memory requires all three budget fields; GPU count may be zero for CPU-only views. gpu_class selects an operator-defined GPU class for this execution only. Omitted options preserve defaults. These options require an Enterprise backend that enforces isolation; Embedded refreshes reject them. Other servers may reject the new options.

LanceDB Enterprise results leave mode, rows_written, and source_version as None because the SQL job adapter does not expose these metrics.

refresh_async

refresh_async(*, full: bool = False, source_version: Optional[int] = None, cpu_millis: Optional[int] = None, memory_bytes: Optional[int] = None, gpu_count: Optional[int] = None, gpu_class: Optional[str] = None) -> 'AsyncJob[RefreshMaterializedViewResult]'

Submit a refresh and return its job without waiting.

The job may already be complete in LanceDB Embedded. In LanceDB Distributed, refreshes are submitted through the SQL service and the returned id is the registry job ID, usable with open_job for progress, events, and cancellation while the refresh runs. Options and result metrics follow refresh.

Servers without early registry-handle support return a connection-scoped query job instead. Its ID cannot be reopened with open_job.

Property Graphs (Asynchronous)

AsyncConnection returns this handle from create_property_graph and open_property_graph.

lancedb.graph.AsyncPropertyGraph

A handle on one property graph: its name and namespace, and the connection it is described through.

Obtained from AsyncConnection.create_property_graph or AsyncConnection.open_property_graph. The graph is queried through SQL GRAPH_TABLE, not through this handle.

name

name: str

The graph's name within its namespace.

namespace_path

namespace_path: List[str]

The namespace holding the graph; empty is the root namespace.

describe

describe() -> PropertyGraphDescription

The graph's node and edge tables, each with the properties create resolved.

Indices (Asynchronous)

Indices can be created on a table to speed up queries. This section lists the indices that LanceDb supports.

IvfHnswPq

IvfHnswPq = HnswPq

IvfHnswSq

IvfHnswSq = HnswSq

IvfHnswFlat

IvfHnswFlat = HnswFlat

IndexConfig

A description of an index currently configured on a column

columns

columns: List[str] = <attribute 'columns' of 'lancedb._lancedb.IndexConfig' objects>

The columns in the index

Currently this is always a list of size 1. In the future there may be more columns to represent composite indices.

created_at

created_at: Optional[datetime] = <attribute 'created_at' of 'lancedb._lancedb.IndexConfig' objects>

When the index was created.

index_details

index_details: Optional[Any] = <attribute 'index_details' of 'lancedb._lancedb.IndexConfig' objects>

Index-type-specific details parsed as a Python object (dict, list, etc.).

Falls back to a raw string if JSON parsing fails. None when unavailable.

index_type

index_type: str = <attribute 'index_type' of 'lancedb._lancedb.IndexConfig' objects>

The type of the index

index_uuid

index_uuid: Optional[str] = <attribute 'index_uuid' of 'lancedb._lancedb.IndexConfig' objects>

The UUID of the first segment of the index.

index_version

index_version: Optional[int] = <attribute 'index_version' of 'lancedb._lancedb.IndexConfig' objects>

The on-disk index format version.

name

name: str = <attribute 'name' of 'lancedb._lancedb.IndexConfig' objects>

Name of the index.

num_indexed_rows

num_indexed_rows: Optional[int] = <attribute 'num_indexed_rows' of 'lancedb._lancedb.IndexConfig' objects>

The number of rows indexed, across all segments.

num_segments

num_segments: Optional[int] = <attribute 'num_segments' of 'lancedb._lancedb.IndexConfig' objects>

The number of segments that make up the index.

num_unindexed_rows

num_unindexed_rows: Optional[int] = <attribute 'num_unindexed_rows' of 'lancedb._lancedb.IndexConfig' objects>

The number of rows not yet covered by this index.

size_bytes

size_bytes: Optional[int] = <attribute 'size_bytes' of 'lancedb._lancedb.IndexConfig' objects>

The total size in bytes of all index files across all segments.

type_url

type_url: Optional[str] = <attribute 'type_url' of 'lancedb._lancedb.IndexConfig' objects>

The protobuf type URL, a precise type identifier for the index.

BTree

Describes a btree index configuration

A btree index is an index on scalar columns. The index stores a copy of the column in sorted order. A header entry is created for each block of rows (currently the block size is fixed at 4096). These header entries are stored in a separate cacheable structure (a btree). To search for data the header is used to determine which blocks need to be read from disk.

For example, a btree index in a table with 1Bi rows requires sizeof(Scalar) * 256Ki bytes of memory and will generally need to read sizeof(Scalar) * 4096 bytes to find the correct row ids.

This index is good for scalar columns with mostly distinct values and does best when the query is highly selective. It works with numeric, temporal, and string columns.

The btree index does not currently have any parameters though parameters such as the block size may be added in the future.

Bitmap

Describe a Bitmap index configuration.

A Bitmap index stores a bitmap for each distinct value in the column for every row.

This index works best for low-cardinality numeric or string columns, where the number of unique values is small (i.e., less than a few thousands). Bitmap index can accelerate the following filters:

  • <, <=, =, >, >=
  • IN (value1, value2, ...)
  • between (value1, value2)
  • is null

For example, a bitmap index with a table with 1Bi rows, and 128 distinct values, requires 128 / 8 * 1Bi bytes on disk.

LabelList

Describe a LabelList index configuration.

LabelList is a scalar index that can be used on List<T> columns to support queries with array_has_all and array_has_any using an underlying bitmap index.

For example, it works with tags, categories, keywords, etc.

Fm

Describe an FM-Index configuration.

Fm is a scalar index on string or binary columns that accelerates substring search, i.e. contains(col, 'needle'). Unlike the tokenized FTS index, it matches arbitrary substrings of the raw bytes.

For example, it works with url, path, content, etc.

NGram

Describe an NGram index configuration.

An NGram index is a scalar index on a UTF-8 string column that accelerates substring filters such as contains(col, 'needle'), along with some LIKE and regular-expression predicates. It tokenizes values into overlapping character n-grams (using Lance's default trigram parameters) and narrows a scan to the rows that share the query's n-grams; the exact predicate is then evaluated on those candidates, so the results are the same with or without the index.

Unlike the Fm index, which matches arbitrary substrings of the raw bytes, NGram is trigram based, so a query shorter than the n-gram length cannot be served by the index.

For example, it works with text, description, comment, etc.

ZoneMap

Describe a ZoneMap index configuration.

A ZoneMap index stores compact min and max summaries for contiguous ranges (zones) of rows. During a scan it skips any zone whose summary cannot satisfy the predicate, so it accelerates range and equality filters such as col > 10, col BETWEEN a AND b, and col = x, without changing the results.

It suits large numeric or temporal columns whose values are correlated with row order (for example timestamp, price, id).

FTS

Describe a FTS index configuration.

FTS supports string columns, lists of strings, and JSON columns created from PyArrow pa.json_(). JSON columns use Lance's JSON-aware tokenizer and are stored as LargeBinary with the lance.json logical type. Arbitrary binary columns are not supported.

JSON queries use path,str,value triples. Paths are relative to the JSON column and use dots for nested object keys. For example, meta.tag,str,nature matches the string value at $.meta.tag; it does not search other paths or JSON key names. The value is processed with the configured text tokenizer. Use str, not string.

Create and query a JSON index through the modern API::

import pyarrow as pa
from lancedb.index import FTS

table = db.create_table(
    "documents",
    pa.table({"doc": pa.array(
        ['{"meta":{"tag":"nature"}}'], type=pa.json_()
    )}),
)
table.create_index("doc", config=FTS())
table.search(
    "meta.tag,str,nature", query_type="fts", fts_columns="doc"
).to_list()

Create the index on the JSON column itself, not a JSON subpath such as doc.meta.tag. JSON-path scalar indexes do not provide FTS path scoping. Plain text such as nature is not a valid JSON FTS query. The current Lance tokenizer can panic on malformed triples; see https://github.com/lance-format/lance/issues/8812.

LanceDB Enterprise uses the same FTS() configuration and requires a server that supports JSON FTS. Wait for table.wait_for_index(["doc_idx"]) before searching a newly created index there with its default name.

Examples:

Create an index configuration that uses 256-document posting blocks:

>>> config = FTS(block_size=256)

Create an index that treats each deepest-list element as one document:

>>> from lancedb.query import DocumentGranularity
>>> config = FTS(document_granularity=DocumentGranularity.LIST_ELEMENT)

Create an index that splits code identifiers into searchable subwords:

>>> config = FTS(analyzer="code", split_identifiers=True)
Notes

Model-backed tokenizers such as jieba/default and lindera/ipadic require tokenizer models in Lance's language model home. Set LANCE_LANGUAGE_MODEL_HOME to override the default platform data directory under lance/language_models.

with_position

with_position: bool = False

base_tokenizer

base_tokenizer: BaseTokenizerType = 'simple'

language

language: str = 'English'

max_token_length

max_token_length: Optional[int] = 40

lower_case

lower_case: bool = True

stem

stem: Optional[bool] = None

remove_stop_words

remove_stop_words: Optional[bool] = None

ascii_folding

ascii_folding: bool = True

ngram_min_length

ngram_min_length: int = 3

ngram_max_length

ngram_max_length: int = 3

prefix_only

prefix_only: bool = False

block_size

block_size: int = 128

custom_stop_words

custom_stop_words: Optional[List[str]] = None

memory_limit

memory_limit: Optional[int] = None

num_workers

num_workers: Optional[int] = None

document_granularity

document_granularity: DocumentGranularity = ROW

analyzer

analyzer: Optional[Literal['text', 'code']] = None

split_identifiers

split_identifiers: Optional[bool] = None

split_on_numerics

split_on_numerics: Optional[bool] = None

preserve_original

preserve_original: Optional[bool] = None

index_operators

index_operators: Optional[bool] = None

HnswPq

Describe a HNSW-PQ index configuration.

HNSW-PQ stands for Hierarchical Navigable Small World - Product Quantization. It is a variant of the HNSW algorithm that uses product quantization to compress the vectors. To create an HNSW-PQ index, you can specify the following parameters:

Parameters:

  • distance_type (Literal['l2', 'cosine', 'dot'], default: 'l2' ) –

    The distance metric used to train the index.

    The following distance types are available:

    "l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).

    "cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].

    "dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.

  • num_partitions (Optional[int], default: None ) –

    The number of IVF partitions to create.

    For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.

  • num_sub_vectors (Optional[int], default: None ) –

    Number of sub-vectors of PQ.

    This value controls how much the vector is compressed during the quantization step. The more sub vectors there are the less the vector is compressed. The default is the dimension of the vector divided by 16. If the dimension is not evenly divisible by 16 we use the dimension divided by 8.

    The above two cases are highly preferred. Having 8 or 16 values per subvector allows us to use efficient SIMD instructions.

    If the dimension is not visible by 8 then we use 1 subvector. This is not ideal and will likely result in poor performance.

  • num_bits (int, default: 8 ) –

    Number of bits to encode each sub-vector.

    This value controls how much the sub-vectors are compressed. The more bits the more accurate the index but the slower search. Only 4 and 8 are supported.

  • max_iterations (int, default: 50 ) –

    Max iterations to train kmeans.

    When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.

    Increasing this might improve the quality of the index but in most cases the parameter is unused because kmeans will converge with fewer iterations. The parameter is only used in cases where kmeans does not appear to converge. In those cases it is unlikely that setting this larger will lead to the index converging anyways.

  • sample_rate (int, default: 256 ) –

    The rate used to calculate the number of training vectors for kmeans.

    When an IVF index is trained, we need to calculate partitions. These are groups of vectors that are similar to each other. To do this we use an algorithm called kmeans.

    Running kmeans on a large dataset can be slow. To speed this up we run kmeans on a random sample of the data. This parameter controls the size of the sample. The total number of vectors used to train the index is sample_rate * num_partitions.

    Increasing this value might improve the quality of the index but in most cases the default should be sufficient.

  • m (int, default: 20 ) –

    The number of neighbors to select for each vector in the HNSW graph.

    This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.

  • ef_construction (int, default: 300 ) –

    The number of candidates to evaluate during the construction of the HNSW graph.

    This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than ef in the search phase.

  • target_partition_size (Optional[int], default: None ) –

    The target size of each partition.

    This value controls the tradeoff between search performance and accuracy. faster search but less accurate results as higher value.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

num_sub_vectors

num_sub_vectors: Optional[int] = None

num_bits

num_bits: int = 8

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

m

m: int = 20

ef_construction

ef_construction: int = 300

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

HnswSq

Describe a HNSW-SQ index configuration.

HNSW-SQ stands for Hierarchical Navigable Small World - Scalar Quantization. It is a variant of the HNSW algorithm that uses scalar quantization to compress the vectors.

Parameters:

  • distance_type (Literal['l2', 'cosine', 'dot'], default: 'l2' ) –

    The distance metric used to train the index.

    The following distance types are available:

    "l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).

    "cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].

    "dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.

  • num_partitions (Optional[int], default: None ) –

    The number of IVF partitions to create.

    For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.

  • max_iterations (int, default: 50 ) –

    Max iterations to train kmeans.

    When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.

    Increasing this might improve the quality of the index but in most cases the parameter is unused because kmeans will converge with fewer iterations. The parameter is only used in cases where kmeans does not appear to converge. In those cases it is unlikely that setting this larger will lead to the index converging anyways.

  • sample_rate (int, default: 256 ) –

    The rate used to calculate the number of training vectors for kmeans.

    When an IVF index is trained, we need to calculate partitions. These are groups of vectors that are similar to each other. To do this we use an algorithm called kmeans.

    Running kmeans on a large dataset can be slow. To speed this up we run kmeans on a random sample of the data. This parameter controls the size of the sample. The total number of vectors used to train the index is sample_rate * num_partitions.

    Increasing this value might improve the quality of the index but in most cases the default should be sufficient.

  • m (int, default: 20 ) –

    The number of neighbors to select for each vector in the HNSW graph.

    This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.

  • ef_construction (int, default: 300 ) –

    The number of candidates to evaluate during the construction of the HNSW graph.

    This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than ef in the search phase.

  • target_partition_size (Optional[int], default: None ) –

    The target size of each partition.

    This value controls the tradeoff between search performance and accuracy. faster search but less accurate results as higher value.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

m

m: int = 20

ef_construction

ef_construction: int = 300

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

HnswFlat

Describe a HNSW-FLAT index configuration.

HNSW-FLAT stands for Hierarchical Navigable Small World without quantization. It stores raw vectors in the HNSW graph, providing the highest recall among the IVF_HNSW family at the cost of more memory and disk space compared to :class:HnswSq or :class:HnswPq.

Parameters:

  • distance_type (Literal['l2', 'cosine', 'dot'], default: 'l2' ) –

    The distance metric used to train the index.

    The following distance types are available:

    "l2" - Euclidean distance. This is a very common distance metric that accounts for both magnitude and direction when determining the distance between vectors. l2 distance has a range of [0, ∞).

    "cosine" - Cosine distance. Cosine distance is a distance metric calculated from the cosine similarity between two vectors. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them. Unlike l2, the cosine distance is not affected by the magnitude of the vectors. Cosine distance has a range of [0, 2].

    "dot" - Dot product. Dot distance is the dot product of two vectors. Dot distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance.

  • num_partitions (Optional[int], default: None ) –

    The number of IVF partitions to create.

    For HNSW, we recommend a small number of partitions. Setting this to 1 works well for most tables. For very large tables, training just one HNSW graph will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training.

  • max_iterations (int, default: 50 ) –

    Max iterations to train kmeans.

    When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run.

  • sample_rate (int, default: 256 ) –

    The rate used to calculate the number of training vectors for kmeans.

  • m (int, default: 20 ) –

    The number of neighbors to select for each vector in the HNSW graph.

    This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be.

  • ef_construction (int, default: 300 ) –

    The number of candidates to evaluate during the construction of the HNSW graph.

    This value controls the tradeoff between build speed and accuracy. The higher the value the more accurate the build but the slower it will be. 150 to 300 is the typical range. 100 is a minimum for good quality search results. In most cases, there is no benefit to setting this higher than 500. This value should be set to a value that is not less than ef in the search phase.

  • target_partition_size (Optional[int], default: None ) –

    The target size of each partition.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

m

m: int = 20

ef_construction

ef_construction: int = 300

target_partition_size

target_partition_size: Optional[int] = None

IvfFlat

Describes an IVF Flat Index

This index stores raw vectors. These vectors are grouped into partitions of similar vectors. Each partition keeps track of a centroid which is the average value of all vectors in the group.

distance_type

distance_type: Literal['l2', 'cosine', 'dot', 'hamming'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

IvfSq

Describes an IVF Scalar Quantization (SQ) index.

This index applies scalar quantization to compress vectors and organizes the quantized vectors into IVF partitions. It offers a balance between search speed and storage efficiency while keeping good recall.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

IvfPq

Describes an IVF PQ Index

This index stores a compressed (quantized) copy of every vector. These vectors are grouped into partitions of similar vectors. Each partition keeps track of a centroid which is the average value of all vectors in the group.

During a query the centroids are compared with the query vector to find the closest partitions. The compressed vectors in these partitions are then searched to find the closest vectors.

The compression scheme is called product quantization. Each vector is divide into subvectors and then each subvector is quantized into a small number of bits. the parameters num_bits and num_subvectors control this process, providing a tradeoff between index size (and thus search speed) and index accuracy.

The partitioning process is called IVF and the num_partitions parameter controls how many groups to create.

Note that training an IVF PQ index on a large dataset is a slow operation and currently is also a memory intensive operation.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

num_sub_vectors

num_sub_vectors: Optional[int] = None

num_bits

num_bits: int = 8

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

IvfRq

Describes an IVF RQ Index

IVF-RQ (RabitQ Quantization) compresses vectors using RabitQ quantization and organizes them into IVF partitions.

The compression scheme is called RabitQ quantization. Each dimension is quantized into a small number of bits. The parameters num_bits and num_partitions control this process, providing a tradeoff between index size (and thus search speed) and index accuracy.

The partitioning process is called IVF and the num_partitions parameter controls how many groups to create.

Note that training an IVF RQ index on a large dataset is a slow operation and currently is also a memory intensive operation.

distance_type

distance_type: Literal['l2', 'cosine', 'dot'] = 'l2'

num_partitions

num_partitions: Optional[int] = None

num_bits

num_bits: Optional[int] = None

max_iterations

max_iterations: int = 50

sample_rate

sample_rate: int = 256

target_partition_size

target_partition_size: Optional[int] = None

accelerator

accelerator: Optional[str] = None

lancedb.table.IndexStatistics

Statistics about an index.

num_indexed_rows

num_indexed_rows: int

num_unindexed_rows

num_unindexed_rows: int

index_type

index_type: Literal['IVF_FLAT', 'IVF_SQ', 'IVF_PQ', 'IVF_RQ', 'IVF_HNSW_SQ', 'IVF_HNSW_PQ', 'IVF_HNSW_FLAT', 'FTS', 'BTREE', 'BITMAP', 'LABEL_LIST', 'NGRAM', 'ZONEMAP']

distance_type

distance_type: Optional[Literal['l2', 'cosine', 'dot']] = None

num_indices

num_indices: Optional[int] = None

Querying (Asynchronous)

Queries allow you to return data from your database. Basic queries can be created with the AsyncTable.query method to return the entire (typically filtered) table. Vector searches return the rows nearest to a query vector and can be created with the AsyncTable.vector_search method.

lancedb.query.AsyncQuery

Bases: AsyncStandardQuery

to_query_object

to_query_object() -> Query

Convert the query into a query object

This is currently experimental but can be useful as the query object is pure python and more easily serializable.

select

select(columns: Union[List[str], dict[str, str]]) -> Self

Return only the specified columns.

By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.

As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.

You can also use this method to create new "dynamic" columns based on your existing columns. For example, you may not care about "a" or "b" but instead simply want "a + b". This is often seen in the SELECT clause of an SQL query (e.g. SELECT a+b FROM my_table).

To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.

For example, an SQL query might state SELECT a + b AS combined, c. The equivalent input to this method would be {"combined": "a + b", "c": "c"}.

Columns will always be returned in the order given, even if that order is different than the order used when adding the data.

with_row_id

with_row_id() -> Self

Include the _rowid column in the results.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Include the _rowaddr column in scanner-backed plain query results.

with_fragments

with_fragments(fragments: Any) -> Self

Restrict scanner-backed plain query results to the given Lance fragments.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Restrict scanner-backed plain query results to the given Lance fragment ids.

to_batches

to_batches(*, max_batch_length: Optional[int] = None, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader

Execute the query and return the results as an Apache Arrow RecordBatchReader.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • max_batch_length (Optional[int], default: None ) –

    The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Use async for batch in await query.to_batches(blob_mode="bytes") to stream blob payloads and convert each batch with batch.to_pandas().

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and collect the results into an Apache Arrow Table.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse "bytes".

to_list

to_list(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Execute the query and collect the results into a pandas DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_polars

to_polars(timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and collect the results into a Polars DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

Examples:

>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())

to_pydantic

to_pydantic(model: Type[LanceModel], *, timeout: Optional[timedelta] = None) -> List[LanceModel]

Convert results to a list of pydantic models.

Parameters:

  • model (Type[LanceModel]) –

    The pydantic model to use.

  • timeout (timedelta, default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • list[LanceModel] –

explain_plan

explain_plan(verbose: Optional[bool] = False)

Return the execution plan for this query.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
...     plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
...     print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and display with runtime metrics.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

where

where(predicate: Union[str, Expr]) -> Self

Only return rows matching the given predicate

The predicate can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

Examples:

>>> predicate = "x > 10"
>>> predicate = "y > 0 AND y < 100"
>>> predicate = "x > 5 OR y = 'test'"

Filtering performance can often be improved by creating a scalar index on the filter column(s).

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

limit

limit(limit: int) -> Self

Set the maximum number of results to return.

By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.

offset

offset(offset: int) -> Self

Set the offset for the results.

Parameters:

  • offset (int) –

    The offset to start fetching results from.

order_by

order_by(ordering: Optional[List[ColumnOrdering]]) -> Self

Set the ordering for the results.

Parameters:

  • ordering (Optional[List[ColumnOrdering]]) –

    The ordering to use for the results. If None, then the default ordering will be used.

fast_search() -> Self

Skip searching un-indexed data.

This can make queries faster, but will miss any data that has not been indexed.

Tip

You can add new data into an existing index by calling AsyncTable.optimize.

use_rte

use_rte(enable: bool) -> Self

Control RTE LSM read routing for this query.

By default (unset), a query against a table with an RTE write spec (see AsyncTable.enable_rte) is routed through the LSM scanner so it also returns data written via the merge_insert LSM path that has not yet been compacted into the base table (the active/frozen in-memory memtables and the SSTables), deduplicated by primary key; a table without a spec reads the base table.

Parameters:

  • enable (bool) –

    True forces the LSM scanner and errors if the table has no RTE write spec. False bypasses the RTE and reads the base table only, even when a spec is present.

postfilter

postfilter() -> Self

If this is called then filtering will happen after the search instead of before. By default filtering will be performed before the search. This is how filtering is typically understood to work. This prefilter step does add some additional latency. Creating a scalar index on the filter column(s) can often improve this latency. However, sometimes a filter is too complex or scalar indices cannot be applied to the column. In these cases postfiltering can be used instead of prefiltering to improve latency. Post filtering applies the filter to the results of the search. This means we only run the filter on a much smaller set of data. However, it can cause the query to return fewer than limit results (or even no results) if none of the nearest results match the filter. Post filtering happens during the "refine stage" (described in more detail in @see {@link VectorQuery#refineFactor}). This means that setting a higher refine factor can often help restore some of the results lost by post filtering.

nearest_to

nearest_to(query_vector: Union[VEC, Tuple, List[VEC]]) -> AsyncVectorQuery

Find the nearest vectors to the given query vector.

This converts the query from a plain query to a vector query.

This method will attempt to convert the input to the query vector expected by the embedding model. If the input cannot be converted then an error will be thrown.

By default, there is no embedding model, and the input should be something that can be converted to a pyarrow array of floats. This includes lists, numpy arrays, and tuples.

If there is only one vector column (a column whose data type is a fixed size list of floats) then the column does not need to be specified. If there is more than one vector column you must use AsyncVectorQuery.column to specify which column you would like to compare with.

If no index has been created on the vector column then a vector query will perform a distance comparison between the query vector and every vector in the database and then sort the results. This is sometimes called a "flat search"

For small databases, with tens of thousands of vectors or less, this can be reasonably fast. In larger databases you should create a vector index on the column. If there is a vector index then an "approximate" nearest neighbor search (frequently called an ANN search) will be performed. This search is much faster, but the results will be approximate.

The query can be further parameterized using the returned builder. There are various ANN search parameters that will let you fine tune your recall accuracy vs search latency.

Vector searches always have a limit. If limit has not been called then a default limit of 10 will be used.

Typically, a single vector is passed in as the query. However, you can also pass in multiple vectors. When multiple vectors are passed in, if the vector column is with multivector type, then the vectors will be treated as a single query. Or the vectors will be treated as multiple queries, this can be useful if you want to find the nearest vectors to multiple query vectors. Flat searches share one table scan across the query vectors, avoiding the scan and memory amplification of making multiple queries concurrently. If multiple vectors are passed in then an additional column query_index will be added to the results. This column will contain the index of the query vector that the result is nearest to.

nearest_to_text

nearest_to_text(query: str | FullTextQuery, columns: Union[str, List[str], None] = None) -> AsyncFTSQuery

Find the documents that are most relevant to the given text query.

This method will perform a full text search on the table and return the most relevant documents. The relevance is determined by BM25.

The columns to search must be with native FTS index (Tantivy-based can't work with this method).

By default, all indexed columns are searched, now only one column can be searched at a time.

Parameters:

  • query (str | FullTextQuery) –

    The text query to search for.

  • columns (Union[str, List[str], None], default: None ) –

    The columns to search in. If None, all indexed columns are searched. For now only one column can be searched at a time.

lancedb.query.AsyncVectorQuery

Bases: AsyncStandardQuery, AsyncVectorQueryBase

column

column(column: str) -> Self

Set the vector column to query

This controls which column is compared to the query vector supplied in the call to AsyncQuery.nearest_to.

This parameter must be specified if the table has more than one column whose data type is a fixed-size-list of floats.

nprobes

nprobes(nprobes: int) -> Self

Set the legacy IVF probe parameter

The number of probes must be greater than 0.

This argument is only used when the vector column has an IVF-based index. If there is no index then this value is ignored.

The IVF stage of IVF PQ divides the input into partitions (clusters) of related values.

The partition whose centroids are closest to the query vector will be exhaustiely searched to find matches. This parameter controls how many partitions will be searched.

Increasing this value will increase the recall of your query but will also increase the latency of your query. If this method is not called, Lance's adaptive probe defaults are used.

For best results we recommend tuning this parameter with a benchmark against your actual data to find the smallest possible value that will still give you the desired recall.

LanceDB retains this as nprobes through LanceDB Embedded and LanceDB Enterprise request construction. Lance sets both probe bounds to this value. Explicit minimum or maximum settings can override their respective bounds.

search_effort

search_effort(search_effort: Optional[float]) -> Self

Set the initial IVF search effort to a finite value in [0, 1].

Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.

None preserves Auto (internally 0.5) and fixed nprobes. Any explicit effort, including 0.5, overrides nprobes in either call order; explicit minimum and maximum bounds remain effective. Zero starts at the caller minimum and one searches all partitions within the maximum. Later probing may continue to obtain enough candidates. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.

Examples:

>>> import asyncio
>>> import lancedb
>>> async def build_query():
...     db = await lancedb.connect_async("memory://")
...     table = await db.create_table(
...         "effort_async", [{"vector": [0.1, 0.2]}]
...     )
...     return table.vector_search([0.1, 0.2]).search_effort(0.75)
>>> query = asyncio.run(build_query())

minimum_nprobes

minimum_nprobes(minimum_nprobes: int) -> Self

Set the minimum number of probes to use.

See nprobes for more details.

These partitions will be searched on every indexed vector query and will increase recall at the expense of latency.

maximum_nprobes

maximum_nprobes(maximum_nprobes: int) -> Self

Set the maximum number of probes to use.

See nprobes for more details.

If this value is greater than minimum_nprobes then the excess partitions will be searched only if we have not found enough results.

This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.

If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.

distance_range

distance_range(lower_bound: Optional[float] = None, upper_bound: Optional[float] = None) -> Self

Set the distance range to use.

Only rows with distances within range [lower_bound, upper_bound) will be returned.

Parameters:

  • lower_bound (Optional[float], default: None ) –

    The lower bound of the distance range.

  • upper_bound (Optional[float], default: None ) –

    The upper bound of the distance range.

Returns:

  • AsyncVectorQuery –

    The AsyncVectorQuery object.

ef

ef(ef: int) -> Self

Set the number of candidates to consider during search

This argument is only used when the vector column has an HNSW index. If there is no index then this value is ignored.

Increasing this value will increase the recall of your query but will also increase the latency of your query. The default value is 1.5 * limit. This default is good for many cases but the best value to use will depend on your data and the recall that you need to achieve.

approx_mode

approx_mode(approx_mode: Literal['fast', 'normal', 'accurate']) -> Self

Set the speed / accuracy tradeoff to use for approximate search

This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.

"fast" favors lower latency and may reduce recall, "normal" (the default) uses the index's default balance, and "accurate" favors higher recall and may increase latency.

Examples:

>>> import asyncio
>>> import lancedb
>>> async def build_query():
...     db = await lancedb.connect_async("memory://")
...     table = await db.create_table(
...         "approx_async", [{"vector": [0.1, 0.2]}]
...     )
...     return table.vector_search([0.1, 0.2]).approx_mode("accurate")
>>> query = asyncio.run(build_query())

refine_factor

refine_factor(refine_factor: int) -> Self

A multiplier to control how many additional rows are taken during the refine step

This argument is only used when the vector column has an IVF PQ index. If there is no index then this value is ignored.

An IVF PQ index stores compressed (quantized) values. They query vector is compared against these values and, since they are compressed, the comparison is inaccurate.

This parameter can be used to refine the results. It can improve both improve recall and correct the ordering of the nearest results.

To refine results LanceDb will first perform an ANN search to find the nearest limit * refine_factor results. In other words, if refine_factor is 3 and limit is the default (10) then the first 30 results will be selected. LanceDb then fetches the full, uncompressed, values for these 30 results. The results are then reordered by the true distance and only the nearest 10 are kept.

Note: there is a difference between calling this method with a value of 1 and never calling this method at all. Calling this method with any value will have an impact on your search latency. When you call this method with a refine_factor of 1 then LanceDb still needs to fetch the full, uncompressed, values so that it can potentially reorder the results.

Note: if this method is NOT called then the distances returned in the _distance column will be approximate distances based on the comparison of the quantized query vector and the quantized result vectors. This can be considerably different than the true distance between the query vector and the actual uncompressed vector.

distance_type

distance_type(distance_type: str) -> Self

Set the distance metric to use

When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use. See @see {@link IvfPqOptions.distanceType} for more details on the different distance metrics available.

Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.

By default "l2" is used.

bypass_vector_index

bypass_vector_index() -> Self

If this is called then any vector index is skipped

An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.

to_query_object

to_query_object() -> Query

Convert the query into a query object

This is currently experimental but can be useful as the query object is pure python and more easily serializable.

select

select(columns: Union[List[str], dict[str, str]]) -> Self

Return only the specified columns.

By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.

As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.

You can also use this method to create new "dynamic" columns based on your existing columns. For example, you may not care about "a" or "b" but instead simply want "a + b". This is often seen in the SELECT clause of an SQL query (e.g. SELECT a+b FROM my_table).

To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.

For example, an SQL query might state SELECT a + b AS combined, c. The equivalent input to this method would be {"combined": "a + b", "c": "c"}.

Columns will always be returned in the order given, even if that order is different than the order used when adding the data.

with_row_id

with_row_id() -> Self

Include the _rowid column in the results.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Include the _rowaddr column in scanner-backed plain query results.

with_fragments

with_fragments(fragments: Any) -> Self

Restrict scanner-backed plain query results to the given Lance fragments.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Restrict scanner-backed plain query results to the given Lance fragment ids.

to_batches

to_batches(*, max_batch_length: Optional[int] = None, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader

Execute the query and return the results as an Apache Arrow RecordBatchReader.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • max_batch_length (Optional[int], default: None ) –

    The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Use async for batch in await query.to_batches(blob_mode="bytes") to stream blob payloads and convert each batch with batch.to_pandas().

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and collect the results into an Apache Arrow Table.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse "bytes".

to_list

to_list(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Execute the query and collect the results into a pandas DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_polars

to_polars(timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and collect the results into a Polars DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

Examples:

>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())

to_pydantic

to_pydantic(model: Type[LanceModel], *, timeout: Optional[timedelta] = None) -> List[LanceModel]

Convert results to a list of pydantic models.

Parameters:

  • model (Type[LanceModel]) –

    The pydantic model to use.

  • timeout (timedelta, default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • list[LanceModel] –

explain_plan

explain_plan(verbose: Optional[bool] = False)

Return the execution plan for this query.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
...     plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
...     print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and display with runtime metrics.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

where

where(predicate: Union[str, Expr]) -> Self

Only return rows matching the given predicate

The predicate can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

Examples:

>>> predicate = "x > 10"
>>> predicate = "y > 0 AND y < 100"
>>> predicate = "x > 5 OR y = 'test'"

Filtering performance can often be improved by creating a scalar index on the filter column(s).

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

limit

limit(limit: int) -> Self

Set the maximum number of results to return.

By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.

offset

offset(offset: int) -> Self

Set the offset for the results.

Parameters:

  • offset (int) –

    The offset to start fetching results from.

order_by

order_by(ordering: Optional[List[ColumnOrdering]]) -> Self

Set the ordering for the results.

Parameters:

  • ordering (Optional[List[ColumnOrdering]]) –

    The ordering to use for the results. If None, then the default ordering will be used.

fast_search() -> Self

Skip searching un-indexed data.

This can make queries faster, but will miss any data that has not been indexed.

Tip

You can add new data into an existing index by calling AsyncTable.optimize.

use_rte

use_rte(enable: bool) -> Self

Control RTE LSM read routing for this query.

By default (unset), a query against a table with an RTE write spec (see AsyncTable.enable_rte) is routed through the LSM scanner so it also returns data written via the merge_insert LSM path that has not yet been compacted into the base table (the active/frozen in-memory memtables and the SSTables), deduplicated by primary key; a table without a spec reads the base table.

Parameters:

  • enable (bool) –

    True forces the LSM scanner and errors if the table has no RTE write spec. False bypasses the RTE and reads the base table only, even when a spec is present.

postfilter

postfilter() -> Self

If this is called then filtering will happen after the search instead of before. By default filtering will be performed before the search. This is how filtering is typically understood to work. This prefilter step does add some additional latency. Creating a scalar index on the filter column(s) can often improve this latency. However, sometimes a filter is too complex or scalar indices cannot be applied to the column. In these cases postfiltering can be used instead of prefiltering to improve latency. Post filtering applies the filter to the results of the search. This means we only run the filter on a much smaller set of data. However, it can cause the query to return fewer than limit results (or even no results) if none of the nearest results match the filter. Post filtering happens during the "refine stage" (described in more detail in @see {@link VectorQuery#refineFactor}). This means that setting a higher refine factor can often help restore some of the results lost by post filtering.

rerank

rerank(reranker: Reranker = RRFReranker(), query_string: Optional[str] = None) -> AsyncHybridQuery

nearest_to_text

nearest_to_text(query: str | FullTextQuery, columns: Union[str, List[str], None] = None) -> AsyncHybridQuery

Find the documents that are most relevant to the given text query, in addition to vector search.

This converts the vector query into a hybrid query.

This search will perform a full text search on the table and return the most relevant documents, combined with the vector query results. The text relevance is determined by BM25.

The columns to search must be with native FTS index (Tantivy-based can't work with this method).

By default, all indexed columns are searched, now only one column can be searched at a time.

Parameters:

  • query (str | FullTextQuery) –

    The text query to search for.

  • columns (Union[str, List[str], None], default: None ) –

    The columns to search in. If None, all indexed columns are searched. For now only one column can be searched at a time.

lancedb.query.AsyncFTSQuery

Bases: AsyncStandardQuery

A query for full text search for LanceDB.

to_query_object

to_query_object() -> Query

Convert the query into a query object

This is currently experimental but can be useful as the query object is pure python and more easily serializable.

select

select(columns: Union[List[str], dict[str, str]]) -> Self

Return only the specified columns.

By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.

As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.

You can also use this method to create new "dynamic" columns based on your existing columns. For example, you may not care about "a" or "b" but instead simply want "a + b". This is often seen in the SELECT clause of an SQL query (e.g. SELECT a+b FROM my_table).

To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.

For example, an SQL query might state SELECT a + b AS combined, c. The equivalent input to this method would be {"combined": "a + b", "c": "c"}.

Columns will always be returned in the order given, even if that order is different than the order used when adding the data.

with_row_id

with_row_id() -> Self

Include the _rowid column in the results.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Include the _rowaddr column in scanner-backed plain query results.

with_fragments

with_fragments(fragments: Any) -> Self

Restrict scanner-backed plain query results to the given Lance fragments.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Restrict scanner-backed plain query results to the given Lance fragment ids.

to_batches

to_batches(*, max_batch_length: Optional[int] = None, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader

Execute the query and return the results as an Apache Arrow RecordBatchReader.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • max_batch_length (Optional[int], default: None ) –

    The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Use async for batch in await query.to_batches(blob_mode="bytes") to stream blob payloads and convert each batch with batch.to_pandas().

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and collect the results into an Apache Arrow Table.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse "bytes".

to_list

to_list(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Execute the query and collect the results into a pandas DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_polars

to_polars(timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and collect the results into a Polars DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

Examples:

>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())

to_pydantic

to_pydantic(model: Type[LanceModel], *, timeout: Optional[timedelta] = None) -> List[LanceModel]

Convert results to a list of pydantic models.

Parameters:

  • model (Type[LanceModel]) –

    The pydantic model to use.

  • timeout (timedelta, default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • list[LanceModel] –

explain_plan

explain_plan(verbose: Optional[bool] = False)

Return the execution plan for this query.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
...     plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
...     print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and display with runtime metrics.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

where

where(predicate: Union[str, Expr]) -> Self

Only return rows matching the given predicate

The predicate can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

Examples:

>>> predicate = "x > 10"
>>> predicate = "y > 0 AND y < 100"
>>> predicate = "x > 5 OR y = 'test'"

Filtering performance can often be improved by creating a scalar index on the filter column(s).

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

limit

limit(limit: int) -> Self

Set the maximum number of results to return.

By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.

offset

offset(offset: int) -> Self

Set the offset for the results.

Parameters:

  • offset (int) –

    The offset to start fetching results from.

order_by

order_by(ordering: Optional[List[ColumnOrdering]]) -> Self

Set the ordering for the results.

Parameters:

  • ordering (Optional[List[ColumnOrdering]]) –

    The ordering to use for the results. If None, then the default ordering will be used.

fast_search() -> Self

Skip searching un-indexed data.

This can make queries faster, but will miss any data that has not been indexed.

Tip

You can add new data into an existing index by calling AsyncTable.optimize.

use_rte

use_rte(enable: bool) -> Self

Control RTE LSM read routing for this query.

By default (unset), a query against a table with an RTE write spec (see AsyncTable.enable_rte) is routed through the LSM scanner so it also returns data written via the merge_insert LSM path that has not yet been compacted into the base table (the active/frozen in-memory memtables and the SSTables), deduplicated by primary key; a table without a spec reads the base table.

Parameters:

  • enable (bool) –

    True forces the LSM scanner and errors if the table has no RTE write spec. False bypasses the RTE and reads the base table only, even when a spec is present.

postfilter

postfilter() -> Self

If this is called then filtering will happen after the search instead of before. By default filtering will be performed before the search. This is how filtering is typically understood to work. This prefilter step does add some additional latency. Creating a scalar index on the filter column(s) can often improve this latency. However, sometimes a filter is too complex or scalar indices cannot be applied to the column. In these cases postfiltering can be used instead of prefiltering to improve latency. Post filtering applies the filter to the results of the search. This means we only run the filter on a much smaller set of data. However, it can cause the query to return fewer than limit results (or even no results) if none of the nearest results match the filter. Post filtering happens during the "refine stage" (described in more detail in @see {@link VectorQuery#refineFactor}). This means that setting a higher refine factor can often help restore some of the results lost by post filtering.

get_query

get_query() -> str

rerank

rerank(reranker: Reranker = RRFReranker()) -> AsyncFTSQuery

nearest_to

nearest_to(query_vector: Union[VEC, Tuple, List[VEC]]) -> AsyncHybridQuery

In addition doing text search on the LanceDB Table, also find the nearest vectors to the given query vector.

This converts the query from a FTS Query to a Hybrid query. Results from the vector search will be combined with results from the FTS query.

This method will attempt to convert the input to the query vector expected by the embedding model. If the input cannot be converted then an error will be thrown.

By default, there is no embedding model, and the input should be something that can be converted to a pyarrow array of floats. This includes lists, numpy arrays, and tuples.

If there is only one vector column (a column whose data type is a fixed size list of floats) then the column does not need to be specified. If there is more than one vector column you must use AsyncVectorQuery.column to specify which column you would like to compare with.

If no index has been created on the vector column then a vector query will perform a distance comparison between the query vector and every vector in the database and then sort the results. This is sometimes called a "flat search"

For small databases, with tens of thousands of vectors or less, this can be reasonably fast. In larger databases you should create a vector index on the column. If there is a vector index then an "approximate" nearest neighbor search (frequently called an ANN search) will be performed. This search is much faster, but the results will be approximate.

The query can be further parameterized using the returned builder. There are various ANN search parameters that will let you fine tune your recall accuracy vs search latency.

Hybrid searches always have a limit. If limit has not been called then a default limit of 10 will be used.

Typically, a single vector is passed in as the query. However, you can also pass in multiple vectors. This can be useful if you want to find the nearest vectors to multiple query vectors. Flat searches share one table scan across the query vectors instead of issuing concurrent full scans. If multiple vectors are passed in then an additional column query_index will be added to the results. This column will contain the index of the query vector that the result is nearest to.

lancedb.query.AsyncHybridQuery

Bases: AsyncStandardQuery, AsyncVectorQueryBase

A query builder that performs hybrid vector and full text search. Results are combined and reranked based on the specified reranker. By default, the results are reranked using the RRFReranker, which uses reciprocal rank fusion score for reranking.

To make the vector and fts results comparable, the scores are normalized. Instead of normalizing scores, the normalize parameter can be set to "rank" in the rerank method to convert the scores to ranks and then normalize them.

column

column(column: str) -> Self

Set the vector column to query

This controls which column is compared to the query vector supplied in the call to AsyncQuery.nearest_to.

This parameter must be specified if the table has more than one column whose data type is a fixed-size-list of floats.

nprobes

nprobes(nprobes: int) -> Self

Set the legacy IVF probe parameter

The number of probes must be greater than 0.

This argument is only used when the vector column has an IVF-based index. If there is no index then this value is ignored.

The IVF stage of IVF PQ divides the input into partitions (clusters) of related values.

The partition whose centroids are closest to the query vector will be exhaustiely searched to find matches. This parameter controls how many partitions will be searched.

Increasing this value will increase the recall of your query but will also increase the latency of your query. If this method is not called, Lance's adaptive probe defaults are used.

For best results we recommend tuning this parameter with a benchmark against your actual data to find the smallest possible value that will still give you the desired recall.

LanceDB retains this as nprobes through LanceDB Embedded and LanceDB Enterprise request construction. Lance sets both probe bounds to this value. Explicit minimum or maximum settings can override their respective bounds.

search_effort

search_effort(search_effort: Optional[float]) -> Self

Set the initial IVF search effort to a finite value in [0, 1].

Efforts 0.1 and 0.25 target empirical mean partition-routing recall of 80% and 90% for L2, cosine and dot through k=100000. The probe budget adapts to the metric and k. Hamming and larger k remain uncalibrated; 100000 is not an API limit. Remote queries also require server support.

None preserves Auto (internally 0.5) and fixed nprobes. Any explicit effort, including 0.5, overrides nprobes in either call order; explicit minimum and maximum bounds remain effective. Zero starts at the caller minimum and one searches all partitions within the maximum. Later probing may continue to obtain enough candidates. This is not a recall guarantee and does not change refinement or HNSW ef. RTE LSM queries do not yet support this option.

Examples:

>>> import asyncio
>>> import lancedb
>>> async def build_query():
...     db = await lancedb.connect_async("memory://")
...     table = await db.create_table(
...         "effort_async", [{"vector": [0.1, 0.2]}]
...     )
...     return table.vector_search([0.1, 0.2]).search_effort(0.75)
>>> query = asyncio.run(build_query())

minimum_nprobes

minimum_nprobes(minimum_nprobes: int) -> Self

Set the minimum number of probes to use.

See nprobes for more details.

These partitions will be searched on every indexed vector query and will increase recall at the expense of latency.

maximum_nprobes

maximum_nprobes(maximum_nprobes: int) -> Self

Set the maximum number of probes to use.

See nprobes for more details.

If this value is greater than minimum_nprobes then the excess partitions will be searched only if we have not found enough results.

This can be useful when there is a narrow filter to allow these queries to spend more time searching and avoid potential false negatives.

If this value is 0 then no limit will be applied and all partitions could be searched if needed to satisfy the limit.

distance_range

distance_range(lower_bound: Optional[float] = None, upper_bound: Optional[float] = None) -> Self

Set the distance range to use.

Only rows with distances within range [lower_bound, upper_bound) will be returned.

Parameters:

  • lower_bound (Optional[float], default: None ) –

    The lower bound of the distance range.

  • upper_bound (Optional[float], default: None ) –

    The upper bound of the distance range.

Returns:

  • AsyncVectorQuery –

    The AsyncVectorQuery object.

ef

ef(ef: int) -> Self

Set the number of candidates to consider during search

This argument is only used when the vector column has an HNSW index. If there is no index then this value is ignored.

Increasing this value will increase the recall of your query but will also increase the latency of your query. The default value is 1.5 * limit. This default is good for many cases but the best value to use will depend on your data and the recall that you need to achieve.

approx_mode

approx_mode(approx_mode: Literal['fast', 'normal', 'accurate']) -> Self

Set the speed / accuracy tradeoff to use for approximate search

This currently affects RQ-quantized indexes, such as IVF_RQ, and prefiltered search on HNSW indexes, where "fast" enables the ACORN traversal. Other index types ignore this setting.

"fast" favors lower latency and may reduce recall, "normal" (the default) uses the index's default balance, and "accurate" favors higher recall and may increase latency.

Examples:

>>> import asyncio
>>> import lancedb
>>> async def build_query():
...     db = await lancedb.connect_async("memory://")
...     table = await db.create_table(
...         "approx_async", [{"vector": [0.1, 0.2]}]
...     )
...     return table.vector_search([0.1, 0.2]).approx_mode("accurate")
>>> query = asyncio.run(build_query())

refine_factor

refine_factor(refine_factor: int) -> Self

A multiplier to control how many additional rows are taken during the refine step

This argument is only used when the vector column has an IVF PQ index. If there is no index then this value is ignored.

An IVF PQ index stores compressed (quantized) values. They query vector is compared against these values and, since they are compressed, the comparison is inaccurate.

This parameter can be used to refine the results. It can improve both improve recall and correct the ordering of the nearest results.

To refine results LanceDb will first perform an ANN search to find the nearest limit * refine_factor results. In other words, if refine_factor is 3 and limit is the default (10) then the first 30 results will be selected. LanceDb then fetches the full, uncompressed, values for these 30 results. The results are then reordered by the true distance and only the nearest 10 are kept.

Note: there is a difference between calling this method with a value of 1 and never calling this method at all. Calling this method with any value will have an impact on your search latency. When you call this method with a refine_factor of 1 then LanceDb still needs to fetch the full, uncompressed, values so that it can potentially reorder the results.

Note: if this method is NOT called then the distances returned in the _distance column will be approximate distances based on the comparison of the quantized query vector and the quantized result vectors. This can be considerably different than the true distance between the query vector and the actual uncompressed vector.

distance_type

distance_type(distance_type: str) -> Self

Set the distance metric to use

When performing a vector search we try and find the "nearest" vectors according to some kind of distance metric. This parameter controls which distance metric to use. See @see {@link IvfPqOptions.distanceType} for more details on the different distance metrics available.

Note: if there is a vector index then the distance type used MUST match the distance type used to train the vector index. If this is not done then the results will be invalid.

By default "l2" is used.

bypass_vector_index

bypass_vector_index() -> Self

If this is called then any vector index is skipped

An exhaustive (flat) search will be performed. The query vector will be compared to every vector in the table. At high scales this can be expensive. However, this is often still useful. For example, skipping the vector index can give you ground truth results which you can use to calculate your recall to select an appropriate value for nprobes.

to_query_object

to_query_object() -> Query

Convert the query into a query object

This is currently experimental but can be useful as the query object is pure python and more easily serializable.

select

select(columns: Union[List[str], dict[str, str]]) -> Self

Return only the specified columns.

By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.

As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.

You can also use this method to create new "dynamic" columns based on your existing columns. For example, you may not care about "a" or "b" but instead simply want "a + b". This is often seen in the SELECT clause of an SQL query (e.g. SELECT a+b FROM my_table).

To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.

For example, an SQL query might state SELECT a + b AS combined, c. The equivalent input to this method would be {"combined": "a + b", "c": "c"}.

Columns will always be returned in the order given, even if that order is different than the order used when adding the data.

with_row_id

with_row_id() -> Self

Include the _rowid column in the results.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Include the _rowaddr column in scanner-backed plain query results.

with_fragments

with_fragments(fragments: Any) -> Self

Restrict scanner-backed plain query results to the given Lance fragments.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Restrict scanner-backed plain query results to the given Lance fragment ids.

to_batches

to_batches(*, max_batch_length: Optional[int] = None, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader

Execute the query and return the results as an Apache Arrow RecordBatchReader.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • max_batch_length (Optional[int], default: None ) –

    The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Use async for batch in await query.to_batches(blob_mode="bytes") to stream blob payloads and convert each batch with batch.to_pandas().

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and collect the results into an Apache Arrow Table.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse "bytes".

to_list

to_list(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Execute the query and collect the results into a pandas DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_polars

to_polars(timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and collect the results into a Polars DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

Examples:

>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())

to_pydantic

to_pydantic(model: Type[LanceModel], *, timeout: Optional[timedelta] = None) -> List[LanceModel]

Convert results to a list of pydantic models.

Parameters:

  • model (Type[LanceModel]) –

    The pydantic model to use.

  • timeout (timedelta, default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • list[LanceModel] –

where

where(predicate: Union[str, Expr]) -> Self

Only return rows matching the given predicate

The predicate can be a SQL string or a type-safe :class:~lancedb.expr.Expr built with :func:~lancedb.expr.col and :func:~lancedb.expr.lit.

Examples:

>>> predicate = "x > 10"
>>> predicate = "y > 0 AND y < 100"
>>> predicate = "x > 5 OR y = 'test'"

Filtering performance can often be improved by creating a scalar index on the filter column(s).

Calling this multiple times combines the filters with a logical AND rather than replacing the previous filter.

limit

limit(limit: int) -> Self

Set the maximum number of results to return.

By default, a plain search has no limit. If this method is not called then every valid row from the table will be returned.

offset

offset(offset: int) -> Self

Set the offset for the results.

Parameters:

  • offset (int) –

    The offset to start fetching results from.

order_by

order_by(ordering: Optional[List[ColumnOrdering]]) -> Self

Set the ordering for the results.

Parameters:

  • ordering (Optional[List[ColumnOrdering]]) –

    The ordering to use for the results. If None, then the default ordering will be used.

fast_search() -> Self

Skip searching un-indexed data.

This can make queries faster, but will miss any data that has not been indexed.

Tip

You can add new data into an existing index by calling AsyncTable.optimize.

use_rte

use_rte(enable: bool) -> Self

Control RTE LSM read routing for this query.

By default (unset), a query against a table with an RTE write spec (see AsyncTable.enable_rte) is routed through the LSM scanner so it also returns data written via the merge_insert LSM path that has not yet been compacted into the base table (the active/frozen in-memory memtables and the SSTables), deduplicated by primary key; a table without a spec reads the base table.

Parameters:

  • enable (bool) –

    True forces the LSM scanner and errors if the table has no RTE write spec. False bypasses the RTE and reads the base table only, even when a spec is present.

postfilter

postfilter() -> Self

If this is called then filtering will happen after the search instead of before. By default filtering will be performed before the search. This is how filtering is typically understood to work. This prefilter step does add some additional latency. Creating a scalar index on the filter column(s) can often improve this latency. However, sometimes a filter is too complex or scalar indices cannot be applied to the column. In these cases postfiltering can be used instead of prefiltering to improve latency. Post filtering applies the filter to the results of the search. This means we only run the filter on a much smaller set of data. However, it can cause the query to return fewer than limit results (or even no results) if none of the nearest results match the filter. Post filtering happens during the "refine stage" (described in more detail in @see {@link VectorQuery#refineFactor}). This means that setting a higher refine factor can often help restore some of the results lost by post filtering.

rerank

rerank(reranker: Reranker = RRFReranker(), normalize: str = 'score') -> AsyncHybridQuery

Rerank the hybrid search results using the specified reranker. The reranker must be an instance of Reranker class.

Parameters:

  • reranker (Reranker, default: RRFReranker() ) –

    The reranker to use. Must be an instance of Reranker class.

  • normalize (str, default: 'score' ) –

    The method to normalize the scores. Can be "rank" or "score". If "rank", the scores are converted to ranks and then normalized. If "score", the scores are normalized directly.

Returns:

  • AsyncHybridQuery –

    The AsyncHybridQuery object.

explain_plan

explain_plan(verbose: Optional[bool] = False)

Return the execution plan for this query.

The output includes both the vector and FTS search plans.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> from lancedb.index import FTS
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", [{"vector": [99.0, 99.0], "text": "hello world"}])
...     await table.create_index("text", config=FTS(with_position=False))
...     plan = await table.query().nearest_to([1.0, 2.0]).nearest_to_text("hello").explain_plan(True)
...     print(plan)
>>> asyncio.run(doctest_example())
RRFReranker(K=60)
    ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance, _rowid@1 as _rowid]
      LanceRead: uri=..., projection=[text], source=stream(_rowid)
        GlobalLimitExec: skip=0, fetch=10
          FilterExec: _distance@2 IS NOT NULL
            SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
              KNNVectorDistance: metric=l2
                LanceRead: uri=..., projection=[vector], num_fragments=..., ...
    ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score, _rowid@0 as _rowid]
      LanceRead: uri=..., projection=[vector, text], source=stream(_rowid)
        GlobalLimitExec: skip=0, fetch=10
          MatchQuery: column=text, query=[hello]

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and return the physical execution plan with runtime metrics.

This runs both the vector and FTS (full-text search) queries and returns detailed metrics for each step of execution—such as rows processed, elapsed time, I/O stats, and more. It’s useful for debugging and performance analysis.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

lancedb.query.AsyncTakeQuery

Bases: AsyncQueryBase

Builder for parameterizing and executing take queries.

to_query_object

to_query_object() -> Query

Convert the query into a query object

This is currently experimental but can be useful as the query object is pure python and more easily serializable.

select

select(columns: Union[List[str], dict[str, str]]) -> Self

Return only the specified columns.

By default a query will return all columns from the table. However, this can have a very significant impact on latency. LanceDb stores data in a columnar fashion. This means we can finely tune our I/O to select exactly the columns we need.

As a best practice you should always limit queries to the columns that you need. If you pass in a list of column names then only those columns will be returned.

You can also use this method to create new "dynamic" columns based on your existing columns. For example, you may not care about "a" or "b" but instead simply want "a + b". This is often seen in the SELECT clause of an SQL query (e.g. SELECT a+b FROM my_table).

To create dynamic columns you can pass in a dict[str, str]. A column will be returned for each entry in the map. The key provides the name of the column. The value is an SQL string used to specify how the column is calculated.

For example, an SQL query might state SELECT a + b AS combined, c. The equivalent input to this method would be {"combined": "a + b", "c": "c"}.

Columns will always be returned in the order given, even if that order is different than the order used when adding the data.

with_row_id

with_row_id() -> Self

Include the _rowid column in the results.

with_row_address

with_row_address(with_row_address: bool = True) -> Self

Include the _rowaddr column in scanner-backed plain query results.

with_fragments

with_fragments(fragments: Any) -> Self

Restrict scanner-backed plain query results to the given Lance fragments.

fragment_ids

fragment_ids(fragment_ids: List[int]) -> Self

Restrict scanner-backed plain query results to the given Lance fragment ids.

to_batches

to_batches(*, max_batch_length: Optional[int] = None, timeout: Optional[timedelta] = None, blob_mode: BlobMode = 'lazy') -> AsyncRecordBatchReader

Execute the query and return the results as an Apache Arrow RecordBatchReader.

A remote table decodes batches as they arrive when the server streams query results. If the response stops partway, reading raises ResponseInterruptedError after the batches already returned, which are valid but incomplete.

Parameters:

  • max_batch_length (Optional[int], default: None ) –

    The maximum number of selected records in a single RecordBatch object. If not specified, a default batch length is used. It is possible for batches to be smaller than the provided length if the underlying data is stored in smaller chunks.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised. The timeout also covers reading the batches, so consuming them slowly can exceed it.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" keeps blob v2 descriptors and auto row ids for manual fetches. "descriptions" omits auto row ids. "bytes" fetches supported row-addressable blob payloads one batch at a time, without requiring pylance. Bytes mode has the same restrictions as :meth:to_arrow.

Examples:

Use async for batch in await query.to_batches(blob_mode="bytes") to stream blob payloads and convert each batch with batch.to_pandas().

output_schema

output_schema() -> Schema

Return the output schema for the query

This does not execute the query.

to_arrow

to_arrow(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> Table

Execute the query and collect the results into an Apache Arrow Table.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    "lazy" returns blob v2 descriptors that carry the row id fetch_blobs reads. "descriptions" returns them without it. "bytes" returns payload bytes for supported row-addressable blob columns and raises for blobs inside lists. LanceDB Enterprise and namespace query pushdown refuse "bytes".

to_list

to_list(timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy') -> List[dict]

Execute the query and return the results as a list of dictionaries.

Each list entry is a dictionary with the selected column names as keys, or all table columns if select is not called. The vector and the "_distance" fields are returned whether or not they're explicitly selected.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Passed through to :meth:to_arrow.

to_pandas

to_pandas(flatten: Optional[Union[int, bool]] = None, timeout: Optional[timedelta] = None, *, blob_mode: BlobMode = 'lazy', **kwargs) -> 'pd.DataFrame'

Execute the query and collect the results into a pandas DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to pandas separately.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = batch.to_pandas()
>>> asyncio.run(doctest_example())

Parameters:

  • flatten (Optional[Union[int, bool]], default: None ) –

    If flatten is True, flatten all nested columns. If flatten is an integer, flatten the nested columns up to the specified depth. If unspecified, do not flatten the nested columns.

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

  • blob_mode (BlobMode, default: 'lazy' ) –

    Controls how blob columns are returned. Local blob v2 queries support "bytes" and "descriptions" without pylance. Eligible local plain scans also support "bytes" for row-addressable legacy blob columns without pylance. Other local blob v2 queries return descriptors in "lazy" and "descriptions" modes, or payloads in "bytes" mode, through :meth:to_arrow. LanceDB Enterprise blob queries support only "descriptions" until a stable table snapshot can bind the query and blob fetches.

  • **kwargs –

    Forwarded to pyarrow.Table.to_pandas after query execution and optional flattening.

to_polars

to_polars(timeout: Optional[timedelta] = None) -> 'pl.DataFrame'

Execute the query and collect the results into a Polars DataFrame.

This method will collect all results into memory before returning. If you expect a large number of results, you may want to use to_batches and convert each batch to polars separately.

Parameters:

  • timeout (Optional[timedelta], default: None ) –

    The maximum time to wait for the query to complete. If not specified, no timeout is applied. If the query does not complete within the specified time, an error will be raised.

Examples:

>>> import asyncio
>>> import polars as pl
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", data=[{"a": 1, "b": 2}])
...     async for batch in await table.query().to_batches():
...         batch_df = pl.from_arrow(batch)
>>> asyncio.run(doctest_example())

to_pydantic

to_pydantic(model: Type[LanceModel], *, timeout: Optional[timedelta] = None) -> List[LanceModel]

Convert results to a list of pydantic models.

Parameters:

  • model (Type[LanceModel]) –

    The pydantic model to use.

  • timeout (timedelta, default: None ) –

    The maximum time to wait for the query to complete. If None, wait indefinitely.

Returns:

  • list[LanceModel] –

explain_plan

explain_plan(verbose: Optional[bool] = False)

Return the execution plan for this query.

Examples:

>>> import asyncio
>>> from lancedb import connect_async
>>> async def doctest_example():
...     conn = await connect_async("./.lancedb")
...     table = await conn.create_table("my_table", [{"vector": [99.0, 99.0]}])
...     plan = await table.query().nearest_to([1.0, 2.0]).explain_plan(True)
...     print(plan)
>>> asyncio.run(doctest_example())
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
  GlobalLimitExec: skip=0, fetch=10
    FilterExec: _distance@2 IS NOT NULL
      SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
        KNNVectorDistance: metric=l2
          LanceRead: uri=..., projection=[vector], num_fragments=..., ...

Parameters:

  • verbose (bool, default: False ) –

    Use a verbose output format.

Returns:

  • plan ( str ) –

analyze_plan

analyze_plan(distributed_metrics: AnalyzePlanDistributedMetrics = 'aggregate') -> str

Execute the query and display with runtime metrics.

Parameters:

  • distributed_metrics (Literal['aggregate', 'per_worker', 'full'], default: 'aggregate' ) –

    Defaults to "aggregate". How distributed worker metrics are displayed for remote query plans.

Returns:

  • plan ( str ) –

use_rte

use_rte(enable: bool) -> 'AsyncTakeQuery'

Control RTE LSM read routing for this take query.

False bypasses the RTE and reads the base table only — the escape hatch, since take-by-row-id/offset is not supported on the LSM scanner.

Storage integrations

lancedb.storage_integration.StorageIntegrationInfo

Registered cloud configuration and setup instructions.

configuration_instructions is a single sentence of setup guidance for display; its wording may change. Use the structured identity and policy fields instead of parsing that text. Creating this object does not verify AWS access or configure a storage operation to use it. Static-key authentication stores only Secret references. No field contains AWS credentials.

name

name: str

database

database: str

namespace_path

namespace_path: list[str]

storage_provider

storage_provider: Literal['S3', 'AZURE', 'GCP', 'HUGGINGFACE']

storage_prefix

storage_prefix: str

aws_region

aws_region: Optional[str]

aws_account_id

aws_account_id: str

aws_iam_principal_arn

aws_iam_principal_arn: str

aws_external_id

aws_external_id: str

created_at_millis

created_at_millis: int

access_verified

access_verified: bool

trust_policy

trust_policy: Optional[Dict[str, Any]]

permissions_policy

permissions_policy: Optional[Dict[str, Any]]

configuration_instructions

configuration_instructions: str

aws_role_arn

aws_role_arn: Optional[str] = None

aws_access_key_id_secret

aws_access_key_id_secret: Optional[SecretReference] = None

aws_secret_access_key_secret

aws_secret_access_key_secret: Optional[SecretReference] = None

azure_storage_account

azure_storage_account: Optional[str] = None

azure_tenant_id_secret

azure_tenant_id_secret: Optional[SecretReference] = None

azure_client_id_secret

azure_client_id_secret: Optional[SecretReference] = None

azure_client_secret_secret

azure_client_secret_secret: Optional[SecretReference] = None

gcp_credentials_json_secret

gcp_credentials_json_secret: Optional[SecretReference] = None

huggingface_token_secret

huggingface_token_secret: Optional[SecretReference] = None

lancedb.storage_integration.SecretReference

A credential Secret, optionally qualified by database and namespace.

name

name: str

database

database: Optional[str] = None

namespace_path

namespace_path: tuple[str, ...] = ()

Integrations belong to a database and follow the same namespace-path naming rules as Secrets, with their own keyspace. Use a regular database under the cluster root for storage configuration needed before opening an external database. default is a convention; no database with that name is required or created automatically. The system database is read-only. Client methods execute SQL through the existing SQL transport.

catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
integration = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
)
print(integration.configuration_instructions)
integration = db.describe_storage_integration("raw")
for name in db.list_storage_integrations():
    print(name)
db.drop_storage_integration("raw")

Static credentials use two existing, single-value Secrets instead of a role:

from lancedb.storage_integration import SecretReference

integration = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_access_key_id_secret=SecretReference("s3-key-id", database="default"),
    aws_secret_access_key_secret=SecretReference("s3-secret-key", database="default"),
)

Pass namespace_path on the object APIs and Secret references to select a namespace. A string Secret reference selects the root namespace of the integration's database. Cross-database references require USAGE on the source database, namespace, and Secret. Role and static-credential modes are mutually exclusive. Credential values are never included in integration metadata.

configuration_instructions is one sentence of display guidance whose wording may change. Use the structured identity and policy fields for programmatic access. Registration does not verify AWS access. AsyncConnection provides the same methods; await create, describe, and drop, and use async for for listing.

Azure storage integrations

Create three Secrets containing the Azure tenant ID, client ID and client secret in an ordinary database. Reference their names when registering Azure Blob access:

integration = db.create_storage_integration(
    "customer_blobs",
    "az://customer-data/raw/",
    storage_provider="AZURE",
    azure_storage_account="customeraccount",
    azure_tenant_id_secret="azure_tenant",
    azure_client_id_secret="azure_client",
    azure_client_secret_secret="azure_credential",
)
print(integration.configuration_instructions)

Use SecretReference to qualify another database or namespace. Sync and async connections use the same options. Secret values are never part of integration metadata. Azure supports service principal client credentials; managed identity, account key and SAS authentication are not integration modes.

The equivalent SQL is:

CREATE STORAGE INTEGRATION customer_blobs
    STORAGE_PROVIDER = 'AZURE'
    STORAGE_PREFIX = 'az://customer-data/raw/'
    STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
    STORAGE_AZURE_TENANT_ID_SECRET = public.azure_tenant
    STORAGE_AZURE_CLIENT_ID_SECRET = public.azure_client
    STORAGE_AZURE_CLIENT_SECRET_SECRET = public.azure_credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;

Registration returns setup instructions and leaves access_verified false. It stores an account-qualified prefix such as az://customer-data@customeraccount.blob.core.windows.net/raw/.

GCP storage integrations

Create one Secret containing the complete GCP application credentials JSON file content, then reference its name when registering Google Cloud Storage access:

from pathlib import Path

db.create_secret("gcp_credentials", Path("credentials.json").read_text())
integration = db.create_storage_integration(
    "customer_gcs",
    "gs://customer-data/raw/",
    storage_provider="GCP",
    gcp_credentials_json_secret="gcp_credentials",
)
print(integration.configuration_instructions)

The equivalent SQL references the existing Secret:

CREATE STORAGE INTEGRATION customer_gcs
    STORAGE_PROVIDER = 'GCP'
    STORAGE_PREFIX = 'gs://customer-data/raw/'
    GCP_CREDENTIALS_JSON_SECRET = public.gcp_credentials;
DESCRIBE STORAGE INTEGRATION customer_gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_gcs;

gcp_credentials_json_secret names a Secret; pass the file content to create_secret, not to create_storage_integration. Use SecretReference to qualify another database or namespace. Sync and async connections accept the same options. The service interprets the stored JSON when accessing storage; registration does not validate its credentials format or verify access, and leaves access_verified false. Integration metadata contains the Secret reference, never its JSON content. GCP options cannot be combined with AWS or Azure options.

Hugging Face storage integrations

Store a Hugging Face token as one Secret, then register the repository prefix with the existing integration methods. Only the Secret name is part of the integration configuration:

import os

from lancedb.storage_integration import SecretReference

db.create_secret("hf_token", os.environ["HF_TOKEN"])
info = db.create_storage_integration(
    "private_hub",
    "hf://datasets/acme/private/raw/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret=SecretReference("hf_token"),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")

The equivalent SQL references the existing Secret:

CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
    HUGGINGFACE_TOKEN_SECRET = public.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;

Omit the token Secret for anonymous access at Hugging Face's default anonymous rate limits. A tokenless integration does not use a cluster's Hugging Face token:

info = db.create_storage_integration(
    "public_hub", "hf://datasets/acme/public/raw/",
    storage_provider="HUGGINGFACE",
)
CREATE STORAGE INTEGRATION public_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';

When provided, the Secret must contain the token itself, with repository read permission. Cross-database and namespace references use the same SecretReference rules as other integrations. Sync and async APIs accept the same options. Registration returns one sentence of instructions and leaves access_verified false; it does not contact Hugging Face. Metadata contains the Secret reference, never its value. Hugging Face options cannot be combined with AWS, Azure or GCP options.

Repository prefixes use hf://datasets/<owner>/<repo>[/<path>] or hf://models/<owner>/<repo>[/<path>]. An optional @revision follows the repository name; omitting it allows matching reads of any revision. Prefixes must be concrete paths without wildcards or traversal. This integration does not make Hugging Face a writable database location for ATTACH DATABASE. Dataset SQL sources select a matching integration automatically, with no integration argument in the query.