Skip to content

Lance Directory Namespace

Lance directory namespace is a lightweight and simple 1-level Lance namespace that only contains a list of tables. People can easily get started with creating and using Lance tables directly on top of any local or remote storage system with a Lance directory namespace.

A directory namespace maps to a directory on storage, we call such directory a namespace directory. A Lance table corresponds to a subdirectory in the namespace directory. We call such a subdirectories table directory. Consider the following example namespace directory layout:

.
└── /my/dir1/
    ├── table1/
    │   ├── data/
    │   │   ├── 0aa36d91-8293-406b-958c-faf9e7547938.lance
    │   │   └── ed7af55d-b064-4442-bcb5-47b524e98d0e.lance
    │   ├── _versions/
    │   │   └── 9223372036854775707.manifest
    │   ├── _indices/
    │   │   └── 85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/
    │   │       └── index.idx
    │   └── _deletions/
    │       └── 75c69434-cde5-4c80-9fe1-e79a6d952fbf.bin
    ├── table2
    └── table3

This describes a Lance directory namespace with the namespace directory at /my/dir1/. It contains tables table1, table2, table3 sitting at table directories /my/dirs/table1, /my/dirs/table2, /my/dirs/table3 respectively.

Directory Path

There are 3 ways to specify a directory path:

  1. URI: a URI that follows the RFC 3986 specification, e.g. s3://mu-bucket/prefix.
  2. Absolute POSIX storage path: an absolute file path in a POSIX standard storage, e.g. /my/dir.
  3. Relative POSIX storage path: a relative file path in a POSIX standard storage, e.g. my/dir2, ./my/dir3. The absolute path of the directory should be based on the current directory of the running process.

Table Existence

A table exists in a Lance directory namespace if a table directory of the specific name exists. This is true even if the directory is empty or the contents in the directory does not follow the Lance table format spec. For such cases, an operation that lists all tables in the directory should show the specific table, and an operation that checks if a table exists should return true. However, an operation that loads the Lance table metadata should fail with error indicating the content in the folder is not compliant with the Lance table format spec.