Manifest

geneva.manifest.mgr.GenevaManifest

A Geneva Manifest represents the files and dependencies used in the execution environment.

name

name: str = field()

version

version: Optional[str] = field(default=None)

pip

pip: list[str] = field(default=[])

py_modules

py_modules: list[str] = field(default=[])

head_image

head_image: Optional[str] = field(default=None)

worker_image

worker_image: Optional[str] = field(default=None)

skip_site_packages

skip_site_packages: bool = field(default=False)

delete_local_zips

delete_local_zips: bool = field(default=False)

local_zip_output_dir

local_zip_output_dir: Optional[str] = field(default=None)

zips

zips: list[list[str]] = field(default=[[]])

checksum

checksum: Optional[str] = field(default=None)

created_at

created_at: datetime = field(
    factory=lambda: now(utc),
    metadata={"pa_type": timestamp("us", tz="UTC")},
)

created_by

created_by: str = field(factory=current_user)

compute_checksum

compute_checksum() -> str

Generate a stable checksum of the manifest, ignoring the checksum field. The zip file names include the checksum of the contents so this hash is comprehensive.

as_dict

as_dict() -> dict

geneva.manifest.builder.GenevaManifestBuilder

Fluent builder for GenevaManifest. name is required, all optional fields will use defaults. Manifests can be saved using db.define_manifest() and loaded using db.context()

Example usage:

import geneva
m = GenevaManifestBuilder
    .create("my-manifest")
    .pip(["numpy", "pandas"])
    .py_modules(["mymodule"])
    .head_image("my-custom-image:latest")
    .skip_site_packages(True)
    .build()

conn = geneva.connect("s3://my-bucket/my-db")
conn.define_manifest("my-manifest", m)
with conn.context(cluster="my-cluster", manifest="my-manifest"):
    conn.open_table("my-table").backfill("my-column")

name

name(name: str) -> GenevaManifestBuilder

Set the manifest name.

version

version(version: str) -> GenevaManifestBuilder

Set the manifest version.

pip

pip(packages: list[str]) -> GenevaManifestBuilder

Set the runtime pip packages list. See Ray's RuntimeEnv doc for more details.

add_pip

add_pip(package: str) -> GenevaManifestBuilder

Add a single pip package to the runtime environment. See Ray's RuntimeEnv doc for more details.

py_modules

py_modules(modules: list[str]) -> GenevaManifestBuilder

Set the Python modules for the runtime environment. See Ray's RuntimeEnv doc for more details.

add_py_module

add_py_module(module: str) -> GenevaManifestBuilder

Add a single Python module to the runtime environment. See Ray's RuntimeEnv doc for more details.

head_image

head_image(head_image: str) -> GenevaManifestBuilder

Set the container image for Ray head. If set, this will take priority over the head image specified in the cluster definition.

worker_image

worker_image(worker_image: str) -> GenevaManifestBuilder

Set the container image for Ray workers. If set, this will take priority over the worker image specified in the cluster definition.

default_head_image

default_head_image() -> GenevaManifestBuilder

Set the container image for Ray head to the default for the current platform

default_worker_image

default_worker_image() -> GenevaManifestBuilder

Set the container image for Ray workers to the default for the current platform.

skip_site_packages

skip_site_packages(
    skip: bool = True,
) -> GenevaManifestBuilder

Set whether to skip site packages during packaging.

delete_local_zips

delete_local_zips(
    delete: bool = True,
) -> GenevaManifestBuilder

Set whether to delete local zip files after upload.

local_zip_output_dir

local_zip_output_dir(
    output_dir: str,
) -> GenevaManifestBuilder

Set the local directory for zip file output.

build

build() -> GenevaManifest

Build the GenevaManifest with the configured settings.

create

create(name: str) -> GenevaManifestBuilder

Create a new builder with the given manifest name.