Manifest
A Geneva Manifest represents the files and dependencies used in the execution environment.
created_at: datetime = field(
factory=lambda: now(utc),
metadata={"pa_type": timestamp("us", tz="UTC")},
)
Generate a stable checksum of the manifest, ignoring the checksum field. The zip file names include the checksum of the contents so this hash is comprehensive.
Fluent builder for GenevaManifest. name is required, all optional
fields will use defaults. Manifests can be saved using db.define_manifest() and
loaded using db.context()
Example usage:
import geneva
m = GenevaManifestBuilder
.create("my-manifest")
.pip(["numpy", "pandas"])
.py_modules(["mymodule"])
.head_image("my-custom-image:latest")
.build()
conn = geneva.connect("s3://my-bucket/my-db")
conn.define_manifest("my-manifest", m)
with conn.context(cluster="my-cluster", manifest="my-manifest"):
conn.open_table("my-table").backfill("my-column")
pip(packages: list[str]) -> GenevaManifestBuilder
Set the runtime pip packages list. Cannot be used with .requirements_path(). Passed through to Ray runtime_env as the "pip" parameter. See Ray's RuntimeEnv doc for more details.
requirements_path(path: str) -> GenevaManifestBuilder
Set the path to a requirements.txt file for the runtime environment. Cannot be used with .pip(). Passed through to Ray runtime_env as the "pip" parameter. See Ray's RuntimeEnv doc for more details.
add_pip(package: str) -> GenevaManifestBuilder
Add a single pip package to the runtime environment. Not usable if .requirements_path() is set. See Ray's RuntimeEnv doc for more details.
conda(
dependencies: dict[str, Any],
) -> GenevaManifestBuilder
Set the conda dependencies for the runtime environment. Cannot be used with .conda_environment_path(). Passed through to Ray runtime_env as the "conda" parameter. See Ray's RuntimeEnv doc for more details.
conda_environment_path(path: str) -> GenevaManifestBuilder
Set the path to a conda environment.yml file for the runtime environment. Cannot be used with .conda(). Passed through to Ray runtime_env as the "conda" parameter. See Ray's RuntimeEnv doc for more details.
py_modules(modules: list[str]) -> GenevaManifestBuilder
Set the Python modules for the runtime environment. See Ray's RuntimeEnv doc for more details.
add_py_module(module: str) -> GenevaManifestBuilder
Add a single Python module to the runtime environment. See Ray's RuntimeEnv doc for more details.
head_image(head_image: str) -> GenevaManifestBuilder
Set the container image for Ray head. If set, this will take priority over the head image specified in the cluster definition.
worker_image(worker_image: str) -> GenevaManifestBuilder
Set the container image for Ray workers. If set, this will take priority over the worker image specified in the cluster definition.
default_head_image() -> GenevaManifestBuilder
Set the container image for Ray head to the default for the current platform
default_worker_image() -> GenevaManifestBuilder
Set the container image for Ray workers to the default for the current platform.
upload_site_packages(
upload: bool = True,
) -> GenevaManifestBuilder
Set whether to upload site packages during packaging. True: upload all local python site packages to Ray workers. False: do not upload site packages (dependencies can be uploaded separately, e.g. via pip(), or by baking into the container image).
upload_site_packages is False by default, but calling builder.upload_site_packages() will set it to True.
delete_local_zips(
delete: bool = True,
) -> GenevaManifestBuilder
Set whether to delete local zip files after upload.
local_zip_output_dir(
output_dir: str,
) -> GenevaManifestBuilder
Set the local directory for zip file output.
create(name: str) -> GenevaManifestBuilder
Create a new builder with the given manifest name.