Skip to content

Development Guide

Lance Java SDK Dependency

This package is dependent on the Lance Java SDK and Lance Namespace Java Modules. You need to build those repositories locally first before building this repository. If your have changes affecting those repositories, the PR in lancedb/lance-spark will only pass CI after the PRs in lancedb/lance and lance/lance-namespace are merged.

Build Commands

This connector is built using Maven. To build everything:

./mvnw clean install

To build everything without running tests:

./mvnw clean install -DskipTests

Multi-Version Support

We offer the following build profiles for you to switch among different build versions:

  • scala-2.12
  • scala-2.13
  • spark-3.4
  • spark-3.5

For example, to use Scala 2.13:

./mvnw clean install -Pscala-2.13

To build a specific version like Spark 3.4:

./mvnw clean install -Pspark-3.4

To build only Spark 3.4:

./mvnw clean install -Pspark-3.4 -pl lance-spark-3.4 -am

Use the shade-jar profile to create the jar with all dependencies for Spark 3.4:

./mvnw clean install -Pspark-3.4 -Pshade-jar -pl lance-spark-3.4 -am

Styling Guide

We use checkstyle and spotless to lint the code.

To verify checkstyle:

./mvnw checkstyle:check

To verify spotless:

./mvnw spotless:check

To apply spotless changes:

./mvnw spotless:apply

Documentation Website

Setup

The documentation website is built using mkdocs-material. The easiest way to setup is to create a Python virtual environment and install the necessary dependencies:

python3 -m venv .env
source .env/bin/activate
pip install mkdocs-material
pip install mkdocs-awesome-pages-plugin

Then you can start the server at http://localhost:8000/lance-spark by:

source .env/bin/activate
mkdocs serve

Contents

In general, we push most of the contents in the website as the single source of truth. The welcome page is the same as the README of the GitHub repository. If you edit one of them, please make sure to update the other document.