Extension Arrays

Lance provides extensions for Arrow arrays and Pandas Series to represent data types for machine learning applications.

BFloat16

BFloat16 is a 16-bit floating point number that is designed for machine learning use cases. Intuitively, it only has 2-3 digits of precision, but it has the same range as a 32-bit float: ~1e-38 to ~1e38. By comparison, a 16-bit float has a range of ~5.96e-8 to 65504.

Lance provides an Arrow extension array (lance.arrow.BFloat16Array) and a Pandas extension array (lance.pandas.BFloat16Dtype) for BFloat16. These are compatible with the ml_dtypes bfloat16 NumPy extension array.

If you are using Pandas, you can use the lance.bfloat16 dtype string to create the array:

import pandas as pd
import lance.arrow

series = pd.Series([1.1, 2.1, 3.4], dtype="lance.bfloat16")
series
0    1.1015625
1      2.09375
2      3.40625
dtype: lance.bfloat16

To create an an arrow array, use the lance.arrow.bfloat16_array() function:

from lance.arrow import bfloat16_array

array = bfloat16_array([1.1, 2.1, 3.4])
array
<lance.arrow.BFloat16Array object at 0x.+>
[1.1015625, 2.09375, 3.40625]

Finally, if you have a pre-existing NumPy array, you can convert it into either:

import numpy as np
from ml_dtypes import bfloat16
from lance.arrow import PandasBFloat16Array, BFloat16Array

np_array = np.array([1.1, 2.1, 3.4], dtype=bfloat16)
PandasBFloat16Array.from_numpy(np_array)
BFloat16Array.from_numpy(np_array)
<PandasBFloat16Array>
[1.1015625, 2.09375, 3.40625]
Length: 3, dtype: lance.bfloat16
<lance.arrow.BFloat16Array object at 0x.+>
[1.1015625, 2.09375, 3.40625]

When reading, these can be converted back to to the NumPy bfloat16 dtype using each array class’s to_numpy method.

ImageURI

lance.arrow.ImageURIArray is an array that stores the URI location of images in some other storage system. For example, file:///path/to/image.png for a local filesystem or s3://bucket/path/image.jpeg for an image on AWS S3. Use this array type when you want to lazily load images from an existing storage medium.

It can be created by calling lance.arrow.ImageURIArray.from_uris() with a list of URIs represented by either pyarrow.StringArray or an iterable that yields strings. Note that the URIs are not strongly validated and images are not read into memory automatically.

from lance.arrow import ImageURIArray

ImageURIArray.from_uris([
    "/tmp/image1.jpg",
    "file:///tmp/image2.jpg",
    "s3://example/image3.jpg"
])
<lance.arrow.ImageURIArray object at 0x.+>
['/tmp/image1.jpg', 'file:///tmp/image2.jpg', 's3://example/image2.jpg']

lance.arrow.ImageURIArray.read_uris() will read images into memory and return them as a new lance.arrow.EncodedImageArray object.

from lance.arrow import ImageURIArray

relative_path = "images/1.png"
uris = [os.path.join(os.path.dirname(__file__), relative_path)]
ImageURIArray.from_uris(uris).read_uris()
<lance.arrow.EncodedImageArray object at 0x...>
[b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00...']

EncodedImage

lance.arrow.EncodedImageArray is an array that stores jpeg and png images in their encoded and compressed representation as they would appear written on disk. Use this array when you want to manipulate images in their compressed format such as when you’re reading them from disk or embedding them into HTML.

It can be created by calling lance.arrow.ImageURIArray.read_uris() on an existing lance.arrow.ImageURIArray. This will read the referenced images into memory. It can also be created by calling lance.arrow.ImageArray.from_array() and passing it an array of encoded images already read into pyarrow.BinaryArray or by calling lance.arrow.ImageTensorArray.to_encoded().

A lance.arrow.EncodedImageArray.image_to_tensor() method is provided to decode encoded images and return them as lance.arrow.FixedShapeImageTensorArray, from which they can be converted to numpy arrays or TensorFlow tensors. For decoding images, it will first attempt to use a decoder provided via the optional function parameter. If decoder is not provided it will attempt to use Pillow and tensorflow in that order. If neither library or custom decoder is available an exception will be raised.

from lance.arrow import ImageURIArray

uris = [os.path.join(os.path.dirname(__file__), "images/1.png")]
encoded_images = ImageURIArray.from_uris(uris).read_uris()
print(encoded_images.image_to_tensor())

def tensorflow_decoder(images):
    import tensorflow as tf
    import numpy as np

    return np.stack(tf.io.decode_png(img.as_py(), channels=3) for img in images.storage)

print(encoded_images.image_to_tensor(tensorflow_decoder))
<lance.arrow.FixedShapeImageTensorArray object at 0x...>
[[42, 42, 42, 255]]
<lance.arrow.FixedShapeImageTensorArray object at 0x...>
[[42, 42, 42, 255]]

FixedShapeImageTensor

lance.arrow.FixedShapeImageTensorArray is an array that stores images as tensors where each individual pixel is represented as a numeric value. Typically images are stored as 3 dimensional tensors shaped (height, width, channels). In color images each pixel is represented by three values (channels) as per RGB color model. Images from this array can be read out as numpy arrays individually or stacked together into a single 4 dimensional numpy array shaped (batch_size, height, width, channels).

It can be created by calling lance.arrow.EncodedImageArray.image_to_tensor() on a previously existing lance.arrow.EncodedImageArray. This will decode encoded images and return them as a lance.arrow.FixedShapeImageTensorArray. It can also be created by calling lance.arrow.ImageArray.from_array() and passing in a pyarrow.FixedShapeTensorArray.

It can be encoded into to lance.arrow.EncodedImageArray by calling lance.arrow.FixedShapeImageTensorArray.to_encoded() and passing custom encoder If encoder is not provided it will attempt to use tensorflow and Pillow in that order. Default encoders will encode to PNG. If neither library is available it will raise an exception.

from lance.arrow import ImageURIArray

def jpeg_encoder(images):
    import tensorflow as tf

    encoded_images = (
        tf.io.encode_jpeg(x).numpy() for x in tf.convert_to_tensor(images)
    )
    return pa.array(encoded_images, type=pa.binary())

uris = [os.path.join(os.path.dirname(__file__), "images/1.png")]
tensor_images = ImageURIArray.from_uris(uris).read_uris().image_to_tensor()
print(tensor_images.to_encoded())
print(tensor_images.to_encoded(jpeg_encoder))
<lance.arrow.EncodedImageArray object at 0x...>
[b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00...']
<lance.arrow.EncodedImageArray object at 0x00007f8d90b91b40>
[b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x01\x01...']