Multimodal Functions

VERA 4.6 adds built-in functions to support image and PDF preprocessing ahead of Vision Language Model (VLM) inference, used together with functions such as ml_predict, ai_extract, and ai_classify.

How Multimodal Data Reaches Flink

VERA 4.6 supports images and PDFs only. Multimodal data arrives in Flink through two paths.

Base64 inline (images under 5MB)

An edge device or camera encodes a frame, base64-encodes it, and posts it to Kafka or a message queue. Flink consumes the message, decodes it, and passes it to the AI function directly.

Trade-off: single-hop, with no object storage dependency, but the base64 encoding adds roughly 33% to the payload size.

S3 upload with event notification (large files)

An edge device uploads the file (for example a large image or a multi-page PDF) to S3. S3 triggers an SQS or SNS event notification carrying the bucket, object key, size, and MIME type. Flink consumes that notification, constructs the S3 URI, and calls FETCH_CONTENT to retrieve the file before passing it to the AI function.

Trade-off: handles large files, at the cost of a second network hop.

Content and image preprocessing functions

These functions preprocess files before AI inference. They're Ververica's implementation of the equivalent Alibaba Cloud Realtime Compute for Apache Flink functions.

FETCH_CONTENT

Downloads a file from a URI and returns its content as a byte array.

Syntax

1VARBINARY FETCH_CONTENT(VARCHAR uri)
2VARBINARY FETCH_CONTENT(VARCHAR uri, INTEGER httpRetryCount)

Parameters

Returns: VARBINARY — the file content. Returns NULL if uri is NULL; a failed download on a non-NULL URI throws an exception. An OSS URI needs bucket access credentials configured.

MIME_TYPE

Returns the MIME type of a file, based on the file extension in a URI.

Syntax

1VARCHAR MIME_TYPE(VARCHAR uri)

Parameters

Supported extensions: BMP, JPEG/JPE/JPG, PNG, TIFF/TIF, WEBP, HEIC, and PDF. An unlisted extension returns application/other.

Returns: VARCHAR — the MIME type. Returns NULL if uri is NULL.

IMAGE_COMPRESS

Compresses an image to a target file size while holding a minimum quality.

Syntax

1ROW<compressed_image_content VARBINARY, result_size INT, result_quality INT, result_long_edge INT>
2IMAGE_COMPRESS(image_content VARBINARY, max_size INT, max_edge INT, min_quality INT)

Parameters

Compression runs in up to three stages, stopping as soon as the result meets max_size: scaling the longest edge down to max_edge, then reducing quality down to min_quality, then scaling resolution further while holding min_quality. Fails if the target size still isn't met once the shortest edge reaches 10 pixels.

Returns: a row with the compressed image content plus its resulting size, quality, and longest-edge length.

Example

1SELECT IMAGE_COMPRESS(FETCH_CONTENT(image_url)) AS compressed FROM T1;

IMAGE_LAPLACE_VAR

Calculates an image's Laplacian variance, a measure of sharpness — lower values indicate a blurrier image.

Syntax

1ROW<score DOUBLE, reason STRING> IMAGE_LAPLACE_VAR(content VARBINARY, max_edge INT, k_size INT)

Parameters

Returns: a row with the Laplacian variance score and a reason string set when the calculation fails.

Example

1SELECT IMAGE_LAPLACE_VAR(FETCH_CONTENT(image_url)) AS sharpness FROM T1;

PDF_TO_IMAGES

Splits a PDF into per-page images, returning one row per page. Use it as a table-valued function with LATERAL TABLE.

Syntax

1PDF_TO_IMAGES(content [, image_format] [, dpi] [, start_page] [, pages])

Parameters

Returns: one row per page, with the page's mime_type, 0-indexed page_no, and image_content (VARBINARY).

Example

1SELECT p.mime_type, p.page_no
2FROM (SELECT FETCH_CONTENT(pdf_url) AS pdf_content FROM T1) AS t,
3  LATERAL TABLE(PDF_TO_IMAGES(t.pdf_content, 'jpg', 150)) AS p(mime_type, page_no, image_content);

Base64 encoding functions

These are part of the Flink string functions set. TO_BASE64 is what you use to build the data URL a VLM API call expects.

Example: FROM_BASE64('aGVsbG8gd29ybGQ=') returns 'hello world'. TO_BASE64('hello world') returns 'aGVsbG8gd29ybGQ='.

Use TO_BASE64 to construct a data URL for OpenAI-compatible VLM API calls, in the form:

1data:<mime_type>;base64,<base64_string>

AI_EXTRACT and AI_CLASSIFY: Image Input

AI_EXTRACT and AI_CLASSIFY accept image input the same way as ML_PREDICT: register a model with content_type = 'image_url', then pass a TO_BASE64-encoded data URL. These functions are part of the engine, so they work on both byoc and self-managed deployments.

1CREATE TEMPORARY MODEL classify_product_image
2INPUT (input_contents STRING)
3OUTPUT (category STRING)
4WITH (
5    'provider' = 'openai',
6    'endpoint' = '<your-endpoint>',
7    'api-key' = '<your-api-key>',
8    'model' = 'gpt-4.1',
9    'content_type' = 'image_url'
10);
11
12SELECT id, AI_CLASSIFY(TO_BASE64(FETCH_CONTENT(s3_uri)), MODEL classify_product_image) AS category
13FROM product_images;

For the full AI_EXTRACT/AI_CLASSIFY signature and the rest of the AI SQL Functions family, see the vvp3 AI SQL Functions reference — the same functions apply to byoc.