Universal Blob Storage

Ververica Platform provides centralized configuration of blob storage for its services.

Configuration

In order to enable universal blob storage configure a base URI for your blob storage provider. Add the following snippet to your Helm values.yaml file:

vvp:
  blobStorage:
    baseUri: s3://my-bucket/vvp

The provided base URI will be picked up by all services that can make use of blob storage Application Manager or Artifact Management.

Storage Providers

Storage Provider	Scheme	Artifact Management	State Snapshots
			Flink 1.10	Flink 1.9	Flink 1.8
File system	file://	✓	✓	✓	✓
AWS S3	s3://	✓	✓	✓	✓
Microsoft ABS	wasbs://	✓	✓	✓	(✓)
Google GCS	gs://	✓	x	x	x
Alibaba OSS	oss://	✓	x	x	x

(✓): With custom Flink image

Additional Provider Configuration

Some supported storage providers have additional options that can be configured in the blobStorage section of the values.yaml file, scoped by provider.

The following is a complete listing of supported additional storage provider configuration options:

blobStorage:
  s3:
    endpoint: ""
    region: ""

Credentials

Ververica Platform supports using a single set of credentials to access your configured blob storage, and will automatically distribute these credentials to Flink jobs that require them.

These credentials can be either specified directly in values.yaml, or added to a Kubernetes secret out-of-band and referenced in values.yaml by name.

The following is a complete listing of the credentials that can be given for each storage provider, with example values:

blobStorageCredentials:
  azure:
    connectionString: DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=vvpArtifacts;AccountKey=VGhpcyBpcyBub3QgYSB2YWxpZCBBQlMga2V5LiAgVGhhbmtzIGZvciB0aG9yb3VnaGx5IHJlYWRpbmcgdGhlIGRvY3MgOikgIA==;
  s3:
    accessKeyId: AKIAEXAMPLEACCESSKEY
    secretAccessKey: qyRRoU+/4d5yYzOGZVz7P9ay9fAAMrexamplesecretkey

To use a pre-created Kubernetes secret, its keys must match the pattern <provider>.<key>. For example, s3.accessKeyId and s3.secretAccessKey. To configure Ververica Platform to use this secret, add the following snippet to your Helm values.yaml file:

blobStorageCredentials:
  existingSecret: my-blob-storage-credentials

Important

The values in a Kubernetes secret must be base64-encoded.

AWS S3

When running on AWS EKS or AWS ECS your Kubernetes Pods inherit the roles attached to the underlying EC2 instances. If these roles already grant access to the required S3 resources you only need to configure vvp.blobStorage.baseUri without configuring any blobStorageCredentials.

Services

The following services make use of the universal blob storage configuration.

Flink Jobs

Flink jobs are configured to store blobs at the following locations:

Blob	Storage Location
Checkpoints	${baseUri}/flink-jobs/namespaces/${ns}/jobs/${jobId}/checkpoints
Savepoints	${baseUri}/flink-savepoints/namespaces/${ns}/deployments/${deploymentId}
High Availability	${baseUri}/flink-jobs/namespaces/${ns}/jobs/${jobId}/ha

User-provided configuration has precedence over universal blob storage.

Artifact Management

Artifacts are stored in the following location:

${baseUri}/artifacts/namespaces/${ns}