Quickstart: Google GKE

Google Kubernetes Engine (GKE) is Google Cloud Platform’s hosted Kubernetes service.

A Google Kubernetes Engine cluster created using the official guide is suitable to run Ververica Platform.

Quickstart

1. Create a Google Kubernetes Engine cluster using the official guide.
2. Follow the Helm installation guide to finish installing Ververica Platform.
3. Follow the steps below to add support for GCS as checkpoint and savepoint storage.

For an in-depth, step-by-step guide on setting up Ververica Platform on GKE, take a look at our knowledge base article: How to set up Ververica Platform with Google Cloud Storage (GCS).

Building Apache Flink® Images with Support for GCS

Apache Flink® currently does not bundle libraries for working with GCS. There is, however, a connector from Google based on Apache Hadoop®. In order to use the gs:// scheme for Universal Blob Storage, it is required to bundle this library along with the shaded Hadoop JAR.

Download GCS dependencies

Please download the following GCS dependencies:

• flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
• gcs-connector-hadoop2-latest.jar

Create Dockerfile

FROM registry.ververica.com/v2.8/flink:1.15.2-stream4-scala_2.12-java11

COPY gcs-connector-hadoop2-latest.jar ../lib/
COPY flink-shaded-hadoop-2-uber-2.8.3-10.0.jar ../lib/

To build a Dockerfile for Apache Flink® 1.15.2 run

$ docker build -t 1.15.2-stream4-scala_2.12-java11-gcs .
$ docker tag 1.15.2-stream4-scala_2.12-java11-gcs <YOUR_DOCKER_REGISTRY>/flink:1.15.2-stream4-scala_2.12-java11-gcs
$ docker push <YOUR_DOCKER_REGISTRY>/flink:1.15.2-stream4-scala_2.12-java11-gcs

For other versions simply adjust the build arguments accordingly.

Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

Prepare Ververica Platform

Amend your values.yaml with the following parameters:

vvp:
  blobStorage:
    baseUri: gs://<YOUR_GCS_BUCKET>

  flinkVersionMetadata:
  - flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
    flinkImageRepository: flink
    flinkVersion: 1.15.2
    imageTag: 1.15.2-stream4-scala_2.12-java11-gcs
    useForSqlDeployments: true
    defaultFor:
    - 1.15

  globalDeploymentDefaults: |
    spec:
      template:
        spec:
          artifact:
            kind: jar
            flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
            flinkImageRepository: flink
            flinkImageTag: 1.15.2-stream4-scala_2.12-java11-gcs
            flinkVersion: '1.15'
          kubernetes:
            pods:
              volumeMounts:
              - name: gcs-key
                volume:
                  name: gcs-key
                  secret:
                    secretName: gcs-key
                volumeMount:
                  mountPath: /flink/gcs-creds/
                  name: gcs-key
              envVars:
                - name: GOOGLE_APPLICATION_CREDENTIALS
                  value: /flink/gcs-creds/gcs-key.json


  globalSessionClusterDefaults: |
    spec:
      kubernetes:
        pods:
          volumeMounts:
            - name: gcs-key
              volume:
                name: gcs-key
                secret:
                  secretName: gcs-key.json
              volumeMount:
                mountPath: /flink/gcs-creds/
                name: gcs-key
          envVars:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /flink/gcs-creds/gcs-key.json

volumes:
- name: "gcs-key"
  secret:
    secretName: "gcs-key"

volumeMounts:
- name: "gcs-key"
  mountPath: "/vvp/gcs-creds"

env:
  - name:  "GOOGLE_APPLICATION_CREDENTIALS"
    value: "/vvp/gcs-creds/gcs-key.json"

Permissions for GCS

Note

This step is optional if you already have service account credentials with the Storage Admin role permissions for your designated GCS bucket.

1. If you do not yet have a service account, create one:

gcloud iam service-accounts create gcs-vvp-service-acc \
--description="Service account for VVP GCS" \
--display-name="gcs-vvp-service-acc"

2. Assuming that you use the “uniform” bucket access control, navigate to Storage Browser, click three dots next to your bucket -> Edit bucket permissions -> + Add member -> and paste your full service account ID (gcs-vvp-service-acc@<YOUR_PROJECT>.iam.gserviceaccount.com).
3. In “Select a role”, pick “Storage Admin”. Save.
4. Navigate to Service Accounts, click three dots next to the gcs-vvp-service-acc service account -> Create key -> JSON -> Store as gcs-key.json.
5. Create a Kubernetes secret in the namespace where Ververica Platform is installed (vvp in this example):

kubectl create secret generic gcs-key --from-file=./gcs-key.json -n vvp

6. Repeat for each namespace that is used as a Deployment Target (vvp-jobs in this example):

kubectl create secret generic gcs-key --from-file=./gcs-key.json -n vvp-jobs