Skip to main content
Version: 2.12

Quickstart: Google GKE

Google Kubernetes Engine (GKE) is Google Cloud Platform's hosted Kubernetes service.

A Google Kubernetes Engine cluster created using the official guide is suitable to run Ververica Platform.

Quickstart​

  1. Create a Google Kubernetes Engine cluster using the official guide.
  2. Follow the Helm installation guide to finish installing Ververica Platform.
  3. Follow the steps below to add support for GCS as checkpoint and savepoint storage.

For an in-depth, step-by-step guide on setting up Ververica Platform on GKE, take a look at our knowledge base article: How to set up Ververica Platform with Google Cloud Storage (GCS).

Apache Flink® currently does not bundle libraries for working with GCS. There is, however, a connector from Google based on Apache Hadoop®. In order to use the gs:// scheme for Universal Blob Storage, it is required to bundle this library along with the shaded Hadoop JAR.

Download GCS dependencies

Please download the following GCS dependencies. Click each link to start the download:

Move the downloaded files so you can access them to build the Dockerfile.

Create Dockerfile

    FROM registry.ververica.com/v{shortVersion}/flink:{ververicaFlinkRelease}-scala_2.12-java11

    COPY gcs-connector-hadoop2-latest.jar ../lib/
    COPY flink-shaded-hadoop-2-uber-2.8.3-10.0.jar ../lib/

To build a Dockerfile for Apache Flink® 1.16.1 run

     docker build -t {ververicaFlinkRelease}-scala_2.12-java11-gcs .
     docker tag {ververicaFlinkRelease}-scala_2.12-java11-gcs <YOUR_DOCKER_REGISTRY>/flink:{ververicaFlinkRelease}-scala_2.12-java11-gcs
     docker push <YOUR_DOCKER_REGISTRY>/flink:{ververicaFlinkRelease}-scala_2.12-java11-gcs

For other versions simply adjust the build arguments accordingly.

Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

Prepare Ververica Platform

Amend your values.yaml with the following parameters:

    vvp:
      blobStorage:
        baseUri: gs://<YOUR_GCS_BUCKET>

      flinkVersionMetadata:
      - flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
        flinkImageRepository: flink
        flinkVersion: {flinkRelease}
        imageTag: {ververicaFlinkRelease}-scala_2.12-java11-gcs
        useForSqlDeployments: true
        defaultFor:
        - {flinkVersion}

      globalDeploymentDefaults: |
        spec:
          template:
            spec:
              artifact:
                kind: jar
                flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
                flinkImageRepository: flink
                flinkImageTag: {ververicaFlinkRelease}-scala_2.12-java11-gcs
                flinkVersion: '{flinkVersion}'
              kubernetes:
                pods:
                  volumeMounts:
                  - name: gcs-key
                    volume:
                      name: gcs-key
                      secret:
                        secretName: gcs-key
                    volumeMount:
                      mountPath: /flink/gcs-creds/
                      name: gcs-key
                  envVars:
                    - name: GOOGLE_APPLICATION_CREDENTIALS
                      value: /flink/gcs-creds/gcs-key.json


      globalSessionClusterDefaults: |
        spec:
          kubernetes:
            pods:
              volumeMounts:
                - name: gcs-key
                  volume:
                    name: gcs-key
                    secret:
                      secretName: gcs-key.json
                  volumeMount:
                    mountPath: /flink/gcs-creds/
                    name: gcs-key
              envVars:
                - name: GOOGLE_APPLICATION_CREDENTIALS
                  value: /flink/gcs-creds/gcs-key.json

    volumes:
    - name: "gcs-key"
      secret:
        secretName: "gcs-key"

    volumeMounts:
    - name: "gcs-key"
      mountPath: "/vvp/gcs-creds"

    env:
      - name:  "GOOGLE_APPLICATION_CREDENTIALS"
        value: "/vvp/gcs-creds/gcs-key.json"

Permissions for GCS

note

This step is optional if you already have service account credentials with the Storage Admin role permissions for your designated GCS bucket.

  1. If you do not yet have a service account, create one:
    gcloud iam service-accounts create gcs-vvp-service-acc \
    --description="Service account for VVP GCS" \
    --display-name="gcs-vvp-service-acc"

  1. Assuming that you use the "uniform" bucket access control, navigate to Storage Browser, click three dots next to your bucket -> Edit bucket permissions -> + Add member -> and paste your full service account ID (gcs-vvp-service-acc@<YOUR_PROJECT>.iam.gserviceaccount.com).

  2. In "Select a role", pick "Storage Admin". Save.

  3. Navigate to Service Accounts, click three dots next to the gcs-vvp-service-acc service account -> Create key -> JSON -> Store as gcs-key.json.

  4. Create a Kubernetes secret in the namespace where Ververica Platform is installed (vvp in this example):

    kubectl create secret generic
    gcs-key --from-file=./gcs-key.json -n vvp
  1. Repeat for each namespace that is used as a Deployment Target (vvp-jobs in this example):
    kubectl create secret generic gcs-key --from-file=./gcs-key.json -n vvp-jobs