Getting Started

In this guide you will setup Ververica Platform, integrate it with Minio for blob storage and run your first stateful Apache Flink Deployment.

This quickstart provides a simple development environment, but is not meant for production. For production deployment information, see the installation guide.

Prerequisites

This quickstart requires access to

  • a Kubernetes cluster,
  • helm to install the platform, and
  • Minio for blob storage.

Check out the appendix for instructions on how to set up the required components on your local machine.

Download, Configure and Install Ververica Platform

First download the Ververica Platform Helm Chart from the customer portal or the Ververica website (non-customers). Then create a helm configuration file values.yaml with the following keys to configure the platform to use Minio for blob storage.

vvp:
  blobStorage:
    baseUri: s3://<your_bucket>

    s3:
      endpoint: <your_minio_endpoint>

blobStorageCredentials:
  s3:
    accessKeyId: <access_key_id>
    secretAccessKey: <secret_access_key>

Note

The bucket already needs to exist in Minio or S3.

The appendix contains a full example configuration. Now you can install the platform using helm:

$ helm install \
  --name vvp \
  --values values.yaml \
  ververica-platform-2.0.1.tgz

You can then access the Ververica Platform UI by port forwarding to localhost:8080.

$ kubectl port-forward service/vvp-ververica-platform 8080:80
Initial Home Screen

Create Your First Deployment

Before you can create a Deployment you need to add a Deployment Target. A Deployment Target is used to select the Kubernetes namespace, which your Apache Flink applications will be deployed into. Choose Deployment Targets in the left side-bar and click Add Deployment Target. Give it a name and use the same Kubernetes namespace as for the platform itself.

Adding Deployment Target

Back under Deployment, click the Create Deployment button on the top right corner of the page to create a new Deployment and provide it a name such as Top Speed Windowing. Then under the Behavior section you can specify how the application should be managed.

Deployment Target: Choose the Deployment Target that you just created to deploy the Flink application into the Kubernetes namespace it references.

Upgrade Strategy: The upgrade strategy should be Stateful so your application state is carried across Flink jobs.

Initial State: The initial state should be Running so that Ververica Platform will automatically deploy a Flink job when you finish creating the Deployment.

Restore Strategy: The restore strategy specifies what state should be used when starting a new Flink job. We will set this to LATEST_SAVEPOINT.

Behavior

We can set up the configuration for your specific job by providing a URI to the Jar with your Flink program. There are two ways to do this: either you switch to the Artifacts screen and upload your artifact directly to the platform, or you point Ververica Platform to an externally stored artifact.

Behavior

Note

If you do not have an artifact to upload, you can use the Top Speed Windowing example which the Flink community uploads to Maven Central for each release:

http://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.11/1.9.0/flink-examples-streaming_2.11-1.9.0-TopSpeedWindowing.jar

Configuration

Under the resources section, you specify the parallelism of your job as 1 and give jobmanager and taskmanagers 0.5 CPUs and 1G of memory each.

Resources

Finally, you press the Create Deployment button to start your job.

Modify Your Deployment

After creating your job, you can try out another feature of the platform and make a change to the Deployment in a stateful and consistent manner. This could be changing the jar to release a new version of the application, or in this case, change a configuration such as the parallelism.

To make a change, you press Modify Deployment which will either take us back to the creation screen you used earlier, or let you edit the deployment as yaml.

Modify Deployment

You can use either, but in this case use Edit as YAML and change the parallelism of your application from 1 to 2 and press Apply Changes.

Deployment Yaml

Once the changes have been applied, Ververica Platform will take care of the rest and transition your application to use the updated configuration.

Transitioning

Suspend Your Deployment

When finished, press the Suspend button to gracefully stop all resources and stop the application.

Suspend Deployment

Next Steps

Now that you have a running Ververica Platform instance there are multiple directions you can go into.

Appendix: Setting Up Minikube and Minio

If you do not have a Kubernetes installation readily available you can use Minikube and Minio to run this quickstart locally.

Install Minikube by following the official instructions in the Kubernetes documentation. Once minikube has been installed you can start your local Kubernetes cluster.

$ minikube start --memory 8192 --cpus 4

Next, you will want to install Helm, a Kubernetes package manager. A full walk through of Helm can be found in the installation instructions.

$ kubectl --namespace kube-system create serviceaccount tiller
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller

With the Helm install command below you can quickly install Minio, an S3 compatible object store. Note that you might need to change the clusterIP adress depending on your installation.

# Install Minio with username: admin and password: password, default bucket: vvp

$ helm install \
  --set accessKey=admin,secretKey=password,service.clusterIP=10.96.1.1,defaultBucket.enabled=true,defaultBucket.name=vvp \
  --name  my-minio stable/minio

To configure Ververica Platform in this particular setup, you can use the following values.yaml:

vvp:
  blobStorage:
    baseUri: s3://vvp
    s3:
      endpoint: http://10.96.1.1
blobStorageCredentials:
  s3:
    accessKeyId: admin
    secretAccessKey: password