Skip to main content
Version: 2.14

Operator Configuration

This guide walks through the steps required to deploy Ververica Platform Operator on Kuberbetes.

In summary the steps are:

  1. Deploy the Custom Resource Definition (CRD)
  2. Configure the Operator
  3. Start Ververica Platform
  4. Configure the CR Deployment
  5. Configure the Deployment in the Platform UI

Deploy the Custom Resource Definition (CRD)​

The Ververica Platform Operator CRD is the custom resource definition for the Operator. It is contained in a Helm chart that ships with the platform, ververica/ververica-platform-crd.

Deploy the CRD using Helm install:

helm install my-vvp-crd ververica/ververica-platform-crd --version 1.0.0

where:

  • my-vvp-crd A user defined name for the Operator, follow Helm naming conventions.
  • ververica/ververica-platform-crd The Helm chart containing the Ververica Platform CRD.
  • --version The CRD version to install, see the table below:
Ververica Platform VersionCRD Helm Chart Version
2.13.x1.0.0

Configure the Operator​

After installing the CRD, configure the Operator. The Operator is bundled in Ververica Platform itself.

To configure and enable the Operator, add the following to your main Ververica Platform installation values YAML file, by default values.yaml:

vvp:
  k8sOperator:
    enabled: true
    identity: "vvp-1"
    watchedNamespaces:
      - <the k8s-namespace for the CR>
      - <the k8s-namespace for the CR>

where:

  • enabled Set true to enable, false to disable.
  • identity A user defined name for the platform instance that includes the Operator, used to distinguish different instances.
  • watchedNamespaces List of the Kubernetes Namespace(s) that the Operator will watch, which must include the Namespace into which you deploy the CR, see below.
note

Only one Operator instance can watch a specific Kubernetes namespace with watchedNamespaces. If more try to watch there will be a conflict and the application won't start. By using the k8sOperator.identity property you can distinguish the Operaror instances associated with specific Ververica Platform instances and avoid potential conflicts.

The Operator creates a ConfigMapLock ververica-operator-leader-election for the specified identity e.g. vvp-1 in each namespace listed under the watchedNamespaces property.

note

The lock lasts forever. To change the Operator identity e.g. to "vvp-2", delete the configMap and reinstall the Operator.

Start Ververica Platform​

Now start Ververica Platform with the helm install command, see the Installation Using Helm documentation.

note

If you are deploying Ververica Platform with a brand new database, we recommend you create a Deployment first, to initialize the default Namespace and a Deployment Target.

Once your Deployment is successfully created and transitioned to the Running state, continue with these steps to configure and deploy the Operator.

Configure the CR Deployment​

Assuming the configuration for this CR is in the file test-cr.yaml:

kubectl -n <namespace> apply -f test-cr.yaml

Be sure that <namespace> is in the watchedNamespaces list you configured for the Operator. In the simple case, set it to be the same as the deploymentTargetName in the CR (line 13 below) where you are deploying your Flink jobs, see More About Namespaces.

An example configuration that applies in patch mode could look like this:

note

Patch mode refers to the PATCH API of Ververica Platform REST API, see spec below.

apiVersion: ververica.platform/v1
kind: VvpDeployment
metadata:
  name: test-cr
spec:
  syncingMode: PATCH
  deployment:
    userMetadata:
      name: test-cr
      namespace: default
    spec:
      state: CANCELLED
      deploymentTargetName: test-target
      maxJobCreationAttempts: 99
      template:
        spec:
          artifact:
            jarUri: >-
              file:///flink/examples/streaming/TopSpeedWindowing.jar
            kind: JAR
  initFields:
    userMetadata:
      displayName: abc
    spec:
      maxSavepointCreationAttempts: 999

The property definitions are as follows:

spec​

  • spec.syncingMode Specify which Deployment API to call. Possible values are PATCH and PUT:

    • PATCH Apply the configuration as a change against the Deployment. The values in the CR will overwrite the values in the referenced Deployment. The fields present in the referenced Deployment but not present in the CR will be unchanged.

    • PUT Replaces the entire referenced Deployment with the data in the CR. The fields present in the referenced Deployment but not present in the CR will be removed. Note that when deploying a CR that refers to an existing Deployment in Ververica Platform, the values will first be patched to the existing Deployment, and the patched result will be synced back to the CR. This way, users don't need to write all the fields of a Deployment in a CR.

  • spec.deployment Specify the Deployment properties. The properties are similar to the fields displayed by the Configure button in the UI but not identical, which enables a CR to control properties that would otherwise be controlled by the Platform. For example, in the Platform:

spec:
  deployment: xxx
    metadata: xxx

and in the CR:

spec:
  deployment: xxx
    metadata: xxx

  • spec.deployment.userMetadata Specify the metadata for the resource we are deploying. Use valid keys only; invalid keys will be erased after reconciliation.

    • Valid keys: name, namespace, displayName, labels, annotations.

    • Note that system metadata properties are moved to status.deploymentSystemMetadata in the CR, see status below.

    • Some annotations used by the Platform are also invalid in a CR:

      • com.dataartisans.appmanager.controller.deployment.spec.version
      • com.dataartisans.appmanager.controller.deployment.transitioning
      • com.dataartisans.appmanager.controller.deployment.transitioning.since

    • The Operator uses spec.deployment.userMetadata.namespace + spec.deployment.userMetadata.name to identify the Ververica Platform Deployment instance.

  • spec.deployment.spec Specify the Deployment properties, has the same meaning as the spec property in a Ververica Platform Deployment, and expects the same keys.

    • For example, the test Deployment above defines the following key/value pairs:

      • state: CANCELLED
      • deploymentTargetName: test-target
      • maxJobCreationAttempts: 99

  • spec.initFields Supported only in patch mode i.e. when the spec.syncingMode value is PATCH.

    • The property is applied to spec.deployment during initialization of the CR. You can use it to set initial values for some keys and allow further changes.
    • For example, the test Deployment above sets initial values for displayName and maxSavepointCreationAttempts.

status​

The system metadata properties:

  • status.deploymentStatus is the same as deployment.status in Ververica Platform.

  • status.deploymentSystemMetadata The Deployment metadata fields are controlled by Ververica Platform and cannot be set by users:

status:
  deploymentSystemMetadata: xxx
    annotations:
      com.dataartisans.appmanager.controller.deployment.spec.version: "4"
      com.dataartisans.appmanager.controller.deployment.transitioning: "true"
      com.dataartisans.appmanager.controller.deployment.transitioning.since: "1689327990185"
    createdAt: "2023-07-14T09:40:41.583995Z"
    id: cf2047ed-cb55-4e41-bb53-9ef91959214e
    labels: {}
    modifiedAt: "2023-07-14T09:46:30.185884Z"
    resourceVersion: 20

  • status.customResourceStatus The status of this CR:

    • customResourceState

      • SYNCING The CR spec is syncing to the referenced Deployment in Ververica Platform.
      • IDLING The syncing is off.

    • deploymentId The observed Deployment ID of the referenced Deployment.

    • observedSpecState The observed spec.state of the referenced Deployment in Ververica Platform.

    • statusState The observed status.state of the referenced Deployment in Ververica Platform.

    • vvpNamespace The vvp-namespace of the referenced Deployment in Ververica Platform, the same as the value of spec.userMetadata.namespace in the CR.

JSON schema for YAML configuration validation​

While configuring, you can also take advantage of a JSON schema for validating your YAML configuration. This schema enforces proper structure, required fields, data types, and value constraints, guaranteeing that all user-provided YAML configurations conform to our standards before processing.

Click to see the JSON schema
{
  "type": "object",
  "required": ["metadata", "spec", "kind"],
  "properties": {
    "kind": {
      "type": "string",
      "enum": ["VvpDeployment"]
    },
    "metadata": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "namespace": {
          "type": "string"
        }
      },
      "spec": {
        "type": "object",
        "required": ["deployment"],
        "properties": {
          "syncingMode": {
            "type": "string",
            "enum": ["PUT", "PATCH"]
          },
          "deployment": {
            "type": "object",
            "required": ["userMetadata", "spec"],
            "properties": {
              "apiVersion": {
                "type": "string"
              },
              "kind": {
                "type": "string"
              },
              "userMetadata": {
                "type": "object",
                "required": ["displayName"],
                "properties": {
                  "annotations": {
                    "type": "object",
                    "additionalProperties": {
                      "type": "string"
                    }
                  },
                  "createdAt": {
                    "type": "string",
                    "format": "date-time"
                  },
                  "displayName": {
                    "type": "string"
                  },
                  "id": {
                    "type": "string",
                    "format": "uuid"
                  },
                  "labels": {
                    "type": "object",
                    "additionalProperties": {
                      "type": "string"
                    }
                  },
                  "modifiedAt": {
                    "type": "string",
                    "format": "date-time"
                  },
                  "name": {
                    "type": "string"
                  },
                  "namespace": {
                    "type": "string"
                  },
                  "resourceVersion": {
                    "type": "integer",
                    "format": "int32",
                    "minimum": -2147483648,
                    "maximum": 2147483647
                  }
                }
              },
              "spec": {
                "type": "object",
                "anyOf": [
                  {
                    "required": ["deploymentTargetName"]
                  },
                  {
                    "required": ["sessionClusterName"]
                  }
                ],
                "required": ["template"],
                "properties": {
                  "deploymentTargetId": {
                    "type": "string",
                    "format": "uuid"
                  },
                  "deploymentTargetName": {
                    "type": "string"
                  },
                  "jobFailureExpirationTime": {
                    "type": "string"
                  },
                  "maxJobCreationAttempts": {
                    "type": "integer",
                    "format": "int32",
                    "minimum": -2147483648,
                    "maximum": 2147483647
                  },
                  "maxSavepointCreationAttempts": {
                    "type": "integer",
                    "format": "int32",
                    "minimum": -2147483648,
                    "maximum": 2147483647
                  },
                  "restoreStrategy": {
                    "type": "object",
                    "properties": {
                      "allowNonRestoredState": {
                        "type": "boolean"
                      },
                      "kind": {
                        "type": "string"
                      }
                    }
                  },
                  "sessionClusterName": {
                    "type": "string"
                  },
                  "state": {
                    "type": "string",
                    "enum": ["RUNNING", "SUSPENDED", "CANCELLED"]
                  },
                  "template": {
                    "type": "object",
                    "required": ["spec"],
                    "properties": {
                      "metadata": {
                        "type": "object",
                        "properties": {
                          "annotations": {
                            "type": "object",
                            "additionalProperties": {
                              "type": "string"
                            }
                          }
                        }
                      },
                      "spec": {
                        "type": "object",
                        "required": ["artifact"],
                        "properties": {
                          "artifact": {
                            "type": "object",
                            "anyOf": [
                              {
                                "properties": {
                                  "jarUri": {
                                    "type": "string"
                                  },
                                  "kind": {
                                    "type": "string",
                                    "enum": ["JAR"]
                                  }
                                },
                                "required": ["kind", "jarUri"]
                              },
                              {
                                "properties": {
                                  "pythonArtifactUri": {
                                    "type": "string"
                                  },
                                  "kind": {
                                    "type": "string",
                                    "enum": ["PYTHON"]
                                  }
                                },
                                "required": ["kind", "pythonArtifactUri"]
                              },
                              {
                                "properties": {
                                  "sqlScript": {
                                    "type": "string"
                                  },
                                  "kind": {
                                    "type": "string",
                                    "enum": ["sqlscript"]
                                  }
                                },
                                "required": ["kind", "sqlScript"]
                              }
                            ],
                            "properties": {
                              "additionalDependencies": {
                                "type": "array",
                                "items": {
                                  "type": "string"
                                }
                              },
                              "additionalPythonArchives": {
                                "type": "array",
                                "items": {
                                  "type": "string"
                                }
                              },
                              "additionalPythonLibraries": {
                                "type": "array",
                                "items": {
                                  "type": "string"
                                }
                              },
                              "artifactKind": {
                                "type": "string",
                                "enum": ["PYTHON", "SQLSCRIPT", "JAR", "UNKNOWN"]
                              },
                              "entryClass": {
                                "type": "string"
                              },
                              "entryModule": {
                                "type": "string"
                              },
                              "flinkImageRegistry": {
                                "type": "string"
                              },
                              "flinkImageRepository": {
                                "type": "string"
                              },
                              "flinkImageTag": {
                                "type": "string"
                              },
                              "flinkVersion": {
                                "type": "string"
                              },
                              "kind": {
                                "type": "string"
                              },
                              "mainArgs": {
                                "type": "string"
                              },
                              "uri": {
                                "type": "string"
                              }
                            }
                          },
                          "flinkConfiguration": {
                            "type": "object",
                            "additionalProperties": {
                              "type": "string"
                            }
                          },
                          "kubernetes": {
                            "type": "object",
                            "properties": {
                              "jobManagerPodTemplate": {
                                "type": "object",
                                "properties": {
                                  "metadata": {
                                    "type": "object"
                                  },
                                  "spec": {
                                    "type": "object"
                                  }
                                }
                              },
                              "labels": {
                                "type": "object",
                                "additionalProperties": {
                                  "type": "string"
                                }
                              },
                              "pods": {
                                "type": "object",
                                "properties": {
                                  "affinity": {
                                    "type": "object"
                                  },
                                  "annotations": {
                                    "type": "object",
                                    "additionalProperties": {
                                      "type": "string"
                                    }
                                  },
                                  "envVars": {
                                    "type": "array",
                                    "items": {
                                      "type": "object",
                                      "properties": {
                                        "name": {
                                          "type": "string"
                                        },
                                        "value": {
                                          "type": "string"
                                        },
                                        "valueFrom": {
                                          "type": "object"
                                        }
                                      }
                                    }
                                  },
                                  "imagePullSecrets": {
                                    "type": "array",
                                    "items": {
                                      "required": ["name"],
                                      "type": "object",
                                      "properties": {
                                        "name": {
                                          "type": "string"
                                        }
                                      }
                                    }
                                  },
                                  "labels": {
                                    "type": "object",
                                    "additionalProperties": {
                                      "type": "string"
                                    }
                                  },
                                  "nodeSelector": {
                                    "type": "object",
                                    "additionalProperties": {
                                      "type": "string"
                                    }
                                  },
                                  "securityContext": {
                                    "type": "object"
                                  },
                                  "serviceAccountName": {
                                    "type": "string"
                                  },
                                  "tolerations": {
                                    "type": "array",
                                    "items": {
                                      "type": "object"
                                    }
                                  },
                                  "volumeMounts": {
                                    "type": "array",
                                    "items": {
                                      "type": "object",
                                      "properties": {
                                        "name": {
                                          "type": "string"
                                        },
                                        "volume": {
                                          "type": "object"
                                        },
                                        "volumeMount": {
                                          "type": "object"
                                        }
                                      }
                                    }
                                  }
                                }
                              },
                              "taskManagerPodTemplate": {
                                "type": "object",
                                "properties": {
                                  "metadata": {
                                    "type": "object"
                                  },
                                  "spec": {
                                    "type": "object"
                                  }
                                }
                              }
                            }
                          },
                          "latestCheckpointFetchInterval": {
                            "type": "integer",
                            "format": "int32",
                            "minimum": -2147483648,
                            "maximum": 2147483647
                          },
                          "logging": {
                            "type": "object",
                            "properties": {
                              "log4j2ConfigurationTemplate": {
                                "type": "string"
                              },
                              "log4jLoggers": {
                                "type": "object",
                                "additionalProperties": {
                                  "type": "string"
                                }
                              },
                              "loggingProfile": {
                                "type": "string"
                              }
                            }
                          },
                          "numberOfTaskManagers": {
                            "type": "integer",
                            "format": "int32",
                            "minimum": -2147483648,
                            "maximum": 2147483647
                          },
                          "parallelism": {
                            "type": "integer",
                            "format": "int32",
                            "minimum": -2147483648,
                            "maximum": 2147483647
                          },
                          "resources": {
                            "type": "object",
                            "additionalProperties": {
                              "required": ["cpu", "memory"],
                              "type": "object",
                              "properties": {
                                "cpu": {
                                  "minimum": 0.1,
                                  "type": "number",
                                  "format": "double",
                                  "maximum": 1.7976931348623157e308
                                },
                                "memory": {
                                  "type": "string"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "upgradeStrategy": {
                    "type": "object",
                    "properties": {
                      "kind": {
                        "type": "string"
                      }
                    }
                  }
                }
              },
              "status": {
                "type": "object",
                "properties": {
                  "running": {
                    "type": "object",
                    "properties": {
                      "conditions": {
                        "type": "array",
                        "items": {
                          "type": "object",
                          "properties": {
                            "lastTransitionTime": {
                              "type": "string",
                              "format": "date-time"
                            },
                            "lastUpdateTime": {
                              "type": "string",
                              "format": "date-time"
                            },
                            "message": {
                              "type": "string"
                            },
                            "reason": {
                              "type": "string"
                            },
                            "status": {
                              "type": "string",
                              "enum": ["True", "False", "Unknown"]
                            },
                            "type": {
                              "type": "string"
                            }
                          }
                        }
                      },
                      "jobId": {
                        "type": "string",
                        "format": "uuid"
                      },
                      "transitionTime": {
                        "type": "string",
                        "format": "date-time"
                      }
                    }
                  },
                  "state": {
                    "type": "string",
                    "enum": [
                      "RUNNING",
                      "SUSPENDED",
                      "CANCELLED",
                      "TRANSITIONING",
                      "FAILED",
                      "FINISHED"
                    ]
                  }
                }
              }
            }
          }
        }
      }
    },
    "$schema": "http://json-schema.org/draft-04/schema#"
  }

More about Namespaces​

Configuring the Namespaces used by the Operator can seem complex, especially because the configuration values need to be set correctly across the Operator configuration, CR manifest, and in the eventual Deployment configurations in the platform UI.

The simple rule to follow is that the Kubernetes Namespace into which you install the CR YAML file, which has a Kubernetes CustomResource type of VvpDeployment, must be in the watchedNamespaces list configured in the vvp.k8sOperator.watchedNamespaces configuration in your values.yaml file.

Operator Authentication and Authorization​

The Kubernetes Operator integrates with Ververica Platform's authentication/authorization and RBAC features.

note

If authentication is already enabled, users will need to update VvpDeployment definition custom resources. The resources now require authentication to be reconciled by the Ververica Platform K8s operator.

For custom resources to be reconciled by the Ververica Platform Kubernetes Operator, the custom resource will now require authentication. Authentication is done by using an API Token stored within a Kubernetes Secret resource. In order to link a custom resource with an API Token, the following steps need to be taken.

Deploy a Kubernetes Secret​

Within the same namespace as the custom resource will be deployed, deploy a Kubernetes Secret. It must be one of the namespaces defined in vvp.k8soperator.watchNamespaces in your values.yaml helm configuration file. The Kubernetes Secret must contain an entry with the key of "apikey" and value of base64 encoded API Token.

To complete the Kubernetes Secret, you need to either create or obtain an API Token from the Ververica Platform and save it to the file with the name apikey. Navigate to Obtaining an API Token for more detailed steps.

Lastly, execute the following command with example-secret-name replaced by any valid Kubernetes Secret name:

kubectl create secret generic example-secret-name --namespace your-k8s-namespace --from-file=apikey

After creating your Kubernetes Secret with the linked API Token, you need to add apikeySecretName to your VvpDeployment Custom Resource. The value for apikeySecretName will be the name of your previously created Kubernetes Secret from above.

For example:

apiVersion: ververica.platform/v1
kind: VvpDeployment
metadata:
  name: your-vvp-custom-resource-name
  namespace: your-k8s-namespace
  annotations:
    apikeySecretName: example-secret-name

This API Token will now be used to secure communication between the Ververica Platform Kubernetes Operator and the Ververica Platform application. If the custom resource is not authenticated, then it will not be reconciled by the Ververica Platform Kubernetes Operator.

Obtaining an API Token​

To use an API Token, a user can create one in the Administration and then API Tokens menus.

API Tokens Menu

A user can then give the token a name and assign to a specific environment role.

API Token Name &amp; Role Assignment

note

You need an API Token with editor or owner role to use it with Ververica Platform K8s-Operator.

Once the token is created and listed in the API Tokens table, it can be copied and used in the apikey annotation of VvpDeployment custom resource. For example:

apiVersion: ververica.platform/v1
kind: VvpDeployment
metadata:
  name: vvp-custom-resource-name
  namespace: vvp-custom-resource-namespace
  annotations:
    apikey: 77ede5cc-a39c-410e-a89f-550b7a9ac22c

A Simple Namespace Example​

Consider the example of a user installing Ververica Platform into the Namespace vvp-some-user and deploying Flink jobs to a Namespace vvp-jobs-some-user.

The Operator must watch the Namespace vvp-jobs-some-user.

In the simple case installing the CR e.g. test-cr.yaml into the Deployment Namespace vvp-jobs-some-user, and including that Namespace in the watchedNamespaces list in the Operator configuration, will achieve the goal.

For example, if we install test-cr.yaml with the following command:

kubectl -n vvp-jobs-some-user apply -f test-cr.yaml

where -n specifies the vvp-jobs-some-user Namespace, we would set the following values in test-cr.yaml:

apiVersion: ververica.platform/v1
  kind: VvpDeployment
  metadata:
    name: test-cr
  spec:
    ...
    deployment:
      userMetadata:
        name: test-cr
        namespace: vvp-some-user
      spec:
        state: CANCELLED
        deploymentTargetName: vvp-jobs-some-user
        ...
  • The installation Namespace vvp-jobs-some-user we specified in the kubectl apply command is where this CustomResource will run. It must be in the list of watchedNamespaces configured for the Operator.
  • The Ververica Platform Namespace (line 10 above), in this case vvp-some-user, or if the value is set to default then the default vvp Namespace, is the Namespace where the Deployment runs.
  • The deploymentTargetName (line 13 above) is the name of the specific DeploymentTarget, which is configured to point to the Kubernetes namespace in which the Deployment will run.

In the simple case, set deploymentTargetName to a DeploymentTarget that points to the Kubernetes Namespace into which you install the CR using kubectl. If you have reason to do so, you can set these to be different Namespaces BUT be sure that the installation namespace for the CR is listed in the CRD watchedNamespaces configuration.

When configured and installed, the Operator will now manage the specified Deployment.

The initial configuration, including creating the DeploymentTarget, should still be done in the Ververica Platform UI.

Configure the Deployment in the Platform UI​

In the Ververica Platform UI, configure the Deployment that will be managed by the Operator.

  • If the CR syncing button is displayed it means the Deployment is already managed by the CR.

  • If no CR syncing button is displayed then the Deployment is not managed by the CR.

If the CR syncing button doesn't display for a Deployment you are trying to configure to be Operator managed, check your Operator configuration. The mostly likely cause for error is mismatched Namespaces.

note

Before making configuration changes to a Deployment in the UI, turn off CR Syncing. With syncing off, the Operator will not sync the CR’s data (i.e. its spec and metadata) to the database even though the Deployment’s status will still be reconciled to the CR.

Configure CR UI

You can also manage the syncing state from the Deployments page:

Deployments UI