- API: Add securityContext to Kubernetes options
- Flink 1.7 deployments
- Implement work around for FLINK-11127 (Flink’s metric query service)
- Don’t report job status as running before all tasks are running
- Fix secret mounting for artifact fetcher init container
- Don’t require JAR entryClass to be explicitly specified if JAR has manifest entry (Note: this is only supported with the 1.7.0-dap2 and 1.7.1-dap2 Flink images)
- User interface
- Add Flink 1.7.0-dap2 and 1.7.1-dap2 image tag to the image drop down
- Add Flink 1.6.3-dap1 image tag to the image drop down
- Respect JAVA_OPTS environment variable for Application Manager JVM
- We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.3 installation
- Please follow the migration guide for 1.3.0 if you are migrating from an Application Manager 1.2 installation
- Add allowNonRestoredState flag to restore strategies LATEST_STATE and LATEST_SAVEPOINT
- Flink 1.7 deployments
- Set default image for Flink 1.7 to 1.7.1
- Don’t use nested volume mount for SSL keys
- Pass resource requests to job clusters
- User interface
- Add Flink 1.7.1 image tag to the image drop down
- Re-add Flink 1.5 images to the image drop down
- Dynamically query Flink for the Flink job ID instead of relying on a static mapping between Application Manager’s job resource and the Flink job
- Please follow the migration guide for 1.3.0 and use 1.3.1 as the image tag for Application Manager (step 5 below)
- End-to-end Flink SSL/TLS allows you to auto provision Flink 1.6 and 1.7 jobs with SSL/TLS support, including mutual authentication.
- New restoreStrategy API for more flexible state restore operations. This include a new mode of operation that allows you to restore your application from both savepoints as well as checkpoints. You have to enable Flink High Availability and checkpoint retention in order to enable restoring from latest checkpoints.
- Application Manager supports Flink 1.7 and manages Flink applications using the container entrypoint. This makes the deployment of applications more robust and less time consuming. Instead of requiring Application Manager to access and submit your JAR artifact to Flink, we let Flink fetch it.
- You can now configure custom imagePullSecrets for each Deployment, making it possible to access custom Docker registries.
- The Settings page of the web user interface now allows you to manage more resources, such as namespaces, API tokens, and seret values.
- Important: Flink 1.7 jobs require entryClass to be configured in Deployment.spec.template.spec.artifact. If you are running Application Manager version greater or equal to 1.3.2 and Flink image tagged as 1.7.0-dap2 or 1.7.1-dap2 (or more recent) this requirement has been dropped. See release notes of version 1.3.2.
- If you are using Flink 1.6, please set your Flink applications to use the image with tag: registry.platform.data-artisans.net/trial/v1.3/flink:1.6.2-dap2-scala_2.11. The dap2 image fixes the built-in InfluxDB metrics reporter that was included in earlier versions. For Flink 1.7, all images already contain the fixed metrics reporter.
- In order to use an existing data volume of an earlier installation please add the following securityContext entry to your Application Manager Deployment:
- Add the following at the root level of your Application Manager ConfigMap for the appmanager.yaml entry:
- Update the image and args for the Application Manager Kubernetes Deployment:
- name: appmanager
- Give access to jobs and secrets Kubernetes resources for Role daplatform-appmanager.
Deployment.spec.startFromSavepoint has been deprecated in favor of
Deployment.spec.restoreStrategy. Operations against startFromSavepoint are reflected in restoreStrategy and vice versa. Only when using the new LATEST_STATE restore strategy will startFromSavepoint be not available.
- Single Sign-on Authentication with OpenID Connect
Single sign-on is implemented through OpenID Connect. Common OpenID connect providers are Google Cloud, Microsoft Azure Active Directory, Keycloak or Dex. Through Dex, further identity providers such as LDAP or SAML 2.0 are available.
- API Tokens for Machine to Machine Authentication
Application Manager now allows to create API tokens for services to authenticate. API tokens are revocable at any time. Access to the system can be restricted similarly to regular users singing on via SSO.
- Role-based Access Control
This new feature allows to restrict access to API resources by defining roles and binding them to users or groups. The access control is based on API resources such as deployments, jobs, savepoints, or events and the HTTP method, such as GET, POST, PATCH.
Namespaces now offer the ability to control visibility and access of API resources. This effectively introduces multi-tenancy into Ververica Platform, so that multiple teams can share a Ververica Platform setup, with strict separation of their resources.
- Secret Values
This release introduces a new API object called “Secret Value”. A secret allows to manage passwords, authentication tokens or secret configuration parameters. In particular, this allows for separating knowledge of a secret from usage of a secret.
- Support for Apache Flink 1.6
- Credentials in the jar URL
- Logging improvements
- User Interface Improvements
- Visualization of Flink Streaming Applications with metrics
- YAML view of Deployments
- Improved “Savepoints” tab
- General API: Deployment and Job resources are expanded eagerly with default values for optional fields. Prior to this version, optional fields of resources were lazily expanded which made it hard to understand the full state of a given resource. With this change, resources will always be fully specified.
- Support for Flink 1.5: Flink 1.5 is the new default version for Deployments. You can specify which Flink version to deploy via the newly introduced
flinkVersion attribute in
Deployment.spec.template.spec.artifact. Note that the specified Flink version and Flink image have to match. If they do not match, deployed jobs will fail with a corresponding error message.
- Custom Kubernetes options: We allow to pass Kubernetes-specific options to created Flink pods via
Deployment.spec.template.spec.kubernetes. This gives you more control about deployed Flink jobs, for instance by attaching volume mounts or a node selector to deployed pods.
- Specify Flink image by digest: In addition to image tags we allow to specify Flink images by image digest in
Deployment.spec.template.spec.artifact.flinkImageTag. In order to specify a digest prefix the Flink image tag with
- Support for finite Flink jobs: Finite Flink jobs such streaming applications consuming from finite sources or batch jobs transition transition to a new terminal state FINISHED. Prior to this release, any terminated job was treated as a failure and led to a job recovery (re-running the finite job).
- Web UI: There have been additions to the web UI to support newly introduced features and multiple minor fixes.
- When modifying an existing Deployment resource, default values will be expanded eagerly as described above. The newly introduced
flinkVersion attribute will default to
1.5 which will lead to your Flink jobs to be upgraded to Flink 1.5. If you don’t want please update the Deployment accordingly with the first modification (PATCH) and set
- Volume mount annotations with key
alpha/k8s-volume-mounts have been deprecated with this release. Please migrate these annotations to the newly introduced Kubernetes options. Support for the annotations may be dropped in future versions.