Releases

Application Manager 1.4.1

Changelog

  • Updated Flink 1.7.2 and 1.8.0 images to stream2 version. In addition to the existing alpine images, we also provide images based on Debian slim:

    flink:1.7.2-stream2-scala_2.11-alpine
    flink:1.7.2-stream2-scala_2.11-slim
    
    flink:1.8.0-stream2-scala_2.11-alpine
    flink:1.8.0-stream2-scala_2.11-slim
    

    Image tags such as 1.8.0-stream2-scala_2.11 that don’t include the distro are still based on alpine.

  • Fix regression in 1.4.0 that resulted in finite jobs being reported as transitioning/running after completion

  • Extract human readable Exception message for reported Flink errors

  • User interface
    • Store Deployment list filters in local storage
    • Add Flink restart strategy configuration helper in Deployment details page
    • Change mainArgs in Deployment details page to text area for long arguments lists
    • Parse namespace from params instead of settings

Migration

  • We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.4 installation

Application Manager 1.4.0

Changelog

  • Add support for Flink 1.8 jobs
  • Add support for Flink’s queryable state via TaskManager service (see Configure Apache Flink for more details)
  • Improve Flink application deployments
    • Detect non progressing deployments
    • More robust Flink job shutdown
    • Faster session cluster spin up
    • Unified artifact fetching between different Flink versions
  • API
    • DeploymentTarget becomes namespaced resource and allows to configure Kubernetes namespace to deploy to (see Deployment Targets for more details). Note that the default permissions allow access to all DeploymentTarget resources and operations. Please restrict access further if required.
    • Allow to specify valueFrom for Kubernetes environment variables (see Configure Kubernetes for more details)
  • User interface
    • Improved defaults for Deployment configuration form
    • Add filters to Deployment overview page (by date, label, DeploymentTarget)
    • Added interface for access control management
    • Various improvements and fixes
    • Rebranded to Ververica

Migration

  • We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.3 installation

Deprecation

  • Docker images are only published to the dist namespace going forward. If you are still using an image from the v1.3 or trial/v1.3 namespace, please replace it with dist. The Helm charts already point to this namespace by default since the 1.3 release. No update is needed if you are not migrating from an older version.
  • The minimum required Kubernetes version has increased to Kubernetes 1.9 for all Flink versions. Earlier versions are compatible with older Kubernetes versions for Flink versions lower or equal to Flink 1.6.

Application Manager 1.3.4

Changelog

  • Add support for Flink 1.8 jobs
  • User interface
    • Add Flink 1.8.0 image tag to the image drop down

Migration

  • We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.3 installation

Application Manager 1.3.3

Changelog

  • Allow to recover from secret value resolution failure
  • Respect finite sources that finish during job startup
  • User interface
    • Add Flink 1.6.4-dap1 image tag to the image drop down
    • Add Flink 1.7.2-dap1 image tag to the image drop down

Migration

  • We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.3 installation

Application Manager 1.3.2

Changelog

  • API: Add securityContext to Kubernetes options
  • Flink 1.7 deployments
    • Implement work around for FLINK-11127 (Flink’s metric query service)
    • Don’t report job status as running before all tasks are running
    • Fix secret mounting for artifact fetcher init container
    • Don’t require JAR entryClass to be explicitly specified if JAR has manifest entry (Note: this is only supported with the 1.7.0-dap2 and 1.7.1-dap2 Flink images)
  • User interface
    • Add Flink 1.7.0-dap2 and 1.7.1-dap2 image tag to the image drop down
    • Add Flink 1.6.3-dap1 image tag to the image drop down
  • Respect JAVA_OPTS environment variable for Application Manager JVM

Migration

  • We recommend using the Helm charts using helm upgrade to upgrade your existing Application Manager 1.3 installation
  • Please follow the migration guide for 1.3.0 if you are migrating from an Application Manager 1.2 installation

Application Manager 1.3.1

Changelog

  • Add allowNonRestoredState flag to restore strategies LATEST_STATE and LATEST_SAVEPOINT
  • Flink 1.7 deployments
    • Set default image for Flink 1.7 to 1.7.1
    • Don’t use nested volume mount for SSL keys
    • Pass resource requests to job clusters
  • User interface
    • Add Flink 1.7.1 image tag to the image drop down
    • Re-add Flink 1.5 images to the image drop down
    • Dynamically query Flink for the Flink job ID instead of relying on a static mapping between Application Manager’s job resource and the Flink job

Migration

  • Please follow the migration guide for 1.3.0 and use 1.3.1 as the image tag for Application Manager (step 5 below)

Application Manager 1.3.0

Changelog

  • End-to-end Flink SSL/TLS allows you to auto provision Flink 1.6 and 1.7 jobs with SSL/TLS support, including mutual authentication.
  • New restoreStrategy API for more flexible state restore operations. This include a new mode of operation that allows you to restore your application from both savepoints as well as checkpoints. You have to enable Flink High Availability and checkpoint retention in order to enable restoring from latest checkpoints.
  • Application Manager supports Flink 1.7 and manages Flink applications using the container entrypoint. This makes the deployment of applications more robust and less time consuming. Instead of requiring Application Manager to access and submit your JAR artifact to Flink, we let Flink fetch it.
  • You can now configure custom imagePullSecrets for each Deployment, making it possible to access custom Docker registries.
  • The Settings page of the web user interface now allows you to manage more resources, such as namespaces, API tokens, and seret values.

Migration

  1. Important: Flink 1.7 jobs require entryClass to be configured in Deployment.spec.template.spec.artifact. If you are running Application Manager version greater or equal to 1.3.2 and Flink image tagged as 1.7.0-dap2 or 1.7.1-dap2 (or more recent) this requirement has been dropped. See release notes of version 1.3.2.
  2. If you are using Flink 1.6, please set your Flink applications to use the image with tag: registry.platform.data-artisans.net/trial/v1.3/flink:1.6.2-dap2-scala_2.11. The dap2 image fixes the built-in InfluxDB metrics reporter that was included in earlier versions. For Flink 1.7, all images already contain the fixed metrics reporter.
  3. In order to use an existing data volume of an earlier installation please add the following securityContext entry to your Application Manager Deployment:
kind: Deployment
spec:
  template:
    spec:
      securityContext:
      fsGroup: 102
  1. Add the following at the root level of your Application Manager ConfigMap for the appmanager.yaml entry:
cluster:
  kubernetes.artifact-fetcher.image-registry: registry.platform.data-artisans.net/trial/v1.3
  kubernetes.artifact-fetcher.image-tag: 1.0.1
  1. Update the image and args for the Application Manager Kubernetes Deployment:
kind: Deployment
metadata:
  name: daplatform-appmanager
spec:
  containers:
  - name: appmanager
    image: registry.platform.data-artisans.net/trial/v1.3/appmanager:1.3.0
    args:
      - server
      - /etc/daplatform/appmanager.yaml
  1. Give access to jobs and secrets Kubernetes resources for Role daplatform-appmanager.
kind: Role
metadata:
  name: daplatform-appmanager
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - "batch"
  resources:
  - jobs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Deprecation

  • Deployment.spec.startFromSavepoint has been deprecated in favor of Deployment.spec.restoreStrategy. Operations against startFromSavepoint are reflected in restoreStrategy and vice versa. Only when using the new LATEST_STATE restore strategy will startFromSavepoint be not available.

Application Manager 1.2.0

Changelog

  • Single Sign-on Authentication with OpenID Connect Single sign-on is implemented through OpenID Connect. Common OpenID connect providers are Google Cloud, Microsoft Azure Active Directory, Keycloak or Dex. Through Dex, further identity providers such as LDAP or SAML 2.0 are available.
  • API Tokens for Machine to Machine Authentication Application Manager now allows to create API tokens for services to authenticate. API tokens are revocable at any time. Access to the system can be restricted similarly to regular users singing on via SSO.
  • Role-based Access Control This new feature allows to restrict access to API resources by defining roles and binding them to users or groups. The access control is based on API resources such as deployments, jobs, savepoints, or events and the HTTP method, such as GET, POST, PATCH.
  • Namespaces Namespaces now offer the ability to control visibility and access of API resources. This effectively introduces multi-tenancy into Ververica Platform, so that multiple teams can share a Ververica Platform setup, with strict separation of their resources.
  • Secret Values This release introduces a new API object called “Secret Value”. A secret allows to manage passwords, authentication tokens or secret configuration parameters. In particular, this allows for separating knowledge of a secret from usage of a secret.
  • Support for Apache Flink 1.6
  • Credentials in the jar URL
  • Logging improvements
  • User Interface Improvements - Visualization of Flink Streaming Applications with metrics - YAML view of Deployments - Improved “Savepoints” tab

Application Manager 1.1.0

Changelog

  • General API: Deployment and Job resources are expanded eagerly with default values for optional fields. Prior to this version, optional fields of resources were lazily expanded which made it hard to understand the full state of a given resource. With this change, resources will always be fully specified.
  • Support for Flink 1.5: Flink 1.5 is the new default version for Deployments. You can specify which Flink version to deploy via the newly introduced flinkVersion attribute in Deployment.spec.template.spec.artifact. Note that the specified Flink version and Flink image have to match. If they do not match, deployed jobs will fail with a corresponding error message.
  • Custom Kubernetes options: We allow to pass Kubernetes-specific options to created Flink pods via Deployment.spec.template.spec.kubernetes. This gives you more control about deployed Flink jobs, for instance by attaching volume mounts or a node selector to deployed pods.
  • Specify Flink image by digest: In addition to image tags we allow to specify Flink images by image digest in Deployment.spec.template.spec.artifact.flinkImageTag. In order to specify a digest prefix the Flink image tag with @, e.g. @sha256:....
  • Support for finite Flink jobs: Finite Flink jobs such streaming applications consuming from finite sources or batch jobs transition transition to a new terminal state FINISHED. Prior to this release, any terminated job was treated as a failure and led to a job recovery (re-running the finite job).
  • Web UI: There have been additions to the web UI to support newly introduced features and multiple minor fixes.

Migration

  • When modifying an existing Deployment resource, default values will be expanded eagerly as described above. The newly introduced flinkVersion attribute will default to 1.5 which will lead to your Flink jobs to be upgraded to Flink 1.5. If you don’t want please update the Deployment accordingly with the first modification (PATCH) and set flinkVersion: 1.4.

Deprecation

  • Volume mount annotations with key alpha/k8s-volume-mounts have been deprecated with this release. Please migrate these annotations to the newly introduced Kubernetes options. Support for the annotations may be dropped in future versions.