Skip to main content

Ververica Runtime Assembly (VERA)

Ververica Runtime Assembly (VERA) is a performance-optimised Flink runtime and includes the custom enterprise-level GeminiStateBackend. In Ververica Cloud, this combination replaces vanilla Apache Flink® and the standard RocksDBStateBackend.

At the application level, Ververica Runtime Assembly (VERA) is 100% compatible with your existing Flink applications. At the same time, the optimised implementation offers significant advantages over vanilla Flink.

note

Ververica Runtime Assembly (VERA) is a core component of Ververica Cloud. Gemini is preconfigured as the default state backend.

Optimised Performance​

Depending on the use case, VERA yields up to twice the stream computing performance of Apache Flink®, and in some cases more, see the Benchmarks section.

The performance improvement is driven by the following features:

Other features​

As well as improved performance, VERA includes some other notable features:

  • Advanced CDC (CDAS & CTAS)
  • Dynamic CEP
  • Built-in Flink ML
  • Additional connectors, specifically Apache Paimon

While some of the above, for example CTAS, may come to Apache Flink® soon, VERA includes them today.

Advanced CDC​

Change Data Capture (CDC) borrows the technology used by database logs. It captures changes from upstream databases and synchronizes the changes to downstream storage in real-time and supports both full and incremental data changes.

VERA integrates with Flink CDC, an open-source CDC technology that supports a large variety of data sources. Flink CDC uniquely leverages an Incremental Snapshot Algorithm which, when paired with the Paimon connector, reads historical data and then updates incremental changes from the Binlog so the database remains unlocked, with parallel reads and automatic switching managed by a single task.

Supported upstream databases include (but are not limited to) MySQL, MariaDB, and MongoDB.

You can leverage CDC to clean, widen, and aggregate full and incremental database data using SQL, while implementing custom logic using the Flink DataStream API.

Use cases include real-time database synchronization and processing of full and incremental table data to obtain real-time consistency snapshots of each table.

To support synchronisation, VERA extends the supported SQL with Create Database as Database (CDAS) and Create Table as Select (CTAS) queries, allowing simple and fast single-statement database and table updates.

Dynamic CEP​

Complex Event Processing (CEP) allows you to dynamically configure rules for a deployment without requiring a restart. In mission-critical scenarios, uninterrupted production deployments ensure business continuity and are a significant gain.

Use cases include online real-time fraud detection, Threat Detection Service (TDS), and real-time marketing.

Flink Machine Learning (ML) is an Apache Flink® library that provides standard ML APIs and a number of ML operators. You can use it to implement ML algorithms based on standard ML APIs and build ML pipelines for training and inference deployments.

VERA is immediately ready for use, including Flink ML pre-installed and all dependencies configured.

Supporting the Ecosystem​

VERA integrates advanced features from the Flink community into production releases to keep you up to date with the latest Flink capabilities.

For example, the Apache Paimon connector is a Flink community project that aims to combine the flexibility and scalability of data lakes on cheap storage like S3 with the optimization and structured queries data warehouses offer. Paimon provides a storage layer that leverages a table format to make intermediate data directly accessible from dynamic tables, enabling high throughput writes and low latency queries in a datalake model.

Autopilot and scheduled tuning​

In most cases, a large amount of time is required for deployment tuning. For example, when you publish a draft, you must configure resources, parallelism, and the number and size of TaskManagers for the draft. When a deployment is running, you must adjust the resources of the deployment to maximize resource utilization. If backpressure occurs on the deployment or the latency increases, you must adjust the configurations of the deployment. With VERA, you can choose a tuning mode based on your business requirements. For example, Autopilot adjusts the parallelism for the deployment to meet the deployment throughput requirements. See this topic for details.

GeminiStateBackend​

Stateful computing in stream processing is complex and raises particular challenges:

  • Typical applications require large numbers of random accesses and few range queries.
  • Data traffic and hotspots change frequently, causing different parallel threads within the same operator to use different data access modes.

VERA's GeminiStateBackend is a key-value storage engine designed for stream processing where the above characteristics hold. It is designed to optimise data processing performance. Its Log-structured merge-tree (LSM) architecture provides three primary capabilities:

  • Adaptation to changes in data volume and access characteristics.
  • Tiered storage of hot and cold data.
  • Switchover between anti-caching and caching architectures.

It supports:

  • Hash storage for fast random access.
  • Compute-storage separation to eliminate local disk dependencies for state data.
  • Key-value separation for improved performance of dual-stream and multi-stream JOIN.
  • Adaptive parameter tuning to automate deployment optimisation.

Benefits of Compute-Storage Separation​

Because local disk storage is typically limited, jobs that store large amounts of state data can run out of disk space. The traditional solution for Flink jobs is to increase the parallelism of threads to reduce the storage requirement per thread.

Compute-storage separation removes the local disk dependency for storing state, preventing job failures caused by a lack of local disk storage.

Benefits of Key-Value Separation​

Dual-stream and multi-stream JOIN is slow in stream processing and is a typical cause of state storage bottlenecks. Key-value separation can improve performance significantly, yielding computing resource utilization increases of 50% on average, and of up to 100% to 200% in specific but typical cases.

Benefits of Adaptive Parameter Tuning​

In stream processing, different operators (projection, filter, joins, and grouping), have different state access modes. Typically, optimal performance of state storage requires different combinations of parameters for different operators. Optimal parameter values also depend on the underlying storage technologies. Manual parameter tuning therefore has high learning and understanding costs.

Adaptive parameter tuning sets parameters based on the current data access mode and traffic while a job is running, optimising state storage performance for the current scenario. It can typically automate up to 95% of parameter tuning that would otherwise have to be performed manually and can yield single-core throughput increases of between 10% to 40%.