Skip to main content

Ververica Platform 3.0.0

Release Notes​

Ververica Platform 3.0.0 introduces major architectural improvements for building, operating, and governing streaming data pipelines. This release supports VERA 4.1 and Apache Flink® 1.20.

Overview​

This release marks a significant shift in the platform architecture with the introduction of VERA (Ververica Runtime Assembly). VERA is an enhanced, Flink-compatible engine designed for large-scale, cloud-native deployments. It enables a Streamhouse architecture that unifies real-time and historical analytics while maintaining open-core principles.

Ververica Platform 3.0.0 maintains full API and SQL compatibility with standard Apache Flink while providing the enhanced performance and elasticity required for enterprise-grade environments.

For details on performance metrics, see the Ververica Platform 3.0 Overview.

Platform Improvements​

Dynamic Parameter Updates Without Restart​

Certain configuration changes can now be applied without restarting the job. This includes parameters for parallelism, checkpoint intervals, timeouts, and minimum pause settings. This allows you to perform quick tuning and troubleshooting without incurring downtime, reloading state, or replaying large data backlogs.

Declarative CDC Pipelines in YAML​

You can now define Change Data Capture (CDC) pipelines declaratively in YAML. This improvement moves configuration away from UI-only workflows, making your jobs version-controlled in Git, reusable across environments (dev/staging/prod), and readable by a wider audience including data engineers and DevOps teams.

Interactive Table & Column-Level Lineage​

A new lineage explorer provides end-to-end table-level lineage across pipelines. It includes column-level lineage tracking for sensitive fields (e.g., PII), allowing you to see upstream and downstream flows, including transformations and associated deployments.

Autopilot 2.0 Intelligent Resource Optimization​

Autopilot 2.0 extends autoscaling beyond Kafka-only sources to work with any Flink source, including CDC, JDBC, files, and REST APIs. The optimization logic analyzes CPU, memory, latency, backpressure, and processing time to adjust both parallelism and memory. You can select between two strategies:

  • Adaptive for workloads with spiky or unpredictable traffic.
  • Stable for steady workloads where minimal restarts and consistent behavior are required.

Scheduled Tuning for Predictable Workloads​

You can now define time-based scaling profiles to align resources with predictable business patterns. You can define schedules to scale resources up during known peak hours (e.g., 9:00–19:00) and scale down during off-peak periods. These profiles support daily, weekly, or monthly schedules to optimize your infrastructure costs.

Deployment Versioning and One-Click Rollback​

The platform now creates an immutable version record for every deployment. You can view a history of changes, compare side-by-side diffs between versions, pin specific stable "known good" versions, and perform one-click rollbacks to previous configurations. This provides a clear audit trail for platform, operations, and governance teams.

Separated Startup vs. Runtime Logs​

Logs are now clearly split into two categories to simplify first-failure analysis:

  • Startup logs cover initialization, configuration, and bootstrapping.
  • Runtime logs cover steady-state execution and processing errors.

SQL Validation with Full Semantics​

The SQL editor now performs full semantic validation against catalogs and connectors prior to deployment. This validation catches missing or unknown tables, incorrect column names, and connector misconfigurations, eliminating common runtime failures and stabilizing CI/CD pipelines.

Auto-Saved SQL Drafts​

SQL drafts are auto-saved on every keystroke. Closing the browser, switching tabs, or losing connectivity no longer risks losing work. Your team can safely work on longer queries and complex refactors without manual save rituals.

Unified Launch Point for All Deployments​

Management of JAR, Python, and SQL deployments is now consolidated into the Deployments page. You no longer need to switch between multiple sections to deploy different artifact types, simplifying user journeys and onboarding.

Safe Job Cancellation With State Preservation​

The platform introduces new options for stopping jobs that preserve state and data integrity:

  • Stop with Savepoint halts the job after creating a savepoint, allowing it to be resumed later with full state.
  • Stop with Drain ensures the job processes all backlog events before stopping, preventing the loss of in-flight data.

SQL Templates for Faster Onboarding​

The SQL editor now includes built-in templates for DDL and table creation, windowed aggregations, and Joins/Top-N queries. These templates encode best practices and reduce the time required for new users to become productive.

Folder Trees and Multi-Tab Editing​

You can organize SQL scripts in nested folder trees by domain, project, or environment. The editor also supports opening and editing multiple scripts in tabs simultaneously, making large projects manageable and reducing time spent searching for the right script.

Debug Mode With Mock Data and Live Charts​

A sandboxed debug mode allows you to upload mock data (e.g., CSV) and execute SQL queries without writing to production sinks. You can inspect live charts, grids, and outputs to validate query logic, shortening the feedback loop during development.

Streaming Data Grid for Real-Time Results​

A real-time data grid renders streaming query results as they arrive, allowing you to verify aggregation logic, filters, and joins immediately without waiting for a full pipeline execution.

Comments, Labels, and Session Cluster Routing​

You can add comments to deployments to capture rationale ("why we configured this this way") and use key-value labels for grouping and filtering by owner, team, or domain. Labels can also be used to route deployments to dedicated session clusters, improving long-term maintainability.

Deep-Linking Between SQL and Deployments​

From a deployment, you can jump directly to the backing SQL script, and from the SQL editor, you can jump to the deployments using that script. This removes the friction of searching for the connection between your code and your running jobs.

Configurable SQL Editor​

You can configure the SQL editor to match your preferences, including font size, line numbers, word wrap, mini-map visibility, and auto-save delay. This improves usability for different working styles and screen sizes.

Named Parameters in UDFs​

User-defined functions (UDFs) now support named parameters. This allows for order-independent arguments and optional parameters without positional NULLs, making queries more readable and safer to refactor.

AI & RAG Support Directly in SQL​

Ververica Platform 3.0.0 adds native AI inference and Retrieval-Augmented Generation (RAG) primitives to the SQL layer. You can register AI models using standard SQL syntax (CREATE MODEL) and invoke them from streaming SQL to perform tasks such as real-time sentiment analysis, embedding generation for semantic search, and vector similarity lookups.

In-Platform Notification Manager​

An in-product notification center streams job and deployment events in real-time. You can filter by status (Running, Succeeded, Failed) and deep-link directly to affected deployments, replacing manual polling and speeding up incident response.

Unified Event View​

A consolidated event view merges platform, operational, and deployment events into a single timeline. This assists you in correlating failures with infrastructure changes significantly faster.

Centralized JobManager & TaskManager Logs​

A unified logs interface provides access to JobManager and TaskManager logs, metrics, configuration, thread dumps, and memory graphs directly within the platform.

Memory Monitoring and Diagnostics​

The platform provides detailed visibility into heap and off-heap memory usage per task and checkpoint. This data supports the analysis of memory-related failures and enables data-driven tuning of checkpoint configurations to prevent garbage collection issues.

Failed TaskManager Archive​

Logs for failed TaskManagers are now archived and retained within the UI. This allows you to perform forensic analysis on transient infrastructure issues even after the associated pods have been terminated.

Full-Screen Metrics Mode​

Any metrics or I/O chart can now be expanded to full-screen mode. This improves readability for incident "war rooms", troubleshooting sessions, and executive reviews.

Search and Auto-Focus in Lineage Graphs​

You can search for any table or column to jump directly to the relevant node in complex lineage graphs. Auto-focus helps you navigate large environments quickly.

Rich Metadata on Hover / Double-Click​

Lineage nodes now display detailed metadata on hover or double-click, including connector types, database names, modification times, and ownership. You can use this to deep-link to the corresponding deployment or artifact.

Field and Node Filtering​

You can filter lineage graphs by specific fields (e.g., PII columns) or specific operators to analyze performance characteristics like CPU, memory usage, and latency.

Lineage Export​

You can export lineage data and related artifacts to JSON or CSV formats for auditing purposes, allowing you to share lineage snapshots with compliance teams without manual redraws.

VERA Engine Improvements​

State Management and Performance​

Gemini State Backend​

The new Gemini state backend significantly improves performance for state-heavy workloads. Benchmarks indicate up to 97% faster snapshots, 32% faster scale-ups, and 47% faster scale-downs. This backend reduces job upgrade interruption times to sub-second pauses.

Tiered Storage​

State management now supports tiered storage, automatically moving hot data to memory or SSDs while offloading cold data to object storage. This prevents local disk exhaustion during emergency rescaling and supports large state sizes without requiring manual storage capacity planning.

Adaptive Key/Value Separation​

The engine enables the separation of keys and values in state storage to optimize caching. This allows the system to avoid reading full records for non-matching joins, potentially doubling performance for low match-rate workloads such as fraud detection. The engine automatically adapts its management of hot and cold data based on access patterns.

Data Architecture​

Streamhouse Architecture​

VVP 3.0 enables a Streamhouse architecture that unifies real-time streaming and lakehouse analytics. Streaming data can be written directly into open formats like Apache Paimon and Iceberg, allowing real-time and historical queries to be run against the same tables. This architecture eliminates the need for duplicated batch and streaming pipelines and leverages ACID semantics within the lakehouse layer.

Unified Data Movement​

Support for CDAS (Create Database As Select) and CTAS (Create Table As Select) commands allows for data movement between systems using single SQL statements. The platform manages schema evolution, offset tracking, and delivery guarantees, simplifying streaming ETL workflows.

SQL and Processing​

Dynamic CEP Rules​

Complex Event Processing (CEP) patterns can now be stored in external tables, such as JDBC or MySQL sources. Running jobs periodically fetch updated rules, allowing logic for use cases like fraud and anomaly detection to evolve without requiring job redeployments.

Execution Optimizations​

Enhancements to the Flink SQL execution path provide faster processing for complex joins and aggregations. These performance improvements apply automatically to SQL pipelines without requiring code changes.

Migration​

If you are planning to upgrade your existing environment to Ververica Platform 3.0.0, review the following requirements carefully to ensure a smooth transition.

important

You must be on VVP 2.15.x running Flink 1.20 to be able to migrate to 3.0. Ensure you have reached this specific baseline before initiating an upgrade.

You are advised to test upgrades in non-production environments before deploying to production, particularly when enabling VERA or Autopilot 2.0 for the first time. Also note that Ververica Platform 2.x licenses are not valid for use on Ververica Platform 3.0.x. You will need to obtain a new license to install Ververica Platform 3.0.x. Please contact your Account Executive for detailed migration guidance.

On this page