Metadata Database Operations

Ververica Platform stores its control-plane state in an external relational database. This page covers what that database holds, what accumulates in it over time, and how to control that growth, reclaim disk space, back it up, and monitor it. For the installation-time configuration of the database itself, see PostgreSQL as Metadata Store.

What the Metadata Database Holds

You configure the metadata database at installation time, through the global.database block of the platform's Helm values. Ververica Platform supports PostgreSQL 15 or later, MySQL 8.x, and MariaDB 11.4.5 or later. PostgreSQL as Metadata Store covers the supported versions, the Helm fields, TLS, manual database creation, and air-gapped installations.

The platform's services each manage their own database on the shared instance. Together those databases hold the control-plane state: your deployments and their configuration history, the jobs that have run for each deployment, the event log you see in the user interface, and the metadata for savepoints and checkpoints.

The vvp-appmanager database holds deployment history, so it accounts for nearly all organic growth. The others stay roughly proportional to the number of users, catalogs, and policies you define. None of these databases store savepoint or checkpoint data itself, which lives in your configured blob storage.

Impact on Running Flink Jobs

Your running Flink jobs never read from or write to the metadata database. The Flink clusters handle data processing, state access, and checkpointing entirely on their own, writing checkpoint and savepoint data to your configured blob storage, and Flink's own failure recovery operates inside the cluster independently of the platform. A slow or temporarily unavailable metadata database does not slow down record processing, does not delay checkpoints, and does not cause running jobs to fail.

Metadata database degradation affects only your ability to manage those jobs. The database backs the control plane, so degradation shows up as a slow user interface, slow API responses, and sluggish deployment state transitions. Ververica Platform might delay actions such as starting or upgrading a deployment, suspending a job with a savepoint, or applying an autoscaling decision until the database is healthy again, and platform upgrade windows grow with table size. During a database outage, running jobs keep processing, but you cannot deploy, upgrade, cancel, or otherwise operate them through the platform until the database recovers.

What Accumulates Over Time

Everything Ververica Platform records about a deployment accumulates for as long as that deployment exists.

Configuration History

Every modification to a deployment creates a new configuration version, visible in the deployment's history in the user interface. Where many teams make frequent changes, and especially with CI/CD pipelines that redeploy on every commit, this is typically the largest long-term contributor to database growth.

Jobs and Events

Each restart, upgrade, or state transition adds job records and event entries for the deployment. A deployment that runs smoothly generates little. A deployment stuck in a restart loop can generate a great deal in a short time, which is one more reason to address unstable deployments promptly.

Snapshot Metadata

Savepoint and checkpoint records accumulate per deployment. This is the one area with a documented retention mechanism that you control, described in the next section.

Control What Accumulates

Snapshot Cleanup Rules

On a deployment's State configuration, you can enable scheduled automatic cleaning of savepoints, capping them by the number of historical snapshots to keep and by a maximum age in hours. Ververica Platform then deletes savepoints automatically whenever either condition is met. You can also delete snapshot resources individually from the user interface or the REST API. See Snapshots.

If you schedule periodic savepoints on a deployment, always pair them with these cleanup rules. Make the cleanup rules part of your standard deployment checklist for every long-running job.

Deployment Lifecycle Hygiene

Deleting a deployment removes its configuration history, job records, and events. This makes deployment lifecycle management your main lever for controlling long-term growth, and it matters most in development and test namespaces, which tend to collect abandoned deployments. Three practices work well:

• Adopt a labeling convention that distinguishes disposable deployments from durable ones, for example an env label.
• Run a periodic sweep, a simple script against the REST API, that deletes cancelled or failed deployments in development namespaces that nobody has touched in a defined number of days.
• For CI/CD-heavy teams, consider whether every pipeline run needs to modify a persistent deployment, because each modification adds a permanent history entry for the life of that deployment. Short-lived deployments that you create and later sweep might fit the workflow better.

Estimate Storage Growth

Growth depends on how your teams use the platform: how often they modify deployments, how stable those deployments are, and what snapshot schedules they set. Measuring your own installation and extrapolating gives you a more reliable figure than any generic estimate.

Establish a Baseline

After the platform has run for a few weeks, capture the size of each platform database and its largest tables, using the queries in Monitor the Metadata Database below, and repeat weekly. Two or three months of trend data gives you an environment-specific growth rate that already reflects how your teams work.

Planning Figures

For initial sizing, before you have measurements, these order-of-magnitude figures work reasonably well. Individual event and snapshot-metadata records are small, a kilobyte or two each. Configuration versions and job records carry a copy of the deployment's resolved configuration and typically run from ten to a few tens of kilobytes, growing with the complexity of your deployment specifications, such as pod templates, Flink configuration, and environment settings.

Example

Take an organization of ten teams with five developers each, where those 50 developers average four configuration changes per working day across the estate, roughly 50,000 modifications per year. At approximately 20 KB per configuration version, that is about 1 GB of configuration history per year, before indexes and storage-engine overhead, plus the job, event, and snapshot activity that accompanies it.

The shape matters more than the total. Growth scales with change activity and deployment count, so an installation that is tidy about its deployment lifecycle grows slowly, while one that accumulates abandoned deployments and redeploys on every commit grows several times faster with the same team size.

Provision the Volume

Whatever your year-one estimate, provision the database volume with generous headroom. Around ten times is a sensible starting point. That headroom is not padding: write-ahead or binary logs, temporary sort space, and table rebuilds during maintenance all draw on the same volume. Alert at 70% volume utilization and treat 85% as an incident, because an out-of-space condition affects every platform service at once.

Budget log space deliberately as well. On MySQL and MariaDB, set binlog_expire_logs_seconds to match your actual replication and point-in-time-recovery needs rather than leaving the default. On PostgreSQL, watch write-ahead log accumulation, particularly from replication slots that have fallen behind or been abandoned, which retain the log indefinitely until you drop them.

Reclaim Disk Space

On every backend, deleting rows does not by itself return disk space to the filesystem. The mechanics differ by engine.

On PostgreSQL

PostgreSQL handles deleted rows through vacuuming. Deletes and updates leave dead tuples behind, and autovacuum reclaims them for reuse within the table. Confirm that autovacuum is enabled, which it is by default, and let it do its work. For the platform's busier tables, you can monitor dead-tuple counts through pg_stat_user_tables and tighten the per-table autovacuum thresholds if dead tuples consistently accumulate faster than autovacuum clears them.

Ordinary vacuuming makes space reusable but does not shrink the files on disk. After a large one-off purge, such as a mass deletion of old deployments, check the bloat on the affected tables. If a large table carries substantial reclaimable space, compact it with VACUUM FULL during a maintenance window, which rewrites the table and takes an exclusive lock, or with an online tool such as pg_repack if a window is hard to schedule. Run ANALYZE afterwards so the planner has fresh statistics. Routine VACUUM FULL provides no benefit, and it only pays off after substantial purges.

Two PostgreSQL-specific points round this out. Watch write-ahead log disk usage and replication slots, because an orphaned slot silently retains the log until the volume fills. And size shared_buffers, along with the surrounding memory settings, so the platform's working set stays cached, which is what keeps the platform's list views fast as history grows.

On MySQL and MariaDB

InnoDB does not shrink its files when you delete rows. Deployment deletions and snapshot cleanup leave free space inside the tablespaces, which InnoDB reuses for new data, but the files on disk never get smaller on their own. This is normal, and most of the time you do not need to take action.

To reclaim disk after a large purge, first check whether it is worth it: compare data_free against the table size in information_schema.TABLES. If a large table carries more than 20 to 30% free space, rebuild it with OPTIMIZE TABLE during a maintenance window, where the rebuild takes time proportional to the table size and briefly locks the table, or use an online rebuild tool such as pt-online-schema-change. Ensure that innodb_file_per_table is enabled, the default on MySQL 8.x, because per-table reclamation is otherwise impossible. Run ANALYZE TABLE on the larger tables after major purges, and size the InnoDB buffer pool to hold the platform's working set.

Back Up and Restore

Take periodic backups of the underlying databases, and capture all of the platform's databases together so that they stay mutually consistent. PostgreSQL as Metadata Store lists the service databases to include.

On PostgreSQL, use pg_dump and pg_restore for self-managed instances, or the automated snapshot features of managed services such as AWS RDS, Azure Database for PostgreSQL, or Google Cloud SQL. On MySQL and MariaDB, take a full-instance snapshot, or a dump of all platform databases at the same point in time.

Platform Upgrades

Platform upgrades run schema migrations against these databases. On a well-kept installation this is quick, but very large tables extend the maintenance window, which is one more reason the lifecycle hygiene above pays off. Check your largest table sizes when planning an upgrade, and take a fresh backup immediately before it.

Monitor the Metadata Database

A small set of signals covers this well. Track the size of the platform's databases and their largest tables on a weekly trend.

On PostgreSQL

1-- Database sizes
2SELECT datname, pg_size_pretty(pg_database_size(datname)) AS size
3FROM   pg_database
4WHERE  datname IN ('vvp-appmanager','vvp-autopilot','vvp-meta','vvp-gateway',
5                   'vvp-advisor','vvp-premise','vvp-k8soperator','accesscontrol')
6ORDER  BY pg_database_size(datname) DESC;
7
8-- Largest tables and dead-tuple counts
9-- (run per database, for example connected to vvp-appmanager)
10SELECT relname,
11       n_live_tup, n_dead_tup,
12       pg_size_pretty(pg_total_relation_size(relid)) AS total_size
13FROM   pg_stat_user_tables
14ORDER  BY pg_total_relation_size(relid) DESC
15LIMIT  20;

On MySQL and MariaDB

1SELECT table_schema, table_name,
2       table_rows,
3       ROUND((data_length+index_length)/1024/1024) AS size_mb,
4       ROUND(data_free/1024/1024)                  AS free_mb
5FROM   information_schema.TABLES
6WHERE  table_schema NOT IN ('mysql','sys','information_schema','performance_schema')
7ORDER  BY (data_length+index_length) DESC
8LIMIT  20;

What Else to Watch

Alongside table sizes, watch volume utilization, log volume (the write-ahead log on PostgreSQL, including replication-slot retention, or the binary log directory on MySQL and MariaDB), the connection count against the configured limit, and, as a leading indicator on the platform side, the responsiveness of the deployment list and event views in the user interface.

Reasonable starting thresholds: alert when the total grows more than 30% month over month while your deployment count stays flat, which usually signals a restart loop or a runaway modification pipeline rather than organic growth; when volume utilization passes 70%; or when list views become noticeably slow. To investigate growth, look at which tables are growing and correlate that with platform activity. An unstable deployment, a chatty pipeline, or missing snapshot cleanup rules are the usual explanations.

Quick Reference