Skip to main content

Ecosystem: Apache Paimon

Apache Paimon is a key component of the Ververica ecosystem, enhancing its capabilities for stream processing and data management by integrating seamlessly with Apache Flink®. It plays a pivotal role in the Streamhouse concept and the broader data infrastructure offered by Ververica.

How Apache Paimon Fits into Ververica's Ecosystem​

Ververica builds on Paimon to expand its ecosystem by providing a robust, stream-native storage layer that enhances Flink’s capabilities. It enables Ververica to offer comprehensive solutions for real-time data processing, advanced analytics, and zero-trust-compliant architectures, aligning with enterprise needs for modern data infrastructure.

Apache Paimon

Key Features​

The key Paimon features and capabilities leveraged in the Ververica ecosystem include:

Unified Data Lakehouse Storage​

Apache Paimon serves as a stream-native data lakehouse that combines the benefits of data lakes (scalability and flexibility) and data warehouses (structured querying and schema enforcement). It bridges the gap between batch and streaming data by providing:

  • Real-time ingestion that supports Flink Change Data Capture (CDC) for incremental updates.
  • Batch and OLAP processing to optimize storage for analytical queries and batch processing, enabling unified workloads.

Integration with Other Technologies​

Apache Paimon integrates tightly with Apache Flink. Flink can use Paimon as both a source (to read data) and a sink (to write data). This integration allows Flink jobs to process data in real-time, update Paimon tables incrementally, and query those tables for analytical insights. It also supports schema evolution and time travel queries, enhancing flexibility in managing data lifecycle changes.

In Ververica’s Streamhouse architecture, Apache Paimon acts as the storage layer, providing low-latency reads and writes for both streaming and batch use cases. By supporting unified stream-batch processing, Flink can seamlessly transition between real-time streams and historical data analysis.

Data Sovereignty and Governance​

Apache Paimon aligns with Ververica's focus on data sovereignty and compliance by keeping data storage within the customer's cloud environment, ensuring that organizations retain full control over their data. It also enforces granular access policies and integrates with cloud-native security tools.

Scalability and Cost Efficiency​

Paimon handles massive-scale data workloads and can support enterprises operating in hybrid or multi-cloud environments. Its optimization for streaming reduces costs associated with traditional batch-processing architectures.

Developer-Friendly Design​

Apache Paimon supports tools and APIs familiar to developers working with Flink, making it easier to adopt within Ververica Unified Streaming Data Platform and simplifying operational complexity for managing data pipelines.

For more information on Apache Paimon, see:

See also Apache Flink, Streamhouse, and Flink CDC.

On this page