Ververica DocsVerverica Docs
Docs Home

Apache Fluss™

4 min read
On this page

Apache Fluss is a unified streaming storage layer built for real-time analytics which can serve as the real-time data layer for Lakehouse architectures. Apache Fluss and its columnar stream and real-time update capabilities integrates seamlessly with Apache Flink to bridge the gap between data streaming and data Lakehouse by enabling low-latency, high-throughput data ingestion and processing.

With Apache Fluss, storing data and making decisions with that data can happen in real time.

How Apache Fluss Fits into Ververica's Ecosystem

Apache Fluss addresses critical challenges in real-time data processing and storage, enhancing the capabilities of Ververica’s Unified Streaming Data Platform by making it a scalable, unified batch and streaming data solution.

Apache Fluss

Key Features

The key Apache Fluss features and capabilities leveraged in the Ververica ecosystem include:

Fast Data Delivery for Time-Sensitive Applications

Apache Fluss ensures sub-second latency streaming reads and writes, enabling immediate read and write operations for fast, actionable insights. Ideal for time-sensitive applications like monitoring and financial platforms, it delivers data as soon as it's ingested.

Unified Processing for Streaming and Batch Workloads

Apache Fluss provides a storage layer optimized for real-time analytics, supporting both streaming and batch workloads to enable a unified approach to data processing. This integration eliminates the complexity of managing disparate storage systems and enhances Apache Flink’s performance in handling both real-time and historical data. Apache Fluss is ideal for organizations requiring a single platform for diverse data analytics needs.

  • By integrating real-time and historical data processing in one platform, Apache Fluss optimizes infrastructure for AI and ML workloads. Its sub-second latency ensures actionable insights for applications such as fraud detection, recommendation engines, and predictive analytics. This capability makes Apache Fluss an ideal solution for organizations seeking both real-time and historical data processing.
  • Because it supports bi-directional communication with lakehouses like Apache Paimon and Apache Iceberg, Apache Fluss enables seamless state initialization and synchronization between batch and streaming jobs. This integration bridges the gap between traditional data lakes and modern stream processing systems, ensuring a cohesive, unified platform for complex data workloads.

Full Visibility into Stream Changes

Apache Fluss provides the ability to view and process data as both a continuous stream of events (a stream) and as a dynamically evolving dataset (a table). This "stream-table duality" capability allows for seamless transitions between the two paradigms, enabling real-time analytics and stateful processing. Because Apache Fluss efficiently manages updates with comprehensive changelogs, it ensures consistent data flow and provides full visibility into stream changes for accurate real-time and historical insights within the same system.

Benefits of stream-table duality include:

  • Real-Time Analytics: It enables systems to perform both real-time data processing (stream) and stateful aggregations or lookups (table) simultaneously.
  • Simplified Data Handling: Developers can work with data as streams or tables depending on the use case, reducing the need for complex data pipelines.
  • Efficient Updates: With changelog streams, only incremental changes are propagated, improving performance and reducing resource consumption.
  • Consistency: Ensures that the stateful computations maintain accuracy and consistency across both views.

Reduced Complexity and Cost

With features like projection pushdown, columnar streaming reads, interactive queries, and native Flink integration, Apache Fluss improves peformance, simplifies the architecture and development, and reduces costs.

  • Apache Fluss uses projection pushdown to optimize streaming reads, fetching only the necessary fields for queries. This reduces data transfer, improving performance up to 10x and lowering network costs.
  • With columnar streaming reads, Apache Fluss enhances performance by storing data in a columnar format. This improves compression and speeds up analytics, making it perfect for data-heavy, real-time applications.
  • Apache Fluss is fully queryable, enabling direct data inspection without extra processing layers. This reduces development complexity, simplifies debugging, and allows for immediate access to live data insights.
  • With its native integration with Flink, Apache Fluss eliminates the need for intermediate Kafka topics and additional OLAP systems. This reduces infrastructure costs, simplifies pipeline architecture, and enhances scalability.

Integration with Emerging Technologies

Apache Fluss provides a scalable, adaptable, and forward-looking data infrastructure designed to support evolving technology trends and business needs.

  • By supporting advanced AI, machine learning, and analytics workloads, Apache Fluss enables your business to leverage cutting-edge tools and methodologies without overhauling your data architecture.
  • Because it integrates seamlessly with popular data lakehouse systems (like Apache Paimon and Apache Iceberg), Apache Fluss ensures that your business can adapt to the latest data management paradigms without disrupting existing workflows.
  • By delivering efficient data processing and reduced network costs, Apache Fluss provides a cost-effective solution, especially for data-intensive industries like finance, retail, and technology.
  • Operating on open-source principles, Apache Fluss ensures your business retains control and avoids vendor lock-in while benefiting from Ververica’s commercial enhancements and support.

Apache Fluss Connector and Catalog

The Apache Fluss DataStream Connector integrates Apache Fluss directly with Apache Flink’s DataStream API, enabling developers to build low-level, fine-grained streaming applications on top of Apache Fluss.

The Apache Fluss Catalog provides a unified metadata layer that makes it easy to manage and discover data stored in Apache Fluss from within Apache Flink and other compatible systems. It acts as a bridge between the Apache Fluss log storage and SQL-based processing by exposing topics, schemas, and configurations as relational objects such as databases and tables.

Related Topics

For more information about Apache Fluss, see:

Was this helpful?