Skip to main content

Ecosystem: Fluss

Fluss is a unified streaming storage layer built for real-time analytics which can serve as the real-time data layer for Lakehouse architectures. Fluss and its columnar stream and real-time update capabilities integrates seamlessly with Apache Flink to bridge the gap between data streaming and data Lakehouse by enabling low-latency, high-throughput data ingestion and processing.

With Fluss, storing data and making decisions with that data can happen in real time.

How Fluss Fits into Ververica's Ecosystem​

Fluss addresses critical challenges in real-time data processing and storage, enhancing the capabilities of Ververica’s Unified Streaming Data Platform by making it a scalable, unified batch and streaming data solution.

Fluss

Key Features​

The key Fluss features and capabilities leveraged in the Ververica ecosystem include:

Fast Data Delivery for Time-Sensitive Applications​

Fluss ensures sub-second latency streaming reads and writes, enabling immediate read and write operations for fast, actionable insights. Ideal for time-sensitive applications like monitoring and financial platforms, it delivers data as soon as it's ingested.

Unified Processing for Streaming and Batch Workloads​

Fluss provides a storage layer optimized for real-time analytics, supporting both streaming and batch workloads to enable a unified approach to data processing. This integration eliminates the complexity of managing disparate storage systems and enhances Apache Flink’s performance in handling both real-time and historical data. Fluss is ideal for organizations requiring a single platform for diverse data analytics needs.

  • By integrating real-time and historical data processing in one platform, Fluss optimizes infrastructure for AI and ML workloads. Its sub-second latency ensures actionable insights for applications such as fraud detection, recommendation engines, and predictive analytics. This capability makes Fluss an ideal solution for organizations seeking both real-time and historical data processing.
  • Because it supports bi-directional communication with lakehouses like Apache Paimon and Apache Iceberg, Fluss enables seamless state initialization and synchronization between batch and streaming jobs. This integration bridges the gap between traditional data lakes and modern stream processing systems, ensuring a cohesive, unified platform for complex data workloads.

Full Visibility into Stream Changes​

Fluss provides the ability to view and process data as both a continuous stream of events (a stream) and as a dynamically evolving dataset (a table). This "stream-table duality" capability allows for seamless transitions between the two paradigms, enabling real-time analytics and stateful processing. Because Fluss efficiently manages updates with comprehensive changelogs, it ensures consistent data flow and provides full visibility into stream changes for accurate real-time and historical insights within the same system.

Benefits of stream-table duality include:

  • Real-Time Analytics: It enables systems to perform both real-time data processing (stream) and stateful aggregations or lookups (table) simultaneously.
  • Simplified Data Handling: Developers can work with data as streams or tables depending on the use case, reducing the need for complex data pipelines.
  • Efficient Updates: With changelog streams, only incremental changes are propagated, improving performance and reducing resource consumption.
  • Consistency: Ensures that the stateful computations maintain accuracy and consistency across both views.

Reduced Complexity and Cost​

With features like projection pushdown, columnar streaming reads, interactive queries, and native Flink integration, Fluss improves peformance, simplifies the architecture and development, and reduces costs.

  • Fluss uses projection pushdown to optimize streaming reads, fetching only the necessary fields for queries. This reduces data transfer, improving performance up to 10x and lowering network costs.
  • With columnar streaming reads, Fluss enhances performance by storing data in a columnar format. This improves compression and speeds up analytics, making it perfect for data-heavy, real-time applications.
  • Fluss is fully queryable, enabling direct data inspection without extra processing layers. This reduces development complexity, simplifies debugging, and allows for immediate access to live data insights.
  • With its native integration with Flink, Fluss eliminates the need for intermediate Kafka topics and additional OLAP systems. This reduces infrastructure costs, simplifies pipeline architecture, and enhances scalability.

Integration with Emerging Technologies​

Fluss provides a scalable, adaptable, and forward-looking data infrastructure designed to support evolving technology trends and business needs.

  • By supporting advanced AI, machine learning, and analytics workloads, Fluss enables your business to leverage cutting-edge tools and methodologies without overhauling your data architecture.
  • Because it integrates seamlessly with popular data lakehouse systems (like Apache Paimon and Apache Iceberg), Fluss ensures that your business can adapt to the latest data management paradigms without disrupting existing workflows.
  • By delivering efficient data processing and reduced network costs, Fluss provides a cost-effective solution, especially for data-intensive industries like finance, retail, and technology.
  • Operating on open-source principles, Fluss ensures your business retains control and avoids vendor lock-in while benefiting from Ververica’s commercial enhancements and support.

For more information about Fluss, see:

See also Apache Paimon and Apache Flink.

On this page