Resource Allocation and Queues

Ververica Cloud provides granular control over computing resources through a simplified allocation system. This topic explains how to use Queues to manage resource consumption, ensuring stability and fairness across your workloads.

Overview

Queues are logical resource pools within a workspace that control how compute resources are allocated for deployments and session clusters. Every workspace provides a built-in default-queue. You can create additional queues to further segment resources, support specific batch workloads, and manage job priorities.

Why Use Queues?

Queues allow you to isolate and control resource consumption for different types of jobs, users, or workloads. Using queues helps prevent:

• Resource Monopolization: Prevents a single resource-intensive deployment from consuming all available Workspace Compute Units (CUs).
• Resource Starvation: Ensures critical deployments have reserved resources, preventing execution failures caused by insufficient capacity.
• Workload Conflict: Eliminates unfair resource competition between batch and real-time workloads.

By assigning batch deployments to dedicated queues and defining scheduling priorities, you can ensure that critical workloads receive higher priority and adequate resources, improving overall system stability and resource utilization.

Managing Queues

Scaling Resources

Adjusting resource allocation requires a specific order of operations to ensure data integrity and quota availability:

• Scaling Up (Top-Down): To add resources to a queue, you must first ensure the Workspace has sufficient quota, and then assign those resources to the Queue.
• Scaling Down (Bottom-Up): To free up resources, you must first reduce the allocation of individual Queues. This returns CUs to the Workspace, which can then be reduced (e.g., by adjusting the quota ceiling).

Creating a Queue

To create a new resource queue:

1. Go to Queue Management.
2. Click Add Queue.
3. Enter a name in the Queue Name field, and the number of Resources (CUs) that you need, and then click OK.

Editing a Queue

To edit an existing queue:

1. Go to Queue Management.
2. Next to the name of the queue you need to edit, click Edit queue.
3. Edit the number in the Resources (CU) field, and then click OK.

Using Queues

You assign workloads to specific queues during their creation or configuration.

• Deployments: In the Deploy Draft dialog, use the drop down in the Deployment Target field to select a queue.
• Session Clusters: In the Create Session Cluster page, select the target queue from the configuration menu.
• Workflows: In the Create Workflow panel, use the Resource Queue parameter to assign the entire workflow to a queue.

note

A queue cannot be deleted if it currently contains active deployments. You must stop or reassign the deployments before deleting the queue.