Skip to main content

Configure Deployment Resources (Expert Mode)

Expert mode is a fine-grained resource allocation method that lets you configure resources for each Slot Sharing Group (SSG) in a SQL deployment. For complex deployments, the default resource mode may result in low resource utilization. Expert mode lets you tune resources per operator group, and optionally per individual operator, to match your workload requirements.

When to Use Expert Mode​

Use expert mode when you need to:

  • Configure JobManager and TaskManager resources independently.
  • Tune parallelism for specific operator groups.
  • Set different state expiration times (TTL) for individual operators.

Configure Expert Mode Resources​

Expert mode is configured from the Resources section of the deployment configuration.

  1. In the Console navigation pane, click Deployments.

  2. Click the name of the SQL deployment you want to configure.

  3. Open the Configuration tab and expand the Resources section.

  4. Configure the following parameters:

    ParameterDescription
    JobManager CPUThe number of CPU cores for the JobManager. Minimum: 0.25. Maximum: 16.
    JobManager MemoryThe amount of memory for the JobManager in GiB. Minimum: 1 GiB. Maximum: 64 GiB.
    TaskManager SlotsThe number of task slots per TaskManager.

  5. Click Get Plan Now to generate an execution plan for your deployment. The plan displays each SSG as a slot in the graph.

  6. To modify a slot's parallelism, click the Edit button on the slot. The parallelism you set applies to all operators in the SSG. Memory is calculated automatically based on the deployment's computational logic.

  7. Click Save.

note

To load a previously saved plan, click Import Plan. To save the current plan as a JSON file, click Export Plan.

note

For source nodes, ensure the parallelism is a divisor of the number of source partitions. For example, for a Kafka topic with 16 partitions, set parallelism to 16, 8, or 4.

Configure Operator Resources (Multiple SSG Mode)​

By default, all operators share one SSG and cannot be configured independently. To configure resources for individual operators, enable Multiple SSG mode, which assigns each operator its own slot.

  1. In the Resources section, enable the Multiple SSG toggle.
  2. Click Re-fetch to regenerate the execution plan with per-operator slots.
  3. In the execution plan graph, click the Edit icon on the operator you want to configure.
  4. In the Modify Operator dialog, set the parameters described in the Parameters section.
  5. Click Save.

Parameters​

ParameterDescription
ParallelismThe parallelism of the operator.
Chaining StrategyHow the operator chains with adjacent operators. ALWAYS (default): the operator chains with both upstream and downstream operators. HEAD: the operator is the head of a chain; upstream operators are disconnected from it. NEVER: the operator does not chain with any adjacent operators.
State Expiration TimeThe TTL for this operator's state, configurable in seconds, minutes, hours, or days. Defaults to the deployment-level expiration time. If no deployment-level TTL is configured, defaults to 1.5 days.

Why Use Per-Operator State TTL​

By default, a single state expiration time applies to all operators in a job. For jobs where different operators have very different state-retention requirements, a single TTL forces a trade-off: either keep all state too long and waste resources, or expire it too soon and break correctness for operators that need longer retention.

For example, consider a job that computes the real-time relationship between orders and logistics data:

  • The order operator processes high-volume, short-lived events. Order state becomes stale quickly and can be expired soon after an order is shipped.
  • The logistics operator tracks updates that can span weeks or months. Its state must be retained much longer to associate late-arriving logistics updates with the correct order.

With a single job-level TTL, you cannot satisfy both requirements simultaneously. Per-operator state TTL lets you set a short expiration time on the order operator and a long expiration time on the logistics operator within the same job.

Limitations​

  • Only stateful operators support the State Expiration Time setting. For stateless operators, the Modify Operator dialog shows only Parallelism and Chaining Strategy.
  • Applies to SQL deployments only.
On this page