Upgrade the Ververica Agent
Applies toBYOC
2 min read
On this page
Upgrading an existing Ververica Agent takes more than a helm upgrade. Some versions change the agent's custom resources or the capabilities of existing workspaces, and you apply those changes yourself as part of the upgrade. This page covers the upgrade from 2.x to 2.5.1-2.
Before you upgrade
Contact Ververica support before you upgrade a production cluster, and run the procedure in a non-production cluster first.
Upgrade from 2.x to 2.5.1-2
Version 2.5.1-2 adds two capabilities to the agent. Workspaces created before the upgrade don't pick them up automatically, so after the Helm upgrade you update the capability keys of each existing workspace and restart the affected pods.
tenantComponentMonitor, introduced in 2.5.1-byoc, applies to AWS and Azure.securityTokenServiceForTenantRole, introduced in 2.5.1-1-byoc, applies to AWS only.
The patch script in step 3 works out which keys each workspace needs, so you don't pick them by hand.
Before you begin
- Helm 3.x.
kubectlaccess to the cluster, with your current kubeconfig context pointing at the BYOC cluster you want to upgrade. Check it withkubectl config current-context.jqon the machine that runs the patch script.
Step 1. Download the chart
BASH
1export VV_AGENT_VERSION=2.5.1-2-byoc
2
3helm pull oci://registry.ververica.cloud/agent-charts/vv-agent \
4 --version ${VV_AGENT_VERSION}Step 2. Upgrade the Helm release
BASH
1helm upgrade <release-name> ./vv-agent-${VV_AGENT_VERSION}.tgz \
2 -n <agent-namespace> \
3 --reuse-values--reuse-values keeps your existing configuration. Complete this step before you run the patch script, which checks that the upgrade has happened and stops if it hasn't.
Step 3. Update the capability keys of existing workspaces
Workspaces created before the upgrade keep the capability keys they had when they were created. The patch script compares each workspace with the configuration the new release installed, and adds or removes keys to match.
Save the script from the end of this page as patch-controlplane-reconcilable-capability.sh and make it executable:
BASH
1chmod +x patch-controlplane-reconcilable-capability.shFirst, check what would change. The dry run only reads and doesn't modify anything:
BASH
1./patch-controlplane-reconcilable-capability.sh -n <agent-namespace> --dry-runFor each workspace, the report lists the keys it has, the keys the new release expects, and what would be added or removed. A workspace marked capability-key-set-in-sync needs no change.
Then add the missing keys to every affected workspace:
BASH
1./patch-controlplane-reconcilable-capability.sh \
2 -n <agent-namespace> --all-affected -a addIf the report also lists keys under remove, run the same command with -a remove.
To update a single workspace instead, pass its control plane name with -c. The control plane name isn't always the workspace ID, so list the names first:
BASH
1kubectl get controlplane -n <agent-namespace>
2
3./patch-controlplane-reconcilable-capability.sh \
4 -n <agent-namespace> -c <control-plane-name> -a addRun the dry run again when you're done. Every workspace now reports capability-key-set-in-sync.
While it updates a workspace, the script temporarily switches on an internal safety setting and restores its previous value afterwards. If another change reaches the workspace at the same moment, it retries up to three times. If it can't restore the setting, it stops, in batch mode as well, and reports which workspace was affected. In that case, contact Ververica support before you continue.
Step 4. Restart the affected pods
The new capabilities only apply to pods created after the update. Existing pods keep running without the new configuration, so restart them for each updated workspace:
BASH
1# vvp-sql, a StatefulSet, is recreated automatically
2kubectl delete pod vvp-sql-0 -n flink-<workspace-id>-vvp-deploy
3
4# Flink JobManager and TaskManager pods
5kubectl delete pod -l app=flink -n <workload-namespace>Verify the upgrade
Run the dry run from step 3. Every workspace reports capability-key-set-in-sync.
Roll back
BASH
1helm rollback <release-name> <previous-revision> -n <agent-namespace>Then restart the affected pods as described in step 4.
The patch script
Save this script as patch-controlplane-reconcilable-capability.sh. Run it with --help for the full list of options.
BASH
1#!/usr/bin/env bash
2
3set -euo pipefail
4
5KUBECTL=${KUBECTL:-kubectl}
6ACTION=""
7CAPABILITY_KEY=""
8NAMESPACE=""
9CONTROL_PLANE=""
10ALL_AFFECTED=false
11VC_TYPE="flink-vc"
12SKIP_PREREQ_CHECK=false
13DRY_RUN=false
14SAFE_MODE_KEY="pyxis.control-plane.safemode"
15EXTENSIONS_JSON=""
16RESTORE_ACTIVE=false
17RESTORE_CP=""
18RESTORE_PATCH=""
19RESTORE_ALLOW_REMOVE_FALLBACK=false
20
21usage() {
22 cat <<'EOF'
23Usage:
24 Single: patch-controlplane-reconcilable-capability.sh -n <ns> -c <cp> -a <add|remove> [-k <key>]
25 Batch: patch-controlplane-reconcilable-capability.sh -n <ns> --all-affected -a <add|remove> [-k <key>] [--type <type>]
26 Dry-run: patch-controlplane-reconcilable-capability.sh -n <ns> --dry-run [-c <cp> | --all-affected] [--type <type>]
27
28Required:
29 -n, --namespace <namespace> ControlPlane namespace
30 -a, --action <add|remove> Action to perform (required outside dry-run)
31 -c, --control-plane <name> ControlPlane name (single mode, mutually
32 exclusive with --all-affected)
33
34Options:
35 -k, --capability-key <key> Capability key to add or remove
36 (optional: if omitted, auto-detect diff
37 from extensions config)
38 --all-affected Patch all Workspaces whose reconcilableCapabilityKeys
39 differ from the extensions config
40 --type <type> Workspace type label to filter (default: flink-vc)
41 Used with batch mode or dry-run
42 --dry-run Show diff report without making any changes
43 (incompatible with -a, -k and prereq options)
44 --skip-prereq-check Skip ConfigMap and parent-chain prechecks
45 -h, --help Show this help
46
47Run this script after applying a target Helm release whenever
48pyxis-extensions changes the desired reconcilableCapabilityKeys for existing
49Workspaces. Use add for keys introduced by the target release and remove for
50keys no longer supported by the target release. Always run dry-run first.
51
52Examples:
53 # Target release adds a specific key to one Workspace
54 $0 -n ververica -c my-workspace -a add -k securityTokenServiceForTenantRole
55
56 # Target release removes stale keys from one Workspace
57 $0 -n ververica -c my-workspace -a remove
58
59 # Target release removes all stale keys from affected Workspaces
60 $0 -n ververica --all-affected -a remove
61
62 # Dry-run: show what would change without making any changes
63 $0 -n ververica --dry-run
64 $0 -n ververica --dry-run --all-affected
65 $0 -n ververica --dry-run -c my-workspace
66EOF
67}
68
69log() {
70 printf '[INFO] %s\n' "$*"
71}
72
73warn() {
74 printf '[WARN] %s\n' "$*" >&2
75}
76
77die() {
78 printf '[ERROR] %s\n' "$*" >&2
79 exit 1
80}
81
82# ── argument parsing ──────────────────────────────────────
83
84while [[ $# -gt 0 ]]; do
85 case "$1" in
86 -n|--namespace)
87 [[ $# -ge 2 ]] || die "$1 requires a value"
88 NAMESPACE=$2; shift 2 ;;
89 -c|--control-plane)
90 [[ $# -ge 2 ]] || die "$1 requires a value"
91 CONTROL_PLANE=$2; shift 2 ;;
92 -a|--action)
93 [[ $# -ge 2 ]] || die "$1 requires a value"
94 ACTION=$2; shift 2 ;;
95 -k|--capability-key)
96 [[ $# -ge 2 ]] || die "$1 requires a value"
97 CAPABILITY_KEY=$2; shift 2 ;;
98 --skip-prereq-check)
99 SKIP_PREREQ_CHECK=true; shift ;;
100 --all-affected)
101 ALL_AFFECTED=true; shift ;;
102 --dry-run)
103 DRY_RUN=true; shift ;;
104 --type)
105 [[ $# -ge 2 ]] || die "$1 requires a value"
106 VC_TYPE=$2; shift 2 ;;
107 -h|--help)
108 usage; exit 0 ;;
109 *)
110 die "unknown argument: $1" ;;
111 esac
112done
113
114[[ -n "$NAMESPACE" ]] || die "namespace is required (-n)"
115if [[ "$DRY_RUN" == true ]]; then
116 [[ -z "$ACTION" ]] || die "--dry-run cannot be combined with -a/--action"
117 [[ -z "$CAPABILITY_KEY" ]] || die "--dry-run cannot be combined with -k/--capability-key"
118 [[ "$SKIP_PREREQ_CHECK" == false ]] || die "--dry-run cannot be combined with --skip-prereq-check"
119fi
120if [[ "$DRY_RUN" == false ]]; then
121 [[ -n "$ACTION" ]] || die "action is required (-a add or -a remove)"
122fi
123if [[ -n "$ACTION" ]]; then
124 [[ "$ACTION" == "add" || "$ACTION" == "remove" ]] || die "action must be 'add' or 'remove', got: $ACTION"
125fi
126
127if [[ "$ALL_AFFECTED" == true ]]; then
128 [[ -z "$CONTROL_PLANE" ]] || die "--all-affected and -c are mutually exclusive"
129elif [[ "$DRY_RUN" == false ]]; then
130 [[ -n "$CONTROL_PLANE" ]] || die "control-plane name is required (-c) or use --all-affected"
131fi
132
133command -v "$KUBECTL" >/dev/null 2>&1 || die "kubectl command not found: $KUBECTL"
134command -v jq >/dev/null 2>&1 || die "jq is required"
135
136# ── helpers ───────────────────────────────────────────────
137
138get_cp() {
139 "$KUBECTL" -n "$NAMESPACE" get controlplane "$1" -o json
140}
141
142load_extensions() {
143 [[ -n "$EXTENSIONS_JSON" ]] && return 0
144 EXTENSIONS_JSON=$("$KUBECTL" -n "$NAMESPACE" get configmap pyxis-extensions -o json 2>/dev/null) || {
145 EXTENSIONS_JSON=""
146 return 1
147 }
148}
149
150extension_definition_contains_key() {
151 local key=$1 vc_type=$2
152 load_extensions || return 1
153 jq -e --arg key "$key" --arg type "$vc_type" '
154 [.data[$type] | fromjson | .defaultSpec.reconcilableCapabilityKeys[]?] | index($key) != null
155 ' <<<"$EXTENSIONS_JSON" >/dev/null
156}
157
158parent_chain_contains_capability() {
159 local vc_name=$1 key=$2
160 local current=$vc_name cp_json parent dependencies
161 local found_capability=false found_dependency=false depth=0
162
163 while [[ -n "$current" && "$depth" -lt 20 ]]; do
164 cp_json=$(get_cp "$current") || return 2
165
166 if jq -e --arg key "$key" \
167 '.spec.capabilities // [] | any(.key == $key)' <<<"$cp_json" >/dev/null; then
168 found_capability=true
169 fi
170
171 dependencies=$(jq -r '.spec.config["pyxis.control-plane.dependencies"] // empty' <<<"$cp_json")
172 if [[ -n "$dependencies" ]] &&
173 jq -e --arg key "$key" '
174 type == "object" and
175 (has($key) or any(.[]; type == "array" and index($key) != null))
176 ' <<<"$dependencies" >/dev/null 2>&1; then
177 found_dependency=true
178 fi
179
180 parent=$(jq -r '.spec.parent // empty' <<<"$cp_json")
181 current=$parent
182 depth=$((depth + 1))
183 done
184
185 [[ "$found_capability" == true && "$found_dependency" == true ]]
186}
187
188parent_chain_references_key() {
189 local vc_name=$1 key=$2
190 local current=$vc_name cp_json parent dependencies depth=0
191
192 while [[ -n "$current" && "$depth" -lt 20 ]]; do
193 cp_json=$(get_cp "$current") || return 2
194
195 dependencies=$(jq -r '.spec.config["pyxis.control-plane.dependencies"] // empty' <<<"$cp_json")
196 if [[ -n "$dependencies" ]]; then
197 jq -e 'type == "object"' <<<"$dependencies" >/dev/null 2>&1 || return 2
198 if jq -e --arg key "$key" '
199 has($key) or any(.[]; type == "array" and index($key) != null)
200 ' <<<"$dependencies" >/dev/null 2>&1; then
201 return 0
202 fi
203 fi
204
205 parent=$(jq -r '.spec.parent // empty' <<<"$cp_json")
206 current=$parent
207 depth=$((depth + 1))
208 done
209
210 [[ -z "$current" ]] || return 2
211 return 1
212}
213
214# ── dry-run: report extensions keys and per-ControlPlane diff ──
215
216dry_run_report() {
217 load_extensions ||
218 die "failed to get ConfigMap $NAMESPACE/pyxis-extensions"
219
220 log "dry-run: read-only report; no ControlPlane will be patched"
221 printf '\nExtensions reconcilableCapabilityKeys:\n'
222 if ! jq -er '
223 .data | to_entries | sort_by(.key)[] |
224 .key as $type |
225 (.value | fromjson | (.defaultSpec.reconcilableCapabilityKeys // []) |
226 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end) as $keys |
227 " \($type): \($keys | tojson)"
228 ' <<<"$EXTENSIONS_JSON"; then
229 die "pyxis-extensions contains invalid JSON or reconcilableCapabilityKeys"
230 fi
231
232 local vc_list
233 if [[ -n "$CONTROL_PLANE" ]]; then
234 vc_list=$CONTROL_PLANE
235 elif [[ -n "$VC_TYPE" ]]; then
236 vc_list=$("$KUBECTL" -n "$NAMESPACE" get controlplane \
237 -l "type=$VC_TYPE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null) ||
238 die "failed to list ControlPlanes with label type=$VC_TYPE"
239 else
240 vc_list=$("$KUBECTL" -n "$NAMESPACE" get controlplane \
241 -o jsonpath='{.items[*].metadata.name}' 2>/dev/null) ||
242 die "failed to list ControlPlanes"
243 fi
244
245 printf '\nControlPlane diffs:\n'
246 if [[ -z "$vc_list" ]]; then
247 printf ' (no ControlPlanes found)\n'
248 return 0
249 fi
250
251 local vc_name vc_json vc_type current_keys desired_keys current_set desired_set
252 local add_keys remove_keys current_duplicates desired_duplicates status
253 local total=0 changed=0 unavailable=0
254 for vc_name in $vc_list; do
255 total=$((total + 1))
256 vc_json=$(get_cp "$vc_name") || {
257 warn "$vc_name: failed to read"
258 unavailable=$((unavailable + 1))
259 continue
260 }
261 vc_type=$(jq -r '.metadata.labels.type // empty' <<<"$vc_json")
262 if ! current_keys=$(jq -ce '
263 (.spec.reconcilableCapabilityKeys // []) |
264 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end
265 ' <<<"$vc_json" 2>/dev/null); then
266 printf ' %s (type=%s): invalid current reconcilableCapabilityKeys\n' \
267 "$vc_name" "${vc_type:-<missing>}"
268 unavailable=$((unavailable + 1))
269 continue
270 fi
271
272 if [[ -z "$vc_type" ]] || ! desired_keys=$(jq -ce --arg type "$vc_type" '
273 .data[$type] | fromjson | (.defaultSpec.reconcilableCapabilityKeys // []) |
274 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end
275 ' <<<"$EXTENSIONS_JSON" 2>/dev/null); then
276 printf ' %s (type=%s): extensions config unavailable\n' \
277 "$vc_name" "${vc_type:-<missing>}"
278 printf ' current: %s\n' "$current_keys"
279 unavailable=$((unavailable + 1))
280 continue
281 fi
282
283 current_set=$(jq -c 'unique' <<<"$current_keys")
284 desired_set=$(jq -c 'unique' <<<"$desired_keys")
285 current_duplicates=$(jq -c 'group_by(.) | map(select(length > 1) | .[0])' <<<"$current_keys")
286 desired_duplicates=$(jq -c 'group_by(.) | map(select(length > 1) | .[0])' <<<"$desired_keys")
287 add_keys=$(jq -nc --argjson desired "$desired_set" --argjson current "$current_set" \
288 '$desired - $current')
289 remove_keys=$(jq -nc --argjson desired "$desired_set" --argjson current "$current_set" \
290 '$current - $desired')
291 status="capability-key-set-in-sync"
292 if [[ "$(jq 'length' <<<"$add_keys")" -gt 0 || "$(jq 'length' <<<"$remove_keys")" -gt 0 ||
293 "$(jq 'length' <<<"$current_duplicates")" -gt 0 || "$(jq 'length' <<<"$desired_duplicates")" -gt 0 ]]; then
294 status="capability-keys-different"
295 changed=$((changed + 1))
296 fi
297
298 printf ' %s (type=%s, source=pyxis-extensions.data.%s): %s\n' \
299 "$vc_name" "$vc_type" "$vc_type" "$status"
300 printf ' current: %s\n' "$current_keys"
301 printf ' desired: %s\n' "$desired_keys"
302 printf ' add: %s\n' "$add_keys"
303 printf ' remove: %s\n' "$remove_keys"
304 if [[ "$current_duplicates" != "[]" || "$desired_duplicates" != "[]" ]]; then
305 printf ' duplicate-current: %s\n' "$current_duplicates"
306 printf ' duplicate-desired: %s\n' "$desired_duplicates"
307 fi
308 done
309
310 printf '\nSummary: total=%d, different=%d, unavailable=%d\n' \
311 "$total" "$changed" "$unavailable"
312}
313
314restore_active_safemode() {
315 [[ "$RESTORE_ACTIVE" == true ]] || return 0
316 if ! "$KUBECTL" -n "$NAMESPACE" patch controlplane "$RESTORE_CP" \
317 --type=json -p "$RESTORE_PATCH" >/dev/null 2>&1; then
318 if [[ "$RESTORE_ALLOW_REMOVE_FALLBACK" == true ]]; then
319 local current_cp fallback_patch
320 current_cp=$(get_cp "$RESTORE_CP" 2>/dev/null) || {
321 warn "$RESTORE_CP: failed to restore internal safety state"
322 return 1
323 }
324 if ! jq -e --arg smk "$SAFE_MODE_KEY" '
325 (.spec.config | type == "object") and (.spec.config | has($smk))
326 ' <<<"$current_cp" >/dev/null; then
327 RESTORE_ACTIVE=false
328 warn "$RESTORE_CP: original internal safety state could not be restored; the temporary switch is already absent"
329 return 0
330 fi
331 fallback_patch='[{"op":"remove","path":"/spec/config/pyxis.control-plane.safemode"}]'
332 if "$KUBECTL" -n "$NAMESPACE" patch controlplane "$RESTORE_CP" \
333 --type=json -p "$fallback_patch" >/dev/null 2>&1; then
334 RESTORE_ACTIVE=false
335 warn "$RESTORE_CP: original internal safety state could not be restored; removed the temporary switch instead"
336 return 0
337 fi
338 fi
339 warn "$RESTORE_CP: failed to restore internal safety state; stop processing"
340 return 1
341 fi
342 RESTORE_ACTIVE=false
343 log "$RESTORE_CP: restored internal safety state"
344}
345
346cleanup() {
347 local exit_code=$?
348 trap - EXIT
349 if [[ "$RESTORE_ACTIVE" == true ]]; then
350 warn "$RESTORE_CP: process exited while internal safety restoration was pending; retrying"
351 restore_active_safemode || exit_code=1
352 fi
353 exit "$exit_code"
354}
355
356trap cleanup EXIT
357trap 'exit 130' INT
358trap 'exit 143' TERM
359
360check_key_prerequisite() {
361 local vc_name=$1 vc_type=$2 key=$3 check_rc
362 [[ "$SKIP_PREREQ_CHECK" == false ]] || return 0
363
364 if [[ "$ACTION" == "add" ]]; then
365 log "$vc_name: checking target release prerequisites for $key"
366 extension_definition_contains_key "$key" "$vc_type" || {
367 warn "$vc_name: pyxis-extensions.data.$vc_type does not contain $key; apply the target Helm release first"
368 return 1
369 }
370 parent_chain_contains_capability "$vc_name" "$key" || {
371 warn "$vc_name: parent chain does not contain $key capability+dependency; verify the target Helm release"
372 return 1
373 }
374 else
375 log "$vc_name: checking target release has removed $key from dependencies"
376 if parent_chain_references_key "$vc_name" "$key"; then
377 warn "$vc_name: $key is still referenced in parent chain dependencies; apply the target Helm release first or use --skip-prereq-check"
378 return 1
379 else
380 check_rc=$?
381 if [[ "$check_rc" -eq 2 ]]; then
382 warn "$vc_name: could not validate parent-chain dependencies for $key"
383 return 1
384 fi
385 fi
386 fi
387}
388
389# Patch all selected keys on one ControlPlane in a single atomic JSON Patch.
390patch_vc_keys() {
391 local vc_name=$1 cp_json=$2 current_keys=$3 keys_to_process=$4
392 local final_keys key key_count resource_version config_field_present config_present
393 local config_value sm_present sm_value vc_type
394 local test_op sm_op cap_op patch verify_ok=false restore_ok=false
395
396 key_count=$(jq 'length' <<<"$keys_to_process")
397 vc_type=$(jq -r '.metadata.labels.type // empty' <<<"$cp_json")
398 while IFS= read -r key; do
399 check_key_prerequisite "$vc_name" "$vc_type" "$key" || return 1
400 done < <(jq -r '.[]' <<<"$keys_to_process")
401
402 # The operator updates ControlPlanes continuously, so re-read right before
403 # each attempt and retry when the resourceVersion test op goes stale.
404 local attempt max_attempts=3 patched=false
405 for ((attempt = 1; attempt <= max_attempts; attempt++)); do
406 cp_json=$(get_cp "$vc_name") || {
407 warn "$vc_name: failed to re-read ControlPlane before patching"
408 return 1
409 }
410 current_keys=$(jq -ce '
411 (.spec.reconcilableCapabilityKeys // []) |
412 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end
413 ' <<<"$cp_json" 2>/dev/null) || {
414 warn "$vc_name: invalid current reconcilableCapabilityKeys"
415 return 1
416 }
417
418 if [[ "$ACTION" == "add" ]]; then
419 final_keys=$(jq -nc --argjson current "$current_keys" --argjson selected "$keys_to_process" \
420 '$current + ($selected - $current)')
421 else
422 final_keys=$(jq -nc --argjson current "$current_keys" --argjson selected "$keys_to_process" \
423 '$current - $selected')
424 fi
425
426 if jq -e '.spec | has("config") and .config != null and (.config | type != "object")' \
427 <<<"$cp_json" >/dev/null; then
428 warn "$vc_name: spec.config is not an object; refusing to overwrite it"
429 return 1
430 fi
431
432 config_field_present=false
433 jq -e '.spec | has("config")' <<<"$cp_json" >/dev/null && config_field_present=true
434 config_present=false
435 jq -e '.spec | has("config") and (.config | type == "object")' <<<"$cp_json" >/dev/null &&
436 config_present=true
437 sm_present=false
438 jq -e --arg smk "$SAFE_MODE_KEY" '(.spec.config // {}) | has($smk)' <<<"$cp_json" >/dev/null &&
439 sm_present=true
440
441 if [[ "$config_present" == true ]]; then
442 sm_op='{"op":"add","path":"/spec/config/pyxis.control-plane.safemode","value":"true"}'
443 else
444 sm_op='{"op":"add","path":"/spec/config","value":{"pyxis.control-plane.safemode":"true"}}'
445 fi
446 cap_op=$(jq -nc --argjson keys "$final_keys" \
447 '{"op":"add","path":"/spec/reconcilableCapabilityKeys","value":$keys}')
448
449 resource_version=$(jq -r '.metadata.resourceVersion // empty' <<<"$cp_json")
450 if [[ -n "$resource_version" ]]; then
451 test_op=$(jq -nc --arg rv "$resource_version" \
452 '{"op":"test","path":"/metadata/resourceVersion","value":$rv}')
453 patch=$(jq -nc --argjson test "$test_op" --argjson sm "$sm_op" --argjson cap "$cap_op" \
454 '[$test, $sm, $cap]')
455 else
456 patch=$(jq -nc --argjson sm "$sm_op" --argjson cap "$cap_op" '[$sm, $cap]')
457 fi
458
459 if [[ "$sm_present" == true ]]; then
460 sm_value=$(jq -c --arg smk "$SAFE_MODE_KEY" '.spec.config[$smk]' <<<"$cp_json")
461 RESTORE_PATCH=$(jq -nc --argjson value "$sm_value" \
462 '[{"op":"add","path":"/spec/config/pyxis.control-plane.safemode","value":$value}]')
463 elif [[ "$config_present" == true ]]; then
464 RESTORE_PATCH='[{"op":"remove","path":"/spec/config/pyxis.control-plane.safemode"}]'
465 elif [[ "$config_field_present" == true ]]; then
466 config_value=$(jq -c '.spec.config' <<<"$cp_json")
467 RESTORE_PATCH=$(jq -nc --argjson value "$config_value" \
468 '[{"op":"add","path":"/spec/config","value":$value}]')
469 else
470 RESTORE_PATCH='[{"op":"remove","path":"/spec/config"}]'
471 fi
472 RESTORE_ALLOW_REMOVE_FALLBACK=true
473 if [[ "$sm_present" == true ]] && jq -e --arg smk "$SAFE_MODE_KEY" '
474 (.spec.config[$smk] | tostring | ascii_downcase) == "true"
475 ' <<<"$cp_json" >/dev/null; then
476 RESTORE_ALLOW_REMOVE_FALLBACK=false
477 fi
478 RESTORE_CP=$vc_name
479 RESTORE_ACTIVE=true
480
481 log "$vc_name: atomically ${ACTION}ing $key_count key(s) (attempt $attempt/$max_attempts)"
482 if "$KUBECTL" -n "$NAMESPACE" patch controlplane "$vc_name" \
483 --type=json -p "$patch" >/dev/null; then
484 patched=true
485 break
486 fi
487 warn "$vc_name: kubectl patch failed (likely a concurrent update changed resourceVersion)"
488 restore_active_safemode || return 1
489 done
490 [[ "$patched" == true ]] || return 1
491
492 get_cp "$vc_name" | jq -e --argjson expected "$final_keys" \
493 '(.spec.reconcilableCapabilityKeys // []) == $expected' >/dev/null && verify_ok=true
494 restore_active_safemode && restore_ok=true
495
496 if [[ "$verify_ok" == true && "$restore_ok" == true ]]; then
497 log "$vc_name: reconcilableCapabilityKeys updated successfully"
498 return 0
499 fi
500 [[ "$verify_ok" == true ]] || warn "$vc_name: patch succeeded but key verification failed"
501 return 1
502}
503
504# ── process_vc: determine keys, patch once per ControlPlane ──
505# Args: $1=vc_name, $2=vc_json
506# Sets: _pc_patched, _pc_failed, _pc_unavailable
507
508process_vc() {
509 local vc_name=$1 vc_json=$2
510 _pc_patched=0
511 _pc_failed=0
512 _pc_unavailable=0
513
514 local keys_to_process current_keys desired_keys vc_type key_count
515
516 if ! current_keys=$(jq -ce '
517 (.spec.reconcilableCapabilityKeys // []) |
518 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end
519 ' <<<"$vc_json" 2>/dev/null); then
520 warn "$vc_name: invalid current reconcilableCapabilityKeys"
521 _pc_unavailable=1
522 return 0
523 fi
524
525 if [[ -n "$CAPABILITY_KEY" ]]; then
526 keys_to_process=$(jq -nc --arg key "$CAPABILITY_KEY" '[$key]')
527 if [[ "$ACTION" == "add" ]]; then
528 keys_to_process=$(jq -nc --argjson selected "$keys_to_process" --argjson current "$current_keys" \
529 '$selected - $current')
530 else
531 keys_to_process=$(jq -nc --argjson selected "$keys_to_process" --argjson current "$current_keys" \
532 '$selected - ($selected - $current)')
533 fi
534 else
535 vc_type=$(jq -r '.metadata.labels.type // empty' <<<"$vc_json")
536 if [[ -z "$vc_type" ]]; then
537 warn "$vc_name: no type label, cannot auto-detect; skipping"
538 _pc_unavailable=1
539 return 0
540 fi
541
542 load_extensions || {
543 warn "$vc_name: failed to get pyxis-extensions configmap; skipping"
544 _pc_unavailable=1
545 return 0
546 }
547 desired_keys=$(jq -ce --arg type "$vc_type" '
548 .data[$type] | fromjson | (.defaultSpec.reconcilableCapabilityKeys // []) |
549 if type == "array" then . else error("reconcilableCapabilityKeys is not an array") end
550 ' <<<"$EXTENSIONS_JSON" 2>/dev/null) || {
551 warn "$vc_name: missing or invalid extensions config for type $vc_type; skipping"
552 _pc_unavailable=1
553 return 0
554 }
555
556 if [[ "$ACTION" == "add" ]]; then
557 keys_to_process=$(jq -nc --argjson d "$desired_keys" --argjson c "$current_keys" '$d - $c')
558 else
559 keys_to_process=$(jq -nc --argjson d "$desired_keys" --argjson c "$current_keys" '$c - $d')
560 fi
561 fi
562
563 key_count=$(jq 'length' <<<"$keys_to_process")
564 if [[ "$key_count" -eq 0 ]]; then
565 log "$vc_name: nothing to do"
566 return 0
567 fi
568
569 log "$vc_name: ${ACTION}ing $key_count key(s): $(jq -r 'join(", ")' <<<"$keys_to_process")"
570
571 if patch_vc_keys "$vc_name" "$vc_json" "$current_keys" "$keys_to_process"; then
572 _pc_patched=$key_count
573 else
574 _pc_failed=$key_count
575 fi
576}
577
578# ── main ─────────────────────────────────────────────────
579
580if [[ "$DRY_RUN" == true ]]; then
581 dry_run_report
582 exit 0
583fi
584
585if [[ "$ALL_AFFECTED" == true ]]; then
586 log "scanning for affected Workspaces (type=$VC_TYPE) in namespace $NAMESPACE"
587
588 vc_list=$("$KUBECTL" -n "$NAMESPACE" get controlplane \
589 -l "type=$VC_TYPE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null) ||
590 die "failed to list ControlPlanes with label type=$VC_TYPE"
591
592 if [[ -z "$vc_list" ]]; then
593 log "no ControlPlanes found with label type=$VC_TYPE in namespace $NAMESPACE"
594 exit 0
595 fi
596
597 total=0 patched=0 skipped=0 failed=0 unavailable=0
598
599 for vc_name in $vc_list; do
600 total=$((total + 1))
601 vc_json=$(get_cp "$vc_name") || { warn "failed to get $vc_name, skipping"; failed=$((failed + 1)); continue; }
602
603 process_vc "$vc_name" "$vc_json"
604 patched=$((patched + _pc_patched))
605 failed=$((failed + _pc_failed))
606 unavailable=$((unavailable + _pc_unavailable))
607 [[ "$_pc_patched" -eq 0 && "$_pc_failed" -eq 0 && "$_pc_unavailable" -eq 0 ]] &&
608 skipped=$((skipped + 1))
609 if [[ "$RESTORE_ACTIVE" == true ]]; then
610 warn "$vc_name: stopping batch because internal safety restoration is still pending"
611 break
612 fi
613 done
614
615 log "batch completed: total=$total, patched=$patched, skipped=$skipped, failed=$failed, unavailable=$unavailable"
616 [[ "$failed" -eq 0 && "$unavailable" -eq 0 ]] || exit 1
617 exit 0
618fi
619
620# single ControlPlane mode
621cp_json=$(get_cp "$CONTROL_PLANE") || die "ControlPlane $NAMESPACE/$CONTROL_PLANE not found"
622
623process_vc "$CONTROL_PLANE" "$cp_json"
624log "completed: patched=$_pc_patched, failed=$_pc_failed, unavailable=$_pc_unavailable"
625[[ "$_pc_failed" -eq 0 && "$_pc_unavailable" -eq 0 ]] || exit 1
626exit 0Was this helpful?