SOP: Aerospike Node Unquiesce Using Ansible Automation Platform (AAP)

1. Purpose

To automate the recovery of an Aerospike node that remains quiesced after a Google Cloud Platform (GCP) live migration event. This procedure replaces the existing manual unquiesce operation while ensuring the node's current state is validated before any action is taken.

2. Scope

This procedure covers:

• Investigation of persistent node-quiesced alerts.
• Validation of the target node's quiesce state.
• Execution of the Aerospike undo-quiesce command.
• Recluster initiation when required.
• Post-operation validation and status reporting.

This procedure does not initiate or manage GCP live migration.

3. Trigger Conditions

Initiate this procedure when:

1. A node-quiesced alert occurs following a GCP live migration event.
2. Automatic recovery does not clear the alert.
3. The alert persists for approximately 45 minutes.
4. The target node is confirmed to require manual recovery.

Do not execute the template solely because a GCP migration event occurred. Always validate the current Aerospike state first.

4. Prerequisites

Before executing the AAP job, ensure that:

• The relevant incident has been raised.
• The target node IP address is available.
• The target Aerospike cluster is identified.
• The AAP execution environment has access to the target cluster.
• The configured Aerospike account has the required privileges to execute quiesce management commands.
• The operator has reviewed any active cluster-health alerts.

5. Procedure

Step 1: Identify the Target Node

Obtain the target node IP address from the alert or incident.

Provide the IP address through the AAP job template input variable:

node_to_unquiesce

The playbook maps the supplied IP address to the corresponding Aerospike node name using info network.

Step 2: Collect Initial Quiesce Statistics

The playbook executes the following Aerospike commands:

info network show statistics like quiesce 

The output is used to identify the target node and determine its current quiesce state.

The following statistics are evaluated:

The playbook must map the target node IP to its node name before selecting the corresponding column from the statistics output.

Step 3: Evaluate the Current State

The playbook checks the target node's current state before executing undo quiesce.

Important: The value of pending_quiesce must be interpreted together with effective_is_quiesced and the observed behavior of the deployed Aerospike version. A pending_quiesce=true value alone does not necessarily indicate a stuck transition.

This validation prevents the template from unnecessarily executing undo quiesce when a webhook or another automation has already performed the recovery.

Step 4: Execute Undo Quiesce

If the checks confirm that the node requires unquiesce, execute:

enable manage quiesce undo with <node-ip> 

The command requests that the effect of quiesce be reverted at the next recluster event.

The playbook must:

• Execute the command using the configured Aerospike credentials.
• Verify the command's return code and response.
• Record the result without exposing credentials.
• Stop and report failure if the command is unsuccessful.

Step 5: Trigger Recluster

After a successful undo-quiesce command, execute:

manage recluster 

Reclustering allows the cluster to apply the updated quiesce state.

The playbook should record the command result and allow sufficient time for the cluster state to settle before performing the final checks.

Step 6: Verify the Final State

Execute the following commands again:

show statistics like quiesce info network 

Verify that:

• The target node's effective_is_quiesced value is false.
• The final pending_quiesce value is consistent with the expected behavior for the deployed Aerospike version.
• The target node is present in the cluster as expected.
• Cluster membership and integrity are healthy.
• Any unexpected