Airlock Gateway failover cluster upgrade with full system installation

The following procedure describes how to upgrade an Airlock Gateway cluster in a production environment, with minimized downtime. This may be required if it is not possible to update the Airlock Gateway installation of the nodes, e.g. if an upgrade to a new major version of Airlock Gateway requires a complete reinstallation of the nodes.

For standard update procedure, see Section – Updates.

A functional Airlock Gateway cluster consists of two different nodes, one active and one passive node. Under normal operating conditions, both nodes are online (attached to the cluster) with the active node handling the traffic and the passive node performing health checks (see drawing).

• The current working state of a node is displayed in the upper left corner of the Airlock Gateway Configuration Center and can assume:
• Active – the node is online and handling the traffic.
• Passive – the node is online and performing health checks on the active node, ready to take over the traffic if the Active node fails.
• Offline – the node is not attached to the cluster. In this state, the node is neither Active nor Passive.

If Airlock Anomaly Shield is enabled on your system, be sure to back up the ColdDB data using the Copy command before performing an Airlock Gateway system upgrade. It may be necessary to re-train the machine learning models of the upgraded Airlock gateways based on the ColdDB data. See also Release notes.

• Procedure summary:
• We will start with upgrading the passive node Bob (see Part 1), so that the traffic is handled by the active node Alice without interruption.
• After upgrading node Bob with the new Airlock Gateway release, Bob has to be synchronized (see Part 2) with the active node Alice. This prepares Bob to take over the traffic from Alice.
• In Part 3, Alice will be set offline (detached from the cluster), causing the upgraded and synchronized node Bob to take over the traffic.
• In Part 4 we upgrade node Alice and import Bob's cluster configuration.
• Finally, the cluster configuration must be applied to both updated nodes (see Part 5). After Alice is set back online, Alice enters the passive state and the full cluster functionality is restored.

• An Airlock Gateway cluster is configured and is fully functional.

• Goals in this section:
• Export and save Bob's current node configuration and SSH keys.
• Bring Bob offline (that detaches the node from the cluster). Alice stays active and handles the traffic.
• Reinstall Airlock Gateway on node Bob and import Bob's old configuration.
• Export and save Bob's node configuration and SSH keys for later use.

1. Log in to the Airlock Gateway Configuration Center on the node Bob. Go to:
   Configuration >> Configuration Files
2. In the section Export current configuration, tick Include sensitive data such as private keys and passphrases and export and save the configuration.
3. Go to:
   System Setup >> System Admin
4. In the section System Control, click the Offline button to detach Bob from the cluster.
5. Bob's State changes to Offline. Alice remains active and handles the traffic. All back-end applications are still available.
6. Install the new Airlock Gateway release on Bob.
7. Wait until the installation has finished. The system reboots automatically and starts up all services.
8. After reboot, go to:
   System Setup >> System Admin
9. In the section System Control, click the Offline button.
10. Deploy add-on modules if necessary.
11. Go to:
    Configuration >> Configuration Files
12. In the section Import a configuration, click the Import button to import the saved configuration from step 2. Resolve errors if necessary.
13. Activate the configuration and add a comment like Full installation upgrade.
14. After activating the configuration, Bob's State remains Offline (detached from the cluster), as configured in step 7.
15. Go to:
    Configuration >> Configuration Files
16. In the section Export current configuration, tick Include sensitive data such as private keys and passphrases to export and save Bob's new configuration from the upgraded system.
17. Bob is upgraded to the latest Airlock Gateway release and is preconfigured as a cluster node, but offline (detached from the cluster).

If no seamless cluster upgrade without session loss is intended, this section may be skipped and continued with Part 3.

• Goals in this section:
• Copy the new SSH public key from the reinstalled node Bob to node Alice.
• Bring node Bob online in the cluster again. The cluster synchronization is performed automatically.

1. Log in to Bob node via SSH.
2. Copy Bob's SSH public key to Alice using ssh-copy-id Alice.
3. Now both nodes store the SSH keys to securely communicate with each other in the cluster.
4. On Bob, log in to the Airlock Gateway Configuration Center. The banner saying Cluster partner node 'Alice' not reachable should disappear within a minute – a page refresh is required to update the banner status.
5. Go to:
   System Setup >> System Admin
6. In the section System Control, click the Online button to bring Bob online (attach him to the cluster).
7. Node synchronization starts automatically. Alice continues to handle the traffic. All back-end applications are still available.
8. Search Bob's logs for the event EVENT_SY-H-DBSYNC-OK.
9. With the EVENT_SY-H-DBSYNC-OK message, both nodes are synchronized. Bob's State changes to Passive.

• Goals in this section:
• Make sure node Bob is online in the cluster.
• Bring node Alice offline (detach it from the cluster). Bob will get active and take over the traffic from Alice.

1. In the Configuration Center of node Bob, check that the State is Passive.
2. If Bob's status is Offline, go to:
   System Setup >> System Admin >> section System Control
   Click the Online button to bring the node online (attach it to the cluster).
3. Node synchronization starts automatically. Bob's State changes to Passive.
4. Change to the Configuration Center of node Alice and go to:
   System Setup >> System Admin
5. In the section System Control, click the Offline button to detach Alice from the cluster.
6. Alice's State changes to Offline. With Alice being offline in the cluster, Bob takes over the traffic without loss.
7. Check that Bob works properly and handles the traffic, i.e. that all back-end applications are available. Check dashboard and logs if required.
8. Node Alice is in Offline state and ready for the system upgrade. Node Bob's state changed to Active, handling the traffic.

• Goals in this section:
• Reinstall Airlock Gateway on node Alice.
• After installation, set Alice offline (detach the node from the cluster).
• Import Bob's configuration to Alice. With this import, node Alice has a complete cluster configuration including the set of SSH keys and the cluster IP configuration.
• Export the configuration of node Alice.

1. Install the new Airlock Gateway release on node Alice.
2. Wait until the installation has finished. The system reboots automatically and starts up all services.
3. In Alice's Configuration Center, go to:
   System Setup >> System Admin
4. In the section System Control, click the Offline button to disconnect Alice from the cluster.
5. Bob remains in Active state and handles the traffic. All back-end applications are still available.
6. Deploy add-on modules if necessary.
7. Go to:
   Configuration >> Configuration Files
8. In the section Import a configuration, click the Import button to upload and import Bob's latest configuration. Resolve errors if necessary.
9. Activate the configuration and add a comment like Full installation upgrade.
10. After activating the configuration, Alice's State remains Offline (detached from the cluster), as configured in step 3. Alice now has a complete cluster configuration including Bob's SSH key and IP failover settings.
11. Go to:
    Configuration >> Configuration Files
12. In the section Export current configuration, tick Include sensitive data such as private keys and passphrases to export and save Alice's configuration.
13. Log out from the Airlock Gateway Configuration Center.
14. Alice upgrade and cluster configuration are finished. Alice is in Offline state.

• Goals in this section:
• Import Alice's complete configuration to node Bob, so that both nodes share the same configuration.
• Activate Bob's configuration and remotely distribute it to Alice. The cluster synchronization is performed automatically.

At this point, the cluster upgrade is complete. Node Bob is in Active state, node Alice in Passive state and both nodes share a complete cluster configuration.

In this section, the configuration of the upgraded active node (see Part 4, step 9) will be reused.

1. In the Configuration Editor of (the active) node Bob, go to:
   Configuration >> Configuration Files
2. In the section Import a configuration, click the Import button to upload and import Alice's configuration file.
3. Activate the configuration on both nodes. Therefore choose the Activate on 'Bob' and 'Alice' radio button on the activation pop-up.
4. The banner saying Cluster partner node 'Alice' not reachable should disappear within a minute (do a page refresh). Cluster synchronization is performed automatically.
5. Check that node Bob works properly and continues to handle the traffic. Check dashboard and logs if required.
6. In the Configuration Center of node Alice, go to:
   System Setup >> System Admin
7. In the section System Control, click the Online button to bring Alice online (attach the node to the cluster).
8. The cluster synchronization starts automatically. Alice's State changes to Passive.

The cluster upgrade is now complete. Both nodes have been upgraded to the latest release and a fully functional cluster configuration has been activated (see drawing).

• Airlock Gateway failover setup for on-premises installations
• Clustering, load-balancing, and failover scenarios for Airlock Gateway setups