🎉 Automating Disaster Recovery for Azure Service Bus: A Seamless Solution ✨

Disaster recovery is a critical component of any robust IT infrastructure. For Azure Service Bus, a highly reliable cloud messaging service, automating the failover process is crucial for minimizing downtime and ensuring business continuity. While the Azure portal provides an easy way to initiate a failover with a click of a button, relying on manual intervention is not ideal for enterprise-grade solutions. Automation is the key.

In this blog post, we’ll explore an automated approach to handling disaster recovery for Azure Service Bus using a PowerShell script. This script seamlessly initiates the failover process and manages the underlying tasks, making it easier for infrastructure architects to ensure continuity without manual intervention.

Why Automate Service Bus Failover?

Azure Service Bus offers geo-disaster recovery capabilities that ensure high availability and data protection. However, initiating a failover manually can be risky and not recommended in production scenarios. Automation provides several benefits:

1. Consistency: Automated scripts ensure that the failover process is executed the same way every time, reducing the risk of human error.
2. Speed: Automation can trigger failover processes instantly, minimizing downtime.
3. Efficiency: Automated processes can handle multiple tasks simultaneously, such as reconfiguring namespaces and cleaning up resources.

Understanding the Burn Down of the Primary Namespace

One critical aspect of Azure Service Bus failover is that after the failover, the primary namespace is essentially “burned down.” This means that the primary namespace becomes inactive, and all its entities, such as queues and topics, need to be cleaned up. This cleanup process ensures that the primary namespace is ready to be reconfigured or decommissioned.

The PowerShell Script

Let’s dive into the PowerShell script that automates the Azure Service Bus failover process. This script ensures that the primary and secondary namespaces are provisioned and manages the failover with minimal manual intervention.

Step 1: Parameter Initialization

First, we set the necessary parameters, such as subscription ID, resource group name, primary and secondary namespaces, and the alias name.

$connection = Connect-AzAccount -ErrorAction Stop
Write-Host "Connected to Azure successfully." -ForegroundColor Yellow

#************** Parameters ********************************************************************************************************************
$subscriptionId = ""
$resourceGroupName = ""
$sbusPrimaryNamespace = ""
$sbusSecondaryNamespace = ""
$sbusAliasName = ""
$partnerId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.ServiceBus/ `
             namespaces/$sbusPrimaryNamespace"
#***********************************************************************************************************************************************

Step 2: Provisioning Functions

We define functions to ensure that both namespaces are fully provisioned before and after the failover. These functions poll the provisioning state and wait until it reaches the ‘Succeeded’ state for both normal and Geo-provisioned.

function Wait-ForNamespaceProvisioning {
    param (
        [string]$resourceGroupName,
        [string]$namespaceName
    )
    
    $maxRetries = 30
    $retryCount = 0
    $delay = 60  # Delay in seconds between retries
    
    do {
        $namespace = Get-AzServiceBusNamespace -ResourceGroupName $resourceGroupName -NamespaceName $namespaceName
        if ($namespace.ProvisioningState -eq "Succeeded") {
            Write-Output "Namespace $namespaceName is provisioned."
            return
        }
        
        Write-Output "Namespace $namespaceName is still in provisioning state: $($namespace.ProvisioningState)."
        Start-Sleep -Seconds $delay
        $retryCount++

    } while ($namespace.ProvisioningState -ne "Succeeded" -and $retryCount -lt $maxRetries)
    
    if ($namespace.ProvisioningState -ne "Succeeded") {
        throw "Namespace $namespaceName did not reach 'Succeeded' state within the allotted time."
    }
}

function Wait-ForNamespaceGeoProvisioning {
    param (
        [string]$resourceGroupName,
        [string]$namespaceName
    )
    
    $maxRetries = 30
    $retryCount = 0
    $delay = 60  # Delay in seconds between retries
    
    do {
        $namespace = Get-AzServiceBusGeoDRConfiguration -ResourceGroupName $resourceGroupName `
                     -NamespaceName $namespaceName

        if ($namespace.ProvisioningState -eq "Succeeded") {
            Write-Output "Namespace $namespaceName is geo provisioned."
            return
        }
        
        if($null -eq $namespace){ 
            Write-Output "Namespace $namespaceName is geo provisioned."
            return 
        }
        
        Write-Output "Namespace $namespaceName is still in geo provisioning state: $($namespace.ProvisioningState)."
        Start-Sleep -Seconds $delay
        $retryCount++

    } while ($namespace.ProvisioningState -ne "Succeeded" -and $retryCount -lt $maxRetries)
    
    if ($namespace.ProvisioningState -ne "Succeeded") {
        throw "Namespace $namespaceName did not reach 'Succeeded' state within the allotted time."
    }
}

Step 3: Combined Provisioning Function

This function ensures that both the primary and secondary namespaces are provisioned.

function Wait-ForNamespaceAndGeoProvisining {
    param (
        [string]$resourceGroupName,
        [string]$PrimaryNamespace,
        [string]$SecondaryNamespace
    )

    Wait-ForNamespaceProvisioning -resourceGroupName $resourceGroupName -namespaceName $PrimaryNamespace
    Wait-ForNamespaceProvisioning -resourceGroupName $resourceGroupName -namespaceName $SecondaryNamespace

    Wait-ForNamespaceGeoProvisioning -resourceGroupName $resourceGroupName -namespaceName $PrimaryNamespace
    Wait-ForNamespaceGeoProvisioning -resourceGroupName $resourceGroupName -namespaceName $SecondaryNamespace

    return
}

Step 4: Initiate Failover

This section of the script handles the failover process itself, ensuring both namespaces are ready, initiating the failover, and then performing post-failover cleanup and reconfiguration.

#************** Initiate failover **************************************

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Write-Output "`nFailing Over : Azure Service Bus $sbusPrimaryNamespace ...`n"

Set-AzServiceBusGeoDRConfigurationFailOver `
    -Name $sbusAliasName `
    -ResourceGroupName $resourceGroupName `
    -NamespaceName $sbusSecondaryNamespace `

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Step 5: Cleanup After Failover

Post-failover, we delete all queues in the primary namespace to clean up resources. This is crucial since the primary namespace will be “burned down” after the failover.

Write-Output "`nDeleting all queues in the primary $sbusPrimaryNamespace ..."

$queues = Get-AzServiceBusQueue -ResourceGroupName $resourceGroupName -NamespaceName $sbusPrimaryNamespace
foreach ($queue in $queues) {
    Remove-AzServiceBusQueue `
        -ResourceGroupName $resourceGroupName `
        -NamespaceName $sbusPrimaryNamespace `
        -QueueName $queue.Name

    Write-Host "Deleted queue: $($queue.Name)"
}

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Step 6: Reconfiguration

After cleaning up the primary namespace, the script sets the alias back to the secondary namespace.

Write-Output "`nSetting the alias back after failover ..."

New-AzServiceBusGeoDRConfiguration `
    -Name $sbusAliasName `
    -NamespaceName $sbusSecondaryNamespace `
    -ResourceGroupName $resourceGroupName `
    -PartnerNamespace $partnerId

Wait-ForNamespaceAndGeoProvisining -resourceGroupName $resourceGroupName -PrimaryNamespace $sbusPrimaryNamespace `
                                   -SecondaryNamespace $sbusSecondaryNamespace

Write-Output "`nService Bus failover process completed successfully !!"

#************** DONE ************************************************************

Disconnect-AzAccount

Full codebase:

https://github.com/AtanuGpt/AzureServiceBusDR/blob/main/failover.ps1

Explanation of the Failover Process

1. Initial Provisioning Check: The script first ensures that both the primary and secondary namespaces are fully provisioned before initiating the failover.
2. Initiate Failover: The failover process is initiated using Set-AzServiceBusGeoDRConfigurationFailOver, which switches the alias to the secondary namespace.
3. Cleanup Primary Namespace: After the failover, the script deletes all queues in the primary namespace. This step is critical because the primary namespace is effectively “burned down” or rendered inactive and cleaned up. This involves removing all entities (queues, topics, etc.) in the primary namespace.
4. Reconfiguration: The alias is reconfigured to point to the secondary namespace, ensuring continued operation.
5. Final Check: The script performs a final check to ensure both namespaces are in the correct state post-failover.

Conclusion

Automating the disaster recovery process for Azure Service Bus is not just a convenience — it’s a necessity for maintaining high availability and ensuring business continuity. This PowerShell script provides a comprehensive solution, making the failover process seamless and efficient. By adopting automation, you can ensure that your Azure Service Bus environment is always prepared for any eventuality, keeping your services running smoothly even in the face of disruptions.