Azure Automation : Periodically delete log files in Virtual Machines

It's vary common that you are running multiple agents in your virtual machines. Microsoft also deploys several agents behind the scene in your VMs to satisfy various configurations requirements. Now the problem starts when these agents start generating logs files locally and eventually starts eating up the disk space. 

You naturally by instinct jumps in to scale up your VMs or starts adding more disks to it inorder to maintain the performance. But this approach would cost your dearly as your Azure bill would shoot up substantially. Why not clean up those logs periodically with a scheduler and make your boss happy 😀

Azure Automation with runbook type PowerShell is there to make you achieve this very comfortably. Create an automation account with a PowerShell runbook. The script is given below. Copy-Paste it >> Test it >> Schedule it. Done! Smart work isn't it 👍

$ResourceGroup = '<<resource group name>>'
$FolderPath = 'rm /<<log folder path>>/*' #E.g. rm /home/user/temp/*

try
{
	Write-Output "Connecting ..."
	$message = Connect-AzAccount -Identity 
	Write-Output "Connection successful"

	#Get all Azure VMs which are in running state and are running Linux
	$myAzureVMs = Get-AzVM -status | Where-Object {$_.PowerState -eq "VM running" -and $_.StorageProfile.OSDisk.OSType -eq "Linux"}
	Write-Output "Total Linux VMs detected : $($myAzureVMs.Name.Count)"
	Write-Output "VM Names : $($myAzureVMs.Name)" 

	# Run the script against all the listed VMs
	if($myAzureVMs.Name.Count -gt 1)
	{
		For($val = 0; $val -le $myAzureVMs.Name.Count-1; $val++)
		{
			Write-Output "Target VM : $($myAzureVMs.Name[$val]), File deletion in progress ..."
			$message = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $myAzureVMs.Name[$val] -CommandId 'RunShellScript' -ScriptString $FolderPath
			Write-Output "Operation completed successfully" 
		}	
	}
	else {
		Write-Output "Target VM : $($myAzureVMs.Name), File deletion in progress ..."
		$message = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $myAzureVMs.Name -CommandId 'RunShellScript' -ScriptString $FolderPath
		Write-Output "Operation completed successfully" 
	}
}
catch
{
	Write-Error -Message $_.Exception
    throw $_.Exception
}

Azure Active Directory Login for Linux Virtual Machines

To improve the security of Linux virtual machines (VMs) in Azure, now we can integrate the VM with Azure Active Directory (Azure AD) authentication. Azure AD can be used as a core authentication platform to SSH into a Linux VM. This functionality allows organizations to manage security access to VMs with Azure role-based access control (RBAC) and Conditional Access policies.

This article shows how to create and configure a Linux VM and log in with Azure AD by using OpenSSH certificate-based authentication.

1.1. Benefits 

• Use your Azure AD credentials to log in to Azure Linux Virtual machines. No need to remember any separate login credentials.
• No separate Bastion Virtual machine is required. 
• No public IP/Port exposure for any Virtual machines to the outside world. 
• Get Open SSH key-based authentication without needing to distribute SSH keys to users or provision SSH public keys on any Azure Linux VMs that you deploy.
• Reduce reliance on local administrator accounts, credential theft, and weak credentials.
• With RBAC, specify who can log in to a virtual machine as a regular user or with administrator privileges. When users join your team, you can update the Azure RBAC policy for the VM to grant access as appropriate. When employees leave your organization and their user accounts are disabled or removed from Azure AD, they no longer have access to your resources.

• With Conditional Access, configure policies to require multifactor authentication or to require that your client device is managed (for example, compliant or hybrid Azure AD joined) before you can use it SSH into Linux VMs.

Note

Azure Active Directory native client for Azure Bastion is officially supported by Microsoft and is now GA. In this article we will use this feature to solve our objective of login to virtual machines with active directory login credentials.

1.2. Installing Azure CLI on Local Laptop/Desktop

• Open command prompt. Type az at the root. This will check if azure cli is already installed in your VM and if found you would see something similar below and you can skip this section and move to next section.          
  

• If not found then go ahead and install Azure CLI from the below link. 

          Windows : https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli

          Linux :  curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

1.3. Configuring AAD login feature on New VMs

• Create a new Linux Virtual Machine. Keeping all other configurations of your choice, make sure to check the "Login with Azure AD" feature on under the Management tab. 💡 This is very  important! The identity will be automatically get selected to system assigned managed identity.                 
  

• Once the VM gets provisioned, you must see the installed Azure AD Login extensions under the Extensions + applications menu under Settings blade of the respective VM. 
  

1.4. Enabling AAD login feature on Existing VMs

• Go to the Setting blade of the respective VM and click on Extensions + application menu. Click +Add to add a new extension          
  

• Select extension  "Azure AD based SSH Login" and click Next and then Review and create button. Click Create after the validation passed message is shown.          
  

• After the installation complete check the Extensions + application menu again under the Settings blade of the VM. This time it must show the installed extension.          
  

1.5. Terraform Script to enable AAD login (Linux VM)

resource "azurerm_linux_virtual_machine" "example" {
   // blah-blah
   identity {
      type = "SystemAssigned" 
   }
}

resource "azurerm_virtual_machine_extension" "example" {
   name = "AADSSHLoginForLinux_extn"
   virtual_machine_id = azurerm_linux_virtual_machine.example.id
   publisher = "Microsoft.Azure.ActiveDirectory"
   type = "AADSSHLoginForLinux"
   type_handler_version = "1.0"
}

1.6. Enabling RBAC (Configure role assignments)

      Any of the following roles must be given to the user accessing the VM.

💡 Assigning any of these roles to a particular user or to a particular group is mandatory inorder to access the VM via Azure AD credentials.

1. Virtual Machine Administrator Login: Users who have this role assigned can log in to an Azure virtual machine with administrator privileges.
2. Virtual Machine User Login: Users who have this role assigned can log in to an Azure virtual machine with regular user privileges 

• From the Access control (IAM) menu of the respective VM; click on +Add and select Add role assignment submenu option. You must have "Owner" or "User Access Administrator" privilege to perform this operation.          
  

• Search for any of the above role and proceed. I have selected the "Virtual Machine Administrator Login" role for this article. Click Next         
  

• Select an user or group to whom you want the role to get assigned. Click Review + assign

1.7. Configuring Azure Bastion (With Native support, New Bastion Host)

• Create a new azure bastion service. Keeping all other configurations of your choice, make sure to check the "Native client support" feature on under the Advanced tab. 💡 This is very  important!         
  

1.8. Configuring Azure Bastion (With Native support, Existing Bastion Host)

• Go to the Configuration option under the Setting tab and check the "Native client support" checkbox. Hit Apply.     
  

1.9. Terraform Script to enable Native client support for Azure Bastion Host

resource "azurerm_bastion_host" "example" {
   // blah-blah
   tunneling_enabled = true
}

1.10. Login to Application VMs with Azure AD credentials via Bastion Host (Local Laptop/Desktop)

• Open Windows Command Prompt or Windows PowerShell IDE in your machine. 
• Type az login and press enter. (Make sure you are disconnected from G+D VPN, 💡 This is important!). This will open up a browser window as would ask for login credentials. Please enter your azure active directory login credentials.         
  

• Upon successful login, it will show all the subscription on which you have access under the tenant. You can execute the following command to list all subscriptions in a readable table format.

az account list -o table  

• Type the following command to check the current subscription you are logged in. 

az account show -o table          

          If it's the right subscription then move to the next step. Else execute the following command to set your correct subscription.

 az account set --subscription "<<subscription_id or subscription_name>>" 

• Get the resource id of your target VM by running the following query. Copy the return value and keep in notepad. You would need this in the next step

 az vm show --name <<vm name>> --resource-group <<resource group name>> --query "id"

• Now the important part which we are waiting for. We are going to login to our target VM via Azure Bastion. Run the following command

 az network bastion ssh \
 --name "<<bastion-host name>>" \
 --resource-group "<<resource group name>>" \
 --target-resource-id "<<vm resource id copied from the previous step>>" \
 --auth-type "AAD"

• Voila!... you are now successfully connected to your target VM via Azure Bastion. 😊 The prompt will quickly change to show your [AAD login id]@[target VM name] 

Congratulations! for coming this far. Hope this article will help you to further explore more on this topic.

Do share with me about your experience and what you have built upon this foundation. I would love to hear from you.

Azure SQL Diagnostics & Event Hub

Auditing & Monitoring is one of the strong pillar in any SDLC process and in today's cloud arena it became even more important. In this article we would learn how to enable auditing capability in Azure SQL database and would also witness some LIVE stream of audit data. When it comes to auditing in Azure SQL, the platform gives you three choices for log destination. You can choose as per your requirement

 

• You can enable auditing either at the server level or at the database level. For this article we would go ahead and enable it at the database level. So, go to your Azure SQL database instance, select Auditing from the left pane under Security and enable auditing.
  

• We are going to use Event Hub for this article to stream all telemetry data from Azure SQL. So, go ahead and select that option and provide the required details. Make sure you have created your Event Hub resource and namespace beforehand in the same region where your Azure SQL service is running. Hit Save.

This will go ahead and add a new log category named "SQLSecurityAuditEvents" in the Azure SQL Diagnostic setting. This is the LIVE telemetry information you need to capture for all your SQL compliance and security audits. For example

1. What are all SQL statements getting executed and on which database, from which IP address those request came and at what datetime, how many rows got affected and much more.....
2. Successful logins 
3. Failed logins

 That's it. You are done with the required configuration.

Now lets run few SQL queries to generate some audit logs. I have used Azure Data Studio to connect and run the queries on my database but you can use any tool of your choice. Make sure you have whitelisted your IP to satisfy the firewall rules.

• Now there's no way you can view the logs by directly going into your Event Hub. So, inorder to view the logs we will take help of Azure Service Bus Explorer desktop application. So, open the explorer, connect to your event hub and fire up a listener to witness live feed the logs.

• Browse through the events and select one as per your requirement. Copy the EventText and view it in any JSON Format Viewer.  Your query and many other valuable information will show up in the logs. 

• Stop the Listener when done.

In real time scenario you won't be using Service Bus Explorer to view and monitor live events. There will be some Big Data analytic service sitting at the other side of Event Hub which will do all the big data processing, monitoring and reporting on the Event Hub logs data.

Hope this article will give you an understand on how to log and monitor telemetry data from Azure SQL.