29 September, 2020

Azure Logic App Integration : Transform XML to JSON

In Enterprise Integration or iPaaS, a scenario that often comes up is to convert a XML payload to Json. You might be receiving XML data from one system but needs to convert that to JSON before pushing it to another system. If both the source and destination systems understands same language then it's OK but that is not always the case. So, there's a challenge.

Azure Logic App has some cool power and we will try to leverage on such in our integration scenario. Here our legacy source system can only send XML data but our destination system only understands JSON. 

Prerequisites:

  • Azure Subscription (Free subscription will also do)
  • Basic XML, XSLT understanding
  • Basic JSON understanding
  • Azure Logic App basic Request/Response understanding
Let's get Started
  • Create a simple xml document (Vendor.xml). This is the payload our source system will send.
<?xml version="1.0" encoding="UTF-8"?>
<Vendor>
   <VendorNumber>VND1</VendorNumber>
   <VendorName>My Vendor</VendorName>
   <GroupId>10</GroupId>
   <Company>USRT</Company>
   <VendorCity>My City</VendorCity>
   <VendorContactPhone>1234567890</VendorContactPhone>
   <VendorContactEmail>myvendor@vendoremail.com</VendorContactEmail>
<VendorCountryRgn>USA</VendorCountryRgn> </Vendor>
  • But our destination system only understands JSON. Now we need to transform the above XML into JSON keeping all other information intact. So after transformation the data should look like this
{
   "VendorNumber":"VND1",
   "VendorName":"My Vendor",
   "Company":"USRT",
   "VendorCity":"My City",
   "VendorContactPhone":"1234567890",
   "VendorContactEmail":"myvendor@vendoremail.com",
   "VendorCountryRgn":"USA"
}
  • Looks simple! but not that simple in Logic App. There's a few marketplace connectors available but that won't help much when it comes to customization and also they not free and has licensing and pricing model complexities. So why to go that route when we have a free yet powerful friend in hand which can do this task quite comfortably and also is a first class citizen in Azure Logic App. XSLT Transformation. 
Here goes our Vendor.xsl document. Save it in your machine as you would need this later. It's quite simple and self-explanatory. XSLT has immense power and hence you can understand what level of control you will have in handling XML complexities.
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output omit-xml-declaration="yes" method="text" media-type="application/json"/>
    <xsl:template match="/">
        <xsl:text>{ </xsl:text>
        
        <xsl:text>"VendorNumber" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorNumber"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorName" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorName"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"Company" : "</xsl:text>
        <xsl:value-of select="Vendor/Company"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorCity" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorCity"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorContactPhone" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorContactPhone"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorContactEmail" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorContactEmail"/>
        <xsl:text>", </xsl:text>
        
        <xsl:text>"VendorCountryRgn" : "</xsl:text>
        <xsl:value-of select="Vendor/VendorCountryRgn"/>
        <xsl:text>" </xsl:text>
        
        <xsl:text> }</xsl:text>
    </xsl:template>
</xsl:stylesheet>
  • We are done with our basic ground work. Now it's time to step into Azure. πŸ˜ƒ
  • Login to Azure portal 
  • We cannot apply this XSLT file straightway on our input XML in Logic App. For that we need to create an "integration accounts". Let's do that. Search for integration in the Azure search bar and select.
  • Click +Add at the top and fill up the necessary details as per your choice. Create the account.
  • Once the integration account is created it will take you to the following landing page. There are lots of components available but for the sake of simplicity I won't go into those as it's not required for your case. It's required for typical B2B scenarios and if you are keep to learn then Azure has good documentation around it.  Here we only need the Maps component. So click on that.
  • Click +Add at the top and upload your Vendor.xsl file. This is the XSLT file we have created earlier. Click OK
  • Great. Now let create the logic app for some real action. Search for Logic App in the Azure search bar and select.
  • Click +Add at the top and fill up the following details as per your choice. One important point to remember here. Make sure your logic app location is same as your integration account location. This is a requirement. Create your logic app.
  • Now the most important part. We need to tell the Logic App to use the integration account so that it could pick up the XSLT transformation map from there. Go to the Settings>>Workflow Settings and map your integration account. Click Save
  • Click Logic App designer under Development Tools and choose "When a HTTP request is received" trigger from the list of common triggers
  • This will take you to the logic app workflow designer with the above trigger as the first element in the workflow. Do nothing here and click the "Next Step"
  • Search for "Transform XML" and select Transform XML from the list of actions.
  • For Contents, select the Body from the Dynamic content list. The is the http request body which we are going to send as an XML payload later.
  • Choose the XSLT map from the "Map" dropdown. This is coming from your integration account where we have uploaded the Vendor.xsl file.
  • Now select the Add new parameter dropdown and select "Transform options" and the value as "Apply XSLT output attributes". This is very important else you will get a runtime error "Code": "InvalidXsltContent", "Message": "An error occurred while processing map. 'Token Text in state Start would result in an invalid XML document. Make sure that the ConformanceLevel setting is set to ConformanceLevel.Fragment or ConformanceLevel.Auto if you want to write an XML fragment. ".  This is to just to inform the runtime to use the output format as defined in the XSLT file which in our case is application/json. So, make sure you do not miss this step πŸ‘
  • Now we do not have any real source and destination system for this demo. So, we would return back the transformed JSON to the caller to simulate the XML to JSON conversion. Click on "+New Step", search for "Response" and select the Response Action
  • Select the "Transformed XML" from the Dynamic content selection window 
  • So, we have 3 steps in the workflow. Save the Logic app. This will generate the HTTP endpoint for us to test. Copy the HTTP POST URL endpoint and open any rest client of your choice. My favorite is Postman
  • Paste the url endpoint to make a POST call. In the body copy-paste the XML data from the Vendor.xml file we have created previously. Hit Send and Voila!! We are successfully getting back JSON response by sending XML request. Isn't that great stuff😚

Great Job and Congratulations! for coming this far. Hope this article will help you to further explore more on this feature.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.

07 September, 2020

Creating ARM Templates with Azure Resource Group Project

Automation is the buzzword nowadays and when it comes to Cloud, it becomes more inevitable. Creating various Azure resources and then logically grouping them inside a Resource group is a very common task we perform while working in Azure. We can go ahead and hit the Azure portal and create all the necessary resources by simple clicks and set up the environment in no time and get things running. Cool!

This may be good and acceptable for your development environment. What about other environments such as QA,Staging or even Production. Is it feasible to repeat the same task over and over again in each environment and create the exact same stuff. What about human error. Manual processes are largely error prone and a slight hiccup could lead to a big costly mistake. Moreover you might not have permission to all the environments and the same you might forget the exact resource configurations few months down the line. So, in short there's hell lot of things that can go wrong if things are manual.

Hence Automation. In this article I will walk you through the steps of creating a simple ARM template containing an Azure Storage resource in VS-2017 solution and then deploying the same with the help of PowerShell to Azure Cloud. Hope you all would enjoy the journey!

Please note that I have purposefully left out some complex areas or haven’t deep dived into details at some places. This is just to make this article an easy read and help as a quick guide from where to start in creating and deploying ARM templates

Prerequisites:

  • Microsoft Visual Studio 2017 with Azure SDK installed.
  • Azure Subscription (Free subscription of 30 days will also do)
  • Basic JSON understanding
  • Windows PowerShell (Command Line or ISE)

Let's get Started

  • Fire up your Visual Studio 2017
  • File > New > Project. Click on "Cloud" in the left panel and then select "Azure Resource Group" project type. Give the project name, location as per your choice. Click OK
No alt text provided for this image
  • Select "Blank Template" from the list of templates as of now. We will add resources later. Click OK
No alt text provided for this image
  • A Resource Group project will be created as below.
Solution Explorer

The main entry point is the azuredeploy.json. This is where we will create all the resources along with its respective configurations. Next is azuredeploy.parameters.json which is basically used to override any parameter that is defined in the azuredeploy.json file. Anything that needs to be customized on a per item basis should be placed here. Deploy-AzureResourceGroup.ps1 is the powershell script require to deploy the resources.

We will only make use the azuredeploy.json in this demo and perform the other two steps a bit differently so to avoid any complexity and help in easy understanding.

  • Double-Click on the azuredeploy.json file to open the code window.
No alt text provided for this image

There are four main sections in the json file.

parameters : Here we will define all the parameters respective to a resource in key-value pair instead of hard-coding in the resource body.

variables : This is optional. These are piece of information which the ARM template can work out on the fly. Specially used to deploy similar resources in different environments.

resources : All resource code blocks will go into this section following the same structure and pattern.

outputs : This is optional. Used when linking other ARM template with the master template or infact creating linked templates. Will not use this in this demo.

  • Once you opened the code view of the azuredeploy.json file, VS-2017 will automatically open a JSON Outline window for powerful navigation purpose.
No alt text provided for this image

If you do not see this windows then goto View > Other Windows > JSON Outline

  • Now our target is to create a Azure Storage resource here. Right click on resource section inside the JSON Outline window and select "Add New Resource"
No alt text provided for this image
  • Scroll down to find the "Storage Account" resource. Give it a name of your choice. I gave demoStorage but it can be any name. Click "Add"
No alt text provided for this image
  • Visual Studio will add all the necessary boilerplate code required for this resource type and will populate all the required sections of the ARM template with some default values.
No alt text provided for this image
  • Now lets do some minor modifications. The Storage name should be unique like any other Azure resources. Visual Studio has tried to make it unique by its own way but what if I want the storage name to start with "demo" as a prefix. Let do this.

We will add one parameter name "demoStoragePrefix" as below in the parameters section

"demoStoragePrefix": {
        "type": "string",
        "minLength": 1,
        "defaultValue": "demo"
      }

In the variables section lets modify the demoStorageName variable to read the prefix from the parameter we just defined above.

"demoStorageName": "[concat(parameters('demoStoragePrefix'), uniqueString(resourceGroup().id))]"

Now also add an additional parameter name "metadata" in the parameters section. This is optional and most people do overlook but I strongly recommend to add this with every template as it makes the code more readable.

"metadata": {
      "type": "string",
      "defaultValue": "Creation of Storage Account"
    }
  • Now the new modified json code block looks like this with the above changes in place.
No alt text provided for this image
  • Save the file and solution. We are done with our ARM template. Now lets dive into the deployment part.

Open PowerShell window (Command line or ISE window will do)

  • Connect to your Azure account. Execute the below command.
Connect-AzureRmAccount

This will prompt for your Azure username and password. Provide the details and login. A successful login with show all your subscriptions. I have only one subscription and hence it shows the details related to that.

Account          : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SubscriptionName : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SubscriptionId   : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
TenantId         : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment      : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • Next set some deployment variables which we will use later in the steps. Select these steps and execute.
## Define Deployment Variables

   ## Your resource location. In my case I placed it in SE Asia
   $location = 'SouthEast Asia' 
   ## Resource Group Name
   $resourceGroupName = 'ARMdemo-simple-storage'
   ## Resource Group Deployment Name
   $resourceDeploymentName = 'ARMdemo-simple-storage-deployment'
   ## The path of the template file in your local drive. Your location folder path can be different
   $templatePath = $env:SystemDrive + '\' + 'Personal\Projects\DemoAzureRmStorageDeployment'
   ## The template filename. It will be same if you have taken the default filename which Visual Studio provides
   $templateFile = 'azuredeploy.json'
   ## The full template path along with the file
   $template = $templatePath + '\' + $templateFile

  • Create a Resource group in Azure under which our Storage resource will be placed. Now here you might be thinking that why we are not creating this resource group through the same ARM template. You are absolutely right but I have purposefully kept this separate to make our ARM template look simple. Otherwise we have to make it a bit complex by adding priority in the template regarding which resource should get created first and depends on what. Hope you understand.
### Create Resource Group
 New-AzureRmResourceGroup `
    -Name $resourceGroupName `
    -Location $location `
    -Verbose -Force

Execute the above script. It will create the resource group name "ARMdemo-simple-storage" in "SouthEast Asia" region. These are coming from the deployment variables we set before. Your name and region can be different based on what you have set in the variables.

  • Now the most important step. Deploying our ARM template with Powershell. Execute the following command. The command "New-AzureRmResourceGroupDeployment" will pick the ARM template from the location of our local drive and would deploy the resource under the resource group created above. The parameters passed are very much self explanatory
### Deploy Resources

New-AzureRmResourceGroupDeployment -Name $resourceDeploymentName -ResourceGroupName $resourceGroupName -TemplateFile $template -Verbose -Force
  • Wait for sometime for the deployment to finish. The command will also validate the ARM template before deployment and if looks fine will display "-Template is valid" else will throw an error message with description of what and where it went wrong.
  • DONE with our automation.

Login to Azure portal and check if everything is created as expected.

First lets check the resource group. Our "ARMdemo-simple-storage" is there as expected.

No alt text provided for this image

Let dive into it. Yes our storage account (provisioned thorough the ARM template) is showing up under the resource group in location : Southeast Asia.

No alt text provided for this image

One thing to notice here is the unique storage name. The prefix is "demo" which we had set in the ARM template and the rest is appended from the resourceGroup id. Recall the below variable we set in the template. Amazing!

"variables": {
     "demoStorageName": "[concat(parameters('demoStoragePrefix'), uniqueString(resourceGroup().id))]"
}

Click on the storage and you can view more details.

No alt text provided for this image

Great Stuff and Well Done! No more manual steps anymore. Going forward any redundant work you do, try to automate it. It will be a blessing.

Do share with me about your experience and what you have built upon this foundation. You can take it upto any level and integrate. I would love to hear from you.