JSON and XML messages deserialization in D365 Finance and Operations

 In this article, I will explain concepts of JSON and XML message deserialization into X++ class contracts for D365 Finance and Operations (FO). And of course, the implementation of it.

A short introduction to the topic

As these days cloud ERP system like D365 FO, provides much more functionality than “old” Axapta, in a complex world of distributed services. It is more dependent on communication with external systems than ever. However, the tooling that is provided and the legacy of X++, puts a lot of grief in complex Integration with the outside world.

Since AX7 has been put into the Azure cloud, the exchange of text files (of various structures) became a nuisance. As we move towards a growing number of API integrations, it puts more pressure on using open standards like JSON and XML.

As I would like to abstract from how the message with XML/JSON information comes into the system, there are many ways of doing this. I will cover only topics dealing with the text content of such a message.

Deserialization of JSON messages into X++ contract classes

The easiest way will be showing this on Sales Order example. Which is most common and familiar concept for everybody interested in this topic.

Let assume the simplest information to be integrated to FO. Which will be Sales Order with basic information and few Sales Lines.

No alt text provided for this image


Sales Order header containing information:

  • Sales Number - tag "SalesNumber" - simple String
  • Customer Name – tag: “Name” - simple type String
  • Customer Account ID - tag: “Account” – simple type String
  • Delivery Date – tag: “DeliveryDate” – simple type String(using of string and parsing it depended of agreed date model is a safe policy that prevents lot of troubles)
  • Delivery Address – tag “DeliveryAddress” -complex type for Address
  • Sales Order lines – tag “Lines” – List of complex type for Sales Line

Implementation of SOHeader contract:

X++
//Sales Order Header Contract
[DataContractAttribute]
class SOHeader
{
    str salesNumber;
    str custName;
    str custAccount;
    str deliveryDate;
    SOAddress deliveryAddress;
    List SOLines;


    [DataMemberAttribute('SalesNumber')]
    public str parmSalesNumber(str _salesNumber = salesNumber)
    {
        salesNumber = _salesNumber;
        return salesNumber;
    }

    [DataMemberAttribute('Name')]
    public str parmCustName(str _CustName = custName)
    {
        custName = _CustName;
        return custName;
    }


    [DataMemberAttribute('Account')]
    public str parmCustAccount(str _CustAccount = custAccount)
    {
        custAccount = _CustAccount;
        return custAccount;
    }


    [DataMemberAttribute('DeliveryDate')]
    public str parmDeliveryDate(str _deliveryDate = deliveryDate)
    {
        deliveryDate = _deliveryDate;
        return deliveryDate;
    }


    [DataMemberAttribute('DeliveryAddress')]
    public SOAddress parmDeliveryAddress(SOAddress _deliveryAddress = deliveryAddress)
    {
        deliveryAddress = _deliveryAddress;
        return deliveryAddress;
    }


    [DataMemberAttribute('Lines'), DataCollectionAttribute(Types::Class, classStr(SOLine))]
    public List parmSOLines(List _SOLines = SOLines)
    {
        SOLines = _SOLines;
        return SOLines;
    }


}

Important parts of this are [Attributes]

[DataContractAttribute] - class atribute indicating that it is a Contract class

[DataMemberAttribute('Name')] or [DataMemberAttribute('DeliveryAddress')] - object attribute - looks the same for simple and complex objects. Return object from such method has to be defined accordingly to type simple or class

 [DataMemberAttribute('Lines'), DataCollectionAttribute(Types::Class, classStr(SOLine))] - Collection attribute which indicate that after that there will be list of objects for defined class. Return object for such method has to be defined as "List"

Value in Attribute e.g. "Name" indicates exact JSON tag, not Class method name or class local variable name. So in JSON it would look like {"Name":"Bulb Inc. ELSON"}.

When processing FormJsonSerializer will find this tag("Name"), decode method name "parmCustName" -as this one has it as DataMemberAttribute - and will assign it to class variable "custName"

Implementation of SOAddress contract:

  • Address Name - tag "AddressName"
  • Street - tag "Street"
  • City - tag "City"
  • ZipCode - tag "PostCode"
  • CountryCode - tag "ISOCountry"

X++
//Sales Order Header Delivery Address Contract
[DataContractAttribute]
class SOAddress
{
    str AName;
    str AStreet;
    str ACity;
    str AZipCode;
    str ACountryCode; 


    [DataMemberAttribute('AddressName')]
    public str parmName(str _AddressName = AName)
    {
        AName = _AddressName;
        return AName;
    }


    [DataMemberAttribute('Street')]
    public str parmStreet(str _AStreet = AStreet)
    {
        AStreet = _AStreet;
        return AStreet;
    }


    [DataMemberAttribute('City')]
    public str parmCity(str _ACity = ACity)
    {
        ACity = _ACity;
        return ACity;
    }


    [DataMemberAttribute('PostCode')]
    public str parmZipCode(str _AZipCode = AZipCode)
    {
        AZipCode = _AZipCode;
        return AZipCode;
    }


    [DataMemberAttribute('ISOCountry')]
    public str parmCountryCode(str _ACountryCode = ACountryCode)
    {
        ACountryCode = _ACountryCode;
        return ACountryCode;
    }


}

And last Contract is SOLine:

  • Item Name - tag "ItemName"
  • ItemCode - tag "ItemCode"
  • LineNum - tag "LineID" - type int
  • Price - tag "Price" type real - item price
  • Qty - tag "Qty" - type real - sales Qty
  • UnitID - tag "UnitID" - unit for sales Qty
  • CurrencyCode - tag "CurrencyISO" - Currency Code for the price

X++
//Sales Order Line Contract
[DataContractAttribute]
class SOLine
{
    str ItemName;
    str ItemCode;
    int LineNum;
    real Price;
    real Qty;
    str UnitID;
    str CurrencyCode;



    [DataMemberAttribute('ItemName')]
    public str parmItemName(str _AddressName = ItemName)
    {
        ItemName = _AddressName;
        return ItemName;
    }


    [DataMemberAttribute('ItemCode')]
    public str parmItemCode(str _ItemCode = ItemCode)
    {
        ItemCode = _ItemCode;
        return ItemCode;
    }


    [DataMemberAttribute('LineID')]
    public int parmLineNum(int _lineNum = LineNum)
    {
        LineNum = _lineNum;
        return LineNum;
    }


    [DataMemberAttribute('Price')]
    public real parmPrice(real _Price = Price)
    {
        Price = _Price;
        return Price;
    }


    [DataMemberAttribute('Qty')]
    public real parmQty(real _Qty = Qty)
    {
        Qty = _Qty;
        return Qty;
    }


    [DataMemberAttribute('UnitID')]
    public str parmUnitID(str _UnitID = UnitID)
    {
        UnitID = _UnitID;
        return UnitID;
    }


    [DataMemberAttribute('CurrencyISO')]
    public str parmCurrencyCode(str _CurrencyCode = CurrencyCode)
    {
        CurrencyCode = _CurrencyCode;
        return CurrencyCode;
    }


}

Those 3 simple contracts allows us to deserialize input JSON message like that:

JSON
{
  "SalesNumber" : "S0001"
  "Account":"UK00BL1",
  "Name":"Bulb Inc. ELSON",
  "DeliveryDate":"19/07/2021",
  "DeliveryAddress":
  {
    "AddressName":"Blub Inc UK",
    "City":"ELSON",
    "ISOCountry":"GB",
    "PostCode":"",
    "Street":"85 Whitchurch Road"
  },
  "Lines":[
    {
       "LineID":1,
       "ItemName":"Bulb 100W",     
       "ItemCode":"BL00100W",   
       "Price":1.5,
       "CurrencyISO":"GBP",
       "Qty":3,
       "UnitID":"ea"
    },
    {
       "LineID":2,
       "ItemName":"Bulb LED 10W",     
       "ItemCode":"LED0010W",   
       "Price":15.45,
       "CurrencyISO":"GBP",
       "Qty":2,       
       "UnitID":"box"
    }
    ]
}

Using just one line of code..

X++
SOHeader SOHeader = FormJsonSerializer::deserializeObject(classNum(SOHeader), serializedJSONstr);

If we want to serialize SOHeader into JSON with values it is one line as well..

X++
str serializedJSON = FormJsonSerializer::serializeClass(SOHeader);

Easy peasy lemon squeezy.. or is it? Check the conclusions at the end :)

Transformation of XML into JSON

Since Microsoft provided us with such a great tool for JSON. One could imagine it will be the same with XML.. but it is not.

However JSON is younger brother of XML ,which mean that transformation from one to another, has been sorted many moons ago. So to deal with XML, we just need to transform it to JSON or back from JSON.

It can be done in 2 ways.. "simple" way as external library in .NET Framework referenced in X++ or D365FO "helper" class in D365 FO.

Simple way of deserializing XML to JSON in .NET Framework using Newtownsof.Json requires just 3 lines of code:

C# Code
XmlDocument doc = new XmlDocument();
doc.LoadXml(xml);


string json = JsonConvert.SerializeXmlNode(doc, Newtonsoft.Json.Formatting.Indented, true);

However it cannot be done like that in X++.. as we can't use SerializeXMLNode we have to use SerializeXnode which is not that pleasant:

X++
stringreader = new System.IO.StringReader(tmpString);
xmlReader = new System.Xml.XmlTextReader(stringreader);
xmlReader.Read();

newNode = System.Xml.Linq.XNode::ReadFrom(xmlReader);

retJSON =  Newtonsoft.Json.JsonConvert::SerializeXNode(newNode, Newtonsoft.Json.Formatting::None);

But to make it really work, we need to do a lot of cleansing manually, as output is a bit rubbish. SerializeXNode also does not allow XML encoding header, so the code need to have a lot of input/output fixes

X++
class SOXMLHelper
{
 
  
    public static str deserializeXML2JSON(str _xmlString)
    {
        System.Xml.XmlTextReader    xmlReader;
        System.IO.StringReader      stringReader;
        System.Xml.Linq.XNode newNode;
        Newtonsoft.Json.Formatting	formatting;
        str							tmpString;
        str							retJSON;
  
        try
        {
            tmpString = strReplace(_xmlString, "\r\n", "");
            tmpString = SOXMLHelper::removeXMLHeader(tmpString);// additional method to deal with XML encoding header <?xml ?>


            stringreader = new System.IO.StringReader(tmpString);
            xmlReader = new System.Xml.XmlTextReader(stringreader);
            xmlReader.Read();


            newNode = System.Xml.Linq.XNode::ReadFrom(xmlReader);


            retJSON =  Newtonsoft.Json.JsonConvert::SerializeXNode(newNode, Newtonsoft.Json.Formatting::None);


            retJSON = SOXMLHelper::cleanCarriageReturn(retJSON); //XNode is creating lots of carriege returns that will be transformed to JSON objects which we don't want


            retJSON = SOXMLHelper::cleanEmptyJSONArrays(retJSON);//XNnode is creating those from all empty spaces which are decoded as JSON collections we don't want either


            xmlReader.Close();
            xmlReader.Dispose();
            stringreader.Dispose();
        }
        catch(Exception::CLRError)
        {
            System.Exception ex = CLRInterop::getLastException();
            throw Error(strFmt("SOXMLHelper::deserializeXML2JSON with error %1", ex.Message));
        }
        catch
        {
            throw Error("SOXMLHelper::deserializeXML2JSON failed");
        }
    
       return retJSON;

    }
}

Luckily serialization from JSON to XML is much easier:

X++
public static str serializeContract2XML(Object _object)
{
        str						serializedJson;
        System.Xml.Linq.XNode	node;
        str						retXML;


        serializedJson = FormJsonSerializer::serializeClass(_object);


        node = Newtonsoft.Json.JsonConvert::DeserializeXNode(serializedJson);
        
        retXML = node.ToString();

        retXML = SOXMLHelper::addXMLHeader(retXML); // we need to add XML header to make it proper XML

        return retXML;

Dealing with XML attributes in X++ contract classes

Another thing is XML requires something that JSON can live without which is ROOT element.

So when we are serializing or deserializing a defined contract that works fine in JSON.. it will not work for XML, as we need to wrap it in ROOT node. Of course, it applies if our JSON does not have ROOT element. But unfortunately in our case, it does not have one yet.

The new contract has to be added to wrap SORoot so we can deal with XML. In our case, we will call it SORoot with the tag "SalesOrder". Which will point to our existing SOHeader contact.

X++
//Sales Order Header Root Contract
[DataContractAttribute]
class SORoot
{
    SOHeader SOHeader;


    [DataMemberAttribute('SalesOrder')]
    public SOHeader parmSOHeader(SOHeader _SOHeader = SOHeader)
    {
        SOHeader = _SOHeader;
        return SOHeader;
    }


}

"SalesOrder" node in JSON will act as ROOT and wrap SOHeader nicely.

JSON
{
  "SalesOrder": {
  "SalesNumber":"S0001",
  "Account":"UK00BL1",
  "Name":"Bulb Inc. ELSON",
  "DeliveryDate":"19/07/2021",
  "DeliveryAddress":
  {
    "AddressName":"Blub Inc UK",
    "City":"ELSON",
    "ISOCountry":"GB",
    "PostCode":"",
    "Street":"85 Whitchurch Road"
  },
  "Lines":[
    {
       "LineID":1,
       "ItemName":"Bulb 100W",     
       "ItemCode":"BL00100W",   
       "Price":1.5,
       "CurrencyISO":"GBP",
       "Qty":3,
       "UnitID":"ea"
    },
    {
       "LineID":2,
       "ItemName":"Bulb LED 10W",     
       "ItemCode":"LED0010W",   
       "Price":15.45,
       "CurrencyISO":"GBP",
       "Qty":2,       
       "UnitID":"box"
    }
    ]
}}

So finally we can have proper XML!!

XML
<?xml version="1.0" encoding="utf-8"?>
<SalesOrder>  
  <SalesNumber>S0001</SalesNumber>
  <Account>UK00BL1</Account>
  <Name>Bulb Inc. ELSON</Name>  
  <DeliveryDate>19/07/2021</DeliveryDate>
  <DeliveryAddress>    
    <AddressName>Blub Inc UK</AddressName>    
    <City>ELSON</City>    
    <ISOCountry>GB</ISOCountry>    
    <PostCode>SY12 5DQ</PostCode>    
    <Street>85  Whitchurch Road</Street>  
  </DeliveryAddress>    
  <Lines>
    <LineID>1</LineID>   
    <ItemCode>BL00100W</ItemCode>    
    <ItemName>Bulb 100W</ItemName>          
    <Price>1.5</Price> 
    <CurrencyISO>GBP</CurrencyISO>        
    <Qty>3</Qty>    
    <UnitID>ea</UnitID>  
  </Lines>  
  <Lines> 
   <LineID>2</LineID>
   <ItemCode>LED0010</ItemCode>    
   <ItemName>Bulb LED 10W</ItemName>           
   <Price>15.45</Price>    
   <CurrencyISO>GBP</CurrencyISO>
   <Qty>2</Qty>    
   <UnitID>box</UnitID>  
  </Lines>  
</SalesOrder>

We could end up with XML here..

But what if some legacy system would like to play with XML attributes? As so far we have operated only on simple XML nodes with values only.

E.g. for Sales Line contract instead "tag->value" pair we will have some values and "XML Attributes"

XML
<Lines>   
  <ItemCode>LED0010</ItemCode>    
  <ItemName>Bulb LED 10W</ItemName>    
  <LineID>2</LineID>    
  <Price>15.45</Price>  
  <CurrencyISO>GBP</CurrencyISO>    
  <Qty>2</Qty>    
  <UnitID>box</UnitID>  
</Lines>

becomes:

<Lines>   
  <ItemCode>LED0010</ItemCode>    
  <ItemName>Bulb LED 10W</ItemName>    
  <LineID>2</LineID>    
  <Price CurrencyISO="GBP">15.45</Price>    
  <Qty UnitID="box">2</Qty>
</Lines>

The answer is self explanatory as soon as we will check, what will happen when we will transform changed XML into JSON.

JSON
{
  "ItemCode":"LED0010",
  "ItemName":"Bulb LED 10W",
  "LineID":2,
  "Price":15.45,
  "CurrencyISO":"GBP",
  "Qty":2,
  "UnitID":"box"
}

becomes:
{
  "ItemCode": "LED0010",
  "ItemName": "Bulb LED 10W",
  "LineID": "2",
  "Price": {
    "@CurrencyISO": "GBP",
     "#text": "15.45"
  },
  "Qty": {
    "@UnitID": "box",
    "#text": "2"
  }
}

So our simple Types in Sales Line like "Qty" -> becomes contract objects. And as we remember from previous paragraph about Contract attributes, that value in attribute is JSON tag

X++
//Sales Order Line Contract
[DataContractAttribute]
class SOLine
{
    str ItemName;
    str ItemCode;
    int LineNum;
    real Price;
    real Qty;
    str UnitID;    
    str CurrencyCode;
}

has to be modified to
[DataContractAttribute]
class SOLine
{
    str ItemName;
    str ItemCode;
    int LineNum;
    SOPrice Price;
    SOQty Qty;
}
and we need need 2 new Contract classes

[DataContractAttribute]
class SOPrice 
{
   str TextValue;
   str CurrencyISOAttr;


    [DataMemberAttribute("@CurrencyISO")]
    public str parmCurrencyISOAttr(str _Code = CurrencyISOAttr)
    {
        CurrencyISOAttr = _Code;
        return CurrencyISOAttr;
    }


    [DataMemberAttribute("#text")]
    public str parmText(str _Text = TextValue
    {
        TextValue= _Text;
        return TextValue;
    }  
  
  }
and we will need another one for SOQty which will have exat the same format value is #text and attribute has @ in front of attibute tag

IMHO using attributes in XML for D365 FO purposes make sense only if we really have to...

E.g. We have to deal with legacy system that deals only with XML. And workaround with changes of legacy system, is to much effort, so we are biting the bullet in X++.

As when we are playing with attributes JSON becomes much more complex needs more processing to be decoded.

Fixes to FormJsonSerializer

Our base in all those exercises is playing with JSON string. And to do that we are doomed to use FormJsonSerializer. However when we use it, we need to remember that it works only with "strong defined" contracts. By which I mean that all fields, objects and collections has to be defined in X++ contract classes for deserialization. Don't worry about serialization as this is always straight forward.

So if external system will start to send new extra tags, or we have missed some tags in initial definition. Our input JSON deserialization into X++ contract classes will fail. As FormJsonSerializer deserialization will fail processing on those not defined. And will finish reading prematurely so our output will be incomplete.

If we want to make it work, we need to fix method deserializeObjectInternal(). And as we can't change it standard one for obvious reasons(no way of extending it). We need our own Deserializer. So we have to create new CUSTFormJsonSerializer copy functionality from FormJsonSerializer and fix those bits:

X++ // Part of  FormJsonSerializer.deserializeObjectInternal()
..
else if (jsonReader.TokenType == Newtonsoft.Json.JsonToken::StartArray)
            {
                // This is the case for collection type properties
                if(currentJsonProperty)
                {
                    if(dataMembers.exists(currentJsonProperty))
                    {
                        // Determine if the property has a colleciton attribute
                        objectMethod = dataMembers.lookup(currentJsonProperty);
                        collectionAttribute = objectMethod.getAttribute(classStr(DataCollectionAttribute));
                        if(collectionAttribute)
                        {
                            // Deserialize the collection
                            propertyValue = CUSTFormJsonSerializer ::deserializeCollectionInternal(
                                objectMethod.returnId(),
                                collectionAttribute.itemType(),
                                collectionAttribute.itemTypeName(),
                                jsonReader);
                            // Set the property
                            objectType.callObject(objectMethod.name(), deserializedObject, propertyValue);
                        }
                    }
//WE NEED TO SERVICE EVERY ELEMENT SINCE IT IS A RECURENCE METHOD
					else
                    {
                        CUSTFormJsonSerializer ::deserializeCollectionDummy(jsonReader);
                    }
                }
            }
            else if (jsonReader.TokenType == Newtonsoft.Json.JsonToken::StartObject)
            {
                // This is the case for nested complex type properties
                if(currentJsonProperty)
                {
                    if(dataMembers.exists(currentJsonProperty))
                    {
                        // Read the object from JSON
                        objectMethod = dataMembers.lookup(currentJsonProperty);
                        // Desrialize the object property
                        propertyValue = CUSTFormJsonSerializer ::deserializeObjectInternal(objectMethod.returnId(), jsonReader);
                        // Set the proeprty value
                        objectType.callObject(objectMethod.name(), deserializedObject, propertyValue);
                    }
//WE NEED TO SERVICE EVERY ELEMENT SINCE IT IS A RECURENCE METHOD
					else
                    {
                        CUSTFormJsonSerializer::deserializeObjectDummy(jsonReader);
                    }
                }
            }
..

Conclusions and remarks

Using contract classes is "must have" for complex integrations. Especially if we don't have "middle man" to transform messages for us and 3rd party APIs has to be dealt with as they are.

Transforming XML to JSON and back, can be done on the fly, e.g. when using API Management. But sometimes messages are so big, that we will end up with shuffling this data using AzureBlobs. So we will always need some internal mechanism inside of X++. However IMO proper external libraries(for XML->JSON and back) using newest NuGet packages are better than X++ workarounds. But I have shown both ways in case someone is not comfortable with .dll addons.

And of course there is issue with FormJsonSerializer that has to be addressed...

But when we will overcome all those problems. We can start decoding and creating messages for any external system fast and reliable. More important functional code in D365 FO becomes nice, clean and efficient. Since we don't have to iterate through XML just play with class objects.

I hope that you will find some of those information useful.

Comments

Post a Comment

Popular posts from this blog

Azure BLOB storage with Dynamics 365 FnO X++

Image
Part-1: Azure BLOB storage with Dynamics 365 for operations 25 May 2020|Azure and Dynamics With the introduction of cloud based applications, Dynamics has opened doors for seamless integration with Azure services. This blog is created to portray one such example of how Azure capability can be utilized within Dynamics 365 for Operations. Lets consider a common scenario where Dynamics 365 for Operations needs exchange of files between different applications. This can be a file produced by your .Net application (or) it can be a file produced by you production system,etc,. In Dynamics AX 2012 this situation was handled by creating common folder and sharing in network path where different applications can place their files. Upon introduction of D365, local paths were not accessible by Dynamics production URL so we need to rely on common internet service for accessing the file. One of the convenient way to handle this situation is by using  Microsoft Azure BLOB storage  as the centra
Read more

Send Email from Ax / D365 FnO using X++

 static void SendEmail(Args _args) { SysEmailParameters parameters = SysEmailParameters::find(); SMTPRelayServerName relayServer; SMTPPortNumber portNumber; SMTPUserName userName; SMTPPassword password; Str1260 subject,body; InteropPermission interopPermission; SysMailer mailer; System.Exception e; ; if (parameters.SMTPRelayServerName) relayServer = parameters.SMTPRelayServerName; else relayServer = parameters.SMTPServerIPAddress; portNumber = parameters.SMTPPortNumber; userName = parameters.SMTPUserName; password = SysEmailParameters::password(); subject = "Subject line for the email"; body = "<B>Body of the email</B>"; CodeAccessPermission::revertAssert(); try { interopPermission = new InteropPermission(InteropKind::ComInterop); interopPermission.assert(); mailer = new SysMailer(); mailer.SMTPRelayServer(relayServer,portNumber,userName,password, parameters.NTLM); //instantiate email mailer.fromaddress("ax.notification@mycompany.com"); mailer.
Read more

Import Files from Blob storage using X++ - D365 FnO

 using Microsoft.WindowsAzure.Storage;  Using Microsoft.WindowsAzure.Storage.Blob;  class RunnableClassBlobStorageDownload  {      ///       /// Runs the class with the specified arguments.     ///       ///  The specified arguments.      public static void main(Args _args)      {          CloudBlobDirectory  cloudBlobDirectory;          CloudBlobClient  cloudBlobClient;          CloudBlobContainer  cloudBlobContainer;          CloudStorageAccount  cloudStorageAccount;          cloudStorageAccount  = CloudStorageAccount::Parse("Azure Blob Connection String");          cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();          cloudBlobContainer  = cloudBlobClient.GetContainerReference("files"); //           System.Collections.IEnumerable lstEnumarable =   cloudBlobContainer.ListBlobs(null, false, 0, null, null);          System.Collections.IEnumerator lstEnumarator = lstEnumarable.GetEnumerator();          List filenames = new List(Types::String);      
Read more