My challenge, opportunity, and resolution for the new year is to update this blog more often. Let's see if I can actually follow through...
I have some new information to post regarding AIF queue management that I've known for a few months now but just have not made the time to formally post.
As with most documentation that you see for setting up a batch server, the instructions either state or imply to run both the AOS service as well as the batch client on the same dedicated server. This may at first seem like a common-sensical notion since the Batch Client, the AOS Service and the data storage all reside on the same system and will ultimately aide in reducing any network bottlenecks of large amounts of batchable transactions running across the network. However, this is not the case because the bottleneck of having both components running on the same system is the memory. Due to the 1.7GB limitation of 32-bit applications, the OS, the AOS, and the client are all contending for the available memory.
Depending on the amount of transactions that you have processing in an hour, or even a day, separating the processes out onto separate servers may not be necessary for you. In our case, we have approximately 1-500 messages measuring between 30-50KB that pass through our AX system via AIF every hour. As a result of the number of messages we process, our 'All-In-One' setup proved to be quite the bottleneck and resulted with us constantly getting 'Out of Memory' errors in our AIF queue. At one point, we had to filter out all messages larger than 200KB simply because even after freeing up all memory available, we still were unable to process those messages and import them through custom file I/O operations. After testing and working Microsoft, we found that separating the AOS Service on a dedicated server and running the client(s) on a separate dedicated server gave us our best potential performance and freed up the client to contend for memory only against the OS and the same with the AOS service.
To further increase our AIF performance, we found that instead of running all four AIF batch processes together in a single client session, separating each of the four out into separate client sessions increased performance. As a result, you are not limited to the four processes being executed in a sequential order but instead they can all three execute independent and even concurrently. Although the issue still exists that the four processes are related and dependent of each other, however, messages that are ready to be processed at the next execution of the batch are not held back from being processed.
For example, the four AIF processing classes are:
- AIFSendGateway
- AIFReceiveGateway
- AIFInboundProcessingService
- AIFOutboundProcessingService
In order to import a message into AX, AIFReceiveGateway must execute to pull the message into the AIF Queue and the AIFInboundProcessingService must execute to process and persist the data to the database. Likewise, to export a message, AIFOutboundProcessingService must retrieve the data from AX package it in the form of a message and write it to the queue before the AIFSendGateway can pick it up and write it to a message. As a result, spliting up the jobs into different batch clients will allow you to make the AIF message passing process closer to a 'real-time' process.
Finally on a separate note, I have just completed reviewing a new book on Dynamics AX that will soon be published. Once the book is officially published, I will post a link to it.
DB
3 comments:
Dave, we have been reluctant to jump into the AIF waters because of its reliance on AX batch processing.
Some of my coworkers have had bad experiences with batch clients requiring too much babysitting so they can be restarted if they fall over. I believe the problems included memory leaks, info logs filling up, and broken connections to the AOS.
Have you found AIF to be reliable? Is it ready to participate in critical business processes? Is there any way to make it somewhat self-repairing so it can recover from things like temporary AOS outtages and crashed batch clients?
Todd,
AIF can be 'flaky' at some times, with most of the flakiness I see being a result of ignorance from other people trying to manage it. I've been the predominate person managing and monitoring our AIF system, but other IT members manage it, too. If they, for instance, kill a process while it's running because they get a white screen and get impatient, you have to go in and reload the class back into the batch job.
We're a fairly small shop with only about four IT guys and only a 300 million dollar company, so it's not too much of a pain for us to monitor it, so full 100% automation is not a necessity for us although it would be nice. We looked at moving all of our transactions over to custom I/O csv-based files but decided to stick with AIF. AIF also does have the advantage that you simply configure it rather than writing code so your time investment is minimal to get it going.
To AIF's credit, however, it wasn't originally designed to handle high transaction loads throughout the day but rather running a few transactions here and there after hours. With some tweaking, I've managed to get it up to the point where it's processing about 6,000 messages per day, with 85% of the messages occuring between 8AM and 5:00PM. I think you'll see in future releases that AIF becomes a stronger integration mechanism than it is today and truely enterprise-strength.
As far as making it self-healing, nothing really there other than the obvious AOS service restart if it fails. I've actually made the launching procedure for the batches easier by changing the launch files so that it automatically launches the batch by name. I've basically got a folder on the desktop filled 12 shortcuts that I just go through and open up one at a time. I could stick them all in a single batch file but sometimes I only start certain batches.
If you have any other questions that I can answer, please let me know.
DB
Hi Dave,
I am facing issue in AIF queue manager when i try to open Queue manager, "Cannot create a record in Queue manager (AifQueueManager)".
May I know, why this error is occuring, and how to rectify this?
Thanks in advance.
Mani
Post a Comment