Software Development Quality Assurance in AX

It's been over a year and a half since my last post, but I'm turning the ship towards a new course: Software Quality. During the hiatus, I've been working on multiple projects simultaneously which has taken me away from the day-to-day habits I used to engage like blogging on AX. My workload increased due to attrition and the economic situations not allowing those individuals to be replaced.

As can be surmised, the more work I take on the more my quality has dipped. This is not as a result of my work ethic but rather the need to compact three full-time equivalents' jobs into a single full-time equivalent's day plus a few hours. Although the quality issues have been extremely minor and have resulted in no downtime, I still feel that zero quality issues are needed in all circumstances.

To illustrate the dynamic nature of the position I have grown into (and many other Dynamics AX developers are facing), in the short time since I started writing this entry this morning, I have already had the following requests:

• Researching a Windows 7 64-bit driver for a USB WiMax adapter
• Reset an AD password
• Clear out space on a linux server drive that had become full
• Get the application back up and running that failed on the linux server as a result of the out of disk space issue
• Have our corporate nagios administrator setup an alert to ensure we are notified when the drive becomes 80% full

As you know, AX development is the majority of my job, but as you can see, systems administrator is now a part-time position for me.

With all that being said, the subject of this blog is changing to focus on helping the small-shop AX developer, who has other duties besides development, move towards having a very low error injection rate into the AX code they deploy.

David Bowles

Convergence 2008 - Day 1

While the first day of Convergence, Partner Day, was slow for a customer like myself, it will pick up tomorrow. This has been my first Convergence attendance so of course like all first-timers, I've been wowed my the amount of cash and effort Microsoft has put into this conference. Dynamics AX 2009 is the focal point on the AX side of the product offerings and judging by the looks of the amount of Marketing material, hand-on labs, and sessions Microsoft is providing, they're releasing it the 'right' way compared to my experience with the 4.0 launch in 2006.

One bit of information that I've found a strong marketing campaign towards during Convergence is Dynamics Communities. I've been a member of the Finance Community for a while now but a 'Sales and Marketing' and 'Customer Service' community has recently launched. I encourage everyone who is currently not a member of at least one of these communities to get involved, especially if you do not have a support contract with either your partner or Microsoft.

Clicking on the link below will take you directly to the registration page. When registering, I kindly ask that you place my Dynamics Community screen name as your referral code, davidbowles.

https://www.dynamics.com/UserProfile/HaveLiveId.aspx

DB

Convergence 2008 - Orlando

It's Friday finally! I haven't posted anything about Convergence, but better late than never. I will be at Convergence for anyone who is interested in meeting while I'm there. To help facilitate this, I have setup a Convergence Connect profile. While a portion of my time will be working at the technical services area of Convergence, I'll have other time to attend sessions and socialize as well.

My main focus at Convergence is SQL Performance and PerformancePoint Server. Those are the two areas I'm hoping to make great strides in by mid-2008.

While I have had many emails and comments asking about the results of the poll I was attempting to take on AIF usage, I have never received any responses on usage. I'm not sure if this was due to not having time or literally no one who reads my blog using it, but if you're of the earlier persuasion, please come by and talk to me for a few minutes. I'd really love to hear your success stories or otherwise in AIF.

Look forward to seeing everyone there.

David Bowles

Do and Don't Lists

As an added service to you the developer or administrator, I have created two lists named "Do" and "Don't", respectively. This is a quick hit list of things that you should at bare minimum entertain the idea of researching the possibility of implementing or definitely not doing at all.

You will see these lists in the right margin at the end of the 'widgets'. If you have a personal lesson learned that you think should be in the list, please send me an email.

DB

Creating 'Permanent' Indexes on DataAreaId in Dynamics AX

Update - 3/3/2008...

I've received a couple of comments regarding this post. Let me add two points to my original post. First, this was more informational than an actual suggestion for your own system. As I stated below, if you have an index with only the DATAAREAID column selected as a 'Non-Clustered' index, this will work. If you add any other additional columns, then AX will remove it when a synchronization is ran. One caveat to this - I am doing this in RTM, so there is always the possibility of this not working in a later release of AX like 4.0 SP1 or SP2. While I'm not expecting to get any performance gain, I am expecting to silence the Query Tuning Wizard on requesting these indexes. As with any performance-related change, I'll monitor how this change affects our installation and adjust accordingly. If I find no harm in their existance, I'll probably leave them there.

I appreciate everyone's feedback on this posting...

DB

Below is the original post:

As I get further from our Implementation Date, I'm transitioning from my original job of developer to more of a DBA/Business Analyst role. The primary reason for the DBA role is due to performance becoming increasingly more of an issue as our database grows and the once amateur users maturing into more novice users demanding even more resources out of the system.

As I work towards optimizing indexes, monitoring resources, and optimizing X++ code, I notice that SQL balks a lot at tables missing indexes with only the 'dataareaid' field within it. As I've always known you cannot add an index to the AX database directly, likewise, you cannot add an index to AX's AOT with only the dataareaid field. Today, I stumbled upon a way to actually accomplish this. Most of you may be aware of this already but I've personally never seen a blog entry or MSDN entry on how doing this to satisfy SQL statistics. While this may not optimize the system at all, I do get asked by my boss a lot why I haven't created these indexes to satisfy what SQL believes it should have. I do know I haven't seen any issues from implementing these indexes.

If you'd like to see if this will help in your server's performance, please try this first in your test/dev environment before even considering making this modification in your production environment. I'm sure this 'hack' is not supported nor advised by Microsoft.

If you are unfamiliar with the 'Missing Indexes' query, try running the following query against your SQL database:

SELECT TOP 20
[Total Cost] = ROUND(avg_total_user_cost * avg_user_impact * (user_seeks + user_scans),0), avg_user_impact, TableName = statement,
[EqualityUsage] = equality_columns, [InequalityUsage] = inequality_columns,
[Include Cloumns] = included_columns
FROM sys.dm_db_missing_index_groups g
INNER JOIN sys.dm_db_missing_index_group_stats s
ON s.group_handle = g.index_group_handle
INNER JOIN sys.dm_db_missing_index_details d
ON d.index_handle = g.index_handle
ORDER BY [Total Cost] DESC;

This query will give you what SQL believes are the top 20 missing index structures in your database.

If your system is anything like ours, at least 1/5 to 1/10 of these will be Indexes with only the 'DataAreaId' column specified. To add this index in, just go right to the Indexes node within the SQL Table and create an index. Only index on the 'DataAreaId' column and set the index type to 'Non-Clustered'. Click Ok. If you run a synchronization against the table in the AOT, it will not delete the index. If you now rerun the query above you will see the record no longer appears.

You may ask why I even bothered doing this. Well the answer is while the Total Cost and Average User Impact are all relative numbers, over the last several months I have lowered the highest Total Cost to around 10 billion and all of the records between 1 and 10 billion were all missing 'DataAreaId' indexes. After implementing four indexes the SQL statistics said I needed, my highest Total Cost is now only 28 million.

I hope this advice may prove to be helpful or if nothing else serve as a 'Nice to Know'.

DB

Happy New Year!

Along with a new year comes new opportunities, new resolutions, and new challenges for everyone to realize.

My challenge, opportunity, and resolution for the new year is to update this blog more often. Let's see if I can actually follow through...

I have some new information to post regarding AIF queue management that I've known for a few months now but just have not made the time to formally post.

As with most documentation that you see for setting up a batch server, the instructions either state or imply to run both the AOS service as well as the batch client on the same dedicated server. This may at first seem like a common-sensical notion since the Batch Client, the AOS Service and the data storage all reside on the same system and will ultimately aide in reducing any network bottlenecks of large amounts of batchable transactions running across the network. However, this is not the case because the bottleneck of having both components running on the same system is the memory. Due to the 1.7GB limitation of 32-bit applications, the OS, the AOS, and the client are all contending for the available memory.

Depending on the amount of transactions that you have processing in an hour, or even a day, separating the processes out onto separate servers may not be necessary for you. In our case, we have approximately 1-500 messages measuring between 30-50KB that pass through our AX system via AIF every hour. As a result of the number of messages we process, our 'All-In-One' setup proved to be quite the bottleneck and resulted with us constantly getting 'Out of Memory' errors in our AIF queue. At one point, we had to filter out all messages larger than 200KB simply because even after freeing up all memory available, we still were unable to process those messages and import them through custom file I/O operations. After testing and working Microsoft, we found that separating the AOS Service on a dedicated server and running the client(s) on a separate dedicated server gave us our best potential performance and freed up the client to contend for memory only against the OS and the same with the AOS service.

To further increase our AIF performance, we found that instead of running all four AIF batch processes together in a single client session, separating each of the four out into separate client sessions increased performance. As a result, you are not limited to the four processes being executed in a sequential order but instead they can all three execute independent and even concurrently. Although the issue still exists that the four processes are related and dependent of each other, however, messages that are ready to be processed at the next execution of the batch are not held back from being processed.

For example, the four AIF processing classes are:

• AIFSendGateway
• AIFReceiveGateway
• AIFInboundProcessingService
• AIFOutboundProcessingService

In order to import a message into AX, AIFReceiveGateway must execute to pull the message into the AIF Queue and the AIFInboundProcessingService must execute to process and persist the data to the database. Likewise, to export a message, AIFOutboundProcessingService must retrieve the data from AX package it in the form of a message and write it to the queue before the AIFSendGateway can pick it up and write it to a message. As a result, spliting up the jobs into different batch clients will allow you to make the AIF message passing process closer to a 'real-time' process.

Finally on a separate note, I have just completed reviewing a new book on Dynamics AX that will soon be published. Once the book is officially published, I will post a link to it.

DB

Comment Spam

Due to a recent surge in the amount of Anonymous comment spam, I am disabling anonymous comments. If you'd like to leave a comment, you will need to register with blogger.com.

Sorry for this inconvenience.

DB

AXUG Summit 2007

As our post-implementation deployment is wrapping up, I am going to make a point of posting more to this blog now that my time is freeing up to do other things. On October 15-17, I will be attending the AXUG Summit in Orlando, FL. There appears to be some great tracks to attend. I will be attending the performance track to see if I can't return with some knowledge on how to speed up some slowness in our invoicing process. If any of you will be attending, let me know and I'll be looking to speak to you.

Just to fill you in now that things have calmed down, here's our implementation story:

My lack of posts for the last four months have not been because I was too lazy but rather because of being overwhelmed with work. Our implementation was rocky but we made it work and persevered as a result. We have a VP who in my opinion is always a stellar performer. He's the sort of guy that even if everyone is looking at doom and gloom, his head is still held very high. If you ever see this guy looking down you better cover your head because the sky is surely falling. In my opinion he had a lot to do with the project's success. When all seemed as if it were about to fall apart, he was there to encourage all of us on the project and cheer us towards success.

My shop is pretty small, less than eight regular IT employees and due to some recent departures is even smaller now. I was and am the only full-time developer for AX in our company during our implementation. I also have been the only person to become technically fluent with AX from installation to configuration to troubleshooting, etc. Why was our implementation this way? To some extent I can blame it on a lean budget but I can also point to our partner.

Hind sight is always 20-20, of course.

We were new to AX. I had just been hired in February 2006 and had the VPC of AX 4.0 Pre-Release given to me in June, no manuals, no documentation, just a job of learning it. Our Project Team was a who's who of the shining stars in the company, one from each functional area of the company. Not one of them had ever heard of AX, and only two or three had ever worked with or knew what an ERP system was. I was the black sheep being from a technical environment and actually having a two month head start on learning it.

Our partner rolled in and immediately setup a project timeline and a go-live date, which we ultimately missed and had to reschedule three times. All seemed as if it were going well until we began official software modification in October. Developer resources from the partner were not cutting it. Each of the three guys who all in total spent less than two weeks with us seemed to have no real idea what was going on. These guys were asking me questions about AX... a scary thought. But the more scary thought was that I actually had the answers. As a result we determined very quickly that either the seasoned resources were tied up (as we were being told, although each of the three guys we received were 'Senior Developers', or seasoned developers did not exist - my personal belief). Before long into the development/modification phase, we hired a previous contractor of another in-house developed system and cut a lower hourly wage for me to teach him AX and X++ and have him at our disposal as needed. This turned out to be a better decision than the partner's own developers.

In the end, we were still allowing our partner to make architectural decisions since they were the experts. They made decisions that I even knew were terrible although my experience in AX was lacking. At the time those design decisions were made, we accepted them and moved forward. Now looking back we all wished we had challenged their solutions.

I share a glimpse into our implementation to convey a notion to you if you are the customer: Know AX.

While I'm sure most implementations are packaged deals (like ours) where a team arrives and educates you first and then immediately begins the project, I suggest a different approach. If you can work it out, install the product first, allow your technical people and key functional people access to it and all of the training manuals for no less than a month. They need to understand what the product is, how it works in its pure state. Then and only then start the implementation project. If you as the customer rely on your partner 100% you, too, will experience modifications/business processes that were implemented but not fully thought out. The ramifications can be from as simple as a quick rewrite of a function to a complete overhaul in the next major AX release.

If you are a partner, I encourage you to encourage your client to learn what AX has to offer and how it works from the beginning. Even if the Sales Order process looks nothing like what your customer will expect in the end, still encourage them to create an Item that they sell, create a customer, create the ledger accounts, execute a sales order. Make them understand how AX works before you begin consulting them on Architectural changes. In the end you will profit from a long-lasting business relationship rather than a continous finger-pointing who-did-what-wrong conflict.

And in the end, isn't your goal a happy customer and continuous reference?

Hope this helps those of you either in or preparing to enter an AX implementation, or for that matter any ERP or large-scale software implementation project.

DB

P.S. Are we happy with AX? Of course we are, but we weren't there on July 1st, the day we went live. Much massaging, cleanup, multiple phone calls to Fargo, several 36 hour+ days, and six weeks of no weekends or weekdays off had to be experienced before happiness was achieved but all in all we have learned from our mistakes and it has made us a better shop in the end.

AIF Email Survey

As you may know, my company went live on AX on July 1st. As we are still working to fully stabilize AX and clean up any loose ends, I am still working to tweak AIF and am 90% toward our goal. As with any application, you can test like there's no tomorrow, test large loads, small loads, erroneous loads, and everything in between but there's no substitute for real life.

I'd like to conduct an email survey for anyone this is applicable to and willing to share usage statistics. I'd like to know how your company is fairing with AIF in a live environment. I'm looking for AIF data only, no Commerce Gateway or custom asciiIO/commaIO input/output processes.

If you're interested in participating, please send an email to: david.bowles@yahoo.com. In the subject line, please put 'AIF Survey', without the quotes. In the body, please copy and paste this template with the information you're willing to share filled out:


***************************
Months Running Live on AIF:



Number of Actions Live on AIF:
Number of 'Out of Box' Actions Live on AIF:
Number of 'Customized From Scratch' Actions Live on AIF:
Number of Sales Order-based Actions on AIF:
Number of Purchase Order-based Actions on AIF:
Number of Production-based Actions on AIF:
Number of Inventory-based Actions on AIF:
Number of Accounting-based (AR/AP/GL) Actions on AIF:



Number of external Systems interfacing AX through AIF:
Using AIF to communicate via EDI? Yes or No
If Yes, What EDI Format and Version do you use:
If Yes, also what Message Brokering/Transformation software do you use, if any:



On a daily average, how many errors do you encounter in your AifQueueManager needing correction?
On a daily or monthly average (please specify which) how many AX AIF transactions do you receive? Inbound _____ Outbound _____
What frequency do you process AIF transactions? Minutely, Quarter Hourly, Hourly, Bi-Daily, Nightly, Monthly



How would you classify the performance of AIF in producing/consuming transactions? Slow, Medium, Fast
Based upon your answer above, does AIF's speed meet your expectations? Yes or No
If AIF's speed is Slow or Medium or it's speed does not meet your expectations, have youworked with a DBA on optimizing database transactions during AIF execution to speed upthe transactions?
If so, in comparison prior to the optimization, does AIF run faster, slower, or the same?



What type of business are you?
***************************


I'd like to thank you in advance for sharing this. After a week or so, I'll post the results of the survey. You're welcome to send this to me totally anonymously. I will not be posting any information regarding the submitter. I'm interested in how many implementations are actually using AIF and to what extent as I'm sure all of you who are using it are interested as well.

DB

No, I haven't abandoned this blog...yet! :-)

Hello All,

I apologize for my nearly two month delay on posting any new updates to the blog. I've been incredibly busy, but isn't everyone? We pushed back the go-live to July 1st due to some 'technical' difficulties - not AX related. In the meantime I've been doing a lot of testing and developing some 'new requirements' also.

To top it all off, I have also been preparing for a wedding as I'll be hanging up my bachelor's cap once and for all on June 16, therefore, I'll be out of contact from June 15 through June 24. Once I get back to the states, I'll be sure to respond to any messages you email to me. The week of June 24th I will be preparing for our July 1st go-live, so I'll be unavailable mostly that week and the week following. Once we get AX stabalized I will be back on board making regular posts.

I can't really think of anything new to talk about in terms of Dynamics AX 4.0. I have been working on some simple data exports outside of AIF using the standard asciiIO/commaIO classes. I am finding that AIF works better with smaller amounts of records rather than larger. We have a couple of outbound files that exports more than 60,000 records. We are seeing minutes worth of difference between AIF and asciiIO exports. I'm hoping that Microsoft realizes an opportunity to optimize the AIF for larger record loads in a later release of AX.

Also, I want to thank everyone for their support. I get emails and instant messages daily from you guys and I'm glad to help as much as possible. I look forward to talking to each of you soon!

David Bowles