I came across this problem late yesterday afternoon and struggled with it again this morning until I received clarification from Microsoft on its intended functionality. I thought I would pass this along for those who may be struggling also.
I was working towards building functionality to test for the existence of line numbers in the Sales Line table where the line number had a .5 in it. For instance, if the line number was 2.5, I was attempting to perform 2.5 MOD 1 in order to get a result of .5. The only problem was that the system would not give me .5 but instead 0. It turns out that when the original AX developers wrote Axapta, they set the MOD function to only work against integers and not reals, hence the reason why I always got the whole number portion of the line number rather than the decimal part.
My workaround was writing a static method that converts the Line Number into a string, determines where the decimal place is in the string, performs a substring to pull out everything to the right of the decimal and convert it back to a real number and return it. This is essentially doing the same thing as MOD 1 would do to a whole number that has a fractional part.
Hope this bit of info helps!