I haven't seen any real documentation on how run 4.0 RTM and 4.0 SP1 side-by-side, so I will address the steps that I took to make this happen. I don't believe that this configuration is supported by Microsoft, so you should only do this in a test or development environment (like I am doing) and make sure you have a backup before proceeding with these steps:
1.) Backup the DB of the environment you are trying to duplicate and upgrade to SP1.
2.) Backup the entire Application folder of the environmnet you are trying to duplicate and upgrade to SP1. If you don't know what or where your Application folder is, you probably should proceed any further.
3.) If you have a 4.0 RTM client on the server where your AOS and File Server is located, you must install it.
4.) Make a copy of your Common Folder in the Microsoft Dynamics\4.0\Server\Common location and call it 'Common RTM'. This will allow you to still configure the 4.0 RTM server environments. Otherwise, you will have problems with the new configuration applet saving the RTM configs correctly since SP1 introduces new Clustering settings.
5.) Install Dynamics AX 4.0 SP1, using the steps in the Microsoft Dynamics AX 4.0 SP1 Upgrade Documentation, specifically page 8 and page 9. (Note only installing the Object Server and File Server and possibly the Dynamics AX 4.0 SP1 client if you want to be able to open ONLY the Dynamics AX 4.0 SP1 environment). Note that on Page 9, point #3, you should not include the axsys.aod file in the files you move to Application\Appl\Standard\Old - this will stop you from being able to create an Upgrade Project.
6.) Before Starting the AOS Service, open the Dynamics AX Server Configuration Utility and import/create a new server configuration that puts the new 4.0 SP1 upgraded environment on a different port than your Dynamics AX 4.0 RTM environment.
7.) Start the AOS Service as indicated on the Microsoft Dynamics 4.0 SP1 upgrade document on page 9 point #5. You can continue on with Microsoft's approved steps from this point.
The biggest two points that I can stress about attempting this is:
A.) BACKUP, BACKUP, BACKUP
B.) Don't forget to make a copy of the Common Folder. as stated in #4 above. If you forget this step and try to open a Dynamics AX 4.0 RTM Server configuration file with the Dynamics AX 4.0 SP1 Server configuration applet and save it, you will hose up your server service the next time you stop and start that server service.
Remember, Microsoft does not endorse my steps. These steps worked great for me but may not work so great for you. Please backup and plan accordingly for issues. Please do not take your issues with these steps to Microsoft.