If anyone is interested there is a ripkit for Rock Band to downsize the ISO to fit on a DVD5.
The ripkit works perfectly I am able to play the entire game with no problems. Here is the read me file that came with the ripkit.
------------------------------------------------
Rock Band DVD9->DVD5 Ripkit
------------------------------------------------
By Psychospacefish
Media: 1 DVD5 (4.36GB)
Ripped: Intro/Bonus/Credits movies, other movies downsampled to 600kbps.
NOTE: The game will "freeze" if it tries to play any of the deleted movies (see above), just press start or the green button on your guitar controller to continue.
NOTE2: Even though the disc image is 4.36GB, it still fully works on Swap Magic 2.
WARNING: Do not change any part of this kit, or else the game will crash.
1.Extract from an ISO image all the files from both layers with Apache 3 into the same folder as the ripkit (you can delete the ISO after this)
2.Run rip.bat
4.At the end IML2ISO will pop up, select rb.iml and click start.
5.Burn the ISO at 4X with DVD Decrypter.
Big thanks to gnie for DVD IML2ISO and neness for Kit Magician, and the makers of PCSX2.
Hopefully I will be able to post the site to go to. The only problem is the files are through rapidshare so it takes a while to download since the file 1.18Gb.