No there doesn't seem to be a file size limit. I copy over 20GB mpeg2 HD movies to watch all the time.
I've read there does seem to be file size limit on mp4 files. You can use Vilvic Media Center to transfer files over your network.
For the PS3 to recognise it as playable it has to be mpeg ps.
This is what I do to make HD mpegs playable.
1. Get VLC.
2. Open up the streaming and transcoding wizard (its under the file menu).
3. Click transode/save to file press next.
4. Then select your stream, press next.
5. On the Transcode page make sure none of the options are on, turn them off if they are. You don't want to transcode your just remuxing. Press next.
6. Chose MPEG PS, press next.
7. Chose your save destination, press next.
8. Check if the details a correct and press finish, then wait while it does it business.
This works on PC and Macs, all you have to do is then transfer it to your PS3.