Please don't take this the wrong way, but you are trying to play the video's via VIDEO and not via MUSIC on your PS3 aren't you?!
(It will only play MP3's/music files via MUSIC, Avi's divx/xvid etc will only play if selected via VIDEO)
(I've been caught out myself by that - browsing through video clips while in the MUSIC menu!)
Re setup & configuration:
I run mine on an ACER REVO Nettop (not very powerful at all) - Generally i don't bother with the transcoding and almost everything plays (very well).
I haven't really bothered with the transcoding, but i've got everything set as per the DEFAULTS and when i do want to transcode on the fly it works (albeit it pretty badly on my machine)
So - my situation:
1. Install it
2. Run it
3. Add some media folders, hide the #transcode# folder, hide transcode engine names, hide media library folder - leave everything else as is.
4. Ensure firewall / router are correctly configured
5. Watch near
HD quality tv shows on my PS3
(Don't get why people slate this so much - its a superb bit of software, my experience has been so much better with this that with TVersity)
Actually, i've just been fiddling with some of the transcode settings here and it's working flawlessly on my Acer Revo (running win7 with a windows rating of 3.3).
If anyone's interested in my settings, let me know and i'll post them for you maybe all you need is to tweak yours and it will work.