One of my main goals for my PS3 was to be a semi-portable high def video player, so these types of questions also interest me.
So far I have not tried linux, but I can say that from the gameOS a 1080p MPEG2 TS format video clip is unwatchable as it cannot sustain the framerate.
What it can do is play numerous 1080p and 480i clips in the small preview mode simultaneously. I do not know enough about the algorithms involved to say what that may or may not mean.
However the poor performance of playing those 1080p TS files leads me to believe that there is no acceleration taking place, even in the gameOS. That could mean that either there is actually no hardware capability to accelerate MPEG2 at the HD level, or the system is just not configured to use it for certain formats.