I think that you should try MKVtools: http://www.macupdate.com/info.php/id/24872/mokgvm2dvd
try to convert the MKV into something else like AVI and then use something like this: http://www.mp4converter.net/video-converter-mac.html
to get the video with the PS3.
Otherwise you can try VMware Fusion on your MAC to install Windows XP just to use mkv2vob on your MAC, but I think that the first solution should be ok.
You can do another thing if yo have a strong spec. Try Tversity which can decode mkv into readable format for the PS3 while streaming it as a media server.