You can add other transcoders to Subsonic. Install the program you want to use to transcode in the folder Subsonic\transcode..
Next in Subsonic navigate to Settings >> Transcoding.
There you are able to configure the new transcoding.
Next under Settings >> Players you can define which transcoder a player uses.
Or as you can read on the Transcodingpage in Subsonic:
Transcoding is the process of converting from one media format to another. Subsonic's transcoding engine allows for the streaming of media that would not otherwise be streamable. The transcoding is performed on-the-fly and doesn't require any disk usage.
The actual transcoding is done by third-party command line programs which must be installed in c:\subsonic\transcode. You may add your own custom transcoder given that it fulfills the following requirements:
•It must have a command line interface.
•It must be able to send output to stdout.
•If used in step 2 it must be able to read input from stdin.
Note that transcodings are activated on a per-player basis from Settings > Players.
I guess if you manage to transcode a file to ogg-format and send it to stdout in your transcoding settings you'll be good.