Low end quad core may not be enough either. I'm suggesting that as a minimum to get perhaps to 720p.
Video performance is dependent on many factors. It's a true "weakest link" scenario that involves CPU (including internal cahe size) power, available memory, network capacity, hard drive speed, and upload bandwidth from and download bandwith at the recieving end. Can't begin to tell you how often I've explained this to users who think there is something wrong with Subsonic. Stop and think how often you get a Youtube video that buffers even with their resources. The typical home network has nowhere near that capacity, yet I often see users post that they think poor video performance = somthing wrong with Subsonic.
After several years of isolating and tweaking each link in the chain, I've finally got what I think are acceptable results on my video server that has dual quad core xeons running at 2.83GHz, 16 GB ram, dual nics aggregated, gigabit lan, connected to the outside with 35/35Mbps Verizon Fios.
Granted your levels of acceptable performance might be found at a less powerful level depending on your useage and typical resolution of the native video. I primarily use Subsonic video for access to recorded TV over the internet when I'm not at home. That's at 1080i. It works well as long as I can get a decent download speed from the remote location.
Can setup and minor tweaks affect things? Yes, certainly, but not to the degree where it will totally compensate for any hardware or bandwidth issues. For video transcoding you need raw power.
That's why I often advise users that have lower power servers should preformat the videos to flv or mp4 and turn off the transcoding and just pass the video directly to JW player or the phone (flv). That way you're only dealing with bandwidth delivery issues and the CPU power is eliminated from the equation. I ran several tests comparing both approaches and it's incredible to see the difference. Preformatted video delivered directly without transcoding drops CPU use to so negligible you won't see it while you monitor the CPU useage compared to seeing it pegged out when transcoding.
Everyone complains that it takes to long to preformat the video. Sure, because then you see how hard the CPU has to work to make the conversion when reformatting. Well that's exactly what you do repeatedly when you ask the server to do on demand transcoding. It's the same issue - it takes a ton of resources. You can do it once deliver many times easily, or struggle every time you play back a video.
I have software that can monitor a directory and automatically convert files to desired formats at predefined load limits in the background and then store the result in a seperate directory thus automating the process, but it's a tedious configuration that I don't use now that I have the server power.
Bottom line - if you're having trouble with video, it's not inherently a Subsonic flaw, but more likely a hardware horsepower issue.