Awfully slow streaming -- oodles of bandwidth

[bdowne01]

I stumbled across Subsonic a few weeks ago, and I absolutely love it. And donated!

I'm having a recurring problem that's very frustrating, and Googling hasn't really gotten me anywhere. Here's my setup:

System / Network Info
Windows 7 Home Premium
Core 2 Duo 2.4 Ghz (2010 Mac Mini) w/4GB RAM
Netgear Stora NAS
Commercial Cable Connection (Charter) 50mb/down - 10mb/ up.
All the above GigE connected with a Dell PowerConnect switch

Subsonic Info (from the About section)
Version 4.7.beta2 (build 3012) – June 9, 2012
Server jetty-6.1.x, java 1.6.0_33, Windows 7 (47.7 MB / 96.7 MB)

In short, I use it to stream my music library from my home to my office. Both my home and office have cable connections from the same ISP (Chater), and general transfers are insanely quick -- mostly because it's never leaving the local network at Charter. Often I get transfer rates 9Mbit/s+ when pulling files from home. Nearly certain it isn't a bandwidth problem. Additional, the Web UI is very responsive and snappy.

If I shut down the Subsonic service and restart it, things will work fine at the office for about an hour or so, then streaming starts to get very very slow. The UI, everything else within the app is very very fast except streaming. Ultimately it gets to the point where it'll take nearly 20 minutes to buffer a 3.5MB MP3. Obviously it's unusable at that point. If I run home and restart the service, it returns to normal for a while and then back to slow.

It's not running on a screaming PC, but generally I'd imagine streaming to only one client shouldn't be all that intensive. I'm also tried/disabled transcoding to no effect--just slow slow.

I found a few posts regarding slow streaming when the music is on the NAS, however I've also reproduced this issue by using a directory on the local C:\ drive of the PC, so I'm pretty sure it's not that either.

Anyone have an idea what I should try next? I love Subsonic, and I even donated, it's just frustrating that it seems to have problems with it's core purpose for me

[GJ51]

I'd start wondering if maybe your ISP isn't throttling the upload side from your home. I'd remote in to the server when it happens and do a speed test from Speedtest.net on the server to get a good reading on the upload speed.

Also, do you have SSL enabled? That might be helpful.

[BKKKPewsey]

Also what memory have you allocated to SS?
Things start getting veeeery slow when SS starts running low on memory.
I would suggest allocating 256Mb unless you have a very large library (200k+) in which case go higher.

[bdowne01]

Thanks for the responses, guys. I don't believe Charter throttles commercial accounts, but I'll call to verify. Also, not running SSL. Didn't think the extra CPU overhead would help at all.

I'll also try increasing the memory limit--that sounds like it might be a culprit. Is that just done in the system tray app?

Edit: Since it's the weekend, I just upped the memory, restarted, and gave it a shot from here at home using the Mac Mini's private (192.168.x.x) address. Still streaming slow to the point of stopping playback! Weird! Interestingly, downloading the song only resulted in about 600kbit/s speed (laptop was physically jacked into the switch, so GigE connected). Watching task manager on the server showed virtually no CPU contention, nor memory problems. Very strange.

[GJ51]

Right click on the sys tray icon, Select Control Panel, Stop the service, go to Settings tab, change mem limit to 256, save settings, go back to Status tab, restart SS service.

EDIT: I was postin while you was edittin'.

Keep in mind that some ISP's specifically state in their service agreements that you're not supposed to be hosting a website on some types of accounts, usually residential, but it may also apply to bus. accts that don't specifically contract for it. That's why I suggested setting up SSL so that your ISP can't see the nature of the content being transmitted.

You also quote transferring files from the NAS, but is that directly or through the PC that hosts SS? We've seen many problems with getting good throughput when connecting to NAS devices. Chances are that if the host is pulling the music fast enough when you test on the local host then you're probably OK.

If all else fails I'd do the old uninstall/reinstall of both the latest Java and SS.

[bdowne01]

GJ51, thanks again for your reply!

I think through process of elimination and using Windows' Resource monitor it does indeed seem to be a problem with Subsonic streaming files off the NAS. I created a local folder on the PC, verified I had a problem streaming some 320kb-encoded MP3s when they're on the NAS. I then added the local folder to Subsonic, copied same MP3s there, rescanned the index, disabled the NAS shares and attempted to play them... worked without issue at all. I repeated this with two other 320kbit-encoded albums with exactly the same results.

Using the process-based network drill down in Resource Monitor, I can see when streaming local MP3's, Subsonic is easily pushing upwards of 4Mbit/s. When attempting to stream off the NAS, it can barely muster 300Kbit/s. Tests outside Subsonic show the NAS is capable of pushing just about 35Mbit/s. I'm a bit of a network guy (I own an ISP ), so I'm often dealing with network-related troubleshooting--and believe I've seen this mistake made by other software projects in dealing with network storage.

I'm nearly certain the method that Subsonic is using to stream MP3's is to open the file, and then do a block-by-block read of the data whilst streaming it. This results in many, many, many tiny I/O's which will completely bottleneck any type of network-attached disk resource due to extrapolated latency problems. Monitoring the process resources indicates that's exactly what I'm seeing: The Subsonic service is almost always in a network I/O-wait situation.

Instead, when dealing with network storage, Subsonic should slurp the entire file in to a memory buffer and stream it from its own in-RAM cache. Of course, this could result in memory problems if streaming massive files--but there's an equally effective solution: If the resource store is a network path, simply OS-copy the file to a local cache folder when it's being streamed, then remove it after moving onto the next file. I noticed Subsonic already has a checkbox regarding network storage, it wouldn't seem all that involved to maybe implement another checkbox next to storage resources indicating it's a "NAS Share" or similar to turn on the aforementioned local-caching behavior.

I'd love to help out and make improvements but my knowledge is limited to Shell scripts and a bit of Perl. If I can help out in anyway to make this project work even better I'm game! I love Subsonic

[GJ51]

I actually think that there is any easier fix that is already available that just involves installing a windows capability to handle the differences in file structures better. I've seen others address this issue before with success, but I'd have to do some searching to find it as it never was a factor on my setup.

It has to do with Windows being NTFS and the NAS using a different file structure and plugging in something that does a better job of getting windows to translate the data. I know it's around here somewher, if I see it afgain, I'll post a link.

[bdowne01]

Alright, so I took some time this evening and decided to dig through the source code. Though I roughly "get it" (seriously, I can't decipher Java), I think I found the culprit which appears to be doing exactly as I thought (but remember, I can't decipher Java):

In ./subsonic/io/InputStreamReaderThread.java, I found this:

Code: Select all

    @Override
    public int read() throws IOException {
        byte[] b = new byte[1];
        int n = read(b);
        return n == -1 ? -1 : b[0];
    }

    @Override
    public int read(byte[] b) throws IOException {
        return read(b, 0, b.length);
    }

From what I can tell (did I mention I can't decipher Java?) it appears to be attempting to read the media files in the playlist one(?) byte at a time. I have no idea where 'b.length' is being set, or if that's just a int that's carrying the length of the data. Trying to read up on the read() & byte functions at docs.oracle.com left my right eye bleeding. Either way this code looks pretty bad from a NAS perspective. If it is indeed reading files one byte at a time, that could result in thousands upon thousands upon thousands of individual, teeny network I/O's to disk which is a horrid waste on Ethernet (remember, a packet is 1500 bytes or so). Even if it's reading files a few hundred bytes at a time, that's not so good.

Disclamer:
* I don't know Java (I think I mentioned something about that)
* I could be talking out my posterior. That code could be calculating pi (from a file?) for all I know.
* I'd even attempt to fiddle at it, but I seriously don't even know where to start with a .java file to make it ... java binaries? Is that a thing?

Anyone willing to delve into this with me and attempt to improve it? What I can offer is network-guru mojo to lend a helping hand and test the bejesus out of it.

[hakko]

I had a look (or I made a search) in the source code of Subsonic 4.5, 4.6, 4.7beta1 and 4.7beta2, in the module subsonic-main. The only place where I found

Code: Select all

byte[] b = new byte[1];

was in java.net.sourceforge.subsonic.io.PlaylistInputStream?

You're using 4.7beta2, right?

[bdowne01]

Hey Hakko, you're correct. I misstated the file, I had the correct directory but you're right on the file: PlayListInputStream.java

Thanks for correcting me. And yes, 4.7-beta2.

[hakko]

It looks like PlaylistInputStream is only being used from StreamController, which calls the read(byte[] b) method with a buffer of size 2048. read() isn't called explicitly, I think it's there simply to fulfill the Inputstream contract. I guess you could add logging to that method if you're suspicious and play with the 2048 constant to see if it makes a difference while communicating with your NAS.

[bdowne01]

Thanks again Hakko. It seems like you're at least roughly familiar with the source. I'm looking for the point in the code at which the file is opened and read in from the filesystem for the purposes of streaming. Do you think this is the right spot?

I'll figure out how to tweak it and test a bunch of different settings in my environment. My only problem is just knowing where to start.

Again, appreciate the help! Hopefully I can get this resolved for lots of other NAS users with a bit of help from the Java-fluent.

[hakko]

It looks right when I had a glance. It keeps a reference to "currentInputStream" and "currentFile" and passes statuses of what's going on from there. Add some logging there and try it.

I guess you should just start off by installing Java and Maven, maybe an IDE like Eclipse or IntelliJ, and check out the whole source code. Go to subsonic-main in a terminal window and issue the command "mvn package" and Maven should build you a brand new subsonic.war in the target directory.

[bdowne01]

Got it. I'll give that a go and report back. You've been a great help!