It all depends on the bandwidth at each end. Upload speed from the server and download speed at the laptop. If your on a 56k modem at the server, it won't handle 320kbps mp3 going out very well.
I recommend doing speedtests at both ends:
http://www.speedtest.net/Use the server at Clifton, NJ for the most consistent measurements in the US. the ping time is very low and I always see my rated bandwidth very accurately.
Once you have established an end to end connection, if there's a bottleneck, you have to examine all the factors through the chain to see what the problem is. If everything works inside the network, then clearly problems outside are most likely bandwidth related. In a work environment, other users on the network can use bandwidth load that is shared with you. Are you all on the same pipe? Through the same switch? What are you listening to? MP3? Flac?
What kbps is the server sending out at?
Some business environments have packet inspection protocols in place that can slow things down, Have or are you using an SSL connection? SSL packets sometimes work better in workplace settings as the encryption prevents packet inspection.
Have you tried connecting from a different setting? Have you tested, connecting to Sindre's demo server or another known good server at work? I can let you connect to my server if you want from work to see if that works. That would let you eliminate you work network as an issue. I'll send you the link on a PM.