Out of memory problems despite increasing mem limit(Unbuntu)

[tvlenthe]

>>Problem Description: VERY LARGE Music library (>92,000 files in hierarchial directories including large artworks). Server starts out fine, including remote web access but then stops responding (even on localhost). Service status shows subsonic service still running. Great while it works, but doesn't last too long (minutes to a few hours). Thanks for any suggestions!

>>Subsonic Version: 4.4 (build 2169) – February 6, 2011

>>Server Version: jetty-6.1.x, java 1.6.0_22, Linux (80.9 MB / 90.4 MB)

>>Troubleshooting Steps: Increased memory to: SUBSONIC_MAX_MEMORY=7500 (m suffix appended in .sh script)

>>>>Last lines subsonic_sh.log:

Code: Select all

46240 [btpool0-0 - Acceptor0 SelectChannelConnector@0.0.0.0:4040] WARN org.mortbay.log - dispatch failed!
Exception in thread "btpool0-0" java.lang.OutOfMemoryError: Java heap space
   at java.lang.String.toCharArray(String.java:2737)
   at java.lang.Thread.init(Thread.java:370)
   at java.lang.Thread.<init>(Thread.java:437)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.<init>(BoundedThreadPool.java:414)
   at org.mortbay.thread.BoundedThreadPool.newThread(BoundedThreadPool.java:374)
   at org.mortbay.thread.BoundedThreadPool.dispatch(BoundedThreadPool.java:94)
   at org.mortbay.jetty.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:86)
   at org.mortbay.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:68)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:502)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:514)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
46242 [btpool0-5] ERROR org.mortbay.log - Error for /dwr/call/plaincall/chatService.getMessages.dwr
java.lang.OutOfMemoryError: Java heap space
   at org.springframework.util.StringUtils.tokenizeToStringArray(StringUtils.java:944)
   at org.springframework.util.StringUtils.tokenizeToStringArray(StringUtils.java:915)
   at org.springframework.util.AntPathMatcher.doMatch(AntPathMatcher.java:98)
   at org.springframework.util.AntPathMatcher.match(AntPathMatcher.java:76)
   at org.acegisecurity.intercept.web.PathBasedFilterInvocationDefinitionMap.lookupAttributes(PathBasedFilterInvocationDefinitionMap.java:115)
   at org.acegisecurity.intercept.web.AbstractFilterInvocationDefinitionSource.getAttributes(AbstractFilterInvocationDefinitionSource.java:39)
   at org.acegisecurity.intercept.AbstractSecurityInterceptor.beforeInvocation(AbstractSecurityInterceptor.java:264)
   at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:104)
   at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
   at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.RequestEncodingFilter.doFilter(RequestEncodingFilter.java:43)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.ParameterDecodingFilter.doFilter(ParameterDecodingFilter.java:54)
66026 [btpool0-4] ERROR org.mortbay.log - Nested in org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space:
java.lang.OutOfMemoryError: Java heap space
   at java.util.regex.Pattern$BnM.optimize(Pattern.java:4929)
   at java.util.regex.Pattern.compile(Pattern.java:1490)
   at java.util.regex.Pattern.<init>(Pattern.java:1150)
   at java.util.regex.Pattern.compile(Pattern.java:840)
   at java.lang.String.split(String.java:2304)
   at net.sourceforge.subsonic.service.SearchService$Line.pars

...

>>>>Error message in localhost browser window:

Code: Select all

Exception    java.lang.OutOfMemoryError
Message    GC overhead limit exceeded
Java version    Sun Microsystems Inc. 1.6.0_22
Operating system    Linux 2.6.38-8-generic-pae
Server    jetty-6.1.x
Memory    Used 82 of 91 MB
Stack trace    java.lang.OutOfMemoryError: GC overhead limit exceeded at org.springframework.util.StringUtils.tokenizeToStringArray(StringUtils.java:944) at org.springframework.util.StringUtils.tokenizeToStringArray(StringUtils.java:915) at org.springframework.util.AntPathMatcher.doMatch(AntPathMatcher.java:99) at org.springframework.util.AntPathMatcher.match(AntPathMatcher.java:76) at org.acegisecurity.intercept.web.PathBasedFilterInvocationDefinitionMap.lookupAttributes(PathBasedFilterInvocationDefinitionMap.java:115) at org.acegisecurity.intercept.web.AbstractFilterInvocationDefinitionSource.getAttributes(AbstractFilterInvocationDefinitionSource.java:39) at org.acegisecurity.intercept.AbstractSecurityInterceptor.beforeInvocation(AbstractSecurityInterceptor.java:264) at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:104) at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at net.sourceforge.subsonic.filter.RequestEncodingFilter.doFilter(RequestEncodingFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at net.sourceforge.subsonic.filter.ParameterDecodingFilter.doFilter(ParameterDecodingFilter.java:54)

>> Playback Application and version: Goal is android, but error happens on LOCALHOST

>> Hardware Platform: Quad core AMD 9570, 8GB RAM

>> Java Memory Limit: ?? Isn't this from SUBSONIC_MAX_MEMORY?

>>Output from ffmpeg -i:

Code: Select all

FFmpeg version 0.6.2-4:0.6.2-1ubuntu1, Copyright (c) 2000-2010 the Libav developers
  built on Mar 22 2011 15:35:22 with gcc 4.5.2
  configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --enable-shared --disable-static
  WARNING: library configuration mismatch
  libavutil   configuration: --extra-version=4:0.6.2-1ubuntu2 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libdirac --enable-libgsm --enable-libopenjpeg --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-libopenjpeg --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libfaad --enable-libdirac --enable-libfaad --enable-libmp3lame --enable-librtmp --enable-libx264 --enable-libxvid --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libavcodec  configuration: --extra-version=4:0.6.2-1ubuntu2 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libdirac --enable-libgsm --enable-libopenjpeg --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-libopenjpeg --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libfaad --enable-libdirac --enable-libfaad --enable-libmp3lame --enable-librtmp --enable-libx264 --enable-libxvid --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libavformat configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libavdevice configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libavfilter configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libswscale  configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libpostproc configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/i686/cmov --cpu=i686 --enable-shared --disable-static --disable-ffmpeg --disable-ffplay
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0

>>Last ten subsonic.log lines:

Code: Select all

[2011-05-22 11:53:35,086] INFO DaoHelper - Checking database schema.
[2011-05-22 11:53:35,458] INFO DaoHelper - Done checking database schema.
[2011-05-22 11:53:35,673] INFO SearchService - Automatic index creation scheduled to run every 1 day(s), starting at Mon May 23 03:00:00 EDT 2011
[2011-05-22 11:53:35,797] INFO PodcastService - Automatic Podcast update scheduled to run every 24 hour(s), starting at Sun May 22 11:58:35 EDT 2011
[2011-05-22 11:53:42,224] INFO NetworkService - Successfully forwarding port 4040.
[2011-05-22 11:53:44,271] INFO VersionService - Resolved local Subsonic version to: 4.4
[2011-05-22 11:53:45,451] INFO VersionService - Resolved latest Subsonic final version to: 4.4
[2011-05-22 11:53:45,451] INFO VersionService - Resolved latest Subsonic beta version to: 4.4.beta1[color=blue][/color][color=blue][/color][color=blue][/color][color=blue][/color]

[sliver4321]

I'll try to write some generic debugging steps for memory and other JVM (Java Virtual Machine) issues here:

Java might complain about low memory due to several conditions:
-Unable to allocate new memory
-Too much CPU time spent in Garbage Collection (cleaning up memory previously used by the JVM) vs Application CPU time

The java heap is divided into several sections, depending on the GC strategy/algorithm used.


A great tool for debugging and management of JVM's is (j)VisualVM.

This tool can be found by direct download (http://visualvm.java.net/) <- Recommended

OR

in the Java Development Kit (http://www.oracle.com/technetwork/java/index.html)

1. Make sure that both VisualVM and Subsonic is running on java 6 or later.

2. Start VisualVM on the same machine that Subsonic is running on
(You can start VisualVM from a remote machine as well, but some of the functions will not work, and you would need to start Subsonic with special parameters).

3. Go to Tools -> Plugins

4. Install all plugins (especially Visual GC)

5. Restart VisualVM

6. On the left side, under "Local", you should see Subsonic/Jetty. Double click to open.

7. In the "overview"-tab, check "JVM arguments". If set on the command line (and they usually are with Subsonic), you should see something like "-Xmx...m". Also check if you see "-XX:MaxPermSize". Those are the 2 primary memory settings in java.
Check that the Xmx parameter contains the expected value (SUBSONIC_MAX_MEMORY)

For applications where the Xmx parameter is not explicitly set on the command line, click on the "Monitor" tab, and check the "heap" section:
"Size" means the memory that the JVM currently has allocated from the host Operating System.
"Used" is the amound of memory currently used by the application
"Max" is the maximum amount of memory that the JVM is allowed to use. This can be set explicity with the "Xmx" parameter. The defaults vary with OS/RAM/JVM version etc.

Note that with Xmx, you can specify a value that is larger that the amount of free memory available on your machine. If java actually starts using more memory than what is available, the OS will usually start "swapping", which means it will use your hard drive as memory. Avoid this!! as it will usually make things run VERY slow.

8. Click on the Visual GC tab to see how the JVM manages the heap space.