Unicode character conversion exception

[subdee]

Hey all,

I searched the forum but couldn't find a solution.

I get the following exception everytime I try to load a file with non-standard characters (aka Unicode). How do I fix this?

Here is the exception:

Java version Sun Microsystems Inc. 1.6.0_12
Operating system Linux 2.6.26-2-686
Server Apache Tomcat/6.0.20

Code: Select all

java.io.CharConversionException: Not an ISO 8859-1 character: ř at javax.servlet.ServletOutputStream.print(ServletOutputStream.java:89) at javax.servlet.ServletOutputStream.println(ServletOutputStream.java:242) at net.sourceforge.subsonic.controller.M3UController.createClientSidePlaylist(M3UController.java:88) at net.sourceforge.subsonic.controller.M3UController.handleRequest(M3UController.java:73) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:874) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:808) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:476) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:431) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107) at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at net.sourceforge.subsonic.filter.RequestEncodingFilter.doFilter(RequestEncodingFilter.java:43) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at net.sourceforge.subsonic.filter.ParameterDecodingFilter.doFilter(ParameterDecodingFilter.java:54) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at net.sourceforge.subsonic.filter.BootstrapVerificationFilter.doFilter(BootstrapVerificationFilter.java:54) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) at java.lang.Thread.run(Thread.java:619)

[subdee]

So is there anyone that can help me please?

[flacflac]

So is there anyone that can help me please?

I think you're encountering a limitation of the XSPF webplayer - assuming you're trying to play this in your browser. I don't think there's a fix as this is not Sindre's player. You could maybe just re-tag that file?! Or try an external player.

BTW, what's the character? A 'y' with 2 dots on top?

ff

[jeremyh]

It is an interesting problem that does seem to be some sort of bug. The character referenced (ř) is not in the ISO 8859-1 character set (I am assuming it is from the Czech alphabet). ISO 8859-1 is not UNICODE, UNICODE was specifically created to avoid shortcomings like this. It looks like the parser is not defaulting its character set to use UNICODE, but rather just using ISO 8859-1 (I think that is settable). You would have to alter the place where the code is used to fix it.
Based on the stack:
java.io.CharConversionException: Not an ISO 8859-1 character: ř at javax.servlet.ServletOutputStream.print(ServletOutputStream.java:89) at javax.servlet.ServletOutputStream.println(ServletOutputStream.java:242) at net.sourceforge.subsonic.controller.M3UController.createClientSidePlaylist(M3UController.java:88) at ...

It looks like it is coming from the playlist controller (which is using the javax servlet output stream and doing a straight java character conversion) I believe you specify the charset when you make the call to do a char conversion (which is likely not set right - although I thought it defaulted to UNICODE if you don't put something explicit in there).

[subdee]

No I'm not using the internal player, it is for external player use(aka downloading a playlist m3u).

I also thought it would default to Unicode but apparently it doesn't. Is there a fix for this or should I report it as a bug?

[jeremyh]

I would say its a bug. You have the line(s) where it flakes:

java.io.CharConversionException: Not an ISO 8859-1 character: ř at
javax.servlet.ServletOutputStream.print(ServletOutputStream.java:89) at
javax.servlet.ServletOutputStream.println(ServletOutputStream.java:242) at
net.sourceforge.subsonic.controller.M3UController.createClientSidePlaylist(M3UController.java:8Cool at ...

When you download the playlist it is calling the M3UController to create the playlist for export (via a servlet request). I would suspect that the coder would specify somewhere that the servlet should use UNICODE, but I am not familiar enough with the specifics to know exactly what to do. Anyone else know for sure? Sindre?

FYI, the character seems to be Unicode 0159

[subdee]

Bump because it's a blocker for me.

Happens on versions 3.6-3.8.

[subdee]

I'd really like to find a solution for this.

The Tomcat is starting up using URI encoding UTF-8. I can only assume that when Subsonic does a GET/POST request, he's not defining the encoding.

Nothing I can do about this from my side though.

[flacflac]

I concur, this bug persists even in the latest beta!

I have trouble playing an album by an artist with a "ğ" in her name.

Code: Select all

[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac)
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac) Selda Ba?can_-_Selda_-_01_-_Meydan Sizindir.flac: ERROR initializing decoder
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac) state = FLAC__FILE_DECODER_ERROR_OPENING_FILE
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac)
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac) An error occurred opening the input file; it is likely that it does not exist
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\flac) or is not readable.
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) Assuming raw pcm input file
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) LAME version 3.96.1 (http://lame.sourceforge.net/)
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) CPU features: MMX (ASM used), SSE, SSE2
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) Using polyphase lowpass filter, transition band: 20094 Hz - 20627 Hz
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) Encoding to
[2/1/10 9:00:39 PM CET]   DEBUG   InputStreamReaderThread   (c:\subsonic\transcode\lame) Encoding as 44.1 kHz 320 kbps j-stereo MPEG-1 Layer III (4.4x) qval=3


Please fix this if possible, Sindre - the eclectic userbase of Subsonic will be very grateful.

BTW, the whole album doesn't even show up when indexing my collection. It is basically being ignored. So mean.

ff

[subdee]

Problem still exists.

Tried also setting file-encoding to utf-8 in java options when starting Tomcat, same issue.

Env language is also UTF-8.

Still a no-go.

[flacflac]

Problem still exists.

Tried also setting file-encoding to utf-8 in java options when starting Tomcat, same issue.

Env language is also UTF-8.

Still a no-go.

Yes same here... I am not sure why this does not get any more attention - Sindre, any way to fix this?

ff

[jeremyh]

Sindre, et al, I was able to take a slightly closer look at the code finally, and the line that it is choking on is this:

Code: Select all

out.println("#EXTINF:" + duration + "," + metaData.getArtist() + " - " + metaData.getTitle());

(line 88 of M3UController) where "out" is the ServletOutputStream, called from:

Code: Select all

createClientSidePlaylist(response.getOutputStream(), player, url);

(line 73)

The characterEncoding is being set on the HttpServletRequest, but then to build the clientSidePlaylist, the outputstream is being used to build the response. If you use the outputStream, no encoding takes place and the characterEncoding is ignored - since you are writing binary directly. You should probably be using the PrintWriter instead of the output stream, that way it uses the character encoding you have specified.

See this: http://www.techienuggets.com/commentList.jsp?tx=8516&tx=8516&d-49653-p=2 for an example of how to fix.

Likely, there will be several places that need to be fixed since you use a similar technique to call the "createServerSitePlaylist" and I suspect other servlet probably use it too.[/url]

[jeremyh]

Sindre, any chance you could toss this into the full release of 4.0?

[sindre_mehus]

Sure, I'll look into it. Sorry for ignoring this for so long

[sindre_mehus]

I've changed it to use utf-8 encoding now, but I actually wasn't able to reproduce the problem. I changed the title of a song to contain an ř, but I didn't get the exception.

Could someone pls send me an offending file so I can verify that my fix works?

Thanks in advance,
Sindre