Error serving static assets

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Error serving static assets

Paul_Fairless
Hi,

Last week or so I've been getting an error loading my site - I'm using the latest snapshot.

The site loads initally, but some of the assets fail to load - style.css for example. If I request the file individually it works fine.

This only seems to be an issue when running from the start scripts and doesn't happen when running locally.

Any ideas?

Aug 09, 2013 2:02:25 PM io.netty.util.concurrent.DefaultPromise notifyListener0
WARNING: An exception was thrown by io.netty.handler.stream.ChunkedWriteHandler$3.operationComplete()
java.lang.IllegalStateException: recycled already
        at io.netty.util.Recycler$Stack.push(Recycler.java:96)
        at io.netty.util.Recycler.recycle(Recycler.java:56)
        at io.netty.util.internal.PendingWrite.recycle(PendingWrite.java:57)
        at io.netty.util.internal.PendingWrite.successAndRecycle(PendingWrite.java:78)
        at io.netty.handler.stream.ChunkedWriteHandler$3.operationComplete(ChunkedWriteHandler.java:284)
        at io.netty.handler.stream.ChunkedWriteHandler$3.operationComplete(ChunkedWriteHandler.java:280)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:621)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:548)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:407)
        at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:548)
        at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:360)
        at io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:486)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:543)
        at io.netty.channel.DefaultChannelPipeline$HeadHandler.close(DefaultChannelPipeline.java:1022)
        at io.netty.channel.DefaultChannelHandlerContext.invokeClose(DefaultChannelHandlerContext.java:553)
        at io.netty.channel.DefaultChannelHandlerContext.close(DefaultChannelHandlerContext.java:538)
        at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
        at io.netty.channel.DefaultChannelHandlerContext.invokeClose(DefaultChannelHandlerContext.java:553)
        at io.netty.channel.DefaultChannelHandlerContext.close(DefaultChannelHandlerContext.java:538)
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
        at io.netty.channel.DefaultChannelHandlerContext.invokeClose(DefaultChannelHandlerContext.java:553)
        at io.netty.channel.DefaultChannelHandlerContext.close(DefaultChannelHandlerContext.java:538)
        at io.netty.channel.DefaultChannelHandlerContext.close(DefaultChannelHandlerContext.java:419)
        at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:830)
        at io.netty.channel.AbstractChannel.close(AbstractChannel.java:177)
        at io.netty.channel.ChannelFutureListener$1.operationComplete(ChannelFutureListener.java:44)
        at io.netty.channel.ChannelFutureListener$1.operationComplete(ChannelFutureListener.java:41)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:621)
        at io.netty.util.concurrent.DefaultPromise.notifyListener(DefaultPromise.java:598)
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:123)
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)
        at org.ratpackframework.file.internal.FileHttpTransmitter.transmit(FileHttpTransmitter.java:91)
        at org.ratpackframework.http.internal.DefaultResponse.sendFile(DefaultResponse.java:118)
        at org.ratpackframework.file.internal.FileStaticAssetRequestHandler.handle(FileStaticAssetRequestHandler.java:85)
        at org.ratpackframework.file.internal.DirectoryStaticAssetRequestHandler.handle(DirectoryStaticAssetRequestHandler.java:54)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:87)
        at org.ratpackframework.file.internal.TargetFileStaticAssetRequestHandler.handle(TargetFileStaticAssetRequestHandler.java:69)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:87)
        at org.ratpackframework.file.internal.FileSystemBindingHandler.handle(FileSystemBindingHandler.java:50)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext$1.handle(DefaultContext.java:169)
        at org.ratpackframework.handling.internal.DefaultContext.next(DefaultContext.java:75)
        at org.ratpackframework.path.internal.PathHandler.handle(PathHandler.java:42)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:79)
        at org.ratpackframework.handling.internal.ChainHandler.handle(ChainHandler.java:32)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:79)
        at org.ratpackframework.handling.internal.ChainHandler.handle(ChainHandler.java:32)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:83)
        at org.ratpackframework.session.internal.SessionBindingHandler.handle(SessionBindingHandler.java:46)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:174)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:83)
        at org.ratpackframework.guice.internal.InjectorBindingHandler.handle(InjectorBindingHandler.java:54)
        at org.ratpackframework.groovy.internal.ScriptBackedApp.handle(ScriptBackedApp.java:97)
        at org.ratpackframework.error.internal.ErrorCatchingHandler.handle(ErrorCatchingHandler.java:33)
        at org.ratpackframework.server.internal.NettyHandlerAdapter.channelRead0(NettyHandlerAdapter.java:95)
        at org.ratpackframework.server.internal.NettyHandlerAdapter.channelRead0(NettyHandlerAdapter.java:50)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:98)
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:333)
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:319)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:333)
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:319)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:333)
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:319)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:333)
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:789)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
        at java.lang.Thread.run(Thread.java:722)
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator
Have raised with Netty folks: https://github.com/netty/netty/issues/1719
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
Latest snapshot makes no difference for me. Initially I thought it was random, however it appears to be a single javascript file and my style.css - the 2 largest files in my project.

When hitting the js file directly it seems to be getting truncated at different points which could indicate a Netty issue serving large files - or files with long line lengths. Still works fine locally.

I also noticed when I included the Netty snapshot in the wrong location (compile not classpath) I re-created the same error locally. Could it be possible the packaged Ratpack app has a dependancy issue with Netty?

Regards
Paul
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator
> I also noticed when I included the Netty snapshot in the wrong location (compile not classpath) I re-created the same error locally. Could it be possible the packaged Ratpack app has a dependancy issue with Netty?

Could you explain this a bit more. This might be key.

It's almost definite that this is a Netty issue, but they can't do much without a reproducible sample that they can play with so that's key.
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
So by I get the error by adding the Netty snapshot into build.gradle as a compile time dependancy. The app must first be run without the snapshot version:

dependencies {
    // SpringLoaded enables runtime hot reloading.
    // It is not part of the app runtime and is not shipped in the distribution.
    springloaded "org.springsource.springloaded:springloaded-core:1.1.1"
    compile "com.codahale.metrics:metrics-core:3.0.0"
    compile "com.codahale.metrics:metrics-jvm:3.0.0"
    compile "joda-time:joda-time:2.2"
    compile 'io.netty:netty-codec-http:4.0.8.Final-SNAPSHOT'


    testCompile "org.gebish:geb-spock:0.9.0"
    testCompile "org.spockframework:spock-core:0.7-groovy-2.0"
    testCompile "org.gebish:geb-junit4:0.9.0"
    testCompile "com.github.detro.ghostdriver:phantomjsdriver:1.0.3"
    testCompile "org.seleniumhq.selenium:selenium-support:2.31.0"
}

then loading the page gives me:

ava.lang.IllegalStateException: recycled already
        at io.netty.util.Recycler$Stack.push(Recycler.java:96)
        at io.netty.util.Recycler.recycle(Recycler.java:56)
        at io.netty.util.internal.PendingWrite.recycle(PendingWrite.java:57)
        at io.netty.util.internal.PendingWrite.recycleAndGet(PendingWrite.java:94)
        at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:325)
        at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:151)
        at io.netty.channel.DefaultChannelHandlerContext.invokeFlush(DefaultChannelHandlerContext.java:676)
        at io.netty.channel.DefaultChannelHandlerContext.writeAndFlush(DefaultChannelHandlerContext.java:693)
        at io.netty.channel.DefaultChannelHandlerContext.writeAndFlush(DefaultChannelHandlerContext.java:716)
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:893)
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:239)
        at org.ratpackframework.file.internal.FileHttptter.transmit(FileHttpTransmitter.java:89)
        at org.ratpackframework.http.internal.DefaultResponse.sendFile(DefaultResponse.java:118)
        at org.ratpackframework.file.internal.FileStaticAssetRequestHandler.handle(FileStaticAssetRequestHandler.java:85)
        at org.ratpackframework.file.internal.DirectoryStaticAssetRequestHandler.handle(DirectoryStaticAssetRequestHandler.java:54)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:187)
        at org.ratpackframework.handling.internal.DefaultContext.insert(DefaultContext.java:95)
        at org.ratpackframework.file.internal.TargetFileStaticAssetRequestHandler.handle(TargetFileStaticAssetRequestHandler.java:69)
        at org.ratpackframework.handling.internal.DefaultContext.doNext(DefaultContext.java:187)
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
Ok I think i had made a mistake when including the dependency. I have tested with the latest snapshot and it is still failing (also locally now) - but the stacktrace is different. I have updated the Netty github issue.

java.lang.IllegalStateException: recycled already
        at io.netty.util.Recycler$Stack.push(Recycler.java:96)
        at io.netty.util.Recycler.recycle(Recycler.java:56)
        at io.netty.util.internal.PendingWrite.recycle(PendingWrite.java:57)
        at io.netty.util.internal.PendingWrite.recycleAndGet(PendingWrite.java:94)
        at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:325)
        at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:151)
        at io.netty.channel.DefaultChannelHandlerContext.invokeFlush(DefaultChannelHandlerContext.java:676)
        at io.netty.channel.DefaultChannelHandlerContext.writeAndFlush(DefaultChannelHandlerContext.java:693)
        at io.netty.channel.DefaultChannelHandlerContext.writeAndFlush(DefaultChannelHandlerContext.java:716)
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:893)
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:239)
        at org.ratpackframework.file.internal.FileHttpTransmitter.transmit(FileHttpTransmitter.java:70)
        at org.ratpackframework.http.internal.DefaultResponse.sendFile(DefaultResponse.java:138)
        at org.ratpackframework.file.internal.FileRenderer.render(FileRenderer.java:76)
        at org.ratpackframework.file.internal.FileRenderer.render(FileRenderer.java:34)
        ...
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator
Thanks, let's see what they say.
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
So They have reverted some code in the latest snapshot and the exception has gone (https://github.com/netty/netty/issues/1743) - however my css & javascript still fail to load - just timeout now. I'm a little reluctant to send them my project.

Do you know of any example apps that also exhibit this behaviour.

Thanks
Paul

Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
digging a bit more it seems unrelated to being a deployed version of the app and more to do with running on an EC2 ubuntu (not tried other variants yet) instance. tried both open-jdk and oracle jdk without luck
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Paul_Fairless
just tried creating a test-app and noticed that Netty 4.0.6-Final is now working again - so removing the updated dependancy has fixed my issue for now. So I've come full circle but now have a working version.

I will still try to provide a test case for 4.0.8 in the hope this issue does not get introduced at that point.
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator


That would be very cool.

I know about the hanging issue with 4.0.7 and 4.0.8. Netty guys are waiting on a test case.
Reply | Threaded
Open this post in threaded view
|

Re: Error serving static assets

Luke Daley
Administrator
Hi Paul,

I found the issue. Ratpack now uses 4.0.7.