title: Random REST API calls fail after upgrading Jersey version
body: <p>Recently had to upgrade version of Jersey from 2.5.1 (reason explained later) to 2.7 and ever since the upgrade, every time I upload code to the server, random API calls start failing with the following stacktrace:</p>
  
  <pre><code>org.glassfish.jersey.internal.Errors logErrors: The following warnings have been detected: WARNING: HK2 service reification failed for [com.example.rest.LeaderboardApi] with an exception:
  MultiException stack 1 of 2
  java.lang.NullPointerException
          at org.jvnet.hk2.internal.Utilities.hasInjectAnnotation(Utilities.java:1484)
          at org.jvnet.hk2.internal.Utilities.findInitializerMethods(Utilities.java:1425)
          at org.jvnet.hk2.internal.DefaultClassAnalyzer.getInitializerMethods(DefaultClassAnalyzer.java:102)
          at org.glassfish.jersey.internal.inject.JerseyClassAnalyzer.getInitializerMethods(JerseyClassAnalyzer.java:239)
          at org.jvnet.hk2.internal.Utilities.getInitMethods(Utilities.java:243)
          at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:163)
          at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:200)
          at org.jvnet.hk2.internal.SystemDescriptor.internalReify(SystemDescriptor.java:649)
          at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:604)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:405)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2046)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.access$700(ServiceLocatorImpl.java:116)
          at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1011)
          at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1006)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:115)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:111)
          at java.util.concurrent.FutureTask.run(FutureTask.java:260)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture.run(LRUHybridCache.java:173)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:292)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1093)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:616)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
          at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:173)
          at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:185)
          at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:94)
          at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:63)
          at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
          at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
          at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
          at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
          at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
          at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
          at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023)
          at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
          at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:535)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:483)
          at com.example.config.JerseyFilter.doFilter(JerseyFilter.java:27)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:420)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
          at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
          at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
          at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:326)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
          at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:657)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:619)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:589)
          at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:453)
          at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:460)
          at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:293)
          at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:319)
          at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:311)
          at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:457)
          at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:263)
          at java.lang.Thread.run(Thread.java:745)
  
  MultiException stack 2 of 2
  java.lang.IllegalArgumentException: Errors were discovered while reifying SystemDescriptor(
          implementation=com.example.rest.LeaderboardApi
          contracts={com.example.rest.LeaderboardApi}
          scope=org.glassfish.jersey.process.internal.RequestScoped
          qualifiers={}
          descriptorType=CLASS
          descriptorVisibility=NORMAL
          metadata=
          rank=0
          loader=null
          proxiable=null
          proxyForSameScope=null
          analysisName=null
          id=156
          locatorId=0
          identityHashCode=1367521629
          reified=false)
          at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:615)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:405)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2046)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.access$700(ServiceLocatorImpl.java:116)
          at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1011)
          at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1006)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:115)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:111)
          at java.util.concurrent.FutureTask.run(FutureTask.java:260)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture.run(LRUHybridCache.java:173)
          at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:292)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1093)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:616)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
          at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:173)
          at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:185)
          at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
          at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:94)
          at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:63)
          at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
          at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
          at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
          at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
          at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
          at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
          at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023)
          at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
          at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:535)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:483)
          at com.example.config.JerseyFilter.doFilter(JerseyFilter.java:27)
          at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:420)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
          at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
          at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
          at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:326)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
          at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:657)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:619)
          at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:589)
          at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:453)
          at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:460)
          at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:293)
          at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:319)
          at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:311)
          at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:457)
          at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:263)
          at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>Every time I do a clean build and re-upload, a new set of methods start failing (and the previously-failing ones start working) without any code change!</p>
  
  <p>After trying 2.7, I switched to 2.25.1 and that opened up a whole another can of worms, so I tried with 2.5.2 (instead of the original 2.5.1) and it had the same problems as 2.5.1. I switched to 2.8, and it had the same random issues as with 2.7.</p>
  
  <p>Current status is that with either version 2.7 or 2.8, a small (at least seemingly) random subset of the REST API methods fails with messages similar to the one above.</p>
  
  <p>I would like to find out why this is and fix it.</p>
  
  <h3>Method Followed While Switching Jersey Versions</h3>
  
  <p>I would clean out the <code>/WEB-INF/lib</code> folder, change the <code>jersey.version</code> variable in <code>pom.xml</code> to the new version. Eclipse would then fetch all the necessary libs and re-build code. Running <code>mvn appengine:update</code> would re-populate the <code>/WEB-INF/lib</code> folder and package everything up.</p>
  
  <p>A few times I did a clean (full) build to ensure that no artefacts from different versions would get mixed.</p>
  
  <h3>Reason for Upgrading from 2.5.1</h3>
  
  <p>Had to add a method (with the following signature):</p>
  
  <pre><code>public Response importChildren(@Context ContainerRequestContext reqCtx,
          @PathParam(ID) String id, @FormDataParam("file") InputStream csvStream,
          @FormDataParam("test") String testStr)
  </code></pre>
  
  <p>that would take an <code>InputStream</code> of a CSV file and process that. Because of that had to add <code>jersey-media-multipart</code>. Used with version 2.5.x (haven't tried with 2.6), this starts giving an error because it tries to write the <code>InputStream</code> to a temporary file (writing to disk is prohibited in Google App Engine (Standard), which is what I am using).</p>
  
  <p>With versions >= 2.7 this tendency to write a temporary file can be avoided by adding file <code>jersey-multipart-config.properties</code> with the content <code>jersey.config.multipart.bufferThreshold = -1</code></p>
  
  <h3>A Desperate Workaround</h3>
  
  <p>If nothing else works, I could move the CSV processing code out of Jersey-mediated methods and into a simple servlet. Then I don't need <code>jersey-media-multipart</code> and can move back to using version 2.5.1, where all the code was working reliably.</p>