{"action":"created","issue":{"url":"https://api.github.com/repos/rails/rails/issues/31028","repository_url":"https://api.github.com/repos/rails/rails","labels_url":"https://api.github.com/repos/rails/rails/issues/31028/labels{/name}","comments_url":"https://api.github.com/repos/rails/rails/issues/31028/comments","events_url":"https://api.github.com/repos/rails/rails/issues/31028/events","html_url":"https://github.com/rails/rails/issues/31028","id":270292017,"number":31028,"title":"\"invalid byte sequence\" in request with proprietary format","user":{"login":"tsujigiri","id":140724,"avatar_url":"https://avatars0.githubusercontent.com/u/140724?v=4","gravatar_id":"","url":"https://api.github.com/users/tsujigiri","html_url":"https://github.com/tsujigiri","followers_url":"https://api.github.com/users/tsujigiri/followers","following_url":"https://api.github.com/users/tsujigiri/following{/other_user}","gists_url":"https://api.github.com/users/tsujigiri/gists{/gist_id}","starred_url":"https://api.github.com/users/tsujigiri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsujigiri/subscriptions","organizations_url":"https://api.github.com/users/tsujigiri/orgs","repos_url":"https://api.github.com/users/tsujigiri/repos","events_url":"https://api.github.com/users/tsujigiri/events{/privacy}","received_events_url":"https://api.github.com/users/tsujigiri/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-11-01T12:24:01Z","updated_at":"2017-11-05T11:38:04Z","closed_at":"2017-11-05T11:38:04Z","author_association":"NONE","body":"In one of our controllers, we process requests from PayPal about processed transactions. This comes in a format we don't particularly care for. We just pass it into the PayPal client to verify it and we are good. The request spec for that randomly fails, though, because Instrumentation tries to parse it as UTF8 and depending on who is faster, the controller or Instrumentation, the request either fails or it doesn't. We haven't seen it in production lately, since not every request contains invalid encoding and we are also not sure whether this is something that was fixed in a later Rails version.\r\n\r\nIs there any way to tell Instrumentation to ignore specific parameters or apply sanitation only for Instrumentation? Any other ideas?\r\n\r\nThe backtrace from the test looks like this:\r\n\r\n```\r\n     Failure/Error: post '/pay_pal_payments/callback', params: \"foo=#{unsanitary_string}\"\r\n\r\n     ActionController::BadRequest:\r\n       Invalid request parameters: invalid byte sequence in UTF-8\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/query_parser.rb:72:in `rescue in parse_nested_query'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/query_parser.rb:61:in `parse_nested_query'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/request.rb:468:in `parse_query'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/request.rb:343:in `POST'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/request.rb:355:in `block (2 levels) in POST'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/parameters.rb:80:in `block in parse_formatted_parameters'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/parameters.rb:80:in `fetch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/parameters.rb:80:in `parse_formatted_parameters'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/request.rb:354:in `block in POST'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/request.rb:57:in `fetch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/request.rb:57:in `fetch_header'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/request.rb:353:in `POST'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/parameters.rb:45:in `parameters'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/http/filter_parameters.rb:41:in `filtered_parameters'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_controller/metal/instrumentation.rb:21:in `process_action'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activerecord-5.0.6/lib/active_record/railties/controller_runtime.rb:18:in `process_action'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/abstract_controller/base.rb:126:in `process'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionview-5.0.6/lib/action_view/rendering.rb:30:in `process'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_controller/metal.rb:190:in `dispatch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_controller/metal.rb:262:in `dispatch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/routing/route_set.rb:32:in `serve'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/journey/router.rb:39:in `block in serve'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/journey/router.rb:26:in `each'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/journey/router.rb:26:in `serve'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/routing/route_set.rb:727:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/cookies.rb:613:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/callbacks.rb:97:in `__run_callbacks__'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/callbacks.rb:90:in `run_callbacks'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/callbacks.rb:36:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/bugsnag-5.3.3/lib/bugsnag/rack.rb:34:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/railties-5.0.6/lib/rails/rack/logger.rb:36:in `call_app'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/railties-5.0.6/lib/rails/rack/logger.rb:24:in `block in call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/tagged_logging.rb:69:in `block in tagged'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/tagged_logging.rb:26:in `tagged'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/tagged_logging.rb:69:in `tagged'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/railties-5.0.6/lib/rails/rack/logger.rb:24:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/request_store-1.3.2/lib/request_store/middleware.rb:9:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/request_id.rb:24:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/executor.rb:12:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/middleware/static.rb:136:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/railties-5.0.6/lib/rails/engine.rb:522:in `call'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-test-0.6.3/lib/rack/mock_session.rb:30:in `request'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-test-0.6.3/lib/rack/test.rb:244:in `process_request'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rack-test-0.6.3/lib/rack/test.rb:124:in `request'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/testing/integration.rb:334:in `process'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/testing/integration.rb:361:in `process_with_kwargs'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/testing/integration.rb:21:in `post'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/actionpack-5.0.6/lib/action_dispatch/testing/integration.rb:442:in `block (2 levels) in <module:Runner>'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/rails-controller-testing-1.0.2/lib/rails/controller/testing/integration.rb:12:in `block (2 levels) in <module:Integration>'\r\n     # ./spec/requests/pay_pal_callback_encoding_spec.rb:17:in `block (2 levels) in <top (required)>'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:287:in `load'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:287:in `block in load'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:259:in `load_dependency'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:287:in `load'\r\n     # /home/helge/.gem/ruby/2.4.0/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'\r\n     # -e:1:in `<main>'\r\n     # ------------------\r\n     # --- Caused by: ---\r\n     # ArgumentError:\r\n     #   invalid byte sequence in UTF-8\r\n     #   /home/helge/.gem/ruby/2.4.0/gems/rack-2.0.3/lib/rack/query_parser.rb:81:in `normalize_params'\r\n\r\n\r\n```\r\n\r\n### System configuration\r\n**Rails version**: 5.0.6\r\n**Ruby version**: 2.4.2p198\r\n"},"comment":{"url":"https://api.github.com/repos/rails/rails/issues/comments/341966600","html_url":"https://github.com/rails/rails/issues/31028#issuecomment-341966600","issue_url":"https://api.github.com/repos/rails/rails/issues/31028","id":341966600,"user":{"login":"tsujigiri","id":140724,"avatar_url":"https://avatars0.githubusercontent.com/u/140724?v=4","gravatar_id":"","url":"https://api.github.com/users/tsujigiri","html_url":"https://github.com/tsujigiri","followers_url":"https://api.github.com/users/tsujigiri/followers","following_url":"https://api.github.com/users/tsujigiri/following{/other_user}","gists_url":"https://api.github.com/users/tsujigiri/gists{/gist_id}","starred_url":"https://api.github.com/users/tsujigiri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsujigiri/subscriptions","organizations_url":"https://api.github.com/users/tsujigiri/orgs","repos_url":"https://api.github.com/users/tsujigiri/repos","events_url":"https://api.github.com/users/tsujigiri/events{/privacy}","received_events_url":"https://api.github.com/users/tsujigiri/received_events","type":"User","site_admin":false},"created_at":"2017-11-05T11:38:04Z","updated_at":"2017-11-05T11:38:04Z","author_association":"NONE","body":"Turns out that's just random data, which explains why its failing randomly. 🤦‍ I should have looked at it earlier. My bad. Investigating it, it turns out that it fails if there is an `&` in it, which makes the next thing a key which may then contain gibberish and makes it fail when Rails tries to comprehend it. I assume. So, yeah. Sorry for stealing your time!"}}
{"actor":{"display_login":"tsujigiri"}}