{"action":"created","issue":{"url":"https://api.github.com/repos/akira/exq/issues/228","repository_url":"https://api.github.com/repos/akira/exq","labels_url":"https://api.github.com/repos/akira/exq/issues/228/labels{/name}","comments_url":"https://api.github.com/repos/akira/exq/issues/228/comments","events_url":"https://api.github.com/repos/akira/exq/issues/228/events","html_url":"https://github.com/akira/exq/issues/228","id":199031745,"number":228,"title":"Jobs randomly get stuck","user":{"login":"mspanc","id":389651,"avatar_url":"https://avatars1.githubusercontent.com/u/389651?v=3","gravatar_id":"","url":"https://api.github.com/users/mspanc","html_url":"https://github.com/mspanc","followers_url":"https://api.github.com/users/mspanc/followers","following_url":"https://api.github.com/users/mspanc/following{/other_user}","gists_url":"https://api.github.com/users/mspanc/gists{/gist_id}","starred_url":"https://api.github.com/users/mspanc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mspanc/subscriptions","organizations_url":"https://api.github.com/users/mspanc/orgs","repos_url":"https://api.github.com/users/mspanc/repos","events_url":"https://api.github.com/users/mspanc/events{/privacy}","received_events_url":"https://api.github.com/users/mspanc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-01-05T18:57:26Z","updated_at":"2017-04-03T17:57:09Z","closed_at":null,"body":"I have found cases when jobs get stuck.\r\n\r\nThis is the sample job (data retreived via exq_ui):\r\n\r\nPID: vault:<0.15483.6>\t\r\nqueue: heavy\t\r\nmodule: Vault.Jobs.Data.Processing.AnalysisAudioDurationJob\r\nargs: 9b091696-5ce7-4e2c-a2f5-a904f784d964,true\r\nstarted at: Wed Jan 04 2017 23:00:52 GMT+0100 (CET)\r\n\r\nIt has ended it with an exception and never got cleared from the queue, blocking slot for other jobs.\r\n\r\n```\r\n4 Jan 2017 14:13:40.956 234 <190>1 2017-01-04T13:13:40.543771+00:00 app web.1 - - 13:13:40.543 [info] [Elixir.Vault.Jobs.Data.Processing.AnalysisAudioDurationJob #PID<0.19642.2> 9b091696-5ce7-4e2c-a2f5-a904f784d964] Saving\r\n» 4 Jan 2017 14:13:40.957 158 <190>1 2017-01-04T13:13:40.544708+00:00 app web.1 - - 13:13:40.544 [error] Process #PID<0.19642.2> raised an exception\r\n» 4 Jan 2017 14:13:40.957 144 <190>1 2017-01-04T13:13:40.544710+00:00 app web.1 - - ** (DBConnection.ConnectionError) ssl send: closed\r\n» 4 Jan 2017 14:13:40.957 194 <190>1 2017-01-04T13:13:40.544711+00:00 app web.1 - - (ecto) lib/ecto/adapters/postgres/connection.ex:115: Ecto.Adapters.Postgres.Connection.execute/4\r\n» 4 Jan 2017 14:13:40.957 163 <190>1 2017-01-04T13:13:40.544712+00:00 app web.1 - - (ecto) lib/ecto/adapters/sql.ex:243: Ecto.Adapters.SQL.sql_call/6\r\n» 4 Jan 2017 14:13:40.957 171 <190>1 2017-01-04T13:13:40.544712+00:00 app web.1 - - (ecto) lib/ecto/adapters/sql.ex:441: Ecto.Adapters.SQL.execute_or_reset/7\r\n» 4 Jan 2017 14:13:40.957 166 <190>1 2017-01-04T13:13:40.544713+00:00 app web.1 - - (ecto) lib/ecto/repo/queryable.ex:130: Ecto.Repo.Queryable.execute/5\r\n» 4 Jan 2017 14:13:40.957 161 <190>1 2017-01-04T13:13:40.544714+00:00 app web.1 - - (ecto) lib/ecto/repo/queryable.ex:35: Ecto.Repo.Queryable.all/4\r\n» 4 Jan 2017 14:13:40.957 161 <190>1 2017-01-04T13:13:40.544715+00:00 app web.1 - - (ecto) lib/ecto/repo/queryable.ex:68: Ecto.Repo.Queryable.one/4\r\n» 4 Jan 2017 14:13:40.958 213 <190>1 2017-01-04T13:13:40.544716+00:00 app web.1 - - (vault) web/jobs/data/processing/result_parser.ex:68: Vault.Jobs.Data.Processing.ResultParser.parse_metadata_list/3\r\n» 4 Jan 2017 14:13:40.958 199 <190>1 2017-01-04T13:13:40.544717+00:00 app web.1 - - (vault) web/jobs/data/processing/result_parser.ex:50: Vault.Jobs.Data.Processing.ResultParser.parse/3\r\n» 4 Jan 2017 14:13:40.958 188 <190>1 2017-01-04T13:13:40.544767+00:00 app web.1 - - 13:13:40.544 [info] Queueing job 1eb24985-d7fc-4fb6-a327-e3606179649c to retry in 44.0 seconds\r\n» 4 Jan 2017 14:13:40.958 174 <190>1 2017-01-04T13:13:40.547998+00:00 app web.1 - - 13:13:40.547 [info] {%DBConnection.ConnectionError{message: \"ssl send: closed\"},\r\n» 4 Jan 2017 14:13:40.958 144 <190>1 2017-01-04T13:13:40.548000+00:00 app web.1 - - [{Ecto.Adapters.Postgres.Connection, :execute, 4,\r\n» 4 Jan 2017 14:13:40.958 160 <190>1 2017-01-04T13:13:40.548001+00:00 app web.1 - - [file: 'lib/ecto/adapters/postgres/connection.ex', line: 115]},\r\n» 4 Jan 2017 14:13:40.958 129 <190>1 2017-01-04T13:13:40.548002+00:00 app web.1 - - {Ecto.Adapters.SQL, :sql_call, 6,\r\n» 4 Jan 2017 14:13:40.958 144 <190>1 2017-01-04T13:13:40.548003+00:00 app web.1 - - [file: 'lib/ecto/adapters/sql.ex', line: 243]},\r\n» 4 Jan 2017 14:13:40.958 137 <190>1 2017-01-04T13:13:40.548003+00:00 app web.1 - - {Ecto.Adapters.SQL, :execute_or_reset, 7,\r\n» 4 Jan 2017 14:13:40.959 144 <190>1 2017-01-04T13:13:40.548004+00:00 app web.1 - - [file: 'lib/ecto/adapters/sql.ex', line: 441]},\r\n» 4 Jan 2017 14:13:40.959 130 <190>1 2017-01-04T13:13:40.548004+00:00 app web.1 - - {Ecto.Repo.Queryable, :execute, 5,\r\n» 4 Jan 2017 14:13:40.959 146 <190>1 2017-01-04T13:13:40.548005+00:00 app web.1 - - [file: 'lib/ecto/repo/queryable.ex', line: 130]},\r\n» 4 Jan 2017 14:13:40.959 126 <190>1 2017-01-04T13:13:40.548005+00:00 app web.1 - - {Ecto.Repo.Queryable, :all, 4,\r\n» 4 Jan 2017 14:13:40.959 145 <190>1 2017-01-04T13:13:40.548006+00:00 app web.1 - - [file: 'lib/ecto/repo/queryable.ex', line: 35]},\r\n» 4 Jan 2017 14:13:40.959 126 <190>1 2017-01-04T13:13:40.548007+00:00 app web.1 - - {Ecto.Repo.Queryable, :one, 4,\r\n» 4 Jan 2017 14:13:40.959 145 <190>1 2017-01-04T13:13:40.548007+00:00 app web.1 - - [file: 'lib/ecto/repo/queryable.ex', line: 68]},\r\n» 4 Jan 2017 14:13:40.959 162 <190>1 2017-01-04T13:13:40.548008+00:00 app web.1 - - {Vault.Jobs.Data.Processing.ResultParser, :parse_metadata_list, 3,\r\n» 4 Jan 2017 14:13:40.959 160 <190>1 2017-01-04T13:13:40.548009+00:00 app web.1 - - [file: 'web/jobs/data/processing/result_parser.ex', line: 68]},\r\n» 4 Jan 2017 14:13:40.959 148 <190>1 2017-01-04T13:13:40.548009+00:00 app web.1 - - {Vault.Jobs.Data.Processing.ResultParser, :parse, 3,\r\n» 4 Jan 2017 14:13:40.959 161 <190>1 2017-01-04T13:13:40.548010+00:00 app web.1 - - [file: 'web/jobs/data/processing/result_parser.ex', line: 50]}]}\r\n» 4 Jan 2017 14:13:40.959 229 <190>1 2017-01-04T13:13:40.548041+00:00 app web.1 - - 13:13:40.547 [info] Elixir.Vault.Jobs.Data.Processing.AnalysisAudioDurationJob[1eb24985-d7fc-4fb6-a327-e3606179649c] fail: 138109ms sec\r\n```\r\n\r\nIt seems that PID in the log and exq_ui do not match."},"comment":{"url":"https://api.github.com/repos/akira/exq/issues/comments/291222093","html_url":"https://github.com/akira/exq/issues/228#issuecomment-291222093","issue_url":"https://api.github.com/repos/akira/exq/issues/228","id":291222093,"user":{"login":"j-mcnally","id":892382,"avatar_url":"https://avatars2.githubusercontent.com/u/892382?v=3","gravatar_id":"","url":"https://api.github.com/users/j-mcnally","html_url":"https://github.com/j-mcnally","followers_url":"https://api.github.com/users/j-mcnally/followers","following_url":"https://api.github.com/users/j-mcnally/following{/other_user}","gists_url":"https://api.github.com/users/j-mcnally/gists{/gist_id}","starred_url":"https://api.github.com/users/j-mcnally/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/j-mcnally/subscriptions","organizations_url":"https://api.github.com/users/j-mcnally/orgs","repos_url":"https://api.github.com/users/j-mcnally/repos","events_url":"https://api.github.com/users/j-mcnally/events{/privacy}","received_events_url":"https://api.github.com/users/j-mcnally/received_events","type":"User","site_admin":false},"created_at":"2017-04-03T17:57:09Z","updated_at":"2017-04-03T17:57:09Z","body":"Based on past experience ive had with sidekiq exhibiting the behaviour, it was often the case of Blocking IO that never finished and didnt have timeout code. The classic example being network / http IO. Where a HTTP request would just wait forever and not realize it was disconnect because Ruby. Could it be similar? Curious if you are doing webscraping / http or any blocking io?\r\n\r\n"}}
{"actor":{"display_login":"j-mcnally"}}