{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/8485","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/8485/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/8485/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/8485/events","html_url":"https://github.com/fchollet/keras/pull/8485","id":273969768,"number":8485,"title":"Rethrow original exception in GeneratorEnqueuer.","user":{"login":"de-vri-es","id":786213,"avatar_url":"https://avatars1.githubusercontent.com/u/786213?v=4","gravatar_id":"","url":"https://api.github.com/users/de-vri-es","html_url":"https://github.com/de-vri-es","followers_url":"https://api.github.com/users/de-vri-es/followers","following_url":"https://api.github.com/users/de-vri-es/following{/other_user}","gists_url":"https://api.github.com/users/de-vri-es/gists{/gist_id}","starred_url":"https://api.github.com/users/de-vri-es/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/de-vri-es/subscriptions","organizations_url":"https://api.github.com/users/de-vri-es/orgs","repos_url":"https://api.github.com/users/de-vri-es/repos","events_url":"https://api.github.com/users/de-vri-es/events{/privacy}","received_events_url":"https://api.github.com/users/de-vri-es/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-11-14T22:41:11Z","updated_at":"2017-11-15T09:00:28Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/fchollet/keras/pulls/8485","html_url":"https://github.com/fchollet/keras/pull/8485","diff_url":"https://github.com/fchollet/keras/pull/8485.diff","patch_url":"https://github.com/fchollet/keras/pull/8485.patch"},"body":"`GeneratorEnqueuer` executes the generator in either background threads or worker processes. If an exception occurs in the generator, the thread/worker prints a traceback and is killed. The consumer of the data receives a `StopIteration` exception instead of the exception thrown by the generator.\r\n\r\nThis causes a lot of confusion on our issue tracker where people quote only the `StopIterator` error and not the interesting exception thrown by the generator. Often if they had seen the original error they could have solved the problem on their own.\r\n\r\nThis PR attempts to solve this by putting `(bool, value)` tuples the data queue currently used for communicating the generator results. Normal values are put in the queue as `(True, generator_output)` while exceptions are put in as `(False, exception)`.\r\n\r\nThere is also some glue code to hide the differences between python2 and python3 in the way backtraces are accessed.\r\n\r\nThis doesn't currently work with `use_multiprocessing=True`. It is not possible to pickle traceback objects, so they can not be put in a `multiprocessing.Queue`. The current workaround for that in this PR is to just print the backtrace in the child process and pickle the exception with backtrace `None`.\r\n\r\nAn alternative would be to use `tblib`, which can pickle/unpickle tracebacks. That would allow the same technique to be used with `use_multiprocessing=True`. I wasn't sure if an extra dependency would be a problem or not.\r\n\r\nExample of an error in a generator with this PR:\r\n```\r\nEpoch 1/50\r\n1/2 [==============>...............] - ETA: 3s - loss: 1.6380 - regression_loss: 0.4514 - classification_loss: 1.1866Traceback (most recent call last):\r\n  File \"../keras-retinanet/examples/train_pascal.py\", line 109, in <module>\r\n    keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0),\r\n  File \"/home/de-vri-es/dev/keras/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/de-vri-es/dev/keras/keras/engine/training.py\", line 2046, in fit_generator\r\n    generator_output = next(output_generator)\r\n  File \"/home/de-vri-es/dev/keras/keras/utils/data_utils.py\", line 678, in get\r\n    six.reraise(value.__class__, value, value.__traceback__)\r\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\r\n    raise value\r\n  File \"/home/de-vri-es/dev/keras/keras/utils/data_utils.py\", line 580, in data_generator_task\r\n    generator_output = next(self._generator)\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 117, in __next__\r\n    return self.next()\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 130, in next\r\n    annotations_group = self.load_annotations_group(group_index)\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 70, in load_annotations_group\r\n    return [self.load_annotations(image_index) for image_index in self.groups[group_index]]\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 70, in <listcomp>\r\n    return [self.load_annotations(image_index) for image_index in self.groups[group_index]]\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/pascal_voc.py\", line 161, in load_annotations\r\n    raise_from(ValueError('invalid annotations file: {}: {}'.format(filename, e)), None)\r\n  File \"/usr/lib/python3.6/site-packages/future/utils/__init__.py\", line 398, in raise_from\r\n    exec(execstr, myglobals, mylocals)\r\n  File \"<string>\", line 1, in <module>\r\nValueError: invalid annotations file: a.xml: could not parse object #1: illegal value for 'truncated': invalid literal for int() with base 10: 'not-an-int'\r\n```\r\n\r\nSame situation without this PR:\r\n```\r\nEpoch 1/50\r\nException in thread Thread-1:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.6/threading.py\", line 916, in _bootstrap_inner\r\n    self.run()\r\n  File \"/usr/lib/python3.6/threading.py\", line 864, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"/home/de-vri-es/dev/keras/keras/utils/data_utils.py\", line 579, in data_generator_task\r\n    generator_output = next(self._generator)\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 117, in __next__\r\n    return self.next()\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 130, in next\r\n    annotations_group = self.load_annotations_group(group_index)\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 70, in load_annotations_group\r\n    return [self.load_annotations(image_index) for image_index in self.groups[group_index]]\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/generator.py\", line 70, in <listcomp>\r\n    return [self.load_annotations(image_index) for image_index in self.groups[group_index]]\r\n  File \"/home/de-vri-es/dev/keras_retinanet/keras_retinanet/preprocessing/pascal_voc.py\", line 161, in load_annotations\r\n    raise_from(ValueError('invalid annotations file: {}: {}'.format(filename, e)), None)\r\n  File \"/usr/lib/python3.6/site-packages/future/utils/__init__.py\", line 398, in raise_from\r\n    exec(execstr, myglobals, mylocals)\r\n  File \"<string>\", line 1, in <module>\r\nValueError: invalid annotations file: a.xml: could not parse object #1: illegal value for 'truncated': invalid literal for int() with base 10: 'not-an-int'\r\n\r\n1/2 [==============>...............] - ETA: 3s - loss: 1.6544 - regression_loss: 0.4738 - classification_loss: 1.1806Traceback (most recent call last):\r\n  File \"../keras-retinanet/examples/train_pascal.py\", line 109, in <module>\r\n    keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0),\r\n  File \"/home/de-vri-es/dev/keras/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/de-vri-es/dev/keras/keras/engine/training.py\", line 2046, in fit_generator\r\n    generator_output = next(output_generator)\r\nStopIteration\r\n```"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/344528018","html_url":"https://github.com/fchollet/keras/pull/8485#issuecomment-344528018","issue_url":"https://api.github.com/repos/fchollet/keras/issues/8485","id":344528018,"user":{"login":"de-vri-es","id":786213,"avatar_url":"https://avatars1.githubusercontent.com/u/786213?v=4","gravatar_id":"","url":"https://api.github.com/users/de-vri-es","html_url":"https://github.com/de-vri-es","followers_url":"https://api.github.com/users/de-vri-es/followers","following_url":"https://api.github.com/users/de-vri-es/following{/other_user}","gists_url":"https://api.github.com/users/de-vri-es/gists{/gist_id}","starred_url":"https://api.github.com/users/de-vri-es/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/de-vri-es/subscriptions","organizations_url":"https://api.github.com/users/de-vri-es/orgs","repos_url":"https://api.github.com/users/de-vri-es/repos","events_url":"https://api.github.com/users/de-vri-es/events{/privacy}","received_events_url":"https://api.github.com/users/de-vri-es/received_events","type":"User","site_admin":false},"created_at":"2017-11-15T09:00:28Z","updated_at":"2017-11-15T09:00:28Z","author_association":"NONE","body":"> Could you add a test to validate the new behaviour?\r\n\r\nGood point, will do.\r\n\r\n> Other than that, the code looks fine, it's pretty much the same as the Sequence now. multiprocessing.pool.AsyncResult is doing the same : keep a (state,result) tuple and throw is state==false.\r\n> \r\n> I don't know if we could change the GeneratorEnqueuer to use AsyncResult in a future PR.\r\n\r\nI looked into directly using `AsyncResult` without using a `multiprocessing.pool.[Thread]Pool` (to keep the PR small), but that is only possible by using private interfaces of `AsyncResult`.\r\n\r\nI do think it is a good idea to use the pool classes provided by `multiprocessing`. That would hide all the details of exception transporting from us. I could look into that instead of this method. I mainly did it this way now to keep the PR small.\r\n\r\n"}}
{"actor":{"display_login":"de-vri-es"}}