{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/4799","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/4799/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/4799/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/4799/events","html_url":"https://github.com/fchollet/keras/pull/4799","id":197119105,"number":4799,"title":"Fix masking for metrics","user":{"login":"dieuwkehupkes","id":2862560,"avatar_url":"https://avatars0.githubusercontent.com/u/2862560?v=3","gravatar_id":"","url":"https://api.github.com/users/dieuwkehupkes","html_url":"https://github.com/dieuwkehupkes","followers_url":"https://api.github.com/users/dieuwkehupkes/followers","following_url":"https://api.github.com/users/dieuwkehupkes/following{/other_user}","gists_url":"https://api.github.com/users/dieuwkehupkes/gists{/gist_id}","starred_url":"https://api.github.com/users/dieuwkehupkes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dieuwkehupkes/subscriptions","organizations_url":"https://api.github.com/users/dieuwkehupkes/orgs","repos_url":"https://api.github.com/users/dieuwkehupkes/repos","events_url":"https://api.github.com/users/dieuwkehupkes/events{/privacy}","received_events_url":"https://api.github.com/users/dieuwkehupkes/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-12-22T08:22:57Z","updated_at":"2017-03-15T23:59:48Z","closed_at":"2017-03-15T21:43:36Z","pull_request":{"url":"https://api.github.com/repos/fchollet/keras/pulls/4799","html_url":"https://github.com/fchollet/keras/pull/4799","diff_url":"https://github.com/fchollet/keras/pull/4799.diff","patch_url":"https://github.com/fchollet/keras/pull/4799.patch"},"body":"During evaluation, the masks of a model are not taken into account,\r\nresulting in incorrect accuracies for sequence-to-sequence models with\r\npadded output values. An example can be found in issue 4752.\r\n\r\nTo solve this issue, treat the metrics in a way analogous to the\r\nobjective functions, by adding an optional axis argument to metrics.py\r\nand computing the overall mean only after multiplying with the model's\r\nmask and dividing by its length in training.py. This required a small\r\nreorganisation of the metric preparation in training.py.\r\n\r\nMetric masking is now fixed for binary_accuracy, categorical_accuracy,\r\nsparse_categorical_accuracy, top_k_categorical_accuracy,\r\nmean_squared_error, mean_absolute_error, mean_absolute_percentage_error,\r\nmean_squared_logarithmic_error, hinge, squared_hinge,\r\ncategorical_crossentropy, sparse_categorical_crossentropy,\r\nbinary_crossentropy, poisson and cosine_proximity. Tests to confirm are\r\nincluded in tests/test_metric_masking.py.\r\n\r\nFixes for matthews_correlation, kullback_leibler_divergence, precision,\r\nrecall,  fbeta_score and fmeasure are not written yet. When masked\r\nmetrics should be computed for these metrics a warning is raised."},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/286917401","html_url":"https://github.com/fchollet/keras/pull/4799#issuecomment-286917401","issue_url":"https://api.github.com/repos/fchollet/keras/issues/4799","id":286917401,"user":{"login":"fchollet","id":710255,"avatar_url":"https://avatars0.githubusercontent.com/u/710255?v=3","gravatar_id":"","url":"https://api.github.com/users/fchollet","html_url":"https://github.com/fchollet","followers_url":"https://api.github.com/users/fchollet/followers","following_url":"https://api.github.com/users/fchollet/following{/other_user}","gists_url":"https://api.github.com/users/fchollet/gists{/gist_id}","starred_url":"https://api.github.com/users/fchollet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fchollet/subscriptions","organizations_url":"https://api.github.com/users/fchollet/orgs","repos_url":"https://api.github.com/users/fchollet/repos","events_url":"https://api.github.com/users/fchollet/events{/privacy}","received_events_url":"https://api.github.com/users/fchollet/received_events","type":"User","site_admin":false},"created_at":"2017-03-15T23:59:48Z","updated_at":"2017-03-15T23:59:48Z","body":"It is fixed, and this PR has extensive conflicts with master."}}
{"actor":{"display_login":"fchollet"}}