{"action":"created","issue":{"url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/19","repository_url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow","labels_url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/19/comments","events_url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/19/events","html_url":"https://github.com/carpedm20/BEGAN-tensorflow/issues/19","id":225313007,"number":19,"title":"Sudden mode collapse after 400k epochs","user":{"login":"stefdoerr","id":7935362,"avatar_url":"https://avatars0.githubusercontent.com/u/7935362?v=3","gravatar_id":"","url":"https://api.github.com/users/stefdoerr","html_url":"https://github.com/stefdoerr","followers_url":"https://api.github.com/users/stefdoerr/followers","following_url":"https://api.github.com/users/stefdoerr/following{/other_user}","gists_url":"https://api.github.com/users/stefdoerr/gists{/gist_id}","starred_url":"https://api.github.com/users/stefdoerr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stefdoerr/subscriptions","organizations_url":"https://api.github.com/users/stefdoerr/orgs","repos_url":"https://api.github.com/users/stefdoerr/repos","events_url":"https://api.github.com/users/stefdoerr/events{/privacy}","received_events_url":"https://api.github.com/users/stefdoerr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-04-30T08:25:47Z","updated_at":"2017-05-09T14:02:04Z","closed_at":null,"body":"I have had this weird behaviour where after 380000 epochs the generator suddenly mode collapses.\r\nFor example see:\r\n\r\nepoch 350k\r\n![349500_g](https://cloud.githubusercontent.com/assets/7935362/25562745/92346624-2d8e-11e7-8ba6-09d4b11823da.png)\r\n\r\nepoch 390k\r\n![389500_g](https://cloud.githubusercontent.com/assets/7935362/25562750/a5936814-2d8e-11e7-8acb-873f25b9e6e8.png)\r\n\r\nepoch 396k\r\n![396000_g](https://cloud.githubusercontent.com/assets/7935362/25562755/afe2587a-2d8e-11e7-9258-48603bdb465c.png)\r\n\r\nepoch 400k\r\n![402000_g](https://cloud.githubusercontent.com/assets/7935362/25562758/b87ccb3c-2d8e-11e7-87f7-cd1bbf9eba2a.png)\r\n\r\nepoch 480k\r\n![483000_g](https://cloud.githubusercontent.com/assets/7935362/25562765/d02af218-2d8e-11e7-97a0-3a05792f7c64.png)\r\n\r\n\r\nI am suspecting that portrait guy in the middle of the lower row as the poltergeist that caused the mode collapse but I can't prove it :laughing: . Anyone seen this happen before? \r\n\r\nYou can also see the g_loss and d_loss_fake increasing around epoch 380k\r\n![screenshot from 2017-04-30 10-23-16](https://cloud.githubusercontent.com/assets/7935362/25562791/3e93dada-2d8f-11e7-94c4-f3609b970783.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/comments/300174080","html_url":"https://github.com/carpedm20/BEGAN-tensorflow/issues/19#issuecomment-300174080","issue_url":"https://api.github.com/repos/carpedm20/BEGAN-tensorflow/issues/19","id":300174080,"user":{"login":"stefdoerr","id":7935362,"avatar_url":"https://avatars0.githubusercontent.com/u/7935362?v=3","gravatar_id":"","url":"https://api.github.com/users/stefdoerr","html_url":"https://github.com/stefdoerr","followers_url":"https://api.github.com/users/stefdoerr/followers","following_url":"https://api.github.com/users/stefdoerr/following{/other_user}","gists_url":"https://api.github.com/users/stefdoerr/gists{/gist_id}","starred_url":"https://api.github.com/users/stefdoerr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stefdoerr/subscriptions","organizations_url":"https://api.github.com/users/stefdoerr/orgs","repos_url":"https://api.github.com/users/stefdoerr/repos","events_url":"https://api.github.com/users/stefdoerr/events{/privacy}","received_events_url":"https://api.github.com/users/stefdoerr/received_events","type":"User","site_admin":false},"created_at":"2017-05-09T14:02:04Z","updated_at":"2017-05-09T14:02:04Z","body":"Ok I think I found the issue. I will have to retrain to make sure but I'm quite convinced.\r\n\r\nIn this line\r\nhttps://github.com/carpedm20/BEGAN-tensorflow/blob/master/models.py#L19\r\nyou were passing all variables, not just the trainable ones. I had modified a bit the code for the nearest-neighbour upscaling and added a new variable which I didn't want to train.\r\nI replaced it now with `get_trainable_variables`"}}
{"actor":{"display_login":"stefdoerr"}}