{"action":"created","issue":{"url":"https://api.github.com/repos/fastai/courses/issues/45","repository_url":"https://api.github.com/repos/fastai/courses","labels_url":"https://api.github.com/repos/fastai/courses/issues/45/labels{/name}","comments_url":"https://api.github.com/repos/fastai/courses/issues/45/comments","events_url":"https://api.github.com/repos/fastai/courses/issues/45/events","html_url":"https://github.com/fastai/courses/pull/45","id":206977118,"number":45,"title":"Add BcolzArrayIterator","user":{"login":"MPJansen","id":7805917,"avatar_url":"https://avatars1.githubusercontent.com/u/7805917?v=3","gravatar_id":"","url":"https://api.github.com/users/MPJansen","html_url":"https://github.com/MPJansen","followers_url":"https://api.github.com/users/MPJansen/followers","following_url":"https://api.github.com/users/MPJansen/following{/other_user}","gists_url":"https://api.github.com/users/MPJansen/gists{/gist_id}","starred_url":"https://api.github.com/users/MPJansen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MPJansen/subscriptions","organizations_url":"https://api.github.com/users/MPJansen/orgs","repos_url":"https://api.github.com/users/MPJansen/repos","events_url":"https://api.github.com/users/MPJansen/events{/privacy}","received_events_url":"https://api.github.com/users/MPJansen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-02-11T12:30:01Z","updated_at":"2017-03-05T16:39:10Z","closed_at":"2017-03-02T00:17:26Z","pull_request":{"url":"https://api.github.com/repos/fastai/courses/pulls/45","html_url":"https://github.com/fastai/courses/pull/45","diff_url":"https://github.com/fastai/courses/pull/45.diff","patch_url":"https://github.com/fastai/courses/pull/45.patch"},"body":"Add an iterator for bcolzArrays to simplify streaming from pre-computed network outputs. This avoids having to load the entire array into ram.\r\n\r\nUsage:\r\n```\r\nX = bcolz.open(path + 'train_convlayer_features.bc', mode='r')\r\ny = bcolz.open(path + 'train_labels.bc', mode='r')\r\ntrn_batches = BcolzArrayIterator(X, y, batch_size=X.chunklen * batch_size, shuffle=True)\r\nmodel.fit_generator(generator=trn_batches, samples_per_epoch=trn_batches.N, nb_epoch=1)\r\n```\r\nThanks to R4mon: http://pastebin.com/y0DskiK6"},"comment":{"url":"https://api.github.com/repos/fastai/courses/issues/comments/284241860","html_url":"https://github.com/fastai/courses/pull/45#issuecomment-284241860","issue_url":"https://api.github.com/repos/fastai/courses/issues/45","id":284241860,"user":{"login":"jph00","id":346999,"avatar_url":"https://avatars2.githubusercontent.com/u/346999?v=3","gravatar_id":"","url":"https://api.github.com/users/jph00","html_url":"https://github.com/jph00","followers_url":"https://api.github.com/users/jph00/followers","following_url":"https://api.github.com/users/jph00/following{/other_user}","gists_url":"https://api.github.com/users/jph00/gists{/gist_id}","starred_url":"https://api.github.com/users/jph00/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jph00/subscriptions","organizations_url":"https://api.github.com/users/jph00/orgs","repos_url":"https://api.github.com/users/jph00/repos","events_url":"https://api.github.com/users/jph00/events{/privacy}","received_events_url":"https://api.github.com/users/jph00/received_events","type":"User","site_admin":false},"created_at":"2017-03-05T16:39:10Z","updated_at":"2017-03-05T16:39:10Z","body":"I've been working on it all week and it keeps giving me wrong answers -\nbut only on large datasets. I just can't figure it out! When I use\nshuffle=True and do fit_generator() on the full imagenet dataset, I get\ntotally wrong results. But when I repeat the exact same code on a subset\nof 20,000 random images, I get great results. I don't want to commit it\nuntil we have it working well...\n"}}
{"actor":{"display_login":"jph00"}}