{"action":"created","issue":{"url":"https://api.github.com/repos/LLNL/lbann/issues/114","repository_url":"https://api.github.com/repos/LLNL/lbann","labels_url":"https://api.github.com/repos/LLNL/lbann/issues/114/labels{/name}","comments_url":"https://api.github.com/repos/LLNL/lbann/issues/114/comments","events_url":"https://api.github.com/repos/LLNL/lbann/issues/114/events","html_url":"https://github.com/LLNL/lbann/issues/114","id":262774867,"number":114,"title":"Data reader update error","user":{"login":"ndryden","id":1119788,"avatar_url":"https://avatars3.githubusercontent.com/u/1119788?v=4","gravatar_id":"","url":"https://api.github.com/users/ndryden","html_url":"https://github.com/ndryden","followers_url":"https://api.github.com/users/ndryden/followers","following_url":"https://api.github.com/users/ndryden/following{/other_user}","gists_url":"https://api.github.com/users/ndryden/gists{/gist_id}","starred_url":"https://api.github.com/users/ndryden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ndryden/subscriptions","organizations_url":"https://api.github.com/users/ndryden/orgs","repos_url":"https://api.github.com/users/ndryden/repos","events_url":"https://api.github.com/users/ndryden/events{/privacy}","received_events_url":"https://api.github.com/users/ndryden/received_events","type":"User","site_admin":false},"labels":[{"id":374636002,"url":"https://api.github.com/repos/LLNL/lbann/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":{"login":"bvanessen","id":6210171,"avatar_url":"https://avatars2.githubusercontent.com/u/6210171?v=4","gravatar_id":"","url":"https://api.github.com/users/bvanessen","html_url":"https://github.com/bvanessen","followers_url":"https://api.github.com/users/bvanessen/followers","following_url":"https://api.github.com/users/bvanessen/following{/other_user}","gists_url":"https://api.github.com/users/bvanessen/gists{/gist_id}","starred_url":"https://api.github.com/users/bvanessen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bvanessen/subscriptions","organizations_url":"https://api.github.com/users/bvanessen/orgs","repos_url":"https://api.github.com/users/bvanessen/repos","events_url":"https://api.github.com/users/bvanessen/events{/privacy}","received_events_url":"https://api.github.com/users/bvanessen/received_events","type":"User","site_admin":false},"assignees":[{"login":"bvanessen","id":6210171,"avatar_url":"https://avatars2.githubusercontent.com/u/6210171?v=4","gravatar_id":"","url":"https://api.github.com/users/bvanessen","html_url":"https://github.com/bvanessen","followers_url":"https://api.github.com/users/bvanessen/followers","following_url":"https://api.github.com/users/bvanessen/following{/other_user}","gists_url":"https://api.github.com/users/bvanessen/gists{/gist_id}","starred_url":"https://api.github.com/users/bvanessen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bvanessen/subscriptions","organizations_url":"https://api.github.com/users/bvanessen/orgs","repos_url":"https://api.github.com/users/bvanessen/repos","events_url":"https://api.github.com/users/bvanessen/events{/privacy}","received_events_url":"https://api.github.com/users/bvanessen/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2017-10-04T12:50:10Z","updated_at":"2017-10-04T17:10:54Z","closed_at":"2017-10-04T17:10:54Z","author_association":"COLLABORATOR","body":"AlexNet, 128 nodes/256 ranks, with the training dataset having\r\n```\r\ntrain_or_test_percent: 0.001\r\nvalidation_percent: 0.01\r\n```\r\nand the testing dataset having\r\n```\r\ntrain_or_test_percent: 0.01\r\nvalidation_percent: 1.0\r\n```\r\nwhich yields\r\n```\r\nTraining using 99.0632% of the training data set, which is 1269 samples.\r\nValidating training using 0.936768% of the training data set, which is 12 samples.\r\n```\r\nand\r\n```\r\n--------------------------------------------------------------------------------\r\n[1] Epoch : stats formated [tr/v/te] iter/epoch = [5/1/2]\r\n            global MB = [ 256/  12/ 256] global last MB = [ 245/  12/ 244]\r\n             local MB = [ 256/  12/ 256]  local last MB = [ 245/  12/ 244]\r\n--------------------------------------------------------------------------------\r\n```\r\ncompletes training and then I get the following error:\r\n```\r\nLBANN: rank 0 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 488 and there are 500 indices\r\nLBANN: rank 9 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 497 and there are 500 indices\r\nLBANN: rank 10 of model 0 caught error message: /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 498 and there are 500 indices\r\nLBANN: rank 4 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 492 and there are 500 indices\r\nLBANN: rank 6 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 494 and there are 500 indices\r\napplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 0\r\nLBANN: rank 8 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 496 and there are 500 indices\r\nLBANN: rank 11 of model 0 caught error message: /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 499 and there are 500 indices\r\nLBANN: rank 5 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 493 and there are 500 indices\r\nLBANN: rank 7 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 495 and there are 500 indices\r\nLBANN: rank 1 of model 0 caught error message:  /usr/workspace/wsa/dryden1/lbann/src/data_readers/data_reader.cpp 216 :: generic data reader update error: the epoch is complete, but not all of the data has been used\r\n -- current pos = 489 and there are 500 indices\r\n```"},"comment":{"url":"https://api.github.com/repos/LLNL/lbann/issues/comments/334224824","html_url":"https://github.com/LLNL/lbann/issues/114#issuecomment-334224824","issue_url":"https://api.github.com/repos/LLNL/lbann/issues/114","id":334224824,"user":{"login":"ndryden","id":1119788,"avatar_url":"https://avatars3.githubusercontent.com/u/1119788?v=4","gravatar_id":"","url":"https://api.github.com/users/ndryden","html_url":"https://github.com/ndryden","followers_url":"https://api.github.com/users/ndryden/followers","following_url":"https://api.github.com/users/ndryden/following{/other_user}","gists_url":"https://api.github.com/users/ndryden/gists{/gist_id}","starred_url":"https://api.github.com/users/ndryden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ndryden/subscriptions","organizations_url":"https://api.github.com/users/ndryden/orgs","repos_url":"https://api.github.com/users/ndryden/repos","events_url":"https://api.github.com/users/ndryden/events{/privacy}","received_events_url":"https://api.github.com/users/ndryden/received_events","type":"User","site_admin":false},"created_at":"2017-10-04T17:10:54Z","updated_at":"2017-10-04T17:10:54Z","author_association":"COLLABORATOR","body":"Yes, seems to work."}}
{"actor":{"display_login":"ndryden"}}