{"action":"created","issue":{"url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/1","repository_url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma","labels_url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/1/comments","events_url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/1/events","html_url":"https://github.com/ekansa/data-rescue-nps-irma/issues/1","id":207705703,"number":1,"title":"Download Progress","user":{"login":"maxogden","id":39759,"avatar_url":"https://avatars.githubusercontent.com/u/39759?v=3","gravatar_id":"","url":"https://api.github.com/users/maxogden","html_url":"https://github.com/maxogden","followers_url":"https://api.github.com/users/maxogden/followers","following_url":"https://api.github.com/users/maxogden/following{/other_user}","gists_url":"https://api.github.com/users/maxogden/gists{/gist_id}","starred_url":"https://api.github.com/users/maxogden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxogden/subscriptions","organizations_url":"https://api.github.com/users/maxogden/orgs","repos_url":"https://api.github.com/users/maxogden/repos","events_url":"https://api.github.com/users/maxogden/events{/privacy}","received_events_url":"https://api.github.com/users/maxogden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-02-15T04:29:07Z","updated_at":"2017-02-15T17:56:42Z","closed_at":null,"body":"Hi, I'm working on an initial download and will post my progress here.\r\n\r\nI have a script that grabs the URLs and makes a file I can feed into my parallel downloader\r\n\r\n```js\r\nvar ids = require('./data/item_ids.json')\r\nvar fs = require('fs')\r\nvar ndjson = require('ndjson')\r\n\r\nvar stream = ndjson.serialize()\r\n\r\nstream.pipe(process.stdout)\r\n\r\nids.forEach(function (id) {\r\n  var file = fs.readFileSync('./data/' + id + '/' + id + '-files.json').toString()\r\n  var json = JSON.parse(file)\r\n  json.forEach(function (j) {\r\n    stream.write({folder: id, url: j.Url, id: j.Id})\r\n  })\r\n})\r\n\r\nstream.end()\r\n```\r\n\r\nInterestingly there are some duplicates:\r\n\r\n```\r\nmax@burrito:/media/hd4/data-rescue-nps-irma$ node ndjson.js > urls.json\r\nmax@burrito:/media/hd4/data-rescue-nps-irma$ wc -l urls.json \r\n42144 urls.json\r\nmax@burrito:/media/hd4/data-rescue-nps-irma$ cat urls.json | jsonfilter url | sort | uniq | wc -l\r\n40600\r\n```\r\n\r\nI'm starting the download of those 40600 now"},"comment":{"url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/comments/280086988","html_url":"https://github.com/ekansa/data-rescue-nps-irma/issues/1#issuecomment-280086988","issue_url":"https://api.github.com/repos/ekansa/data-rescue-nps-irma/issues/1","id":280086988,"user":{"login":"maxogden","id":39759,"avatar_url":"https://avatars.githubusercontent.com/u/39759?v=3","gravatar_id":"","url":"https://api.github.com/users/maxogden","html_url":"https://github.com/maxogden","followers_url":"https://api.github.com/users/maxogden/followers","following_url":"https://api.github.com/users/maxogden/following{/other_user}","gists_url":"https://api.github.com/users/maxogden/gists{/gist_id}","starred_url":"https://api.github.com/users/maxogden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxogden/subscriptions","organizations_url":"https://api.github.com/users/maxogden/orgs","repos_url":"https://api.github.com/users/maxogden/repos","events_url":"https://api.github.com/users/maxogden/events{/privacy}","received_events_url":"https://api.github.com/users/maxogden/received_events","type":"User","site_admin":false},"created_at":"2017-02-15T17:56:42Z","updated_at":"2017-02-15T17:56:42Z","body":"Seems like the error rate is pretty low. Here's a status update:\r\n\r\n```\r\nmax@burrito:/media/hd4$ du -sh data\r\n239G\tdata\r\nmax@burrito:/media/hd4$ wc -l results.json \r\n11425 results.json\r\nmax@burrito:/media/hd4$ cat results.json | grep errMsg | wc -l\r\n74\r\n```\r\n\r\nSo assuming some uniformity in file sizes it's looking around 1TB"}}
{"actor":{"display_login":"maxogden"}}