{"action":"created","comment":{"url":"https://api.github.com/repos/mediachain/aleph/pulls/comments/102686564","pull_request_review_id":23447288,"id":102686564,"diff_hunk":"@@ -115,22 +131,30 @@ function writeDataObjectsToTarball (client: RestClient, tarball: Object, objectI\n     if (batch.length >= OBJECT_BATCH_SIZE) {\n       const ids = batch\n       batch = []\n-      batchPromises.push(fetchObjectBatch(client, tarball, ids))\n+      batchPromises.push(fetchObjectBatch(client, tarball, ids, allowErrors))\n     }\n   }\n \n-  batchPromises.push(fetchObjectBatch(client, tarball, batch))\n+  batchPromises.push(fetchObjectBatch(client, tarball, batch, allowErrors))\n   return Promise.all(batchPromises)\n }\n \n-function fetchObjectBatch (client: RestClient, tarball: Object, objectIds: Array<string>): Promise<*> {\n+function fetchObjectBatch (client: RestClient, tarball: Object, objectIds: Array<string>, allowErrors: boolean): Promise<*> {\n   if (objectIds.length < 1) return Promise.resolve()\n \n   return client.batchGetDataStream(objectIds, false)\n     .then(stream => new Promise((resolve, reject) => {\n       stream.on('data', dataResult => {\n         const key = objectIds.shift()\n-        if (dataResult == null || typeof dataResult !== 'object' || dataResult.data == null) return\n+        if (dataResult == null || typeof dataResult !== 'object' || dataResult.data == null) {\n+          const msg = (dataResult && dataResult.error) ? dataResult.error : 'Unknown error'\n+          if (allowErrors) {\n+            printlnErr(`Error fetching object for ${key}: ${msg}`)\n+            return","path":"src/client/cli/commands/archive.js","position":153,"original_position":106,"commit_id":"eccd844ca1a71d4181a88f59fffb629072ac2c55","original_commit_id":"eccd844ca1a71d4181a88f59fffb629072ac2c55","user":{"login":"vyzo","id":36426,"avatar_url":"https://avatars.githubusercontent.com/u/36426?v=3","gravatar_id":"","url":"https://api.github.com/users/vyzo","html_url":"https://github.com/vyzo","followers_url":"https://api.github.com/users/vyzo/followers","following_url":"https://api.github.com/users/vyzo/following{/other_user}","gists_url":"https://api.github.com/users/vyzo/gists{/gist_id}","starred_url":"https://api.github.com/users/vyzo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vyzo/subscriptions","organizations_url":"https://api.github.com/users/vyzo/orgs","repos_url":"https://api.github.com/users/vyzo/repos","events_url":"https://api.github.com/users/vyzo/events{/privacy}","received_events_url":"https://api.github.com/users/vyzo/received_events","type":"User","site_admin":false},"body":"so the promise stays unresolved here; is this desired behaviour?","created_at":"2017-02-23T10:47:55Z","updated_at":"2017-02-23T10:47:55Z","html_url":"https://github.com/mediachain/aleph/pull/185#discussion_r102686564","pull_request_url":"https://api.github.com/repos/mediachain/aleph/pulls/185","_links":{"self":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/comments/102686564"},"html":{"href":"https://github.com/mediachain/aleph/pull/185#discussion_r102686564"},"pull_request":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/185"}}},"pull_request":{"url":"https://api.github.com/repos/mediachain/aleph/pulls/185","id":107300498,"html_url":"https://github.com/mediachain/aleph/pull/185","diff_url":"https://github.com/mediachain/aleph/pull/185.diff","patch_url":"https://github.com/mediachain/aleph/pull/185.patch","issue_url":"https://api.github.com/repos/mediachain/aleph/issues/185","number":185,"state":"open","locked":false,"title":"Add `mcclient archive` command to generate archive tarball","user":{"login":"yusefnapora","id":678715,"avatar_url":"https://avatars.githubusercontent.com/u/678715?v=3","gravatar_id":"","url":"https://api.github.com/users/yusefnapora","html_url":"https://github.com/yusefnapora","followers_url":"https://api.github.com/users/yusefnapora/followers","following_url":"https://api.github.com/users/yusefnapora/following{/other_user}","gists_url":"https://api.github.com/users/yusefnapora/gists{/gist_id}","starred_url":"https://api.github.com/users/yusefnapora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yusefnapora/subscriptions","organizations_url":"https://api.github.com/users/yusefnapora/orgs","repos_url":"https://api.github.com/users/yusefnapora/repos","events_url":"https://api.github.com/users/yusefnapora/events{/privacy}","received_events_url":"https://api.github.com/users/yusefnapora/received_events","type":"User","site_admin":false},"body":"I figured that it would be easier to just make the tarball in JS using the [tar-stream](https://github.com/mafintosh/tar-stream) module vs writing some shell or python scripts, since we already have the code to extract and request the associated objects.\r\n\r\nThis adds an `mcclient archive <queryString>` command that will write a gzipped tarball to stdout (or you can give a `--output|-o` flag).  In the tarball will be a `stmt/<statementId>` entry for each statement, and a `data/<objectId>` entry for each data object.  The statements are stringified JSON objects, but could easily do protobufs instead.\r\n\r\nHappy to tweak the archive format in the morning (multiple statements per entry is probably a good idea).","created_at":"2017-02-21T23:13:31Z","updated_at":"2017-02-23T10:47:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c91cae47528bbc363f22c3bad5465afc4fe4099a","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/mediachain/aleph/pulls/185/commits","review_comments_url":"https://api.github.com/repos/mediachain/aleph/pulls/185/comments","review_comment_url":"https://api.github.com/repos/mediachain/aleph/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mediachain/aleph/issues/185/comments","statuses_url":"https://api.github.com/repos/mediachain/aleph/statuses/eccd844ca1a71d4181a88f59fffb629072ac2c55","head":{"label":"mediachain:yn-archive","ref":"yn-archive","sha":"eccd844ca1a71d4181a88f59fffb629072ac2c55","user":{"login":"mediachain","id":16711838,"avatar_url":"https://avatars.githubusercontent.com/u/16711838?v=3","gravatar_id":"","url":"https://api.github.com/users/mediachain","html_url":"https://github.com/mediachain","followers_url":"https://api.github.com/users/mediachain/followers","following_url":"https://api.github.com/users/mediachain/following{/other_user}","gists_url":"https://api.github.com/users/mediachain/gists{/gist_id}","starred_url":"https://api.github.com/users/mediachain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediachain/subscriptions","organizations_url":"https://api.github.com/users/mediachain/orgs","repos_url":"https://api.github.com/users/mediachain/repos","events_url":"https://api.github.com/users/mediachain/events{/privacy}","received_events_url":"https://api.github.com/users/mediachain/received_events","type":"Organization","site_admin":false},"repo":{"id":66977529,"name":"aleph","full_name":"mediachain/aleph","owner":{"login":"mediachain","id":16711838,"avatar_url":"https://avatars.githubusercontent.com/u/16711838?v=3","gravatar_id":"","url":"https://api.github.com/users/mediachain","html_url":"https://github.com/mediachain","followers_url":"https://api.github.com/users/mediachain/followers","following_url":"https://api.github.com/users/mediachain/following{/other_user}","gists_url":"https://api.github.com/users/mediachain/gists{/gist_id}","starred_url":"https://api.github.com/users/mediachain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediachain/subscriptions","organizations_url":"https://api.github.com/users/mediachain/orgs","repos_url":"https://api.github.com/users/mediachain/repos","events_url":"https://api.github.com/users/mediachain/events{/privacy}","received_events_url":"https://api.github.com/users/mediachain/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mediachain/aleph","description":"א: The mediachain universe manipulation engine","fork":false,"url":"https://api.github.com/repos/mediachain/aleph","forks_url":"https://api.github.com/repos/mediachain/aleph/forks","keys_url":"https://api.github.com/repos/mediachain/aleph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mediachain/aleph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mediachain/aleph/teams","hooks_url":"https://api.github.com/repos/mediachain/aleph/hooks","issue_events_url":"https://api.github.com/repos/mediachain/aleph/issues/events{/number}","events_url":"https://api.github.com/repos/mediachain/aleph/events","assignees_url":"https://api.github.com/repos/mediachain/aleph/assignees{/user}","branches_url":"https://api.github.com/repos/mediachain/aleph/branches{/branch}","tags_url":"https://api.github.com/repos/mediachain/aleph/tags","blobs_url":"https://api.github.com/repos/mediachain/aleph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mediachain/aleph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mediachain/aleph/git/refs{/sha}","trees_url":"https://api.github.com/repos/mediachain/aleph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mediachain/aleph/statuses/{sha}","languages_url":"https://api.github.com/repos/mediachain/aleph/languages","stargazers_url":"https://api.github.com/repos/mediachain/aleph/stargazers","contributors_url":"https://api.github.com/repos/mediachain/aleph/contributors","subscribers_url":"https://api.github.com/repos/mediachain/aleph/subscribers","subscription_url":"https://api.github.com/repos/mediachain/aleph/subscription","commits_url":"https://api.github.com/repos/mediachain/aleph/commits{/sha}","git_commits_url":"https://api.github.com/repos/mediachain/aleph/git/commits{/sha}","comments_url":"https://api.github.com/repos/mediachain/aleph/comments{/number}","issue_comment_url":"https://api.github.com/repos/mediachain/aleph/issues/comments{/number}","contents_url":"https://api.github.com/repos/mediachain/aleph/contents/{+path}","compare_url":"https://api.github.com/repos/mediachain/aleph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mediachain/aleph/merges","archive_url":"https://api.github.com/repos/mediachain/aleph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mediachain/aleph/downloads","issues_url":"https://api.github.com/repos/mediachain/aleph/issues{/number}","pulls_url":"https://api.github.com/repos/mediachain/aleph/pulls{/number}","milestones_url":"https://api.github.com/repos/mediachain/aleph/milestones{/number}","notifications_url":"https://api.github.com/repos/mediachain/aleph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mediachain/aleph/labels{/name}","releases_url":"https://api.github.com/repos/mediachain/aleph/releases{/id}","deployments_url":"https://api.github.com/repos/mediachain/aleph/deployments","created_at":"2016-08-30T21:01:45Z","updated_at":"2017-02-12T16:29:55Z","pushed_at":"2017-02-22T21:40:51Z","git_url":"git://github.com/mediachain/aleph.git","ssh_url":"git@github.com:mediachain/aleph.git","clone_url":"https://github.com/mediachain/aleph.git","svn_url":"https://github.com/mediachain/aleph","homepage":"","size":1419,"stargazers_count":16,"watchers_count":16,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3,"mirror_url":null,"open_issues_count":24,"forks":3,"open_issues":24,"watchers":16,"default_branch":"master"}},"base":{"label":"mediachain:master","ref":"master","sha":"0bf6b22785b553273e94a3ee1b7e7aff15bf667a","user":{"login":"mediachain","id":16711838,"avatar_url":"https://avatars.githubusercontent.com/u/16711838?v=3","gravatar_id":"","url":"https://api.github.com/users/mediachain","html_url":"https://github.com/mediachain","followers_url":"https://api.github.com/users/mediachain/followers","following_url":"https://api.github.com/users/mediachain/following{/other_user}","gists_url":"https://api.github.com/users/mediachain/gists{/gist_id}","starred_url":"https://api.github.com/users/mediachain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediachain/subscriptions","organizations_url":"https://api.github.com/users/mediachain/orgs","repos_url":"https://api.github.com/users/mediachain/repos","events_url":"https://api.github.com/users/mediachain/events{/privacy}","received_events_url":"https://api.github.com/users/mediachain/received_events","type":"Organization","site_admin":false},"repo":{"id":66977529,"name":"aleph","full_name":"mediachain/aleph","owner":{"login":"mediachain","id":16711838,"avatar_url":"https://avatars.githubusercontent.com/u/16711838?v=3","gravatar_id":"","url":"https://api.github.com/users/mediachain","html_url":"https://github.com/mediachain","followers_url":"https://api.github.com/users/mediachain/followers","following_url":"https://api.github.com/users/mediachain/following{/other_user}","gists_url":"https://api.github.com/users/mediachain/gists{/gist_id}","starred_url":"https://api.github.com/users/mediachain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediachain/subscriptions","organizations_url":"https://api.github.com/users/mediachain/orgs","repos_url":"https://api.github.com/users/mediachain/repos","events_url":"https://api.github.com/users/mediachain/events{/privacy}","received_events_url":"https://api.github.com/users/mediachain/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mediachain/aleph","description":"א: The mediachain universe manipulation engine","fork":false,"url":"https://api.github.com/repos/mediachain/aleph","forks_url":"https://api.github.com/repos/mediachain/aleph/forks","keys_url":"https://api.github.com/repos/mediachain/aleph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mediachain/aleph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mediachain/aleph/teams","hooks_url":"https://api.github.com/repos/mediachain/aleph/hooks","issue_events_url":"https://api.github.com/repos/mediachain/aleph/issues/events{/number}","events_url":"https://api.github.com/repos/mediachain/aleph/events","assignees_url":"https://api.github.com/repos/mediachain/aleph/assignees{/user}","branches_url":"https://api.github.com/repos/mediachain/aleph/branches{/branch}","tags_url":"https://api.github.com/repos/mediachain/aleph/tags","blobs_url":"https://api.github.com/repos/mediachain/aleph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mediachain/aleph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mediachain/aleph/git/refs{/sha}","trees_url":"https://api.github.com/repos/mediachain/aleph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mediachain/aleph/statuses/{sha}","languages_url":"https://api.github.com/repos/mediachain/aleph/languages","stargazers_url":"https://api.github.com/repos/mediachain/aleph/stargazers","contributors_url":"https://api.github.com/repos/mediachain/aleph/contributors","subscribers_url":"https://api.github.com/repos/mediachain/aleph/subscribers","subscription_url":"https://api.github.com/repos/mediachain/aleph/subscription","commits_url":"https://api.github.com/repos/mediachain/aleph/commits{/sha}","git_commits_url":"https://api.github.com/repos/mediachain/aleph/git/commits{/sha}","comments_url":"https://api.github.com/repos/mediachain/aleph/comments{/number}","issue_comment_url":"https://api.github.com/repos/mediachain/aleph/issues/comments{/number}","contents_url":"https://api.github.com/repos/mediachain/aleph/contents/{+path}","compare_url":"https://api.github.com/repos/mediachain/aleph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mediachain/aleph/merges","archive_url":"https://api.github.com/repos/mediachain/aleph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mediachain/aleph/downloads","issues_url":"https://api.github.com/repos/mediachain/aleph/issues{/number}","pulls_url":"https://api.github.com/repos/mediachain/aleph/pulls{/number}","milestones_url":"https://api.github.com/repos/mediachain/aleph/milestones{/number}","notifications_url":"https://api.github.com/repos/mediachain/aleph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mediachain/aleph/labels{/name}","releases_url":"https://api.github.com/repos/mediachain/aleph/releases{/id}","deployments_url":"https://api.github.com/repos/mediachain/aleph/deployments","created_at":"2016-08-30T21:01:45Z","updated_at":"2017-02-12T16:29:55Z","pushed_at":"2017-02-22T21:40:51Z","git_url":"git://github.com/mediachain/aleph.git","ssh_url":"git@github.com:mediachain/aleph.git","clone_url":"https://github.com/mediachain/aleph.git","svn_url":"https://github.com/mediachain/aleph","homepage":"","size":1419,"stargazers_count":16,"watchers_count":16,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3,"mirror_url":null,"open_issues_count":24,"forks":3,"open_issues":24,"watchers":16,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/185"},"html":{"href":"https://github.com/mediachain/aleph/pull/185"},"issue":{"href":"https://api.github.com/repos/mediachain/aleph/issues/185"},"comments":{"href":"https://api.github.com/repos/mediachain/aleph/issues/185/comments"},"review_comments":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/185/comments"},"review_comment":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mediachain/aleph/pulls/185/commits"},"statuses":{"href":"https://api.github.com/repos/mediachain/aleph/statuses/eccd844ca1a71d4181a88f59fffb629072ac2c55"}}}}
{"actor":{"display_login":"vyzo"}}