{"action":"created","comment":{"url":"https://api.github.com/repos/opendatakit/docs/pulls/comments/141933187","pull_request_review_id":66230579,"id":141933187,"diff_hunk":"@@ -0,0 +1,41 @@\n+Aggregate Limitations\n+========================\n+\n+- Push of data to the server should be serialized. Within ODK Aggregate, there is a global mutex (`TaskLock` across all server instances, mediated by the datastore layer) in the server when inserting submissions. Having multiple push requests occurring simultaneously will cause them to block on the mutex, chewing up their 60-second request limit, as they get processed in single file no matter how many server instances are spun up.\n+\n+- The 60-second request limit can be very commonly exceeded over low-bandwidth connections, and even text-only submissions can be impacted on satellite connections. That is why ODK Collect splits submissions into multiple 10MB submission requests. The timer starts upon receipt of the first byte, so a slow connection can account for a sizeable portion of those 60 seconds. Same applies for sending a response. The processing on the server is generally negligible in relation to the transmission times.\n+\n+.. note::\n+   \n+   - The above two limitations, the global mutex and the in-memory copies/full-packet-assembly, are a result of implementing on top of AppEngine and its Datastore.\n+   - A server that used database transactions and that used streaming servlet 3.0 functionality would have less trouble with concurrent requests.\n+\n+\n+- Pull of data can be overlapped. However, for submissions with many media attachments, keep in mind that ODK Aggregate holds all media attachments in memory, then copies them into the HttpEntity, so this can consume a fair amount of memory (everything is held in memory at least twice). The same memory consumption applies on pushes / submission of forms. i.e., you probably want to manage how many media attachment download requests you have in play at the same time.\n+\n+- Spinning up of copies of the frontend will incur faster quota usage on AppEngine. For that reason, the Aggregate configuration here specifies a 14-second queuing time threshold before a new instance is spun up. Only if at least one request is queued for longer than 14 seconds will a new instance be spun up, and then that new instance will take about 30 seconds to become live. Leaving a 15-second processing interval. This is why ODK Collect tried twice before failing a submit.\n+\n+\n+- Aggregate 0.9.x has a 1000-record limit to the number of records it can display (this is by design); it does, however, still retain all records i.e. it will hold as many records as you've uploaded and does not loose or ignore the 1001st record, etc; in general, any webserver will have a limit to what it can display interactively (at some point it will run out of memory or time out). \n+\n+.. tip::\n+  \n+  - On Aggregate 0.9.x, we provide the Briefcase applet (one of the links on the top row) to allow you to extract your data from the server into a local CSV file for local processing. That should retrieve all data within the server, and should work regardless of the number of submissions you have uploaded.","path":"aggregate-limitations.rst","position":23,"original_position":23,"commit_id":"02968c26830c591aae57596fc3c5d683a5fa8e81","original_commit_id":"02968c26830c591aae57596fc3c5d683a5fa8e81","user":{"login":"ankita240796","id":15226041,"avatar_url":"https://avatars3.githubusercontent.com/u/15226041?v=4","gravatar_id":"","url":"https://api.github.com/users/ankita240796","html_url":"https://github.com/ankita240796","followers_url":"https://api.github.com/users/ankita240796/followers","following_url":"https://api.github.com/users/ankita240796/following{/other_user}","gists_url":"https://api.github.com/users/ankita240796/gists{/gist_id}","starred_url":"https://api.github.com/users/ankita240796/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankita240796/subscriptions","organizations_url":"https://api.github.com/users/ankita240796/orgs","repos_url":"https://api.github.com/users/ankita240796/repos","events_url":"https://api.github.com/users/ankita240796/events{/privacy}","received_events_url":"https://api.github.com/users/ankita240796/received_events","type":"User","site_admin":false},"body":"@adammichaelwood I meant to say here that the server shows the link at the top.\r\n![selection_001](https://user-images.githubusercontent.com/15226041/31029690-559381fa-a570-11e7-89a0-6def8c124d82.png)\r\nDo I need to insert the link here?\r\n","created_at":"2017-09-29T18:16:18Z","updated_at":"2017-09-29T18:16:18Z","html_url":"https://github.com/opendatakit/docs/pull/235#discussion_r141933187","pull_request_url":"https://api.github.com/repos/opendatakit/docs/pulls/235","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/comments/141933187"},"html":{"href":"https://github.com/opendatakit/docs/pull/235#discussion_r141933187"},"pull_request":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/235"}},"in_reply_to_id":141909666},"pull_request":{"url":"https://api.github.com/repos/opendatakit/docs/pulls/235","id":143899146,"html_url":"https://github.com/opendatakit/docs/pull/235","diff_url":"https://github.com/opendatakit/docs/pull/235.diff","patch_url":"https://github.com/opendatakit/docs/pull/235.patch","issue_url":"https://api.github.com/repos/opendatakit/docs/issues/235","number":235,"state":"open","locked":false,"title":"Added limitations of Aggregate","user":{"login":"ankita240796","id":15226041,"avatar_url":"https://avatars3.githubusercontent.com/u/15226041?v=4","gravatar_id":"","url":"https://api.github.com/users/ankita240796","html_url":"https://github.com/ankita240796","followers_url":"https://api.github.com/users/ankita240796/followers","following_url":"https://api.github.com/users/ankita240796/following{/other_user}","gists_url":"https://api.github.com/users/ankita240796/gists{/gist_id}","starred_url":"https://api.github.com/users/ankita240796/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankita240796/subscriptions","organizations_url":"https://api.github.com/users/ankita240796/orgs","repos_url":"https://api.github.com/users/ankita240796/repos","events_url":"https://api.github.com/users/ankita240796/events{/privacy}","received_events_url":"https://api.github.com/users/ankita240796/received_events","type":"User","site_admin":false},"body":"closes #26 \r\n\r\n\r\n\r\n## What is included in this PR?\r\nAdded limitations of Aggregate and issues with different versions. \r\n\r\n\r\n","created_at":"2017-09-29T14:15:47Z","updated_at":"2017-09-29T18:16:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0a8253340682a92f87173a803e0f332a296dcc44","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/opendatakit/docs/pulls/235/commits","review_comments_url":"https://api.github.com/repos/opendatakit/docs/pulls/235/comments","review_comment_url":"https://api.github.com/repos/opendatakit/docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/opendatakit/docs/issues/235/comments","statuses_url":"https://api.github.com/repos/opendatakit/docs/statuses/02968c26830c591aae57596fc3c5d683a5fa8e81","head":{"label":"ankita240796:aggregate-limit","ref":"aggregate-limit","sha":"02968c26830c591aae57596fc3c5d683a5fa8e81","user":{"login":"ankita240796","id":15226041,"avatar_url":"https://avatars3.githubusercontent.com/u/15226041?v=4","gravatar_id":"","url":"https://api.github.com/users/ankita240796","html_url":"https://github.com/ankita240796","followers_url":"https://api.github.com/users/ankita240796/followers","following_url":"https://api.github.com/users/ankita240796/following{/other_user}","gists_url":"https://api.github.com/users/ankita240796/gists{/gist_id}","starred_url":"https://api.github.com/users/ankita240796/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankita240796/subscriptions","organizations_url":"https://api.github.com/users/ankita240796/orgs","repos_url":"https://api.github.com/users/ankita240796/repos","events_url":"https://api.github.com/users/ankita240796/events{/privacy}","received_events_url":"https://api.github.com/users/ankita240796/received_events","type":"User","site_admin":false},"repo":{"id":103161803,"name":"docs","full_name":"ankita240796/docs","owner":{"login":"ankita240796","id":15226041,"avatar_url":"https://avatars3.githubusercontent.com/u/15226041?v=4","gravatar_id":"","url":"https://api.github.com/users/ankita240796","html_url":"https://github.com/ankita240796","followers_url":"https://api.github.com/users/ankita240796/followers","following_url":"https://api.github.com/users/ankita240796/following{/other_user}","gists_url":"https://api.github.com/users/ankita240796/gists{/gist_id}","starred_url":"https://api.github.com/users/ankita240796/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankita240796/subscriptions","organizations_url":"https://api.github.com/users/ankita240796/orgs","repos_url":"https://api.github.com/users/ankita240796/repos","events_url":"https://api.github.com/users/ankita240796/events{/privacy}","received_events_url":"https://api.github.com/users/ankita240796/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ankita240796/docs","description":"Open Data Kit Documentation, built in Sphinx.","fork":true,"url":"https://api.github.com/repos/ankita240796/docs","forks_url":"https://api.github.com/repos/ankita240796/docs/forks","keys_url":"https://api.github.com/repos/ankita240796/docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ankita240796/docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ankita240796/docs/teams","hooks_url":"https://api.github.com/repos/ankita240796/docs/hooks","issue_events_url":"https://api.github.com/repos/ankita240796/docs/issues/events{/number}","events_url":"https://api.github.com/repos/ankita240796/docs/events","assignees_url":"https://api.github.com/repos/ankita240796/docs/assignees{/user}","branches_url":"https://api.github.com/repos/ankita240796/docs/branches{/branch}","tags_url":"https://api.github.com/repos/ankita240796/docs/tags","blobs_url":"https://api.github.com/repos/ankita240796/docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ankita240796/docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ankita240796/docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/ankita240796/docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ankita240796/docs/statuses/{sha}","languages_url":"https://api.github.com/repos/ankita240796/docs/languages","stargazers_url":"https://api.github.com/repos/ankita240796/docs/stargazers","contributors_url":"https://api.github.com/repos/ankita240796/docs/contributors","subscribers_url":"https://api.github.com/repos/ankita240796/docs/subscribers","subscription_url":"https://api.github.com/repos/ankita240796/docs/subscription","commits_url":"https://api.github.com/repos/ankita240796/docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/ankita240796/docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/ankita240796/docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/ankita240796/docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/ankita240796/docs/contents/{+path}","compare_url":"https://api.github.com/repos/ankita240796/docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ankita240796/docs/merges","archive_url":"https://api.github.com/repos/ankita240796/docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ankita240796/docs/downloads","issues_url":"https://api.github.com/repos/ankita240796/docs/issues{/number}","pulls_url":"https://api.github.com/repos/ankita240796/docs/pulls{/number}","milestones_url":"https://api.github.com/repos/ankita240796/docs/milestones{/number}","notifications_url":"https://api.github.com/repos/ankita240796/docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ankita240796/docs/labels{/name}","releases_url":"https://api.github.com/repos/ankita240796/docs/releases{/id}","deployments_url":"https://api.github.com/repos/ankita240796/docs/deployments","created_at":"2017-09-11T16:43:26Z","updated_at":"2017-09-11T16:43:27Z","pushed_at":"2017-09-29T14:12:02Z","git_url":"git://github.com/ankita240796/docs.git","ssh_url":"git@github.com:ankita240796/docs.git","clone_url":"https://github.com/ankita240796/docs.git","svn_url":"https://github.com/ankita240796/docs","homepage":"http://docs.opendatakit.org/","size":316,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"opendatakit:master","ref":"master","sha":"3727ea0834c1a2791b9ce2d573cc940b1d526871","user":{"login":"opendatakit","id":6222985,"avatar_url":"https://avatars0.githubusercontent.com/u/6222985?v=4","gravatar_id":"","url":"https://api.github.com/users/opendatakit","html_url":"https://github.com/opendatakit","followers_url":"https://api.github.com/users/opendatakit/followers","following_url":"https://api.github.com/users/opendatakit/following{/other_user}","gists_url":"https://api.github.com/users/opendatakit/gists{/gist_id}","starred_url":"https://api.github.com/users/opendatakit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opendatakit/subscriptions","organizations_url":"https://api.github.com/users/opendatakit/orgs","repos_url":"https://api.github.com/users/opendatakit/repos","events_url":"https://api.github.com/users/opendatakit/events{/privacy}","received_events_url":"https://api.github.com/users/opendatakit/received_events","type":"Organization","site_admin":false},"repo":{"id":92860132,"name":"docs","full_name":"opendatakit/docs","owner":{"login":"opendatakit","id":6222985,"avatar_url":"https://avatars0.githubusercontent.com/u/6222985?v=4","gravatar_id":"","url":"https://api.github.com/users/opendatakit","html_url":"https://github.com/opendatakit","followers_url":"https://api.github.com/users/opendatakit/followers","following_url":"https://api.github.com/users/opendatakit/following{/other_user}","gists_url":"https://api.github.com/users/opendatakit/gists{/gist_id}","starred_url":"https://api.github.com/users/opendatakit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opendatakit/subscriptions","organizations_url":"https://api.github.com/users/opendatakit/orgs","repos_url":"https://api.github.com/users/opendatakit/repos","events_url":"https://api.github.com/users/opendatakit/events{/privacy}","received_events_url":"https://api.github.com/users/opendatakit/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/opendatakit/docs","description":"Open Data Kit Documentation, built in Sphinx.","fork":false,"url":"https://api.github.com/repos/opendatakit/docs","forks_url":"https://api.github.com/repos/opendatakit/docs/forks","keys_url":"https://api.github.com/repos/opendatakit/docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/opendatakit/docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/opendatakit/docs/teams","hooks_url":"https://api.github.com/repos/opendatakit/docs/hooks","issue_events_url":"https://api.github.com/repos/opendatakit/docs/issues/events{/number}","events_url":"https://api.github.com/repos/opendatakit/docs/events","assignees_url":"https://api.github.com/repos/opendatakit/docs/assignees{/user}","branches_url":"https://api.github.com/repos/opendatakit/docs/branches{/branch}","tags_url":"https://api.github.com/repos/opendatakit/docs/tags","blobs_url":"https://api.github.com/repos/opendatakit/docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/opendatakit/docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/opendatakit/docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/opendatakit/docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/opendatakit/docs/statuses/{sha}","languages_url":"https://api.github.com/repos/opendatakit/docs/languages","stargazers_url":"https://api.github.com/repos/opendatakit/docs/stargazers","contributors_url":"https://api.github.com/repos/opendatakit/docs/contributors","subscribers_url":"https://api.github.com/repos/opendatakit/docs/subscribers","subscription_url":"https://api.github.com/repos/opendatakit/docs/subscription","commits_url":"https://api.github.com/repos/opendatakit/docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/opendatakit/docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/opendatakit/docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/opendatakit/docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/opendatakit/docs/contents/{+path}","compare_url":"https://api.github.com/repos/opendatakit/docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/opendatakit/docs/merges","archive_url":"https://api.github.com/repos/opendatakit/docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/opendatakit/docs/downloads","issues_url":"https://api.github.com/repos/opendatakit/docs/issues{/number}","pulls_url":"https://api.github.com/repos/opendatakit/docs/pulls{/number}","milestones_url":"https://api.github.com/repos/opendatakit/docs/milestones{/number}","notifications_url":"https://api.github.com/repos/opendatakit/docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/opendatakit/docs/labels{/name}","releases_url":"https://api.github.com/repos/opendatakit/docs/releases{/id}","deployments_url":"https://api.github.com/repos/opendatakit/docs/deployments","created_at":"2017-05-30T17:45:43Z","updated_at":"2017-09-20T10:12:02Z","pushed_at":"2017-09-29T17:45:00Z","git_url":"git://github.com/opendatakit/docs.git","ssh_url":"git@github.com:opendatakit/docs.git","clone_url":"https://github.com/opendatakit/docs.git","svn_url":"https://github.com/opendatakit/docs","homepage":"http://docs.opendatakit.org/","size":312,"stargazers_count":5,"watchers_count":5,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":36,"mirror_url":null,"open_issues_count":74,"forks":36,"open_issues":74,"watchers":5,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/235"},"html":{"href":"https://github.com/opendatakit/docs/pull/235"},"issue":{"href":"https://api.github.com/repos/opendatakit/docs/issues/235"},"comments":{"href":"https://api.github.com/repos/opendatakit/docs/issues/235/comments"},"review_comments":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/235/comments"},"review_comment":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/opendatakit/docs/pulls/235/commits"},"statuses":{"href":"https://api.github.com/repos/opendatakit/docs/statuses/02968c26830c591aae57596fc3c5d683a5fa8e81"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"ankita240796"}}