{"action":"created","comment":{"url":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments/127028053","pull_request_review_id":49571335,"id":127028053,"diff_hunk":"@@ -3,79 +3,119 @@\n Scaling Out Data Ingestion\n ##########################\n \n-Citus lets you scale out data ingestion to very high rates, but there are several trade-offs to consider in terms of the throughput, durability, consistency and latency. In this section, we discuss several approaches to data ingestion and give examples of how to use them.\n+Citus lets you scale out data ingestion to very high rates, but there are several trade-offs to consider in terms of application integration, throughput, and latency. In this section, we discuss different approaches to data ingestion, and provide guidelines for expected throughput and latency numbers.\n \n-Real-time Inserts (0-50k/s)\n----------------------------\n+Real-time Insert and Updates\n+----------------------------\n \n-On the Citus coordinator, you can perform INSERT commands directly on hash distributed tables. The advantage of using INSERT is that the new data is immediately visible to SELECT queries, and durably stored on multiple replicas.\n+On the Citus coordinator, you can perform INSERT, INSERT .. ON CONFLICT, UPDATE, and DELETE commands directly on distributed tables. When you issue one of these commands, the changes are immediately visible to the user.\n \n-When processing an INSERT, Citus first finds the right shard placements based on the value in the distribution column, then it connects to the workers storing the shard placements, and finally performs an INSERT on each of them. From the perspective of the user, the INSERT takes several milliseconds to process because of the round-trips to the workers, but the coordinator can process other INSERTs in other sessions while waiting for a response. The coordinator also keeps connections to the workers open within the same session, which means subsequent queries will see lower response times.\n+When you run an INSERT (or another ingest command), Citus first finds the right shard placements based on the value in the distribution column. Citus then connects to the worker nodes storing the shard placements, and performs an INSERT on each of them. From the perspective of the user, the INSERT takes several milliseconds to process because of the network latency to worker nodes. The Citus coordinator node however can process concurrent INSERTs to reach high throughputs.\n \n-::\n+Insert Throughput\n+~~~~~~~~~~~~~~~~~\n \n-    -- Set up a distributed table containing counters\n-    CREATE TABLE counters (c_key text, c_date date, c_value int, primary key (c_key, c_date));\n-    SELECT create_distributed_table('counters', 'c_key');\n+To measure data ingest rates with Citus, we use a standard tool called pgbench and provide :ref:`repeatable benchmarking steps <citus_write_throughput_benchmark>`.\n \n-    -- Enable timing to see reponse times\n-    \\timing on\n+We also used these steps to run pgbench across different Citus Cloud formations on AWS and observed the following ingest rates for transactional INSERT statements. For these benchmark results, we used Citus Cloud formations' default configuration, and set pgbench's concurrent thread count to 64 and client count to 256. We didn't apply any optimizations to improve performance numbers; and you can get higher ingest ratios by tuning your database setup.","path":"performance/scaling_data_ingestion.rst","position":29,"original_position":29,"commit_id":"11a9a656caf3f3ff84ef0b2e342b20fabb19b368","original_commit_id":"11a9a656caf3f3ff84ef0b2e342b20fabb19b368","user":{"login":"begriffs","id":911911,"avatar_url":"https://avatars1.githubusercontent.com/u/911911?v=3","gravatar_id":"","url":"https://api.github.com/users/begriffs","html_url":"https://github.com/begriffs","followers_url":"https://api.github.com/users/begriffs/followers","following_url":"https://api.github.com/users/begriffs/following{/other_user}","gists_url":"https://api.github.com/users/begriffs/gists{/gist_id}","starred_url":"https://api.github.com/users/begriffs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/begriffs/subscriptions","organizations_url":"https://api.github.com/users/begriffs/orgs","repos_url":"https://api.github.com/users/begriffs/repos","events_url":"https://api.github.com/users/begriffs/events{/privacy}","received_events_url":"https://api.github.com/users/begriffs/received_events","type":"User","site_admin":false},"body":"I think it reads better as \"the default configuration for Citus Cloud formations\" rather than \"Citus Cloud formations' default configuration\"","created_at":"2017-07-12T18:02:02Z","updated_at":"2017-07-12T18:02:02Z","html_url":"https://github.com/citusdata/citus_docs/pull/402#discussion_r127028053","pull_request_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/402","_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments/127028053"},"html":{"href":"https://github.com/citusdata/citus_docs/pull/402#discussion_r127028053"},"pull_request":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/402"}}},"pull_request":{"url":"https://api.github.com/repos/citusdata/citus_docs/pulls/402","id":129539276,"html_url":"https://github.com/citusdata/citus_docs/pull/402","diff_url":"https://github.com/citusdata/citus_docs/pull/402.diff","patch_url":"https://github.com/citusdata/citus_docs/pull/402.patch","issue_url":"https://api.github.com/repos/citusdata/citus_docs/issues/402","number":402,"state":"open","locked":false,"title":"Initial updates to scaling data ingestion branch","user":{"login":"ozgune","id":2545443,"avatar_url":"https://avatars0.githubusercontent.com/u/2545443?v=3","gravatar_id":"","url":"https://api.github.com/users/ozgune","html_url":"https://github.com/ozgune","followers_url":"https://api.github.com/users/ozgune/followers","following_url":"https://api.github.com/users/ozgune/following{/other_user}","gists_url":"https://api.github.com/users/ozgune/gists{/gist_id}","starred_url":"https://api.github.com/users/ozgune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozgune/subscriptions","organizations_url":"https://api.github.com/users/ozgune/orgs","repos_url":"https://api.github.com/users/ozgune/repos","events_url":"https://api.github.com/users/ozgune/events{/privacy}","received_events_url":"https://api.github.com/users/ozgune/received_events","type":"User","site_admin":false},"body":"This branch still needs to include the following changes before it can be reviewed:\r\n\r\n1. Add two tables that provide guidelines for INSERT and UPDATEs\r\n2. Decide if we need to provide any benchmarking steps for COPY\r\n3. Incorporate the section that has repeatable benchmark steps: INSERT and UPDATE\r\n4. Draw a benchmark diagram for the repeatable benchmark steps document\r\n5. Run numbers of update throughput","created_at":"2017-07-08T19:16:35Z","updated_at":"2017-07-12T18:02:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6fb628a5119675928cc89a29e857019ac9561e2e","assignee":{"login":"ozgune","id":2545443,"avatar_url":"https://avatars0.githubusercontent.com/u/2545443?v=3","gravatar_id":"","url":"https://api.github.com/users/ozgune","html_url":"https://github.com/ozgune","followers_url":"https://api.github.com/users/ozgune/followers","following_url":"https://api.github.com/users/ozgune/following{/other_user}","gists_url":"https://api.github.com/users/ozgune/gists{/gist_id}","starred_url":"https://api.github.com/users/ozgune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozgune/subscriptions","organizations_url":"https://api.github.com/users/ozgune/orgs","repos_url":"https://api.github.com/users/ozgune/repos","events_url":"https://api.github.com/users/ozgune/events{/privacy}","received_events_url":"https://api.github.com/users/ozgune/received_events","type":"User","site_admin":false},"assignees":[{"login":"ozgune","id":2545443,"avatar_url":"https://avatars0.githubusercontent.com/u/2545443?v=3","gravatar_id":"","url":"https://api.github.com/users/ozgune","html_url":"https://github.com/ozgune","followers_url":"https://api.github.com/users/ozgune/followers","following_url":"https://api.github.com/users/ozgune/following{/other_user}","gists_url":"https://api.github.com/users/ozgune/gists{/gist_id}","starred_url":"https://api.github.com/users/ozgune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozgune/subscriptions","organizations_url":"https://api.github.com/users/ozgune/orgs","repos_url":"https://api.github.com/users/ozgune/repos","events_url":"https://api.github.com/users/ozgune/events{/privacy}","received_events_url":"https://api.github.com/users/ozgune/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/402/commits","review_comments_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/402/comments","review_comment_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/issues/402/comments","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/11a9a656caf3f3ff84ef0b2e342b20fabb19b368","head":{"label":"citusdata:ozgune-scaling-data-ingestion-update","ref":"ozgune-scaling-data-ingestion-update","sha":"11a9a656caf3f3ff84ef0b2e342b20fabb19b368","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars1.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":56884660,"name":"citus_docs","full_name":"citusdata/citus_docs","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars1.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus_docs","description":"Citus Documentation","fork":false,"url":"https://api.github.com/repos/citusdata/citus_docs","forks_url":"https://api.github.com/repos/citusdata/citus_docs/forks","keys_url":"https://api.github.com/repos/citusdata/citus_docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus_docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus_docs/teams","hooks_url":"https://api.github.com/repos/citusdata/citus_docs/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus_docs/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus_docs/events","assignees_url":"https://api.github.com/repos/citusdata/citus_docs/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus_docs/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus_docs/tags","blobs_url":"https://api.github.com/repos/citusdata/citus_docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus_docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus_docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus_docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus_docs/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus_docs/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus_docs/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus_docs/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus_docs/subscription","commits_url":"https://api.github.com/repos/citusdata/citus_docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus_docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus_docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus_docs/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus_docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus_docs/merges","archive_url":"https://api.github.com/repos/citusdata/citus_docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus_docs/downloads","issues_url":"https://api.github.com/repos/citusdata/citus_docs/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus_docs/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus_docs/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus_docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus_docs/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus_docs/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus_docs/deployments","created_at":"2016-04-22T20:54:52Z","updated_at":"2017-06-29T12:27:55Z","pushed_at":"2017-07-11T20:30:14Z","git_url":"git://github.com/citusdata/citus_docs.git","ssh_url":"git@github.com:citusdata/citus_docs.git","clone_url":"https://github.com/citusdata/citus_docs.git","svn_url":"https://github.com/citusdata/citus_docs","homepage":null,"size":6815,"stargazers_count":12,"watchers_count":12,"language":"CSS","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"open_issues_count":110,"forks":9,"open_issues":110,"watchers":12,"default_branch":"master"}},"base":{"label":"citusdata:master","ref":"master","sha":"93adc3a65bc043d1fabeb9d7630ae4f469133da2","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars1.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":56884660,"name":"citus_docs","full_name":"citusdata/citus_docs","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars1.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus_docs","description":"Citus Documentation","fork":false,"url":"https://api.github.com/repos/citusdata/citus_docs","forks_url":"https://api.github.com/repos/citusdata/citus_docs/forks","keys_url":"https://api.github.com/repos/citusdata/citus_docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus_docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus_docs/teams","hooks_url":"https://api.github.com/repos/citusdata/citus_docs/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus_docs/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus_docs/events","assignees_url":"https://api.github.com/repos/citusdata/citus_docs/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus_docs/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus_docs/tags","blobs_url":"https://api.github.com/repos/citusdata/citus_docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus_docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus_docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus_docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus_docs/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus_docs/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus_docs/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus_docs/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus_docs/subscription","commits_url":"https://api.github.com/repos/citusdata/citus_docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus_docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus_docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus_docs/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus_docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus_docs/merges","archive_url":"https://api.github.com/repos/citusdata/citus_docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus_docs/downloads","issues_url":"https://api.github.com/repos/citusdata/citus_docs/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus_docs/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus_docs/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus_docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus_docs/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus_docs/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus_docs/deployments","created_at":"2016-04-22T20:54:52Z","updated_at":"2017-06-29T12:27:55Z","pushed_at":"2017-07-11T20:30:14Z","git_url":"git://github.com/citusdata/citus_docs.git","ssh_url":"git@github.com:citusdata/citus_docs.git","clone_url":"https://github.com/citusdata/citus_docs.git","svn_url":"https://github.com/citusdata/citus_docs","homepage":null,"size":6815,"stargazers_count":12,"watchers_count":12,"language":"CSS","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"open_issues_count":110,"forks":9,"open_issues":110,"watchers":12,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/402"},"html":{"href":"https://github.com/citusdata/citus_docs/pull/402"},"issue":{"href":"https://api.github.com/repos/citusdata/citus_docs/issues/402"},"comments":{"href":"https://api.github.com/repos/citusdata/citus_docs/issues/402/comments"},"review_comments":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/402/comments"},"review_comment":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/402/commits"},"statuses":{"href":"https://api.github.com/repos/citusdata/citus_docs/statuses/11a9a656caf3f3ff84ef0b2e342b20fabb19b368"}}}}
{"actor":{"display_login":"begriffs"}}