{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb/issues/470","repository_url":"https://api.github.com/repos/apache/couchdb","labels_url":"https://api.github.com/repos/apache/couchdb/issues/470/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb/issues/470/comments","events_url":"https://api.github.com/repos/apache/couchdb/issues/470/events","html_url":"https://github.com/apache/couchdb/pull/470","id":219412450,"number":470,"title":"Scheduling Replicator","user":{"login":"nickva","id":211822,"avatar_url":"https://avatars0.githubusercontent.com/u/211822?v=3","gravatar_id":"","url":"https://api.github.com/users/nickva","html_url":"https://github.com/nickva","followers_url":"https://api.github.com/users/nickva/followers","following_url":"https://api.github.com/users/nickva/following{/other_user}","gists_url":"https://api.github.com/users/nickva/gists{/gist_id}","starred_url":"https://api.github.com/users/nickva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickva/subscriptions","organizations_url":"https://api.github.com/users/nickva/orgs","repos_url":"https://api.github.com/users/nickva/repos","events_url":"https://api.github.com/users/nickva/events{/privacy}","received_events_url":"https://api.github.com/users/nickva/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-04-04T23:07:07Z","updated_at":"2017-04-06T16:55:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/couchdb/pulls/470","html_url":"https://github.com/apache/couchdb/pull/470","diff_url":"https://github.com/apache/couchdb/pull/470.diff","patch_url":"https://github.com/apache/couchdb/pull/470.patch"},"body":"Introduce Scheduling CouchDB Replicator\r\n\r\nJira: https://issues.apache.org/jira/browse/COUCHDB-3324\r\n\r\nThe core of the new replicator is a scheduler. It which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: `_scheduler/jobs` and `_scheduler/docs`.\r\n\r\nReplication protocol hasn't change so it is possible to replicate between CouchDB 1.x, 2.x, PouchDB, and other implementations of CouchDB replication protocol.\r\n\r\n## Scheduler\r\n\r\nScheduler allows running a large number of replication jobs. Tested with up to 100k replication jobs in 3 node cluster. Replication jobs are run in a fair, round-robin fashion. Scheduler behavior can be configured by these configuration options in `[replicator]` sections:\r\n\r\n   * `max_jobs` : Number of actively running replications. Making this too high\r\n     could cause performance issues. Making it too low could mean replications jobs\r\n     might not have enough time to make progress before getting unscheduled again.\r\n     This parameter can be adjusted at runtime and will take effect during next\r\n     rescheduling cycle.\r\n\r\n   * `interval` : Scheduling interval in milliseconds. During each reschedule\r\n       cycle scheduler might start or stop up to \"max_churn\" number of jobs.\r\n\r\n   * `max_churn` : Maximum number of replications to start and stop during\r\n      rescheduling. This parameter along with \"interval\" defines the rate of job\r\n      replacement. During startup, however a much larger number of jobs could be\r\n      started (up to max_jobs) in a short period of time.\r\n\r\n## _scheduler/{jobs,docs} API\r\n\r\nThere is an improved replication state querying API, with a focus on ease of use and performance. The new API avoids having to update the replication document with transient state updates. In production that can lead to conflicts and performance issues. The two new APIs are:\r\n\r\n   * `_scheduler/jobs` : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to `_active_tasks` but focuses only on replications. Jobs which have completed or which were never created because of malformed replication document will not be shown here as they are not managed by the scheduler. `_replicate` replications, started form `_replicate` endpoint not from a document in a `_replicator` db, will also show up here.\r\n\r\n   *  `_scheduler/docs` :  This endpoint is an improvement on having to go back and re-read replication document to query their state. It represents the state of all the replications started from documents in `_replicator` dbs. Unlike `_scheduler/jobs` it will also show jobs which have failed or completed (that is, which are not managed by the scheduler anymore). \r\n\r\n## Compatibility Mode\r\n\r\nUnderstandably some customers are using the document-based API to query replication states (`triggered`, `error`, `completed` etc). To ease the upgrade path, there is a compatibility configuration setting:\r\n\r\n```\r\n[replicator]\r\nupdate_docs = false | true\r\n```\r\nIt defaults to `false` but when set to `true` it will continue updating replication document with the state of the replication jobs.\r\n\r\n\r\n## Other Improvements\r\n\r\n * Network resource usage and performance was improved by implementing a common connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.\r\n\r\n * Improved rate limiting handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.\r\n\r\n * Improved performance by avoiding repeatedly retrying failing replication jobs. Instead use exponential backoff. In a large multi-user cluster, quite a few replication jobs are invalid, are crashing or failing (for various reasons such as inability to checkpoint to source, mismatched credentials, missing databases). Penalizing failing replication will free up system resources for more useful work. \r\n\r\n * Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some scenarios (for example, after a sustained DNS failure which eventually recovers), replications reach their retry limit and cease to work. Previously it required operator intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job. So it should recover automatically.\r\n\r\n * Better handling of filtered replications: Failing to fetch filters could block couch replicator manager, lead to message queue backups and memory exhaustion. Also, when replication filter code changes update replication accordingly (replication job ID should change in that case). This PR fixes both of those issues. Filters and replication ID calculation are managed in separate worker threads (in couch replicator doc processor). Also, when filter code changes on the source, replication jobs will restart and re-calculate their new ID automatically.\r\n\r\n## Related PR:\r\n\r\nDocumentation PR:  https://github.com/apache/couchdb-documentation/pull/123\r\n\r\n## Tests\r\n\r\n* EUnit test coverage. Some modules such as multidb_changes, have close to 100% coverage. Some have less. All previous replication tests have been updated to work with the new scheduling replicator. \r\n\r\n* Except for one modification, existing Javascript tests pass.\r\n\r\n* Additional integration tests. To validate, test and benchmark some edge cases, an additional toolkit was created: https://github.com/cloudant-labs/couchdyno/blob/master/README_tests.md which allows testing scenarios where nodes fail, creation of large number of replication jobs, manipulation of cluster configurations, and setting up long running (soak) tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/couchdb/issues/comments/292237042","html_url":"https://github.com/apache/couchdb/pull/470#issuecomment-292237042","issue_url":"https://api.github.com/repos/apache/couchdb/issues/470","id":292237042,"user":{"login":"nickva","id":211822,"avatar_url":"https://avatars0.githubusercontent.com/u/211822?v=3","gravatar_id":"","url":"https://api.github.com/users/nickva","html_url":"https://github.com/nickva","followers_url":"https://api.github.com/users/nickva/followers","following_url":"https://api.github.com/users/nickva/following{/other_user}","gists_url":"https://api.github.com/users/nickva/gists{/gist_id}","starred_url":"https://api.github.com/users/nickva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickva/subscriptions","organizations_url":"https://api.github.com/users/nickva/orgs","repos_url":"https://api.github.com/users/nickva/repos","events_url":"https://api.github.com/users/nickva/events{/privacy}","received_events_url":"https://api.github.com/users/nickva/received_events","type":"User","site_admin":false},"created_at":"2017-04-06T16:55:48Z","updated_at":"2017-04-06T16:55:48Z","body":"Some additional information from testing how new scheduling replicator handles a large amount of jobs.\r\n\r\nThis is a graph of adding 1M replication jobs to a cluster of 3 nodes (testy011). Replication jobs were added using dyno/rep tool : https://github.com/cloudant/dyno/blob/master/README_rep.md\r\n\r\n```\r\nrep.replicate_all_and_compare(n=1000, num=1, normal=False) \r\n```\r\n\r\nThat creates a connected cluster of replications for for n=1000, it would create 1000*1000=1M replication jobs.\r\n\r\n![db_changes](https://cloud.githubusercontent.com/assets/211822/24765972/c1fb151c-1ac7-11e7-8fd5-45e44d1c4bc2.png)\r\n\r\n`couch_replicator.docs.db_changes` tracks the total number of changes seen by the replicator. In this case the replicator has seen the 1M replication documents created above.\r\n\r\n![jobs_total](https://cloud.githubusercontent.com/assets/211822/24766116/231f61e0-1ac8-11e7-9e32-8c95490df4f6.png)\r\n\r\n`couch_replicator.jobs.total` tracks the number of replication jobs managed by the scheduler. Out of the 1M jobs, each node in the cluster picked up about 330K each.\r\n\r\n![scheduler](https://cloud.githubusercontent.com/assets/211822/24766148/3daacf4a-1ac8-11e7-9577-60d502321804.png)\r\n\r\n`couch_replicator.jobs.starts` counts the number of jobs which have been started by the scheduler. In this case a time derivative was applied. So it shows how the scheduler interval works. Every time it runs it start another 20 jobs (this is specified by the max_churn parameter).\r\n\r\n\r\n"}}
{"actor":{"display_login":"nickva"}}