{"action":"created","issue":{"url":"https://api.github.com/repos/couchbase/sync_gateway/issues/2607","repository_url":"https://api.github.com/repos/couchbase/sync_gateway","labels_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/2607/labels{/name}","comments_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/2607/comments","events_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/2607/events","html_url":"https://github.com/couchbase/sync_gateway/issues/2607","id":232556390,"number":2607,"title":"Replicating between databases causes SG to go OOM","user":{"login":"b-smets","id":1068509,"avatar_url":"https://avatars1.githubusercontent.com/u/1068509?v=3","gravatar_id":"","url":"https://api.github.com/users/b-smets","html_url":"https://github.com/b-smets","followers_url":"https://api.github.com/users/b-smets/followers","following_url":"https://api.github.com/users/b-smets/following{/other_user}","gists_url":"https://api.github.com/users/b-smets/gists{/gist_id}","starred_url":"https://api.github.com/users/b-smets/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-smets/subscriptions","organizations_url":"https://api.github.com/users/b-smets/orgs","repos_url":"https://api.github.com/users/b-smets/repos","events_url":"https://api.github.com/users/b-smets/events{/privacy}","received_events_url":"https://api.github.com/users/b-smets/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"tleyden","id":296876,"avatar_url":"https://avatars2.githubusercontent.com/u/296876?v=3","gravatar_id":"","url":"https://api.github.com/users/tleyden","html_url":"https://github.com/tleyden","followers_url":"https://api.github.com/users/tleyden/followers","following_url":"https://api.github.com/users/tleyden/following{/other_user}","gists_url":"https://api.github.com/users/tleyden/gists{/gist_id}","starred_url":"https://api.github.com/users/tleyden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tleyden/subscriptions","organizations_url":"https://api.github.com/users/tleyden/orgs","repos_url":"https://api.github.com/users/tleyden/repos","events_url":"https://api.github.com/users/tleyden/events{/privacy}","received_events_url":"https://api.github.com/users/tleyden/received_events","type":"User","site_admin":false},"assignees":[{"login":"tleyden","id":296876,"avatar_url":"https://avatars2.githubusercontent.com/u/296876?v=3","gravatar_id":"","url":"https://api.github.com/users/tleyden","html_url":"https://github.com/tleyden","followers_url":"https://api.github.com/users/tleyden/followers","following_url":"https://api.github.com/users/tleyden/following{/other_user}","gists_url":"https://api.github.com/users/tleyden/gists{/gist_id}","starred_url":"https://api.github.com/users/tleyden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tleyden/subscriptions","organizations_url":"https://api.github.com/users/tleyden/orgs","repos_url":"https://api.github.com/users/tleyden/repos","events_url":"https://api.github.com/users/tleyden/events{/privacy}","received_events_url":"https://api.github.com/users/tleyden/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/couchbase/sync_gateway/milestones/24","html_url":"https://github.com/couchbase/sync_gateway/milestone/24","labels_url":"https://api.github.com/repos/couchbase/sync_gateway/milestones/24/labels","id":2420585,"number":24,"title":"1.5.0","description":"","creator":{"login":"djpongh","id":21346920,"avatar_url":"https://avatars0.githubusercontent.com/u/21346920?v=3","gravatar_id":"","url":"https://api.github.com/users/djpongh","html_url":"https://github.com/djpongh","followers_url":"https://api.github.com/users/djpongh/followers","following_url":"https://api.github.com/users/djpongh/following{/other_user}","gists_url":"https://api.github.com/users/djpongh/gists{/gist_id}","starred_url":"https://api.github.com/users/djpongh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djpongh/subscriptions","organizations_url":"https://api.github.com/users/djpongh/orgs","repos_url":"https://api.github.com/users/djpongh/repos","events_url":"https://api.github.com/users/djpongh/events{/privacy}","received_events_url":"https://api.github.com/users/djpongh/received_events","type":"User","site_admin":false},"open_issues":56,"closed_issues":27,"state":"open","created_at":"2017-03-29T22:31:04Z","updated_at":"2017-06-16T18:57:26Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2017-05-31T12:57:53Z","updated_at":"2017-06-19T12:34:58Z","closed_at":"2017-06-16T17:33:28Z","body":"### Sync Gateway version\r\nWe are currently running a 1.5 master build: \r\nhttp://latestbuilds.hq.couchbase.com/couchbase-sync-gateway/1.5.0/1.5.0-381/\r\nWe do this in order to work around https://github.com/couchbase/sync_gateway/issues/2587  \r\nAccessing / results in:\r\n```{\"couchdb\":\"Welcome\",\"vendor\":{\"name\":\"Couchbase Sync Gateway\",\"version\":1.5},\"version\":\"Couchbase Sync Gateway/1.5.0(381;6a1598b)\"}```\r\n\r\n### Operating system\r\nUbuntu 14.04\r\n\r\n### Config file\r\nRegular config file with 8 databases and the following memory tweaks:\r\n```\r\n              \"feed_type\": \"TAP\",\r\n              \"rev_cache_size\": 10,\r\n              \"cache\": {\r\n                \"channel_cache_max_length\": 10,\r\n                \"channel_cache_min_length\": 10\r\n\t        }\r\n```\r\nNote: We added these only after the first OOMs were encountered.\r\n\r\n### Log output\r\nhttps://gist.github.com/b-smets/50f07bfee28349a4cf3b113eafa7d6e6\r\n\r\nIn order to work around https://github.com/couchbase/sync_gateway/issues/1648 and shrink our bucket size, we attempted moving all documents of our big bucket into a new bucket, hoping the obsolete attachments would stay behind. The big bucket contains about 1.3M items (according to our Couchbase Web interface) of which are 2507 json documents with 1-5 attachments (<2MB) each, and the rest is obsolete garbage. The target bucket is empty.\r\n\r\nWe tried running the following command:\r\n```\r\ncurl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ \"source\": \"<sourcebucket>\",  \"target\": \"<targetbucket>\" }' 'http://localhost:4985/_replicate'\r\n```\r\nThis command ran for approx. 5 minutes before resulting in replication terminated response from the server. However, the replication was still going strong and items were being transferred between the buckets. After the replication finished, we checked the number of documents in the target bucket but this contained only about 350 documents so it didn't process everything.\r\nAny attempt to run the replication a second time now results in an out of memory error (see gist) after a couple of minutes. No new documents are being transferred.\r\n\r\nOther things we tried:\r\n- Use \"async\": true => Same OOM errors\r\n- Use \"continuous\": true => This results in nothing happening. No documents being transfered but no oom.\r\n- Replicate into the target database on another SG instance => Same OOM\r\n"},"comment":{"url":"https://api.github.com/repos/couchbase/sync_gateway/issues/comments/309426384","html_url":"https://github.com/couchbase/sync_gateway/issues/2607#issuecomment-309426384","issue_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/2607","id":309426384,"user":{"login":"ArihantRk","id":13716346,"avatar_url":"https://avatars3.githubusercontent.com/u/13716346?v=3","gravatar_id":"","url":"https://api.github.com/users/ArihantRk","html_url":"https://github.com/ArihantRk","followers_url":"https://api.github.com/users/ArihantRk/followers","following_url":"https://api.github.com/users/ArihantRk/following{/other_user}","gists_url":"https://api.github.com/users/ArihantRk/gists{/gist_id}","starred_url":"https://api.github.com/users/ArihantRk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ArihantRk/subscriptions","organizations_url":"https://api.github.com/users/ArihantRk/orgs","repos_url":"https://api.github.com/users/ArihantRk/repos","events_url":"https://api.github.com/users/ArihantRk/events{/privacy}","received_events_url":"https://api.github.com/users/ArihantRk/received_events","type":"User","site_admin":false},"created_at":"2017-06-19T12:34:58Z","updated_at":"2017-06-19T12:34:58Z","body":"@tleyden \r\nWe tested http://latestbuilds.hq.couchbase.com/couchbase-sync-gateway/1.5.0/1.5.0-407/ patch and not able to reproduce Out of Memory issue in our 72 hrs traffic run. \r\nBelow are the Observation is during traffic run\r\n1. Document conflicts (revision limit is not working and older revisions (more than revision limit) are getting added) are happening due to SG Replication lagging and conflicts intern causes document size increase but did not observe out of memory issue in this patch. \r\n2. Sync Gateway memory increased from 2 GB to  5 GB over 72 hrs and we suspect it might be due to document size grow. \r\nIs this expected behavior? whether memory will keeps on increase if so what is the upper limit?\r\n"}}
{"actor":{"display_login":"ArihantRk"}}