{"action":"created","issue":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/4012","repository_url":"https://api.github.com/repos/arangodb/arangodb","labels_url":"https://api.github.com/repos/arangodb/arangodb/issues/4012/labels{/name}","comments_url":"https://api.github.com/repos/arangodb/arangodb/issues/4012/comments","events_url":"https://api.github.com/repos/arangodb/arangodb/issues/4012/events","html_url":"https://github.com/arangodb/arangodb/issues/4012","id":281396327,"number":4012,"title":"RocksDB Async Replication breaking","user":{"login":"Dan-Wolfy","id":7839746,"avatar_url":"https://avatars2.githubusercontent.com/u/7839746?v=4","gravatar_id":"","url":"https://api.github.com/users/Dan-Wolfy","html_url":"https://github.com/Dan-Wolfy","followers_url":"https://api.github.com/users/Dan-Wolfy/followers","following_url":"https://api.github.com/users/Dan-Wolfy/following{/other_user}","gists_url":"https://api.github.com/users/Dan-Wolfy/gists{/gist_id}","starred_url":"https://api.github.com/users/Dan-Wolfy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dan-Wolfy/subscriptions","organizations_url":"https://api.github.com/users/Dan-Wolfy/orgs","repos_url":"https://api.github.com/users/Dan-Wolfy/repos","events_url":"https://api.github.com/users/Dan-Wolfy/events{/privacy}","received_events_url":"https://api.github.com/users/Dan-Wolfy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-12-12T14:06:41Z","updated_at":"2017-12-13T12:31:14Z","closed_at":null,"author_association":"NONE","body":"## my environment running ArangoDB\r\nI'm using the latest ArangoDB of the respective release series:\r\n- [ ] 3.2.8\r\n\r\nMode:\r\n- [ ] Single ( Master ) with 2 Async ( Slaves ) Replication\r\n\r\nStorage-Engine:\r\n- [ ] rocksdb\r\n\r\nOn this operating system:\r\n  - [ ] Azure\r\n  - [ ] docker - official docker library\r\n\r\n### this is an installation-related issue:\r\nDescribe which steps you carried out, what you expected to happen and what actually happened.\r\n\r\nHi,\r\n\r\nRaising this as an Installation issue as think its my knowledge of ArangoDB; RocksDB in Async replication. I've tried several options but it randomly breaks replication even when idling.\r\n\r\nBefore we where using MMFiles and had no problem with replication but we require non deadlock on reads for performance increase so switched to RocksDB and been plagued with problems.\r\n\r\nMaster started with blank DB then db and collections created.\r\nSlaves both started up and start replicating from Master okay then insert X seconds one or both break on numerous rebuilds.\r\n\r\nAlso checking the journal on the Master server its the same file and hasn't rotated out as well.\r\n\r\nOn the Master we have tried;- \r\nsudo bash -c \"echo madvise >/sys/kernel/mm/transparent_hugepage/enabled\"\r\nsudo bash -c \"echo madvise >/sys/kernel/mm/transparent_hugepage/defrag\"\r\nsudo bash -c \"sysctl -w 'vm.max_map_count=512000'\".\r\nexport GLIBCXX_FORCE_NEW=1\r\n\r\n--rocksdb.write-buffer-size default and 512 MiB\r\n--rocksdb.max-total-wal-size default and 1024 MiB\r\n--rocksdb.wal-file-timeout default and 900\r\n\r\nCurrent setup is;-\r\n--rocksdb.wal-file-timeout default 900\r\nsudo bash -c \"echo madvise >/sys/kernel/mm/transparent_hugepage/enabled\"\r\nsudo bash -c \"echo madvise >/sys/kernel/mm/transparent_hugepage/defrag\"\r\nsudo bash -c \"sysctl -w 'vm.max_map_count=512000'\".\r\nexport GLIBCXX_FORCE_NEW=1\r\n\r\nOn the Slaves;-\r\n    require('@arangodb/replication').setupReplication({\r\n        endpoint: 'tcp://{{ arangodb.replication.master }}',\r\n        username: '{{ arangodb.admins.replication.user }}',\r\n        password: '{{ arangodb.admins.replication.pwd }}',\r\n        verbose: true,\r\n        includeSystem: false,\r\n        incremental: true,\r\n        maxConnectRetries: 3,\r\n        requireFromPresent: true,\r\n        autoResync: true,\r\n        autoResyncRetires: 3,\r\n    });\r\n\r\nSlave-1;-\r\n...\r\n2017-12-12T14:02:23Z [1] INFO {replication} fetching master log from tick 19522, first regular tick 19522, barrier: 0, open transactions: 0\r\n{ \r\n  \"state\" : { \r\n    \"running\" : true, \r\n    \"lastAppliedContinuousTick\" : null, \r\n    \"lastProcessedContinuousTick\" : \"21258\", \r\n    \"lastAvailableContinuousTick\" : \"21258\", \r\n    \"safeResumeTick\" : \"19528\", \r\n    \"progress\" : { \r\n      \"time\" : \"2017-12-12T13:49:46Z\", \r\n      \"message\" : \"fetching master log from tick 19522, first regular tick 19522, barrier: 0, open transactions: 0\", \r\n      \"failedConnects\" : 0 \r\n    }, \r\n    \"totalRequests\" : 122, \r\n    \"totalFailedConnects\" : 0, \r\n    \"totalEvents\" : 0, \r\n    \"totalOperationsExcluded\" : 0, \r\n    \"lastError\" : { \r\n      \"errorNum\" : 0 \r\n    }, \r\n    \"time\" : \"2017-12-12T13:49:48Z\" \r\n  }, \r\n  \"server\" : { \r\n    \"version\" : \"3.2.8\", \r\n    \"serverId\" : \"206855583103809\" \r\n  }, \r\n  \"endpoint\" : \"tcp://arangodb-master-0\", \r\n  \"database\" : \"$database\" \r\n}\r\n\r\nSlave 0;\r\n{ \r\n  \"state\" : { \r\n    \"running\" : false, \r\n    \"lastAppliedContinuousTick\" : null, \r\n    \"lastProcessedContinuousTick\" : null, \r\n    \"lastAvailableContinuousTick\" : \"19522\", \r\n    \"safeResumeTick\" : null, \r\n    \"progress\" : { \r\n      \"time\" : \"2017-12-12T13:45:07Z\", \r\n      \"message\" : \"applier shut down\", \r\n      \"failedConnects\" : 0 \r\n    }, \r\n    \"totalRequests\" : 0, \r\n    \"totalFailedConnects\" : 0, \r\n    \"totalEvents\" : 0, \r\n    \"totalOperationsExcluded\" : 0, \r\n    \"lastError\" : { \r\n      \"errorNum\" : 1414, \r\n      \"time\" : \"2017-12-12T13:45:07Z\", \r\n      \"errorMessage\" : \"required follow tick value '19522' is not present (anymore?) on master at tcp://arangodb-master-0. Last tick available on master is 19522. It may be required to do a full resync and increase the number of historic logfiles on the master.\" \r\n    }, \r\n    \"time\" : \"2017-12-12T13:49:48Z\" \r\n  }, \r\n  \"server\" : { \r\n    \"version\" : \"3.2.8\", \r\n    \"serverId\" : \"129997755601041\" \r\n  }, \r\n  \"endpoint\" : \"tcp://arangodb-master-0\", \r\n  \"database\" : \"$database\" \r\n}\r\n2017-12-12T13:45:07Z [1] INFO {replication} starting with from tick 19522, fetch tick 19522, open transactions: 0\r\n2017-12-12T13:45:07Z [1] INFO {replication} fetching master log from tick 19522, first regular tick 19522, barrier: 0, open transactions: 0\r\n2017-12-12T13:45:07Z [1] ERROR {replication} replication applier error for database '$database': required follow tick value '19522' is not present (anymore?) on master at tcp://arangodb-master-0. Last tick available on master is 19522. It may be required to do a full resync and increase the number of historic logfiles on the master.\r\n2017-12-12T13:45:07Z [1] INFO {replication} stopped replication applier for database '$database'\r\n2017-12-12T13:45:07Z [1] WARNING {replication} replication applier stopped for database '$database' because required tick is not present on master\r\n2017-12-12T13:45:07Z [1] WARNING {replication} aborting automatic resynchronization for database '$database' after 3 retries\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/comments/351377679","html_url":"https://github.com/arangodb/arangodb/issues/4012#issuecomment-351377679","issue_url":"https://api.github.com/repos/arangodb/arangodb/issues/4012","id":351377679,"user":{"login":"Dan-Wolfy","id":7839746,"avatar_url":"https://avatars2.githubusercontent.com/u/7839746?v=4","gravatar_id":"","url":"https://api.github.com/users/Dan-Wolfy","html_url":"https://github.com/Dan-Wolfy","followers_url":"https://api.github.com/users/Dan-Wolfy/followers","following_url":"https://api.github.com/users/Dan-Wolfy/following{/other_user}","gists_url":"https://api.github.com/users/Dan-Wolfy/gists{/gist_id}","starred_url":"https://api.github.com/users/Dan-Wolfy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dan-Wolfy/subscriptions","organizations_url":"https://api.github.com/users/Dan-Wolfy/orgs","repos_url":"https://api.github.com/users/Dan-Wolfy/repos","events_url":"https://api.github.com/users/Dan-Wolfy/events{/privacy}","received_events_url":"https://api.github.com/users/Dan-Wolfy/received_events","type":"User","site_admin":false},"created_at":"2017-12-13T12:31:14Z","updated_at":"2017-12-13T12:31:14Z","author_association":"NONE","body":"I have not mentioned it here before, it a issue we have seen before on our side ( one a month if that), however since the move from MM to RocksDB its happening a lot. \r\n\r\nI have seen other tickets mentioning OOM with V8, I have pushed our management team for sign off for consultancy and support with Arangodb where we can send our source code for review. However its done this when idling too .\r\n\r\nLastest OOM issue also had also had;-\r\nINFO started email-process-transaction\r\n<--- Last few GCs --->\r\n\r\n[1:0x7f653918aa80] 368 ms: Scavenge 3.3 (7.0) > 3.2 (7.5) MB, 1.0 / 0.0 allocation failure\r\n[1:0x7f653918aa80] 445 ms: Scavenge 4.2 (7.5) > 3.8 (10.0) MB, 0.5 / 0.0 allocation failure\r\n\r\n\r\n<--- JS stacktrace --->\r\nCannot get stack trace in GC.\r\n\r\n\r\nThen the docker container restarted, however sometime it just sits there and doesn't exit"}}
{"actor":{"display_login":"Dan-Wolfy"}}