{"action":"created","issue":{"url":"https://api.github.com/repos/github/gh-ost/issues/475","repository_url":"https://api.github.com/repos/github/gh-ost","labels_url":"https://api.github.com/repos/github/gh-ost/issues/475/labels{/name}","comments_url":"https://api.github.com/repos/github/gh-ost/issues/475/comments","events_url":"https://api.github.com/repos/github/gh-ost/issues/475/events","html_url":"https://github.com/github/gh-ost/issues/475","id":252904067,"number":475,"title":"Row counts not matching at cutover and then cutover continually fails and loops","user":{"login":"ekirkland","id":7440894,"avatar_url":"https://avatars1.githubusercontent.com/u/7440894?v=4","gravatar_id":"","url":"https://api.github.com/users/ekirkland","html_url":"https://github.com/ekirkland","followers_url":"https://api.github.com/users/ekirkland/followers","following_url":"https://api.github.com/users/ekirkland/following{/other_user}","gists_url":"https://api.github.com/users/ekirkland/gists{/gist_id}","starred_url":"https://api.github.com/users/ekirkland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ekirkland/subscriptions","organizations_url":"https://api.github.com/users/ekirkland/orgs","repos_url":"https://api.github.com/users/ekirkland/repos","events_url":"https://api.github.com/users/ekirkland/events{/privacy}","received_events_url":"https://api.github.com/users/ekirkland/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2017-08-25T13:39:56Z","updated_at":"2017-08-30T13:07:45Z","closed_at":null,"author_association":"NONE","body":"We've been trying to migrate a fairly large table, it has about 100 million rows and its about 150 gb.  We've done two attempts at adding the columns and in both cases when we got to the postponed cutover I manually ran count(1) queries on the original table and the _gho table and the row counts mismatched the first time by 300k rows and the second time by 1M rows.  We're able to stop writes to this table and so we still attempted the cutover process to see if it would catch things up, but it seemed to loop with the following error:\r\n\r\n```\r\n2017-08-25 13:26:47 INFO Looking for magic cut-over table\r\n2017-08-25 13:26:47 INFO Will now proceed to drop magic table and unlock tables\r\n2017-08-25 13:26:47 INFO Dropping magic cut-over table\r\n2017-08-25 13:26:47 INFO Releasing lock from `db`.`table_a`, `db`.`_table_a_del`\r\n2017-08-25 13:26:47 INFO Tables unlocked\r\n2017-08-25 13:26:48 INFO Grabbing voluntary lock: gh-ost.125413.lock\r\n2017-08-25 13:26:48 INFO Setting LOCK timeout as 6 seconds\r\n2017-08-25 13:26:48 INFO Looking for magic cut-over table\r\n2017-08-25 13:26:48 INFO Creating magic cut-over table `db`.`_table_a_del`\r\n2017-08-25 13:26:48 INFO Magic cut-over table created\r\n2017-08-25 13:26:48 INFO Locking `db`.`table_a`, `db`.`_table_a_del`\r\n2017-08-25 13:26:48 INFO Tables locked\r\n2017-08-25 13:26:48 INFO Session locking original & magic tables is 125413\r\n2017-08-25 13:26:48 INFO Writing changelog state: AllEventsUpToLockProcessed:1503667608367587880\r\n2017-08-25 13:26:48 INFO Waiting for events up to lock\r\n2017-08-25 13:26:51 ERROR Timeout while waiting for events up to lock\r\n2017-08-25 13:26:51 ERROR 2017-08-25 13:26:51 ERROR Timeout while waiting for events up to lock\r\n2017-08-25 13:26:51 INFO Looking for magic cut-over table\r\n2017-08-25 13:26:51 INFO Will now proceed to drop magic table and unlock tables\r\n2017-08-25 13:26:51 INFO Dropping magic cut-over table\r\n2017-08-25 13:26:51 INFO Releasing lock from `db`.`table_a`, `db`.`_table_a_del`\r\n2017-08-25 13:26:51 INFO Tables unlocked\r\n```\r\nIt repeats this output over and over.  The first time around it eventually failed due to the default 60 retries, but the second time we increased that so it retried longer.  It wasn't making any progress on catching up the missing rows.\r\n\r\nWe're running MySQL 5.7.17 in Amazon RDS.\r\n\r\nI currently still have the migration attempt running with the cutover postponed, so if there are things to try out or information gathered that can be done without restarting it that would be great, since it took 88 hours to get to this point.\r\n\r\nWe run ghost with the following params, and have had alot of success until this table.\r\n\r\n```\r\n    --host=${slave_host} \\\r\n   --table=\"${table_name}\" \\\r\n    --alter=\"${alter_statement}\" \\\r\n    --assume-master-host=${master_host} \\\r\n    --database=\"${database_name}\" \\\r\n    --throttle-control-replicas=${slave_host} \\\r\n    --assume-rbr \\\r\n    --initially-drop-ghost-table \\\r\n    --initially-drop-socket-file \\\r\n    --max-load=Threads_running=15 \\\r\n    --max-lag-millis=1000 \\\r\n    --chunk-size=1000 \\\r\n    --nice-ratio=2.5 \\\r\n    --verbose \\\r\n    --cut-over=default \\\r\n    --cut-over-lock-timeout-seconds=3 \\\r\n    --default-retries=1000000 \\\r\n    --panic-flag-file=/tmp/${script_identifier}-gh-ost.panic.flag \\\r\n    --serve-socket-file=/tmp/${script_identifier}-gh-ost.sock \\\r\n    --throttle-flag-file=/tmp/${script_identifier}-gh-ost.throttle \\\r\n    --postpone-cut-over-flag-file=/tmp/${script_identifier}-gh-ost.postpone \\\r\n    --execute\r\n```"},"comment":{"url":"https://api.github.com/repos/github/gh-ost/issues/comments/325984421","html_url":"https://github.com/github/gh-ost/issues/475#issuecomment-325984421","issue_url":"https://api.github.com/repos/github/gh-ost/issues/475","id":325984421,"user":{"login":"ekirkland","id":7440894,"avatar_url":"https://avatars1.githubusercontent.com/u/7440894?v=4","gravatar_id":"","url":"https://api.github.com/users/ekirkland","html_url":"https://github.com/ekirkland","followers_url":"https://api.github.com/users/ekirkland/followers","following_url":"https://api.github.com/users/ekirkland/following{/other_user}","gists_url":"https://api.github.com/users/ekirkland/gists{/gist_id}","starred_url":"https://api.github.com/users/ekirkland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ekirkland/subscriptions","organizations_url":"https://api.github.com/users/ekirkland/orgs","repos_url":"https://api.github.com/users/ekirkland/repos","events_url":"https://api.github.com/users/ekirkland/events{/privacy}","received_events_url":"https://api.github.com/users/ekirkland/received_events","type":"User","site_admin":false},"created_at":"2017-08-30T13:07:45Z","updated_at":"2017-08-30T13:07:45Z","author_association":"NONE","body":"We've been using ghost for a while now and have had a lot of success with it, even on large tables.  One characteristic of this database that differs from our other database that we modify more frequently is that it does some work in large transactions and we do see much higher replica lag (it can spike to over 1 minute under normal conditions).  Could that be causing issues?"}}
{"actor":{"display_login":"ekirkland"}}