{"action":"created","issue":{"url":"https://api.github.com/repos/cf-tm-bot/services/issues/224","repository_url":"https://api.github.com/repos/cf-tm-bot/services","labels_url":"https://api.github.com/repos/cf-tm-bot/services/issues/224/labels{/name}","comments_url":"https://api.github.com/repos/cf-tm-bot/services/issues/224/comments","events_url":"https://api.github.com/repos/cf-tm-bot/services/issues/224/events","html_url":"https://github.com/cf-tm-bot/services/issues/224","id":199327255,"number":224,"title":"Investigate implications of failed replication when using TOI - Story Id: 137200329","user":{"login":"cf-tm-bot","id":20690070,"avatar_url":"https://avatars.githubusercontent.com/u/20690070?v=3","gravatar_id":"","url":"https://api.github.com/users/cf-tm-bot","html_url":"https://github.com/cf-tm-bot","followers_url":"https://api.github.com/users/cf-tm-bot/followers","following_url":"https://api.github.com/users/cf-tm-bot/following{/other_user}","gists_url":"https://api.github.com/users/cf-tm-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/cf-tm-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cf-tm-bot/subscriptions","organizations_url":"https://api.github.com/users/cf-tm-bot/orgs","repos_url":"https://api.github.com/users/cf-tm-bot/repos","events_url":"https://api.github.com/users/cf-tm-bot/events{/privacy}","received_events_url":"https://api.github.com/users/cf-tm-bot/received_events","type":"User","site_admin":false},"labels":[{"id":446225076,"url":"https://api.github.com/repos/cf-tm-bot/services/labels/feature","name":"feature","color":"ededed","default":false},{"id":446963063,"url":"https://api.github.com/repos/cf-tm-bot/services/labels/started","name":"started","color":"c8cbd0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-01-07T00:57:21Z","updated_at":"2017-01-16T18:57:44Z","closed_at":null,"body":"In Galera, when running under TOI, it's possible for a single node to fail applying a DDL due to insufficient disk space, RAM, etc. In TOI, this is a silent failure.\n\nWe believe that Galera's method of handling this is to allow the node to continue running until it attempts to apply a writeset that conflicts with the old schema. (In this case, it should crash and SST.)\n\nWe further believe, then, that this means that if a failover occurs such that reads are happening on this node, or even writes that are still compatible with the old schema, it's possible that apps will get old or inconsistent data.\n\nHere are four tests.\n\nAssume a column that is consistent across all three nodes:\n\n| COL1 | COL 2 |\n|  --- | --- |\n| A | DZ |\n| B | MN |\n\nAnd it is transformed on two of three nodes (you can probably use RSU to simulate a DDL fail):\n\n| COL1 | COL2 | COL3 |\n| --- | --- | --- |\n| A | DZ | 1 |\n| B | MN | 2 |\n\n... but does not occur on one of the three nodes, does this cause Galera any issue? \n\nScenario 1: \nOur theory is that if a select query runs against the node that did not complete the DDL, then it will happily return old data (for instance if an app selects COL1,COL2 but not COL3). Further, we believe that the app will receive an error if it does query COL3, this is a fail, since that's incorrect data.\n\nScenario 2: Same as above, but now an additional row is being written to the cluster:\n\n| COL1 | COL2 | COL3 |\n| --- | --- | --- |\n| C | RDZ | 3 |\n\nWhen you write this new row to the cluster (using a master that has the new schema), does the node that did not complete the DDL crash? Or worse, does the master crash due to failed certification?\n\nScenario 3: Vice versa, if the node that does not complete the DDL receives the insert, what happens? Does it crash? Does it tell the app invalid schema? If the latter, this is a fail, because the app is actually issuing valid queries.\n\nScenario 4: If you're renaming a column (COL2 -> COLNEW), if the node that does not complete the DDL becomes master, and an app writes to that node using the **old** column name, what happens to the two other nodes that have complete the DDL?\n\n\n---\n\nMirrors: [story 137200329](https://www.pivotaltracker.com/story/show/137200329) submitted on Jan 6, 2017 UTC\n- **Requester**: Marco Nicosia\n- **Owners**: Difan Zhao, Laine Campbell\n- **Estimate**: 3.0"},"comment":{"url":"https://api.github.com/repos/cf-tm-bot/services/issues/comments/272938821","html_url":"https://github.com/cf-tm-bot/services/issues/224#issuecomment-272938821","issue_url":"https://api.github.com/repos/cf-tm-bot/services/issues/224","id":272938821,"user":{"login":"cf-tm-bot","id":20690070,"avatar_url":"https://avatars.githubusercontent.com/u/20690070?v=3","gravatar_id":"","url":"https://api.github.com/users/cf-tm-bot","html_url":"https://github.com/cf-tm-bot","followers_url":"https://api.github.com/users/cf-tm-bot/followers","following_url":"https://api.github.com/users/cf-tm-bot/following{/other_user}","gists_url":"https://api.github.com/users/cf-tm-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/cf-tm-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cf-tm-bot/subscriptions","organizations_url":"https://api.github.com/users/cf-tm-bot/orgs","repos_url":"https://api.github.com/users/cf-tm-bot/repos","events_url":"https://api.github.com/users/cf-tm-bot/events{/privacy}","received_events_url":"https://api.github.com/users/cf-tm-bot/received_events","type":"User","site_admin":false},"created_at":"2017-01-16T18:57:44Z","updated_at":"2017-01-16T18:57:44Z","body":"Difan Zhao [commented](https://www.pivotaltracker.com/n/projects/969486/stories/137200329/comments/160493719) on Jan 16, 2017, 6.39 PM UTC:\n\nSo next, we want to create an experiment where the insert statement can't be silently ignored. To do this, was have to add a column in the middle of the table rather than at the end. Additionally, we make sure that col 2 is int one one node and varchar on the other. This creates an error that should cause node 2 to extract from the cluster in a state that requires SST.\n\nMariaDB [test]> SET wsrep_osu_method='RSU';\nQuery OK, 0 rows affected (0.00 sec)\n\nMariaDB [test]> ALTER TABLE tab DROP COLUMN num;\nQuery OK, 0 rows affected (0.03 sec)\nRecords: 0  Duplicates: 0  Warnings: 0\n\nMariaDB [test]> show create table tab;\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Table | Create Table                                                                                                                                                                              |\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| tab   | CREATE TABLE `tab` (\n  `id` int(11) NOT NULL,\n  `str` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,\n  PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n1 row in set (0.00 sec)\n\nMariaDB [test]> ALTER TABLE tab ADD COLUMN num int DEFAULT NULL AFTER id;\nQuery OK, 0 rows affected (0.04 sec)\nRecords: 0  Duplicates: 0  Warnings: 0\n\nMariaDB [test]> show create table tab;\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Table | Create Table                                                                                                                                                                                                            |\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| tab   | CREATE TABLE `tab` (\n  `id` int(11) NOT NULL,\n  `num` int(11) DEFAULT NULL,\n  `str` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,\n  PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |\n+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n1 row in set (0.00 sec)\n\nMariaDB [test]>  INSERT INTO tab (id, num, str) VALUES (10, 1, 'str2');\nQuery OK, 1 row affected (0.00 sec)\n\n<!--\nComment Id: 160493719\n-->"}}
{"actor":{"display_login":"cf-tm-bot"}}