{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/2443","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/2443/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/2443/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/2443/events","html_url":"https://github.com/scylladb/scylla/issues/2443","id":233423960,"number":2443,"title":"Migrating data from Cassandra 2.1 to Scylla using sstableloader fails","user":{"login":"tzach","id":170200,"avatar_url":"https://avatars2.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"labels":[{"id":160570956,"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true}],"state":"closed","locked":false,"assignee":{"login":"elcallio","id":7113590,"avatar_url":"https://avatars2.githubusercontent.com/u/7113590?v=3","gravatar_id":"","url":"https://api.github.com/users/elcallio","html_url":"https://github.com/elcallio","followers_url":"https://api.github.com/users/elcallio/followers","following_url":"https://api.github.com/users/elcallio/following{/other_user}","gists_url":"https://api.github.com/users/elcallio/gists{/gist_id}","starred_url":"https://api.github.com/users/elcallio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elcallio/subscriptions","organizations_url":"https://api.github.com/users/elcallio/orgs","repos_url":"https://api.github.com/users/elcallio/repos","events_url":"https://api.github.com/users/elcallio/events{/privacy}","received_events_url":"https://api.github.com/users/elcallio/received_events","type":"User","site_admin":false},"assignees":[{"login":"elcallio","id":7113590,"avatar_url":"https://avatars2.githubusercontent.com/u/7113590?v=3","gravatar_id":"","url":"https://api.github.com/users/elcallio","html_url":"https://github.com/elcallio","followers_url":"https://api.github.com/users/elcallio/followers","following_url":"https://api.github.com/users/elcallio/following{/other_user}","gists_url":"https://api.github.com/users/elcallio/gists{/gist_id}","starred_url":"https://api.github.com/users/elcallio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elcallio/subscriptions","organizations_url":"https://api.github.com/users/elcallio/orgs","repos_url":"https://api.github.com/users/elcallio/repos","events_url":"https://api.github.com/users/elcallio/events{/privacy}","received_events_url":"https://api.github.com/users/elcallio/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/20","html_url":"https://github.com/scylladb/scylla/milestone/20","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/20/labels","id":2124118,"number":20,"title":"1.7","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars2.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":43,"state":"open","created_at":"2016-11-09T09:28:02Z","updated_at":"2017-06-08T18:19:31Z","due_on":"2017-02-15T08:00:00Z","closed_at":null},"comments":9,"created_at":"2017-06-04T10:05:11Z","updated_at":"2017-06-08T18:19:31Z","closed_at":"2017-06-08T18:19:31Z","body":"*Installation details*\r\nScylla version (or git commit hash): 1.7\r\nCluster size: 1\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): Docker\r\n\r\nUploading an SSTable generated on Cassandra 2.1,17 to Scylla 1.7 using sstableloader not all data correctly migrate.\r\n\r\n### Reproduce\r\n\r\n#### Cassandra 2.17  \r\n```\r\nsudo docker run --name some-cassandra-2.1 -d cassandra:2.1.17\r\nsudo docker exec -it some-cassandra-2.1 cqlsh\r\ncqlsh>\r\nCREATE KEYSPACE mykeyspace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };\r\nUSE mykeyspace;\r\n\r\nCREATE TABLE all_types (aascii ascii, abigint bigint, ablob blob, aboolean boolean, adecimal decimal, adouble double, afloat float, ainet inet, aint int, atext text, atimestamp timestamp, atimeuuid timeuuid, auuid uuid, avarchar varchar, avarint varint, alist list<int>, amap map<int,int>, aset set<int>, PRIMARY KEY (aascii, abigint)\r\n);\r\n\r\nINSERT INTO all_types (aascii, abigint, ablob, aboolean, adecimal, adouble, afloat, ainet, aint, atext, atimestamp, atimeuuid, auuid, avarchar, avarint, alist, amap, aset) VALUES ('tzach', 1999, bigintAsBlob(3), true, 10 , 10.10, 11.11 , '204.202.130.223', 17, 'text' ,'2016-08-30 07:01:00', maxTimeuuid('2013-01-01 00:05+0000'), 123e4567-e89b-12d3-a456-426655440000, 'tzachvarchar', 17, [1,2,3], {1 : 2}, {1,2,3,4});\r\nINSERT INTO all_types (aascii, abigint, ablob, aboolean, adecimal, adouble, afloat, ainet, aint, atext, atimestamp, atimeuuid, auuid, avarchar, avarint, alist, amap, aset) VALUES ('tzach', 2000, bigintAsBlob(3), true, 10 , 10.10, 11.11 , '204.202.130.223', 17, 'text' ,'2016-08-30 07:01:00', maxTimeuuid('2013-01-01 00:05+0000'), 123e4567-e89b-12d3-a456-426655440000, 'tzachvarchar', 17, [1,2,3], {1 : 2}, {1,2,3,4});\r\nINSERT INTO all_types (aascii, abigint, ablob, aboolean, adecimal, adouble, afloat, ainet, aint, atext, atimestamp, atimeuuid, auuid, avarchar, avarint, alist, amap, aset) VALUES ('livyatan', 2001, bigintAsBlob(3), true, 10 , 10.10, 11.11 , '204.202.130.223', 17, 'text' ,'2016-08-30 07:01:00', maxTimeuuid('2013-01-01 00:05+0000'), 123e4567-e89b-12d3-a456-426655440000, 'tzachvarchar', 17, [1,2,3], {1 : 2}, {1,2,3,4});\r\n\r\ncqlsh> select * from mykeyspace.all_types ;\r\n\r\n aascii   | abigint | ablob              | aboolean | adecimal | adouble | afloat | ainet           | aint | alist     | amap   | aset         | atext | atimestamp               | atimeuuid                            | auuid                                | avarchar     | avarint\r\n----------+---------+--------------------+----------+----------+---------+--------+-----------------+------+-----------+--------+--------------+-------+--------------------------+--------------------------------------+--------------------------------------+--------------+---------\r\n    tzach |    1999 | 0x0000000000000003 |     True |       10 |    10.1 |  11.11 | 204.202.130.223 |   17 | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  text | 2016-08-30 07:01:00+0000 | e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f | 123e4567-e89b-12d3-a456-426655440000 | tzachvarchar |      17\r\n    tzach |    2000 | 0x0000000000000003 |     True |       10 |    10.1 |  11.11 | 204.202.130.223 |   17 | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  text | 2016-08-30 07:01:00+0000 | e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f | 123e4567-e89b-12d3-a456-426655440000 | tzachvarchar |      17\r\n livyatan |    2001 | 0x0000000000000003 |     True |       10 |    10.1 |  11.11 | 204.202.130.223 |   17 | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  text | 2016-08-30 07:01:00+0000 | e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f | 123e4567-e89b-12d3-a456-426655440000 | tzachvarchar |      17\r\n\r\n(3 rows)\r\n\r\nsudo docker exec -it some-cassandra-2.1 nodetool flush\r\nsudo docker cp some-cassandra-2.1:/var/lib/cassandra/data/mykeyspace/ mykeyspace\r\n```\r\n\r\n#### Scylla\r\n\r\n```\r\nsudo docker run --name some-scylla -d scylladb/scylla:1.7.0 --memory 1G --smp 1\r\nsudo docker exec -it some-scylla cqlsh\r\nsudo docker exec -it some-scylla cqlsh\r\ncqlsh>\r\nCREATE KEYSPACE mykeyspace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };\r\nUSE mykeyspace;\r\n\r\nCREATE TABLE all_types (aascii ascii, abigint bigint, ablob blob, aboolean boolean, adecimal decimal, adouble double, afloat float, ainet inet, aint int, atext text, atimestamp timestamp, atimeuuid timeuuid, auuid uuid, avarchar varchar, avarint varint, alist list<int>, amap map<int,int>, aset set<int>, PRIMARY KEY (aascii, abigint)\r\n);\r\n\r\nsudo docker cp mykeyspace some-scylla:/\r\n~$ sudo docker exec -it some-scylla sstableloader -v -d 172.17.0.3 /mykeyspace/all_types-a70a2510490a11e7bc32091830ac5256\r\nCQL: 'INSERT INTO mykeyspace.all_types (ablob,aboolean,adecimal,adouble,afloat,ainet,aint,atext,atimestamp,atimeuuid,auuid,avarchar,avarint,aascii) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?) USING TIMESTAMP ?;' [java.nio.HeapByteBuffer[pos=0 lim=8 cap=8], true, 10, 10.1, 11.11, /204.202.130.223, 17, text, Tue Aug 30 07:01:00 UTC 2016, e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f, 123e4567-e89b-12d3-a456-426655440000, tzachvarchar, 17, tzach, 1496569584310912]\r\nCQL: 'UPDATE mykeyspace.all_types USING TIMESTAMP ? SET alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, amap[?] = ?, aset = aset + ?, aset = aset + ?, aset = aset + ?, aset = aset + ? WHERE aascii = ? AND abigint = ?;' [1496569584310912, ac1557f0-490a-11e7-bc32-091830ac5256, 1, ac1557f1-490a-11e7-bc32-091830ac5256, 2, ac1557f2-490a-11e7-bc32-091830ac5256, 3, 1, 2, [1], [2], [3], [4], tzach, 1999]\r\nCQL: 'INSERT INTO mykeyspace.all_types (ablob,aboolean,adecimal,adouble,afloat,ainet,aint,atext,atimestamp,atimeuuid,auuid,avarchar,avarint,aascii) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?) USING TIMESTAMP ?;' [java.nio.HeapByteBuffer[pos=0 lim=8 cap=8], true, 10, 10.1, 11.11, /204.202.130.223, 17, text, Tue Aug 30 07:01:00 UTC 2016, e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f, 123e4567-e89b-12d3-a456-426655440000, tzachvarchar, 17, tzach, 1496569584386539]\r\ncom.datastax.driver.core.exceptions.InvalidQueryException: Missing mandatory PRIMARY KEY part abigint\r\nCQL: 'UPDATE mykeyspace.all_types USING TIMESTAMP ? SET alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, amap[?] = ?, aset = aset + ?, aset = aset + ?, aset = aset + ?, aset = aset + ? WHERE aascii = ? AND abigint = ?;' [1496569584386539, ac1976a0-490a-11e7-bc32-091830ac5256, 1, ac1976a1-490a-11e7-bc32-091830ac5256, 2, ac1976a2-490a-11e7-bc32-091830ac5256, 3, 1, 2, [1], [2], [3], [4], tzach, 2000]\r\ncom.datastax.driver.core.exceptions.InvalidQueryException: Missing mandatory PRIMARY KEY part abigint\r\nCQL: 'INSERT INTO mykeyspace.all_types (ablob,aboolean,adecimal,adouble,afloat,ainet,aint,atext,atimestamp,atimeuuid,auuid,avarchar,avarint,aascii) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?) USING TIMESTAMP ?;' [java.nio.HeapByteBuffer[pos=0 lim=8 cap=8], true, 10, 10.1, 11.11, /204.202.130.223, 17, text, Tue Aug 30 07:01:00 UTC 2016, e23f450f-53a6-11e2-7f7f-7f7f7f7f7f7f, 123e4567-e89b-12d3-a456-426655440000, tzachvarchar, 17, livyatan, 1496569584403084]\r\nCQL: 'UPDATE mykeyspace.all_types USING TIMESTAMP ? SET alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, alist[SCYLLA_TIMEUUID_LIST_INDEX(?)] = ?, amap[?] = ?, aset = aset + ?, aset = aset + ?, aset = aset + ?, aset = aset + ? WHERE aascii = ? AND abigint = ?;' [1496569584403084, ac1b9980-490a-11e7-bc32-091830ac5256, 1, ac1b9981-490a-11e7-bc32-091830ac5256, 2, ac1b9982-490a-11e7-bc32-091830ac5256, 3, 1, 2, [1], [2], [3], [4], livyatan, 2001]\r\ncom.datastax.driver.core.exceptions.InvalidQueryException: Missing mandatory PRIMARY KEY part abigint\r\n~$ sudo docker exec -it some-scylla cqlsh\r\nConnected to Test Cluster at 172.17.0.3:9042.\r\n[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]\r\nUse HELP for help.\r\ncqlsh> use mykeyspace ;\r\ncqlsh:mykeyspace> select * from all_types ;\r\n\r\n aascii   | abigint | ablob | aboolean | adecimal | adouble | afloat | ainet | aint | alist     | amap   | aset         | atext | atimestamp | atimeuuid | auuid | avarchar | avarint\r\n----------+---------+-------+----------+----------+---------+--------+-------+------+-----------+--------+--------------+-------+------------+-----------+-------+----------+---------\r\n    tzach |    1999 |  null |     null |     null |    null |   null |  null | null | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  null |       null |      null |  null |     null |    null\r\n    tzach |    2000 |  null |     null |     null |    null |   null |  null | null | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  null |       null |      null |  null |     null |    null\r\n livyatan |    2001 |  null |     null |     null |    null |   null |  null | null | [1, 2, 3] | {1: 2} | {1, 2, 3, 4} |  null |       null |      null |  null |     null |    null\r\n\r\n(3 rows)\r\n```\r\n\r\nNote the errors in the sstableloader and the fact many of the values in Scylla are null.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/307185801","html_url":"https://github.com/scylladb/scylla/issues/2443#issuecomment-307185801","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/2443","id":307185801,"user":{"login":"tzach","id":170200,"avatar_url":"https://avatars2.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"created_at":"2017-06-08T18:19:31Z","updated_at":"2017-06-08T18:19:31Z","body":"Tested with 1.7.1 and, as promised, problem is solved."}}
{"actor":{"display_login":"tzach"}}