{"action":"created","issue":{"url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/264","repository_url":"https://api.github.com/repos/Stratio/cassandra-lucene-index","labels_url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/264/labels{/name}","comments_url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/264/comments","events_url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/264/events","html_url":"https://github.com/Stratio/cassandra-lucene-index/issues/264","id":203811193,"number":264,"title":"Index a document only if necessary","user":{"login":"fdimuccio","id":3441610,"avatar_url":"https://avatars2.githubusercontent.com/u/3441610?v=3","gravatar_id":"","url":"https://api.github.com/users/fdimuccio","html_url":"https://github.com/fdimuccio","followers_url":"https://api.github.com/users/fdimuccio/followers","following_url":"https://api.github.com/users/fdimuccio/following{/other_user}","gists_url":"https://api.github.com/users/fdimuccio/gists{/gist_id}","starred_url":"https://api.github.com/users/fdimuccio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fdimuccio/subscriptions","organizations_url":"https://api.github.com/users/fdimuccio/orgs","repos_url":"https://api.github.com/users/fdimuccio/repos","events_url":"https://api.github.com/users/fdimuccio/events{/privacy}","received_events_url":"https://api.github.com/users/fdimuccio/received_events","type":"User","site_admin":false},"labels":[{"id":352316429,"url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/labels/performance","name":"performance","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-01-28T14:54:17Z","updated_at":"2017-03-13T17:04:36Z","closed_at":null,"body":"Hello,\r\n\r\nthank you for this great project. We are using Cassandra 3.7 alongside cassandra-lucene-index 3.7.5 to index few teras of data. By tracing the indexing process after an UPDATE I noticed that a row is indexed or deleted even if no relevant fields for the lucene index are affected.\r\n\r\nIn our use case the schema is something like that:\r\n\r\n```\r\nCREATE TABLE ks.tbl (\r\n    pk bigint PRIMARY KEY,\r\n    field1 frozen<udt1>,\r\n    field2 frozen<udt2>,\r\n    field3 frozen<udt3>,\r\n    field4 frozen<udt4>,\r\n    field5 frozen<udt5>\r\n) WITH bloom_filter_fp_chance = 0.01\r\n    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}\r\n    AND comment = ''\r\n    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}\r\n    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}\r\n    AND crc_check_chance = 1.0\r\n    AND dclocal_read_repair_chance = 0.1\r\n    AND default_time_to_live = 0\r\n    AND gc_grace_seconds = 864000\r\n    AND max_index_interval = 2048\r\n    AND memtable_flush_period_in_ms = 0\r\n    AND min_index_interval = 128\r\n    AND read_repair_chance = 0.0\r\n    AND speculative_retry = '99PERCENTILE';\r\n```\r\n```\r\nCREATE CUSTOM INDEX idx on ks.tbl()\r\nUSING 'com.stratio.cassandra.lucene.Index'\r\nWITH OPTIONS = {\r\n  'refresh_seconds': '60',\r\n  'schema': '{\r\n    fields: {\r\n        \"field3.subfield1\": {type: \"text\"},\r\n        \"field3.subfield2\": {type: \"text\"},\r\n        \"field3.subfield3\": {type: \"text\"},\r\n        \"field3.subfield4\": {type: \"text\"},\r\n        \"field3.subfield5\": {type: \"text\"},\r\n        \"field3.subfield6\": {type: \"string\", case_sensitive: \"false\"}\r\n    }\r\n  }'\r\n};\r\n```\r\n\r\nOnly field3 has data to be indexed, and the dataset is quite sparse, only a small percentage of rows has that field set. I was wondering if it is possible to skip the index operation in case the update does not touch any of the fields that must be indexed: org.apache.cassandra.index.Indexer.updateRow(Row oldRowData, Row newRowData) gives the opportunity to check which fields have been updated. By skipping the operation IndexWriterSkinny does not even need to do a read before write. Probably even the initial compaction (Secondary index creation) could benefits from this optimization.\r\n\r\nIs it correct or I'm missing some aspects of the indexing process?"},"comment":{"url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/comments/286173572","html_url":"https://github.com/Stratio/cassandra-lucene-index/issues/264#issuecomment-286173572","issue_url":"https://api.github.com/repos/Stratio/cassandra-lucene-index/issues/264","id":286173572,"user":{"login":"ealonsodb","id":11426807,"avatar_url":"https://avatars0.githubusercontent.com/u/11426807?v=3","gravatar_id":"","url":"https://api.github.com/users/ealonsodb","html_url":"https://github.com/ealonsodb","followers_url":"https://api.github.com/users/ealonsodb/followers","following_url":"https://api.github.com/users/ealonsodb/following{/other_user}","gists_url":"https://api.github.com/users/ealonsodb/gists{/gist_id}","starred_url":"https://api.github.com/users/ealonsodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ealonsodb/subscriptions","organizations_url":"https://api.github.com/users/ealonsodb/orgs","repos_url":"https://api.github.com/users/ealonsodb/repos","events_url":"https://api.github.com/users/ealonsodb/events{/privacy}","received_events_url":"https://api.github.com/users/ealonsodb/received_events","type":"User","site_admin":false},"created_at":"2017-03-13T17:04:36Z","updated_at":"2017-03-13T17:04:36Z","body":"Hi @fdimuccio: \r\nYour last changes are great. I had merged the PR. \r\n\r\nThank you for your effort "}}
{"actor":{"display_login":"ealonsodb"}}