{"action":"opened","issue":{"url":"https://api.github.com/repos/searchkit/searchkit/issues/386","repository_url":"https://api.github.com/repos/searchkit/searchkit","labels_url":"https://api.github.com/repos/searchkit/searchkit/issues/386/labels{/name}","comments_url":"https://api.github.com/repos/searchkit/searchkit/issues/386/comments","events_url":"https://api.github.com/repos/searchkit/searchkit/issues/386/events","html_url":"https://github.com/searchkit/searchkit/issues/386","id":205636685,"number":386,"title":"Improving aggregations and filtering performance","user":{"login":"sadovnychyi","id":193864,"avatar_url":"https://avatars.githubusercontent.com/u/193864?v=3","gravatar_id":"","url":"https://api.github.com/users/sadovnychyi","html_url":"https://github.com/sadovnychyi","followers_url":"https://api.github.com/users/sadovnychyi/followers","following_url":"https://api.github.com/users/sadovnychyi/following{/other_user}","gists_url":"https://api.github.com/users/sadovnychyi/gists{/gist_id}","starred_url":"https://api.github.com/users/sadovnychyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sadovnychyi/subscriptions","organizations_url":"https://api.github.com/users/sadovnychyi/orgs","repos_url":"https://api.github.com/users/sadovnychyi/repos","events_url":"https://api.github.com/users/sadovnychyi/events{/privacy}","received_events_url":"https://api.github.com/users/sadovnychyi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-06T16:27:17Z","updated_at":"2017-02-06T16:27:17Z","closed_at":null,"body":"I got pretty bad performance when I turned searchkit on production index with ~1M documents and ~20 tags per document. But here's good news – I managed to improve it by up to 11 times (from tens of seconds to under a second). I do understand that you can just make your cluster bigger but it seems like waste of resources. So the bottleneck is how most of aggregations are made, e.g. by `RefinementListFilter`:\r\n\r\n```json\r\n{\r\n  \"aggs\": {\r\n    \"actors.raw6\": {\r\n      \"filter\": {},\r\n      \"aggs\": {\r\n        \"actors.raw\": {\r\n          \"terms\": {\r\n            \"field\": \"actors.raw\",\r\n            \"size\": 10\r\n          }\r\n        },\r\n        \"actors.raw_count\": {\r\n          \"cardinality\": {\r\n            \"field\": \"actors.raw\"\r\n          }\r\n        }\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nWhen you apply filters – aggregations are filtered individually, instead of filtering all documents to make search request much lighter, we also have another aggregation here to count number of documents to provide pagination – which is very useful, but we don't really need another aggregation for it.\r\n\r\nImproved query with applied filter:\r\n```json\r\n{\r\n  \"aggs\": {\r\n    \"actors.raw6\": {\r\n      \"terms\": {\r\n        \"field\": \"actors.raw\",\r\n        \"size\": 10\r\n      }\r\n    }\r\n  },\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"filter\": [{\r\n        \"term\": {\r\n          \"actors.raw\": \"Naveen Andrews\"\r\n        }\r\n      }]\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nThis provides filtering for both aggregations and documents (before we have `post_filter` applied to filter out the documents).\r\nThe workaround for pagination is to use `sum_other_doc_count` value provided for each aggregation. It's going to be zero if there's no more possible aggregations, and going to be some other number if there is. It's likely that we won't be able to correctly display `View all`, but we will be able to display `View more` and `View less` when it's zero, and it's more than enough.\r\n\r\nI don't know about other use cases for those additional aggregations, but I think it's should be possible to avoid them, at least with dedicated component. I'm providing the code that I use to build a custom component below. I did similar RangeAccessor modifications as well. Note that all this applied only for \"AND\" queries, I have no idea about \"OR\".\r\n\r\nWould you consider improving this either by improving existing components or by adding additional ones e.g. `HighPerformanceRefinementListFilter` or something?\r\n\r\n```javascript\r\nimport {map, omitBy, isUndefined, isEmpty, isArray, filter} from 'lodash'\r\nimport {FacetAccessor, TermsBucket, TermQuery, RefinementListFilter} from 'searchkit'\r\n\r\n\r\nfunction BoolFilter(val) {\r\n  if (isArray(val)) {\r\n    // Remove empty filters\r\n    val = filter(val, f => !isEmpty(f))\r\n    if (val.length === 0) {\r\n      return {}\r\n    }\r\n  }\r\n  return {\r\n    bool: {\r\n      'filter': val\r\n    }\r\n  }\r\n}\r\n\r\n\r\nclass CustomFacetAccessor extends FacetAccessor {\r\n  getRawBuckets() {\r\n    return this.getAggregations([\r\n      this.uuid,\r\n      this.fieldContext.getAggregationPath(),\r\n      'buckets'], [])\r\n  }\r\n\r\n  getDocCount() {\r\n    // not used at all afaik\r\n    return this.getCount()\r\n  }\r\n\r\n  getCount() {\r\n    return this.getAggregations([\r\n      this.uuid,\r\n      this.fieldContext.getAggregationPath(),\r\n      'sum_other_doc_count'], 0)\r\n  }\r\n\r\n  buildSharedQuery(query) {\r\n    var filters = this.state.getValue()\r\n    var filterTerms = map(filters, (filter)=> {\r\n      return this.fieldContext.wrapFilter(TermQuery(this.key, filter))\r\n    })\r\n    var selectedFilters = map(filters, (filter) => {\r\n      return {\r\n        name: this.options.title || this.translate(this.key),\r\n        value: this.translate(filter),\r\n        id: this.options.id,\r\n        remove: () => this.state = this.state.remove(filter)\r\n      }\r\n    })\r\n    if (filterTerms.length > 0) {\r\n      query = query.addQuery(BoolFilter(filterTerms))\r\n        .addSelectedFilters(selectedFilters)\r\n    }\r\n    return query\r\n  }\r\n\r\n  buildOwnQuery(query) {\r\n    if (!this.loadAggregations){\r\n      return query\r\n    } else {\r\n      return query\r\n        .setAggs(TermsBucket(\r\n          this.uuid,\r\n          this.key,\r\n          omitBy({\r\n            size: this.size,\r\n            order: this.getOrder(),\r\n            include: this.options.include,\r\n            exclude: this.options.exclude,\r\n            min_doc_count: this.options.min_doc_count\r\n          }, isUndefined)))\r\n    }\r\n  }\r\n}\r\n\r\n\r\nexport class CustomRefinementListFilter extends RefinementListFilter {\r\n  accessor: CustomFacetAccessor\r\n\r\n  defineAccessor() {\r\n    return new CustomFacetAccessor(\r\n      this.props.field, this.getAccessorOptions())\r\n  }\r\n}\r\n```"}}
{"actor":{"display_login":"sadovnychyi"}}