{"action":"opened","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/24620","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/24620/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/24620/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/24620/events","html_url":"https://github.com/elastic/elasticsearch/issues/24620","id":228000011,"number":24620,"title":"Wrong fieldLength in _explain","user":{"login":"Corei13","id":4954380,"avatar_url":"https://avatars1.githubusercontent.com/u/4954380?v=3","gravatar_id":"","url":"https://api.github.com/users/Corei13","html_url":"https://github.com/Corei13","followers_url":"https://api.github.com/users/Corei13/followers","following_url":"https://api.github.com/users/Corei13/following{/other_user}","gists_url":"https://api.github.com/users/Corei13/gists{/gist_id}","starred_url":"https://api.github.com/users/Corei13/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Corei13/subscriptions","organizations_url":"https://api.github.com/users/Corei13/orgs","repos_url":"https://api.github.com/users/Corei13/repos","events_url":"https://api.github.com/users/Corei13/events{/privacy}","received_events_url":"https://api.github.com/users/Corei13/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-05-11T13:55:36Z","updated_at":"2017-05-11T13:55:36Z","closed_at":null,"body":"I have a document with title **Women's Funny T-Shirt (#wine (Hashtag Tee Shirt) Drinking) Ladies Shirt**\r\n\r\nThe analyze api shows there are 11 terms.\r\n```\r\nGET /products/_analyze\r\n{\r\n  \"field\": \"custom.title\",\r\n  \"text\": \"Women's Funny T-Shirt (#wine (Hashtag Tee Shirt) Drinking) Ladies Shirt\"\r\n}\r\n```\r\n```json\r\n{\r\n  \"tokens\": [\r\n    {\r\n      \"token\": \"women's\",\r\n      \"start_offset\": 0,\r\n      \"end_offset\": 7,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 0\r\n    },\r\n    {\r\n      \"token\": \"funny\",\r\n      \"start_offset\": 8,\r\n      \"end_offset\": 13,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 1\r\n    },\r\n    {\r\n      \"token\": \"t\",\r\n      \"start_offset\": 14,\r\n      \"end_offset\": 15,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 2\r\n    },\r\n    {\r\n      \"token\": \"shirt\",\r\n      \"start_offset\": 16,\r\n      \"end_offset\": 21,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 3\r\n    },\r\n    {\r\n      \"token\": \"wine\",\r\n      \"start_offset\": 24,\r\n      \"end_offset\": 28,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 4\r\n    },\r\n    {\r\n      \"token\": \"hashtag\",\r\n      \"start_offset\": 30,\r\n      \"end_offset\": 37,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 5\r\n    },\r\n    {\r\n      \"token\": \"tee\",\r\n      \"start_offset\": 38,\r\n      \"end_offset\": 41,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 6\r\n    },\r\n    {\r\n      \"token\": \"shirt\",\r\n      \"start_offset\": 42,\r\n      \"end_offset\": 47,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 7\r\n    },\r\n    {\r\n      \"token\": \"drinking\",\r\n      \"start_offset\": 49,\r\n      \"end_offset\": 57,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 8\r\n    },\r\n    {\r\n      \"token\": \"ladies\",\r\n      \"start_offset\": 59,\r\n      \"end_offset\": 65,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 9\r\n    },\r\n    {\r\n      \"token\": \"shirt\",\r\n      \"start_offset\": 66,\r\n      \"end_offset\": 71,\r\n      \"type\": \"<ALPHANUM>\",\r\n      \"position\": 10\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nBut the explain api tells that the `title` field has fieldLength of 16\r\n```\r\nGET /products/custom/1971/_explain\r\n{\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"should\": {\r\n          \"match\": {\r\n            \"title\": {\r\n              \"query\": \"women t shirt\"\r\n            }\r\n          }\r\n        }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\n```json\r\n{\r\n  \"_index\": \"products\",\r\n  \"_type\": \"custom\",\r\n  \"_id\": \"1971\",\r\n  \"matched\": true,\r\n  \"explanation\": {\r\n    \"value\": 5.2311754,\r\n    \"description\": \"sum of:\",\r\n    \"details\": [\r\n      {\r\n        \"value\": 5.2311754,\r\n        \"description\": \"sum of:\",\r\n        \"details\": [\r\n          {\r\n            \"value\": 2.0648851,\r\n            \"description\": \"weight(title:t in 278491) [PerFieldSimilarity], result of:\",\r\n            \"details\": [\r\n              {\r\n                \"value\": 2.0648851,\r\n                \"description\": \"score(doc=278491,freq=1.0 = termFreq=1.0\\n), product of:\",\r\n                \"details\": [\r\n                  {\r\n                    \"value\": 2.2896154,\r\n                    \"description\": \"idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:\",\r\n                    \"details\": [\r\n                      {\r\n                        \"value\": 446517,\r\n                        \"description\": \"docFreq\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 4407636,\r\n                        \"description\": \"docCount\",\r\n                        \"details\": []\r\n                      }\r\n                    ]\r\n                  },\r\n                  {\r\n                    \"value\": 0.901848,\r\n                    \"description\": \"tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:\",\r\n                    \"details\": [\r\n                      {\r\n                        \"value\": 1,\r\n                        \"description\": \"termFreq=1.0\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 1.2,\r\n                        \"description\": \"parameter k1\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 0.75,\r\n                        \"description\": \"parameter b\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 12.637836,\r\n                        \"description\": \"avgFieldLength\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 16,\r\n                        \"description\": \"fieldLength\",\r\n                        \"details\": []\r\n                      }\r\n                    ]\r\n                  }\r\n                ]\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"value\": 3.1662903,\r\n            \"description\": \"weight(title:shirt in 278491) [PerFieldSimilarity], result of:\",\r\n            \"details\": [\r\n              {\r\n                \"value\": 3.1662903,\r\n                \"description\": \"score(doc=278491,freq=3.0 = termFreq=3.0\\n), product of:\",\r\n                \"details\": [\r\n                  {\r\n                    \"value\": 2.1297789,\r\n                    \"description\": \"idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:\",\r\n                    \"details\": [\r\n                      {\r\n                        \"value\": 523907,\r\n                        \"description\": \"docFreq\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 4407636,\r\n                        \"description\": \"docCount\",\r\n                        \"details\": []\r\n                      }\r\n                    ]\r\n                  },\r\n                  {\r\n                    \"value\": 1.4866756,\r\n                    \"description\": \"tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:\",\r\n                    \"details\": [\r\n                      {\r\n                        \"value\": 3,\r\n                        \"description\": \"termFreq=3.0\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 1.2,\r\n                        \"description\": \"parameter k1\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 0.75,\r\n                        \"description\": \"parameter b\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 12.637836,\r\n                        \"description\": \"avgFieldLength\",\r\n                        \"details\": []\r\n                      },\r\n                      {\r\n                        \"value\": 16,\r\n                        \"description\": \"fieldLength\",\r\n                        \"details\": []\r\n                      }\r\n                    ]\r\n                  }\r\n                ]\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      },\r\n      {\r\n        \"value\": 0,\r\n        \"description\": \"match on required clause, product of:\",\r\n        \"details\": [\r\n          {\r\n            \"value\": 0,\r\n            \"description\": \"# clause\",\r\n            \"details\": []\r\n          },\r\n          {\r\n            \"value\": 1,\r\n            \"description\": \"_type:amazon, product of:\",\r\n            \"details\": [\r\n              {\r\n                \"value\": 1,\r\n                \"description\": \"boost\",\r\n                \"details\": []\r\n              },\r\n              {\r\n                \"value\": 1,\r\n                \"description\": \"queryNorm\",\r\n                \"details\": []\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}\r\n```"}}
{"actor":{"display_login":"Corei13"}}