{"action":"created","issue":{"url":"https://api.github.com/repos/fstpackage/fst/issues/77","repository_url":"https://api.github.com/repos/fstpackage/fst","labels_url":"https://api.github.com/repos/fstpackage/fst/issues/77/labels{/name}","comments_url":"https://api.github.com/repos/fstpackage/fst/issues/77/comments","events_url":"https://api.github.com/repos/fstpackage/fst/issues/77/events","html_url":"https://github.com/fstpackage/fst/issues/77","id":249306119,"number":77,"title":"Option strings as factors=FALSE","user":{"login":"pmakai","id":22682349,"avatar_url":"https://avatars2.githubusercontent.com/u/22682349?v=4","gravatar_id":"","url":"https://api.github.com/users/pmakai","html_url":"https://github.com/pmakai","followers_url":"https://api.github.com/users/pmakai/followers","following_url":"https://api.github.com/users/pmakai/following{/other_user}","gists_url":"https://api.github.com/users/pmakai/gists{/gist_id}","starred_url":"https://api.github.com/users/pmakai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmakai/subscriptions","organizations_url":"https://api.github.com/users/pmakai/orgs","repos_url":"https://api.github.com/users/pmakai/repos","events_url":"https://api.github.com/users/pmakai/events{/privacy}","received_events_url":"https://api.github.com/users/pmakai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-10T10:44:32Z","updated_at":"2017-08-10T11:49:53Z","closed_at":null,"body":"The current version reads in factors to R, even if I select only 1000 rows. That means that ALL factor levels are read in, and this extremely memory inefficient. \r\n\r\nAn option strings as factors might be benefitial. "},"comment":{"url":"https://api.github.com/repos/fstpackage/fst/issues/comments/321529533","html_url":"https://github.com/fstpackage/fst/issues/77#issuecomment-321529533","issue_url":"https://api.github.com/repos/fstpackage/fst/issues/77","id":321529533,"user":{"login":"MarcusKlik","id":24658401,"avatar_url":"https://avatars3.githubusercontent.com/u/24658401?v=4","gravatar_id":"","url":"https://api.github.com/users/MarcusKlik","html_url":"https://github.com/MarcusKlik","followers_url":"https://api.github.com/users/MarcusKlik/followers","following_url":"https://api.github.com/users/MarcusKlik/following{/other_user}","gists_url":"https://api.github.com/users/MarcusKlik/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcusKlik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcusKlik/subscriptions","organizations_url":"https://api.github.com/users/MarcusKlik/orgs","repos_url":"https://api.github.com/users/MarcusKlik/repos","events_url":"https://api.github.com/users/MarcusKlik/events{/privacy}","received_events_url":"https://api.github.com/users/MarcusKlik/received_events","type":"User","site_admin":false},"created_at":"2017-08-10T11:49:53Z","updated_at":"2017-08-10T11:49:53Z","body":"Hi @pmakai, thanks for reporting your issue. Indeed, when a factor column has e.g. 1e6 levels and only 1e3 rows are actually read, all factor levels have to be imported. So in effect, a large read is done for a small subset of rows.\r\n\r\nFor those cases, adding a parameter `stringsAsFactors` (or `factorsAsStrings` really) to `fstread` will be useful.\r\n\r\nThe downside is that for a column that has been serialized as a `factor`, `fst` writes the levels and the actual values to disk in separate blocks. That means that given a subset of a stored table, `fst` has to 'search' the stored levels for the levels that correspond to the actual values. We can make that search memory efficient, but for randomly ordered data, it still requires reading most of the level data, so it will not be very performant. In the example above, there is a large probability that all 1e6 levels have to be read to lookup the 1e3 values selected.\r\n\r\nSo, the user has a few options:\r\n\r\n* Write the table without `factor` columns: reading small subsets will be fast but the `fst` file may be (much) larger for `character` columns with large elements.\r\n* Write the table with `factor` columns and read a subset with option _factorsAsStrings == TRUE_. The `fst` file will probably be small but due to the look-ups required, reading will be slow for small subsets (but it will be memory efficient).\r\n* Read subsets of the table that are equal in size or larger than the number of levels. That way, reading will probably be as fast as for subsets of non-factor columns.\r\n\r\nI have to think this on through!"}}
{"actor":{"display_login":"MarcusKlik"}}