{"action":"created","issue":{"url":"https://api.github.com/repos/Creepsky/creepMiner/issues/86","repository_url":"https://api.github.com/repos/Creepsky/creepMiner","labels_url":"https://api.github.com/repos/Creepsky/creepMiner/issues/86/labels{/name}","comments_url":"https://api.github.com/repos/Creepsky/creepMiner/issues/86/comments","events_url":"https://api.github.com/repos/Creepsky/creepMiner/issues/86/events","html_url":"https://github.com/Creepsky/creepMiner/issues/86","id":211805013,"number":86,"title":"Multiple plot files handling","user":{"login":"Axadiw","id":1938540,"avatar_url":"https://avatars3.githubusercontent.com/u/1938540?v=3","gravatar_id":"","url":"https://api.github.com/users/Axadiw","html_url":"https://github.com/Axadiw","followers_url":"https://api.github.com/users/Axadiw/followers","following_url":"https://api.github.com/users/Axadiw/following{/other_user}","gists_url":"https://api.github.com/users/Axadiw/gists{/gist_id}","starred_url":"https://api.github.com/users/Axadiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Axadiw/subscriptions","organizations_url":"https://api.github.com/users/Axadiw/orgs","repos_url":"https://api.github.com/users/Axadiw/repos","events_url":"https://api.github.com/users/Axadiw/events{/privacy}","received_events_url":"https://api.github.com/users/Axadiw/received_events","type":"User","site_admin":false},"labels":[{"id":547626046,"url":"https://api.github.com/repos/Creepsky/creepMiner/labels/change","name":"change","color":"fbca04","default":false},{"id":553203677,"url":"https://api.github.com/repos/Creepsky/creepMiner/labels/done","name":"done","color":"c2e0c6","default":false},{"id":471005996,"url":"https://api.github.com/repos/Creepsky/creepMiner/labels/feature","name":"feature","color":"0e8a16","default":false},{"id":548157747,"url":"https://api.github.com/repos/Creepsky/creepMiner/labels/in%20progress","name":"in progress","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/Creepsky/creepMiner/milestones/2","html_url":"https://github.com/Creepsky/creepMiner/milestone/2","labels_url":"https://api.github.com/repos/Creepsky/creepMiner/milestones/2/labels","id":2361265,"number":2,"title":"1.5.0","description":"","creator":{"login":"Creepsky","id":8005422,"avatar_url":"https://avatars1.githubusercontent.com/u/8005422?v=3","gravatar_id":"","url":"https://api.github.com/users/Creepsky","html_url":"https://github.com/Creepsky","followers_url":"https://api.github.com/users/Creepsky/followers","following_url":"https://api.github.com/users/Creepsky/following{/other_user}","gists_url":"https://api.github.com/users/Creepsky/gists{/gist_id}","starred_url":"https://api.github.com/users/Creepsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Creepsky/subscriptions","organizations_url":"https://api.github.com/users/Creepsky/orgs","repos_url":"https://api.github.com/users/Creepsky/repos","events_url":"https://api.github.com/users/Creepsky/events{/privacy}","received_events_url":"https://api.github.com/users/Creepsky/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":3,"state":"open","created_at":"2017-03-03T14:23:25Z","updated_at":"2017-03-04T23:45:26Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2017-03-03T21:20:15Z","updated_at":"2017-03-05T11:39:00Z","closed_at":null,"body":"Hi,\r\n\r\nI have been experimenting with Burst cloud mining with couple thousands of 5-10GB plot files.\r\nAll of them are mounted using FUSE driver under certain directory on my filesystem.\r\n\r\nThe problem I'm experiencing is very low performance of plots reading processes. I believe that cause of this that plot readers are reading files sequentially (and in my scenario parallel approach is better). \r\nI have been experimenting with this problem with grouping plot files into separate folders, and place them separately in `mining.conf` file like:\r\n\r\n```\r\n\"plots\": [\r\n        \"path_to_directory1\",\r\n        \"path_to_directory2\",\r\n        \"path_to_directory3\",\r\n        \"path_to_directory4\",\r\n...\r\n    ],\r\n```\r\n\r\nUnfortunately (no matter how many `maxPlotReaders` I have selected) performance was the same as when one all of them were placed in the same folder.\r\n\r\nI have been able to increase performance when I have scattered plot files into multiple folders (with ~100 of plot files in single directory), and launched multiple instances of creepMiner on the same machine with different mining configurations like:\r\n\r\n```\r\n\"plots\": [\r\n        \"path_to_directory1\"\r\n    ],\r\n```\r\n\r\n,\r\n\r\n```\r\n\"plots\": [\r\n        \"path_to_directory2\"\r\n    ],\r\n```\r\n\r\nand so on.\r\n\r\nWith this approach I have been able to increase performance drastically, as well as CPU and RAM :)\r\nI guess that large amount of these resources could be saved if I would use single instance of creepMiner that will spawn multiple plot reading threads that will scan single plot file directory.\r\n\r\nI know that creep miner wasn't designed for this kind of usage (generally miners are using couple of plot files instead of thousands of them), but could you implement functionality that will allow scan single plot file directories with multiple reading threads?\r\n\r\nRegards"},"comment":{"url":"https://api.github.com/repos/Creepsky/creepMiner/issues/comments/284222213","html_url":"https://github.com/Creepsky/creepMiner/issues/86#issuecomment-284222213","issue_url":"https://api.github.com/repos/Creepsky/creepMiner/issues/86","id":284222213,"user":{"login":"Creepsky","id":8005422,"avatar_url":"https://avatars1.githubusercontent.com/u/8005422?v=3","gravatar_id":"","url":"https://api.github.com/users/Creepsky","html_url":"https://github.com/Creepsky","followers_url":"https://api.github.com/users/Creepsky/followers","following_url":"https://api.github.com/users/Creepsky/following{/other_user}","gists_url":"https://api.github.com/users/Creepsky/gists{/gist_id}","starred_url":"https://api.github.com/users/Creepsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Creepsky/subscriptions","organizations_url":"https://api.github.com/users/Creepsky/orgs","repos_url":"https://api.github.com/users/Creepsky/repos","events_url":"https://api.github.com/users/Creepsky/events{/privacy}","received_events_url":"https://api.github.com/users/Creepsky/received_events","type":"User","site_admin":false},"created_at":"2017-03-05T11:39:00Z","updated_at":"2017-03-05T11:39:00Z","body":"Ok, you can find the changes in the branch [plotfile-types](https://github.com/Creepsky/creepMiner/tree/plotfile-types).\r\n\r\nYou can now declare some more things in the \"plots\" section in the configuration.\r\n\r\nThere are 2 types of plot dirs/files: **sequiential** and **parallel**.\r\n\r\nAll files of a **sequential** one are read one by one, by one plot reader. This is the default type, what is useful if you have a normal HDD.\r\n\r\nFiles from a **parallel** one are all read together by all plot reader side by side. SSDs or network dirs would benefit from this setting.\r\n\r\nYou can declare the type of the dir like this\r\n\r\n```\r\n\"plots\" : [\r\n    {\r\n        \"path\" : \"I:\\\\Plotfiles\",\r\n        \"type\" : \"sequential\"\r\n    },\r\n    {\r\n        \"path\" : \"J:\\\\Plotfiles\",\r\n        \"type\" : \"paralell\"\r\n    },\r\n    \"K:\\\\Plotfiles\"\r\n]\r\n```\r\n\r\nThe old form (that is still valid for backward compatibility)\r\n```\r\n\"plots\" : [\r\n    \"I:\\\\Plotfiles\"\r\n]\r\n```\r\nis equivalent to\r\n```\r\n\"plots\" : [\r\n    {\r\n        \"path\" : \"I:\\\\Plotfiles\",\r\n        \"type\" : \"sequential\"\r\n    }\r\n]\r\n```\r\n\r\nAlso have a look at #67. You can create related plot dirs, but this is described in the other issue."}}
{"actor":{"display_login":"Creepsky"}}