{"action":"created","issue":{"url":"https://api.github.com/repos/Norconex/importer/issues/73","repository_url":"https://api.github.com/repos/Norconex/importer","labels_url":"https://api.github.com/repos/Norconex/importer/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/importer/issues/73/comments","events_url":"https://api.github.com/repos/Norconex/importer/issues/73/events","html_url":"https://github.com/Norconex/importer/issues/73","id":283680005,"number":73,"title":"Re-crawl doesn't recognize all files already indexed ","user":{"login":"jmrichardson","id":6565528,"avatar_url":"https://avatars3.githubusercontent.com/u/6565528?v=4","gravatar_id":"","url":"https://api.github.com/users/jmrichardson","html_url":"https://github.com/jmrichardson","followers_url":"https://api.github.com/users/jmrichardson/followers","following_url":"https://api.github.com/users/jmrichardson/following{/other_user}","gists_url":"https://api.github.com/users/jmrichardson/gists{/gist_id}","starred_url":"https://api.github.com/users/jmrichardson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmrichardson/subscriptions","organizations_url":"https://api.github.com/users/jmrichardson/orgs","repos_url":"https://api.github.com/users/jmrichardson/repos","events_url":"https://api.github.com/users/jmrichardson/events{/privacy}","received_events_url":"https://api.github.com/users/jmrichardson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-12-20T20:05:02Z","updated_at":"2017-12-26T04:54:42Z","closed_at":null,"author_association":"NONE","body":"Hi, I have ~4.7M files already indexed and re-ran the crawler to see how long it would take to crawl on second attempt.  The first (initial) crawl took 1 day 10 hours.  The second attempt I started last night and it is still running.  When I started the crawler with the \"-a start\", it did detect a previous run and after a bit of time it started and showed the files had been indexed with the \"REJECTED_UNMODIFIED\" tag in the log file.  However,  after about 8% of the latest log file (line 1070927 of 11906185), files are now being committed \"DOCUMENT_COMMITTED_ADD\" even though they have been indexed from the first run.  All of the files since then have not been marked as \"REJECTED_UNMODIFIED\".  The only thing that may have been an issue was during the 2nd crawl  it was stopped pre-maturely stopped due to a server re-location during its run (and re-started with the \"start\" parameter). \r\n\r\nAfter this crawl completes (sometime tomorrow) I will re-run it again and make sure it doesn't get stopped prematurely.  However, I was expecting the previous files indexed would remain in the database.  Is there any way to inspect the database or do more error testing to understand why this happened?\r\n\r\nThanks for your help\r\nJohn\r\n"},"comment":{"url":"https://api.github.com/repos/Norconex/importer/issues/comments/353917468","html_url":"https://github.com/Norconex/importer/issues/73#issuecomment-353917468","issue_url":"https://api.github.com/repos/Norconex/importer/issues/73","id":353917468,"user":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars1.githubusercontent.com/u/2326274?v=4","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"created_at":"2017-12-26T04:54:42Z","updated_at":"2017-12-26T04:54:42Z","author_association":"CONTRIBUTOR","body":"I am assuming you are talking about the \"crawl store\"?  If so, what about using the default (MVStore) implementation?  Did you get performance issues there?\r\n\r\nUsing an embedded SQL database like H2 is fine for relatively small sites, but on large ones, you will start to see performance degradation for sure.\r\n\r\nWhat kind of server do you have? Do you monitor the server resources when you run it with over 50 threads? How does it behave? 50 threads seem like a lot.  How many CPUs do you have?  If you specify more threads than CPUs, and all/many threads are busy, are you maxing your CPUs?  If so, performance will be impacted for sure.  Also, do you have lots of RAM?  The parsing of files can be CPU and RAM intensive, so if having too many threads results in taking too much memory, it may result in disk swapping as well.  Speaking of disks, if you have 50 threads writing heavily to disk, plus your H2 database... do you have an ultra-fast disk?  \r\n\r\nI would recommend lowering your number of threads and see if you get better performance.\r\n\r\nAlso, if you are maxing your server capacity and speed is essential, I would recommend you split your crawling into a few Collector instances running on different servers.\r\n\r\nAs far as not picking up files properly when you rerun it after a crash/stop, it is likely because you launched it with \"start\".  When doing so, it starts from the beginning, but the \"cache\" from the previous run was incomplete, so it may cause issues.  You are best to launch it using \"resume\" whenever it stops before completion so it picks up exactly where it was."}}
{"actor":{"display_login":"essiembre"}}