{"action":"created","issue":{"url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/668","repository_url":"https://api.github.com/repos/jeremylong/DependencyCheck","labels_url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/668/labels{/name}","comments_url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/668/comments","events_url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/668/events","html_url":"https://github.com/jeremylong/DependencyCheck/pull/668","id":209168707,"number":668,"title":"Misc minor performance tweaking and cleanup","user":{"login":"stefanneuhaus","id":8896988,"avatar_url":"https://avatars.githubusercontent.com/u/8896988?v=3","gravatar_id":"","url":"https://api.github.com/users/stefanneuhaus","html_url":"https://github.com/stefanneuhaus","followers_url":"https://api.github.com/users/stefanneuhaus/followers","following_url":"https://api.github.com/users/stefanneuhaus/following{/other_user}","gists_url":"https://api.github.com/users/stefanneuhaus/gists{/gist_id}","starred_url":"https://api.github.com/users/stefanneuhaus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stefanneuhaus/subscriptions","organizations_url":"https://api.github.com/users/stefanneuhaus/orgs","repos_url":"https://api.github.com/users/stefanneuhaus/repos","events_url":"https://api.github.com/users/stefanneuhaus/events{/privacy}","received_events_url":"https://api.github.com/users/stefanneuhaus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-02-21T14:54:13Z","updated_at":"2017-02-21T16:00:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/jeremylong/DependencyCheck/pulls/668","html_url":"https://github.com/jeremylong/DependencyCheck/pull/668","diff_url":"https://github.com/jeremylong/DependencyCheck/pull/668.diff","patch_url":"https://github.com/jeremylong/DependencyCheck/pull/668.patch"},"body":"My aim was to further speedup the CVE DB update of the OOTB setup using H2 by making CveDB thread-safe and processing the update files in parallel.\r\n\r\nThough I tried several things the speedup is not very impressive. The mean runtime of `gradlew --no-daemon dependencyCheckPurge dependencyCheckUpdate` decreased by ~3 sec on my machine/internet connection.\r\n\r\nNevertheless the PR could be valuable:\r\n* refactoring of CveDB eases further optimizations/parallelization\r\n* small performance improvement\r\n* minor cleanup\r\n\r\n\r\nI want to share some observations which you might find interesting:\r\n* Performance of the DB updates only plays an important role in the OOTB setup, as in this case the effect of the cached DB is restricted (lack of a central DB and varying build servers).\r\n* Though CveDB is thread-safe now, increasing parallelism by means of `NvdCveUpdater.PROCESSING_THREAD_POOL_SIZE` even degrades performance at the moment. This is probably not very surprising, as CveDB maintains a single connection and thus uses synchronization.\r\n* Using a connection pool is probably a good idea, as this would require far less synchronization. Advantages:\r\n** Enterprise setup: parallel processing should scale very well\r\n** H2: ~5-10% speedup (testrunning to estimate the speedup required some tweaks)\r\n* Submitting DB requests is not the bottleneck in the fast H2 embedded mode as no network IO is involved.\r\n* H2 connection string\r\n** `FILE_LOCK=SERIALIZED`: omitting this brings some speedup but is probably not an option, as there could be builds running in parallel\r\n** also using `LOCK_MODE=0` (isolation level Read Uncommitted): speedup only in conjunction with omitting `FILE_LOCK=SERIALIZED`. The same drawbacks as above apply.\r\n* Enabling batch support for H2 did not bring any improvements. Tested with autocommit enabled and disabled."},"comment":{"url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/comments/281387120","html_url":"https://github.com/jeremylong/DependencyCheck/pull/668#issuecomment-281387120","issue_url":"https://api.github.com/repos/jeremylong/DependencyCheck/issues/668","id":281387120,"user":{"login":"stefanneuhaus","id":8896988,"avatar_url":"https://avatars.githubusercontent.com/u/8896988?v=3","gravatar_id":"","url":"https://api.github.com/users/stefanneuhaus","html_url":"https://github.com/stefanneuhaus","followers_url":"https://api.github.com/users/stefanneuhaus/followers","following_url":"https://api.github.com/users/stefanneuhaus/following{/other_user}","gists_url":"https://api.github.com/users/stefanneuhaus/gists{/gist_id}","starred_url":"https://api.github.com/users/stefanneuhaus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stefanneuhaus/subscriptions","organizations_url":"https://api.github.com/users/stefanneuhaus/orgs","repos_url":"https://api.github.com/users/stefanneuhaus/repos","events_url":"https://api.github.com/users/stefanneuhaus/events{/privacy}","received_events_url":"https://api.github.com/users/stefanneuhaus/received_events","type":"User","site_admin":false},"created_at":"2017-02-21T16:00:17Z","updated_at":"2017-02-21T16:00:17Z","body":"* \"file lock in the JDBC URL\": I fully agree, we should not trade performance for reliability.\r\n* I meant \"connection pool\", not \"thread pool\" (I corrected it above). And yes, this still would require some testing."}}
{"actor":{"display_login":"stefanneuhaus"}}