{"action":"closed","issue":{"url":"https://api.github.com/repos/spacetelescope/hstcal/issues/152","repository_url":"https://api.github.com/repos/spacetelescope/hstcal","labels_url":"https://api.github.com/repos/spacetelescope/hstcal/issues/152/labels{/name}","comments_url":"https://api.github.com/repos/spacetelescope/hstcal/issues/152/comments","events_url":"https://api.github.com/repos/spacetelescope/hstcal/issues/152/events","html_url":"https://github.com/spacetelescope/hstcal/issues/152","id":236245656,"number":152,"title":"Rethink CTE column parallel decomposition scheduled: static vs dynamic","user":{"login":"jamienoss","id":5013975,"avatar_url":"https://avatars1.githubusercontent.com/u/5013975?v=3","gravatar_id":"","url":"https://api.github.com/users/jamienoss","html_url":"https://github.com/jamienoss","followers_url":"https://api.github.com/users/jamienoss/followers","following_url":"https://api.github.com/users/jamienoss/following{/other_user}","gists_url":"https://api.github.com/users/jamienoss/gists{/gist_id}","starred_url":"https://api.github.com/users/jamienoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamienoss/subscriptions","organizations_url":"https://api.github.com/users/jamienoss/orgs","repos_url":"https://api.github.com/users/jamienoss/repos","events_url":"https://api.github.com/users/jamienoss/events{/privacy}","received_events_url":"https://api.github.com/users/jamienoss/received_events","type":"User","site_admin":false},"labels":[{"id":341367902,"url":"https://api.github.com/repos/spacetelescope/hstcal/labels/ACS","name":"ACS","color":"bfd4f2","default":false},{"id":328428006,"url":"https://api.github.com/repos/spacetelescope/hstcal/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"closed","locked":false,"assignee":{"login":"jamienoss","id":5013975,"avatar_url":"https://avatars1.githubusercontent.com/u/5013975?v=3","gravatar_id":"","url":"https://api.github.com/users/jamienoss","html_url":"https://github.com/jamienoss","followers_url":"https://api.github.com/users/jamienoss/followers","following_url":"https://api.github.com/users/jamienoss/following{/other_user}","gists_url":"https://api.github.com/users/jamienoss/gists{/gist_id}","starred_url":"https://api.github.com/users/jamienoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamienoss/subscriptions","organizations_url":"https://api.github.com/users/jamienoss/orgs","repos_url":"https://api.github.com/users/jamienoss/repos","events_url":"https://api.github.com/users/jamienoss/events{/privacy}","received_events_url":"https://api.github.com/users/jamienoss/received_events","type":"User","site_admin":false},"assignees":[{"login":"jamienoss","id":5013975,"avatar_url":"https://avatars1.githubusercontent.com/u/5013975?v=3","gravatar_id":"","url":"https://api.github.com/users/jamienoss","html_url":"https://github.com/jamienoss","followers_url":"https://api.github.com/users/jamienoss/followers","following_url":"https://api.github.com/users/jamienoss/following{/other_user}","gists_url":"https://api.github.com/users/jamienoss/gists{/gist_id}","starred_url":"https://api.github.com/users/jamienoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamienoss/subscriptions","organizations_url":"https://api.github.com/users/jamienoss/orgs","repos_url":"https://api.github.com/users/jamienoss/repos","events_url":"https://api.github.com/users/jamienoss/events{/privacy}","received_events_url":"https://api.github.com/users/jamienoss/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2017-06-15T16:24:06Z","updated_at":"2017-06-15T20:27:00Z","closed_at":"2017-06-15T20:27:00Z","body":"The difference between a ``static`` & ``dynamic`` queues is that for the ``static`` the total number of columns are divided by the total number of threads and then allocates this specific subdivision between each of the threads - thus equally balancing the load. ``Dynamic`` on the other hand simply leaves all of the columns in the queue and then allows each thread to pull a column off of the queue dynamically, at run-time, when it has finished processing the previous column - thus allowing for some threads to do more work than others.\r\n\r\nThe original CTE [code](https://github.com/spacetelescope/hstcal/blob/master/pkg/wfc3/calwf3/wf3cte/wf3cte.c#L874-L877) decomposes the CTE correction on a per column basis and then schedules the parallel queue as ``dynamic``. The reason it does this is due to the treatment of subarrays being 'pasted' into a full frame image space and thus creating padding. This padding is obviously not processed and as such means, entire columns are not needing processing. To accommodate for this, ``dynamic`` scheduling is required to correctly balance the workload otherwise a ``static`` schedule would open the possibility of some threads receiving nothing but padded columns and therefore not being able to process anything, rendering them a wasted resource. The new implementation does not paste and pad subarrays into full frames and therefore doesn't suffer from this issue, because of this I changed the schedule to ``static`` to avoid the slight overhead.\r\n\r\nHowever, without using the ``--nthreads <n>`` option to specify the number of threads to use, it runs _greedy_ and will use all available threads. In the case that this is not being run a  shared machine and nothing else is running, there is zero or minimal cpu contention, then ``static`` scheduling is great. In the case that it is a shared machine it will use all of the threads regardless of who else is running code on the said machine and thus contention is a possibility. Now in the static case, this can be a big issue. If the machine has 8 cores (not hyperthreaded) and 8 try to run ``acscte`` however 2 are already being used, the columns distributed to those 2 cores are in contention with tht already running and thus have to wait for the previous process to complete. This, therefore means that the entire ``acscte`` process gets held up by these 2 cores.\r\n\r\nThe solution here is to simply make this main loop over each column dynamically scheduled and thus allowing the finished and waiting cores to process the columns that would have otherwise been distributed to those to contended cores.\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"jamienoss"}}