{"action":"opened","issue":{"url":"https://api.github.com/repos/cxcsds/ciao-contrib/issues/53","repository_url":"https://api.github.com/repos/cxcsds/ciao-contrib","labels_url":"https://api.github.com/repos/cxcsds/ciao-contrib/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/cxcsds/ciao-contrib/issues/53/comments","events_url":"https://api.github.com/repos/cxcsds/ciao-contrib/issues/53/events","html_url":"https://github.com/cxcsds/ciao-contrib/issues/53","id":238886294,"number":53,"title":"srcflux using dmmerge to combine temporary files","user":{"login":"kglotfelty","id":6662179,"avatar_url":"https://avatars2.githubusercontent.com/u/6662179?v=3","gravatar_id":"","url":"https://api.github.com/users/kglotfelty","html_url":"https://github.com/kglotfelty","followers_url":"https://api.github.com/users/kglotfelty/followers","following_url":"https://api.github.com/users/kglotfelty/following{/other_user}","gists_url":"https://api.github.com/users/kglotfelty/gists{/gist_id}","starred_url":"https://api.github.com/users/kglotfelty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kglotfelty/subscriptions","organizations_url":"https://api.github.com/users/kglotfelty/orgs","repos_url":"https://api.github.com/users/kglotfelty/repos","events_url":"https://api.github.com/users/kglotfelty/events{/privacy}","received_events_url":"https://api.github.com/users/kglotfelty/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-06-27T15:14:00Z","updated_at":"2017-06-27T15:14:00Z","closed_at":null,"body":"Based on email from Dong-Woo (Ref: \"Fwd: Re: source pipe\" 6/26/2017) \r\n\r\n`srcflux` runs `dmextract` to compute the PSF fraction in the source and background regions for each source individually.  These individual output files are then combined into a single table using `dmmerge`.\r\n\r\nHowever, `dmmerge` doesn't just append the rows, it also merges the subspace.  `dmextract` adds the region subspace into the output file.  Thus when `dmmerge` runs, as in DWK's example with 450+ sources, the region intersection is extremely messy; and ultimately useless since we only care about the PSF fractions.  In this case, the merge ends up exhausting system memory (though I'm somewhat surprised why) and the script fails.\r\n\r\nWe should be able to do something like tack on a `[subspace -sky]` to the files being `dmmerge`d together.  My off-to-the-side test showed that this ran to completion.  I also went ahead and disable the header lookup table since we don't care about the header in this case either.\r\n\r\nBut I want to see if there are any bad side effects of doing this.  Also I'm wondering if just opening the files via crates would work and appending the rows together that was works -- or will we hit a system limit on number of open file handles (which I'm thinking is what sent me down the `dmmerge` route to begin with).\r\n\r\nNote:  There are other places in the script where dmmerge is used in a similar way to collect temp files before adding them to the final output table.  Those need to be reviewed too.\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"kglotfelty"}}