{"action":"created","issue":{"url":"https://api.github.com/repos/dpla/ingestion3/issues/12","repository_url":"https://api.github.com/repos/dpla/ingestion3","labels_url":"https://api.github.com/repos/dpla/ingestion3/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/dpla/ingestion3/issues/12/comments","events_url":"https://api.github.com/repos/dpla/ingestion3/issues/12/events","html_url":"https://github.com/dpla/ingestion3/pull/12","id":227436076,"number":12,"title":"get muliple pages from oai harvest","user":{"login":"AudreyAltman","id":3615206,"avatar_url":"https://avatars0.githubusercontent.com/u/3615206?v=3","gravatar_id":"","url":"https://api.github.com/users/AudreyAltman","html_url":"https://github.com/AudreyAltman","followers_url":"https://api.github.com/users/AudreyAltman/followers","following_url":"https://api.github.com/users/AudreyAltman/following{/other_user}","gists_url":"https://api.github.com/users/AudreyAltman/gists{/gist_id}","starred_url":"https://api.github.com/users/AudreyAltman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AudreyAltman/subscriptions","organizations_url":"https://api.github.com/users/AudreyAltman/orgs","repos_url":"https://api.github.com/users/AudreyAltman/repos","events_url":"https://api.github.com/users/AudreyAltman/events{/privacy}","received_events_url":"https://api.github.com/users/AudreyAltman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-05-09T17:12:09Z","updated_at":"2017-05-10T20:46:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dpla/ingestion3/pulls/12","html_url":"https://github.com/dpla/ingestion3/pull/12","diff_url":"https://github.com/dpla/ingestion3/pull/12.diff","patch_url":"https://github.com/dpla/ingestion3/pull/12.patch"},"body":"This uses the DataSource interface to harvest multiple pages from an OAI feed.\r\n\r\n* The process of collecting HTTP responses from the OAI feed cannot be parallelized due the nature of the OAI feed.  To reduce memory load prior to parallelization, the responses are treated as Strings rather than XML objects.  They are parsed into XML object after parallelization has occurred.\r\n* The `OaiResponseProcessor` has been changed from a class to a static object in order to make it easily serializable.\r\n* At this time, the only two methods of `OaiResponseProcessor` that are being used are `getRecordsAsTuples` and `getResumptionToken`.\r\n\r\nThis has been tested locally with the following options:\r\n```\r\n--executor-memory \"1g\" \r\n--driver-memory \"2g\"\r\nendpoint: http://oai.artstor.org/oaicatmuseum/OAIHandler\r\nmetadataPrefix: oai_dc\r\nverb: ListRecords\r\n```\r\n\r\nThis addresses [ticket DT-1279](https://digitalpubliclibraryofamerica.atlassian.net/browse/DT-1279)."},"comment":{"url":"https://api.github.com/repos/dpla/ingestion3/issues/comments/300607258","html_url":"https://github.com/dpla/ingestion3/pull/12#issuecomment-300607258","issue_url":"https://api.github.com/repos/dpla/ingestion3/issues/12","id":300607258,"user":{"login":"mdellabitta","id":925372,"avatar_url":"https://avatars1.githubusercontent.com/u/925372?v=3","gravatar_id":"","url":"https://api.github.com/users/mdellabitta","html_url":"https://github.com/mdellabitta","followers_url":"https://api.github.com/users/mdellabitta/followers","following_url":"https://api.github.com/users/mdellabitta/following{/other_user}","gists_url":"https://api.github.com/users/mdellabitta/gists{/gist_id}","starred_url":"https://api.github.com/users/mdellabitta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdellabitta/subscriptions","organizations_url":"https://api.github.com/users/mdellabitta/orgs","repos_url":"https://api.github.com/users/mdellabitta/repos","events_url":"https://api.github.com/users/mdellabitta/events{/privacy}","received_events_url":"https://api.github.com/users/mdellabitta/received_events","type":"User","site_admin":false},"created_at":"2017-05-10T20:46:00Z","updated_at":"2017-05-10T20:46:00Z","body":"Great work!"}}
{"actor":{"display_login":"mdellabitta"}}