{"action":"opened","issue":{"url":"https://api.github.com/repos/openaustralia/morph/issues/1120","repository_url":"https://api.github.com/repos/openaustralia/morph","labels_url":"https://api.github.com/repos/openaustralia/morph/issues/1120/labels{/name}","comments_url":"https://api.github.com/repos/openaustralia/morph/issues/1120/comments","events_url":"https://api.github.com/repos/openaustralia/morph/issues/1120/events","html_url":"https://github.com/openaustralia/morph/issues/1120","id":205760383,"number":1120,"title":"Extract data store into standalone API","user":{"login":"auxesis","id":12306,"avatar_url":"https://avatars.githubusercontent.com/u/12306?v=3","gravatar_id":"","url":"https://api.github.com/users/auxesis","html_url":"https://github.com/auxesis","followers_url":"https://api.github.com/users/auxesis/followers","following_url":"https://api.github.com/users/auxesis/following{/other_user}","gists_url":"https://api.github.com/users/auxesis/gists{/gist_id}","starred_url":"https://api.github.com/users/auxesis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/auxesis/subscriptions","organizations_url":"https://api.github.com/users/auxesis/orgs","repos_url":"https://api.github.com/users/auxesis/repos","events_url":"https://api.github.com/users/auxesis/events{/privacy}","received_events_url":"https://api.github.com/users/auxesis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-07T01:36:17Z","updated_at":"2017-02-07T01:36:17Z","closed_at":null,"body":"As @mlandauer wrote in #647: \r\n\r\n> ## Have no need to maintain state between scraper runs\r\n> \r\n> Currently in morph.io the `/data` directory in each container is mounted to the host filesystem to ensure that the SQLite database for each scraper is persistent. This is currently stopping the docker server from running on a different server to the web app server.\r\n> \r\n> So, for simplicity, what if the necessary data and code is transferred to the container before the run and the result is transferred out of the container at the end of the run? This way, there is little that the morph server needs to know and makes it very easy for anyone to run one. Also, if third parties are running docker servers that morph.io is using we shouldn't be depending on them always being there. If they can disappear at any time they also can't store information we need.\r\n\r\nTo allow scrapers to run on separate nodes to the Morph web app, we must split the scraper database fetch/store operations out into a web service that can be queried by the Sidekiq workers. \r\n\r\nOn the server side, [these methods](https://github.com/openaustralia/morph/blob/428503978a0f17e585b650464019ee100564e382/lib/morph/runner.rb#L195-L220) need to be refactored into API endpoints. \r\n\r\nOn the client side, [these methods](https://github.com/openaustralia/morph/blob/428503978a0f17e585b650464019ee100564e382/lib/morph/runner.rb#L195-L220) need to be refactored to make a request to the web service. \r\n\r\nTo implement this, I propose:\r\n\r\n - Creating a standalone Sinatra API for these fetch/store operations\r\n - Mounting the Sinatra API into Morph's Rails app\r\n - Deploying the Sinatra API early to verify fetch/store operations \r\n - Adding a feature toggle on the client side to switch between existing and new behaviour (if possible)\r\n\r\nAcceptance criteria: \r\n\r\n - [ ] All Morph scraper workers fetch all their data through the standalone API\r\n - [ ] All store operations are authenticated with an API key\r\n - [ ] Metrics for fetch/store operations are published at `/metrics`\r\n - [ ] API keys can be managed via an API\r\n\r\nOnce we have this in place, we can investigate: \r\n\r\n - Adding ability to fetch/store scraper databases on S3\r\n - Running scrapers on separate nodes"}}
{"actor":{"display_login":"auxesis"}}