{"action":"created","issue":{"url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/34","repository_url":"https://api.github.com/repos/ManageIQ/manageiq-design","labels_url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/34/labels{/name}","comments_url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/34/comments","events_url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/34/events","html_url":"https://github.com/ManageIQ/manageiq-design/issues/34","id":277540425,"number":34,"title":"[RFC] Collect, manage, and provision Cloud provider managed (PaaS) services","user":{"login":"01100010011001010110010101110000","id":6529397,"avatar_url":"https://avatars0.githubusercontent.com/u/6529397?v=4","gravatar_id":"","url":"https://api.github.com/users/01100010011001010110010101110000","html_url":"https://github.com/01100010011001010110010101110000","followers_url":"https://api.github.com/users/01100010011001010110010101110000/followers","following_url":"https://api.github.com/users/01100010011001010110010101110000/following{/other_user}","gists_url":"https://api.github.com/users/01100010011001010110010101110000/gists{/gist_id}","starred_url":"https://api.github.com/users/01100010011001010110010101110000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/01100010011001010110010101110000/subscriptions","organizations_url":"https://api.github.com/users/01100010011001010110010101110000/orgs","repos_url":"https://api.github.com/users/01100010011001010110010101110000/repos","events_url":"https://api.github.com/users/01100010011001010110010101110000/events{/privacy}","received_events_url":"https://api.github.com/users/01100010011001010110010101110000/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"Fryguy","id":52120,"avatar_url":"https://avatars3.githubusercontent.com/u/52120?v=4","gravatar_id":"","url":"https://api.github.com/users/Fryguy","html_url":"https://github.com/Fryguy","followers_url":"https://api.github.com/users/Fryguy/followers","following_url":"https://api.github.com/users/Fryguy/following{/other_user}","gists_url":"https://api.github.com/users/Fryguy/gists{/gist_id}","starred_url":"https://api.github.com/users/Fryguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fryguy/subscriptions","organizations_url":"https://api.github.com/users/Fryguy/orgs","repos_url":"https://api.github.com/users/Fryguy/repos","events_url":"https://api.github.com/users/Fryguy/events{/privacy}","received_events_url":"https://api.github.com/users/Fryguy/received_events","type":"User","site_admin":false},"assignees":[{"login":"Fryguy","id":52120,"avatar_url":"https://avatars3.githubusercontent.com/u/52120?v=4","gravatar_id":"","url":"https://api.github.com/users/Fryguy","html_url":"https://github.com/Fryguy","followers_url":"https://api.github.com/users/Fryguy/followers","following_url":"https://api.github.com/users/Fryguy/following{/other_user}","gists_url":"https://api.github.com/users/Fryguy/gists{/gist_id}","starred_url":"https://api.github.com/users/Fryguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fryguy/subscriptions","organizations_url":"https://api.github.com/users/Fryguy/orgs","repos_url":"https://api.github.com/users/Fryguy/repos","events_url":"https://api.github.com/users/Fryguy/events{/privacy}","received_events_url":"https://api.github.com/users/Fryguy/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2017-11-28T21:15:19Z","updated_at":"2017-12-06T17:28:59Z","closed_at":null,"author_association":"NONE","body":"# Missing Feature\r\nCurrently ManageIQ does not collect most managed services from cloud providers, some examples being databases such as Azure database for SQL, Amazon RDS instances, HDInsight, Elastic Map Reduce, etc.\r\n\r\nAs a result, if one wishes to provision these services from within ManageIQ, one must use a generic service template item and use some combination of tags and custom attributes on the resulting service to control the object in the provider, retire it, or perform other operations. This has a number of downsides, such as the services being static, unable to collect metrics, etc.\r\n\r\n# Proposed Solution\r\nI would like to begin this by implementing models, refreshers, provisioning workflows for the database and map reduce services. In Azure, these are Azure Database for *Engine* and HDInsight, and in AWS these are RDS for *Engine* and EMR. I'll break out the specifics into sections for each service.\r\n\r\n## Cloud Database\r\nThere are currently `CloudDatabase` and `CloudDatabaseFlavor` models present in the core ManageIQ repo. These should be sufficient as they are to handle the different tooling available in AWS, Azure, Google Cloud, and OpenStack. Azure has a fair amount of tooling built up around their PaaS offering for MS SQL; it is split out into three different 'services' which all tie together.\r\n\r\n* SQL Servers\r\n* SQL Databases\r\n* SQL Elastic Pools\r\n\r\nThe SQL server defines admin credentials and an endpoint for management, equivalent to an AWS `DBInstance`. The databases are databases within a server. These databases have flavors, called a `ServiceObjectiveName` in Azure parlance. These flavors can be applied to individual databases, or to a collection of databases via an elastic pool, so a database flavor would be either a specific `ServiceObjectiveName`, or the database's elastic pool. \r\n\r\nThis is much more straightforward in AWS. Here, the flavor of a database is simply the sizing of the `DBInstance`, and the individual databases are not exposed. This is also how Azure handles all other DB engines it offers, such as MySQL and PostgreSQL.\r\n\r\nGoogle Cloud is essentially a simplified AWS in its database offerings. It offers MySQL and has PostgreSQL in beta, with specific database instance sizes and database engine versions. Like Azure, it is possible to enumerate the databases in an instance via the API. \r\n\r\nTrevor, OpenStack's DBaaS plugin, has a [REST API](https://developer.openstack.org/api-ref/database/) that allows one to list database instances, the databases in those instances, engine versions, etc. The information available there fits nicely into the models already present.\r\n\r\nTo gather the most accurate information, I believe it is appropriate to map each `DBInstance` in AWS into a `CloudDatabase`, and do the same in Azure for all PaaS database offerings expcept MS SQL. For MS SQL, we should map the individual databases into a `CloudDatabase`, because the performance level of a database is tied to the database itself, not to the server. \r\n\r\n## EMR\r\nAzure and AWS both have PaaS offerings for clusters running various big data processing software and their supporting applications: Hadoop, Spark, Hive, Zookeeper, etc. They differ in that in Azure one must deploy a cluster per processing engine, e.g. one for Hadoop MapReduce and one for Spark, but in AWS it seems to be possible to deploy all supported software onto a single cluster. However, they have in common a release version for whatever versions of the software have been packaged by AWS and Azure, and an applications array containing the applications installed on the cluster.\r\n\r\nAgain it seems that Google Cloud is a simplified AWS. One chooses an image version, which contains all available applications (Hadoop, Spark, etc). It does not seem to be possible to query what software is actually installed, only the image version number, so `applications` would likely be `nil` here.\r\n\r\nSahara, OpenStack's big data plugin, has a [REST API](https://developer.openstack.org/api-ref/data-processing/) that exposes clusters and the templates used to create them. The templates are prebuilt node configurations, but the software installed on those nodes comes from so-called plugins, which exist for vanilla Apache Hadoop, Spark, and other processing engines. So the `version` will come from the plugin associated with a cluster.\r\n\r\nWith that in mind, I think something like this would work well for the schema of a CloudMapReduce model:\r\n\r\nColumn | Type\r\n--- | ---\r\nid | bigint\r\nems_id | bigint\r\ncloud_tenant_id | bigint\r\nresource_group_id | bigint\r\nems_ref | string\r\nname | string\r\nversion | string\r\nextra_attributes | text\r\nstatus | string\r\nstatus_reason | string\r\napplications | array\r\nnodes | array\r\n\r\nWhere nodes are the `Flavors` that the nodes are running on, and applications are the applications installed into the cluster. For Azure, this will be the kind of cluster and whatever additional apps are installed, and for AWS will simply be the apps. I am uncertain if we will need a separate model for this, or if it can simply be an array of hashes along the lines of:\r\n\r\n```ruby\r\n[{\r\n    name: \"app_name\",\r\n    version: 1.0.0\r\n}]\r\n```\r\n\r\n## PRs\r\n\r\n- [ ] Models for CloudMapReduce\r\n- [ ] Refresher for databases\r\n- [ ] Refresher for MR clusters\r\n- [ ] Expose models to automate\r\n- [ ] Trigger provider refresh upon relevant events for these models\r\n- [ ] Provisioning workflow for databases\r\n- [ ] Provisioning workflow for MR clusters"},"comment":{"url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/comments/349714162","html_url":"https://github.com/ManageIQ/manageiq-design/issues/34#issuecomment-349714162","issue_url":"https://api.github.com/repos/ManageIQ/manageiq-design/issues/34","id":349714162,"user":{"login":"01100010011001010110010101110000","id":6529397,"avatar_url":"https://avatars0.githubusercontent.com/u/6529397?v=4","gravatar_id":"","url":"https://api.github.com/users/01100010011001010110010101110000","html_url":"https://github.com/01100010011001010110010101110000","followers_url":"https://api.github.com/users/01100010011001010110010101110000/followers","following_url":"https://api.github.com/users/01100010011001010110010101110000/following{/other_user}","gists_url":"https://api.github.com/users/01100010011001010110010101110000/gists{/gist_id}","starred_url":"https://api.github.com/users/01100010011001010110010101110000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/01100010011001010110010101110000/subscriptions","organizations_url":"https://api.github.com/users/01100010011001010110010101110000/orgs","repos_url":"https://api.github.com/users/01100010011001010110010101110000/repos","events_url":"https://api.github.com/users/01100010011001010110010101110000/events{/privacy}","received_events_url":"https://api.github.com/users/01100010011001010110010101110000/received_events","type":"User","site_admin":false},"created_at":"2017-12-06T17:28:59Z","updated_at":"2017-12-06T17:28:59Z","author_association":"NONE","body":"Updated. I also looked into VMware's vCloud offerings for these and they don't seem to have anything of note yet. I couldn't find anything for Air SQL beyond a tech preview announcement in 2015, and their Big Data Extensions is a virtual appliance running on vSphere, that I think would require a separate provider.\r\n\r\nAlso, I'm concerned that I may be shoehorning the databases into `CloudDatabase` because of the odd way Azure treats MS SQL databases. Might it be better to add a `CloudDatabaseInstance` model that `has_many` `CloudDatabases`? That would leave the issue of where to associate flavors, however, as flavors are associated directly to the database in Azure MS SQL, but to the database instance everywhere else, even in Azure's other DBaaS offerings. Thoughts?"}}
{"actor":{"display_login":"01100010011001010110010101110000"}}