{"action":"created","comment":{"url":"https://api.github.com/repos/apache/incubator-metron/pulls/comments/97836741","pull_request_review_id":18459681,"id":97836741,"diff_hunk":"@@ -317,6 +316,8 @@ This package installs the Metron Profiler %{metron_home}\n # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n \n %changelog\n+* Thu Jan 19 2017 Justin Leet <justinjleet@gmail.com> - 0.3.0","path":"metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec","position":28,"original_position":28,"commit_id":"7d00731ef00736cf868ff08bad0d607e43c97721","original_commit_id":"248fd5c72b2a1f3cb3bcf8d938b4ffc96abf5525","user":{"login":"cestella","id":540359,"avatar_url":"https://avatars.githubusercontent.com/u/540359?v=3","gravatar_id":"","url":"https://api.github.com/users/cestella","html_url":"https://github.com/cestella","followers_url":"https://api.github.com/users/cestella/followers","following_url":"https://api.github.com/users/cestella/following{/other_user}","gists_url":"https://api.github.com/users/cestella/gists{/gist_id}","starred_url":"https://api.github.com/users/cestella/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cestella/subscriptions","organizations_url":"https://api.github.com/users/cestella/orgs","repos_url":"https://api.github.com/users/cestella/repos","events_url":"https://api.github.com/users/cestella/events{/privacy}","received_events_url":"https://api.github.com/users/cestella/received_events","type":"User","site_admin":false},"body":"+1 to that @dlyle65535 ","created_at":"2017-01-25T17:46:58Z","updated_at":"2017-01-25T17:46:58Z","html_url":"https://github.com/apache/incubator-metron/pull/421#discussion_r97836741","pull_request_url":"https://api.github.com/repos/apache/incubator-metron/pulls/421","_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/comments/97836741"},"html":{"href":"https://github.com/apache/incubator-metron/pull/421#discussion_r97836741"},"pull_request":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/421"}}},"pull_request":{"url":"https://api.github.com/repos/apache/incubator-metron/pulls/421","id":102462669,"html_url":"https://github.com/apache/incubator-metron/pull/421","diff_url":"https://github.com/apache/incubator-metron/pull/421.diff","patch_url":"https://github.com/apache/incubator-metron/pull/421.patch","issue_url":"https://api.github.com/repos/apache/incubator-metron/issues/421","number":421,"state":"open","locked":false,"title":"METRON-283 Migrate Geo Enrichment outside of MySQL","user":{"login":"justinleet","id":5077341,"avatar_url":"https://avatars.githubusercontent.com/u/5077341?v=3","gravatar_id":"","url":"https://api.github.com/users/justinleet","html_url":"https://github.com/justinleet","followers_url":"https://api.github.com/users/justinleet/followers","following_url":"https://api.github.com/users/justinleet/following{/other_user}","gists_url":"https://api.github.com/users/justinleet/gists{/gist_id}","starred_url":"https://api.github.com/users/justinleet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinleet/subscriptions","organizations_url":"https://api.github.com/users/justinleet/orgs","repos_url":"https://api.github.com/users/justinleet/repos","events_url":"https://api.github.com/users/justinleet/events{/privacy}","received_events_url":"https://api.github.com/users/justinleet/received_events","type":"User","site_admin":false},"body":"## MySQL Removed\r\nDrops MySQL entirely from the project.  This is done for a couple reasons outlined in a discussion thread on the dev lists.  They boil down to a combination of licensing, eliminating a single point of failure, and using MaxMind's libraries for handling GeoLite2 data, and a couple other concerns.\r\n\r\nThis PR includes dependencies, installation paths, READMEs, etc.  The only places left are MySQLConfig and MySQLConfigTest for if anyone wants to use them.  The vast majority of removed files / code are simply from stripping this out.  If any traces of MySQL outside of this are found in review, they should almost certainly be removed.\r\n\r\nThis moves to a system based on using [MaxMind's binary DB](http://dev.maxmind.com/geoip/geoip2/geolite2/) directly. Per their page:\r\n\r\n> The GeoLite2 databases are distributed under the Creative Commons Attribution-ShareAlike 4.0 International License. \r\n\r\nOur LICENSE file has been updated with the notification that we include use their database (and some of their [test data](https://github.com/maxmind/MaxMind-DB), which is CCA ShareAlike 3.0).  Both of these licenses are acceptable for us as stated on [Apache Legal](https://www.apache.org/legal/resolved#cc-sa).\r\n\r\nLet me know if that notification should be in a different spot or spots, and I can adjust it appropriately.\r\n\r\n## GeoLite2 Database\r\nThe main portion of the PR is in `GeoLiteDatabaseTest.java`, which manages access to the GeoLite2 database.\r\n\r\n### Raw Database\r\nThe raw database is stored on HDFS.  By default, it will be in `/apps/metron/geo`.  If no explicit location is given, `/apps/metron/geo/default/<db_filename>` will be used.  Otherwise, updates will use `/apps/metron/geo/<millis>/<db_filename>`.  Given the low rate of churn on the DB (updated once per week) and the potential for replay use cases , I haven't implemented any pruning or anything fancy on top of this.\r\n\r\n#### Updating DB\r\nA script is provided for updates `geo_enrichment_load.sh` in `metron-data-management`.  Usage details are provided in `metron-data-management/README.md`.  Note that the original didn't appear to have update capabilities, \r\n\r\nThe script will pull down a new instance of GeoLite2 database.  This location can be either their standard web address (or somewhere else hosted), or even a file:// URL.  Once the db file is pulled down, it will push to the appropriate HDFS location.  Finally, it will pull down and update the global config with the new location.  This will not require a topology restart.\r\n\r\nNote that there have been conversations about how we manage config updates (specifically leaning towards Ambari).  This has not been finalized, and we have two non Ambari testing environments (quickdev and docker-metron) so this just hits ZK.  Ambari is not updated based on this script, and it is the user's responsibility to update global.json.\r\n\r\nThis leads to a questions people may have preferences on\r\n\r\n- Do we want the script to always update? Should there be a flag to stage the file, but not update configs?\r\n\r\n### Code\r\nIt is a singleton that allows for the database to be updated when a global config is updated.  It is (hopefully!) correctly locked to avoid threading issues when updating or reading from the DB (and I've been able to update without issues.\r\n\r\nThe various Bolts have been updated to make sure they initialize the adapter to have it grab the current data appropriately.\r\n\r\nIn addition, a Stellar function has been provided GEO_GET(), which takes an IPV4 address.  It probably works with an IPV6 address, but I didn't really dig into it, given that the goal was to initially match parity.\r\n\r\nGiven the somewhat core nature of this, and my relative unfamiliarity going in with how all these pieces tie together, I'm definitely looking for feedback on how things are implemented, or if I missed conventions we've used in the code.\r\n\r\n## Testing\r\nUnit testing is added for the database and Stellar portions of the code as needed.  The DB testing uses one of MaxMind's test DB's that they've published, because we can't create the binary format correctly.  It does not use the full (20+ MB) version of the data, but rather a stripped down version (on the order of several KB).\r\n\r\nThree environments were tested during this.  Having these three disparate environments make features that cut across like this more complicated to test, so additional scrutiny would be merited (I would definitely like at least one person to run through one of these themselves and make sure it's transparent).  Notably, quickdev requires Ansible setup scripts to align; the mpack requires layout, internal configuration, and handling of additional files and ownership of scripts to work properly; and docker-metron requires essentially cheating the scripts and just running a wget on the file because things aren't actually setup.\r\n\r\n- quickdev\r\n- Ambari Management Pack\r\n- docker-metron\r\n\r\n### QuickDev\r\nAnsible scripts are updated.  Running data through topologies kicked out the data.\r\n\r\n### Ambari Management Pack\r\nRPMs updated where needed. Config Screen layout changed, updates made to properly handle configs and ownership.  Ran Stellar on this install.  Again, ran data through the topology.\r\n\r\n### Docker\r\nEssentially this just involved cheating the scripts and running a wget on the GeoLite2 dbfile, because there's no Hadoop.  Ran through the instructions to run the topologies (which are a little different than the others because Docker) and again was able to get data back out.\r\n\r\n## Additional Notes\r\n- As noted above, do we want the DB script to always update? Should there be a flag to stage the file, but not update configs?  I primarily see this affecting the mpack because of the Ambari management behind it.\r\n- ~~Increased `withMaxTimeMS` in the indexing integration test.  This seems unrelated to my changes ( and I believe had been seen elsewhere), so if anybody has found the root cause, I can adjust my code appropriately.~~ Outside issue (and code is reverted in this PR), see: https://issues.apache.org/jira/browse/METRON-672.  \r\n- LocID doesn't technically exist in the new data, and I suspect it was never meant to be relied upon anyway outside of being a join key.  The same applies to the new field that is replacing it in this context.  It seems like we were mostly just passing that field along because it was available, and it seems like it should be refactored to be more useful.  I didn't take on that analysis here, it's the slightly more validated version of a gut feeling.\r\n- The newer form of the MaxMind info has more data available than the old source we were using.  We should also consider passing (at least some) of this data along. See MaxMind's [What's New in GeoIp2](http://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/).  One of the ones that leapt out at me as potentially interesting was a field containing where an IP was registered, rather than just where the IP actually is.  Another is fields for `is_anonymous_proxy` key, etc.  I didn't validate if everything new is in the free version of the dataset.\r\n","created_at":"2017-01-20T13:29:45Z","updated_at":"2017-01-25T17:46:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"02403ffe61a7087f5524936ac8562fe9cce4d401","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/incubator-metron/pulls/421/commits","review_comments_url":"https://api.github.com/repos/apache/incubator-metron/pulls/421/comments","review_comment_url":"https://api.github.com/repos/apache/incubator-metron/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/incubator-metron/issues/421/comments","statuses_url":"https://api.github.com/repos/apache/incubator-metron/statuses/7d00731ef00736cf868ff08bad0d607e43c97721","head":{"label":"justinleet:geo_mmdb","ref":"geo_mmdb","sha":"7d00731ef00736cf868ff08bad0d607e43c97721","user":{"login":"justinleet","id":5077341,"avatar_url":"https://avatars.githubusercontent.com/u/5077341?v=3","gravatar_id":"","url":"https://api.github.com/users/justinleet","html_url":"https://github.com/justinleet","followers_url":"https://api.github.com/users/justinleet/followers","following_url":"https://api.github.com/users/justinleet/following{/other_user}","gists_url":"https://api.github.com/users/justinleet/gists{/gist_id}","starred_url":"https://api.github.com/users/justinleet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinleet/subscriptions","organizations_url":"https://api.github.com/users/justinleet/orgs","repos_url":"https://api.github.com/users/justinleet/repos","events_url":"https://api.github.com/users/justinleet/events{/privacy}","received_events_url":"https://api.github.com/users/justinleet/received_events","type":"User","site_admin":false},"repo":{"id":62403147,"name":"incubator-metron","full_name":"justinleet/incubator-metron","owner":{"login":"justinleet","id":5077341,"avatar_url":"https://avatars.githubusercontent.com/u/5077341?v=3","gravatar_id":"","url":"https://api.github.com/users/justinleet","html_url":"https://github.com/justinleet","followers_url":"https://api.github.com/users/justinleet/followers","following_url":"https://api.github.com/users/justinleet/following{/other_user}","gists_url":"https://api.github.com/users/justinleet/gists{/gist_id}","starred_url":"https://api.github.com/users/justinleet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinleet/subscriptions","organizations_url":"https://api.github.com/users/justinleet/orgs","repos_url":"https://api.github.com/users/justinleet/repos","events_url":"https://api.github.com/users/justinleet/events{/privacy}","received_events_url":"https://api.github.com/users/justinleet/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/justinleet/incubator-metron","description":"Mirror of Apache Metron (Incubating)","fork":true,"url":"https://api.github.com/repos/justinleet/incubator-metron","forks_url":"https://api.github.com/repos/justinleet/incubator-metron/forks","keys_url":"https://api.github.com/repos/justinleet/incubator-metron/keys{/key_id}","collaborators_url":"https://api.github.com/repos/justinleet/incubator-metron/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/justinleet/incubator-metron/teams","hooks_url":"https://api.github.com/repos/justinleet/incubator-metron/hooks","issue_events_url":"https://api.github.com/repos/justinleet/incubator-metron/issues/events{/number}","events_url":"https://api.github.com/repos/justinleet/incubator-metron/events","assignees_url":"https://api.github.com/repos/justinleet/incubator-metron/assignees{/user}","branches_url":"https://api.github.com/repos/justinleet/incubator-metron/branches{/branch}","tags_url":"https://api.github.com/repos/justinleet/incubator-metron/tags","blobs_url":"https://api.github.com/repos/justinleet/incubator-metron/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/justinleet/incubator-metron/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/justinleet/incubator-metron/git/refs{/sha}","trees_url":"https://api.github.com/repos/justinleet/incubator-metron/git/trees{/sha}","statuses_url":"https://api.github.com/repos/justinleet/incubator-metron/statuses/{sha}","languages_url":"https://api.github.com/repos/justinleet/incubator-metron/languages","stargazers_url":"https://api.github.com/repos/justinleet/incubator-metron/stargazers","contributors_url":"https://api.github.com/repos/justinleet/incubator-metron/contributors","subscribers_url":"https://api.github.com/repos/justinleet/incubator-metron/subscribers","subscription_url":"https://api.github.com/repos/justinleet/incubator-metron/subscription","commits_url":"https://api.github.com/repos/justinleet/incubator-metron/commits{/sha}","git_commits_url":"https://api.github.com/repos/justinleet/incubator-metron/git/commits{/sha}","comments_url":"https://api.github.com/repos/justinleet/incubator-metron/comments{/number}","issue_comment_url":"https://api.github.com/repos/justinleet/incubator-metron/issues/comments{/number}","contents_url":"https://api.github.com/repos/justinleet/incubator-metron/contents/{+path}","compare_url":"https://api.github.com/repos/justinleet/incubator-metron/compare/{base}...{head}","merges_url":"https://api.github.com/repos/justinleet/incubator-metron/merges","archive_url":"https://api.github.com/repos/justinleet/incubator-metron/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/justinleet/incubator-metron/downloads","issues_url":"https://api.github.com/repos/justinleet/incubator-metron/issues{/number}","pulls_url":"https://api.github.com/repos/justinleet/incubator-metron/pulls{/number}","milestones_url":"https://api.github.com/repos/justinleet/incubator-metron/milestones{/number}","notifications_url":"https://api.github.com/repos/justinleet/incubator-metron/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/justinleet/incubator-metron/labels{/name}","releases_url":"https://api.github.com/repos/justinleet/incubator-metron/releases{/id}","deployments_url":"https://api.github.com/repos/justinleet/incubator-metron/deployments","created_at":"2016-07-01T15:41:46Z","updated_at":"2016-07-01T15:41:49Z","pushed_at":"2017-01-25T17:05:22Z","git_url":"git://github.com/justinleet/incubator-metron.git","ssh_url":"git@github.com:justinleet/incubator-metron.git","clone_url":"https://github.com/justinleet/incubator-metron.git","svn_url":"https://github.com/justinleet/incubator-metron","homepage":null,"size":47842,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"368e7ad63641e3c505ee6e3d335ecabfa60b8003","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":47607561,"name":"incubator-metron","full_name":"apache/incubator-metron","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/incubator-metron","description":"Mirror of Apache Metron (Incubating)","fork":false,"url":"https://api.github.com/repos/apache/incubator-metron","forks_url":"https://api.github.com/repos/apache/incubator-metron/forks","keys_url":"https://api.github.com/repos/apache/incubator-metron/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/incubator-metron/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/incubator-metron/teams","hooks_url":"https://api.github.com/repos/apache/incubator-metron/hooks","issue_events_url":"https://api.github.com/repos/apache/incubator-metron/issues/events{/number}","events_url":"https://api.github.com/repos/apache/incubator-metron/events","assignees_url":"https://api.github.com/repos/apache/incubator-metron/assignees{/user}","branches_url":"https://api.github.com/repos/apache/incubator-metron/branches{/branch}","tags_url":"https://api.github.com/repos/apache/incubator-metron/tags","blobs_url":"https://api.github.com/repos/apache/incubator-metron/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/incubator-metron/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/incubator-metron/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/incubator-metron/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/incubator-metron/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/incubator-metron/languages","stargazers_url":"https://api.github.com/repos/apache/incubator-metron/stargazers","contributors_url":"https://api.github.com/repos/apache/incubator-metron/contributors","subscribers_url":"https://api.github.com/repos/apache/incubator-metron/subscribers","subscription_url":"https://api.github.com/repos/apache/incubator-metron/subscription","commits_url":"https://api.github.com/repos/apache/incubator-metron/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/incubator-metron/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/incubator-metron/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/incubator-metron/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/incubator-metron/contents/{+path}","compare_url":"https://api.github.com/repos/apache/incubator-metron/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/incubator-metron/merges","archive_url":"https://api.github.com/repos/apache/incubator-metron/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/incubator-metron/downloads","issues_url":"https://api.github.com/repos/apache/incubator-metron/issues{/number}","pulls_url":"https://api.github.com/repos/apache/incubator-metron/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/incubator-metron/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/incubator-metron/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/incubator-metron/labels{/name}","releases_url":"https://api.github.com/repos/apache/incubator-metron/releases{/id}","deployments_url":"https://api.github.com/repos/apache/incubator-metron/deployments","created_at":"2015-12-08T08:00:05Z","updated_at":"2017-01-24T14:23:59Z","pushed_at":"2017-01-25T17:05:24Z","git_url":"git://github.com/apache/incubator-metron.git","ssh_url":"git@github.com:apache/incubator-metron.git","clone_url":"https://github.com/apache/incubator-metron.git","svn_url":"https://github.com/apache/incubator-metron","homepage":null,"size":48714,"stargazers_count":165,"watchers_count":165,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":157,"mirror_url":"git://git.apache.org/incubator-metron.git","open_issues_count":19,"forks":157,"open_issues":19,"watchers":165,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/421"},"html":{"href":"https://github.com/apache/incubator-metron/pull/421"},"issue":{"href":"https://api.github.com/repos/apache/incubator-metron/issues/421"},"comments":{"href":"https://api.github.com/repos/apache/incubator-metron/issues/421/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/421/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/incubator-metron/pulls/421/commits"},"statuses":{"href":"https://api.github.com/repos/apache/incubator-metron/statuses/7d00731ef00736cf868ff08bad0d607e43c97721"}}}}
{"actor":{"display_login":"cestella"}}