{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-metron/issues/438","repository_url":"https://api.github.com/repos/apache/incubator-metron","labels_url":"https://api.github.com/repos/apache/incubator-metron/issues/438/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-metron/issues/438/comments","events_url":"https://api.github.com/repos/apache/incubator-metron/issues/438/events","html_url":"https://github.com/apache/incubator-metron/pull/438","id":205250071,"number":438,"title":"METRON-686 Record Rule Set that Fired During Threat Triage","user":{"login":"nickwallen","id":2475409,"avatar_url":"https://avatars.githubusercontent.com/u/2475409?v=3","gravatar_id":"","url":"https://api.github.com/users/nickwallen","html_url":"https://github.com/nickwallen","followers_url":"https://api.github.com/users/nickwallen/followers","following_url":"https://api.github.com/users/nickwallen/following{/other_user}","gists_url":"https://api.github.com/users/nickwallen/gists{/gist_id}","starred_url":"https://api.github.com/users/nickwallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickwallen/subscriptions","organizations_url":"https://api.github.com/users/nickwallen/orgs","repos_url":"https://api.github.com/users/nickwallen/repos","events_url":"https://api.github.com/users/nickwallen/events{/privacy}","received_events_url":"https://api.github.com/users/nickwallen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2017-02-03T19:39:18Z","updated_at":"2017-02-22T19:07:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-metron/pulls/438","html_url":"https://github.com/apache/incubator-metron/pull/438","diff_url":"https://github.com/apache/incubator-metron/pull/438.diff","patch_url":"https://github.com/apache/incubator-metron/pull/438.patch"},"body":"The goal of threat triage is to prioritize the alerts that pose the greatest threat and thus need urgent attention.  To perform threat triage, a set of rules are applied to each message.  Each rule has a predicate to determine if the rule applies or not.  The threat score from each applied rule is aggregated into a single threat triage score that can be used to prioritize high risk threats.\r\n\r\n### Changes\r\n* All of the threat triage rules that were applied to a message during threat triage are captured and recorded.\r\n* A new 'reason' field was added to the threat triage rules to allow a rule author to provide contextual information to assist in actioning a threat. \r\n\r\n### Implementation\r\nThe `ThreatTriageProcessor` attempts to apply each `RiskLevelRule` to a message being triaged.  If the rule's predicate returns true, the rule is applied to the message resulting in a `RuleScore`.  The `RuleScore` represents the portion of the threat score contributed by a single rule.  Each of the `RuleScore`s are aggregated into an overall `ThreatScore`, which captures the outcome of the threat triage process.  This `ThreatScore` is then attached to the message.\r\n\r\n### Example\r\n\r\nPreviously the result of threat triage was a single threat score represented as a double.  \r\n```\r\n\"threat.triage.level\": 10.0\r\n```\r\n\r\nThe result of threat triage now provides greater information about the triage process.  The rule set...\r\n```\r\n    \"triageConfig\" : {\r\n      \"riskLevelRules\" : [ {\r\n        \"name\" : \"Abnormal DNS Port\",\r\n        \"rule\" : \"source.type == \\\"bro\\\" and protocol == \\\"dns\\\" and ip_dst_port != 53\",\r\n        \"score\" : 10.0,\r\n        \"reason\" : \"FORMAT(\\\"Abnormal DNS Port: expected: 53, found: %s:%d\\\", ip_dst_addr, ip_dst_port)\"\r\n      } ],\r\n      \"aggregator\" : \"MAX\",\r\n      \"aggregationConfig\" : { }\r\n    }\r\n```\r\n\r\nproduces a result like...\r\n```\r\n\"threat.triage.level\":{\r\n   \"score\":10.0,\r\n   \"rules\":[\r\n      { \r\n         \"name\":\"Abnormal DNS Port\",\r\n         \"comment\":null\r\n         \"score\":10.0,\r\n         \"reason\":\"Abnormal DNS Port: expected: 53, found: 224.0.0.251:5353\",\r\n      }\r\n   ]\r\n}\r\n```\r\n\r\n### Testing\r\nSetup a Metron environment with the default Bro sensor running.  The following commands can be executed in the Stellar REPL.\r\n\r\n```\r\n# fetch the current config\r\nconf := CONFIG_GET(\"ENRICHMENT\", \"bro\")\r\n\r\n# triage all bro telemetry\r\nis_alert := source.type == 'bro'\r\nconf := ENRICHMENT_STELLAR_TRANSFORM_ADD(conf, 'ENRICHMENT', SHELL_VARS2MAP('is_alert'))\r\n\r\n# create a rule that looks for DNS running on non-standard ports\r\nrule := source.type == \"bro\" and protocol == \"dns\" and ip_dst_port != 53\r\ntriage := { \"name\":\"Abnormal DNS Port\", \"rule\": SHELL_GET_EXPRESSION('rule'), \"reason\": SHELL_GET_EXPRESSION('reason'), \"score\":\"10\" }\r\nconf := THREAT_TRIAGE_ADD(conf, [triage])\r\n\r\n# save the configuration\r\nCONFIG_PUT(\"ENRICHMENT\", conf, \"bro\")\r\n```\r\n\r\nMonitor the messages being pushed to the 'indexing' topic in Kafka.  You should expect to see a message where the 'threat.triage.level' is like the example shown above.\r\n\r\nPrinting the threat triage rule in the REPL is going to look something like this.\r\n```\r\n> THREAT_TRIAGE_PRINT(conf)\r\n╔═══════════════════╤═════════╤══════════════════════════════════════════════════════════════════╤═══════╤═══════════════════════════════════════════════════════════════════════════════════╗\r\n║ Name              │ Comment │ Triage Rule                                                      │ Score │ Reason                                                                            ║\r\n╠═══════════════════╪═════════╪══════════════════════════════════════════════════════════════════╪═══════╪═══════════════════════════════════════════════════════════════════════════════════╣\r\n║ Abnormal DNS Port │         │ source.type == \"bro\" and protocol == \"dns\" and ip_dst_port != 53 │ 10    │ FORMAT(\"Abnormal DNS Port: expected: 53, found: %s:%d\", ip_dst_addr, ip_dst_port) ║\r\n╚═══════════════════╧═════════╧══════════════════════════════════════════════════════════════════╧═══════╧═══════════════════════════════════════════════════════════════════════════════════╝\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/incubator-metron/issues/comments/281769050","html_url":"https://github.com/apache/incubator-metron/pull/438#issuecomment-281769050","issue_url":"https://api.github.com/repos/apache/incubator-metron/issues/438","id":281769050,"user":{"login":"nickwallen","id":2475409,"avatar_url":"https://avatars.githubusercontent.com/u/2475409?v=3","gravatar_id":"","url":"https://api.github.com/users/nickwallen","html_url":"https://github.com/nickwallen","followers_url":"https://api.github.com/users/nickwallen/followers","following_url":"https://api.github.com/users/nickwallen/following{/other_user}","gists_url":"https://api.github.com/users/nickwallen/gists{/gist_id}","starred_url":"https://api.github.com/users/nickwallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickwallen/subscriptions","organizations_url":"https://api.github.com/users/nickwallen/orgs","repos_url":"https://api.github.com/users/nickwallen/repos","events_url":"https://api.github.com/users/nickwallen/events{/privacy}","received_events_url":"https://api.github.com/users/nickwallen/received_events","type":"User","site_admin":false},"created_at":"2017-02-22T19:07:16Z","updated_at":"2017-02-22T19:07:16Z","body":"@ottobackwards @cestella @justinleet  - Let me know what you guys think on my previous comment about how to move forward on this PR.  \r\n\r\nI think we have a rough outline of the \"flattening\" stuff, but how should we move forward on this PR specifically.  See options outlined in my previous comment."}}
{"actor":{"display_login":"nickwallen"}}