{"action":"created","issue":{"url":"https://api.github.com/repos/GeoNode/geonode/issues/3137","repository_url":"https://api.github.com/repos/GeoNode/geonode","labels_url":"https://api.github.com/repos/GeoNode/geonode/issues/3137/labels{/name}","comments_url":"https://api.github.com/repos/GeoNode/geonode/issues/3137/comments","events_url":"https://api.github.com/repos/GeoNode/geonode/issues/3137/events","html_url":"https://github.com/GeoNode/geonode/issues/3137","id":237890757,"number":3137,"title":"GNIP: GeoNode monitoring","user":{"login":"cezio","id":116021,"avatar_url":"https://avatars1.githubusercontent.com/u/116021?v=3","gravatar_id":"","url":"https://api.github.com/users/cezio","html_url":"https://github.com/cezio","followers_url":"https://api.github.com/users/cezio/followers","following_url":"https://api.github.com/users/cezio/following{/other_user}","gists_url":"https://api.github.com/users/cezio/gists{/gist_id}","starred_url":"https://api.github.com/users/cezio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cezio/subscriptions","organizations_url":"https://api.github.com/users/cezio/orgs","repos_url":"https://api.github.com/users/cezio/repos","events_url":"https://api.github.com/users/cezio/events{/privacy}","received_events_url":"https://api.github.com/users/cezio/received_events","type":"User","site_admin":false},"labels":[{"id":7008294,"url":"https://api.github.com/repos/GeoNode/geonode/labels/enhancement","name":"enhancement","color":"02d7e1","default":true},{"id":6356197,"url":"https://api.github.com/repos/GeoNode/geonode/labels/gnip","name":"gnip","color":"d7e102","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-06-22T15:47:54Z","updated_at":"2017-07-02T13:13:57Z","closed_at":null,"body":"GNIP: geonode monitoring\r\n========================\r\n\r\nOverview\r\n--------\r\nGeoNode monitoring is an infrastructure to extract and present information on installation's health status and resource (layers, maps, documents) usage. Monitoring is an additional Django/GeoNode application which will:\r\n * collect data (events) from software components\r\n * calculate usage statistics (metrics)\r\n * display it in human-friendly form\r\n * provide a way to set fault thresholds and send alerts in case of reaching those\r\nGeoNode monitoring functionality is not limited to plain GeoNode, but it will also collect data from accompanying GeoServer instances, and from operating system on hardware resources usage. \r\n\r\nProposed by\r\n===========\r\n\r\nCezary Statkiewicz [GeoSolutions](http://www.geo-solutions.it/)\r\n\r\n \r\nAssigned to release\r\n-------------------\r\n \r\nNone yet.\r\n \r\nMotivation\r\n----------\r\n\r\nGeoNode lacks information on system health and resources usage. Data needed to deduce such information could be extracted in various ways (client-side analytics, log parsing, external monitoring), but each way has it's drawbacks, and none would show full picture. Also, existing or previous attempts ([GeoHealthCheck](https://github.com/geopython/GeoHealthCheck), [geonode-monitor](https://github.com/GFDRR/geonode-monitor) are quite incomplete and focus on measuring external visibility/state only.\r\n\r\nThis proposal introduces contrib monitoring application, which would provide insights into actual usage of data and do health check of underlying system. This would be beneficial especially for smaller installations, where there are no resources for bigger monitoring infrastructure. Application should be optional, although there are few integration points in GeoNode core.\r\n\r\nNote, this is not a replacement for full-fledge monitoring systems like Zabbix or Nagios. While GeoNode monitoring can work in stand-alone mode, it could be also integrated with 3rd party systems as well, as a data source.\r\n\r\nProposal\r\n--------\r\n\r\nGN monitoring has two main tasks:\r\n * collecting data from probes,\r\n * calculate usage stats and present them.\r\n\r\nCollecting data starts with recording request with context: besides basic http context, it should also contain information about used resources, service which was used, more detailed information on client etc. Similar data structure is already available in [GeoServer](http://docs.geoserver.org/stable/en/user/extensions/monitoring/audit.html#log-files). \r\n\r\nData collection may be implemented in several ways. By default, data will be pulled from probes, although there should be a way for probes to push data to collector. Also, monitoring should be ready to handle data exchange through AMQP, which will be future default way of notifications handling. Data collection can be performed periodically or persistently in real time.\r\n\r\nStatistics calculation is performed periodically, into fixed length periods with aggregated data. Aggregated data would contain general statistics and per-resource statistics, so presentation layer can present system status from overview to layer-level without much of recalculation.\r\n\r\n## Architecture overview\r\n\r\n![geonode monitoring architecture ](https://user-images.githubusercontent.com/116021/27441875-0be66012-576f-11e7-9599-cdf6612f24cf.png)\r\n\r\nMonitoring is composed of several components, described below. Note, that those are logical units. Code should reside in ``geonode.contrib.monitoring`` module as a Django application.\r\n \r\n### GeoNode probes\r\n\r\nProbes are points of integration in GeoNode core, which will record it's activity. This is build with:\r\n * middleware - requests are marked with start/end time, and after view is processed, request is recorded to database with context information,\r\n * views - core views for layers, maps and documents should mark request with resources affected by this request.\r\n\r\n### GeoServer probes\r\n\r\nGeoServer provides Monitoring/Audit API, which can be used. GeoServer improvements will be handled outside this GNIP.\r\n\r\n### System-level probes\r\n\r\nGeoNode monitoring can collect system-level data (cpu usage, memory usage, disks usage). System-level data can be extracted by reading system indicators from GeoNode and GeoServer processes and expose with Status API in GeoServer. GeoNode would have Expose API, which is a set of views which will present system-level data at the moment of request. \r\n\r\n### Collector\r\n\r\nThis is the core element of monitoring, because it connects both main functionalities. Collector provides following facilities:\r\n * receive or acquire raw data from probes\r\n   This can be any of following:\r\n   * actively query (over HTTP or other transport) probes for data (pull),\r\n   * expose view reachable from probes, which will report their data to it (push),\r\n   * hybrid, utilizing incoming AMQP infrastructure, probes will publish events to queue, collector will act as a consumer and collect it from broker,\r\n * normalize, calculate and store metrics,\r\n * expose metrics for status UI and notifications.\r\n \r\nCollector can be run as:\r\n * periodical command, pulling data from probes (implemented as ``collect_metrics``),\r\n * long-running process (as a AMQP consumer),\r\n * as a view, passively receive data from probes.\r\n\r\n### Dashboard/Status UI\r\n\r\n![monitoring](https://user-images.githubusercontent.com/116021/27442945-f4b7345e-5771-11e7-93fb-804c0c80e53f.png)\r\n\r\nStatus UI is a set of views and client-side application that will present metrics. User should get main indicators in simplified form (to judge if system is working properly), and have a way to see more detailed data few clicks away. Status UI should also provide a way to configure notifications and collector.\r\n\r\n### Notifications\r\n\r\nMonitoring Notifications shouldn't be confused with GeoNode notifications app, which is a separate entity. However, Monitoring Notifications will use general notifications as a backend for sending alerts. User should be able to configure thresholds for certain indicators, which can consist of several metrics. Notifications will check metrics for each indicator after each metrics calculation, and send alerts in alarm conditions. \r\n\r\n### Beacon\r\n\r\nBeacon is an API that exposes current status of GeoNode for external monitoring.\r\n\r\n## Data model\r\n\r\n![monitoring data model](https://user-images.githubusercontent.com/116021/27441994-538d4d86-576f-11e7-8bbf-14763c469079.png)\r\n\r\n### Collected data \r\n\r\nThere are different types of probes and data they provide. Basically, two base types are distinguished, service  type and host type. Service type provides stream of events from service (GeoNode, GeoSever). Stream can contain data from past or be provided in real-time. Host type probe provides only data for current moment.\r\n\r\nCollector will get following data from probes:\r\n * request with context (client location, affected resources, timing, errors),\r\n * exception information for errors occured during request procesing,\r\n * system-level data.\r\n\r\nData will be aggregated and stored in fixed-lenght periods. For near-present data, periods should be 1-5 minutes, for older data periods could be longer.\r\n\r\n### Metrics\r\n\r\nMetric is an aggregated value for specific indicator. There are three types of metrics:\r\n * value (where we store value and count occurences within a period of time, for example: request method)\r\n * rate (where we store average rate within a period of time)\r\n * count (where we count occurences within specific period of time)\r\n\r\nWhile metric types seems similar, they are handled differently during stats calculation and presentation.\r\n\r\nA metric has several main properties:\r\n * valid period (``valid_from``, ``valid_to``),\r\n * service, for which it is calculated,\r\n * a name, like ``request.ip``, or ``request.count`` (which is defined in MetricName model),\r\n * numeric value.\r\nAdditionally, metric can be associated to:\r\n * specific resource (layer, map, document), \r\n * OWS service type (names stored in OWSService model),\r\n * free-text label.\r\n\r\nFollowing metric organization allows to have different levels of granularity (per-service, per-metric, per-resource etc) and further aggregation (increased intervals, aggregating total request count from sum of requests to specific resources etc)."},"comment":{"url":"https://api.github.com/repos/GeoNode/geonode/issues/comments/312491184","html_url":"https://github.com/GeoNode/geonode/issues/3137#issuecomment-312491184","issue_url":"https://api.github.com/repos/GeoNode/geonode/issues/3137","id":312491184,"user":{"login":"safezpa","id":15123869,"avatar_url":"https://avatars2.githubusercontent.com/u/15123869?v=3","gravatar_id":"","url":"https://api.github.com/users/safezpa","html_url":"https://github.com/safezpa","followers_url":"https://api.github.com/users/safezpa/followers","following_url":"https://api.github.com/users/safezpa/following{/other_user}","gists_url":"https://api.github.com/users/safezpa/gists{/gist_id}","starred_url":"https://api.github.com/users/safezpa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/safezpa/subscriptions","organizations_url":"https://api.github.com/users/safezpa/orgs","repos_url":"https://api.github.com/users/safezpa/repos","events_url":"https://api.github.com/users/safezpa/events{/privacy}","received_events_url":"https://api.github.com/users/safezpa/received_events","type":"User","site_admin":false},"created_at":"2017-07-02T13:13:57Z","updated_at":"2017-07-02T13:13:57Z","body":"I think may be ELK will be another nice solution for m\r\n![_2017-07-02t13-11-17 154z](https://user-images.githubusercontent.com/15123869/27770101-5862a8ac-5f6b-11e7-900d-02ad7671063a.png)\r\nonitoring ."}}
{"actor":{"display_login":"safezpa"}}