{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/PHEDEX/issues/1085","repository_url":"https://api.github.com/repos/dmwm/PHEDEX","labels_url":"https://api.github.com/repos/dmwm/PHEDEX/issues/1085/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/PHEDEX/issues/1085/comments","events_url":"https://api.github.com/repos/dmwm/PHEDEX/issues/1085/events","html_url":"https://github.com/dmwm/PHEDEX/issues/1085","id":227177075,"number":1085,"title":"Pass PhEDEx metadata to FTS tasks for hadoop monitoring.","user":{"login":"nataliaratnikova","id":4698564,"avatar_url":"https://avatars1.githubusercontent.com/u/4698564?v=3","gravatar_id":"","url":"https://api.github.com/users/nataliaratnikova","html_url":"https://github.com/nataliaratnikova","followers_url":"https://api.github.com/users/nataliaratnikova/followers","following_url":"https://api.github.com/users/nataliaratnikova/following{/other_user}","gists_url":"https://api.github.com/users/nataliaratnikova/gists{/gist_id}","starred_url":"https://api.github.com/users/nataliaratnikova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nataliaratnikova/subscriptions","organizations_url":"https://api.github.com/users/nataliaratnikova/orgs","repos_url":"https://api.github.com/users/nataliaratnikova/repos","events_url":"https://api.github.com/users/nataliaratnikova/events{/privacy}","received_events_url":"https://api.github.com/users/nataliaratnikova/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vkuznet","id":125138,"avatar_url":"https://avatars3.githubusercontent.com/u/125138?v=3","gravatar_id":"","url":"https://api.github.com/users/vkuznet","html_url":"https://github.com/vkuznet","followers_url":"https://api.github.com/users/vkuznet/followers","following_url":"https://api.github.com/users/vkuznet/following{/other_user}","gists_url":"https://api.github.com/users/vkuznet/gists{/gist_id}","starred_url":"https://api.github.com/users/vkuznet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkuznet/subscriptions","organizations_url":"https://api.github.com/users/vkuznet/orgs","repos_url":"https://api.github.com/users/vkuznet/repos","events_url":"https://api.github.com/users/vkuznet/events{/privacy}","received_events_url":"https://api.github.com/users/vkuznet/received_events","type":"User","site_admin":false},"assignees":[{"login":"vkuznet","id":125138,"avatar_url":"https://avatars3.githubusercontent.com/u/125138?v=3","gravatar_id":"","url":"https://api.github.com/users/vkuznet","html_url":"https://github.com/vkuznet","followers_url":"https://api.github.com/users/vkuznet/followers","following_url":"https://api.github.com/users/vkuznet/following{/other_user}","gists_url":"https://api.github.com/users/vkuznet/gists{/gist_id}","starred_url":"https://api.github.com/users/vkuznet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkuznet/subscriptions","organizations_url":"https://api.github.com/users/vkuznet/orgs","repos_url":"https://api.github.com/users/vkuznet/repos","events_url":"https://api.github.com/users/vkuznet/events{/privacy}","received_events_url":"https://api.github.com/users/vkuznet/received_events","type":"User","site_admin":false},{"login":"nataliaratnikova","id":4698564,"avatar_url":"https://avatars1.githubusercontent.com/u/4698564?v=3","gravatar_id":"","url":"https://api.github.com/users/nataliaratnikova","html_url":"https://github.com/nataliaratnikova","followers_url":"https://api.github.com/users/nataliaratnikova/followers","following_url":"https://api.github.com/users/nataliaratnikova/following{/other_user}","gists_url":"https://api.github.com/users/nataliaratnikova/gists{/gist_id}","starred_url":"https://api.github.com/users/nataliaratnikova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nataliaratnikova/subscriptions","organizations_url":"https://api.github.com/users/nataliaratnikova/orgs","repos_url":"https://api.github.com/users/nataliaratnikova/repos","events_url":"https://api.github.com/users/nataliaratnikova/events{/privacy}","received_events_url":"https://api.github.com/users/nataliaratnikova/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2017-05-08T21:16:14Z","updated_at":"2017-05-24T15:55:37Z","closed_at":null,"body":"Valentin requests that PhEDEx metadata are propagated to FTS.\r\nThe FTS logs will appear on HDFS, then he can run spark job over FTS logs to extract some context for the global transfers monitoring. \r\n\r\nHere are the metadata Valentin is interested in: \r\n\r\n- attach PhEDEx request metadata to FTS request\r\n  - metadata should include\r\n    - issuer of the PhEDEx request\r\n    - timestamp of the request\r\n    - DN of the user of the request\r\n    - details of the request, e.g. transfer dataset /a/b/c from site A to site B\r\n\r\nHow such metadata should be structure is a subject of FTS metadata\r\nformat/schema. I can't tell you more and you'll need to coordinate with FTS\r\ndevelopers. It would be nice to use the same schema among different CMS FTS\r\nusers, e.g. PhEDEx and ASO. This is why I CC'ed Diego since according to him\r\nand, in fact, the FTS logs already carry ASO metadata, e.g.\r\njob_metadata.issuer=ASO.\r\n\r\nSince we're talking about meta-data its structure may change/adjusted over\r\ntime.\r\n\r\nHere are typical examples:\r\n- if I'm placing a request to PhEDEx then metadata would be something like:\r\n{\r\n \"issuer\": \"PHEDEX/user\",\r\n \"time\": 123456,\r\n \"client\": \"PHEDEX-client-version-1\",\r\n \"dn\": \"/my/DN/here\",\r\n \"request\": {\r\n    \"dataset\":\"/a/b/c\",\r\n    \"source\": \"T1_XXX\",\r\n    \"destination\": \"T2_XXX\"\r\n }\r\n}\r\n\r\n- if DDM placing a request you'll create\r\n{\r\n \"issuer\":\"PHEDEX/DDM\",\r\n ...\r\n}"},"comment":{"url":"https://api.github.com/repos/dmwm/PHEDEX/issues/comments/303768505","html_url":"https://github.com/dmwm/PHEDEX/issues/1085#issuecomment-303768505","issue_url":"https://api.github.com/repos/dmwm/PHEDEX/issues/1085","id":303768505,"user":{"login":"vkuznet","id":125138,"avatar_url":"https://avatars3.githubusercontent.com/u/125138?v=3","gravatar_id":"","url":"https://api.github.com/users/vkuznet","html_url":"https://github.com/vkuznet","followers_url":"https://api.github.com/users/vkuznet/followers","following_url":"https://api.github.com/users/vkuznet/following{/other_user}","gists_url":"https://api.github.com/users/vkuznet/gists{/gist_id}","starred_url":"https://api.github.com/users/vkuznet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkuznet/subscriptions","organizations_url":"https://api.github.com/users/vkuznet/orgs","repos_url":"https://api.github.com/users/vkuznet/repos","events_url":"https://api.github.com/users/vkuznet/events{/privacy}","received_events_url":"https://api.github.com/users/vkuznet/received_events","type":"User","site_admin":false},"created_at":"2017-05-24T15:55:37Z","updated_at":"2017-05-24T15:55:37Z","body":"I don't mind, but data-wise it is redundant. If user know the dataset\n(s)he may construct a filter simply using primds,procds,tier parts.\n\nThe only difference for users would be either use a string\n/a/b/c or parts.\n\nBut storage wise (if we do really care) additional strings (dataset)\nwill add some overhead.\n\nOn  0, Stefano Belforte <notifications@github.com> wrote:\n> Hi @vkuznet, what if we report 4 things, the full dataset name and the three pieces,\n> would that be a problem ? In the spirit that IIUC we are encouraging every operator,\n> coordinator, user to build their favorite ES search of kibana/grafana dashboard,\n> I would not mind looking also at \"convenience\".\n> \n> \n> -- \n> You are receiving this because you were mentioned.\n> Reply to this email directly or view it on GitHub:\n> https://github.com/dmwm/PHEDEX/issues/1085#issuecomment-303764910\n"}}
{"actor":{"display_login":"vkuznet"}}