{"action":"created","issue":{"url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/350","repository_url":"https://api.github.com/repos/ppwwyyxx/tensorpack","labels_url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/350/labels{/name}","comments_url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/350/comments","events_url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/350/events","html_url":"https://github.com/ppwwyyxx/tensorpack/issues/350","id":246534914,"number":350,"title":"gradient stats","user":{"login":"a-maci","id":23125439,"avatar_url":"https://avatars2.githubusercontent.com/u/23125439?v=4","gravatar_id":"","url":"https://api.github.com/users/a-maci","html_url":"https://github.com/a-maci","followers_url":"https://api.github.com/users/a-maci/followers","following_url":"https://api.github.com/users/a-maci/following{/other_user}","gists_url":"https://api.github.com/users/a-maci/gists{/gist_id}","starred_url":"https://api.github.com/users/a-maci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-maci/subscriptions","organizations_url":"https://api.github.com/users/a-maci/orgs","repos_url":"https://api.github.com/users/a-maci/repos","events_url":"https://api.github.com/users/a-maci/events{/privacy}","received_events_url":"https://api.github.com/users/a-maci/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-07-29T17:41:20Z","updated_at":"2017-07-29T18:19:20Z","closed_at":null,"body":"I want to collect stats for gradients.\r\n\r\nI put stat collection code at couple places (method 1 and method 2 below). The stats coming from these two methods are very different. **Any idea why and which one is correct?** If both methods are incorrect then any suggestions of where to put the stat collection code for gradients?\r\n\r\n```\r\nMethod 1:\r\n@tf.RegisterGradient(\"FGGrad\")        #following the DoReFa style of code of monkey-patching\r\n     def grad_fg(op, x):           # x is the grad tensor, DoReFa quantizes this but we compute stat and return\r\n         add_activation_summary(x)\r\n         if bitG == 32:\r\n             return x\r\n\t\t pass     \r\n```                    \r\n\t\t\r\n```\r\nMethod 2:\r\nclass SummaryGradient(MapGradient):       #code in tensorpack/tensorpack/tfutils/gradproc.py\r\n    \"\"\"\r\n    Summary histogram, sparsity and RMS for each gradient variable.\r\n    \"\"\"\r\n\r\n    def __init__(self):\r\n        super(SummaryGradient, self).__init__(self._mapper)\r\n\r\n    def _mapper(self, grad, var):\r\n        name = var.op.name\r\n        if name not in _summaried_gradient:\r\n            _summaried_gradient.add(name)\r\n            tf.summary.histogram(name + '-grad', grad)\r\n            tf.summary.scalar(name + '-sparsity', tf.nn.zero_fraction(grad))\r\n            add_moving_summary(rms(grad, name=name + '/rms'))\r\n            add_moving_summary(tf.nn.zero_fraction(grad, name=name + '/sparsity'))\r\n        return grad\r\n```\t\r\n\t\r\nResults:\t\r\n```\r\n SummaryGradient/bn1/beta-sparsity: 0\r\n SummaryGradient/bn1/beta/sparsity/Mean: 0\r\n SummaryGradient/bn1/gamma-sparsity: 0\r\n SummaryGradient/bn1/gamma/sparsity/Mean: 0\r\n SummaryGradient/bn2/beta-sparsity: 0\r\n SummaryGradient/bn2/beta/sparsity/Mean: 0\r\n SummaryGradient/bn2/gamma-sparsity: 0\r\n SummaryGradient/bn2/gamma/sparsity/Mean: 0\r\n SummaryGradient/bn3/beta-sparsity: 0\r\n SummaryGradient/bn3/beta/sparsity/Mean: 0\r\n SummaryGradient/bn3/gamma-sparsity: 0\r\n SummaryGradient/bn3/gamma/sparsity/Mean: 0\r\n SummaryGradient/bn4/beta-sparsity: 0\r\n SummaryGradient/bn4/beta/sparsity/Mean: 0\r\n SummaryGradient/bn4/gamma-sparsity: 0\r\n SummaryGradient/bn4/gamma/sparsity/Mean: 0\r\n SummaryGradient/bn5/beta-sparsity: 0\r\n SummaryGradient/bn5/beta/sparsity/Mean: 0\r\n SummaryGradient/bn5/gamma-sparsity: 0\r\n SummaryGradient/bn5/gamma/sparsity/Mean: 0\r\n SummaryGradient/bn6/beta-sparsity: 0\r\n SummaryGradient/bn6/beta/sparsity/Mean: 0\r\n SummaryGradient/bn6/gamma-sparsity: 0\r\n SummaryGradient/bn6/gamma/sparsity/Mean: 2.1251e-07\r\n SummaryGradient/bn7/beta-sparsity: 0\r\n SummaryGradient/bn7/beta/sparsity/Mean: 0\r\n SummaryGradient/bn7/gamma-sparsity: 0\r\n SummaryGradient/bn7/gamma/sparsity/Mean: 0\r\n SummaryGradient/conv1/W-sparsity: 0\r\n SummaryGradient/conv1/W/sparsity/Mean: 0\r\n SummaryGradient/conv2/W-sparsity: 0\r\n SummaryGradient/conv2/W/sparsity/Mean: 1.0765e-07\r\n SummaryGradient/conv3/W-sparsity: 0\r\n SummaryGradient/conv3/W/sparsity/Mean: 0\r\n SummaryGradient/conv4/W-sparsity: 0\r\n SummaryGradient/conv4/W/sparsity/Mean: 2.4277e-09\r\n SummaryGradient/conv5/W-sparsity: 0\r\n SummaryGradient/conv5/W/sparsity/Mean: 0\r\n SummaryGradient/conv6/W-sparsity: 0\r\n SummaryGradient/conv6/W/sparsity/Mean: 7.4114e-09\r\n SummaryGradient/conv7/W-sparsity: 0\r\n SummaryGradient/conv7/W/sparsity/Mean: 1.1209e-08\r\n SummaryGradient/conv8/W-sparsity: 0\r\n SummaryGradient/conv8/W/sparsity/Mean: 3.6441e-10\r\n SummaryGradient/conv8/b-sparsity: 0\r\n SummaryGradient/conv8/b/sparsity/Mean: 0\r\n \r\n gradients/Identity_1_grad/activation-summary/gradients/pool2/output_grad/MaxPoolGrad_0-sparsity: 0.83238\r\n gradients/Identity_2_grad/activation-summary/gradients/bn3/FusedBatchNorm_grad/FusedBatchNormGrad_0-sparsity: 0\r\n gradients/Identity_3_grad/activation-summary/gradients/bn4/FusedBatchNorm_grad/FusedBatchNormGrad_0-sparsity: 0\r\n gradients/Identity_4_grad/activation-summary/gradients/pool3/output_grad/MaxPoolGrad_0-sparsity: 0.84553\r\n gradients/Identity_5_grad/activation-summary/gradients/bn6/FusedBatchNorm_grad/FusedBatchNormGrad_0-sparsity: 0\r\n gradients/Identity_6_grad/activation-summary/gradients/bn7/FusedBatchNorm_grad/FusedBatchNormGrad_0-sparsity: 0\r\n gradients/Identity_grad/activation-summary/gradients/pool1/output_grad/MaxPoolGrad_0-sparsity: 0.8218\r\n```\r\n \r\nSummaryGradients* are output based on Method 2 and gradients/Identity* are outputs based on method 1.\r\nHere is the application code:  \r\n \r\n```\r\n logits = (LinearWrap(image)\r\n              .Conv2D('conv1', 96, 11, stride=4, padding='VALID')\r\n              .apply(fg)\r\n              .MaxPooling('pool1', 3, 2, padding='VALID')\r\n              .BatchNorm('bn1')\r\n              .apply(activate)\r\n\r\n              .Conv2D('conv2', 256, 5, padding='SAME', split=2)\r\n              .apply(fg)\r\n              .MaxPooling('pool2', 3, 2, padding='VALID')\r\n              .BatchNorm('bn2')\r\n              .apply(activate)\r\n\r\n              .Conv2D('conv3', 384, 3)\r\n              .apply(fg)\r\n              .BatchNorm('bn3')\r\n              .apply(activate)\r\n\r\n              .Conv2D('conv4', 384, 3, split=2)\r\n              .apply(fg)\r\n              .BatchNorm('bn4')\r\n              .apply(activate)\r\n              \r\n              .Conv2D('conv5', 256, 3, split=2)\r\n              .apply(fg)\r\n              .MaxPooling('pool3', 3, 2, padding='VALID')\r\n              .BatchNorm('bn5')\r\n              .apply(activate)\r\n          \r\n              .Conv2D('conv6', 4096, 5, padding='VALID')\r\n              .apply(fg)\r\n              .BatchNorm('bn6')\r\n              .apply(activate)\r\n          \t  \r\n\t      .Conv2D('conv7', 4096, 1)\r\n\t      .apply(fg)\r\n              .BatchNorm('bn7')\r\n\r\n\t      .Conv2D('conv8', 1000, 1, stride=2, use_bias=True, nl=tf.identity)())\r\n```"},"comment":{"url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/comments/318848870","html_url":"https://github.com/ppwwyyxx/tensorpack/issues/350#issuecomment-318848870","issue_url":"https://api.github.com/repos/ppwwyyxx/tensorpack/issues/350","id":318848870,"user":{"login":"a-maci","id":23125439,"avatar_url":"https://avatars2.githubusercontent.com/u/23125439?v=4","gravatar_id":"","url":"https://api.github.com/users/a-maci","html_url":"https://github.com/a-maci","followers_url":"https://api.github.com/users/a-maci/followers","following_url":"https://api.github.com/users/a-maci/following{/other_user}","gists_url":"https://api.github.com/users/a-maci/gists{/gist_id}","starred_url":"https://api.github.com/users/a-maci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-maci/subscriptions","organizations_url":"https://api.github.com/users/a-maci/orgs","repos_url":"https://api.github.com/users/a-maci/repos","events_url":"https://api.github.com/users/a-maci/events{/privacy}","received_events_url":"https://api.github.com/users/a-maci/received_events","type":"User","site_admin":false},"created_at":"2017-07-29T18:19:20Z","updated_at":"2017-07-29T18:19:20Z","body":"Actually I need gradient stats for grads wrt variables(weights) and the grads that get back-propagated to the previous layer. \n\nGoal is to look at both the grad stats - grads that are involved with weight update and the grad stats that get back-prop'ed to the previous layer (to update weights in that layer). \n\nAny suggestions?\n\n> On Jul 29, 2017, at 11:00 AM, Yuxin Wu <notifications@github.com> wrote:\n> \n> Your method summarizes gradients w.r.t layer outputs, not variables.\n> \n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub, or mute the thread.\n> \n"}}
{"actor":{"display_login":"a-maci"}}