{"action":"created","issue":{"url":"https://api.github.com/repos/numba/numba/issues/2454","repository_url":"https://api.github.com/repos/numba/numba","labels_url":"https://api.github.com/repos/numba/numba/issues/2454/labels{/name}","comments_url":"https://api.github.com/repos/numba/numba/issues/2454/comments","events_url":"https://api.github.com/repos/numba/numba/issues/2454/events","html_url":"https://github.com/numba/numba/issues/2454","id":241029333,"number":2454,"title":"nopython mode generator function does NOT update structured scalar argument","user":{"login":"complyue","id":15646573,"avatar_url":"https://avatars1.githubusercontent.com/u/15646573?v=3","gravatar_id":"","url":"https://api.github.com/users/complyue","html_url":"https://github.com/complyue","followers_url":"https://api.github.com/users/complyue/followers","following_url":"https://api.github.com/users/complyue/following{/other_user}","gists_url":"https://api.github.com/users/complyue/gists{/gist_id}","starred_url":"https://api.github.com/users/complyue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/complyue/subscriptions","organizations_url":"https://api.github.com/users/complyue/orgs","repos_url":"https://api.github.com/users/complyue/repos","events_url":"https://api.github.com/users/complyue/events{/privacy}","received_events_url":"https://api.github.com/users/complyue/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"sklam","id":1929845,"avatar_url":"https://avatars2.githubusercontent.com/u/1929845?v=3","gravatar_id":"","url":"https://api.github.com/users/sklam","html_url":"https://github.com/sklam","followers_url":"https://api.github.com/users/sklam/followers","following_url":"https://api.github.com/users/sklam/following{/other_user}","gists_url":"https://api.github.com/users/sklam/gists{/gist_id}","starred_url":"https://api.github.com/users/sklam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklam/subscriptions","organizations_url":"https://api.github.com/users/sklam/orgs","repos_url":"https://api.github.com/users/sklam/repos","events_url":"https://api.github.com/users/sklam/events{/privacy}","received_events_url":"https://api.github.com/users/sklam/received_events","type":"User","site_admin":false},"assignees":[{"login":"sklam","id":1929845,"avatar_url":"https://avatars2.githubusercontent.com/u/1929845?v=3","gravatar_id":"","url":"https://api.github.com/users/sklam","html_url":"https://github.com/sklam","followers_url":"https://api.github.com/users/sklam/followers","following_url":"https://api.github.com/users/sklam/following{/other_user}","gists_url":"https://api.github.com/users/sklam/gists{/gist_id}","starred_url":"https://api.github.com/users/sklam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklam/subscriptions","organizations_url":"https://api.github.com/users/sklam/orgs","repos_url":"https://api.github.com/users/sklam/repos","events_url":"https://api.github.com/users/sklam/events{/privacy}","received_events_url":"https://api.github.com/users/sklam/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2017-07-06T17:25:46Z","updated_at":"2017-07-07T20:50:05Z","closed_at":null,"body":"I guess this has sth to do with the implementation details with good reason, but I'd argue the behaviours should be more consistent:\r\n\r\n|  | non-generator | generator |\r\n| --- | --- | --- |\r\n| np.ndarray arg | ✅ will update | ✅ will update |\r\n| np.record arg | ✅ will update | ❌ no update |\r\n\r\nversion 0.33.0\r\n\r\nI need the nopython/nogil mode function to be a generator so that it stays away from python interpreter until a rolling buffer (represented by several fixed ndarray) is filled up, then yield out some intermediate data, for the calling python (slow object mode) code to handle all the data, empty the rolling buffer, then resume the nopython mode generator function (again in performance). A  structured scalar argument to the nopython/nogil function is used to pass range info of the rolling buffer (3 value for a specific case) into and out of the nopython context, since both inside/outside code need such info to address data correctly. \r\n\r\nI can replace the structured scalar arg with a few (at least one array per distinct dtype) ndarray args as value containers for this purpose, but that harms a lot to readability and brain health, so hope this is not too hard to fix.\r\n\r\nstructured scalar proof code:\r\n```Python\r\nimport numba as nb\r\nimport numpy as np\r\n\r\npoint_dtype = np.dtype([\r\n    ('x', np.float32),\r\n    ('y', np.float32),\r\n])\r\n\r\n\r\n@nb.njit\r\ndef mv(pt, v, steps):\r\n    print(\r\n        ' --- moving point x=', pt.x, ', y=', pt.y,\r\n        'with vector x=', v.x, ', y=', v.y,\r\n        'for ', steps, 'steps'\r\n    )\r\n    d = np.sqrt(v.x ** 2 + v.y ** 2)\r\n    for i in range(steps):\r\n        pt.x += v.x\r\n        pt.y += v.y\r\n        print('  -- step', i, 'to (', pt.x, ',', pt.y, ')')\r\n\r\n\r\n@nb.njit\r\ndef mv_yielding_d(pt, v, steps):\r\n    print(\r\n        ' --- moving point x=', pt.x, ', y=', pt.y,\r\n        'with vector x=', v.x, ', y=', v.y,\r\n        'for ', steps, 'steps'\r\n    )\r\n    d = np.sqrt(v.x ** 2 + v.y ** 2)\r\n    for i in range(steps):\r\n        pt.x += v.x\r\n        pt.y += v.y\r\n        print('  -- step', i, 'to (', pt.x, ',', pt.y, ')')\r\n        yield d\r\n\r\n\r\npt = np.rec.array([(0, 0)], point_dtype)[0]\r\nv = np.rec.array([(3, 7)], point_dtype)[0]\r\n\r\nprint(' *** initial point at', pt)\r\n\r\nmv(pt, v, 5)\r\n\r\nprint(' *** moved to', pt)\r\n\r\nfor d in mv_yielding_d(pt, v, 5) or ():\r\n    print(\r\n        '  ** moved distance=', d,\r\n        'now at', pt\r\n    )\r\n\r\nprint(' *** final point at', pt)\r\n\r\n```\r\nwhich outputs:\r\n```pycon\r\n *** initial point at (0.0, 0.0)\r\n --- moving point x= 0.0 , y= 0.0 with vector x= 3.0 , y= 7.0 for  5 steps\r\n  -- step 0 to ( 3.0 , 7.0 )\r\n  -- step 1 to ( 6.0 , 14.0 )\r\n  -- step 2 to ( 9.0 , 21.0 )\r\n  -- step 3 to ( 12.0 , 28.0 )\r\n  -- step 4 to ( 15.0 , 35.0 )\r\n *** moved to (15.0, 35.0)\r\n --- moving point x= 15.0 , y= 35.0 with vector x= 3.0 , y= 7.0 for  5 steps\r\n  -- step 0 to ( 18.0 , 42.0 )\r\n  ** moved distance= 7.6157732009887695 now at (15.0, 35.0)\r\n  -- step 1 to ( 21.0 , 49.0 )\r\n  ** moved distance= 7.6157732009887695 now at (15.0, 35.0)\r\n  -- step 2 to ( 24.0 , 56.0 )\r\n  ** moved distance= 7.6157732009887695 now at (15.0, 35.0)\r\n  -- step 3 to ( 27.0 , 63.0 )\r\n  ** moved distance= 7.6157732009887695 now at (15.0, 35.0)\r\n  -- step 4 to ( 30.0 , 70.0 )\r\n  ** moved distance= 7.6157732009887695 now at (15.0, 35.0)\r\n *** final point at (15.0, 35.0)\r\n```\r\n\r\nndarray proof code:\r\n```Python\r\nimport numba as nb\r\nimport numpy as np\r\n\r\n\r\n@nb.njit\r\ndef mv(pt, v, steps):\r\n    print(\r\n        ' --- moving point x=', pt[0], ', y=', pt[1],\r\n        'with vector x=', v[0], ', y=', v[1],\r\n        'for ', steps, 'steps'\r\n    )\r\n    d = np.sqrt(v[0] ** 2 + v[1] ** 2)\r\n    for i in range(steps):\r\n        pt[0] += v[0]\r\n        pt[1] += v[1]\r\n        print('  -- step', i, 'to (', pt[0], ',', pt[1], ')')\r\n\r\n\r\n@nb.njit\r\ndef mv_yielding_d(pt, v, steps):\r\n    print(\r\n        ' --- moving point x=', pt[0], ', y=', pt[1],\r\n        'with vector x=', v[0], ', y=', v[1],\r\n        'for ', steps, 'steps'\r\n    )\r\n    d = np.sqrt(v[0] ** 2 + v[1] ** 2)\r\n    for i in range(steps):\r\n        pt[0] += v[0]\r\n        pt[1] += v[1]\r\n        print('  -- step', i, 'to (', pt[0], ',', pt[1], ')')\r\n        yield d\r\n\r\n\r\npt = np.zeros(2, np.float32)\r\nv = np.array([3, 7], np.float32)\r\n\r\nprint(' *** initial point at', pt)\r\n\r\nmv(pt, v, 5)\r\n\r\nprint(' *** moved to', pt)\r\n\r\nfor d in mv_yielding_d(pt, v, 5) or ():\r\n    print(\r\n        '  ** moved distance=', d,\r\n        'now at', pt\r\n    )\r\n\r\nprint(' *** final point at', pt)\r\n```\r\nwhich will outputs:\r\n```pycon\r\n *** initial point at [ 0.  0.]\r\n --- moving point x= 0.0 , y= 0.0 with vector x= 3.0 , y= 7.0 for  5 steps\r\n  -- step 0 to ( 3.0 , 7.0 )\r\n  -- step 1 to ( 6.0 , 14.0 )\r\n  -- step 2 to ( 9.0 , 21.0 )\r\n  -- step 3 to ( 12.0 , 28.0 )\r\n  -- step 4 to ( 15.0 , 35.0 )\r\n *** moved to [ 15.  35.]\r\n --- moving point x= 15.0 , y= 35.0 with vector x= 3.0 , y= 7.0 for  5 steps\r\n  -- step 0 to ( 18.0 , 42.0 )\r\n  ** moved distance= 7.6157732009887695 now at [ 18.  42.]\r\n  -- step 1 to ( 21.0 , 49.0 )\r\n  ** moved distance= 7.6157732009887695 now at [ 21.  49.]\r\n  -- step 2 to ( 24.0 , 56.0 )\r\n  ** moved distance= 7.6157732009887695 now at [ 24.  56.]\r\n  -- step 3 to ( 27.0 , 63.0 )\r\n  ** moved distance= 7.6157732009887695 now at [ 27.  63.]\r\n  -- step 4 to ( 30.0 , 70.0 )\r\n  ** moved distance= 7.6157732009887695 now at [ 30.  70.]\r\n *** final point at [ 30.  70.]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/numba/numba/issues/comments/313789664","html_url":"https://github.com/numba/numba/issues/2454#issuecomment-313789664","issue_url":"https://api.github.com/repos/numba/numba/issues/2454","id":313789664,"user":{"login":"sklam","id":1929845,"avatar_url":"https://avatars2.githubusercontent.com/u/1929845?v=3","gravatar_id":"","url":"https://api.github.com/users/sklam","html_url":"https://github.com/sklam","followers_url":"https://api.github.com/users/sklam/followers","following_url":"https://api.github.com/users/sklam/following{/other_user}","gists_url":"https://api.github.com/users/sklam/gists{/gist_id}","starred_url":"https://api.github.com/users/sklam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklam/subscriptions","organizations_url":"https://api.github.com/users/sklam/orgs","repos_url":"https://api.github.com/users/sklam/repos","events_url":"https://api.github.com/users/sklam/events{/privacy}","received_events_url":"https://api.github.com/users/sklam/received_events","type":"User","site_admin":false},"created_at":"2017-07-07T20:50:05Z","updated_at":"2017-07-07T20:50:05Z","body":"The reason the generator version didn't reflect the assigned value is that the record is \"copied\" when the generator \"freezes\".  This is due to the lack of reference-counted pointer to the *parent* data buffer inside the record object, which was introduced prior to numba's refct implementation.\r\n\r\nI'm working on a solution."}}
{"actor":{"display_login":"sklam"}}