{"action":"created","issue":{"url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/31","repository_url":"https://api.github.com/repos/onnx/onnx-tensorflow","labels_url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/31/comments","events_url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/31/events","html_url":"https://github.com/onnx/onnx-tensorflow/issues/31","id":282303977,"number":31,"title":"Force NCHW format","user":{"login":"lukemurray","id":156625,"avatar_url":"https://avatars0.githubusercontent.com/u/156625?v=4","gravatar_id":"","url":"https://api.github.com/users/lukemurray","html_url":"https://github.com/lukemurray","followers_url":"https://api.github.com/users/lukemurray/followers","following_url":"https://api.github.com/users/lukemurray/following{/other_user}","gists_url":"https://api.github.com/users/lukemurray/gists{/gist_id}","starred_url":"https://api.github.com/users/lukemurray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukemurray/subscriptions","organizations_url":"https://api.github.com/users/lukemurray/orgs","repos_url":"https://api.github.com/users/lukemurray/repos","events_url":"https://api.github.com/users/lukemurray/events{/privacy}","received_events_url":"https://api.github.com/users/lukemurray/received_events","type":"User","site_admin":false},"labels":[{"id":711826260,"url":"https://api.github.com/repos/onnx/onnx-tensorflow/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-12-15T02:46:07Z","updated_at":"2017-12-15T14:32:04Z","closed_at":null,"author_association":"NONE","body":"Hi there, this seems to add a bunch of transpose ops into the graph, can we force it to build the graph in NCHW format?"},"comment":{"url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/comments/352019711","html_url":"https://github.com/onnx/onnx-tensorflow/issues/31#issuecomment-352019711","issue_url":"https://api.github.com/repos/onnx/onnx-tensorflow/issues/31","id":352019711,"user":{"login":"LucasMahieu","id":12006816,"avatar_url":"https://avatars3.githubusercontent.com/u/12006816?v=4","gravatar_id":"","url":"https://api.github.com/users/LucasMahieu","html_url":"https://github.com/LucasMahieu","followers_url":"https://api.github.com/users/LucasMahieu/followers","following_url":"https://api.github.com/users/LucasMahieu/following{/other_user}","gists_url":"https://api.github.com/users/LucasMahieu/gists{/gist_id}","starred_url":"https://api.github.com/users/LucasMahieu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LucasMahieu/subscriptions","organizations_url":"https://api.github.com/users/LucasMahieu/orgs","repos_url":"https://api.github.com/users/LucasMahieu/repos","events_url":"https://api.github.com/users/LucasMahieu/events{/privacy}","received_events_url":"https://api.github.com/users/LucasMahieu/received_events","type":"User","site_admin":false},"created_at":"2017-12-15T14:32:04Z","updated_at":"2017-12-15T14:32:04Z","author_association":"NONE","body":"I have though a lot about this problem. Here is a proposal of how I think things have to be done to be optimal.\r\n\r\nI will give an example to explain my idea:\r\nI suggest that the user have the choice of two things:\r\n\r\n    - The device on which he wants to run the TF graph\r\n          (the source graph for front-end or the destination graph for back-end) (GPU or CPU)\r\n    - The data format that he will put as input of the TF graph \r\n          (the source graph for front-end or the destination graph for back-end)  (NCHW or NHWC)\r\n\r\nBack-end\r\n--- \r\n\r\nNow, assume that the ONNX graph is : \r\n**-> MaxPool(k, s, p) -> Concat(axis = 1) ->**\r\nk, p, s is kernel, stride and pad for H and W dimension it will not be impact by the thereafter explanation.\r\nThis example illustrate how to deal with ops that have specific constrains on data format and others that don't have.\r\n\r\nAssuming that the input given in the ONNX graph is always in 'NCHW' format.\r\nSo if user makes choice to give 'NHWC' data to the TF graph, he has to transpose all the inputs taken from ONNX.\r\n\r\nAccording to the 2 choices of the user, if he wants to convert this ONNX graph to Tensorflow Graph, we have 4 different cases to support.\r\n\r\n**ONNX -> Tensorflow: Each cell of the Table is the TF graph generated**\r\n\r\n| device\\input data <br>of TF graph\t| NCHW                      | NHWC \t|\r\n|-------------------------                |------                          |-----               |\r\n| GPU                                       | maxpool<br>concat(axis=1) | transpose(0,3,1,2)<br>maxpool<br>transpose(0,2,3,1)<br>concat(axis=3)|\r\n| CPU                                        |transpose(0,2,3,1)<br>maxpool<br>transpose(0,3,1,2)<br>concat(axis=1) | maxpool<br>concat(axis=3) |\r\n\r\nFront-end\r\n--- \r\n\r\nFor the front-end, is the same, the user has to said if the TF graph is made for CPU or GPU and if the tensors of the graph are in format NCHW or NHWC.\r\n\r\nNow, assume that the TF graph is : \r\n**-> MaxPool(k, s, p, d) -> Concat(axis=channel) ->**\r\nk, p, s is kernel, stride and padding type for all dimensions. d in the data_format of the op's input.\r\nIn TF, k=[k0, k1, k2, k3] ; s=[s0, s1, s2, s3] ; one value for each axis, so they will be impacted by user's choices.\r\n\r\n**Tensorflow -> ONNX: Each cell of the Table is the ONNX graph generated**\r\n\r\n| device\\ data format <br>of TF graph | NCHW                      | NHWC \t|\r\n|-------------------------                |------                          |-----               |\r\n| GPU                                       | maxpool((k2,k3),(s2,s3))<br>concat(axis=1) | \\* note 1 |\r\n| CPU                                        | \\* note 2 | maxpool((k1,k2),(s1,s2))<br>concat(axis=1) |\r\n\r\n- note 1 : This case is special. If the TF graph is just like this : \r\n*-> MaxPool(k, s, p, 'NHWC') -> Concat(axis=3) ->* \r\nIt would not be executable on CPU with TF. But if  the user takes care of that and use transposed just before and after the MaxPool(k, s, p, 'NHWC') (or if the tf graph was generated by our backend for example) the graph would be correct.\r\nAnd so, the onnx-tf front-end should parse and transform : \r\n\r\n    **This tf graph part:**\r\n*-> transpose(0,3,1,2) -> MaxPool((k0,k1,k2,k3), (s0,s1,s2,s3), p, 'NHWC') -> transpose(0,2,3,1) -> Concat(axis=3) ->*\r\n    **Into this onnx graph part**\r\n*-> MaxPool((k1,k2), (s1,s2), p) -> Concat(axis=1) ->*\r\n\r\n\r\n- note 2 : This case is special. If the TF graph is just like this : \r\n*-> MaxPool(k, s, p, 'NCHW') -> Concat(axis=1) ->* \r\nIt would not be executable on CPU with TF. But if  the user takes care of that and use transposed just before and after the MaxPool(k, s, p, 'NHWC') (or if the tf graph was generated by our backend for example) the graph would be correct.\r\nAnd so, the onnx-tf front-end should parse and transform : \r\n\r\n    **This tf graph part:**\r\n*-> transpose(0,2,3,1) -> MaxPool((k0,k1,k2,k3), (s0,s1,s2,s3), p, 'NHWC') -> transpose(0,3,1,2) -> Concat(axis=1) ->*\r\n    **Into this onnx graph part**\r\n*-> MaxPool((k2,k3), (s2,s3), p) -> Concat(axis=1) ->*\r\n\r\n\r\nTo conclude, we have to add the 2 possible choices for the user when converting a graph.\r\nAnd we have to parse graph in non one after one order, but in a more intelligent way according to the user choices in order to support special cases \"note 1\" and \"note 2\". This could be made by two different methods : \r\n    - Adding a transpose op around max_pool, then remove 2 opposite successive transpose ops from the final graph.\r\n    - Identify a pattern \"transpose + max_pool + transpose\" and generate only a max_pool with the rights parameters "}}
{"actor":{"display_login":"LucasMahieu"}}