{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/5476","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/5476/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/5476/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/5476/events","html_url":"https://github.com/dmlc/mxnet/issues/5476","id":215221435,"number":5476,"title":"Linear Regression Model Comparison Btwn TF, Theano, Caffe, & MXNet","user":{"login":"projectlouis","id":26049835,"avatar_url":"https://avatars1.githubusercontent.com/u/26049835?v=3","gravatar_id":"","url":"https://api.github.com/users/projectlouis","html_url":"https://github.com/projectlouis","followers_url":"https://api.github.com/users/projectlouis/followers","following_url":"https://api.github.com/users/projectlouis/following{/other_user}","gists_url":"https://api.github.com/users/projectlouis/gists{/gist_id}","starred_url":"https://api.github.com/users/projectlouis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/projectlouis/subscriptions","organizations_url":"https://api.github.com/users/projectlouis/orgs","repos_url":"https://api.github.com/users/projectlouis/repos","events_url":"https://api.github.com/users/projectlouis/events{/privacy}","received_events_url":"https://api.github.com/users/projectlouis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-18T21:02:59Z","updated_at":"2017-03-19T13:43:32Z","closed_at":null,"body":"## Background\r\nCurrently I'm running some linear regression tests for a non-thesis paper... I'm still naive in the ML realm, so I'm relatively sure I may have misinterpreted something. I'd like to best represent each framework appropriately to ensure a consistent evaluation approach.\r\n\r\nI've managed to get a simple example using [airfoil self-noise data](https://archive.ics.uci.edu/ml/datasets/Airfoil+Self-Noise).\r\n\r\nThe specific data file I use (for my standard pandas read-in format) is at: [airfoil self noise.csv](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/data/airfoil_self_noise.csv)\r\n\r\nInside the script, you'll notice that all inputs are pushed into a z-score format and this is done across all 4 frameworks. For Tensorflow, since it's iterating and doesn't allow an 'num_epoch' variable, I assumed the following would work:\r\n5000 Epochs --> Iterations = (# Epochs * # Samples) / Batch Size\r\n5000 Epochs --> 58710 Iterations\r\n\r\n## Model Used\r\nThe Framework Set up is as follows: \r\n- MLP (3 Layers, 5 Inputs, 1 Output)\r\n- Layer 1 - 25 Hidden, ReLu Activation\r\n- Layer 2 - 5 Hidden, ReLu Activation\r\n- Layer 3 - Final Output -- No Activation\r\n- Batch Size: 128\r\n- Learning Rate: 0.001\r\n- Num Epochs: 5000\r\n- Optimizer: Adam\r\n- Metric: Mean Squared Error \r\n- Train-Test Split: 80/20\r\n\r\nI've highlighted my network creations for each of the four frameworks below:\r\n [Theano](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/airfoil_regress_theano.py#L20-L42), \r\n[TF](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/airfoil_regress_tf.py#L69-L104), \r\n[MXNet](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/airfoil_regress_mxnet.py#L10-L29), \r\nand [Caffe](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/airfoil_regress_caffe.py#L101-L113).\r\n\r\n## The Issue\r\n\r\n**As for the specific question in mind, I'm seeing relatively different R^2 values specifically for MXNet. (Comparing my Predicted versus ideal values inside excel)**\r\n\r\nR^2 for MXNet: [0.6463](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/data/airfoil_regression_mxnet.xlsx)\r\nR^2 for TF: [0.8797](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/data/airfoil_regression_tf.xlsx)\r\nR^2 for Theano: [0.845](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/data/airfoil_regression_theano.xlsx)\r\nR^2 for Caffe: [0.5979](https://github.com/projectlouis/mlbenchmark/blob/master/airfoil/data/airfoil_regression_caffe.xlsx)\r\n\r\nIf anyone could point out issues with the MXNet code, that would be highly appreciative. Caffe seems to act the same as MXNet.... So maybe this isn't an issue?\r\n\r\nCheers,\r\nLouis\r\n\r\n## Environment info\r\nOperating System: Windows (Docker Containers)\r\n\r\nPackage used (Python/R/Scala/Julia): Python\r\n\r\nMXNet version: https://hub.docker.com/r/kaixhin/mxnet/\r\n\r\nPython version and distribution: Python 2.7.6\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/287617586","html_url":"https://github.com/dmlc/mxnet/issues/5476#issuecomment-287617586","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/5476","id":287617586,"user":{"login":"sxjscience","id":5178350,"avatar_url":"https://avatars2.githubusercontent.com/u/5178350?v=3","gravatar_id":"","url":"https://api.github.com/users/sxjscience","html_url":"https://github.com/sxjscience","followers_url":"https://api.github.com/users/sxjscience/followers","following_url":"https://api.github.com/users/sxjscience/following{/other_user}","gists_url":"https://api.github.com/users/sxjscience/gists{/gist_id}","starred_url":"https://api.github.com/users/sxjscience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sxjscience/subscriptions","organizations_url":"https://api.github.com/users/sxjscience/orgs","repos_url":"https://api.github.com/users/sxjscience/repos","events_url":"https://api.github.com/users/sxjscience/events{/privacy}","received_events_url":"https://api.github.com/users/sxjscience/received_events","type":"User","site_admin":false},"created_at":"2017-03-19T13:43:32Z","updated_at":"2017-03-19T13:43:32Z","body":"I guess it's due to the initialization method."}}
{"actor":{"display_login":"sxjscience"}}