{"action":"opened","issue":{"url":"https://api.github.com/repos/cjlin1/libsvm/issues/98","repository_url":"https://api.github.com/repos/cjlin1/libsvm","labels_url":"https://api.github.com/repos/cjlin1/libsvm/issues/98/labels{/name}","comments_url":"https://api.github.com/repos/cjlin1/libsvm/issues/98/comments","events_url":"https://api.github.com/repos/cjlin1/libsvm/issues/98/events","html_url":"https://github.com/cjlin1/libsvm/issues/98","id":238795593,"number":98,"title":"NU_SVC training is affected by order of training samples","user":{"login":"alexisrozhkov","id":6758087,"avatar_url":"https://avatars1.githubusercontent.com/u/6758087?v=3","gravatar_id":"","url":"https://api.github.com/users/alexisrozhkov","html_url":"https://github.com/alexisrozhkov","followers_url":"https://api.github.com/users/alexisrozhkov/followers","following_url":"https://api.github.com/users/alexisrozhkov/following{/other_user}","gists_url":"https://api.github.com/users/alexisrozhkov/gists{/gist_id}","starred_url":"https://api.github.com/users/alexisrozhkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexisrozhkov/subscriptions","organizations_url":"https://api.github.com/users/alexisrozhkov/orgs","repos_url":"https://api.github.com/users/alexisrozhkov/repos","events_url":"https://api.github.com/users/alexisrozhkov/events{/privacy}","received_events_url":"https://api.github.com/users/alexisrozhkov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-06-27T09:50:04Z","updated_at":"2017-06-27T09:50:04Z","closed_at":null,"body":"I have originally discovered this effect using C version of libsvm, wrapped into C++ code, using real data. \r\n\r\nThen I created a minimal snippet to reproduce it using libsvm in sklearn with random data for training and testing. \r\n\r\nSo far I have checked only NU_SVC and C_SVC, and it seems that C_SVC is invariant to order of training samples, but NU_SVC isn't, which is counterintuitive (at least for me, since there's no notion of sample ordering in SVM problem formulation).\r\n\r\nHere's a snippet to reproduce:\r\n```\r\nimport numpy as np\r\nfrom sklearn import svm\r\n\r\n# training/testing set params\r\nn = 1000  # num of samples\r\nd = 2  # num of dimensions\r\n\r\n# random training set (balanced)\r\ntrain_feats = np.random.uniform(size=(n, d))\r\ntrain_labels = np.array([i < n/2 for i in xrange(n)]).reshape(n, 1)\r\n\r\n# combine to retain connection between features and labels during shuffling\r\ntrain = np.hstack([train_feats, train_labels])\r\n\r\n# 2 svms to compare\r\ncsvm = svm.SVC(kernel='rbf',\r\n               gamma=0.5, \r\n               degree=3,\r\n               cache_size=100,\r\n               tol=0.001, \r\n               C=1.0, \r\n               shrinking=False)\r\n\r\nnsvm = svm.NuSVC(kernel='rbf',\r\n                 gamma=0.5, \r\n                 degree=3,\r\n                 cache_size=100,\r\n                 tol=0.001, \r\n                 nu=0.1, \r\n                 shrinking=False)\r\n\r\n# random testing set (balanced)\r\ntest_feats = np.random.uniform(size=(n, d))\r\ntest_labels = [i < n/2 for i in xrange(n)]\r\n\r\nfor i in xrange(10):\r\n    # shuffle train set\r\n    train_shuffled = train.copy()\r\n    np.random.shuffle(train_shuffled)\r\n    \r\n    # split into features and labels\r\n    shuffled_train_feats = train_shuffled[:, :-1]\r\n    shuffled_train_labels = train_shuffled[:, -1]\r\n    \r\n    # train both svms individually\r\n    csvm.fit(shuffled_train_feats, shuffled_train_labels)  \r\n    nsvm.fit(shuffled_train_feats, shuffled_train_labels)  \r\n    \r\n    # make predictions individually\r\n    pred_c = csvm.predict(test_feats)\r\n    pred_n = nsvm.predict(test_feats)\r\n\r\n    acc_c = sum(pred_c == test_labels)/float(n)\r\n    acc_n = sum(pred_n == test_labels)/float(n)\r\n    \r\n    print acc_c, acc_n\r\n```\r\n\r\nAnd corresponding output:\r\n```\r\n0.502 0.49\r\n0.502 0.495\r\n0.502 0.504\r\n0.502 0.5\r\n0.502 0.526\r\n0.502 0.496\r\n0.502 0.505\r\n0.502 0.507\r\n0.502 0.533\r\n0.502 0.516\r\n```\r\nFirst column - accuracies using C_SVC, second - N_SVC using different shufflings of training samples. As you can see C_SVC is invariant to sample order, NU_SVC is not. \r\n\r\nI would rather use NU_SVC, since it is simpler to perform grid search on, but this effect makes it almost pointless. Am I missing something?"}}
{"actor":{"display_login":"alexisrozhkov"}}