{"action":"created","issue":{"url":"https://api.github.com/repos/davidsandberg/facenet/issues/452","repository_url":"https://api.github.com/repos/davidsandberg/facenet","labels_url":"https://api.github.com/repos/davidsandberg/facenet/issues/452/labels{/name}","comments_url":"https://api.github.com/repos/davidsandberg/facenet/issues/452/comments","events_url":"https://api.github.com/repos/davidsandberg/facenet/issues/452/events","html_url":"https://github.com/davidsandberg/facenet/issues/452","id":256217773,"number":452,"title":"An issue with the accuracy of facenet","user":{"login":"github4f","id":6980446,"avatar_url":"https://avatars3.githubusercontent.com/u/6980446?v=4","gravatar_id":"","url":"https://api.github.com/users/github4f","html_url":"https://github.com/github4f","followers_url":"https://api.github.com/users/github4f/followers","following_url":"https://api.github.com/users/github4f/following{/other_user}","gists_url":"https://api.github.com/users/github4f/gists{/gist_id}","starred_url":"https://api.github.com/users/github4f/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/github4f/subscriptions","organizations_url":"https://api.github.com/users/github4f/orgs","repos_url":"https://api.github.com/users/github4f/repos","events_url":"https://api.github.com/users/github4f/events{/privacy}","received_events_url":"https://api.github.com/users/github4f/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-09-08T11:00:35Z","updated_at":"2017-09-11T12:30:17Z","closed_at":null,"author_association":"NONE","body":"Hi- I use a pipeline based on mtcnn for face detection and then facenet for face recognition (I did test both 20170511-185253.pb and 20170512-110547.pb). I perform this process in a while loop for 2 minutes in which frames/images are captured using opencv video capture function! Finally, I compare my own images/faces during this period of time using cosine and euclidean functions! But the results are not promising at all. Indeed, the error can reach up to 50% sometime among frames! Any idea why is that and how I can improve it? \r\n\r\n1) Do I need to do normalization, or rgb2gray before giving my frames to mtcnn?\r\n\r\n2) Does mtccn already perform alignment?\r\n\r\n\r\n3) Based on  align_dlib.py, I use the following line: scaled = misc.imresize(img, prealigned_scale, interp='bilinear')\r\n\r\nrather than using: align.align(image_size, img, landmarkIndices=landmarkIndices, skipMulti=False, scale=scale)\r\n\r\n\r\n\r\n4) What about alignment ( 3d alignment such as fb deep face)?  \r\n\r\n\r\n5) I use the following code in the while loop\r\n\r\n\r\ninput_image_size = 160\r\n\r\n\r\n\r\n\r\n\r\n     ret, frame = video_capture.read()\r\n      bounding_boxes, _ = detect_face.detect_face(frame, minsize, pnet, rnet, onet, threshold, factor)\r\n      nrof_faces = bounding_boxes.shape[0]\r\n               if nrof_faces > 0:\r\n                     det = bounding_boxes[:, 0:4]\r\n                    img_size = np.asarray(frame.shape)[0:2]\r\n                    cropped = []\r\n                    scaled = []\r\n                    scaled_reshape = []\r\n                    bb = np.zeros((nrof_faces,4), dtype=np.int32)    \r\n            \r\n                     for i in range(nrof_faces):\r\n                        emb_array = np.zeros((1, embedding_size))\r\n\r\n\r\n                        bb[i][0] = det[i][0]\r\n                        bb[i][1] = det[i][1]\r\n                        bb[i][2] = det[i][2]\r\n                        bb[i][3] = det[i][3]\r\n\r\n\r\n                        cv2.rectangle(frame, (bb[i][0], bb[i][1]), (bb[i][2], bb[i][3]), (0, 255, 0), 2)    #boxing face             \r\n                        cropped.append(frame[bb[i][1]:bb[i][3], bb[i][0]:bb[i][2], :])\r\n                        cropped[i] = facenet.flip(cropped[i], False)\r\n\r\n                        scaled.append(misc.imresize(cropped[i], (image_size, image_size), interp='bilinear'))\r\n                    \r\n                        scaled[i] = cv2.resize(scaled[i], (input_image_size,input_image_size), interpolation=cv2.INTER_CUBIC)\r\n\r\n\r\n                        scaled[i] = facenet.prewhiten(scaled[i])\r\n                        scaled_reshape.append(scaled[i].reshape(-1,input_image_size,input_image_size,3))\r\n                        feed_dict = {images_placeholder: scaled_reshape[i], phase_train_placeholder: False}\r\n                        emb_array[0, :] = sess.run(embeddings, feed_dict=feed_dict)\r\n\r\n                       #then I compare (cosine, etc.) with embeddings of my own face pictures (captured with same camera, same background, etc.):\r\n                       dist_cosine= distance.cosine(saved_embedding ,emb_array[0, :])\r\n                       dist_euclidean = distance.euclidean(embedding,emb_array[0, :])\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/davidsandberg/facenet/issues/comments/328514132","html_url":"https://github.com/davidsandberg/facenet/issues/452#issuecomment-328514132","issue_url":"https://api.github.com/repos/davidsandberg/facenet/issues/452","id":328514132,"user":{"login":"github4f","id":6980446,"avatar_url":"https://avatars3.githubusercontent.com/u/6980446?v=4","gravatar_id":"","url":"https://api.github.com/users/github4f","html_url":"https://github.com/github4f","followers_url":"https://api.github.com/users/github4f/followers","following_url":"https://api.github.com/users/github4f/following{/other_user}","gists_url":"https://api.github.com/users/github4f/gists{/gist_id}","starred_url":"https://api.github.com/users/github4f/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/github4f/subscriptions","organizations_url":"https://api.github.com/users/github4f/orgs","repos_url":"https://api.github.com/users/github4f/repos","events_url":"https://api.github.com/users/github4f/events{/privacy}","received_events_url":"https://api.github.com/users/github4f/received_events","type":"User","site_admin":false},"created_at":"2017-09-11T12:30:17Z","updated_at":"2017-09-11T12:30:17Z","author_association":"NONE","body":"Hi- I did an experiment using about 120 figures of myself. You can find the hist in the following figure. According to the results, normally I see more than 75% similarity between my pictures, but sometime it drops below 70%!\r\n\r\n<img width=\"360\" alt=\"myface\" src=\"https://user-images.githubusercontent.com/6980446/30273993-c66b40d0-96fb-11e7-8338-baaaa32cd141.PNG\">\r\n\r\nIn the following figure, I compare my own figures (120 figures) vs 120 figures of a friend of mine:\r\n\r\n<img width=\"359\" alt=\"i_vs_my friend\" src=\"https://user-images.githubusercontent.com/6980446/30274091-33d2bb30-96fc-11e7-883a-b4f42377001a.PNG\">\r\n\r\nIt is very interesting that sometime the similarity between I and my friend is beyond 60%. Based on the above figures, I decided to use 75% as the threshold in my code. \r\n\r\n@vudung45 : I am wondering your code is consistent with my results or you reach better similarity among your figures (for just one person)? I noticed that you use a different approach for alignment. \r\n\r\nI also did some investigation to see why there are so much similarity between I and my friend. I noticed that most of the time, the problem is rooted in wrong captured figures. for example, there are some figures that both eyes are not visible in the figures. Something like below: \r\n\r\n![figure drawing for dummies -mantesh-80-1](https://user-images.githubusercontent.com/6980446/30274213-b9d717b2-96fc-11e7-8a21-e58b224eea4b.jpg)\r\n\r\nI noticed that openface (http://cmusatyalab.github.io/openface/), does use opencv cascade (eye), to detect false figures. But mtcnn cannot do that! I suggest that we also include eye cascade to remove those figures. Please let me know your idea (@fgervais ).  \r\n\r\n"}}
{"actor":{"display_login":"github4f"}}