{"action":"created","issue":{"url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/2","repository_url":"https://api.github.com/repos/rpryzant/japanese_corpus","labels_url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/2/comments","events_url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/2/events","html_url":"https://github.com/rpryzant/japanese_corpus/issues/2","id":208165909,"number":2,"title":"Investigate data quality","user":{"login":"dennybritz","id":403133,"avatar_url":"https://avatars3.githubusercontent.com/u/403133?v=3","gravatar_id":"","url":"https://api.github.com/users/dennybritz","html_url":"https://github.com/dennybritz","followers_url":"https://api.github.com/users/dennybritz/followers","following_url":"https://api.github.com/users/dennybritz/following{/other_user}","gists_url":"https://api.github.com/users/dennybritz/gists{/gist_id}","starred_url":"https://api.github.com/users/dennybritz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dennybritz/subscriptions","organizations_url":"https://api.github.com/users/dennybritz/orgs","repos_url":"https://api.github.com/users/dennybritz/repos","events_url":"https://api.github.com/users/dennybritz/events{/privacy}","received_events_url":"https://api.github.com/users/dennybritz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":24,"created_at":"2017-02-16T16:05:53Z","updated_at":"2017-03-29T01:38:00Z","closed_at":null,"body":"We should figure out of what quality our aligned sentences are and how we can make them better. I think a simple way to do this is to just sample a couple (50 or so?) translation from each of the corpora and create a spreadsheet. I can go through the translations manually and mark the ones that are wrong, and why they are wrong (e.g. misaligned, garbage text, etc). That should give us an idea of how we can improve the data, and which corpora are the most noisy and need more work."},"comment":{"url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/comments/289955930","html_url":"https://github.com/rpryzant/japanese_corpus/issues/2#issuecomment-289955930","issue_url":"https://api.github.com/repos/rpryzant/japanese_corpus/issues/2","id":289955930,"user":{"login":"rpryzant","id":8572027,"avatar_url":"https://avatars2.githubusercontent.com/u/8572027?v=3","gravatar_id":"","url":"https://api.github.com/users/rpryzant","html_url":"https://github.com/rpryzant","followers_url":"https://api.github.com/users/rpryzant/followers","following_url":"https://api.github.com/users/rpryzant/following{/other_user}","gists_url":"https://api.github.com/users/rpryzant/gists{/gist_id}","starred_url":"https://api.github.com/users/rpryzant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rpryzant/subscriptions","organizations_url":"https://api.github.com/users/rpryzant/orgs","repos_url":"https://api.github.com/users/rpryzant/repos","events_url":"https://api.github.com/users/rpryzant/events{/privacy}","received_events_url":"https://api.github.com/users/rpryzant/received_events","type":"User","site_admin":false},"created_at":"2017-03-29T01:38:00Z","updated_at":"2017-03-29T01:38:00Z","body":"Sounds good. Good luck.\r\n\r\nOn my end, I noticed two things:\r\n1. Sub files have a ton of information in their filenames that we're ignoring\r\n2. Sub files, even if they correspond to the same thing, may be shifted, so under the current alignment scheme they would result in completely invalid alignments that the system would think are great\r\n\r\nSo I'm trying out a new style of sub file alignment that I think will be much more powerful. I\r\n\r\n1. do some fuzzy guessing/matching on the subfile name to align files\r\n2. find the \"first content caption\" in both files, and shift all of their timestamps so that the entire file lines up. \r\n3. walk through both files, and as long as captions stay aligned, match them\r\n\r\nFinding the first \"content caption\" is tough because you have to sort out what's a real caption vs what's msc stuff the subtitler threw in (\"directed by...\", \"SUBS BY XXX\", \"yes/no\", etc.). \r\n\r\nRight now I'm thinking of doing this by getting \r\n* lemmatized english definitions for all the content words of the first few japanese captions\r\n* lemmatized content words for the first few english captions\r\n\r\nThen I'll say that the \"first matching caption\" is the pair of sentences in the above lists that share the most lemmatized content words (nouns, verbs, adjectives). "}}
{"actor":{"display_login":"rpryzant"}}