{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/2960","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/2960/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/2960/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/2960/events","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2960","id":202661279,"number":2960,"title":"Question: How to optimize load_data-Operation","user":{"login":"inkrement","id":604528,"avatar_url":"https://avatars0.githubusercontent.com/u/604528?v=3","gravatar_id":"","url":"https://api.github.com/users/inkrement","html_url":"https://github.com/inkrement","followers_url":"https://api.github.com/users/inkrement/followers","following_url":"https://api.github.com/users/inkrement/following{/other_user}","gists_url":"https://api.github.com/users/inkrement/gists{/gist_id}","starred_url":"https://api.github.com/users/inkrement/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inkrement/subscriptions","organizations_url":"https://api.github.com/users/inkrement/orgs","repos_url":"https://api.github.com/users/inkrement/repos","events_url":"https://api.github.com/users/inkrement/events{/privacy}","received_events_url":"https://api.github.com/users/inkrement/received_events","type":"User","site_admin":false},"labels":[{"id":82126549,"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/labels/api:%20bigquery","name":"api: bigquery","color":"c0c0c0","default":false},{"id":77438584,"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/labels/type:%20question","name":"type: question","color":"8bc34a","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-01-23T21:51:37Z","updated_at":"2017-03-13T21:13:28Z","closed_at":"2017-01-24T17:37:27Z","body":"I want to copy my MySQL data (>200 Mio. rows) to BigQuery. Therefore I created a python script, which uses this library. At the moment it streams 1000 rows with one request and it generates about 1,1 requests/second. This is not really fast and it would take me days to transfer the whole dataset. I am sure that this can be optimized, but I don't know how. Would you have some suggestions? You can find my source code [here](https://github.com/inkrement/MySQLbq/blob/e8f484a5aac93dc77f687457424995a72ad4460b/run.py)\r\n\r\nI thought about the following points:\r\n\r\n * Each request contains 1000 rows, should I choose a bigger number?\r\n * Does this library use gzip per default?"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/comments/286245583","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2960#issuecomment-286245583","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-python/issues/2960","id":286245583,"user":{"login":"inkrement","id":604528,"avatar_url":"https://avatars0.githubusercontent.com/u/604528?v=3","gravatar_id":"","url":"https://api.github.com/users/inkrement","html_url":"https://github.com/inkrement","followers_url":"https://api.github.com/users/inkrement/followers","following_url":"https://api.github.com/users/inkrement/following{/other_user}","gists_url":"https://api.github.com/users/inkrement/gists{/gist_id}","starred_url":"https://api.github.com/users/inkrement/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inkrement/subscriptions","organizations_url":"https://api.github.com/users/inkrement/orgs","repos_url":"https://api.github.com/users/inkrement/repos","events_url":"https://api.github.com/users/inkrement/events{/privacy}","received_events_url":"https://api.github.com/users/inkrement/received_events","type":"User","site_admin":false},"created_at":"2017-03-13T21:13:28Z","updated_at":"2017-03-13T21:13:28Z","body":"@tswast haha.. thanks, but this is my issue. But as I mentioned in this thread this is not a nice solution. I would have to scan all my data and find a symbol, which does not occur.. but whats next? If I add a new row to my database the next day, which includes exactly this symbol I have to find another one. MySQL is a really famous database and its format is used by a lot of other databases (MemSQL, MariaDB,...). Would be great if google would support some tools to migrate data. I would like to spend my money on queries and google storage and not for \"services\" like alooma. "}}
{"actor":{"display_login":"inkrement"}}