{"action":"opened","issue":{"url":"https://api.github.com/repos/kennethreitz/requests/issues/3870","repository_url":"https://api.github.com/repos/kennethreitz/requests","labels_url":"https://api.github.com/repos/kennethreitz/requests/issues/3870/labels{/name}","comments_url":"https://api.github.com/repos/kennethreitz/requests/issues/3870/comments","events_url":"https://api.github.com/repos/kennethreitz/requests/issues/3870/events","html_url":"https://github.com/kennethreitz/requests/issues/3870","id":207235685,"number":3870,"title":"Uploading large video files","user":{"login":"dvigneshwer","id":5075497,"avatar_url":"https://avatars.githubusercontent.com/u/5075497?v=3","gravatar_id":"","url":"https://api.github.com/users/dvigneshwer","html_url":"https://github.com/dvigneshwer","followers_url":"https://api.github.com/users/dvigneshwer/followers","following_url":"https://api.github.com/users/dvigneshwer/following{/other_user}","gists_url":"https://api.github.com/users/dvigneshwer/gists{/gist_id}","starred_url":"https://api.github.com/users/dvigneshwer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dvigneshwer/subscriptions","organizations_url":"https://api.github.com/users/dvigneshwer/orgs","repos_url":"https://api.github.com/users/dvigneshwer/repos","events_url":"https://api.github.com/users/dvigneshwer/events{/privacy}","received_events_url":"https://api.github.com/users/dvigneshwer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-13T14:37:29Z","updated_at":"2017-02-13T14:37:29Z","closed_at":null,"body":"Hi,\r\n\r\nThe code snippet below works for uploading a small video file(15MB) to my storage service.\r\n\r\n```\r\nurl = self.swift_public_url+'/'+self.swift_container+'/'+self.video_name\t\t\t \r\nfiles = {'file': open(self.local_video_loc, 'rb')}\r\nheaders = {\r\n\t'x-auth-token': self.swift_auth_token,\r\n         'content-type': \"application/x-www-form-urlencoded\",\r\n}\r\nresponse = requests.request(\"PUT\",url,files=files,headers=headers)\r\n```\r\n\r\nThe code crashes my computer(becomes unresponsive) when I am trying to upload a video of say 2+ GB of size.\r\n\r\n**_How to perform the following:_**\r\n* Chunk the data to smaller units and upload to the service\r\n* Is it possible to parallel run  \r\n\r\n**Attempts:**\r\n* Opened the video file and started reading it in chunks\r\n* Created a generator using the yield keyword\r\n* read_as_gen should send me each chunk one by one \r\n~~~~\r\nurl = self.swift_public_url+'/'+self.swift_container+'/'+self.video_name\r\n\t\t\t\t\r\ndef read_as_gen(filename, chunksize=-1):\r\n\t  with open(filename, mode='rb') as f:\r\n\t         while True:\r\n\t\t\t  chunk = f.read(chunksize)\r\n\t\t\t  if len(chunk) > 0:\r\n\t\t\t            yield chunk\r\n\t\t\t   else:\r\n\t\t\t             raise StopIteration\r\n\r\nfiles = {'file': read_as_gen(self.local_video_loc, 64*1024)}\r\n\r\n# files = {'file': open(self.local_video_loc, 'rb')}\r\nheaders = {\r\n\t          'x-auth-token': self.swift_auth_token,\r\n\t          'content-type': \"application/x-www-form-urlencoded\",\r\n\t\t}\r\nresponse = requests.request(\"PUT\",url,files=files,headers=headers)\r\n~~~~\r\n\r\nAbove attempt did not work for me, only a 70 bytes video file was created in my storage server.  \r\n\r\n~~~~\r\nhdfs_client.upload('/'+self.hfds_video_loc+'/'+self.video_name,self.local_video_loc,overwrite=False,n_threads=1, temp_dir=None, chunk_size=65536, progress=None, cleanup=True)\r\n~~~~\r\nAbove functionality can be implemented with the hdfs client.\r\nRef: https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.Client.upload\r\n\r\nI am basically trying to create the above client using the request module for another storage service.\r\n"}}
{"actor":{"display_login":"dvigneshwer"}}