fs.cp_file: out of memory error (don't store file in RAM)

[casperdcl]

PyDrive2/pydrive2/fs/spec.py

Lines 358 to 365 in 378c6e8

	def cp_file(self, lpath, rpath, **kwargs):
	"""In-memory streamed copy"""
	with self.open(lpath) as stream:
	# IterStream objects doesn't support full-length
	# seek() calls, so we have to wrap the data with
	# an external buffer.
	buffer = io.BytesIO(stream.read())
	self.upload_fobj(buffer, rpath)

ref. #119 (review) suggestion:

-    """In-memory streamed copy""" 
-    with self.open(lpath) as stream: 
-        # IterStream objects doesn't support full-length 
-        # seek() calls, so we have to wrap the data with 
-        # an external buffer. 
-        buffer = io.BytesIO(stream.read()) 
-        self.upload_fobj(buffer, rpath) 
+        with self.open(lpath) as stream:
+            self.upload_fobj(stream, rpath)

and fix self.open to return a proper stream.

[shcheklein]

@isidentical could you give more context on this?

[isidentical]

Normally a on-server copy was not available when migrating from the dvc's pydrivefilesystem, so I added it with a simple interface for the sake of simplicty in order to share test suites with sshfs. Though we do not depend on that method (it is only used by the DVC when doing external outputs flow AFAIK, and gdrive does not support it).

[casperdcl]

hmm so "external outputs" will affect iterative/dvc#6485, right?

[isidentical]

I don't think that command involves any on the server copy, does it? I think it would be only relevant here iterative/dvc#5811

[efiop]

yeah, it is not related to put-url, not right now at least. Still would be nice not to load the whole thing into memory, of course.

[shcheklein]

PyDrive2 interface has native CopyFile implementation now - https://docs.iterative.ai/PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.Copy