Opened 4 years ago

Last modified 18 months ago

#93 new enhancement

Support a callback for comparing files in `upload` and `download`

Reported by: schwa Owned by: schwa
Priority: minor Milestone: 4.1.0
Component: Library Version:
Keywords: upload, download, condition Cc:

Description

This is based on an idea from Adrian Jones from the ftputil mailing list. See the thread "Conditional upload/download based on file size".

To upload/download a file after an incomplete upload/download, it should be possible to compare the sizes in addition to the timestamps of the files. upload_if_newer and download_if_newer only decide based on the timestamp.

Adrian suggested introducing a condition callback to let the caller decide when the file should be transferred again.

Change History (4)

comment:1 Changed 4 years ago by schwa

I think the condition callback argument (maybe with a different name, but let's see) should be added to upload and download, not upload_if_newer and download_if_newer. Adding it to the latter methods would mix two ways to define the condition, once by the method name, once by the callback argument. Users might wonder whether a given condition argument is applied in addition to the timestamp check or if the callback has to implement the timestamp check on its own.

Therefore, I'd add the callback argument to upload and download and implement upload_if_newer and download_if_newer in terms of that. To make the new feature directly useful, ftputil should provide a callback to compare timestamps and one to compare sizes. The *_if_newer methods would then use the pre-defined timestamp comparison callback.

It might be tricky to come up with a good interface for the callbacks. At the moment, the comparison is done on file_transfer.LocalFile and file_transfer.RemoteFile objects and I'm not sure of how much of this should be exposed to ftputil client code. So far, the module isn't described in the official documentation.

When adding the condition callback to upload and download, there's a potential source of confusion because these methods already have a callback argument that works on chunks of the transferred data. (This can be used for GUI progress bars, for example.) For backward compatibility, I wouldn't want to rename the existing callback argument.

comment:2 Changed 4 years ago by ftputiluser

To change the conditional logic, or to supply a function to provide the conditional logic, it seems LocalFile & RemoteFile objects must exposed for useful properties to be available.

If so, perhaps expose the RemoteFile object then add a transfer_condition method containing the existing logic which is called by copy_file, this could be subclassed and overridden to extend the conditional logic.

e.g.

class MyRemoteFile(ftuputil.file_transfer.Remote_file):
    def transfer_condition(self, source_file, target_file):
        return target_file.exists() or \
           ... other logic

Although I'm not sure how the FTPHost would then use this subclass, perhaps some kind of file_factory of the FTPHost? And this really depends on the decision whether the LocalFile & RemoteFile objects should be exposed. Otherwise, it seems the only way is to add methods for each conditional case, e.g., checking file size, upload_if_different ?

comment:3 Changed 3 years ago by schwa

Milestone: 3.4

comment:4 Changed 18 months ago by schwa

Milestone: 3.44.1.0
Note: See TracTickets for help on using tickets.