Opened 7 years ago

Closed 3 years ago

#61 closed enhancement (fixed)

Add 'rest' argument to file()

Reported by: ftputiluser Owned by: schwa
Priority: minor Milestone: 3.3
Component: Library Version: 2.6
Keywords: ftputil, file, rest Cc:

Description (last modified by schwa)

The method file() of FTPHost lacks extra numeric argument "rest" (or "offset") which is given to _FTPFile method _open() in line 228:

host._file._open(effective_file, mode)

The method _open() uses this argument for transfercmd() method in lines 85-86:

self._conn = ftp_error._try_with_ioerror(
               self._session.transfercmd, command)

This will give functionality of byte offset supported by transfercmd (http://docs.python.org/library/ftplib.html#ftplib.FTP.transfercmd). It does not fully cover ticket #14, however it will save lots of time of developers who simply need byte offset when reading files.

Change History (7)

comment:1 Changed 7 years ago by schwa

  • Status changed from new to assigned

comment:2 Changed 7 years ago by schwa

  • Milestone changed from 2.7 to 2.8

comment:3 Changed 6 years ago by schwa

  • Milestone 2.8 deleted

comment:4 Changed 4 years ago by ftputiluser

This is critical for me too.

Regards, Łukasz (lm at zork.pl)

comment:5 Changed 3 years ago by schwa

  • Description modified (diff)
  • Milestone set to 3.3

The FTP REST command was originally described in RFC 959, section 3.5. The RFC describes REST only in the context of the "block" and "compressed" modes, which aren't used by Python's ftplib. RFC 959 describes REST markers which don't have to consist of digits.

On the other hand, RFC 3659, chapter 5 describes an extension for the use of REST in "stream" mode. For binary transfer, the REST parameter is the 0-based byte count from where reading from or writing to a remote file should start.

For example, if a remote file contains the bytes

abcdefghijkl

(represented as ASCII letters here), a REST 3, followed by a RETR command, will read the string "defghijkl". A REST 3, followed by a STOR command, will overwrite the bytes starting at "d".

If the data written to the server has fewer bytes than the remaining bytes on the remote side, the "extra" remote bytes (here "jkl") will get lost. For example, writing "123" to the above remote file "abcdefghijkl" will result in the new remote content "abc123". I don't know if the shortening of the remote file is officially specified anywhere, but it's the behavior I see with my FTP server (Pure FTPd).

comment:6 Changed 3 years ago by schwa

As suggested in the ticket description, I'm going to add a rest argument to FTPHost.open (formerly file). For now, this argument will only be supported for binary transfer modes since applying the concept to text files may give confusing results.

Since ftputil is about simulating Python's API for local file systems, I had considered implementing the "offset" functionality with a seek method on FTPFile objects (the type of object returned by FTPHost.open). However, this is far more complicated to implement:

  • seek would have to reopen the file with a REST argument and an identical path.
  • Having a seek without tell (local files have a tell method) would be awkward.
  • Implementing tell would require keeping track of a counter when reading from or writing to a remote file.

All in all, implementing the "rest" functionality via a file seek method may be an interesting experiment, but offers little gain compared to the implementation effort.

Last edited 3 years ago by schwa (previous) (diff)

comment:7 Changed 3 years ago by schwa

  • Resolution set to fixed
  • Status changed from assigned to closed

I implemented passing rest from FTPHost.open to FTP.transfercmd in [3cb69eca3e41] and [66ef713f4fe8], [525a118b3085]. The documentation update is in [60662e7b943d].

The documentation contains the following caveat:

If you pass rest values which point after the file, the behavior is undefined and may even differ from one FTP server to another. Therefore, use the rest argument only for error recovery in case of interrupted transfers. You need to keep track of the transferred data so that you can provide a valid rest argument for a resumed transfer.

Note: See TracTickets for help on using tickets.