What’s new in ftputil 4.0.0?

Version: 4.0.0
Date: 2020-06-13
Author: Stefan Schwarzer

Supported Python versions

Support for Python 2 is dropped. The minimum required Python 3 version is 3.6.

Find more details in Questions and answers.

Path-like objects

Methods that take directory and file names as str or bytes objects now also accept path-like objects.

Time shift handling

ftputil uses the notion of “time shift” to deal with time zone differences between client and server. This is important for the methods upload_if_newer and download_if_newer.

The defintion of “time shift” changed from earlier ftputil versions to ftputil 4.0.0.

Previously, the time shift was defined as

time_used_by_server – local_time_used_by_client

The new definition is

time_used_by_server – UTC

Both definitions have their pros and cons (detailed in the Questions and answers).

Porting

If you don’t use any methods that deal with time stamps from the FTP server, you can ignore the time shift redefinition. The affected methods are upload_if_newer, download_if_newer and using the timestamp values from the stat and lstat methods.

Ideally, you have write access on the server in the current directory. In this case you can call synchronize_times:

with ftputil.FTPHost(host, user, password) as ftp_host:
    ftp_host.synchronize_times()
    ...

If using synchronize_times isn’t an option, you have to set the time shift explicitly with set_time_shift:

with ftputil.FTPHost(host, user, password) as ftp_host:
    ftp_host.set_time_shift(new_time_shift)
    ...

If you’re sure the server in its directory listings uses the same timezone as the client, you can use

with ftputil.FTPHost(host, user, password) as ftp_host:
    ftp_host.set_time_shift(
      round((datetime.datetime.now() - datetime.datetime.utcnow()).seconds, -2)
    )
    ...

This is roughly equivalent to the old ftputil behavior. The only difference is that the new behavior requires that you adapt the time shift value if there’s a switch to or from daylight saving time.

If you want to cover such daylight saving time switches as well, you could override time_shift() to return the above value. This will automatically adapt to daylight saving time switches because datetime.datetime.now() will return the time for the changed UTC offset. That said, I think this approach is a hack because it means that calls to set_time_shift will effectively be ignored. So don’t take this approach as a recommendation.

You can use this script to scan your code for uses of time_shift() or set_time_shift. However, if you implicitly relied on the old default behavior (time shift is 0.0 if client and server use the same time zone), you’ll need additional calls for set_time_shift.

ftputil no longer uses the -a option by default

Earlier ftputil versions by default sent an -a option with the FTP DIR command to include “hidden” directories and files (names starting with a dot) in the listing.

That led to problems when the server didn’t understand the option and treated it as a directory or file name.

Therefore, ftputil no longer uses the -a option by default.

Porting

You can enable the old behavior by setting use_list_a_option on the FTPHost instance to True:

with ftputil.FTPHost(host, user, password) as ftp_host:
    ftp_host.use_list_a_option = True
    ...

However, do this only if you’re sure the server interprets the option correctly!

makedirs behaves as in Python 3

In Python 2, os.makedirs didn’t complain if any directory in the path to create already existed. Since ftputil was originally based on Python 2 behavior, this was also the behavior in FTPHost.makedirs.

Python 3 added an optional argument exist_ok with the default False. With this default, os.makedirs raises an exception if any directory but the last in the path argument exists.

Since Python 2 is used less and less, ftputil 4.0.0 follows the Python 3 semantics.

Porting

If you want the old behavior of FTPHost.makedirs, pass exist_ok=True. Note that there’s also an unused mode argument for consistency with the os.makedirs API, so make sure you pass exist_ok as a keyword argument.

You can use this script to scan your code for uses of makedirs().

Questions and answers

Why is Python 2 no longer supported?

Since the start of the year Python 2 officially is no longer maintained and supporting it in combination with Python 3 led to lots of extra work. Therefore, I decided to drop support for Python 2.

Why is the minimum version Python 3.6?

Python 3.4 and older versions are no longer supported by the CPython team.

My plan was to support Python 3.5 since it’s not yet end-of-life’d and it may still be used in some LTS Linux/Unix distributions. However, dropping Python 3.5 support made it much easier to implement ticket #119, support for path-like objects. Python 3.6 introduced some infrastructure so that code that used to use only str and bytes paths can now use path-like objects as well. I considered it more important to support path-like objects than Python 3.5. I guess it might have been possible to add support for path-like objects on top of Python 3.5, but it would have been a hassle and Python 3.5 support officially ends in just a few months.

Why a new time shift definition?

Both the old and the new approach have their pros and cons:

Regarding the old approach:

  • Pro: If the server uses the time zone of the client in directory listings, the default time shift of 0.0 will do the right thing.

  • Con: Since the time shift depends on two time zones, it’s more difficult to reason about the (ftputil) code and write correct code. In the past, there have been multiple bugs in the time zone / time shift handling in ftputil, some of them occuring only under “interesting” conditions (around daylight saving time changes or year changes).

    Actually, I decided to implement the new approach when I ran into a bug around the last new year change.

Regarding the new approach:

  • Pro: The new behavior works better when the server is set to UTC. This is a sane setting because it avoids that an hour interval is used twice when there’s a switch from daylight saving time to “normal” time. There are even more subtle problems with daylight saving time switches.
  • Pro: Since the time shift calculation depends on only one time zone (that of the server), the code in ftputil is easier to reason about and now hopefully more robust.
  • Con: The new approach is backward-incompatible, so users may have to adapt their code.
  • Con: If the server uses the time zone of the client in directory listings, the time shift must be adjusted whenever there’s a switch to or from daylight saving time.

Neither of the two approaches is foolproof. For example, timestamps that are older than the last daylight saving time switch may be calculated wrongly because they may use a different time zone than the one currently set by set_time_shift.