Opened 22 months ago

Closed 3 months ago

#110 closed defect (fixed)

Directory listings are empty if a server doesn't support the`-a` option

Reported by: schwa Owned by: schwa
Priority: major Milestone: 4.0.0
Component: Library Version: 3.3.1
Keywords: hiffen files, -a option, listdir Cc:

Description (last modified by schwa)

This issue comes up from time to time (for example http://lists.sschwarzer.net/pipermail/ftputil/2017q2/thread.html).

Since ftputil 2.8 the library by default tries to list hidden files (i. e. files starting with a dot) by using the -a option. (Everything in this ticket also applies to hidden directories.)

There are three cases:

  • The server supports the -a option and is configured to list hidden files. This is what we ideally want.
  • The server supports the -a option but is configured to not list hidden files. In this case, the directory listing misses hidden files, which is what the server intended.
  • The server does not support the -a option. In this case, the server understands -a as a file name.

This third case usually means that the returned listing won't have any entries. After all, ftputil told the server (from its perspective) to list the file -a and the file isn't there. This behavior is most obvious in FTPHost.listdir, but probably effects other functionality as well.

Change History (10)

comment:1 Changed 22 months ago by schwa

Summary: Directory listings fail if server doesn't support `-a` optionDirectory listings are empty if a server doesn't support the`-a` option

comment:2 Changed 22 months ago by schwa

You can switch off the automatic addition of the -a option by using

ftp_host = ftputil.FTPHost(server, user, password)
ftp_host.use_list_a_option = False

Although this kind of solves the problem, it's not what a user thinks of in the first place. With FTPHost.listdir, the returned list will be empty and from there it's not intuitive where to look for a solution. A user can read the documentation where the problem and workaround are described, but this probably won't happen smoothly because the user doesn't expect the problem to be related to hidden files.

comment:3 Changed 22 months ago by schwa

The most obvious remedy to this problem is not to use -a by default. This will have the consequence that even if the server supports the -a option and is configured to list hidden files when using the option, these files won't be listed. But in this case it's relatively clear that the problem is related to hidden files and I could describe in the documentation how to use the -a option.

A problem with this remedy, and the reason why I didn't implement it, is that it breaks backward compatibility. When some code used to use ftputil on a server supporting the -a option and allows to list hidden files, these files will be unexpectedly absent if ftputil changes the default to not include the -a option for listings.

comment:4 Changed 22 months ago by schwa

Mattias Bokesand came up with another suggestion to use a similar approach as for the "user-supported" timezone detection. By using some user-visible method, ftputil could try to write a "hidden" file to the current directory, check if it gets listed with -a and if yes, use the -a option from then on.

Obviously, this requires that the user has write permissions for the current directory.

If it's added, such a detection should not happen by default because I think it's bad practice to write something on the host without the user knowing about it. Moreover, some servers are configured to offer only write-only access to some directories and if ftputil is logged into such a directory, it won't be able to delete the file.

comment:5 Changed 22 months ago by schwa

Description: modified (diff)

comment:6 Changed 22 months ago by schwa

Description: modified (diff)

comment:7 Changed 18 months ago by schwa

Added deprecation warning in [93401904f9bc].

comment:8 in reply to:  3 Changed 4 months ago by schwa

Replying to schwa:

The most obvious remedy to this problem is not to use -a by default. This will have the consequence that even if the server supports the -a option and is configured to list hidden files when using the option, these files won't be listed. But in this case it's relatively clear that the problem is related to hidden files and I could describe in the documentation how to use the -a option.

A problem with this remedy, and the reason why I didn't implement it, is that it breaks backward compatibility. When some code used to use ftputil on a server supporting the -a option and allows to list hidden files, these files will be unexpectedly absent if ftputil changes the default to not include the -a option for listings.

ftputil 4.x will set use_list_a_option to False by default.

comment:9 Changed 3 months ago by schwa

Milestone: 4.0.0

comment:10 Changed 3 months ago by schwa

Resolution: fixed
Status: newclosed

This is changed in master (changeset [766c15f83205]).

Note: See TracTickets for help on using tickets.