Opened 6 years ago

Closed 6 years ago

#65 closed defect (fixed)

Auto-probing of LIST's -a option is faulty (Ticket #23)

Reported by: ftputiluser Owned by: schwa
Priority: major Milestone: 2.8
Component: Library Version: 2.7.1
Keywords: list, -a, hidden files Cc:

Description (last modified by schwa)

My FTP server does not seem to like the -a option but ftplib does not throw an exception which is what ftputil is expecting. The result is that an empty list is returned whenever LIST -a is used.

I made a local fix by commenting out the call to _check_list_a_option in FTPHost.__init__.

Change History (8)

comment:1 Changed 6 years ago by ftputiluser

  • Summary changed from auto-probing of LIST's -a option is faulty (Ticket #23) to Auto-probing of LIST's -a option is faulty (Ticket #23)

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win 32 Type "help", "copyright", "credits" or "license" for more information.

import ftplib ftp = ftplib.FTP(..) ftp.cwd(..)

'250 CWD command successful'

ftp.retrlines('LIST')

drw-rw-rw- 1 cidsauto cidsauto 1 Sep 24 17:51 Agents drw-rw-rw- 1 cidsauto cidsauto 1 Sep 24 17:51 CentralServer? drw-rw-rw- 1 cidsauto cidsauto 1 Sep 24 17:51 Packages drw-rw-rw- 1 cidsauto cidsauto 1 Sep 24 17:51 Working -rw-rw-rw- 1 cidsauto cidsauto 3099 Aug 1 19:39 LUXStyleSheet.xsl '226 Transfer complete.'

ftp.retrlines('LIST -a')

'226 Transfer complete.'

comment:2 follow-up: Changed 6 years ago by schwa

  • Description modified (diff)
  • Keywords list -a hidden files added
  • Status changed from new to assigned

Thanks for the report!

What you describe is a general limitation of the auto-probing. When I implemented it, I wasn't able to come up with a reliable way so that it would work under all circumstances, for all servers (see last comment in ticket #23). On the other hand, I still added support for -a because not supporting it had other implications (see ticket #63). I assumed that most likely a server would either support the -a option, ignore it or cause an error condition if used.

Which FTP server and version do you use? Is there any documentation on how the server interprets the LIST -a option or even if there's a supposed way to handle any LIST options?

I can think of two approaches now:

  • Remove auto-probing and let the user switch on using -a explicitly if he/she wants to get hidden files or directories.
  • Auto-probe by default, but give the user a way not to use it, for example with a keyword argument in the FTPHost constructor.

I tend to use the second approach because then ftputil will behave "as expected" for most users. On the other hand, if I knew that there were many servers behaving as yours, I would prefer the first approach. But maybe there are other approaches?

(I edited the description field to use code markup, so __init__ doesn't appear as underlined init. Unfortunately, I can't edit your comment as easily.)

comment:3 in reply to: ↑ 2 Changed 6 years ago by schwa

Replying to schwa:

I can think of two approaches now:

  • Remove auto-probing and let the user switch on using -a explicitly if he/she wants to get hidden files or directories.
  • Auto-probe by default, but give the user a way not to use it, for example with a keyword argument in the FTPHost constructor.

Given the unrealiability of auto-probing, I'm aware that from a software design point of view the first approach is clearly preferable. But the second approach might still be better from the point of view of overall usefulness for all users. In my experience, many users don't look at the documentation if something doesn't behave as it "obviously" should and maybe won't use the library at all.

comment:4 follow-up: Changed 6 years ago by ftputiluser

This is a Perforce FTP server. It is managed by our IT so unfortunately I do not have intimate knowledge of the actual host.

I am not sure about the prevalence of this problem. I will say that if a new user falls victim to this, it is very likely that they won't use the library either.

comment:5 in reply to: ↑ 4 ; follow-up: Changed 6 years ago by schwa

Replying to ftputiluser:

This is a Perforce FTP server. It is managed by our IT so unfortunately I do not have intimate knowledge of the actual host.

Thanks nonetheless. :-)

I am not sure about the prevalence of this problem. I will say that if a new user falls victim to this, it is very likely that they won't use the library either.

That's true. On the other hand, I assume it's very unlikely that LIST -a leads to empty directory listings. So my assumption is that of the two cases

  • a user doesn't see hidden files and abandons the library vs.
  • a user gets an empty directory listing because of use of LIST -a and abandons the library,

the first one is more likely.

Besides, after I wrote my reply I thought more about the problem. Maybe it's a reasonable compromise to add a check in the auto-probing, so that if there are fewer entries in the directory listing if LIST -a is used, I'll drop the -a option.

In the "extreme" case that the directory is empty even if using LIST alone, I could attempt to write a "hidden" file like ._ftputil_hidden_ and if the writing succeeds but the file still doesn't show up, don't use the -a option. If the user doesn't have write permission, I would rather stick with using the -a option because of what I said above. (Note that this default then only applies if the directory seemed empty even without -a.)

In addition to that I'm still not against a way to override the auto-probing result. :-)

What do you think?

comment:6 in reply to: ↑ 5 ; follow-up: Changed 6 years ago by ftputiluser

Well, I think those are interesting ideas but I would not try to fix something like this by adding more code. I was able to figure out this problem because the code was well written and straightforward. Adding more layers on top layers to hide a basic limitation is not the path I would take.

comment:7 in reply to: ↑ 6 ; follow-up: Changed 6 years ago by schwa

Replying to ftputiluser:

Well, I think those are interesting ideas but I would not try to fix something like this by adding more code.

Thanks for your feedback.

I was able to figure out this problem because the code was well written and straightforward.

Thanks again! :-)

Adding more layers on top layers to hide a basic limitation is not the path I would take.

I think for a library like ftputil to be useful you have to jump through hoops (at least a few). Imagine a web server that serves only clients strictly adhering to the applicable RFCs. I don't fulfill each and every wish though (see #26, #58, #60). ;-)

Regarding the LIST -a heuristics, it came to my mind that in addition to being more code, the LIST calls upon connection burden everyone making a connection with ftputil, even if LIST -a would work without problems or not having any effect. Clearly, for only a few entries in the login directory it won't matter, but for larger directories it might.

So I guess I'll remove the test I now have, activate use of -a by default, offer a boolean attribute to deactivate the use of -a and document this flag.

comment:8 in reply to: ↑ 7 Changed 6 years ago by schwa

  • Milestone set to 2.8
  • Resolution set to fixed
  • Status changed from assigned to closed

Replying to schwa:

So I guess I'll remove the test I now have, activate use of -a by default, offer a boolean attribute to deactivate the use of -a and document this flag.

I did this in changesets [74bde432d0ec] for the code and [6d7ff24a9288] for the documentation.

Note: See TracTickets for help on using tickets.