path.isdir raises exception for dir in non-existent path, unlike os.path

Hi Julian,

I can't confirm the bug:

>>> import ftputil
>>> h = ftputil.FTPHost("localhost", ..., ...)
>>> h.path.isdir("hrgknebkekrghi3w")
False

There are also testcases for that in _test_ftp_path.py.

I notice a change with respect to ftputil 2.2.2 in the traceback line

ftp_error._try_with_oserror(self._session.dir, '-a '+path, callback)

ftputil originally doesn't use "-a" here. Perhaps that's the cause of your observation?

Hello Julian,

I just see that the "-a patch" is mentioned in issue #23. I've looked at the defects before the enhancement request, so I saw the latter only now.

hi schwa,

yeah sorry about the "-a" in the Traceback output, forgot about that ;). However, I had the bug already before I had added that. And I hadn't changed any other part of the module.

Can you please try to change your test like this (i.e. put a '/' in the path):

>>> import ftputil
>>> h = ftputil.FTPHost("localhost", ..., ...)
>>> h.path.isdir("hrgkneb/kekrghi3w") # <---- notice the slash!!!

Your example runs fine, according to my report, because the dirname of "hrgknebkekrghi3w" is ""(the current working dir) which does exists. Problems arise when then dirname doesn't exist, such as for "hrgkneb/kekrghi3w" where the dirname of the path is "hrgkneb". Here ftputil does throw an exception where os wouldn't. By the way, i have removed the whole ftputil directory and downloaded version 2.2.3 just to be sure I didn't adjust any of the original code, and I reproduced this bug.

Thanks for the help

julian

hi schwa,

My name is Stefan Schwarzer. I thought this would be obvious from the documentation and the source files. ;-)

Regarding your report, I get with ftputil 2.2.3:

>>> import ftputil
>>> h = ftputil.FTPHost("localhost", 'ftptest', 'd605581757de5eb56d568a4419f4126e')
>>> h.path.isdir("hrgkneb/kekrghi3w")
False

Could you please post the full traceback you get with ftputil 2.2.3?

I just think schwa is a neat nick :D. Ok Stefan, here is my traceback:

julian@julian-laptop:~/develop/clftp/trunk/tests$ python
Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ftputil
>>> h = ftputil.FTPHost("localhost", 'julian', 'thisaintit')
>>> h.path.isdir("hrgkneb/kekrghi3w")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_path.py", line 116, in isdir
    path, _exception_for_missing_path=False)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 794, in stat
    return self._stat.stat(path, _exception_for_missing_path)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 537, in stat
    _exception_for_missing_path)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 514, in __call_with_parser_retry
    result = method(*args, **kwargs)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 481, in _real_stat
    lstat_result = self._real_lstat(path, _exception_for_missing_path)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 436, in _real_lstat
    lines = self._host_dir(dirname)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 367, in _host_dir
    return self._host._dir(path)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 751, in _dir
    lines = self._robust_ftp_command(command, path, descend_deeply=True)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 551, in _robust_ftp_command
    return command(self, path)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 749, in command
    ftp_error._try_with_oserror(self._session.dir, path, callback)
  File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_error.py", line 84, in _try_with_oserror
    raise TemporaryError(obj)
ftputil.ftp_error.TemporaryError: 450 /hrgkneb: No such file or directory
Debugging info: ftputil 2.2.3, Python 2.5.1 (linux2)
>>> 

Also, I logged into the computers at my university and download (wget), extracted ftputil on a machine there, renamed ftputil-2.2.3 to ftputil, and repeated the test. This machine was using python 2.3.4:

Linux ow126 2.6.12-1.1381_FC3smp #1 SMP Fri Oct 21 04:03:26 EDT 2005 i686 i686 i386 GNU/Linux

and i got in fact the exact same error, and traceback. Just for the record, till now I worked at home with ftputil as a subdir in the folder I'm working, so without installing it. So far, it has worked great. To be sure I installed it with "sudo setup.py install" too, restarted the terminal ... but no luck. Couldn't install it at the university because of rights and everything.

Any other suggestions / test code?

julian

Julian,

Could you send me a small standalone script (i. e. runnable with python mytest.py) which reproduces the problem? Try to minimize setup preconditions; possibly, you can contact a public FTP server for that, so that I don't have anything to have installed but Python and ftputil. I suggest you attach the script to this ticket.

Regarding the installation of ftputil at the university: You can pass additional options to python setup.py install to install a package in your home directory or somewhere else. After setting $PYTHONPATH, this should work fine.

Stefan

Besides, what's you last name, if you don't mind telling me? (And is Julian your real name?) ;-)

He Stefan, I attached a small test program i wrote (mytest.py).

As I was testing it I found out that a lot of ftp sites, especially public ones, indeed don't give any problems. On most sites the FTPHost.path.isdir function works fine (so you were right)! However, my localhost and the remote non-public servers I actually always use for testing do suffer from this bug (i was right too!). So I have to say this bug is clearly not universal, but server specific. And it's not the OS. Maybe some apache setting(?), but I don't know enough about the technical details to say and I'm to tired now to find out.

Well I used http://www.ftp-sites.org/ to try different to look for public ftp sites that would raise the error and luckily I found some to share with you (although, again, most sites deed indeed work fine).

The code will try some path.isdir tests on several sites. The first site, ftp.gnome.org, doesn't raise an exception. But then it will go on and try these bad sites:

ftp.adsl4linux.de
ftp.chello.nl
ftp.2600.com
ftp.3k.com

now for me, the first one of these will already result in abnormal program termination with the traceback i posted earlier. Please tell me you get the same this time :D. I hope it helps!!!

My full name is Julian Kooij by the way, but my friends call me ftputiluser.

right, so it's just a FTP error (450: File not Found) raised by ftplib.FTP. You can obtain this "bug" without ftputil:

import ftplib

lines = list()

host = ftplib.FTP('ftp.gnome.org', 'anonymous', '')
host.dir('xyz', lines.append) # <-- appends nothing to lines
host.close()
print lines

host = ftplib.FTP('ftp.adsl4linux.de', 'anonymous', '')
host.dir('xyz', lines.append) # <-- raises the ftplib.error_temp 450 exception
host.close()
print lines

I guess some servers just return an empty dir listing for non existent directories and others a specific FTP error code: it's not a bug, it's a feature. Anyway, I believe ftputil should handle these discrepancies, right?

Thanks,

Julian

Hi Julian,

Thanks for following up. Your friends calling you ftputiluser is nice. Perhaps you want to enter your name and your ftputil uses under FtputilUsers. :-)

"Originally", the traceback occurs in ftp_stat.py in line 436 (method _real_lstat), when self._host_dir(dirname) is called. The problematic servers use a status code 450 for paths that are not found. The FTP server I use, Pure-FTPd, selects 550.

The RFC 959 which defines the FTP standard, describe the status codes 450 and 550 as:

• 450: Requested file action not taken. File unavailable (e.g. file busy).
• 550: Requested action not taken. File unavailable (e.g. file not found, no access).

So, in my opinion, 550 for a missing directory is a much better choice.

That said, so far I haven't managed to come up with a solid way to fix this problem. Since ftputil's stat call isn't a "atomic" function but uses several FTP commands, these individual commands may cause their own temporary or permanent errors. Furthermore, since I can't determine the real cause of such errors reliably, it's brittle to interpret in the context of the called ftputil method what a particular exception actually means.

For example, if I change the above-mentioned _host_dir call to

try:
    lines = self._host_dir(dirname)
except (ftp_error.TemporaryError, ftp_error.PermanentError):
    if _exception_for_missing_path:
        raise
    else:
        return None

I'd implicitly assume that every FTP error in the DIR command stems from an unfound path while it may really come from a directory which is present but not accessible for the client.

The inconsistency enters in line 451 when I raise a "synthetic" PermanentError for names which aren't found in the directory listing. (Until now, I've assumed 550 would be the "obvious" status code here.) Determining in advance for a server if it uses 450 or 550 for missing paths to be consistent would be rather difficult or impossible. I don't think it's worth the effort.

At the moment, I tend to close this bug as "wontfix" and just describe the problem in the documentation. What do you think?

Thanks for your interest in examining this problem. :-)

A side note: The list() in your ftplib example code is a rather non-idiomatic way to say []. ;-)