ftputil version: 2.2.2
python: 2.5.1
ftpserver: proftpd on localhost
OS: Linux julian-laptop 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
The os and ftputil modules don't work the same when it comes to isdir: ftputil's path.isdir(d) raises an TemporaryError? if the os.path.dirname of the variable d is non-existent path. Thus, assuming the working dirs of os and ftputil are empty directories, this would be an example:
import ftputil
import os
h = ftputil.FTPHost(server, user, password)
print os.path.isdir('foo') # outputs False
print h.path.isdir('foo') # outputs False
print os.path.isdir('foo/bar') # outputs False
print h.path.isdir('foo/bar')
# raises ftputil.ftp_error.TemporaryError on 'foo'
# another example:
print os.path.isdir('foo/bar/helloworld') # outputs False
print h.path.isdir('foo/bar/helloworld')
# raises ftputil.ftp_error.TemporaryError on 'foo/bar'
traceback output from python console:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/julian/develop/python/ftputil/ftp_path.py", line 116, in isdir
path, _exception_for_missing_path=False)
File "/home/julian/develop/python/ftputil/ftputil.py", line 793, in stat
return self._stat.stat(path, _exception_for_missing_path)
File "/home/julian/develop/python/ftputil/ftp_stat.py", line 537, in stat
_exception_for_missing_path)
File "/home/julian/develop/python/ftputil/ftp_stat.py", line 514, in __call_with_parser_retry
result = method(*args, **kwargs)
File "/home/julian/develop/python/ftputil/ftp_stat.py", line 481, in _real_stat
lstat_result = self._real_lstat(path, _exception_for_missing_path)
File "/home/julian/develop/python/ftputil/ftp_stat.py", line 436, in _real_lstat
lines = self._host_dir(dirname)
File "/home/julian/develop/python/ftputil/ftp_stat.py", line 367, in _host_dir
return self._host._dir(path)
File "/home/julian/develop/python/ftputil/ftputil.py", line 750, in _dir
lines = self._robust_ftp_command(command, path, descend_deeply=True)
File "/home/julian/develop/python/ftputil/ftputil.py", line 551, in _robust_ftp_command
return command(self, path)
File "/home/julian/develop/python/ftputil/ftputil.py", line 748, in command
ftp_error._try_with_oserror(self._session.dir, '-a '+path, callback)
File "/home/julian/develop/python/ftputil/ftp_error.py", line 83, in _try_with_oserror
raise TemporaryError(obj)
ftputil.ftp_error.TemporaryError: 450 /foo: No such file or directory
Debugging info: ftputil 2.2.2, Python 2.5.1 (linux2)
>>>
ok, thanks ;)
julian
Hi Julian,
I can't confirm the bug:
>>> import ftputil >>> h = ftputil.FTPHost("localhost", ..., ...) >>> h.path.isdir("hrgknebkekrghi3w") False
There are also testcases for that in
_test_ftp_path.py
.I notice a change with respect to ftputil 2.2.2 in the traceback line
ftp_error._try_with_oserror(self._session.dir, '-a '+path, callback)
ftputil originally doesn't use "-a" here. Perhaps that's the cause of your observation?
Hello Julian,
I just see that the "-a patch" is mentioned in issue #23. I've looked at the defects before the enhancement request, so I saw the latter only now.
hi schwa,
yeah sorry about the "-a" in the Traceback output, forgot about that ;). However, I had the bug already before I had added that. And I hadn't changed any other part of the module.
Can you please try to change your test like this (i.e. put a '/' in the path):
>>> import ftputil >>> h = ftputil.FTPHost("localhost", ..., ...) >>> h.path.isdir("hrgkneb/kekrghi3w") # <---- notice the slash!!!
Your example runs fine, according to my report, because the dirname of "hrgknebkekrghi3w" is ""(the current working dir) which does exists. Problems arise when then dirname doesn't exist, such as for "hrgkneb/kekrghi3w" where the dirname of the path is "hrgkneb". Here ftputil does throw an exception where os wouldn't. By the way, i have removed the whole ftputil directory and downloaded version 2.2.3 just to be sure I didn't adjust any of the original code, and I reproduced this bug.
Thanks for the help
julian
hi schwa,
My name is Stefan Schwarzer. I thought this would be obvious from the documentation and the source files. ;-)
Regarding your report, I get with ftputil 2.2.3:
>>> import ftputil >>> h = ftputil.FTPHost("localhost", 'ftptest', 'd605581757de5eb56d568a4419f4126e') >>> h.path.isdir("hrgkneb/kekrghi3w") False
Could you please post the full traceback you get with ftputil 2.2.3?
I just think schwa is a neat nick :D. Ok Stefan, here is my traceback:
julian@julian-laptop:~/develop/clftp/trunk/tests$ python Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import ftputil >>> h = ftputil.FTPHost("localhost", 'julian', 'thisaintit') >>> h.path.isdir("hrgkneb/kekrghi3w") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_path.py", line 116, in isdir path, _exception_for_missing_path=False) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 794, in stat return self._stat.stat(path, _exception_for_missing_path) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 537, in stat _exception_for_missing_path) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 514, in __call_with_parser_retry result = method(*args, **kwargs) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 481, in _real_stat lstat_result = self._real_lstat(path, _exception_for_missing_path) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 436, in _real_lstat lines = self._host_dir(dirname) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_stat.py", line 367, in _host_dir return self._host._dir(path) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 751, in _dir lines = self._robust_ftp_command(command, path, descend_deeply=True) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 551, in _robust_ftp_command return command(self, path) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftputil.py", line 749, in command ftp_error._try_with_oserror(self._session.dir, path, callback) File "/home/julian/develop/clftp/trunk/tests/ftputil/ftp_error.py", line 84, in _try_with_oserror raise TemporaryError(obj) ftputil.ftp_error.TemporaryError: 450 /hrgkneb: No such file or directory Debugging info: ftputil 2.2.3, Python 2.5.1 (linux2) >>>
Also, I logged into the computers at my university and download (wget), extracted ftputil on a machine there, renamed ftputil-2.2.3 to ftputil, and repeated the test. This machine was using python 2.3.4:
Linux ow126 2.6.12-1.1381_FC3smp <a href="/~sschwarzer/ftputil/1" title="~sschwarzer/ftputil#1: Add russian documentation to wiki (Anton Stepanov)">#1</a> SMP Fri Oct 21 04:03:26 EDT 2005 i686 i686 i386 GNU/Linux
and i got in fact the exact same error, and traceback. Just for the record, till now I worked at home with ftputil as a subdir in the folder I'm working, so without installing it. So far, it has worked great. To be sure I installed it with "sudo setup.py install" too, restarted the terminal ... but no luck. Couldn't install it at the university because of rights and everything.
Any other suggestions / test code?
julian
Julian,
Could you send me a small standalone script (i. e. runnable with
python mytest.py
) which reproduces the problem? Try to minimize setup preconditions; possibly, you can contact a public FTP server for that, so that I don't have anything to have installed but Python and ftputil. I suggest you attach the script to this ticket.Regarding the installation of ftputil at the university: You can pass additional options to
python setup.py install
to install a package in your home directory or somewhere else. After setting$PYTHONPATH
, this should work fine.Stefan
Besides, what's you last name, if you don't mind telling me? (And is Julian your real name?) ;-)
He Stefan, I attached a small test program i wrote (mytest.py).
As I was testing it I found out that a lot of ftp sites, especially public ones, indeed don't give any problems. On most sites the FTPHost.path.isdir function works fine (so you were right)! However, my localhost and the remote non-public servers I actually always use for testing do suffer from this bug (i was right too!). So I have to say this bug is clearly not universal, but server specific. And it's not the OS. Maybe some apache setting(?), but I don't know enough about the technical details to say and I'm to tired now to find out.
Well I used http://www.ftp-sites.org/ to try different to look for public ftp sites that would raise the error and luckily I found some to share with you (although, again, most sites deed indeed work fine).
The code will try some path.isdir tests on several sites. The first site, ftp.gnome.org, doesn't raise an exception. But then it will go on and try these bad sites:
ftp.adsl4linux.de ftp.chello.nl ftp.2600.com ftp.3k.com
now for me, the first one of these will already result in abnormal program termination with the traceback i posted earlier. Please tell me you get the same this time :D. I hope it helps!!!
My full name is Julian Kooij by the way, but my friends call me ftputiluser.
right, so it's just a FTP error (450: File not Found) raised by ftplib.FTP. You can obtain this "bug" without ftputil:
import ftplib lines = list() host = ftplib.FTP('ftp.gnome.org', 'anonymous', '') host.dir('xyz', lines.append) # <-- appends nothing to lines host.close() print lines host = ftplib.FTP('ftp.adsl4linux.de', 'anonymous', '') host.dir('xyz', lines.append) # <-- raises the ftplib.error_temp 450 exception host.close() print lines
I guess some servers just return an empty dir listing for non existent directories and others a specific FTP error code: it's not a bug, it's a feature. Anyway, I believe ftputil should handle these discrepancies, right?
Thanks,
Julian
Hi Julian,
Thanks for following up. Your friends calling you ftputiluser is nice. Perhaps you want to enter your name and your ftputil uses under FtputilUsers. :-)
"Originally", the traceback occurs in ftp_stat.py in line 436 (method
_real_lstat
), whenself._host_dir(dirname)
is called. The problematic servers use a status code 450 for paths that are not found. The FTP server I use, Pure-FTPd, selects 550.The RFC 959 which defines the FTP standard, describe the status codes 450 and 550 as:
- 450: Requested file action not taken. File unavailable (e.g. file busy).
- 550: Requested action not taken. File unavailable (e.g. file not found, no access).
So, in my opinion, 550 for a missing directory is a much better choice.
That said, so far I haven't managed to come up with a solid way to fix this problem. Since ftputil's
stat
call isn't a "atomic" function but uses several FTP commands, these individual commands may cause their own temporary or permanent errors. Furthermore, since I can't determine the real cause of such errors reliably, it's brittle to interpret in the context of the called ftputil method what a particular exception actually means.For example, if I change the above-mentioned
_host_dir
call totry: lines = self._host_dir(dirname) except (ftp_error.TemporaryError, ftp_error.PermanentError): if _exception_for_missing_path: raise else: return None
I'd implicitly assume that every FTP error in the DIR command stems from an unfound path while it may really come from a directory which is present but not accessible for the client.
The inconsistency enters in line 451 when I raise a "synthetic"
PermanentError
for names which aren't found in the directory listing. (Until now, I've assumed 550 would be the "obvious" status code here.) Determining in advance for a server if it uses 450 or 550 for missing paths to be consistent would be rather difficult or impossible. I don't think it's worth the effort.At the moment, I tend to close this bug as "wontfix" and just describe the problem in the documentation. What do you think?
Thanks for your interest in examining this problem. :-)
A side note: The
list()
in your ftplib example code is a rather non-idiomatic way to say[]
. ;-)