When syncing, some dir named ".hg" 's isdir
test will throw
PermanentError
, which it should not (the dir is there on FTP), and it
breaks the sync process.
It might be the same problem as ticket
#39,
"isdir and isfile on broken links with special targets cause a
PermanentError
".
Remark by ftputil author: It turned out the exception wasn't raised
by isdir
but by the subsequent _mkdir
call. isdir
gives a
seemingly wrong result here because the directory does exist, - but
it's not "seen" by isdir
as the directory with a leading dot doesn't
show up in the parent directory listing.
As this ticket turned to a discussion about showing hidden files and directories, please add further comments to ticket #23 if necessary.
Thanks for the report.
What's the ftputil version you're using? Could you please add the full traceback for the exception to this ticket? As for ticket #62, a small runnable script surely would help reproduce the problem and make sure we're on the same page. :-)
Replying to schwa:
Thanks for the report.
What's the ftputil version you're using? Could you please add the full traceback for the exception to this ticket? As for ticket #62, a small runnable script surely would help reproduce the problem and make sure we're on the same page. :-)
Thanks for looking into this. Below is the reproduce scenario:
- Remote FTP Server: Mac OS X Lion
- Local Client: Window 7
- ftputil verison: current tip of hg repository
- Local directory: dir/.hg
- Remote directory already has dir/.hg
The script is as simple as:
with FTPHost(host, user, password) as remote: local = LocalHost() syncer = Syncer(local, remote) syncer.sync(local_directory, remote_path)
The program will break at ftp_sync.py line 79:
if not self._target.path.isdir(target_dir): self._target.mkdir(target_dir)
isdir should return True because ".hg" already exists on remote FTP server, however, it return False. and the following mkdir call will throw PermanentError?.
My quick fix is:
if not self._target.path.isdir(target_dir): try: self._target.mkdir(target_dir) except ftp_error.PermanentError, e: # print 'target dir exists', target_dir pass
Maybe I should use a
try/except
clause instead of theif
statement to begin with. But first I have to know what exactly happens.Can you please add a
mkdir
call for me?
The argument for mkdir call is:
upload/.hg
The first sync runs OK since "upload/.hg" dir is not yet exists:
Making /upload Making /upload\.hg Syncing e:\a\.hg\file.name -> /upload\.hg\file.name Syncing e:\a\.hg\thgstatus -> /upload\.hg\thgstatus
However, on second run, it will throw PermanentError? since that directory already exists on FTP.
Traceback (most recent call last): File "E:\work\ftp_uploader\test.py", line 3, in <module> sync('e:\\a', 'localhost', '/upload', 'will', 'zhuo') File "E:\work\ftp_uploader\ftp_uploader.py", line 27, in sync n = syncer.sync(local_directory, remote_path) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 149, in sync return self._sync_tree(source_path, target_path) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 125, in _sync_tree self._mkdir(inner_target_dir) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 80, in _mkdir self._target.mkdir(target_dir) File "E:\work\ftp_uploader\ftputil\__init__.py", line 622, in mkdir self._robust_ftp_command(command, path) File "E:\work\ftp_uploader\ftputil\__init__.py", line 587, in _robust_ftp_comm and return command(self, tail) File "E:\work\ftp_uploader\ftputil\__init__.py", line 621, in command return ftp_error._try_with_oserror(self._session.mkd, path) File "E:\work\ftp_uploader\ftputil\ftp_error.py", line 136, in _try_with_oserr or raise PermanentError(*exc.args) ftputil.ftp_error.PermanentError: 550 Directory already exists
I attached an
ftp_sync.py
to ticket #62 which should fix this bug as well. Please try that file as described there in ticket #62, [comment 8](https://todo.sr.ht/~sschwarzer/ftputil/62 "Comment 8 for #62: defect: FTP upload path not correct using ftp_sync.py under win32 (closed: fixed)").
Thanks for your effort, however, the fix does not work. In case ".hg" is already an existing dir on FTP server, try to sync ".hg" shows the following error:
Making /upload Fixing inner_target_dir from /upload\.hg to /upload/.hg Making /upload/.hg Traceback (most recent call last): File "E:\work\ftp_uploader\test.py", line 3, in <module> sync('e:\\a', '10.64.81.67', '/upload', 'will', 'zhuo') File "E:\work\ftp_uploader\ftp_uploader.py", line 27, in sync n = syncer.sync(local_directory, remote_path) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 153, in sync self._sync_tree(source_path, target_path) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 130, in _sync_tree self._mkdir(inner_target_dir) File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 80, in _mkdir self._target.mkdir(target_dir) File "E:\work\ftp_uploader\ftputil\__init__.py", line 622, in mkdir self._robust_ftp_command(command, path) File "E:\work\ftp_uploader\ftputil\__init__.py", line 587, in _robust_ftp_comm and return command(self, tail) File "E:\work\ftp_uploader\ftputil\__init__.py", line 621, in command return ftp_error._try_with_oserror(self._session.mkd, path) File "E:\work\ftp_uploader\ftputil\ftp_error.py", line 136, in _try_with_oserr or raise PermanentError(*exc.args) ftputil.ftp_error.PermanentError: 550 Can't create directory: File exists Debugging info: ftputil 2.6, Python 2.7 (win32)
Regards, ZHUO Qiang <zhuo.qiang@…>
Replying to ftputiluser:
Thanks for your effort, however, the fix does not work. In case ".hg" is already an existing dir on FTP server, try to sync ".hg" shows the following error:
I wonder why the code
if not self._target.path.isdir(target_dir): self._target.mkdir(target_dir)
in
_mkdir
doesn't work. This is supposed to handle the "already existing directory" case.Would you please add a
if
statement like this:print "isdir(%r): %s" % (target_dir, self._target.path.isdir(target_dir))
and put the output in this ticket?
Maybe it's more "pythonic" to use a
try/except
statement like you suggested. But before I change the current code I'd like to find out why it doesn't work as expected. If I just change the code it might mask other problems we haven't thought of yet. (I try to avoid programming by coincidence. :-) )
I think the problem is, for those hidden directories, FTPHost.path.isdir returns False even if they exists.
Here's my test scripts:
with FTPHost('localhost', 'will', 'zhuo') as remote: print 'Normal Dir ', remote.path.isdir('normal') print 'Hidden Dir ', remote.path.isdir('.hidden')
"normal" and ".hidden" are all existing directories. However, the output is:
Normal Dir True Hidden Dir False
Replying to ftputiluser:
I think the problem is, for those hidden directories, FTPHost.path.isdir returns False even if they exists. "normal" and ".hidden" are all existing directories. However, the output is: [...]
Wow, I didn't expect this at all.
Here are some other tests, using my local FTP server (PureFTPd 1.0.28):
root /home/ftptest# mkdir .hidden
$ ipypython ... >>> import ftputil >>> h = ftputil.FTPHost("localhost", 'ftptest', 'd605581757de5eb56d568a4419f4126e') >>> h.listdir(".") ['CONTENTS', 'broken_link', 'debian-keyring.tar.gz', 'dir with spaces', 'dir_with_broken_link', 'rootdir1', 'rootdir2', 'valid_link', 'walk_test']
The directory
.hidden
isn't in the list.In a command line FTP client I get:
$ ftp localhost Connected to localhost.localdomain. ... 230-User ftptest has group access to: 1004 230 OK. Current directory is / Remote system type is UNIX. Using binary mode to transfer files. ftp> dir 200 PORT command successful 150 Connecting to port 54770 -rw-r--r-- 1 0 0 641 Jul 22 2009 CONTENTS lrwxrwxrwx 1 0 0 11 Jul 22 2009 broken_link -> nonexistent -rw-r--r-- 1 1004 1004 13705975 Aug 17 2006 debian-keyring.tar.gz drwxr-xr-x 3 1004 1004 4096 Jun 3 2007 dir with spaces drwxr-xr-x 2 0 0 4096 Jul 22 2009 dir_with_broken_link drwxr-xr-x 2 0 0 4096 Jan 28 2006 rootdir1 drwxr-xr-x 3 0 0 4096 Jan 28 2006 rootdir2 lrwxrwxrwx 1 0 0 21 Jul 22 2009 valid_link -> debian-keyring.tar.gz drwxr-xr-x 5 1004 1004 4096 Apr 5 2009 walk_test 226-Options: -l 226 9 matches total
Again, the directory doesn't show up. If I try to create it, I get an error message:
ftp> mkdir .hidden 553 Prohibited directory name
Also, it seems I can't create any other directory whose name has a leading dot.
Of course, this complicates things. I think whether dot files/directories are treated as hidden or "normal" depends on an FTP server setting. I think I saw such a setting for my server.
If I replace the current ftputil code with a
try/except
statement, it seemingly will work for your case, but it won't necessarily work for other users, depending on their server and configuration.Even if the upload works for you with
try/except
, there might still be a "symmetry problem": If you try to mirror the directory tree that contains the.hg
directory from the remote to the local system, the.hg
directory might be missing from the target directory (if it wasn't there already, but then you won't get updates from the remote side).
It just occured to me that just catching an
OSError
from themkdir
call and ignoring it won't do it since there might be other reasons for the failure. For example, if themkdir
call fails because you don't have write permission in the parent directory of the one to be created, you do want to see the exception instead of having it silently ignored.Unfortunately, handling exceptions differently based on something else than the exception class often isn't reliable.
By the way, I'm interested in two things in your setup:
Does the hidden directory show up in a directory listing?
What happens if you try to remove the directory?
Anyway, I think the best way to deal with the issues is to either configure the server to make it able to transparently work with "hidden" files or not to do anything with "hidden" files or directories because the behavior or the consistency of operations may be undefined (see my previous comment).
Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError ?
Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting:
ftp> dir 200 PORT command successful 150 Connecting to port 56982 drwxr-xr-x 2 502 20 68 May 30 21:22 normal 226-Options: -l 226 1 matches total
Change the command to "dir -a" will show all including hidden ones:
ftp> dir -a 200 PORT command successful 150 Connecting to port 56983 drwxrwxrwx 5 502 20 170 May 30 21:23 . drwxrwxrwx 5 502 20 170 May 30 21:23 .. -rw-r--r-- 1 502 20 6148 May 31 10:36 .DS_Store drwxr-xr-x 2 502 20 68 May 30 21:22 .hidden drwxr-xr-x 2 502 20 68 May 30 21:22 normal 226-Options: -a -l 226 5 matches total
This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.
change init.py line 809 as following also fix this bug.
def _FTPHost_dir_command(self, path): """Callback function.""" lines = [] def callback(line): """Callback function.""" lines.append(line) #ftp_error._try_with_oserror(self._session.dir, path, callback) ftp_error._try_with_oserror(lambda path, callback: self._session.dir(path, '-a', callback), path, callback) # fix bug, listdir not show .hidden file/dir return lines
It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.
Replying to ftputiluser:
Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError?
To my knowledge, there are no specific
PermanentError
exceptions. (ftputil uses a specialized classCommandNotImplementedError
but this is generated by ftputil itself).Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting: [...] This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.
I looked at the page, but all it suggests is that the option is supported by most clients. The real problem, as far as ftputil is concerned, is server-side support for the
-a
option. (Actually, client support of-a
isn't important at all becauseftputil
is the client in our case. Ok, I rely onftplib
, but you can useftplib
to send arbitrary commands to the server.) I just checked several FTP-related RFCs on the server support ofLIST -a
, but didn't find anything helpful. Moreover, even if we find that many servers supportLIST -a
, this support may be configurable, so you can't rely on the option being functional the moment you connect to a server.That said, I'm now thinking about some "auto-probing" functionality for the
ftplib.dir
command, similar to what I do for the directory listing format determination. So I might issue anLIST -a
command on login and continue to use the-a
option if I don't get a an error. (There's some chance of a "false negative" if you don't have read access for the login directory, but that should be rare. Even if this happens, you'll only lose the ability to see the dot files/directories.)It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.
The other direction will still be a problem though (see comment:9). Although I can display a hidden directory with
LIST -a
from the ftp client now, I don't seem to be able to create a "hidden" directory with my current server settings (553 Prohibited directory name
).
Replying to schwa:
To my knowledge, there are no specific
PermanentError
exceptions. (ftputil uses a specialized classCommandNotImplementedError
but this is generated by ftputil itself).In some cases one can get away by checking the server return code, but, looking at http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes , this won't apply here. There's no specific code for something like "directory already exists".
Replying to schwa:
Replying to ftputiluser:
Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError?
To my knowledge, there are no specific
PermanentError
exceptions. (ftputil uses a specialized classCommandNotImplementedError
but this is generated by ftputil itself).Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting: [...] This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.
I looked at the page, but all it suggests is that the option is supported by most clients. The real problem, as far as ftputil is concerned, is server-side support for the
-a
option. (Actually, client support of-a
isn't important at all becauseftputil
is the client in our case. Ok, I rely onftplib
, but you can useftplib
to send arbitrary commands to the server.) I just checked several FTP-related RFCs on the server support ofLIST -a
, but didn't find anything helpful. Moreover, even if we find that many servers supportLIST -a
, this support may be configurable, so you can't rely on the option being functional the moment you connect to a server.That said, I'm now thinking about some "auto-probing" functionality for the
ftplib.dir
command, similar to what I do for the directory listing format determination. So I might issue anLIST -a
command on login and continue to use the-a
option if I don't get a an error. (There's some chance of a "false negative" if you don't have read access for the login directory, but that should be rare. Even if this happens, you'll only lose the ability to see the dot files/directories.)It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.
The other direction will still be a problem though (see comment:9). Although I can display a hidden directory with
LIST -a
from the ftp client now, I don't seem to be able to create a "hidden" directory with my current server settings (553 Prohibited directory name
).Auto-probing is really a nice tradeoff between standard-compliance and usefulness. I like this solution.
As far as creating .hidden directory concerns, at least it works perfectly fine for PureFTPd with the default setting on Mac.
Thansk for your investigation on this issue.