Ticket #63 (closed defect: invalid)

Opened 6 years ago

Last modified 6 years ago

ftp_sync break on wrong mkdir exception

Reported by: ftputiluser Owned by: schwa
Priority: major Milestone: 2.7
Component: Library Version:
Keywords: Cc:

Description (last modified by schwa) (diff)

When syncing, some dir named ".hg" 's isdir test will throw PermanentError, which it should not (the dir is there on FTP), and it breaks the sync process.

It might be the same problem as ticket #39, "isdir and isfile on broken links with special targets cause a PermanentError".

Remark by ftputil author: It turned out the exception wasn't raised by isdir but by the subsequent _mkdir call. isdir gives a seemingly wrong result here because the directory does exist, - but it's not "seen" by isdir as the directory with a leading dot doesn't show up in the parent directory listing.

As this ticket turned to a discussion about showing hidden files and directories, please add further comments to ticket #23 if necessary.

Change History

comment:1 follow-up: ↓ 2 Changed 6 years ago by schwa

  • Status changed from new to assigned
  • Description modified (diff)
  • Milestone set to 2.7

Thanks for the report.

What's the ftputil version you're using? Could you please add the full traceback for the exception to this ticket? As for ticket #62, a small runnable script surely would help reproduce the problem and make sure we're on the same page. :-)

comment:2 in reply to: ↑ 1 Changed 6 years ago by ftputiluser

Replying to schwa:

Thanks for the report.

What's the ftputil version you're using? Could you please add the full traceback for the exception to this ticket? As for ticket #62, a small runnable script surely would help reproduce the problem and make sure we're on the same page. :-)

Thanks for looking into this. Below is the reproduce scenario:

  • Remote FTP Server: Mac OS X Lion
  • Local Client: Window 7
  • ftputil verison: current tip of hg repository
  • Local directory: dir/.hg
  • Remote directory already has dir/.hg

The script is as simple as:

with FTPHost(host, user, password) as remote:
    local = LocalHost()
    syncer = Syncer(local, remote)
    syncer.sync(local_directory, remote_path)

The program will break at ftp_sync.py line 79:

        if not self._target.path.isdir(target_dir):
                self._target.mkdir(target_dir)

isdir should return True because ".hg" already exists on remote FTP server, however, it return False. and the following mkdir call will throw PermanentError?.

My quick fix is:

        if not self._target.path.isdir(target_dir):
            try:
                self._target.mkdir(target_dir)
            except ftp_error.PermanentError, e:
                # print 'target dir exists', target_dir
                pass


comment:3 Changed 6 years ago by schwa

Maybe I should use a try/except clause instead of the if statement to begin with. But first I have to know what exactly happens.

Can you please add a print statement (or use a debugger) to find out the argument to the failing mkdir call for me?

comment:4 Changed 6 years ago by ftputiluser

The argument for mkdir call is:

upload/.hg

The first sync runs OK since "upload/.hg" dir is not yet exists:

Making /upload
Making /upload\.hg
Syncing e:\a\.hg\file.name -> /upload\.hg\file.name
Syncing e:\a\.hg\thgstatus -> /upload\.hg\thgstatus

However, on second run, it will throw PermanentError? since that directory already exists on FTP.

Traceback (most recent call last):
  File "E:\work\ftp_uploader\test.py", line 3, in <module>
    sync('e:\\a', 'localhost', '/upload', 'will', 'zhuo')
  File "E:\work\ftp_uploader\ftp_uploader.py", line 27, in sync
    n = syncer.sync(local_directory, remote_path)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 149, in sync
    return self._sync_tree(source_path, target_path)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 125, in _sync_tree
    self._mkdir(inner_target_dir)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 80, in _mkdir
    self._target.mkdir(target_dir)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 622, in mkdir
    self._robust_ftp_command(command, path)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 587, in _robust_ftp_comm
and
    return command(self, tail)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 621, in command
    return ftp_error._try_with_oserror(self._session.mkd, path)
  File "E:\work\ftp_uploader\ftputil\ftp_error.py", line 136, in _try_with_oserr
or
    raise PermanentError(*exc.args)
ftputil.ftp_error.PermanentError: 550 Directory already exists

comment:5 Changed 6 years ago by schwa

I attached an ftp_sync.py to ticket #62 which should fix this bug as well. Please try that file as described there in ticket #62, comment 8.

comment:6 follow-up: ↓ 7 Changed 6 years ago by ftputiluser

Thanks for your effort, however, the fix does not work. In case ".hg" is already an existing dir on FTP server, try to sync ".hg" shows the following error:

Making /upload
Fixing inner_target_dir from /upload\.hg to /upload/.hg
Making /upload/.hg
Traceback (most recent call last):
  File "E:\work\ftp_uploader\test.py", line 3, in <module>
    sync('e:\\a', '10.64.81.67', '/upload', 'will', 'zhuo')
  File "E:\work\ftp_uploader\ftp_uploader.py", line 27, in sync
    n = syncer.sync(local_directory, remote_path)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 153, in sync
    self._sync_tree(source_path, target_path)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 130, in _sync_tree
    self._mkdir(inner_target_dir)
  File "E:\work\ftp_uploader\ftputil\ftp_sync.py", line 80, in _mkdir
    self._target.mkdir(target_dir)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 622, in mkdir
    self._robust_ftp_command(command, path)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 587, in _robust_ftp_comm
and
    return command(self, tail)
  File "E:\work\ftp_uploader\ftputil\__init__.py", line 621, in command
    return ftp_error._try_with_oserror(self._session.mkd, path)
  File "E:\work\ftp_uploader\ftputil\ftp_error.py", line 136, in _try_with_oserr
or
    raise PermanentError(*exc.args)
ftputil.ftp_error.PermanentError: 550 Can't create directory: File exists
Debugging info: ftputil 2.6, Python 2.7 (win32)

Regards, ZHUO Qiang <zhuo.qiang@…>

comment:7 in reply to: ↑ 6 Changed 6 years ago by schwa

Replying to ftputiluser:

Thanks for your effort, however, the fix does not work. In case ".hg" is already an existing dir on FTP server, try to sync ".hg" shows the following error:

I wonder why the code

        if not self._target.path.isdir(target_dir):
            self._target.mkdir(target_dir)

in _mkdir doesn't work. This is supposed to handle the "already existing directory" case.

Would you please add a print statement before the if statement like this:

        print "isdir(%r): %s" % (target_dir, self._target.path.isdir(target_dir))

and put the output in this ticket?

Maybe it's more "pythonic" to use a try/except statement like you suggested. But before I change the current code I'd like to find out why it doesn't work as expected. If I just change the code it might mask other problems we haven't thought of yet. (I try to avoid programming by coincidence. :-) )

comment:8 follow-up: ↓ 9 Changed 6 years ago by ftputiluser

I think the problem is, for those hidden directories, FTPHost.path.isdir returns False even if they exists.

Here's my test scripts:

with FTPHost('localhost', 'will', 'zhuo') as remote:
    print 'Normal Dir ', remote.path.isdir('normal')
    print 'Hidden Dir ', remote.path.isdir('.hidden')

"normal" and ".hidden" are all existing directories. However, the output is:

Normal Dir  True
Hidden Dir  False

comment:9 in reply to: ↑ 8 Changed 6 years ago by schwa

Replying to ftputiluser:

I think the problem is, for those hidden directories, FTPHost.path.isdir returns False even if they exists. "normal" and ".hidden" are all existing directories. However, the output is: [...]

Wow, I didn't expect this at all.

Here are some other tests, using my local FTP server (PureFTPd 1.0.28):

root /home/ftptest# mkdir .hidden
$ ipypython
...
>>> import ftputil
>>> h = ftputil.FTPHost("localhost", 'ftptest', 'd605581757de5eb56d568a4419f4126e')
>>> h.listdir(".")
['CONTENTS',
 'broken_link',
 'debian-keyring.tar.gz',
 'dir with spaces',
 'dir_with_broken_link',
 'rootdir1',
 'rootdir2',
 'valid_link',
 'walk_test']

The directory .hidden isn't in the list.

In a command line FTP client I get:

$ ftp localhost
Connected to localhost.localdomain.
...
230-User ftptest has group access to:  1004
230 OK. Current directory is /   
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
200 PORT command successful
150 Connecting to port 54770
-rw-r--r--    1 0          0                 641 Jul 22  2009 CONTENTS
lrwxrwxrwx    1 0          0                  11 Jul 22  2009 broken_link -> nonexistent
-rw-r--r--    1 1004       1004         13705975 Aug 17  2006 debian-keyring.tar.gz
drwxr-xr-x    3 1004       1004             4096 Jun  3  2007 dir with spaces
drwxr-xr-x    2 0          0                4096 Jul 22  2009 dir_with_broken_link
drwxr-xr-x    2 0          0                4096 Jan 28  2006 rootdir1
drwxr-xr-x    3 0          0                4096 Jan 28  2006 rootdir2
lrwxrwxrwx    1 0          0                  21 Jul 22  2009 valid_link -> debian-keyring.tar.gz
drwxr-xr-x    5 1004       1004             4096 Apr  5  2009 walk_test
226-Options: -l
226 9 matches total

Again, the directory doesn't show up. If I try to create it, I get an error message:

ftp> mkdir .hidden
553 Prohibited directory name

Also, it seems I can't create any other directory whose name has a leading dot.

Of course, this complicates things. I think whether dot files/directories are treated as hidden or "normal" depends on an FTP server setting. I think I saw such a setting for my server.

If I replace the current ftputil code with a try/except statement, it seemingly will work for your case, but it won't necessarily work for other users, depending on their server and configuration.

Even if the upload works for you with try/except, there might still be a "symmetry problem": If you try to mirror the directory tree that contains the .hg directory from the remote to the local system, the .hg directory might be missing from the target directory (if it wasn't there already, but then you won't get updates from the remote side).

comment:10 Changed 6 years ago by schwa

It just occured to me that just catching an OSError from the mkdir call and ignoring it won't do it since there might be other reasons for the failure. For example, if the mkdir call fails because you don't have write permission in the parent directory of the one to be created, you do want to see the exception instead of having it silently ignored.

Unfortunately, handling exceptions differently based on something else than the exception class often isn't reliable.

By the way, I'm interested in two things in your setup:

  • Does the hidden directory show up in a directory listing?
  • What happens if you try to remove the directory?

Anyway, I think the best way to deal with the issues is to either configure the server to make it able to transparently work with "hidden" files or not to do anything with "hidden" files or directories because the behavior or the consistency of operations may be undefined (see my previous comment).

comment:11 follow-up: ↓ 12 Changed 6 years ago by ftputiluser

Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError ?

Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting:

ftp> dir
200 PORT command successful
150 Connecting to port 56982
drwxr-xr-x    2 502      20             68 May 30 21:22 normal
226-Options: -l
226 1 matches total

Change the command to "dir -a" will show all including hidden ones:

ftp> dir -a
200 PORT command successful
150 Connecting to port 56983
drwxrwxrwx    5 502      20            170 May 30 21:23 .
drwxrwxrwx    5 502      20            170 May 30 21:23 ..
-rw-r--r--    1 502      20           6148 May 31 10:36 .DS_Store
drwxr-xr-x    2 502      20             68 May 30 21:22 .hidden
drwxr-xr-x    2 502      20             68 May 30 21:22 normal
226-Options: -a -l
226 5 matches total

This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.

change init.py line 809 as following also fix this bug.

        def _FTPHost_dir_command(self, path):
            """Callback function."""
            lines = []
            def callback(line):
                """Callback function."""
                lines.append(line)
            #ftp_error._try_with_oserror(self._session.dir, path, callback)
            ftp_error._try_with_oserror(lambda path, callback: self._session.dir(path, '-a', callback), path, callback) # fix bug, listdir not show .hidden file/dir
            return lines

It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.

comment:12 in reply to: ↑ 11 ; follow-ups: ↓ 13 ↓ 14 Changed 6 years ago by schwa

Replying to ftputiluser:

Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError?

To my knowledge, there are no specific PermanentError exceptions. (ftputil uses a specialized class CommandNotImplementedError but this is generated by ftputil itself).

Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting: [...] This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.

I looked at the page, but all it suggests is that the option is supported by most clients. The real problem, as far as ftputil is concerned, is server-side support for the -a option. (Actually, client support of -a isn't important at all because ftputil is the client in our case. Ok, I rely on ftplib, but you can use ftplib to send arbitrary commands to the server.) I just checked several FTP-related RFCs on the server support of LIST -a, but didn't find anything helpful. Moreover, even if we find that many servers support LIST -a, this support may be configurable, so you can't rely on the option being functional the moment you connect to a server.

That said, I'm now thinking about some "auto-probing" functionality for the ftplib.dir command, similar to what I do for the directory listing format determination. So I might issue an LIST -a command on login and continue to use the -a option if I don't get a an error. (There's some chance of a "false negative" if you don't have read access for the login directory, but that should be rare. Even if this happens, you'll only lose the ability to see the dot files/directories.)

It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.

The other direction will still be a problem though (see comment:9). Although I can display a hidden directory with LIST -a from the ftp client now, I don't seem to be able to create a "hidden" directory with my current server settings (553 Prohibited directory name).

comment:13 in reply to: ↑ 12 Changed 6 years ago by schwa

Replying to schwa:

To my knowledge, there are no specific PermanentError exceptions. (ftputil uses a specialized class CommandNotImplementedError but this is generated by ftputil itself).

In some cases one can get away by checking the server return code, but, looking at http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes , this won't apply here. There's no specific code for something like "directory already exists".

comment:14 in reply to: ↑ 12 Changed 6 years ago by ftputiluser

Replying to schwa:

Replying to ftputiluser:

Actually you only need to catch an more specific PermanentError?, may it be safer than general OSError?

To my knowledge, there are no specific PermanentError exceptions. (ftputil uses a specialized class CommandNotImplementedError but this is generated by ftputil itself).

Get to the original problem, FTP list dir won't show hidden dot files and direcotries. This behavior is depends not only on FTP server setting but also on client setting: [...] This at least works for PureFTPd on Mac as I just tested. And according to http://www.freehostia.com/blog/webhosting/ftp-and-hidden-filesfolders.html this "-a" option seems FTP standard.

I looked at the page, but all it suggests is that the option is supported by most clients. The real problem, as far as ftputil is concerned, is server-side support for the -a option. (Actually, client support of -a isn't important at all because ftputil is the client in our case. Ok, I rely on ftplib, but you can use ftplib to send arbitrary commands to the server.) I just checked several FTP-related RFCs on the server support of LIST -a, but didn't find anything helpful. Moreover, even if we find that many servers support LIST -a, this support may be configurable, so you can't rely on the option being functional the moment you connect to a server.

That said, I'm now thinking about some "auto-probing" functionality for the ftplib.dir command, similar to what I do for the directory listing format determination. So I might issue an LIST -a command on login and continue to use the -a option if I don't get a an error. (There's some chance of a "false negative" if you don't have read access for the login directory, but that should be rare. Even if this happens, you'll only lose the ability to see the dot files/directories.)

It's a more general solution, won't "eat up" the real error, plus, it also works when sync from FTP to local.

The other direction will still be a problem though (see comment:9). Although I can display a hidden directory with LIST -a from the ftp client now, I don't seem to be able to create a "hidden" directory with my current server settings (553 Prohibited directory name).

Auto-probing is really a nice tradeoff between standard-compliance and usefulness. I like this solution.

As far as creating .hidden directory concerns, at least it works perfectly fine for PureFTPd with the default setting on Mac.

Thansk for your investigation on this issue.

comment:15 Changed 6 years ago by schwa

  • Status changed from assigned to closed
  • Resolution set to invalid
  • Description modified (diff)
Note: See TracTickets for help on using tickets.