Opened 4 months ago

Last modified 4 months ago

#147 new defect

ftphost.path.exists isdir return True until ftphost.chdir

Reported by: ftputiluser Owned by: schwa
Priority: major Milestone:
Component: Library Version:
Keywords: Cc:

Description

python 3.6.2

ftputil 5.0.1

I've tried to find out what is going on and why but haven't been successful.

I've seen if I have a remote directory '/solr/someDir'

ftpHost.getcwd() = '/'
ftpHost.path.exists('/solr/someDir') = False
ftpHost.path.isdir('/solr/someDir') = False
ftpHost.path.exists('/solr') = False
ftpHost.path.isdir('/solr') = False
then I do
ftpHost.chdir('/solr')
ftpHost.getcwd() = '/solr'
ftpHost.path.exists('/solr') = True
ftpHost.path.isdir('/solr') = True
ftpHost.chdir('/solr/someDir')
ftpHost.path.exists('/solr/someDir') = True
ftpHost.path.isdir('/solr/someDir') = True

In addition to this, I've found when I use ftp.path.exists, the script sometimes will get hung up on the exists command and not move on but when I change the script to do try/catch on chdir:

try:
    ftpHost.chdir('someDirPath')
    destFolderExists = True
except Exception:
    destFolderExists = False

Then it doesn't hang. I'd like to use .path.exists but I currently only trust the try/catch on chdir.

Is there an particular issues with .path.exists and .path.isdir?

Change History (4)

comment:1 Changed 4 months ago by ftputiluser

My example had one error when you chdir in /solr, .exists still returns False for /solr but .isdir now returns True, but if you chdir into /solr/somedir, .exists and .isdir is True for /solr/somedir

ftpHost.chdir('/solr')
ftpHost.getcwd() = '/solr'
ftpHost.path.exists('/solr') = False
ftpHost.path.isdir('/solr') = True

I wonder if /solr is a 'special' dir on remote.

I also wonder about the hanging. Could .exists cause a lock condition in multithreaded ftping?

comment:2 Changed 4 months ago by schwa

That exists and isdir only return True if you change inside the directory, reminds me of these tickets:

https://ftputil.sschwarzer.net/trac/ticket/86

https://ftputil.sschwarzer.net/trac/ticket/87

This may also be related:

https://ftputil.sschwarzer.net/trac/ticket/66

Although this ticket is marked as fixed, there still may be a similar bug.

Can you please cross-check with ftplib what you see when you list the contents in / while / is the current directory?

Regarding the hangs: You write,

the script sometimes will get hung up on the exists command.

Does "sometimes" mean exists always hangs for some certain directory/directories or does it hang intermittently even for a specific directory? Also it may be helpful if you can fix the current directory from which you invoke exists. As in your description, there may be different behavior depending on the current directory.

Orthogonally to that, you can try to increase the debug level for the FTP session factory to check more precisely in which low-level FTP operation exists hangs.

Could .exists cause a lock condition in multithreaded ftping?

With how ftputil is designed, the low-level FTP connections ("sessions") should be completely separate once they're created, and they're only created if you open a remote file (say, for a download). By the way, ftputil doesn't use multithreading internally. Do you use an FTPHost connection from different threads? :-) This would result in unpredictable behavior.

Something that might also help with debugging is creating small standalone scripts to hopefully reproduce the behavior you're seeing. That way, you're less likely to confuse different circumstances which might influence the behavior. If something like the hangs happens intermittently even under the seemingly same conditions, you can write a loop that repeats some code until you see a hang.

Last edited 4 months ago by schwa (previous) (diff)

comment:3 Changed 4 months ago by schwa

By the way, you showed the code

try:
    ftpHost.chdir('someDirPath')
    destFolderExists = True
except Exception:
    destFolderExists = False

You shouldn't catch the very general Exception; be more specific to avoid catching too much. For example, this code would catch a NameError or AttributeError from a typo in a name. (Although as long as you only call into FTPHost, there shouldn't be NameErrors or AttributeErrors, but then, ftpHost at some point may be a subclass where you added your own code.) And, of course, although ftputil is quite thoroughly tested, it might actually raise a NameError or AttributeError if this happens for an edge case that wasn't covered by the automated tests.

That said, the code also made me think which actual exception you want to catch here. Or, in other words, which exception type did you see here before wrapping the call in the try/except? On a related note, can you show the full traceback? This might help with debugging.

comment:4 Changed 4 months ago by schwa

Another problem of the try/except workaround is that a failing chdir might have other reasons than the directory not existing. For example, the directory may be there, but you may not be allowed to change into it and/or read its contents.

Note: See TracTickets for help on using tickets.