python 3.6.2
ftputil 5.0.1
I've tried to find out what is going on and why but haven't been successful.
I've seen if I have a remote directory '/solr/someDir'
ftpHost.getcwd() = '/'
ftpHost.path.exists('/solr/someDir') = False
ftpHost.path.isdir('/solr/someDir') = False
ftpHost.path.exists('/solr') = False
ftpHost.path.isdir('/solr') = False
then I do
ftpHost.chdir('/solr')
ftpHost.getcwd() = '/solr'
ftpHost.path.exists('/solr') = True
ftpHost.path.isdir('/solr') = True
ftpHost.chdir('/solr/someDir')
ftpHost.path.exists('/solr/someDir') = True
ftpHost.path.isdir('/solr/someDir') = True
In addition to this, I've found when I use ftp.path.exists, the script sometimes will get hung up on the exists command and not move on but when I change the script to do try/catch on chdir:
try:
ftpHost.chdir('someDirPath')
destFolderExists = True
except Exception:
destFolderExists = False
Then it doesn't hang. I'd like to use .path.exists but I currently only trust the try/catch on chdir.
Is there an particular issues with .path.exists and .path.isdir?
My example had one error when you chdir in /solr, .exists still returns False for /solr but .isdir now returns True, but if you chdir into /solr/somedir, .exists and .isdir is True for /solr/somedir
ftpHost.chdir('/solr') ftpHost.getcwd() = '/solr' ftpHost.path.exists('/solr') = False ftpHost.path.isdir('/solr') = True
I wonder if /solr is a 'special' dir on remote.
I also wonder about the hanging. Could .exists cause a lock condition in multithreaded ftping?
That
exists
andisdir
only returnTrue
if you change inside the directory, reminds me of these tickets:https://todo.sr.ht/~sschwarzer/ftputil/86
https://todo.sr.ht/~sschwarzer/ftputil/87
This may also be related:
https://todo.sr.ht/~sschwarzer/ftputil/66
Although this ticket is marked as fixed, there still may be a similar bug.
Can you please cross-check with
ftplib
what you see when you list the contents in/
while/
is the current directory?Regarding the hangs: You write,
the script sometimes will get hung up on the exists command.
Does "sometimes" mean
exists
always hangs for some certain directory/directories or does it hang intermittently even for a specific directory? Also it may be helpful if you can fix the current directory from which you invokeexists
. As in your description, there may be different behavior depending on the current directory.Orthogonally to that, you can try to increase the debug level for the FTP session factory to check more precisely in which low-level FTP operation
exists
hangs.Could .exists cause a lock condition in multithreaded ftping?
With how ftputil is designed, the low-level FTP connections ("sessions") should be completely separate once they're created, and they're only created if you open a remote file (say, for a download). By the way, ftputil doesn't use multithreading internally. Do you use an
FTPHost
connection from different threads? :-) This would result in unpredictable behavior.Something that might also help with debugging is creating small standalone scripts to hopefully reproduce the behavior you're seeing. That way, you're less likely to confuse different circumstances which might influence the behavior. If something like the hangs happens intermittently even under the seemingly same conditions, you can write a loop that repeats some code until you see a hang.
By the way, you showed the code
try: ftpHost.chdir('someDirPath') destFolderExists = True except Exception: destFolderExists = False
You shouldn't catch the very general
Exception
; be more specific to avoid catching too much. For example, this code would catch aNameError
orAttributeError
from a typo in a name. (Although as long as you only call intoFTPHost
, there shouldn't beNameError
s orAttributeError
s, but then,ftpHost
at some point may be a subclass where you added your own code.) And, of course, although ftputil is quite thoroughly tested, it might actually raise aNameError
orAttributeError
if this happens for an edge case that wasn't covered by the automated tests.That said, the code also made me think which actual exception you want to catch here. Or, in other words, which exception type did you see here before wrapping the call in the
try
/except
? On a related note, can you show the full traceback? This might help with debugging.
Another problem of the
try
/except
workaround is that a failingchdir
might have other reasons than the directory not existing. For example, the directory may be there, but you may not be allowed to change into it and/or read its contents.
It's unclear whether this is a bug in ftputil or something else. I'd need more information from the reporter to continue, but haven't gotten any feedback since over a year. Therefore, I'm closing this ticket.