~sschwarzer/ftputil#9: 
Access Restriction Problems

When 'ftputil' library (or, rather, underlying "ftplib") issues PORT command on the server (Example: 'ls' or 'dir' ftp commands) - each such statement results in opening NEW socket. All of these sockets stay open until process explicitly closes ftp connection. The problem occurs when the long-running process constantly polls server for a file (executes listdir() to see if required file has arrived). When such process runs for days, or when many such processes run simultaneously on the server - this causes server eventually run out of available sockets and crash.

Status
RESOLVED INVALID
Submitter
Valeriy Pogrebitskiy (vpogrebi@iname.com) (unverified)
Assigned to
No-one
Submitted
18 years ago
Updated
18 years ago
Labels
bug library

Valeriy Pogrebitskiy (vpogrebi@iname.com) (unverified) 18 years ago · edit

Majority of FTP servers (at least, within financial industry) are configured in such a way that user LOGIN directory and HOME directory are not the same. On many servers, LOGIN directory is a root (/) directory where user privileges are restricted. For example, 'ls' and 'dir' commands produce exceptions. Often, the ONLY command available to user in / directory is 'cd' - to let user step into a folder within his/her own HOME directory.

When this is the case, attempts to use 'ftputil' package to execute DIR, LS, GET, PUT, etc. statement while being "phisically" in the directory with restricted access - results in "ftputil.ftp_error.PermanentError?: 550 Permission denied" exceptions.

Temporary solution for this - is to execute explicit CHDIR command to "step" into the directory with sufficient user privileges.

FTPUTIL can be (potentially) modified to do that automatically... When process calls any 'ftphost' procedures, providing PATH to function cal - 'ftputil' may execute explicit CHDIR to step into that directory first, before executing requested function call.

schwa (unverified) 18 years ago · edit

Unless the problem lies in an old version of ftplib, this is an operating system issue. You (or the sysadmin) has to tune the OS's parameters.

It's true that ftplib.dir opens a new socket for each call, but the socket is closed as soon as the listing has been retrieved. What you probably observe is that the connections, after having been closed, stay in TIME_WAIT state. (On Unix systems, you can see this with the netstat command. I think, it exists on Windows as well.) This lasts for a minute or so while the OS waits for an opportunity to re-use the already-closed connections.

I did a test on a Linux system with a 2.6.x kernel:

host = ftputil.FTPHost(...)
for i in range(10000):
    print i, host.listdir(host.curdir)

When this code runs, the number of connections in TIME_WAIT increases quickly, as one can see with the Unix command

netstat -a | grep TIME_WAIT | wc -l

However, when the number of "half-open" connections rises to about 1200, the OS starts to shut down the ones that were first opened. With the shown command pipe I can see that the connections in TIME_WAIT state vary between 1200 and 1400, but never rise above this upper limit, so the above loop (10000 listdir calls) can finish. After a while the remaining connections are shut down completely, and those in TIME_WAIT state eventually decrease to 0.

Register here or Log in to comment, or comment via email.