Opened 14 years ago

Closed 14 years ago

#9 closed defect (invalid)

Access Restriction Problems

Reported by: Valeriy Pogrebitskiy (vpogrebi@… Owned by: schwa
Priority: major Milestone: 2.1
Component: Library Version:
Keywords: ftputil, DIR, LS, privileges, access, PermanentError Cc:

Description (last modified by schwa)

When 'ftputil' library (or, rather, underlying "ftplib") issues PORT command on the server (Example: 'ls' or 'dir' ftp commands) - each such statement results in opening NEW socket. All of these sockets stay open until process explicitly closes ftp connection. The problem occurs when the long-running process constantly polls server for a file (executes listdir() to see if required file has arrived). When such process runs for days, or when many such processes run simultaneously on the server - this causes server eventually run out of available sockets and crash.

Change History (3)

comment:1 Changed 14 years ago by Valeriy Pogrebitskiy (vpogrebi@…

Component: OtherLibrary
Keywords: LS PREIVILEGES ACCESS PermanentError added; PORT SOCKET removed
Priority: minormajor
Summary: Socket (PORT) connection problemsAccess Restriction Problems
Type: defectenhancement

Majority of FTP servers (at least, within financial industry) are configured in such a way that user LOGIN directory and HOME directory are not the same. On many servers, LOGIN directory is a root (/) directory where user privileges are restricted. For example, 'ls' and 'dir' commands produce exceptions. Often, the ONLY command available to user in / directory is 'cd' - to let user step into a folder within his/her own HOME directory.

When this is the case, attempts to use 'ftputil' package to execute DIR, LS, GET, PUT, etc. statement while being "phisically" in the directory with restricted access - results in "ftputil.ftp_error.PermanentError?: 550 Permission denied" exceptions.

Temporary solution for this - is to execute explicit CHDIR command to "step" into the directory with sufficient user privileges.

FTPUTIL can be (potentially) modified to do that automatically... When process calls any 'ftphost' procedures, providing PATH to function cal - 'ftputil' may execute explicit CHDIR to step into that directory first, before executing requested function call.

comment:2 Changed 14 years ago by schwa

Keywords: ftputil privileges access added; FTPUTIL PREIVILEGES ACCESS removed
Milestone: 2.1
Status: newassigned

comment:3 Changed 14 years ago by schwa

Component: LibraryOther
Description: modified (diff)
Resolution: invalid
Status: assignedclosed

Unless the problem lies in an old version of ftplib, this is an operating system issue. You (or the sysadmin) has to tune the OS's parameters.

It's true that ftplib.dir opens a new socket for each call, but the socket is closed as soon as the listing has been retrieved. What you probably observe is that the connections, after having been closed, stay in TIME_WAIT state. (On Unix systems, you can see this with the netstat command. I think, it exists on Windows as well.) This lasts for a minute or so while the OS waits for an opportunity to re-use the already-closed connections.

I did a test on a Linux system with a 2.6.x kernel:

host = ftputil.FTPHost(...)
for i in range(10000):
    print i, host.listdir(host.curdir)

When this code runs, the number of connections in TIME_WAIT increases quickly, as one can see with the Unix command

netstat -a | grep TIME_WAIT | wc -l

However, when the number of "half-open" connections rises to about 1200, the OS starts to shut down the ones that were first opened. With the shown command pipe I can see that the connections in TIME_WAIT state vary between 1200 and 1400, but never rise above this upper limit, so the above loop (10000 listdir calls) can finish. After a while the remaining connections are shut down completely, and those in TIME_WAIT state eventually decrease to 0.

Note: See TracTickets for help on using tickets.