Error when using ftputil with M2Crypto

[schwa]

​Reported by Roger Demetrescu:

"""

I'm trying to connect to a FTP server that only allow encrypted connection. Here is the screenshot of a WinSCP configuration that works perfectly:

​http://postimg.org/image/8yqzs2gwp/

Problem is I am unable to do this kind of connection with ftputil using the recipe from:

http://ftputil.sschwarzer.net/trac/wiki/Documentation#does-ftputil-support-ssl

I am using:

• ftputil 3.0
• python 2.7.5+
• Ubuntu 13.10

The production environment will be running python 2.6.

When I use de above recipe, I can connect to the ftp server, but when ftputil tries to send commands, it receives this kind of errors:

>>> host.chdir('/imprensa')
PermanentError: 500 Unknown command.

>>> host.listdir('.')
InaccessibleLoginDirError: directory '/' is not accessible

Just to make it clear, when I connect to the server using WinSCP, I am allowed to list the "/" directory.

"""

[schwa]

This is the attempt of a summary of the following mails.

The exceptions above happen with this session class for M2Crypto:

import ftputil
from M2Crypto import ftpslib

from config import host, username, password

class SSLFTPSession(ftpslib.FTP_TLS):

    def __init__(self, host, userid, password):
        ftpslib.FTP_TLS.__init__(self)
        self.set_debuglevel(2)
        self.connect(host, 21)
        self.auth_tls()
        self.login(userid, password)
        self.prot_p()

On the other hand, this equivalent code with the ftplib.FTP_TLS class from Python 2.7 works:

import ftplib
import ftputil

from config import host, username, password

class SSLFTPSession(ftplib.FTP_TLS):

    def __init__(self, host, userid, password):
        ftplib.FTP_TLS.__init__(self)
        self.set_debuglevel(2)
        self.connect(host, 21)
        self.login(userid, password)
        self.prot_p()

When using the above two sessions, they at first result in the same debug trace:

*get* '220-IBM Portal\r\n'
*get* '220 \r\n'
*resp* '220-IBM Portal\n220 '
*cmd* 'AUTH TLS'
*put* 'AUTH TLS\r\n'
*get* '234 Proceed with negotiation.\r\n'
*resp* '234 Proceed with negotiation.'
*cmd* 'USER myusername'
*put* 'USER myusername\r\n'
*get* '331 Please specify the password.\r\n'
*resp* '331 Please specify the password.'
*cmd* 'PASS ********'
*put* 'PASS ********\r\n'
*get* '230 Login successful.\r\n'
*resp* '230 Login successful.'
*cmd* 'PBSZ 0'
*put* 'PBSZ 0\r\n'
*get* '200 PBSZ set to 0.\r\n'
*resp* '200 PBSZ set to 0.'
*cmd* 'PROT P'
*put* 'PROT P\r\n'
*get* '200 PROT now Private.\r\n'
*resp* '200 PROT now Private.'
*cmd* 'PWD'
*put* 'PWD\r\n'
*get* '257 "/"\r\n'
*resp* '257 "/"'

(verified by Roger with diff).

When, after that, f.listdir('.') is used, the M2Crypto version gives this output:

*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '500 Unknown command.\r\n'
*resp* '500 Unknown command.'
---------------------------------------------------------------------------
InaccessibleLoginDirError                 Traceback (most recent call last)
<ipython-input-1-1646fe8e3e06> in <module>()
----> 1 f.listdir('.')

On the other hand, the ftplib.FTP_TLS session for the same listdir call results in this output:

*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 Directory successfully changed.\r\n'
*resp* '250 Directory successfully changed.'
*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 Directory successfully changed.\r\n'
*resp* '250 Directory successfully changed.'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Switching to ASCII mode.\r\n'
*resp* '200 Switching to ASCII mode.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (***,***,***,***,250,31).\r\n'
*resp* '227 Entering Passive Mode (***,***,***,***,250,31).'
*cmd* u'LIST -a'
*put* u'LIST -a\r\n'
*get* '150 Here comes the directory listing.\r\n'
*resp* '150 Here comes the directory listing.'
*retr* 'drwxrwxrwx    9 0        0            4096 Apr 02 19:44 .\r\n'
*retr* 'drwxrwxrwx    9 0        0            4096 Apr 02 19:44 ..\r\n'
*retr* 'drwxrwxrwx    3 0        0            4096 Nov 27 18:36 backup\r\n'
*retr* 'drwxrwxrwx    2 0        0            4096 Nov 27 18:37 css\r\n'
<<< LOTS OF DIRECTORIES AND FILES >>>
*retr* ''
*get* '226 Directory send OK.\r\n'
*resp* '226 Directory send OK.'
*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 Directory successfully changed.\r\n'
*resp* '250 Directory successfully changed.'

[schwa]

Interestingly, when using M2Crypto outside of ftputil, everything seems to work fine:

import ftputil

from M2Crypto import ftpslib

from config import host, username, password

f = ftpslib.FTP_TLS()
f.set_debuglevel(2)
f.connect(host, 21)
f.auth_tls()
f.login(username, password)
f.prot_p()
f.retrlines('LIST')

results in the log

*get* '220-IBM Portal\r\n'
*get* '220 \r\n'
*resp* '220-IBM Portal\n220 '
*cmd* 'AUTH TLS'
*put* 'AUTH TLS\r\n'
*get* '234 Proceed with negotiation.\r\n'
*resp* '234 Proceed with negotiation.'
*cmd* 'USER myusername'
*put* 'USER myusername\r\n'
*get* '331 Please specify the password.\r\n'
*resp* '331 Please specify the password.'
*cmd* 'PASS ********'
*put* 'PASS ********\r\n'
*get* '230 Login successful.\r\n'
*resp* '230 Login successful.'
*cmd* 'PBSZ 0'
*put* 'PBSZ 0\r\n'
*get* '200 PBSZ set to 0.\r\n'
*resp* '200 PBSZ set to 0.'
*cmd* 'PROT P'
*put* 'PROT P\r\n'
*get* '200 PROT now Private.\r\n'
*resp* '200 PROT now Private.'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Switching to ASCII mode.\r\n'
*resp* '200 Switching to ASCII mode.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (xxx,xxx,xxx,xxx,250,175).\r\n'
*resp* '227 Entering Passive Mode (xxx,xxx,xxx,xxx,250,175).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Here comes the directory listing.\r\n'
*resp* '150 Here comes the directory listing.'
*retr* 'drwxrwxrwx    3 0        0            4096 Nov 27 18:36 backup\r\n'
drwxrwxrwx    3 0        0            4096 Nov 27 18:36 backup
*retr* 'drwxrwxrwx    2 0        0            4096 Nov 27 18:37 css\r\n'
<LOT OF DIRS AND FILES>
*get* '226 Directory send OK.\r\n'
*resp* '226 Directory send OK.'

The used version of M2Crypto is 0.22.3.

[schwa]

Roger will need 2.6 for the final hosting of his project.

At the moment he's using ​a workaround that takes the ftplib.FTP class from Python 2.6 and lets ftplib.FTP_TLS of Python 2.7 use this as the base class instead of the native FTP class from Python 2.7. See the referenced mail for details.

[schwa]

I configured my local FTP server for TLS and was able to find out why the session based on M2Crypto didn't work while the session based on ftplib.FTP_TLS did. The used M2Crypto version is 0.21.1-11 on Fedora 19.

After instantiating an ftputil.FTPHost object based on the M2Crypto session, I successfully logged in. Then I ran host.chdir(".") in the debugger.

Since ftputil 3.0 is supposed to work in a rather "Python 3 style", it converts the string argument "." to a unicode string, then it calls self._session.cwd(u"."). self._session is an instance of SSLFTPSession, which was defined by Roger (see above).

M2Crypto.ftpslib.FTP_TLS inherits from ftplib.FTP the following calls are all in ftplib.

-> cwd(dirname)                 dirname is u"."
-> voidcmd(cmd)                 cmd is u"CWD ."
-> putcmd(cmd)                  cmd is u"CWD ."
-> putline(line)                line is u"CWD ."
-> sock.sendall(line)           line is u"CWD .\r\n"

The arrows on the left mean that this is a call that doesn't return (yet). I'd denote a return with <-.

Up to this point the calls are the same for both session factories. For the session factory based on ftplib.FTP_TLS, sock is a normal Python socket object. On the other hand, for the session factory based on M2Crypto.ftpslib.FTP_TLS, sock is an instance of M2Crypto.SSL.Connection.Connection.

The sendall call then triggers these calls inside the M2Crypto code:

-> write(data)                  data is u"CWD .\r\n"
-> _write_bio(data)             data is u"CWD .\r\n"
-> m2.ssl_write(self.ssl, data, self._timeout)
                                data is u"CWD .\r\n"

The latter function is in compiled code and can't be stepped into from the Python debugger.

Interestingly, the return value of the ssl_write call is 28, which is four times the length of the string "CWD .\r\n". Hence it could be that ssl_write sends an UCS4 version of the unicode string.

Whatever the bytes are, they seem to "confuse" the FTP server and it sends the error message "500 ?\r\n" instead of the desired 250 success message.

[schwa]

Debug log for M2Crypto-based session.

[schwa]

Above, I debugged only the FTPHost.chdir call. But I can imagine that few or no calls accessible via M2Crypto.ftpslib process unicode strings correctly. Probably all calls that send data are affected.

These are the approaches that I currently see to deal with the problem:

Approach 1. File an ​M2Crypto issue that unicode strings can't be sent and ask for adding this feature. From my point of view, this is the desirable long-term solution. On the other hand, it may not be a practical solution, since many deployment environments will use an older version of M2Crypto. Still it should make sense to file the issue, even if it's rather for documentation on M2Crypto.

Approach 2. Mention in the ftputil documentation that session factories need to be able to work with unicode strings, as both ftplib.FTP and ftplib.FTP_TLS in Python 2 and 3 do. Add a note that M2Crypto isn't supported for this reason. This approach isn't very desirable because many Python 2.6-only installations (e. g. on RHEL) will want to use M2Crypto for FTP/TLS.

Approach 3. Change ftputil so that FTP session instances always get byte strings. This isn't attractive because some potential session factories usable with Python 3 may expect unicode strings in some places, as M2Crypto on Python 2 expects byte strings.

Approach 4. Provide a helper class or class decorator that can wrap another session factory (say M2Crypto.ftpslib.FTP_TLS) and make sure that certain methods of a session instance are only called with byte strings. This is my favorite solution for now despite that it requires an ftputil user to deliberately use this adapter code.

[schwa]

By the way, there's a ​related issue on the ​M2Crypto GitHub site. I added a comment there.

[schwa]

M2Crypto session that modifies the socket's sendall method.

[schwa]

The attached m2crypto_session.py contains a session factory class as a workaround for the problem discussed above. I also put the file in the sandbox directory of the repository.

The class M2CryptoSession inherits from M2Crypto.ftpslib.FTP_TLS. After the instantiation of the class the sendall method of self.sock is replaced with a variant that encodes its argument to a byte string before sending it. Note that the code implicitly uses ISO-8859-1 (Latin1) encoding. This is fine if you originally passed byte strings to an ftputil API because then ftputil will have used ISO-8859-1 for decoding the string, so the decoding and the encoding step are complementary.

At the moment, I don't include this file in the ftputil distribution, but I'll consider it when a second person runs into the problem. :-)

[schwa]

I added a module `session.py` to be used as a "universal" "session factory factory". Please check the docstring of the function session_factory.

You should be able to create a session class according to this ticket like this:

import M2Crypto.ftpslib

import ftputil
# Will later change to `ftputil.session`
import sandbox.session


SSLFTPSession = sandbox.session.session_factory(
                  base_class=M2Crypto.ftpslib.FTP_TLS,
                  encrypt_data_channel=True,
                  debug_level=2)
with ftputil.FTPHost(host, user, password,
                     session_factory=SSLFTPSession) as host:
    ...

This code isn't integrated into the ftputil distribution yet and tests are still missing.

[schwa]

There are unit tests for the session factory factory now. I plan to include this in ftputil 3.1, but I still need to write the documentation.

[schwa]

I removed support for M2Crypto in [a10aefe0f71f] while preparing ftputil 4.0.0 because I assumed that M2Crypto was Python-2-only (which I think was the case for a long time) and that it was unnecessary since Python's ftplib got the FTP_TLS class.

As I just found out, M2Crypto (now ​moved to Gitlab) has been ported and according to its ​PyPI page it has a lot of features. Also the ​earlier issue that required a workaround in ftputil seems to have been fixed.

For now, I assume that the M2Crypto issue has been fixed properly, so M2Crypto.ftpslib.FTP_TLS can be used as a drop-in replacement for ftplib.FTP_TLS and thus doesn't need any workaround code in ftputil. Therefore, I leave the explicit M2Crypto support removed and this ticket closed. If you run into problems with M2Crypto, please reopen this ticket.