#78 closed defect (fixed)
Error when using ftputil with M2Crypto
Reported by: | schwa | Owned by: | schwa |
---|---|---|---|
Priority: | major | Milestone: | 3.1 |
Component: | Library | Version: | 3.0 |
Keywords: | FTP_TLS, M2Crypto, unicode string, byte string | Cc: |
Description
Reported by Roger Demetrescu:
"""
I'm trying to connect to a FTP server that only allow encrypted connection. Here is the screenshot of a WinSCP configuration that works perfectly:
http://postimg.org/image/8yqzs2gwp/
Problem is I am unable to do this kind of connection with ftputil using the recipe from:
http://ftputil.sschwarzer.net/trac/wiki/Documentation#does-ftputil-support-ssl
I am using:
- ftputil 3.0
- python 2.7.5+
- Ubuntu 13.10
The production environment will be running python 2.6.
When I use de above recipe, I can connect to the ftp server, but when ftputil tries to send commands, it receives this kind of errors:
>>> host.chdir('/imprensa') PermanentError: 500 Unknown command. >>> host.listdir('.') InaccessibleLoginDirError: directory '/' is not accessible
Just to make it clear, when I connect to the server using WinSCP, I am allowed to list the "/" directory.
"""
Attachments (2)
Change History (13)
comment:1 Changed 6 years ago by
comment:2 Changed 6 years ago by
Interestingly, when using M2Crypto outside of ftputil, everything seems to work fine:
import ftputil from M2Crypto import ftpslib from config import host, username, password f = ftpslib.FTP_TLS() f.set_debuglevel(2) f.connect(host, 21) f.auth_tls() f.login(username, password) f.prot_p() f.retrlines('LIST')
results in the log
*get* '220-IBM Portal\r\n' *get* '220 \r\n' *resp* '220-IBM Portal\n220 ' *cmd* 'AUTH TLS' *put* 'AUTH TLS\r\n' *get* '234 Proceed with negotiation.\r\n' *resp* '234 Proceed with negotiation.' *cmd* 'USER myusername' *put* 'USER myusername\r\n' *get* '331 Please specify the password.\r\n' *resp* '331 Please specify the password.' *cmd* 'PASS ********' *put* 'PASS ********\r\n' *get* '230 Login successful.\r\n' *resp* '230 Login successful.' *cmd* 'PBSZ 0' *put* 'PBSZ 0\r\n' *get* '200 PBSZ set to 0.\r\n' *resp* '200 PBSZ set to 0.' *cmd* 'PROT P' *put* 'PROT P\r\n' *get* '200 PROT now Private.\r\n' *resp* '200 PROT now Private.' *cmd* 'TYPE A' *put* 'TYPE A\r\n' *get* '200 Switching to ASCII mode.\r\n' *resp* '200 Switching to ASCII mode.' *cmd* 'PASV' *put* 'PASV\r\n' *get* '227 Entering Passive Mode (xxx,xxx,xxx,xxx,250,175).\r\n' *resp* '227 Entering Passive Mode (xxx,xxx,xxx,xxx,250,175).' *cmd* 'LIST' *put* 'LIST\r\n' *get* '150 Here comes the directory listing.\r\n' *resp* '150 Here comes the directory listing.' *retr* 'drwxrwxrwx 3 0 0 4096 Nov 27 18:36 backup\r\n' drwxrwxrwx 3 0 0 4096 Nov 27 18:36 backup *retr* 'drwxrwxrwx 2 0 0 4096 Nov 27 18:37 css\r\n' <LOT OF DIRS AND FILES> *get* '226 Directory send OK.\r\n' *resp* '226 Directory send OK.'
The used version of M2Crypto is 0.22.3.
comment:3 Changed 6 years ago by
Roger will need 2.6 for the final hosting of his project.
At the moment he's using a workaround that takes the ftplib.FTP
class from Python 2.6 and lets ftplib.FTP_TLS
of Python 2.7 use this as the base class instead of the native FTP
class from Python 2.7. See the referenced mail for details.
comment:4 Changed 6 years ago by
I configured my local FTP server for TLS and was able to find out why the session based on M2Crypto didn't work while the session based on ftplib.FTP_TLS
did. The used M2Crypto version is 0.21.1-11 on Fedora 19.
After instantiating an ftputil.FTPHost
object based on the M2Crypto session, I successfully logged in. Then I ran host.chdir(".")
in the debugger.
Since ftputil 3.0 is supposed to work in a rather "Python 3 style", it converts the string argument "." to a unicode string, then it calls self._session.cwd(u".")
. self._session
is an instance of SSLFTPSession
, which was defined by Roger (see above).
M2Crypto.ftpslib.FTP_TLS
inherits from ftplib.FTP
the following calls are all in ftplib
.
-> cwd(dirname) dirname is u"." -> voidcmd(cmd) cmd is u"CWD ." -> putcmd(cmd) cmd is u"CWD ." -> putline(line) line is u"CWD ." -> sock.sendall(line) line is u"CWD .\r\n"
The arrows on the left mean that this is a call that doesn't return (yet). I'd denote a return with <-
.
Up to this point the calls are the same for both session factories. For the session factory based on ftplib.FTP_TLS
, sock
is a normal Python socket
object. On the other hand, for the session factory based on M2Crypto.ftpslib.FTP_TLS
, sock
is an instance of M2Crypto.SSL.Connection.Connection
.
The sendall
call then triggers these calls inside the M2Crypto code:
-> write(data) data is u"CWD .\r\n" -> _write_bio(data) data is u"CWD .\r\n" -> m2.ssl_write(self.ssl, data, self._timeout) data is u"CWD .\r\n"
The latter function is in compiled code and can't be stepped into from the Python debugger.
Interestingly, the return value of the ssl_write
call is 28, which is four times the length of the string "CWD .\r\n"
. Hence it could be that ssl_write
sends an UCS4 version of the unicode string.
Whatever the bytes are, they seem to "confuse" the FTP server and it sends the error message "500 ?\r\n"
instead of the desired 250 success message.
Changed 6 years ago by
Attachment: | ticket78_m2crypto_debug_log.txt added |
---|
Debug log for M2Crypto-based session.
comment:5 Changed 6 years ago by
Above, I debugged only the FTPHost.chdir
call. But I can imagine that few or no calls accessible via M2Crypto.ftpslib
process unicode strings correctly. Probably all calls that send data are affected.
These are the approaches that I currently see to deal with the problem:
Approach 1. File an M2Crypto issue that unicode strings can't be sent and ask for adding this feature. From my point of view, this is the desirable long-term solution. On the other hand, it may not be a practical solution, since many deployment environments will use an older version of M2Crypto. Still it should make sense to file the issue, even if it's rather for documentation on M2Crypto.
Approach 2. Mention in the ftputil documentation that session factories need to be able to work with unicode strings, as both ftplib.FTP
and ftplib.FTP_TLS
in Python 2 and 3 do. Add a note that M2Crypto isn't supported for this reason. This approach isn't very desirable because many Python 2.6-only installations (e. g. on RHEL) will want to use M2Crypto for FTP/TLS.
Approach 3. Change ftputil so that FTP session instances always get byte strings. This isn't attractive because some potential session factories usable with Python 3 may expect unicode strings in some places, as M2Crypto on Python 2 expects byte strings.
Approach 4. Provide a helper class or class decorator that can wrap another session factory (say M2Crypto.ftpslib.FTP_TLS
) and make sure that certain methods of a session instance are only called with byte strings. This is my favorite solution for now despite that it requires an ftputil user to deliberately use this adapter code.
comment:6 Changed 6 years ago by
By the way, there's a related issue on the M2Crypto GitHub site. I added a comment there.
comment:7 Changed 6 years ago by
Status: | new → assigned |
---|
comment:8 Changed 6 years ago by
Keywords: | unicode string byte string added |
---|
Changed 6 years ago by
Attachment: | m2crypto_session.py added |
---|
M2Crypto session that modifies the socket's sendall
method.
comment:9 Changed 6 years ago by
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
The attached m2crypto_session.py
contains a session factory class as a workaround for the problem discussed above. I also put the file in the sandbox
directory of the repository.
The class M2CryptoSession
inherits from M2Crypto.ftpslib.FTP_TLS
. After the instantiation of the class the sendall
method of self.sock
is replaced with a variant that encodes its argument to a byte string before sending it. Note that the code implicitly uses ISO-8859-1 (Latin1) encoding. This is fine if you originally passed byte strings to an ftputil API because then ftputil will have used ISO-8859-1 for decoding the string, so the decoding and the encoding step are complementary.
At the moment, I don't include this file in the ftputil distribution, but I'll consider it when a second person runs into the problem. :-)
comment:10 Changed 6 years ago by
I added a module `session.py` to be used as a "universal" "session factory factory". Please check the docstring of the function session_factory
.
You should be able to create a session class according to this ticket like this:
import M2Crypto.ftpslib import ftputil # Will later change to `ftputil.session` import sandbox.session SSLFTPSession = sandbox.session.session_factory( base_class=M2Crypto.ftpslib.FTP_TLS, encrypt_data_channel=True, debug_level=2) with ftputil.FTPHost(host, user, password, session_factory=SSLFTPSession) as host: ...
This code isn't integrated into the ftputil distribution yet and tests are still missing.
comment:11 Changed 6 years ago by
There are unit tests for the session factory factory now. I plan to include this in ftputil 3.1, but I still need to write the documentation.
This is the attempt of a summary of the following mails.
The exceptions above happen with this session class for M2Crypto:
On the other hand, this equivalent code with the
ftplib.FTP_TLS
class from Python 2.7 works:When using the above two sessions, they at first result in the same debug trace:
(verified by Roger with
diff
).When, after that,
f.listdir('.')
is used, the M2Crypto version gives this output:On the other hand, the
ftplib.FTP_TLS
session for the samelistdir
call results in this output: