`UnicodeDecodeError` when server sends non-ASCII error messages

[schwa]

​reported by Roger Demetrescu:

"""

I've found an issue with ftputil 3.0 when trying to use host.makedirs() and part of the path already exists AND the FTP server gives error messages with accented characters.

I created 2 virtualenvs:

• python 2.7 (also happens with python 2.6)
• ftputil 2.8 and ftputil 3.0

FTP Server messages are in brazilian portuguese language.

When I use:

host.makedirs('/aaa/bbb/ccc')

and /aaa doesn't exist, all directories are created successfully.

BUT, when "/aaa" already exists, that's what happens:

ftputil 2.8

>>> host.makedirs('/aaa/bbb/ccc')
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'MKD aaa'
*put* 'MKD aaa\r\n'
*get* '550 aaa: N\xe3o \xe9 poss\xedvel criar um arquivo j\xe1
existente.\r\n'
*resp* '550 aaa: N\xe3o \xe9 poss\xedvel criar um arquivo j\xe1 existente.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'CWD /aaa'
*put* 'CWD /aaa\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'MKD bbb'
*put* 'MKD bbb\r\n'
*get* '257 "bbb" directory created.\r\n'
*resp* '257 "bbb" directory created.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'CWD /aaa/bbb'
*put* 'CWD /aaa/bbb\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* 'MKD ccc'
*put* 'MKD ccc\r\n'
*get* '257 "ccc" directory created.\r\n'
*resp* '257 "ccc" directory created.'
*cmd* 'CWD /'
*put* 'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'

ftputil 3.0

>>> host.makedirs('/aaa/bbb/ccc')
*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
*cmd* u'MKD aaa'
*put* u'MKD aaa\r\n'
*get* '550 aaa: N\xe3o \xe9 poss\xedvel criar um arquivo j\xe1
existente.\r\n'
*resp* '550 aaa: N\xe3o \xe9 poss\xedvel criar um arquivo j\xe1 existente.'
*cmd* u'CWD /'
*put* u'CWD /\r\n'
*get* '250 CWD command successful.\r\n'
*resp* '250 CWD command successful.'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/roger/.virtualenvs/version30/local/lib/python2.7/site-packages/ftputil/host.py", line 628, in makedirs
    self.mkdir(next_directory)
  File "/home/roger/.virtualenvs/version30/local/lib/python2.7/site-packages/ftputil/host.py", line 608, in mkdir
    self._robust_ftp_command(command, path)
  File "/home/roger/.virtualenvs/version30/local/lib/python2.7/site-packages/ftputil/host.py", line 574, in _robust_ftp_command
    return command(self, tail)
  File "/home/roger/.virtualenvs/version30/local/lib/python2.7/site-packages/ftputil/host.py", line 607, in command
    self._session.mkd(path)
  File "/home/roger/.virtualenvs/version30/local/lib/python2.7/site-packages/ftputil/error.py", line 128, in __exit__
    if exc_value.args and exc_value.args[0].startswith("502"):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 10: ordinal not in range(128)

The error message in pt_br is "Não é possível criar um arquivo já existente."

"""

[schwa]

It seems the actual error message is a byte string and ftputil checks if the string starts with the string "502".

In ftputil 2.8, "502" is a byte string, so everything works.

On the other hand, in ftputil 3.0, "502" is a unicode string, since I have from __future__ import unicode_literals at the top of the module. For the startswith check, Python implicitly tries to convert the server message, a byte string, to a unicode string and fails because of the non-ASCII characters.

The responsible code is

class FtplibErrorToFTPOSError(object):
    """
    Context manager to convert `ftplib` exceptions to exceptions
    derived from `FTPOSError`.
    """

    ...

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is None:
            # No exception
            return
        if isinstance(exc_value, ftplib.error_temp):
            raise TemporaryError(*exc_value.args)
        elif isinstance(exc_value, ftplib.error_perm):
            # If `exc_value.args[0]` is present, assume it's a byte or
            # unicode string.
            if exc_value.args and exc_value.args[0].startswith("502"):
                raise CommandNotImplementedError(*exc_value.args)
            else:
                raise PermanentError(*exc_value.args)
        elif isinstance(exc_value, ftplib.all_errors):
            raise FTPOSError(*exc_value.args)
        else:
            raise

in error.py. Note the startswith call about in the middle of the __exit__ method.

[schwa]

Here's an untested workaround until a fix is available. Use this code in one of your modules to monkey-patch error.ftplib_error_to_ftp_os_error.__exit__.

  import ftputil.error

  _original_exit = ftputil.error.ftplib_error_to_ftp_os_error.__exit__

  # Bound method, don't use `self` as first argument.
  def _new_exit(exc_type, exc_value, traceback):
      # `exc_value.args` is a tuple and thus can't be
      # modified in-place below.
      args = []
      for arg in exc_value.args:
          if isinstance(arg, str):
              # Since latin1 is an 8-bit encoding, the
              # `decode` call should never cause an exception.
              arg = arg.decode("latin1")
          args.append(arg)
      exc_value.args = tuple(args)
      return _original_exit(exc_type, exc_value, traceback)

  ftputil.error.ftplib_error_to_ftp_os_error.__exit__ = _new_exit

If you use the workaround and it fails, please tell me the error you get.

[schwa]

Fixed in [0079a3a8be44].