Opened 13 months ago

Last modified 13 months ago

#139 new defect

FTP/TLS 403 Forbidden Error for Most Operations

Reported by: ftputiluser Owned by: schwa
Priority: major Milestone:
Component: Library Version: 4.0.0
Keywords: Cc: greg.fernlund@…

Description (last modified by schwa)

Python 3.6.8;3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux;FTP Server Version - Unknown

The login,CHDIR, GETCWD, isDir commands appear to work, but when I execute a Listdir, Download or IsFile?, I get 403 Forbidden.

Here is the output; my FTPLSSession class is attached.

*cmd* 'pbsz 0'
*resp* '200 PBSZ command successful (PBSZ=0).'
*cmd* 'PWD'
*resp* '257 "/" is current directory.'
about to change remote directory
*cmd* 'CWD /BFTP/2020/For Greg'
*resp* '250 CWD command successful.'
changed remote directory1 - /BFTP/2020/For Greg
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
changed remote directory2 - /BFTP/2020/For Greg/Automation
is directory - /BFTP/2020/For Greg/Automation
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
*cmd* 'TYPE A'
*resp* '200 Type set to A.'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (172,20,25,52,218,63)'
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
Traceback (most recent call last):
  File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 809, in connect
    negotiate(self, dest_addr, dest_port)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 718, in _negotiate_HTTP
    raise HTTPError(error)
socks.HTTPError: 403: Forbidden

[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
    self._session.dir(path, callback)
  File "/usr/lib64/python3.6/ftplib.py", line 575, in dir
    self.retrlines(cmd, func)
  File "/usr/lib64/python3.6/ftplib.py", line 468, in retrlines
    with self.transfercmd(cmd) as conn, \
  File "/usr/lib64/python3.6/ftplib.py", line 399, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib64/python3.6/ftplib.py", line 798, in ntransfercmd
    conn, size = FTP.ntransfercmd(self, cmd, rest)
  File "/usr/lib64/python3.6/ftplib.py", line 361, in ntransfercmd
    source_address=self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 47, in wrapper
    return function(*args, **kwargs)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 814, in connect
    raise GeneralProxyError("Socket error", error)
socks.GeneralProxyError: Socket error: 403: Forbidden

[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "GetFTP.py", line 155, in <module>
    if ftp_host.path.isfile(source):
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/path.py", line 155, in isfile
    return self._is_file_system_entity(path, "file")
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/path.py", line 125, in _is_file_system_entity
    stat_result = self._host.stat(path, _exception_for_missing_path=False)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 937, in stat
    return self._stat._stat(path, _exception_for_missing_path)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 846, in _stat
    self._real_stat, path, _exception_for_missing_path
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
    result = method(*args, **kwargs)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 768, in _real_stat
    lstat_result = self._real_lstat(path, _exception_for_missing_path)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 729, in _real_lstat
    for stat_result in self._stat_results_from_dir(dirname):
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 620, in _stat_results_from_dir
    lines = self._host_dir(path)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 613, in _host_dir
    return self._host._dir(path)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 889, in _dir
    _FTPHost_dir_command, path, descend_deeply=True
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 610, in _robust_ftp_command
    return command(self, "")
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
    self._session.dir(path, callback)
  File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/error.py", line 172, in __exit__
    raise FTPOSError(*exc_value.args, original_exception=exc_value)
ftputil.error.FTPOSError: Socket error: 403: Forbidden

[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)
Debugging info: ftputil 4.0.0, Python 3.6.8 (linux)

Attachments (1)

ClassFTPLSSession.txt (389 bytes) - added by ftputiluser 13 months ago.
This is the FTPLSSession class I use

Download all attachments as: .zip

Change History (6)

Changed 13 months ago by ftputiluser

Attachment: ClassFTPLSSession.txt added

This is the FTPLSSession class I use

comment:1 Changed 13 months ago by schwa

Description: modified (diff)

comment:2 Changed 13 months ago by schwa

Hi Greg, thanks for using ftputil and for your report.

I've never seen this kind of exception, but I never used a PBSZ command either. ;-) I also haven't seen it in any examples for setting up FTP TLS connections. Why are you using it? Have you tried using ftputil without the PBSZ command? If yes, what happens then?

I found something about PBSZ in RFC 2228:

   PROTECTION BUFFER SIZE (PBSZ)

      The argument is a decimal integer representing the maximum size,
      in bytes, of the encoded data blocks to be sent or received during
      file transfer.  This number shall be no greater than can be
      represented in a 32-bit unsigned integer.

      This command allows the FTP client and server to negotiate a
      maximum protected buffer size for the connection.  There is no
      default size; the client must issue a PBSZ command before it can
      issue the first PROT command.

      The PBSZ command must be preceded by a successful security data
      exchange.

      If the server cannot parse the argument, or if it will not fit in
      32 bits, it should respond with a 501 reply code.

      If the server has not completed a security data exchange with the
      client, it should respond with a 503 reply code.

      Otherwise, the server must reply with a 200 reply code.  If the
      size provided by the client is too large for the server, it must
      use a string of the form "PBSZ=number" in the text part of the
      reply to indicate a smaller buffer size.  The client and the
      server must use the smaller of the two buffer sizes if both buffer
      sizes are specified.

Although I'm not familiar with any details, the value 0 you set looks suspicious, even if the server initially replies with 200 PBSZ command successful (PBSZ=0).

Please let me know if this helps.

comment:3 Changed 13 months ago by ftputiluser

The sendcmd('pbsz 0') does not appear to have any impact on the exchange with the host - when it is removed, there is no difference; when the value is changed (512), the host replies with:

*cmd* 'pbsz 512'
*put* 'pbsz 512\r\n'
*get* '200 PBSZ command successful (PBSZ=0).\n'
*resp* '200 PBSZ command successful (PBSZ=0).'

Is there any way i can get more information about the error? Have a feeling it is on the host server side, but can't tell. I successfully used ftputil to upload a file to this same server, so I am perplexed why i would have a security issue downloading a file.

Last edited 13 months ago by schwa (previous) (diff)

comment:4 Changed 13 months ago by ftputiluser

I should have definitely prefaced this with the fact that I am going through a local proxy server of indeterminate capabilities, with port 3128. I am using PySocks? to set_default the proxy address which definitely works for the authentication and directory operations. The [*] Note: The HTTP proxy server may not be supported by PySocks? (must be a CONNECT tunnel proxy) may be an indication that my proxy server does not allow these other activities, but I don't understand why I could do anything at all if that were the case. Additionally, the proxy server allows me to perform pip operations (when the proxy info is provided), so it obviously allows for files to be downloaded over what I assume is an http connection. I have verified that both put and get operations work using the ftputil functions from my laptop, even when I simulate a proxy address in Windows.

comment:5 Changed 13 months ago by schwa

When you post program output, you can put it between triple curly brackets to switch on code block formatting, like this:

{{{
Program output
}}}

I had done this for your ticket description and your first comment after that. :-)

I should have definitely prefaced this with the fact that I am going through a local proxy server of indeterminate capabilities

Yes, you should have. ;-)

but I don't understand why I could do anything at all if that were the case. Additionally, the proxy server allows me to perform pip operations (when the proxy info is provided), so it obviously allows for files to be downloaded over what I assume is an http connection.

FTP is tricky because it uses a so-called control connection for the commands and a data connection for the data that is transferred. Also you have active and passive mode. When you create an FTPHost instance, this opens one TCP/IP connection. To allow use of the FTPHost instance while transferring files, ftputil creates a new FTP control connection and a data connection for the file transfer.

It's a bit simpler if you use a client that blocks while doing file transfers (e. g. a command line FTP client or a single ftplib.FTP_TLS session). Have you tried using an FTP command line client for testing to see whether this works? I'm not suggesting to stop there, but it may help trying a simplified test case instead of debugging a full ftputil session.

My suggestion is that you try an FTP command line client to navigate the remote file system and do a download (and maybe upload if you have the permission). If you get errors, try it with the other mode (active or passive) before starting file transfers. I don't know if you can switch the mode in the same FTP session but between file transfers. Hence I suggest you restart the client before trying the other mode. Depending on your FTP client, you may be able to switch on debugging output to see the messages sent/received between server and client.

Let me know how that goes. :-)

Note: See TracTickets for help on using tickets.