Python 3.6.8;3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux;FTP Server Version - Unknown
The login,CHDIR, GETCWD, isDir commands appear to work, but when I execute a Listdir, Download or IsFile?, I get 403 Forbidden.
Here is the output; my FTPLSSession class is attached.
*cmd* 'pbsz 0'
*resp* '200 PBSZ command successful (PBSZ=0).'
*cmd* 'PWD'
*resp* '257 "/" is current directory.'
about to change remote directory
*cmd* 'CWD /BFTP/2020/For Greg'
*resp* '250 CWD command successful.'
changed remote directory1 - /BFTP/2020/For Greg
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
changed remote directory2 - /BFTP/2020/For Greg/Automation
is directory - /BFTP/2020/For Greg/Automation
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
*cmd* 'TYPE A'
*resp* '200 Type set to A.'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (172,20,25,52,218,63)'
*cmd* 'CWD /BFTP/2020/For Greg/Automation'
*resp* '250 CWD command successful.'
Traceback (most recent call last):
File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 809, in connect
negotiate(self, dest_addr, dest_port)
File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 718, in _negotiate_HTTP
raise HTTPError(error)
socks.HTTPError: 403: Forbidden
[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
self._session.dir(path, callback)
File "/usr/lib64/python3.6/ftplib.py", line 575, in dir
self.retrlines(cmd, func)
File "/usr/lib64/python3.6/ftplib.py", line 468, in retrlines
with self.transfercmd(cmd) as conn, \
File "/usr/lib64/python3.6/ftplib.py", line 399, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib64/python3.6/ftplib.py", line 798, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "/usr/lib64/python3.6/ftplib.py", line 361, in ntransfercmd
source_address=self.source_address)
File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 47, in wrapper
return function(*args, **kwargs)
File "/home/cloud-user/.local/lib/python3.6/site-packages/socks.py", line 814, in connect
raise GeneralProxyError("Socket error", error)
socks.GeneralProxyError: Socket error: 403: Forbidden
[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "GetFTP.py", line 155, in <module>
if ftp_host.path.isfile(source):
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/path.py", line 155, in isfile
return self._is_file_system_entity(path, "file")
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/path.py", line 125, in _is_file_system_entity
stat_result = self._host.stat(path, _exception_for_missing_path=False)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 937, in stat
return self._stat._stat(path, _exception_for_missing_path)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 846, in _stat
self._real_stat, path, _exception_for_missing_path
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 801, in __call_with_parser_retry
result = method(*args, **kwargs)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 768, in _real_stat
lstat_result = self._real_lstat(path, _exception_for_missing_path)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 729, in _real_lstat
for stat_result in self._stat_results_from_dir(dirname):
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 620, in _stat_results_from_dir
lines = self._host_dir(path)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/stat.py", line 613, in _host_dir
return self._host._dir(path)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 889, in _dir
_FTPHost_dir_command, path, descend_deeply=True
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 610, in _robust_ftp_command
return command(self, "")
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/host.py", line 885, in _FTPHost_dir_command
self._session.dir(path, callback)
File "/home/cloud-user/.local/lib/python3.6/site-packages/ftputil/error.py", line 172, in __exit__
raise FTPOSError(*exc_value.args, original_exception=exc_value)
ftputil.error.FTPOSError: Socket error: 403: Forbidden
[*] Note: The HTTP proxy server may not be supported by PySocks (must be a CONNECT tunnel proxy)
Debugging info: ftputil 4.0.0, Python 3.6.8 (linux)
Hi Greg, thanks for using ftputil and for your report.
I've never seen this kind of exception, but I never used a
PBSZ
command either. ;-) I also haven't seen it in any examples for setting up FTP TLS connections. Why are you using it? Have you tried using ftputil without thePBSZ
command? If yes, what happens then?I found something about
PBSZ
in RFC 2228:PROTECTION BUFFER SIZE (PBSZ) The argument is a decimal integer representing the maximum size, in bytes, of the encoded data blocks to be sent or received during file transfer. This number shall be no greater than can be represented in a 32-bit unsigned integer. This command allows the FTP client and server to negotiate a maximum protected buffer size for the connection. There is no default size; the client must issue a PBSZ command before it can issue the first PROT command. The PBSZ command must be preceded by a successful security data exchange. If the server cannot parse the argument, or if it will not fit in 32 bits, it should respond with a 501 reply code. If the server has not completed a security data exchange with the client, it should respond with a 503 reply code. Otherwise, the server must reply with a 200 reply code. If the size provided by the client is too large for the server, it must use a string of the form "PBSZ=number" in the text part of the reply to indicate a smaller buffer size. The client and the server must use the smaller of the two buffer sizes if both buffer sizes are specified.
Although I'm not familiar with any details, the value 0 you set looks suspicious, even if the server initially replies with
200 PBSZ command successful (PBSZ=0).
Please let me know if this helps.
The sendcmd('pbsz 0') does not appear to have any impact on the exchange with the host - when it is removed, there is no difference; when the value is changed (512), the host replies with:
*cmd* 'pbsz 512' *put* 'pbsz 512\r\n' *get* '200 PBSZ command successful (PBSZ=0).\n' *resp* '200 PBSZ command successful (PBSZ=0).'
Is there any way i can get more information about the error? Have a feeling it is on the host server side, but can't tell. I successfully used ftputil to upload a file to this same server, so I am perplexed why i would have a security issue downloading a file.
I should have definitely prefaced this with the fact that I am going through a local proxy server of indeterminate capabilities, with port 3128. I am using PySocks? to set_default the proxy address which definitely works for the authentication and directory operations. The [*] Note: The HTTP proxy server may not be supported by PySocks? (must be a CONNECT tunnel proxy) may be an indication that my proxy server does not allow these other activities, but I don't understand why I could do anything at all if that were the case. Additionally, the proxy server allows me to perform pip operations (when the proxy info is provided), so it obviously allows for files to be downloaded over what I assume is an http connection. I have verified that both put and get operations work using the ftputil functions from my laptop, even when I simulate a proxy address in Windows.
When you post program output, you can put it between triple curly brackets to switch on code block formatting, like this:
{{{ Program output }}}
I had done this for your ticket description and your first comment after that. :-)
I should have definitely prefaced this with the fact that I am going through a local proxy server of indeterminate capabilities
Yes, you should have. ;-)
but I don't understand why I could do anything at all if that were the case. Additionally, the proxy server allows me to perform pip operations (when the proxy info is provided), so it obviously allows for files to be downloaded over what I assume is an http connection.
FTP is tricky because it uses a so-called control connection for the commands and a data connection for the data that is transferred. Also you have active and passive mode. When you create an
FTPHost
instance, this opens one TCP/IP connection. To allow use of theFTPHost
instance while transferring files, ftputil creates a new FTP control connection and a data connection for the file transfer.It's a bit simpler if you use a client that blocks while doing file transfers (e. g. a command line FTP client or a single
ftplib.FTP_TLS
session). Have you tried using an FTP command line client for testing to see whether this works? I'm not suggesting to stop there, but it may help trying a simplified test case instead of debugging a full ftputil session.My suggestion is that you try an FTP command line client to navigate the remote file system and do a download (and maybe upload if you have the permission). If you get errors, try it with the other mode (active or passive) before starting file transfers. I don't know if you can switch the mode in the same FTP session but between file transfers. Hence I suggest you restart the client before trying the other mode. Depending on your FTP client, you may be able to switch on debugging output to see the messages sent/received between server and client.
Let me know how that goes. :-)
I'm closing this ticket since I need more information from the reporter and I haven't gottten a reply since over a year.