Changes between Version 3 and Version 4 of PreReleaseDocumentation


Ignore:
Timestamp:
May 31, 2020, 11:08:37 PM (14 months ago)
Author:
schwa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PreReleaseDocumentation

    v3 v4  
    11{{{
    22#!rst
    3 
     3``ftputil`` -- a high-level FTP client library
     4==============================================
     5
     6:Version:   4.0.0-beta
     7:Date:      2020-05-31
     8:Summary:   high-level FTP client library for Python
     9:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
     10:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
     11
     12.. contents::
     13
     14
     15Introduction
     16------------
     17
     18The ``ftputil`` module is a high-level interface to the ftplib_
     19module. The `FTPHost objects`_ generated from it allow many operations
     20similar to those of os_, `os.path`_ and `shutil`_.
     21
     22.. _ftplib: https://docs.python.org/library/ftplib.html
     23.. _os: https://docs.python.org/library/os.html
     24.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
     25.. _`os.path`: https://docs.python.org/library/os.path.html
     26.. _`shutil`: https://docs.python.org/library/shutil.html
     27
     28Example::
     29
     30    import ftputil
     31
     32    # Download some files from the login directory.
     33    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
     34        names = ftp_host.listdir(ftp_host.curdir)
     35        for name in names:
     36            if ftp_host.path.isfile(name):
     37                ftp_host.download(name, name)  # remote, local
     38        # Make a new directory and copy a remote file into it.
     39        ftp_host.mkdir("newdir")
     40        with ftp_host.open("index.html", "rb") as source:
     41            with ftp_host.open("newdir/index.html", "wb") as target:
     42                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
     43
     44Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
     45modification time of a file. The latter can also follow links, similar
     46to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
     47
     48
     49``ftputil`` features
     50--------------------
     51
     52* Method names are familiar from Python's ``os``, ``os.path`` and
     53  ``shutil`` modules. For example, use ``os.path.join`` to join
     54  paths for a local file system and ``ftp_host.path.join`` to join
     55  paths for a remote FTP file system.
     56
     57* Remote file system navigation (``getcwd``, ``chdir``)
     58
     59* Upload and download files (``upload``, ``upload_if_newer``,
     60  ``download``, ``download_if_newer``)
     61
     62* Time zone synchronization between client and server (needed
     63  for ``upload_if_newer`` and ``download_if_newer``)
     64
     65* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
     66  ``rmtree``) and remove files (``remove``)
     67
     68* Get information about directories, files and links (``listdir``,
     69  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
     70  ``abspath``, ``dirname``, ``basename`` etc.)
     71
     72* Iterate over remote file systems (``walk``)
     73
     74* Local caching of results from ``lstat`` and ``stat`` calls to reduce
     75  network access (also applies to ``exists``, ``getmtime`` etc.).
     76
     77* Read files from and write files to remote hosts via
     78  file-like objects (``FTPHost.open``; the generated file-like objects
     79  have the familiar methods like ``read``, ``readline``, ``readlines``,
     80  ``write``, ``writelines`` and ``close``. You can also iterate over
     81  these files line by line in a ``for`` loop.
     82
     83
     84Exception hierarchy
     85-------------------
     86
     87The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
     88``ftputil.error.TemporaryError``.
     89
     90The exception classes are organized as follows::
     91
     92    FTPError
     93        FTPOSError(FTPError, OSError)
     94            PermanentError(FTPOSError)
     95                CommandNotImplementedError(PermanentError)
     96            TemporaryError(FTPOSError)
     97        FTPIOError(FTPError)
     98        InternalError(FTPError)
     99            InaccessibleLoginDirError(InternalError)
     100            ParserError(InternalError)
     101            RootDirError(InternalError)
     102            TimeShiftError(InternalError)
     103
     104and are described here:
     105
     106- ``FTPError``
     107
     108  is the root of the exception hierarchy of the module.
     109
     110- ``FTPOSError``
     111
     112  is derived from ``OSError``. This is for similarity between the
     113  os module and ``FTPHost`` objects. Compare
     114
     115  ::
     116
     117    try:
     118        os.chdir("nonexisting_directory")
     119    except OSError:
     120        ...
     121
     122  with
     123
     124  ::
     125
     126    host = ftputil.FTPHost("host", "user", "password")
     127    try:
     128        host.chdir("nonexisting_directory")
     129    except OSError:
     130        ...
     131
     132  Imagine a function
     133
     134  ::
     135
     136    def func(path, file):
     137        ...
     138
     139  which works on the local file system and catches ``OSErrors``. If you
     140  change the parameter list to
     141
     142  ::
     143
     144    def func(path, file, os=os):
     145        ...
     146
     147  where ``os`` denotes the ``os`` module, you can call the function also as
     148
     149  ::
     150
     151    host = ftputil.FTPHost("host", "user", "password")
     152    func(path, file, os=host)
     153
     154  to use the same code for both a local and remote file system.
     155  Another similarity between ``OSError`` and ``FTPOSError`` is that
     156  the latter holds the FTP server return code in the ``errno``
     157  attribute of the exception object and the error text in
     158  ``strerror``.
     159
     160- ``PermanentError``
     161
     162  is raised for 5xx return codes from the FTP server. This
     163  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
     164  ``ftplib.error_perm`` are *not* identical).
     165
     166- ``CommandNotImplementedError``
     167
     168  indicates that an underlying command the code tries to use is not
     169  implemented. For an example, see the description of the
     170  `FTPHost.chmod`_ method.
     171
     172- ``TemporaryError``
     173
     174  is raised for FTP return codes from the 4xx category. This
     175  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
     176  ``ftplib.error_temp`` are *not* identical).
     177
     178- ``FTPIOError``
     179
     180  denotes an I/O error on the remote host. This appears
     181  mainly with file-like objects that are retrieved by calling
     182  ``FTPHost.open``. Compare
     183
     184  ::
     185
     186    >>> try:
     187    ...     f = open("not_there")
     188    ... except IOError as obj:
     189    ...     print(obj.errno)
     190    ...     print(obj.strerror)
     191    ...
     192    2
     193    No such file or directory
     194
     195  with
     196
     197  ::
     198
     199    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
     200    >>> try:
     201    ...     f = ftp_host.open("not_there")
     202    ... except IOError as obj:
     203    ...     print(obj.errno)
     204    ...     print(obj.strerror)
     205    ...
     206    550
     207    550 not_there: No such file or directory.
     208
     209  As you can see, both code snippets are similar. However, the error
     210  codes aren't the same.
     211
     212- ``InternalError``
     213
     214  subsumes exception classes for signaling errors due to limitations
     215  of the FTP protocol or the concrete implementation of ``ftputil``.
     216
     217- ``InaccessibleLoginDirError``
     218
     219  This exception is raised if the directory in which "you" are placed
     220  upon login is not accessible, i. e. a ``chdir`` call with the
     221  directory as argument would fail.
     222
     223- ``ParserError``
     224
     225  is used for errors during the parsing of directory
     226  listings from the server. This exception is used by the ``FTPHost``
     227  methods ``stat``, ``lstat``, and ``listdir``.
     228
     229- ``RootDirError``
     230
     231  Because of the implementation of the ``lstat`` method it is not
     232  possible to do a ``stat`` call on the root directory ``/``.
     233  If you know *any* way to do it, please let me know. :-)
     234
     235  This problem does *not* affect stat calls on items *in* the root
     236  directory.
     237
     238- ``TimeShiftError``
     239
     240  is used to denote errors which relate to setting the `time shift`_.
     241
     242
     243Directory and file names
     244------------------------
     245
     246.. note::
     247
     248   Keep in mind that this section only applies to directory and file
     249   *names*, not file *contents*. Encoding and decoding for file
     250   contents is handled by the ``encoding`` argument for
     251   `FTPHost.open`_.
     252
     253First off: If your directory and file names (both as arguments and on
     254the server) contain only ISO 8859-1 (latin-1) characters, you can use
     255such names in the form of ``bytes`` or ``str`` objects. However, you
     256can't mix different string types (``bytes`` and ``str``) in one call
     257(for example in ``FTPHost.path.join``).
     258
     259If you have directory or file names with characters that aren't in
     260latin-1, it's recommended to use ``bytes`` objects. In that case,
     261returned paths will be ``bytes`` objects, too.
     262
     263Read on for details.
     264
     265.. note::
     266
     267   The approach described below may look awkward and in a way it is.
     268   The intention of ``ftputil`` is to behave like the local file
     269   system APIs of Python 3 as far as it makes sense. Moreover, the
     270   taken approach makes sure that directory and file names that were
     271   used with Python 3's native ``ftplib`` module will be compatible
     272   with ``ftputil`` and vice versa. Otherwise you may be able to use a
     273   file name with ``ftputil``, but get an exception when trying to
     274   read the same file with Python 3's ``ftplib`` module.
     275
     276Methods that take paths of directories and/or files can take either
     277``bytes`` or ``str`` objects, or `PathLike`_ objects that can be
     278converted to ``bytes`` or ``str``.
     279
     280.. _PathLike: https://docs.python.org/3/library/os.html#os.PathLike
     281
     282If a method gets a string argument (or a string argument wrapped in a
     283PathLike_ object) and returns one or more strings, these strings will
     284have the same string type (``bytes`` or ``str``) as the argument(s).
     285Mixing different string types in one call (for example in
     286``FTPHost.path.join``) isn't allowed and will cause a ``TypeError``.
     287These rules are the same as for local file system operations in Python 3.
     288
     289``bytes`` objects for directory and file names will be sent to the
     290server as-is. On the other hand, ``str`` objects will be encoded to
     291``bytes`` objects, assuming latin-1 encoding. This implies that such
     292``str`` objects must only contain code points 0-255 for the latin-1
     293character set. Using any other characters will result in a
     294``UnicodeEncodeError`` exception.
     295
     296If you have directory or file names as ``str`` objects with
     297non-latin-1 characters, encode the strings to ``bytes`` yourself,
     298using the encoding you know the server uses for its file system.
     299Decode received paths with the same encoding. Encapsulate these
     300conversions as far as you can. Otherwise, you'd have to adapt
     301potentially a lot of code if the server encoding changes.
     302
     303If you *don't* know the encoding on the server side, it's probably the
     304best to only use ``bytes`` for directory and file names. That said, as
     305soon as you *show* the names to a user, you -- or the library you use
     306for displaying the names -- has to guess an encoding.
     307
     308If you can decide about paths yourself, it's generally safest to use
     309only ASCII characters in FTP paths.
     310
     311
     312``FTPHost`` objects
     313-------------------
     314
     315.. _`FTPHost construction`:
     316
     317Construction
     318~~~~~~~~~~~~
     319
     320Introduction
     321````````````
     322
     323``FTPHost`` instances can be created with the following call::
     324
     325    ftp_host = ftputil.FTPHost(server, user, password, account,
     326                               session_factory=ftplib.FTP)
     327
     328The first four parameters are strings with the same meaning as for the
     329FTP class in the ``ftplib`` module. Usually the ``account`` and
     330``session_factory`` arguments aren't needed though.
     331
     332``FTPHost`` objects can also be used in a ``with`` statement::
     333
     334    import ftputil
     335
     336    with ftputil.FTPHost(server, user, password) as ftp_host:
     337        print(ftp_host.listdir(ftp_host.curdir))
     338
     339After the ``with`` block, the ``FTPHost`` instance and the
     340associated FTP sessions will be closed automatically.
     341
     342If something goes wrong during the ``FTPHost`` construction or in the
     343body of the ``with`` statement, the instance is closed as well.
     344Exceptions will be propagated (as with ``try ... finally``).
     345
     346Session factories
     347`````````````````
     348
     349The keyword argument ``session_factory`` may be used to generate FTP
     350connections with other factories than the default ``ftplib.FTP``. For
     351example, the standard library of Python 3 contains a class
     352``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
     353connection.
     354
     355In fact, all positional and keyword arguments other than
     356``session_factory`` are passed to the factory to generate a new
     357background session. This also happens for every remote file that is
     358opened; see below.
     359
     360This functionality of the constructor also allows to wrap
     361``ftplib.FTP`` objects to do something that wouldn't be possible with
     362the ``ftplib.FTP`` constructor alone.
     363
     364As an example, assume you want to connect to another than the default
     365port, but ``ftplib.FTP`` only offers this by means of its ``connect``
     366method, not via its constructor. One solution is to use a custom
     367class as a session factory::
     368
     369    import ftplib
     370    import ftputil
     371
     372    EXAMPLE_PORT = 50001
     373
     374    class MySession(ftplib.FTP):
     375
     376        def __init__(self, host, userid, password, port):
     377            """Act like ftplib.FTP's constructor but connect to another port."""
     378            ftplib.FTP.__init__(self)
     379            self.connect(host, port)
     380            self.login(userid, password)
     381
     382    # Try _not_ to use an _instance_ `MySession()` as factory, -
     383    # use the class itself.
     384    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
     385                         session_factory=MySession) as ftp_host:
     386        # Use `ftp_host` as usual.
     387        ...
     388
     389On login, the format of the directory listings (needed for stat'ing
     390files and directories) should be determined automatically. If not,
     391please `enter a ticket`_.
     392
     393.. _`enter a ticket`: https://ftputil.sschwarzer.net/issuetrackernotes
     394
     395For the most common uses you don't need to create your own session
     396factory class though. The ``ftputil.session`` module has a function
     397``session_factory`` that can create session factories for a variety
     398of parameters::
     399
     400    session_factory(base_class=ftplib.FTP,
     401                    port=21,
     402                    use_passive_mode=None,
     403                    encrypt_data_channel=True,
     404                    debug_level=None)
     405
     406with
     407
     408- ``base_class`` is a base class to inherit a new session factory
     409  class from. By default, this is ``ftplib.FTP`` from the Python
     410  standard library.
     411
     412- ``port`` is the command channel port. The default is 21, used in most
     413  FTP server configurations.
     414
     415- ``use_passive_mode`` is either a boolean that determines whether
     416  passive mode should be used or ``None``. ``None`` means to let the
     417  base class choose active or passive mode.
     418
     419- ``encrypt_data_channel`` defines whether to encrypt the data channel
     420  for secure connections. This is only supported for the base classes
     421  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
     422  parameter is ignored.
     423
     424- ``debug_level`` sets the debug level for FTP session instances. The
     425  semantics is defined by the base class. For example, a debug level
     426  of 2 causes the most verbose output for Python's ``ftplib.FTP``
     427  class.
     428
     429All of these parameters can be combined. For example, you could use
     430
     431::
     432
     433    import ftplib
     434
     435    import ftputil
     436    import ftputil.session
     437
     438
     439    my_session_factory = ftputil.session.session_factory(
     440                           base_class=ftpslib.FTP_TLS,
     441                           port=31,
     442                           encrypt_data_channel=True,
     443                           debug_level=2)
     444
     445    with ftputil.FTPHost(server, user, password,
     446                         session_factory=my_session_factory) as ftp_host:
     447        ...
     448
     449to create and use a session factory derived from ``ftplib.FTP_TLS``
     450that connects on command channel 31, will encrypt the data channel and
     451print output for debug level 2.
     452
     453Note: Generally, you can achieve everything you can do with
     454``ftputil.session.session_factory`` with an explicit session factory
     455as described at the start of this section. However, the class
     456``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
     457it with ftputil out of the box. The function ``session_factory``
     458contains a workaround for this limitation. For details refer to `this
     459ticket`_.
     460
     461.. _`this ticket`: https://ftputil.sschwarzer.net/trac/ticket/78
     462
     463Hidden files and directories
     464~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     465
     466Whether ftputil sees "hidden" files and directories (usually files or
     467directories whose names start with a dot) depends on the FTP server
     468configuration. By default, ftputil does *not* use the ``-a`` option in
     469the FTP ``LIST`` command to find hidden files.
     470
     471To tell the server to list hidden directories and files, set
     472``FTPHost.use_list_a_option`` to ``True``::
     473
     474    ftp_host = ftputil.FTPHost(server, user, password, account,
     475                               session_factory=ftplib.FTP)
     476    ftp_host.use_list_a_option = True
     477
     478Caveats:
     479
     480- If the server doesn't understand the ``-a`` option at all, the
     481  server may interpret ``-a`` as the name of a file or directory,
     482  which can result in odd behavior. Therefore, use ``-a`` only if
     483  you're sure the server you're talking to supports it. Another
     484  approach is to have test code for ``-a`` support and fall back to
     485  not using the option.
     486
     487- Even if the server knows about the ``-a`` option, the server may
     488  be configured to ignore it.
     489
     490``FTPHost`` attributes and methods
     491~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     492
     493Attributes
     494``````````
     495
     496- ``curdir``, ``pardir``, ``sep``
     497
     498  are strings which denote the current and the parent directory on the
     499  remote server. ``sep`` holds the path separator. Though `RFC 959`_
     500  (File Transfer Protocol) notes that these values may depend on the
     501  FTP server implementation, the Unix variants seem to work well in
     502  practice, even for non-Unix servers.
     503
     504  Nevertheless, it's recommended that you don't hardcode these values
     505  for remote paths, but use `FTPHost.path`_ as you would use
     506  ``os.path`` to write platform-independent Python code for local
     507  filesystems. Keep in mind that most, *but not all*, arguments of
     508  ``FTPHost`` methods refer to remote directories or files. For
     509  example, in `FTPHost.upload`_, the first argument is a local
     510  path and the second a remote path. Both of these should use their
     511  respective path separators.
     512
     513.. _`FTPHost.upload`: `Uploading and downloading files`_
     514
     515Remote file system navigation
     516`````````````````````````````
     517
     518- ``getcwd()``
     519
     520  returns the absolute current directory on the remote host. This
     521  method works like ``os.getcwd``.
     522
     523- ``chdir(directory)``
     524
     525  sets the current directory on the FTP server. This resembles
     526  ``os.chdir``, as you may have expected.
     527
     528.. _`callback function`:
     529
     530Uploading and downloading files
     531```````````````````````````````
     532
     533- ``upload(source, target, callback=None)``
     534
     535  copies a local source file (given by a filename, i. e. a string)
     536  to the remote host under the name target. Both ``source`` and
     537  ``target`` may be absolute paths or relative to their corresponding
     538  current directory (on the local or the remote host, respectively).
     539
     540  The file content is always transferred in binary mode.
     541
     542  The callback, if given, will be invoked for each transferred chunk
     543  of data::
     544
     545    callback(chunk)
     546
     547  where ``chunk`` is a bytestring. An example usage of a callback
     548  method is to display a progress indicator.
     549
     550- ``download(source, target, callback=None)``
     551
     552  performs a download from the remote source file to a local target
     553  file. Both ``source`` and ``target`` are strings. See the
     554  description of ``upload`` for more details.
     555
     556.. _`upload_if_newer`:
     557
     558- ``upload_if_newer(source, target, callback=None)``
     559
     560  is similar to the ``upload`` method. The only difference is that the
     561  upload is only invoked if the time of the last modification for the
     562  source file is more recent than that of the target file or the
     563  target doesn't exist at all. The check for the last modification
     564  time considers the precision of the timestamps and transfers a file
     565  "if in doubt". Consequently the code
     566
     567  ::
     568
     569    ftp_host.upload_if_newer("source_file", "target_file")
     570    time.sleep(10)
     571    ftp_host.upload_if_newer("source_file", "target_file")
     572
     573  might upload the file again if the timestamp of the target file is
     574  precise up to a minute, which is typically the case because the
     575  remote datetime is determined by parsing a directory listing from
     576  the server. To avoid unnecessary transfers, wait at least a minute
     577  between calls of ``upload_if_newer`` for the same file. If it still
     578  seems that a file is uploaded unnecessarily (or not when it should),
     579  read the subsection on `time shift`_ settings.
     580
     581  If an upload actually happened, the return value of
     582  ``upload_if_newer`` is a ``True``, else ``False``.
     583
     584  Note that the method only checks the existence and/or the
     585  modification time of the source and target file; it doesn't
     586  compare any other file properties, say, the file size.
     587
     588  This also means that if a transfer is interrupted, the remote file
     589  will have a newer modification time than the local file, and thus
     590  the transfer won't be repeated if ``upload_if_newer`` is used a
     591  second time. There are at least two possibilities after a failed
     592  upload:
     593
     594  - use ``upload`` instead of ``upload_if_newer``, or
     595
     596  - remove the incomplete target file with ``FTPHost.remove``, then
     597    use ``upload`` or ``upload_if_newer`` to transfer it again.
     598
     599.. _`download_if_newer`:
     600
     601- ``download_if_newer(source, target, callback=None)``
     602
     603  corresponds to ``upload_if_newer`` but performs a download from the
     604  server to the local host. Read the descriptions of download and
     605  ``upload_if_newer`` for more information. If a download actually
     606  happened, the return value is ``True``, else ``False``.
     607
     608.. _`time shift`:
     609.. _`time zone correction`:
     610
     611Time zone correction
     612````````````````````
     613
     614For ``upload_if_newer`` and ``download_if_newer`` to work correctly,
     615the time zone of the server must be taken into account. By default,
     616ftputil assumes that the timestamps in server listings are in UTC_.
     617
     618.. _UTC: https://en.wikipedia.org/wiki/Utc
     619
     620.. _`set_time_shift`:
     621
     622- ``set_time_shift(time_shift)``
     623
     624  sets the so-called time shift value, measured in seconds. The time
     625  shift here is defined as the difference between the time used in
     626  server listings and UTC.
     627
     628  ::
     629
     630    time_shift = server_time - utc_time
     631
     632  For example, a server in Berlin/Germany set to the local time
     633  (currently UTC+03:00), would require a time shift value of 3 *
     634  3600.0 = 10800.0 seconds to be handled correctly by ftputil's
     635  ``upload_if_newer`` and ``download_if_newer``, as well as the
     636  ``stat`` and ``lstat`` calls.
     637
     638  Note that servers don't necessarily send their file system listings
     639  in their local time zone. Some use UTC, which actually makes sense
     640  because UTC doesn't lead to an ambiguity when there's a switch back
     641  from the daylight saving time to the "normal" time of the server
     642  location.
     643
     644  If the time shift value is invalid, for example its absolute value
     645  is larger than 24 hours, a ``TimeShiftError`` is raised.
     646
     647  .. note::
     648
     649     Versions of ftputil before 4.0.0 used a different definition of
     650     "time shift", server_time - local_client_time.
     651
     652     This had the advantage that the default of 0.0 would be correct
     653     *if* the server was set to the same time zone as the client
     654     where ftputil runs. On the other hand, this approach meant that
     655     the time shift depended on *two* time zones, not only the one
     656     used on the server side. This could be confusing if server
     657     and client *didn't* use the same time zone.
     658
     659  See also `synchronize_times`_ for a way to set the time shift with a
     660  simple method call. If you can't use ``synchronize_times`` *and* the
     661  server uses the same time zone as the client, you can set the time
     662  shift value with
     663
     664  ::
     665
     666    set_time_shift(
     667      round( (datetime.datetime.now() - datetime.datetime.utcnow()).seconds, -2 )
     668    )
     669
     670- ``time_shift()``
     671
     672  returns the currently-set time shift value. See ``set_time_shift``
     673  above for its definition.
     674
     675.. _`synchronize_times`:
     676
     677- ``synchronize_times()``
     678
     679  synchronizes the local times of the server and the client, so that
     680  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
     681  if the client and the server use different time zones. For this
     682  to work, *all* of the following conditions must be true:
     683
     684  - The connection between server and client is established.
     685
     686  - The client has write access to the directory that is current when
     687    ``synchronize_times`` is called.
     688
     689  If you can't fulfill these conditions, you can nevertheless set the
     690  time shift value explicitly with `set_time_shift`_. Trying to call
     691  ``synchronize_times`` if the above conditions aren't met results in
     692  a ``TimeShiftError`` exception.
     693
     694Creating and removing directories
     695`````````````````````````````````
     696
     697- ``mkdir(path, [mode])``
     698
     699  makes the given directory on the remote host. This does *not*
     700  construct "intermediate" directories that don't already exist. The
     701  ``mode`` parameter is ignored; this is for compatibility with
     702  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
     703  instead of the ``os`` module. See the explanation in the subsection
     704  `Exception hierarchy`_.
     705
     706- ``makedirs(path, [mode])``
     707
     708  works similar to ``mkdir`` (see above), but also makes intermediate
     709  directories like ``os.makedirs``. The ``mode`` parameter is only
     710  there for compatibility with ``os.makedirs`` and is ignored.
     711
     712- ``rmdir(path)``
     713
     714  removes the given remote directory. If it's not empty, raise
     715  a ``PermanentError``.
     716
     717- ``rmtree(path, ignore_errors=False, onerror=None)``
     718
     719  removes the given remote, possibly non-empty, directory tree.
     720  The interface of this method is rather complex, in favor of
     721  compatibility with ``shutil.rmtree``.
     722
     723  If ``ignore_errors`` is set to a true value, errors are ignored.
     724  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
     725  set, all exceptions occurring during the tree iteration and
     726  processing are raised. These exceptions are all of type
     727  ``PermanentError``.
     728
     729  To distinguish between different kinds of errors, pass in a callable
     730  for ``onerror``. This callable must accept three arguments:
     731  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
     732  object, *for example* ``your_host_object.listdir``. ``path`` is the
     733  path that was the recent argument of the respective method
     734  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
     735  info as it is gotten from ``sys.exc_info``.
     736
     737  The code of ``rmtree`` is taken from Python's ``shutil`` module
     738  and adapted for ``ftputil``.
     739
     740Removing files and links
     741````````````````````````
     742
     743- ``remove(path)``
     744
     745  removes a file or link on the remote host, similar to ``os.remove``.
     746
     747- ``unlink(path)``
     748
     749  is an alias for ``remove``.
     750
     751Retrieving information about directories, files and links
     752`````````````````````````````````````````````````````````
     753
     754- ``listdir(path)``
     755
     756  returns a list containing the names of the files and directories
     757  in the given path, similar to `os.listdir`_. The special names
     758  ``.`` and ``..`` are not in the list.
     759
     760The methods ``lstat`` and ``stat`` (and some others) rely on the
     761directory listing format used by the FTP server. When connecting to a
     762host, ``FTPHost``'s constructor tries to guess the right format, which
     763succeeds in most cases. However, if you get strange results or
     764``ParserError`` exceptions by a mere ``lstat`` call, please `enter a
     765ticket`_.
     766
     767If ``lstat`` or ``stat`` give wrong modification dates or times, look
     768at the methods that deal with time zone differences (`time zone
     769correction`_).
     770
     771.. _`FTPHost.lstat`:
     772
     773- ``lstat(path)``
     774
     775  returns an object similar to that from `os.lstat`_. This is a kind
     776  of tuple with additional attributes; see the documentation of the
     777  ``os`` module for details.
     778
     779  The result is derived by parsing the output of a ``LIST`` command on
     780  the server. Therefore, the result from ``FTPHost.lstat`` can not
     781  contain more information than the received text. In particular:
     782
     783  - User and group ids can only be determined as strings, not as
     784    numbers, and that only if the server supplies them. This is
     785    usually the case with Unix servers but maybe not for other FTP
     786    servers.
     787
     788  - Values for the time of the last modification may be rough,
     789    depending on the information from the server. For timestamps
     790    older than a year, this usually means that the precision of the
     791    modification timestamp value is not better than a day. For newer
     792    files, the information may be accurate to a minute.
     793
     794    If the time of the last modification is before the epoch (usually
     795    1970-01-01 UTC), set the time of the last modification to 0.0.
     796
     797  - Links can only be recognized on servers that provide this
     798    information in the ``LIST`` output.
     799
     800  - Stat attributes that can't be determined at all are set to
     801        ``None``. For example, a line of a directory listing may not
     802        contain the date/time of a directory's last modification.
     803
     804  - There's a special problem with stat'ing the root directory.
     805    (Stat'ing things *in* the root directory is fine though.) In
     806    this case, a ``RootDirError`` is raised. This has to do with the
     807    algorithm used by ``(l)stat``, and I know of no approach which
     808    mends this problem.
     809
     810  Currently, ``ftputil`` recognizes the common Unix-style and
     811  Microsoft/DOS-style directory formats. If you need to parse output
     812  from another server type, please write to the `ftputil mailing
     813  list`_. You may consider `writing your own parser`_.
     814
     815.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
     816.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
     817.. _`ftputil mailing list`: https://ftputil.sschwarzer.net/mailinglist
     818.. _`writing your own parser`: `Writing directory parsers`_
     819
     820.. _`FTPHost.stat`:
     821
     822- ``stat(path)``
     823
     824  returns ``stat`` information also for files which are pointed to by a
     825  link. This method follows multiple links until a regular file or
     826  directory is found. If an infinite link chain is encountered or the
     827  target of the last link in the chain doesn't exist, a
     828  ``PermanentError`` is raised.
     829
     830  The limitations of the ``lstat`` method also apply to ``stat``.
     831
     832.. _`FTPHost.path`:
     833
     834``FTPHost`` objects contain an attribute named ``path``, similar to
     835`os.path`_. The following methods can be applied to the remote host
     836with the same semantics as for ``os.path``:
     837
     838::
     839
     840    abspath(path)
     841    basename(path)
     842    commonprefix(path_list)
     843    dirname(path)
     844    exists(path)
     845    getmtime(path)
     846    getsize(path)
     847    isabs(path)
     848    isdir(path)
     849    isfile(path)
     850    islink(path)
     851    join(path1, path2, ...)
     852    normcase(path)
     853    normpath(path)
     854    split(path)
     855    splitdrive(path)
     856    splitext(path)
     857    walk(path, func, arg)
     858
     859Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
     860methods return ``False`` if they can't find the path given by their
     861argument.
     862
     863Local caching of file system information
     864````````````````````````````````````````
     865
     866Many of the above methods need access to the remote file system to
     867obtain data on directories and files. To get the most recent data,
     868*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
     869would require to fetch a directory listing from the server, which can
     870make the program *very* slow. This effect is more pronounced for
     871operations which mostly scan the file system rather than transferring
     872file data.
     873
     874For this reason, ``ftputil`` by default saves the results from
     875directory listings locally and reuses those results. This reduces
     876network accesses and so speeds up the software a lot. However, since
     877data is more rarely fetched from the server, the risk of obsolete data
     878also increases. This will be discussed below.
     879
     880Caching can be controlled -- if necessary at all -- via the
     881``stat_cache`` object in an ``FTPHost``'s namespace. For example,
     882after calling
     883
     884::
     885
     886    ftp_host = ftputil.FTPHost(host, user, password)
     887
     888the cache can be accessed as ``ftp_host.stat_cache``.
     889
     890While ``ftputil`` usually manages the cache quite well, there are two
     891possible reasons for modifying cache parameters.
     892
     893The first is when the number of possible entries is too low. You may
     894notice that when you are processing very large directories and the
     895program becomes much slower than before. It's common for code to read
     896a directory with ``listdir`` and then process the found directories
     897and files. This can also happen implicitly by a call to
     898``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
     899increases the cache size if directories with more entries than the
     900current maximum cache size are to be scanned. Most of the time, this
     901works fine.
     902
     903However, if you need access to stat data for several directories at
     904the same time, you may need to increase the cache explicitly. This is
     905done by the ``resize`` method::
     906
     907    ftp_host.stat_cache.resize(20000)
     908
     909where the argument is the maximum number of ``lstat`` results to store
     910(the default is 5000, in versions before 2.6 it was 1000). Note that
     911each path on the server, e. g. "/home/schwa/some_dir", corresponds to
     912a single cache entry. Methods like ``exists`` or ``getmtime`` all
     913derive their results from a previously fetched ``lstat`` result.
     914
     915The value 5000 above means that the cache will hold *at most* 5000
     916entries (unless increased automatically by an explicit or implicit
     917``listdir`` call, see above). If more are about to be stored, the
     918entries which haven't been used for the longest time will be deleted
     919to make place for newer entries.
     920
     921The second possible reason to change the cache parameters is to avoid
     922stale cache data. Caching is so effective because it reduces network
     923accesses. This can also be a disadvantage if the file system data on
     924the remote server changes after a stat result has been retrieved; the
     925client, when looking at the cached stat data, will use obsolete
     926information.
     927
     928There are two potential ways to get such out-of-date stat data. The
     929first happens when an ``FTPHost`` instance modifies a file path for
     930which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
     931Such changes are handled transparently; the path will be deleted from
     932the cache. A different matter are changes unknown to the ``FTPHost``
     933object which inspects its cache. Obviously, for example, these are
     934changes by programs running on the remote host. On the other hand,
     935cache inconsistencies can also occur if two ``FTPHost`` objects change
     936a file system simultaneously::
     937
     938    with (
     939      ftputil.FTPHost(server, user1, password1) as ftp_host1,
     940      ftputil.FTPHost(server, user1, password1) as ftp_host2
     941    ):
     942        stat_result1 = ftp_host1.stat("some_file")
     943        stat_result2 = ftp_host2.stat("some_file")
     944        ftp_host2.remove("some_file")
     945        # `ftp_host1` will still see the obsolete cache entry!
     946        print(ftp_host1.stat("some_file"))
     947        # Will raise an exception since an `FTPHost` object
     948        # knows of its own changes.
     949        print(ftp_host2.stat("some_file"))
     950
     951At first sight, it may appear to be a good idea to have a shared cache
     952among several ``FTPHost`` objects. After some thinking, this turns out
     953to be very error-prone. For example, it won't help with different
     954processes using ``ftputil``. So, if you have to deal with concurrent
     955write/read accesses to a server, you have to handle them explicitly.
     956
     957The most useful tool for this is the ``invalidate`` method. In the
     958example above, it could be used like this::
     959
     960    with (
     961      ftputil.FTPHost(server, user1, password1) as ftp_host1,
     962      ftputil.FTPHost(server, user1, password1) as ftp_host2
     963    ):
     964        stat_result1 = ftp_host1.stat("some_file")
     965        stat_result2 = ftp_host2.stat("some_file")
     966        ftp_host2.remove("some_file")
     967        # Invalidate using an absolute path.
     968        absolute_path = ftp_host1.path.abspath(
     969                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
     970        ftp_host1.stat_cache.invalidate(absolute_path)
     971        # Will now raise an exception as it should.
     972        print(ftp_host1.stat("some_file"))
     973        # Would raise an exception since an `FTPHost` object
     974        # knows of its own changes, even without `invalidate`.
     975        print(ftp_host2.stat("some_file"))
     976
     977The method ``invalidate`` can be used on any *absolute* path, be it a
     978directory, a file or a link.
     979
     980By default, the cache entries (if not replaced by newer ones) are
     981stored for an infinite time. That is, if you start your Python process
     982using ``ftputil`` and let it run for three days a stat call may still
     983access cache data that old. To avoid this, you can set the ``max_age``
     984attribute::
     985
     986    with ftputil.FTPHost(server, user, password) as ftp_host:
     987        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
     988
     989This sets the maximum age of entries in the cache to an hour. This
     990means any entry older won't be retrieved from the cache but its data
     991instead fetched again from the remote host and then again stored for
     992up to an hour. To reset `max_age` to the default of unlimited age,
     993i. e. cache entries never expire, use ``None`` as value.
     994
     995If you are certain that the cache will be in the way, you can disable
     996and later re-enable it completely with ``disable`` and ``enable``::
     997
     998    with ftputil.FTPHost(server, user, password) as ftp_host:
     999        ftp_host.stat_cache.disable()
     1000        ...
     1001        ftp_host.stat_cache.enable()
     1002
     1003During that time, the cache won't be used; all data will be fetched
     1004from the network. After enabling the cache again, its entries will be
     1005the same as when the cache was disabled, that is, entries won't get
     1006updated with newer data during this period. Note that even when the
     1007cache is disabled, the file system data in the code can become
     1008inconsistent::
     1009
     1010    with ftputil.FTPHost(server, user, password) as ftp_host:
     1011        ftp_host.stat_cache.disable()
     1012        if ftp_host.path.exists("some_file"):
     1013            mtime = ftp_host.path.getmtime("some_file")
     1014
     1015In that case, the file ``some_file`` may have been removed by another
     1016process between the calls to ``exists`` and ``getmtime``!
     1017
     1018Iteration over directories
     1019``````````````````````````
     1020
     1021.. _`FTPHost.walk`:
     1022
     1023- ``walk(top, topdown=True, onerror=None, followlinks=False)``
     1024
     1025  iterates over a directory tree, similar to `os.walk`_. Actually,
     1026  ``FTPHost.walk`` uses the code from Python with just the necessary
     1027  modifications, so see the linked documentation.
     1028
     1029.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
     1030
     1031.. _`FTPHost.path.walk`:
     1032
     1033- ``path.walk(path, func, arg)``
     1034
     1035  Similar to ``os.path.walk``, the ``walk`` method in
     1036  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
     1037  easier to use.
     1038
     1039Other methods
     1040`````````````
     1041
     1042- ``close()``
     1043
     1044  closes the connection to the remote host. After this, no more
     1045  interaction with the FTP server is possible with this ``FTPHost``
     1046  object. Usually you don't need to close an ``FTPHost`` instance
     1047  with ``close`` if you set up the instance in a ``with`` statement.
     1048
     1049- ``rename(source, target)``
     1050
     1051  renames the source file (or directory) on the FTP server.
     1052
     1053.. _`FTPHost.chmod`:
     1054
     1055- ``chmod(path, mode)``
     1056
     1057  sets the access mode (permission flags) for the given path. The mode
     1058  is an integer as returned for the mode by the ``stat`` and ``lstat``
     1059  methods. Be careful: Usually, mode values are written as octal
     1060  numbers, for example 0755 to make a directory readable and writable
     1061  for the owner, but not writable for the group and others. If you
     1062  want to use such octal values, rely on Python's support for them::
     1063
     1064    ftp_host.chmod("some_directory", 0o755)
     1065
     1066  Not all FTP servers support the ``chmod`` command. In case of
     1067  an exception, how do you know if the path doesn't exist or if
     1068  the command itself is invalid? If the FTP server complies with
     1069  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
     1070  command isn't allowed. ``ftputil`` maps this special error
     1071  response to a ``CommandNotImplementedError`` which is derived from
     1072  ``PermanentError``.
     1073
     1074  So you need to code like this::
     1075
     1076    with ftputil.FTPHost(server, user, password) as ftp_host:
     1077        try:
     1078            ftp_host.chmod("some_file", 0o644)
     1079        except ftputil.error.CommandNotImplementedError:
     1080            # `chmod` not supported
     1081            ...
     1082        except ftputil.error.PermanentError:
     1083            # Possibly a non-existent file
     1084            ...
     1085
     1086  Because the ``CommandNotImplementedError`` is more specific, you
     1087  have to test for it first.
     1088
     1089.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
     1090
     1091- ``copyfileobj(source, target, length=64*1024)``
     1092
     1093  copies the contents from the file-like object ``source`` to the
     1094  file-like object ``target``. The only difference to
     1095  ``shutil.copyfileobj`` is the default buffer size. Note that
     1096  arbitrary file-like objects can be used as arguments (e. g. local
     1097  files, remote FTP files).
     1098
     1099  However, the interfaces of ``source`` and ``target`` have to match;
     1100  the string type read from ``source`` must be an accepted string type
     1101  when written to ``target``. For example, if you open ``source`` in
     1102  Python 3 as a local text file and ``target`` as a remote file object
     1103  in binary mode, the transfer will fail since ``source.read`` gives
     1104  unicode strings (``str``) whereas ``target.write`` only accepts byte
     1105  strings (``bytes``).
     1106
     1107  See `File-like objects`_ for the construction and use of remote
     1108  file-like objects.
     1109
     1110.. _`set_parser`:
     1111
     1112- ``set_parser(parser)``
     1113
     1114  sets a custom parser for FTP directories. Note that you have to pass
     1115  in a parser *instance*, not the class.
     1116
     1117  An `extra section`_ shows how to write own parsers if the default
     1118  parsers in ``ftputil`` don't work for you.
     1119
     1120.. _`extra section`: `Writing directory parsers`_
     1121
     1122.. _`keep_alive`:
     1123
     1124- ``keep_alive()``
     1125
     1126  attempts to keep the connection to the remote server active in order
     1127  to prevent timeouts from happening. This method is primarily
     1128  intended to keep the underlying FTP connection of an ``FTPHost``
     1129  object alive while a file is uploaded or downloaded. This will
     1130  require either an extra thread while the upload or download is in
     1131  progress or calling ``keep_alive`` from a `callback function`_.
     1132
     1133  The ``keep_alive`` method won't help if the connection has already
     1134  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
     1135
     1136  If you want to use this method, keep in mind that FTP servers define
     1137  a timeout for a reason. A timeout prevents running out of server
     1138  connections because of clients that never disconnect on their own.
     1139
     1140  Note that the ``keep_alive`` method does *not* affect the "hidden"
     1141  FTP child connections established by ``FTPHost.open`` (see section
     1142  `FTPHost instances vs. FTP connections`_ for details). You *can't*
     1143  use ``keep_alive`` to avoid a timeout in a stalling transfer like
     1144  this::
     1145
     1146      with ftputil.FTPHost(server, userid, password) as ftp_host:
     1147          with ftp_host.open("some_remote_file", "rb") as fobj:
     1148              data = fobj.read(100)
     1149              # _Futile_ attempt to avoid file connection timeout.
     1150              for i in range(15):
     1151                  time.sleep(60)
     1152                  ftp_host.keep_alive()
     1153              # Will raise an `ftputil.error.TemporaryError`.
     1154              data += fobj.read()
     1155
     1156
     1157.. _`FTPHost.open`:
     1158
     1159File-like objects
     1160-----------------
     1161
     1162Construction
     1163~~~~~~~~~~~~
     1164
     1165Basics
     1166``````
     1167
     1168``FTPFile`` objects are returned by a call to ``FTPHost.open``;
     1169never use the ``FTPFile`` constructor directly.
     1170
     1171The APIs for remote file-like objects is modeled after the APIs of
     1172the built-in ``open`` function and its return value.
     1173
     1174- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
     1175  errors=None, newline=None, rest=None)``
     1176
     1177  returns a file-like object that refers to the path on the remote
     1178  host. This path may be absolute or relative to the current directory
     1179  on the remote host (this directory can be determined with the
     1180  ``getcwd`` method). As with local file objects, the default mode is
     1181  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
     1182  "wb".
     1183
     1184  If a file is opened in binary mode, you *must not* specify an
     1185  encoding. On the other hand, if you open a file in text mode, an
     1186  encoding is used. By default, this is the return value of
     1187  ``locale.getpreferredencoding``, but you can (and probably should)
     1188  specify a distinct encoding.
     1189
     1190  If you open a file in binary mode, the read and write operations use
     1191  ``bytes`` objects. That is, read operations return ``bytes`` and
     1192  write operations only accept ``bytes``.
     1193
     1194  Similarly, text files always work with strings (``str``). Here, read
     1195  operations return string and write operations only accept strings.
     1196
     1197  The arguments ``buffering``, ``errors`` and ``newline`` have the
     1198  same semantics as in open_.
     1199
     1200  If the file is opened in binary mode, you may pass 0 or a positive
     1201  integer for the ``rest`` argument. The argument is passed to the
     1202  underlying FTP session instance (for example an instance of
     1203  ``ftplib.FTP``) to start reading or writing at the given byte
     1204  offset. For example, if a remote file contains the letters
     1205  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".
     1206
     1207  .. warning::
     1208
     1209     If you pass ``rest`` values which point *after* the file, the
     1210     behavior is undefined and may even differ from one FTP server to
     1211     another. Therefore, use the ``rest`` argument only for error
     1212     recovery in case of interrupted transfers. You need to keep track
     1213     of the transferred data so that you can provide a valid ``rest``
     1214     argument for a resumed transfer.
     1215
     1216.. _`open`: https://docs.python.org/3/library/functions.html#open
     1217
     1218``FTPHost.open`` can also be used in a ``with`` statement::
     1219
     1220    import ftputil
     1221
     1222    with ftputil.FTPHost(...) as ftp_host:
     1223        ...
     1224        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
     1225            fobj.write("This is some text.")
     1226
     1227At the end of the ``with`` block, the remote file will be closed
     1228automatically.
     1229
     1230If something goes wrong during the construction of the file or in the
     1231body of the ``with`` statement, the file will be closed as well.
     1232Exceptions will be propagated as with ``try ... finally``.
     1233
     1234Attributes and methods
     1235~~~~~~~~~~~~~~~~~~~~~~
     1236
     1237The methods
     1238
     1239::
     1240
     1241    close()
     1242    read([count])
     1243    readline([count])
     1244    readlines()
     1245    write(data)
     1246    writelines(string_sequence)
     1247
     1248and the attribute ``closed`` have the same semantics as for file
     1249objects of a local disk file system. The iterator protocol is
     1250supported as well, i. e. you can use a loop to read a file line by
     1251line::
     1252
     1253    with ftputil.FTPHost(server, user, password) as ftp_host:
     1254        with ftp_host.open("some_file") as input_file:
     1255            for line in input_file:
     1256                # Do something with the line, e. g.
     1257                print(line.strip().replace("ftplib", "ftputil"))
     1258
     1259For more on file objects, see the section `File objects`_ in the
     1260Python Library Reference.
     1261
     1262.. _`file objects`: https://docs.python.org/3/glossary.html#term-file-object
     1263
     1264
     1265.. _`child_connections`:
     1266
     1267``FTPHost`` instances vs. FTP connections
     1268-----------------------------------------
     1269
     1270This section explains why keeping an ``FTPHost`` instance "alive"
     1271without timing out sometimes isn't trivial. If you always finish your
     1272FTP operations in time, you don't need to read this section.
     1273
     1274The file transfer protocol is a stateful protocol. That means an FTP
     1275connection always is in a certain state. Each of these states can only
     1276change to certain other states under certain conditions triggered by
     1277the client or the server.
     1278
     1279One of the consequences is that a single FTP connection can't be used
     1280at the same time, say, to transfer data on the FTP data channel and to
     1281create a directory on the remote host.
     1282
     1283For example, consider this::
     1284
     1285    >>> import ftplib
     1286    >>> ftp = ftplib.FTP(server, user, password)
     1287    >>> ftp.pwd()
     1288    '/'
     1289    >>> # Start transfer. `CONTENTS` is a text file on the server.
     1290    >>> socket = ftp.transfercmd("RETR CONTENTS")
     1291    >>> socket
     1292    <socket._socketobject object at 0x7f801a6386e0>
     1293    >>> ftp.pwd()
     1294    Traceback (most recent call last):
     1295      File "<stdin>", line 1, in <module>
     1296      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
     1297        return parse257(resp)
     1298      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
     1299        raise error_reply, resp
     1300    ftplib.error_reply: 226-File successfully transferred
     1301    226 0.000 seconds (measured here), 5.60 Mbytes per second
     1302    >>>
     1303
     1304Note that ``ftp`` is a single FTP connection, represented by an
     1305``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
     1306
     1307On the other hand, consider this::
     1308
     1309    >>> import ftputil
     1310    >>> ftp_host = ftputil.FTPHost(server, user, password)
     1311    >>> ftp_host.getcwd()
     1312    >>> fobj = ftp_host.open("CONTENTS")
     1313    >>> fobj
     1314    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
     1315    >>> ftp_host.getcwd()
     1316    u'/'
     1317    >>> fobj.readline()
     1318    u'Contents of FTP test directory\n'
     1319    >>> fobj.close()
     1320    >>>
     1321
     1322To be able to start a file transfer (i. e. open a remote file for
     1323reading or writing) and still be able to use other FTP commands,
     1324ftputil uses a trick. For every remote file, ftputil creates a new FTP
     1325connection, called a child connection in the ftputil source code.
     1326(Actually, FTP connections belonging to closed remote files are
     1327re-used if they haven't timed out yet.)
     1328
     1329In most cases this approach isn't noticeable by code using ftputil.
     1330However, the nice abstraction of dealing with a single FTP connection
     1331falls apart if one of the child connections times out. For example, if
     1332you open a remote file and work only with the initial "main"
     1333connection to navigate the file system, the FTP connection for the
     1334remote file may eventually time out.
     1335
     1336While it's often relatively easy to prevent the "main" connection from
     1337timing out it's unfortunately practically impossible to do this for a
     1338remote file connection (apart from transferring some data, of course).
     1339For this reason, `FTPHost.keep_alive`_ affects only the main
     1340connection. Child connections may still time out if they're idle for
     1341too long.
     1342
     1343.. _`FTPHost.keep_alive`: `keep_alive`_
     1344
     1345Some more details:
     1346
     1347- A kind of "straightforward" way of keeping the main connection alive
     1348  would be to call ``ftp_host.getcwd()``. However, this doesn't work
     1349  because ftputil caches the current directory and returns it without
     1350  actually contacting the server. That's the main reason why there's
     1351  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
     1352  (i. e. the session object), which isn't a public attribute.
     1353
     1354- Some servers define not only an idle timeout but also a transfer
     1355  timeout. This means the connection times out unless there's some
     1356  transfer on the data channel for this connection. So ftputil's
     1357  ``keep_alive`` doesn't prevent this timeout, but an
     1358  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
     1359  this transfers the data for the whole directory listing which might
     1360  take some time if the directory has many entries.
     1361
     1362Bottom line: If you can, you should organize your FTP actions so that
     1363you finish everything before a timeout happens.
     1364
     1365
     1366Writing directory parsers
     1367-------------------------
     1368
     1369``ftputil`` recognizes the two most widely-used FTP directory formats,
     1370Unix and MS style, and adjusts itself automatically. Almost every FTP
     1371server uses one of these formats.
     1372
     1373However, if your server uses a format which is different from the two
     1374provided by ``ftputil``, you can plug in a custom parser with a single
     1375method call and have ``ftputil`` use this parser.
     1376
     1377For this, you need to write a parser class by inheriting from the
     1378class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
     1379
     1380    import ftputil.error
     1381    import ftputil.stat
     1382
     1383    class XyzParser(ftputil.stat.Parser):
     1384        """
     1385        Parse the default format of the FTP server of the XYZ
     1386        corporation.
     1387        """
     1388
     1389        def parse_line(self, line, time_shift=0.0):
     1390            """
     1391            Parse a `line` from the directory listing and return a
     1392            corresponding `StatResult` object. If the line can't
     1393            be parsed, raise `ftputil.error.ParserError`.
     1394
     1395            The `time_shift` argument can be used to fine-tune the
     1396            parsing of dates and times. See the class
     1397            `ftputil.stat.UnixParser` for an example.
     1398            """
     1399            # Split the `line` argument and examine it further; if
     1400            # something goes wrong, raise an `ftputil.error.ParserError`.
     1401            ...
     1402            # Make a `StatResult` object from the parts above.
     1403            stat_result = ftputil.stat.StatResult(...)
     1404            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
     1405            stat_result._st_name = ...
     1406            stat_result._st_target = ...
     1407            stat_result._st_mtime_precision = ...
     1408            return stat_result
     1409
     1410        # Define `ignores_line` only if the default in the base class
     1411        # doesn't do enough!
     1412        def ignores_line(self, line):
     1413            """
     1414            Return a true value if the line should be ignored. For
     1415            example, the implementation in the base class handles
     1416            lines like "total 17". On the other hand, if the line
     1417            should be used for stat'ing, return a false value.
     1418            """
     1419            is_total_line = super().ignores_line(line)
     1420            my_test = ...
     1421            return is_total_line or my_test
     1422
     1423A ``StatResult`` object is similar to the value returned by
     1424`os.stat`_ and is usually built with statements like
     1425
     1426::
     1427
     1428    stat_result = StatResult(
     1429                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
     1430                     st_gid, st_size, st_atime, st_mtime, st_ctime))
     1431    stat_result._st_name = ...
     1432    stat_result._st_target = ...
     1433    stat_result._st_mtime_precision = ...
     1434
     1435with the arguments of the ``StatResult`` constructor described in
     1436the following table.
     1437
     1438===== =================== ============ =================== =======================
     1439Index Attribute           os.stat type ``StatResult`` type Notes
     1440===== =================== ============ =================== =======================
     14410     st_mode             int          int
     14421     st_ino              long         long
     14432     st_dev              long         long
     14443     st_nlink            int          int
     14454     st_uid              int          str                 usually only available as string
     14465     st_gid              int          str                 usually only available as string
     14476     st_size             long         long
     14487     st_atime            int/float    float
     14498     st_mtime            int/float    float
     14509     st_ctime            int/float    float
     1451\-    _st_name            \-           str                 file name without directory part
     1452\-    _st_target          \-           str                 link target (may be absolute or relative)
     1453\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
     1454===== =================== ============ =================== =======================
     1455
     1456If you can't extract all the desirable data from a line (for
     1457example, the MS format doesn't contain any information about the
     1458owner of a file), set the corresponding values in the ``StatResult``
     1459instance to ``None``.
     1460
     1461Parser classes can use several helper methods which are defined in
     1462the class ``Parser``:
     1463
     1464- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
     1465  an appropriate ``st_mode`` integer value.
     1466
     1467- ``parse_unix_time`` returns a float number usable for the
     1468  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
     1469  "May"/"26"/"2005". Note that the method expects the timestamp string
     1470  already split at whitespace.
     1471
     1472- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
     1473  returns a float number like from ``time.mktime``. Note that the
     1474  method expects the timestamp string already split at whitespace.
     1475
     1476Additionally, there's an attribute ``_month_numbers`` which maps
     1477lowercase three-letter month abbreviations to integers.
     1478
     1479For more details, see the two "standard" parsers ``UnixParser`` and
     1480``MSParser`` in the module ``ftputil/stat.py``.
     1481
     1482To actually *use* the parser, call the method `set_parser`_ of the
     1483``FTPHost`` instance.
     1484
     1485If you can't write a parser or don't want to, please ask on the
     1486`ftputil mailing list`_. Possibly someone has already written a parser
     1487for your server or can help with it.
     1488
     1489
     1490FAQ / Tips and tricks
     1491---------------------
     1492
     1493Where can I get the latest version?
     1494~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1495
     1496See the `download page`_. Announcements will be sent to the `mailing
     1497list`_. Announcements on major updates will also be posted to the
     1498`Python announcements list`_.
     1499
     1500.. _`download page`: https://ftputil.sschwarzer.net/download
     1501.. _`mailing list`: https://ftputil.sschwarzer.net/mailinglist
     1502.. _`Python announcements list`: https://mail.python.org/mailman3/lists/python-announce-list.python.org/
     1503
     1504Is there a mailing list on ``ftputil``?
     1505~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1506
     1507Yes, please visit https://ftputil.sschwarzer.net/mailinglist to
     1508subscribe or read the archives.
     1509
     1510Though you can *technically* post without subscribing first I can't
     1511recommend it: The mails from non-subscribers have to be approved by
     1512me and because the arriving mails contain *lots* of spam, I rarely go
     1513through these mails.
     1514
     1515I found a bug! What now?
     1516~~~~~~~~~~~~~~~~~~~~~~~~
     1517
     1518Before reporting a bug, make sure that you already read this manual
     1519and tried the `latest version`_ of ``ftputil``. There the bug might
     1520have already been fixed.
     1521
     1522.. _`latest version`: https://ftputil.sschwarzer.net/download
     1523
     1524Please see https://ftputil.sschwarzer.net/issuetrackernotes for
     1525guidelines on entering a bug in ``ftputil``'s ticket system. If you
     1526are unsure if the behaviour you found is a bug or not, you should write
     1527to the `ftputil mailing list`_. *Never* include confidential information
     1528(user id, password, file names, etc.) in the problem report! Be
     1529careful!
     1530
     1531Does ``ftputil`` support TLS?
     1532~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1533
     1534``ftputil`` has no *built-in* TLS support.
     1535
     1536On the other hand, there are two ways to get TLS support with
     1537ftputil:
     1538
     1539- The ``ftplib`` library has a class ``FTP_TLS`` that you can use for
     1540  the ``session_factory`` keyword argument in the ``FTPHost``
     1541  constructor. You can't use the class directly though *if* you need
     1542  additional setup code in comparison to ``ftplib.FTP``, for example
     1543  calling ``prot_p``, to secure the data connection. On the other
     1544  hand, `ftputil.session.session_factory`_ can be used to create a
     1545  custom session factory.
     1546
     1547- If you have other requirements that ``session_factory`` can't
     1548  fulfill, you may create your own session factory by inheriting from
     1549  ``ftplib.FTP_TLS``::
     1550
     1551    import ftplib
     1552
     1553    import ftputil
     1554
     1555
     1556    class FTPTLSSession(ftplib.FTP_TLS):
     1557
     1558        def __init__(self, host, user, password):
     1559            ftplib.FTP_TLS.__init__(self)
     1560            self.connect(host, port)
     1561            self.login(user, password)
     1562            # Set up encrypted data connection.
     1563            self.prot_p()
     1564            ...
     1565
     1566    # Note the `session_factory` parameter. Pass the class, not
     1567    # an instance.
     1568    with ftputil.FTPHost(server, user, password,
     1569                         session_factory=FTPTLSSession) as ftp_host:
     1570        # Use `ftp_host` as usual.
     1571        ...
     1572
     1573.. _`ftputil.session.session_factory`: `Session factories`_
     1574
     1575
     1576How do I connect to a non-default port?
     1577~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1578
     1579By default, an instantiated ``FTPHost`` object connects on the usual
     1580FTP port. If you have to use a different port, refer to the section
     1581`Session factories`_.
     1582
     1583How do I set active or passive mode?
     1584~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1585
     1586Please see the section `Session factories`_.
     1587
     1588How can I debug an FTP connection problem?
     1589~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1590
     1591You can do this with a session factory. See `Session factories`_.
     1592
     1593If you want to change the debug level only temporarily after the
     1594connection is established, you can reach the `session object`_ as the
     1595``_session`` attribute of the ``FTPHost`` instance and call
     1596``_session.set_debuglevel``. Note that the ``_session`` attribute
     1597should *only* be accessed for debugging. Calling arbitrary
     1598``ftplib.FTP`` methods on the session object may *cause* bugs!
     1599
     1600.. _`session object`: `Session factories`_
     1601
     1602Conditional upload/download to/from a server in a different time zone
     1603~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1604
     1605You may find that ``ftputil`` uploads or downloads files
     1606unnecessarily, or not when it should. Please see the section on `time
     1607zone correction`_. It may even be sufficient to call
     1608`synchronize_times`_.
     1609
     1610When I use ``ftputil``, all I get is a ``ParserError`` exception
     1611~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1612
     1613The FTP server you connect to may use a directory format that
     1614``ftputil`` doesn't understand. You can either write and
     1615`plug in your own parser`_ or ask on the `mailing list`_ for
     1616help.
     1617
     1618.. _`plug in your own parser`: `Writing directory parsers`_
     1619
     1620``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
     1621~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1622
     1623Like Python's counterparts under `os.path`_, ``ftputil``'s methods
     1624return ``False`` if they can't find the given path.
     1625
     1626Probably you used ``listdir`` on a directory and called ``is...()`` on
     1627the returned names. But if the argument for ``listdir`` wasn't the
     1628current directory, the paths won't be found and so all ``is...()``
     1629variants will return ``False``.
     1630
     1631I don't find an answer to my problem in this document
     1632~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1633
     1634Please send an email with your problem report or question to the
     1635`ftputil mailing list`_, and we'll see what we can do for you. :-)
     1636
     1637
     1638Bugs and limitations
     1639--------------------
     1640
     1641- ``ftputil`` needs at least Python 3.5 to work.
     1642
     1643- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
     1644  names starting with a dot) depends on the configuration of the FTP
     1645  server. See `Hidden files and directories`_ for details.
     1646
     1647- Due to the implementation of ``lstat`` it can not return a sensible
     1648  value for the root directory ``/`` though stat'ing entries *in* the
     1649  root directory isn't a problem. If you know an implementation that
     1650  can do this, please let me know. The root directory is handled
     1651  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
     1652
     1653- In multithreaded programs, you can have each thread use one or more
     1654  ``FTPHost`` instances as long as no instance is shared with other
     1655  threads.
     1656
     1657- Currently, it is not possible to continue an interrupted upload or
     1658  download. Contact me if this causes problems for you.
     1659
     1660- There's exactly one cache for ``lstat`` results for each ``FTPHost``
     1661  object, i. e. there's no sharing of cache results determined by
     1662  several ``FTPHost`` objects. See `Local caching of file system
     1663  information`_ for the reasons.
     1664
     1665
     1666Files
     1667-----
     1668
     1669If not overwritten via installation options, the ``ftputil`` files
     1670reside in the ``ftputil`` package. There's also documentation in
     1671`reStructuredText`_ and in HTML format. The locations of these
     1672files after installation is system-dependent.
     1673
     1674.. _`reStructuredText`: https://docutils.sourceforge.net/rst.html
     1675
     1676The files ``test_*.py`` and ``scripted_session.py`` are for
     1677unit-testing. If you only *use* ``ftputil``, i. e. *don't* modify it,
     1678you can delete these files.
     1679
     1680
     1681References
     1682----------
     1683
     1684- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
     1685
     1686- Python Software Foundation. 2020. `The Python Standard Library`_.
     1687
     1688.. _`RFC 959 - File Transfer Protocol (FTP)`: https://www.ietf.org/rfc/rfc959.txt
     1689.. _`The Python Standard Library`: https://docs.python.org/library/index.html
     1690
     1691
     1692Authors
     1693-------
     1694
     1695``ftputil`` is written by Stefan Schwarzer
     1696<sschwarzer@sschwarzer.net> and contributors (see
     1697``doc/contributors.txt``).
     1698
     1699The original ``lrucache`` module was written by Evan Prodromou
     1700<evan@prodromou.name>.
     1701
     1702Feedback is appreciated. :-)
    41703}}}