source: doc/ftputil.txt @ 1940:b0cb3871ed21

Last change on this file since 1940:b0cb3871ed21 was 1940:b0cb3871ed21, checked in by Stefan Schwarzer <sschwarzer@…>, 16 months ago
Adapt documentation to changed time shift semantics ticket: 134
File size: 61.5 KB
Line 
1``ftputil`` -- a high-level FTP client library
2==============================================
3
4:Version:   3.4
5:Date:      2017-11-08
6:Summary:   high-level FTP client library for Python
7:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
8:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
9
10.. contents::
11
12
13Introduction
14------------
15
16The ``ftputil`` module is a high-level interface to the ftplib_
17module. The `FTPHost objects`_ generated from it allow many operations
18similar to those of os_, `os.path`_ and `shutil`_.
19
20.. _ftplib: https://docs.python.org/library/ftplib.html
21.. _os: https://docs.python.org/library/os.html
22.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
23.. _`os.path`: https://docs.python.org/library/os.path.html
24.. _`shutil`: https://docs.python.org/library/shutil.html
25
26Example::
27
28    import ftputil
29
30    # Download some files from the login directory.
31    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
32        names = ftp_host.listdir(ftp_host.curdir)
33        for name in names:
34            if ftp_host.path.isfile(name):
35                ftp_host.download(name, name)  # remote, local
36        # Make a new directory and copy a remote file into it.
37        ftp_host.mkdir("newdir")
38        with ftp_host.open("index.html", "rb") as source:
39            with ftp_host.open("newdir/index.html", "wb") as target:
40                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
41
42Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
43modification time of a file. The latter can also follow links, similar
44to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
45
46
47``ftputil`` features
48--------------------
49
50* Method names are familiar from Python's ``os``, ``os.path`` and
51  ``shutil`` modules. For example, use ``os.path.join`` to join
52  paths for a local file system and ``ftp_host.path.join`` to join
53  paths for a remote FTP file system.
54
55* Remote file system navigation (``getcwd``, ``chdir``)
56
57* Upload and download files (``upload``, ``upload_if_newer``,
58  ``download``, ``download_if_newer``)
59
60* Time zone synchronization between client and server (needed
61  for ``upload_if_newer`` and ``download_if_newer``)
62
63* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
64  ``rmtree``) and remove files (``remove``)
65
66* Get information about directories, files and links (``listdir``,
67  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
68  ``abspath``, ``dirname``, ``basename`` etc.)
69
70* Iterate over remote file systems (``walk``)
71
72* Local caching of results from ``lstat`` and ``stat`` calls to reduce
73  network access (also applies to ``exists``, ``getmtime`` etc.).
74
75* Read files from and write files to remote hosts via
76  file-like objects (``FTPHost.open``; the generated file-like objects
77  have the familiar methods like ``read``, ``readline``, ``readlines``,
78  ``write``, ``writelines`` and ``close``. You can also iterate over
79  these files line by line in a ``for`` loop.
80
81
82Exception hierarchy
83-------------------
84
85The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
86``ftputil.error.TemporaryError``.
87
88The exception classes are organized as follows::
89
90    FTPError
91        FTPOSError(FTPError, OSError)
92            PermanentError(FTPOSError)
93                CommandNotImplementedError(PermanentError)
94            TemporaryError(FTPOSError)
95        FTPIOError(FTPError)
96        InternalError(FTPError)
97            InaccessibleLoginDirError(InternalError)
98            ParserError(InternalError)
99            RootDirError(InternalError)
100            TimeShiftError(InternalError)
101
102and are described here:
103
104- ``FTPError``
105
106  is the root of the exception hierarchy of the module.
107
108- ``FTPOSError``
109
110  is derived from ``OSError``. This is for similarity between the
111  os module and ``FTPHost`` objects. Compare
112
113  ::
114
115    try:
116        os.chdir("nonexisting_directory")
117    except OSError:
118        ...
119
120  with
121
122  ::
123
124    host = ftputil.FTPHost("host", "user", "password")
125    try:
126        host.chdir("nonexisting_directory")
127    except OSError:
128        ...
129
130  Imagine a function
131
132  ::
133
134    def func(path, file):
135        ...
136
137  which works on the local file system and catches ``OSErrors``. If you
138  change the parameter list to
139
140  ::
141
142    def func(path, file, os=os):
143        ...
144
145  where ``os`` denotes the ``os`` module, you can call the function also as
146
147  ::
148
149    host = ftputil.FTPHost("host", "user", "password")
150    func(path, file, os=host)
151
152  to use the same code for both a local and remote file system.
153  Another similarity between ``OSError`` and ``FTPOSError`` is that
154  the latter holds the FTP server return code in the ``errno``
155  attribute of the exception object and the error text in
156  ``strerror``.
157
158- ``PermanentError``
159
160  is raised for 5xx return codes from the FTP server. This
161  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
162  ``ftplib.error_perm`` are *not* identical).
163
164- ``CommandNotImplementedError``
165
166  indicates that an underlying command the code tries to use is not
167  implemented. For an example, see the description of the
168  `FTPHost.chmod`_ method.
169
170- ``TemporaryError``
171
172  is raised for FTP return codes from the 4xx category. This
173  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
174  ``ftplib.error_temp`` are *not* identical).
175
176- ``FTPIOError``
177
178  denotes an I/O error on the remote host. This appears
179  mainly with file-like objects that are retrieved by calling
180  ``FTPHost.open``. Compare
181
182  ::
183
184    >>> try:
185    ...     f = open("not_there")
186    ... except IOError as obj:
187    ...     print(obj.errno)
188    ...     print(obj.strerror)
189    ...
190    2
191    No such file or directory
192
193  with
194
195  ::
196
197    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
198    >>> try:
199    ...     f = ftp_host.open("not_there")
200    ... except IOError as obj:
201    ...     print(obj.errno)
202    ...     print(obj.strerror)
203    ...
204    550
205    550 not_there: No such file or directory.
206
207  As you can see, both code snippets are similar. However, the error
208  codes aren't the same.
209
210- ``InternalError``
211
212  subsumes exception classes for signaling errors due to limitations
213  of the FTP protocol or the concrete implementation of ``ftputil``.
214
215- ``InaccessibleLoginDirError``
216
217  This exception is raised if the directory in which "you" are placed
218  upon login is not accessible, i. e. a ``chdir`` call with the
219  directory as argument would fail.
220
221- ``ParserError``
222
223  is used for errors during the parsing of directory
224  listings from the server. This exception is used by the ``FTPHost``
225  methods ``stat``, ``lstat``, and ``listdir``.
226
227- ``RootDirError``
228
229  Because of the implementation of the ``lstat`` method it is not
230  possible to do a ``stat`` call on the root directory ``/``.
231  If you know *any* way to do it, please let me know. :-)
232
233  This problem does *not* affect stat calls on items *in* the root
234  directory.
235
236- ``TimeShiftError``
237
238  is used to denote errors which relate to setting the `time shift`_.
239
240
241Directory and file names
242------------------------
243
244.. note::
245
246   Keep in mind that this section only applies to directory and file
247   *names*, not file *contents*. Encoding and decoding for file
248   contents is handled by the ``encoding`` argument for
249   `FTPHost.open`_.
250
251First off: If your directory and file names (both as arguments and on
252the server) contain only ISO 8859-1 (latin-1) characters, you can use
253such names in the form of ``bytes`` or ``str`` objects. However, you
254can't mix different string types (``bytes`` and ``str``) in one call
255(for example in ``FTPHost.path.join``).
256
257If you have directory or file names with characters that aren't in
258latin-1, it's recommended to use ``bytes`` objects. In that case,
259returned paths will be ``bytes`` objects, too.
260
261Read on for details.
262
263.. note::
264
265   The approach described below may look awkward and in a way it is.
266   The intention of ``ftputil`` is to behave like the local file
267   system APIs of Python 3 as far as it makes sense. Moreover, the
268   taken approach makes sure that directory and file names that were
269   used with Python 3's native ``ftplib`` module will be compatible
270   with ``ftputil`` and vice versa. Otherwise you may be able to use a
271   file name with ``ftputil``, but get an exception when trying to
272   read the same file with Python 3's ``ftplib`` module.
273
274Methods that take paths of directories and/or files can take either
275``bytes`` or ``str`` objects, or `PathLike`_ objects that can be
276converted to ``bytes`` or ``str``.
277
278.. _PathLike: https://docs.python.org/3/library/os.html#os.PathLike
279
280If a method gets a string argument (or a string argument wrapped in a
281PathLike_ object) and returns one or more strings, these strings will
282have the same string type (``bytes`` or ``str``) as the argument(s).
283Mixing different string types in one call (for example in
284``FTPHost.path.join``) isn't allowed and will cause a ``TypeError``.
285These rules are the same as for local file system operations in Python 3.
286
287``bytes`` objects for directory and file names will be sent to the
288server as-is. On the other hand, ``str`` objects will be encoded to
289``bytes`` objects, assuming latin-1 encoding. This implies that such
290``str`` objects must only contain code points 0-255 for the latin-1
291character set. Using any other characters will result in a
292``UnicodeEncodeError`` exception.
293
294If you have directory or file names as ``str`` objects with
295non-latin-1 characters, encode the strings to ``bytes`` yourself,
296using the encoding you know the server uses for its file system.
297Decode received paths with the same encoding. Encapsulate these
298conversions as far as you can. Otherwise, you'd have to adapt
299potentially a lot of code if the server encoding changes.
300
301If you *don't* know the encoding on the server side, it's probably the
302best to only use ``bytes`` for directory and file names. That said, as
303soon as you *show* the names to a user, you -- or the library you use
304for displaying the names -- has to guess an encoding.
305
306If you can decide about paths yourself, it's generally safest to use
307only ASCII characters in FTP paths.
308
309
310``FTPHost`` objects
311-------------------
312
313.. _`FTPHost construction`:
314
315Construction
316~~~~~~~~~~~~
317
318Introduction
319````````````
320
321``FTPHost`` instances can be created with the following call::
322
323    ftp_host = ftputil.FTPHost(server, user, password, account,
324                               session_factory=ftplib.FTP)
325
326The first four parameters are strings with the same meaning as for the
327FTP class in the ``ftplib`` module. Usually the ``account`` and
328``session_factory`` arguments aren't needed though.
329
330``FTPHost`` objects can also be used in a ``with`` statement::
331
332    import ftputil
333
334    with ftputil.FTPHost(server, user, password) as ftp_host:
335        print(ftp_host.listdir(ftp_host.curdir))
336
337After the ``with`` block, the ``FTPHost`` instance and the
338associated FTP sessions will be closed automatically.
339
340If something goes wrong during the ``FTPHost`` construction or in the
341body of the ``with`` statement, the instance is closed as well.
342Exceptions will be propagated (as with ``try ... finally``).
343
344Session factories
345`````````````````
346
347The keyword argument ``session_factory`` may be used to generate FTP
348connections with other factories than the default ``ftplib.FTP``. For
349example, the standard library of Python 3 contains a class
350``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
351connection.
352
353In fact, all positional and keyword arguments other than
354``session_factory`` are passed to the factory to generate a new
355background session. This also happens for every remote file that is
356opened; see below.
357
358This functionality of the constructor also allows to wrap
359``ftplib.FTP`` objects to do something that wouldn't be possible with
360the ``ftplib.FTP`` constructor alone.
361
362As an example, assume you want to connect to another than the default
363port, but ``ftplib.FTP`` only offers this by means of its ``connect``
364method, not via its constructor. One solution is to use a custom
365class as a session factory::
366
367    import ftplib
368    import ftputil
369
370    EXAMPLE_PORT = 50001
371
372    class MySession(ftplib.FTP):
373
374        def __init__(self, host, userid, password, port):
375            """Act like ftplib.FTP's constructor but connect to another port."""
376            ftplib.FTP.__init__(self)
377            self.connect(host, port)
378            self.login(userid, password)
379
380    # Try _not_ to use an _instance_ `MySession()` as factory, -
381    # use the class itself.
382    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
383                         session_factory=MySession) as ftp_host:
384        # Use `ftp_host` as usual.
385        ...
386
387On login, the format of the directory listings (needed for stat'ing
388files and directories) should be determined automatically. If not,
389please `enter a ticket`_.
390
391.. _`enter a ticket`: https://ftputil.sschwarzer.net/issuetrackernotes
392
393For the most common uses you don't need to create your own session
394factory class though. The ``ftputil.session`` module has a function
395``session_factory`` that can create session factories for a variety
396of parameters::
397
398    session_factory(base_class=ftplib.FTP,
399                    port=21,
400                    use_passive_mode=None,
401                    encrypt_data_channel=True,
402                    debug_level=None)
403
404with
405
406- ``base_class`` is a base class to inherit a new session factory
407  class from. By default, this is ``ftplib.FTP`` from the Python
408  standard library.
409
410- ``port`` is the command channel port. The default is 21, used in most
411  FTP server configurations.
412
413- ``use_passive_mode`` is either a boolean that determines whether
414  passive mode should be used or ``None``. ``None`` means to let the
415  base class choose active or passive mode.
416
417- ``encrypt_data_channel`` defines whether to encrypt the data channel
418  for secure connections. This is only supported for the base classes
419  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
420  parameter is ignored.
421
422- ``debug_level`` sets the debug level for FTP session instances. The
423  semantics is defined by the base class. For example, a debug level
424  of 2 causes the most verbose output for Python's ``ftplib.FTP``
425  class.
426
427All of these parameters can be combined. For example, you could use
428
429::
430
431    import ftplib
432
433    import ftputil
434    import ftputil.session
435
436
437    my_session_factory = ftputil.session.session_factory(
438                           base_class=ftpslib.FTP_TLS,
439                           port=31,
440                           encrypt_data_channel=True,
441                           debug_level=2)
442
443    with ftputil.FTPHost(server, user, password,
444                         session_factory=my_session_factory) as ftp_host:
445        ...
446
447to create and use a session factory derived from ``ftplib.FTP_TLS``
448that connects on command channel 31, will encrypt the data channel and
449print output for debug level 2.
450
451Note: Generally, you can achieve everything you can do with
452``ftputil.session.session_factory`` with an explicit session factory
453as described at the start of this section. However, the class
454``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
455it with ftputil out of the box. The function ``session_factory``
456contains a workaround for this limitation. For details refer to `this
457ticket`_.
458
459.. _`this ticket`: https://ftputil.sschwarzer.net/trac/ticket/78
460
461Hidden files and directories
462~~~~~~~~~~~~~~~~~~~~~~~~~~~~
463
464Whether ftputil sees "hidden" files and directories (usually files or
465directories whose names start with a dot) depends on the FTP server
466configuration. By default, ftputil does *not* use the ``-a`` option in
467the FTP ``LIST`` command to find hidden files.
468
469To tell the server to list hidden directories and files, set
470``FTPHost.use_list_a_option`` to ``True``::
471
472    ftp_host = ftputil.FTPHost(server, user, password, account,
473                               session_factory=ftplib.FTP)
474    ftp_host.use_list_a_option = True
475
476Caveats:
477
478- If the server doesn't understand the ``-a`` option at all, the
479  server may interpret ``-a`` as the name of a file or directory,
480  which can result in odd behavior. Therefore, use ``-a`` only if
481  you're sure the server you're talking to supports it. Another
482  approach is to have test code for ``-a`` support and fall back to
483  not using the option.
484
485- Even if the server knows about the ``-a`` option, the server may
486  be configured to ignore it.
487
488``FTPHost`` attributes and methods
489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
490
491Attributes
492``````````
493
494- ``curdir``, ``pardir``, ``sep``
495
496  are strings which denote the current and the parent directory on the
497  remote server. ``sep`` holds the path separator. Though `RFC 959`_
498  (File Transfer Protocol) notes that these values may depend on the
499  FTP server implementation, the Unix variants seem to work well in
500  practice, even for non-Unix servers.
501
502  Nevertheless, it's recommended that you don't hardcode these values
503  for remote paths, but use `FTPHost.path`_ as you would use
504  ``os.path`` to write platform-independent Python code for local
505  filesystems. Keep in mind that most, *but not all*, arguments of
506  ``FTPHost`` methods refer to remote directories or files. For
507  example, in `FTPHost.upload`_, the first argument is a local
508  path and the second a remote path. Both of these should use their
509  respective path separators.
510
511.. _`FTPHost.upload`: `Uploading and downloading files`_
512
513Remote file system navigation
514`````````````````````````````
515
516- ``getcwd()``
517
518  returns the absolute current directory on the remote host. This
519  method works like ``os.getcwd``.
520
521- ``chdir(directory)``
522
523  sets the current directory on the FTP server. This resembles
524  ``os.chdir``, as you may have expected.
525
526.. _`callback function`:
527
528Uploading and downloading files
529```````````````````````````````
530
531- ``upload(source, target, callback=None)``
532
533  copies a local source file (given by a filename, i. e. a string)
534  to the remote host under the name target. Both ``source`` and
535  ``target`` may be absolute paths or relative to their corresponding
536  current directory (on the local or the remote host, respectively).
537
538  The file content is always transferred in binary mode.
539
540  The callback, if given, will be invoked for each transferred chunk
541  of data::
542
543    callback(chunk)
544
545  where ``chunk`` is a bytestring. An example usage of a callback
546  method is to display a progress indicator.
547
548- ``download(source, target, callback=None)``
549
550  performs a download from the remote source file to a local target
551  file. Both ``source`` and ``target`` are strings. See the
552  description of ``upload`` for more details.
553
554.. _`upload_if_newer`:
555
556- ``upload_if_newer(source, target, callback=None)``
557
558  is similar to the ``upload`` method. The only difference is that the
559  upload is only invoked if the time of the last modification for the
560  source file is more recent than that of the target file or the
561  target doesn't exist at all. The check for the last modification
562  time considers the precision of the timestamps and transfers a file
563  "if in doubt". Consequently the code
564
565  ::
566
567    ftp_host.upload_if_newer("source_file", "target_file")
568    time.sleep(10)
569    ftp_host.upload_if_newer("source_file", "target_file")
570
571  might upload the file again if the timestamp of the target file is
572  precise up to a minute, which is typically the case because the
573  remote datetime is determined by parsing a directory listing from
574  the server. To avoid unnecessary transfers, wait at least a minute
575  between calls of ``upload_if_newer`` for the same file. If it still
576  seems that a file is uploaded unnecessarily (or not when it should),
577  read the subsection on `time shift`_ settings.
578
579  If an upload actually happened, the return value of
580  ``upload_if_newer`` is a ``True``, else ``False``.
581
582  Note that the method only checks the existence and/or the
583  modification time of the source and target file; it doesn't
584  compare any other file properties, say, the file size.
585
586  This also means that if a transfer is interrupted, the remote file
587  will have a newer modification time than the local file, and thus
588  the transfer won't be repeated if ``upload_if_newer`` is used a
589  second time. There are at least two possibilities after a failed
590  upload:
591
592  - use ``upload`` instead of ``upload_if_newer``, or
593
594  - remove the incomplete target file with ``FTPHost.remove``, then
595    use ``upload`` or ``upload_if_newer`` to transfer it again.
596
597.. _`download_if_newer`:
598
599- ``download_if_newer(source, target, callback=None)``
600
601  corresponds to ``upload_if_newer`` but performs a download from the
602  server to the local host. Read the descriptions of download and
603  ``upload_if_newer`` for more information. If a download actually
604  happened, the return value is ``True``, else ``False``.
605
606.. _`time shift`:
607.. _`time zone correction`:
608
609Time zone correction
610````````````````````
611
612For ``upload_if_newer`` and ``download_if_newer`` to work correctly,
613the time zone of the server must be taken into account. By default,
614ftputil assumes that the timestamps in server listings are in UTC_.
615
616.. _UTC: https://en.wikipedia.org/wiki/Utc
617
618.. _`set_time_shift`:
619
620- ``set_time_shift(time_shift)``
621
622  sets the so-called time shift value, measured in seconds. The time
623  shift here is defined as the difference between the time used in
624  server listings and UTC.
625
626  ::
627
628    time_shift = server_time - utc_time
629
630  For example, a server in Berlin/Germany set to the local time
631  (currently UTC+03:00), would require a time shift value of 3 *
632  3600.0 = 10800.0 seconds to be handled correctly by ftputil's
633  ``upload_if_newer`` and ``download_if_newer``, as well as the
634  ``stat`` and ``lstat`` calls.
635
636  Note that servers don't necessarily send their file system listings
637  in their local time zone. Some use UTC, which actually makes sense
638  because UTC doesn't lead to an ambiguity when there's a switch back
639  from the daylight saving time to the "normal" time of the server
640  location.
641
642  If the time shift value is invalid, for example its absolute value
643  is larger than 24 hours, a ``TimeShiftError`` is raised.
644
645  .. note::
646
647     Versions of ftputil before 4.0.0 used a different definition of
648     "time shift", server_time - local_client_time.
649
650     This had the advantage that the default of 0.0 would be correct
651     *if* the server was set to the same time zone as the client
652     where ftputil runs. On the other hand, this approach meant that
653     the time shift depended on *two* time zones, not only the one
654     used on the server side. This could be confusing if server
655     and client *didn't* use the same time zone.
656
657  See also `synchronize_times`_ for a way to set the time shift with a
658  simple method call. If you can't use ``synchronize_times`` *and* the
659  server uses the same time zone as the client, you can set the time
660  shift value with
661
662  ::
663
664    set_time_shift(
665      round( (datetime.datetime.now() - datetime.datetime.utcnow()).seconds, -2 )
666    )
667
668- ``time_shift()``
669
670  returns the currently-set time shift value. See ``set_time_shift``
671  above for its definition.
672
673.. _`synchronize_times`:
674
675- ``synchronize_times()``
676
677  synchronizes the local times of the server and the client, so that
678  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
679  if the client and the server use different time zones. For this
680  to work, *all* of the following conditions must be true:
681
682  - The connection between server and client is established.
683
684  - The client has write access to the directory that is current when
685    ``synchronize_times`` is called.
686
687  If you can't fulfill these conditions, you can nevertheless set the
688  time shift value explicitly with `set_time_shift`_. Trying to call
689  ``synchronize_times`` if the above conditions aren't met results in
690  a ``TimeShiftError`` exception.
691
692Creating and removing directories
693`````````````````````````````````
694
695- ``mkdir(path, [mode])``
696
697  makes the given directory on the remote host. This does *not*
698  construct "intermediate" directories that don't already exist. The
699  ``mode`` parameter is ignored; this is for compatibility with
700  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
701  instead of the ``os`` module. See the explanation in the subsection
702  `Exception hierarchy`_.
703
704- ``makedirs(path, [mode])``
705
706  works similar to ``mkdir`` (see above), but also makes intermediate
707  directories like ``os.makedirs``. The ``mode`` parameter is only
708  there for compatibility with ``os.makedirs`` and is ignored.
709
710- ``rmdir(path)``
711
712  removes the given remote directory. If it's not empty, raise
713  a ``PermanentError``.
714
715- ``rmtree(path, ignore_errors=False, onerror=None)``
716
717  removes the given remote, possibly non-empty, directory tree.
718  The interface of this method is rather complex, in favor of
719  compatibility with ``shutil.rmtree``.
720
721  If ``ignore_errors`` is set to a true value, errors are ignored.
722  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
723  set, all exceptions occurring during the tree iteration and
724  processing are raised. These exceptions are all of type
725  ``PermanentError``.
726
727  To distinguish between different kinds of errors, pass in a callable
728  for ``onerror``. This callable must accept three arguments:
729  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
730  object, *for example* ``your_host_object.listdir``. ``path`` is the
731  path that was the recent argument of the respective method
732  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
733  info as it is gotten from ``sys.exc_info``.
734
735  The code of ``rmtree`` is taken from Python's ``shutil`` module
736  and adapted for ``ftputil``.
737
738Removing files and links
739````````````````````````
740
741- ``remove(path)``
742
743  removes a file or link on the remote host, similar to ``os.remove``.
744
745- ``unlink(path)``
746
747  is an alias for ``remove``.
748
749Retrieving information about directories, files and links
750`````````````````````````````````````````````````````````
751
752- ``listdir(path)``
753
754  returns a list containing the names of the files and directories
755  in the given path, similar to `os.listdir`_. The special names
756  ``.`` and ``..`` are not in the list.
757
758The methods ``lstat`` and ``stat`` (and some others) rely on the
759directory listing format used by the FTP server. When connecting to a
760host, ``FTPHost``'s constructor tries to guess the right format, which
761succeeds in most cases. However, if you get strange results or
762``ParserError`` exceptions by a mere ``lstat`` call, please `enter a
763ticket`_.
764
765If ``lstat`` or ``stat`` give wrong modification dates or times, look
766at the methods that deal with time zone differences (`time zone
767correction`_).
768
769.. _`FTPHost.lstat`:
770
771- ``lstat(path)``
772
773  returns an object similar to that from `os.lstat`_. This is a kind
774  of tuple with additional attributes; see the documentation of the
775  ``os`` module for details.
776
777  The result is derived by parsing the output of a ``LIST`` command on
778  the server. Therefore, the result from ``FTPHost.lstat`` can not
779  contain more information than the received text. In particular:
780
781  - User and group ids can only be determined as strings, not as
782    numbers, and that only if the server supplies them. This is
783    usually the case with Unix servers but maybe not for other FTP
784    servers.
785
786  - Values for the time of the last modification may be rough,
787    depending on the information from the server. For timestamps
788    older than a year, this usually means that the precision of the
789    modification timestamp value is not better than a day. For newer
790    files, the information may be accurate to a minute.
791
792    If the time of the last modification is before the epoch (usually
793    1970-01-01 UTC), set the time of the last modification to 0.0.
794
795  - Links can only be recognized on servers that provide this
796    information in the ``LIST`` output.
797
798  - Stat attributes that can't be determined at all are set to
799        ``None``. For example, a line of a directory listing may not
800        contain the date/time of a directory's last modification.
801
802  - There's a special problem with stat'ing the root directory.
803    (Stat'ing things *in* the root directory is fine though.) In
804    this case, a ``RootDirError`` is raised. This has to do with the
805    algorithm used by ``(l)stat``, and I know of no approach which
806    mends this problem.
807
808  Currently, ``ftputil`` recognizes the common Unix-style and
809  Microsoft/DOS-style directory formats. If you need to parse output
810  from another server type, please write to the `ftputil mailing
811  list`_. You may consider `writing your own parser`_.
812
813.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
814.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
815.. _`ftputil mailing list`: https://ftputil.sschwarzer.net/mailinglist
816.. _`writing your own parser`: `Writing directory parsers`_
817
818.. _`FTPHost.stat`:
819
820- ``stat(path)``
821
822  returns ``stat`` information also for files which are pointed to by a
823  link. This method follows multiple links until a regular file or
824  directory is found. If an infinite link chain is encountered or the
825  target of the last link in the chain doesn't exist, a
826  ``PermanentError`` is raised.
827
828  The limitations of the ``lstat`` method also apply to ``stat``.
829
830.. _`FTPHost.path`:
831
832``FTPHost`` objects contain an attribute named ``path``, similar to
833`os.path`_. The following methods can be applied to the remote host
834with the same semantics as for ``os.path``:
835
836::
837
838    abspath(path)
839    basename(path)
840    commonprefix(path_list)
841    dirname(path)
842    exists(path)
843    getmtime(path)
844    getsize(path)
845    isabs(path)
846    isdir(path)
847    isfile(path)
848    islink(path)
849    join(path1, path2, ...)
850    normcase(path)
851    normpath(path)
852    split(path)
853    splitdrive(path)
854    splitext(path)
855    walk(path, func, arg)
856
857Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
858methods return ``False`` if they can't find the path given by their
859argument.
860
861Local caching of file system information
862````````````````````````````````````````
863
864Many of the above methods need access to the remote file system to
865obtain data on directories and files. To get the most recent data,
866*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
867would require to fetch a directory listing from the server, which can
868make the program *very* slow. This effect is more pronounced for
869operations which mostly scan the file system rather than transferring
870file data.
871
872For this reason, ``ftputil`` by default saves the results from
873directory listings locally and reuses those results. This reduces
874network accesses and so speeds up the software a lot. However, since
875data is more rarely fetched from the server, the risk of obsolete data
876also increases. This will be discussed below.
877
878Caching can be controlled -- if necessary at all -- via the
879``stat_cache`` object in an ``FTPHost``'s namespace. For example,
880after calling
881
882::
883
884    ftp_host = ftputil.FTPHost(host, user, password)
885
886the cache can be accessed as ``ftp_host.stat_cache``.
887
888While ``ftputil`` usually manages the cache quite well, there are two
889possible reasons for modifying cache parameters.
890
891The first is when the number of possible entries is too low. You may
892notice that when you are processing very large directories and the
893program becomes much slower than before. It's common for code to read
894a directory with ``listdir`` and then process the found directories
895and files. This can also happen implicitly by a call to
896``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
897increases the cache size if directories with more entries than the
898current maximum cache size are to be scanned. Most of the time, this
899works fine.
900
901However, if you need access to stat data for several directories at
902the same time, you may need to increase the cache explicitly. This is
903done by the ``resize`` method::
904
905    ftp_host.stat_cache.resize(20000)
906
907where the argument is the maximum number of ``lstat`` results to store
908(the default is 5000, in versions before 2.6 it was 1000). Note that
909each path on the server, e. g. "/home/schwa/some_dir", corresponds to
910a single cache entry. Methods like ``exists`` or ``getmtime`` all
911derive their results from a previously fetched ``lstat`` result.
912
913The value 5000 above means that the cache will hold *at most* 5000
914entries (unless increased automatically by an explicit or implicit
915``listdir`` call, see above). If more are about to be stored, the
916entries which haven't been used for the longest time will be deleted
917to make place for newer entries.
918
919The second possible reason to change the cache parameters is to avoid
920stale cache data. Caching is so effective because it reduces network
921accesses. This can also be a disadvantage if the file system data on
922the remote server changes after a stat result has been retrieved; the
923client, when looking at the cached stat data, will use obsolete
924information.
925
926There are two potential ways to get such out-of-date stat data. The
927first happens when an ``FTPHost`` instance modifies a file path for
928which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
929Such changes are handled transparently; the path will be deleted from
930the cache. A different matter are changes unknown to the ``FTPHost``
931object which inspects its cache. Obviously, for example, these are
932changes by programs running on the remote host. On the other hand,
933cache inconsistencies can also occur if two ``FTPHost`` objects change
934a file system simultaneously::
935
936    with (
937      ftputil.FTPHost(server, user1, password1) as ftp_host1,
938      ftputil.FTPHost(server, user1, password1) as ftp_host2
939    ):
940        stat_result1 = ftp_host1.stat("some_file")
941        stat_result2 = ftp_host2.stat("some_file")
942        ftp_host2.remove("some_file")
943        # `ftp_host1` will still see the obsolete cache entry!
944        print(ftp_host1.stat("some_file"))
945        # Will raise an exception since an `FTPHost` object
946        # knows of its own changes.
947        print(ftp_host2.stat("some_file"))
948
949At first sight, it may appear to be a good idea to have a shared cache
950among several ``FTPHost`` objects. After some thinking, this turns out
951to be very error-prone. For example, it won't help with different
952processes using ``ftputil``. So, if you have to deal with concurrent
953write/read accesses to a server, you have to handle them explicitly.
954
955The most useful tool for this is the ``invalidate`` method. In the
956example above, it could be used like this::
957
958    with (
959      ftputil.FTPHost(server, user1, password1) as ftp_host1,
960      ftputil.FTPHost(server, user1, password1) as ftp_host2
961    ):
962        stat_result1 = ftp_host1.stat("some_file")
963        stat_result2 = ftp_host2.stat("some_file")
964        ftp_host2.remove("some_file")
965        # Invalidate using an absolute path.
966        absolute_path = ftp_host1.path.abspath(
967                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
968        ftp_host1.stat_cache.invalidate(absolute_path)
969        # Will now raise an exception as it should.
970        print(ftp_host1.stat("some_file"))
971        # Would raise an exception since an `FTPHost` object
972        # knows of its own changes, even without `invalidate`.
973        print(ftp_host2.stat("some_file"))
974
975The method ``invalidate`` can be used on any *absolute* path, be it a
976directory, a file or a link.
977
978By default, the cache entries (if not replaced by newer ones) are
979stored for an infinite time. That is, if you start your Python process
980using ``ftputil`` and let it run for three days a stat call may still
981access cache data that old. To avoid this, you can set the ``max_age``
982attribute::
983
984    with ftputil.FTPHost(server, user, password) as ftp_host:
985        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
986
987This sets the maximum age of entries in the cache to an hour. This
988means any entry older won't be retrieved from the cache but its data
989instead fetched again from the remote host and then again stored for
990up to an hour. To reset `max_age` to the default of unlimited age,
991i. e. cache entries never expire, use ``None`` as value.
992
993If you are certain that the cache will be in the way, you can disable
994and later re-enable it completely with ``disable`` and ``enable``::
995
996    with ftputil.FTPHost(server, user, password) as ftp_host:
997        ftp_host.stat_cache.disable()
998        ...
999        ftp_host.stat_cache.enable()
1000
1001During that time, the cache won't be used; all data will be fetched
1002from the network. After enabling the cache again, its entries will be
1003the same as when the cache was disabled, that is, entries won't get
1004updated with newer data during this period. Note that even when the
1005cache is disabled, the file system data in the code can become
1006inconsistent::
1007
1008    with ftputil.FTPHost(server, user, password) as ftp_host:
1009        ftp_host.stat_cache.disable()
1010        if ftp_host.path.exists("some_file"):
1011            mtime = ftp_host.path.getmtime("some_file")
1012
1013In that case, the file ``some_file`` may have been removed by another
1014process between the calls to ``exists`` and ``getmtime``!
1015
1016Iteration over directories
1017``````````````````````````
1018
1019.. _`FTPHost.walk`:
1020
1021- ``walk(top, topdown=True, onerror=None, followlinks=False)``
1022
1023  iterates over a directory tree, similar to `os.walk`_. Actually,
1024  ``FTPHost.walk`` uses the code from Python with just the necessary
1025  modifications, so see the linked documentation.
1026
1027.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
1028
1029.. _`FTPHost.path.walk`:
1030
1031- ``path.walk(path, func, arg)``
1032
1033  Similar to ``os.path.walk``, the ``walk`` method in
1034  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
1035  easier to use.
1036
1037Other methods
1038`````````````
1039
1040- ``close()``
1041
1042  closes the connection to the remote host. After this, no more
1043  interaction with the FTP server is possible with this ``FTPHost``
1044  object. Usually you don't need to close an ``FTPHost`` instance
1045  with ``close`` if you set up the instance in a ``with`` statement.
1046
1047- ``rename(source, target)``
1048
1049  renames the source file (or directory) on the FTP server.
1050
1051.. _`FTPHost.chmod`:
1052
1053- ``chmod(path, mode)``
1054
1055  sets the access mode (permission flags) for the given path. The mode
1056  is an integer as returned for the mode by the ``stat`` and ``lstat``
1057  methods. Be careful: Usually, mode values are written as octal
1058  numbers, for example 0755 to make a directory readable and writable
1059  for the owner, but not writable for the group and others. If you
1060  want to use such octal values, rely on Python's support for them::
1061
1062    ftp_host.chmod("some_directory", 0o755)
1063
1064  Not all FTP servers support the ``chmod`` command. In case of
1065  an exception, how do you know if the path doesn't exist or if
1066  the command itself is invalid? If the FTP server complies with
1067  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
1068  command isn't allowed. ``ftputil`` maps this special error
1069  response to a ``CommandNotImplementedError`` which is derived from
1070  ``PermanentError``.
1071
1072  So you need to code like this::
1073
1074    with ftputil.FTPHost(server, user, password) as ftp_host:
1075        try:
1076            ftp_host.chmod("some_file", 0o644)
1077        except ftputil.error.CommandNotImplementedError:
1078            # `chmod` not supported
1079            ...
1080        except ftputil.error.PermanentError:
1081            # Possibly a non-existent file
1082            ...
1083
1084  Because the ``CommandNotImplementedError`` is more specific, you
1085  have to test for it first.
1086
1087.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
1088
1089- ``copyfileobj(source, target, length=64*1024)``
1090
1091  copies the contents from the file-like object ``source`` to the
1092  file-like object ``target``. The only difference to
1093  ``shutil.copyfileobj`` is the default buffer size. Note that
1094  arbitrary file-like objects can be used as arguments (e. g. local
1095  files, remote FTP files).
1096
1097  However, the interfaces of ``source`` and ``target`` have to match;
1098  the string type read from ``source`` must be an accepted string type
1099  when written to ``target``. For example, if you open ``source`` in
1100  Python 3 as a local text file and ``target`` as a remote file object
1101  in binary mode, the transfer will fail since ``source.read`` gives
1102  unicode strings (``str``) whereas ``target.write`` only accepts byte
1103  strings (``bytes``).
1104
1105  See `File-like objects`_ for the construction and use of remote
1106  file-like objects.
1107
1108.. _`set_parser`:
1109
1110- ``set_parser(parser)``
1111
1112  sets a custom parser for FTP directories. Note that you have to pass
1113  in a parser *instance*, not the class.
1114
1115  An `extra section`_ shows how to write own parsers if the default
1116  parsers in ``ftputil`` don't work for you.
1117
1118.. _`extra section`: `Writing directory parsers`_
1119
1120.. _`keep_alive`:
1121
1122- ``keep_alive()``
1123
1124  attempts to keep the connection to the remote server active in order
1125  to prevent timeouts from happening. This method is primarily
1126  intended to keep the underlying FTP connection of an ``FTPHost``
1127  object alive while a file is uploaded or downloaded. This will
1128  require either an extra thread while the upload or download is in
1129  progress or calling ``keep_alive`` from a `callback function`_.
1130
1131  The ``keep_alive`` method won't help if the connection has already
1132  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
1133
1134  If you want to use this method, keep in mind that FTP servers define
1135  a timeout for a reason. A timeout prevents running out of server
1136  connections because of clients that never disconnect on their own.
1137
1138  Note that the ``keep_alive`` method does *not* affect the "hidden"
1139  FTP child connections established by ``FTPHost.open`` (see section
1140  `FTPHost instances vs. FTP connections`_ for details). You *can't*
1141  use ``keep_alive`` to avoid a timeout in a stalling transfer like
1142  this::
1143
1144      with ftputil.FTPHost(server, userid, password) as ftp_host:
1145          with ftp_host.open("some_remote_file", "rb") as fobj:
1146              data = fobj.read(100)
1147              # _Futile_ attempt to avoid file connection timeout.
1148              for i in range(15):
1149                  time.sleep(60)
1150                  ftp_host.keep_alive()
1151              # Will raise an `ftputil.error.TemporaryError`.
1152              data += fobj.read()
1153
1154
1155.. _`FTPHost.open`:
1156
1157File-like objects
1158-----------------
1159
1160Construction
1161~~~~~~~~~~~~
1162
1163Basics
1164``````
1165
1166``FTPFile`` objects are returned by a call to ``FTPHost.open``;
1167never use the ``FTPFile`` constructor directly.
1168
1169The APIs for remote file-like objects is modeled after the APIs of
1170the built-in ``open`` function and its return value.
1171
1172- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
1173  errors=None, newline=None, rest=None)``
1174
1175  returns a file-like object that refers to the path on the remote
1176  host. This path may be absolute or relative to the current directory
1177  on the remote host (this directory can be determined with the
1178  ``getcwd`` method). As with local file objects, the default mode is
1179  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
1180  "wb".
1181
1182  If a file is opened in binary mode, you *must not* specify an
1183  encoding. On the other hand, if you open a file in text mode, an
1184  encoding is used. By default, this is the return value of
1185  ``locale.getpreferredencoding``, but you can (and probably should)
1186  specify a distinct encoding.
1187
1188  If you open a file in binary mode, the read and write operations use
1189  ``bytes`` objects. That is, read operations return ``bytes`` and
1190  write operations only accept ``bytes``.
1191
1192  Similarly, text files always work with strings (``str``). Here, read
1193  operations return string and write operations only accept strings.
1194
1195  The arguments ``buffering``, ``errors`` and ``newline`` have the
1196  same semantics as in open_.
1197
1198  If the file is opened in binary mode, you may pass 0 or a positive
1199  integer for the ``rest`` argument. The argument is passed to the
1200  underlying FTP session instance (for example an instance of
1201  ``ftplib.FTP``) to start reading or writing at the given byte
1202  offset. For example, if a remote file contains the letters
1203  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".
1204
1205  .. warning::
1206
1207     If you pass ``rest`` values which point *after* the file, the
1208     behavior is undefined and may even differ from one FTP server to
1209     another. Therefore, use the ``rest`` argument only for error
1210     recovery in case of interrupted transfers. You need to keep track
1211     of the transferred data so that you can provide a valid ``rest``
1212     argument for a resumed transfer.
1213
1214.. _`open`: https://docs.python.org/3/library/functions.html#open
1215
1216``FTPHost.open`` can also be used in a ``with`` statement::
1217
1218    import ftputil
1219
1220    with ftputil.FTPHost(...) as ftp_host:
1221        ...
1222        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
1223            fobj.write("This is some text.")
1224
1225At the end of the ``with`` block, the remote file will be closed
1226automatically.
1227
1228If something goes wrong during the construction of the file or in the
1229body of the ``with`` statement, the file will be closed as well.
1230Exceptions will be propagated as with ``try ... finally``.
1231
1232Attributes and methods
1233~~~~~~~~~~~~~~~~~~~~~~
1234
1235The methods
1236
1237::
1238
1239    close()
1240    read([count])
1241    readline([count])
1242    readlines()
1243    write(data)
1244    writelines(string_sequence)
1245
1246and the attribute ``closed`` have the same semantics as for file
1247objects of a local disk file system. The iterator protocol is
1248supported as well, i. e. you can use a loop to read a file line by
1249line::
1250
1251    with ftputil.FTPHost(server, user, password) as ftp_host:
1252        with ftp_host.open("some_file") as input_file:
1253            for line in input_file:
1254                # Do something with the line, e. g.
1255                print(line.strip().replace("ftplib", "ftputil"))
1256
1257For more on file objects, see the section `File objects`_ in the
1258Python Library Reference.
1259
1260.. _`file objects`: https://docs.python.org/3/glossary.html#term-file-object
1261
1262
1263.. _`child_connections`:
1264
1265``FTPHost`` instances vs. FTP connections
1266-----------------------------------------
1267
1268This section explains why keeping an ``FTPHost`` instance "alive"
1269without timing out sometimes isn't trivial. If you always finish your
1270FTP operations in time, you don't need to read this section.
1271
1272The file transfer protocol is a stateful protocol. That means an FTP
1273connection always is in a certain state. Each of these states can only
1274change to certain other states under certain conditions triggered by
1275the client or the server.
1276
1277One of the consequences is that a single FTP connection can't be used
1278at the same time, say, to transfer data on the FTP data channel and to
1279create a directory on the remote host.
1280
1281For example, consider this::
1282
1283    >>> import ftplib
1284    >>> ftp = ftplib.FTP(server, user, password)
1285    >>> ftp.pwd()
1286    '/'
1287    >>> # Start transfer. `CONTENTS` is a text file on the server.
1288    >>> socket = ftp.transfercmd("RETR CONTENTS")
1289    >>> socket
1290    <socket._socketobject object at 0x7f801a6386e0>
1291    >>> ftp.pwd()
1292    Traceback (most recent call last):
1293      File "<stdin>", line 1, in <module>
1294      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
1295        return parse257(resp)
1296      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
1297        raise error_reply, resp
1298    ftplib.error_reply: 226-File successfully transferred
1299    226 0.000 seconds (measured here), 5.60 Mbytes per second
1300    >>>
1301
1302Note that ``ftp`` is a single FTP connection, represented by an
1303``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
1304
1305On the other hand, consider this::
1306
1307    >>> import ftputil
1308    >>> ftp_host = ftputil.FTPHost(server, user, password)
1309    >>> ftp_host.getcwd()
1310    >>> fobj = ftp_host.open("CONTENTS")
1311    >>> fobj
1312    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
1313    >>> ftp_host.getcwd()
1314    u'/'
1315    >>> fobj.readline()
1316    u'Contents of FTP test directory\n'
1317    >>> fobj.close()
1318    >>>
1319
1320To be able to start a file transfer (i. e. open a remote file for
1321reading or writing) and still be able to use other FTP commands,
1322ftputil uses a trick. For every remote file, ftputil creates a new FTP
1323connection, called a child connection in the ftputil source code.
1324(Actually, FTP connections belonging to closed remote files are
1325re-used if they haven't timed out yet.)
1326
1327In most cases this approach isn't noticeable by code using ftputil.
1328However, the nice abstraction of dealing with a single FTP connection
1329falls apart if one of the child connections times out. For example, if
1330you open a remote file and work only with the initial "main"
1331connection to navigate the file system, the FTP connection for the
1332remote file may eventually time out.
1333
1334While it's often relatively easy to prevent the "main" connection from
1335timing out it's unfortunately practically impossible to do this for a
1336remote file connection (apart from transferring some data, of course).
1337For this reason, `FTPHost.keep_alive`_ affects only the main
1338connection. Child connections may still time out if they're idle for
1339too long.
1340
1341.. _`FTPHost.keep_alive`: `keep_alive`_
1342
1343Some more details:
1344
1345- A kind of "straightforward" way of keeping the main connection alive
1346  would be to call ``ftp_host.getcwd()``. However, this doesn't work
1347  because ftputil caches the current directory and returns it without
1348  actually contacting the server. That's the main reason why there's
1349  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
1350  (i. e. the session object), which isn't a public attribute.
1351
1352- Some servers define not only an idle timeout but also a transfer
1353  timeout. This means the connection times out unless there's some
1354  transfer on the data channel for this connection. So ftputil's
1355  ``keep_alive`` doesn't prevent this timeout, but an
1356  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
1357  this transfers the data for the whole directory listing which might
1358  take some time if the directory has many entries.
1359
1360Bottom line: If you can, you should organize your FTP actions so that
1361you finish everything before a timeout happens.
1362
1363
1364Writing directory parsers
1365-------------------------
1366
1367``ftputil`` recognizes the two most widely-used FTP directory formats,
1368Unix and MS style, and adjusts itself automatically. Almost every FTP
1369server uses one of these formats.
1370
1371However, if your server uses a format which is different from the two
1372provided by ``ftputil``, you can plug in a custom parser with a single
1373method call and have ``ftputil`` use this parser.
1374
1375For this, you need to write a parser class by inheriting from the
1376class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
1377
1378    import ftputil.error
1379    import ftputil.stat
1380
1381    class XyzParser(ftputil.stat.Parser):
1382        """
1383        Parse the default format of the FTP server of the XYZ
1384        corporation.
1385        """
1386
1387        def parse_line(self, line, time_shift=0.0):
1388            """
1389            Parse a `line` from the directory listing and return a
1390            corresponding `StatResult` object. If the line can't
1391            be parsed, raise `ftputil.error.ParserError`.
1392
1393            The `time_shift` argument can be used to fine-tune the
1394            parsing of dates and times. See the class
1395            `ftputil.stat.UnixParser` for an example.
1396            """
1397            # Split the `line` argument and examine it further; if
1398            # something goes wrong, raise an `ftputil.error.ParserError`.
1399            ...
1400            # Make a `StatResult` object from the parts above.
1401            stat_result = ftputil.stat.StatResult(...)
1402            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
1403            stat_result._st_name = ...
1404            stat_result._st_target = ...
1405            stat_result._st_mtime_precision = ...
1406            return stat_result
1407
1408        # Define `ignores_line` only if the default in the base class
1409        # doesn't do enough!
1410        def ignores_line(self, line):
1411            """
1412            Return a true value if the line should be ignored. For
1413            example, the implementation in the base class handles
1414            lines like "total 17". On the other hand, if the line
1415            should be used for stat'ing, return a false value.
1416            """
1417            is_total_line = super().ignores_line(line)
1418            my_test = ...
1419            return is_total_line or my_test
1420
1421A ``StatResult`` object is similar to the value returned by
1422`os.stat`_ and is usually built with statements like
1423
1424::
1425
1426    stat_result = StatResult(
1427                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
1428                     st_gid, st_size, st_atime, st_mtime, st_ctime))
1429    stat_result._st_name = ...
1430    stat_result._st_target = ...
1431    stat_result._st_mtime_precision = ...
1432
1433with the arguments of the ``StatResult`` constructor described in
1434the following table.
1435
1436===== =================== ============ =================== =======================
1437Index Attribute           os.stat type ``StatResult`` type Notes
1438===== =================== ============ =================== =======================
14390     st_mode             int          int
14401     st_ino              long         long
14412     st_dev              long         long
14423     st_nlink            int          int
14434     st_uid              int          str                 usually only available as string
14445     st_gid              int          str                 usually only available as string
14456     st_size             long         long
14467     st_atime            int/float    float
14478     st_mtime            int/float    float
14489     st_ctime            int/float    float
1449\-    _st_name            \-           str                 file name without directory part
1450\-    _st_target          \-           str                 link target (may be absolute or relative)
1451\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
1452===== =================== ============ =================== =======================
1453
1454If you can't extract all the desirable data from a line (for
1455example, the MS format doesn't contain any information about the
1456owner of a file), set the corresponding values in the ``StatResult``
1457instance to ``None``.
1458
1459Parser classes can use several helper methods which are defined in
1460the class ``Parser``:
1461
1462- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
1463  an appropriate ``st_mode`` integer value.
1464
1465- ``parse_unix_time`` returns a float number usable for the
1466  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
1467  "May"/"26"/"2005". Note that the method expects the timestamp string
1468  already split at whitespace.
1469
1470- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
1471  returns a float number like from ``time.mktime``. Note that the
1472  method expects the timestamp string already split at whitespace.
1473
1474Additionally, there's an attribute ``_month_numbers`` which maps
1475lowercase three-letter month abbreviations to integers.
1476
1477For more details, see the two "standard" parsers ``UnixParser`` and
1478``MSParser`` in the module ``ftputil/stat.py``.
1479
1480To actually *use* the parser, call the method `set_parser`_ of the
1481``FTPHost`` instance.
1482
1483If you can't write a parser or don't want to, please ask on the
1484`ftputil mailing list`_. Possibly someone has already written a parser
1485for your server or can help with it.
1486
1487
1488FAQ / Tips and tricks
1489---------------------
1490
1491Where can I get the latest version?
1492~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1493
1494See the `download page`_. Announcements will be sent to the `mailing
1495list`_. Announcements on major updates will also be posted to the
1496`Python announcements list`_.
1497
1498.. _`download page`: https://ftputil.sschwarzer.net/download
1499.. _`mailing list`: https://ftputil.sschwarzer.net/mailinglist
1500.. _`Python announcements list`: https://mail.python.org/mailman3/lists/python-announce-list.python.org/
1501
1502Is there a mailing list on ``ftputil``?
1503~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1504
1505Yes, please visit https://ftputil.sschwarzer.net/mailinglist to
1506subscribe or read the archives.
1507
1508Though you can *technically* post without subscribing first I can't
1509recommend it: The mails from non-subscribers have to be approved by
1510me and because the arriving mails contain *lots* of spam, I rarely go
1511through these mails.
1512
1513I found a bug! What now?
1514~~~~~~~~~~~~~~~~~~~~~~~~
1515
1516Before reporting a bug, make sure that you already read this manual
1517and tried the `latest version`_ of ``ftputil``. There the bug might
1518have already been fixed.
1519
1520.. _`latest version`: https://ftputil.sschwarzer.net/download
1521
1522Please see https://ftputil.sschwarzer.net/issuetrackernotes for
1523guidelines on entering a bug in ``ftputil``'s ticket system. If you
1524are unsure if the behaviour you found is a bug or not, you should write
1525to the `ftputil mailing list`_. *Never* include confidential information
1526(user id, password, file names, etc.) in the problem report! Be
1527careful!
1528
1529Does ``ftputil`` support TLS?
1530~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1531
1532``ftputil`` has no *built-in* TLS support.
1533
1534On the other hand, there are two ways to get TLS support with
1535ftputil:
1536
1537- The ``ftplib`` library has a class ``FTP_TLS`` that you can use for
1538  the ``session_factory`` keyword argument in the ``FTPHost``
1539  constructor. You can't use the class directly though *if* you need
1540  additional setup code in comparison to ``ftplib.FTP``, for example
1541  calling ``prot_p``, to secure the data connection. On the other
1542  hand, `ftputil.session.session_factory`_ can be used to create a
1543  custom session factory.
1544
1545- If you have other requirements that ``session_factory`` can't
1546  fulfill, you may create your own session factory by inheriting from
1547  ``ftplib.FTP_TLS``::
1548
1549    import ftplib
1550
1551    import ftputil
1552
1553
1554    class FTPTLSSession(ftplib.FTP_TLS):
1555
1556        def __init__(self, host, user, password):
1557            ftplib.FTP_TLS.__init__(self)
1558            self.connect(host, port)
1559            self.login(user, password)
1560            # Set up encrypted data connection.
1561            self.prot_p()
1562            ...
1563
1564    # Note the `session_factory` parameter. Pass the class, not
1565    # an instance.
1566    with ftputil.FTPHost(server, user, password,
1567                         session_factory=FTPTLSSession) as ftp_host:
1568        # Use `ftp_host` as usual.
1569        ...
1570
1571.. _`ftputil.session.session_factory`: `Session factories`_
1572
1573
1574How do I connect to a non-default port?
1575~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1576
1577By default, an instantiated ``FTPHost`` object connects on the usual
1578FTP port. If you have to use a different port, refer to the section
1579`Session factories`_.
1580
1581How do I set active or passive mode?
1582~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1583
1584Please see the section `Session factories`_.
1585
1586How can I debug an FTP connection problem?
1587~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1588
1589You can do this with a session factory. See `Session factories`_.
1590
1591If you want to change the debug level only temporarily after the
1592connection is established, you can reach the `session object`_ as the
1593``_session`` attribute of the ``FTPHost`` instance and call
1594``_session.set_debuglevel``. Note that the ``_session`` attribute
1595should *only* be accessed for debugging. Calling arbitrary
1596``ftplib.FTP`` methods on the session object may *cause* bugs!
1597
1598.. _`session object`: `Session factories`_
1599
1600Conditional upload/download to/from a server in a different time zone
1601~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1602
1603You may find that ``ftputil`` uploads or downloads files
1604unnecessarily, or not when it should. Please see the section on `time
1605zone correction`_. It may even be sufficient to call
1606`synchronize_times`_.
1607
1608When I use ``ftputil``, all I get is a ``ParserError`` exception
1609~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1610
1611The FTP server you connect to may use a directory format that
1612``ftputil`` doesn't understand. You can either write and
1613`plug in your own parser`_ or ask on the `mailing list`_ for
1614help.
1615
1616.. _`plug in your own parser`: `Writing directory parsers`_
1617
1618``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
1619~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1620
1621Like Python's counterparts under `os.path`_, ``ftputil``'s methods
1622return ``False`` if they can't find the given path.
1623
1624Probably you used ``listdir`` on a directory and called ``is...()`` on
1625the returned names. But if the argument for ``listdir`` wasn't the
1626current directory, the paths won't be found and so all ``is...()``
1627variants will return ``False``.
1628
1629I don't find an answer to my problem in this document
1630~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1631
1632Please send an email with your problem report or question to the
1633`ftputil mailing list`_, and we'll see what we can do for you. :-)
1634
1635
1636Bugs and limitations
1637--------------------
1638
1639- ``ftputil`` needs at least Python 3.6 to work.
1640
1641- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
1642  names starting with a dot) depends on the configuration of the FTP
1643  server. See `Hidden files and directories`_ for details.
1644
1645- Due to the implementation of ``lstat`` it can not return a sensible
1646  value for the root directory ``/`` though stat'ing entries *in* the
1647  root directory isn't a problem. If you know an implementation that
1648  can do this, please let me know. The root directory is handled
1649  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
1650
1651- In multithreaded programs, you can have each thread use one or more
1652  ``FTPHost`` instances as long as no instance is shared with other
1653  threads.
1654
1655- Currently, it is not possible to continue an interrupted upload or
1656  download. Contact me if this causes problems for you.
1657
1658- There's exactly one cache for ``lstat`` results for each ``FTPHost``
1659  object, i. e. there's no sharing of cache results determined by
1660  several ``FTPHost`` objects. See `Local caching of file system
1661  information`_ for the reasons.
1662
1663
1664Files
1665-----
1666
1667If not overwritten via installation options, the ``ftputil`` files
1668reside in the ``ftputil`` package. There's also documentation in
1669`reStructuredText`_ and in HTML format. The locations of these
1670files after installation is system-dependent.
1671
1672.. _`reStructuredText`: https://docutils.sourceforge.net/rst.html
1673
1674The files ``test_*.py`` and ``scripted_session.py`` are for
1675unit-testing. If you only *use* ``ftputil``, i. e. *don't* modify it,
1676you can delete these files.
1677
1678
1679References
1680----------
1681
1682- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
1683
1684- Python Software Foundation. 2020. `The Python Standard Library`_.
1685
1686.. _`RFC 959 - File Transfer Protocol (FTP)`: https://www.ietf.org/rfc/rfc959.txt
1687.. _`The Python Standard Library`: https://docs.python.org/library/index.html
1688
1689
1690Authors
1691-------
1692
1693``ftputil`` is written by Stefan Schwarzer
1694<sschwarzer@sschwarzer.net> and contributors (see
1695``doc/contributors.txt``).
1696
1697The original ``lrucache`` module was written by Evan Prodromou
1698<evan@prodromou.name>.
1699
1700Feedback is appreciated. :-)
Note: See TracBrowser for help on using the repository browser.