source: doc/ftputil.txt @ 1890:e067bc331d9e

Last change on this file since 1890:e067bc331d9e was 1890:e067bc331d9e, checked in by Stefan Schwarzer <sschwarzer@…>, 21 months ago
Remove arguments from `super()` This is no longer necessary in Python 3.
File size: 60.7 KB
Line 
1``ftputil`` -- a high-level FTP client library
2==============================================
3
4:Version:   3.4
5:Date:      2017-11-08
6:Summary:   high-level FTP client library for Python
7:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
8:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
9
10.. contents::
11
12
13Introduction
14------------
15
16The ``ftputil`` module is a high-level interface to the ftplib_
17module. The `FTPHost objects`_ generated from it allow many operations
18similar to those of os_, `os.path`_ and `shutil`_.
19
20.. _ftplib: https://docs.python.org/library/ftplib.html
21.. _os: https://docs.python.org/library/os.html
22.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
23.. _`os.path`: https://docs.python.org/library/os.path.html
24.. _`shutil`: https://docs.python.org/library/shutil.html
25
26Example::
27
28    import ftputil
29
30    # Download some files from the login directory.
31    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
32        names = ftp_host.listdir(ftp_host.curdir)
33        for name in names:
34            if ftp_host.path.isfile(name):
35                ftp_host.download(name, name)  # remote, local
36        # Make a new directory and copy a remote file into it.
37        ftp_host.mkdir("newdir")
38        with ftp_host.open("index.html", "rb") as source:
39            with ftp_host.open("newdir/index.html", "wb") as target:
40                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
41
42Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
43modification time of a file. The latter can also follow links, similar
44to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
45
46
47``ftputil`` features
48--------------------
49
50* Method names are familiar from Python's ``os``, ``os.path`` and
51  ``shutil`` modules. For example, use ``os.path.join`` to join
52  paths for a local file system and ``ftp_host.path.join`` to join
53  paths for a remote FTP file system.
54
55* Remote file system navigation (``getcwd``, ``chdir``)
56
57* Upload and download files (``upload``, ``upload_if_newer``,
58  ``download``, ``download_if_newer``)
59
60* Time zone synchronization between client and server (needed
61  for ``upload_if_newer`` and ``download_if_newer``)
62
63* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
64  ``rmtree``) and remove files (``remove``)
65
66* Get information about directories, files and links (``listdir``,
67  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
68  ``abspath``, ``dirname``, ``basename`` etc.)
69
70* Iterate over remote file systems (``walk``)
71
72* Local caching of results from ``lstat`` and ``stat`` calls to reduce
73  network access (also applies to ``exists``, ``getmtime`` etc.).
74
75* Read files from and write files to remote hosts via
76  file-like objects (``FTPHost.open``; the generated file-like objects
77  have the familiar methods like ``read``, ``readline``, ``readlines``,
78  ``write``, ``writelines`` and ``close``. You can also iterate over
79  these files line by line in a ``for`` loop.
80
81
82Exception hierarchy
83-------------------
84
85The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
86``ftputil.error.TemporaryError``.
87
88The exception classes are organized as follows::
89
90    FTPError
91        FTPOSError(FTPError, OSError)
92            PermanentError(FTPOSError)
93                CommandNotImplementedError(PermanentError)
94            TemporaryError(FTPOSError)
95        FTPIOError(FTPError)
96        InternalError(FTPError)
97            InaccessibleLoginDirError(InternalError)
98            ParserError(InternalError)
99            RootDirError(InternalError)
100            TimeShiftError(InternalError)
101
102and are described here:
103
104- ``FTPError``
105
106  is the root of the exception hierarchy of the module.
107
108- ``FTPOSError``
109
110  is derived from ``OSError``. This is for similarity between the
111  os module and ``FTPHost`` objects. Compare
112
113  ::
114
115    try:
116        os.chdir("nonexisting_directory")
117    except OSError:
118        ...
119
120  with
121
122  ::
123
124    host = ftputil.FTPHost("host", "user", "password")
125    try:
126        host.chdir("nonexisting_directory")
127    except OSError:
128        ...
129
130  Imagine a function
131
132  ::
133
134    def func(path, file):
135        ...
136
137  which works on the local file system and catches ``OSErrors``. If you
138  change the parameter list to
139
140  ::
141
142    def func(path, file, os=os):
143        ...
144
145  where ``os`` denotes the ``os`` module, you can call the function also as
146
147  ::
148
149    host = ftputil.FTPHost("host", "user", "password")
150    func(path, file, os=host)
151
152  to use the same code for both a local and remote file system.
153  Another similarity between ``OSError`` and ``FTPOSError`` is that
154  the latter holds the FTP server return code in the ``errno``
155  attribute of the exception object and the error text in
156  ``strerror``.
157
158- ``PermanentError``
159
160  is raised for 5xx return codes from the FTP server. This
161  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
162  ``ftplib.error_perm`` are *not* identical).
163
164- ``CommandNotImplementedError``
165
166  indicates that an underlying command the code tries to use is not
167  implemented. For an example, see the description of the
168  `FTPHost.chmod`_ method.
169
170- ``TemporaryError``
171
172  is raised for FTP return codes from the 4xx category. This
173  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
174  ``ftplib.error_temp`` are *not* identical).
175
176- ``FTPIOError``
177
178  denotes an I/O error on the remote host. This appears
179  mainly with file-like objects that are retrieved by calling
180  ``FTPHost.open``. Compare
181
182  ::
183
184    >>> try:
185    ...     f = open("not_there")
186    ... except IOError as obj:
187    ...     print(obj.errno)
188    ...     print(obj.strerror)
189    ...
190    2
191    No such file or directory
192
193  with
194
195  ::
196
197    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
198    >>> try:
199    ...     f = ftp_host.open("not_there")
200    ... except IOError as obj:
201    ...     print(obj.errno)
202    ...     print(obj.strerror)
203    ...
204    550
205    550 not_there: No such file or directory.
206
207  As you can see, both code snippets are similar. However, the error
208  codes aren't the same.
209
210- ``InternalError``
211
212  subsumes exception classes for signaling errors due to limitations
213  of the FTP protocol or the concrete implementation of ``ftputil``.
214
215- ``InaccessibleLoginDirError``
216
217  This exception is raised if the directory in which "you" are placed
218  upon login is not accessible, i. e. a ``chdir`` call with the
219  directory as argument would fail.
220
221- ``ParserError``
222
223  is used for errors during the parsing of directory
224  listings from the server. This exception is used by the ``FTPHost``
225  methods ``stat``, ``lstat``, and ``listdir``.
226
227- ``RootDirError``
228
229  Because of the implementation of the ``lstat`` method it is not
230  possible to do a ``stat`` call on the root directory ``/``.
231  If you know *any* way to do it, please let me know. :-)
232
233  This problem does *not* affect stat calls on items *in* the root
234  directory.
235
236- ``TimeShiftError``
237
238  is used to denote errors which relate to setting the `time shift`_.
239
240
241Directory and file names
242------------------------
243
244.. note::
245
246   Keep in mind that this section only applies to directory and file
247   *names*, not file *contents*. Encoding and decoding for file
248   contents is handled by the ``encoding`` argument for
249   `FTPHost.open`_.
250
251First off: If your directory and file names (both as arguments and on
252the server) contain only ISO 8859-1 (latin-1) characters, you can use
253such names in the form of ``bytes`` or ``str`` objects. However, you
254can't mix different string types (``bytes`` and ``str``) in one call
255(for example in ``FTPHost.path.join``).
256
257If you have directory or file names with characters that aren't in
258latin-1, it's recommended to use ``bytes`` objects. In that case,
259returned paths will be ``bytes`` objects, too.
260
261Read on for details.
262
263.. note::
264
265   The approach described below may look awkward and in a way it is.
266   The intention of ``ftputil`` is to behave like the local file
267   system APIs of Python 3 as far as it makes sense. Moreover, the
268   taken approach makes sure that directory and file names that were
269   used with Python 3's native ``ftplib`` module will be compatible
270   with ``ftputil`` and vice versa. Otherwise you may be able to use a
271   file name with ``ftputil``, but get an exception when trying to
272   read the same file with Python 3's ``ftplib`` module.
273
274Methods that take paths of directories and/or files can take either
275``bytes`` or ``str`` objects, or `PathLike`_ objects that can be
276converted to ``bytes`` or ``str``.
277
278.. _PathLike: https://docs.python.org/3/library/os.html#os.PathLike
279
280If a method gets a string argument (or a string argument wrapped in a
281PathLike_ object) and returns one or more strings, these strings will
282have the same string type (``bytes`` or ``str``) as the argument(s).
283Mixing different string types in one call (for example in
284``FTPHost.path.join``) isn't allowed and will cause a ``TypeError``.
285These rules are the same as for local file system operations in Python 3.
286
287``bytes`` objects for directory and file names will be sent to the
288server as-is. On the other hand, ``str`` objects will be encoded to
289``bytes`` objects, assuming latin-1 encoding. This implies that such
290``str`` objects must only contain code points 0-255 for the latin-1
291character set. Using any other characters will result in a
292``UnicodeEncodeError`` exception.
293
294If you have directory or file names as ``str`` objects with
295non-latin-1 characters, encode the strings to ``bytes`` yourself,
296using the encoding you know the server uses for its file system.
297Decode received paths with the same encoding. Encapsulate these
298conversions as far as you can. Otherwise, you'd have to adapt
299potentially a lot of code if the server encoding changes.
300
301If you *don't* know the encoding on the server side, it's probably the
302best to only use ``bytes`` for directory and file names. That said, as
303soon as you *show* the names to a user, you -- or the library you use
304for displaying the names -- has to guess an encoding.
305
306If you can decide about paths yourself, it's generally safest to use
307only ASCII characters in FTP paths.
308
309
310``FTPHost`` objects
311-------------------
312
313.. _`FTPHost construction`:
314
315Construction
316~~~~~~~~~~~~
317
318Introduction
319````````````
320
321``FTPHost`` instances can be created with the following call::
322
323    ftp_host = ftputil.FTPHost(server, user, password, account,
324                               session_factory=ftplib.FTP)
325
326The first four parameters are strings with the same meaning as for the
327FTP class in the ``ftplib`` module. Usually the ``account`` and
328``session_factory`` arguments aren't needed though.
329
330``FTPHost`` objects can also be used in a ``with`` statement::
331
332    import ftputil
333
334    with ftputil.FTPHost(server, user, password) as ftp_host:
335        print(ftp_host.listdir(ftp_host.curdir))
336
337After the ``with`` block, the ``FTPHost`` instance and the
338associated FTP sessions will be closed automatically.
339
340If something goes wrong during the ``FTPHost`` construction or in the
341body of the ``with`` statement, the instance is closed as well.
342Exceptions will be propagated (as with ``try ... finally``).
343
344Session factories
345`````````````````
346
347The keyword argument ``session_factory`` may be used to generate FTP
348connections with other factories than the default ``ftplib.FTP``. For
349example, the standard library of Python 3 contains a class
350``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
351connection.
352
353In fact, all positional and keyword arguments other than
354``session_factory`` are passed to the factory to generate a new
355background session. This also happens for every remote file that is
356opened; see below.
357
358This functionality of the constructor also allows to wrap
359``ftplib.FTP`` objects to do something that wouldn't be possible with
360the ``ftplib.FTP`` constructor alone.
361
362As an example, assume you want to connect to another than the default
363port, but ``ftplib.FTP`` only offers this by means of its ``connect``
364method, not via its constructor. One solution is to use a custom
365class as a session factory::
366
367    import ftplib
368    import ftputil
369
370    EXAMPLE_PORT = 50001
371
372    class MySession(ftplib.FTP):
373
374        def __init__(self, host, userid, password, port):
375            """Act like ftplib.FTP's constructor but connect to another port."""
376            ftplib.FTP.__init__(self)
377            self.connect(host, port)
378            self.login(userid, password)
379
380    # Try _not_ to use an _instance_ `MySession()` as factory, -
381    # use the class itself.
382    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
383                         session_factory=MySession) as ftp_host:
384        # Use `ftp_host` as usual.
385        ...
386
387On login, the format of the directory listings (needed for stat'ing
388files and directories) should be determined automatically. If not,
389please `enter a ticket`_.
390
391.. _`enter a ticket`: https://ftputil.sschwarzer.net/issuetrackernotes
392
393For the most common uses you don't need to create your own session
394factory class though. The ``ftputil.session`` module has a function
395``session_factory`` that can create session factories for a variety
396of parameters::
397
398    session_factory(base_class=ftplib.FTP,
399                    port=21,
400                    use_passive_mode=None,
401                    encrypt_data_channel=True,
402                    debug_level=None)
403
404with
405
406- ``base_class`` is a base class to inherit a new session factory
407  class from. By default, this is ``ftplib.FTP`` from the Python
408  standard library.
409
410- ``port`` is the command channel port. The default is 21, used in most
411  FTP server configurations.
412
413- ``use_passive_mode`` is either a boolean that determines whether
414  passive mode should be used or ``None``. ``None`` means to let the
415  base class choose active or passive mode.
416
417- ``encrypt_data_channel`` defines whether to encrypt the data channel
418  for secure connections. This is only supported for the base classes
419  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
420  parameter is ignored.
421
422- ``debug_level`` sets the debug level for FTP session instances. The
423  semantics is defined by the base class. For example, a debug level
424  of 2 causes the most verbose output for Python's ``ftplib.FTP``
425  class.
426
427All of these parameters can be combined. For example, you could use
428
429::
430
431    import ftplib
432
433    import ftputil
434    import ftputil.session
435
436
437    my_session_factory = ftputil.session.session_factory(
438                           base_class=ftpslib.FTP_TLS,
439                           port=31,
440                           encrypt_data_channel=True,
441                           debug_level=2)
442
443    with ftputil.FTPHost(server, user, password,
444                         session_factory=my_session_factory) as ftp_host:
445        ...
446
447to create and use a session factory derived from ``ftplib.FTP_TLS``
448that connects on command channel 31, will encrypt the data channel and
449print output for debug level 2.
450
451Note: Generally, you can achieve everything you can do with
452``ftputil.session.session_factory`` with an explicit session factory
453as described at the start of this section. However, the class
454``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
455it with ftputil out of the box. The function ``session_factory``
456contains a workaround for this limitation. For details refer to `this
457ticket`_.
458
459.. _`this ticket`: https://ftputil.sschwarzer.net/trac/ticket/78
460
461Hidden files and directories
462~~~~~~~~~~~~~~~~~~~~~~~~~~~~
463
464Whether ftputil sees "hidden" files and directories (usually files or
465directories whose names start with a dot) depends on the FTP server
466configuration. By default, ftputil does *not* use the ``-a`` option in
467the FTP ``LIST`` command to find hidden files.
468
469To tell the server to list hidden directories and files, set
470``FTPHost.use_list_a_option`` to ``True``::
471
472    ftp_host = ftputil.FTPHost(server, user, password, account,
473                               session_factory=ftplib.FTP)
474    ftp_host.use_list_a_option = True
475
476Caveats:
477
478- If the server doesn't understand the ``-a`` option at all, the
479  server may interpret ``-a`` as the name of a file or directory,
480  which can result in odd behavior. Therefore, use ``-a`` only if
481  you're sure the server you're talking to supports it. Another
482  approach is to have test code for ``-a`` support and fall back to
483  not using the option.
484
485- Even if the server knows about the ``-a`` option, the server may
486  be configured to ignore it.
487
488``FTPHost`` attributes and methods
489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
490
491Attributes
492``````````
493
494- ``curdir``, ``pardir``, ``sep``
495
496  are strings which denote the current and the parent directory on the
497  remote server. ``sep`` holds the path separator. Though `RFC 959`_
498  (File Transfer Protocol) notes that these values may depend on the
499  FTP server implementation, the Unix variants seem to work well in
500  practice, even for non-Unix servers.
501
502  Nevertheless, it's recommended that you don't hardcode these values
503  for remote paths, but use `FTPHost.path`_ as you would use
504  ``os.path`` to write platform-independent Python code for local
505  filesystems. Keep in mind that most, *but not all*, arguments of
506  ``FTPHost`` methods refer to remote directories or files. For
507  example, in `FTPHost.upload`_, the first argument is a local
508  path and the second a remote path. Both of these should use their
509  respective path separators.
510
511.. _`FTPHost.upload`: `Uploading and downloading files`_
512
513Remote file system navigation
514`````````````````````````````
515
516- ``getcwd()``
517
518  returns the absolute current directory on the remote host. This
519  method works like ``os.getcwd``.
520
521- ``chdir(directory)``
522
523  sets the current directory on the FTP server. This resembles
524  ``os.chdir``, as you may have expected.
525
526.. _`callback function`:
527
528Uploading and downloading files
529```````````````````````````````
530
531- ``upload(source, target, callback=None)``
532
533  copies a local source file (given by a filename, i. e. a string)
534  to the remote host under the name target. Both ``source`` and
535  ``target`` may be absolute paths or relative to their corresponding
536  current directory (on the local or the remote host, respectively).
537
538  The file content is always transferred in binary mode.
539
540  The callback, if given, will be invoked for each transferred chunk
541  of data::
542
543    callback(chunk)
544
545  where ``chunk`` is a bytestring. An example usage of a callback
546  method is to display a progress indicator.
547
548- ``download(source, target, callback=None)``
549
550  performs a download from the remote source file to a local target
551  file. Both ``source`` and ``target`` are strings. See the
552  description of ``upload`` for more details.
553
554.. _`upload_if_newer`:
555
556- ``upload_if_newer(source, target, callback=None)``
557
558  is similar to the ``upload`` method. The only difference is that the
559  upload is only invoked if the time of the last modification for the
560  source file is more recent than that of the target file or the
561  target doesn't exist at all. The check for the last modification
562  time considers the precision of the timestamps and transfers a file
563  "if in doubt". Consequently the code
564
565  ::
566
567    ftp_host.upload_if_newer("source_file", "target_file")
568    time.sleep(10)
569    ftp_host.upload_if_newer("source_file", "target_file")
570
571  might upload the file again if the timestamp of the target file is
572  precise up to a minute, which is typically the case because the
573  remote datetime is determined by parsing a directory listing from
574  the server. To avoid unnecessary transfers, wait at least a minute
575  between calls of ``upload_if_newer`` for the same file. If it still
576  seems that a file is uploaded unnecessarily (or not when it should),
577  read the subsection on `time shift`_ settings.
578
579  If an upload actually happened, the return value of
580  ``upload_if_newer`` is a ``True``, else ``False``.
581
582  Note that the method only checks the existence and/or the
583  modification time of the source and target file; it doesn't
584  compare any other file properties, say, the file size.
585
586  This also means that if a transfer is interrupted, the remote file
587  will have a newer modification time than the local file, and thus
588  the transfer won't be repeated if ``upload_if_newer`` is used a
589  second time. There are at least two possibilities after a failed
590  upload:
591
592  - use ``upload`` instead of ``upload_if_newer``, or
593
594  - remove the incomplete target file with ``FTPHost.remove``, then
595    use ``upload`` or ``upload_if_newer`` to transfer it again.
596
597.. _`download_if_newer`:
598
599- ``download_if_newer(source, target, callback=None)``
600
601  corresponds to ``upload_if_newer`` but performs a download from the
602  server to the local host. Read the descriptions of download and
603  ``upload_if_newer`` for more information. If a download actually
604  happened, the return value is ``True``, else ``False``.
605
606.. _`time shift`:
607.. _`time zone correction`:
608
609Time zone correction
610````````````````````
611
612If the client where ``ftputil`` runs and the server have different
613understandings of their local times, this has to be taken into account
614for ``upload_if_newer`` and ``download_if_newer`` to work correctly.
615
616Note that even if the client and the server are in the same time zone
617(or even on the same computer), the time shift value (see below) may
618be different from zero. For example, my computer is set to use local
619time whereas the server running on the very same host uses UTC time.
620
621.. _`set_time_shift`:
622
623- ``set_time_shift(time_shift)``
624
625  sets the so-called time shift value, measured in seconds. The time
626  shift is the difference between the local time of the server and the
627  local time of the client at a given moment, i. e. by definition
628
629  ::
630
631    time_shift = server_time - client_time
632
633  Setting this value is important for `upload_if_newer`_ and
634  `download_if_newer`_ to work correctly even if the time zone of the
635  FTP server differs from that of the client. Note that the time shift
636  value *can be negative*.
637
638  If the time shift value is invalid, for example its absolute value
639  is larger than 24 hours, a ``TimeShiftError`` is raised.
640
641  See also `synchronize_times`_ for a way to set the time shift with a
642  simple method call.
643
644- ``time_shift()``
645
646  returns the currently-set time shift value. See ``set_time_shift``
647  above for its definition.
648
649.. _`synchronize_times`:
650
651- ``synchronize_times()``
652
653  synchronizes the local times of the server and the client, so that
654  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
655  if the client and the server use different time zones. For this
656  to work, *all* of the following conditions must be true:
657
658  - The connection between server and client is established.
659
660  - The client has write access to the directory that is current when
661    ``synchronize_times`` is called.
662
663  If you can't fulfill these conditions, you can nevertheless set the
664  time shift value explicitly with `set_time_shift`_. Trying to call
665  ``synchronize_times`` if the above conditions aren't met results in
666  a ``TimeShiftError`` exception.
667
668Creating and removing directories
669`````````````````````````````````
670
671- ``mkdir(path, [mode])``
672
673  makes the given directory on the remote host. This does *not*
674  construct "intermediate" directories that don't already exist. The
675  ``mode`` parameter is ignored; this is for compatibility with
676  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
677  instead of the ``os`` module. See the explanation in the subsection
678  `Exception hierarchy`_.
679
680- ``makedirs(path, [mode])``
681
682  works similar to ``mkdir`` (see above), but also makes intermediate
683  directories like ``os.makedirs``. The ``mode`` parameter is only
684  there for compatibility with ``os.makedirs`` and is ignored.
685
686- ``rmdir(path)``
687
688  removes the given remote directory. If it's not empty, raise
689  a ``PermanentError``.
690
691- ``rmtree(path, ignore_errors=False, onerror=None)``
692
693  removes the given remote, possibly non-empty, directory tree.
694  The interface of this method is rather complex, in favor of
695  compatibility with ``shutil.rmtree``.
696
697  If ``ignore_errors`` is set to a true value, errors are ignored.
698  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
699  set, all exceptions occurring during the tree iteration and
700  processing are raised. These exceptions are all of type
701  ``PermanentError``.
702
703  To distinguish between different kinds of errors, pass in a callable
704  for ``onerror``. This callable must accept three arguments:
705  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
706  object, *for example* ``your_host_object.listdir``. ``path`` is the
707  path that was the recent argument of the respective method
708  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
709  info as it is gotten from ``sys.exc_info``.
710
711  The code of ``rmtree`` is taken from Python's ``shutil`` module
712  and adapted for ``ftputil``.
713
714Removing files and links
715````````````````````````
716
717- ``remove(path)``
718
719  removes a file or link on the remote host, similar to ``os.remove``.
720
721- ``unlink(path)``
722
723  is an alias for ``remove``.
724
725Retrieving information about directories, files and links
726`````````````````````````````````````````````````````````
727
728- ``listdir(path)``
729
730  returns a list containing the names of the files and directories
731  in the given path, similar to `os.listdir`_. The special names
732  ``.`` and ``..`` are not in the list.
733
734The methods ``lstat`` and ``stat`` (and some others) rely on the
735directory listing format used by the FTP server. When connecting to a
736host, ``FTPHost``'s constructor tries to guess the right format, which
737succeeds in most cases. However, if you get strange results or
738``ParserError`` exceptions by a mere ``lstat`` call, please `enter a
739ticket`_.
740
741If ``lstat`` or ``stat`` give wrong modification dates or times, look
742at the methods that deal with time zone differences (`time zone
743correction`_).
744
745.. _`FTPHost.lstat`:
746
747- ``lstat(path)``
748
749  returns an object similar to that from `os.lstat`_. This is a kind
750  of tuple with additional attributes; see the documentation of the
751  ``os`` module for details.
752
753  The result is derived by parsing the output of a ``LIST`` command on
754  the server. Therefore, the result from ``FTPHost.lstat`` can not
755  contain more information than the received text. In particular:
756
757  - User and group ids can only be determined as strings, not as
758    numbers, and that only if the server supplies them. This is
759    usually the case with Unix servers but maybe not for other FTP
760    servers.
761
762  - Values for the time of the last modification may be rough,
763    depending on the information from the server. For timestamps
764    older than a year, this usually means that the precision of the
765    modification timestamp value is not better than a day. For newer
766    files, the information may be accurate to a minute.
767
768    If the time of the last modification is before the epoch (usually
769    1970-01-01 UTC), set the time of the last modification to 0.0.
770
771  - Links can only be recognized on servers that provide this
772    information in the ``LIST`` output.
773
774  - Stat attributes that can't be determined at all are set to
775        ``None``. For example, a line of a directory listing may not
776        contain the date/time of a directory's last modification.
777
778  - There's a special problem with stat'ing the root directory.
779    (Stat'ing things *in* the root directory is fine though.) In
780    this case, a ``RootDirError`` is raised. This has to do with the
781    algorithm used by ``(l)stat``, and I know of no approach which
782    mends this problem.
783
784  Currently, ``ftputil`` recognizes the common Unix-style and
785  Microsoft/DOS-style directory formats. If you need to parse output
786  from another server type, please write to the `ftputil mailing
787  list`_. You may consider `writing your own parser`_.
788
789.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
790.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
791.. _`ftputil mailing list`: https://ftputil.sschwarzer.net/mailinglist
792.. _`writing your own parser`: `Writing directory parsers`_
793
794.. _`FTPHost.stat`:
795
796- ``stat(path)``
797
798  returns ``stat`` information also for files which are pointed to by a
799  link. This method follows multiple links until a regular file or
800  directory is found. If an infinite link chain is encountered or the
801  target of the last link in the chain doesn't exist, a
802  ``PermanentError`` is raised.
803
804  The limitations of the ``lstat`` method also apply to ``stat``.
805
806.. _`FTPHost.path`:
807
808``FTPHost`` objects contain an attribute named ``path``, similar to
809`os.path`_. The following methods can be applied to the remote host
810with the same semantics as for ``os.path``:
811
812::
813
814    abspath(path)
815    basename(path)
816    commonprefix(path_list)
817    dirname(path)
818    exists(path)
819    getmtime(path)
820    getsize(path)
821    isabs(path)
822    isdir(path)
823    isfile(path)
824    islink(path)
825    join(path1, path2, ...)
826    normcase(path)
827    normpath(path)
828    split(path)
829    splitdrive(path)
830    splitext(path)
831    walk(path, func, arg)
832
833Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
834methods return ``False`` if they can't find the path given by their
835argument.
836
837Local caching of file system information
838````````````````````````````````````````
839
840Many of the above methods need access to the remote file system to
841obtain data on directories and files. To get the most recent data,
842*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
843would require to fetch a directory listing from the server, which can
844make the program *very* slow. This effect is more pronounced for
845operations which mostly scan the file system rather than transferring
846file data.
847
848For this reason, ``ftputil`` by default saves the results from
849directory listings locally and reuses those results. This reduces
850network accesses and so speeds up the software a lot. However, since
851data is more rarely fetched from the server, the risk of obsolete data
852also increases. This will be discussed below.
853
854Caching can be controlled -- if necessary at all -- via the
855``stat_cache`` object in an ``FTPHost``'s namespace. For example,
856after calling
857
858::
859
860    ftp_host = ftputil.FTPHost(host, user, password)
861
862the cache can be accessed as ``ftp_host.stat_cache``.
863
864While ``ftputil`` usually manages the cache quite well, there are two
865possible reasons for modifying cache parameters.
866
867The first is when the number of possible entries is too low. You may
868notice that when you are processing very large directories and the
869program becomes much slower than before. It's common for code to read
870a directory with ``listdir`` and then process the found directories
871and files. This can also happen implicitly by a call to
872``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
873increases the cache size if directories with more entries than the
874current maximum cache size are to be scanned. Most of the time, this
875works fine.
876
877However, if you need access to stat data for several directories at
878the same time, you may need to increase the cache explicitly. This is
879done by the ``resize`` method::
880
881    ftp_host.stat_cache.resize(20000)
882
883where the argument is the maximum number of ``lstat`` results to store
884(the default is 5000, in versions before 2.6 it was 1000). Note that
885each path on the server, e. g. "/home/schwa/some_dir", corresponds to
886a single cache entry. Methods like ``exists`` or ``getmtime`` all
887derive their results from a previously fetched ``lstat`` result.
888
889The value 5000 above means that the cache will hold *at most* 5000
890entries (unless increased automatically by an explicit or implicit
891``listdir`` call, see above). If more are about to be stored, the
892entries which haven't been used for the longest time will be deleted
893to make place for newer entries.
894
895The second possible reason to change the cache parameters is to avoid
896stale cache data. Caching is so effective because it reduces network
897accesses. This can also be a disadvantage if the file system data on
898the remote server changes after a stat result has been retrieved; the
899client, when looking at the cached stat data, will use obsolete
900information.
901
902There are two potential ways to get such out-of-date stat data. The
903first happens when an ``FTPHost`` instance modifies a file path for
904which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
905Such changes are handled transparently; the path will be deleted from
906the cache. A different matter are changes unknown to the ``FTPHost``
907object which inspects its cache. Obviously, for example, these are
908changes by programs running on the remote host. On the other hand,
909cache inconsistencies can also occur if two ``FTPHost`` objects change
910a file system simultaneously::
911
912    with (
913      ftputil.FTPHost(server, user1, password1) as ftp_host1,
914      ftputil.FTPHost(server, user1, password1) as ftp_host2
915    ):
916        stat_result1 = ftp_host1.stat("some_file")
917        stat_result2 = ftp_host2.stat("some_file")
918        ftp_host2.remove("some_file")
919        # `ftp_host1` will still see the obsolete cache entry!
920        print(ftp_host1.stat("some_file"))
921        # Will raise an exception since an `FTPHost` object
922        # knows of its own changes.
923        print(ftp_host2.stat("some_file"))
924
925At first sight, it may appear to be a good idea to have a shared cache
926among several ``FTPHost`` objects. After some thinking, this turns out
927to be very error-prone. For example, it won't help with different
928processes using ``ftputil``. So, if you have to deal with concurrent
929write/read accesses to a server, you have to handle them explicitly.
930
931The most useful tool for this is the ``invalidate`` method. In the
932example above, it could be used like this::
933
934    with (
935      ftputil.FTPHost(server, user1, password1) as ftp_host1,
936      ftputil.FTPHost(server, user1, password1) as ftp_host2
937    ):
938        stat_result1 = ftp_host1.stat("some_file")
939        stat_result2 = ftp_host2.stat("some_file")
940        ftp_host2.remove("some_file")
941        # Invalidate using an absolute path.
942        absolute_path = ftp_host1.path.abspath(
943                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
944        ftp_host1.stat_cache.invalidate(absolute_path)
945        # Will now raise an exception as it should.
946        print(ftp_host1.stat("some_file"))
947        # Would raise an exception since an `FTPHost` object
948        # knows of its own changes, even without `invalidate`.
949        print(ftp_host2.stat("some_file"))
950
951The method ``invalidate`` can be used on any *absolute* path, be it a
952directory, a file or a link.
953
954By default, the cache entries (if not replaced by newer ones) are
955stored for an infinite time. That is, if you start your Python process
956using ``ftputil`` and let it run for three days a stat call may still
957access cache data that old. To avoid this, you can set the ``max_age``
958attribute::
959
960    with ftputil.FTPHost(server, user, password) as ftp_host:
961        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
962
963This sets the maximum age of entries in the cache to an hour. This
964means any entry older won't be retrieved from the cache but its data
965instead fetched again from the remote host and then again stored for
966up to an hour. To reset `max_age` to the default of unlimited age,
967i. e. cache entries never expire, use ``None`` as value.
968
969If you are certain that the cache will be in the way, you can disable
970and later re-enable it completely with ``disable`` and ``enable``::
971
972    with ftputil.FTPHost(server, user, password) as ftp_host:
973        ftp_host.stat_cache.disable()
974        ...
975        ftp_host.stat_cache.enable()
976
977During that time, the cache won't be used; all data will be fetched
978from the network. After enabling the cache again, its entries will be
979the same as when the cache was disabled, that is, entries won't get
980updated with newer data during this period. Note that even when the
981cache is disabled, the file system data in the code can become
982inconsistent::
983
984    with ftputil.FTPHost(server, user, password) as ftp_host:
985        ftp_host.stat_cache.disable()
986        if ftp_host.path.exists("some_file"):
987            mtime = ftp_host.path.getmtime("some_file")
988
989In that case, the file ``some_file`` may have been removed by another
990process between the calls to ``exists`` and ``getmtime``!
991
992Iteration over directories
993``````````````````````````
994
995.. _`FTPHost.walk`:
996
997- ``walk(top, topdown=True, onerror=None, followlinks=False)``
998
999  iterates over a directory tree, similar to `os.walk`_. Actually,
1000  ``FTPHost.walk`` uses the code from Python with just the necessary
1001  modifications, so see the linked documentation.
1002
1003.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
1004
1005.. _`FTPHost.path.walk`:
1006
1007- ``path.walk(path, func, arg)``
1008
1009  Similar to ``os.path.walk``, the ``walk`` method in
1010  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
1011  easier to use.
1012
1013Other methods
1014`````````````
1015
1016- ``close()``
1017
1018  closes the connection to the remote host. After this, no more
1019  interaction with the FTP server is possible with this ``FTPHost``
1020  object. Usually you don't need to close an ``FTPHost`` instance
1021  with ``close`` if you set up the instance in a ``with`` statement.
1022
1023- ``rename(source, target)``
1024
1025  renames the source file (or directory) on the FTP server.
1026
1027.. _`FTPHost.chmod`:
1028
1029- ``chmod(path, mode)``
1030
1031  sets the access mode (permission flags) for the given path. The mode
1032  is an integer as returned for the mode by the ``stat`` and ``lstat``
1033  methods. Be careful: Usually, mode values are written as octal
1034  numbers, for example 0755 to make a directory readable and writable
1035  for the owner, but not writable for the group and others. If you
1036  want to use such octal values, rely on Python's support for them::
1037
1038    ftp_host.chmod("some_directory", 0o755)
1039
1040  Not all FTP servers support the ``chmod`` command. In case of
1041  an exception, how do you know if the path doesn't exist or if
1042  the command itself is invalid? If the FTP server complies with
1043  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
1044  command isn't allowed. ``ftputil`` maps this special error
1045  response to a ``CommandNotImplementedError`` which is derived from
1046  ``PermanentError``.
1047
1048  So you need to code like this::
1049
1050    with ftputil.FTPHost(server, user, password) as ftp_host:
1051        try:
1052            ftp_host.chmod("some_file", 0o644)
1053        except ftputil.error.CommandNotImplementedError:
1054            # `chmod` not supported
1055            ...
1056        except ftputil.error.PermanentError:
1057            # Possibly a non-existent file
1058            ...
1059
1060  Because the ``CommandNotImplementedError`` is more specific, you
1061  have to test for it first.
1062
1063.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
1064
1065- ``copyfileobj(source, target, length=64*1024)``
1066
1067  copies the contents from the file-like object ``source`` to the
1068  file-like object ``target``. The only difference to
1069  ``shutil.copyfileobj`` is the default buffer size. Note that
1070  arbitrary file-like objects can be used as arguments (e. g. local
1071  files, remote FTP files).
1072
1073  However, the interfaces of ``source`` and ``target`` have to match;
1074  the string type read from ``source`` must be an accepted string type
1075  when written to ``target``. For example, if you open ``source`` in
1076  Python 3 as a local text file and ``target`` as a remote file object
1077  in binary mode, the transfer will fail since ``source.read`` gives
1078  unicode strings (``str``) whereas ``target.write`` only accepts byte
1079  strings (``bytes``).
1080
1081  See `File-like objects`_ for the construction and use of remote
1082  file-like objects.
1083
1084.. _`set_parser`:
1085
1086- ``set_parser(parser)``
1087
1088  sets a custom parser for FTP directories. Note that you have to pass
1089  in a parser *instance*, not the class.
1090
1091  An `extra section`_ shows how to write own parsers if the default
1092  parsers in ``ftputil`` don't work for you.
1093
1094.. _`extra section`: `Writing directory parsers`_
1095
1096.. _`keep_alive`:
1097
1098- ``keep_alive()``
1099
1100  attempts to keep the connection to the remote server active in order
1101  to prevent timeouts from happening. This method is primarily
1102  intended to keep the underlying FTP connection of an ``FTPHost``
1103  object alive while a file is uploaded or downloaded. This will
1104  require either an extra thread while the upload or download is in
1105  progress or calling ``keep_alive`` from a `callback function`_.
1106
1107  The ``keep_alive`` method won't help if the connection has already
1108  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
1109
1110  If you want to use this method, keep in mind that FTP servers define
1111  a timeout for a reason. A timeout prevents running out of server
1112  connections because of clients that never disconnect on their own.
1113
1114  Note that the ``keep_alive`` method does *not* affect the "hidden"
1115  FTP child connections established by ``FTPHost.open`` (see section
1116  `FTPHost instances vs. FTP connections`_ for details). You *can't*
1117  use ``keep_alive`` to avoid a timeout in a stalling transfer like
1118  this::
1119
1120      with ftputil.FTPHost(server, userid, password) as ftp_host:
1121          with ftp_host.open("some_remote_file", "rb") as fobj:
1122              data = fobj.read(100)
1123              # _Futile_ attempt to avoid file connection timeout.
1124              for i in range(15):
1125                  time.sleep(60)
1126                  ftp_host.keep_alive()
1127              # Will raise an `ftputil.error.TemporaryError`.
1128              data += fobj.read()
1129
1130
1131.. _`FTPHost.open`:
1132
1133File-like objects
1134-----------------
1135
1136Construction
1137~~~~~~~~~~~~
1138
1139Basics
1140``````
1141
1142``FTPFile`` objects are returned by a call to ``FTPHost.open``;
1143never use the ``FTPFile`` constructor directly.
1144
1145The APIs for remote file-like objects is modeled after the APIs of
1146the built-in ``open`` function and its return value.
1147
1148- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
1149  errors=None, newline=None, rest=None)``
1150
1151  returns a file-like object that refers to the path on the remote
1152  host. This path may be absolute or relative to the current directory
1153  on the remote host (this directory can be determined with the
1154  ``getcwd`` method). As with local file objects, the default mode is
1155  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
1156  "wb".
1157
1158  If a file is opened in binary mode, you *must not* specify an
1159  encoding. On the other hand, if you open a file in text mode, an
1160  encoding is used. By default, this is the return value of
1161  ``locale.getpreferredencoding``, but you can (and probably should)
1162  specify a distinct encoding.
1163
1164  If you open a file in binary mode, the read and write operations use
1165  ``bytes`` objects. That is, read operations return ``bytes`` and
1166  write operations only accept ``bytes``.
1167
1168  Similarly, text files always work with strings (``str``). Here, read
1169  operations return string and write operations only accept strings.
1170
1171  The arguments ``buffering``, ``errors`` and ``newline`` have the
1172  same semantics as in open_.
1173
1174  If the file is opened in binary mode, you may pass 0 or a positive
1175  integer for the ``rest`` argument. The argument is passed to the
1176  underlying FTP session instance (for example an instance of
1177  ``ftplib.FTP``) to start reading or writing at the given byte
1178  offset. For example, if a remote file contains the letters
1179  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".
1180
1181  .. warning::
1182
1183     If you pass ``rest`` values which point *after* the file, the
1184     behavior is undefined and may even differ from one FTP server to
1185     another. Therefore, use the ``rest`` argument only for error
1186     recovery in case of interrupted transfers. You need to keep track
1187     of the transferred data so that you can provide a valid ``rest``
1188     argument for a resumed transfer.
1189
1190.. _`open`: https://docs.python.org/3/library/functions.html#open
1191
1192``FTPHost.open`` can also be used in a ``with`` statement::
1193
1194    import ftputil
1195
1196    with ftputil.FTPHost(...) as ftp_host:
1197        ...
1198        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
1199            fobj.write("This is some text.")
1200
1201At the end of the ``with`` block, the remote file will be closed
1202automatically.
1203
1204If something goes wrong during the construction of the file or in the
1205body of the ``with`` statement, the file will be closed as well.
1206Exceptions will be propagated as with ``try ... finally``.
1207
1208Attributes and methods
1209~~~~~~~~~~~~~~~~~~~~~~
1210
1211The methods
1212
1213::
1214
1215    close()
1216    read([count])
1217    readline([count])
1218    readlines()
1219    write(data)
1220    writelines(string_sequence)
1221
1222and the attribute ``closed`` have the same semantics as for file
1223objects of a local disk file system. The iterator protocol is
1224supported as well, i. e. you can use a loop to read a file line by
1225line::
1226
1227    with ftputil.FTPHost(server, user, password) as ftp_host:
1228        with ftp_host.open("some_file") as input_file:
1229            for line in input_file:
1230                # Do something with the line, e. g.
1231                print(line.strip().replace("ftplib", "ftputil"))
1232
1233For more on file objects, see the section `File objects`_ in the
1234Python Library Reference.
1235
1236.. _`file objects`: https://docs.python.org/3/glossary.html#term-file-object
1237
1238
1239.. _`child_connections`:
1240
1241``FTPHost`` instances vs. FTP connections
1242-----------------------------------------
1243
1244This section explains why keeping an ``FTPHost`` instance "alive"
1245without timing out sometimes isn't trivial. If you always finish your
1246FTP operations in time, you don't need to read this section.
1247
1248The file transfer protocol is a stateful protocol. That means an FTP
1249connection always is in a certain state. Each of these states can only
1250change to certain other states under certain conditions triggered by
1251the client or the server.
1252
1253One of the consequences is that a single FTP connection can't be used
1254at the same time, say, to transfer data on the FTP data channel and to
1255create a directory on the remote host.
1256
1257For example, consider this::
1258
1259    >>> import ftplib
1260    >>> ftp = ftplib.FTP(server, user, password)
1261    >>> ftp.pwd()
1262    '/'
1263    >>> # Start transfer. `CONTENTS` is a text file on the server.
1264    >>> socket = ftp.transfercmd("RETR CONTENTS")
1265    >>> socket
1266    <socket._socketobject object at 0x7f801a6386e0>
1267    >>> ftp.pwd()
1268    Traceback (most recent call last):
1269      File "<stdin>", line 1, in <module>
1270      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
1271        return parse257(resp)
1272      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
1273        raise error_reply, resp
1274    ftplib.error_reply: 226-File successfully transferred
1275    226 0.000 seconds (measured here), 5.60 Mbytes per second
1276    >>>
1277
1278Note that ``ftp`` is a single FTP connection, represented by an
1279``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
1280
1281On the other hand, consider this::
1282
1283    >>> import ftputil
1284    >>> ftp_host = ftputil.FTPHost(server, user, password)
1285    >>> ftp_host.getcwd()
1286    >>> fobj = ftp_host.open("CONTENTS")
1287    >>> fobj
1288    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
1289    >>> ftp_host.getcwd()
1290    u'/'
1291    >>> fobj.readline()
1292    u'Contents of FTP test directory\n'
1293    >>> fobj.close()
1294    >>>
1295
1296To be able to start a file transfer (i. e. open a remote file for
1297reading or writing) and still be able to use other FTP commands,
1298ftputil uses a trick. For every remote file, ftputil creates a new FTP
1299connection, called a child connection in the ftputil source code.
1300(Actually, FTP connections belonging to closed remote files are
1301re-used if they haven't timed out yet.)
1302
1303In most cases this approach isn't noticeable by code using ftputil.
1304However, the nice abstraction of dealing with a single FTP connection
1305falls apart if one of the child connections times out. For example, if
1306you open a remote file and work only with the initial "main"
1307connection to navigate the file system, the FTP connection for the
1308remote file may eventually time out.
1309
1310While it's often relatively easy to prevent the "main" connection from
1311timing out it's unfortunately practically impossible to do this for a
1312remote file connection (apart from transferring some data, of course).
1313For this reason, `FTPHost.keep_alive`_ affects only the main
1314connection. Child connections may still time out if they're idle for
1315too long.
1316
1317.. _`FTPHost.keep_alive`: `keep_alive`_
1318
1319Some more details:
1320
1321- A kind of "straightforward" way of keeping the main connection alive
1322  would be to call ``ftp_host.getcwd()``. However, this doesn't work
1323  because ftputil caches the current directory and returns it without
1324  actually contacting the server. That's the main reason why there's
1325  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
1326  (i. e. the session object), which isn't a public attribute.
1327
1328- Some servers define not only an idle timeout but also a transfer
1329  timeout. This means the connection times out unless there's some
1330  transfer on the data channel for this connection. So ftputil's
1331  ``keep_alive`` doesn't prevent this timeout, but an
1332  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
1333  this transfers the data for the whole directory listing which might
1334  take some time if the directory has many entries.
1335
1336Bottom line: If you can, you should organize your FTP actions so that
1337you finish everything before a timeout happens.
1338
1339
1340Writing directory parsers
1341-------------------------
1342
1343``ftputil`` recognizes the two most widely-used FTP directory formats,
1344Unix and MS style, and adjusts itself automatically. Almost every FTP
1345server uses one of these formats.
1346
1347However, if your server uses a format which is different from the two
1348provided by ``ftputil``, you can plug in a custom parser with a single
1349method call and have ``ftputil`` use this parser.
1350
1351For this, you need to write a parser class by inheriting from the
1352class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
1353
1354    import ftputil.error
1355    import ftputil.stat
1356
1357    class XyzParser(ftputil.stat.Parser):
1358        """
1359        Parse the default format of the FTP server of the XYZ
1360        corporation.
1361        """
1362
1363        def parse_line(self, line, time_shift=0.0):
1364            """
1365            Parse a `line` from the directory listing and return a
1366            corresponding `StatResult` object. If the line can't
1367            be parsed, raise `ftputil.error.ParserError`.
1368
1369            The `time_shift` argument can be used to fine-tune the
1370            parsing of dates and times. See the class
1371            `ftputil.stat.UnixParser` for an example.
1372            """
1373            # Split the `line` argument and examine it further; if
1374            # something goes wrong, raise an `ftputil.error.ParserError`.
1375            ...
1376            # Make a `StatResult` object from the parts above.
1377            stat_result = ftputil.stat.StatResult(...)
1378            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
1379            stat_result._st_name = ...
1380            stat_result._st_target = ...
1381            stat_result._st_mtime_precision = ...
1382            return stat_result
1383
1384        # Define `ignores_line` only if the default in the base class
1385        # doesn't do enough!
1386        def ignores_line(self, line):
1387            """
1388            Return a true value if the line should be ignored. For
1389            example, the implementation in the base class handles
1390            lines like "total 17". On the other hand, if the line
1391            should be used for stat'ing, return a false value.
1392            """
1393            is_total_line = super().ignores_line(line)
1394            my_test = ...
1395            return is_total_line or my_test
1396
1397A ``StatResult`` object is similar to the value returned by
1398`os.stat`_ and is usually built with statements like
1399
1400::
1401
1402    stat_result = StatResult(
1403                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
1404                     st_gid, st_size, st_atime, st_mtime, st_ctime))
1405    stat_result._st_name = ...
1406    stat_result._st_target = ...
1407    stat_result._st_mtime_precision = ...
1408
1409with the arguments of the ``StatResult`` constructor described in
1410the following table.
1411
1412===== =================== ============ =================== =======================
1413Index Attribute           os.stat type ``StatResult`` type Notes
1414===== =================== ============ =================== =======================
14150     st_mode             int          int
14161     st_ino              long         long
14172     st_dev              long         long
14183     st_nlink            int          int
14194     st_uid              int          str                 usually only available as string
14205     st_gid              int          str                 usually only available as string
14216     st_size             long         long
14227     st_atime            int/float    float
14238     st_mtime            int/float    float
14249     st_ctime            int/float    float
1425\-    _st_name            \-           str                 file name without directory part
1426\-    _st_target          \-           str                 link target (may be absolute or relative)
1427\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
1428===== =================== ============ =================== =======================
1429
1430If you can't extract all the desirable data from a line (for
1431example, the MS format doesn't contain any information about the
1432owner of a file), set the corresponding values in the ``StatResult``
1433instance to ``None``.
1434
1435Parser classes can use several helper methods which are defined in
1436the class ``Parser``:
1437
1438- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
1439  an appropriate ``st_mode`` integer value.
1440
1441- ``parse_unix_time`` returns a float number usable for the
1442  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
1443  "May"/"26"/"2005". Note that the method expects the timestamp string
1444  already split at whitespace.
1445
1446- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
1447  returns a float number like from ``time.mktime``. Note that the
1448  method expects the timestamp string already split at whitespace.
1449
1450Additionally, there's an attribute ``_month_numbers`` which maps
1451lowercase three-letter month abbreviations to integers.
1452
1453For more details, see the two "standard" parsers ``UnixParser`` and
1454``MSParser`` in the module ``ftputil/stat.py``.
1455
1456To actually *use* the parser, call the method `set_parser`_ of the
1457``FTPHost`` instance.
1458
1459If you can't write a parser or don't want to, please ask on the
1460`ftputil mailing list`_. Possibly someone has already written a parser
1461for your server or can help with it.
1462
1463
1464FAQ / Tips and tricks
1465---------------------
1466
1467Where can I get the latest version?
1468~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1469
1470See the `download page`_. Announcements will be sent to the `mailing
1471list`_. Announcements on major updates will also be posted to the
1472`Python announcements list`_.
1473
1474.. _`download page`: https://ftputil.sschwarzer.net/download
1475.. _`mailing list`: https://ftputil.sschwarzer.net/mailinglist
1476.. _`Python announcements list`: https://mail.python.org/mailman3/lists/python-announce-list.python.org/
1477
1478Is there a mailing list on ``ftputil``?
1479~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1480
1481Yes, please visit https://ftputil.sschwarzer.net/mailinglist to
1482subscribe or read the archives.
1483
1484Though you can *technically* post without subscribing first I can't
1485recommend it: The mails from non-subscribers have to be approved by
1486me and because the arriving mails contain *lots* of spam, I rarely go
1487through these mails.
1488
1489I found a bug! What now?
1490~~~~~~~~~~~~~~~~~~~~~~~~
1491
1492Before reporting a bug, make sure that you already read this manual
1493and tried the `latest version`_ of ``ftputil``. There the bug might
1494have already been fixed.
1495
1496.. _`latest version`: https://ftputil.sschwarzer.net/download
1497
1498Please see https://ftputil.sschwarzer.net/issuetrackernotes for
1499guidelines on entering a bug in ``ftputil``'s ticket system. If you
1500are unsure if the behaviour you found is a bug or not, you should write
1501to the `ftputil mailing list`_. *Never* include confidential information
1502(user id, password, file names, etc.) in the problem report! Be
1503careful!
1504
1505Does ``ftputil`` support TLS?
1506~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1507
1508``ftputil`` has no *built-in* TLS support.
1509
1510On the other hand, there are two ways to get TLS support with
1511ftputil:
1512
1513- The ``ftplib`` library has a class ``FTP_TLS`` that you can use for
1514  the ``session_factory`` keyword argument in the ``FTPHost``
1515  constructor. You can't use the class directly though *if* you need
1516  additional setup code in comparison to ``ftplib.FTP``, for example
1517  calling ``prot_p``, to secure the data connection. On the other
1518  hand, `ftputil.session.session_factory`_ can be used to create a
1519  custom session factory.
1520
1521- If you have other requirements that ``session_factory`` can't
1522  fulfill, you may create your own session factory by inheriting from
1523  ``ftplib.FTP_TLS``::
1524
1525    import ftplib
1526
1527    import ftputil
1528
1529
1530    class FTPTLSSession(ftplib.FTP_TLS):
1531
1532        def __init__(self, host, user, password):
1533            ftplib.FTP_TLS.__init__(self)
1534            self.connect(host, port)
1535            self.login(user, password)
1536            # Set up encrypted data connection.
1537            self.prot_p()
1538            ...
1539
1540    # Note the `session_factory` parameter. Pass the class, not
1541    # an instance.
1542    with ftputil.FTPHost(server, user, password,
1543                         session_factory=FTPTLSSession) as ftp_host:
1544        # Use `ftp_host` as usual.
1545        ...
1546
1547.. _`ftputil.session.session_factory`: `Session factories`_
1548
1549
1550How do I connect to a non-default port?
1551~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1552
1553By default, an instantiated ``FTPHost`` object connects on the usual
1554FTP port. If you have to use a different port, refer to the section
1555`Session factories`_.
1556
1557How do I set active or passive mode?
1558~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1559
1560Please see the section `Session factories`_.
1561
1562How can I debug an FTP connection problem?
1563~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1564
1565You can do this with a session factory. See `Session factories`_.
1566
1567If you want to change the debug level only temporarily after the
1568connection is established, you can reach the `session object`_ as the
1569``_session`` attribute of the ``FTPHost`` instance and call
1570``_session.set_debuglevel``. Note that the ``_session`` attribute
1571should *only* be accessed for debugging. Calling arbitrary
1572``ftplib.FTP`` methods on the session object may *cause* bugs!
1573
1574.. _`session object`: `Session factories`_
1575
1576Conditional upload/download to/from a server in a different time zone
1577~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1578
1579You may find that ``ftputil`` uploads or downloads files
1580unnecessarily, or not when it should. This can happen when the FTP
1581server is in a different time zone than the client on which
1582``ftputil`` runs. Please see the section on `time zone correction`_.
1583It may even be sufficient to call `synchronize_times`_.
1584
1585When I use ``ftputil``, all I get is a ``ParserError`` exception
1586~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1587
1588The FTP server you connect to may use a directory format that
1589``ftputil`` doesn't understand. You can either write and
1590`plug in your own parser`_ or ask on the `mailing list`_ for
1591help.
1592
1593.. _`plug in your own parser`: `Writing directory parsers`_
1594
1595``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
1596~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1597
1598Like Python's counterparts under `os.path`_, ``ftputil``'s methods
1599return ``False`` if they can't find the given path.
1600
1601Probably you used ``listdir`` on a directory and called ``is...()`` on
1602the returned names. But if the argument for ``listdir`` wasn't the
1603current directory, the paths won't be found and so all ``is...()``
1604variants will return ``False``.
1605
1606I don't find an answer to my problem in this document
1607~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1608
1609Please send an email with your problem report or question to the
1610`ftputil mailing list`_, and we'll see what we can do for you. :-)
1611
1612
1613Bugs and limitations
1614--------------------
1615
1616- ``ftputil`` needs at least Python 3.6 to work.
1617
1618- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
1619  names starting with a dot) depends on the configuration of the FTP
1620  server. See `Hidden files and directories`_ for details.
1621
1622- Due to the implementation of ``lstat`` it can not return a sensible
1623  value for the root directory ``/`` though stat'ing entries *in* the
1624  root directory isn't a problem. If you know an implementation that
1625  can do this, please let me know. The root directory is handled
1626  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
1627
1628- In multithreaded programs, you can have each thread use one or more
1629  ``FTPHost`` instances as long as no instance is shared with other
1630  threads.
1631
1632- Currently, it is not possible to continue an interrupted upload or
1633  download. Contact me if this causes problems for you.
1634
1635- There's exactly one cache for ``lstat`` results for each ``FTPHost``
1636  object, i. e. there's no sharing of cache results determined by
1637  several ``FTPHost`` objects. See `Local caching of file system
1638  information`_ for the reasons.
1639
1640
1641Files
1642-----
1643
1644If not overwritten via installation options, the ``ftputil`` files
1645reside in the ``ftputil`` package. There's also documentation in
1646`reStructuredText`_ and in HTML format. The locations of these
1647files after installation is system-dependent.
1648
1649.. _`reStructuredText`: https://docutils.sourceforge.net/rst.html
1650
1651The files ``test_*.py`` and ``scripted_session.py`` are for
1652unit-testing. If you only *use* ``ftputil``, i. e. *don't* modify it,
1653you can delete these files.
1654
1655
1656References
1657----------
1658
1659- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
1660
1661- Python Software Foundation. 2020. `The Python Standard Library`_.
1662
1663.. _`RFC 959 - File Transfer Protocol (FTP)`: https://www.ietf.org/rfc/rfc959.txt
1664.. _`The Python Standard Library`: https://docs.python.org/library/index.html
1665
1666
1667Authors
1668-------
1669
1670``ftputil`` is written by Stefan Schwarzer
1671<sschwarzer@sschwarzer.net> and contributors (see
1672``doc/contributors.txt``).
1673
1674The original ``lrucache`` module was written by Evan Prodromou
1675<evan@prodromou.name>.
1676
1677Feedback is appreciated. :-)
Note: See TracBrowser for help on using the repository browser.