source: doc/ftputil.txt @ 1698:97f5bfb96c1a

Last change on this file since 1698:97f5bfb96c1a was 1698:97f5bfb96c1a, checked in by Stefan Schwarzer <sschwarzer@…>, 3 years ago
Patch version and release date
File size: 62.9 KB
Line 
1``ftputil`` -- a high-level FTP client library
2==============================================
3
4:Version:   3.4
5:Date:      2017-11-08
6:Summary:   high-level FTP client library for Python
7:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
8:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
9
10.. contents::
11
12
13Introduction
14------------
15
16The ``ftputil`` module is a high-level interface to the ftplib_
17module. The `FTPHost objects`_ generated from it allow many operations
18similar to those of os_, `os.path`_ and `shutil`_.
19
20.. _ftplib: https://docs.python.org/library/ftplib.html
21.. _os: https://docs.python.org/library/os.html
22.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
23.. _`os.path`: https://docs.python.org/library/os.path.html
24.. _`shutil`: https://docs.python.org/library/shutil.html
25
26Example::
27
28    import ftputil
29
30    # Download some files from the login directory.
31    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
32        names = ftp_host.listdir(ftp_host.curdir)
33        for name in names:
34            if ftp_host.path.isfile(name):
35                ftp_host.download(name, name)  # remote, local
36        # Make a new directory and copy a remote file into it.
37        ftp_host.mkdir("newdir")
38        with ftp_host.open("index.html", "rb") as source:
39            with ftp_host.open("newdir/index.html", "wb") as target:
40                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
41
42Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
43modification time of a file. The latter can also follow links, similar
44to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
45
46
47``ftputil`` features
48--------------------
49
50* Method names are familiar from Python's ``os``, ``os.path`` and
51  ``shutil`` modules. For example, use ``os.path.join`` to join
52  paths for a local file system and ``ftp_host.path.join`` to join
53  paths for a remote FTP file system.
54
55* Remote file system navigation (``getcwd``, ``chdir``)
56
57* Upload and download files (``upload``, ``upload_if_newer``,
58  ``download``, ``download_if_newer``)
59
60* Time zone synchronization between client and server (needed
61  for ``upload_if_newer`` and ``download_if_newer``)
62
63* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
64  ``rmtree``) and remove files (``remove``)
65
66* Get information about directories, files and links (``listdir``,
67  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
68  ``abspath``, ``split``, ``join``, ``dirname``, ``basename`` etc.)
69
70* Iterate over remote file systems (``walk``)
71
72* Local caching of results from ``lstat`` and ``stat`` calls to reduce
73  network access (also applies to ``exists``, ``getmtime`` etc.).
74
75* Read files from and write files to remote hosts via
76  file-like objects (``FTPHost.open``; the generated file-like objects
77  have the familiar methods like ``read``, ``readline``, ``readlines``,
78  ``write``, ``writelines`` and ``close``. You can also iterate over
79  these files line by line in a ``for`` loop.
80
81
82Exception hierarchy
83-------------------
84
85The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
86``ftputil.error.TemporaryError``.
87
88The exception classes are organized as follows::
89
90    FTPError
91        FTPOSError(FTPError, OSError)
92            PermanentError(FTPOSError)
93                CommandNotImplementedError(PermanentError)
94            TemporaryError(FTPOSError)
95        FTPIOError(FTPError)
96        InternalError(FTPError)
97            InaccessibleLoginDirError(InternalError)
98            ParserError(InternalError)
99            RootDirError(InternalError)
100            TimeShiftError(InternalError)
101
102and are described here:
103
104- ``FTPError``
105
106  is the root of the exception hierarchy of the module.
107
108- ``FTPOSError``
109
110  is derived from ``OSError``. This is for similarity between the
111  os module and ``FTPHost`` objects. Compare
112
113  ::
114
115    try:
116        os.chdir("nonexisting_directory")
117    except OSError:
118        ...
119
120  with
121
122  ::
123
124    host = ftputil.FTPHost("host", "user", "password")
125    try:
126        host.chdir("nonexisting_directory")
127    except OSError:
128        ...
129
130  Imagine a function
131
132  ::
133
134    def func(path, file):
135        ...
136
137  which works on the local file system and catches ``OSErrors``. If you
138  change the parameter list to
139
140  ::
141
142    def func(path, file, os=os):
143        ...
144
145  where ``os`` denotes the ``os`` module, you can call the function also as
146
147  ::
148
149    host = ftputil.FTPHost("host", "user", "password")
150    func(path, file, os=host)
151
152  to use the same code for both a local and remote file system.
153  Another similarity between ``OSError`` and ``FTPOSError`` is that
154  the latter holds the FTP server return code in the ``errno``
155  attribute of the exception object and the error text in
156  ``strerror``.
157
158- ``PermanentError``
159
160  is raised for 5xx return codes from the FTP server. This
161  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
162  ``ftplib.error_perm`` are *not* identical).
163
164- ``CommandNotImplementedError``
165
166  indicates that an underlying command the code tries to use is not
167  implemented. For an example, see the description of the
168  `FTPHost.chmod`_ method.
169
170- ``TemporaryError``
171
172  is raised for FTP return codes from the 4xx category. This
173  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
174  ``ftplib.error_temp`` are *not* identical).
175
176- ``FTPIOError``
177
178  denotes an I/O error on the remote host. This appears
179  mainly with file-like objects that are retrieved by calling
180  ``FTPHost.open``. Compare
181
182  ::
183
184    >>> try:
185    ...     f = open("not_there")
186    ... except IOError as obj:
187    ...     print obj.errno
188    ...     print obj.strerror
189    ...
190    2
191    No such file or directory
192
193  with
194
195  ::
196
197    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
198    >>> try:
199    ...     f = ftp_host.open("not_there")
200    ... except IOError as obj:
201    ...     print obj.errno
202    ...     print obj.strerror
203    ...
204    550
205    550 not_there: No such file or directory.
206
207  As you can see, both code snippets are similar. However, the error
208  codes aren't the same.
209
210- ``InternalError``
211
212  subsumes exception classes for signaling errors due to limitations
213  of the FTP protocol or the concrete implementation of ``ftputil``.
214
215- ``InaccessibleLoginDirError``
216
217  This exception is raised if the directory in which "you" are placed
218  upon login is not accessible, i. e. a ``chdir`` call with the
219  directory as argument would fail.
220
221- ``ParserError``
222
223  is used for errors during the parsing of directory
224  listings from the server. This exception is used by the ``FTPHost``
225  methods ``stat``, ``lstat``, and ``listdir``.
226
227- ``RootDirError``
228
229  Because of the implementation of the ``lstat`` method it is not
230  possible to do a ``stat`` call on the root directory ``/``.
231  If you know *any* way to do it, please let me know. :-)
232
233  This problem does *not* affect stat calls on items *in* the root
234  directory.
235
236- ``TimeShiftError``
237
238  is used to denote errors which relate to setting the `time shift`_.
239
240
241Directory and file names
242------------------------
243
244.. note::
245
246   Keep in mind that this section only applies to directory and file
247   *names*, not file *contents*. Encoding and decoding for file
248   contents is handled by the ``encoding`` argument for
249   `FTPHost.open`_.
250
251First off: If your directory and file names (both as
252arguments and on the server) contain only ISO 8859-1 (latin-1)
253characters, you can use such names in the form of byte strings or
254unicode strings. However, you can't mix different string types (bytes
255and unicode) in one call (for example in ``FTPHost.path.join``).
256
257If you have directory or file names with characters that aren't in
258latin-1, it's recommended to use byte strings. In that case,
259returned paths will be byte strings, too.
260
261Read on for details.
262
263.. note::
264
265   The approach described below may look awkward and in a way it is.
266   The intention of ``ftputil`` is to behave like the local file
267   system APIs of Python 3 as far as it makes sense. Moreover, the
268   taken approach makes sure that directory and file names that were
269   used with Python 3's native ``ftplib`` module will be compatible
270   with ``ftputil`` and vice versa. Otherwise you may be able to use a
271   file name with ``ftputil``, but get an exception when trying to
272   read the same file with Python 3's ``ftplib`` module.
273
274Methods that take names of directories and/or files can take either
275byte or unicode strings. If a method got a string argument and returns
276one or more strings, these strings will have the same string type as
277the argument(s). Mixing different string arguments in one call (for
278example in ``FTPHost.path.join``) isn't allowed and will cause a
279``TypeError``. These rules are the same as for local file system
280operations in Python 3. Since ``ftputil`` uses the same API for Python
2812, ``ftputil`` will do the same when run on Python 2.
282
283Byte strings for directory and file names will be sent to the server
284as-is. On the other hand, unicode strings will be encoded to byte
285strings, assuming latin-1 encoding. This implies that such unicode
286strings must only contain code points 0-255 for the latin-1 character
287set. Using any other characters will result in a
288``UnicodeEncodeError`` exception.
289
290If you have directory or file names as unicode strings with non-latin-1
291characters, encode the unicode strings to byte strings yourself, using
292the encoding you know the server uses. Decode received paths with the
293same encoding. Encapsulate these conversions as far as you can.
294Otherwise, you'd have to adapt potentially a lot of code if the server
295encoding changes.
296
297If you *don't* know the encoding on the server side,
298it's probably the best to only use byte strings for directory and file
299names. That said, as soon as you *show* the names to a user, you -- or
300the library you use for displaying the names -- has to guess an
301encoding.
302
303
304``FTPHost`` objects
305-------------------
306
307.. _`FTPHost construction`:
308
309Construction
310~~~~~~~~~~~~
311
312Introduction
313````````````
314
315``FTPHost`` instances can be created with the following call::
316
317    ftp_host = ftputil.FTPHost(server, user, password, account,
318                               session_factory=ftplib.FTP)
319
320The first four parameters are strings with the same meaning as for the
321FTP class in the ``ftplib`` module. Usually the ``account`` and
322``session_factory`` arguments aren't needed though.
323
324``FTPHost`` objects can also be used in a ``with`` statement::
325
326    import ftputil
327
328    with ftputil.FTPHost(server, user, password) as ftp_host:
329        print ftp_host.listdir(ftp_host.curdir)
330
331After the ``with`` block, the ``FTPHost`` instance and the
332associated FTP sessions will be closed automatically.
333
334If something goes wrong during the ``FTPHost`` construction or in the
335body of the ``with`` statement, the instance is closed as well.
336Exceptions will be propagated (as with ``try ... finally``).
337
338Session factories
339`````````````````
340
341The keyword argument ``session_factory`` may be used to generate FTP
342connections with other factories than the default ``ftplib.FTP``. For
343example, the standard library of Python 2.7 contains a class
344``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
345connection.
346
347In fact, all positional and keyword arguments other than
348``session_factory`` are passed to the factory to generate a new
349background session. This also happens for every remote file that is
350opened; see below.
351
352This functionality of the constructor also allows to wrap
353``ftplib.FTP`` objects to do something that wouldn't be possible with
354the ``ftplib.FTP`` constructor alone.
355
356As an example, assume you want to connect to another than the default
357port, but ``ftplib.FTP`` only offers this by means of its ``connect``
358method, not via its constructor. One solution is to use a custom
359class as a session factory::
360
361    import ftplib
362    import ftputil
363
364    EXAMPLE_PORT = 50001
365
366    class MySession(ftplib.FTP):
367
368        def __init__(self, host, userid, password, port):
369            """Act like ftplib.FTP's constructor but connect to another port."""
370            ftplib.FTP.__init__(self)
371            self.connect(host, port)
372            self.login(userid, password)
373
374    # Try _not_ to use an _instance_ `MySession()` as factory, -
375    # use the class itself.
376    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
377                         session_factory=MySession) as ftp_host:
378        # Use `ftp_host` as usual.
379        ...
380
381On login, the format of the directory listings (needed for stat'ing
382files and directories) should be determined automatically. If not,
383please `file a bug report`_.
384
385.. _`file a bug report`: http://ftputil.sschwarzer.net/issuetrackernotes
386
387For the most common uses you don't need to create your own session
388factory class though. The ``ftputil.session`` module has a function
389``session_factory`` that can create session factories for a variety
390of parameters::
391
392    session_factory(base_class=ftplib.FTP,
393                    port=21,
394                    use_passive_mode=None,
395                    encrypt_data_channel=True,
396                    debug_level=None)
397
398with
399
400- ``base_class`` is a base class to inherit a new session factory
401  class from. By default, this is ``ftplib.FTP`` from the Python
402  standard library.
403
404- ``port`` is the command channel port. The default is 21, used in most
405  FTP server configurations.
406
407- ``use_passive_mode`` is either a boolean that determines whether
408  passive mode should be used or ``None``. ``None`` means to let the
409  base class choose active or passive mode.
410
411- ``encrypt_data_channel`` defines whether to encrypt the data channel
412  for secure connections. This is only supported for the base classes
413  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
414  the parameter is ignored.
415
416- ``debug_level`` sets the debug level for FTP session instances. The
417  semantics is defined by the base class. For example, a debug level
418  of 2 causes the most verbose output for Python's ``ftplib.FTP``
419  class.
420
421All of these parameters can be combined. For example, you could use
422
423::
424
425    import ftplib
426
427    import ftputil
428    import ftputil.session
429
430
431    my_session_factory = ftputil.session.session_factory(
432                           base_class=ftpslib.FTP_TLS,
433                           port=31,
434                           encrypt_data_channel=True,
435                           debug_level=2)
436
437    with ftputil.FTPHost(server, user, password,
438                         session_factory=my_session_factory) as ftp_host:
439        ...
440
441to create and use a session factory derived from ``ftplib.FTP_TLS``
442that connects on command channel 31, will encrypt the data channel and
443print output for debug level 2.
444
445Note: Generally, you can achieve everything you can do with
446``ftputil.session.session_factory`` with an explicit session factory
447as described at the start of this section. However, the class
448``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
449it with ftputil out of the box. The function ``session_factory``
450contains a workaround for this limitation. For details refer to `this
451bug report`_.
452
453.. _`this bug report`: http://ftputil.sschwarzer.net/trac/ticket/78
454
455Hidden files and directories
456~~~~~~~~~~~~~~~~~~~~~~~~~~~~
457
458Whether ftputil sees "hidden" files and directories (usually files or
459directories whose names start with a dot) depends on the FTP server
460configuration. By default, ftputil uses the ``-a`` option in the FTP
461``LIST`` command to find hidden files. However, the server may ignore
462this.
463
464If using the ``-a`` option leads to problems, for example if an
465FTP server causes an exception, you may switch off the use of the
466option::
467
468    ftp_host = ftputil.FTPHost(server, user, password, account,
469                               session_factory=ftplib.FTP)
470    ftp_host.use_list_a_option = False
471
472``FTPHost`` attributes and methods
473~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
474
475Attributes
476``````````
477
478- ``curdir``, ``pardir``, ``sep``
479
480  are strings which denote the current and the parent directory on the
481  remote server. ``sep`` holds the path separator. Though `RFC 959`_
482  (File Transfer Protocol) notes that these values may depend on the
483  FTP server implementation, the Unix variants seem to work well in
484  practice, even for non-Unix servers.
485
486  Nevertheless, it's recommended that you don't hardcode these values
487  for remote paths, but use `FTPHost.path`_ as you would use
488  ``os.path`` to write platform-independent Python code for local
489  filesystems. Keep in mind that most, *but not all*, arguments of
490  ``FTPHost`` methods refer to remote directories or files. For
491  example, in `FTPHost.upload`_, the first argument is a local
492  path and the second a remote path. Both of these should use their
493  respective path separators.
494
495.. _`FTPHost.upload`: `Uploading and downloading files`_
496
497Remote file system navigation
498`````````````````````````````
499
500- ``getcwd()``
501
502  returns the absolute current directory on the remote host. This
503  method works like ``os.getcwd``.
504
505- ``chdir(directory)``
506
507  sets the current directory on the FTP server. This resembles
508  ``os.chdir``, as you may have expected.
509
510.. _`callback function`:
511
512Uploading and downloading files
513```````````````````````````````
514
515- ``upload(source, target, callback=None)``
516
517  copies a local source file (given by a filename, i. e. a string)
518  to the remote host under the name target. Both ``source`` and
519  ``target`` may be absolute paths or relative to their corresponding
520  current directory (on the local or the remote host, respectively).
521
522  The file content is always transferred in binary mode.
523
524  The callback, if given, will be invoked for each transferred chunk
525  of data::
526
527    callback(chunk)
528
529  where ``chunk`` is a bytestring. An example usage of a callback
530  method is to display a progress indicator.
531
532- ``download(source, target, callback=None)``
533
534  performs a download from the remote source file to a local target
535  file. Both ``source`` and ``target`` are strings. See the
536  description of ``upload`` for more details.
537
538.. _`upload_if_newer`:
539
540- ``upload_if_newer(source, target, callback=None)``
541
542  is similar to the ``upload`` method. The only difference is that the
543  upload is only invoked if the time of the last modification for the
544  source file is more recent than that of the target file or the
545  target doesn't exist at all. The check for the last modification
546  time considers the precision of the timestamps and transfers a file
547  "if in doubt". Consequently the code
548
549  ::
550
551    ftp_host.upload_if_newer("source_file", "target_file")
552    time.sleep(10)
553    ftp_host.upload_if_newer("source_file", "target_file")
554
555  might upload the file again if the timestamp of the target file is
556  precise up to a minute, which is typically the case because the
557  remote datetime is determined by parsing a directory listing from
558  the server. To avoid unnecessary transfers, wait at least a minute
559  between calls of ``upload_if_newer`` for the same file. If it still
560  seems that a file is uploaded unnecessarily (or not when it should),
561  read the subsection on `time shift`_ settings.
562
563  If an upload actually happened, the return value of
564  ``upload_if_newer`` is a ``True``, else ``False``.
565
566  Note that the method only checks the existence and/or the
567  modification time of the source and target file; it doesn't
568  compare any other file properties, say, the file size.
569
570  This also means that if a transfer is interrupted, the remote file
571  will have a newer modification time than the local file, and thus
572  the transfer won't be repeated if ``upload_if_newer`` is used a
573  second time. There are at least two possibilities after a failed
574  upload:
575
576  - use ``upload`` instead of ``upload_if_newer``, or
577
578  - remove the incomplete target file with ``FTPHost.remove``, then
579    use ``upload`` or ``upload_if_newer`` to transfer it again.
580
581.. _`download_if_newer`:
582
583- ``download_if_newer(source, target, callback=None)``
584
585  corresponds to ``upload_if_newer`` but performs a download from the
586  server to the local host. Read the descriptions of download and
587  ``upload_if_newer`` for more information. If a download actually
588  happened, the return value is ``True``, else ``False``.
589
590.. _`time shift`:
591.. _`time zone correction`:
592
593Time zone correction
594````````````````````
595
596If the client where ``ftputil`` runs and the server have a different
597understanding of their local times, this has to be taken into account
598for ``upload_if_newer`` and ``download_if_newer`` to work correctly.
599
600Note that even if the client and the server are in the same time zone
601(or even on the same computer), the time shift value (see below) may
602be different from zero. For example, my computer is set to use local
603time whereas the server running on the very same host insists on using
604UTC time.
605
606.. _`set_time_shift`:
607
608- ``set_time_shift(time_shift)``
609
610  sets the so-called time shift value, measured in seconds. The time
611  shift is the difference between the local time of the server and the
612  local time of the client at a given moment, i. e. by definition
613
614  ::
615
616    time_shift = server_time - client_time
617
618  Setting this value is important for `upload_if_newer`_ and
619  `download_if_newer`_ to work correctly even if the time zone of the
620  FTP server differs from that of the client. Note that the time shift
621  value *can be negative*.
622
623  If the time shift value is invalid, for example its absolute value
624  is larger than 24 hours, a ``TimeShiftError`` is raised.
625
626  See also `synchronize_times`_ for a way to set the time shift with a
627  simple method call.
628
629- ``time_shift()``
630
631  returns the currently-set time shift value. See ``set_time_shift``
632  above for its definition.
633
634.. _`synchronize_times`:
635
636- ``synchronize_times()``
637
638  synchronizes the local times of the server and the client, so that
639  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
640  if the client and the server use different time zones. For this
641  to work, *all* of the following conditions must be true:
642
643  - The connection between server and client is established.
644
645  - The client has write access to the directory that is current when
646    ``synchronize_times`` is called.
647
648  If you can't fulfill these conditions, you can nevertheless set the
649  time shift value explicitly with `set_time_shift`_. Trying to call
650  ``synchronize_times`` if the above conditions aren't met results in
651  a ``TimeShiftError`` exception.
652
653Creating and removing directories
654`````````````````````````````````
655
656- ``mkdir(path, [mode])``
657
658  makes the given directory on the remote host. This does *not*
659  construct "intermediate" directories that don't already exist. The
660  ``mode`` parameter is ignored; this is for compatibility with
661  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
662  instead of the ``os`` module. See the explanation in the subsection
663  `Exception hierarchy`_.
664
665- ``makedirs(path, [mode])``
666
667  works similar to ``mkdir`` (see above), but also makes intermediate
668  directories like ``os.makedirs``. The ``mode`` parameter is only
669  there for compatibility with ``os.makedirs`` and is ignored.
670
671- ``rmdir(path)``
672
673  removes the given remote directory. If it's not empty, raise
674  a ``PermanentError``.
675
676- ``rmtree(path, ignore_errors=False, onerror=None)``
677
678  removes the given remote, possibly non-empty, directory tree.
679  The interface of this method is rather complex, in favor of
680  compatibility with ``shutil.rmtree``.
681
682  If ``ignore_errors`` is set to a true value, errors are ignored.
683  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
684  set, all exceptions occurring during the tree iteration and
685  processing are raised. These exceptions are all of type
686  ``PermanentError``.
687
688  To distinguish between different kinds of errors, pass in a callable
689  for ``onerror``. This callable must accept three arguments:
690  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
691  object, *for example* ``your_host_object.listdir``. ``path`` is the
692  path that was the recent argument of the respective method
693  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
694  info as it is gotten from ``sys.exc_info``.
695
696  The code of ``rmtree`` is taken from Python's ``shutil`` module
697  and adapted for ``ftputil``.
698
699Removing files and links
700````````````````````````
701
702- ``remove(path)``
703
704  removes a file or link on the remote host, similar to ``os.remove``.
705
706- ``unlink(path)``
707
708  is an alias for ``remove``.
709
710Retrieving information about directories, files and links
711`````````````````````````````````````````````````````````
712
713- ``listdir(path)``
714
715  returns a list containing the names of the files and directories
716  in the given path, similar to `os.listdir`_. The special names
717  ``.`` and ``..`` are not in the list.
718
719The methods ``lstat`` and ``stat`` (and some others) rely on the
720directory listing format used by the FTP server. When connecting to a
721host, ``FTPHost``'s constructor tries to guess the right format, which
722succeeds in most cases. However, if you get strange results or
723``ParserError`` exceptions by a mere ``lstat`` call, please `file a
724bug report`_.
725
726If ``lstat`` or ``stat`` give wrong modification dates or times, look
727at the methods that deal with time zone differences (`time zone
728correction`_).
729
730.. _`FTPHost.lstat`:
731
732- ``lstat(path)``
733
734  returns an object similar to that from `os.lstat`_. This is a
735  "tuple" with additional attributes; see the documentation of the
736  ``os`` module for details.
737
738  The result is derived by parsing the output of a ``LIST`` command on
739  the server. Therefore, the result from ``FTPHost.lstat`` can not
740  contain more information than the received text. In particular:
741
742  - User and group ids can only be determined as strings, not as
743    numbers, and that only if the server supplies them. This is
744    usually the case with Unix servers but maybe not for other FTP
745    server programs.
746
747  - Values for the time of the last modification may be rough,
748    depending on the information from the server. For timestamps
749    older than a year, this usually means that the precision of the
750    modification timestamp value is not better than days. For newer
751    files, the information may be accurate to a minute.
752
753    If the time of the last modification is before the epoch (usually
754    1970-01-01 UTC), set the time of the last modification to 0.0.
755
756  - Links can only be recognized on servers that provide this
757    information in the ``LIST`` output.
758
759  - Stat attributes that can't be determined at all are set to
760        ``None``. For example, a line of a directory listing may not
761        contain the date/time of a directory's last modification.
762
763  - There's a special problem with stat'ing the root directory.
764    (Stat'ing things *in* the root directory is fine though.) In
765    this case, a ``RootDirError`` is raised. This has to do with the
766    algorithm used by ``(l)stat``, and I know of no approach which
767    mends this problem.
768
769  Currently, ``ftputil`` recognizes the common Unix-style and
770  Microsoft/DOS-style directory formats. If you need to parse output
771  from another server type, please write to the `ftputil mailing
772  list`_. You may consider `writing your own parser`_.
773
774.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
775.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
776.. _`ftputil mailing list`: http://ftputil.sschwarzer.net/mailinglist
777.. _`writing your own parser`: `Writing directory parsers`_
778
779.. _`FTPHost.stat`:
780
781- ``stat(path)``
782
783  returns ``stat`` information also for files which are pointed to by a
784  link. This method follows multiple links until a regular file or
785  directory is found. If an infinite link chain is encountered or the
786  target of the last link in the chain doesn't exist, a
787  ``PermanentError`` is raised.
788
789  The limitations of the ``lstat`` method also apply to ``stat``.
790
791.. _`FTPHost.path`:
792
793``FTPHost`` objects contain an attribute named ``path``, similar to
794`os.path`_. The following methods can be applied to the remote host
795with the same semantics as for ``os.path``:
796
797::
798
799    abspath(path)
800    basename(path)
801    commonprefix(path_list)
802    dirname(path)
803    exists(path)
804    getmtime(path)
805    getsize(path)
806    isabs(path)
807    isdir(path)
808    isfile(path)
809    islink(path)
810    join(path1, path2, ...)
811    normcase(path)
812    normpath(path)
813    split(path)
814    splitdrive(path)
815    splitext(path)
816    walk(path, func, arg)
817
818Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
819methods return ``False`` if they can't find the path given by their
820argument.
821
822Local caching of file system information
823````````````````````````````````````````
824
825Many of the above methods need access to the remote file system to
826obtain data on directories and files. To get the most recent data,
827*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
828would require to fetch a directory listing from the server, which can
829make the program *very* slow. This effect is more pronounced for
830operations which mostly scan the file system rather than transferring
831file data.
832
833For this reason, ``ftputil`` by default saves the results from
834directory listings locally and reuses those results. This reduces
835network accesses and so speeds up the software a lot. However, since
836data is more rarely fetched from the server, the risk of obsolete data
837also increases. This will be discussed below.
838
839Caching can be controlled -- if necessary at all -- via the
840``stat_cache`` object in an ``FTPHost``'s namespace. For example,
841after calling
842
843::
844
845    ftp_host = ftputil.FTPHost(host, user, password)
846
847the cache can be accessed as ``ftp_host.stat_cache``.
848
849While ``ftputil`` usually manages the cache quite well, there are two
850possible reasons for modifying cache parameters.
851
852The first is when the number of possible entries is too low. You may
853notice that when you are processing very large directories and the
854program becomes much slower than before. It's common for code to read
855a directory with ``listdir`` and then process the found directories
856and files. This can also happen implicitly by a call to
857``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
858increases the cache size if directories with more entries than the
859current maximum cache size are to be scanned. Most of the time, this
860works fine.
861
862However, if you need access to stat data for several directories at
863the same time, you may need to increase the cache explicitly. This is
864done by the ``resize`` method::
865
866    ftp_host.stat_cache.resize(20000)
867
868where the argument is the maximum number of ``lstat`` results to store
869(the default is 5000, in versions before 2.6 it was 1000). Note that
870each path on the server, e. g. "/home/schwa/some_dir", corresponds to
871a single cache entry. Methods like ``exists`` or ``getmtime`` all
872derive their results from a previously fetched ``lstat`` result.
873
874The value 5000 above means that the cache will hold *at most* 5000
875entries (unless increased automatically by an explicit or implicit
876``listdir`` call, see above). If more are about to be stored, the
877entries which haven't been used for the longest time will be deleted
878to make place for newer entries.
879
880The second possible reason to change the cache parameters is to avoid
881stale cache data. Caching is so effective because it reduces network
882accesses. This can also be a disadvantage if the file system data on
883the remote server changes after a stat result has been retrieved; the
884client, when looking at the cached stat data, will use obsolete
885information.
886
887There are two potential ways to get such out-of-date stat data. The
888first happens when an ``FTPHost`` instance modifies a file path for
889which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
890Such changes are handled transparently; the path will be deleted from
891the cache. A different matter are changes unknown to the ``FTPHost``
892object which inspects its cache. Obviously, for example, these are
893changes by programs running on the remote host. On the other hand,
894cache inconsistencies can also occur if two ``FTPHost`` objects change
895a file system simultaneously::
896
897    with (
898      ftputil.FTPHost(server, user1, password1) as ftp_host1,
899      ftputil.FTPHost(server, user1, password1) as ftp_host2
900    ):
901        stat_result1 = ftp_host1.stat("some_file")
902        stat_result2 = ftp_host2.stat("some_file")
903        ftp_host2.remove("some_file")
904        # `ftp_host1` will still see the obsolete cache entry!
905        print ftp_host1.stat("some_file")
906        # Will raise an exception since an `FTPHost` object
907        # knows of its own changes.
908        print ftp_host2.stat("some_file")
909
910At first sight, it may appear to be a good idea to have a shared cache
911among several ``FTPHost`` objects. After some thinking, this turns out
912to be very error-prone. For example, it won't help with different
913processes using ``ftputil``. So, if you have to deal with concurrent
914write/read accesses to a server, you have to handle them explicitly.
915
916The most useful tool for this is the ``invalidate`` method. In the
917example above, it could be used like this::
918
919    with (
920      ftputil.FTPHost(server, user1, password1) as ftp_host1,
921      ftputil.FTPHost(server, user1, password1) as ftp_host2
922    ):
923        stat_result1 = ftp_host1.stat("some_file")
924        stat_result2 = ftp_host2.stat("some_file")
925        ftp_host2.remove("some_file")
926        # Invalidate using an absolute path.
927        absolute_path = ftp_host1.path.abspath(
928                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
929        ftp_host1.stat_cache.invalidate(absolute_path)
930        # Will now raise an exception as it should.
931        print ftp_host1.stat("some_file")
932        # Would raise an exception since an `FTPHost` object
933        # knows of its own changes, even without `invalidate`.
934        print ftp_host2.stat("some_file")
935
936The method ``invalidate`` can be used on any *absolute* path, be it a
937directory, a file or a link.
938
939By default, the cache entries (if not replaced by newer ones) are
940stored for an infinite time. That is, if you start your Python process
941using ``ftputil`` and let it run for three days a stat call may still
942access cache data that old. To avoid this, you can set the ``max_age``
943attribute::
944
945    with ftputil.FTPHost(server, user, password) as ftp_host:
946        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
947
948This sets the maximum age of entries in the cache to an hour. This
949means any entry older won't be retrieved from the cache but its data
950instead fetched again from the remote host and then again stored for
951up to an hour. To reset `max_age` to the default of unlimited age,
952i. e. cache entries never expire, use ``None`` as value.
953
954If you are certain that the cache will be in the way, you can disable
955and later re-enable it completely with ``disable`` and ``enable``::
956
957    with ftputil.FTPHost(server, user, password) as ftp_host:
958        ftp_host.stat_cache.disable()
959        ...
960        ftp_host.stat_cache.enable()
961
962During that time, the cache won't be used; all data will be fetched
963from the network. After enabling the cache again, its entries will be
964the same as when the cache was disabled, that is, entries won't get
965updated with newer data during this period. Note that even when the
966cache is disabled, the file system data in the code can become
967inconsistent::
968
969    with ftputil.FTPHost(server, user, password) as ftp_host:
970        ftp_host.stat_cache.disable()
971        if ftp_host.path.exists("some_file"):
972            mtime = ftp_host.path.getmtime("some_file")
973
974In that case, the file ``some_file`` may have been removed by another
975process between the calls to ``exists`` and ``getmtime``!
976
977Iteration over directories
978``````````````````````````
979
980.. _`FTPHost.walk`:
981
982- ``walk(top, topdown=True, onerror=None, followlinks=False)``
983
984  iterates over a directory tree, similar to `os.walk`_. Actually,
985  ``FTPHost.walk`` uses the code from Python with just the necessary
986  modifications, so see the linked documentation.
987
988.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
989
990.. _`FTPHost.path.walk`:
991
992- ``path.walk(path, func, arg)``
993
994  Similar to ``os.path.walk``, the ``walk`` method in
995  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
996  easier to use.
997
998Other methods
999`````````````
1000
1001- ``close()``
1002
1003  closes the connection to the remote host. After this, no more
1004  interaction with the FTP server is possible with this ``FTPHost``
1005  object. Usually you don't need to close an ``FTPHost`` instance
1006  with ``close`` if you set up the instance in a ``with`` statement.
1007
1008- ``rename(source, target)``
1009
1010  renames the source file (or directory) on the FTP server.
1011
1012.. _`FTPHost.chmod`:
1013
1014- ``chmod(path, mode)``
1015
1016  sets the access mode (permission flags) for the given path. The mode
1017  is an integer as returned for the mode by the ``stat`` and ``lstat``
1018  methods. Be careful: Usually, mode values are written as octal
1019  numbers, for example 0755 to make a directory readable and writable
1020  for the owner, but not writable for the group and others. If you
1021  want to use such octal values, rely on Python's support for them::
1022
1023    ftp_host.chmod("some_directory", 0o755)
1024
1025  Not all FTP servers support the ``chmod`` command. In case of
1026  an exception, how do you know if the path doesn't exist or if
1027  the command itself is invalid? If the FTP server complies with
1028  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
1029  command isn't allowed. ``ftputil`` maps this special error
1030  response to a ``CommandNotImplementedError`` which is derived from
1031  ``PermanentError``.
1032
1033  So you need to code like this::
1034
1035    with ftputil.FTPHost(server, user, password) as ftp_host:
1036        try:
1037            ftp_host.chmod("some_file", 0o644)
1038        except ftputil.error.CommandNotImplementedError:
1039            # `chmod` not supported
1040            ...
1041        except ftputil.error.PermanentError:
1042            # Possibly a non-existent file
1043            ...
1044
1045  Because the ``CommandNotImplementedError`` is more specific, you
1046  have to test for it first.
1047
1048.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
1049
1050- ``copyfileobj(source, target, length=64*1024)``
1051
1052  copies the contents from the file-like object ``source`` to the
1053  file-like object ``target``. The only difference to
1054  ``shutil.copyfileobj`` is the default buffer size. Note that
1055  arbitrary file-like objects can be used as arguments (e. g. local
1056  files, remote FTP files).
1057
1058  However, the interfaces of ``source`` and ``target`` have to match;
1059  the string type read from ``source`` must be an accepted string type
1060  when written to ``target``. For example, if you open ``source`` in
1061  Python 3 as a local text file and ``target`` as a remote file object
1062  in binary mode, the transfer will fail since ``source.read`` gives
1063  unicode strings whereas ``target.write`` only accepts byte strings.
1064
1065  See `File-like objects`_ for the construction and use of remote
1066  file-like objects.
1067
1068.. _`set_parser`:
1069
1070- ``set_parser(parser)``
1071
1072  sets a custom parser for FTP directories. Note that you have to pass
1073  in a parser *instance*, not the class.
1074
1075  An `extra section`_ shows how to write own parsers if the default
1076  parsers in ``ftputil`` don't work for you.
1077
1078.. _`extra section`: `Writing directory parsers`_
1079
1080.. _`keep_alive`:
1081
1082- ``keep_alive()``
1083
1084  attempts to keep the connection to the remote server active in order
1085  to prevent timeouts from happening. This method is primarily
1086  intended to keep the underlying FTP connection of an ``FTPHost``
1087  object alive while a file is uploaded or downloaded. This will
1088  require either an extra thread while the upload or download is in
1089  progress or calling ``keep_alive`` from a `callback function`_.
1090
1091  The ``keep_alive`` method won't help if the connection has already
1092  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
1093
1094  If you want to use this method, keep in mind that FTP servers define
1095  a timeout for a reason. A timeout prevents running out of server
1096  connections because of clients that never disconnect on their own.
1097
1098  Note that the ``keep_alive`` method does *not* affect the "hidden"
1099  FTP child connections established by ``FTPHost.open`` (see section
1100  `FTPHost instances vs. FTP connections`_ for details). You *can't*
1101  use ``keep_alive`` to avoid a timeout in a stalling transfer like
1102  this::
1103
1104      with ftputil.FTPHost(server, userid, password) as ftp_host:
1105          with ftp_host.open("some_remote_file", "rb") as fobj:
1106              data = fobj.read(100)
1107              # _Futile_ attempt to avoid file connection timeout.
1108              for i in xrange(15):
1109                  time.sleep(60)
1110                  ftp_host.keep_alive()
1111              # Will raise an `ftputil.error.TemporaryError`.
1112              data += fobj.read()
1113
1114
1115.. _`FTPHost.open`:
1116
1117File-like objects
1118-----------------
1119
1120Construction
1121~~~~~~~~~~~~
1122
1123Basics
1124``````
1125
1126``FTPFile`` objects are returned by a call to ``FTPHost.open``;
1127never use the ``FTPFile`` constructor directly.
1128
1129The API of remote file-like objects are is modeled after the API of
1130the io_ module in Python 3, which has also been backported to Python
11312.6 and 2.7.
1132
1133.. _io: http://docs.python.org/library/io.html
1134
1135- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
1136  errors=None, newline=None, rest=None)``
1137
1138  returns a file-like object that refers to the path on the remote
1139  host. This path may be absolute or relative to the current directory
1140  on the remote host (this directory can be determined with the
1141  ``getcwd`` method). As with local file objects, the default mode is
1142  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
1143  "wb".
1144
1145  If a file is opened in binary mode, you *must not* specify an
1146  encoding. On the other hand, if you open a file in text mode, an
1147  encoding is used. By default, this is the return value of
1148  ``locale.getpreferredencoding``, but you can (and probably should)
1149  specify a distinct encoding.
1150
1151  If you open a file in binary mode, the read and write operations use
1152  byte strings (``str`` in Python 2, ``bytes`` in Python 3). That is,
1153  read operations return byte strings and write operations only accept
1154  byte strings.
1155
1156  Similarly, text files always work with unicode strings (``unicode``
1157  in Python 2, ``str`` in Python 3). Here, read operations return
1158  unicode strings and write operations only accept unicode strings.
1159
1160  .. warning::
1161
1162     Note that the semantics of "text mode" has changed fundamentally
1163     from ftputil 2.8 and earlier. Previously, "text mode" implied
1164     converting newline characters to ``\r\n`` when writing remote
1165     files and converting newlines to ``\n`` when reading remote
1166     files. This is in line with the "text mode" notion of FTP command
1167     line clients. Now, "text mode" follows the semantics in Python's
1168     ``io`` module.
1169
1170  The arguments ``errors`` and ``newline`` have the same semantics as
1171  in `io.open`_. The argument ``buffering`` currently is ignored.
1172  It's only there for compatibility with the ``io.open`` interface.
1173
1174  If the file is opened in binary mode, you may pass 0 or a positive
1175  integer for the ``rest`` argument. The argument is passed to the
1176  underlying FTP session instance (for example an instance of
1177  ``ftplib.FTP``) to start reading or writing at the given byte
1178  offset. For example, if a remote file contains the letters
1179  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".
1180
1181  .. warning::
1182
1183     If you pass ``rest`` values which point *after* the file, the
1184     behavior is undefined and may even differ from one FTP server to
1185     another. Therefore, use the ``rest`` argument only for error
1186     recovery in case of interrupted transfers. You need to keep track
1187     of the transferred data so that you can provide a valid ``rest``
1188     argument for a resumed transfer.
1189
1190.. _`io.open`: http://docs.python.org/library/io.html#io.open
1191
1192``FTPHost.open`` can also be used in a ``with`` statement::
1193
1194    import ftputil
1195
1196    with ftputil.FTPHost(...) as ftp_host:
1197        ...
1198        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
1199            fobj.write("This is some text.")
1200
1201At the end of the ``with`` block, the remote file will be closed
1202automatically.
1203
1204If something goes wrong during the construction of the file or in the
1205body of the ``with`` statement, the file will be closed as well.
1206Exceptions will be propagated as with ``try ... finally``.
1207
1208Attributes and methods
1209~~~~~~~~~~~~~~~~~~~~~~
1210
1211The methods
1212
1213::
1214
1215    close()
1216    read([count])
1217    readline([count])
1218    readlines()
1219    write(data)
1220    writelines(string_sequence)
1221
1222and the attribute ``closed`` have the same semantics as for file
1223objects of a local disk file system. The iterator protocol is
1224supported as well, i. e. you can use a loop to read a file line by
1225line::
1226
1227    with ftputil.FTPHost(server, user, password) as ftp_host:
1228        with ftp_host.open("some_file") as input_file:
1229            for line in input_file:
1230                # Do something with the line, e. g.
1231                print line.strip().replace("ftplib", "ftputil")
1232
1233For more on file objects, see the section `File objects`_ in the
1234Python Library Reference.
1235
1236.. _`file objects`: https://docs.python.org/2.7/library/stdtypes.html#file-objects
1237
1238
1239.. _`child_connections`:
1240
1241``FTPHost`` instances vs. FTP connections
1242-----------------------------------------
1243
1244This section explains why keeping an ``FTPHost`` instance "alive"
1245without timing out sometimes isn't trivial. If you always finish your
1246FTP operations in time, you don't need to read this section.
1247
1248The file transfer protocol is a stateful protocol. That means an FTP
1249connection always is in a certain state. Each of these states can only
1250change to certain other states under certain conditions triggered by
1251the client or the server.
1252
1253One of the consequences is that a single FTP connection can't be used
1254at the same time, say, to transfer data on the FTP data channel and to
1255create a directory on the remote host.
1256
1257For example, consider this::
1258
1259    >>> import ftplib
1260    >>> ftp = ftplib.FTP(server, user, password)
1261    >>> ftp.pwd()
1262    '/'
1263    >>> # Start transfer. `CONTENTS` is a text file on the server.
1264    >>> socket = ftp.transfercmd("RETR CONTENTS")
1265    >>> socket
1266    <socket._socketobject object at 0x7f801a6386e0>
1267    >>> ftp.pwd()
1268    Traceback (most recent call last):
1269      File "<stdin>", line 1, in <module>
1270      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
1271        return parse257(resp)
1272      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
1273        raise error_reply, resp
1274    ftplib.error_reply: 226-File successfully transferred
1275    226 0.000 seconds (measured here), 5.60 Mbytes per second
1276    >>>
1277
1278Note that ``ftp`` is a single FTP connection, represented by an
1279``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
1280
1281On the other hand, consider this::
1282
1283    >>> import ftputil
1284    >>> ftp_host = ftputil.FTPHost(server, user, password)
1285    >>> ftp_host.getcwd()
1286    >>> fobj = ftp_host.open("CONTENTS")
1287    >>> fobj
1288    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
1289    >>> ftp_host.getcwd()
1290    u'/'
1291    >>> fobj.readline()
1292    u'Contents of FTP test directory\n'
1293    >>> fobj.close()
1294    >>>
1295
1296To be able to start a file transfer (i. e. open a remote file for
1297reading or writing) and still be able to use other FTP commands,
1298ftputil uses a trick. For every remote file, ftputil creates a new FTP
1299connection, called a child connection in the ftputil source code.
1300(Actually, FTP connections belonging to closed remote files are
1301re-used if they haven't timed out yet.)
1302
1303In most cases this approach isn't noticeable by code using ftputil.
1304However, the nice abstraction of dealing with a single FTP connection
1305falls apart if one of the child connections times out. For example, if
1306you open a remote file and work only with the initial "main"
1307connection to navigate the file system, the FTP connection for the
1308remote file may eventually time out.
1309
1310While it's often relatively easy to prevent the "main" connection from
1311timing out it's unfortunately practically impossible to do this for a
1312remote file connection (apart from transferring some data, of course).
1313For this reason, `FTPHost.keep_alive`_ affects only the main
1314connection. Child connections may still time out if they're idle for
1315too long.
1316
1317.. _`FTPHost.keep_alive`: `keep_alive`_
1318
1319Some more details:
1320
1321- A kind of "straightforward" way of keeping the main connection alive
1322  would be to call ``ftp_host.getcwd()``. However, this doesn't work
1323  because ftputil caches the current directory and returns it without
1324  actually contacting the server. That's the main reason why there's
1325  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
1326  (i. e. the session object), which isn't a public attribute.
1327
1328- Some servers define not only an idle timeout but also a transfer
1329  timeout. This means the connection times out unless there's some
1330  transfer on the data channel for this connection. So ftputil's
1331  ``keep_alive`` doesn't prevent this timeout, but an
1332  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
1333  this transfers the data for the whole directory listing which might
1334  take some time if the directory has many entries.
1335
1336Bottom line: If you can, you should organize your FTP actions so that
1337you finish everything before a timeout happens.
1338
1339
1340Writing directory parsers
1341-------------------------
1342
1343``ftputil`` recognizes the two most widely-used FTP directory formats,
1344Unix and MS style, and adjusts itself automatically. Almost every FTP
1345server uses one of these formats.
1346
1347However, if your server uses a format which is different from the two
1348provided by ``ftputil``, you can plug in a custom parser with a single
1349method call and have ``ftputil`` use this parser.
1350
1351For this, you need to write a parser class by inheriting from the
1352class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
1353
1354    import ftputil.error
1355    import ftputil.stat
1356
1357    class XyzParser(ftputil.stat.Parser):
1358        """
1359        Parse the default format of the FTP server of the XYZ
1360        corporation.
1361        """
1362
1363        def parse_line(self, line, time_shift=0.0):
1364            """
1365            Parse a `line` from the directory listing and return a
1366            corresponding `StatResult` object. If the line can't
1367            be parsed, raise `ftputil.error.ParserError`.
1368
1369            The `time_shift` argument can be used to fine-tune the
1370            parsing of dates and times. See the class
1371            `ftputil.stat.UnixParser` for an example.
1372            """
1373            # Split the `line` argument and examine it further; if
1374            # something goes wrong, raise an `ftputil.error.ParserError`.
1375            ...
1376            # Make a `StatResult` object from the parts above.
1377            stat_result = ftputil.stat.StatResult(...)
1378            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
1379            stat_result._st_name = ...
1380            stat_result._st_target = ...
1381            stat_result._st_mtime_precision = ...
1382            return stat_result
1383
1384        # Define `ignores_line` only if the default in the base class
1385        # doesn't do enough!
1386        def ignores_line(self, line):
1387            """
1388            Return a true value if the line should be ignored. For
1389            example, the implementation in the base class handles
1390            lines like "total 17". On the other hand, if the line
1391            should be used for stat'ing, return a false value.
1392            """
1393            is_total_line = super(XyzParser, self).ignores_line(line)
1394            my_test = ...
1395            return is_total_line or my_test
1396
1397A ``StatResult`` object is similar to the value returned by
1398`os.stat`_ and is usually built with statements like
1399
1400::
1401
1402    stat_result = StatResult(
1403                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
1404                     st_gid, st_size, st_atime, st_mtime, st_ctime))
1405    stat_result._st_name = ...
1406    stat_result._st_target = ...
1407    stat_result._st_mtime_precision = ...
1408
1409with the arguments of the ``StatResult`` constructor described in
1410the following table.
1411
1412===== =================== ============ =================== =======================
1413Index Attribute           os.stat type ``StatResult`` type Notes
1414===== =================== ============ =================== =======================
14150     st_mode             int          int
14161     st_ino              long         long
14172     st_dev              long         long
14183     st_nlink            int          int
14194     st_uid              int          str                 usually only available as string
14205     st_gid              int          str                 usually only available as string
14216     st_size             long         long
14227     st_atime            int/float    float
14238     st_mtime            int/float    float
14249     st_ctime            int/float    float
1425\-    _st_name            \-           str                 file name without directory part
1426\-    _st_target          \-           str                 link target (may be absolute or relative)
1427\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
1428===== =================== ============ =================== =======================
1429
1430If you can't extract all the desirable data from a line (for
1431example, the MS format doesn't contain any information about the
1432owner of a file), set the corresponding values in the ``StatResult``
1433instance to ``None``.
1434
1435Parser classes can use several helper methods which are defined in
1436the class ``Parser``:
1437
1438- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
1439  an appropriate ``st_mode`` integer value.
1440
1441- ``parse_unix_time`` returns a float number usable for the
1442  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
1443  "May"/"26"/"2005". Note that the method expects the timestamp string
1444  already split at whitespace.
1445
1446- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
1447  returns a float number like from ``time.mktime``. Note that the
1448  method expects the timestamp string already split at whitespace.
1449
1450Additionally, there's an attribute ``_month_numbers`` which maps
1451lowercase three-letter month abbreviations to integers.
1452
1453For more details, see the two "standard" parsers ``UnixParser`` and
1454``MSParser`` in the module ``ftputil/stat.py``.
1455
1456To actually *use* the parser, call the method `set_parser`_ of the
1457``FTPHost`` instance.
1458
1459If you can't write a parser or don't want to, please ask on the
1460`ftputil mailing list`_. Possibly someone has already written a parser
1461for your server or can help with it.
1462
1463
1464FAQ / Tips and tricks
1465---------------------
1466
1467Where can I get the latest version?
1468~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1469
1470See the `download page`_. Announcements will be sent to the `mailing
1471list`_. Announcements on major updates will also be posted to the
1472newsgroup `comp.lang.python.announce`_ .
1473
1474.. _`download page`: http://ftputil.sschwarzer.net/download
1475.. _`mailing list`: http://ftputil.sschwarzer.net/mailinglist
1476.. _`comp.lang.python.announce`: news:comp.lang.python.announce
1477
1478Is there a mailing list on ``ftputil``?
1479~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1480
1481Yes, please visit http://ftputil.sschwarzer.net/mailinglist to
1482subscribe or read the archives.
1483
1484Though you can *technically* post without subscribing first I can't
1485recommend it: The mails from non-subscribers have to be approved by
1486me and because the arriving mails contain *lots* of spam, I rarely go
1487through these mails.
1488
1489I found a bug! What now?
1490~~~~~~~~~~~~~~~~~~~~~~~~
1491
1492Before reporting a bug, make sure that you already read this manual
1493and tried the `latest version`_ of ``ftputil``. There the bug might
1494have already been fixed.
1495
1496.. _`latest version`: http://ftputil.sschwarzer.net/download
1497
1498Please see http://ftputil.sschwarzer.net/issuetrackernotes for
1499guidelines on entering a bug in ``ftputil``'s ticket system. If you
1500are unsure if the behaviour you found is a bug or not, you should write
1501to the `ftputil mailing list`_. In *either* case you *must not*
1502include confidential information (user id, password, file names, etc.)
1503in the problem report! Be careful!
1504
1505Does ``ftputil`` support SSL/TLS?
1506~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1507
1508``ftputil`` has no *built-in* SSL/TLS support.
1509
1510On the other hand, there are two ways to get TLS support with
1511ftputil:
1512
1513- In Python 2.7 and Python 3.2 and up, the ``ftplib`` library has a
1514  class ``FTP_TLS`` that you can use for the ``session_factory``
1515  keyword argument in the ``FTPHost`` constructor. You can't use the
1516  class directly though if you need additional setup code in
1517  comparison to ``ftplib.FTP``, for example calling ``prot_p``, to
1518  secure the data connection. On the other hand,
1519  `ftputil.session.session_factory`_ can be used to create a custom
1520  session factory.
1521 
1522  If you have other requirements that ``session_factory`` can't
1523  fulfill, you may create your own session factory by inheriting from
1524  ``ftplib.FTP_TLS``::
1525
1526    import ftplib
1527
1528    import ftputil
1529
1530
1531    class FTPTLSSession(ftplib.FTP_TLS):
1532
1533        def __init__(self, host, user, password):
1534            ftplib.FTP_TLS.__init__(self)
1535            self.connect(host, port)
1536            self.login(user, password)
1537            # Set up encrypted data connection.
1538            self.prot_p()
1539            ...
1540
1541    # Note the `session_factory` parameter. Pass the class, not
1542    # an instance.
1543    with ftputil.FTPHost(server, user, password,
1544                         session_factory=FTPTLSSession) as ftp_host:
1545        # Use `ftp_host` as usual.
1546        ...
1547
1548.. _`ftputil.session.session_factory`: `Session factories`_
1549
1550- If you need to work with Python 2.6, you can use the
1551  ``ftpslib.FTP_TLS`` class from the M2Crypto_ project. Again, you
1552  can't use the class directly but need to use
1553  ``ftputil.session.session_factory`` or a recipe similar to that
1554  above.
1555
1556  Unfortunately, ``M2Crypto.ftpslib.FTP_TLS`` (at least in version
1557  0.22.3) doesn't work correctly if you pass unicode strings to its
1558  methods. Since ``ftputil`` does exactly that at some point (even if
1559  you used byte strings in ``ftputil`` calls) you need a workaround in
1560  the session factory class::
1561
1562    import M2Crypto
1563
1564    import ftputil
1565    import ftputil.tool
1566
1567
1568    class M2CryptoSession(M2Crypto.ftpslib.FTP_TLS):
1569
1570        def __init__(self, host, user, password):
1571            M2Crypto.ftpslib.FTP_TLS.__init__(self)
1572            # Change the port number if needed.
1573            self.connect(host, 21)
1574            self.auth_tls()
1575            self.login(user, password)
1576            self.prot_p()
1577            self._fix_socket()
1578            ...
1579
1580        def _fix_socket(self):
1581            """
1582            Change the socket object so that arguments to `sendall`
1583            are converted to byte strings before being used.
1584            """
1585            original_sendall = self.sock.sendall
1586            # Bound method, therefore no `self` argument.
1587            def sendall(data):
1588                data = ftputil.tool.as_bytes(data)
1589                return original_sendall(data)
1590            self.sock.sendall = sendall
1591
1592    # Note the `session_factory` parameter. Pass the class, not
1593    # an instance.
1594    with ftputil.FTPHost(server, user, password,
1595                         session_factory=M2CryptoSession) as ftp_host:
1596        # Use `ftp_host` as usual.
1597        ...
1598
1599  That said, ``session_factory`` has this workaround built in, so
1600  normally you don't need to define the session factory yourself!
1601
1602.. _M2Crypto: https://github.com/martinpaljak/M2Crypto
1603
1604How do I connect to a non-default port?
1605~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1606
1607By default, an instantiated ``FTPHost`` object connects on the usual
1608FTP port. If you have to use a different port, refer to the section
1609`Session factories`_.
1610
1611How do I set active or passive mode?
1612~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1613
1614Please see the section `Session factories`_.
1615
1616How can I debug an FTP connection problem?
1617~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1618
1619You can do this with a session factory. See `Session factories`_.
1620
1621If you want to change the debug level only temporarily after the
1622connection is established, you can reach the `session object`_ as the
1623``_session`` attribute of the ``FTPHost`` instance and call
1624``_session.set_debuglevel``. Note that the ``_session`` attribute
1625should *only* be accessed for debugging. Calling arbitrary
1626``ftplib.FTP`` methods on the session object may *cause* bugs!
1627
1628.. _`session object`: `Session factories`_
1629
1630Conditional upload/download to/from a server in a different time zone
1631~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1632
1633You may find that ``ftputil`` uploads or downloads files
1634unnecessarily, or not when it should. This can happen when the FTP
1635server is in a different time zone than the client on which
1636``ftputil`` runs. Please see the section on `time zone correction`_.
1637It may even be sufficient to call `synchronize_times`_.
1638
1639When I use ``ftputil``, all I get is a ``ParserError`` exception
1640~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1641
1642The FTP server you connect to may use a directory format that
1643``ftputil`` doesn't understand. You can either write and
1644`plug in an own parser`_ or ask on the `mailing list`_ for
1645help.
1646
1647.. _`plug in an own parser`: `Writing directory parsers`_
1648
1649``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
1650~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1651
1652Like Python's counterparts under `os.path`_, ``ftputil``'s methods
1653return ``False`` if they can't find the given path.
1654
1655Probably you used ``listdir`` on a directory and called ``is...()`` on
1656the returned names. But if the argument for ``listdir`` wasn't the
1657current directory, the paths won't be found and so all ``is...()``
1658variants will return ``False``.
1659
1660I don't find an answer to my problem in this document
1661~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1662
1663Please send an email with your problem report or question to the
1664`ftputil mailing list`_, and we'll see what we can do for you. :-)
1665
1666
1667Bugs and limitations
1668--------------------
1669
1670- ``ftputil`` needs at least Python 2.6 to work.
1671
1672- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
1673  names starting with a dot) depends on the configuration of the FTP
1674  server. See `Hidden files and directories`_ for details.
1675
1676- Due to the implementation of ``lstat`` it can not return a sensible
1677  value for the root directory ``/`` though stat'ing entries *in* the
1678  root directory isn't a problem. If you know an implementation that
1679  can do this, please let me know. The root directory is handled
1680  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
1681
1682- In multithreaded programs, you can have each thread use one or more
1683  ``FTPHost`` instances as long as no instance is shared with other
1684  threads.
1685
1686- Currently, it is not possible to continue an interrupted upload or
1687  download. Contact me if this causes problems for you.
1688
1689- There's exactly one cache for ``lstat`` results for each ``FTPHost``
1690  object, i. e. there's no sharing of cache results determined by
1691  several ``FTPHost`` objects. See `Local caching of file system
1692  information`_ for the reasons.
1693
1694
1695Files
1696-----
1697
1698If not overwritten via installation options, the ``ftputil`` files
1699reside in the ``ftputil`` package. There's also documentation in
1700`reStructuredText`_ and in HTML format. The locations of these
1701files after installation is system-dependent.
1702
1703.. _`reStructuredText`: http://docutils.sourceforge.net/rst.html
1704
1705The files ``test_*.py`` and ``mock_ftplib.py`` are for unit-testing.
1706If you only *use* ``ftputil``, i. e. *don't* modify it, you can
1707delete these files.
1708
1709
1710References
1711----------
1712
1713- Mackinnon T, Freeman S, Craig P. 2000. `Endo-Testing:
1714  Unit Testing with Mock Objects`_.
1715
1716- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
1717
1718- Van Rossum G et al. 2013. `Python Library Reference`_.
1719
1720.. _`Endo-Testing: Unit Testing with Mock Objects`:
1721   http://www.connextra.com/aboutUs/mockobjects.pdf
1722.. _`RFC 959 - File Transfer Protocol (FTP)`: http://www.ietf.org/rfc/rfc959.txt
1723.. _`Python Library Reference`: https://docs.python.org/library/index.html
1724
1725
1726Authors
1727-------
1728
1729``ftputil`` is written by Stefan Schwarzer
1730<sschwarzer@sschwarzer.net> and contributors (see
1731``doc/contributors.txt``).
1732
1733The original ``lrucache`` module was written by Evan Prodromou
1734<evan@prodromou.name>.
1735
1736Feedback is appreciated. :-)
Note: See TracBrowser for help on using the repository browser.