source: doc/ftputil.txt @ 1727:2652fb65ded9

Last change on this file since 1727:2652fb65ded9 was 1727:2652fb65ded9, checked in by Stefan Schwarzer <sschwarzer@…>, 4 months ago
Use `print` as a function in non-code Add parentheses for `print` calls in - documentation - docstrings - comments
File size: 63.3 KB
Line 
1``ftputil`` -- a high-level FTP client library
2==============================================
3
4:Version:   3.4
5:Date:      2017-11-08
6:Summary:   high-level FTP client library for Python
7:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
8:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
9
10.. contents::
11
12
13Introduction
14------------
15
16The ``ftputil`` module is a high-level interface to the ftplib_
17module. The `FTPHost objects`_ generated from it allow many operations
18similar to those of os_, `os.path`_ and `shutil`_.
19
20.. _ftplib: https://docs.python.org/library/ftplib.html
21.. _os: https://docs.python.org/library/os.html
22.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
23.. _`os.path`: https://docs.python.org/library/os.path.html
24.. _`shutil`: https://docs.python.org/library/shutil.html
25
26Example::
27
28    import ftputil
29
30    # Download some files from the login directory.
31    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
32        names = ftp_host.listdir(ftp_host.curdir)
33        for name in names:
34            if ftp_host.path.isfile(name):
35                ftp_host.download(name, name)  # remote, local
36        # Make a new directory and copy a remote file into it.
37        ftp_host.mkdir("newdir")
38        with ftp_host.open("index.html", "rb") as source:
39            with ftp_host.open("newdir/index.html", "wb") as target:
40                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
41
42Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
43modification time of a file. The latter can also follow links, similar
44to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
45
46
47``ftputil`` features
48--------------------
49
50* Method names are familiar from Python's ``os``, ``os.path`` and
51  ``shutil`` modules. For example, use ``os.path.join`` to join
52  paths for a local file system and ``ftp_host.path.join`` to join
53  paths for a remote FTP file system.
54
55* Remote file system navigation (``getcwd``, ``chdir``)
56
57* Upload and download files (``upload``, ``upload_if_newer``,
58  ``download``, ``download_if_newer``)
59
60* Time zone synchronization between client and server (needed
61  for ``upload_if_newer`` and ``download_if_newer``)
62
63* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
64  ``rmtree``) and remove files (``remove``)
65
66* Get information about directories, files and links (``listdir``,
67  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
68  ``abspath``, ``split``, ``join``, ``dirname``, ``basename`` etc.)
69
70* Iterate over remote file systems (``walk``)
71
72* Local caching of results from ``lstat`` and ``stat`` calls to reduce
73  network access (also applies to ``exists``, ``getmtime`` etc.).
74
75* Read files from and write files to remote hosts via
76  file-like objects (``FTPHost.open``; the generated file-like objects
77  have the familiar methods like ``read``, ``readline``, ``readlines``,
78  ``write``, ``writelines`` and ``close``. You can also iterate over
79  these files line by line in a ``for`` loop.
80
81
82Exception hierarchy
83-------------------
84
85The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
86``ftputil.error.TemporaryError``.
87
88The exception classes are organized as follows::
89
90    FTPError
91        FTPOSError(FTPError, OSError)
92            PermanentError(FTPOSError)
93                CommandNotImplementedError(PermanentError)
94            TemporaryError(FTPOSError)
95        FTPIOError(FTPError)
96        InternalError(FTPError)
97            InaccessibleLoginDirError(InternalError)
98            ParserError(InternalError)
99            RootDirError(InternalError)
100            TimeShiftError(InternalError)
101
102and are described here:
103
104- ``FTPError``
105
106  is the root of the exception hierarchy of the module.
107
108- ``FTPOSError``
109
110  is derived from ``OSError``. This is for similarity between the
111  os module and ``FTPHost`` objects. Compare
112
113  ::
114
115    try:
116        os.chdir("nonexisting_directory")
117    except OSError:
118        ...
119
120  with
121
122  ::
123
124    host = ftputil.FTPHost("host", "user", "password")
125    try:
126        host.chdir("nonexisting_directory")
127    except OSError:
128        ...
129
130  Imagine a function
131
132  ::
133
134    def func(path, file):
135        ...
136
137  which works on the local file system and catches ``OSErrors``. If you
138  change the parameter list to
139
140  ::
141
142    def func(path, file, os=os):
143        ...
144
145  where ``os`` denotes the ``os`` module, you can call the function also as
146
147  ::
148
149    host = ftputil.FTPHost("host", "user", "password")
150    func(path, file, os=host)
151
152  to use the same code for both a local and remote file system.
153  Another similarity between ``OSError`` and ``FTPOSError`` is that
154  the latter holds the FTP server return code in the ``errno``
155  attribute of the exception object and the error text in
156  ``strerror``.
157
158- ``PermanentError``
159
160  is raised for 5xx return codes from the FTP server. This
161  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
162  ``ftplib.error_perm`` are *not* identical).
163
164- ``CommandNotImplementedError``
165
166  indicates that an underlying command the code tries to use is not
167  implemented. For an example, see the description of the
168  `FTPHost.chmod`_ method.
169
170- ``TemporaryError``
171
172  is raised for FTP return codes from the 4xx category. This
173  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
174  ``ftplib.error_temp`` are *not* identical).
175
176- ``FTPIOError``
177
178  denotes an I/O error on the remote host. This appears
179  mainly with file-like objects that are retrieved by calling
180  ``FTPHost.open``. Compare
181
182  ::
183
184    >>> try:
185    ...     f = open("not_there")
186    ... except IOError as obj:
187    ...     print(obj.errno)
188    ...     print(obj.strerror)
189    ...
190    2
191    No such file or directory
192
193  with
194
195  ::
196
197    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
198    >>> try:
199    ...     f = ftp_host.open("not_there")
200    ... except IOError as obj:
201    ...     print(obj.errno)
202    ...     print(obj.strerror)
203    ...
204    550
205    550 not_there: No such file or directory.
206
207  As you can see, both code snippets are similar. However, the error
208  codes aren't the same.
209
210- ``InternalError``
211
212  subsumes exception classes for signaling errors due to limitations
213  of the FTP protocol or the concrete implementation of ``ftputil``.
214
215- ``InaccessibleLoginDirError``
216
217  This exception is raised if the directory in which "you" are placed
218  upon login is not accessible, i. e. a ``chdir`` call with the
219  directory as argument would fail.
220
221- ``ParserError``
222
223  is used for errors during the parsing of directory
224  listings from the server. This exception is used by the ``FTPHost``
225  methods ``stat``, ``lstat``, and ``listdir``.
226
227- ``RootDirError``
228
229  Because of the implementation of the ``lstat`` method it is not
230  possible to do a ``stat`` call on the root directory ``/``.
231  If you know *any* way to do it, please let me know. :-)
232
233  This problem does *not* affect stat calls on items *in* the root
234  directory.
235
236- ``TimeShiftError``
237
238  is used to denote errors which relate to setting the `time shift`_.
239
240
241Directory and file names
242------------------------
243
244.. note::
245
246   Keep in mind that this section only applies to directory and file
247   *names*, not file *contents*. Encoding and decoding for file
248   contents is handled by the ``encoding`` argument for
249   `FTPHost.open`_.
250
251First off: If your directory and file names (both as
252arguments and on the server) contain only ISO 8859-1 (latin-1)
253characters, you can use such names in the form of byte strings or
254unicode strings. However, you can't mix different string types (bytes
255and unicode) in one call (for example in ``FTPHost.path.join``).
256
257If you have directory or file names with characters that aren't in
258latin-1, it's recommended to use byte strings. In that case,
259returned paths will be byte strings, too.
260
261Read on for details.
262
263.. note::
264
265   The approach described below may look awkward and in a way it is.
266   The intention of ``ftputil`` is to behave like the local file
267   system APIs of Python 3 as far as it makes sense. Moreover, the
268   taken approach makes sure that directory and file names that were
269   used with Python 3's native ``ftplib`` module will be compatible
270   with ``ftputil`` and vice versa. Otherwise you may be able to use a
271   file name with ``ftputil``, but get an exception when trying to
272   read the same file with Python 3's ``ftplib`` module.
273
274Methods that take names of directories and/or files can take either
275byte or unicode strings. If a method got a string argument and returns
276one or more strings, these strings will have the same string type as
277the argument(s). Mixing different string arguments in one call (for
278example in ``FTPHost.path.join``) isn't allowed and will cause a
279``TypeError``. These rules are the same as for local file system
280operations in Python 3. Since ``ftputil`` uses the same API for Python
2812, ``ftputil`` will do the same when run on Python 2.
282
283Byte strings for directory and file names will be sent to the server
284as-is. On the other hand, unicode strings will be encoded to byte
285strings, assuming latin-1 encoding. This implies that such unicode
286strings must only contain code points 0-255 for the latin-1 character
287set. Using any other characters will result in a
288``UnicodeEncodeError`` exception.
289
290If you have directory or file names as unicode strings with non-latin-1
291characters, encode the unicode strings to byte strings yourself, using
292the encoding you know the server uses. Decode received paths with the
293same encoding. Encapsulate these conversions as far as you can.
294Otherwise, you'd have to adapt potentially a lot of code if the server
295encoding changes.
296
297If you *don't* know the encoding on the server side,
298it's probably the best to only use byte strings for directory and file
299names. That said, as soon as you *show* the names to a user, you -- or
300the library you use for displaying the names -- has to guess an
301encoding.
302
303
304``FTPHost`` objects
305-------------------
306
307.. _`FTPHost construction`:
308
309Construction
310~~~~~~~~~~~~
311
312Introduction
313````````````
314
315``FTPHost`` instances can be created with the following call::
316
317    ftp_host = ftputil.FTPHost(server, user, password, account,
318                               session_factory=ftplib.FTP)
319
320The first four parameters are strings with the same meaning as for the
321FTP class in the ``ftplib`` module. Usually the ``account`` and
322``session_factory`` arguments aren't needed though.
323
324``FTPHost`` objects can also be used in a ``with`` statement::
325
326    import ftputil
327
328    with ftputil.FTPHost(server, user, password) as ftp_host:
329        print(ftp_host.listdir(ftp_host.curdir))
330
331After the ``with`` block, the ``FTPHost`` instance and the
332associated FTP sessions will be closed automatically.
333
334If something goes wrong during the ``FTPHost`` construction or in the
335body of the ``with`` statement, the instance is closed as well.
336Exceptions will be propagated (as with ``try ... finally``).
337
338Session factories
339`````````````````
340
341The keyword argument ``session_factory`` may be used to generate FTP
342connections with other factories than the default ``ftplib.FTP``. For
343example, the standard library of Python 2.7 contains a class
344``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
345connection.
346
347In fact, all positional and keyword arguments other than
348``session_factory`` are passed to the factory to generate a new
349background session. This also happens for every remote file that is
350opened; see below.
351
352This functionality of the constructor also allows to wrap
353``ftplib.FTP`` objects to do something that wouldn't be possible with
354the ``ftplib.FTP`` constructor alone.
355
356As an example, assume you want to connect to another than the default
357port, but ``ftplib.FTP`` only offers this by means of its ``connect``
358method, not via its constructor. One solution is to use a custom
359class as a session factory::
360
361    import ftplib
362    import ftputil
363
364    EXAMPLE_PORT = 50001
365
366    class MySession(ftplib.FTP):
367
368        def __init__(self, host, userid, password, port):
369            """Act like ftplib.FTP's constructor but connect to another port."""
370            ftplib.FTP.__init__(self)
371            self.connect(host, port)
372            self.login(userid, password)
373
374    # Try _not_ to use an _instance_ `MySession()` as factory, -
375    # use the class itself.
376    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
377                         session_factory=MySession) as ftp_host:
378        # Use `ftp_host` as usual.
379        ...
380
381On login, the format of the directory listings (needed for stat'ing
382files and directories) should be determined automatically. If not,
383please `file a bug report`_.
384
385.. _`file a bug report`: http://ftputil.sschwarzer.net/issuetrackernotes
386
387For the most common uses you don't need to create your own session
388factory class though. The ``ftputil.session`` module has a function
389``session_factory`` that can create session factories for a variety
390of parameters::
391
392    session_factory(base_class=ftplib.FTP,
393                    port=21,
394                    use_passive_mode=None,
395                    encrypt_data_channel=True,
396                    debug_level=None)
397
398with
399
400- ``base_class`` is a base class to inherit a new session factory
401  class from. By default, this is ``ftplib.FTP`` from the Python
402  standard library.
403
404- ``port`` is the command channel port. The default is 21, used in most
405  FTP server configurations.
406
407- ``use_passive_mode`` is either a boolean that determines whether
408  passive mode should be used or ``None``. ``None`` means to let the
409  base class choose active or passive mode.
410
411- ``encrypt_data_channel`` defines whether to encrypt the data channel
412  for secure connections. This is only supported for the base classes
413  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
414  the parameter is ignored.
415
416- ``debug_level`` sets the debug level for FTP session instances. The
417  semantics is defined by the base class. For example, a debug level
418  of 2 causes the most verbose output for Python's ``ftplib.FTP``
419  class.
420
421All of these parameters can be combined. For example, you could use
422
423::
424
425    import ftplib
426
427    import ftputil
428    import ftputil.session
429
430
431    my_session_factory = ftputil.session.session_factory(
432                           base_class=ftpslib.FTP_TLS,
433                           port=31,
434                           encrypt_data_channel=True,
435                           debug_level=2)
436
437    with ftputil.FTPHost(server, user, password,
438                         session_factory=my_session_factory) as ftp_host:
439        ...
440
441to create and use a session factory derived from ``ftplib.FTP_TLS``
442that connects on command channel 31, will encrypt the data channel and
443print output for debug level 2.
444
445Note: Generally, you can achieve everything you can do with
446``ftputil.session.session_factory`` with an explicit session factory
447as described at the start of this section. However, the class
448``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
449it with ftputil out of the box. The function ``session_factory``
450contains a workaround for this limitation. For details refer to `this
451bug report`_.
452
453.. _`this bug report`: http://ftputil.sschwarzer.net/trac/ticket/78
454
455Hidden files and directories
456~~~~~~~~~~~~~~~~~~~~~~~~~~~~
457
458Whether ftputil sees "hidden" files and directories (usually files or
459directories whose names start with a dot) depends on the FTP server
460configuration. By default, ftputil does *not* use the ``-a`` option in
461the FTP ``LIST`` command to find hidden files.
462
463To tell the server to list hidden directories and files, set
464``FTPHost.use_list_a_option`` to ``True``::
465
466    ftp_host = ftputil.FTPHost(server, user, password, account,
467                               session_factory=ftplib.FTP)
468    ftp_host.use_list_a_option = True
469
470Caveats:
471
472- If the server doesn't understand the ``-a`` option at all, the
473  server may interpret ``-a`` as the name of a file or directory,
474  which can result in odd behavior. Therefore, use ``-a`` only if
475  you're sure the server you're talking to supports it. Another
476  approach is to have test code for ``-a`` support and fall back to
477  not using the option.
478
479- Even if the server knows about the ``-a`` option, the server may
480  be configured to ignore it.
481
482``FTPHost`` attributes and methods
483~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
484
485Attributes
486``````````
487
488- ``curdir``, ``pardir``, ``sep``
489
490  are strings which denote the current and the parent directory on the
491  remote server. ``sep`` holds the path separator. Though `RFC 959`_
492  (File Transfer Protocol) notes that these values may depend on the
493  FTP server implementation, the Unix variants seem to work well in
494  practice, even for non-Unix servers.
495
496  Nevertheless, it's recommended that you don't hardcode these values
497  for remote paths, but use `FTPHost.path`_ as you would use
498  ``os.path`` to write platform-independent Python code for local
499  filesystems. Keep in mind that most, *but not all*, arguments of
500  ``FTPHost`` methods refer to remote directories or files. For
501  example, in `FTPHost.upload`_, the first argument is a local
502  path and the second a remote path. Both of these should use their
503  respective path separators.
504
505.. _`FTPHost.upload`: `Uploading and downloading files`_
506
507Remote file system navigation
508`````````````````````````````
509
510- ``getcwd()``
511
512  returns the absolute current directory on the remote host. This
513  method works like ``os.getcwd``.
514
515- ``chdir(directory)``
516
517  sets the current directory on the FTP server. This resembles
518  ``os.chdir``, as you may have expected.
519
520.. _`callback function`:
521
522Uploading and downloading files
523```````````````````````````````
524
525- ``upload(source, target, callback=None)``
526
527  copies a local source file (given by a filename, i. e. a string)
528  to the remote host under the name target. Both ``source`` and
529  ``target`` may be absolute paths or relative to their corresponding
530  current directory (on the local or the remote host, respectively).
531
532  The file content is always transferred in binary mode.
533
534  The callback, if given, will be invoked for each transferred chunk
535  of data::
536
537    callback(chunk)
538
539  where ``chunk`` is a bytestring. An example usage of a callback
540  method is to display a progress indicator.
541
542- ``download(source, target, callback=None)``
543
544  performs a download from the remote source file to a local target
545  file. Both ``source`` and ``target`` are strings. See the
546  description of ``upload`` for more details.
547
548.. _`upload_if_newer`:
549
550- ``upload_if_newer(source, target, callback=None)``
551
552  is similar to the ``upload`` method. The only difference is that the
553  upload is only invoked if the time of the last modification for the
554  source file is more recent than that of the target file or the
555  target doesn't exist at all. The check for the last modification
556  time considers the precision of the timestamps and transfers a file
557  "if in doubt". Consequently the code
558
559  ::
560
561    ftp_host.upload_if_newer("source_file", "target_file")
562    time.sleep(10)
563    ftp_host.upload_if_newer("source_file", "target_file")
564
565  might upload the file again if the timestamp of the target file is
566  precise up to a minute, which is typically the case because the
567  remote datetime is determined by parsing a directory listing from
568  the server. To avoid unnecessary transfers, wait at least a minute
569  between calls of ``upload_if_newer`` for the same file. If it still
570  seems that a file is uploaded unnecessarily (or not when it should),
571  read the subsection on `time shift`_ settings.
572
573  If an upload actually happened, the return value of
574  ``upload_if_newer`` is a ``True``, else ``False``.
575
576  Note that the method only checks the existence and/or the
577  modification time of the source and target file; it doesn't
578  compare any other file properties, say, the file size.
579
580  This also means that if a transfer is interrupted, the remote file
581  will have a newer modification time than the local file, and thus
582  the transfer won't be repeated if ``upload_if_newer`` is used a
583  second time. There are at least two possibilities after a failed
584  upload:
585
586  - use ``upload`` instead of ``upload_if_newer``, or
587
588  - remove the incomplete target file with ``FTPHost.remove``, then
589    use ``upload`` or ``upload_if_newer`` to transfer it again.
590
591.. _`download_if_newer`:
592
593- ``download_if_newer(source, target, callback=None)``
594
595  corresponds to ``upload_if_newer`` but performs a download from the
596  server to the local host. Read the descriptions of download and
597  ``upload_if_newer`` for more information. If a download actually
598  happened, the return value is ``True``, else ``False``.
599
600.. _`time shift`:
601.. _`time zone correction`:
602
603Time zone correction
604````````````````````
605
606If the client where ``ftputil`` runs and the server have a different
607understanding of their local times, this has to be taken into account
608for ``upload_if_newer`` and ``download_if_newer`` to work correctly.
609
610Note that even if the client and the server are in the same time zone
611(or even on the same computer), the time shift value (see below) may
612be different from zero. For example, my computer is set to use local
613time whereas the server running on the very same host insists on using
614UTC time.
615
616.. _`set_time_shift`:
617
618- ``set_time_shift(time_shift)``
619
620  sets the so-called time shift value, measured in seconds. The time
621  shift is the difference between the local time of the server and the
622  local time of the client at a given moment, i. e. by definition
623
624  ::
625
626    time_shift = server_time - client_time
627
628  Setting this value is important for `upload_if_newer`_ and
629  `download_if_newer`_ to work correctly even if the time zone of the
630  FTP server differs from that of the client. Note that the time shift
631  value *can be negative*.
632
633  If the time shift value is invalid, for example its absolute value
634  is larger than 24 hours, a ``TimeShiftError`` is raised.
635
636  See also `synchronize_times`_ for a way to set the time shift with a
637  simple method call.
638
639- ``time_shift()``
640
641  returns the currently-set time shift value. See ``set_time_shift``
642  above for its definition.
643
644.. _`synchronize_times`:
645
646- ``synchronize_times()``
647
648  synchronizes the local times of the server and the client, so that
649  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
650  if the client and the server use different time zones. For this
651  to work, *all* of the following conditions must be true:
652
653  - The connection between server and client is established.
654
655  - The client has write access to the directory that is current when
656    ``synchronize_times`` is called.
657
658  If you can't fulfill these conditions, you can nevertheless set the
659  time shift value explicitly with `set_time_shift`_. Trying to call
660  ``synchronize_times`` if the above conditions aren't met results in
661  a ``TimeShiftError`` exception.
662
663Creating and removing directories
664`````````````````````````````````
665
666- ``mkdir(path, [mode])``
667
668  makes the given directory on the remote host. This does *not*
669  construct "intermediate" directories that don't already exist. The
670  ``mode`` parameter is ignored; this is for compatibility with
671  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
672  instead of the ``os`` module. See the explanation in the subsection
673  `Exception hierarchy`_.
674
675- ``makedirs(path, [mode])``
676
677  works similar to ``mkdir`` (see above), but also makes intermediate
678  directories like ``os.makedirs``. The ``mode`` parameter is only
679  there for compatibility with ``os.makedirs`` and is ignored.
680
681- ``rmdir(path)``
682
683  removes the given remote directory. If it's not empty, raise
684  a ``PermanentError``.
685
686- ``rmtree(path, ignore_errors=False, onerror=None)``
687
688  removes the given remote, possibly non-empty, directory tree.
689  The interface of this method is rather complex, in favor of
690  compatibility with ``shutil.rmtree``.
691
692  If ``ignore_errors`` is set to a true value, errors are ignored.
693  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
694  set, all exceptions occurring during the tree iteration and
695  processing are raised. These exceptions are all of type
696  ``PermanentError``.
697
698  To distinguish between different kinds of errors, pass in a callable
699  for ``onerror``. This callable must accept three arguments:
700  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
701  object, *for example* ``your_host_object.listdir``. ``path`` is the
702  path that was the recent argument of the respective method
703  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
704  info as it is gotten from ``sys.exc_info``.
705
706  The code of ``rmtree`` is taken from Python's ``shutil`` module
707  and adapted for ``ftputil``.
708
709Removing files and links
710````````````````````````
711
712- ``remove(path)``
713
714  removes a file or link on the remote host, similar to ``os.remove``.
715
716- ``unlink(path)``
717
718  is an alias for ``remove``.
719
720Retrieving information about directories, files and links
721`````````````````````````````````````````````````````````
722
723- ``listdir(path)``
724
725  returns a list containing the names of the files and directories
726  in the given path, similar to `os.listdir`_. The special names
727  ``.`` and ``..`` are not in the list.
728
729The methods ``lstat`` and ``stat`` (and some others) rely on the
730directory listing format used by the FTP server. When connecting to a
731host, ``FTPHost``'s constructor tries to guess the right format, which
732succeeds in most cases. However, if you get strange results or
733``ParserError`` exceptions by a mere ``lstat`` call, please `file a
734bug report`_.
735
736If ``lstat`` or ``stat`` give wrong modification dates or times, look
737at the methods that deal with time zone differences (`time zone
738correction`_).
739
740.. _`FTPHost.lstat`:
741
742- ``lstat(path)``
743
744  returns an object similar to that from `os.lstat`_. This is a
745  "tuple" with additional attributes; see the documentation of the
746  ``os`` module for details.
747
748  The result is derived by parsing the output of a ``LIST`` command on
749  the server. Therefore, the result from ``FTPHost.lstat`` can not
750  contain more information than the received text. In particular:
751
752  - User and group ids can only be determined as strings, not as
753    numbers, and that only if the server supplies them. This is
754    usually the case with Unix servers but maybe not for other FTP
755    server programs.
756
757  - Values for the time of the last modification may be rough,
758    depending on the information from the server. For timestamps
759    older than a year, this usually means that the precision of the
760    modification timestamp value is not better than days. For newer
761    files, the information may be accurate to a minute.
762
763    If the time of the last modification is before the epoch (usually
764    1970-01-01 UTC), set the time of the last modification to 0.0.
765
766  - Links can only be recognized on servers that provide this
767    information in the ``LIST`` output.
768
769  - Stat attributes that can't be determined at all are set to
770        ``None``. For example, a line of a directory listing may not
771        contain the date/time of a directory's last modification.
772
773  - There's a special problem with stat'ing the root directory.
774    (Stat'ing things *in* the root directory is fine though.) In
775    this case, a ``RootDirError`` is raised. This has to do with the
776    algorithm used by ``(l)stat``, and I know of no approach which
777    mends this problem.
778
779  Currently, ``ftputil`` recognizes the common Unix-style and
780  Microsoft/DOS-style directory formats. If you need to parse output
781  from another server type, please write to the `ftputil mailing
782  list`_. You may consider `writing your own parser`_.
783
784.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
785.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
786.. _`ftputil mailing list`: http://ftputil.sschwarzer.net/mailinglist
787.. _`writing your own parser`: `Writing directory parsers`_
788
789.. _`FTPHost.stat`:
790
791- ``stat(path)``
792
793  returns ``stat`` information also for files which are pointed to by a
794  link. This method follows multiple links until a regular file or
795  directory is found. If an infinite link chain is encountered or the
796  target of the last link in the chain doesn't exist, a
797  ``PermanentError`` is raised.
798
799  The limitations of the ``lstat`` method also apply to ``stat``.
800
801.. _`FTPHost.path`:
802
803``FTPHost`` objects contain an attribute named ``path``, similar to
804`os.path`_. The following methods can be applied to the remote host
805with the same semantics as for ``os.path``:
806
807::
808
809    abspath(path)
810    basename(path)
811    commonprefix(path_list)
812    dirname(path)
813    exists(path)
814    getmtime(path)
815    getsize(path)
816    isabs(path)
817    isdir(path)
818    isfile(path)
819    islink(path)
820    join(path1, path2, ...)
821    normcase(path)
822    normpath(path)
823    split(path)
824    splitdrive(path)
825    splitext(path)
826    walk(path, func, arg)
827
828Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
829methods return ``False`` if they can't find the path given by their
830argument.
831
832Local caching of file system information
833````````````````````````````````````````
834
835Many of the above methods need access to the remote file system to
836obtain data on directories and files. To get the most recent data,
837*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
838would require to fetch a directory listing from the server, which can
839make the program *very* slow. This effect is more pronounced for
840operations which mostly scan the file system rather than transferring
841file data.
842
843For this reason, ``ftputil`` by default saves the results from
844directory listings locally and reuses those results. This reduces
845network accesses and so speeds up the software a lot. However, since
846data is more rarely fetched from the server, the risk of obsolete data
847also increases. This will be discussed below.
848
849Caching can be controlled -- if necessary at all -- via the
850``stat_cache`` object in an ``FTPHost``'s namespace. For example,
851after calling
852
853::
854
855    ftp_host = ftputil.FTPHost(host, user, password)
856
857the cache can be accessed as ``ftp_host.stat_cache``.
858
859While ``ftputil`` usually manages the cache quite well, there are two
860possible reasons for modifying cache parameters.
861
862The first is when the number of possible entries is too low. You may
863notice that when you are processing very large directories and the
864program becomes much slower than before. It's common for code to read
865a directory with ``listdir`` and then process the found directories
866and files. This can also happen implicitly by a call to
867``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
868increases the cache size if directories with more entries than the
869current maximum cache size are to be scanned. Most of the time, this
870works fine.
871
872However, if you need access to stat data for several directories at
873the same time, you may need to increase the cache explicitly. This is
874done by the ``resize`` method::
875
876    ftp_host.stat_cache.resize(20000)
877
878where the argument is the maximum number of ``lstat`` results to store
879(the default is 5000, in versions before 2.6 it was 1000). Note that
880each path on the server, e. g. "/home/schwa/some_dir", corresponds to
881a single cache entry. Methods like ``exists`` or ``getmtime`` all
882derive their results from a previously fetched ``lstat`` result.
883
884The value 5000 above means that the cache will hold *at most* 5000
885entries (unless increased automatically by an explicit or implicit
886``listdir`` call, see above). If more are about to be stored, the
887entries which haven't been used for the longest time will be deleted
888to make place for newer entries.
889
890The second possible reason to change the cache parameters is to avoid
891stale cache data. Caching is so effective because it reduces network
892accesses. This can also be a disadvantage if the file system data on
893the remote server changes after a stat result has been retrieved; the
894client, when looking at the cached stat data, will use obsolete
895information.
896
897There are two potential ways to get such out-of-date stat data. The
898first happens when an ``FTPHost`` instance modifies a file path for
899which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
900Such changes are handled transparently; the path will be deleted from
901the cache. A different matter are changes unknown to the ``FTPHost``
902object which inspects its cache. Obviously, for example, these are
903changes by programs running on the remote host. On the other hand,
904cache inconsistencies can also occur if two ``FTPHost`` objects change
905a file system simultaneously::
906
907    with (
908      ftputil.FTPHost(server, user1, password1) as ftp_host1,
909      ftputil.FTPHost(server, user1, password1) as ftp_host2
910    ):
911        stat_result1 = ftp_host1.stat("some_file")
912        stat_result2 = ftp_host2.stat("some_file")
913        ftp_host2.remove("some_file")
914        # `ftp_host1` will still see the obsolete cache entry!
915        print(ftp_host1.stat("some_file"))
916        # Will raise an exception since an `FTPHost` object
917        # knows of its own changes.
918        print(ftp_host2.stat("some_file"))
919
920At first sight, it may appear to be a good idea to have a shared cache
921among several ``FTPHost`` objects. After some thinking, this turns out
922to be very error-prone. For example, it won't help with different
923processes using ``ftputil``. So, if you have to deal with concurrent
924write/read accesses to a server, you have to handle them explicitly.
925
926The most useful tool for this is the ``invalidate`` method. In the
927example above, it could be used like this::
928
929    with (
930      ftputil.FTPHost(server, user1, password1) as ftp_host1,
931      ftputil.FTPHost(server, user1, password1) as ftp_host2
932    ):
933        stat_result1 = ftp_host1.stat("some_file")
934        stat_result2 = ftp_host2.stat("some_file")
935        ftp_host2.remove("some_file")
936        # Invalidate using an absolute path.
937        absolute_path = ftp_host1.path.abspath(
938                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
939        ftp_host1.stat_cache.invalidate(absolute_path)
940        # Will now raise an exception as it should.
941        print(ftp_host1.stat("some_file"))
942        # Would raise an exception since an `FTPHost` object
943        # knows of its own changes, even without `invalidate`.
944        print(ftp_host2.stat("some_file"))
945
946The method ``invalidate`` can be used on any *absolute* path, be it a
947directory, a file or a link.
948
949By default, the cache entries (if not replaced by newer ones) are
950stored for an infinite time. That is, if you start your Python process
951using ``ftputil`` and let it run for three days a stat call may still
952access cache data that old. To avoid this, you can set the ``max_age``
953attribute::
954
955    with ftputil.FTPHost(server, user, password) as ftp_host:
956        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
957
958This sets the maximum age of entries in the cache to an hour. This
959means any entry older won't be retrieved from the cache but its data
960instead fetched again from the remote host and then again stored for
961up to an hour. To reset `max_age` to the default of unlimited age,
962i. e. cache entries never expire, use ``None`` as value.
963
964If you are certain that the cache will be in the way, you can disable
965and later re-enable it completely with ``disable`` and ``enable``::
966
967    with ftputil.FTPHost(server, user, password) as ftp_host:
968        ftp_host.stat_cache.disable()
969        ...
970        ftp_host.stat_cache.enable()
971
972During that time, the cache won't be used; all data will be fetched
973from the network. After enabling the cache again, its entries will be
974the same as when the cache was disabled, that is, entries won't get
975updated with newer data during this period. Note that even when the
976cache is disabled, the file system data in the code can become
977inconsistent::
978
979    with ftputil.FTPHost(server, user, password) as ftp_host:
980        ftp_host.stat_cache.disable()
981        if ftp_host.path.exists("some_file"):
982            mtime = ftp_host.path.getmtime("some_file")
983
984In that case, the file ``some_file`` may have been removed by another
985process between the calls to ``exists`` and ``getmtime``!
986
987Iteration over directories
988``````````````````````````
989
990.. _`FTPHost.walk`:
991
992- ``walk(top, topdown=True, onerror=None, followlinks=False)``
993
994  iterates over a directory tree, similar to `os.walk`_. Actually,
995  ``FTPHost.walk`` uses the code from Python with just the necessary
996  modifications, so see the linked documentation.
997
998.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
999
1000.. _`FTPHost.path.walk`:
1001
1002- ``path.walk(path, func, arg)``
1003
1004  Similar to ``os.path.walk``, the ``walk`` method in
1005  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
1006  easier to use.
1007
1008Other methods
1009`````````````
1010
1011- ``close()``
1012
1013  closes the connection to the remote host. After this, no more
1014  interaction with the FTP server is possible with this ``FTPHost``
1015  object. Usually you don't need to close an ``FTPHost`` instance
1016  with ``close`` if you set up the instance in a ``with`` statement.
1017
1018- ``rename(source, target)``
1019
1020  renames the source file (or directory) on the FTP server.
1021
1022.. _`FTPHost.chmod`:
1023
1024- ``chmod(path, mode)``
1025
1026  sets the access mode (permission flags) for the given path. The mode
1027  is an integer as returned for the mode by the ``stat`` and ``lstat``
1028  methods. Be careful: Usually, mode values are written as octal
1029  numbers, for example 0755 to make a directory readable and writable
1030  for the owner, but not writable for the group and others. If you
1031  want to use such octal values, rely on Python's support for them::
1032
1033    ftp_host.chmod("some_directory", 0o755)
1034
1035  Not all FTP servers support the ``chmod`` command. In case of
1036  an exception, how do you know if the path doesn't exist or if
1037  the command itself is invalid? If the FTP server complies with
1038  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
1039  command isn't allowed. ``ftputil`` maps this special error
1040  response to a ``CommandNotImplementedError`` which is derived from
1041  ``PermanentError``.
1042
1043  So you need to code like this::
1044
1045    with ftputil.FTPHost(server, user, password) as ftp_host:
1046        try:
1047            ftp_host.chmod("some_file", 0o644)
1048        except ftputil.error.CommandNotImplementedError:
1049            # `chmod` not supported
1050            ...
1051        except ftputil.error.PermanentError:
1052            # Possibly a non-existent file
1053            ...
1054
1055  Because the ``CommandNotImplementedError`` is more specific, you
1056  have to test for it first.
1057
1058.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
1059
1060- ``copyfileobj(source, target, length=64*1024)``
1061
1062  copies the contents from the file-like object ``source`` to the
1063  file-like object ``target``. The only difference to
1064  ``shutil.copyfileobj`` is the default buffer size. Note that
1065  arbitrary file-like objects can be used as arguments (e. g. local
1066  files, remote FTP files).
1067
1068  However, the interfaces of ``source`` and ``target`` have to match;
1069  the string type read from ``source`` must be an accepted string type
1070  when written to ``target``. For example, if you open ``source`` in
1071  Python 3 as a local text file and ``target`` as a remote file object
1072  in binary mode, the transfer will fail since ``source.read`` gives
1073  unicode strings whereas ``target.write`` only accepts byte strings.
1074
1075  See `File-like objects`_ for the construction and use of remote
1076  file-like objects.
1077
1078.. _`set_parser`:
1079
1080- ``set_parser(parser)``
1081
1082  sets a custom parser for FTP directories. Note that you have to pass
1083  in a parser *instance*, not the class.
1084
1085  An `extra section`_ shows how to write own parsers if the default
1086  parsers in ``ftputil`` don't work for you.
1087
1088.. _`extra section`: `Writing directory parsers`_
1089
1090.. _`keep_alive`:
1091
1092- ``keep_alive()``
1093
1094  attempts to keep the connection to the remote server active in order
1095  to prevent timeouts from happening. This method is primarily
1096  intended to keep the underlying FTP connection of an ``FTPHost``
1097  object alive while a file is uploaded or downloaded. This will
1098  require either an extra thread while the upload or download is in
1099  progress or calling ``keep_alive`` from a `callback function`_.
1100
1101  The ``keep_alive`` method won't help if the connection has already
1102  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
1103
1104  If you want to use this method, keep in mind that FTP servers define
1105  a timeout for a reason. A timeout prevents running out of server
1106  connections because of clients that never disconnect on their own.
1107
1108  Note that the ``keep_alive`` method does *not* affect the "hidden"
1109  FTP child connections established by ``FTPHost.open`` (see section
1110  `FTPHost instances vs. FTP connections`_ for details). You *can't*
1111  use ``keep_alive`` to avoid a timeout in a stalling transfer like
1112  this::
1113
1114      with ftputil.FTPHost(server, userid, password) as ftp_host:
1115          with ftp_host.open("some_remote_file", "rb") as fobj:
1116              data = fobj.read(100)
1117              # _Futile_ attempt to avoid file connection timeout.
1118              for i in xrange(15):
1119                  time.sleep(60)
1120                  ftp_host.keep_alive()
1121              # Will raise an `ftputil.error.TemporaryError`.
1122              data += fobj.read()
1123
1124
1125.. _`FTPHost.open`:
1126
1127File-like objects
1128-----------------
1129
1130Construction
1131~~~~~~~~~~~~
1132
1133Basics
1134``````
1135
1136``FTPFile`` objects are returned by a call to ``FTPHost.open``;
1137never use the ``FTPFile`` constructor directly.
1138
1139The API of remote file-like objects are is modeled after the API of
1140the io_ module in Python 3, which has also been backported to Python
11412.6 and 2.7.
1142
1143.. _io: http://docs.python.org/library/io.html
1144
1145- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
1146  errors=None, newline=None, rest=None)``
1147
1148  returns a file-like object that refers to the path on the remote
1149  host. This path may be absolute or relative to the current directory
1150  on the remote host (this directory can be determined with the
1151  ``getcwd`` method). As with local file objects, the default mode is
1152  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
1153  "wb".
1154
1155  If a file is opened in binary mode, you *must not* specify an
1156  encoding. On the other hand, if you open a file in text mode, an
1157  encoding is used. By default, this is the return value of
1158  ``locale.getpreferredencoding``, but you can (and probably should)
1159  specify a distinct encoding.
1160
1161  If you open a file in binary mode, the read and write operations use
1162  byte strings (``str`` in Python 2, ``bytes`` in Python 3). That is,
1163  read operations return byte strings and write operations only accept
1164  byte strings.
1165
1166  Similarly, text files always work with unicode strings (``unicode``
1167  in Python 2, ``str`` in Python 3). Here, read operations return
1168  unicode strings and write operations only accept unicode strings.
1169
1170  .. warning::
1171
1172     Note that the semantics of "text mode" has changed fundamentally
1173     from ftputil 2.8 and earlier. Previously, "text mode" implied
1174     converting newline characters to ``\r\n`` when writing remote
1175     files and converting newlines to ``\n`` when reading remote
1176     files. This is in line with the "text mode" notion of FTP command
1177     line clients. Now, "text mode" follows the semantics in Python's
1178     ``io`` module.
1179
1180  The arguments ``errors`` and ``newline`` have the same semantics as
1181  in `io.open`_. The argument ``buffering`` currently is ignored.
1182  It's only there for compatibility with the ``io.open`` interface.
1183
1184  If the file is opened in binary mode, you may pass 0 or a positive
1185  integer for the ``rest`` argument. The argument is passed to the
1186  underlying FTP session instance (for example an instance of
1187  ``ftplib.FTP``) to start reading or writing at the given byte
1188  offset. For example, if a remote file contains the letters
1189  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".
1190
1191  .. warning::
1192
1193     If you pass ``rest`` values which point *after* the file, the
1194     behavior is undefined and may even differ from one FTP server to
1195     another. Therefore, use the ``rest`` argument only for error
1196     recovery in case of interrupted transfers. You need to keep track
1197     of the transferred data so that you can provide a valid ``rest``
1198     argument for a resumed transfer.
1199
1200.. _`io.open`: http://docs.python.org/library/io.html#io.open
1201
1202``FTPHost.open`` can also be used in a ``with`` statement::
1203
1204    import ftputil
1205
1206    with ftputil.FTPHost(...) as ftp_host:
1207        ...
1208        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
1209            fobj.write("This is some text.")
1210
1211At the end of the ``with`` block, the remote file will be closed
1212automatically.
1213
1214If something goes wrong during the construction of the file or in the
1215body of the ``with`` statement, the file will be closed as well.
1216Exceptions will be propagated as with ``try ... finally``.
1217
1218Attributes and methods
1219~~~~~~~~~~~~~~~~~~~~~~
1220
1221The methods
1222
1223::
1224
1225    close()
1226    read([count])
1227    readline([count])
1228    readlines()
1229    write(data)
1230    writelines(string_sequence)
1231
1232and the attribute ``closed`` have the same semantics as for file
1233objects of a local disk file system. The iterator protocol is
1234supported as well, i. e. you can use a loop to read a file line by
1235line::
1236
1237    with ftputil.FTPHost(server, user, password) as ftp_host:
1238        with ftp_host.open("some_file") as input_file:
1239            for line in input_file:
1240                # Do something with the line, e. g.
1241                print(line.strip().replace("ftplib", "ftputil"))
1242
1243For more on file objects, see the section `File objects`_ in the
1244Python Library Reference.
1245
1246.. _`file objects`: https://docs.python.org/2.7/library/stdtypes.html#file-objects
1247
1248
1249.. _`child_connections`:
1250
1251``FTPHost`` instances vs. FTP connections
1252-----------------------------------------
1253
1254This section explains why keeping an ``FTPHost`` instance "alive"
1255without timing out sometimes isn't trivial. If you always finish your
1256FTP operations in time, you don't need to read this section.
1257
1258The file transfer protocol is a stateful protocol. That means an FTP
1259connection always is in a certain state. Each of these states can only
1260change to certain other states under certain conditions triggered by
1261the client or the server.
1262
1263One of the consequences is that a single FTP connection can't be used
1264at the same time, say, to transfer data on the FTP data channel and to
1265create a directory on the remote host.
1266
1267For example, consider this::
1268
1269    >>> import ftplib
1270    >>> ftp = ftplib.FTP(server, user, password)
1271    >>> ftp.pwd()
1272    '/'
1273    >>> # Start transfer. `CONTENTS` is a text file on the server.
1274    >>> socket = ftp.transfercmd("RETR CONTENTS")
1275    >>> socket
1276    <socket._socketobject object at 0x7f801a6386e0>
1277    >>> ftp.pwd()
1278    Traceback (most recent call last):
1279      File "<stdin>", line 1, in <module>
1280      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
1281        return parse257(resp)
1282      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
1283        raise error_reply, resp
1284    ftplib.error_reply: 226-File successfully transferred
1285    226 0.000 seconds (measured here), 5.60 Mbytes per second
1286    >>>
1287
1288Note that ``ftp`` is a single FTP connection, represented by an
1289``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
1290
1291On the other hand, consider this::
1292
1293    >>> import ftputil
1294    >>> ftp_host = ftputil.FTPHost(server, user, password)
1295    >>> ftp_host.getcwd()
1296    >>> fobj = ftp_host.open("CONTENTS")
1297    >>> fobj
1298    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
1299    >>> ftp_host.getcwd()
1300    u'/'
1301    >>> fobj.readline()
1302    u'Contents of FTP test directory\n'
1303    >>> fobj.close()
1304    >>>
1305
1306To be able to start a file transfer (i. e. open a remote file for
1307reading or writing) and still be able to use other FTP commands,
1308ftputil uses a trick. For every remote file, ftputil creates a new FTP
1309connection, called a child connection in the ftputil source code.
1310(Actually, FTP connections belonging to closed remote files are
1311re-used if they haven't timed out yet.)
1312
1313In most cases this approach isn't noticeable by code using ftputil.
1314However, the nice abstraction of dealing with a single FTP connection
1315falls apart if one of the child connections times out. For example, if
1316you open a remote file and work only with the initial "main"
1317connection to navigate the file system, the FTP connection for the
1318remote file may eventually time out.
1319
1320While it's often relatively easy to prevent the "main" connection from
1321timing out it's unfortunately practically impossible to do this for a
1322remote file connection (apart from transferring some data, of course).
1323For this reason, `FTPHost.keep_alive`_ affects only the main
1324connection. Child connections may still time out if they're idle for
1325too long.
1326
1327.. _`FTPHost.keep_alive`: `keep_alive`_
1328
1329Some more details:
1330
1331- A kind of "straightforward" way of keeping the main connection alive
1332  would be to call ``ftp_host.getcwd()``. However, this doesn't work
1333  because ftputil caches the current directory and returns it without
1334  actually contacting the server. That's the main reason why there's
1335  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
1336  (i. e. the session object), which isn't a public attribute.
1337
1338- Some servers define not only an idle timeout but also a transfer
1339  timeout. This means the connection times out unless there's some
1340  transfer on the data channel for this connection. So ftputil's
1341  ``keep_alive`` doesn't prevent this timeout, but an
1342  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
1343  this transfers the data for the whole directory listing which might
1344  take some time if the directory has many entries.
1345
1346Bottom line: If you can, you should organize your FTP actions so that
1347you finish everything before a timeout happens.
1348
1349
1350Writing directory parsers
1351-------------------------
1352
1353``ftputil`` recognizes the two most widely-used FTP directory formats,
1354Unix and MS style, and adjusts itself automatically. Almost every FTP
1355server uses one of these formats.
1356
1357However, if your server uses a format which is different from the two
1358provided by ``ftputil``, you can plug in a custom parser with a single
1359method call and have ``ftputil`` use this parser.
1360
1361For this, you need to write a parser class by inheriting from the
1362class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
1363
1364    import ftputil.error
1365    import ftputil.stat
1366
1367    class XyzParser(ftputil.stat.Parser):
1368        """
1369        Parse the default format of the FTP server of the XYZ
1370        corporation.
1371        """
1372
1373        def parse_line(self, line, time_shift=0.0):
1374            """
1375            Parse a `line` from the directory listing and return a
1376            corresponding `StatResult` object. If the line can't
1377            be parsed, raise `ftputil.error.ParserError`.
1378
1379            The `time_shift` argument can be used to fine-tune the
1380            parsing of dates and times. See the class
1381            `ftputil.stat.UnixParser` for an example.
1382            """
1383            # Split the `line` argument and examine it further; if
1384            # something goes wrong, raise an `ftputil.error.ParserError`.
1385            ...
1386            # Make a `StatResult` object from the parts above.
1387            stat_result = ftputil.stat.StatResult(...)
1388            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
1389            stat_result._st_name = ...
1390            stat_result._st_target = ...
1391            stat_result._st_mtime_precision = ...
1392            return stat_result
1393
1394        # Define `ignores_line` only if the default in the base class
1395        # doesn't do enough!
1396        def ignores_line(self, line):
1397            """
1398            Return a true value if the line should be ignored. For
1399            example, the implementation in the base class handles
1400            lines like "total 17". On the other hand, if the line
1401            should be used for stat'ing, return a false value.
1402            """
1403            is_total_line = super(XyzParser, self).ignores_line(line)
1404            my_test = ...
1405            return is_total_line or my_test
1406
1407A ``StatResult`` object is similar to the value returned by
1408`os.stat`_ and is usually built with statements like
1409
1410::
1411
1412    stat_result = StatResult(
1413                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
1414                     st_gid, st_size, st_atime, st_mtime, st_ctime))
1415    stat_result._st_name = ...
1416    stat_result._st_target = ...
1417    stat_result._st_mtime_precision = ...
1418
1419with the arguments of the ``StatResult`` constructor described in
1420the following table.
1421
1422===== =================== ============ =================== =======================
1423Index Attribute           os.stat type ``StatResult`` type Notes
1424===== =================== ============ =================== =======================
14250     st_mode             int          int
14261     st_ino              long         long
14272     st_dev              long         long
14283     st_nlink            int          int
14294     st_uid              int          str                 usually only available as string
14305     st_gid              int          str                 usually only available as string
14316     st_size             long         long
14327     st_atime            int/float    float
14338     st_mtime            int/float    float
14349     st_ctime            int/float    float
1435\-    _st_name            \-           str                 file name without directory part
1436\-    _st_target          \-           str                 link target (may be absolute or relative)
1437\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
1438===== =================== ============ =================== =======================
1439
1440If you can't extract all the desirable data from a line (for
1441example, the MS format doesn't contain any information about the
1442owner of a file), set the corresponding values in the ``StatResult``
1443instance to ``None``.
1444
1445Parser classes can use several helper methods which are defined in
1446the class ``Parser``:
1447
1448- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
1449  an appropriate ``st_mode`` integer value.
1450
1451- ``parse_unix_time`` returns a float number usable for the
1452  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
1453  "May"/"26"/"2005". Note that the method expects the timestamp string
1454  already split at whitespace.
1455
1456- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
1457  returns a float number like from ``time.mktime``. Note that the
1458  method expects the timestamp string already split at whitespace.
1459
1460Additionally, there's an attribute ``_month_numbers`` which maps
1461lowercase three-letter month abbreviations to integers.
1462
1463For more details, see the two "standard" parsers ``UnixParser`` and
1464``MSParser`` in the module ``ftputil/stat.py``.
1465
1466To actually *use* the parser, call the method `set_parser`_ of the
1467``FTPHost`` instance.
1468
1469If you can't write a parser or don't want to, please ask on the
1470`ftputil mailing list`_. Possibly someone has already written a parser
1471for your server or can help with it.
1472
1473
1474FAQ / Tips and tricks
1475---------------------
1476
1477Where can I get the latest version?
1478~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1479
1480See the `download page`_. Announcements will be sent to the `mailing
1481list`_. Announcements on major updates will also be posted to the
1482newsgroup `comp.lang.python.announce`_ .
1483
1484.. _`download page`: http://ftputil.sschwarzer.net/download
1485.. _`mailing list`: http://ftputil.sschwarzer.net/mailinglist
1486.. _`comp.lang.python.announce`: news:comp.lang.python.announce
1487
1488Is there a mailing list on ``ftputil``?
1489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1490
1491Yes, please visit http://ftputil.sschwarzer.net/mailinglist to
1492subscribe or read the archives.
1493
1494Though you can *technically* post without subscribing first I can't
1495recommend it: The mails from non-subscribers have to be approved by
1496me and because the arriving mails contain *lots* of spam, I rarely go
1497through these mails.
1498
1499I found a bug! What now?
1500~~~~~~~~~~~~~~~~~~~~~~~~
1501
1502Before reporting a bug, make sure that you already read this manual
1503and tried the `latest version`_ of ``ftputil``. There the bug might
1504have already been fixed.
1505
1506.. _`latest version`: http://ftputil.sschwarzer.net/download
1507
1508Please see http://ftputil.sschwarzer.net/issuetrackernotes for
1509guidelines on entering a bug in ``ftputil``'s ticket system. If you
1510are unsure if the behaviour you found is a bug or not, you should write
1511to the `ftputil mailing list`_. In *either* case you *must not*
1512include confidential information (user id, password, file names, etc.)
1513in the problem report! Be careful!
1514
1515Does ``ftputil`` support SSL/TLS?
1516~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1517
1518``ftputil`` has no *built-in* SSL/TLS support.
1519
1520On the other hand, there are two ways to get TLS support with
1521ftputil:
1522
1523- In Python 2.7 and Python 3.2 and up, the ``ftplib`` library has a
1524  class ``FTP_TLS`` that you can use for the ``session_factory``
1525  keyword argument in the ``FTPHost`` constructor. You can't use the
1526  class directly though if you need additional setup code in
1527  comparison to ``ftplib.FTP``, for example calling ``prot_p``, to
1528  secure the data connection. On the other hand,
1529  `ftputil.session.session_factory`_ can be used to create a custom
1530  session factory.
1531 
1532  If you have other requirements that ``session_factory`` can't
1533  fulfill, you may create your own session factory by inheriting from
1534  ``ftplib.FTP_TLS``::
1535
1536    import ftplib
1537
1538    import ftputil
1539
1540
1541    class FTPTLSSession(ftplib.FTP_TLS):
1542
1543        def __init__(self, host, user, password):
1544            ftplib.FTP_TLS.__init__(self)
1545            self.connect(host, port)
1546            self.login(user, password)
1547            # Set up encrypted data connection.
1548            self.prot_p()
1549            ...
1550
1551    # Note the `session_factory` parameter. Pass the class, not
1552    # an instance.
1553    with ftputil.FTPHost(server, user, password,
1554                         session_factory=FTPTLSSession) as ftp_host:
1555        # Use `ftp_host` as usual.
1556        ...
1557
1558.. _`ftputil.session.session_factory`: `Session factories`_
1559
1560- If you need to work with Python 2.6, you can use the
1561  ``ftpslib.FTP_TLS`` class from the M2Crypto_ project. Again, you
1562  can't use the class directly but need to use
1563  ``ftputil.session.session_factory`` or a recipe similar to that
1564  above.
1565
1566  Unfortunately, ``M2Crypto.ftpslib.FTP_TLS`` (at least in version
1567  0.22.3) doesn't work correctly if you pass unicode strings to its
1568  methods. Since ``ftputil`` does exactly that at some point (even if
1569  you used byte strings in ``ftputil`` calls) you need a workaround in
1570  the session factory class::
1571
1572    import M2Crypto
1573
1574    import ftputil
1575    import ftputil.tool
1576
1577
1578    class M2CryptoSession(M2Crypto.ftpslib.FTP_TLS):
1579
1580        def __init__(self, host, user, password):
1581            M2Crypto.ftpslib.FTP_TLS.__init__(self)
1582            # Change the port number if needed.
1583            self.connect(host, 21)
1584            self.auth_tls()
1585            self.login(user, password)
1586            self.prot_p()
1587            self._fix_socket()
1588            ...
1589
1590        def _fix_socket(self):
1591            """
1592            Change the socket object so that arguments to `sendall`
1593            are converted to byte strings before being used.
1594            """
1595            original_sendall = self.sock.sendall
1596            # Bound method, therefore no `self` argument.
1597            def sendall(data):
1598                data = ftputil.tool.as_bytes(data)
1599                return original_sendall(data)
1600            self.sock.sendall = sendall
1601
1602    # Note the `session_factory` parameter. Pass the class, not
1603    # an instance.
1604    with ftputil.FTPHost(server, user, password,
1605                         session_factory=M2CryptoSession) as ftp_host:
1606        # Use `ftp_host` as usual.
1607        ...
1608
1609  That said, ``session_factory`` has this workaround built in, so
1610  normally you don't need to define the session factory yourself!
1611
1612.. _M2Crypto: https://github.com/martinpaljak/M2Crypto
1613
1614How do I connect to a non-default port?
1615~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1616
1617By default, an instantiated ``FTPHost`` object connects on the usual
1618FTP port. If you have to use a different port, refer to the section
1619`Session factories`_.
1620
1621How do I set active or passive mode?
1622~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1623
1624Please see the section `Session factories`_.
1625
1626How can I debug an FTP connection problem?
1627~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1628
1629You can do this with a session factory. See `Session factories`_.
1630
1631If you want to change the debug level only temporarily after the
1632connection is established, you can reach the `session object`_ as the
1633``_session`` attribute of the ``FTPHost`` instance and call
1634``_session.set_debuglevel``. Note that the ``_session`` attribute
1635should *only* be accessed for debugging. Calling arbitrary
1636``ftplib.FTP`` methods on the session object may *cause* bugs!
1637
1638.. _`session object`: `Session factories`_
1639
1640Conditional upload/download to/from a server in a different time zone
1641~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1642
1643You may find that ``ftputil`` uploads or downloads files
1644unnecessarily, or not when it should. This can happen when the FTP
1645server is in a different time zone than the client on which
1646``ftputil`` runs. Please see the section on `time zone correction`_.
1647It may even be sufficient to call `synchronize_times`_.
1648
1649When I use ``ftputil``, all I get is a ``ParserError`` exception
1650~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1651
1652The FTP server you connect to may use a directory format that
1653``ftputil`` doesn't understand. You can either write and
1654`plug in an own parser`_ or ask on the `mailing list`_ for
1655help.
1656
1657.. _`plug in an own parser`: `Writing directory parsers`_
1658
1659``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
1660~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1661
1662Like Python's counterparts under `os.path`_, ``ftputil``'s methods
1663return ``False`` if they can't find the given path.
1664
1665Probably you used ``listdir`` on a directory and called ``is...()`` on
1666the returned names. But if the argument for ``listdir`` wasn't the
1667current directory, the paths won't be found and so all ``is...()``
1668variants will return ``False``.
1669
1670I don't find an answer to my problem in this document
1671~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1672
1673Please send an email with your problem report or question to the
1674`ftputil mailing list`_, and we'll see what we can do for you. :-)
1675
1676
1677Bugs and limitations
1678--------------------
1679
1680- ``ftputil`` needs at least Python 2.6 to work.
1681
1682- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
1683  names starting with a dot) depends on the configuration of the FTP
1684  server. See `Hidden files and directories`_ for details.
1685
1686- Due to the implementation of ``lstat`` it can not return a sensible
1687  value for the root directory ``/`` though stat'ing entries *in* the
1688  root directory isn't a problem. If you know an implementation that
1689  can do this, please let me know. The root directory is handled
1690  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
1691
1692- In multithreaded programs, you can have each thread use one or more
1693  ``FTPHost`` instances as long as no instance is shared with other
1694  threads.
1695
1696- Currently, it is not possible to continue an interrupted upload or
1697  download. Contact me if this causes problems for you.
1698
1699- There's exactly one cache for ``lstat`` results for each ``FTPHost``
1700  object, i. e. there's no sharing of cache results determined by
1701  several ``FTPHost`` objects. See `Local caching of file system
1702  information`_ for the reasons.
1703
1704
1705Files
1706-----
1707
1708If not overwritten via installation options, the ``ftputil`` files
1709reside in the ``ftputil`` package. There's also documentation in
1710`reStructuredText`_ and in HTML format. The locations of these
1711files after installation is system-dependent.
1712
1713.. _`reStructuredText`: http://docutils.sourceforge.net/rst.html
1714
1715The files ``test_*.py`` and ``mock_ftplib.py`` are for unit-testing.
1716If you only *use* ``ftputil``, i. e. *don't* modify it, you can
1717delete these files.
1718
1719
1720References
1721----------
1722
1723- Mackinnon T, Freeman S, Craig P. 2000. `Endo-Testing:
1724  Unit Testing with Mock Objects`_.
1725
1726- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
1727
1728- Van Rossum G et al. 2013. `Python Library Reference`_.
1729
1730.. _`Endo-Testing: Unit Testing with Mock Objects`:
1731   http://www.connextra.com/aboutUs/mockobjects.pdf
1732.. _`RFC 959 - File Transfer Protocol (FTP)`: http://www.ietf.org/rfc/rfc959.txt
1733.. _`Python Library Reference`: https://docs.python.org/library/index.html
1734
1735
1736Authors
1737-------
1738
1739``ftputil`` is written by Stefan Schwarzer
1740<sschwarzer@sschwarzer.net> and contributors (see
1741``doc/contributors.txt``).
1742
1743The original ``lrucache`` module was written by Evan Prodromou
1744<evan@prodromou.name>.
1745
1746Feedback is appreciated. :-)
Note: See TracBrowser for help on using the repository browser.