source: doc/ftputil.txt @ 1593:6b49dedd1dcf

Last change on this file since 1593:6b49dedd1dcf was 1593:6b49dedd1dcf, checked in by Stefan Schwarzer <sschwarzer@…>, 6 years ago
Add example for using `FTPHost.operation` vs. `os.operation` This should help with ticket #92.
File size: 60.0 KB
Line 
1``ftputil`` -- a high-level FTP client library
2==============================================
3
4:Version:   3.2
5:Date:      2014-10-12
6:Summary:   high-level FTP client library for Python
7:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
8:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>
9
10.. contents::
11
12
13Introduction
14------------
15
16The ``ftputil`` module is a high-level interface to the ftplib_
17module. The `FTPHost objects`_ generated from it allow many operations
18similar to those of os_, `os.path`_ and `shutil`_.
19
20.. _ftplib: https://docs.python.org/library/ftplib.html
21.. _os: https://docs.python.org/library/os.html
22.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
23.. _`os.path`: https://docs.python.org/library/os.path.html
24.. _`shutil`: https://docs.python.org/library/shutil.html
25
26Example::
27
28    import ftputil
29
30    # Download some files from the login directory.
31    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
32        names = ftp_host.listdir(ftp_host.curdir)
33        for name in names:
34            if ftp_host.path.isfile(name):
35                ftp_host.download(name, name)  # remote, local
36        # Make a new directory and copy a remote file into it.
37        ftp_host.mkdir("newdir")
38        with ftp_host.open("index.html", "rb") as source:
39            with ftp_host.open("newdir/index.html", "wb") as target:
40                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj
41
42Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
43modification time of a file. The latter can also follow links, similar
44to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.
45
46
47``ftputil`` features
48--------------------
49
50* Method names are familiar from Python's ``os``, ``os.path`` and
51  ``shutil`` modules. For example, use ``os.path.join`` to join
52  paths for a local file system and ``ftp_host.path.join`` to join
53  paths for a remote FTP file system.
54
55* Remote file system navigation (``getcwd``, ``chdir``)
56
57* Upload and download files (``upload``, ``upload_if_newer``,
58  ``download``, ``download_if_newer``)
59
60* Time zone synchronization between client and server (needed
61  for ``upload_if_newer`` and ``download_if_newer``)
62
63* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
64  ``rmtree``) and remove files (``remove``)
65
66* Get information about directories, files and links (``listdir``,
67  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
68  ``abspath``, ``split``, ``join``, ``dirname``, ``basename`` etc.)
69
70* Iterate over remote file systems (``walk``)
71
72* Local caching of results from ``lstat`` and ``stat`` calls to reduce
73  network access (also applies to ``exists``, ``getmtime`` etc.).
74
75* Read files from and write files to remote hosts via
76  file-like objects (``FTPHost.open``; the generated file-like objects
77  have the familiar methods like ``read``, ``readline``, ``readlines``,
78  ``write``, ``writelines`` and ``close``. You can also iterate over
79  these files line by line in a ``for`` loop.
80
81
82Exception hierarchy
83-------------------
84
85The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
86``ftputil.error.TemporaryError``.
87
88The exception classes are organized as follows::
89
90    FTPError
91        FTPOSError(FTPError, OSError)
92            PermanentError(FTPOSError)
93                CommandNotImplementedError(PermanentError)
94            TemporaryError(FTPOSError)
95        FTPIOError(FTPError)
96        InternalError(FTPError)
97            InaccessibleLoginDirError(InternalError)
98            ParserError(InternalError)
99            RootDirError(InternalError)
100            TimeShiftError(InternalError)
101
102and are described here:
103
104- ``FTPError``
105
106  is the root of the exception hierarchy of the module.
107
108- ``FTPOSError``
109
110  is derived from ``OSError``. This is for similarity between the
111  os module and ``FTPHost`` objects. Compare
112
113  ::
114
115    try:
116        os.chdir("nonexisting_directory")
117    except OSError:
118        ...
119
120  with
121
122  ::
123
124    host = ftputil.FTPHost("host", "user", "password")
125    try:
126        host.chdir("nonexisting_directory")
127    except OSError:
128        ...
129
130  Imagine a function
131
132  ::
133
134    def func(path, file):
135        ...
136
137  which works on the local file system and catches ``OSErrors``. If you
138  change the parameter list to
139
140  ::
141
142    def func(path, file, os=os):
143        ...
144
145  where ``os`` denotes the ``os`` module, you can call the function also as
146
147  ::
148
149    host = ftputil.FTPHost("host", "user", "password")
150    func(path, file, os=host)
151
152  to use the same code for both a local and remote file system.
153  Another similarity between ``OSError`` and ``FTPOSError`` is that
154  the latter holds the FTP server return code in the ``errno``
155  attribute of the exception object and the error text in
156  ``strerror``.
157
158- ``PermanentError``
159
160  is raised for 5xx return codes from the FTP server. This
161  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
162  ``ftplib.error_perm`` are *not* identical).
163
164- ``CommandNotImplementedError``
165
166  indicates that an underlying command the code tries to use is not
167  implemented. For an example, see the description of the
168  `FTPHost.chmod`_ method.
169
170- ``TemporaryError``
171
172  is raised for FTP return codes from the 4xx category. This
173  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
174  ``ftplib.error_temp`` are *not* identical).
175
176- ``FTPIOError``
177
178  denotes an I/O error on the remote host. This appears
179  mainly with file-like objects that are retrieved by calling
180  ``FTPHost.open``. Compare
181
182  ::
183
184    >>> try:
185    ...     f = open("not_there")
186    ... except IOError as obj:
187    ...     print obj.errno
188    ...     print obj.strerror
189    ...
190    2
191    No such file or directory
192
193  with
194
195  ::
196
197    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
198    >>> try:
199    ...     f = ftp_host.open("not_there")
200    ... except IOError as obj:
201    ...     print obj.errno
202    ...     print obj.strerror
203    ...
204    550
205    550 not_there: No such file or directory.
206
207  As you can see, both code snippets are similar. However, the error
208  codes aren't the same.
209
210- ``InternalError``
211
212  subsumes exception classes for signaling errors due to limitations
213  of the FTP protocol or the concrete implementation of ``ftputil``.
214
215- ``InaccessibleLoginDirError``
216
217  This exception is raised if the directory in which "you" are placed
218  upon login is not accessible, i. e. a ``chdir`` call with the
219  directory as argument would fail.
220
221- ``ParserError``
222
223  is used for errors during the parsing of directory
224  listings from the server. This exception is used by the ``FTPHost``
225  methods ``stat``, ``lstat``, and ``listdir``.
226
227- ``RootDirError``
228
229  Because of the implementation of the ``lstat`` method it is not
230  possible to do a ``stat`` call on the root directory ``/``.
231  If you know *any* way to do it, please let me know. :-)
232
233  This problem does *not* affect stat calls on items *in* the root
234  directory.
235
236- ``TimeShiftError``
237
238  is used to denote errors which relate to setting the `time shift`_,
239  *for example* trying to set a value which is no multiple of a full
240  hour.
241
242
243Directory and file names
244------------------------
245
246Methods that take names of directories and files can take either byte
247strings (``str`` on Python 2, ``bytes`` on Python 3) or unicode
248strings (``unicode`` on Python 2, ``str`` on Python 3).
249
250Byte strings will be sent to the FTP server as-is. Unicode strings
251will be encoded with the encoding returned from
252``locale.getpreferredencoding``. This is the same semantics as for
253locally used names in Python 3.
254
255Methods that take and return a directory or file name will return the
256same string type as they're given. For example, if the argument to
257``FTPHost.path.abspath`` is a byte string, you'll get a byte string
258back. This behavior is the same as for the local file system API in
259Python 2 and 3.
260
261
262``FTPHost`` objects
263-------------------
264
265.. _`FTPHost construction`:
266
267Construction
268~~~~~~~~~~~~
269
270Basics
271``````
272
273``FTPHost`` instances can be generated with the following call::
274
275    ftp_host = ftputil.FTPHost(server, user, password, account,
276                               session_factory=ftplib.FTP)
277
278The first four parameters are strings with the same meaning as for the
279FTP class in the ``ftplib`` module.
280
281``FTPHost`` objects can also be used in a ``with`` statement::
282
283    import ftputil
284
285    with ftputil.FTPHost(server, user, password) as ftp_host:
286        print ftp_host.listdir(ftp_host.curdir)
287
288After the ``with`` block, the ``FTPHost`` instance and the
289associated FTP sessions will be closed automatically.
290
291If something goes wrong during the ``FTPHost`` construction or in the
292body of the ``with`` statement, the instance is closed as well.
293Exceptions will be propagated (as with ``try ... finally``).
294
295Session factories
296`````````````````
297
298The keyword argument ``session_factory`` may be used to generate FTP
299connections with other factories than the default ``ftplib.FTP``. For
300example, the standard library of Python 2.7 contains a class
301``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
302connection.
303
304In fact, all positional and keyword arguments other than
305``session_factory`` are passed to the factory to generate a new
306background session. This also happens for every remote file that is
307opened; see below.
308
309This functionality of the constructor also allows to wrap
310``ftplib.FTP`` objects to do something that wouldn't be possible with
311the ``ftplib.FTP`` constructor alone.
312
313As an example, assume you want to connect to another than the default
314port, but ``ftplib.FTP`` only offers this by means of its ``connect``
315method, not via its constructor. One solution is to use a custom
316class as a session factory::
317
318    import ftplib
319    import ftputil
320
321    EXAMPLE_PORT = 50001
322
323    class MySession(ftplib.FTP):
324
325        def __init__(self, host, userid, password, port):
326            """Act like ftplib.FTP's constructor but connect to another port."""
327            ftplib.FTP.__init__(self)
328            self.connect(host, port)
329            self.login(userid, password)
330
331    # Try _not_ to use an _instance_ `MySession()` as factory, -
332    # use the class itself.
333    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
334                         session_factory=MySession) as ftp_host:
335        # Use `ftp_host` as usual.
336        ...
337
338On login, the format of the directory listings (needed for stat'ing
339files and directories) should be determined automatically. If not,
340please `file a bug report`_.
341
342.. _`file a bug report`: http://ftputil.sschwarzer.net/issuetrackernotes
343
344For the most common uses you don't need to create your own session
345factory class though. The ``ftputil.session`` module has a function
346``session_factory`` that can create session factories for a variety
347of parameters::
348
349    session_factory(base_class=ftplib.FTP,
350                    port=21,
351                    use_passive_mode=None,
352                    encrypt_data_channel=True,
353                    debug_level=None)
354
355with
356
357- ``base_class`` is a base class to inherit a new session factory
358  class from. By default, this is ``ftplib.FTP`` from the Python
359  standard library.
360
361- ``port`` is the command channel port. The default is 21, used in most
362  FTP server configurations.
363
364- ``use_passive_mode`` is either a boolean that determines whether
365  passive mode should be used or ``None``. ``None`` means to let the
366  base class choose active or passive mode.
367
368- ``encrypt_data_channel`` defines whether to encrypt the data channel
369  for secure connections. This is only supported for the base classes
370  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
371  the parameter is ignored.
372
373- ``debug_level`` sets the debug level for FTP session instances. The
374  semantics is defined by the base class. For example, a debug level
375  of 2 causes the most verbose output for Python's ``ftplib.FTP``
376  class.
377
378All of these parameters can be combined. For example, you could use
379
380::
381
382    import ftplib
383
384    import ftputil
385    import ftputil.session
386
387
388    my_session_factory = ftputil.session.session_factory(
389                           base_class=ftpslib.FTP_TLS,
390                           port=31,
391                           encrypt_data_channel=True,
392                           debug_level=2)
393
394    with ftputil.FTPHost(server, user, password,
395                         session_factory=my_session_factory) as ftp_host:
396        ...
397
398to create and use a session factory derived from ``ftplib.FTP_TLS``
399that connects on command channel 31, will encrypt the data channel and
400print output for debug level 2.
401
402Note: Generally, you can achieve everything you can do with
403``ftputil.session.session_factory`` with an explicit session factory
404as described at the start of this section. However, the class
405``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
406it with ftputil out of the box. The function ``session_factory``
407contains a workaround for this limitation. For details refer to `this
408bug report`_.
409
410.. _`this bug report`: http://ftputil.sschwarzer.net/trac/ticket/78
411
412Hidden files and directories
413~~~~~~~~~~~~~~~~~~~~~~~~~~~~
414
415Whether ftputil sees "hidden" files and directories (usually files or
416directories whose names start with a dot) depends on the FTP server
417configuration. By default, ftputil uses the ``-a`` option in the FTP
418``LIST`` command to find hidden files. However, the server may ignore
419this.
420
421If using the ``-a`` option leads to problems, for example if an
422FTP server causes an exception, you may switch off the use of the
423option::
424
425    ftp_host = ftputil.FTPHost(server, user, password, account,
426                               session_factory=ftplib.FTP)
427    ftp_host.use_list_a_option = False
428
429``FTPHost`` attributes and methods
430~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
431
432Attributes
433``````````
434
435- ``curdir``, ``pardir``, ``sep``
436
437  are strings which denote the current and the parent directory on the
438  remote server. ``sep`` holds the path separator. Though `RFC 959`_
439  (File Transfer Protocol) notes that these values may depend on the
440  FTP server implementation, the Unix variants seem to work well in
441  practice, even for non-Unix servers.
442
443  Nevertheless, it's recommended that you don't hardcode these values
444  for remote paths, but use `FTPHost.path`_ as you would use
445  ``os.path`` to write platform-independent Python code for local
446  filesystems. Keep in mind that most, *but not all*, arguments of
447  ``FTPHost`` methods refer to remote directories or files. For
448  example, in `FTPHost.upload`_, the first argument is a local
449  path and the second a remote path. Both of these should use their
450  respective path separators.
451
452.. _`FTPHost.upload`: `Uploading and downloading files`_
453
454Remote file system navigation
455`````````````````````````````
456
457- ``getcwd()``
458
459  returns the absolute current directory on the remote host. This
460  method works like ``os.getcwd``.
461
462- ``chdir(directory)``
463
464  sets the current directory on the FTP server. This resembles
465  ``os.chdir``, as you may have expected.
466
467.. _`callback function`:
468
469Uploading and downloading files
470```````````````````````````````
471
472- ``upload(source, target, callback=None)``
473
474  copies a local source file (given by a filename, i. e. a string)
475  to the remote host under the name target. Both ``source`` and
476  ``target`` may be absolute paths or relative to their corresponding
477  current directory (on the local or the remote host, respectively).
478
479  The file content is always transferred in binary mode.
480
481  The callback, if given, will be invoked for each transferred chunk
482  of data::
483
484    callback(chunk)
485
486  where ``chunk`` is a bytestring. An example usage of a callback
487  method is to display a progress indicator.
488
489- ``download(source, target, callback=None)``
490
491  performs a download from the remote source file to a local target
492  file. Both ``source`` and ``target`` are strings. See the
493  description of ``upload`` for more details.
494
495.. _`upload_if_newer`:
496
497- ``upload_if_newer(source, target, callback=None)``
498
499  is similar to the ``upload`` method. The only difference is that the
500  upload is only invoked if the time of the last modification for the
501  source file is more recent than that of the target file or the
502  target doesn't exist at all. The check for the last modification
503  time considers the precision of the timestamps and transfers a file
504  "if in doubt". Consequently the code
505
506  ::
507
508    ftp_host.upload_if_newer("source_file", "target_file")
509    time.sleep(10)
510    ftp_host.upload_if_newer("source_file", "target_file")
511
512  might upload the file again if the timestamp of the target file is
513  precise up to a minute, which is typically the case because the
514  remote datetime is determined by parsing a directory listing from
515  the server. To avoid unnecessary transfers, wait at least a minute
516  between calls of ``upload_if_newer`` for the same file. If it still
517  seems that a file is uploaded unnecessarily (or not when it should),
518  read the subsection on `time shift`_ settings.
519
520  If an upload actually happened, the return value of
521  ``upload_if_newer`` is a ``True``, else ``False``.
522
523  Note that the method only checks the existence and/or the
524  modification time of the source and target file; it doesn't
525  compare any other file properties, say, the file size.
526
527  This also means that if a transfer is interrupted, the remote file
528  will have a newer modification time than the local file, and thus
529  the transfer won't be repeated if ``upload_if_newer`` is used a
530  second time. There are at least two possibilities after a failed
531  upload:
532
533  - use ``upload`` instead of ``upload_if_newer``, or
534
535  - remove the incomplete target file with ``FTPHost.remove``, then
536    use ``upload`` or ``upload_if_newer`` to transfer it again.
537
538.. _`download_if_newer`:
539
540- ``download_if_newer(source, target, callback=None)``
541
542  corresponds to ``upload_if_newer`` but performs a download from the
543  server to the local host. Read the descriptions of download and
544  ``upload_if_newer`` for more information. If a download actually
545  happened, the return value is ``True``, else ``False``.
546
547.. _`time shift`:
548.. _`time zone correction`:
549
550Time zone correction
551````````````````````
552
553If the client where ``ftputil`` runs and the server have a different
554understanding of their local times, this has to be taken into account
555for ``upload_if_newer`` and ``download_if_newer`` to work correctly.
556
557Note that even if the client and the server are in the same time zone
558(or even on the same computer), the time shift value (see below) may
559be different from zero. For example, my computer is set to use local
560time whereas the server running on the very same host insists on using
561UTC time.
562
563.. _`set_time_shift`:
564
565- ``set_time_shift(time_shift)``
566
567  sets the so-called time shift value, measured in seconds. The time
568  shift is the difference between the local time of the server and the
569  local time of the client at a given moment, i. e. by definition
570
571  ::
572
573    time_shift = server_time - client_time
574
575  Setting this value is important for `upload_if_newer`_ and
576  `download_if_newer`_ to work correctly even if the time zone of the
577  FTP server differs from that of the client. Note that the time shift
578  value *can be negative*.
579
580  If the time shift value is invalid, e. g. no multiple of a full hour
581  or its absolute value larger than 24 hours, a ``TimeShiftError`` is
582  raised.
583
584  See also `synchronize_times`_ for a way to set the time shift with a
585  simple method call.
586
587- ``time_shift()``
588
589  returns the currently-set time shift value. See ``set_time_shift``
590  above for its definition.
591
592.. _`synchronize_times`:
593
594- ``synchronize_times()``
595
596  synchronizes the local times of the server and the client, so that
597  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
598  if the client and the server use different time zones. For this
599  to work, *all* of the following conditions must be true:
600
601  - The connection between server and client is established.
602
603  - The client has write access to the directory that is current when
604    ``synchronize_times`` is called.
605
606  If you can't fulfill these conditions, you can nevertheless set the
607  time shift value explicitly with `set_time_shift`_. Trying to call
608  ``synchronize_times`` if the above conditions aren't met results in
609  a ``TimeShiftError`` exception.
610
611Creating and removing directories
612`````````````````````````````````
613
614- ``mkdir(path, [mode])``
615
616  makes the given directory on the remote host. This does *not*
617  construct "intermediate" directories that don't already exist. The
618  ``mode`` parameter is ignored; this is for compatibility with
619  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
620  instead of the ``os`` module. See the explanation in the subsection
621  `Exception hierarchy`_.
622
623- ``makedirs(path, [mode])``
624
625  works similar to ``mkdir`` (see above), but also makes intermediate
626  directories like ``os.makedirs``. The ``mode`` parameter is only
627  there for compatibility with ``os.makedirs`` and is ignored.
628
629- ``rmdir(path)``
630
631  removes the given remote directory. If it's not empty, raise
632  a ``PermanentError``.
633
634- ``rmtree(path, ignore_errors=False, onerror=None)``
635
636  removes the given remote, possibly non-empty, directory tree.
637  The interface of this method is rather complex, in favor of
638  compatibility with ``shutil.rmtree``.
639
640  If ``ignore_errors`` is set to a true value, errors are ignored.
641  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
642  set, all exceptions occurring during the tree iteration and
643  processing are raised. These exceptions are all of type
644  ``PermanentError``.
645
646  To distinguish between different kinds of errors, pass in a callable
647  for ``onerror``. This callable must accept three arguments:
648  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
649  object, *for example* ``your_host_object.listdir``. ``path`` is the
650  path that was the recent argument of the respective method
651  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
652  info as it is gotten from ``sys.exc_info``.
653
654  The code of ``rmtree`` is taken from Python's ``shutil`` module
655  and adapted for ``ftputil``.
656
657Removing files and links
658````````````````````````
659
660- ``remove(path)``
661
662  removes a file or link on the remote host, similar to ``os.remove``.
663
664- ``unlink(path)``
665
666  is an alias for ``remove``.
667
668Retrieving information about directories, files and links
669`````````````````````````````````````````````````````````
670
671- ``listdir(path)``
672
673  returns a list containing the names of the files and directories
674  in the given path, similar to `os.listdir`_. The special names
675  ``.`` and ``..`` are not in the list.
676
677The methods ``lstat`` and ``stat`` (and some others) rely on the
678directory listing format used by the FTP server. When connecting to a
679host, ``FTPHost``'s constructor tries to guess the right format, which
680succeeds in most cases. However, if you get strange results or
681``ParserError`` exceptions by a mere ``lstat`` call, please `file a
682bug report`_.
683
684If ``lstat`` or ``stat`` give wrong modification dates or times, look
685at the methods that deal with time zone differences (`time zone
686correction`_).
687
688.. _`FTPHost.lstat`:
689
690- ``lstat(path)``
691
692  returns an object similar to that from `os.lstat`_. This is a
693  "tuple" with additional attributes; see the documentation of the
694  ``os`` module for details.
695
696  The result is derived by parsing the output of a ``LIST`` command on
697  the server. Therefore, the result from ``FTPHost.lstat`` can not
698  contain more information than the received text. In particular:
699
700  - User and group ids can only be determined as strings, not as
701    numbers, and that only if the server supplies them. This is
702    usually the case with Unix servers but maybe not for other FTP
703    server programs.
704
705  - Values for the time of the last modification may be rough,
706    depending on the information from the server. For timestamps
707    older than a year, this usually means that the precision of the
708    modification timestamp value is not better than days. For newer
709    files, the information may be accurate to a minute.
710
711    If the time of the last modification is before the epoch (usually
712    1970-01-01 UTC), set the time of the last modification to 0.0.
713
714  - Links can only be recognized on servers that provide this
715    information in the ``LIST`` output.
716
717  - Stat attributes that can't be determined at all are set to
718        ``None``. For example, a line of a directory listing may not
719        contain the date/time of a directory's last modification.
720
721  - There's a special problem with stat'ing the root directory.
722    (Stat'ing things *in* the root directory is fine though.) In
723    this case, a ``RootDirError`` is raised. This has to do with the
724    algorithm used by ``(l)stat``, and I know of no approach which
725    mends this problem.
726
727  Currently, ``ftputil`` recognizes the common Unix-style and
728  Microsoft/DOS-style directory formats. If you need to parse output
729  from another server type, please write to the `ftputil mailing
730  list`_. You may consider `writing your own parser`_.
731
732.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
733.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
734.. _`ftputil mailing list`: http://ftputil.sschwarzer.net/mailinglist
735.. _`writing your own parser`: `Writing directory parsers`_
736
737.. _`FTPHost.stat`:
738
739- ``stat(path)``
740
741  returns ``stat`` information also for files which are pointed to by a
742  link. This method follows multiple links until a regular file or
743  directory is found. If an infinite link chain is encountered or the
744  target of the last link in the chain doesn't exist, a
745  ``PermanentError`` is raised.
746
747  The limitations of the ``lstat`` method also apply to ``stat``.
748
749.. _`FTPHost.path`:
750
751``FTPHost`` objects contain an attribute named ``path``, similar to
752`os.path`_. The following methods can be applied to the remote host
753with the same semantics as for ``os.path``:
754
755::
756
757    abspath(path)
758    basename(path)
759    commonprefix(path_list)
760    dirname(path)
761    exists(path)
762    getmtime(path)
763    getsize(path)
764    isabs(path)
765    isdir(path)
766    isfile(path)
767    islink(path)
768    join(path1, path2, ...)
769    normcase(path)
770    normpath(path)
771    split(path)
772    splitdrive(path)
773    splitext(path)
774    walk(path, func, arg)
775
776Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
777methods return ``False`` if they can't find the path given by their
778argument.
779
780Local caching of file system information
781````````````````````````````````````````
782
783Many of the above methods need access to the remote file system to
784obtain data on directories and files. To get the most recent data,
785*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
786would require to fetch a directory listing from the server, which can
787make the program *very* slow. This effect is more pronounced for
788operations which mostly scan the file system rather than transferring
789file data.
790
791For this reason, ``ftputil`` by default saves the results from
792directory listings locally and reuses those results. This reduces
793network accesses and so speeds up the software a lot. However, since
794data is more rarely fetched from the server, the risk of obsolete data
795also increases. This will be discussed below.
796
797Caching can be controlled -- if necessary at all -- via the
798``stat_cache`` object in an ``FTPHost``'s namespace. For example,
799after calling
800
801::
802
803    ftp_host = ftputil.FTPHost(host, user, password)
804
805the cache can be accessed as ``ftp_host.stat_cache``.
806
807While ``ftputil`` usually manages the cache quite well, there are two
808possible reasons for modifying cache parameters.
809
810The first is when the number of possible entries is too low. You may
811notice that when you are processing very large directories and the
812program becomes much slower than before. It's common for code to read
813a directory with ``listdir`` and then process the found directories
814and files. This can also happen implicitly by a call to
815``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
816increases the cache size if directories with more entries than the
817current maximum cache size are to be scanned. Most of the time, this
818works fine.
819
820However, if you need access to stat data for several directories at
821the same time, you may need to increase the cache explicitly. This is
822done by the ``resize`` method::
823
824    ftp_host.stat_cache.resize(20000)
825
826where the argument is the maximum number of ``lstat`` results to store
827(the default is 5000, in versions before 2.6 it was 1000). Note that
828each path on the server, e. g. "/home/schwa/some_dir", corresponds to
829a single cache entry. Methods like ``exists`` or ``getmtime`` all
830derive their results from a previously fetched ``lstat`` result.
831
832The value 5000 above means that the cache will hold *at most* 5000
833entries (unless increased automatically by an explicit or implicit
834``listdir`` call, see above). If more are about to be stored, the
835entries which haven't been used for the longest time will be deleted
836to make place for newer entries.
837
838The second possible reason to change the cache parameters is to avoid
839stale cache data. Caching is so effective because it reduces network
840accesses. This can also be a disadvantage if the file system data on
841the remote server changes after a stat result has been retrieved; the
842client, when looking at the cached stat data, will use obsolete
843information.
844
845There are two potential ways to get such out-of-date stat data. The
846first happens when an ``FTPHost`` instance modifies a file path for
847which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
848Such changes are handled transparently; the path will be deleted from
849the cache. A different matter are changes unknown to the ``FTPHost``
850object which inspects its cache. Obviously, for example, these are
851changes by programs running on the remote host. On the other hand,
852cache inconsistencies can also occur if two ``FTPHost`` objects change
853a file system simultaneously::
854
855    with (
856      ftputil.FTPHost(server, user1, password1) as ftp_host1,
857      ftputil.FTPHost(server, user1, password1) as ftp_host2
858    ):
859        stat_result1 = ftp_host1.stat("some_file")
860        stat_result2 = ftp_host2.stat("some_file")
861        ftp_host2.remove("some_file")
862        # `ftp_host1` will still see the obsolete cache entry!
863        print ftp_host1.stat("some_file")
864        # Will raise an exception since an `FTPHost` object
865        # knows of its own changes.
866        print ftp_host2.stat("some_file")
867
868At first sight, it may appear to be a good idea to have a shared cache
869among several ``FTPHost`` objects. After some thinking, this turns out
870to be very error-prone. For example, it won't help with different
871processes using ``ftputil``. So, if you have to deal with concurrent
872write/read accesses to a server, you have to handle them explicitly.
873
874The most useful tool for this is the ``invalidate`` method. In the
875example above, it could be used like this::
876
877    with (
878      ftputil.FTPHost(server, user1, password1) as ftp_host1,
879      ftputil.FTPHost(server, user1, password1) as ftp_host2
880    ):
881        stat_result1 = ftp_host1.stat("some_file")
882        stat_result2 = ftp_host2.stat("some_file")
883        ftp_host2.remove("some_file")
884        # Invalidate using an absolute path.
885        absolute_path = ftp_host1.path.abspath(
886                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
887        ftp_host1.stat_cache.invalidate(absolute_path)
888        # Will now raise an exception as it should.
889        print ftp_host1.stat("some_file")
890        # Would raise an exception since an `FTPHost` object
891        # knows of its own changes, even without `invalidate`.
892        print ftp_host2.stat("some_file")
893
894The method ``invalidate`` can be used on any *absolute* path, be it a
895directory, a file or a link.
896
897By default, the cache entries (if not replaced by newer ones) are
898stored for an infinite time. That is, if you start your Python process
899using ``ftputil`` and let it run for three days a stat call may still
900access cache data that old. To avoid this, you can set the ``max_age``
901attribute::
902
903    with ftputil.FTPHost(server, user, password) as ftp_host:
904        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds
905
906This sets the maximum age of entries in the cache to an hour. This
907means any entry older won't be retrieved from the cache but its data
908instead fetched again from the remote host and then again stored for
909up to an hour. To reset `max_age` to the default of unlimited age,
910i. e. cache entries never expire, use ``None`` as value.
911
912If you are certain that the cache will be in the way, you can disable
913and later re-enable it completely with ``disable`` and ``enable``::
914
915    with ftputil.FTPHost(server, user, password) as ftp_host:
916        ftp_host.stat_cache.disable()
917        ...
918        ftp_host.stat_cache.enable()
919
920During that time, the cache won't be used; all data will be fetched
921from the network. After enabling the cache again, its entries will be
922the same as when the cache was disabled, that is, entries won't get
923updated with newer data during this period. Note that even when the
924cache is disabled, the file system data in the code can become
925inconsistent::
926
927    with ftputil.FTPHost(server, user, password) as ftp_host:
928        ftp_host.stat_cache.disable()
929        if ftp_host.path.exists("some_file"):
930            mtime = ftp_host.path.getmtime("some_file")
931
932In that case, the file ``some_file`` may have been removed by another
933process between the calls to ``exists`` and ``getmtime``!
934
935Iteration over directories
936``````````````````````````
937
938.. _`FTPHost.walk`:
939
940- ``walk(top, topdown=True, onerror=None, followlinks=False)``
941
942  iterates over a directory tree, similar to `os.walk`_. Actually,
943  ``FTPHost.walk`` uses the code from Python with just the necessary
944  modifications, so see the linked documentation.
945
946.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk
947
948.. _`FTPHost.path.walk`:
949
950- ``path.walk(path, func, arg)``
951
952  Similar to ``os.path.walk``, the ``walk`` method in
953  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
954  easier to use.
955
956Other methods
957`````````````
958
959- ``close()``
960
961  closes the connection to the remote host. After this, no more
962  interaction with the FTP server is possible with this ``FTPHost``
963  object. Usually you don't need to close an ``FTPHost`` instance
964  with ``close`` if you set up the instance in a ``with`` statement.
965
966- ``rename(source, target)``
967
968  renames the source file (or directory) on the FTP server.
969
970.. _`FTPHost.chmod`:
971
972- ``chmod(path, mode)``
973
974  sets the access mode (permission flags) for the given path. The mode
975  is an integer as returned for the mode by the ``stat`` and ``lstat``
976  methods. Be careful: Usually, mode values are written as octal
977  numbers, for example 0755 to make a directory readable and writable
978  for the owner, but not writable for the group and others. If you
979  want to use such octal values, rely on Python's support for them::
980
981    ftp_host.chmod("some_directory", 0o755)
982
983  Not all FTP servers support the ``chmod`` command. In case of
984  an exception, how do you know if the path doesn't exist or if
985  the command itself is invalid? If the FTP server complies with
986  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
987  command isn't allowed. ``ftputil`` maps this special error
988  response to a ``CommandNotImplementedError`` which is derived from
989  ``PermanentError``.
990
991  So you need to code like this::
992
993    with ftputil.FTPHost(server, user, password) as ftp_host:
994        try:
995            ftp_host.chmod("some_file", 0o644)
996        except ftputil.error.CommandNotImplementedError:
997            # `chmod` not supported
998            ...
999        except ftputil.error.PermanentError:
1000            # Possibly a non-existent file
1001            ...
1002
1003  Because the ``CommandNotImplementedError`` is more specific, you
1004  have to test for it first.
1005
1006.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_
1007
1008- ``copyfileobj(source, target, length=64*1024)``
1009
1010  copies the contents from the file-like object ``source`` to the
1011  file-like object ``target``. The only difference to
1012  ``shutil.copyfileobj`` is the default buffer size. Note that
1013  arbitrary file-like objects can be used as arguments (e. g. local
1014  files, remote FTP files).
1015
1016  However, the interfaces of ``source`` and ``target`` have to match;
1017  the string type read from ``source`` must be an accepted string type
1018  when written to ``target``. For example, if you open ``source`` in
1019  Python 3 as a local text file and ``target`` as a remote file object
1020  in binary mode, the transfer will fail since ``source.read`` gives
1021  unicode strings whereas ``target.write`` only accepts byte strings.
1022
1023  See `File-like objects`_ for the construction and use of remote
1024  file-like objects.
1025
1026.. _`set_parser`:
1027
1028- ``set_parser(parser)``
1029
1030  sets a custom parser for FTP directories. Note that you have to pass
1031  in a parser *instance*, not the class.
1032
1033  An `extra section`_ shows how to write own parsers if the default
1034  parsers in ``ftputil`` don't work for you.
1035
1036.. _`extra section`: `Writing directory parsers`_
1037
1038.. _`keep_alive`:
1039
1040- ``keep_alive()``
1041
1042  attempts to keep the connection to the remote server active in order
1043  to prevent timeouts from happening. This method is primarily
1044  intended to keep the underlying FTP connection of an ``FTPHost``
1045  object alive while a file is uploaded or downloaded. This will
1046  require either an extra thread while the upload or download is in
1047  progress or calling ``keep_alive`` from a `callback function`_.
1048
1049  The ``keep_alive`` method won't help if the connection has already
1050  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.
1051
1052  If you want to use this method, keep in mind that FTP servers define
1053  a timeout for a reason. A timeout prevents running out of server
1054  connections because of clients that never disconnect on their own.
1055
1056  Note that the ``keep_alive`` method does *not* affect the "hidden"
1057  FTP child connections established by ``FTPHost.open`` (see section
1058  `FTPHost instances vs. FTP connections`_ for details). You *can't*
1059  use ``keep_alive`` to avoid a timeout in a stalling transfer like
1060  this::
1061
1062      with ftputil.FTPHost(server, userid, password) as ftp_host:
1063          with ftp_host.open("some_remote_file", "rb") as fobj:
1064              data = fobj.read(100)
1065              # _Futile_ attempt to avoid file connection timeout.
1066              for i in xrange(15):
1067                  time.sleep(60)
1068                  ftp_host.keep_alive()
1069              # Will raise an `ftputil.error.TemporaryError`.
1070              data += fobj.read()
1071
1072
1073File-like objects
1074-----------------
1075
1076Construction
1077~~~~~~~~~~~~
1078
1079Basics
1080``````
1081
1082``FTPFile`` objects are returned by a call to ``FTPHost.open``;
1083never use the ``FTPFile`` constructor directly.
1084
1085The API of remote file-like objects are is modeled after the API of
1086the io_ module in Python 3, which has also been backported to Python
10872.6 and 2.7.
1088
1089.. _io: http://docs.python.org/library/io.html
1090
1091- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
1092  errors=None, newline=None)``
1093
1094  returns a file-like object that refers to the path on the remote
1095  host. This path may be absolute or relative to the current directory
1096  on the remote host (this directory can be determined with the
1097  ``getcwd`` method). As with local file objects, the default mode is
1098  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
1099  "wb".
1100
1101  If a file is opened in binary mode, you *must not* specify an
1102  encoding. On the other hand, if you open a file in text mode, an
1103  encoding is used. By default, this is the return value of
1104  ``locale.getpreferredencoding``, but you can (and probably should)
1105  specify a distinct encoding.
1106
1107  If you open a file in binary mode, the read and write operations use
1108  byte strings (``str`` in Python 2, ``bytes`` in Python 3). That is,
1109  read operations return byte strings and write operations only accept
1110  byte strings.
1111
1112  Similarly, text files always work with unicode strings (``unicode``
1113  in Python 2, ``str`` in Python 3). Here, read operations return
1114  unicode strings and write operations only accept unicode strings.
1115
1116  The arguments ``errors`` and ``newline`` have the same semantics as
1117  in `io.open`_. The argument ``buffering`` currently is ignored.
1118  It's only there for compatibility with the ``io.open`` interface.
1119
1120.. _`io.open`: http://docs.python.org/library/io.html#io.open
1121
1122Note that the semantics of "text mode" has changed fundamentally from
1123ftputil 2.8 and earlier. Previously, "text mode" implied converting
1124newline characters to ``\r\n`` when writing remote files and
1125converting newlines to ``\n`` when reading remote files. This is
1126in line with the "text mode" notion of FTP command line clients.
1127Now, "text mode" follows the semantics in Python's ``io`` module.
1128
1129``FTPHost.open`` can also be used in a ``with`` statement::
1130
1131    import ftputil
1132
1133    with ftputil.FTPHost(...) as ftp_host:
1134        ...
1135        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
1136            fobj.write("This is some text.")
1137
1138At the end of the ``with`` block, the remote file will be closed
1139automatically.
1140
1141If something goes wrong during the construction of the file or in the
1142body of the ``with`` statement, the file will be closed as well.
1143Exceptions will be propagated as with ``try ... finally``.
1144
1145Attributes and methods
1146~~~~~~~~~~~~~~~~~~~~~~
1147
1148The methods
1149
1150::
1151
1152    close()
1153    read([count])
1154    readline([count])
1155    readlines()
1156    write(data)
1157    writelines(string_sequence)
1158
1159and the attribute ``closed`` have the same semantics as for file
1160objects of a local disk file system. The iterator protocol is
1161supported as well, i. e. you can use a loop to read a file line by
1162line::
1163
1164    with ftputil.FTPHost(server, user, password) as ftp_host:
1165        with ftp_host.open("some_file") as input_file:
1166            for line in input_file:
1167                # Do something with the line, e. g.
1168                print line.strip().replace("ftplib", "ftputil")
1169
1170For more on file objects, see the section `File objects`_ in the
1171Python Library Reference.
1172
1173.. _`file objects`: https://docs.python.org/2.7/library/stdtypes.html#file-objects
1174
1175
1176.. _`child_connections`:
1177
1178``FTPHost`` instances vs. FTP connections
1179-----------------------------------------
1180
1181This section explains why keeping an ``FTPHost`` instance "alive"
1182without timing out sometimes isn't trivial. If you always finish your
1183FTP operations in time, you don't need to read this section.
1184
1185The file transfer protocol is a stateful protocol. That means an FTP
1186connection always is in a certain state. Each of these states can only
1187change to certain other states under certain conditions triggered by
1188the client or the server.
1189
1190One of the consequences is that a single FTP connection can't be used
1191at the same time, say, to transfer data on the FTP data channel and to
1192create a directory on the remote host.
1193
1194For example, consider this::
1195
1196    >>> import ftplib
1197    >>> ftp = ftplib.FTP(server, user, password)
1198    >>> ftp.pwd()
1199    '/'
1200    >>> # Start transfer. `CONTENTS` is a text file on the server.
1201    >>> socket = ftp.transfercmd("RETR CONTENTS")
1202    >>> socket
1203    <socket._socketobject object at 0x7f801a6386e0>
1204    >>> ftp.pwd()
1205    Traceback (most recent call last):
1206      File "<stdin>", line 1, in <module>
1207      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
1208        return parse257(resp)
1209      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
1210        raise error_reply, resp
1211    ftplib.error_reply: 226-File successfully transferred
1212    226 0.000 seconds (measured here), 5.60 Mbytes per second
1213    >>>
1214
1215Note that ``ftp`` is a single FTP connection, represented by an
1216``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.
1217
1218On the other hand, consider this::
1219
1220    >>> import ftputil
1221    >>> ftp_host = ftputil.FTPHost(server, user, password)
1222    >>> ftp_host.getcwd()
1223    >>> fobj = ftp_host.open("CONTENTS")
1224    >>> fobj
1225    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
1226    >>> ftp_host.getcwd()
1227    u'/'
1228    >>> fobj.readline()
1229    u'Contents of FTP test directory\n'
1230    >>> fobj.close()
1231    >>>
1232
1233To be able to start a file transfer (i. e. open a remote file for
1234reading or writing) and still be able to use other FTP commands,
1235ftputil uses a trick. For every remote file, ftputil creates a new FTP
1236connection, called a child connection in the ftputil source code.
1237(Actually, FTP connections belonging to closed remote files are
1238re-used if they haven't timed out yet.)
1239
1240In most cases this approach isn't noticeable by code using ftputil.
1241However, the nice abstraction of dealing with a single FTP connection
1242falls apart if one of the child connections times out. For example, if
1243you open a remote file and work only with the initial "main"
1244connection to navigate the file system, the FTP connection for the
1245remote file may eventually time out.
1246
1247While it's often relatively easy to prevent the "main" connection from
1248timing out it's unfortunately practically impossible to do this for a
1249remote file connection (apart from transferring some data, of course).
1250For this reason, `FTPHost.keep_alive`_ affects only the main
1251connection. Child connections may still time out if they're idle for
1252too long.
1253
1254.. _`FTPHost.keep_alive`: `keep_alive`_
1255
1256Some more details:
1257
1258- A kind of "straightforward" way of keeping the main connection alive
1259  would be to call ``ftp_host.getcwd()``. However, this doesn't work
1260  because ftputil caches the current directory and returns it without
1261  actually contacting the server. That's the main reason why there's
1262  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
1263  (i. e. the session object), which isn't a public attribute.
1264
1265- Some servers define not only an idle timeout but also a transfer
1266  timeout. This means the connection times out unless there's some
1267  transfer on the data channel for this connection. So ftputil's
1268  ``keep_alive`` doesn't prevent this timeout, but an
1269  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
1270  this transfers the data for the whole directory listing which might
1271  take some time if the directory has many entries.
1272
1273Bottom line: If you can, you should organize your FTP actions so that
1274you finish everything before a timeout happens.
1275
1276
1277Writing directory parsers
1278-------------------------
1279
1280``ftputil`` recognizes the two most widely-used FTP directory formats,
1281Unix and MS style, and adjusts itself automatically. Almost every FTP
1282server uses one of these formats.
1283
1284However, if your server uses a format which is different from the two
1285provided by ``ftputil``, you can plug in a custom parser with a single
1286method call and have ``ftputil`` use this parser.
1287
1288For this, you need to write a parser class by inheriting from the
1289class ``Parser`` in the ``ftputil.stat`` module. Here's an example::
1290
1291    import ftputil.error
1292    import ftputil.stat
1293
1294    class XyzParser(ftputil.stat.Parser):
1295        """
1296        Parse the default format of the FTP server of the XYZ
1297        corporation.
1298        """
1299
1300        def parse_line(self, line, time_shift=0.0):
1301            """
1302            Parse a `line` from the directory listing and return a
1303            corresponding `StatResult` object. If the line can't
1304            be parsed, raise `ftputil.error.ParserError`.
1305
1306            The `time_shift` argument can be used to fine-tune the
1307            parsing of dates and times. See the class
1308            `ftputil.stat.UnixParser` for an example.
1309            """
1310            # Split the `line` argument and examine it further; if
1311            # something goes wrong, raise an `ftputil.error.ParserError`.
1312            ...
1313            # Make a `StatResult` object from the parts above.
1314            stat_result = ftputil.stat.StatResult(...)
1315            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
1316            stat_result._st_name = ...
1317            stat_result._st_target = ...
1318            stat_result._st_mtime_precision = ...
1319            return stat_result
1320
1321        # Define `ignores_line` only if the default in the base class
1322        # doesn't do enough!
1323        def ignores_line(self, line):
1324            """
1325            Return a true value if the line should be ignored. For
1326            example, the implementation in the base class handles
1327            lines like "total 17". On the other hand, if the line
1328            should be used for stat'ing, return a false value.
1329            """
1330            is_total_line = super(XyzParser, self).ignores_line(line)
1331            my_test = ...
1332            return is_total_line or my_test
1333
1334A ``StatResult`` object is similar to the value returned by
1335`os.stat`_ and is usually built with statements like
1336
1337::
1338
1339    stat_result = StatResult(
1340                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
1341                     st_gid, st_size, st_atime, st_mtime, st_ctime))
1342    stat_result._st_name = ...
1343    stat_result._st_target = ...
1344    stat_result._st_mtime_precision = ...
1345
1346with the arguments of the ``StatResult`` constructor described in
1347the following table.
1348
1349===== =================== ============ =================== =======================
1350Index Attribute           os.stat type ``StatResult`` type Notes
1351===== =================== ============ =================== =======================
13520     st_mode             int          int
13531     st_ino              long         long
13542     st_dev              long         long
13553     st_nlink            int          int
13564     st_uid              int          str                 usually only available as string
13575     st_gid              int          str                 usually only available as string
13586     st_size             long         long
13597     st_atime            int/float    float
13608     st_mtime            int/float    float
13619     st_ctime            int/float    float
1362\-    _st_name            \-           str                 file name without directory part
1363\-    _st_target          \-           str                 link target (may be absolute or relative)
1364\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
1365===== =================== ============ =================== =======================
1366
1367If you can't extract all the desirable data from a line (for
1368example, the MS format doesn't contain any information about the
1369owner of a file), set the corresponding values in the ``StatResult``
1370instance to ``None``.
1371
1372Parser classes can use several helper methods which are defined in
1373the class ``Parser``:
1374
1375- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
1376  an appropriate ``st_mode`` integer value.
1377
1378- ``parse_unix_time`` returns a float number usable for the
1379  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
1380  "May"/"26"/"2005". Note that the method expects the timestamp string
1381  already split at whitespace.
1382
1383- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
1384  returns a float number like from ``time.mktime``. Note that the
1385  method expects the timestamp string already split at whitespace.
1386
1387Additionally, there's an attribute ``_month_numbers`` which maps
1388lowercase three-letter month abbreviations to integers.
1389
1390For more details, see the two "standard" parsers ``UnixParser`` and
1391``MSParser`` in the module ``ftputil/stat.py``.
1392
1393To actually *use* the parser, call the method `set_parser`_ of the
1394``FTPHost`` instance.
1395
1396If you can't write a parser or don't want to, please ask on the
1397`ftputil mailing list`_. Possibly someone has already written a parser
1398for your server or can help with it.
1399
1400
1401FAQ / Tips and tricks
1402---------------------
1403
1404Where can I get the latest version?
1405~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1406
1407See the `download page`_. Announcements will be sent to the `mailing
1408list`_. Announcements on major updates will also be posted to the
1409newsgroup `comp.lang.python.announce`_ .
1410
1411.. _`download page`: http://ftputil.sschwarzer.net/download
1412.. _`mailing list`: http://ftputil.sschwarzer.net/mailinglist
1413.. _`comp.lang.python.announce`: news:comp.lang.python.announce
1414
1415Is there a mailing list on ``ftputil``?
1416~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1417
1418Yes, please visit http://ftputil.sschwarzer.net/mailinglist to
1419subscribe or read the archives.
1420
1421Though you can *technically* post without subscribing first I can't
1422recommend it: The mails from non-subscribers have to be approved by
1423me and because the arriving mails contain *lots* of spam, I rarely go
1424through these mails.
1425
1426I found a bug! What now?
1427~~~~~~~~~~~~~~~~~~~~~~~~
1428
1429Before reporting a bug, make sure that you already read this manual
1430and tried the `latest version`_ of ``ftputil``. There the bug might
1431have already been fixed.
1432
1433.. _`latest version`: http://ftputil.sschwarzer.net/download
1434
1435Please see http://ftputil.sschwarzer.net/issuetrackernotes for
1436guidelines on entering a bug in ``ftputil``'s ticket system. If you
1437are unsure if the behaviour you found is a bug or not, you should write
1438to the `ftputil mailing list`_. In *either* case you *must not*
1439include confidential information (user id, password, file names, etc.)
1440in the problem report! Be careful!
1441
1442Does ``ftputil`` support SSL/TLS?
1443~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1444
1445``ftputil`` has no *built-in* SSL/TLS support.
1446
1447On the other hand, there are two ways to get TLS support with
1448ftputil:
1449
1450- In Python 2.7 and Python 3.2 and up, the ``ftplib`` library has a
1451  class ``FTP_TLS`` that you can use for the ``session_factory``
1452  keyword argument in the ``FTPHost`` constructor. You can't use the
1453  class directly though if you need additional setup code in
1454  comparison to ``ftplib.FTP``, for example calling ``prot_p``, to
1455  secure the data connection. On the other hand,
1456  `ftputil.session.session_factory`_ can be used to create a custom
1457  session factory.
1458 
1459  If you have other requirements that ``session_factory`` can't
1460  fulfill, you may create your own session factory by inheriting from
1461  ``ftplib.FTP_TLS``::
1462
1463    import ftplib
1464
1465    import ftputil
1466
1467
1468    class FTPTLSSession(ftplib.FTP_TLS):
1469
1470        def __init__(self, host, user, password):
1471            ftplib.FTP_TLS.__init__(self)
1472            self.connect(host, port)
1473            self.login(user, password)
1474            # Set up encrypted data connection.
1475            self.prot_p()
1476            ...
1477
1478    # Note the `session_factory` parameter. Pass the class, not
1479    # an instance.
1480    with ftputil.FTPHost(server, user, password,
1481                         session_factory=FTPTLSSession) as ftp_host:
1482        # Use `ftp_host` as usual.
1483        ...
1484
1485.. _`ftputil.session.session_factory`: `Session factories`_
1486
1487- If you need to work with Python 2.6, you can use the
1488  ``ftpslib.FTP_TLS`` class from the M2Crypto_ project. Again, you
1489  can't use the class directly but need to use
1490  ``ftputil.session.session_factory`` or a recipe similar to that
1491  above.
1492
1493  Unfortunately, ``M2Crypto.ftpslib.FTP_TLS`` (at least in version
1494  0.22.3) doesn't work correctly if you pass unicode strings to its
1495  methods. Since ``ftputil`` does exactly that at some point (even if
1496  you used byte strings in ``ftputil`` calls) you need a workaround in
1497  the session factory class::
1498
1499    import M2Crypto
1500
1501    import ftputil
1502    import ftputil.tool
1503
1504
1505    class M2CryptoSession(M2Crypto.ftpslib.FTP_TLS):
1506
1507        def __init__(self, host, user, password):
1508            M2Crypto.ftpslib.FTP_TLS.__init__(self)
1509            # Change the port number if needed.
1510            self.connect(host, 21)
1511            self.auth_tls()
1512            self.login(user, password)
1513            self.prot_p()
1514            self._fix_socket()
1515            ...
1516
1517        def _fix_socket(self):
1518            """
1519            Change the socket object so that arguments to `sendall`
1520            are converted to byte strings before being used.
1521            """
1522            original_sendall = self.sock.sendall
1523            # Bound method, therefore no `self` argument.
1524            def sendall(data):
1525                data = ftputil.tool.as_bytes(data)
1526                return original_sendall(data)
1527            self.sock.sendall = sendall
1528
1529    # Note the `session_factory` parameter. Pass the class, not
1530    # an instance.
1531    with ftputil.FTPHost(server, user, password,
1532                         session_factory=M2CryptoSession) as ftp_host:
1533        # Use `ftp_host` as usual.
1534        ...
1535
1536  That said, ``session_factory`` has this workaround built in, so
1537  normally you don't need to define the session factory yourself!
1538
1539.. _M2Crypto: https://github.com/martinpaljak/M2Crypto
1540
1541How do I connect to a non-default port?
1542~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1543
1544By default, an instantiated ``FTPHost`` object connects on the usual
1545FTP port. If you have to use a different port, refer to the section
1546`Session factories`_.
1547
1548How do I set active or passive mode?
1549~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1550
1551Please see the section `Session factories`_.
1552
1553How can I debug an FTP connection problem?
1554~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1555
1556You can do this with a session factory. See `Session factories`_.
1557
1558If you want to change the debug level only temporarily after the
1559connection is established, you can reach the `session object`_ as the
1560``_session`` attribute of the ``FTPHost`` instance and call
1561``_session.set_debuglevel``. Note that the ``_session`` attribute
1562should *only* be accessed for debugging. Calling arbitrary
1563``ftplib.FTP`` methods on the session object may *cause* bugs!
1564
1565.. _`session object`: `Session factories`_
1566
1567Conditional upload/download to/from a server in a different time zone
1568~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1569
1570You may find that ``ftputil`` uploads or downloads files
1571unnecessarily, or not when it should. This can happen when the FTP
1572server is in a different time zone than the client on which
1573``ftputil`` runs. Please see the section on `time zone correction`_.
1574It may even be sufficient to call `synchronize_times`_.
1575
1576When I use ``ftputil``, all I get is a ``ParserError`` exception
1577~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1578
1579The FTP server you connect to may use a directory format that
1580``ftputil`` doesn't understand. You can either write and
1581`plug in an own parser`_ or ask on the `mailing list`_ for
1582help.
1583
1584.. _`plug in an own parser`: `Writing directory parsers`_
1585
1586``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
1587~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1588
1589Like Python's counterparts under `os.path`_, ``ftputil``'s methods
1590return ``False`` if they can't find the given path.
1591
1592Probably you used ``listdir`` on a directory and called ``is...()`` on
1593the returned names. But if the argument for ``listdir`` wasn't the
1594current directory, the paths won't be found and so all ``is...()``
1595variants will return ``False``.
1596
1597I don't find an answer to my problem in this document
1598~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1599
1600Please send an email with your problem report or question to the
1601`ftputil mailing list`_, and we'll see what we can do for you. :-)
1602
1603
1604Bugs and limitations
1605--------------------
1606
1607- ``ftputil`` needs at least Python 2.6 to work.
1608
1609- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
1610  names starting with a dot) depends on the configuration of the FTP
1611  server. See `Hidden files and directories`_ for details.
1612
1613- Due to the implementation of ``lstat`` it can not return a sensible
1614  value for the root directory ``/`` though stat'ing entries *in* the
1615  root directory isn't a problem. If you know an implementation that
1616  can do this, please let me know. The root directory is handled
1617  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.
1618
1619- In multithreaded programs, you can have each thread use one or more
1620  ``FTPHost`` instances as long as no instance is shared with other
1621  threads.
1622
1623- Currently, it is not possible to continue an interrupted upload or
1624  download. Contact me if this causes problems for you.
1625
1626- There's exactly one cache for ``lstat`` results for each ``FTPHost``
1627  object, i. e. there's no sharing of cache results determined by
1628  several ``FTPHost`` objects. See `Local caching of file system
1629  information`_ for the reasons.
1630
1631
1632Files
1633-----
1634
1635If not overwritten via installation options, the ``ftputil`` files
1636reside in the ``ftputil`` package. There's also documentation in
1637`reStructuredText`_ and in HTML format. The locations of these
1638files after installation is system-dependent.
1639
1640.. _`reStructuredText`: http://docutils.sourceforge.net/rst.html
1641
1642The files ``test_*.py`` and ``mock_ftplib.py`` are for unit-testing.
1643If you only *use* ``ftputil``, i. e. *don't* modify it, you can
1644delete these files.
1645
1646
1647References
1648----------
1649
1650- Mackinnon T, Freeman S, Craig P. 2000. `Endo-Testing:
1651  Unit Testing with Mock Objects`_.
1652
1653- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.
1654
1655- Van Rossum G et al. 2013. `Python Library Reference`_.
1656
1657.. _`Endo-Testing: Unit Testing with Mock Objects`:
1658   http://www.connextra.com/aboutUs/mockobjects.pdf
1659.. _`RFC 959 - File Transfer Protocol (FTP)`: http://www.ietf.org/rfc/rfc959.txt
1660.. _`Python Library Reference`: https://docs.python.org/library/index.html
1661
1662
1663Authors
1664-------
1665
1666``ftputil`` is written by Stefan Schwarzer
1667<sschwarzer@sschwarzer.net> and contributors (see
1668``doc/contributors.txt``).
1669
1670The original ``lrucache`` module was written by Evan Prodromou
1671<evan@prodromou.name>.
1672
1673Feedback is appreciated. :-)
Note: See TracBrowser for help on using the repository browser.