error when openening mvs file for writing

[ftputiluser]

When trying to open a file on a mvs-host for writing with the following commands

>>> from ftputil import *
>>> datei = FTPHost("hostname", "username", "password").open("versuch", "w")

I get the following traceback:

Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    datei = FTPHost("hostname", "username", "password").open("versuch", "w")
  File "C:\Program Files\Python25\lib\site-packages\ftputil\ftputil.py", line 230, in open
    return self.file(path, mode)
  File "C:\Program Files\Python25\lib\site-packages\ftputil\ftputil.py", line 225, in file
    self.stat_cache.invalidate(effective_path)
  File "C:\Program Files\Python25\Lib\site-packages\ftputil\ftp_stat_cache.py", line 128, in invalidate
    assert path.startswith("/"), "%s must be an absolute path" % path
AssertionError: 'username.'/versuch must be an absolute path

There is no problem to open a (existing) file with the above commandline for reading. The problem seems to be the naming convention on mvs systems where no root directory called / exists. There ist an ugly workaround for this problem:

datei = FTPHost("hostname", "username", "password").open("/'username.versuch'", "w")

forces the / to be the first character of the filename which is required by ftp_stat_cache.py.

[schwa]

Thanks for your issue report.

Could you please substitute the method FTPHost.file with the following debug version and run the code in your bug description (above) again? Please add a comment containing the output.

    def file(self, path, mode='r'):
        """
        Return an open file(-like) object which is associated with
        this `FTPHost` object.

        This method tries to reuse a child but will generate a new one
        if none is available.
        """
        host = self._available_child()
        if host is None:
            host = self._copy()
            self._children.append(host)
            host._file = ftp_file._FTPFile(host)
        basedir = self.getcwd()
        print "basedir:", basedir
        # prepare for changing the directory (see whitespace workaround
        #  in method `_dir`)
        if host.path.isabs(path):
            print "is absolute"
            effective_path = path
        else:
            print "is not absolute"
            effective_path = host.path.join(basedir, path)
        print "effective path:", effective_path
        effective_dir, effective_file = host.path.split(effective_path)
        print "effective dir:", effective_dir
        print "effective file:", effective_file
        try:
            # this will fail if we can't access the directory at all
            host.chdir(effective_dir)
        except ftp_error.PermanentError:
            # similarly to a failed `file` in a local filesystem, we
            #  raise an `IOError`, not an `OSError`
            raise ftp_error.FTPIOError("remote directory '%s' doesn't exist "
                  "or has insufficient access rights" % effective_dir)
        host._file._open(effective_file, mode)
        if 'w' in mode:
            self.stat_cache.invalidate(effective_path)
        return host._file

Stefan

[ftputiluser]

Hi Stefan, here is the result (traceback omitted):

FTPHost("hostname", "username", "password").file("versuch", "w")
basedir: 'username.'
is not absolute
effective path: 'username.'/versuch
effective dir: 'username.'
effective file: versuch

Regards, Axel

[schwa]

Hi Axel,

In comparison to Posix or Windows, this path syntax is quite unfamiliar to me. I don't have an idea yet to support this out-of-the-box and cleanly in ftputil.

Some questions that might help:

• Can you configure the FTP server? In particular, can you configure the server to emit Posix path names instead of the MVS syntax?
• Is the path syntax used by the server the same as that of MVS' file system?
• Are there resources about the path syntax on the web?
• Can you tell something about the FTP server software? What's its name and version?
• What is the output of the FTP STAT, SITE and SYST commands if the server accepts them? You can invoke the commands via ftplib 's sendcmd command.

As you set the priority of this bug report to "minor" it might suffice to apply a workaround without modifying ftputil. For example, could you write a utility function to change the path as you described in the initial report?

Stefan

[ftputiluser]

Hi Stefan, here are the answers to your questions:

1st I can't configure the FTP server, I am just an ordinary user on the host.

2nd/3rd As far as I understand MVS the is nothing like a path structure at all. I found a short description in the german wikipedia:

Hier ist Dataset eine andere Bezeichnung für eine Datei, die auf einem
IBM-Großrechnersystem existiert. Ein Dataset-Name (DSN) kann maximal 44 Zeichen
lang sein und besteht aus mehreren Qualifiern (Namensteilen), die durch Punkte 
voneinander getrennt werden. Jeder Qualifier kann maximal acht Stellen lang sein. 
Beispiel: MEIN.PRIVATES.TEST.DATASET.V1

http://de.wikipedia.org/wiki/Dataset

If the DSN is not surrounded by apostrophes (') the name of the current user is added as the first qualifier (i.e. MEIN.DATASET.V1 is the same as 'USERNAME.MEIN.DATASET.V1' ). Additionally, the DSNs can include a member name in braces: 'USERNAME.DATASET.V1(MEMBER1)' )

The FTP server has a meaning of a current directory which is described by adding the current "path", which consists of one or more qualifiers, to a given name.

4th/5th

>>> print xxx.sendcmd("stat")
211-Server FTP talking to host xx.yy.zz.aa, port 1858
211-User: xyz  Working directory: xyz.
211-The control connection has transferred 2636 bytes
211-There is no current data connection.
211-The next data connection will be actively opened
211-to host xxx.yy.zz.aa, port 1858, 
211-using Mode Stream, Structure File, type ASCII, byte-size 8
211-Automatic recall of migrated data sets.
211-Automatic mount of direct access volumes.
211-Auto tape mount is allowed.
211-Inactivity timer is set to 700
211-Timer FTPKEEPALIVE is set to 3600
211-VCOUNT is 59
211-ASA control characters in ASA files opened for text processing 
211-will be transferred as ASA control characters.
211-Trailing blanks are not removed from a fixed format 
211-data set when it is retrieved.
211-Data set mode.  (Do not treat each qualifier as a directory.)
211-ISPFSTATS is set to FALSE
211-Primary allocation 5 cylinders.  Secondary allocation 5 cylinders.
211-Partitioned data sets will be created with 15 directory blocks.
211-FileType SEQ (Sequential - default).
211-Number of access method buffers is 5
211-RDWs from variable format data sets are discarded.
211-Records on input tape are unspecified format
211-SITE DB2 subsystem name is DB2
211-Data not wrapped into next record.
211-Tape write is not allowed to use BSAM I/O
211-Truncated records will not be treated as an error
211-JESLRECL is 80
211-JESRECFM is Fixed
211-JESINTERFACELEVEL is 1
211-Server site variable JESTRAILINGBLANKS is set to TRUE
211-Confidence level in data transfers is neither checked nor reported
211-ENcoding is set to SBCS
211-Outbound SBCS ASCII data uses CRLF line terminator
211-Outbound MBCS ASCII data uses CRLF line terminator
211-Server site variable MBREQUIRELASTEOL is set to TRUE
211-Server site variable UNICODEFILESYSTEMBOM is set to ASIS
211-DBSUB is set to FALSE
211-SBSUB is set to FALSE
211-SBSUBCHAR is set to SPACE
211-SMS is active.
211-Mgmtclass for new data sets is MCS30570
211-Data sets will be allocated using unit SYSDA
211-New data sets will be catalogued if a store operation ends abnormally
211-Single quotes will override the current working directory.
211-UMASK value is 027
211-Process id is 50791100
211-Checkpoint interval is 0
211-Authentication type: None
211-Port of Entry resource class for IPv4 clients is: TERMINAL
211-Record format FB, Lrecl: 80, Blocksize: 29440
211-Server site variable LISTSUBDIR is set to TRUE
211 *** end of status ***

>>> print xxx.sendcmd("site")
202 SITE not necessary; you may proceed

>>> print xxx.sendcmd("syst")
215 MVS is the operating system of this server. FTP Server is running on z/OS.

Of course it would be possible to write a helper function to mask the special mvs behaviour. But of course, I would prefer a solution in the ftplib itself. If you need more information or debugging support, don't hesitate to ask me.

Best regards, Axel

[schwa]

Hello Axel,

sorry, I forgot to follow up.

I think I don't want to add a larger additional abstraction level to ftputil's navigation. However, if you can think of a way to do add the MVS compatibility with minimal drawbacks to the code's clarity/understandability, you may suggest possible approaches.

Stefan

[schwa]

Hi Axel,

If I understood correctly from the description of issue #28, there's a not-too-hard workaround for opening files under MVS. Therefore, I'd like to close this issue (#26), at least for now.

Stefan