WebSphere Application Server: Jython Throws an IOException While Accessing Environment Variables

If you try to access an environment variable via Jython’s os.environ in wsadmin of WebSphere AS and you are on the “wrong” Windows release you will get an IOException telling you “Cannot run program “sh”: CreateProcess error=2, The system cannot find the file specified.”. On a Unix or Linux system running the same script in wsadmin you won’t have any problems. A strange situation caused by Jython, not by WebSphere itself. — Here you will see why and you will find a solution.

While porting a configuration and deployment process from AIX and Linux to Windows I encountered a really strange situation. While all Jython scripts executed in wsadmin on AIX and Linux worked fine, the same scripts executed in wsadmin on Windows Server 2008 R2 and Windows 2012 R2 respectively throwed this exception:

Failed to get environment, environ will be empty: (0, 'Failed to execute command ([\'sh\', \'-c\', \'env\']): java.io.IOException: Cannot run program "sh": CreateProcess error=2, The system cannot find the file specified.')

Why does Jython try to execute a Unix shell (sh) on Windows to read an environment variable?

Time for further inspection…

The Situation

In a new and fresh test installation the situation shows that the problem is definitely caused by accessing os.environ. As we see in this wsadmin -lang jython session:

wsadmin>import os
wsadmin>os.environ['USERPROFILE']
Failed to get environment, environ will be empty: (0, 'Failed to execute command ([\'sh\', \'-c\', \'env\']): java.io.IOException: Cannot run program "sh": CreateProcess error=2, The system cannot find the file specified.')
WASX7015E: Exception running command: "os.environ['USERPROFILE']"; exception information:
 com.ibm.bsf.BSFException: exception from Jython:
Traceback (innermost last):
  File "<input>", line 1, in ?
  File "C:\Program Files\IBM\WebSphere\AppServer\optionalLibraries\jython\Lib\javaos.py", line 137, in __getitem__
  File "C:\Program Files\IBM\WebSphere\AppServer\optionalLibraries\jython\Lib\UserDict.py", line 14, in __getitem__
KeyError: USERPROFILE

A simple access to the existing environment variable %USERPROFILE% ($ENV:USERPROFILE for our PowerShell using friends) can reproduce this situation.

The Reason

As mentioned in the error message the problem is caused in %WASHOME%\optionalLibraries\jython\Lib\javaos.py which contains the Jython code of module os. In this file we see that there is an attribute _shellCmd which holds the name of the command processor:

# default to None/empty for shell and environment behavior
_shellCmd = None
_envCmd = None
_envTransform = None

# override defaults based on _osType
if _osType == "nt":
    _shellCmd = ["cmd", "/c"]
    _envCmd = "set"
    _envTransform = string.upper
elif _osType == "dos":
    _shellCmd = ["command.com", "/c"]
    _envCmd = "set"
    _envTransform = string.upper
elif _osType == "posix":
    _shellCmd = ["sh", "-c"]
    _envCmd = "env"
elif _osType == "mac":
    curdir = ':'  # override Posix directories
    pardir = '::' 
elif _osType == "None":
    pass

The initialization of _shellCmd depends on the value of _osType which is assigned a few lines above the previously shown block in javaos.py as follows:

_osType = _getOsType()

Last but not least the method _getOsType() shows us why sh is used as command shell:

 def _getOsType( os=None ):
    """Select the OS behavior based on os argument, 'python.os' registry
    setting and 'os.name' Java property.
    os: explicitly select desired OS. os=None to autodetect, os='None' to
    disable 
    """
    os = os or sys.registry.getProperty( "python.os" ) or \
               java.lang.System.getProperty( "os.name" )
        
    _osTypeMap = (
        ( "nt", r"(nt)|(Windows NT)|(Windows NT 4.0)|(WindowsNT)|"
                r"(Windows 2000)|(Windows XP)|(Windows CE)" ),
        ( "dos", r"(dos)|(Windows 95)|(Windows 98)|(Windows ME)" ),
        ( "mac", r"(mac)|(MacOS.*)|(Darwin)" ),
        ( "None", r"(None)" ),
        ( "posix", r"(.*)" ), # default - posix seems to vary mast widely
        )
    for osType, pattern in _osTypeMap:
        if re.match( pattern, os ):
            break
    return osType

As you see only a few Windows releases are supported and recognized as "nt". You’ll find for example Windows NT, Windows 2000 and Windows XP. Even the old MS-DOS-based systems Windows 95/98/ME are supported. Looks a bit like a list of exhibits from a computer museum. More recent systems such as Windows Server 2008 R2, Windows Server 2012, Windows Vista, Windows 7, and Windows 8 are missing.

If a system cannot be determined it is treated as a POSIX system as we see in this line:

( "posix", r"(.*)" ), # default - posix seems to vary mast widely

So my Windows 2008 R2 and Windows 2012 R2 systems become suddenly Unix-like systems which shall provide sh.

Let’s check the value of the system property os.name and the value returned by _getOsType() on a Windows Server 2008 R2:

wsadmin>import os
wsadmin>java.lang.System.getProperty('os.name')
'Windows Server 2008 R2'
wsadmin>os._getOsType()
'posix'

The Solution(s)

The first solution which comes to mind is, of course, the extension of the "nt" entry in _osTypeMap of _getOsType(). You could extend this entry by adding other releases like that:

( "nt", r"(nt)|(Windows NT)|(Windows NT 4.0)|(WindowsNT)|"
        r"(Windows 2000)|(Windows XP)|(Windows CE)|(Windows Server 2008 R2)" ),

This would work but it needs a lot of effort for maintaining this patch:

• Whenever a new Windows version is released (and used by you) you have to extend the pattern again.
• You will have to check this code every time you install a WebSphere fixpack. (Who knows when this module is overwritten by a fixpack?)
• If you develop code for customers and products “off the shell” you will have to support a wide range of Windows versions. You have to get all the possible values for os.name and test it against all these Windows versions.

Even that is all managable it is not very charming.

The second solution is installing UNIX Services for Windows and Subsystems for UNIX-based Application respectively or Cygwin or Mint. Afterwards you put sh‘s directory in to PATH. Now that Jython’s javaos.py finds sh in PATH everything will work.

But do you really want to install a Unix shell only because Jython is not able to determine the operating system correctly? — No, of course not. (Even it’s always a good idea to have a Unix shell available. — But here speaks the Unix guy out of me… 😉 )

The last and easiest solution is Jython’s registry! If you put the right operating system value into python.os you can force it to be an “nt”.

Why? — Have look at this line of _getOsType() and you will understand:

    os = os or sys.registry.getProperty( "python.os" ) or \
               java.lang.System.getProperty( "os.name" )

But where to find the registry? — http://www.jython.org/archive/21/docs/registry.html tells us that Jython will use the values of the system properties python.home or install.root if present. Otherwise it will look for a file called registry in a directory which is part of CLASSPATH and contains the jython.jar.

To make it short: In WebSphere’s Jython the properties python.home and install.root won’t be present by default. So let’s find the directroy which holds jython.jar: It is %WASHOME%\optionalLibraries\jython.

So create simply a file %WASHOME%\optionalLibraries\jython\registry with your favorite text editor and put the following line into it:

python.os=nt

Final Test

Let’s have a look whether it worked. In a new spawned wsadmin we enter:

wsadmin>import os
wsadmin>os.environ['USERPROFILE']
'C:\\Users\\JohnnyBQuick'
wsadmin>java.lang.System.getProperty('os.name')
'Windows Server 2008'
wsadmin>os._getOsType()
'nt'

One simple line in a newly created file and the problems are gone.