Preparing Python Scripts For Python 3

Porting to Python 3

Starting with MeVisLab 3.1, the Python interpreter in MeVisLab is Python 3. This requires porting old Python 2 scripts to Python 3. The porting can be very easy, especially on scripts that mainly use the MeVisLab Python APIs (which basically are the same for Python 2 and 3). It can get more complex when you are using a lot of Python libraries and work with binary data and file IO (because of unicode and bytes datatypes). To facilitate the porting, MeVisLab comes with the python-future library and a Wizard to automatically convert scripts.

python-future Library

The python-future library is used to make Python scripts compatible to Python 2 and Python 3. It is integrated into MeVisLab and available to all MeVisLab modules. python-future provides an automatic conversion, which can be run from the Futurize Wizard in MeVisLab.

See the python-future quick-start guide for more details about how it works.

The Futurize Wizard

The Futurize Wizard is a MeVisLab user script that allows you to run the automatic conversion provided by python-future.

It can be started from the MeVisLab Menu: Scripting → Python 3 Support → Futurize Python Scripts:

Futurize Options

By default, the futurize script is run with the options --all-imports --both-stages --unicode-literals. You can customize the options in the Futurize Options dialog:

Detect Python Scripts

The first step of the wizard is to choose how the Python scripts should be detected.

If one or more modules are selected in the network, then it is possible to change

• the scripts that belong to the related files of the modules or
• the scripts that the ModuleDependencyAnalyzer will determine after analyzing the modules.

Note

inline scripts in *.script and *.pri files are not automatically found and converted. You have to do this manually. If you have Mako templates, you must also port them by hand.

It is also possible to select a directory, which is recursively scanned for Python scripts:

Select Python Scripts And Start Conversion

Finally, you can preview what changes will be applied and select which of the detected scripts will be changed. Then you can start the conversion:

The Automatic Script Conversion Has Finished. What Now?

The automatic conversion makes it a lot easier to make Python scripts compatible to Python 2 and Python 3. But it cannot guarantee that the converted code is correct. Thus it is necessary to test the converted code and maybe to fix some problems by hand. The next section explains some of the issues that can occur.

Common Issues

• Imports via the importPath MDL tag in MeVisLab modules are not automatically adapted by futurize. They must be replaced by a relative import statement: from . import SomeLocalModule.
• Since more relative imports of the form from . import SomeLocalModule are used, it is more likely to get problems with circular dependencies between Python scripts. These dependencies must be resolved by hand. One solution might be to move some imports to the end of the Python script.
• Code that needs to handle unicode decoding/encoding should be carefully reviewed. For example, this can be an issue when handling the output of processes or reading/writing files.
• The split() function of the string module has been removed in Python 3. Use the split() method on string objects instead.
• The built-in function sorted() requires a key parameter and does not support the cmp function anymore, see here for more details.
• string.atoi() does no more exist, use int() instead, it works also in Python 2.7. string.atof() must also be replaced by float().
• Futurize converts dict.keys() to list(dict.keys()), because keys() returned a list in Python 2. It should be checked if this is really necessary. If the list of keys is not modified, but if there is a read only iteration over the keys(), then it is not necessary. The keys() function could be omitted entirely. This issue is also applies to items() and maybe other collection functions.
• ElementTree.tostring() behaves differently in Python 3: the encoding header (e.g. <?xml version='1.0' encoding='UTF-8'?>) is no more included in the string.
• The dircache module was removed, because it was rarely used and its implementation is very simple. It can simply replaced by the os module, or reimplemented, if it is really needed as a performance optimization.
• Print statements of the form print >> targetFile, line, `` are converted to the form ``print(line, end=' ', file=targetFile). This not equivalent if the line ends with a newline, because the trailing comma in Python 2 caused no white space after newlines, which the converted code does. Instead end should be an empty string: print(line, end='', file=targetFile) (only if line ends with \n, otherwise the transformation is correct).
• The behavior of inspect.ismethod has changed. Occurrences may need to be replaced by a combination of inspect.getmembers() and inspect.ismethod().
• The message attribute on exception objects has been deprecated since Python 2.6 and was removed in Python 3. Replace it by args[0].
• Qt functions that require a QByteArray accept a unicode string in Python 2 and Python 3 and automatically convert to a UTF-8 encoded QByteArray. If you need a different encoding, call encode() on your unicode string to get a bytes object.
• If you want to convert a QByteArray to a Python bytes object, you need to call data() on the QByteArray. Calling str(bytearray) returns a bytes object in Python 2 and a unicode string in Python 3, so it is not portable. Use QByteArray.data() instead!
• The int type will be future.types.newint.newint instead of int, so the following assert (typically used in test cases) will fail if variable is an int: assert(type(variable) == int). This can be fixed by using isinstance: assert(isinstance(status, (int, long))).

Further Reading

• Porting Python 2 Code to Python 3
• Python-Future
• A Developer's Guide to Python 3.0:
  • Core Language Changes
  • Numbers, Strings, and Data
  • Standard Library