11 December 2021
Nuitka Release 0.6.18
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release has a focus on new features of all kinds, and then also new kinds of performance improvements, some of which enable static optimization of what normally would be dynamic imports, while also polishing plugins and adding also many new features and a huge amount of organizational changes. Python3.6+: Fixes to asyncgen, need to raise Onefile: Fix, LTO mode was always enabled for onefile compilation, but not all compilers support it yet, e.g. MinGW64 did not. Fixed in 0.6.17.1 already. Fix, Fix, trusted module constants were not working properly in all cases. Fixed in 0.6.17.2 already. Fix, Standalone: Prevent the inclusion of Standalone: Added missing implicit dependency for Fix, Qt webengine support for everything but Windows: Fix, bootstrap splash screen code for Windows was missing in release packages. Fixed in 0.6.17.3 already. Fix, could crash on known implicit data directories not present. Fixed in 0.6.17.3 already. macOS: Disable download of Standalone: The Onefile: Make sure the child process is cleaned up even after its successful exit. Fixed in 0.6.17.4 already. Standalone: Added support for Standalone: Added support for Standalone: Added support for Python3.4+: Up until Python3.7 inclusive, a workaround for stream encoding (was ASCII), causing crashes on output of non-ASCII, other Python versions are not affected. Fixed in 0.6.17.5 already. Python2: Workaround for LTO error messages from older gcc versions. Fixed in 0.6.17.5 already. Standalone: Added support for Fix, need to prevent usage of static Standalone: Do not load Fix, could crash with generator expressions in finally blocks of tried blocks that return. Fixed in 0.6.17.7 already. Python3.5+: Compatibility of comparisons with Standalone: Added support for Distutils: Fix, when building more than one package, things could go wrong. Fixed in 0.6.17.7 already. Fix, for module mode filenames are used, and for packages, you can specify a directory, however, a trailing slash was not working. Fixed in 0.6.17.7 already. Compatibility: Fix, when locating modules, a package directory and an extension module of the same name were not used according to priority. Fixed in 0.6.17.7 already. Standalone: Added workaround Standalone: Properly detect usage of hard imports from standard library in Standalone: Added data files for MSYS2: Fix, need to normalize compiler paths before comparing. Anaconda: For accelerated binaries, the created macOS: Set minimum OS version derived from the Python executable used, this should make it work on all supported platforms (of that Python). Standalone: Added support for automatic inclusion of Standalone: Added support for automatic inclusion of Standalone: Added support for Standalone: Added support for Standalone: Added support for Standalone: Added Standalone: Added necessary datafiles for Standalone: Added support for Python3: Fix, the Windows: The LTO was supposed to be used automatically on with MSVC 14.2 or higher, but that was regressed and has been repaired now. Standalone: Extension modules contained in packages, depending on their mode of loading had the Windows: The Setuptools support. Documented Added experimental support for Python 3.10, there are however still important issues with compatibility with the CPython 3.9 test suite with at least asyncgen and coroutines. macOS: For app bundles, version information can be provided with the new option Added Python vendor detection of Plugins: Also handle the usage of Plugins: The Windows: Allow enforcing usage of MSVC with Windows: Added support for LTO with MinGW64 on Windows, this was previously limited to the MSVC compiler only. Windows: Added support for using Note It doesn`t work when executed from a Git bash prompt, but e.g. from a standard command prompt. Added new experimental flag for compiled types to inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses Plugins: The Qt binding plugins now resolve macOS: Provide UI: Added new option Plugins: Added experimental support for UI: Added option Added dedicated attribute nodes for attribute values that match names of dictionary operations. These are optimized into dedicate nodes for methods of dictionaries should their expression have an exact dictionary shape. These in turn optimize calls on them statically into dictionary operations. This is done for all methods of The new operation nodes also add compile time optimization for being used on constant values where possible. Also added dedicated attribute nodes for string operations. For operations, currently only part of the methods are done. These are currently only Annotate type shape for dictionary Faster processing of “expression only” statement nodes. These are nodes, where a value is computed, but then not used, it still needs to be accounted for though, representing the value release. Windows: Enabled LTO by default with MinGW64, which makes it produce much faster results. It now yield faster binaries than MSVC 2019 with pystone. Windows: Added support for C level PGO (Profile Guided Optimization) with MSVC and MinGW64, allowing extra speed boosts from the C compilation on Windows as well. Standalone: Better handling of Avoid useless exception checks in our dictionary helpers, as these could only occur when working with dictionary overloads, which we know to not be the case. For nodes, have dedicated child mixin classes for nodes with a single child value and for nodes with a tuple of children, so that these common kind of nodes operate faster and don’t have to check at run time what type they are during access. Actually make use of the egg cache. Nuitka was unpacking eggs in every compilation, but in wheel installs, these can be quite common and should be faster. Star arguments annotated their type shape, but the methods to check for dictionary exactly were not affected by this preventing optimization in some cases. Added Using static libpython with Plugins: Added improvements to the Python3.9+: Faster calls into uncompiled functions from compiled code using newly introduced API of that version. Statically optimize Added support for Added hard import and static optimization for Also compute named import lookup through variables, expanding their use to more cases, e.g. like this: Also optimize compile time comparisons through variable names if possible, i.e. the value cannot have changed. Faster calls of uncompiled code with Python3.9 or higher avoiding DLL call overhead. Commercial: There are The main script runners for Python2 have been renamed to Windows: Added support for Visual Studio 2022 by updating the inline copy of Scons used for Windows to version 4.3.0, on non Windows, the other ones will keep being used. Windows: Requiring latest MinGW64 with version 11.2 as released by winlibs, because this is known to allow LTO, where previous releases were missing needed binaries. Reject standalone mode usage with Apple Python, as it works only with the other supported Pythons, avoiding pitfalls in attempting to distribute it. Move hosting of documentation to Sphinx, added Changelog and some early parts of API documentation there too. This gives much more readable results than what we have done so far with Nikola. More things will move there. User Manual: Add description how to access code attributes in User Manual: Added commands used to generate performance numbers for Python. User Manual: List other Python’s for which static linking is supposed to work. Improved help for Started using Jinja2 in code templates with a few types, adding basic infrastructure to do that. This will be expanded in the future. Updated plugin documentation with more recent information. Added Python flavor as detected to the Linux: Added distribution name to Always enable the Added project URLs for PyPI, so people looking at it from there have some immediate places to checkout. Debian: Use common code for included PDF files, which have page styles and automatic corrections for Updated to latest The binary names for Python2 changed from Note You ought to be using Added support for Fedora 34 and Fedora 35. In a change of mind Massive cleanup of Massive cleanups of all documentation in ReST format. Plenty of formatting errors were resolved. Many typos were identified and globally fixed. Spellings e.g. of “Developer Manual” are now enforced with automatic replacements. Also missing or wrong quotes were turned to proper methods. Also enforce code language for shell scripts to be the same everywhere. Removed last usages of Avoid string comparison with Replaced usages of string tests in list of python flags specified, with functions that check for a specific name with a speaking function name. Added mixin for expressions that have no side effect outside of their value, providing common method implementation more consistently. Remove code geared to using old PyLint and on Python2, we no longer use that. Also removed annotations only used for overriding Python2 builtins from Nuitka code. The PDF specific annotations were moved into being applied only in the PDF building step, avoiding errors for raw PDF directives. Apply Visual Code auto-format to our Yaml files. This is unfortunately not and automatic formatting yet. Introduce dedicated Replacing remaining usages of Massive cleanup of the Using own helper to update More consistent naming of plugin classes, and enforce a relationship of detector class names to the names of detected plugins. The new naming consistency is now enforced. Added CPython 3.10 test suite, it needs more work though. Added generated test that exercises dictionary methods in multiple variations. Test suite names were specified wrongly in a few of them. This release is again a huge step forward. It refines on PGO and LTO for C level to work with all relevant compilers. Internally Python level PGO is prepared, but only a future release will feature it. With that, scalability improvements as well as even more performance improvements will be unlocked. The amount of optimization added this time is even bigger, some of which unlocks static optimization of module imports, that previously would have to be considered implicit. This work will need one extra step, namely to also trace hard imports on the function level, then this will be an extremely powerful tool to solve these kinds of issues in the future. The next release will have this and go even further in this area. With the dictionary methods, and some string methods, also a whole new kind of optimization has been started. These will make working with Type specialization for Python3 has not progressed though, and will have to be featured in a future releases though. For scalability, the The support for macOS has been refined, with version information being possible to add, and adding information to the binary about which OSes are supported, as well as rejecting Apple Python, which is only a trap if you want to deploy to other OS versions. More work will be needed to support This release achieves major compatibility improvements. And of course, the experimental support for 3.10 is not the least. The next release will strive to complete the support for it fully, but this should be usable at least, for now please stay on 3.9 if you can.Bug Fixes
StopAsyncIteration rather than StopIteration in some situations to be fully compatible.type calls with 3 arguments didn’t annotate their potential exception exit. Fixed in 0.6.17.2 already.pkg-resources exiting with error at compile time for unresolved requirements in compiled code, but these can of course still be optional, i.e. that code would never run. Instead give only a warning, and run time fail on these. Fixed in 0.6.17.2 already.drm libraries on Linux, they have to come from the target OS at run time. Fixed in 0.6.17.2 already.ipcqueue module. Fixed in 0.6.17.3 already.PySide2 wasn’t working properly. Partially fixed in 0.6.17.3 already.ccache binary for M1 architecture and systems before macOS 10.14 as it doesn’t work on these. Fixed in 0.6.17.3 already.pendulum.locals handling for Python 3.6 was regressed. Fixed in 0.6.17.4 already.xmlschema. Fixed in 0.6.17.4 already.curses on Windows. Fixed in 0.6.17.4 already.coincurve module. Fixed in 0.6.17.5 already.win32print. Fixed in 0.6.17.6 already.libpython in module mode or else on some Python versions, linker errors can happen. Fixed in 0.6.17.6 already.site module early anymore. This might have caused issues in some configurations, but really only would be needed for loading inspect which doesn`t depend on it in standalone mode. Fixed in 0.6.17.6 already.try:
return 9
finally:
"".join(x for x in b"some_iterable")
types.CoroutineType and types.AsyncGeneratorType types was not yet implemented. Fixed in 0.6.17.7 already.# These already worked:
assert isinstance(compiledCoroutine(), types.CoroutineType) is True
assert isinstance(compiledAsyncgen(), types.AsyncGeneratorType) is True
# These now work too:
assert type(compiledCoroutine()) == types.CoroutineType
assert type(compiledAsyncgen()) == types.AsyncGeneratorType
ruamel.yaml. Fixed in 0.6.17.7 already.importlib_resources insisting on Python source files to exist to be able to load datafiles. Fixed in 0.6.17.7 already.--follow-stdlib mode.opensapi_spec_validator..cmd file wasn’t containing all needed environment.xmlschema package datafiles.eel package datafiles.h5py package.phonenumbers package.feedparser package, this currently depends on the anti-bloat plugin to be enabled, which will become enabled by default in the future.gi plugin for said package that copies typelib files and sets the search path for them in standalone mode.eel package.QtWebEngine to all Qt bindings and also make it work on Linux. Before only PySide2 on Windows was supported.all built-in was wrongly assuming that bytes values could not be false, but in fact they are if they contain \0 which is actually false. The same does not happen for string values, but that’s a difference to be considered.__package__ value set to a wrong value, which at least impacted new matplotlib detection of Qt backend.python setup.py install was installing binaries for no good reason.New Features
bdist_nuitka and bdist_wheel integration and added support for Nuitka as a build package backend in pyproject.toml files. Using Nuitka to build your wheels is supposed to be easy now.--macos-app-version.Anaconda, pyenv, Apple Python, and pyenv and output the result in version output, this should make it easiert to analyse reported issues.__name__ for metadata version resolution of the pkg-resources standard plugin.data-files standard plugin now reads configuration from a Yaml file that data-files.yml making it more accessible for contributions.--msvc=latest. This allows you to prevent accidental usage of MinGW64 on Windows, when MSVC is intended, but achieves that without fixing the version to use.--debugger with the downloaded MinGW64 provided gdb.exe.PyObject_IsInstance work, providing support for packages like pydantic.pyqtgraph selection of binding by hard coding QT_LIB. This will allow to resolve its own dynamic imports depending on that variable at compile time. At this time, the compile time analysis is not covering all cases yet, but we hope to get there.minOS for standalone builds, derived from the setting of the Python used to create it.--disable-ccache to prevent Nuitka from injecting ccache (Clang, gcc) and clcache (MSVC) for caching the C results of the compilation.PyQt6. While using PySide2 or PySide6 is very much recommended with Nuitka, this allows its use.--low-memory to allow the user to specify that the compilation should attempt to use less memory where possible, this increases compile times, but might enable compilation on some weaker machines.Optimization
dict for both Python2 and Python3, namely get, items, iteritems, itervalues, iterkeys, viewvalues, viewkeys, pop, setdefault, has_key, clear, copy, update.join, strip, lstrip, rstrip, partition, rpartition. Besides performance, this subset was enough to cover compile time evaluation of module name computation for importlib.import_module as done by SWIG bindings, allowing these implicit dependencies to be discovered at compile time without any help, marking a significant improvement for standalone usage.in/not in nodes, this was missing unlike in the generic in/not in nodes.something() # ignores return value, means statement only node
requests.packages and six.moves. The old handling could duplicate their code. Now uses a new mechanism to resolve metapath based importer effects at compile time.anti-bloat configuration for main programs present in the modules of the standard library, these can be removed from the compilation and should lower dependencies detected.pyenv automatically. This should give both smaller (standalone mode) and faster results as is the case when using this feature..anti-bloat plugin for gevent to avoid including its testing framework.importlib.import_module calls with constant args into fixed name imports.sys.version_info to be used as a compile time constant. This should enable many checks to be done at compile time.typing.TYPE_CHECKING.import sys
...
if sys.version_info.major >= 3:
...
Organizational
Buy Now buttons available now for the direct purchase of the Nuitka Commercial offering. Finally Credit Card, Google Pay, and Apple Pay are all possible. This is using Stripe. Get in touch with me if you want to use bank transfer, which is of course still best for me.nuitka2 and nuitka2-run, which is consistent with what we do for Python3, and avoids issues where bin folder ends up in sys.path and prevents the loading of nuitka package.nuitka-project style options.--include-package with a hint how to exclude some of the subpackages.--version output for improved bug reports.--version output for improved bug reports.gevent plugin, we want to achieve this for all plugins, and this is only a step in that direction.rst2pdf applied.black, isort, pylint versions.nuitka and nuitka-run to nuitka2 and nuitka2-run. This harmonizes it with Python2 and avoids issues, where the bin folder in sys.path can cause issues with re-execution of Nuitka finding those to import.python -m nuitka style of calling Nuitka anyway, as it gives you best control over what Python is used to run Nuitka, you can pick python2 there if you want it to run with that, even with full path. Check the relevant section in the User Manual too.Cleanups
--enable-plugin has become the only form to enable a plugin used in documentation and tests.numpy and Qt binding plugins, e.g. pyside2. Data files and DLLs are now provided through proper declarative objects rather than copied manually. The handling of PyQt5 from the plugin should have improved as a side effect.getPythonFlags() and made the function private, replacing their use with dedicated function to check for individual flags.nuitka.utils.getOS() and instead add accessors that are more readable, e.g. nuitka.utils.isMacOS() and put them to use where it makes sense.nuitka.utils.Json module, as we intend to expand its usage, e.g. for caching.print functions with uses of nuitka.Tracing instead.gevent plugin, user proper method to execute code after module load, rather than source patching without need. The plugin no longer messes with inclusions that other code already provides for standalone.sys module attributes, to avoid errors from old C compilers, and also cleaning up using code to not have to cast on string constants.Tests
Summary
dict containers faster, but obviously a lot of ground is to cover there still, e.g. list values are a natural target not yet started. Future releases will progress here.anti-bloat work has continued, and this should be the last release, where this is not on by default. Compiling without it is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.pyenv or even Homebrew there too, for now CPython is still the recommended platform to use.