11 December 2021
Nuitka Release 0.6.18
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release has a focus on new features of all kinds, and then also new kinds of performance improvements, some of which enable static optimization of what normally would be dynamic imports, while also polishing plugins and adding also many new features and a huge amount of organizational changes. Python3.6+: Fixes to asyncgen, need to raise Onefile: Fix, LTO mode was always enabled for onefile compilation, but not all compilers support it yet, e.g. MinGW64 did not. Fixed in 0.6.17.1 already. Fix, Fix, trusted module constants were not working properly in all cases. Fixed in 0.6.17.2 already. Fix, Standalone: Prevent the inclusion of Standalone: Added missing implicit dependency for Fix, Qt webengine support for everything but Windows: Fix, bootstrap splash screen code for Windows was missing in release packages. Fixed in 0.6.17.3 already. Fix, could crash on known implicit data directories not present. Fixed in 0.6.17.3 already. macOS: Disable download of Standalone: The Onefile: Make sure the child process is cleaned up even after its successful exit. Fixed in 0.6.17.4 already. Standalone: Added support for Standalone: Added support for Standalone: Added support for Python3.4+: Up until Python3.7 inclusive, a workaround for stream encoding (was ASCII), causing crashes on output of non-ASCII, other Python versions are not affected. Fixed in 0.6.17.5 already. Python2: Workaround for LTO error messages from older gcc versions. Fixed in 0.6.17.5 already. Standalone: Added support for Fix, need to prevent usage of static Standalone: Do not load Fix, could crash with generator expressions in finally blocks of tried blocks that return. Fixed in 0.6.17.7 already. Python3.5+: Compatibility of comparisons with Standalone: Added support for Distutils: Fix, when building more than one package, things could go wrong. Fixed in 0.6.17.7 already. Fix, for module mode filenames are used, and for packages, you can specify a directory, however, a trailing slash was not working. Fixed in 0.6.17.7 already. Compatibility: Fix, when locating modules, a package directory and an extension module of the same name were not used according to priority. Fixed in 0.6.17.7 already. Standalone: Added workaround Standalone: Properly detect usage of hard imports from standard library in Standalone: Added data files for MSYS2: Fix, need to normalize compiler paths before comparing. Anaconda: For accelerated binaries, the created macOS: Set minimum OS version derived from the Python executable used, this should make it work on all supported platforms (of that Python). Standalone: Added support for automatic inclusion of Standalone: Added support for automatic inclusion of Standalone: Added support for Standalone: Added support for Standalone: Added support for Standalone: Added Standalone: Added necessary datafiles for Standalone: Added support for Python3: Fix, the Windows: The LTO was supposed to be used automatically on with MSVC 14.2 or higher, but that was regressed and has been repaired now. Standalone: Extension modules contained in packages, depending on their mode of loading had the Windows: The Setuptools support. Documented Added experimental support for Python 3.10, there are however still important issues with compatibility with the CPython 3.9 test suite with at least asyncgen and coroutines. macOS: For app bundles, version information can be provided with the new option Added Python vendor detection of Plugins: Also handle the usage of Plugins: The Windows: Allow enforcing usage of MSVC with Windows: Added support for LTO with MinGW64 on Windows, this was previously limited to the MSVC compiler only. Windows: Added support for using Note It doesn`t work when executed from a Git bash prompt, but e.g. from a standard command prompt. Added new experimental flag for compiled types to inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses Plugins: The Qt binding plugins now resolve macOS: Provide UI: Added new option Plugins: Added experimental support for UI: Added option Added dedicated attribute nodes for attribute values that match names of dictionary operations. These are optimized into dedicate nodes for methods of dictionaries should their expression have an exact dictionary shape. These in turn optimize calls on them statically into dictionary operations. This is done for all methods of The new operation nodes also add compile time optimization for being used on constant values where possible. Also added dedicated attribute nodes for string operations. For operations, currently only part of the methods are done. These are currently only Annotate type shape for dictionary Faster processing of “expression only” statement nodes. These are nodes, where a value is computed, but then not used, it still needs to be accounted for though, representing the value release. Windows: Enabled LTO by default with MinGW64, which makes it produce much faster results. It now yield faster binaries than MSVC 2019 with pystone. Windows: Added support for C level PGO (Profile Guided Optimization) with MSVC and MinGW64, allowing extra speed boosts from the C compilation on Windows as well. Standalone: Better handling of Avoid useless exception checks in our dictionary helpers, as these could only occur when working with dictionary overloads, which we know to not be the case. For nodes, have dedicated child mixin classes for nodes with a single child value and for nodes with a tuple of children, so that these common kind of nodes operate faster and don’t have to check at run time what type they are during access. Actually make use of the egg cache. Nuitka was unpacking eggs in every compilation, but in wheel installs, these can be quite common and should be faster. Star arguments annotated their type shape, but the methods to check for dictionary exactly were not affected by this preventing optimization in some cases. Added Using static libpython with Plugins: Added improvements to the Python3.9+: Faster calls into uncompiled functions from compiled code using newly introduced API of that version. Statically optimize Added support for Added hard import and static optimization for Also compute named import lookup through variables, expanding their use to more cases, e.g. like this: Also optimize compile time comparisons through variable names if possible, i.e. the value cannot have changed. Faster calls of uncompiled code with Python3.9 or higher avoiding DLL call overhead. Commercial: There are The main script runners for Python2 have been renamed to Windows: Added support for Visual Studio 2022 by updating the inline copy of Scons used for Windows to version 4.3.0, on non Windows, the other ones will keep being used. Windows: Requiring latest MinGW64 with version 11.2 as released by winlibs, because this is known to allow LTO, where previous releases were missing needed binaries. Reject standalone mode usage with Apple Python, as it works only with the other supported Pythons, avoiding pitfalls in attempting to distribute it. Move hosting of documentation to Sphinx, added Changelog and some early parts of API documentation there too. This gives much more readable results than what we have done so far with Nikola. More things will move there. User Manual: Add description how to access code attributes in User Manual: Added commands used to generate performance numbers for Python. User Manual: List other Python’s for which static linking is supposed to work. Improved help for Started using Jinja2 in code templates with a few types, adding basic infrastructure to do that. This will be expanded in the future. Updated plugin documentation with more recent information. Added Python flavor as detected to the Linux: Added distribution name to Always enable the Added project URLs for PyPI, so people looking at it from there have some immediate places to checkout. Debian: Use common code for included PDF files, which have page styles and automatic corrections for Updated to latest The binary names for Python2 changed from Note You ought to be using Added support for Fedora 34 and Fedora 35. In a change of mind Massive cleanup of Massive cleanups of all documentation in ReST format. Plenty of formatting errors were resolved. Many typos were identified and globally fixed. Spellings e.g. of “Developer Manual” are now enforced with automatic replacements. Also missing or wrong quotes were turned to proper methods. Also enforce code language for shell scripts to be the same everywhere. Removed last usages of Avoid string comparison with Replaced usages of string tests in list of python flags specified, with functions that check for a specific name with a speaking function name. Added mixin for expressions that have no side effect outside of their value, providing common method implementation more consistently. Remove code geared to using old PyLint and on Python2, we no longer use that. Also removed annotations only used for overriding Python2 builtins from Nuitka code. The PDF specific annotations were moved into being applied only in the PDF building step, avoiding errors for raw PDF directives. Apply Visual Code auto-format to our Yaml files. This is unfortunately not and automatic formatting yet. Introduce dedicated Replacing remaining usages of Massive cleanup of the Using own helper to update More consistent naming of plugin classes, and enforce a relationship of detector class names to the names of detected plugins. The new naming consistency is now enforced. Added CPython 3.10 test suite, it needs more work though. Added generated test that exercises dictionary methods in multiple variations. Test suite names were specified wrongly in a few of them. This release is again a huge step forward. It refines on PGO and LTO for C level to work with all relevant compilers. Internally Python level PGO is prepared, but only a future release will feature it. With that, scalability improvements as well as even more performance improvements will be unlocked. The amount of optimization added this time is even bigger, some of which unlocks static optimization of module imports, that previously would have to be considered implicit. This work will need one extra step, namely to also trace hard imports on the function level, then this will be an extremely powerful tool to solve these kinds of issues in the future. The next release will have this and go even further in this area. With the dictionary methods, and some string methods, also a whole new kind of optimization has been started. These will make working with Type specialization for Python3 has not progressed though, and will have to be featured in a future releases though. For scalability, the The support for macOS has been refined, with version information being possible to add, and adding information to the binary about which OSes are supported, as well as rejecting Apple Python, which is only a trap if you want to deploy to other OS versions. More work will be needed to support This release achieves major compatibility improvements. And of course, the experimental support for 3.10 is not the least. The next release will strive to complete the support for it fully, but this should be usable at least, for now please stay on 3.9 if you can.Bug Fixes
StopAsyncIteration
rather than StopIteration
in some situations to be fully compatible.type
calls with 3 arguments didn’t annotate their potential exception exit. Fixed in 0.6.17.2 already.pkg-resources
exiting with error at compile time for unresolved requirements in compiled code, but these can of course still be optional, i.e. that code would never run. Instead give only a warning, and run time fail on these. Fixed in 0.6.17.2 already.drm
libraries on Linux, they have to come from the target OS at run time. Fixed in 0.6.17.2 already.ipcqueue
module. Fixed in 0.6.17.3 already.PySide2
wasn’t working properly. Partially fixed in 0.6.17.3 already.ccache
binary for M1 architecture and systems before macOS 10.14 as it doesn’t work on these. Fixed in 0.6.17.3 already.pendulum.locals
handling for Python 3.6 was regressed. Fixed in 0.6.17.4 already.xmlschema
. Fixed in 0.6.17.4 already.curses
on Windows. Fixed in 0.6.17.4 already.coincurve
module. Fixed in 0.6.17.5 already.win32print
. Fixed in 0.6.17.6 already.libpython
in module mode or else on some Python versions, linker errors can happen. Fixed in 0.6.17.6 already.site
module early anymore. This might have caused issues in some configurations, but really only would be needed for loading inspect
which doesn`t depend on it in standalone mode. Fixed in 0.6.17.6 already.try:
return 9
finally:
"".join(x for x in b"some_iterable")
types.CoroutineType
and types.AsyncGeneratorType
types was not yet implemented. Fixed in 0.6.17.7 already.# These already worked:
assert isinstance(compiledCoroutine(), types.CoroutineType) is True
assert isinstance(compiledAsyncgen(), types.AsyncGeneratorType) is True
# These now work too:
assert type(compiledCoroutine()) == types.CoroutineType
assert type(compiledAsyncgen()) == types.AsyncGeneratorType
ruamel.yaml
. Fixed in 0.6.17.7 already.importlib_resources
insisting on Python source files to exist to be able to load datafiles. Fixed in 0.6.17.7 already.--follow-stdlib
mode.opensapi_spec_validator
..cmd
file wasn’t containing all needed environment.xmlschema
package datafiles.eel
package datafiles.h5py
package.phonenumbers
package.feedparser
package, this currently depends on the anti-bloat
plugin to be enabled, which will become enabled by default in the future.gi
plugin for said package that copies typelib
files and sets the search path for them in standalone mode.eel
package.QtWebEngine
to all Qt bindings and also make it work on Linux. Before only PySide2 on Windows was supported.all
built-in was wrongly assuming that bytes values could not be false, but in fact they are if they contain \0
which is actually false. The same does not happen for string values, but that’s a difference to be considered.__package__
value set to a wrong value, which at least impacted new matplotlib detection of Qt backend.python setup.py install
was installing binaries for no good reason.New Features
bdist_nuitka
and bdist_wheel
integration and added support for Nuitka as a build
package backend in pyproject.toml
files. Using Nuitka to build your wheels is supposed to be easy now.--macos-app-version
.Anaconda
, pyenv
, Apple Python
, and pyenv
and output the result in version output, this should make it easiert to analyse reported issues.__name__
for metadata version resolution of the pkg-resources
standard plugin.data-files
standard plugin now reads configuration from a Yaml file that data-files.yml
making it more accessible for contributions.--msvc=latest
. This allows you to prevent accidental usage of MinGW64 on Windows, when MSVC is intended, but achieves that without fixing the version to use.--debugger
with the downloaded MinGW64 provided gdb.exe
.PyObject_IsInstance
work, providing support for packages like pydantic
.pyqtgraph
selection of binding by hard coding QT_LIB
. This will allow to resolve its own dynamic imports depending on that variable at compile time. At this time, the compile time analysis is not covering all cases yet, but we hope to get there.minOS
for standalone builds, derived from the setting of the Python used to create it.--disable-ccache
to prevent Nuitka from injecting ccache
(Clang, gcc) and clcache
(MSVC) for caching the C results of the compilation.PyQt6
. While using PySide2
or PySide6
is very much recommended with Nuitka, this allows its use.--low-memory
to allow the user to specify that the compilation should attempt to use less memory where possible, this increases compile times, but might enable compilation on some weaker machines.Optimization
dict
for both Python2 and Python3, namely get
, items
, iteritems
, itervalues
, iterkeys
, viewvalues
, viewkeys
, pop
, setdefault
, has_key
, clear
, copy
, update
.join
, strip
, lstrip
, rstrip
, partition
, rpartition
. Besides performance, this subset was enough to cover compile time evaluation of module name computation for importlib.import_module
as done by SWIG bindings, allowing these implicit dependencies to be discovered at compile time without any help, marking a significant improvement for standalone usage.in
/not in
nodes, this was missing unlike in the generic in
/not in
nodes.something() # ignores return value, means statement only node
requests.packages
and six.moves
. The old handling could duplicate their code. Now uses a new mechanism to resolve metapath based importer effects at compile time.anti-bloat
configuration for main programs present in the modules of the standard library, these can be removed from the compilation and should lower dependencies detected.pyenv
automatically. This should give both smaller (standalone mode) and faster results as is the case when using this feature..anti-bloat
plugin for gevent
to avoid including its testing framework.importlib.import_module
calls with constant args into fixed name imports.sys.version_info
to be used as a compile time constant. This should enable many checks to be done at compile time.typing.TYPE_CHECKING
.import sys
...
if sys.version_info.major >= 3:
...
Organizational
Buy Now
buttons available now for the direct purchase of the Nuitka Commercial offering. Finally Credit Card, Google Pay, and Apple Pay are all possible. This is using Stripe. Get in touch with me if you want to use bank transfer, which is of course still best for me.nuitka2
and nuitka2-run
, which is consistent with what we do for Python3, and avoids issues where bin
folder ends up in sys.path
and prevents the loading of nuitka
package.nuitka-project
style options.--include-package
with a hint how to exclude some of the subpackages.--version
output for improved bug reports.--version
output for improved bug reports.gevent
plugin, we want to achieve this for all plugins, and this is only a step in that direction.rst2pdf
applied.black
, isort
, pylint
versions.nuitka
and nuitka-run
to nuitka2
and nuitka2-run
. This harmonizes it with Python2 and avoids issues, where the bin
folder in sys.path
can cause issues with re-execution of Nuitka finding those to import.python -m nuitka
style of calling Nuitka anyway, as it gives you best control over what Python is used to run Nuitka, you can pick python2
there if you want it to run with that, even with full path. Check the relevant section in the User Manual too.Cleanups
--enable-plugin
has become the only form to enable a plugin used in documentation and tests.numpy
and Qt binding plugins, e.g. pyside2
. Data files and DLLs are now provided through proper declarative objects rather than copied manually. The handling of PyQt5 from the plugin should have improved as a side effect.getPythonFlags()
and made the function private, replacing their use with dedicated function to check for individual flags.nuitka.utils.getOS()
and instead add accessors that are more readable, e.g. nuitka.utils.isMacOS()
and put them to use where it makes sense.nuitka.utils.Json
module, as we intend to expand its usage, e.g. for caching.print
functions with uses of nuitka.Tracing
instead.gevent
plugin, user proper method to execute code after module load, rather than source patching without need. The plugin no longer messes with inclusions that other code already provides for standalone.sys
module attributes, to avoid errors from old C compilers, and also cleaning up using code to not have to cast on string constants.Tests
Summary
dict
containers faster, but obviously a lot of ground is to cover there still, e.g. list
values are a natural target not yet started. Future releases will progress here.anti-bloat
work has continued, and this should be the last release, where this is not on by default. Compiling without it is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.pyenv
or even Homebrew there too, for now CPython is still the recommended platform to use.