Nuitka Release 1.0
This release contains a large amount of new features, while consolidating what we have with many bug fixes. Scalability should be dramatically better, as well as new optimization that will accelerate some code quite a bit. See the summary, how this release is paving the way forward.
errorsargument given was not working. Fixed in 0.9.1 already.
MSYS2: Fix, the accelerate mode
.cmdfile was not working correctly. Fixed in 0.9.1 already.
Onefile: Fix, the bootstrap when waiting for the child, didn’t protect against signals that interrupt this call. This only affected users of the non-public
--onefile-tempdiroption on Linux, but with that becoming the default in 1.0, this was discovered. Fixed in 0.9.1 already.
pkg_resourcescompile time generated
Distributionvalues could cause issues with code that put it into calls, or in tried blocks. Fixed in 0.9.1 already.
Standalone: Added implicit dependencies of
Xlibpackage. Fixed in 0.9.1 already.
macOS: Fix, the package configuration for
wxhad become invalid when restructuring the Yaml with code and schema disagreeing on allowed values. Fixed in 0.9.1 already.
str.formatwith a single positional argument didn’t generate proper code and failed to compile on the C level. Fixed in 0.9.1 already.
Fix, the type shape of
str.countresult was wrong. Fixed in 0.9.1 already.
UI: Fix, the warning about collision of just compiled package and original package in the same folder hiding the compiled package should not apply to packages without an
__init__.pyfile, as those do not take precedence. Fixed in 0.9.2 already.
Debugging: Fix, the fallback to
gdbwhen using the option
--debuggerwas broken on anything but Windows. Fixed in 0.9.2 already.
Python3.8: The module
importlib.metadatawas not recognized before 3.9, but actually 3.8 already has it, causing the compile time resolution of package versions to not work there. Fixed in 0.9.3 already.
Standalone: Fix, at least on macOS we should also scan from parent folders of DLLs, since they may contain sub-directories in their names. This is mostly the case, when using frameworks. Fixed in 0.9.2 already.
Standalone: Added package configuration for
PyQt5to require onefile bundle mode on macOS, and recommend to disable console for PyQt6. This is same as we already do for
PySide6. Fixed in 0.9.2 already.
Standalone: Removed stray macOS onefile bundle package configuration for
picklemodule which must have been added in error. Fixed in 0.9.2 already.
UI: Catch user error of attempting to compile the
__init__.pyrather than the package directory. Fixed in 0.9.2 already.
Fix, hard name import nodes failed to clone, causing issues in optimization phase. Fixed in 0.9.2 already.
Fix, avoid warnings given with gcc 11. Fixed in 0.9.2 already.
Fix, dictionary nodes where the operation itself has no effect, e.g.
dict.copywere not properly annotating that their dictionary argument could still cause a raise and have side effects, triggering an assertion violation in Nuitka. Fixed in 0.9.2 already.
pynputimplicit dependencies on Linux. Fixed in 0.9.2 already.
Fix, boolean condition checks on variables converted immutable constant value assignments to boolean values, leading to incorrect code execution. Fixed in 0.9.2 already.
Python3.9: Fix, could crash on generic aliases with non-hashable values. Fixed in 0.9.3 already.
Python3: Fix, an iteration over
sys.version_infowas falsely optimized into a tuple, which is not always compatible. Fixed in 0.9.3 already.
Standalone: Added support for
xgboostpackage. Fixed in 0.9.3 already.
Standalone: Added data file for
text_unidecodepackage. Fixed in 0.9.4 already.
Standalone: Added data files for
swagger_ui_bundlepackage. Fixed in 0.9.4 already.
Standalone: Added data files for
connexionpackage. Fixed in 0.9.4 already.
Standalone: Added implicit dependencies for
rapidfuzz. Fixed in 0.9.4 already.
Python3.10: Fix, the reformulation of
matchstatements could create nodes that are used twice, causing code generation to assert. Fixed in 0.9.4 already.
Fix, module objects removed from
sys.modulesbut still used could lack a reference to themselves, and therefore crash due to working on a released module variables dictionary. Fixed in 0.9.5 already.
Fix, the MSVC compiles code generated for SciPy 1.8 wrongly. Added a workaround for that code to avoid triggering it. Fixed in 0.9.6 already.
Fix, calls to
str.formatwhere the result is not used, could crash the compiler during code generation. Fixed in 0.9.6 already.
Standalone: For DLLs on macOS and Anaconda, also consider the
libdirectory of the root environment, as some DLLs are otherwise not found.
__class__to be used on the class level.
xrangewith large values didn’t work on all platforms. This affected at least Python2 on macOS, but potentially others as well.
Windows: When scanning for installed Pythons to e.g. run Scons or onefile compression, it was attempting to use installations that got deleted manually and could crash.
macOS: Fix, DLL conflicts are now resolved by checking the version information too, also all cases that previously errored out after a conflict was reported, will now work.
Fix, conditional expressions whose statically decided condition picking a branch will raise an exception could crash the compilation.
# Would previously crash Nuitka during optimization. return 1/0 if os.name == "nt" else 1/0
Windows: Make sure we set C level standard file handles too
At least newer subprocess was affected by this, being unable to provide working handles to child processes that pass their current handles through, and also this should help DLL code to use it as level.
Standalone: Added support for
Standalone: Added support for
dipyby anti-bloat removal of its testing framework that wants to do unsupported stuff.
UI: Could still give warnings about modules not being followed, where that was not true.
--include-modulewas not working for non-automatic standard library paths.
Onefile: Recognize a non-changing path from
--onefile-tempdir-specand then use cached mode. By default a temporary folder is used in the spec value, make it delete the files afterwards.
The cached mode is not necessarily faster, but it is not going to change files already there, leaving the binaries there intact. In the future it may also become faster to execute, but right now checking the validity of the file takes about as long as re-creating it, therefore no gain yet. The main point, is to not change where it runs from.
Standalone: Added option to exclude DLLs. You can npw use
--noinclude-dllsto exclude DLLs by filename patterns.
The may e.g. come from Qt plugins, where you know, or experimented, that it is not going to be used in your specific application. Use with care, removing DLLs will lead to very hard to recognize errors.
CondaCCfrom environment variables for Linux and macOS, in case it is installed. This can be done with e.g.
conda install gcc_linux-64on Linux or
conda install clang_osx-64on macOS.
Added new option
--nowarn-mnemonicto disable warnings that use mnemonics, there is currently not that many yet, but it’s going to expand. You can use this to acknowledge the ones you accept, and not get that warning with the information pointer anymore.
Added method for resolving DLL conflicts on macOS too. This is using version information and picks the newer one where possible.
--user-package-configuration-filefor user provided Yaml files, which can be used to provide package configuration to Nuitka, to e.g. add DLLs, data files, do some anti-bloat work, or add missing dependencies locally. The documentation for this does not yet exist though, but Nuitka contains a Yaml schema in the
nuitka-project-elseto avoid repeating conditions in Nuitka project configuration, this can e.g. be used like this:
# nuitka-project-if: os.getenv("TEST_VARIANT", "pyside2") == "pyside2": # nuitka-project: --enable-plugin=no-qt # nuitka-project-else: # nuitka-project: --enable-plugin=no-qt # nuitka-project: --noinclude-data-file=*.svg
Previously, the inverted condition had to be used in another
nuitka-project-ifwhich is no big deal, but less readable.
Added support for deep copying uncompiled functions. There is now a section in the User Manual that explains how to clone compiled functions. This allows a workaround like this:
def binder(func, name): try: result = func.clone() except AttributeError: result = types.FunctionType(func.__code__, func.__globals__, name=func.__name__, argdefs=func.__defaults__, closure=func.__closure__) result = functools.update_wrapper(result, func) result.__kwdefaults__ = func.__kwdefaults__ result.__name__ = name return result
Plugins: Added explicit deprecation status of a plugin. We now have a few that do nothing, and are just there for compatibility with existing users, and this now informs the user properly rather than just saying it is not relevant.
Fix, some Python installations crash when attempting to import modules, such as
ModuleNameobject, because we limit string operations done, and e.g. refuse to do
.startswithwhich of course, other loaders that your installation has added, might still use.
Windows: In case of not found DLLs, we can still examine the run time of the currently compiling Python process of Nuitka, and locate them that way, which helps for some Python configurations to support standalone, esp. to find CPython DLL in unusual spots.
Debian: Workaround for
lib2to3data files. These are from stdlib and therefore the patched code from Debian needs to be undone, to make these portable again.
Scalability: Avoid merge traces of initial variable versions, which came into play when merging a variable used in only one branch. These are useless and only made other optimization slower or impossible.
Scalability: Also avoid merge traces of merge traces, instead flatten merge traces and avoid the duplication doing so. There were pathological cases, where this reduced optimization time for functions from infinite to instant.
For comparison helpers, switch comparison where possible, such that there are only 3 variants, rather than 6. Instead the boolean result is inverted, e.g. changing
not <effectively. Of course this can only be done for types, where we know that nothing special, i.e. no method overloads of
__gte__is going on.
For binary operations that are commutative with the selected types, in mixed type cases, swap the arguments during code generation, such that e.g.
long_a + float_bis actually computed as
float_b + long_a. This again avoids many helpers. It also can be done for
*with integers and container types.
In cases, where a comparison (or one of the few binary operation where we consider it useful), is used in a boolean context, but we know it is impossible to raise an exception, a C boolean result type is used rather than a
nuitka_boolwhich is now only used when necessary, because it can indicate the exception result.
Anti-Bloat: More anti-bloat work was done for popular packages, covering also uses of
nose2package removals and warnings. There was also a focus on making
tifffilecompile well, removing e.g. the uses of the tensorflow testing framework.
Faster comparison of
intvalues with constant values, this uses helpers that work with C
longvalues that represent a single “digit” of a value, or ones that use the full value space of C
Faster comparison of
floatvalues with constant values, this uses helpers that work with C
floatvalues, avoiding the useless Python level constant objects.
Python2: Comparison of
longnow has specialized helpers that avoids converting the
longthrough coercion. This takes advantage of code to compare C
longvalues (which are at the core of Python2
For binary operation on mixed types, e.g.
int * bytesthe slot of the first function was still considered, and called to give a
Py_NotImplementedreturn value for no good reason. This also applies to mixed operations of
floattypes, and for
unicodevalues on Python2.
Added missing helper for
**operation with floats, this had been overlooked so far.
Added dedicated nodes for
ctypes.CDLLwhich aims to allow us to detect used DLLs at compile time in the future, and to move closer to support its bindings more efficiently.
Added specialized nodes for
dict.popitemas well. With this, now all of the dictionary methods are specialized.
Added specialized nodes for
str.splitlines. While these are barely performance relevant, this completes all
removesuffixthat are Python3.9 or higher.
Added type shape for result of
str.indexoperation as well, this was missing so far.
dictmethod calls through variables.
Optimize calls through variables containing e.g. mutable constant values, these will be rare, because they all become exceptions.
Optimize calls through variables containing built-in values, unlocking optimization of such calls, where it is assigned to a local variable.
For generated attribute nodes, avoid local doing import statements on the function level. While these were easier to generate, they can only be slow at run time.
strbuilt-in annotate its value as derived from
str, which unfortunately does not allow much optimization, since that can still change many things, but it was still a missing attribute.
For variable value release nodes, specialize them by value type as well, enhancing the scalability, because e.g. parameter variable specific tests, need not be considered for all other variable types as well.
Plugins: Major changes to the Yaml file content, cleaning up some of the DLL configuration to more easy to use.
The DLL configuration has two flavors, one from code and one from filename matching, and these got separated into distinct items in the Yaml configuration. Also how source and dest paths get provided got simplified, with a relative path now being used consistently and with sane defaults, deriving the destination path from where the module lives. Also what we called patterns, are actually prefixes, as there is still the platform specific DLL file naming appended.
Plugins: Move mode checks to dedicated plugin called
options-nannythat is always enabled, giving also much cleaner Yaml configuration with a new section added specifically for these. It controls advice on the optional or required use of
--disable-consoleand the like. Some packages, e.g.
wxare known to crash on macOS when the console is enabled, so this advice is now done with saner configuration.
Plugins: Also for all Yaml configuration sub-items where is now a consistent
whenfield, that allows checking Python version, OS, Nuitka modes such as standalone, and only apply configuration when matching this criterion, with that the anti-bloat options to allow certain bloat, should now have proper effect as well.
The use of
AppImageon Linux is no more. The performance for startup was always slower, while having lost the main benefit of avoiding IO at startup, due to new cached mode, so now we always use the same bootstrap binary as on macOS and Windows.
UI: Do not display implicit reports reported by plugins by default anymore. These have become far too many, esp. with the recent stdlib work, and often do not add any value. The compilation report will become where to turn to find out why a module in included.
UI: Ask the user to install the ordered set package that will actually work for the specific Python version, rather than making him try one of two, where sometimes only one can work, esp. with Python 3.10 allowing only one.
GitHub: More clear wording in the issue template that
python -m nuitka --versionoutput is really required for support to given.
Attempt to use Anaconda
ccachebinary if installed on non-Windows. This is esp. handy on macOS, where it is harder to get it.
Windows: Avoid byte-compiling the inline copy of Scons that uses Python3 when installing for Python2.
Added experimental switches to disable certain optimization in order to try out their impact, e.g. on corruption bugs.
Reports: Added included DLLs for standalone mode to compilation report.
Reports: Added control tags influencing plugin decisions to the compilation report.
Plugins: Make the
implicit-importsdependency section in the Yaml package configuration a list, for consistency with other blocks.
Plugins: Added checking of tags such from the package configuration, so that for things dependent on python version (e.g.
before_python39), the usage of Anaconda (
anaconda) or certain OS (e.g.
macos), or modes (e.g.
standalone), expressions in
whencan limit a configuration item.
Quality: Re-enabled string normalization from black, the issues with changes that are breaking to Python2 have been worked around.
User Manual: Describe using a minimal virtualenv as a possible help low memory situations as well.
Quality: The yaml auto-format now properly preserves comments, being based on
Nuitka-Python: Added support for the Linux build with Nuitka-Python for our own CPython fork as well, previously only Windows was working, amd macOS will follow later.
The commit hook when installed from git bash was working, but doing so from
cmd.exedidn’t find a proper path for shell from the
Debugging: A lot of experimental toggles were added, that allow control over the use of certain optimization, e.g. use of dict, list, iterators, subscripts, etc. internals, to aid in debugging in situations where it’s not clear, if these are causing the issue or not.
Added support for Fedora 36, which requires some specific linker options, also recognize Fedora based distributions as such.
Removed long deprecated option
--noinclude-matplotlibfrom numpy plugin, as it hasn’t had an effect for a long time now.
Visual Code: Added extension for editing Jinja2 templates. This one even detects that we are editing C or Python and properly highlights accordingly.
Standalone: Major cleanup of the dependency analysis for standalone. There is no longer a distinction between entry points (main binary, extension modules) and DLLs that they depend on. The OS specific parts got broken out into dedicated modules as well and decisions are now taken immediately.
Plugins: Split the Yaml package configuration files into 3 files. One contains now Python2 only stdlib configuration, and another one general stdlib.
Plugins: Also cleanup the
zmqplugin, which was one the last holdouts of now removed plugin method, moving parts to the Yaml configuration. We therefore no longer have
considerExtraDllswhich used to work on the standalone folder, but instead only plugin code that provides included DLL or binary objects from
getExtraDllswhich gives Nuitka much needed control over DLL copying. This was a long lasting battle finally won, and will allow many new features to come.
UI: Avoid changing whitespace in warnings, where we have intended line breaks, e.g. in case of duplicate DLLs. Went over all warnings and made sure to either avoid new-lines or have them, depending on wanted output.
Iterator end check code now uses the same code as rich comparison expressions and can benefit from optimization being done there as well.
Solved TODO item about code generation time C types to specify if they have error checking or not, rather than hard coding it.
Production of binary helper function set was cleaned up massively, but still needs more work, comparison helper function set was also redesigned.
Changing the spelling of our container package to become more clear.
namedtupleobjects for storing used DLL information for more clear code.
Added spellchecker ignores for all attribute and argument names of generated fixed attribute nodes.
In auto-format make sure the imports float to the top. That very much cleans up generated attribute nodes code, allowing also to combine the many ones it makes, but also cleans up some of our existing code.
The package configuration Yaml files are now sorted according to module names. This will help to avoid merge conflicts during hotfixes merge back to develop and automatically group related entries in a sane way.
Moved large amounts of code producing implicit imports to Yaml configuration files.
tensorflowplugin to Yaml based configuration, making it a deprecated do nothing plugin, that only remains there for a few releases, to not crash existing build scripts.
Lots of spelling cleanups, e.g. renaming
Added generated test to cover
bytesmethod. This would have found the issue with
Enhanced standalone test for
ctypeson Linux to actually have something to test.
This release improves on many things at once. A lot of work has been put into polishing the Yaml configuration that now only lacks documentation and examples, such that the community as a whole should become capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.
Then a lot of new optimization has been done, to close the missing gaps
str methods, but before completing
is already a work in progress pull request, and
bytes, we want to
start and generate the node classes that form the link or basis of
dedicated nodes. This will be an area to work on more.
The many improvements to existing code helpers, and them being able to
pick target types for the arguments of comparisons and binary
operations, is a pre-cursor to universal optimization of this kind. What
is currently only done for constant values, will in the future be
interesting for picking specific C types for use. That will then be a
huge difference from what we are doing now, where most things still have
PyObject * based types.
Scalability has again seen very real improvements, memory usage of Nuitka itself, as well as compile time inside Nuitka are down by a lot for some cases, very noticeably. There is never enough of this, but it appears, in many cases now, large compilations run much faster.
For macOS specifically, the new DLL dependency analysis, is much more capable or resolving conflicts all by itself. Many of the more complex packages with some variants of Python, specifically Anaconda will now be working a lot better.
And then, of course there is the big improvement for Onefile, that allows to use cached paths. This will make it more usable in the general case, e.g. where the firewall of Windows hate binaries that change their path each time they run.
Future directions will aim to make the compilation report more concise, and given reasons and dependencies as they are known on the inside more clearly, such that is can be a major tool for testing, bug reporting and analysis of the compilation result.