23 March 2024
Nuitka Release 2.1
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release had focus on new features and new optimization. There is a also a large amount of compatibility with things newly added to support anti-bloat better, and workaround problems with newer package versions that would otherwise need source code at run-time. Windows: Using older MSVC before 14.3 was not working anymore. Fixed in 2.0.1 already. Compatibility: The Windows: Fix, sometimes UI: The help output for Standalone: Added support for the Standalone: Added Standalone: Added support for Standalone: Added missing data file for Standalone: Added missing DLLs for Standalone: Added missing DLL for UI: Using reporting path in macOS dependency scan error message, otherwise these contain home directory paths for no good reason. Fixed in 2.0.2 already. UI: Fix, could crash when compiling directories with trailing slashes used. At least on Windows, this happened for the “/” slash value. Fixed in 2.0.2 already. Module: Fix, convenience option Python2: Avoid values for Windows: Fix, short paths with and without dir name cached wrongly, which could lead to shorted paths even where not asked for them. Fixed in 2.0.2 already. Fix, comparing list values that changed could segfault. This is a bug fix Python did, that we didn’t follow yet and that became apparent after using our dedicated list helpers more often. Fixed in 2.0.2 already. Standalone: Added support for Standalone: Fix, namespace packages had wrong runtime Python3.11: Fix, was using tuples from freelist of the wrong size CPython changed the index for the size, to not use zero, which was wasteful when introduced with 3.10, but to As a result, code producing a lot short living tuples could end up creating new ones over and over, causing bad memory allocations and slow performance. Fixed in 2.0.2 already. macOS: Fix, need to allow non-existent and versioned dependencies of DLLs to themselves. Fixed in 2.0.2 already. Windows: Fix PGO (Profile Guided Optimization) build errors with MinGW64, this feature is not yet ready for general use, but these errors shouldn’t happen. Fixed in 2.0.2 already. Plugins: Fix, do not load The Plugins: Avoid warnings from plugin evaluated code, it could happen that a Fix, loading pickles with compiled functions in module mode was not working. Fixed in 2.0.3 already. Standalone: Added data files for Fix, variable assignment from variables that started to raise were not recognized. When a variable assignment from a variable became a raise expression, that wasn’t caught and propagated as it should have been. Fixed in 2.0.3 already. Make the Fix, PySide2/6 argument name for slot connection and disconnect should be Standalone: Added support for Standalone: Added support for Standalone: Added support for Standalone: Added support for Fix, added handling of Fix, need to annotate potential exception exit when making a fixed import from hard module attribute. Fixed in 2.0.4 already. Fix, didn’t consider Nuitka project options with Scons: Fix, need to close progress bar when about to error exit. Otherwise error outputs will be garbled by incomplete progress bar. Fixed in 2.0.4 already. Fix, need to convert relative from imports to hard imports too, or else packages needed to be followed are not included. Fixed in 2.0.5 already. Standalone: Added Windows: Fix, wasn’t working when compiling on network mounted drive letters. Fixed in 2.0.6 already. Fix, the Actions: Fix, some yaml configs could fail to load plugins. Fixed in 2.0.6 already. Standalone: Added support for newer Fix, inline copies of Plugins: Added support for Plugins: Make Plugins: Make not only booleans related to the python version available, but also strings Plugins: Added helper function macOS: Added support for Tcl/Tk detection on Homebrew Python. Added So far it was impossible to distinguish non-standalone, i.e. accelerated mode and module compilation by looking at the Plugins: Added Plugins: Added ability to specify modules to not follow when a module is used. This Plugins: Added values UI: Added more control over caching with per cache category environment variables, as documented in the User Manual.. Plugins: Added support for reporting module detections The Scalability: Do not make loop analysis at all for very trusted value traces, their point is to not change, and waiting for that to be confirmed has no point. Use very trusted value traces in functions not just as mere assign traces or else expected optimization will not be done on them in many cases. With this a lot more cases of hard values are optimized leading also to generally more compact and correct results in terms of imports, metadata, code avoided on the wrong OS, etc. Scalability: When specializing assignments, make sure to have the proper value trace immediately. When changing to a hard value, the value trace was still an assign trace and not very trusted for one for micro pass of the module. This had the effect to need one more micro pass to get to benefiting of the unescapable nature of those values, which meant more micro passes than necessary and those being more complex due to escaped traces, and therefore taking longer for affected modules. Scalability: The code trying avoid merge traces of merge traces, and to instead flatten merge traces was only handling part of these correctly, and correcting it reduced optimization time for some functions from infinite to instant. Less memory usage should also come out of this, even where this was not affecting compile time as much. Added in 2.0.1 already. Scalability: Some codes that checked for variables were testing for temporary variable and normal variable both one after another, making some optimization steps and code generation slower than necessary due to the extra calls. Scalability: A variable assignment from variable that were later recognized to become a raise was not recognized as such, and this then wasn’t caught and propagated as it should, preventing more optimization of the affected code. Make sure to convert more directly when observing things to change, rather than doing it one pass later. The fix proper reuse of tuples released to the freelist with matching sizes causes less memory usage and faster performance for the 3.11 version. Added in 2.0.2 already. Statically optimize This should make a bunch of dead code obvious to Nuitka, it can now tell this aborts execution of a branch, potentially eliminating imports, etc. macOS: Enable python static link library for Homebrew too. Added in 2.0.1 already. Added in 2.0.3 already. Avoid compiling bloated module namespace of Anti-Bloat: Avoid including Anti-Bloat: Avoid including setuptools for Anti-Bloat: Avoid Anti-Bloat: Avoid including Anti-Bloat: Changes to avoid Anti-Bloat: Avoid Anti-Bloat: Made more packages equivalent to using Moved the changelog of Nuitka to the website, just point to there from Nuitka repo. UI: Proper error message from Nuitka when scons build fails with a detail mnemonic page. Read more on the info page for detailed information. Windows: Reject all MinGW64 that are not are not the Quality: Added auto-format of PNG and JPEG images. This aims at making it simpler to add images to our repositories, esp. Nuitka Website. This now makes User Manual: Be more clear about compiler version needs on Windows for Python 3.11. User Manual: Added examples for error message with low C compiler memory, such that maybe they can be found via search by users. User Manual: Removed sections that are unnecessary or better maintained as separate pages on the website. Quality: Avoid empty Quality: Enforce normalized paths for Quality: Check more keys with expressions for syntax errors, to catch these mistakes in configuration sooner. Quality: Scanning through all files with the auto-format tool should now be faster, and CPython test suite directories (test submodules) if present are ignored. Release: Remove month from manpage generation, that’s only noise in diffs. Removed digital art folders, these were only making checkouts larger for no good reason. We will have better ones on the website in the future. Scons: Allow C warnings when compiling for running in debugger automatically. UI: The macOS app bundle option is not experimental at all. This has been untrue for years now, remove that cautioning. macOS: Discontinue support for PyQt6. With newer PyQt6 we would have to package frameworks properly, and we don’t have that yet and it will be a lot of developer time to get it. Instead point people to PySide6 which is the better choice and is perfectly supported by Qt company and Nuitka. Removed version numbering, month of creation, etc. from the man pages generated. Moved Bumped copyright year and split the license text such that it is now at the bottom of the files rather than eating up the first page, this is aimed at making the code more readable. With Scons: Don’t scan for Minor cleanups coming from trying out Removed test with chinese filenames, we need to avoid chinese names in the repo. These have been seen as preventing installation on some systems that are not capable of handling them in the git, zip, pip tooling, so lets avoid them entirely now that Nuitka handles these just fine. Tests: More macOS standalone tests that need to be bundles were getting the project configuration to do it. This release added much needed tools for our Nuitka Package configuration, but also cleans up scalability and optimization that was supposed to work, but did not yet, or not anymore. The usability improved again, as it does always, but the big improvements for scalability that will implement existing algorithms more efficient, are yet to come, this release was mainly driven by the need to get Bug Fixes
dill-compat
plugin didn’t work for functions with closure variables taken. Fixed in 2.0.1 already.def get_local_closure(b):
def _local_multiply(x, y):
return x * y + b
return _local_multiply
fn = get_local_closure(1)
fn2 = dill.loads(dill.dumps(fn))
print(fn2(2, 3))
kernel32.dll
is actually reported as a dependency, remove assertion against that. Fixed in 2.0.1 already.--output-filename
was not formatted properly. Fixed in 2.0.1 already.scapy
package. Fixed in 2.0.2 already.PonyORM
implicit dependencies. Fixed in 2.0.2 already.cryptoauthlib
, betterproto
, tracerite
, sklearn.util
, and qt_material
packages. Fixed in 2.0.2 already.scipy
package. Fixed in 2.0.2 already.speech_recognition
package. Fixed in 2.0.2 already.gmsh
package. Fixed in 2.0.2 already.--run
was not considering --output-dir
directory to load the result module. Without this, the check for un-replaced module was always triggering for module source in current directory, despite doing the right thing and putting it elsewhere. Fixed in 2.0.2 already.__file__
of modules that are unicode and solve a TODO that restores consistency over modules mode __file__
values. Fixed in 2.0.2 already.tiktoken
package. Fixed in 2.0.2 already.__path__
value. Fixed in 2.0.2 already.size-1
but we did not follow that and then used a tuple one bit larger than necessary.importlib_metadata
unless really necessary.pkg_resources
plugin used to load it, and that then had harmful effects for our handling of distribution information in some configurations. Fixed in 2.0.3 already.UserWarning
would be displayed during compilation. Fixed in 2.0.3 already.h2o
package. Fixed in 2.0.3 already.NUITKA_PYTHONPATH
usage more robust. Fixed in 2.0.3 already.slot
, wasn’t working with keyword argument calls. Fixed in 2.0.3 already.paddle
and paddleocr
packages. Fixed in 2.0.4 already.diatheke
. Fixed in 2.0.4 already.zaber-motion
package. Fixed in 2.0.4 already.plyer
package. Fixed in 2.0.4 already.OSError
for metadata read, otherwise corrupt packages can have Nuitka crashing. Fixed in 2.0.4 already.--main
and --script-path
. This is of course the only way Nuitka-Action does call it, so they didn’t work there at all. Fixed in 2.0.4 already.pygame_menu
data files. Fixed in 2.0.6 already..pyi
parser was crashing on some comments with a leading from
in the line, recognize these better. Fixed in 2.0.6 already.torch
packages that otherwise require source code.tqdm
etc. left sub-modules behind, removing only the top level sys.modules
entry may not be enough.New Features
constants
in Nuitka package configurations. We can now using when
clauses, define variable values to be defined, e.g. to specify the DLL suffix, or the DLL path, based on platform dependent properties.relative_path
, suffix
, prefix
in DLL Nuitka package configurations allowed to be an expression rather than just a constant value.python_version_str
and python_version_full_str
, to use them when constructing e.g. DLL paths in Nuitka Package Configuration.iterate_modules
for producing the submodules of a given package, for using in expressions of Nuitka package configuration.module
attribute to __compiled__
values__compiled__
attribute, so we add an indicator for module mode that closes this gap.appdirs
and importlib
for use in Nuitka package config expressions.nofollow
configuration is for rare use cases only.extension_std_suffix
and extension_suffix
for use in expressions, to e.g. construct DLL suffix patterns from it.delvewheel
plugin now puts the version of that packaging tool used by a particular module in the report rather than tracing it to the user, that in the normal case won’t care. This is more for debugging purposes of Nuitka.Optimization
sys.exit
into exception raise of SystemExit
.altair
package. Added in 2.0.3 already.kubernetes
for tensorflow
unless used otherwise. Added in 2.0.3 already.tqdm
. Added in 2.0.3 already.IPython
in fire
package. Added in 2.0.3 already.Cython
for pydantic
package. Added in 2.0.3 already.triton
in newer torch
as well. Added in 2.0.5 already.setuptools
via setuptools_scm
in pyarrow
.setuptools
which we want to avoid, all of Cython
, cython
, pyximport
, paddle.utils.cpp_extension
, torch.utils.cpp_extension
were added for better reports of the actual causes.Organizational
winlibs
that Nuitka itself downloaded. As these packages break very easily, we need to control if it’s a working set of ccache
, make
, binutils
and gcc with all the necessary workarounds and features like LTO
working on Windows properly.optipng
and jpegoptim
calls as necessary. Previously this was manual steps for the website to be applied.no-auto-follow
values, for silently ignoring it there is a dedicated string ignore
that must be used.dest_path
and relative_path
. Users were uncertain if a leading dot made sense, but we now disallow it for clarity.Credits.rst
file to be on the website and maintain it there rather than syncing of from the Nuitka repository.Cleanups
sys.exit
being optimized, we were able to make our trick to avoid following nuitka
because of accidentally finding the setup
as an import more simple.# Don't allow importing this, and make recognizable that
# the above imports are not to follow. Sometimes code imports
# setup and then Nuitka ends up including itself.
if __name__ != "__main__":
sys.exit("Cannot import 'setup' module of Nuitka")
ccache
on Windows, the winlibs
package contains it nowadays, and since it’s now required to be used, there is no point for this code anymore.ruff
as a linter on Nuitka, it found a few uses of not using not in
, but that was it.Tests
Summary
torch
to work in its latest version out of the box with stable Nuitka, but this couldn’t be done as a hotfix