30 July 2024
Nuitka Release 2.4
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release largely contains bug fixes for the previous changes, but also finishes full compatibility with the match statements of 3.10, something that was long overdue since there were always some incompatible behaviors there.
In terms of bug fixes, it’s also huge. An upgrade is required, especially for new UI: Fix, we had reversed Python3.10+: Fix, must not check for Python3.10+: Fix, Python3.8+: Fix, calls to variable args functions now need to be done differently, or else they can crash, as was observed with 3.10 in PGO instrumentation, at least. Fixed in 2.3.1 already. PGO: Fix, using Linux: Support extension modules used as DLLs by other DLLs or extension modules. That makes newer Python3.10+: Matches classes were not fully compatible. We need to check against case-defined class Also, properly annotating the exception exit of subscript matches; the subscript value can indeed raise an exception. Collect keyword and positional match values in one go and detect duplicate attributes used, which we previously did not. Scons: Fix, do not crash when Debian: Fix, detecting the Debian flavor of Python was not working anymore, and as a result, the intended defaults were no longer applied by Nuitka, leading to incorrect suggestions that didn’t work. Fixed in 2.3.3 already. Ubuntu: Fix, the static link library for Python 3.12 is not usable unless we provide parts of HACL for the Standalone: Fix, importing newer Python3.11+: Added support for newer Python with Standalone: Support locating Windows icons for Standalone: Added support for Python3.12: Fix, our workaround for Compatibility: Added support for Standalone: Added missing dependencies for standard library extension modules, mainly exhibited on macOS. Fixed in 2.3.4 already. Windows: Fix build failures on mapped network drives. Fixed in 2.3.4 already. Python3.12: Fix, need to set frame Fix, fork bomb in Nuitka-Python: Fix, cannot ask for shared library prefixes. Fixed in 2.3.8 already. Standalone: Make sure Linux: Fix, for static executables we should ignore errors setting a DLL load path. Fixed in 2.3.10 already. Compatibility: Fix, nuitka resource readers also need to have Fix, need to force no-locale language outputs for tools outputs on non-Windows. Our previous methods were not forcing enough. For non-Windows this makes Nuitka work on systems with locales active for message outputs only. Fixed in 2.3.10 already. Fix, was not using proper result value for Windows: Retry deleting dist and build folders, allowing users to recognize still running programs and not crashing on Anti-Virus software still locking parts of them. Fix, Fix, output correct unsupported exception messages for in-place operations For in-place Fix, included metadata could lead to instable code generation. We were using a dictionary for it, but that is not as stable order for the C compiler to fully benefit. Fix, including data files for packages that are extension modules was not working yet. macOS: Detect the DLL path of Fix, need to prefer extension modules over Python code for packages. Fix, immutable constant values are not to be treated as very trusted. Python3: Fix, the Python3: Added Fix, the For Standalone: Fix, adding missing dependencies for some crypto packages. Python3.12: Need to write to thread local variable during import. This however doesn’t work for Windows and non-static libpython flavors in general. macOS: Enforce using system Fix, we need to read Standalone: Fix, some packages query private values for distribution objects, so use the same attribute name for the path. Multidist: Make sure to follow the multidist reformulation modules. Otherwise in accelerated mode, these could end up not being included. Fix, need to hold a reference of the iterable while converting it to Plugins: Fix, this wasn’t properly ignoring macOS: Need to allow DLLs from all Homebrew paths. Reports: Do not crash during report writing for very early errors. Python3.11+: Fix, need to make sure we have Debian: More robust against problematic distribution folders with no metadata, these apparently can happen with OS upgrades. Fix, was leaking exception in case of Compatibility: Close standard file handles on process forks as CPython does. This should enhance things for compilations using Standalone: Added data file for older Standalone: Support older Standalone: Added data files for Standalone: Added support for the newer Standalone: Include all Standalone: Added implicit dependencies of Standalone: Added implicit dependencies for Standalone: Added support for Standalone: More complete support for Standalone: Added implicit dependencies for Standalone: Added support for Standalone: Added support for newer Standalone: Added data files for Standalone: Added support for Standalone: Added support for Standalone: Added support for Standalone: Added support for Standalone: Added implicit dependencies for Standalone: Added support for newer Standalone: Added support for Standalone: Added support for Standalone: Added workaround for Standalone: Added support for newer Experimental support for Python 3.13 beta 3. We try to follow its release cycle closely and aim to support it at the time of CPython release. We also detect no-GIL Python and can make use of it. The GIL status is output in the Scons: Added experimental option Debugging: A new non-deployment handler helps when segmentation faults occurred. The crashing program then outputs a message pointing to a page with helpful information unless the deployment mode is active. Begin merging changes for WASI support. Parts of the C changes were merged and for other parts, command line option PGO: Added support for using it in standalone mode as well, so once we use it more, it will immediately be practical. Make the Plugins: The Reports: Added included metadata and reasons for it. Standalone: Added support for Compatibility: Use existing source files as if they were Plugins: Make version information, onefile mode, and onefile cached mode indication available in Nuitka Package Configuration, too. Onefile: Warn about using Tendo uses the running binary name for locking by default. So it’s not going to work if that changes for each execution, make the user aware of that, so they can use cached mode instead. Reports: Include the micro pass counts and tracing merge statistics so we can see the impact of new optimization. Plugins: Allow to specify modes in the Nuitka Package Configuration for Standalone: Added module configuration for Django to find commands and load its engine. Allow negative values for –jobs to be relative to the system core count so that you can tell Nuitka to use all but two cores with Python3.12: Annotate libraries that are currently not supported We will need to provide our own Python3.12 variant to make them work. Python3.11+: Catch calls to uncompiled function objects with compiled code objects. We now raise a Statically optimize constant subscripts of variables with immutable constant values. Forward propagate very trusted values for variable references enabling a lot more optimization. Python3.8+: Calls of C functions are faster and more compact code using vector calls, too. Python3.10+: Mark our compiled types as immutable. Python3.12: Constant returning functions are dealing with immortal values only. Makes their usage slightly faster since no reference count handling is needed. Python3.10+: Faster attribute descriptor lookups. Have our own replacement of Avoid using Python API function for determining sequence sizes when getting a length size for list creations. Data Composer: More compact and portable Python3 Rather than fixed native length 8 or 4 bytes, we use variable length encoding which for small values uses only a single byte. This also avoids using Large values are also more compact because middle 31-bit portions can be less than 4 bytes and save space on average. Data Composer: Store bytecode blob size more efficient and portable, too. Prepare having knowledge of Added more hard trust for the The Added the other types to be known to exist. That should help scalability for types intensive code somewhat by removing error handling for them. macOS: Should use static libpython with Anaconda as it works there too, and reduces issues with Python3.12 and extension module imports. Standalone: Statically optimize by OS in Consequently, standalone distributions can exclude OS-specific packages such as Avoid changing code names for complex call helpers The numbering of complex call helper as normally applied to all functions are, caused this issue. When part of the code is used from the bytecode cache, they never come to exist and the C code of modules using them then didn’t match. This avoids an extra C re-compilation for some modules that were using renumbered function the second time around a compilation happens. Added in 2.3.10 already. Avoid using C-API when creating Faster indentation of generated code. Add new Recognize usage of Avoid compiling large Avoid Changes for not using Avoid Avoid using Avoid using Avoid Avoid using Avoid using Avoid Added Avoid Avoid Avoid Avoid using Python3.12: Added annotations of official support for Nuitka PyPI package and test runner options that were still missing. Fixed in 2.3.1 already. UI: Change runner scripts. The UI: Check filenames for data files for illegal paths on the respective platforms. Some user errors with data file options become more apparent this way. UI: Check spec paths more for illegal paths as well. Also do not accept system paths like UI: Handle report writing interrupt with CTRL-C more gracefully. No need to present this this as a general problem, rather inform the user that he did it. NoGIL: Warn if using a no-GIL Python version, as this mode is not yet officially supported by Nuitka. Added badges to the UI: Use the retry decorator when removing directories in general. It will be more thorough with properly annotated retries on Windows. For the dist folder, mention the running program as a probable cause. Quality: Check Quality: Catch backlashes in paths provided in Nuitka Package Configuration values for Debugging: Disable pagination in PGO: Warn if the PGO binary does not run successfully. UI: The new console mode option is a Windows-specific option now, move it to that group. UI: Detect “rye python” on macOS. Added in 2.3.8 already. UI: Be forgiving about release candidates; Ubuntu shipped one in a LTS release. Changed in 2.3.8 already. Debugging: Allow fine-grained debug control for immortal checks Can use UI: Avoid leaking compile time Nuitka environment variables to the child processes. They were primarily visible with For non-Windows, we now recognize if we are the exact re-execution and otherwise, reject them. Watch: Delete the existing Watch: Keep track of Nuitka compiled program exit code in newly added result files, too. Watch: Redo compilations in case of previous errors when executing the compile program. Quality: Wasn’t detecting files to ignore for PyLint on Windows properly, also detect crashes of PyLint. Added test to cover the macOS: Make actual use of Make compile extension module test work on macOS, too. Avoid using Python3.11+: Make large constants test executable for as well. We no longer can easily create those values on the fly and output them due to security enhancements. Python3.3: Remove support from the test runner as well. Tests: Added construct-based tests for coroutines so we can compare their performance as well. Make try/finally variable releases through common code. It will allow us to apply special exception value trace handling for only those for scalability improvements, while also making many re-formulations simpler. Avoid using Avoid Python3 and Python3.5+ specific Jinja2 modules on versions before that, and consequently, avoid warning about the Moved code object extraction of Also call Don’t pass main filename to the tree building anymore, and make Change comments that still mentioned Python 3.3 as where a change in Python happened since we no longer support this version. Now, we consider what’s first seen in Python 3.4 is a Python3 change. Cleanup, change Python 3.4 checks to 3.0 checks as Python3.3 is no longer supported. Cleans up version checks, as we now treat The usual flow of spelling cleanups, this time for C codes. This release cycle was a longer than usual, with much new optimization and package support requiring attention. For optimization we got quite a few things going, esp. with more forward propagation, but the big ones for scalability are still all queued up and things are only prepared. The 3.13 work was continuing smoothly and seems to be doing fine. We are still on track for supporting it right after release. The parts where we try and address WASI prepare cross-compilation, but we will not aim at it generally immediately, and target our own Nuitka standalone backend Python that is supposed to be added in coming releases.setuptools that made compiled programs segfault at startup.Bug Fixes
disable / force and wrong option name recommendation for --windows-console-mode when the user used old-style options.len greater or equal of 0 or for sequence match cases. That is unnecessary and incompatible and can raise exceptions with custom sequences not implementing __len__. Fixed in 2.3.1 already.match sequence with final star arguments failed in some cases to capture the rest. The assigned value then was empty.when it shouldn’t have been. Fixed in 2.3.1 already.nuitka-run did not execute the program created as expected. Fixed in 2.3.1 already.tensorflow and potentially more packages work again. Fixed in 2.3.1 already.__match_args__, not the matched value type __match_args that is not necessarily the same.clang is not reporting its version correctly. It happened if Clang usage was required with --clang option but not installed. Fixed in 2.3.2 already.sha2 module so as not to cause link errors. Fixed in 2.3.3 already.pkg_resources was crashing. Fixed in 2.3.3 already.dill-compat. Fixed in 2.3.4 already.pywebview. Fixed in 2.3.4 already.spacy related packages. Fixed in 2.3.4 already.cv2 support cannot use the imp module anymore. Fixed in 2.3.4 already.__init__ files that are extension modules. Architecture checks for macOS were false negatives for them, and the case insensitive import scan failed to find them on Windows. Fixed in 2.3.4 already.prev_inst or else f_lasti is random. Some packages; for example PySide6; use this to check what bytecode calls them or how they import them and it could crash when attempting it. Fixed in 2.3.6 already.cpuinfo package no longer happens. Fixed in 2.3.8 already.keras package dependency for tensorflow is visible. Fixed in 2.3.10 already..parent attribute. Fixed in 2.3.10 already.SET_ATTRIBUTE to check success in a few corner cases. Fixed in 2.3.10 already.dict.fromkeys didn’t give compatible error messages for no args given.**, it was also incompatible, since it must not mention the pow function.libpython (if used) by looking at dependencies of the running Python binary rather than encoding what CPython does. Doing that covers other Python flavors as well.__loader__ attribute of a module should be an object and not only the class, otherwise only static methods can work..name and .path attributes to Nuitka loader objects for enhanced compatibility with code that expects source code loaders.sys.argv[0] needs to be absolute for best usability.dirname(sys.argv[0]) to be usable even if the program is launched via PATH environment by a shell, we cannot rely on how we are launched since that won’t be a good path, unlike with Python interpreter, where it always is.codesign as the Anaconda one is not working for us..pyi files as source code. Otherwise unicode characters can cause crashes.list.None values in load descriptions as intended.split as a constant value when using exception groups.--python-flag=-m mode that could cause errors.attach on Windows.Package Support
bokeh version. Fixed in 2.3.1 already.pandas versions as well.panel package.kivy version and added macOS support as well. Fixed in 2.3.4 already.kivy.uix packages with kivy, so their typical config driven usage is not too hard.lxml.sax module. Fixed in 2.3.4 already.zeroconf package. Fixed in 2.3.4 already.numpy version 2. Fixed in 2.3.7 already.tables package. Fixed in 2.3.8 already.scipy.signal package. Fixed in 2.3.8 already.moviepy and imageio_ffmeg packages. Fixed in 2.3.8 already.scipy. Fixed in 2.3.10 already.bpy package. For full support more work will be needed.nes_py and gym_tetris packages.dash and plotly.usb1 package.azure.cognitiveservices.speech package.tinycudann package.win32com.server.register.jaxtyping package.open3d package.torch submodule import function.paddleocr.New Features
--version format and the GIL usage is available as a new {GIL} variable for project options.--experimental=force-system-scons to enforce system Scons to be used. That allows for the non-use of inline copy, which can be interesting for experiments with newer Scons releases. Added in 2.3.2 already.--target=wasi was added, and we are starting to address cross platform compilation for it. More work will be necessary to fully merge it, right not it doesn’t work at all yet.--list-package-dlls use plugins as well, and make delvewheel and announce its DLL path internally, too. Listing DLLs for packages using plugins can use these paths for more complete outputs.no-qt plugin was usable in accelerated mode.spacy with a new plugin..pyi files for extension modules. That gives us dependencies for code that installs source code and extension modules.tendo.singleton in non-cached onefile mode.annotations, doc_strings, and asserts. These overrule global configuration, which is often not practical. Some modules may require annotations, but for other packages, we will know they are fine without them. Simply disabling annotations globally barely works. For some modules, removing annotations can give a 30% compile-time speedup.--jobs=-2 and need not hardcode your current code count.RuntimeError in the bytecode making it easier to catch them rather than segfaulting.Optimization
PyDesc_IsData that had become an API call, making it very slow on Windows specifically.int (Python2 long) value representation.struct.pack with C types, as we might be doing cross platform, so this makes part of the WASI changes unnecessary at the same time.__prepare__ result to be dictionaries per compile time decisions.typing module.typing.Text is a constant too. In debug mode, we now check all exports of typing for constant values. This will allow to find missing values sooner in the future.sysconfig._aix_support and _osx_support.__path__ value.Anti-Bloat
pydoc bloat mode to trigger warnings when using it.numpy.distutils as setuptools bloat for more direct reporting.opcua modules that generate huge C files much like asyncua package. Added in 2.3.1 already.shiboken2 and shiboken6 modules from matplotlib package when the no-qt plugin is used. Added in 2.3.6 already.pydoc and distutils in numpy version 2. Added in 2.3.7 already.numpy and packaging dependencies from PIL package.webbrowser module from pydoc.unittest in keras package. Added in 2.3.1 already.distutils from _oxs_support (used by sysconfig) module on macOS.pydoc for werkzeug package. Fixed in 2.3.10 already.pydoc for site module. Fixed in 2.3.10 already.pydoc from xmlrpc.server. Fixed in 2.3.10 already.no_docstrings support for numpy2 as well. Fixed in 2.3.10 already.pydoc in joblib.memory.setuptools in gsplat package.dask and jax in scipy package.matplotlib for networkx package.Organizational
nuitka3 is no more. Instead, we have nuitka2 where it applies. Also, we now use CMD files rather than batch files.{TEMP} and no path separator after it.README.rst of Nuitka to display package support and more. Added in 2.3.1 already.replacements and replacements_plain Nuitka package configuration values.dest_path, relative_path, dirs, raw_dirs and empty_dirs.gdb with the --debugger option.--no-debug-immortal-assumptions to allow for corrupted immortal objects, which might be done by non-Nuitka code and then break the debug mode.--run, but we should avoid it for everything.virtualenv in case of errors updating or upgrading it.Tests
dill-compat plugin.ctypes in its standalone test to ensure correctness on that OS, too.2to3 in our tests since newer Python no longer contains it by default, we split up tests with mixed contents into two tests instead.Cleanups
anti-bloat configuration values replacements where replacements_plain is good enough. A lot of config pre-date its addition.SyntaxError given.dill-compat plugin from Python module template to C code helper for shared usage and better editing.va_end for standards compliance when using va_start. Some C compilers may need that, so we better do it even if what we have seen so far doesn’t need it.nuitka.Options functions usage explicit when importing.>=3.4 either as >=3 or can drop checks entirely.Summary