27 November 2024
Nuitka Release 2.5
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release focused on Python 3.13 support, but also on improved compatibility, made many performance optimizations, enhanced error reporting, and better debugging support. Windows: Fixed a regression in onefile mode that incorrectly handled program and command line paths containing spaces. Fixed in 2.4.4 already. Windows: Corrected an issue where console output handles were being opened with closed file handles. Fixed in 2.4.2 already. Standalone: Restored the ability to use trailing slashes on the command line to specify the target directory for data files on Windows. Fixed in 2.4.2 already. Compatibility: Fixed a parsing error that occurred with relative imports in Modules: Ensured that extension modules load correctly into packages when using Python 3.12. Fixed in 2.4.4 already. Windows: Improved command line handling for onefile mode to ensure full compatibility with quoting. Fixed in 2.4.4 already. Data Directories: Allowed the use of non-normalized paths on the command line when specifying data directories. Fixed in 2.4.5 already. Python 3.11+: Fixed an issue where importlib_metadata: Improved compatibility with Plugins: Fixed a crash that occurred when using the Scons: Improved error tolerance when reading Scons: Prevented the C standard option from being applied to C++ compilers, resolving an issue with the splash screen on Windows when using Clang. Fixed in 2.4.8 already. macOS: Enhanced handling of DLL self-dependencies to accommodate cases where DLLs use both Compatibility: Fixed a memory leak that occurred when using MSYS2: Excluded the Python 3.10+: Fixed a crash that could occur when a MSYS2: Implemented a workaround for non-normalized paths returned by Python 3.12: Resolved an issue where Nuitka’s constant code was triggering assertions in Python 3.12.7. Fixed in 2.4.10 already. UI: Ensured that the Windows: Prevented encoding issues with CMD files used for accelerated mode on Windows. Standalone: Improved the standard library scan to avoid assuming the presence of specific files, which might have been deleted by the user or a Python distribution. Compatibility: Added Compatibility: Backported the error message change for Compatibility: Fixed an issue where calling built-in functions with keyword-only arguments could result in errors due to incorrect argument passing. Compatibility: Fixed reference leaks that occurred when using Windows: Prioritized relative paths over absolute paths for the result executable when absolute paths are not file system encodable. This helps address issues related to non-ASCII short paths on some Chinese systems. Compatibility: Improved compatibility with C extensions by handling cases where the attribute slot is not properly implemented, preventing potential segfaults. Compatibility: Prevent the leakage of Compatibility: Fixed an issue where matching calls with keyword-only arguments could lead to incorrect optimization and argument passing errors. Compatibility: Corrected the handling of iterators in for loops to avoid assuming the presence of slots, preventing potential issues. macOS: Added support for cyclic DLL dependencies, where DLLs have circular references. Compatibility: Ensured the use of updated expressions during optimization phase for side effects to prevent crashes caused by referencing obsolete information. Python 3.10+: Fixed a crash that could occur in complex cases when re-formulating Python 3.4-3.5: Corrected an issue in Nuitka’s custom Python 3.10+: Ensured that Plugins: Added a data file function that avoids loading packages, preventing potential crashes caused by incompatible dependencies (e.g., Compatibility: Ensured that Nuitka’s package reader closes data files after reading them to prevent resource warnings in certain Python configurations. Standalone: Exposed Accelerated Mode: Enabled the Compatibility: Prevented resource warnings for unclosed files when trace outputs are sent to files via command line options. Compatibility: Enabled the use of Plugins: Fixed an issue in the Python 3.12 (Linux): Enhanced static Standalone: Further improved the standard library scan to avoid assuming the presence of files that might have been manually deleted. UI: Fixed the arcade: Improved standalone configuration for the license-expression: Added a missing data file for the pydantic: Included a missing implicit dependency required for deprecated decorators in the spacy: Added a missing implicit dependency for the trio: Updated standalone support for newer versions of the tensorflow: Updated standalone support for newer versions of the pygame-ce: Added standalone support for the toga: Added standalone support for newer versions of the django: Implemented a workaround for a PySide6: Improved standalone support for polars: Added a missing dependency for the django: Enhanced handling of cases where the win32ctypes: Included missing implicit dependencies for arcade: Added a missing data file for the PySide6: Allowed driverless-selenium: Added standalone support for the tkinterdnd2: Updated standalone support for newer versions of the kivymd: Updated standalone support for newer versions of the gssapi: Added standalone support for the azure.cognitiveservices.speech: Added standalone support for the mne: Added standalone support for the fastapi: Added a missing dependency for the pyav: Updated standalone support for newer versions of the py_mini_racer: Added standalone support for the keras: Improved standalone support for transformers: Updated standalone support for newer versions of the win32com.server.register: Updated standalone support for newer versions of the Python 3.12+: Added support for cv2: Enabled automatic scanning of missing implicit imports for the lttbc: Added standalone support for the win32file: Added a missing dependency for the kivy: Fixed an issue where the paddleocr: Added missing data files for the playwright: Added standalone support for the PySide6: Allowed Python 3.13: Added experimental support for Python 3.13. Warning Python 3.13 support is not yet recommended for production use due to limited testing. On Windows, only MSVC and ClangCL are currently supported due to workarounds needed for incompatible structure layouts. UI: Introduced a new Note The Windows: Added a new UI: Added the Modules: Enabled the generation of type stubs ( Note Nuitka also adds implicit imports to compiled extension modules, ensuring that dependencies are not hidden. Plugins: Changed the data files configuration to a list of items, allowing the use of Onefile: Removed the MSVC requirement for the splash screen in onefile mode. It now works with MinGW64, Clang, and ClangCL. Done for 2.4.8 already. Reports: Added information about the file system encoding used during compilation to help debug encoding issues. Windows: Improved the Distutils: Added the ability to disable Nuitka in Distutils: Added support for commercial file embedding in Linux: Added support for using uninstalled self-compiled Python installations on Linux. Plugins: Enabled the Runtime: Added a new Reports: Added a list of DLLs that are actively excluded because they are located outside of the PyPI package. Plugins: Allowed plugins to override the compilation mode for standard library modules when necessary. Performance: Implemented experimental support for “dual types”, which can significantly speed up integer operations in specific cases (achieving speedups of 12x or more in some very specific loops). This feature is still under development but shows promising potential for future performance gains, esp. when combined with future PGO (Profile Guided Optimization) work revealing likely runtime types more often and more types being covered. Performance: Improved the speed of module variable access. For Python 3.6 to 3.10, this optimization utilizes dictionary version tags but may be less effective when module variables are frequently written to. For Python 3.11+, it relies on dictionary key versions, making it less susceptible to dictionary changes but potentially slightly slower for cache hits compared to Python 3.10. Performance: Accelerated string dictionary lookups for Python 3.11+ by leveraging knowledge about the key and the module dictionary’s likely structure. This also resolves a previous TODO item, where initial 3.11 support was not as fast as our support for 3.10 was in this domain. Performance: Optimized module dictionary updates to occur only when values actually change, improving caching efficiency. Performance: Enhanced exception handling by removing bloat in the abstracted differences between Python 3.12 and earlier versions. This simplifies the generated C code, reduces conversions, and improves efficiency for all Python versions. This affects both C compile time and runtime performance favorably and solves a huge TODO for Python 3.12 performance. Performance: Removed the use of CPython APIs calls for accessing exception context and cause values, which can be slow. Performance: Utilized Nuitka’s own faster methods for creating Performance: Implemented a custom variant of Windows: Aligned with CPython’s change in reference counting implementation on Windows for Python 3.12+, which improves performance with LTO (Link Time Optimization) enabled. Optimization: Expanded static optimization to include unary operations, improving the handling of number operations and preparing for full support of dual types. Optimization: Added static optimization for Performance: Passed the exception state directly into unpacking functions, eliminating redundant exception fetching and improving code efficiency. Performance: Introduced a dedicated helper for unpacking length checks, resulting in faster and more compact code helping scalability as well. Performance: Generated more efficient code for raising built-in exceptions by directly creating them through the base exception’s Performance: Optimized exception creation by avoiding unnecessary tuple allocations for empty exceptions. This hack avoids hitting the memory allocator as much. Performance: Replaced remaining uses of Code Generation: Replaced implicit exception raise nodes with direct exception creation nodes for improved C code generation. Windows: Aligned with CPython’s change in managing object reference counters on Windows for Python 3.12+, improving performance with LTO enabled. Performance: Removed remaining CPython API calls when creating Windows: Avoided scanning for DLLs in the Windows: Updated to a newer MinGW64 version from 13.2 to 14.2 for potentially improved binary code generation with that compiler. Code Size: Reduced the size of constant blobs by avoiding module-level constants for the global values Code Generation: Improved code generation for variables by directly placing Optimization: Statically optimized the Performance: Limited the use of Optimization: Implemented a faster check for non-frame statement sequences by decoupling frames and normal statement sequences and using dedicated accessors. This improves performance during the optimization phase. Prevented the inclusion of Avoided the use of Removed Prevented attempts to include both Python 2 and Python 3 code for the Enhanced handling for the Allowed Avoided Avoided Avoided Updated anti-bloat measures for newer versions of the Avoided Allowed Allowed Avoided test modules when using the Treated the use of the Development Environment: Added experimental support for a devcontainer to the repository, providing an easier way to set up a Linux-based development environment. This feature is still under development and may require further refinement. Issue Reporting: Clarified the issue reporting process on GitHub, emphasizing the importance of testing reproducers against Python first to ensure the issue is related to Nuitka. Issue Reporting: Discouraged the use of UI: Improved the clarity of help message of the option for marking data files as external, emphasizing that files must be included before being used. UI: Added checks to the Qt plugins to ensure that specified plugin families exist, preventing unnoticed errors. UI: Implemented heuristic detection of terminal link support, paving the way for adding links to options and groups in the command line interface. UI: Removed obsolete caching-related options from the help output, as they have been replaced by more general options. Plugins: Improved error messages when retrieving information from packages during compilation. Quality: Implemented a workaround for an Quality: Updated GitHub actions to use Quality: Updated to the latest version of Release Process: Updated the release script tests for Debian and PyPI to use the correct runner names. (Changed in 2.4.1 already. UI: Disabled progress bar locking, as Nuitka currently doesn’t utilize threads. UI: Added heuristic detection of terminal link support and introduced an experimental terminal link as a first step towards a more interactive command line interface. Debugging: Fixed a crash in the “explain reference counts” feature that could occur with unusual Debugging: Included reference counts of tracebacks when dumping reference counts at program end. Debugging: Added assertions and traces to improve debugging of input/output handling. Quality: Added checks for configuration module names in Nuitka package configuration to catch errors caused by using filenames instead of module names. UI: Removed obsolete options controlling cache behavior, directing users to the more general cache options. Scons: Ensured that the Distutils: Added the Dependency Management: Started experimental support for downloading Nuitka dependencies like Added Python 3.13 to the GitHub Actions test matrix. Significantly enhanced construct-based tests for clearer results. The new approach executes code with a boolean flag instead of generating different code, potentially leading to the removal of custom templating. Removed the Fixed an issue where the test runner did not discover and use Python 3.12+, resulting in insufficient test coverage for those versions on GitHub Actions. Ensured that the Set up continuous integration with Azure Pipelines to run Nuitka tests against the factory branch on each commit. Enforced the use of static Improved the robustness of many construct tests, making them less sensitive to unrelated optimization changes. Removed a test that was only applicable to Nuitka Commercial, as it was not useful to always skip it in the standard version. Commercial tests are now also recognized by their names. Added handling for segmentation faults in Prevented resource warnings for unclosed files in a reflected test. WASI: Corrected the signatures of C function getters and setters for compiled types in WASI: Improved code cleanliness by adhering to Code Generation: Fixed a regression in code generation that caused misaligned indentation in some cases. Code Formatting: Changed some code for identical formatting with Caching: Enforced proper indentation in Nuitka cache files stored in JSON format. Code Cleanliness: Replaced checks for Python 3.4 or higher with checks for Python 3, simplifying the code and reflecting the fact that Python 3.3 is no longer supported. Code Cleanliness: Removed remaining Python 3.3 specific code from frame templates. Code Cleanliness: Performed numerous spelling corrections and renamed internal helper functions for consistency and clarity. Plugins: Renamed the Plugins: Moved the This a major release that brings support for Python 3.13, relatively soon after its release. Our plugin system and Nuitka plugin configuration was used a lot for support of many more third-party packages, and numerous other enhancements in the domain of avoiding bloat. This release focuses on improved compatibility, new break through performance optimizations, to build on in the future, enhanced error reporting, and better debugging support.Bug Fixes
.pyi
files, which could affect some extension modules with available source code. Fixed in 2.4.3 already.inspect
module functions could raise StopIteration
when examining compiled functions on the stack. Fixed in 2.4.5 already.importlib_metadata
by handling cases where it might be broken, preventing potential compilation crashes. Fixed in 2.4.5 already.no_asserts
YAML configuration option. Fixed in 2.4.6 already.ccache
log files to prevent crashes on Windows caused by non-ASCII module names or paths. Fixed in 2.4.11 already..so
and .dylib
extensions for self-references. Fixed in 2.4.8 already.deepcopy
on compiled methods. Fixed in 2.4.9 already.bin
directory from being considered a system DLL folder when determining DLL inclusion. Fixed in 2.4.9 already.match
statement failed to match a class with arguments. Fixed in 2.4.9 already.os.path.normpath
in MSYS2 Python environments. Fixed in 2.4.11 already.--include-package
option includes both Python modules and extension modules that are sub-modules of the specified package. Fixed in 2.4.11 already.suffix
, suffixes
, and stem
attributes to Nuitka resource readers to improve compatibility with file objects.yield from
used at the module level, using dynamic detection instead of hardcoded text per version.list.insert
and list.index
with 2 or 3 arguments.sys.frozen
when using the multiprocessing
module and its plugin, resolving a long-standing TODO and potentially breaking compatibility with packages that relied on this behavior.match
statements.PyDict_Next
implementation that could lead to incorrect results in older Python 3 versions.AttributeError
is raised with the correct keyword arguments, avoiding a TypeError
that occurred previously.numpy
versions).setuptools
contained vendor packages in standalone distributions to match the behavior of the setuptools
package.django
module parameter in accelerated mode to correctly detect used extensions.xmlrpc.server
without requiring the pydoc
module.anti-bloat
configuration where change_function
and change_classes
ignored “when” clauses, leading to unintended changes.libpython
handling for Linux. Static libpython
is now used only when the inline copy is available (not in official Debian packages). The inline copy of hacl
is used for all Linux static libpython
uses with Python 3.12 or higher.--include-raw-dir
option, which was not functioning correctly. Only the Nuitka Package configuration was being used previously.Package Support
arcade
package. Added in 2.4.3 already.license-expression
package in standalone distributions. Added in 2.4.6 already.pydantic
package to function correctly in standalone mode. Fixed in 2.4.5 already.spacy
package in standalone distributions. Added in 2.4.7 already.trio
package. Added in 2.4.8 already.tensorflow
package. Added in 2.4.8 already.pygame-ce
package. Added in 2.4.8 already.toga
package on Windows. Added in 2.4.9 already.django
debug feature that attempted to extract column numbers from compiled frames. Added in 2.4.9 already.PySide6
on macOS by allowing the recognition of potentially unusable plugins. Added in 2.4.9 already.polars
package in standalone distributions. Added in 2.4.9 already.django
settings module parameter is absent in standalone distributions. Added in 2.4.9 already.win32ctypes
modules on Windows in standalone distributions. Added in 2.4.9 already.arcade
package in standalone distributions. Added in 2.4.9 already.PySide6
extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed. Added in 2.4.11 already.driverless-selenium
package. Added in 2.4.11 already.tkinterdnd2
package. Added in 2.4.11 already.kivymd
package. Added in 2.4.11 already.gssapi
package. Added in 2.4.11 already.azure.cognitiveservices.speech
package on macOS.mne
package.fastapi
package in standalone distributions.pyav
package.py_mini_racer
package.keras
by extending its sub-modules path to include the keras.api
sub-package.transformers
package.win32com.server.register
package.distutils
in setuptools
for Python 3.12 and later.cv2
package in standalone distributions.lttbc
package.win32file
package in standalone distributions.kivy
clipboard was not working on Linux due to missing dependencies in standalone distributions.paddleocr
package in standalone distributions.playwright
package with a new plugin.PySide6
extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed.New Features
--mode
selector to replace the options --standalone
, --onefile
, --module
, and --macos-create-app-bundle
.app
mode creates an app bundle on macOS and a onefile binary on other operating systems to provide the best deployment option for each platform.hide
choice for the --windows-console-mode
option. This generates a console program that hides the console window as soon as possible, although it may still briefly flash.--python-flag=-B
option to disable the use of bytecode cache (.pyc
) files during imports. This is mainly relevant for accelerated mode and dynamic imports in non-isolated standalone mode..pyi
files) for compiled modules using an inline copy of stubgen
. This provides more accurate and informative type hints for compiled code.when
conditions for more flexible control. Done in 2.4.6 already.attach
mode for --windows-console-mode
when forced redirects are used.pyproject.toml
builds using the build_with_nuitka
setting. This allows falling back to the standard build
backend without modifying code or configuration. This setting can also be passed on the command line using --config-setting
.distutils
packages.matplotlib
plugin to react to active Qt and tkinter
plugins for backend selection.original_argv0
attribute to the __compiled__
value to provide access to the original start value of sys.argv[0]
, which might be needed by applications when Nuitka modifies it to an absolute path.Optimization
int
and long
values, avoiding slower CPython API calls._PyGen_FetchStopIterationValue
to avoid CPython API calls in generator handling, further improving performance on generators, coroutines and asyncgen.os.stat
and os.lstat
calls.new
method instead of calling them as functions. This can speed up some things by a lot.PyTuple_Pack
with Nuitka’s own helpers to avoid CPython API calls.int
values in various parts of the code, including specialization code, helpers, and constants loading.PATH
environment variable when they are not intended to be used from the system. This prevents potential crashes related to non-encodable DLL paths and makes those scans faster too.-1
, 0
, and 1
.NameError
exceptions into the thread state when raised, making for more compact C code.sys.ps1
and sys.ps2
values to not exist (unless in module mode), potentially enabling more static optimization in packages that detect interactive usage checking them.tqdm
locking to no-GIL and Scons builds where threading is actively used.Anti-Bloat
importlib_metadata
for the numpy
package. Added in 2.4.2 already.dask
in the pandera
package. Added in 2.4.5 already.numba
for newer versions of the shap
package. Added in 2.4.6 already.aenum
package, avoiding SyntaxError
warnings. Added in 2.4.7 already.sympy
package. Added in 2.4.7 already.pydoc
for the pyqtgraph
package. Added in 2.4.7 already.pytest
in the time_machine
package. Added in 2.4.9 already.pytest
in the anyio
package.numba
in the pandas
package.torch
package with increased coverage.pygame.tests
and cv2
for the pygame
package.unittest
in the absl.testing
package.setuptools
in the tufup
package.bsdiff4
package.wheel
module the same as using the setuptools
package.Organizational
--deployment
in issue reports, as it hinders the automatic identification of issues, that should be the first thing to remove.isort
bug that prevented it from handling UTF-8 comments.clang-format-20
.black
for code formatting.dict
values mistaken for modules.CC
environment variable is used consistently for --version
and onefile bootstrap builds, as well as the Python build, preventing inconsistencies in compiler usage and outputs.compiled-package-hidden-by-package
mnemonic for use in distutils
to handle the expected warning when a Python package is replaced with a compiled package and the Python code is yet to be deleted.ordered-set
. This feature is not yet ready for general use.Tests
2to3
conversion code from the test suite, as it is being removed from newer Python versions. Tests are now split with version requirements as needed.compare_with_cpython
test function defaults to executing the system’s Python interpreter instead of relying on the PYTHON
environment variable.libpython
for construct-based tests to eliminate DLL call overhead and provide more accurate performance measurements.distutils
test cases, providing debug output for easier diagnosis of these failures.Cleanups
WASI
to ensure they match the calling conventions. Casts are now performed locally to the compiled types instead of in the function signature. Call entries also have the correct signature used by Python C code.PyCFunction
signatures in WASI
.clang-format-20
to eliminate differences between the new and old versions.get_module_directory
helper function in the Nuitka Package configuration to remove the leading underscore, improving readability.numexpr.cpuinfo
workaround to the appropriate location in the Nuitka Package configuration, resolving an old TODO item.Summary