26 May 2022
Nuitka Release 0.8
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release has a massive amount of bug fixes, builds on existing features, and adds new ones. Windows: Fix, need to ignore cases where shorting in path for external use during compilation gives an permission error. Fixed in 0.7.1 already. Compatibility: Added workaround for Windows: Fix, detect ARM64 arch of MSVC properly, such that we can give a proper mismatch for the Python architecture. Fixed in 0.7.1 already. Standalone: Fix, using macOS: Fix, the dependency parser was using normalized paths where original paths must be used. Fixed in 0.7.1 already. Standalone: Fix, using Onefile: Fix, the detection of a usable Python for compression could crash. Fixed in 0.7.2 already. Onefile: Adding the payload on Windows could run into locks still being held, need to wait in that case. Fixed in 0.7.2 already. Fix, need to include Release: The wheels built for Nuitka when installed through URLs were not version specific, but due to different inline copies per OS and Python version, they must not be reused. Therefore we now pretend to contain an extension module, which handles that. Fixed in 0.7.3 already. Standalone: Fix, using Standalone: Do not compress MSVC run time library when using Standalone: Fix, on Windows more files should be included for TkInter to work with all software. Fixed in 0.7.5 already. Distutils: Added support for Standalone: Fix, using Standalone: Added support for Standalone: Fix, bytecode demotions should use relative filenames rather than original ones. Fixed in 0.7.7 already. Standalone: Fix, must remove extension module objects from Windows: In case of an exception, Windows: Improved Standalone: Added support for Standalone: Added support for newer Module: Fix, need to restore the Standalone: Added support for Fix, Windows: Fix, could crash on when the Scons report was written due to directories in Compatibility: Fix, positive Fix, Python3.10+: Fix, Standalone: Added support for newer Python3: Fix, failed to read bytecode only stdlib files. This affect mostly Fedora Python which does this for encodings. Python3.5+: Fix, two phase loading of modules could release it immediately. Standalone: Added missing dependency for Fix, the Standalone: Added missing dependency of Standalone: Added support for Standalone: Added support for Fix, need to mark assigned from variables as escaped. Without it, some aliased loop variables could be misunderstood and falsely statically optimized. Standalone: Added support for newer Standalone: Added data files for the Standalone: Added support for newer Standalone: Added support for Python3.7+: Fix, the error message wasn’t fully compatible for unsubscriptable type exception messages. Standalone: Fix, Python3.4+: Fix, the Standalone: Disable QtPDF plugin of PySide 6.3.0, because it’s failing dependency checks. On macOS this was blocking, we will change it to detection if that is necessary in a future release. Standalone: Added support for Standalone: Added support for Standalone: Added support for newer Added new option Added support for Added support for Added support for requiring and suggesting modes. In part this was added to 0.7.3 already, and is used e.g. to enforce that on macOS the Debian: Detect version information for “Debian Sid”. Added in 0.7.4 already, and also improved how Debian/Ubuntu versions are output. Added new option Distutils: When using The option names are the same, but without leading dashes. Lists are only needed when passing multiple values with the same option. macOS: Add support for specifying signing identity with Included data files are now reported in the compilation report XML as well. Accept absolute paths for searching paths of binaries. This allows e.g. the Python3.10: Added support for positional matching of classes, so far only keyword matching was working. Added support for path spec values Standalone: Detect missing Anti-Bloat: Added support for plain replacements in the Python3.10+: Added support for Optimization: Added Much more The Do not compile the meta path based loader separate, which allows us to not expose functions and values only used by it. Also spares the C compiler one file. Added various dedicated nodes for querying package resources data, e.g. Added more hard imports, some of which help scalability in the compilation, because these are then known to exist in standalone mode, others are used for package resource specific operations. Onefile: Releasing decompression buffers avoiding unnecessary high memory usage of bootstrap binary. Standalone: Avoid proving directories with no DLLs (e.g. from packages) towards For Standalone: Avoid including Removed references to Removed uses of Made standard plugin file naming consistent, their name should be Cleaned up Massive cleanups of data file handling in plugins. Adding methods for producing the now required objects. The Scons file handling got further cleaned up and unified, doing more things in common code. Avoid Split the allowance check from the encountering. Allow plugins and options all to say if an import should be followed, and only when that is decided, to complain about it. Previously the attempt was causing an error, even if another plugin were to decide against it later. Python2.6: Avoid warnings from MSVC for out specialized UI: Warning tests are now wrapped to multiple lines if necessary. That makes it more accessible for larger messages that contain more guiding information. Documented how to use local Nuitka checkout with Added instructions on how to pass extra C and linker flags and to the User Manual. Made our auto-format usable for the Nuitka website code too. Removed dependencies on Updated to latest versions of requirements for development, esp. Renamed Unified how dependencies are installed in GitHub actions. Updated man page contents for option name changes from last releases. Updated the MinGW64 winlibs download used on Windows to the latest version. Updated the Updated Visual Code C config to use Python3.10 and MSVC 2022 include paths. Better outputs from standalone library compilation test, esp. when finding a problem, present the script to reproduce it immediately. Enhanced generated tests to cover Added automatic execution of Enhanced output in case of This release has seen a lot of consolidation. The plugins layer for data files is now all powerful, allowing much nicer handling of them by the plugins, they are better reported in normal output, and they are also part of the report file that Nuitka can create. You may now also inhibit their inclusion from the command line, if you decide otherwise. The Generally many features got more polished, e.g. non-automatic inclusion of stdlib modules has most problems fixed up. An important area of improvement, are the hard imports. These will be used to replace the source based resolution of package requirements with ones that are proper nodes in the tree. Getting these hard imports to still retain full compatibility with existing imports, that are more or less For onefile, the cached mode is being prepared with the variables added, but will only be in a later release. Also a bunch of new or upgraded packages are working now, and the push for Bug Fixes
scipy.stats function copying. Fixed in 0.7.1 already.importlib.metadata module failed to include email from standard library parts no longer included by default. Fixed in 0.7.1 already.shiboken6 module (mostly due to PySide6) failed to include argparse from the standard library from standard library parts no longer included by default. Fixed in 0.7.1 already.pkg_resources as well, we need it for when we use Jinja2, which is more often now. For Python3 this was fixed in 0.7.3 already. Later a version to use with Python2 was added as well.urllib.requests module failed to include http.client from standard library parts no longer included by default. Fixed in 0.7.3 already. Later http.cookiejar was added too.upx plugin, that is not going to work. Fixed in 0.7.4 already.package_dir directive to specify where source code lives. Fixed in 0.7.6 already.shelve module failed to include dbm from standard library parts no longer included by default. Fixed in 0.7.6 already.arcade data files. Fixed in 0.7.7 already.sys.modules before executing an extension module that will create it. This fixes cases of cyclic dependencies from modules loaded by the extension module.clcache was itself triggering one during its handling, hiding the real exception behind a TypeError.clcache locking behavior, avoiding a race. Also increase characters used for key from 2 to 3 chars, making collisions far more rare.persistent package.tensorflow package.__file__ and __spec__ values of top level module. It is changed by CPython after import to an incompatible file name, and not our loader, preventing package resources to be found.Crpytodome.Cipher.PKCS1_v1_5.pkgutil.iter_modules without arguments was not working properly with our meta path based loader.PATH that failed to encode.divmod and modulo % with negative remainders of positive floats was not correct.str.encode with only errors value, but default value for encoding was crashing the compilation.match statement sliced values must be lists, not tuples.glfw and OpenGL packages.pydantic.str.split rejected default sep value with only maxsplit given as a keyword argument.wsgiref module.falcon module.eliot module.uvicorn package.accessible_output2, babel, frozendict, and sound_lib package.sklearn package.tkinterdnd2 package.idlelib from stdlib was always ignored.__spec__.origin as produced by find_spec of our meta path based loader, didn’t have the correct origin attribute value.orderedmultidict.clr module.cv2 package.New Features
--module-name-choice to select what value __name__ and __package__ are going to be. With --module-name-choice=runtime (default for --module mode), the created module uses the parent package to deduce the value of __package__, to be fully compatible. The value --module-name-choice=original (default for other modes) allows for more static optimization to happen.get_resource_reader to our meta path based loader. This allows to avoid useless temporary files in case importlib.resources.path is used, due to a bad interaction with the fallback implementation used without it.--force-stdout-spec```and ``--force-stderr-spec on all platforms, this was previously limited to Windows.wx will only work as a GUI program and crash unless --disable-console is specified. These will warn the user or outright error the compilation if something is known to be needed or useful.--noinclude-data-files to instruct Nuitka to not include data files matching patterns given. Also attached loggers and tags to included data file and include them in the compilation report.pyproject.toml without setup.py so far it was not possible to pass arguments. This is now possible by adding a section like this.[tool.nuitka]
# options without an argument are passed as boolean value
show-scons = true
# options with single values, e.g. enable a plugin of Nuitka
enable-plugin = "pyside2"
# options with several values, e.g. avoiding including modules, accepts
# list argument.
nofollow-import-to = ["*.tests", "*.distutils"]
--macos-sign-identity and access to protected resources --macos-app-protected-resource.upx plugin to accept both a folder path and the full path including the binary name to work when you specify the binary location with --upx-binary making it more user friendly.%CACHE_DIR, %COMPANY%, %PRODUCT%, %VERSION%, and %HOME in preparation of onefile once again being able to be cached and not unpacked repeatedly for each execution.tk-inter plugin at run time. When TCL fails to load, it then outputs a more helpful error. This ought to be done for all plugins, where it’s not clear if they are needed.anti-bloat.yml file. Before with replacement`, the new value had to be produced by an eval, which makes for less readable values due to extra quoting. for plain values.Optimization
ordered-set PyPI package to speed up compilation on these versions too, adding a warning if no accelerated form of OrderedSet is used, but believed to be usable.bytes.decode operations. This is only a start and we needed this for internal usage, more should follow later.anti-bloat work was added. Avoiding ipython, unittest, and sometimes even doctest usage for some more packages.ccache was not always used, sometimes it believed to catch a potential race, that we need to tell it to ignore. This will speed up re-compilation of the C side in many cases.pkgutil.get_data. This will make it easier to detect cases of missing data files in the future.ldd, this should avoid exceeding command line limits.clcache remove writing of the stats file before Scons has completed, which avoids IO and locking churn.wsgiref from stdlib by default.Cleanups
chrpath and dead code around it, it was still listed as a dependency, although we stopped using it a while ago.anti-bloat in examples and tests, it is now enabled by default.*Plugin.py.tensorflow plugin. The source modification was moved to anti-bloat where it is easy to do. The implicit dependencies are now in the config file of implicit-imports plugin.#ifdefs usages with new helper function Nuitka_String_FromFormat that implies them for more readable code.long code. In testing it worked correctly, but this is more explicit and doesn’t rely on C implementation specific behavior, although it appears to be universal.Organizational
pyproject.toml files, this makes debugging Nuitka straightforward in these setups.chrpath and the now dead code that would use it, we are happy with patchelf.black and pylint.--macos-onefile-icon to --macos-app-icon because that is what it is really used for.ccache binary used on Windows with MinGW64. This is in preparation of using it potentially for MSVC as well.Tests
str methods to use keyword arguments.pyproject.toml driven test case.optimization test failures, dumping what value is there that has not become a compile time constant.Summary
pyproject.toml now supporting Nuitka arguments is closing an important gap.__import__ uses only, was revealing quite a bit of technical debt, that has been addressed with this release.anti-bloat work has increased, making many compilations even more lean, but scalability is still an issue.