16 October 2022
Nuitka Release 1.1
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release contains a large amount of new compatibility features, while consolidating what we have. Scalability should be better in some cases. Standalone: Enhanced dependency scan of dependent DLLs to forward the containing package, so it can be searched in as well. This fixed at least PySide on macOS. Fixed in 1.0.1 already. macOS: Enhanced dependency detection to use normalized paths and therefore to be more stable. Fixed in 1.0.1 already. Standalone: Added support for the Standalone: Include data files for Python3.10: Fix, could crash in case a class was not giving Standalone: Added data files needed for Python3.10: Fix, matching sequence with Standalone: Fix, do not attempt to list non-existent Standalone: Fix, on newer Linux, Fix, comparison expressions could give wrong results as a regression of the new release. Fixed in 1.0.3 already. Fix, on older Python (before 3.6), it could crash on data files defined in the Yaml config. Fixed in 1.0.4 already. Fix, binary operations could give wrong results as a regression of the new release. Fixed in 1.0.4 already. Standalone: Added support for Standalone: Fix, empty directory structures were not working anymore due to a regression in the last release. Fixed in 1.0.5 already. Windows: Fix, detected Pythons from Windows registry may of course fail to execute, because they were e.g. manually deleted. This would show e.g. in onefile compression. Fixed in 1.0.5 already. Onefile: Fix, using a too old Fix, the inline copy of Jinja2 imported Fix, disable LTO mode for Anaconda Python, it is known to not work. Fixed in 1.0.6 already. Linux: Fix, no need to insist on icon path for onefile anymore. Fixed in 1.0.6 already. Standalone: Fix, the new version Standalone: Added support for more Standalone: Added support for Fix, must not expose module name objects to Python import hooks. Fixed in 1.0.8 already. Fix, calls to bound methods of string values generated incorrect calls. Fixed in 1.0.8 already. Fix, do not crash in version detection on Standalone: Added back support for older versions of the Standalone: Ignore Standalone: Added support for Python3.10: Fix, Fix, Standalone: Added support for the Standalone: Added support for using Debian: Fix, versioned dependency for Standalone: Added support for Windows: Fix, avoid using Standalone: Added support for more Standalone: Adding support for Standalone: Added support for Standalone: Added support for Scons: Avoid gcc linker command line length limits for module mode too. Standalone: Added data file of Standalone: Add support for Fix, the anti-bloat configuration for Standalone: Fix DLL dependency caching on Windows, need to consider DLL content of course too. Standalone: Added missing dependency for Standalone: Added support for Standalone: Added support for Windows: Fix, need to make sure to use UTF-8 encoding for define values like company name. Otherwise the local system encoding is used, but the C compiler expects UTF-8 in wide literals. This may crash of give wrong results. Standalone: Added Anaconda: Include Windows: Fix, can have file version without a company name. Python3.10: Added support for assignments in Compatibility: Register Nuitka meta path based loader with Compatibility: Make function Compatibility: Added support for Onefile: Added support for including other binaries than the main executable in the payload. So far on non-Windows, we only made the main binary executable, hard coded, and nothing else. But Some things, e.g. Qt web engine, do require binaries to be used, and these no longer have the issue of missing x-bit on macOS and Linux now. Standalone: Resolve executable path when called through symbolic link, which makes file resolutions work properly for it, for this type of installing it in Python3.9+: Added support for It returns traversable objects, which can be used to opens files, checks them, etc. and this e.g. allows UI: Added interface method to specify filename patterns with package data inclusion option, making macOS: Make Consistently allow Plugins: Added method to overload to work on standalone binary specifically. This makes it easier to only modify that specific binary. Plugins: Added support for regular expressions in anti-bloat replacements, with new Add support for Standalone: Do not include Standalone: Do not include Standalone: Do not include Avoid compilation of large generated codes in the Compile time optimize For known truth values of the right hand side of Added dedicated assignment node for hard imports, which then are propagated in classes as well, allowing for more static optimization for code on the class level. Added linker options to make static macOS: Make sure macOS: Fix, need to search environment specific DLL paths and only then global paths, otherwise mixed Python versions will not work correctly. Anti-Bloat: Remove IPython usage in Anti-Bloat: Avoid Anti-Bloat: Some Keep the Scons report items sorted, or else it varies for the hashing of dependencies with Python versions before 3.6, causing cache misses without need. UI: Output the UI: Enhanced command line option description of UI: Specified needed Python version for use in Reports: Added information about data files including, optimization times per module, active plugins. Debugging: Repaired offline DLL dependency listing tool, such that it can be used during Windows DLL analysis. Make Plugins: Avoid useless warning about PySide2 plugin usage if another Qt plugin is actually selected. UI: Catch error of directories being used as data files where plain files are expected and point out that other options must be used. User Manual: Added section about accessing files in standalone mode too, so people can make sure it works properly. Onefile: Using UI: Catch user error of using elements, that resolve to absolute values in the middle of path specs, so using e.g. Quality: Updated to newer version of UI: Nicer error message when a forbidden import is requested as an implicit import by a plugin. Python3.11: Adapted to allocator and exception state changes, but more will be needed to compile at all. Visual Code: Find UI: Quote command line argument values as necessary when stating them in the logging. Otherwise they are not directly usable on the shell and also less readable. Debian: Do not list fake modules as used debian packages codes, which could e.g. happen with the pre-load code of Nuitka-Python: Added support to set link time flags coming from statically included packages. For our Added more usable form UI: Check if output filename’s directory exists ahead of time, and error exit if not, otherwise compilation crashed only in the very end, trying to create the final result. UI: When exiting with no error code, do not use red color or Quality: Make sure the Yaml auto-format does not change effective contents. Quality: Added ability to limit autoformat by file type, which can be handy when e.g. only the yaml files should be scanned. UI: Removed Prefer single quotes rather than double quotes in our package configuration Yaml files, otherwise esp. regular expressions with escapes become very confusing. Move import hacks to general mechanism in Yaml package configuration files. This is for extra paths from package names or from directory paths relative to the package. This removes special purpose code from core code paths and allows their reuse. Again more spelling cleanups have been done, to make the code cleaner to read and search. Unified how plugins treat iteration over their value list, and how the Output compilation command that failed during coverage taking, which makes it unnecessary to attempt to reconstruct what happened from test modes. Added coverage for comparisons that need argument swaps. Allow more time in onefile keyboard signal test, otherwise it can be a race on slow machines, e.g. emulated machines. Tests: Added support for running a local web server. This release is mainly a consolidation of previous release. Optimization added in previous release did in fact introduce regressions, that needed to be addressed and were cause for relatively many hotfixes. The Yaml Nuitka Package Configuration feature is getting ever more powerful, but is not one bit more documented, such that the community as a whole is not yet capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches. New optimization was focused around compatibility with very few exceptions, where the non-automatic standard library work is standing out, and allows for smaller binaries in many cases. Scalability has seen improvements through a few optimization, but mainly again with anti-bloat work being done. This is owed to the fact that consolidation was the name of the game. For Anaconda specifically, a lot more software is covered, and generally, The compilation report contains much more information and is getting there is terms of completeness. At some point, we should ask for it in bug reports.Bug Fixes
networkx package which uses new support for a function decorator trying to copy function default values. Fixed in 1.0.1 already.pandas.io.format package. This one has Jinja2 template files that will be needed when using this package.match arguments, but the user did attempt to match them. This happened e.g. with range objects. Fixed in 1.0.2 already.pyenchant package. Fixed in 1.0.2 already.as assignments in them didn’t check for sub-pattern given. Fixed in 1.0.2 already.PATH entries on Windows, these can crash the dependency detection otherwise. Fixed in 1.0.2 already.linux-vdso.so.1 appears in output of ldd in a way that suggests it may exist, which of course it does not, this is a kernel virtual DLL. Fixed in 1.0.3 already.pyzbar package. Fixed in 1.0.5 already.zstandard without finding another Python with a suitable one, lead to run time unpacking errors. Fixed in 1.0.6 already.logging for no good reason, which lead to errors for users who have a module of the same name, that it was then using instead. Fixed in 1.0.6 already.certifi was not working on Windows and 3.10 anymore. Fixed in 1.0.7 already.rapidfuzz implicit dependencies. Fixed in 1.0.8 already.vibora. Fixed in 1.0.8 already.gcc error exit querying of its version.pyzmq package.PATH elements that fail to be listed as a directory. It appears e.g. on Windows, folders can exist, despite being unusable in fact. These can then cause errors in DLL dependency scan. Also avoid having PATH set when executing dependency walker, it appears to use it even if not asked to.tzlocal package.complex literals were not working for mappings in match statements.bool built-in expressions were not properly annotating exception raises, where the value cannot raise on truth check.agscheduler package. Plugins must be done manually still with explicit --include-module calls.shapely in Anaconda as well.libzstd should also be in package, this should restore Nuitka package builds for Debian Jessie.vtk package.pywin32 in our appdirs usage, it might be a broken installation and is optional to appdirs anyway, which then will fallback to using ctypes to make the lookups.pandas versions.mkl implicit DLL usage in Anaconda.jsonschema with Python 3.10.pyfiglet fonts data files.distributed.config.cv2 GUI on Linux, the Qt platform plugin is now included.numpy.testing tools exposed an incomplete suppress_warnings replacement that could lead to errors in some functions of numpy.torchvision.torchvision on Anaconda as well.panda3d.facenet_torch data files.libstdc++.so on Linux or else e.g. cv2 will not work with system library.New Features
match alternatives | syntax.pkg_resources such that checking resource presence with has_resource works too. This should also add support for using jinja2.PackageLoader, previously only jinja2.FileSystemLoader worked. Fixed in 1.0.1 already.__defaults__ attribute size changeable. For a long time, this was a relatively big issue for some packages, but now this is supported as well.importlib.metadata.distribution and importlib_metadata.distribution functions as well importlib.metadata.metadata and importlib_metadata.metadata functions.%PATH%.importlib.resources.files with compiled modules.jsonschema to work with Python 3.10, despite bugs in CPython’s compatibility layer.--include-package-data usable in many more cases, picking the only files or file types you want. You can now use --include-package-data=package_name=*.txt and select only a subset of package data files in this way. Before this, it included everything and --noinclude-data-files would have to be used.runtime signing an experimental option.when conditions for all package configuration elements, e.g. also DLLs.replacements_re code.Optimization
os.path hard module imports along with specialized nodes for file tests os.path.exists, os.path.isfile, and os.path.isdir aiming at tracking used files, producing warnings about missing files in the future.concurrent standard library package automatically. This avoids the inclusion of multiprocessing which we essentially had reverted during last release cycle.zoneinfo standard library package automatically. It has many data files and is often not used (yet).asyncio standard library package automatically anymore.asyncua package.pkg_resources.iter_entry_points too, such that these can be used to resolve plugin modules, which helps with adding support for agscheduler package plugins. Note that these still need to be manually included with --include-module but now that works.and or or conditions, reduce the expression as far as possible.--static-libpython work with clang on Linux as well.libpython is loaded relative to the executable. This is needed for at least Anaconda Python.rich package.doctest dependency when using pyrect.unittest removals from pytorch using libraries.Organizational
.cmd file created (if any) on Windows, e.g. when run in a virtualenv or for uninstalled Python versions, it will otherwise not run in accelerated mode, but previously the output suggested to run the executable directly.--include-plugin-directory which is frequently misunderstood. That option barely does what people want it to do. Point them to using the other options that are easy to use and will work.--python-for-scons so users can know ahead of time what versions are suitable.--xml accept a filename for the node tree dump, and change it so it can be executed in addition to actual compilation. This way we need not be super-robust about keeping stdout clean, to not break XML parsing.%TEMP% folder should not by itself prevent cached onefile mode, only really variable paths should. People may want to have this as some kind of temporary cache still.something/%PROGRAM% is now a mistake caught at compile time. These values can only be at the start of spec values naturally.rstfmt.clang-format from the recommended C++ extension of Visual Code, which makes it finally available on macOS easily too.pkg_resources if that is from a Debian package. Fake packages should not be mentioned for these lists though.isort trick of splitting files in two parts (mostly to setup import paths for nuitka package), make sure the second parts starts with a new line.--output-filename to specify the output filename, the short form has become barely usable after we switched to enforcing no space separation for command line arguments.FATAL error annotation, that is not justified.--windows-onefile-tempdir-spec alias of --onefile-tempdir-spec option, it is no longer Windows specific.Cleanups
when condition is applied for the various kinds of sections.Tests
Summary
cv2 and torch related tools are now working better, but it seems DLL handling will remain problematic in many instances.