Nuitka Blog - Posts tagged compiler

Nuitka Release 2.8

2025-11-15T13:52:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release adds a ton of new features and corrections.

Bug Fixes

Standalone: For the “Python Build Standalone” flavor ensured that debug builds correctly recognize all their specific built-in modules, preventing potential errors. (Fixed in 2.7.2 already.)
Linux: Fixed a crash when attempting to modify the RPATH of statically linked executables (e.g., from imageio-ffmpeg). (Fixed in 2.7.2 already.)
Anaconda: Updated PySide2 support to correctly handle path changes in newer Conda packages and improved path normalization for robustness. (Fixed in 2.7.2 already.)
macOS: Corrected handling of QtWebKit framework resources. Previous special handling was removed as symlinking is now default, which also resolved an issue of file duplication. (Fixed in 2.7.2 already.)
Debugging: Resolved an issue in debug builds where an incorrect assertion was done during the addition of distribution metadata. (Fixed in 2.7.1 already.)
Module: Corrected an issue preventing stubgen from functioning with Python versions earlier than 3.6. (Fixed in 2.7.1 already.)
UI: Prevented Nuitka from crashing when --include-module was used with a built-in module. (Fixed in 2.7.1 already.)
Module: Addressed a compatibility issue where the code mode for the constants blob failed with the C++ fallback. This fallback is utilized on very old GCC versions (e.g., default on CentOS7), which are generally not recommended. (Fixed in 2.7.1 already.)
Standalone: Resolved an assertion error that could occur in certain Python setups due to extension module suffix ordering. The issue involved incorrect calculation of the derived module name when the wrong suffix was applied (e.g., using .so to derive a module name like gdbmmodule instead of just gdbm). This was observed with Python 2 on CentOS7 but could potentially affect other versions with unconventional extension module configurations. (Fixed in 2.7.1 already.)
Python 3.12.0: Corrected the usage of an internal structure identifier that is only available in Python 3.12.1 and later versions. (Fixed in 2.7.1 already.)
Plugins: Prevented crashes in Python setups where importing pkg_resources results in a PermissionError. This typically occurs in broken installations, for instance, where some packages are installed with root privileges. (Fixed in 2.7.1 already.)
macOS: Implemented a workaround for data file names that previously could not be signed within app bundles. The attempt in release 2.7 to sign these files inadvertently caused a regression for cases involving illegal filenames. (Fixed in 2.7.1 already.)
Python 2.6: Addressed an issue where staticmethod objects lacked the __func__ attribute. Nuitka now tracks the original function as a distinct value. (Fixed in 2.7.1 already.)
Corrected behavior for orderedset implementations that lack a union method, ensuring Nuitka does not attempt to use it. (Fixed in 2.7.1 already.)
Python 2.6: Ensured compatibility for setups where the _PyObject_GC_IS_TRACKED macro is unavailable. This macro is now used beyond assertions, necessitating support outside of debug mode. (Fixed in 2.7.1 already.)
Python 2.6: Resolved an issue caused by the absence of sys.version_info.releaselevel by utilizing a numeric index instead and adding a new helper function to access it. (Fixed in 2.7.1 already.)
Module: Corrected the __compiled__.main value to accurately reflects the package in which a module is loaded, this was not the case for Python versions prior to 3.12. (Fixed in 2.7.1 already.)
Plugins: Further improved the dill-compat plugin by preventing assertions related to empty annotations and by removing hard-coded module names for greater flexibility. (Fixed in 2.7.1 already.)
Windows: For onefile mode using DLL mode, ensure all necessary environment variables are correctly set for QtWebEngine. Previously, default Qt paths could point incorrectly near the onefile binary. (Fixed in 2.7.3 already.)
PySide6: Fixed an issue with PySide6 where slots defined in base classes might not be correctly handled, leading to them only working for the first class that used them. (Fixed in 2.7.3 already.)
Plugins: Enhanced Qt binding plugin support by checking for module presence without strictly requiring metadata. This improves compatibility with environments like Homebrew or uv where package metadata might be absent. (Fixed in 2.7.3 already.)
macOS: Ensured the apple target is specified during linking to prevent potential linker warnings about using an unknown target in certain configurations. (Fixed in 2.7.3 already.)
macOS: Disabled the use of static libpython with pyenv installations, as this configuration is currently broken. (Fixed in 2.7.3 already.)
macOS: Improved error handling for the --macos-app-protected-resource option by catching cases where a description is not provided. (Fixed in 2.7.3 already.)
Plugins: Enhanced workarounds for PySide6, now also covering single-shot timer callbacks. (Fixed in 2.7.4 already.)
Plugins: Ensured that the Qt binding module is included when using accelerated mode with Qt bindings. (Fixed in 2.7.4 already.)
macOS: Avoided signing through symlinks and minimized their use to prevent potential issues, especially during code signing of application bundles. (Fixed in 2.7.4 already.)
Windows: Implemented path shortening for paths used in onefile DLL mode to prevent issues with long or Unicode paths. This also benefits module mode. (Fixed in 2.7.4 already.)
UI: The options nanny plugin no longer uses a deprecated option for macOS app bundles, preventing potential warnings or issues. (Fixed in 2.7.4 already.)
Plugins: Ensured the correct macOS target architecture is used. This particularly useful for PySide2 with universal CPython binaries, to prevent compile time crashes e.g. when cross-compiling for a different architecture. (Fixed in 2.7.4 already.)
UI: Fixed a crash that occurred on macOS if the ccache download was rejected by the user. (Fixed in 2.7.4 already.)
UI: Improved the warning message related to macOS application icons for better clarity. (Added in 2.7.4 already.)
Standalone: Corrected an issue with QML plugins on macOS when using newer PySide6 versions. (Fixed in 2.7.4 already.)
Python 3.10+: Fixed a memory leak where the matched value in pattern matching constructs was not being released. (Fixed in 2.7.4 already.)
Python3: Fixed an issue where exception exits for larger range objects, which are not optimized away, were not correctly annotated by the compiler. (Fixed in 2.7.4 already.)
Windows: Corrected an issue with the automatic use of icons for PySide6 applications on non-Windows if Windows icon options were used. (Fixed in 2.7.4 already.)
Onefile: When using DLL mode there was a load error for the DLL with MSVC 14.2 or earlier, but older MSVC is to be supported. (Fixed in 2.7.5 already.)
Onefile: Fix, the splash screen was showing in DLL mode twice or more; these extra copies couldn’t be stopped. (Fixed in 2.7.5 already.)
Standalone: Fixed an issue where data files were no longer checked for conflicts with included DLLs. The order of data file and DLL copying was restored, and macOS app signing was made a separate step to remove the order dependency. (Fixed in 2.7.6 already.)
macOS: Corrected our workaround using symlinks for files that cannot be signed. When --output-directory was used, as it made incorrect assumptions about the dist folder path. (Fixed in 2.7.6 already.)
UI: Prevented checks on onefile target specifications when not actually compiling in onefile mode, e.g. on macOS with --mode=app. (Fixed in 2.7.6 already.)
UI: Improved error messages for data directory options by include the relevant part in the output. (Fixed in 2.7.6 already.)
Plugins: Suppressed UserWarning messages from the pkg_resources module during compilation. (Fixed in 2.7.6 already.)
Python3.11+: Fixed an issue where descriptors for compiled methods were incorrectly exposed for Python 3.11 and 3.12. (Fixed in 2.7.7 already.)
Plugins: Avoided loading modules when checking for data file existence. This prevents unnecessary module loading and potential crashes in broken installations. (Fixed in 2.7.9 already.)
Plugins: The global_change_function anti-bloat feature now operates on what should be the qualified names (__qualname__) instead of just function names, preventing incorrect replacements of methods with the same name in different classes. (Fixed in 2.7.9 already.)
Onefile: The containing_dir attribute of the __compiled__ object was regressed in DLL mode on Windows, pointing to the temporary DLL directory instead of the directory containing the onefile binary. (Fixed in 2.7.10 already, note that the solution in 2.7.9 had a regression.)
Compatibility: Fixed a crash that occurred when an import attempted to go outside its package boundaries. (Fixed in 2.7.11 already.)
macOS: Ignored a warning from codesign when using self-signed certificates. (Fixed in 2.7.11 already.)
Onefile: Fixed an issue in DLL mode where environment variables from other onefile processes (related to temporary paths and process IDs) were not being ignored, which could lead to conflicts. (Fixed in 2.7.12 already.)
Compatibility: Fixed a potential crash that could occur when processing an empty code body. (Fixed in 2.7.13 already.)
Plugins: Ensured that DLL directories created by plugins could be at the top level when necessary, improving flexibility. (Fixed in 2.7.13 already.)
Onefile: On Windows, corrected an issue in DLL mode where original_argv0 was None; it is now properly set. (Fixed in 2.7.13 already.)
macOS: Avoided a warning that appeared on newer macOS versions. (Fixed in 2.7.13 already.)
macOS: Allowed another DLL to be missing for PySide6 to support more setups. (Fixed in 2.7.13 already.)
Standalone: Corrected the existing import workaround for Python 3.12 that was incorrectly renaming existing modules of matching names into sub-modules of the currently imported module. (Fixed in 2.7.14 already.)
Standalone: On Windows, ensured that the DLL search path correctly uses the proper DLL directory. (Fixed in 2.7.14 already.)
Python 3.5+: Fixed a memory leak where the called object could be leaked in calls with keyword arguments following a star dict argument. (Fixed in 2.7.14 already.)
Python 3.13: Fixed an issue where PyState_FindModule was not working correctly with extension modules due to sub-interpreter changes. (Fixed in 2.7.14 already.)
Onefile: Corrected an issue where the process ID (PID) was not set in a timely manner, which could affect onefile operations. (Fixed in 2.7.14 already.)
Compatibility: Fixed a crash that could occur when a function with both a star-list argument and keyword-only arguments was called without any arguments. (Fixed in 2.7.16 already.)
Standalone: Corrected an issue where distribution names were not checked case-insensitively, which could lead to metadata not being included. (Fixed in 2.7.16 already.)
Linux: Avoid using full zlib with extern declarations but instead only the CRC32 functions we need. Otherwise conflicts with OS headers could occur.
Standalone: Fixed an issue where scanning for standard library dependencies was unnecessarily performed.
Plugins: Made the runtime query code robust against modules that in stdout during import

This affected at least toga giving some warnings on Windows with mere stdout prints. We now have a marker for the start of our output that we look for and safely ignore them.
Windows: Do not attempt to attach to the console when running in DLL mode. For onefile with DLL mode, this was unnecessary as the bootstrap already handles it, and for pure DLL mode, it is not desired.
Onefile: Removed unnecessary parent process monitoring in onefile mode, as there is no child process launched.
Anaconda: Determine version and project name for conda packages more reliably

It seems Anaconda is giving variables in package metadata and often no project name, so we derive it from the conda files and its meta data in those cases.
macOS: Make sure the SSL certificates are found when downloading on macOS, ensuring successful downloads.
Windows: Fixed an issue where console mode attach was not working in onefile DLL mode.
Scons: Fixed an issue where pragma was used with older gcc gcc can give warnings about them. This fixes building on older OSes with the system gcc.
Compatibility: Fix, need to avoid using filenames with more than 250 chars for long module names.
- For cache files, const files, and C files, we need to make sure, we don’t exceed the 255 char limits per path element that literally every OS has.
- Also enhanced the check code for legal paths to cover this, so user options are covered from this errors too.
- Moved file hashing to file operations where it makes more sense to allow module names to use hashing to provide a legal filename to refer to themselves.
Compatibility: Fixed an issue where walking included compiled packages through the Nuitka loader could produce incorrect names in some cases.
Windows: Fixed wrong calls made when checking stderr properties during launch if it was None.
Debugging: Fixed an issue where the segfault non-deployment disable itself before doing anything else.
Plugins: Fix, the warning to choose a GUI plugin for matplotlib was given with tk-inter plugin enabled still, which is of course not appropriate.
Distutils: Fix, do not recreate the build folder with a .gitignore file.

We were re-creating it as soon as we looked at what it would be, now it’s created only when asking for that to happen.
No-GIL: Addressed compile errors for the free-threaded dictionary implementation that were introduced by necessary hot-fixes in the version 2.7.
Compatibility: Fixed handling of generic classes and generic type declarations in Python 3.12.
macOS: Fixed an issue where entitlements were not properly provided for code signing.
Onefile: Fixed delayed shutdown for terminal applications in onefile DLL mode.

Was waiting for non-used child processes, which don’t exist and then the timeout for that operation, which is always happening on CTRL-C or terminal shutdown.
Python3.13: Fix, seems interpreter frames with None code objects exist and need to be handled as well.
Standalone: Fix, need to allow for setuptools package to be user provided.
Windows: Avoided using non-encodable dist and build folder names.

Some paths don’t become short, but still be non-encodable from the file system for tools. In these cases, temporary filenames are used to avoid errors from C compilers and other tools.
Python3.13: Fix, ignore stdlib cgi module that might be left over from previous installs

The module was removed during development, and if you install over an old alpha version of 3.13 a newer Python, Nuitka would crash on it.
macOS: Allowed the lib folder for the Python Build Standalone flavor, improving compatibility.
macOS: Allowed libraries for rpath resolution to be found in all Homebrew folders and not just lib.
Onefile: Need to allow .. in paths to allow outside installation paths.

Package Support

Standalone: Introduced support for the nicegui package. (Added in 2.7.1 already.)
Standalone: Extended support to include xgboost.core on macOS. (Added in 2.7.1 already.)
Standalone: Added needed data files for ursina package. (Added in 2.7.1 already.)
Standalone: Added support for newer versions of the pydantic package. (Added in 2.7.4 already.)
Standalone: Extended libonnxruntime support to macOS, enabling its use in compiled applications on this platform. (Added in 2.7.4 already.)
Standalone: Added necessary data files for the pygameextra package. (Added in 2.7.4 already.)
Standalone: Included GL backends for the arcade package. (Added in 2.7.4 already.)
Standalone: Added more data directories for the ursina and panda3d packages, improving their out-of-the-box compatibility. (Added in 2.7.4 already.)
Standalone: Added support for newer skimage package. (Added in 2.7.5 already.)
Standalone: Added support for the PyTaskbar package. (Added in 2.7.6 already.)
macOS: Added tk-inter support for Python 3.13 with official CPython builds, which now use framework files for Tcl/Tk. (Added in 2.7.6 already.)
Standalone: Added support for the paddlex package. (Added in 2.7.6 already.)
Standalone: Added support for the jinxed package, which dynamically loads terminal information. (Added in 2.7.6 already.)
Windows: Added support for the ansicon package by including a missing DLL. (Added in 2.7.6 already.)
macOS: Enhanced configuration for the pypylon package, however, it’s not sufficient. (Added in 2.7.6 already.)
Standalone: Added support for newer numpy versions. (Added in 2.7.7 already.)
Standalone: Added support for older vtk package. (Added in 2.7.8 already.)
Standalone: Added support for newer certifi versions that use importlib.resources. (Added in 2.7.9 already.)
Standalone: Added support for the reportlab.graphics.barcode module. (Added in 2.7.9 already.)
Standalone: Added support for newer versions of the transformers package. (Added in 2.7.11 already.)
Standalone: Added support for newer versions of the sklearn package. (Added in 2.7.12 already.)
Standalone: Added support for newer versions of the scipy package. (Added in 2.7.12 already.)
Standalone: Added support for older versions of the cv2 package (specifically version 4.4). (Added in 2.7.12 already.)
Standalone: Added initial support for the vllm package. (Added in 2.7.12 already.)
Standalone: Ensured all necessary DLLs for the pygame package are included. (Added in 2.7.12 already.)
Standalone: Added support for newer versions of the zaber_motion package. (Added in 2.7.13 already.)
Standalone: Added missing dependencies for the pymediainfo package. (Added in 2.7.13 already.)
Standalone: Added support for newer versions of the sklearn package by including a missing dependency. (Added in 2.7.13 already.)
Standalone: Added support for newer versions of the toga package. (Added in 2.7.14 already.)
Standalone: Added support for the wordninja-enhanced package. (Added in 2.7.14 already.)
Standalone: Added support for the Fast-SSIM package. (Added in 2.7.14 already.)
Standalone: Added a missing data file for the rfc3987_syntax package. (Added in 2.7.14 already.)
Standalone: Added missing data files for the trimesh package. (Added in 2.7.15 already.)
Standalone: Added support for the gdsfactory, klayout, and kfactory packages. (Added in 2.7.15 already.)
Standalone: Added support for the vllm package. (Added in 2.7.16 already.)
Standalone: Added support for newer versions of the tkinterweb package. (Added in 2.7.15 already.)
Standalone: Added support for newer versions of the cmsis_pack_manager package. (Added in 2.7.15 already.)
Standalone: Added missing data files for the idlelib package. (Added in 2.7.15 already.)
Standalone: Avoid including debug binary on non-Windows for Qt Webkit.
Standalone: Add dependencies for pymediainfo package.
Standalone: Added support for the winpty package.
Standalone: Added support for newer versions of the gi package.
Standalone: Added support for newer versions of the litellm package.
Standalone: Added support for the traits and pyface packages.
Standalone: Added support for newer versions of the transformers package.
Standalone: Added data files for rasterio package.
Standalone: Added support for ortools package.
Standalone: Added support newer “vtk” package

New Features

Python3.14: Added experimental support for Python3.14, not recommended for use yet, as this is very fresh and might be missing a lot of fixes.
Release: Added an extra dependency group for the Nuitka build-backend, intended for use in pyproject.toml and other build-system dependencies. To use it depend in Nuitka[build-wheel] instead of Nuitka. (Added in 2.7.7 already.)

For release we also added Nuitka[onefile], Nuitka[standalone], Nuitka[app] as extra dependency groups. If icon conversions are used, e.g. Nuitka[onefile,icon-conversion] adds the necessary packages for that. If you don’t care about what’s being pulled in Nuitka[all] can be used, by default Nuitka only comes with the bare minimum needed and will inform about missing packages.
macOS: Added --macos-sign-keyring-filename and --macos-sign-keyring-password to automatically unlock a keyring for use during signing. This is very useful for CI where no UI prompt can be used.
Windows: Detect when input cannot be used due to no console or the console not providing proper standard input and produce a dialog for entry instead. Shells like cmd.exe execute inputs as commands entered when attaching to them. With this, the user is informed to make the input into the dialog instead. In case of no terminal, this just brings up the dialog for GUI mode.
Plugins: Introduced global_change_function to the anti-bloat engine, allowing function replacements across all sub-modules of a package at once. (Added in 2.7.6 already.)
Reports: For Python 3.13+, the compilation report now includes information on GIL usage. (Added in 2.7.7 already.)
macOS: Added an option to prevent an application from running in multiple instances. (Added in 2.7.7 already.)
AIX: Added support for this OS as well, now standalone and module mode work there too.
Scons: When C a compilation fails to due warnings in --debug mode, recognize that and provide the proper extra options to use if you want to ignore that.
Non-Deployment: Added a non-deployment handler to catch modules
Non-Deployment: Added non-deployment handler to catch modules that error exit on import, while assumed to work perfectly.

This will give people an indication that the numpy module is expected to work and that maybe just the newest version is not and we need to be told about it.
Non-Deployment: Added a non-deployment handler for DistributionNotFound exceptions in the main program, which now points the user to the necessary metadata options.
UI: Made --include-data-files-external the primary option for placing data files alongside the created program.

This now works with standalone mode too, and is no longer onefile specific, the name should reflect that and people can now use it more broadly.
Plugins: Added support for multiple warnings of the same kind. The dill-compat plugin needs that as it supports multiple packages.
Plugins: Added detector for the dill-compat plugin that detects usages of dill, cloudpickle and ray.cloudpickle.
Standalone: Add support for including Visual Code runtime dlls on Windows.
- When MSVC (Visual Studio) is installed, we take the runtime DLLs from its folders. We cannot take the ones from the redist packages installed to system folders for license reasons.
- Gives a warning when these DLLs would be needed, but were not found.
- We might want to add an option later to exclude them again, for size purposes, but correctness out of the box is more important for now.
UI: Make sure the distribution name is correct for --include-distribution-metadata option values.
Plugins: Added support for configuring re-compilation of extension modules from their source code.
- When we have both Python code and an extension module, we only had a global option available on the command line.
- This adds --recompile-extension-modules for more fine grained choices as it allows to specify names and patterns.
- For zmq, we need to enforce it to never be compiled, as it checks if it is compiled with Cython at runtime, so re-compilation is never possible.
Reports: Include environment flags for C compiler and linker picked up for the compilation. Sometimes these cause compilation errors that and this will reveal there presence.

Optimization

Enhanced detection of raise statements that use compile-time constant values which are not actual exception instances.

This improvement prevents Nuitka from crashing during code generation when encountering syntactically valid but semantically incorrect code, such as raise NotImplemented. While such code is erroneous, it should not cause a compiler crash. (Added in 2.7.1 already.)
With unknown locals dictionary variables trust very hard values there too.
- With this using hard import names also optimize inside of classes.
- This makes gcloud metadata work, which previously wasn’t resolved in their code.
macOS: Enhanced PySide2 support by removing the general requirement for onefile mode. Onefile mode is now only enforced for QtWebEngine due to its specific stability issues when not bundled this way. (Added in 2.7.4 already.)
Scons: Added support for C23 embedding of the constants blob with ClangCL, avoiding the use of resources. Since the onefile bootstrap does not yet honor this for its payload, this feature is not yet complete but could help with size limitations in the future.
Plugins: Overhauled the UPX plugin.

Use better compression than before, hint the user at disabling onefile compression where applicable to avoid double compression. Output warnings for files that are not considered compressible. Check for upx binary sooner.
Scons: Avoid compiling hacl code for macOS where it’s not needed.

Anti-Bloat

Improved handling of the astropy package by implementing global replacements instead of per-module ones. Similar global handling has also been applied to IPython to reduce overhead. (Added in 2.7.1 already.)
Avoid docutils usage in the markdown2 package. (Added in 2.7.1 already.)
Reduced compiled size by avoiding the use of “docutils” within the markdown2 package. (Added in 2.7.1 already.)
Avoid including the testing framework from the langsmith package. (Added in 2.7.6 already.)
Avoid including setuptools from jax.version. (Added in 2.7.6 already.)
Avoid including unittest from the reportlab package. (Added in 2.7.6 already.)
Avoid including IPython for the keras package using a more global approach. (Added in 2.7.11 already.)
Avoid including the triton package when compiling transformers. (Added in 2.7.11 already.)
Avoid a bloat warning for an optional import in the seaborn package. (Added in 2.7.13 already.)
Avoid compiling generated google.protobuf.*_pb2 files. (Added in 2.7.7 already.)
Avoid including triton and setuptools when using the xformers package. (Added in 2.7.16 already.)
Refined dask support to not remove pandas.testing when pytest usage is allowed. (Added in 2.7.16 already.)
Avoid compiling the tensorflow module that is very slow and contains generated code.
Avoid using setuptools in cupy package.
Avoid false bloat warning in seadoc package.
Avoid using dask in sklearn package.
Avoid using cupy.testing in the cupy package.
Avoid using IPython in the roboflow package.
Avoid including ray for the vllm package.
Avoid using dill in the torch package.

Organizational

UI: Remove obsolete options to control the compilation mode from help output. We are keeping them only to not break existing workflows, but --mode=... should be used now, and these options will start triggering warnings soon.
Python3.13.4: Reject broken CPython official release for Windows.

The link library included is not the one needed for GIL, and as such it breaks Nuitka heavily and must be errored out on, all smaller or larger micro versions work, but this one does not.
Release: Do not use Nuitka 2.7.9 as it broke data file access via __file__ in onefile mode on Windows. This is a brown paper bag release, with 2.7.10 containing only the fix for that. Sorry for the inconvenience.
Release: Ensured proper handling of newer setuptools versions during Nuitka installation. (Fixed in 2.7.4 already.)
UI: Sort --list-distribution-metadata output and remove duplicates. (Changed in 2.7.8 already.)
Visual Code: Added a Python 2.6 configuration for Win32 to aid in comparisons and legacy testing.
UI: Now lists available Qt plugin families if --include-qt-plugin cannot find one.
UI: Warn about compiling a file named __main__.py which should be avoided, instead you should specify the package directory in that case.
- UI: Make it an error to compile a file named __init__.py for standalone mode.
Debugging: The --edit option now correctly finds files even when using long, non-shortened temporary file paths.
Debugging: The pyside6 plugin now enforces --no-debug-immortal-assumptions when --debug is on because PySide6 violates these and we don’t need Nuitka to check for that then as it will abort when it finds them.
Quality: Avoid writing auto-formatted files with same contents
- That avoids stirring up tools that listen to changes.
- For example the Nuitka website auto-builder otherwise rebuilt per release post on docs update.
Quality: Use latest version of deepdiff.
Quality: Added autoformat for JSON files.
Release: The man pages were using outdated options and had no example for standalone or app modes. Also the actual options were no longer included.
GitHub: Use the --mode options in the issue template as well.
GitHub: Enhanced wordings for bug report template to give more directions and more space for excellent reports to be made.
GitHub: The bug report template now requests the output of our package metadata listing tool, as it provides more insight into how Nuitka perceives the environment.
Debugging: Re-enabled important warnings for Clang, which had unnoticed for a long time and prevented a few things from being recognized.
Debugging: Support arbitrary debuggers through –debugger-choice.

Support arbitrary debuggers for use in the --debugger mode, if you specify all of their command line you can do anything there.

Also added predefined valgrind-memcheck mode for memory checker tool of Valgrind to be used.
UI: Added rich as a progress bar that can be used. Since it’s available via pip, it can likely be found and requires no inline copy. Added colors and similar behavior for tqdm as well.
UI: Remove obsolete warning for Linux with upx plugin.

We don’t use appimage anymore for a while now, so its constraints no longer apply.
UI: Add warnings for module specific options too. The logic to not warn on GitHub Actions was inverted, this restores warnings for normal users.
UI: Output the module name in question for options-nanny plugin and parameter warnings.
UI: When a forbidden import comes from an implicit import, report it properly.

Sometimes .pyi files from extension modules cause an import, but it was not clear which one; now it will indicate the module causing it.
UI: More clear error message in case a Python for scons was not found.
Actions: Cover debug mode compilation at least once.
Quality: Resolve paths from all OSes in --edit. Sometime I want to look at a file on a different OS, and there is no need to enforce being on the same one for path resolution to work.
Actions: Updated to a newer Ubuntu version for testing, as to get clang-format installed anymore.
Debugging: Allow for C stack output in signal handlers, this is most useful when doing the non-deployment handler that catches them to know where they came from more precisely.
UI: Show no-GIL in output of Python flavor in compilation if relevant.
License: Changed the license of remaining project files to AGPLv3 granting an exception for created binaries much like gcc does. As a result, going forward we need everybody to sign a CLA that allows re-licensing.

Tests

Removed Azure CI configuration, as testing has been fully migrated to GitHub Actions. (Changed in 2.7.9 already.)
Improved test robustness against short paths for package-containing directories. (Added in 2.7.4 already.)
Prevented test failures caused by rejected download prompts during test execution, making CI more stable. (Added in 2.7.4 already.)
Refactored common testing code to avoid using doctests, preventing warnings in specific standalone mode test scenarios related to reference counting. (Added in 2.7.4 already.)
Tests: Cover the memory leaking call re-formulation with a reference count test.

Cleanups

Plugins: Improved pkg_resources integration by using the __loader__ attribute of the registering module for loader type registration, avoiding modification of the global builtins dictionary. (Fixed in 2.7.2 already.)
Improved the logging mechanism for module search scans. It is now possible to enable tracing for individual locateModule calls, significantly enhancing readability and aiding debugging efforts.
Scons: Refactored architecture specific options into dedicated functions to improve code clarity.
Spelling: Various spelling and wording cleanups.
- Avoid using #ifdef in C code templates, and let’s just avoid it generally.
- Added missing slot function names to the ignored word list.
- Renamed variables related to slots to be more verbose and proper spelling as a result, as that’s for better understanding of their use anyway.
Scons: Specify versions supported for Scons by excluding the ones that are not, rather than manually maintaining a list. This adds automatic support for Python 3.14.
Plugins: Removed a useless call to intern as it did not have thought it does.
Attach copyright during code generation for code specializations
- This also enhances the formatting for almost all files by making leading and trailing new lines more consistent.
- One C file turns out unused and was removed as a left over from a previous refactoring.

Summary

This release was supposed to focus on scalability, but that didn’t happen again due to a variety of important issues coming up as well as a created downtime after high private difficulties after a planned surgery. However, the upcoming release will have it finally.

The onefile DLL mode as used on Windows has driven a lot of need for corrections, some of which are only in the final release, and this is probably the first time it should be usable for everything.

For compatibility, working with the popular (yet - not yes recommended UV-Python), Windows UI fixes for temporary onefile and macOS improvements, as well as improved Android support are excellent.

The next release of Nuitka however will have to focus on scalability and maintenance only. But as usual, not sure if it can happen.

Nuitka Release 2.7

2025-05-30T07:02:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release adds a ton of new features and corrections.

Bug Fixes

macOS: Correctly recognize self-dependencies of DLLs that include an architecture suffix on x86_64. (Fixed in 2.6.1 already.)
Standalone: Resolved an issue where .pyi files associated with versioned extension module filenames were not detected. (Fixed in 2.6.1 already.)
Standalone: Fixed .pyi file parsing failures caused by single-line triple quotes. (Fixed in 2.6.1 already.)
Standalone: Corrected the __spec__.origin path for packages where __init__ is an extension module.

While this primarily affected the module’s repr string, code relying on origin for resource location could previously encounter errors. (Fixed in 2.6.1 already.)
Multidist: Ensured created binaries use the name they were launched with, rather than the path of the actual binary file.

This allows entry points invoked via subprocess with different process names to function correctly from a single binary distribution. (Fixed in 2.6.1 already.)
Windows: Fixed unusable sys.stdin when attaching to a console (--windows-console-mode=attach) without an active terminal, which previously led to errors when forking processes. (Fixed in 2.6.1 already.)
Modules: Prevented crashes in module mode when encountering potentially optimizable importlib distribution calls that are non-optimizable in this mode. (Fixed in 2.6.1 already.)
Python3: Resolved errors with newer setuptools versions caused by namespace packages not providing a path_finder. (Fixed in 2.6.1 already.)
Windows: Removed a UTF-8 comment from a C header file that could prevent MSVC from compiling correctly in certain system locales. (Fixed in 2.6.2 already.)
Standalone: Ensured that user-inhibited data files have their associated tags cleared to prevent confusion in plugins. (Fixed in 2.6.2 already.)
Standalone: Corrected pkgutil.iter_modules to prevent incomplete module listings.

Previously, if a package directory existed, Python’s file finder could interfere, yielding incomplete results that excluded compiled modules. (Fixed in 2.6.2 already.)
Windows: Addressed a C compiler warning related to onefile compression on 32-bit Windows during bootstrap compilation.
Python3.12+: Resolved a SystemError when accessing the module attribute of type variables.
Metadata: Handled cases where reading record data from distribution files might fail because the files do not always exist. (Fixed in 2.6.3 already.)
Compatibility: Fixed dependency resolution for shared libraries that are symbolic links on non-Windows platforms.

Resolution failed if dependencies originated from the symlink target and that target was not in the system library path or nearby as another symlink. (Fixed in 2.6.3 already.)
Standalone: Prevented the use of msvcp140.dll from PySide6 (specifically shiboken6) by other packages to avoid potential compatibility crashes. (Fixed in 2.6.3 already.)
Standalone: Fixed .pyi file parsing failures caused by comments appearing after import statements. (Fixed in 2.6.3 already.)
UI: Corrected the output of DLL and EXE listings, which no longer provided the correct sub-folder information. (Fixed in 2.6.3 already.)
macOS: Restored the attachment of icons to application bundles. (Fixed in 2.6.3 already.)
Onefile: Resolved a C compiler warning on 32-bit Windows related to missing type conversion for the decompression buffer size. (Fixed in 2.6.3 already.)
Python3.12+: Ensured type aliases have a usable module attribute containing the correct module name. (Fixed in 2.6.3 already.)
Python3.12+: Improved compatibility for type alias values.

Corrected the creation process for compound type aliases, resolving errors in libraries like pydantic when used in schemas. (Fixed in 2.6.5 already.)
Python3.12+: Implemented a workaround for extension modules failing to set the correct package context, which previously caused significant compatibility issues (e.g., namespace collisions).

This was particularly problematic when static libpython was unavailable (common on Windows and official macOS CPython). The workaround involves saving/restoring the module context and correcting potentially wrongly named sub-modules created during extension module loading.

This improves compatibility for packages like PySide6QtAds, onnx, mediapipe, paddleocr, and newer scipy versions, which were previously affected. (Fixed in 2.6.6 already.)
Plugins: Prevented a crash in the Nuitka Package Configuration helper function get_data when falling back to pkgutil for data files not found naively. (Fixed in 2.6.6 already.)
Standalone: Corrected the rpath value used for finding dependent DLLs in sub-folders on non-Windows platforms.

It previously excluded the DLL’s own folder ($ORIGIN), sometimes preventing DLLs from loading. (Fixed in 2.6.7 already.)
Standalone: Preserved existing $ORIGIN-relative rpaths in DLLs on Linux.

Some PyPI packages rely on these existing paths to reference content in other packages; replacing them previously broke these setups. (Fixed in 2.6.7 already.)
Standalone: Now treats shared library dependencies specified with paths as implicit rpaths.

This fixes builds using Python distributions like Python Build Standalone (installed by uv), which may have an $ORIGIN-relative libpython dependency that needs to be respected. (Fixed in 2.6.7 already.)
Python3: Ensured generators preserve external exceptions and restore their own upon resuming.

This fixes issues where generators used as context managers, handling an exception via yield, could prevent an outer with statement from correctly re-raising its own exception. (Fixed in 2.6.7 already.)
Android: Removed the Termux rpath value pointing into its installation from standalone builds.

While ineffective after APK packaging due to Android security, this value should not have been present. (Fixed in 2.6.7 already.)
Python Build Standalone: Added the rpath to libpython by default in all modes for Python Build Standalone distributions.

This resolves issues with uninstalled libpython on Linux. (Fixed in 2.6.8 already.)
Standalone: Resolved incompatibility with older Linux distributions caused by using newer patchelf options introduced in 2.6.8’s rpath changes. (Fixed in 2.6.9 already.)
Python3.9: Fixed errors in the spacy plugin when using older importlib.metadata versions. (Fixed in 2.6.8 already.)
Standalone: Prevented requests package imports from being incorrectly treated as sub-packages. (Fixed in 2.6.8 already.)
Distutils on macOS: Improved integration for scanned extension modules where determining the correct architecture can be difficult. (Fixed in 2.6.8 already.)
Windows: Defined dotnet as a dependency to ensure all UI features requiring it are properly enabled.
Scons: Ensured the correct link.exe executable is used for the MSVC backend, avoiding potential conflicts with linkers added to the PATH (e.g., by git). (Fixed in 2.6.9 already.)
Scons: Avoided using config.txt with clcache when using MSVC.

This prevents potential race conditions during the first use where multiple clcache threads might attempt to create the file simultaneously. (Fixed in 2.6.9 already.)
Standalone: Ensured extension modules are loaded during the create_module phase of the Nuitka loader for better compatibility.

Loading them later during exec_module caused issues with some extension modules, such as those created by mypy (used in black).
Python3.13: Corrected the workaround for extension module package context issues, resolving errors that occurred when the module and package names were identical.
Module: Prevented stub generation attempts for namespace packages, which previously resulted in warnings as there is no source code to process.
Debian: Ensured consistent casing for the installer name used in Debian package metadata.
Poetry: Updated detection logic for newer poetry versions to handle changes in installer name casing, which could previously impact system DLL usage determination.
Module: Improved stub generation (stubgen) for generics, handling of missing typing imports, and other cases.
Plugins: Fixed potential corruption and crashes in the dill-compat plugin when handling functions with keyword defaults.
Standalone: Added support for newer py-cpuinfo versions on non-Windows platforms.
Accelerated: Prevented Nuitka’s sys.path_hook from overriding standard Python path loader hooks, as it doesn’t yet support all their functionalities.
Python3.12.7+: Set additional unicode immortal attributes (including for non-attributes) to prevent triggering Python core assertions when enabled.
Compatibility: Ensured errors are properly fetched during class variable lookups.

Previously, an error exit could occur without an exception being set, leading to crashes when attempting to attach tracebacks.
Python3.13: Adapted dictionary value creation and copying to follow internal layout changes, preventing potential crashes and corruption caused by using obsolete Python 3.11/3.12 code.
Scons: Corrected the default LTO module count calculation to refer to the number of compiled modules.
Package: Ensured namespace parent modules are included in compiled packages.

These were previously missed due to the removal of reliance on --include-package for delayed namespace package handling.
macOS: Ensured data files included in application bundles are also signed.
Windows: Applied short path conversion to the directory part of sys.argv[0] as well.

This prevents issues with tools called using this path that might not handle non-shortened (potentially unicode) paths correctly.

Package Support

Standalone: Included necessary data files for the blib2to3 package. (Added in 2.6.1 already.)
Standalone: Added support for newer numba versions. (Added in 2.6.2 already.)
Standalone: Added support for newer huggingface_hub versions. (Added in 2.6.2 already.)
Anti-Bloat: Provided additional numpy.testing stubs required for proper execution of some sklearn modules. (Fixed in 2.6.2 already.)
Standalone: Enhanced configuration for fontTools. Avoided configuring hidden dependencies now detected by parsing provided Python files like .pyi files. (Fixed in 2.6.2 already.)
Standalone: Corrected plugin configuration for newer PySide6 sqldrivers on macOS. (Fixed in 2.6.3 already.)
Python3.12+: Introduced standalone support for mediapipe, including a workaround for extension module sub-module creation issues. (Fixed in 2.6.3 already.)
Python3.12+: Introduced standalone support for onnx, including a workaround for extension module sub-module creation issues. (Fixed in 2.6.3 already.)
Standalone: Added support for newer sqlglot versions. (Added in 2.6.5 already.)
Standalone: Included asset data files for the arcade package. (Added in 2.6.5 already.)
Standalone: Added implicit dependencies for sqlalchemy.orm. (Added in 2.6.5 already.)
macOS: Included additional frameworks required for PySide 6.8 web-engine support. (Added in 2.6.5 already.)
Standalone: Enhanced cv2 support to handle potentially Python minor version-specific config files by allowing optional data file discovery in plugins. (Added in 2.6.6 already.)
Standalone: Added support for the scipy sub-module loader mechanism.

By treating it as a lazy loader, implicit dependencies within scipy are now correctly detected without requiring explicit configuration. (Added in 2.6.7 already.)
Standalone: Automatically include Django database engine modules. (Added in 2.6.7 already.)
Homebrew: Added tk-inter support for Python versions using Tcl/Tk version 9.
Standalone: Included a missing data file for the jenn package.
Standalone: Added support for newer scipy.optimize._cobyla versions. (Fixed in 2.6.8 already.)
Anaconda: Fixed issues with bare mkl usage (without numpy).
Standalone: Included a missing data file for the cyclonedx package.
Compatibility: Enabled pickling of local compiled functions using cloudpickle and ray.cloudpickle.
Standalone: Added support for mitmproxy on macOS.
Standalone: Included necessary data files for python-docs and mne.
Standalone: Added support for newer toga versions, requiring handling of its lazy loader.
Standalone: Introduced support for the black code formatter package.
Standalone: Included metadata when the travertino package is used.
Standalone: Significantly enhanced support for detecting dependencies derived from django settings.
Standalone: Added support for the duckdb package.

New Features

DLL Mode: Introduced a new experimental mode (--mode=dll) to create standalone DLL distributions.

While functional for many cases, documentation is currently limited, and features like multiprocessing require further work involving interaction with the launching binary.

This mode is intended to improve Windows GUI compatibility (tray icons, notifications) for onefile applications by utilizing an internal DLL structure.
Windows: Onefile mode now internally uses the new DLL mode by default, interacting with a DLL instead of an executable in temporary mode.

Use --onefile-no-dll to revert to the previous behavior if issues arise.
Windows: Added support for dependency analysis on Windows ARM builds using pefile (as Dependency Walker lacks ARM support).
Android: Enabled module mode support when using Termux Python. (Added in 2.6.7 already.)
Compatibility: Added support for Python Build Standalone distributions (e.g., as downloaded by uv).

Note that static libpython is not supported with these distributions as the included static library is currently unusable. (Added in 2.6.7 already.)
Windows: Enabled taskbar grouping for compiled applications if product and company names are provided in the version information. (Added in 2.6.4 already.)
Windows: Automatically use icons provided via --windows-icon-from-ico for PySide6 applications.

This eliminates the need to separately provide the icon as a PNG file, avoiding duplication.
Nuitka Package Configuration: Allowed using values from constants and variable declarations within when conditions where feasible.
Reports: Clearly indicate if an included package is “vendored” (e.g., packages bundled within setuptools).
Compatibility: Added support for the safe_path (-P) Python flag, preventing the use of the current directory in module searches.
Compatibility: Added support for the dont_write_bytecode (-B) Python flag, disabling the writing of .pyc files at runtime (primarily for debugging purposes, as compiled code doesn’t generate them).
UI: Introduced a new experimental tool for scanning distribution metadata, producing output similar to pip list -v. Intended for debugging metadata scan results.
Plugins: Enhanced the dill-compat plugin to transfer __annotations__ and __qualname__.

Added an option to control whether the plugin should also handle cloudpickle and ray.cloudpickle.
AIX: Implemented initial enhancements towards enabling Nuitka usage on AIX, although further work is required.

Optimization

Optimized finalizer handling in compiled generators, coroutines, and asyncgens by avoiding slower C API calls introduced in 2.6, restoring performance for these objects.
Implemented a more compact encoding for empty strings in data blobs.

Instead of 2 bytes (unicode) + null terminator, a dedicated type indicator reduces this frequent value to a single byte.

Anti-Bloat

Avoided including matplotlib when used by the tqdm package. (Added in 2.6.2 already.)
Avoided including matplotlib when used by the scipy package. (Added in 2.6.2 already.)
Avoided including cython when used by the fontTools package. (Added in 2.6.2 already.)
Avoided including sparse when used by the scipy package. (Added in 2.6.3 already.)
Avoided including ndonnx when used by the scipy package. (Added in 2.6.3 already.)
Avoided including setuptools for the jaxlib package.

Also prevented attempts to query the version from jaxlib source code using git. (Added in 2.6.3 already.)
Avoided including yaml when used by the scipy package. (Added in 2.6.4 already.)
Avoided including charset_normalizer for the numpy package. (Added in 2.6.5 already.)
Avoided including lxml for the pandas package. (Added in 2.6.5 already.)
Avoided including PIL (Pillow) for the sklearn package. (Added in 2.6.5 already.)
Avoided including numba when used by the smt package. (Added in 2.6.7 already.)
Avoided including more optional pygame dependencies. (Added in 2.6.8 already.)
Avoided including setuptools, tomli, and tomllib for the incremental package.
Avoided including IPython when used by the rich package vendored within pip.
For reporting purposes, treated usage of ipywidgets as equivalent to using IPython.
Added support for assert_raises within Nuitka’s numpy.testing stub.

Organizational

UI: Improved the output format for used command line options.

Filenames provided as positional arguments now use the report path format. Info traces now support an optional leader for intended value output, enhancing readability.
Reports: Saved and restored timing information for cached modules.

This eliminates timing differences based on whether a module was loaded from cache, reducing noise in Nuitka-Watch comparisons where cached module timings previously changed with every new compilation.
Actions: Added compilation report artifacts to all CI runs involving empty module compilations.
Debugging: Enabled the --edit option to find modules within .app bundles created on macOS. (Added in 2.6.1 already.)
User Manual: Updated the Nuitka-Action example; linking directly to its documentation might be preferable. (Changed in 2.6.1 already.)
Quality: Enforced ASCII encoding for all Nuitka C files to prevent accidental inclusion of non-ASCII characters.
Quality: Added syntax validation for global_replacements result values, similar to existing checks for replacements.

Also added validation to ensure global_replacements_re and replacements_re result values are valid regular expressions.
Plugins: Ensured error messages for illegal module names in implicit imports correctly report the originating plugin name.
Quality: Enabled use of clang-format-21 if available and applied formatting changes specific to this newer version.
Quality: Suppressed pylint warnings related to setuptools usage when running with Python 3.12+.
UI: Disallowed mixed usage of Anaconda and Poetry without an active Poetry virtual environment.

This avoids issues caused by a Poetry bug where it incorrectly sets the INSTALLER metadata for Conda packages in this scenario, making reliable detection of Conda packages impossible.
macOS: Deprecated --macos-target-arch in favor of the standard --target-arch option, with plans for eventual removal.
Release: Ensured usage of a compatible setuptools version during osc uploads.
UI: Improved the error message for invalid custom anti-bloat modes by listing the allowed values.
Release: Removed CPython test git submodules from the repository.

These submodules caused issues, such as being cloned during pip install and sometimes failing, potentially breaking Nuitka installation.

Tests

Enabled passing extra options via the NUITKA_EXTRA_OPTIONS environment variable for distutils test cases involving pyproject.toml.
Removed the standalone test for the gi package, as it’s better covered by Nuitka-Watch and prone to failures in CI due to lack of an X11 display.
Ensured tests correctly ignore the current directory when necessary by using the new --python-flag=safe_path.

This forces the use of the original source code as intended, rather than potentially finding modules in the current directory.
Corrected the implementation of the retry mechanism for pipenv installation within the nuitka-watch tool.
Added support for passing extra options via environment variables to the nuitka-watch tool.

Cleanups

Distutils: Standardized usage to --module=package where appropriate, instead of manually adding package contents, resulting in more conventional Nuitka command lines.
Refactored .pyi file creation into a dedicated function, simplifying the post-processing code.

Summary

This release was supposed to focus on scalability, but that didn’t happen due to a variety of important issues coming up as well as unplanned private difficulties.

The added DLL mode will be very interesting to many users, but needs more polish in future releases.

The next release of Nuitka however will have to focus on scalability and maintenance only. But as usual, not sure if it can happen.

Nuitka this week #16

2025-02-15T00:00:00+00:00

Hey Nuitka users! This started out as an idea of a weekly update, but that hasn’t happened, and so we will switch it over to just writing up when something interesting happens and then push it out relatively immediately when it happens.

Nuitka Onefile Gets More Flexible: `--onefile-cache-mode` and `{PROGRAM_DIR}`

We’ve got a couple of exciting updates to Nuitka’s onefile mode that give you more control and flexibility in how you deploy your applications. These enhancements stem from real-world needs and demonstrate Nuitka’s commitment to providing powerful and adaptable solutions.

Taking Control of Onefile Unpacking: `--onefile-cache-mode`

Onefile mode is fantastic for creating single-file executables, but the management of the unpacking directory where the application expands has sometimes been a bit… opaque. Previously, Nuitka would decide whether to clean up this directory based on whether the path used runtime-dependent variables. This made sense in theory, but in practice, it could lead to unexpected behavior and made debugging onefile issues harder.

Now, you have complete control! The new --onefile-cache-mode option lets you explicitly specify what happens to the unpacking directory:

--onefile-cache-mode=auto: This is the default behavior. Nuitka will remove the unpacking directory unless runtime-dependent values were used in the path specification. This is the same behavior as previous versions.
--onefile-cache-mode=cached: The unpacking directory is not removed and becomes a persistent, cached directory. This is useful for debugging, inspecting the unpacked files, or if you have a use case that benefits from persistent caching of the unpacked data. The files will remain available for subsequent runs.
--onefile-cache-mode=temporary: The unpacking directory is removed after the program exits.

This gives you the power to choose the behavior that best suits your needs. No more guessing!

Relative Paths with `{PROGRAM_DIR}`

Another common request, particularly from users deploying applications in more restricted environments, was the ability to specify the onefile unpacking directory relative to the executable itself. Previously, you were limited to absolute paths or paths relative to the user’s temporary directory space.

We’ve introduced a new variable, {PROGRAM_DIR}, that you can use in the --onefile-tempdir-spec option. This variable is dynamically replaced at runtime with the full path to the directory containing the onefile executable.

For example:

nuitka --onefile --onefile-tempdir-spec="{PROGRAM_DIR}/.myapp_data" my_program.py

This would create a directory named .myapp_data inside the same directory as the my_program.exe (or my_program on Linux/macOS) and unpack the application there. This is perfect for creating truly self-contained applications where all data and temporary files reside alongside the executable.

Nuitka Commercial and Open Source

These features, like many enhancements to Nuitka, originated from a request by a Nuitka commercial customer. This highlights the close relationship between the commercial offerings and the open-source core. While commercial support helps drive development and ensures the long-term sustainability of Nuitka, the vast majority of features are made freely available to all users.

Give it a Try!

This change will be in 2.7 and is currently

We encourage you to try out these new features and let us know what you think! As always, bug reports, feature requests, and contributions are welcome on GitHub.

Nuitka Release 2.6

2025-02-13T00:00:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has all-around improvements, with a lot effort spent on bug fixes in the memory leak domain, and preparatory actions for scalability improvements.

Bug Fixes

MSYS2: Path normalization to native Windows format was required in more places for the MinGW variant of MSYS2.

The os.path.normpath function doesn’t normalize to native Win32 paths with MSYS2, instead using forward slashes. This required manual normalization in additional areas. (Fixed in 2.5.1)
UI: Fix, give a proper error when extension modules asked to include failed to be located. instead of a proper error message. (Fixed in 2.5.1)
Fix, files with illegal module names (containing .) in their basename were incorrectly considered as potential sub-modules for --include-package. These are now skipped. (Fixed in 2.5.1)
Stubgen: Improved stability by preventing crashes when stubgen encounters code it cannot handle. Exceptions from it are now ignored. (Fixed in 2.5.1)
Stubgen: Addressed a crash that occurred when encountering assignments to non-variables. (Fixed in 2.5.1)
Python 3: Fixed a regression introduced in 2.5 release that could lead to segmentation faults in exception handling for generators. (Fixed in 2.5.2)
Python 3.11+: Corrected an issue where dictionary copies of large split directories could become corrupted. This primarily affected instance dictionaries, which are created as copies until updated, potentially causing problems when adding new keys. (Fixed in 2.5.2)
Python 3.11+: Removed the assumption that module dictionaries always contain only strings as keys. Some modules, like Foundation on macOS, use non-string keys. (Fixed in 2.5.2)
Deployment: Ensured that the --deployment option correctly affects the C compilation process. Previously, only individual disables were applied. (Fixed in 2.5.2)
Compatibility: Fixed a crash that could occur during compilation when unary operations were used within binary operations. (Fixed in 2.5.3)
Onefile: Corrected the handling of __compiled__.original_argv0, which could lead to crashes. (Fixed in 2.5.4)
Compatibility: Resolved a segmentation fault occurring at runtime when calling tensorflow.function with only keyword arguments. (Fixed in 2.5.5)
macOS: Harmless warnings generated for x64 DLLs on arm64 with newer macOS versions are now ignored. (Fixed in 2.5.5)
Python 3.13: Addressed a crash in Nuitka’s dictionary code that occurred when copying dictionaries due to internal changes in Python 3.13. (Fixed in 2.5.6)
macOS: Improved onefile mode signing by applying --macos-signed-app-name to the signature of binaries, not just app bundles. (Fixed in 2.5.6)
Standalone: Corrected an issue where too many paths were added as extra directories from the Nuitka package configuration. This primarily affected the win32com package, which currently relies on the package-dirs import hack. (Fixed in 2.5.6)
Python 2: Prevented crashes on macOS when creating onefile bundles with Python 2 by handling negative CRC32 values. This issue may have affected other versions as well. (Fixed in 2.5.6)
Plugins: Restored the functionality of code provided in pre-import-code, which was no longer being applied due to a regression. (Fixed in 2.5.6)
macOS: Suppressed the app bundle mode recommendation when it is already in use. (Fixed in 2.5.6)
macOS: Corrected path normalization when the output directory argument includes “~”.
macOS: GitHub Actions Python is now correctly identified as a Homebrew Python to ensure proper DLL resolution. (Fixed in 2.5.7)
Compatibility: Fixed a reference leak that could occur with values sent to generator objects. Asyncgen and coroutines were not affected. (Fixed in 2.5.7)
Standalone: The --include-package scan now correctly handles cases where both a package init file and competing Python files exist, preventing compile-time conflicts. (Fixed in 2.5.7)
Modules: Resolved an issue where handling string constants in modules created for Python 3.12 could trigger assertions, and modules created with 3.12.7 or newer failed to load on older Python 3.12 versions when compiled with Nuitka 2.5.5-2.5.6. (Fixed in 2.5.7)
Python 3.10+: Corrected the tuple code used when calling certain method descriptors. This issue primarily affected a Python 2 assertion, which was not impacted in practice. (Fixed in 2.5.7)
Python 3.13: Updated resource readers to accept multiple arguments for importlib.resources.read_text, and correctly handle encoding and errors as keyword-only arguments.
Scons: The platform encoding is no longer used to decode ccache logs. Instead, latin1 is used, as it is sufficient for matching filenames across log lines and avoids potential encoding errors. (Fixed in 2.5.7)
Python 3.12+: Requests to statically link libraries for hacl are now ignored, as these libraries do not exist. (Fixed in 2.5.7)
Compatibility: Fixed a memory leak affecting the results of functions called via specs. This primarily impacted overloaded hard import operations. (Fixed in 2.5.7)
Standalone: When multiple distributions for a package are found, the one with the most accurate file matching is now selected. This improves handling of cases where an older version of a package (e.g., python-opencv) is overwritten with a different variant (e.g., python-opencv-headless), ensuring the correct version is used for Nuitka package configuration and reporting. (Fixed in 2.5.8)
Python 2: Prevented a potential crash during onefile initialization on Python 2 by passing the directory name directly from the onefile bootstrap, avoiding the use of os.dirname which may not be fully loaded at that point. (Fixed in 2.5.8)
Anaconda: Preserved necessary PATH environment variables on Windows for packages that require loading DLLs from those locations. Only PATH entries not pointing inside the installation prefix are removed. (Fixed in 2.5.8)
Anaconda: Corrected the is_conda_package check to function properly when distribution names and package names differ. (Fixed in 2.5.8)
Anaconda: Improved package name resolution for Anaconda distributions by checking conda metadata when file metadata is unavailable through the usual methods. (Fixed in 2.5.8)
MSYS2: Normalized the downloaded gcc path to use native Windows slashes, preventing potential compilation failures. (Fixed in 2.5.9)
Python 3.13: Restored static libpython functionality on Linux by adapting to a signature change in an unexposed API. (Fixed in 2.5.9)
Python 3.6+: Prevented asyncgen from being resurrected when a finalizer is attached, resolving memory leaks that could occur with asyncio in the presence of exceptions. (Fixed in 2.5.10)
UI: Suppressed the gcc download prompt that could appear during --version output on Windows systems without MSVC or with an improperly installed gcc.
Ensured compatibility with monkey patched os.lstat or os.stat functions, which are used in some testing scenarios.
Data Composer: Improved the determinism of the JSON statistics output by sorting keys, enabling reliable build comparisons.
Python 3.6+: Fixed a memory leak in asyncgen with finalizers, which could lead to significant memory consumption when using asyncio and encountering exceptions.
Scons: Optimized empty generators (an optimization result) to avoid generating unused context code, eliminating C compilation warnings.
Python 3.6+: Fixed a reference leak affecting the asend value in asyncgen. While typically None, this could lead to observable reference leaks in certain cases.
Python 3.5+: Improved handling of coroutine and asyncgen resurrection, preventing memory leaks with asyncio and asyncgen, and ensuring correct execution of finally code in coroutines.
Python 3: Corrected the handling of generator objects resurrecting during deallocation. While not explicitly demonstrated, this addresses potential issues similar to those encountered with coroutines, particularly for old-style coroutines created with the types.coroutine decorator.
PGO: Fixed a potential crash during runtime trace collection by ensuring timely initialization of the output mechanism.

Package Support

Standalone: Added inclusion of metadata for jupyter_client to support its own usage of metadata. (Added in 2.5.1)
Standalone: Added support for the llama_cpp package. (Added in 2.5.1)
Standalone: Added support for the litellm package. (Added in 2.5.2)
Standalone: Added support for the lab_lamma package. (Added in 2.5.2)
Standalone: Added support for docling metadata. (Added in 2.5.5)
Standalone: Added support for pypdfium on Linux. (Added in 2.5.5)
Standalone: Added support for using the debian package. (Added in 2.5.5)
Standalone: Added support for the pdfminer package. (Added in 2.5.5)
Standalone: Included missing dependencies for the torch._dynamo.polyfills package. (Added in 2.5.6)
Standalone: Added support for rtree on Linux. The previous static configuration only worked on Windows and macOS; this update detects it from the module code. (Added in 2.5.6)
Standalone: Added missing pywebview JavaScript data files. (Added in 2.5.7)
Standalone: Added support for newer versions of the sklearn package. (Added in 2.5.7)
Standalone: Added support for newer versions of the dask package. (Added in 2.5.7)
Standalone: Added support for newer versions of the transformers package. (Added in 2.5.7)
Windows: Placed numpy DLLs at the top level for improved support in the Nuitka VM. (Added in 2.5.7)
Standalone: Allowed excluding browsers when including playwright. (Added in 2.5.7)
Standalone: Added support for newer versions of the sqlfluff package. (Added in 2.5.8)
Standalone: Added support for the opencv conda package, disabling unnecessary workarounds for its dependencies. (Added in 2.5.8)
Standalone: Added support for newer versions of the soundfile package.
Standalone: Added support for newer versions of the coincurve package.
Standalone: Added support for newer versions of the apscheduler package.
macOS: Removed the error and workaround forcing that required bundle mode for PyQt5 on macOS, as standalone mode now appears to function correctly.
Standalone: Added support for seleniumbase package downloads.

New Features

Module: Implemented 2-phase loading for all modules in Python 3.5 and higher. This improves loading modules as sub-packages in Python 3.12+, where the loading context is no longer accessible.
UI: Introduced the app value for the --mode parameter. This creates an app bundle on macOS and a onefile binary on other platforms, replacing the --macos-create-app-bundle option. (Added in 2.5.5)
UI: Added a package mode, similar to module, which automatically includes all sub-modules of a package without requiring manual specification with --include-package.
Module: Added an option to completely disable the use of stubgen. (Added in 2.5.1)
Homebrew: Added support for tcl9 with the tk-inter plugin.
Package Resolution: Improved handling of multiple distributions installed for the same package name. Nuitka now attempts to identify the most recently installed distribution, enabling proper recognition of different versions in scenarios like python-opencv and python-opencv-headless.
Python 3.13.1 Compatibility: Addressed an issue where a workaround introduced for Python 3.10.0 broke standalone mode in Python 3.13.1. (Added in 2.5.6)
Plugins: Introduced a new feature for absolute source paths (typically derived from variables or relative to constants). This offers greater flexibility compared to the by_code DLL feature, which may be removed in the future. (Added in 2.5.6)
Plugins: Added support for when conditions in variable sections within Nuitka Package configuration.
macOS: App bundles now automatically switch to the containing directory when not launched from the command line. This prevents the current directory from defaulting to /, which is rarely correct and can be unexpected for users. (Added in 2.5.6)
Compatibility: Relaxed the restriction on setting the compiled frame f_trace. Instead of outright rejection, the deployment flag --no-deployment-flag=frame-useless-set-trace can be used to allow it, although it will be ignored.
Windows: Added the ability to detect extension module entry points using an inline copy of pefile. This enables --list-package-dlls to verify extension module validity on the platform. It also opens possibilities for automatic extension module detection on major operating systems.
Watch: Added support for using conda packages instead of PyPI packages.
UI: Introduced --list-package-exe to complement --list-package-dlls for package analysis when creating Nuitka Package Configuration.
Windows ARM: Removed workarounds that are no longer necessary for compilation. While the lack of dependency analysis might require correction in a hotfix, this configuration should now be supported.

Optimization

Scalability: Implemented experimental code for more compact code object usage, leading to more scalable C code and constants usage. This is expected to speed up C compilation and code generation in the future once fully validated.
Scons: Added support for C23 embedding of the constants blob. This will be utilized with Clang 19+ and GCC 15+, except on Windows and macOS where other methods are currently employed.
Compilation: Improved performance by avoiding redundant path checks in cases of duplicated package directories. This significantly speeds up certain scenarios where file system access is slow.
Scons: Enhanced detection of static libpython, including for self-compiled, uninstalled Python installations.

Anti-Bloat

Improved no_docstrings support for the xgboost package. (Added in 2.5.7)
Avoided unnecessary usage of numpy for the PIL package.
Avoided unnecessary usage of yaml for the numpy package.
Excluded tcltest TCL code when using tk-inter, as these TCL files are unused.
Avoided using IPython from the comm package.
Avoided using pytest from the pdbp package.

Organizational

UI: Added categories for plugins in the --help output. Non-package support plugin options are now shown by default. Introduced a dedicated --help-plugins option and highlighted it in the general --help output. This allows viewing all plugin options without needing to enable a specific plugin.
UI: Improved warnings for onefile and OS-specific options. These warnings are now displayed unless the command originates from a Nuitka-Action context, where users typically build for different modes with a single configuration set.
Nuitka-Action: The default mode is now app, building an application bundle on macOS and a onefile binary on other platforms.
UI: The executable path in --version output now uses the report path. This avoids exposing the user’s home directory, encouraging more complete output sharing.
UI: The Python flavor name is now included in the startup compilation message.
UI: Improved handling of missing Windows version information. If only partial version information (e.g., product or file version) is provided, an explicit error is given instead of an assertion error during post-processing.
UI: Corrected an issue where the container argument for run-inside-nuitka-container could not be a non-template file. (Fixed in 2.5.2)
Release: The PyPI upload sdist creation now uses a virtual environment. This ensures consistent project name casing, as it is determined by the setuptools version. While currently using the deprecated filename format, this change prepares for the new format.
Release: The osc binary is now used from the virtual environment to avoid potential issues with a broken system installation, as currently observed on Ubuntu.
Debugging: Added an experimental option to disable the automatic conversion to short paths on Windows.
UI: Improved handling of external data files that overwrite the original file. Nuitka now prompts the user to provide an output directory to prevent unintended overwrites. (Added in 2.5.6)
UI: Introduced the alias --include-data-files-external for the external data files option. This clarifies that the feature is not specific to onefile mode and encourages its wider use.
UI: Allowed none as a valid value for the macOS icon option. This disables the warning about a missing icon when intentionally not providing one.
UI: Added an error check for icon filenames without suffixes, preventing cases where the file type cannot be inferred.
UI: Corrected the examples for --include-package-data with file patterns, which used incorrect delimiters.
Scons: Added a warning about using gcc with LTO when make is unavailable, as this combination will not work. This provides a clearer message than the standard gcc warnings, which can be difficult for Python users to interpret.
Debugging: Added an option to preserve printing during reference count tests. This can be helpful for debugging by providing additional trace information.
Debugging: Added a small code snippet for module reference leak testing to the Developer Manual.

Tests

Temporarily disabled tests that expose regressions in Python 3.13.1 that mean not to follow.
Improved test organization by using more common code for package tests. The scanning for test cases and main files now utilizes shared code.
Added support for testing variations of a test with different extra flags. This is achieved by exposing a NUITKA_TEST_VARIANT environment variable.
Improved detection of commercial-only test cases by identifying them through their names rather than hardcoding them in the runner. These tests are now removed from the standard distribution to reduce clutter.
Utilized --mode options in tests for better control and clarity. Standalone mode tests now explicitly check for the application of the mode and error out if it’s missing. Mode options are added to the project options of each test case instead of requiring global configuration.
Added a test case to ensure comprehensive coverage of external data file usage in onefile mode. This helps detect regressions that may have gone unnoticed previously.
Increased test coverage for coroutines and async generators, including checks for inspect.isawaitable and testing both function and context objects.

Cleanups

Unified the code used for generating source archives for PyPI uploads, ensuring consistency between production and standard archives.
Harmonized the usage of include <...> vs include "..." based on the origin of the included files, improving code style consistency.
Removed code duplication in the exception handler generator code by utilizing the DROP_GENERATOR_EXCEPTION functions.
Updated Python version checks to reflect current compatibility. Checks for >=3.4 were changed to >=3, and outdated references to Python 3.3 in comments were updated to simply “Python 3”.
Scons: Simplified and streamlined the code for the command options. An OrderedDict is now used to ensure more stable build outputs and prevent unnecessary differences in recorded output.
Improved the executeToolChecked function by adding an argument to indicate whether decoding of returned bytes output to unicode is desired. This eliminates redundant decoding in many places.

Summary

This a major release that it consolidates Nuitka big time.

The scalability work has progressed, even if no immediately visible effects are there yet, the next releases will have them, as this is the main area of improvement these days.

The memory leaks found are very important and very old, this is the first time that asyncio should be working perfect with Nuitka, it was usable before, but compatibility is now much higher.

Also, this release puts out a much nicer help output and handling of plugins help, which no longer needs tricks to see a plugin option that is not enabled (yet), during --help. The user interface is hopefully more clean due to it.

Nuitka Release 2.5

2024-11-27T00:00:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release focused on Python 3.13 support, but also on improved compatibility, made many performance optimizations, enhanced error reporting, and better debugging support.

Bug Fixes

Windows: Fixed a regression in onefile mode that incorrectly handled program and command line paths containing spaces. Fixed in 2.4.4 already.
Windows: Corrected an issue where console output handles were being opened with closed file handles. Fixed in 2.4.2 already.
Standalone: Restored the ability to use trailing slashes on the command line to specify the target directory for data files on Windows. Fixed in 2.4.2 already.
Compatibility: Fixed a parsing error that occurred with relative imports in .pyi files, which could affect some extension modules with available source code. Fixed in 2.4.3 already.
Modules: Ensured that extension modules load correctly into packages when using Python 3.12. Fixed in 2.4.4 already.
Windows: Improved command line handling for onefile mode to ensure full compatibility with quoting. Fixed in 2.4.4 already.
Data Directories: Allowed the use of non-normalized paths on the command line when specifying data directories. Fixed in 2.4.5 already.
Python 3.11+: Fixed an issue where inspect module functions could raise StopIteration when examining compiled functions on the stack. Fixed in 2.4.5 already.
importlib_metadata: Improved compatibility with importlib_metadata by handling cases where it might be broken, preventing potential compilation crashes. Fixed in 2.4.5 already.
Plugins: Fixed a crash that occurred when using the no_asserts YAML configuration option. Fixed in 2.4.6 already.
Scons: Improved error tolerance when reading ccache log files to prevent crashes on Windows caused by non-ASCII module names or paths. Fixed in 2.4.11 already.
Scons: Prevented the C standard option from being applied to C++ compilers, resolving an issue with the splash screen on Windows when using Clang. Fixed in 2.4.8 already.
macOS: Enhanced handling of DLL self-dependencies to accommodate cases where DLLs use both .so and .dylib extensions for self-references. Fixed in 2.4.8 already.
Compatibility: Fixed a memory leak that occurred when using deepcopy on compiled methods. Fixed in 2.4.9 already.
MSYS2: Excluded the bin directory from being considered a system DLL folder when determining DLL inclusion. Fixed in 2.4.9 already.
Python 3.10+: Fixed a crash that could occur when a match statement failed to match a class with arguments. Fixed in 2.4.9 already.
MSYS2: Implemented a workaround for non-normalized paths returned by os.path.normpath in MSYS2 Python environments. Fixed in 2.4.11 already.
Python 3.12: Resolved an issue where Nuitka’s constant code was triggering assertions in Python 3.12.7. Fixed in 2.4.10 already.
UI: Ensured that the --include-package option includes both Python modules and extension modules that are sub-modules of the specified package. Fixed in 2.4.11 already.
Windows: Prevented encoding issues with CMD files used for accelerated mode on Windows.
Standalone: Improved the standard library scan to avoid assuming the presence of specific files, which might have been deleted by the user or a Python distribution.
Compatibility: Added suffix, suffixes, and stem attributes to Nuitka resource readers to improve compatibility with file objects.
Compatibility: Backported the error message change for yield from used at the module level, using dynamic detection instead of hardcoded text per version.
Compatibility: Fixed an issue where calling built-in functions with keyword-only arguments could result in errors due to incorrect argument passing.
Compatibility: Fixed reference leaks that occurred when using list.insert and list.index with 2 or 3 arguments.
Windows: Prioritized relative paths over absolute paths for the result executable when absolute paths are not file system encodable. This helps address issues related to non-ASCII short paths on some Chinese systems.
Compatibility: Improved compatibility with C extensions by handling cases where the attribute slot is not properly implemented, preventing potential segfaults.
Compatibility: Prevent the leakage of sys.frozen when using the multiprocessing module and its plugin, resolving a long-standing TODO and potentially breaking compatibility with packages that relied on this behavior.
Compatibility: Fixed an issue where matching calls with keyword-only arguments could lead to incorrect optimization and argument passing errors.
Compatibility: Corrected the handling of iterators in for loops to avoid assuming the presence of slots, preventing potential issues.
macOS: Added support for cyclic DLL dependencies, where DLLs have circular references.
Compatibility: Ensured the use of updated expressions during optimization phase for side effects to prevent crashes caused by referencing obsolete information.
Python 3.10+: Fixed a crash that could occur in complex cases when re-formulating match statements.
Python 3.4-3.5: Corrected an issue in Nuitka’s custom PyDict_Next implementation that could lead to incorrect results in older Python 3 versions.
Python 3.10+: Ensured that AttributeError is raised with the correct keyword arguments, avoiding a TypeError that occurred previously.
Plugins: Added a data file function that avoids loading packages, preventing potential crashes caused by incompatible dependencies (e.g., numpy versions).
Compatibility: Ensured that Nuitka’s package reader closes data files after reading them to prevent resource warnings in certain Python configurations.
Standalone: Exposed setuptools contained vendor packages in standalone distributions to match the behavior of the setuptools package.
Accelerated Mode: Enabled the django module parameter in accelerated mode to correctly detect used extensions.
Compatibility: Prevented resource warnings for unclosed files when trace outputs are sent to files via command line options.
Compatibility: Enabled the use of xmlrpc.server without requiring the pydoc module.
Plugins: Fixed an issue in the anti-bloat configuration where change_function and change_classes ignored “when” clauses, leading to unintended changes.
Python 3.12 (Linux): Enhanced static libpython handling for Linux. Static libpython is now used only when the inline copy is available (not in official Debian packages). The inline copy of hacl is used for all Linux static libpython uses with Python 3.12 or higher.
Standalone: Further improved the standard library scan to avoid assuming the presence of files that might have been manually deleted.
UI: Fixed the --include-raw-dir option, which was not functioning correctly. Only the Nuitka Package configuration was being used previously.

Package Support

arcade: Improved standalone configuration for the arcade package. Added in 2.4.3 already.
license-expression: Added a missing data file for the license-expression package in standalone distributions. Added in 2.4.6 already.
pydantic: Included a missing implicit dependency required for deprecated decorators in the pydantic package to function correctly in standalone mode. Fixed in 2.4.5 already.
spacy: Added a missing implicit dependency for the spacy package in standalone distributions. Added in 2.4.7 already.
trio: Updated standalone support for newer versions of the trio package. Added in 2.4.8 already.
tensorflow: Updated standalone support for newer versions of the tensorflow package. Added in 2.4.8 already.
pygame-ce: Added standalone support for the pygame-ce package. Added in 2.4.8 already.
toga: Added standalone support for newer versions of the toga package on Windows. Added in 2.4.9 already.
django: Implemented a workaround for a django debug feature that attempted to extract column numbers from compiled frames. Added in 2.4.9 already.
PySide6: Improved standalone support for PySide6 on macOS by allowing the recognition of potentially unusable plugins. Added in 2.4.9 already.
polars: Added a missing dependency for the polars package in standalone distributions. Added in 2.4.9 already.
django: Enhanced handling of cases where the django settings module parameter is absent in standalone distributions. Added in 2.4.9 already.
win32ctypes: Included missing implicit dependencies for win32ctypes modules on Windows in standalone distributions. Added in 2.4.9 already.
arcade: Added a missing data file for the arcade package in standalone distributions. Added in 2.4.9 already.
PySide6: Allowed PySide6 extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed. Added in 2.4.11 already.
driverless-selenium: Added standalone support for the driverless-selenium package. Added in 2.4.11 already.
tkinterdnd2: Updated standalone support for newer versions of the tkinterdnd2 package. Added in 2.4.11 already.
kivymd: Updated standalone support for newer versions of the kivymd package. Added in 2.4.11 already.
gssapi: Added standalone support for the gssapi package. Added in 2.4.11 already.
azure.cognitiveservices.speech: Added standalone support for the azure.cognitiveservices.speech package on macOS.
mne: Added standalone support for the mne package.
fastapi: Added a missing dependency for the fastapi package in standalone distributions.
pyav: Updated standalone support for newer versions of the pyav package.
py_mini_racer: Added standalone support for the py_mini_racer package.
keras: Improved standalone support for keras by extending its sub-modules path to include the keras.api sub-package.
transformers: Updated standalone support for newer versions of the transformers package.
win32com.server.register: Updated standalone support for newer versions of the win32com.server.register package.
Python 3.12+: Added support for distutils in setuptools for Python 3.12 and later.
cv2: Enabled automatic scanning of missing implicit imports for the cv2 package in standalone distributions.
lttbc: Added standalone support for the lttbc package.
win32file: Added a missing dependency for the win32file package in standalone distributions.
kivy: Fixed an issue where the kivy clipboard was not working on Linux due to missing dependencies in standalone distributions.
paddleocr: Added missing data files for the paddleocr package in standalone distributions.
playwright: Added standalone support for the playwright package with a new plugin.
PySide6: Allowed PySide6 extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed.

New Features

Python 3.13: Added experimental support for Python 3.13.

Warning

Python 3.13 support is not yet recommended for production use due to limited testing. On Windows, only MSVC and ClangCL are currently supported due to workarounds needed for incompatible structure layouts.
UI: Introduced a new --mode selector to replace the options --standalone, --onefile, --module, and --macos-create-app-bundle.

Note

The app mode creates an app bundle on macOS and a onefile binary on other operating systems to provide the best deployment option for each platform.
Windows: Added a new hide choice for the --windows-console-mode option. This generates a console program that hides the console window as soon as possible, although it may still briefly flash.
UI: Added the --python-flag=-B option to disable the use of bytecode cache (.pyc) files during imports. This is mainly relevant for accelerated mode and dynamic imports in non-isolated standalone mode.
Modules: Enabled the generation of type stubs (.pyi files) for compiled modules using an inline copy of stubgen. This provides more accurate and informative type hints for compiled code.

Note

Nuitka also adds implicit imports to compiled extension modules, ensuring that dependencies are not hidden.
Plugins: Changed the data files configuration to a list of items, allowing the use of when conditions for more flexible control. Done in 2.4.6 already.
Onefile: Removed the MSVC requirement for the splash screen in onefile mode. It now works with MinGW64, Clang, and ClangCL. Done for 2.4.8 already.
Reports: Added information about the file system encoding used during compilation to help debug encoding issues.
Windows: Improved the attach mode for --windows-console-mode when forced redirects are used.
Distutils: Added the ability to disable Nuitka in pyproject.toml builds using the build_with_nuitka setting. This allows falling back to the standard build backend without modifying code or configuration. This setting can also be passed on the command line using --config-setting.
Distutils: Added support for commercial file embedding in distutils packages.
Linux: Added support for using uninstalled self-compiled Python installations on Linux.
Plugins: Enabled the matplotlib plugin to react to active Qt and tkinter plugins for backend selection.
Runtime: Added a new original_argv0 attribute to the __compiled__ value to provide access to the original start value of sys.argv[0], which might be needed by applications when Nuitka modifies it to an absolute path.
Reports: Added a list of DLLs that are actively excluded because they are located outside of the PyPI package.
Plugins: Allowed plugins to override the compilation mode for standard library modules when necessary.

Optimization

Performance: Implemented experimental support for “dual types”, which can significantly speed up integer operations in specific cases (achieving speedups of 12x or more in some very specific loops). This feature is still under development but shows promising potential for future performance gains, esp. when combined with future PGO (Profile Guided Optimization) work revealing likely runtime types more often and more types being covered.
Performance: Improved the speed of module variable access.
- For Python 3.6 to 3.10, this optimization utilizes dictionary version tags but may be less effective when module variables are frequently written to.
- For Python 3.11+, it relies on dictionary key versions, making it less susceptible to dictionary changes but potentially slightly slower for cache hits compared to Python 3.10.
Performance: Accelerated string dictionary lookups for Python 3.11+ by leveraging knowledge about the key and the module dictionary’s likely structure. This also resolves a previous TODO item, where initial 3.11 support was not as fast as our support for 3.10 was in this domain.
Performance: Optimized module dictionary updates to occur only when values actually change, improving caching efficiency.
Performance: Enhanced exception handling by removing bloat in the abstracted differences between Python 3.12 and earlier versions. This simplifies the generated C code, reduces conversions, and improves efficiency for all Python versions. This affects both C compile time and runtime performance favorably and solves a huge TODO for Python 3.12 performance.
Performance: Removed the use of CPython APIs calls for accessing exception context and cause values, which can be slow.
Performance: Utilized Nuitka’s own faster methods for creating int and long values, avoiding slower CPython API calls.
Performance: Implemented a custom variant of _PyGen_FetchStopIterationValue to avoid CPython API calls in generator handling, further improving performance on generators, coroutines and asyncgen.
Windows: Aligned with CPython’s change in reference counting implementation on Windows for Python 3.12+, which improves performance with LTO (Link Time Optimization) enabled.
Optimization: Expanded static optimization to include unary operations, improving the handling of number operations and preparing for full support of dual types.
Optimization: Added static optimization for os.stat and os.lstat calls.
Performance: Passed the exception state directly into unpacking functions, eliminating redundant exception fetching and improving code efficiency.
Performance: Introduced a dedicated helper for unpacking length checks, resulting in faster and more compact code helping scalability as well.
Performance: Generated more efficient code for raising built-in exceptions by directly creating them through the base exception’s new method instead of calling them as functions. This can speed up some things by a lot.
Performance: Optimized exception creation by avoiding unnecessary tuple allocations for empty exceptions. This hack avoids hitting the memory allocator as much.
Performance: Replaced remaining uses of PyTuple_Pack with Nuitka’s own helpers to avoid CPython API calls.
Code Generation: Replaced implicit exception raise nodes with direct exception creation nodes for improved C code generation.
Windows: Aligned with CPython’s change in managing object reference counters on Windows for Python 3.12+, improving performance with LTO enabled.
Performance: Removed remaining CPython API calls when creating int values in various parts of the code, including specialization code, helpers, and constants loading.
Windows: Avoided scanning for DLLs in the PATH environment variable when they are not intended to be used from the system. This prevents potential crashes related to non-encodable DLL paths and makes those scans faster too.
Windows: Updated to a newer MinGW64 version from 13.2 to 14.2 for potentially improved binary code generation with that compiler.
Code Size: Reduced the size of constant blobs by avoiding module-level constants for the global values -1, 0, and 1.
Code Generation: Improved code generation for variables by directly placing NameError exceptions into the thread state when raised, making for more compact C code.
Optimization: Statically optimized the sys.ps1 and sys.ps2 values to not exist (unless in module mode), potentially enabling more static optimization in packages that detect interactive usage checking them.
Performance: Limited the use of tqdm locking to no-GIL and Scons builds where threading is actively used.
Optimization: Implemented a faster check for non-frame statement sequences by decoupling frames and normal statement sequences and using dedicated accessors. This improves performance during the optimization phase.

Anti-Bloat

Prevented the inclusion of importlib_metadata for the numpy package. Added in 2.4.2 already.
Avoided the use of dask in the pandera package. Added in 2.4.5 already.
Removed numba for newer versions of the shap package. Added in 2.4.6 already.
Prevented attempts to include both Python 2 and Python 3 code for the aenum package, avoiding SyntaxError warnings. Added in 2.4.7 already.
Enhanced handling for the sympy package. Added in 2.4.7 already.
Allowed pydoc for the pyqtgraph package. Added in 2.4.7 already.
Avoided pytest in the time_machine package. Added in 2.4.9 already.
Avoided pytest in the anyio package.
Avoided numba in the pandas package.
Updated anti-bloat measures for newer versions of the torch package with increased coverage.
Avoided pygame.tests and cv2 for the pygame package.
Allowed unittest in the absl.testing package.
Allowed setuptools in the tufup package.
Avoided test modules when using the bsdiff4 package.
Treated the use of the wheel module the same as using the setuptools package.

Organizational

Development Environment: Added experimental support for a devcontainer to the repository, providing an easier way to set up a Linux-based development environment. This feature is still under development and may require further refinement.
Issue Reporting: Clarified the issue reporting process on GitHub, emphasizing the importance of testing reproducers against Python first to ensure the issue is related to Nuitka.
Issue Reporting: Discouraged the use of --deployment in issue reports, as it hinders the automatic identification of issues, that should be the first thing to remove.
UI: Improved the clarity of help message of the option for marking data files as external, emphasizing that files must be included before being used.
UI: Added checks to the Qt plugins to ensure that specified plugin families exist, preventing unnoticed errors.
UI: Implemented heuristic detection of terminal link support, paving the way for adding links to options and groups in the command line interface.
UI: Removed obsolete caching-related options from the help output, as they have been replaced by more general options.
Plugins: Improved error messages when retrieving information from packages during compilation.
Quality: Implemented a workaround for an isort bug that prevented it from handling UTF-8 comments.
Quality: Updated GitHub actions to use clang-format-20.
Quality: Updated to the latest version of black for code formatting.
Release Process: Updated the release script tests for Debian and PyPI to use the correct runner names. (Changed in 2.4.1 already.
UI: Disabled progress bar locking, as Nuitka currently doesn’t utilize threads.
UI: Added heuristic detection of terminal link support and introduced an experimental terminal link as a first step towards a more interactive command line interface.
Debugging: Fixed a crash in the “explain reference counts” feature that could occur with unusual dict values mistaken for modules.
Debugging: Included reference counts of tracebacks when dumping reference counts at program end.
Debugging: Added assertions and traces to improve debugging of input/output handling.
Quality: Added checks for configuration module names in Nuitka package configuration to catch errors caused by using filenames instead of module names.
UI: Removed obsolete options controlling cache behavior, directing users to the more general cache options.
Scons: Ensured that the CC environment variable is used consistently for --version and onefile bootstrap builds, as well as the Python build, preventing inconsistencies in compiler usage and outputs.
Distutils: Added the compiled-package-hidden-by-package mnemonic for use in distutils to handle the expected warning when a Python package is replaced with a compiled package and the Python code is yet to be deleted.
Dependency Management: Started experimental support for downloading Nuitka dependencies like ordered-set. This feature is not yet ready for general use.

Tests

Added Python 3.13 to the GitHub Actions test matrix.
Significantly enhanced construct-based tests for clearer results. The new approach executes code with a boolean flag instead of generating different code, potentially leading to the removal of custom templating.
Removed the 2to3 conversion code from the test suite, as it is being removed from newer Python versions. Tests are now split with version requirements as needed.
Fixed an issue where the test runner did not discover and use Python 3.12+, resulting in insufficient test coverage for those versions on GitHub Actions.
Ensured that the compare_with_cpython test function defaults to executing the system’s Python interpreter instead of relying on the PYTHON environment variable.
Set up continuous integration with Azure Pipelines to run Nuitka tests against the factory branch on each commit.
Enforced the use of static libpython for construct-based tests to eliminate DLL call overhead and provide more accurate performance measurements.
Improved the robustness of many construct tests, making them less sensitive to unrelated optimization changes.
Removed a test that was only applicable to Nuitka Commercial, as it was not useful to always skip it in the standard version. Commercial tests are now also recognized by their names.
Added handling for segmentation faults in distutils test cases, providing debug output for easier diagnosis of these failures.
Prevented resource warnings for unclosed files in a reflected test.

Cleanups

WASI: Corrected the signatures of C function getters and setters for compiled types in WASI to ensure they match the calling conventions. Casts are now performed locally to the compiled types instead of in the function signature. Call entries also have the correct signature used by Python C code.
WASI: Improved code cleanliness by adhering to PyCFunction signatures in WASI.
Code Generation: Fixed a regression in code generation that caused misaligned indentation in some cases.
Code Formatting: Changed some code for identical formatting with clang-format-20 to eliminate differences between the new and old versions.
Caching: Enforced proper indentation in Nuitka cache files stored in JSON format.
Code Cleanliness: Replaced checks for Python 3.4 or higher with checks for Python 3, simplifying the code and reflecting the fact that Python 3.3 is no longer supported.
Code Cleanliness: Removed remaining Python 3.3 specific code from frame templates.
Code Cleanliness: Performed numerous spelling corrections and renamed internal helper functions for consistency and clarity.
Plugins: Renamed the get_module_directory helper function in the Nuitka Package configuration to remove the leading underscore, improving readability.
Plugins: Moved the numexpr.cpuinfo workaround to the appropriate location in the Nuitka Package configuration, resolving an old TODO item.

Summary

This a major release that brings support for Python 3.13, relatively soon after its release.

Our plugin system and Nuitka plugin configuration was used a lot for support of many more third-party packages, and numerous other enhancements in the domain of avoiding bloat.

This release focuses on improved compatibility, new break through performance optimizations, to build on in the future, enhanced error reporting, and better debugging support.

Nuitka Release 2.4

2024-07-30T09:09:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release largely contains bug fixes for the previous changes, but also finishes full compatibility with the match statements of 3.10, something that was long overdue since there were always some incompatible behaviors there.

In terms of bug fixes, it’s also huge. An upgrade is required, especially for new setuptools that made compiled programs segfault at startup.

Bug Fixes

UI: Fix, we had reversed disable / force and wrong option name recommendation for --windows-console-mode when the user used old-style options.
Python3.10+: Fix, must not check for len greater or equal of 0 or for sequence match cases. That is unnecessary and incompatible and can raise exceptions with custom sequences not implementing __len__. Fixed in 2.3.1 already.
Python3.10+: Fix, match sequence with final star arguments failed in some cases to capture the rest. The assigned value then was empty.when it shouldn’t have been. Fixed in 2.3.1 already.
Python3.8+: Fix, calls to variable args functions now need to be done differently, or else they can crash, as was observed with 3.10 in PGO instrumentation, at least. Fixed in 2.3.1 already.
PGO: Fix, using nuitka-run did not execute the program created as expected. Fixed in 2.3.1 already.
Linux: Support extension modules used as DLLs by other DLLs or extension modules. That makes newer tensorflow and potentially more packages work again. Fixed in 2.3.1 already.
Python3.10+: Matches classes were not fully compatible.

We need to check against case-defined class __match_args__, not the matched value type __match_args that is not necessarily the same.

Also, properly annotating the exception exit of subscript matches; the subscript value can indeed raise an exception.

Collect keyword and positional match values in one go and detect duplicate attributes used, which we previously did not.
Scons: Fix, do not crash when clang is not reporting its version correctly. It happened if Clang usage was required with --clang option but not installed. Fixed in 2.3.2 already.
Debian: Fix, detecting the Debian flavor of Python was not working anymore, and as a result, the intended defaults were no longer applied by Nuitka, leading to incorrect suggestions that didn’t work. Fixed in 2.3.3 already.
Ubuntu: Fix, the static link library for Python 3.12 is not usable unless we provide parts of HACL for the sha2 module so as not to cause link errors. Fixed in 2.3.3 already.
Standalone: Fix, importing newer pkg_resources was crashing. Fixed in 2.3.3 already.
Python3.11+: Added support for newer Python with dill-compat. Fixed in 2.3.4 already.
Standalone: Support locating Windows icons for pywebview. Fixed in 2.3.4 already.
Standalone: Added support for spacy related packages. Fixed in 2.3.4 already.
Python3.12: Fix, our workaround for cv2 support cannot use the imp module anymore. Fixed in 2.3.4 already.
Compatibility: Added support for __init__ files that are extension modules. Architecture checks for macOS were false negatives for them, and the case insensitive import scan failed to find them on Windows. Fixed in 2.3.4 already.
Standalone: Added missing dependencies for standard library extension modules, mainly exhibited on macOS. Fixed in 2.3.4 already.
Windows: Fix build failures on mapped network drives. Fixed in 2.3.4 already.
Python3.12: Fix, need to set frame prev_inst or else f_lasti is random. Some packages; for example PySide6; use this to check what bytecode calls them or how they import them and it could crash when attempting it. Fixed in 2.3.6 already.
Fix, fork bomb in cpuinfo package no longer happens. Fixed in 2.3.8 already.
Nuitka-Python: Fix, cannot ask for shared library prefixes. Fixed in 2.3.8 already.
Standalone: Make sure keras package dependency for tensorflow is visible. Fixed in 2.3.10 already.
Linux: Fix, for static executables we should ignore errors setting a DLL load path. Fixed in 2.3.10 already.
Compatibility: Fix, nuitka resource readers also need to have .parent attribute. Fixed in 2.3.10 already.
Fix, need to force no-locale language outputs for tools outputs on non-Windows. Our previous methods were not forcing enough.

For non-Windows this makes Nuitka work on systems with locales active for message outputs only. Fixed in 2.3.10 already.
Fix, was not using proper result value for SET_ATTRIBUTE to check success in a few corner cases. Fixed in 2.3.10 already.
Windows: Retry deleting dist and build folders, allowing users to recognize still running programs and not crashing on Anti-Virus software still locking parts of them.
Fix, dict.fromkeys didn’t give compatible error messages for no args given.
Fix, output correct unsupported exception messages for in-place operations

For in-place **, it was also incompatible, since it must not mention the pow function.
Fix, included metadata could lead to instable code generation. We were using a dictionary for it, but that is not as stable order for the C compiler to fully benefit.
Fix, including data files for packages that are extension modules was not working yet.
macOS: Detect the DLL path of libpython (if used) by looking at dependencies of the running Python binary rather than encoding what CPython does. Doing that covers other Python flavors as well.
Fix, need to prefer extension modules over Python code for packages.
Fix, immutable constant values are not to be treated as very trusted.
Python3: Fix, the __loader__ attribute of a module should be an object and not only the class, otherwise only static methods can work.
Python3: Added .name and .path attributes to Nuitka loader objects for enhanced compatibility with code that expects source code loaders.
Fix, the sys.argv[0] needs to be absolute for best usability.

For dirname(sys.argv[0]) to be usable even if the program is launched via PATH environment by a shell, we cannot rely on how we are launched since that won’t be a good path, unlike with Python interpreter, where it always is.
Standalone: Fix, adding missing dependencies for some crypto packages.
Python3.12: Need to write to thread local variable during import. This however doesn’t work for Windows and non-static libpython flavors in general.
macOS: Enforce using system codesign as the Anaconda one is not working for us.
Fix, we need to read .pyi files as source code. Otherwise unicode characters can cause crashes.
Standalone: Fix, some packages query private values for distribution objects, so use the same attribute name for the path.
Multidist: Make sure to follow the multidist reformulation modules. Otherwise in accelerated mode, these could end up not being included.
Fix, need to hold a reference of the iterable while converting it to list.
Plugins: Fix, this wasn’t properly ignoring None values in load descriptions as intended.
macOS: Need to allow DLLs from all Homebrew paths.
Reports: Do not crash during report writing for very early errors.
Python3.11+: Fix, need to make sure we have split as a constant value when using exception groups.
Debian: More robust against problematic distribution folders with no metadata, these apparently can happen with OS upgrades.
Fix, was leaking exception in case of --python-flag=-m mode that could cause errors.
Compatibility: Close standard file handles on process forks as CPython does. This should enhance things for compilations using attach on Windows.

Package Support

Standalone: Added data file for older bokeh version. Fixed in 2.3.1 already.
Standalone: Support older pandas versions as well.
Standalone: Added data files for panel package.
Standalone: Added support for the newer kivy version and added macOS support as well. Fixed in 2.3.4 already.
Standalone: Include all kivy.uix packages with kivy, so their typical config driven usage is not too hard.
Standalone: Added implicit dependencies of lxml.sax module. Fixed in 2.3.4 already.
Standalone: Added implicit dependencies for zeroconf package. Fixed in 2.3.4 already.
Standalone: Added support for numpy version 2. Fixed in 2.3.7 already.
Standalone: More complete support for tables package. Fixed in 2.3.8 already.
Standalone: Added implicit dependencies for scipy.signal package. Fixed in 2.3.8 already.
Standalone: Added support for moviepy and imageio_ffmeg packages. Fixed in 2.3.8 already.
Standalone: Added support for newer scipy. Fixed in 2.3.10 already.
Standalone: Added data files for bpy package. For full support more work will be needed.
Standalone: Added support for nes_py and gym_tetris packages.
Standalone: Added support for dash and plotly.
Standalone: Added support for usb1 package.
Standalone: Added support for azure.cognitiveservices.speech package.
Standalone: Added implicit dependencies for tinycudann package.
Standalone: Added support for newer win32com.server.register.
Standalone: Added support for jaxtyping package.
Standalone: Added support for open3d package.
Standalone: Added workaround for torch submodule import function.
Standalone: Added support for newer paddleocr.

New Features

Experimental support for Python 3.13 beta 3. We try to follow its release cycle closely and aim to support it at the time of CPython release. We also detect no-GIL Python and can make use of it. The GIL status is output in the --version format and the GIL usage is available as a new {GIL} variable for project options.
Scons: Added experimental option --experimental=force-system-scons to enforce system Scons to be used. That allows for the non-use of inline copy, which can be interesting for experiments with newer Scons releases. Added in 2.3.2 already.
Debugging: A new non-deployment handler helps when segmentation faults occurred. The crashing program then outputs a message pointing to a page with helpful information unless the deployment mode is active.
Begin merging changes for WASI support. Parts of the C changes were merged and for other parts, command line option --target=wasi was added, and we are starting to address cross platform compilation for it. More work will be necessary to fully merge it, right not it doesn’t work at all yet.
PGO: Added support for using it in standalone mode as well, so once we use it more, it will immediately be practical.
Make the --list-package-dlls use plugins as well, and make delvewheel and announce its DLL path internally, too. Listing DLLs for packages using plugins can use these paths for more complete outputs.
Plugins: The no-qt plugin was usable in accelerated mode.
Reports: Added included metadata and reasons for it.
Standalone: Added support for spacy with a new plugin.
Compatibility: Use existing source files as if they were .pyi files for extension modules. That gives us dependencies for code that installs source code and extension modules.
Plugins: Make version information, onefile mode, and onefile cached mode indication available in Nuitka Package Configuration, too.
Onefile: Warn about using tendo.singleton in non-cached onefile mode.

Tendo uses the running binary name for locking by default. So it’s not going to work if that changes for each execution, make the user aware of that, so they can use cached mode instead.
Reports: Include the micro pass counts and tracing merge statistics so we can see the impact of new optimization.
Plugins: Allow to specify modes in the Nuitka Package Configuration for annotations, doc_strings, and asserts. These overrule global configuration, which is often not practical. Some modules may require annotations, but for other packages, we will know they are fine without them. Simply disabling annotations globally barely works. For some modules, removing annotations can give a 30% compile-time speedup.
Standalone: Added module configuration for Django to find commands and load its engine.
Allow negative values for –jobs to be relative to the system core count so that you can tell Nuitka to use all but two cores with --jobs=-2 and need not hardcode your current code count.
Python3.12: Annotate libraries that are currently not supported

We will need to provide our own Python3.12 variant to make them work.
Python3.11+: Catch calls to uncompiled function objects with compiled code objects. We now raise a RuntimeError in the bytecode making it easier to catch them rather than segfaulting.

Optimization

Statically optimize constant subscripts of variables with immutable constant values.
Forward propagate very trusted values for variable references enabling a lot more optimization.
Python3.8+: Calls of C functions are faster and more compact code using vector calls, too.
Python3.10+: Mark our compiled types as immutable.
Python3.12: Constant returning functions are dealing with immortal values only. Makes their usage slightly faster since no reference count handling is needed.
Python3.10+: Faster attribute descriptor lookups. Have our own replacement of PyDesc_IsData that had become an API call, making it very slow on Windows specifically.
Avoid using Python API function for determining sequence sizes when getting a length size for list creations.
Data Composer: More compact and portable Python3 int (Python2 long) value representation.

Rather than fixed native length 8 or 4 bytes, we use variable length encoding which for small values uses only a single byte.

This also avoids using struct.pack with C types, as we might be doing cross platform, so this makes part of the WASI changes unnecessary at the same time.

Large values are also more compact because middle 31-bit portions can be less than 4 bytes and save space on average.
Data Composer: Store bytecode blob size more efficient and portable, too.
Prepare having knowledge of __prepare__ result to be dictionaries per compile time decisions.
Added more hard trust for the typing module.

The typing.Text is a constant too. In debug mode, we now check all exports of typing for constant values. This will allow to find missing values sooner in the future.

Added the other types to be known to exist. That should help scalability for types intensive code somewhat by removing error handling for them.
macOS: Should use static libpython with Anaconda as it works there too, and reduces issues with Python3.12 and extension module imports.
Standalone: Statically optimize by OS in sysconfig.

Consequently, standalone distributions can exclude OS-specific packages such as _aix_support and _osx_support.
Avoid changing code names for complex call helpers

The numbering of complex call helper as normally applied to all functions are, caused this issue. When part of the code is used from the bytecode cache, they never come to exist and the C code of modules using them then didn’t match.

This avoids an extra C re-compilation for some modules that were using renumbered function the second time around a compilation happens. Added in 2.3.10 already.
Avoid using C-API when creating __path__ value.
Faster indentation of generated code.

Anti-Bloat

Add new pydoc bloat mode to trigger warnings when using it.
Recognize usage of numpy.distutils as setuptools bloat for more direct reporting.
Avoid compiling large opcua modules that generate huge C files much like asyncua package. Added in 2.3.1 already.
Avoid shiboken2 and shiboken6 modules from matplotlib package when the no-qt plugin is used. Added in 2.3.6 already.
Changes for not using pydoc and distutils in numpy version 2. Added in 2.3.7 already.
Avoid numpy and packaging dependencies from PIL package.
Avoid using webbrowser module from pydoc.
Avoid using unittest in keras package. Added in 2.3.1 already.
Avoid distutils from _oxs_support (used by sysconfig) module on macOS.
Avoid using pydoc for werkzeug package. Fixed in 2.3.10 already.
Avoid using pydoc for site module. Fixed in 2.3.10 already.
Avoid pydoc from xmlrpc.server. Fixed in 2.3.10 already.
Added no_docstrings support for numpy2 as well. Fixed in 2.3.10 already.
Avoid pydoc in joblib.memory.
Avoid setuptools in gsplat package.
Avoid dask and jax in scipy package.
Avoid using matplotlib for networkx package.

Organizational

Python3.12: Added annotations of official support for Nuitka PyPI package and test runner options that were still missing. Fixed in 2.3.1 already.
UI: Change runner scripts. The nuitka3 is no more. Instead, we have nuitka2 where it applies. Also, we now use CMD files rather than batch files.
UI: Check filenames for data files for illegal paths on the respective platforms. Some user errors with data file options become more apparent this way.
UI: Check spec paths more for illegal paths as well. Also do not accept system paths like {TEMP} and no path separator after it.
UI: Handle report writing interrupt with CTRL-C more gracefully. No need to present this this as a general problem, rather inform the user that he did it.
NoGIL: Warn if using a no-GIL Python version, as this mode is not yet officially supported by Nuitka.
Added badges to the README.rst of Nuitka to display package support and more. Added in 2.3.1 already.
UI: Use the retry decorator when removing directories in general. It will be more thorough with properly annotated retries on Windows. For the dist folder, mention the running program as a probable cause.
Quality: Check replacements and replacements_plain Nuitka package configuration values.
Quality: Catch backlashes in paths provided in Nuitka Package Configuration values for dest_path, relative_path, dirs, raw_dirs and empty_dirs.
Debugging: Disable pagination in gdb with the --debugger option.
PGO: Warn if the PGO binary does not run successfully.
UI: The new console mode option is a Windows-specific option now, move it to that group.
UI: Detect “rye python” on macOS. Added in 2.3.8 already.
UI: Be forgiving about release candidates; Ubuntu shipped one in a LTS release. Changed in 2.3.8 already.
Debugging: Allow fine-grained debug control for immortal checks

Can use --no-debug-immortal-assumptions to allow for corrupted immortal objects, which might be done by non-Nuitka code and then break the debug mode.
UI: Avoid leaking compile time Nuitka environment variables to the child processes.

They were primarily visible with --run, but we should avoid it for everything.

For non-Windows, we now recognize if we are the exact re-execution and otherwise, reject them.
Watch: Delete the existing virtualenv in case of errors updating or upgrading it.
Watch: Keep track of Nuitka compiled program exit code in newly added result files, too.
Watch: Redo compilations in case of previous errors when executing the compile program.
Quality: Wasn’t detecting files to ignore for PyLint on Windows properly, also detect crashes of PyLint.

Tests

Added test to cover the dill-compat plugin.
macOS: Make actual use of ctypes in its standalone test to ensure correctness on that OS, too.
Make compile extension module test work on macOS, too.
Avoid using 2to3 in our tests since newer Python no longer contains it by default, we split up tests with mixed contents into two tests instead.
Python3.11+: Make large constants test executable for as well. We no longer can easily create those values on the fly and output them due to security enhancements.
Python3.3: Remove support from the test runner as well.
Tests: Added construct-based tests for coroutines so we can compare their performance as well.

Cleanups

Make try/finally variable releases through common code. It will allow us to apply special exception value trace handling for only those for scalability improvements, while also making many re-formulations simpler.
Avoid using anti-bloat configuration values replacements where replacements_plain is good enough. A lot of config pre-date its addition.
Avoid Python3 and Python3.5+ specific Jinja2 modules on versions before that, and consequently, avoid warning about the SyntaxError given.
Moved code object extraction of dill-compat plugin from Python module template to C code helper for shared usage and better editing.
Also call va_end for standards compliance when using va_start. Some C compilers may need that, so we better do it even if what we have seen so far doesn’t need it.
Don’t pass main filename to the tree building anymore, and make nuitka.Options functions usage explicit when importing.
Change comments that still mentioned Python 3.3 as where a change in Python happened since we no longer support this version. Now, we consider what’s first seen in Python 3.4 is a Python3 change.
Cleanup, change Python 3.4 checks to 3.0 checks as Python3.3 is no longer supported. Cleans up version checks, as we now treat >=3.4 either as >=3 or can drop checks entirely.
The usual flow of spelling cleanups, this time for C codes.

Summary

This release cycle was a longer than usual, with much new optimization and package support requiring attention.

For optimization we got quite a few things going, esp. with more forward propagation, but the big ones for scalability are still all queued up and things are only prepared.

The 3.13 work was continuing smoothly and seems to be doing fine. We are still on track for supporting it right after release.

The parts where we try and address WASI prepare cross-compilation, but we will not aim at it generally immediately, and target our own Nuitka standalone backend Python that is supposed to be added in coming releases.

Nuitka Release 2.3

2024-06-12T00:00:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release bumps the long-awaited 3.12 support to a complete level. Now, Nuitka behaves identically to CPython 3.12 for the most part.

In terms of bug fixes, it’s also huge. Especially for Unicode paths and software with Unicode extension module names and Unicode program names, and even non-UTF8 code names, there have been massive amounts of improvements.

Bug Fixes

Standalone: Added support for python-magic-bin package. Fixed in 2.2.1 already.
Fix: The cache directory creation could fail when multiple compilations started simultaneously. Fixed in 2.2.1 already.
macOS: For arm64 builds, DLLs can also have an architecture dependent suffix; check that as well. Makes the soundfile dependency scan work. Fixed in 2.2.1 already.
Fix: Modules where lazy loaders handling adds hard imports when a module is first processed did not affect the current module, potentially causing it not to resolve hidden imports. Fixed in 2.2.1 already.
macOS: The use of libomp in numba needs to cause the extension module not to be included and not to look elsewhere. Fixed in 2.2.1 already.
Python3.6+: Fix, added support for keyword arguments of ModuleNotFoundError. Fixed in 2.2.1 already.
macOS: Detect more versioned DLLs and arm64 specific filenames. Fixed in 2.2.1 already.
Fix, was not annotating exception exit when converting an import to a hard submodule import. Fixed in 2.2.2 already.
Fix, branches that became empty can still have traces that need to be merged.

Otherwise, usages outside the branch will not see propagated assignment statements. As a result, these falsely became unassigned instead. Fixed in 2.2.2 already.
Windows: Fix, uninstalled self-compiled Python didn’t have proper installation prefix added for DLL scan, resulting in runtime DLLs not picked up from there. Fixed in 2.2.2 already.
Standalone: Added support for newer PySide6 version 6.7. It needed correction on macOS and has a new data file type. Fixed in 2.2.3 already.
Standalone: Complete support for pyocd package. Fixed in 2.2.3 already.
Module: Fix, the created .pyi files were incomplete.

The list of imported modules created in the finalization step was incomplete, we now go over the actual done modules and mark all non-included modules as dependencies.
Scons: Fix, need to avoid using Unicode paths towards the linker on Windows. Instead, use a temporary output filename and rename it to the actual filename after Scons has completed.
Windows: Avoid passing Unicode paths to the dependency walker on Windows, as it cannot handle those. Also, the temporary filenames in the build folder must be in short paths, as it cannot handle them in case that is a Unicode path.
Scons: For ccache on Windows, the log filename must be a short path too, if the build folder is a Unicode path.
Windows: Make sure the Scons build executes inside a short path as well, so that a potential Unicode path is visible to the C compiler when resolving the current directory.
Windows: The encoding of Unicode paths for accelerated mode values of __file__ was not making sure that hex sequences were correctly terminated, so in some cases, it produced ambiguous C literals.
Windows: Execute binaries created with --windows-uac-admin with and --run options with proper UAC prompt.
Fix, need to allow for non-UTF8 Unicode in variable names, function names, class names, and method names.

Python3.10+: Fix, match statements that captured the rest of mapping checks were not working yet.

match value:
   case {"key1": 5, **rest}:
      ... # rest was not assigned here

Windows: When deleting build folders, make sure the retries leading to a complete deletion always.
Python2: Fix, could crash with non-unicode program paths on Windows.
Avoid giving SyntaxWarning from reading source code

For example, the standard site module of Python 3.12 gives warnings about illegal escape sequences that nobody cares about apparently.
Fix, the matplotlib warnings by options-nanny were still given even if the no-qt plugin was used, since the variable name referenced there was not actually set yet by that plugin.
Windows: Fix, when using the uninstalled self-compiled Python, we need python.exe to find DLL dependencies. Otherwise it doesn’t locate the MSVC runtime and Python DLL properly.
Standalone: Added support for freetype package.

New Features

Support for Python 3.12 is finally there. We focused on scalability first and because we did things the correct way immediately, rather than rushing to get it working and improving only later.

As a result, the correctness and performance of Nuitka with previous Python releases are improved as well.

Some things got delayed, though. We need to do more work to take advantage of other core changes. Concerning exceptions normalized at creation time, the created module code doesn’t yet take advantage. Also, more efficient two-digit long handling is possible with Python 3.12, but not implemented. It will take more time before we have these changes completed.
Experimental support for Python 3.13 beta 1 is also there, and potentially surprising, but we will try and follow its release cycle closely and aim to support it at the time of release.

Nuitka has followed all of its core changes so far, and basic tests are passing; the accelerated, module, standalone, and onefile modes all work as expected. The only thing delayed is the uncompiled generator integration, where we need to replicate the exact CPython behavior. We need to have perfect integration only for working with the asyncio loop, so we wait with it until release candidates appear.
Plugins: Added support to include directories entirely unchanged by adding raw_dir values for data-files section, see .
UI: The new command line option --include-raw-dir was added to allow including directories entirely unchanged.
Module: Added support for creating modules with Unicode names. Needs a different DLL entry function name and to make use of two-phase initialization for the created extension module.
Added support for OpenBSD standalone mode.

Optimization

Python3: Avoid API calls for allocators

Most effective with Python 3.11 or higher but also many other types like bytes, dict keys, float, and list objects are faster to create with all Python3 versions.
Python3.5+: Directly use the Python allocator functions for object creation, avoiding the DLL API calls. The coverage is complete with Python3.11 or higher, but many object types like float, dict, list, bytes benefit even before that version.
Python3: Faster creation of StopIteration objects.

With Python 3.12, the object is created directly and set as the current exception without normalization checks.

We also added a new specialized function to create the exception object and populate it directly, avoiding the overhead of calling of the StopIteration type.
Python3.10+: When accessing freelists, we were not passing for tstate but locally getting the interpreter object, which can be slower by a few percent in some configurations. We now use the free lists more efficient with tuple, list, and dict objects.
Python3.8+: Call uncompiled functions via vector calls.

We avoid an API call that ends up being slower than using the same function via the vector call directly.
Python3.4+: Avoid using _PyObject_LengthHint API calls in list.extend and have our variant that is faster to call.
Added specialization for os.path.normpath. We might benefit from compile time analysis of it once we want to detect file accesses.
Avoid using module constants accessor for global constant values

For example, with (), we used the module-level accessor for no reason, as it is already available as a global value. As a result, constant blobs shrink, and the compiled code becomes slightly smaller , too.
Anti-Bloat: Avoid using dask from the sparse module. Added in 2.2.2 already.

Organizational

UI: Major change in console handling.

Compiled programs on Windows now have a third mode, besides console or not. You can now create GUI applications that attach to an available console and output there.

The new option --console controls this and allows to enforce console with the force value and disable using it with the disable value, the attach value activates the new behavior.

Note

Redirection of outputs to a file in attach mode only works if it is launched correctly, for example, interactively in a shell, but some forms of invocation will not work; prominently, subprocess.call without inheritable outputs will still output to a terminal.

On macOS, the distinction doesn’t exist anymore; technically it wasn’t valid for a while already; you need to use bundles for non-console applications, though, by default otherwise a console is forced by macOS itself.
Detect patchelf usage in buggy version 0.18.0 and ask the user to upgrade or downgrade it, as this specific version is known to be broken.
UI: Make clear that the --nofollow-import-to option accepts patters.
UI: Added warning for module mode and usage of the options to force outputs as they don’t have any effect.
UI: Check the success of Scons in creating the expected binary immediately after running it and not only once we reach post-processing.
UI: Detect empty user package configuration files
UI: Do not output module ast when a plugin reports an error for the module, for example, a forbidden import.
Actions: Update from deprecated action versions to the latest versions.

Tests

Use Nuitka Project Options for the user plugin test rather than passing by environment variables to the test runner.
Added a new search mode, skip, to complement resume which resumes right after the last test resume stopped on. We can use that while support for a Python version is not complete.

Cleanups

Solved a TODO about using unified code for setting the StopIteration, coroutines, generators, and asyncgen used to be different.
Unified how the binary result filename is passed to Scons for modules and executables to use the same result_exe key.

Summary

This release marks a huge step in catching up with compatibility of Python. After being late with 3.12 support, we will now be early with 3.13 support if all goes well.

The many Unicode support related changes also enhanced Nuitka to generate 2 phase loading extension modules, which also will be needed for sub-interpreter support later on.

From here on, we need to re-visit compatibility. A few more obscured 3.10 features are missing, the 3.11 compatibility is not yet complete, and we need to take advantage of the new caching possibilities to enhance performance for example with attribute lookups to where it can be with the core changes there.

For the coming releases until 3.13 is released, we hope to focus on scalability a lot more and get a much needed big improvement there, and complete these other tasks on the side.

Nuitka Release 2.2

2024-04-12T11:37:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release focused on compatibility and significant optimization progress for loops, such as list operations within. The main line of change is to be able to support Python 3.12 in the next release.

Bug Fixes

Standalone: Added support for pypdfium2 package. Fixed in 2.1.1 already.
Standalone: Make cefpython3 work on Linux. Fixed in 2.1.1 already.
ArchLinux: Added platform linker option to be usable with their current Arch Python package. Fixed in 2.1.1 already.
Fix, ctypes.CDLL optimization used a misspelled argument name for use_last_error, such that keyword argument calls were statically optimized into TypeError at compile-time. Fixed in 2.1.1 already.
Fix, list.insert was not properly annotating exceptions. Raises by producing the inserted value raised or the index was not annotated and, therefore, could fail to be caught locally. Fixed in 2.1.1 already.
Standalone: Added support for selenium package. Fixed in 2.1.2 already.
Standalone: Added support for hydra package. Fixed in 2.1.2 already.
Standalone: Updated dotenv workaround for newer version. Fixed in 2.1.3 already.
Fix, PySide6 slots failed to be moved between threads. For that we need to make function renames visible in the owning class as well. Fixed in 2.1.3 already.
Standalone: Added support for win32com.server.register. Fixed in 2.1.3 already.
Standalone: Handle “string” import errors of uvicorn gracefully. Fixed in 2.1.3 already.
Fix, the dill-compat plugin needs also needs to expose the compiled type names as built-ins for the pickle module to find them.
Standalone: Added support for gruut package. Fixed in 2.1.3 already.
Standalone: Added support for newer toga to also include toga_winforms metadata. Fixed in 2.1.3 already.
Standalone: Added support for newer tensorflow package. Fixed in 2.1.4 already.
Standalone: Fix, matplotlib needs to emit a dependency on the backend to be included. Otherwise it could be missing at run-time in some cases. Fixed in 2.1.4 already.
Onefile: Respect XDG_CACHE_HOME variable on non-Windows platforms. Some users might configure that to not to be ~/.cache, respect that. Fixed in 2.1.4 already.
Python2: Some cases of list.insert were not properly handling all index types. Fixed in 2.1.4 already.
Fix, optimized list.remove failed to handle tuple arguments properly. Removing tuple values from lists could cause errors. Fixed in 2.1.4 already.
Standalone: Added missing implicit dependencies for pyarrow.datasets. Fixed in 2.1.4 already.
Standalone: Added support for dask.dataframe module. Fixed in 2.1.4 already.
Standalone: Added DLLs for tensorrt_libs package. Fixed in 2.1.4 already.
Standalone: Added missing metadata of numpy for xarray package. Fixed in 2.1.4 already.
Standalone: Added support for newer scipy. Fixed in 2.1.5 already.
Standalone: Fix, older gcc could give warning about C code to work with PYTHONPATH which caused build errors on older systems. Fixed in 2.1.5 already.
Fix, locals representing nodes could not be cloned, and as a result, some code re-formulations failed to compile in try constructs. Fixed in 2.1.5 already.
Standalone: Added data files for names package. Fixed in 2.1.5 already.
Standalone: Added data files for randomname package. Fixed in 2.1.5 already.
Standalone: Fix, the standalone standard library scan was not fully ignoring git folders, subfolders were still looked at and could cause issues. Fixed in 2.1.5 already.
Standalone: Added support for newer transformers. Fixed in 2.1.5 already.
Standalone: Add support for newer bitsandbytes. Fixed in 2.1.5 already.
Scons: Fix, when locating binaries, do not use directories but only files.

A directory on PATH that was named gcc could be mistaken to be a gcc binary causing errors. Fixed in 2.1.6 already.
Windows: Fix, by default, scan only for .bin and .exe binaries for Nuitka Package Configuration EXE dependency patterns. This was the intended value, but it had not taken effect yet. Fixed in 2.1.6 already.
Fix, the __compiled__.containing_dir should be an absolute path. For it to be usable after a change of directory is done by the program that is required. Fixed in 2.1.6 already.
Standalone: Added support for more parts of networkx package. Fixed in 2.1.6 already.
Windows: Fix working with UNC paths and re-parse points at compile time.

Now Nuitka should work with mapped and even unmapped to drive paths like \\some-hostname\unc-test as they are common in some VM setups.
Windows: Make sure the download path is an external use path in scons as well, otherwise the home directory could be an unusable path for MinGW64, causing it not to find files.
Standalone: Added missing dependency of sspilib that prevented requests-ntlm from working on Windows.
Python3.5+: Add support for using dictionary un-packings in class declarations. That is a rarely used in actual Python code, but was found missing by tests recently.
Python3.11: Fix, code objects co_qualname attribute was not actually the qualified name, but the same as co_name only.
Anaconda: Fix, must not consider the Anaconda lib directory as a system directory, because then those DLLs that are not included.
Fix, cannot trust dynamic hard modules as much otherwise, huggingface_hub.utils.tqdm ended up being a module and not the class it’s supposed to be.
macOS: Fix, on newer macOS the libc++ and libz DLLs cannot be found anymore, we need to ignore that actively as our code insists on full resolution to catch bugs.
Fix, support for newer zaber_motion was not really working.
Standalone: Added required data files for pyviz_comms.
Standalone: Added required data files for panel package.
Standalone: Added required data files for bokeh package.
Standalone: Fixup scipy for Anaconda.
Fix, need to make parent module usages more explicit.

Otherwise, plugin mechanisms like no-follow from a parent module cannot affect its child modules, as they can end up being followed to only after them.
Fix, the dill-compat plugin in module mode cannot assume the main module name to be the one from compile time, need to look the actual one up at runtime.

New Features

Added experimental support for Python 3.12, this is passing basic tests, but known to crash a lot at run-time still, you are recommended to use pre-releases of Nuitka, as official support is not going to happen before 2.3 release.
Standalone: Added support for tensorflow.function JIT

With preserved source code of decorated functions and we can provide it at run-time to tensorflow JIT so it can do its tracing executions.
For Nuitka Package Configuration, we now have change_class similar to change_function to replace a full class definition with something else, this can be used to modify classes to become stubs or even unusable.
For the experimental @pyqtSlot decorator, we also should handle the @asyncSlot the same way. Added in 2.1.1 already.
Added new kind of warning of plugin category and use it in the Nuitka Package Configuration to inform matplotlib users to select a GUI backend via plugin selection. Added in 2.1.4 already.
Zig: Added support for zig as CC value. Due to it not supporting C11 fully yet, we need to use the C++ workaround and cannot compile for Python 3.11 or higher yet.
For the __compiled__ value, we now have a __compiled__.main that is the name of the compiled module. For modules, Nuitka determines this at run time; in other modes, it is the name of the main module.

Optimization

Use set specific API in contains tests, rather than generic sequence one.
Lower value in something tests for known set and list values to use frozenset and tuple respectively.
Recognize exact type shapes of loop variables where possible. This enables appends to list to be optimized to their dedicated nodes among other things, with those often being a lot faster than generic code. This speeds up e.g. list append tests by a significant amount.
Optimization: Have dedicated helper for list.remove, such that it is not using a Python DLL call where that is slow.
ArchLinux: Enable static libpython by default, it is usable indeed. Added in 2.1.2 already.
Anti-Bloat: Avoid unittest usage in antlr package.
Anti-Bloat: Avoid IPython in celery package. Added in 2.1.2 already.
Anti-Bloat: Avoid using setuptools in transformers package for more modules. Added in 2.1.3 already.
Anti-Bloat: Avoid testing packages for newer tensorflow package as well. Added in 2.1.4 already.
Optimization: Avoid recompiling azure package which is not performance relevant. Added in 2.1.4 already.
Avoid packages owned by Nuitka plugins in matploblib backends unless the corresponding plugin is actually active. Added in 2.1.4 already.
Anti-Bloat: Avoid setuptools in deepspeed package. Added in 2.1.4 already.
Anti-Bloat: Avoid setuptools in transformers package. Added in 2.1.4 already.
Anti-Bloat: Avoid scipy usage causing torch or cupy usage. Added in 2.1.4 already.
Anti-Bloat: Recognize keras testing modules as unittest bloat.
Faster code generation due to enhancements in how identifiers are cached for module names and the indentation codes.
Optimization: Handle no_docstrings issue for torio package.
Anti-Bloat: Avoid IPython from imgui_bundle package.
Anti-Bloat: Remove testing module usage when dask is used.
Anti-Bloat: Avoid unitest usage in tf_keras package as well.
Anti-Bloat: Avoid IPython from bokeh package.

Organizational

UI: Catch conflicts between data files and EXE/DLLs/extension module filenames. Previously, you could overwrite binaries with data files, but that is now rejected as an explicit error.
Onefile: Avoid using the program name without suffix inside the dist folder, as that avoids collisions with data file directories of the same name, e.g., if the package and main binary have the same name, they would clash previously, but adding a .bin suffix to the binary avoids that entirely.
UI: Don’t force {VERSION} in specs to be resolved to four digits.

That made it hard for users, who will be surprised to see 1.0 become 1.0.0.0 when that is only needed for Windows version information really.
UI: Catch wrong values for --jobs value sooner, negative and non-integer values error exit immediately. Added in 2.1.1 already.
UI: Nicer usage name when invoked with python -m nuitka

The recommended form of invocation of Nuitka should not have an ugly invocation reference mentioning __main__.py instead put the python -m nuitka notion there.
UI: Reorder options for the plugins group to be more readable.
Plugins: Remove obsolete plugins from standard plugin documentation. Removed in 2.1.4 already.
UI: The Windows release was coming from the compiling Python and as such wrong, for example, Windows 11 always showed up as Windows 10, and some older versions of Python didn’t know Windows 10, yet, so this could be confusing in issue analysis.
UI: Do not warn about static libpython for Python debug mode compilation. It is misleading as often it doesn’t work for that configuration, and it’s only a distraction since debugging Python reference counts is not about performance. Changed in 2.1.4 already.
UI: Catch newlines in spec values. They break code C code generation potentially; they also are likely copy&paste mistakes that won’t do what the user expects. Added in 2.1.4 already.
Quality: Updated to the latest version of black.
Quality: Fix, isort and black can corrupt outputs, catch that.
Debugging: Generate Scons debug script

It can serve to quickly re-execute a Scons compilation without re-executing Nuitka again. This is best used where there is no Python level change but only C changes and no expectation of producing a usable result.

Because no post-processing is applied, and as a consequence this is not usable to produce binaries that work. In the future, we might expand this to be able to run post-processing still.
Debugging: Disabling all freelists is now honored for more code, tuples and empty dictionaries as well.
UI: Add macOS version to help output, which is sometimes vital for issue analysis.
Reports: Add the OS release to reports as well.
Reports: Exclude parent path imports from compilation reports for module usages that are found and end up not being excluded.
Watch: Reporting more problems, catching more errors, and adding the ability to create PRs from changes. However, it does not yet do it automatically.
Visual Code: Have plugins C files in the include path as well.

Tests

Tests: Fix, cannot assume setuptools to be installed, some RPM based systems don’t have it.
Run commercial code signing test only on Windows.
Allow for standalone testing file access to the Azure agent folders. For tests on Azure, it’s like the home directory.
Make sure optimization tests are named to make it clear that they are tests.

Cleanups

Remove useless --execute-with-pythonpath option, we don’t use that anymore at all.

Summary

The JIT mechanism added for tensorflow should be possible to generalize and will be applied to other JITs, like numba and others in the future as well.

The road to Python 3.12 is not fully complete, but the end feels closer now, and the subsequent release hopefully will add the official support for it.

Nuitka Release 2.1

2024-03-23T15:11:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release had focus on new features and new optimization. There is a also a large amount of compatibility with things newly added to support anti-bloat better, and workaround problems with newer package versions that would otherwise need source code at run-time.

Bug Fixes

Windows: Using older MSVC before 14.3 was not working anymore. Fixed in 2.0.1 already.

Compatibility: The dill-compat plugin didn’t work for functions with closure variables taken. Fixed in 2.0.1 already.

def get_local_closure(b):
  def _local_multiply(x, y):
    return x * y + b
  return _local_multiply

fn = get_local_closure(1)
fn2 = dill.loads(dill.dumps(fn))
print(fn2(2, 3))

Windows: Fix, sometimes kernel32.dll is actually reported as a dependency, remove assertion against that. Fixed in 2.0.1 already.
UI: The help output for --output-filename was not formatted properly. Fixed in 2.0.1 already.
Standalone: Added support for the scapy package. Fixed in 2.0.2 already.
Standalone: Added PonyORM implicit dependencies. Fixed in 2.0.2 already.
Standalone: Added support for cryptoauthlib, betterproto, tracerite, sklearn.util, and qt_material packages. Fixed in 2.0.2 already.
Standalone: Added missing data file for scipy package. Fixed in 2.0.2 already.
Standalone: Added missing DLLs for speech_recognition package. Fixed in 2.0.2 already.
Standalone: Added missing DLL for gmsh package. Fixed in 2.0.2 already.
UI: Using reporting path in macOS dependency scan error message, otherwise these contain home directory paths for no good reason. Fixed in 2.0.2 already.
UI: Fix, could crash when compiling directories with trailing slashes used. At least on Windows, this happened for the “/” slash value. Fixed in 2.0.2 already.
Module: Fix, convenience option --run was not considering --output-dir directory to load the result module. Without this, the check for un-replaced module was always triggering for module source in current directory, despite doing the right thing and putting it elsewhere. Fixed in 2.0.2 already.
Python2: Avoid values for __file__ of modules that are unicode and solve a TODO that restores consistency over modules mode __file__ values. Fixed in 2.0.2 already.
Windows: Fix, short paths with and without dir name cached wrongly, which could lead to shorted paths even where not asked for them. Fixed in 2.0.2 already.
Fix, comparing list values that changed could segfault. This is a bug fix Python did, that we didn’t follow yet and that became apparent after using our dedicated list helpers more often. Fixed in 2.0.2 already.
Standalone: Added support for tiktoken package. Fixed in 2.0.2 already.
Standalone: Fix, namespace packages had wrong runtime __path__ value. Fixed in 2.0.2 already.
Python3.11: Fix, was using tuples from freelist of the wrong size
- CPython changed the index for the size, to not use zero, which was wasteful when introduced with 3.10, but to size-1 but we did not follow that and then used a tuple one bit larger than necessary.
- As a result, code producing a lot short living tuples could end up creating new ones over and over, causing bad memory allocations and slow performance.
Fixed in 2.0.2 already.
macOS: Fix, need to allow non-existent and versioned dependencies of DLLs to themselves. Fixed in 2.0.2 already.
Windows: Fix PGO (Profile Guided Optimization) build errors with MinGW64, this feature is not yet ready for general use, but these errors shouldn’t happen. Fixed in 2.0.2 already.
Plugins: Fix, do not load importlib_metadata unless really necessary.

The pkg_resources plugin used to load it, and that then had harmful effects for our handling of distribution information in some configurations. Fixed in 2.0.3 already.
Plugins: Avoid warnings from plugin evaluated code, it could happen that a UserWarning would be displayed during compilation. Fixed in 2.0.3 already.
Fix, loading pickles with compiled functions in module mode was not working. Fixed in 2.0.3 already.
Standalone: Added data files for h2o package. Fixed in 2.0.3 already.
Fix, variable assignment from variables that started to raise were not recognized.

When a variable assignment from a variable became a raise expression, that wasn’t caught and propagated as it should have been. Fixed in 2.0.3 already.
Make the NUITKA_PYTHONPATH usage more robust. Fixed in 2.0.3 already.
Fix, PySide2/6 argument name for slot connection and disconnect should be slot, wasn’t working with keyword argument calls. Fixed in 2.0.3 already.
Standalone: Added support for paddle and paddleocr packages. Fixed in 2.0.4 already.
Standalone: Added support for diatheke. Fixed in 2.0.4 already.
Standalone: Added support for zaber-motion package. Fixed in 2.0.4 already.
Standalone: Added support for plyer package. Fixed in 2.0.4 already.
Fix, added handling of OSError for metadata read, otherwise corrupt packages can have Nuitka crashing. Fixed in 2.0.4 already.
Fix, need to annotate potential exception exit when making a fixed import from hard module attribute. Fixed in 2.0.4 already.
Fix, didn’t consider Nuitka project options with --main and --script-path. This is of course the only way Nuitka-Action does call it, so they didn’t work there at all. Fixed in 2.0.4 already.
Scons: Fix, need to close progress bar when about to error exit. Otherwise error outputs will be garbled by incomplete progress bar. Fixed in 2.0.4 already.
Fix, need to convert relative from imports to hard imports too, or else packages needed to be followed are not included. Fixed in 2.0.5 already.
Standalone: Added pygame_menu data files. Fixed in 2.0.6 already.
Windows: Fix, wasn’t working when compiling on network mounted drive letters. Fixed in 2.0.6 already.
Fix, the .pyi parser was crashing on some comments with a leading from in the line, recognize these better. Fixed in 2.0.6 already.
Actions: Fix, some yaml configs could fail to load plugins. Fixed in 2.0.6 already.
Standalone: Added support for newer torch packages that otherwise require source code.
Fix, inline copies of tqdm etc. left sub-modules behind, removing only the top level sys.modules entry may not be enough.

New Features

Plugins: Added support for constants in Nuitka package configurations. We can now using when clauses, define variable values to be defined, e.g. to specify the DLL suffix, or the DLL path, based on platform dependent properties.
Plugins: Make relative_path, suffix, prefix in DLL Nuitka package configurations allowed to be an expression rather than just a constant value.
Plugins: Make not only booleans related to the python version available, but also strings python_version_str and python_version_full_str, to use them when constructing e.g. DLL paths in Nuitka Package Configuration.
Plugins: Added helper function iterate_modules for producing the submodules of a given package, for using in expressions of Nuitka package configuration.
macOS: Added support for Tcl/Tk detection on Homebrew Python.
Added module attribute to __compiled__ values

So far it was impossible to distinguish non-standalone, i.e. accelerated mode and module compilation by looking at the __compiled__ attribute, so we add an indicator for module mode that closes this gap.
Plugins: Added appdirs and importlib for use in Nuitka package config expressions.
Plugins: Added ability to specify modules to not follow when a module is used. This nofollow configuration is for rare use cases only.
Plugins: Added values extension_std_suffix and extension_suffix for use in expressions, to e.g. construct DLL suffix patterns from it.
UI: Added more control over caching with per cache category environment variables, as documented in the User Manual..
Plugins: Added support for reporting module detections

The delvewheel plugin now puts the version of that packaging tool used by a particular module in the report rather than tracing it to the user, that in the normal case won’t care. This is more for debugging purposes of Nuitka.

Optimization

Scalability: Do not make loop analysis at all for very trusted value traces, their point is to not change, and waiting for that to be confirmed has no point.
Use very trusted value traces in functions not just as mere assign traces or else expected optimization will not be done on them in many cases. With this a lot more cases of hard values are optimized leading also to generally more compact and correct results in terms of imports, metadata, code avoided on the wrong OS, etc.
Scalability: When specializing assignments, make sure to have the proper value trace immediately.

When changing to a hard value, the value trace was still an assign trace and not very trusted for one for micro pass of the module.

This had the effect to need one more micro pass to get to benefiting of the unescapable nature of those values, which meant more micro passes than necessary and those being more complex due to escaped traces, and therefore taking longer for affected modules.
Scalability: The code trying avoid merge traces of merge traces, and to instead flatten merge traces was only handling part of these correctly, and correcting it reduced optimization time for some functions from infinite to instant. Less memory usage should also come out of this, even where this was not affecting compile time as much. Added in 2.0.1 already.
Scalability: Some codes that checked for variables were testing for temporary variable and normal variable both one after another, making some optimization steps and code generation slower than necessary due to the extra calls.
Scalability: A variable assignment from variable that were later recognized to become a raise was not recognized as such, and this then wasn’t caught and propagated as it should, preventing more optimization of the affected code. Make sure to convert more directly when observing things to change, rather than doing it one pass later.
The fix proper reuse of tuples released to the freelist with matching sizes causes less memory usage and faster performance for the 3.11 version. Added in 2.0.2 already.
Statically optimize sys.exit into exception raise of SystemExit.

This should make a bunch of dead code obvious to Nuitka, it can now tell this aborts execution of a branch, potentially eliminating imports, etc.
macOS: Enable python static link library for Homebrew too. Added in 2.0.1 already. Added in 2.0.3 already.
Avoid compiling bloated module namespace of altair package. Added in 2.0.3 already.
Anti-Bloat: Avoid including kubernetes for tensorflow unless used otherwise. Added in 2.0.3 already.
Anti-Bloat: Avoid including setuptools for tqdm. Added in 2.0.3 already.
Anti-Bloat: Avoid IPython in fire package. Added in 2.0.3 already.
Anti-Bloat: Avoid including Cython for pydantic package. Added in 2.0.3 already.
Anti-Bloat: Changes to avoid triton in newer torch as well. Added in 2.0.5 already.
Anti-Bloat: Avoid setuptools via setuptools_scm in pyarrow.
Anti-Bloat: Made more packages equivalent to using setuptools which we want to avoid, all of Cython, cython, pyximport, paddle.utils.cpp_extension, torch.utils.cpp_extension were added for better reports of the actual causes.

Organizational

Moved the changelog of Nuitka to the website, just point to there from Nuitka repo.
UI: Proper error message from Nuitka when scons build fails with a detail mnemonic page. Read more on for detailed information.
Windows: Reject all MinGW64 that are not are not the WinLibs that Nuitka itself downloaded. As these packages break very easily, we need to control if it’s a working set of ccache, make, binutils and gcc with all the necessary workarounds and features like LTO working on Windows properly.
Quality: Added auto-format of PNG and JPEG images. This aims at making it simpler to add images to our repositories, esp. Nuitka Website. This now makes optipng and jpegoptim calls as necessary. Previously this was manual steps for the website to be applied.
User Manual: Be more clear about compiler version needs on Windows for Python 3.11.
User Manual: Added examples for error message with low C compiler memory, such that maybe they can be found via search by users.
User Manual: Removed sections that are unnecessary or better maintained as separate pages on the website.
Quality: Avoid empty no-auto-follow values, for silently ignoring it there is a dedicated string ignore that must be used.
Quality: Enforce normalized paths for dest_path and relative_path. Users were uncertain if a leading dot made sense, but we now disallow it for clarity.
Quality: Check more keys with expressions for syntax errors, to catch these mistakes in configuration sooner.
Quality: Scanning through all files with the auto-format tool should now be faster, and CPython test suite directories (test submodules) if present are ignored.
Release: Remove month from manpage generation, that’s only noise in diffs.
Removed digital art folders, these were only making checkouts larger for no good reason. We will have better ones on the website in the future.
Scons: Allow C warnings when compiling for running in debugger automatically.
UI: The macOS app bundle option is not experimental at all. This has been untrue for years now, remove that cautioning.
macOS: Discontinue support for PyQt6.

With newer PyQt6 we would have to package frameworks properly, and we don’t have that yet and it will be a lot of developer time to get it.

Instead point people to PySide6 which is the better choice and is perfectly supported by Qt company and Nuitka.
Removed version numbering, month of creation, etc. from the man pages generated.
Moved Credits.rst file to be on the website and maintain it there rather than syncing of from the Nuitka repository.
Bumped copyright year and split the license text such that it is now at the bottom of the files rather than eating up the first page, this is aimed at making the code more readable.

Cleanups

With sys.exit being optimized, we were able to make our trick to avoid following nuitka because of accidentally finding the setup as an import more simple.

# Don't allow importing this, and make recognizable that
# the above imports are not to follow. Sometimes code imports
# setup and then Nuitka ends up including itself.
if __name__ != "__main__":
   sys.exit("Cannot import 'setup' module of Nuitka")

Scons: Don’t scan for ccache on Windows, the WinLibs package contains it nowadays, and since it’s now required to be used, there is no point for this code anymore.
Minor cleanups coming from trying out ruff as a linter on Nuitka, it found a few uses of not using not in, but that was it.

Tests

Removed test with chinese filenames, we need to avoid chinese names in the repo. These have been seen as preventing installation on some systems that are not capable of handling them in the git, zip, pip tooling, so lets avoid them entirely now that Nuitka handles these just fine.
Tests: More macOS standalone tests that need to be bundles were getting the project configuration to do it.

Summary

This release added much needed tools for our Nuitka Package configuration, but also cleans up scalability and optimization that was supposed to work, but did not yet, or not anymore.

The usability improved again, as it does always, but the big improvements for scalability that will implement existing algorithms more efficient, are yet to come, this release was mainly driven by the need to get torch to work in its latest version out of the box with stable Nuitka, but this couldn’t be done as a hotfix

Nuitka Release 2.0

2024-02-05T00:00:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release had focus on new features and new optimization. There is a really large amount of compatibility with things newly added, but also massive amounts of new features, and esp. for macOS and Windows, lot of platform specified new abilities and corrections.

Bug Fixes

Fix, workaround for private functions as Qt slots not having names mangled. Fixed in 1.9.1 already.
Fix, when using Nuitka with pdm it was not detected as using pip packages. Fixed in 1.9.1 already.
Fix, for pydantic our lazy loader parser didn’t handle all cases properly yet. Fixed in 1.9.1 already.
Standalone: Added data files for pyocd package. Fixed in 1.9.1 already.
Standalone: Added DLL for cmsis_pack_manager package. Fixed in 1.9.1 already.
Standalone: Fix, the specs expanded at run time in some causes could contain random characters. Fixed in 1.9.2 already.
Fix, {"a":b, ...}.get("b") could crash at runtime. Fixed in 1.9.2 already.
Standalone: Added data files for pyproj package. Fixed in 1.9.2 already.
Standalone: Added more metadata requirements for transformers package. Fixed in 1.9.2 already.
Plugins: Fix, could crash when including packages from the command line, if they had yaml configuration that requires checking the using module, e.g. anti-bloat work. Fixed in 1.9.3 already.
Standalone: Added support for delphifmx package. Fixed in 1.9.4 already.
Android: Fix, cannot exclude libz on that platform, it’s not a full Linux OS. Fixed in 1.9.3 already.
Standalone: Add needed DLLs for bitsandbytes package. Fixed in 1.9.3 already.
Windows: Fix, newer joblib was not working anymore. Fixed in 1.9.3 already.
Windows: Fix, could crash when working with junctions that switch drives. Fixed in 1.9.3 already.
Fix, was crashing with poetry installed environments. Fixed in 1.9.3 already.
Standalone: Added support for newer chromadb package. Fixed in 1.9.3 already.
Fix, could crash in report creation on modules excluded that were asked via command line for inclusion. Fixed in 1.9.3 already.
Anti-Bloat: Fix for newer streamlit, it was causing SyntaxError for the compilation. Fixed in 1.9.4 already.
Arch: Added support for their OS release file location too. Fixed in 1.9.4 already.
Windows: Fix, MinGW64 doesn’t accept chinese module names a C source files. Use short paths for these instead. Fixed in 1.9.4 already.
Standalone: Added missing DLL for libusb_package package. Fixed in 1.9.4 already.
Fix, properly skip directories with non-module top level names when trying to find top level packages of distributions. Fixed in 1.9.4 already.
Fix, avoid memory leak bug in triggered by rich package. Fixed in 1.9.4 already.
Python3.11+: Fix, didn’t detect non-keywords on star dict calls in some cases. Fixed in 1.9.4 already.
Fix, avoid crashes due to unrecognized installers on macOS and Windows, some packages that are built via legacy fallbacks of certain pip versions do not leave any indication of their origin at all. Fixed in 1.9.4 already.
Windows: Fix, need to indicate that the program is long path aware or else it cannot work with the paths. Fixed in 1.9.4 already.
Debian: The extern namespace might not exist in the pkg_resources module, make the code work with versions that remove it and use the proper external package names then. Fixed in 1.9.6 already.
Compatibility: Fix, need to also have .exists method in our files reader objects. Fixed in 1.9.5 already.
macOS: Fix, PyQt5 standalone can fail due to libqpdf too.
Compatibility: Make dill-compat plugin support module mode too, previously this only worked for executables only. Fixed in 1.9.6 already.
Standalone: Added data file for curl_cffi package. Fixed in 1.9.6 already.
Windows: Fix warnings given by MinGW64 in debug mode for onefile compilation. Fixed in 1.9.6 already.
Python2: The handling of DLL permission changes was not robust against using unicode filenames. Fixed in 1.9.7 already.
Python2: Fix, could crash on Debian packages when detecting their installer. Fixed in 1.9.7 already.
Standalone: Added required data file for astor package. Fixed in 1.9.7 already.
Reports: Fix, in case of build crashes during optimization, the bug report creation could be crashing because the module is not in the list of done modules yet. Fixed in 1.9.7 already.
Python2: Fix, unittest.mock was not yet available, code attempting to use it was crashing the compilation. Fixed in 1.9.7 already.
Accelerated: Fix, tensorflow configuration removing site usage needs to apply only to standalone mode. Fixed in 1.9.7 already.
Plugins: Fix, the get_dist_name Nuitka Package Configuration function could crash in some rare configurations. Fixed in 1.9.7 already.
Standalone: Added necessary data file for pygame package. Added in 1.9.7 already.
Standalone: Fix, was not properly handling standard library overloading module names for decisions. Inclusion and compilation mode were made as if the module was part of the standard library, rather than user code. This is now properly checking if it’s also an actual standard library module.
Plugins: Fix, crashing on missing absence message with no UPX binary was found.
Windows: Fix, couldn’t load extension modules from UNC paths, so standalone distributions failed to launch from network drives. This now works again and was a regression from adding support for symlinks on Windows.
Standalone: Added support for non-legacy pillow in imageio package.
Standalone: Added required easyOCR data file.
Nuitka-Python: Fix, do not demote to non-LTO for “too many” modules there in the default auto mode, it doesn’t work without it.
Fix, python setup.py install could fail. Apparently it tries to lookup Nuitka during installation, which then could fail, due to hacks we due to make sure wheels are platform dependent. That hack is of course not really needed for install, since no collision is going to happen there.
macOS: Fix, the standard matplotlib plugin that uses native UI was not included yet, and it was also not working due to bindings requiring uncompiled functions, which is now worked around.
Compatibility: Add back PySide6 workaround for overloading names like update with slots.
Standalone: Added geopandas data files.
Python2: Fix, code objects must be made from str exactly, unicode however was used in some configurations after recent improvements to the run time path handling.
Standalone: Added missing data files for boto, the predecessor of boto3 as well.
Standalone: Added missing DLL for tensorflow factorization module.
Compatibility: Fix, PySide2 and PySide6 signal disconnection without arguments were not working yet.
Standalone: Added support for toga.
Scons: Fix, need to Avoid picking up clang from PATH on Windows with --clang provided, as only our WinLibs version is really working.
Fix, version of setuptools when included (which we try to avoid very much) was None which breaks some users of it, now it’s the correct version so checks of e.g. setuptools_scm can succeed.
Fix, icon options for platforms were conflated, so what should be windows only icon could get used on other platforms as well.
Fix, could not create compiled methods from compiled methods. Also now errors out for invalid types given properly.

New Features

Plugins: Added support for module decisions, these are parameters provided by the user which can be used to influence the Nuitka per package configuration with a new get_parameter function. We are using these to control important choices in the user, sometimes warning it to make that decision, if the default can be considered problematic.
Plugins: Added support for variables in Nuitka package configurations. We can now query at compile time, values from installed packages and use them, e.g. to know what backend is to be used.
Standalone: Added module decision to disable Torch JIT. This is generally the right idea, but the decision is still asked for since some packages and programs want to do Torch Tracing, and that is then disabled as well. This makes a bunch of transformers programs work.
Standalone: Added module decision to disable Numba JIT. This makes numba work in some cases, but not all. Some packages go very deep with JIT integration, but simpler uses will now compile.
New option --include-onefile-external-data allows you to specify file patterns that you included by other data files others, but to put those files not inside, but on the outside of the onefile binary. This makes it easier to create deployments fully within Nuitka project configuration, and to change your mind back and forth without adding/removing the data file option.
macOS: Added new value auto for detecting signing identity, if only one is available in the system.
macOS: Added support for --copyright and --trademark information to be in app bundles as well, this was previously Windows only.
Windows: Added support for using junctions in the Python environment, these are used e.g. when installing via scoop. Added in 1.9.2 already.
Added option --cf-protection to select the control flow protection mode for the GCC compiler and deviate from default values of some environments to less strict values.
Reports: Added output filename to report, mainly intended for automatically locating the compilation result independent of options used.
Plugins: Now provides a checksum for yaml files, but not yet verifies them at runtime, to ask the user to run the checker tool to update it when they make modifications.
Windows: Detect when we create too large compiled executables. There is a limit of 2GB that you might e.g. violate by attempting to embed very large files. This doesn’t cover onefile yet.
Watch: The tool can now create PRs with the changes in Nuitka-Watch for merging, this is for using it in the CI.
Watch: Scanning for Python versions now requires pipenv to be installed in them to be found.
Watch: Added ability to create branch and PR from watch run results.
Plugins: Added overridden-environment-variables feature to package configuration. These are environment variable changes that only last during the import of that module and are undone later.
Plugins: Added force-environment-variables feature to package configuration. These are environment variable changes done on module import that are not undone.
Nuitka-Action: Nuitka options that can be given multiple times, cannot be specified multiple times in your workflow. As a workaround, Nuitka now allows in Actions, to use new lines as separator. This is best done with this kind of quoting a multiline string.
```
include-data-dir: |
   a=b
   c=d
```
The Nuitka Package Configuration no-auto-follow now applies recursively, i.e. that a top level package can have it, and not every sub-package that uses a package but should not be automatically followed, does have to say this. With this e.g. networkx configuration became simpler, and yet covered automatically older versions as well, and future changes too.
Windows: Added support for compiling in case sensitive folders. When this option is enabled, using os.path.normcase can make filenames not found, so with a few cleanups, for lazy code that wasn’t really using the APIs designed for comparisons and filename suffix testing, this works now better.
The __compiled__ value has a new attribute containing_dir that allows to find where a module, accelerate executable, a standalone dist folder, a macOS app bundle, or the onefile binary lives in a consistent fashion. This allows esp. better use than sys.argv[0] which points deep into the .app bundle, and can be used cross platform.

Optimization

Scalability: Avoid variables that are not shared to be treated as if they were, marking their type shape as tshape_unknown in the first micro pass. These micro passes are not visible, but basically constitute a full visit of the module tree over and over, until no more optimization is changing it. This can lead to quicker resolution, as that unknown type shape effectively disallowed all optimization for variables and reduce the number of necessary micro passes by one.
Escaped variables did provide a type shape tshape_unknown and while a lot of optimization looks for value knowledge, and gets by the escaped nature of the value, sometimes, this was seriously inhibiting some of the type based optimization.

Loop type shape analysis now succeeds in detecting the types for this code example, which is sort of a break-through for future performance enhancements in generated code.

# Initial the value of "i" is "NUITKA_NINT_UNASSIGNED" in its
# indicator part. The C compiler will remove that assignment
# as it's only checked in the assignment coming up.
i = 0
# Assignment from a constant, produces a value where both the C
# and the object value are value. This is indicated by a value
# of "NUITKA_NINT_BOTH_VALID". The code generation will assign
# both the object member from a prepared value, and the clong
# member to 0.

# For the conditional check, "NUITKA_NINT_CLONG_VALID" will
# always be set, and therefore function will resort to comparing
# that clong member against 9 simply, that will always be very
# fast. Depending on how well the C compiler can tell if an overflow
# can even occur, such that an object might get created, it can even
# optimize that statically. In this case it probably could, but we
# do not rely on that to be fast.
while i < 9:  # RICH_COMPARE_LT_CBOOL_NINT_CLONG
   # Here, we might change the type of the object. In Python2,
   # this can change from ``int`` to ``long``, and our type
   # analysis tells us that. We can consider another thing,
   # not "NINT", but "NINTLONG" or so, to special case that
   # code. We ignore Python2 here, but multiple possible types
   # will be an issue, e.g. list or tuple, float or complex.
   # So this calls a function, that returns a value of type
   # "NINT" (actually it will become an in-place operation
   # but lets ignore that too).
   # That function is "BINARY_OPERATION_ADD_NINT_NINT_CLONG"(i, 1)
   # and it is going to check if the CLONG is valid, add the one,
   # and set to result to a new int. It will reset the
   # "NUITKA_NINT_OBJECT_VALID" flag, since the object will not be
   # bothered to create.
   i = i + 1

# Since "NUITKA_INT_OBJECT_VALID" not given, need to create the
# PyObject and return it.
return i

Python3.11+: Use tomllib from standard library for our distutils integration into pyproject based builds.
Avoid late specialization for None returns in generators and do it during tree building already, to remove noise.
Added successful detection of static libpython for self compiled Python Linux and macOS. This makes it work with pyenv as well.
Standalone: Avoid including .pyx files when scanning for data files, these are code files too, in this case source files that are definitely unused most of the time.
macOS: Make static libpython default with CPython for more compact standalone distribution and faster binaries.
Remove non-existent entries from sys.path, avoiding many file system lookups during import scans.
Anti-Bloat: Avoid using triton in torch package in more cases. Added in 1.9.2 already.
Anti-Bloat: Avoid using pytest in knetworkx package in more cases. Added in 1.9.2 already.
Anti-Bloat: Avoid using IPython in distributed package. Added in 1.9.3 already.
Anti-Bloat: Avoid using dask in skimage. Added in 1.9.3 already.
Anti-Bloat: Avoid using triton in the bitsandbytes package. Added in 1.9.3 already.
Anti-Bloat: Avoid IPython in tf_keras package as well. Added in 1.9.6 already.
Anti-Bloat: Avoid unittest in mock.mock module. Added in 1.9.7 already.
Avoid importing setuptools_scm during compilation when using the tqdm inline copy, this also avoids a warning on Ubuntu. Added in 1.9.7 already.
Anti-Bloat: Avoid doctest in skimage in their tifffile inline copy as well. Added in 1.9.7 already.
Anti-Bloat: Avoid h5py.tests with older h5py as well. Added in 1.9.7 already.
Anti-Bloat: Using distributed.utils_test is also considered using pytest.
Anti-Bloat: Avoid IPython in the pip package.
Anti-Bloat: Avoid site module for older tensorflow versions too.
Anti-Bloat: Avoid more unittest usages in tensorflow packages.
Anti-Bloat: Avoid nose in skimage package.
Anti-Bloat: Avoid nose in networkx package.
Anti-Bloat: Avoid nose in pywt package.

Organizational

UI: Change template paths over from %VAR% to {VAR}.

The old spec values are migrated transparently and continue to work, but get a warning when used.

The new code detects unknown variable names and more formatting issues than before.

Using only the {PID} value for process ID, is now making it temporary value for onefile, that was previously a bug.

The main benefit and reason of doing this, is that Windows CMD.EXE does expand those values before Nuitka sees them as even with quoting %TEMP% is the current one on the building machine, a recipe for disaster. As some people still use that, and e.g. os.system or subprocess with shell=True will use it too, this is just not sustainable for a good user experience.

As a result, compile time and run time variables now clash, there is e.g. {VERSION} (program version information given) and {Version} (Nuitka version), and we should clean that up too.
Project: Added Code of Conduct. Adapted from the one used in the Linux kernel.
UI: Warnings given by Nuitka used to be in red color, changed those to be yellow for consistency.
User Manual: Added pointer for Nuitka-Action Nuitka-Action for users interested in using Nuitka in GitHub workflows.
Added .gitignore to build folder that just causes these folders to be ignored by git.
User Manual: Added information on how to debug fork bombs from created binaries.
Debugging: The output of --experimental=--report-refcounts that we use to show leaks of compiled time objects at program exit, now counts and reports on functions, generator objects and compiled cells as well.
Quality: Warnings from yamllint not disabled are errors. These were only output, but didn’t cause the autoformat to error exit yet.
UI: Enhanced formatting of info traces, drop the :INFO part that shouts, and reserve that for errors and warnings. Also format info messages to make sure they fit into the line.
UI: Changed --show-source-changes to accept module pattern to make it easier to only see the ones currently being worked on. To get the old behavior of showing everything, use * as a pattern.
UI: Allow using ~ in data files source path for command line options and expand it properly.
Quality: Enhanced schema for our package configuration yaml files to detect suffixes with leading dots, that is not wanted. These now fail checks, but we also tolerate them now.
Quality: Check module names used in the package configuration yaml files for validity, this catches e.g. trailing dots.
Quality: Make sure to really prefer clang-format from Visual Code and MSVC for formatting C code, otherwise a system installed one could be used that gives slightly different outputs.
Scons: Allow disabling to enforce no warnings for C compilation

Currently only for gcc, where we need it until loop tracing is better, we can now use --experimental=allow-c-warnings options to make --debug work for some known currently unavoidable warnings.
macOS: Make --macos-create-app-bundle imply standalone mode, it’s not working or useful for accelerated mode anyway.
Standalone: Added support for using self-compiled Python versions that are not installed on Linux and macOS. This avoids having to do make install and can ease debugging with changes made in Python core itself. Added in 1.9.6 already.
Release: Added ability to simple re-date hotfixes. Previously the version bump commit needed to be dropped, now a fixup commit is easy to generate.
Release: Man pages are no longer built during package builds, but are available statically in the git, which should make it easier.
Release: Disable verbose output in package installation of Nuitka, it never was any use, and just makes things hard to read.
UI: Check user yaml file present immediately. Otherwise it was crashing when parsing yaml files first time with less comprehensible exceptions. Added in 1.9.7 already.
Quality: Updated to latest rstfmt, black and isort versions.
Debian: Remove references to PDF documentation that no longer exists.
Quality: Do not crash when collecting modified files due to deleted files.
UI: Detect the Alpine flavor of Python as well.
UI: Detect manylinux Pythons as a Python flavors as well.
UI: Detect self compiled uninstalled Python as a dedicated flavor.

Cleanups

For the Nuitka-Action part of the available options is now generated from Nuitka option definitions itself, adding some previously missing options as a result. As a result, adding --include-onefile-external-data was automatic this time.
The warnings for onefile only options without onefile mode provided have been moved to common code, and in some cases were having wrong texts corrected.
Use enum definitions in the Nuitka Package Configuration schema rather than manual oneOf types.
The User Manual was proof read and had a bunch of wordings improved.
Cleanup, avoid “unused but set variable” warning from the C compiler for hard some forms of hard imports.
Prefer os.getenv over os.environ.get for readability.
Changed parts of the C codes that clang-format had a hard time with to something more normal.

Tests

When locating the standalone binary created, use a compilation report and resolve the path specified there. This allows macOS app bundles to be used in these tests as well.
Made the PyQt tests executable on macOS too adding necessary options.
Added reference test case for unpacking into a list, this was not covered but under suspect of reference leaking which turns out to be wrong.
Much enhanced usage of virtualenv in the distutils test cases. We make more sure to delete them even in case of issues. We disable warnings during Nuitka package installation. The code to execute a case was factored out and became more clear. We now handle errors in execution with stating what case actually failed, this was a bit hard to tell previously. Also do not install Nuitka when a pyproject case is used, since the build tool installs Nuitka itself.

Summary

This release deserves the 2.0 marker, as it is ground breaking in many ways. The loop type analysis stands out on the optimization front. This will open an avenue for much optimized code at least for some benchmark examples this summer.

The new features for package configuration, demonstrate abilities to avoid plugins for Nuitka, where those previously would have been used. The new variables and parameters made it unnecessary to have them, and still add compile time variable use and user decisions and information, without them.

The scope of supported Python configurations got expanded a bit, and the the usual slew of anti-bloat work and new packages supported, makes Nuitka an ever more round package.

The improved user dialog with less noisy messages and slightly better coloring, continues a trend, where Nuitka becomes more and more easy to use.

Nuitka this week #15

2024-01-30T00:00:00+00:00

This is a weekly update, or at least it’s supposed to be of what’s going on in Nuitka land, for you to learn about ongoing developments and important changes to the project.

In this issue, I am first going to cover a bit of backlog from news update missed in the past, but also covering very exciting changes from this week.

Nuitka Action

A GitHub Action is a component used in GitHub workflows. These are yaml driven configurations that can cause GitHub to do automatic building of your software.

Many of the more professional users build their binaries as part of GitHub workflows, and Nuitka and Nuitka commercial are both used in that way a lot. Many times people do it on their own, i.e. install Nuitka by hand, and call it by hand, which is kind of not the preferred way for many people.

Enter the great Nuitka Action which was originally created by Jim Kring, who handed over the maintenance of it to the Nuitka organization that has further refined it. This was a great contribution that makes it everything easier for Nuitka users on GitHub if they want to use it.

- name: Build Executable
  uses: Nuitka/Nuitka-Action@main
  with:
    nuitka-version: main
    script-name: kasa_cli
    onefile: true

- name: Upload Artifacts
  uses: actions/upload-artifact@v3
  with:
    name: ${{ runner.os }} Build
    path: |
      build/*.exe
      build/*.bin
      build/*.app/**/*

Options of Nuitka are exposed as yaml attributes. The documentation of this mapping could be very much enhanced, but basically it’s just dropping the -- part from e.g. --onefile and for toggles, you say true.

Now one interesting limitation of GitHub Action, I have come across this week and that is that it’s not easily possible to specify an option twice. For some values in Nuitka, that however is necessary. Where module names are acceptable, a , separation os supported, but with file paths, we don’t do that, e.g. not with --include-data-dir (note --include-package-data is much better to use) but that is the one it came up for.

But now we support splitting by new-line from GitHub actions for everything that produces a list value as a Nuitka option. See below for a very nice example of how the | in Yaml makes that even easy to read.

- name: Build Executable
  uses: Nuitka/Nuitka-Action@main
  with:
    nuitka-version: main
    script-name: kasa_cli
    onefile: true
    include-data-dir: |
      source_path_dir1=dest_path_dir1
      source_path_dir2=dest_path_dir2
      source_path_dir3=dest_path_dir3

Note

This works with Nuitka 2.0 or higher.

The Nuitka-Action is permanently refined. Just today we updated its caching action to latest, and there is an ongoing activity to improve options. We started to generate options from Nuitka help output directly, so that it is easier to add support for new Options in Nuitka, and to generally make them more consistent.

Community

On the Discord server, you can get in touch with an ever more vibrant community of Nuitka users. You are welcome to join us on the Discord server for Nuitka community where you can hang out with the developers and ask questions. It’s not intended as an interactive manual. You are supposed to read the docs for yourself first. And issues are best reported to GitHub.

I am also now occasionally on the Python Discord server. Mostly when I get summoned to answer questions that my community thinks make sense, and have been awarded the community role there, which is pretty nice. I seem to make new connections there.

Optimization Work

For me this is extremely exciting, this has been on my nerves for a long time, and I didn’t have the time to figure it out. Now for the scalability work, I wanted to make sure the algorithm used for loop type analysis is actually going to be sustainable, before I optimize the implementation to scale better.

And low and behold, one of my oldest code examples, the one I mean to demonstrate C type performance from Python code with, has failed to get proper results for a long time now. But this changed this week and it’s part of the 2.0 release, making it my mind worth the bump itself. Checkout this annotated code.

# Initially the value of undefined "i" is "NUITKA_NINT_UNASSIGNED"
# in its indicator part. The C compiler will remove that assignment
# as it's only checked in the assignment coming up.

i = 0
# Assignment from a constant, produces a value where both the C
# and the object value are value. This is indicated by a value
# of "NUITKA_NINT_BOTH_VALID". The code generation will assign
# both the object member from a prepared value, and the clong
# member to 0.

# For the conditional check, "NUITKA_NINT_CLONG_VALID" will
# always be set, and therefore function will resort to comparing
# that clong member against 9 simply, that will always be very
# fast. Depending on how well the C compiler can tell if an overflow
# can even occur, such that an object might get created, it can even
# optimize that statically. In this case it probably could, but we
# do not rely on that to be fast.
while i < 9:  # RICH_COMPARE_LT_CBOOL_NINT_CLONG
    # Here, we might change the type of the object. In Python2,
    # this can change from ``int`` to ``long``, and our type
    # analysis tells us that. We can consider another thing,
    # not "NINT", but "NINTLONG" or so, to special case that
    # code. We ignore Python2 here, but multiple possible types
    # will be an issue, e.g. list or tuple, float or complex.
    # So this calls a function, that returns a value of type
    # "NINT" (actually it will become an in-place operation
    # but lets ignore that too).
    # That function is "BINARY_OPERATION_ADD_NINT_NINT_CLONG"(i, 1)
    # and it is going to check if the CLONG is valid, add the one,
    # and set to result to a new int. It will reset the
    # "NUITKA_NINT_OBJECT_VALID" flag, since the object will not be
    # bothered to create.
    i = i + 1

# Since "NUITKA_INT_OBJECT_VALID" not given, need to create the
# PyObject and return it.
return i

Now that the loop analysis works, I will be much happier to make the value trace collection faster. I will describe it when I do it. From here on for optimization, the C type NINT needs to be created and code generation for the branching helper functions be added, and then the should see this perform perfectly.

Functions like RICH_COMPARE_LT_CBOOL_NINT_CLONG will look like this. We do not yet have RICH_COMPARE_LT_CBOOL_LONG_CLONG which it will fall back to, but we did RICH_COMPARE_LT_CBOOL_INT_CLONG for Python2 a while ago, and we could expand that no problem.

extern bool RICH_COMPARE_LT_CBOOL_NINT_CLONG(nuitka_long *operand1, long operand2) {
   if (operand1->validity & NUITKA_LONG_VALUE_VALID) {
      return operand1->long_value < operand2;
   } else {
      return RICH_COMPARE_LT_CBOOL_LONG_CLONG(operand1->long_object, operand2);
   }
}

Once I get to that, performance will get a hot topic. From there then, adding sources of type information, be it profile guided compilation, be it type annotations, be it ever better compile time type inference, will start to make a lot more sense.

Nuitka 2.0

The 2.0 release has been made. I am going to announce it separately. I am usually waiting for a few days, to settle potentially regressions. This time older C compiler support needed a fixup, there is always something. And then I announce it when I feel that the regressions are gone and that new users will not encounter obvious breakage at all.

Technical Writer

When I first launched Nuitka commercial, I needed to get myself financially supported, dropping my day job after 20 years. I am willing to say that has happened.

Now as you all know, Nuitka is technically excellent. I cannot say the same thing of the documentation. Large parts of Nuitka commercial are still not even publicly described. The User Manual of Nuitka is good, but not nearly as good as it should be. The website is kind of unorganized. It’s pretty obvious that my talent is not there. I have received plenty of help over the years, but it’s not been enough to match Nuitka and Nuitka commercial outstanding quality.

So, what I did this week, after seeing that my financial projection for the coming years seems to allow it, is to attempt and hire people on a free lancing basis. The first step is a technical writer. She will know very little of Python and even the terminal, but she will know how to organize and improve the content of Nuitka.

It will take time for her to get there and this is very fresh.

Nuitka Website as a DevContainer

As a first necessary step to make it easier to contribute to the Nuitka documentation, the website repo, has gained DevContainer configuration. It will install a small Ubuntu via docker (or podman if you configured Visual Code like that), and run the pipenv environment and a daemon to open the website.

The docs for that are spotty right now, and the Technical Writer that is using that, is tasked to improve this right now.

It should become really easy that way to contribute enhancements to the documentation.

I have yet to figure out how to handle the release matching documentation vs. website documentation for user manual. But the idea is certainly that the Nuitka documentation is edited on the website.

Nuitka Package Configuration

So I added a post that explains variables, but the one for parameters, I still need to do that and also update the actual reference documentation.

Teasers

Future TWN still have a lot to talk about, we will speak about Nuitka-Python (our own Python fork with incredible capabilities), about Nuitka-Watch (our way of making sure Nuitka works with PyPI packages and hot-fixes to not regress), about compilation reports as a new feature, Windows AV stuff, onefile improvements, and so on and so on. I got interesting stuff for many weeks. Limiting myself for now or I will never publish this.

Twitter and Mastodon

I should be more active there, although often I fail due to not wanting to talk about unfinished things, so actually I do not post there as much.

And lets not forget, having followers make me happy. So do re-tweets. Esp. those, please do them.

Help Wanted

Nuitka definitely needs more people to work on it. I hope the technical writer will aid us in better laying how ways for you to help.

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka Package Configuration Part 3

2024-01-24T00:00:00+00:00

This is the third part of a post series under the tag package_config that explains the Nuitka Package Configuration in more detail. To recap, Nuitka package configuration is the way Nuitka learns about hidden dependencies, needed DLLs, data files, and just generally avoids bloat in the compilation. The details are here on a dedicate page on the web site in Nuitka Package Configuration but reading on will be just fine.

Problem Package

Each post will feature one package that caused a particular problem. In this case, we are talking about the package toga.

Problems like with this package are typically encountered in standalone mode only, but they also affect accelerated mode, since it doesn’t compile all the things desired in that case. Some packages, and in this instance look at what OS they are running on, environment variables, etc. and then in a relatively static fashion, but one that Nuitka cannot see through, loads a what it calls “backend” module.

We are going to look at that in some detail, and will see a workaround applied with the anti-bloat engine doing code modification on the fly that make the choice determined at compile time, and visible to Nuitka is this way.

Initial Symptom

The initial symptom reported was that toga did suffer from broken version lookups and therefor did not work, and we encountered even two things, that prevented it, one was about the version number. It was trying to do int after resolving the version of toga by itself to None.

Traceback (most recent call last):
  File "C:\py\dist\toga1.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\toga\__init__.py", line 1, in <module toga>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\toga\app.py", line 20, in <module toga.app>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\toga\widgets\base.py", line 7, in <module toga.widgets.base>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\travertino\__init__.py", line 4, in <module travertino>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\setuptools_scm\__init__.py", line 7, in <module setuptools_scm>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\setuptools_scm\_config.py", line 15, in <module setuptools_scm._config>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\setuptools_scm\_integration\pyproject_reading.py", line 8, in <module setuptools_scm._integration.pyproject_reading>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\py\dist\setuptools_scm\_integration\setuptools.py", line 62, in <module setuptools_scm._integration.setuptools>
  File "C:\py\dist\setuptools_scm\_integration\setuptools.py", line 29, in _warn_on_old_setuptools
ValueError: invalid literal for int() with base 10: 'unknown'

So, this is clearly something that we consider bloat in the first place, to runtime lookup your own version number. The use of setuptools_scm is implying the use of setuptools, for which the version cannot be determined, and that’s crashing.

Step 1 - Analysis of initial crashing

So first thing, we did was to repair setuptools, to know its version. It is doing it a bit different, because it cannot use itself. Our compile time optimization failed there, but also would be overkill. We never came across this, since we avoid setuptools very hard normally, but it’s not good to be incompatible.

- module-name: 'setuptools.version'
  anti-bloat:
    - description: 'workaround for metadata version of setuptools'
      replacements:
        "pkg_resources.get_distribution('setuptools').version": "repr(__import__('setuptools.version').version.__version__)"

We do not have to include all metadata for setuptools here, just to get that one item, so we chose to make a simple string replacement here, that just looks the value up at compile time and puts it into the source code automatically. That removes the pkg_resources.get_distribution() call entirely.

With that, setuptools_scm was not crashing anymore. That’s good. But we don’t really want it to be included, since it’s good for dynamically detecting the version from git, and what not, but including the framework for building C extensions, not a good idea in the general case. Nuitka therefore said this:

Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'setuptools_scm' (intending to
Nuitka-Plugins:WARNING: avoid 'setuptools') in 'toga' (at
Nuitka-Plugins:WARNING: 'c:\3\Lib\site-packages\toga\__init__.py:99') encountered. It may
Nuitka-Plugins:WARNING: slow down compilation.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at
Nuitka-Plugins:WARNING: https://nuitka.net/info/unwanted-module.html

So that’s informing the user to take action. And in the case of optional imports, i.e. ones where using code will handle the ImportError just fine and work without it, we can use do this.

- module-name: 'toga'
  anti-bloat:
    - description: 'remove setuptools usage'
      no-auto-follow:
        'setuptools_scm': ''
      when: 'not use_setuptools'

He we say, no not automatically follow setuptools_scm reports, unless there is other code that still does it. In that way, the import still happens if some other part of the code imports the module, but only then. We no longer enforce the non-usage of a module here, we just make that decision based on other uses being present.

With this the bloat warning, and the inclusion of setuptools_scm into the compilation is removed, and you always want to make as small as possible and remove those packages that do not contribute anything but overhead, aka bloat.

The next thing discovered was that toga needs the toga-core distribution to version check. For that, we use the common solution, and tell that we want to include the metadata of it, for when toga is part of a compilation.

- module-name: 'toga'
  data-files:
    include-metadata:
      - 'toga-core'

So that moved the entire issue of version looks to resolved.

Step 2 - Dynamic Backend dependency

Now on to the backend issue. What remained was a need for including the platform specific backend. One that can even be overridden by an environment variable. For full compatibility, we invented something new. Typically what we would have done is to create a toga plugin for the following snippet.

- module-name: 'toga.platform'
  variables:
    setup_code: 'import toga.platform'
    declarations:
      'toga_backend_module_name': 'toga.platform.get_platform_factory().__name__'
  anti-bloat:
    - change_function:
        'get_platform_factory': "'importlib.import_module(%r)' % get_variable('toga_backend_module_name')"

There is a whole new thing here, a new feature that was added specifically for this to be easy to do. And with the backend selection being complex and partially dynamic code, we didn’t want to hard code that. So we added support for variables and their use in Nuitka Package Configuration.

The first block variables defines a mapping of expressions in declarations that will be evaluated at compile time given the setup code under setup_code.

This then allows us to have a variable with the name of the backend that toga decides to use. We then change the very complex function get_platform_factory that we used used, for compilation, to be replacement that Nuitka will be able to statically optimize and see the backend as a dependency and use it directly at run time, which is what we want.

Final remarks

I am hoping you will find this very helpful information and will join the effort to make packaging for Python work out of the box. Adding support for toga was a bit more complex, but with the new tool, once identified to be that kind of backend issue, it might have become a lot more easy.

Lessons learned. We should cover packages that we routinely remove from compilation, like setuptools, but e.g. also IPython. This will have to added, such that setuptools_scm cannot cloud the vision to actual issues.

Nuitka this week #14

2024-01-12T00:00:00+00:00

This is a weekly update, or at least it’s supposed to be of what’s going on in Nuitka land, for you to learn about ongoing developments and important changes to the project.

Communication vs. Coding

After GSoC 2019, it seems I dropped off with communication about Nuitka quite a lot, e.g. I stopped “Nuitka This Week”. The reasons are multi-facetted. I think part of the reason is that I was getting busy, part of it clearly also was Corona. But also a more dreadful change in my private life, where the real life Nuitka, my wife, became ill for a long time. Effectively it’s only become really better mid last year.

I think, this caused me to go full into code for Nuitka, and to launch Nuitka commercial, but generally to become more quiet. I have already relaxed this for a bit, e.g. about Python 3.11, I made a bunch of postings.

So this one is a bit general to start off, but I also provide fairly recent details about what I worked on for 2.0 as well.

Nuitka has evolved a lot

From a usability standpoint, ever since I went all in with Nuitka, but also before, the out of the box experience of Nuitka has become ever better. And even 2.0 will take it noticeable further. It’s the premier choice for Python deployment if you want efficiency. Its onefile mode is pretty great already and is continuously getting better.

It’s fair to say that Nuitka was great in 2019. I think in 2023 it became almost amazing for deployment. This is in large part due to working on the Yaml configuration and these things. In 2024 I hope to get it really smooth.

I actually made posts about the Yaml stuff, and I will resume the series shortly, basically it allows people to help improve the deployment side of Nuitka, e.g. missing DLLs and data files, hacks needed, etc. for packages, and it’s quickly becoming better and more complete.

Community

On the Discord server, I have been in touch with users of Nuitka a lot more. You are welcome to join us on the Discord server for Nuitka community where you can hang out with the developers and ask questions. It’s not intended as an interactive manual. You are supposed to read the docs for yourself first.

Optimization Work

I think, it’s in vain to explain what I did for performance in all that time. Mostly, some technical debts for Python3 were collected, extending to advantage in in speed of Nuitka over pure Python again. The advantage compared on Python2 was not as present, and still is not, on Python3, but for 3.10 it’s pretty good.

The major breakthroughs have not happened. But I will be talking about the plans, these sure are exciting. Lots of things are in place, some are not, but I hope to get there.

Current Evolutions

So many things on my mind right now, for one, I guess, 4 plugin changes that I have yet to document in new postings. Two are visible here in this code.

- module-name: 'toga.platform' # checksum: 4db91cac
  variables:
    setup_code: 'import toga.platform'
    declarations:
      'toga_backend_module_name': 'toga.platform.get_platform_factory().__name__'
  anti-bloat:
    - change_function:
        'get_platform_factory': "'importlib.import_module(%r)' % get_variable('toga_backend_module_name')"

First, the checksum. Nuitka is going to warn you about checking your user yaml files for correctness in the future. Since it often finds structural problems, very much needed, since yaml is whitespace sensitive, and you never know what it is: a list, a dict, etc. but the schema we created, can tell.

Second, variables are a new section, and in fact so new, they are not even documented. They can be used to query values from code at compile time. In this case we are using it to get at the backend to use, so we can tell it at runtime. Otherwise, it’s hidden to Nuitka, and could e.g. still be subject to a changed decision from environment variables, something we typically do not want.

For the third and fourth thing, we need another example. Torch can use a JIT to speed up some things, with a compilation very similar to what Nuitka does. That however needs a compiler and the source code on the target platform. Not an easy ask for all kinds of deployments. A new feature makes this easier than before.

- module-name: 'torch' # checksum: 84315db6
  parameters:
    - 'name': 'disable-jit'
      'values': 'value in ("yes", "no")'
  options:
    checks:
      - description: "Torch JIT is disabled by default, make a choice explicit with '--module-parameter=torch-disable-jit=yes|no'"
        support_info: 'parameter'
        when: 'standalone and get_parameter("disable-jit", None) is None'
  import-hacks:
    - force-environment-variables:
        'PYTORCH_JIT': '0'
      when: 'get_parameter("disable-jit", "no" if standalone else "yes") == "yes"'

So, what this does, is to make Nuitka accept parameters. The options part is designed to complain when the default value is used in standalone mode, kind of making the user acknowledge that it’s the intended value. For accelerated mode, we do not disable the JIT, since we can expect to be in the same environment with source code intact.

With get_parameter you get the option value, and can be conditional on it in the when block. That is the 3rd new thing.

The fourth new thing, is the forcing of environment variables. We have so far done this, including in plugins like tk-inter manually with post-load-code. The above is the same, effectively doing os.environ["PYTORCH_JIT"] = "0" if the JIT is to be disabled.

These changes are designed to avoid having to do plugins again. Historically for toga support, we should have created a new plugin, but now it’s not necessary anymore, since the Yaml mechanism can cover retrieval of compile time values from modules. And for torch and the JIT, a plugin would have been needed to provide the command line control for that decision.

This increased power of the Yaml will make it even less often the case that a plugin must be written. But of course docs will have to be added and maybe more places will need to work with variables to make that true even more often. More on that in the future.

Teasers

Future TWN will speak about Nuitka-Python (our own Python fork with incredible capabilities), about Nuitka-Watch (our way of making sure Nuitka works with PyPI packages and hot-fixes to not regress), about compilation reports as a new feature, Windows AV stuff, onefile improvements, and so on and so on. I got interesting stuff for many weeks. Limiting myself for now or I will never publish this.

Twitter and Mastodon

I should be more active there, although often I fail due to not wanting to talk about unfinished things, so actually I do not post there as much.

And lets not forget, having followers make me happy. So do re-tweets. Esp. those, please do them.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Nuitka Release 1.9

2023-12-17T14:25:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has focused on improved startup time and compatibility with lazy loaders which has resulted in some optimization. There are also the usual amounts of bug fixes. For macOS and Linux there are lots of improvements that should make standalone mode for them robust with many more configurations.

Bug Fixes

Nuitka Action: Fix, the parsing code intended for the github action was not working as advertised. Fixed in 1.8.1 already.
Standalone: Follow soundfile change for their DLL names. Fixed in 1.8.1 already.
MSYS: Fix, the recent change to detect their Python flavor with 3.11 was done wrong. Fixed in 1.8.1 already.
Windows: Ignore MS API DLLs found from %PATH%. We only ignored them because they come from the Windows system folder, but if any program has them, then we did include them. Fixed in 1.8.1 already.
Standalone: Fix, calendar is used by time built-in module actually and therefore must be included. Fixed in 1.8.1 already.
Standalone: Added data file for unstructured package. Fixed in 1.8.1 already.
Standalone: Added data file for grpc package. Fixed in 1.8.1 already.
Standalone: Added missing dependency for skimage. Fixed in 1.8.1 already.
Python3.11: The dictionary copy code could crash on special kinds of dictionaries. Fixed in 1.8.2 already.
Standalone: Added data file required by ens of web3 package. Fixed in 1.8.2 already.
Fix, multiprocessing could not access attributes living in __main__ module, but only things elsewhere, breaking minimal examples. Fixed in 1.8.2 already.
Reports: Fix, the license of some packages in case it is UNKNOWN was not handling all the cases that wheels expose. Fixed in 1.8.2 already.
Fix, using --include-module and --include-package was behaving identical for packages. Made the former not include all of the package, but only the top level and what that uses.
Standalone: Added support for the lightning package. Fixed in 1.8.3 already.
Distutils: Fix, the platform tag was sometimes incorrect for wheels built. Fixed in 1.8.3 already.
Compatibility: Make the PySide2/PySide6 signal connection workaround more robust. It was not handling reuse of the same method properly and insisted on changing __name__ which some objects apparently dislike a lot. Fixed in 1.8.4 already.
Windows: Fix, need to use short path for the Python installation prefix, as it might be Unicode path as well. Fixed in 1.8.4 already.
Fix, output spec %NONE% was not compiling anymore. Fixed in 1.8.4 already.
Reports: Avoid having short paths for DLL sources on Windows. Fixed in 1.8.4 already.
Fix, catch provided metadata from command line --include-distribution-metadata without including the package at runtime. Fixed in 1.8.4 already.
Python3.10+: Fix, was not properly initializing indicator variable used in the match re-formulation. The generated code still work, but this was an error on the logical level to use a variable un-initialized. Fixed in 1.8.4 already.
Standalone: Added missing DLLs for rlottie-python. Fixed in 1.8.4 already.
Standalone: Added missing implicit dependencies and also avoid duplication of DLLs for the av package. Fixed in 1.8.4 already.
Fix, was not handling errors when creating distribution objects. Fixed in 1.8.4 already.
macOS: Remove extended attributes from DLLs, e.g. finder can add them and it prevents code signing. Fixed in 1.8.4 already.
macOS: Workaround for signing tkinter data files properly, we just exclude the problematic ones, as they are going to be unused. Fixed in 1.8.4 already.
Standalone: Added hidden dependency of curl_cffi package. Fixed in 1.8.5 already.
Standalone: Added hidden dependency of tensorflow package. Fixed in 1.8.5 already.
Standalone: Added more kivymd data files. Fixed in 1.8.5 already.
Standalone: Added implicit dependency for winloop package. Fixed in 1.8.6 already.
Windows: Fix, do not resolve main program executable filename to long filename. Fixed in 1.8.5 already.
Windows: Fix, ignore ucrtbase runtime DLLs found from %PATH% as well. Fixed in 1.8.6 already.
Compatibility: Fix, the dill-compat plugin was regressed and support for dill version 0.3 was added.
Fix, need to include package name for joblib usage with --python-flag=-m to work properly.
Windows: Added support for newest joblib too, we no longer need to error for using the latest one.
Fix, attribute lookups becoming hard through node factories didn’t annotate possible exceptions.
Standalone: Added support for huggingface_hub vendored lazy loader variant.
Standalone: Added support for datasets module.
Plugins: Handle default plugin of matplotlib a lot better. Be more graceful when the query of the default one fails, and point to MPLBACKEND usage. Otherwise inform the user of the backend used for matplotlib so it can be checked.
Fix, --include-module on a package was including all of it rather than just top level, which was what it should do. For including the full package, there is --include-package instead.
macOS: Fix, need to check dependencies for the selected target arch precisely, otherwise DLLs and extension modules for the other arch can cause errors for our dependency analysis in standalone mode.
- Also added support for getting DLL exported symbols on macOS which then allows to properly distinguish extension modules from mere DLLs on macOS, and not just Linux.
MSYS2: Fix, the GTK DLL name changed again.
Compatibility: Added support for .zip files being in python path as well.
Fix, when a sub-package module import is rejected for whatever reason, the programs attempt to import it, still implies an attempt to import the parent module. For extension modules in accelerated mode, this is of course common, but the containing package if any, is of course still to be included.
Fix, PySide6 in accelerated mode needs workarounds too, previously only standalone mode was avoiding the corruptions it was causing.
Fix, make the PySide2/PySide6 signal connection workaround also fix disconnection only. For signals that only ever got disconnected, but never connected, the workaround was not applied.
MSYS2: For standalone add more GI dlls.
Fix, inline copies of e.g. tqdm could be detected during compilation and even in place of the real package.
Standalone: Added proper support for timm without disabling JIT generally.
Python3.11: Fix, frozen stdlib modules must be turned off

Otherwise the value of os.__file__ becomes wrong, and maybe more issues, as Nuitka is more compatible to full modules than the frozen modules are for standalone mode at least.
Python2: Avoid RuntimeWarning when using inline copy of tqdm.
Standalone: Added support for newer pydantic and its lazy loader.
Standalone: Add config for font data files of qtawesome package.
Plugins: Added workaround for PySide6 enums checking bytecode when some older enum values are used. The PySide6 means to detect method calls vs. type lookups to decide if to inject the default value for a flag value. With our workaround, enums behave as expected without that check being possible.
Standalone: Added support for gradio package.
Python3.6+: Added support for non-latin (for example Chinese) module names, these were not working correctly yet.
Python3: Fix, star importing from modules with non-UTF8 encodable names in the module dictionary crashed.
Python2: Fix, couldn’t list directories with unicode filenames in them, so that e.g. a Python3 created build directory with unicode module names was not possible to fully delete.
Compatibility: Added missing as_posix method to our resource reader objects.
Standalone: Added missing DLLs for PyAutoIt package.
Standalone: Added data files for the flask_restx package, for which --include-package-data also wouldn’t work, due to its strange handling when running in frozen mode.
Standalone: Added more metadata requirements for transformers package.
Standalone: Added support for newer transformers package.
Standalone: Added data files for yapf vendored lib2to3 package.
Plugins: Fix, was crashing on module patterns of submodules not existing in the yaml config implicit dependencies.

New Features

Plugins: Introduce an explicit hard import registry, that now can be expanded at compile time by plugins.
Plugins: Added support for lazy delayed loading, which removes the need for include-pyi-file as we inline its effect at compile time. Also, the dependencies of these kinds of packages no longer need to be overreaching and can analyze the code again. This is using the hard import registry plugin interface.
Linux: Standalone builds with PyPI packages no longer include system DLLs unless a new Yaml configuration for DLLs called package-system-dlls is configured, which will be necessary for GTK bindings probably. With this the included DLLs will more often be only ones suitable for use on other OSes. This should make Linux standalone somewhat easier, but still need to compile on old OS.
Reports: For distributions include the installer name, so we can tell pip, conda or system packages apart better.
Reports: For included modules, we now also attribute the distributions it directly uses modules from and the distribution the module itself belongs to was added as an attribute as well.
Reports: Added excluded module reasons to reporting, so that it can be told directly, which imports were found, but not followed to. Also added report reader capable of providing information from a compilation report.
Added support for FIPS compliance, a US security standard by NIST, that caused parts of Python used by Nuitka to be flagged.
Watch: Added option to control the update mode, handles now rc versions, so it can be used before and after Nuitka releases easily.
Watch: Added timeout for how long programs are allow to run.
Watch: Added ability to recognize fork loops happening, so test cases of e.g. joblib do not suddenly go wild on a break change in that or other packages.
macOS: The --list-package-dlls now needs to check target arch options, so we now delay the non-compiling options execution until it’s set, which also makes it cleaner code. Also, we can now distinguish real Python extension modules from mere DLLs on macOS too.
Standalone: Also ignore av and cv2 DLL collisions, making it more generic.
Standalone: Make tk-inter plugin more robust. Detect the tkinter version used and scan for its paths. Use path used when compiling tcl from source and check data directory paths for tcl and tk for expected files, and error out if they are not found. With these changes self-compiled Python as e.g. used in our commercial Linux container is supported too now.
Enhanced support for self compiled Python by using link libraries needed for static linked extension modules. This allows a better commercial Linux container build mainly.

Optimization

Optimization: Enhanced handling of aliased variables.

Was not converting variable assignments from variables created during re-formulations to the dedicated nodes, potentially missing out on optimizations specific to that case, because it was then not recognized to be non-generic anymore later.

Was not optimizing comparisons and truth checks for temporary variable references, missing out a lot of opportunities for optimization of code coming from re-formulations.

When a variable is aliased, but the source variable is one that cannot escape or is even very hard value, we were not annotating that as well as possible, but now e.g. comparisons with constant values that are immutable are done even if aliased.

Remove knowledge of variables assigned to other variables only if that value can actually escape, otherwise that has no real point.
Use variable length encoding for data blob size values. This removes size constraints in some cases, but also makes the representation of list, tuple, dict more compact, since they commonly have only a few elements, but we used 4 bytes for length, where the average should be close to one 1 byte per length item now.
Faster CRC32 with zlib, leading to much faster program startup, and faster checksums for cached mode of onefile, improving that a lot as well.
Windows: Updated MinGW64 to latest WinLibs package, should produce even faster code and show stopping bugs in its binutils have apparently been fixed. This should now link a lot faster with LTO, due to using multiple processes.
Added support for builtins.open as hard import to open.
Scalability: Make sure we actually use __slots__ for our classes. Variables, code generation context, iteration handles, and type shapes didn’t really use those and that should speed their use up and therefore reduce Python compile time and memory usage.
Standalone: Removed one more automatic stdlib module textwrap as it otherwise uses a runner code with test code that is bloating with hello world code.
Fedora: Enabled LTO linking by default as well, it’s working, but Fedora Python is still not really good to use, since it doesn’t allow static linking of libpython.
Anti-Bloat: Avoid pytest usage in pooch package. Added in 1.8.1 already.
Anti-Bloat: Remove pdb usage from pyparsing package. Added in 1.8.2 already.
Anti-Bloat: Remove unittest usage in bitarray. module. Added in 1.8.2 already.
Anti-Bloat: Avoid lightning to cause use of its lightning.testing framework.
Anti-Bloat: Added override that that torch but only it can use unittest, it will not work otherwise.
Anti-Bloat: Avoid using IPython in gradio package.
Anti-Bloat: Avoid using IPython in altair package.
Anti-Bloat: Avoid using numba in pyqtgraph package.
Anti-Bloat: Avoid using triton in torch package.
Anti-Bloat: Avoid using unittest in multiprocess package.
Anti-Bloat: Avoid using setuptools with new mmcv package as well.
Anti-Bloat: Avoid URLs in numpy messages.
Code Generation: Dedicated helper function for fixed imports, that uses less C code for standard imports.
Standalone: Avoid including libz on Linux.
Quality: Use latest isort and rstchk versions.

Organizational

Python3.12: Mark as unsupported for now, it does not yet compile on the C level again.
User Manual: Added description of deployment mode, this was not documented so far, but for some programs, dealing with them is now required.
User Manual: Improved --include-plugin-directory documentation to make it more clear what it is usable for and what not.
UI: Reject standard library paths for plugin directories given to --include-plugin-directory which is a frequent user error.
UI: When interrupting during Scons build with CTRL-C do not give a Nuitka call stack, there is no point in that one, rather just exit with a message saying the user interrupted the scons build.
UI: Make package data output from --list-package-data more understandable.

We already had a count for DLLs too, and should not list directory name in case it’s empty and has no data files, otherwise this can confuse people.
UI: Make the progress bar react to terminal resizes. This avoids many of the distortions seen in Visual Code that seems to do it a lot.
UI: Added a mnemonic warning for macOS architecture cross compilation, that it will only work as well as Python does when limited to that arch. Read more on the info page for detailed information. Added in 1.8.4 already.
UI: Error exit for wrong/non-existent input files first. Otherwise e.g. complaints about not including anything can be given where project options were intended to solve that.
UI: Enhanced error message in case of not included imageio plugins. Added in 1.8.4 already.
UI: Enhanced messages from options nanny, showing the condition that was not passed.
UI: Improved download experience. When hitting CTRL-C during a download, delete the incomplete file immediately, otherwise it’s causing an error next run. Also added progress for downloads as well, so they do not sit there silent without a way to know how much is remaining.
UI: Also report errors happening during plugin init nicely.
Visual Code: Added ignore paths code spell checking. This only adds the most obvious things, more to come later.
Visual Code: Use environment for C include path configurations, and add one for use on macOS, this cleaned up a lot of inconsistencies in paths for the various existing platforms.
Debugging: Added experimental switch to disable free lists, so memory corruption issues can become easier to debug.
UI: Output clang and gcc versions in --version output as well.
UI: Add hint how to disable the warning message that asks to disable the console during debugging by explicit --enable-console usage.
UI: Do not consider aliases of options for ambiguous option error. Without this --no-progressbar and --no-progress-bar being both accepted, forced long version of options for no good reason.
Debugging: With --debug output failed query command that a plugin made. In this way it is easier to check what is wrong about it for the user already.
UI: Check if metadata included has the actual distribution package included. Otherwise we error out, as this would result in a RuntimeError when the program is attempting to use it.
UI: Harmonized help text quoting. We will also need that in order to generate the help texts for Nuitka-Action in the future. Currently this is not perfect yet.
Debugging: Added trace that allows us to see how long Py_Exit call takes, which might be a while in some cases.
User Manual: Made the Nuitka requirements top level chapter.
User Manual: Added promise to support newer Python versions as soon as possible.
User Manual: Added section about how Linux standalone is hard and needs special care.
Python3.11: Disallow to switch to g++ for too old gcc, with this Python version we have to use C11.
Quality: Remove inconsistencies with C python hex version literals in auto-format, which will also make searching code easier.
UI: More clear error message for asking package data of a module name that is not a package.

Cleanups

Dedicated node for fixed and built-in imports were added, which allow the general import node to be cleaner code.
Scons: Removed remaining win_target mode, this is long obsolete.
Spelling improvements by newer codespell, and generally, partially ported to 1.8.4 already so the Actions pass again.
Plugins: Move python code of dill-compat run time hook to separate file.

Tests

Run the distutils tests on macOS as well, so it’s made sure wheel creation is working there too, which it was though.
Avoid relative URLs in use during pyproject.toml tests, these fail to work on macOS at least.
Add GI GTK/GDK/Cairo standalone test for use with MSYS2. Eventually this should be run inside Nuitka-Watch against MSYS2 on a regular basis, but it doesn’t support this Python flavor yet.
Added test case with Chinese module names and identifiers that exposed issues.
Completed the PGO test case and actually verify it does what we want.
Added standalone test for setuptools. Since our anti-bloat works makes it not compiled with most packages, when it is, make sure it doesn’t expose Nuitka to some sort of issue by explicitly covering it.
Show tracebacks made in report creations on GitHub Actions and during RPM builds.

Summary

This is again massive in terms of new features supported. The lazy loader support is very important as it allows to handle more packages in better ways than just including everything.

The new added optimization are nice, esp. startup time will make a huge difference for many people, but mainly the focus was on supporting packages properly, and getting Nuitka-Watch to be able to detect breaking of packages from PyPI closer to when it happens.

And then of course, there is a tremendous amount of improvements for the UI, with lots features become even more rounded.

For Python 3.12 work has begun, but there is more to do for it. At this time it’s not clear how long it takes to add it. Stay tuned.

Nuitka Release 1.8

2023-09-30T07:59:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

Bug Fixes

Standalone: Added support for opentelemetry package. Added in 1.7.1 already.
Reports: Fix, do not report plugin influence when there are not no-auto-follow in an anti-bloat section. Fixed in 1.7.2 already.
Anti-Bloat: Add missing usage tag use_pytest for anti-bloat changes that remove pytest related codes. Fixed in 1.7.2 already.
Standalone: Added support for newer jsonschema package. Fixed in 1.7.2 already.
Standalone: Fix, our iterdir implementation was crashing in files for packages that don’t actually have a directory for data files to live in. Fixed in 1.7.2 already.
Fix, parent package imports could pick the wrong name internally and then collide with sub-packages of that package during collision. Fixed in 1.7.3 already.
Standalone: Added support for pymssql package. Fixed in 1.7.3 already.
Standalone: Added support for cvxpy package. Fixed in 1.7.4 already.
Standalone: Added missing dependencies of lib2to3.refactor. Fixed in 1.7.4 already.
Standalone: Fix, data files for lib2to3.pgen were regressed. Fixed in 1.7.4 already.
Standalone: Added missing dependency of cairo package. Fixed in 1.7.4 already.
Standalone: Added support for new trio package. Fixed in 1.7.4 already.
Standalone: Added support for markdown package. Fixed in 1.7.4 already.
Standalone: Added support to eventlet package. Fixed in 1.7.4 already.
Standalone: Added support for more newer sklearn package. Fixed in 1.7.5 already.
Standalone: Added support for more newer skimage package. Fixed in 1.7.5 already.
Standalone: Added support for more newer transformers package. Fixed in 1.7.5 already.
Standalone: Added support for torch_scatter package. Fixed in 1.7.6 already.
Standalone: Added missing DLL for wx.html2 to work well on Windows. Fixed in 1.7.6 already.
Fix, the @pyqtSlot decoration could crash the compilation and was effective even if no pyqt plugin was active. Fixed in 1.7.6 already.
Python3.11: Fix, need to support BaseExceptionGroup for code generation too, otherwise the exceptiongroup backport was not working. Fixed in 1.7.7 already.
MSYS2: Fix usage of deprecated sysconfig variable with mingw. After their switch to Python 3.11, it is no longer available. Fixed in 1.7.7 already.
Distutils: Do not compile empty directories found in package scan as namespaces. Fixed in 1.7.7 already.
Python3.7+: Fix, need to follow dict internal structure more correctly, otherwise we over-allocate and copy more data than necessary. Fixed in 1.7.7 already.
Python3.8: Fix, the new pyqt plugin workaround requires 3.9 or higher and could causes compile time crashes with the @pyqtSlot decorator. Fixed in 1.7.7 already.
Modules: Fix, the .pyi file created was using default encoding which can vary and potentially even crash on other systems. Enforcing utf-8 now. Fixed in 1.7.8 already.
Fix, only failed relative imports should become package relative. This was giving wrong names for attempts imports in these cases. Mostly only affected dependency caching correctness and reporting at this time. Fixed in 1.7.8 already.
Standalone: Added missing metadata dependencies for transformers package. Fixed in 1.7.9 already, but more added for release.
Fix, need to ignore folders that cannot be module names in stdlib. Could e.g. crash when encountering folders like .idea which cannot be module names. Fixed in 1.7.9 already.
Standalone: Added data files for langchain package. Fixed in 1.7.10 already.
Fix, forced output paths didn’t work without C11 mode. This mainly affected older MSVC users, with newer MSVC and good enough Windows SDK, it’s not using C++ anymore. Fixed in 1.7.10 already.
Fix, was using int values for boolean returns, something that was giving warnings with at least older MSVC not in C11 mode. Fixed in 1.7.10 already.
Fix, failed hard name imports could crash with segfault trying to release their value. Fixed in 1.7.10 already.
Standalone: Added missing implicit dependency for xml.sax in stdlib. Fixed in 1.7.10 already.
Windows: Fix, --mingw64 mode was not working if MSVC was installed, but not acceptable for use. Fixed in 1.7.10 already.
Standalone: Fix, onnxruntime had too few DLLs included. Fixed in 1.7.10 already.
Standalone: Added support for moviepy. Fixed in 1.7.10 already.
Python3.10+: Fix, matching empty sequences was not considering length, leading to incorrect code execution for that case.
```
match x:
   case []:
         ... # non-empty sequences matched here
```
UI: Fix, some error outputs didn’t work nicely with progress bars, need to use our own print function that temporarily disables them or else outputs get corrupted.
Linux: Sync output for data composer. This is to avoid race conditions that we might have been seeing occasionally.
Compatibility: Fix, the sys.flags.optimize value for --python-flag=-OO didn’t match what Python does.
Standalone: Fix, packages have no __file__ if imported from frozen, these was causing issues for some packages that scan all modules and expect those to be there.
Fix, the dict built-in could crash if its argument self-destructs during usage.
Fix, the PySide2/PySide6 workaround for connecting compiled class methods without crashing were not handling its optional type argument.
Enhanced non-commercial PySide2 support by adding yet another class to be hooked. This was ironically contributed by a commercial user.
Standalone: Added support for newer delvewheel version as used in newest scipy and probably more packages in the future.
Compatibility: The pkgutil.iter_modules function now works without importing the module first. The makes Faker work on Windows as well.
Reports: Detect top level packages even with broken packaging. Some packages will not reveal through installed files or top level what package they are for, and as a result, they cannot be uninstalled, but we need to still be able guess what package they are responsible for, so we go by their PyPI name, which works for tensorflow.
Compatibility: More robust way of allowing iteration of compiled packages via file path.

Rather than pre-populating the cache, we should provide the hook function to check if we are responsible for a given path. With this, the Faker package works on Windows as well now, and probably other packages benefit too. This then works on paths rather than strings, which due to short paths, etc. can be non-unique on Windows easily.
Standalone: Added support for the opencc package.
Compatibility: Fix, import name resolving done for things like six and others should be done as soon as possible, and not just during optimization, or else some imports can become just wrong as a result.
Python3.11: Added support for the new closure keyword only argument in exec built-in.
Standalone: Added support for pythonnet on Linux as well.
Debian: Fix, do not give false alarms for root pip installed packages, they get a similar path component, but are not actually Debian packages of course, this was mostly affecting builds inside containers of course.
Compatibility: Added support for comparing results from our resource reader file interfaces. This is needed for when people want to e.g. sort the the file list.
Python3.6+: Fix, didn’t catch await on module level as a syntax error.
Compatibility: Added support for joblib with loky backend as well.
Standalone: Added support for newer chromadb adding missing dependencies and data files.
Python3.9+: Fix, importlib.resources.files() was not fully compatible

Need to provide basename for .name attribute rather than an absolute path. And in some cases, a leading trailing slashes was produced for the full path, which caused trouble for file iteration of filenames.
Standalone: Added support for newer importlib_resources as well. We now need to expose the files functionality even before Python 3.9 for this to be possible.
Standalone: Added support for newer rapidfuzz package.
Added support for newer PyOpenGL package.

New Features

Plugins: Added support to specify embedding of metadata for given packages via the package configuration. With this, entry points, version, etc. can even be resolved if not currently possible at compile time to so through the code with static optimization. Added in 1.7.1 already.
```
- module-name: 'opentelemetry.propagate'
  data-files:
    include-metadata:
      - 'opentelemetry-api'
```
Distutils: Add PEP 660 editable install support. With this pdm can be used for building wheels with Nuitka compilation. Added in 1.7.8 already.
Haiku: Added support for accelerated mode, standalone will need more work, also recognize its form of the site-packages folder, named vendor-packages.
Disable misleading initial import exception handling in numpy, all what it says detracts only.
Added python flags given for no_asserts, no_docstrings and no_annotations to the __compiled__ attribute values of modules and functions to fully expose the information.
Watch: Added capability to specify what nuitka binary to use in nuitka-watch so we can use enhanced nuitka-watch from develop branch with older versions of Nuitka with no issues.
Watch: Now evaluates the minimum version needed for Nuitka, and skips test cases, allowing nuitka-watch to be run with versions that do not yet handle cases that e.g. develop already can, i.e. next Nuitka version.
Watch: Now evaluates if a compilation with Nuitka needs to be done at all, as it’s only necessary if the PyPI config changed, or if Nuitka version changed.
Reports: Added source path for modules, so it’s easier to tell where something came from, and esp. in case of bugs in the import location of Nuitka.
Reports: In case of a crash, always write report file for use in bug reporting. This is now done even if no report was asked for.
Reports: Include error exit message from Nuitka in case of explicit exits.
UI: Added new --deployment and --no-deployment-flag that disables certain debugging helpers.

Right now, we use this to control a hook that prevents execution of itself with -c which is used by e.g. joblib and that potentially can turns Nuitka created programs into a fork bombs, when they use sys.executable -c .... This can be disabled with --no-deployment-flag=self-execution or --deployment.

The plan is to expand this to cover FileNotFoundError and similar exception exits pointing to compilation issues with helpful more annotations.
Catch attempts to exec compiled function bytecodes.

This segfaults otherwise with at least Python3.11 and is probably a good idea to catch for all versions, as it doesn’t do anything.
Windows: Remove unnecessary .\ in CMD files generated, these will otherwise show up in sys.argv[0] too, making them more ugly than necessary.
Scons: Also respect CFLAGS setting. It’s rarely used, but for completeness sake we should have that too. The effects are the same as CCFLAGS it seems.

Optimization

Added type shape for built-in hash operation, these must indeed be of int type either way.
Anti-Bloat: Avoid using unittest in future and multiprocessing package. Added in 1.7.3 already.
Anti-Bloat: Avoid using unittest in git package. Added in 1.7.3 already.
Anti-Bloat: Avoid IPython in streamlit package.
Standalone: Make transformers work with no_docstrings mode. Added in 1.7.7 already.
Anti-Bloat: Avoid more IPython usage in transformers package.
Anti-Bloat: Avoid using pytest in polyfactory package.
Anti-Bloat: Expand the list of modules that are in the unittest group by the ones Python provides itself, test.support, test.test_support and future.moves.test.support, so the culprits are more easily recognizable.
Anti-Bloat: Treat ipykernel and jupyter_client as equal to IPython for usage, so the bloat warning about IPython becomes more meaningful in that case too.
Anti-Bloat: Avoid using IPython in plumbum package.
Statically optimize the value of sys.byteorder as well.
Anti-Bloat: Added no-auto-follow for tornado in joblib package. The user is informed of that happening if nothing else imports tornado in case he wants to enable it.
Standalone: Avoid including standard library zipapp or calendar automatically and remove their runners through anti-bloat configuration. This got rid of argparse for hello world compilation.
Standalone: Do not auto include standard library json.tool which is a binary only.
Standalone: Avoid automatic inclusion a _json extension module for the json module and do not automatically include it as part of stdlib anymore, this can reduce the size of standalone distributions.
Standalone: Avoid the standard library audioop extension module by making all audio related modules non-automatically included.
Standalone: Avoid the _contextvars standard library extension module. Explicit and implicit imports of contextvar module will continue to work and hopefully give proper errors until we do ourselves raise such errors.
Standalone: Avoid also the “_crypt” standard library extension module, and make the crypt module raise an error where we modify the message to not be as misleading.
Standalone: On macOS we also saw _bisect, _opcode and more modules that are optional extension modules, that we no longer do automatically use if they are that way.
Standalone: Added more modules like mailbox, grp, etc. to exclusion from standard library when they trigger dependencies on other things, or are an extension themselves.
Anti-Bloat: Avoid using sqlalchmy.testing and therefore pytest in sqlalchemy package. Also added that testing package to be treated as using pytest. Added in 1.7.10 already.
Anti-Bloat: Avoid IPython in distributed package. Added in 1.7.10 already.
Anti-Bloat: Avoid dask usage in skimage. Added in 1.7.10 already.
Anti-Bloat: More changes needed for newer sympy to avoid IPython. Added in 1.7.10 already.
Anti-Bloat: Enhanced handling of PIL.ImageQt even without the Qt binding plugins being active.
Anti-Bloat: Do not automatically follow matplotlib from scipy as that is code that will only be used if other code using it exists too.
Anti-Bloat: Avoid pandas and matplotlib for sklearn package. Availability checks of third-party packages should be counted as real usage.
Anti-Bloat: Avoid IPython in newer keras module too.
Anti-Bloat: Updated for newer tensorflow package, also using more robust new form of no-auto-follow to achieve that.
Anti-Bloat: Avoid using Qt bindings for pandas.io.clipboard as it’s only useful if one of our Qt plugins is active.

Organizational

User Manual: Make it clear in the example that renaming created extension modules to change their name does not work, such that the user has to first rename the Python module properly.
macOS: Pronounce Homebrew as somewhat support but not recommended due to its limited results for portability.
UI: Added mnemonic for unsupported Windows store Python, so we have a place to give more information. Read more on the info page for detailed information.
UI: Disable warning for numpy/scipy DLL non-identity conflicts. These are very common unfortunately and known to be harmless.
Stop creating PDFs for release. They are not really needed, but cause extra effort that makes no sense.
Quality: Updated to latest black which removes some leading new lines in blocks, changing a bunch of files. Bumped development requirements file Python version to 3.8, since black won’t do 3.7 anymore.
Quality: Updated to latest PyLint, no changes from that.
Quality: Auto-format the markdown files used for GitHub templates as well.
Debugging: Catch errors during data composer phase cleaner. Added in 1.7.1 already.
Plugins: More clear error messages for Yaml files checker. Added in 1.7.5 already.
Release: Avoid DNS lookup by container, these sometimes failed.
UI: Catch user error of compiling in module mode with unknown file kinds, it needs to be Python code of course.
UI: In case of SyntaxError in main file, always suggest latest supported version. Previous it was toggling between Python2 and Python3, but that’s no longer the main reason this happens.
UI: Fix typo in help output for --trademarks option. Added in 1.7.8 already.
UI: Fix, need to enforce version information completeness only on Windows, other platforms can be more forgiving. Added in 1.7.8 already.
Visual Code: Enable black formatter as default for Python.
UI: Disallow --follow-stdlib with --standalone mode. This is now the default, and just generally makes no sense anymore.
Plugins: Warn if Qt qml plugins are not included, but qml files are. This has been a trap for first time users for a while now, that now have a way of knowing that they need to enable that Qt plugin feature.
Plugins: Enhanced Qt binding plugins selection by the various qt plugins

Now can also ask to not include specified plugins with --noinclude-qt-plugins and by now include sensible by default, with the --include-qt-plugins=qml line not replacing it, but rather extending it. That makes it easier to handle and catches a common trap, where users would only specify the missing plugin, but remove required plugins like platform making it stop to work.
Plugins: Allow plugins provide None for flags not just by return value length, but also an explicit value, so plugin code can make a difference in a consistent way.
UI: Lets have the options-nanny output the failed condition, so it’s more clear what the issue is.
Quality: Unified spell checker markers to same form in all files through auto-format for more consistency.
Quality: Always avoid attempting to format executables, much like we already do for bytecode, otherwise some attempts on them can crash.
Windows: Only change directory to short path during execution of Scons, we are otherwise leaking it to --run execution in tests, giving their output comparison a harder time than necessary.
Scons: Use report paths for outputs of filenames in slow compilation messages as well.
WinPython: Adapted detection of this flavor to changes made in that project.

Cleanups

Major Cleanup, do not treat technical modules special anymore

Previously the immediate demotion of standard library to bytecode is not really needed and prevented dependency analysis. We have had plenty issues with that ever since not all stdlib modules were automatic anymore, there was a risk of missing some of them, just because this analysis was not done.

Moved the import detection code to a dedicated module cleaning up the size of the standalone mechanics, as it also is not exclusive to it.

Adding “reasons” to modules, different from “decision reasons” why something was allowed to be included, these give the technical reason why something is added. This is needed for anti-bloat to be able to ignore stdlib being added only for being frozen.

Now we are correctly annotating why an extension module was included, e.g. is it technical or not, that solves a TODO we had.

Removes a lot of code duplication for reading source and bytecode of modules and the separate handling of uncompiled modules as a category in the module registry is no more necessary.

The detection logic for technical modules itself was apparently not robust and had bugs to be fixed that became visible now, and that make it unclear how it ever worked as well.
Refactor towards unification of statement and expression.

Make sure Make existing statement operations, i.e. use the function intended for them so they are immediately closer to what expressions do, and don’t visit their own children themselves anymore.

Remove checks for expression or statement, we won’t use that anymore, and it’s only costing performance until we merge them.
The caching (currently only used when demoting to bytecode), was not keeping track of distributions attempted to be used, but then being not found. That could have led to errors when using the cached result.
Again some more spelling fixes in code were identified and fixed.
Removed now unused user provided flag from uncompiled module nodes.
Removed 3.3 support from test runner as well.
Avoid potential slur word from one of the tests.

Tests

Sometimes the pickle from cached CPython executions cannot be read due to protocol version differences, then of course it’s also not usable.
Added CPython311 test suite, but it is not yet completely integrated.
Tests: Salvage one test for dateutil from a GSoC 2019 PR, we can use that.

Summary

This is massive in terms of new features supported. The deployment mode being added, provides us with a framework to make new user experience with e.g. the missing data files, much more generous and help them by pointing to the right solution.

The technical debt of immediate bytecode demotion being removed, is huge for reliability of Nuitka. We now really only have to deal with actual hidden dependencies in stdlib, and not just ones caused by us trying to exclude parts of it and missing internal dependencies.

Python 3.11 and Nuitka full support

2023-08-01T00:00:00+01:00

I kind of noticed, that I never really updated this post series to its conclusion. Python 3.11 is now fully supported by Nuitka, that is the TLDR. This has been the case since 1.6, and the 1.7 release improved on it, and 1.8 well is around the corner, so it’s about time to state it now in dedicated form.

So, this post is kind of too late. My excuse is that well, there are very exciting improvements in kind of all areas of Nuitka, and esp. even recently, with long standing fixes for some memory leaks in 1.7 hot fixes. These are usually very intense, adding new packages and fixing old bugs usually, since there are usually very few regressions in Nuitka.

Lessons Learned

On thing I want to do better when 3.12 comes, is to be prepared for the core changes, so work on 3.12 support has already started with the first alpha, with Nuitka having a few commits (but not yet enough) to make it work for basic compilation.

It will be more of a priority to follow CPython closer. It seems core changes are becoming more involved, and it’s better to not fall behind too much. The delay in supporting 3.11 must have hurt Nuitka, since bleeding edge users are also those reporting some import issues long before commercial users of Nuitka even considering the environment.

So, well, maybe what I will be aiming at for a new Nuitka release 1.8 e.g. is to support the current pre-release of CPython. But I also want to make releases more time based, i.e. not delay new features, and sometimes important improvements longer than necessary. We shall see how it goes.

Nuitka Release 1.7

2023-07-15T09:21:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

There release is focused on adding plenty of new features in Nuitka, with the new isolated mode for standalone being headliners, but there are beginnings for including functions as not compiled, and really a lot of new anti-bloat new features for improved handling, and improving user interaction.

Also many packages were improved specifically to use less unnecessary stuff, some of which are commonly used. For some things, e.g. avoiding tkinter, this got also down to polishing modules that have GUI plugins to avoid those if another GUI toolkit is used.

In terms of bug fixes, it’s also a lot, and macOS got again a lot of improvements that solve issues in our dependency detection. But also a long standing corruption for code generation of cell variables of contractions in loops has finally been solved.

Bug Fixes

Python3.11: The MSVC compiler for Windows will not work before 14.3 (Visual Studio 2022) if used in conjunction with Python 3.11, point it out to the user an ignore older versions. Fixed in 1.6.1 already.
Standalone: Added support for the pint package. Fixed in 1.6.1 already.
Standalone: Added missing standard library dependency for statistics. Fixed in 1.6.1 already.
Compatibility: Fix, the transformers auto models were copying invalid bytecode from compiled functions. Added workaround to use compiled function .clone() method. Fixed in 1.6.1 already.
Compatibility: Added workaround for scipy.optimize.cobyla package. Fixed in 1.6.1 already.
Anaconda: Detect Anaconda package from conda install vs. PyPI package from pip install, the specifics should only be applied to those. Adapted our configurations to make the difference. Fixed in 1.6.1 already.
Anaconda: Do not search DLLs for newer shapely versions. Fixed in 1.6.1 already.
Standalone: Add new implicit dependencies for pycrytodome.ECC module. Fixed in 1.6.1 already.
Standalone: Fix tls_client for Linux by not non-Linux DLLs. Fixed in 1.6.1 already.
MacOS: When using --macos-app-name, the executable name of a bundle could become wrong and prevent the launch of the program. Now uses the actual executable name. Fixed in 1.6.1 already.
Multidist: The docs didn’t properly state the option name to use which is --main and also it didn’t show up in help output. Fixed in 1.6.2 already.
Standalone: Added support for polars package. Fixed in 1.6.3 already.
Standalone: Added implicit imports for apscheduler triggers. Fixed in 1.6.3 already.
Standalone: Add data files to AXML parser packages. Added in 1.6.4 already.
Fix, exec nodes didn’t annotate their exception exit. Fixed in 1.6.4 already.
Standalone: Added data files for open_clip package. Fixed in 1.6.4 already.
Standalone: Avoid data files warning with old pendulum package. Fixed in 1.6.4 already.
Standalone: Added implicit dependencies for faker module. Fixed in 1.6.4 already.
Added workaround for opentele exception raising trying to look at the exception frame before its raised. Fixed in 1.6.4 already.
Nuitka-Python: Do not check for unknown built-in modules. Fixed in 1.6.4 already.
Scons: Fix, the total ccache file number given could be wrong. Ignored messages were counted still as compiled, leading to larger sum of files than actually there was. Fixed in 1.6.5 already.
Fix, multiprocessing resource tracker was not properly initialized. On at least macOS this was causing it to work relatively badly, because it could fail to actually use it. Fixed in 1.6.5 already.
Standalone: Added support for cassandra-driver package. Fixed in 1.6.5 already.
Onefile: Have Python process suicide when bootstrap surprisingly died, respecting the provided grace time for shutdown. Fixed in 1.6.5 already.
Plugins: Fix, package versions for at least Ubuntu packages can be broken, such that at least pkg_resources rejects them. Handle that and use fallback to next version detection method. Fixed in 1.6.5 already.
Onefile: Handle SIGTERM and SIGQUIT just like SIGINT on non-Windows. The Python code with see KeyboardInterrupt for all 3 signals, so it’s easier to implement. Previously onefile would exit without cleanup being performed. Fixed in 1.6.5 already.
Standalone: Fix, need to add more implicit dependencies for pydantic because we do no longer include e.g. decimal and uuid automatically.
Standalone: Added missing implicit dependencies for fiona package. Added in 1.6.6 already.
Standalone: Added missing implicit dependencies for rasterio package. Added in 1.6.6 already.
Standalone: Fix, need to add more implicit dependencies for pydantic. Added in 1.6.6 already.
Fix, the data composer used a signed value for encoding constant blob sizes, limiting it needlessly to half the size possible.
Windows: Avoid dependency on API not available on all versions, specifically Windows 7 didn’t work anymore. With this, symlinks are only resolved where they actually exist, and MinGW64 does it too now.
Standalone: Added support for .location attribute for pkg_resources distribution objects.
Anti-Bloat: Avoid using dask and numba in the tsfresh package.
Fix, outline cell variables must be re-initialized on entry. The code would be crashing for for outlines used in a loop, since the cleanup code for these cell variables would release the cell that was created during containing scope setup.
Standalone: Added missing dependency of pygeos package.
Standalone: Added sqlalchemy implicit dependency.
Standalone: Added data files for mnemonic package.
Fix, attribute checks could cause corruption when used on objects that raise exceptions during __getattr__.
Python2: Fix, wasn’t making sure instance attribute lookups were actually only done with str attributes.
macOS: Fix, need to allow versioned DLL dependency from un-versioned DLLs packaged.
Standalone: Added DLLs for rtree package.
Standalone: Added support for newer skimage package.
Standalone: Added support for newer matplotlib package.
Standalone: Fix, our numpy.testing replacement, was lacking a function assert_array_almost_equal used in at least the pytransform3d package.

New Features

Added support for --python-flag=isolated mode. In this mode, packages are not expandable via environment variable provided paths and sys.path is emptied which makes imports from the file system not work.
The options for forcing outputs were renamed to --force-stdout-spec and force-stderr-spec to force output to files and now work on non-Windows as well. They kind of were before, but e.g. %PROGRAM% was not implemented for all OSes yet.
Capturing of all outputs now extends beyond the Python level outputs is now attempting to capture C level outputs as well. These can be traces of Nuitka itself, but also messages from C libraries. On Windows, with MinGW64 this does not work, and it still only captures MinGW64, due to limitations of using different C run-times. With MSVC it works for the compiled program and C, but DLLs can have their own C runtime outputs that are still not caught.
Added new spec value %PROGRAM_BASE% which will avoid the suffix .exe or .bin of binaries that %PROGRAM% will still give.
Plugins: Added ability to query if a package in an Anaconda package or not, with the new is_conda_package() function in Nuitka package configuration. Added in 1.6.1 already.
Plugins: Provide control tags during plugin startup with new interface, such that these become globally visible.
Plugins: Allow to give --include-qt-plugins options of Qt binding plugins to be given multiple times. This is for consistency with other options. These now expand the list of plugins rather than replacing it.
Added experimental code to include functions decorated in certain ways to be included as bytecode. Prepare the inclusion as source code in a similar fashion. This was used to make example PyQt5 code work properly with timers where it doesn’t normally work, but is still in development before it will be generally useful. For that it reacts to @pyqtSlot decorators.
Plugins: Make anti-bloat not warn when bloating modules include their group. This helps when e.g. distributed is going to use dask, then we warn about distributed, but not anymore, when that then uses dask. And that intention to avoid dask is now in the warning given for distributed.
Plugins: Added ability to decide module inclusion based on using module name and not only the used name. This will be super useful to make some imports not count per se for inclusion.
Plugins: Added new no-auto-follow Yaml configuration for anti-bloat, that makes imports from one module not automatically included. That can make optional import removal much easier.
Plugins: Added new function for when clauses, such that it now can be tested if this Python version has a certain built-in name, e.g. when: 'not has_builtin_module("_socket")' will not apply configuration _socket is an extension module rather than built-in. This can be used to avoid unnecessary changes.

Optimization

Optimization: Better hasattr handling. Added ability for generated expression base class to monitor the attribute name for becoming constant and then calling a new abstract method due to auto_compute_handling saying wait_constant:name.
Optimization: Added type shapes for setattr and hasattr built-ins as well as the attribute check node for better code generation.
Optimization: Added dedicated nodes for importlib.resources.files to allow including the used package automatically.
Standalone: Include only platform DLLs for tls_client rather than all DLLs for all platforms. Added in 1.6.1 already.
Anti-Bloat: Avoid including sympy.testing for sympy package. Added in 1.6.3 already.
Anti-Bloat: Avoid IPython in transformers package. Added in 1.6.3 already.
Anti-Bloat: Avoid transformers.testing_util inclusion for transformers package as it will trigger pytest inclusion.
Anti-Bloat: Added missing method to our numpy.testing stub, so it can be used with more packages. Added in 1.6.4 already.
Anti-Bloat: Avoid numba usage from parts of pandas. Added in 1.6.4 already.
Anti-Bloat: Avoid pytest usage in patsy more completely. Added in 1.6.4 already.
Standalone: Added data files needed for pycountry package. Added in 1.6.4 already.
Anti-Bloat: Avoid unittest usage in numpy package. Added in 1.6.4 already.
Anti-Bloat: Avoid using pytest in statsmodels package. Added in 1.6.4 already.
Anti-Bloat: Avoid including PIL.ImageQt when no-qt plugin is used. Added in 1.6.4 already.
Anti-Bloat: Avoid IPython usage in dask. We do not cover bloat with dask allowed well yet, more like this should be added. Added in 1.6.5 already.
Anti-Bloat: Avoid dask via distributed in fsspec package. Added in 1.6.5 already.
Anti-Bloat: Avoid IPython in patsy package. Added in 1.6.5 already.
Anti-Bloat: Avoid setuptools in newer torch as well. Added in 1.6.5 already.
Anti-Bloat: Avoid tkinter inclusion in PIL and matplotlib if another GUI plugin is active. This is using the control tags made available by GUI plugins.
Anti-Bloat: Avoid warning for from unittest import mock imports. These are common, and not considered actual usage of unittest anymore.
Anti-Bloat: Avoid pandas usage in tqdm. This uses the new no-auto-follow feature that will enable the optional integration of tqdm if pandas is included by other means only.
Anti-Bloat: Better method of avoiding socket in email.utils. With changing the source code to delay the import of socket to the only function using it. Socket is now included only if used elsewhere. These changes however, are only done if _socket if is not a built-in module, because only then they really matters. And using a simple --include-module=socket will restore this. This approach is more robust and less invasive.

Organizational

Added run-inside-nuitka-container for use in CI scripts. With this, dependencies of package building and testing from correct system installation should go away.
Release: Add CI container for use with run-inside-nuitka-container to make Debian package releases. This provides a more stable and flexible environment rather than building through ansible maintained environments, since different branches can more easily use different versions, or new features for the container handling.
Release: Use upload tokens rather than PyPI password in uploads, and secure the account with 2FA.
UI: Avoid duplicate warnings for anti-bloat detected imports. In case of from unittest import mock there were 2 warnings given, for unittest and unittest.mock but that is superfluous.
macOS: More beginner friendly version of Apple Python standalone error. They won’t know why it is, and where to get a working Python version, so we explain more and added a download link.
Scons: Consider only 5 minutes slow for a module compilation in backend. Many machines are busy or slow by nature, so don’t warn that much.
GitHub: Actions no longer work (easily) with Python2, so we removed those and need to test it elsewhere.
UI: Output the filename of the XML node dump from --xml as well.
UI: Make --edit-module-code work with onefile outputs as well.
Debugging: Allow yaml condition traceback to go through in --debug mode, so exception causes are visible.
Plugins: Make more clear what is the forbidden module user, such that it is possible to debug it.
UI: Inform user about slow linking, and --lto=no choice in case auto was used. This should make this option more obvious for new users that somehow victim of not defaulting to no, but still having a slow link.
Debugging: Include PDBs for DLLs in unstripped mode already. Previously this was only done for debug mode, but that’s a bit high of a requirement, and we sometimes need to debug where things do not happen in debug mode.
User Manual: Added typical problem with python -m compiled_module execution not working and why that is so.
Debian: Do not include PDF files in packages. These are probably not used that much, but they cause issues at times, that are likely not worth the effort.

Cleanups

Moved OS error reporting as done in onefile binary to common code for easier reuse in plugins.
Moved helper codes for expanding paths and for getting the path to the running executable to file path common code for clearer code structure.
Removed x-bits from files that do not need them. For __main__ files, they are not needed, and for some files they were outright wrong.
Python3.12: Avoid usage of distutils.utils which were using to disable bytecode compilation for things we expect to not work.
Solve TODO and use more modern git command git branch --show-current to detect branch, our CI will have this for sure.
In our Yaml configuration prefer the GUI toolkit control tags, e.g. use_pyside6 over the plugin("pyside6") method.

Tests

Release: Use CI container for linter checks, so different branches can use different versions with less pain involved.
macOS: Allow all system library frameworks to be used, not just a few selected ones, there is many of them and they should all exist on every system. Added in 1.6.1 already.
Made the pendulum test actually useful to cover new and old pendulum actually working properly.

Summary

This release really polished anti-bloat to the point where we now have all the tools needed. Also torch in newest version is now working nicely again with it, and a few rough edges of what we did with 1.6 for not including extension modules were removed. This polishing will go on, but has reached really high levels. More and more people are capable of helping with PRs here.

The optimization work outside of anti-bloat was really minor, with only the two attribute built-in nodes being worked on, and only hasattr seeing real improvements. However, this was more of a structural thing. The wait_constant technique will not get applied more often, but it also will need a wait_all_constant companion, before we can expect scalability improvements.

Restoring Windows 7 is important to many people deploying to old systems, and the like.

However, in the coming release, we need to attack loop tracing. The only bugs currently remaining are related to wrong tracing of items, and it also is a limitation for hard imports to work. So scalability from doing more of the wait_constant work, and from more clever loop tracing shall be the focus of the 1.8 release.

Next Nuitka Live Stream

2023-06-28T00:00:00+01:00

Tomorrow, Thursday 30.06.2023, there will be the forth live stream of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 11-14 CEST and potentially in the evening, maybe 18-20 CEST as well, but it seems I tend to go overtime.

Last time

So, with me starting to stream last year, then I went all in on Nuitka with the commercial offering, and suddenly, there was a lot of things to do, while at the same time, the 3.11 support was harder than ever before to add for a new minor version of Python.

So the last four streams are on my Youtube, these are around 4h videos, always done two on the same day. Lets see how long I last this time.

I honestly forgot where I was last time, but I know that I started a kind of optimization, where iteration over randomly accessible types, gets converted to integer lookups, avoiding an iterator creation and release.

This time

That unfinished optimization, as far as I recall, with automatically unlock optimization in terms of standard library parts becoming smaller, and probably should be on the radar if only because of that.

It may only lack the releases to be converted to something, I will need to check though.

I have been tagging issues with stream tag too, so maybe that is also something to look at.

Not sure yet, what to do. I will definitely start with a quick look at what new toys there are in Nuitka, esp. the Yaml stuff is very nice.

How to Join

There is a dedicated page on the web site which has the details. Spoiler, it’s free and I have no plans for anything that involves a subscription of any kind. Of course, talking of subscription, do also checkout the Nuitka commercial offering. That is a subscription with adds that protect your IP even more than regular Nuitka.

Join me

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

Nuitka Release 1.6

2023-06-09T15:41:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release bumps the much awaited 3.11 support to full level. This means Nuitka is now expected to behave identical to CPython3.11 for the largest part.

There is plenty of new features in Nuitka, e.g. a new testing approach with reproducible compilation reports, support for including the metadata if an distribution, and more.

In terms of bug fixes, it’s also huge, and esp. macOS got a lot of improvements that solve issues with prominent packages in our dependency detection. And then for PySide we found a corruption issue, that got workarounds.

Bug Fixes

The new dict in optimization was compile time crashing on code where the dictionary shaped value checked for a key was actually an conditional expression
```
# Was crashing
"value" in some_dict if condition else other_dict
```
Fixed in 1.5.1 already.
Standalone: Added support for openvino. This also required to make sure to keep used DLLs and their dependencies in the same folder. Before they were put on the top level. Fixed in 1.5.1 already.
Android: Convert RPATH to RUNPATH such that standalone binaries need no LD_LIBRARY_PATH guidance anymore. Fixed in 1.5.1 already.
Standalone: Added support for newer skimage. Fixed in 1.5.1 already.
Standalone: Fix, new data file type .json needed to be added to the list of extensions used for the Qt plugin bindings. Fixed in 1.5.2 already.
Standalone: Fix, the nuitka_types_patch module using during startup was released, which can have bad effects. Fixed in 1.5.2 already.
Android: More reliable detection of the Android based Python Flavor. Fixed in 1.5.2 already.
Standalone: Added data files for pytorch_lightning and lightning_fabric packages. Added in 1.5.2 already.
Windows: Fix, the preservation of PATH didn’t work on systems where this could lead to encoding issues due to reading a MBCS value and writing it as a unicode string. We now read and write the environment value as unicode both. Fixed in 1.5.3 already.
Plugins: Fix, the scons report values were not available in case of removed --remove-output deleting it before use. It is now read in case if will be used. Fixed in 1.5.3 already.
Python3.11: Added support for ExceptionGroup built-in type. Fixed in 1.5.4 already.
Anaconda: Fix, using numpy in a virtualenv and not from conda package was crashing. Fixed in 1.5.4 already.
Standalone: Added support for setuptools. Due to the anti-bloat work, we didn’t notice that if that was not sufficiently usable, the compiled result was not usable. Fixed in 1.5.4 already.
Distutils: Added support for pyproject with src folders. This supports now tool.setuptools.packages.find with a where value with pyproject files, where it typically is used like this:
```
[tool.setuptools.packages.find]
where = ["src"]
```
Windows: Fix, the nuitka-run batch file was not working. Fixed in 1.5.4 already.
Standalone: Add pymoo implicit dependencies. Fixed in 1.5.5 already.
macOS: Avoid deprecated API, this should fix newer Xcode being used. Fixed in 1.5.5 already.
Fix, the multiprocessing in spawn mode didn’t handle relative paths that become invalid after process start. Fixed in 1.5.5 already.
Fix, spec %CACHE_DIR% was not given the correct folder on non-Windows. Fixed in 1.5.5 already.
Fix, special float values like nan and inf didn’t properly generate code for C values. Fixed in 1.5.5 already.
Standalone: Add missing DLL for onnxruntime on Linux too. Fixed in 1.5.5 already.
UI: Fix, illegal python flags value could enable site mode. by mistake and were not caught. Fixed in 1.5.6 already.
Windows: Fix, user names with spaces failed with MinGW64 during linking. Fixed in 1.5.6 already.
Linux: Fix, was not excluding all libraries from glibc, which could cause crashes on newer systems. Fixed in 1.5.6 already.
Windows: Fix, could still pickup SxS libraries distributed by other software when found in PATH. Fixed in 1.5.6 already.
Windows: Fix, do not use cache DLL dependencies if one the files listed there went missing. Fixed in 1.5.6 already.
Onefile: Reject path spec that points to a system folder. We do not want to delete those when cleaning up clearly. Added in 1.5.6 already.
Plugins: Fix, the dill-compat was broken by code object changes. Fixed in 1.5.6 already.
Standalone: Added workaround for networkx decorator issues. Fixed in 1.5.7 already.
Standalone: Added workaround for PySide6 problem with disconnecting signals from methods. Fixed in 1.5.7 already.
Standalone: Added workaround for PySide2 problem with disconnecting signals.
Fix, need to make sure the yaml package is located absolutely or else case insensitive file systems can confuse things. Fixed in 1.5.7 already.
Standalone: Fix, extra scan paths were not considered in caching of module imports, breaking the feature in many cases. Fixed in 1.5.7 already.
Windows: Fix, avoid system installed appdirs package as it is frequently broken. Fixed in 1.5.7 already.
Standalone: The bytecode cache check needs to handle re-checking relative imports found in the cache better. Otherwise some standard library modules were always recompiled due to apparent import changes. Fixed in 1.5.7 already.
Nuitka-Python: Fix, do not insist on PYTHONHOME making it to os.environ in order to delete it again. Fixed in 1.5.7 already.
Nuitka-Python: Allow builtin modules of all names. This is of course what it does. Fixed in 1.5.7 already.
Nuitka-Python: Ignore empty extension module suffix. Was confusing Nuitka to consider every file an extension module potentially. Fixed in 1.5.7 already.
Plugins: Properly merge code coming from distinct plugins. The __future__ imports need to be moved to the start. Added in 1.5.7 already.
Standalone: Added support for opentele package. Fixed in 1.5.7 already.
Standalone: Added support for newer pandas and pyarrow usage. Fixed in 1.5.7 already.
Standalone: Added missing implicit dependency for PySide6. Fixed in 1.5.7 already.
Fix, the pyi-file parser didn’t handle doc strings, and could be crash for comment contents not conforming to be import statement code. Fixed in 1.5.8 already.
Standalone: Added support for pyqtlet2 data files.
Python2: Fix, PermissionError doesn’t exist on that version, which could lead to issues with retries for locked files e.g. but was also observed with symlinks.
Plugins: Recognize the error given by with upx if a file is already compressed.
Fix, so called “fixed” imports were not properly tracking their use, such that they then didn’t show up in reports, and didn’t cause dependencies on the module, which could e.g. impact importlib to not be included even if still being used.
Windows: Fix, retries for payload attachment were crashing when maximum number of retries were reached. Using the common code for retries solves that, since that code handles it just fine.
Standalone: Added support for the av module.
Distutils: Fix, should build from files in build folder rather than source files. This allows tools like versioneer that integrate with setuptools to do their thing, and get the result of that to compilation rather than the original source files.
Standalone: Added support for the Equation module.
Windows/macOS: Avoid problems with case insensitive file systems. The nuitka.Constants module and nuitka.constants package could collide, so we now avoid that package, there was only what is now nuitka.Serialization in there anyway. Also similar problem with nuitka.utils.Json and json standard library module.
Standalone: Added support transformers package.
Standalone: Fix for PyQt5 which needs a directory to exist.
macOS: Fix, was crashing with PyQt6 in standalone mode when trying to register plugins to non-default path. We now try to skip the need, which also makes it work.
Fix, recursion error for complex code that doesn’t happen in ast module, but during conversion of the node tree it gives to our own tree, were not handled, and crashed with RecursionError. This is now also handled, just like the error from ast.
Standalone: Added support for sqlfluff.
Standalone: Added support for PySide 6.5 on macOS solving DLL dependency issues.
Scons: Recognize more ccache outputs properly, their logging changed and provided irrelevant states, and ones not associated so far.
Onefile: Fix, could do random exit codes when failing to fork for whatever reason.
Standalone: Added support for pysnmp package.
Standalone: Added support for torchaudio and tensorflow on macOS. These contain broken DLL dependencies as relative paths, that are apparently ignored by macOS, so we do that too now.
Onefile: Use actual rather than guessed standalone binary name for multiprocessing spawns. Without this, a renamed onefile binary, didn’t work.
Fix, side effect nodes, that are typically created when an expression raises, were use in optimization contexts, where they do not work.
Standalone: Added missing implicit dependency for sentence_transformers package.
macOS: Fix, added missing dependency for platform module.

New Features

Support for Python 3.11 is finally there. This took very long, because there were way more core changes than with previous releases. Nuitka integrates close to that core, and is as such very affected by this. Also a lot of missed opportunities to improve 3.7 or higher, 3.9 or higher, and 3.10 or higher were implemented right away, as they were discovered on the way. Those had core changes not yet taken advantage of and as a result got faster with Nuitka too.
Reports: Added option --report-diffable to make the XML report created with --report become usable for comparison across different machine installations, users compiling, etc. so it can be used to compare versions of Nuitka and versions of packages being compiled for changes. Also avoid short names in reports, and resolve them back to long names, so they become more portable too.
Reports: Added option to provide custom data from the user. We use it in out testing to record the pipenv state used with things like --report-user-provided=pipenv-lock-hash=64a5e4 with this data ending up inside of reports, where tools like the new testing tool nuitka-watch can use it to decide if upstream packages changed or not. These are free form, just needs to fit XML rules.
Plugins: Added include-pyi-file flag to data-files section. If provided, the .pyi file belonging to a specific module is included. Some packages, e.g. skimage depend at runtime on them. For data file options and configuration, these files are excluded, but this is now the way to force their inclusion. Added in 1.5.1 already.
Compatibility: Added support for including distribution metadata with new option --include-distribution-metadata.

This allows generic walks over distributions and their entry points to succeed, as well as version checks with the metadata packages that are not compile time optimized.
Distutils: Handle extension modules in build tasks. Also recognize if we built it ourselves, in which case we remove it for rebuild. Added in 1.5.7 already.
Linux: Detect DLL like filenames that are Python extension modules, and ignore them when listing DLLs of a package with --list-package-dlls option. So far, this was a manual task to figure out actual DLLs. This will of course improve the Yaml package configuration tooling .
Onefile: Allow forcing to use no compression for the onefile payload, useful for debugging, to avoid long compression times and for test coverage of the rare case of not compressing if the bootstrap handles that correctly too.
Need to resolve symlinks that were used to call the application binary in some places on macOS at least. We therefore implemented the previously experimental and Windows only feature for all platforms.
Standalone: Added support including symlinks on non-Windows in standalone distribution, if they still point to a path that is inside the distribution. This can save a bunch of disk space used for some packages that e.g. distribute DLL links on Linux.
Onefile: Added support for including symlinks from the standalone distribution as such on non-Windows. Previously they were resolved to complete copies.
UI: Respect code suffixes in package data patterns. With this e.g. --include-package-data=package_name:*.py is doing what you say, even if of course, that might not be working.
UI: Added option --edit-module-code option.

To avoid manually locating code to open it in Visual Code replaced old find-module helper to be a main Nuitka option, where it is more accessible. This also goes beyond it it, such that it resolves standalone file paths to module names to make debugging easier, and that it opens the file right away.
Standalone: Added support for handling missing DLLs. Needed for macOS PySide6.5.0 from PyPI, which contains DLL references that are broken. With this feature, we can exclude DLLs that wouldn’t work anyway.

Optimization

Anti-Bloat: Remove IPython usage in huggingface_hub package versions. Added in 1.5.2 already.
Anti-Bloat: Avoid IPython usage in tokenizers module. Added in 1.5.4 already.
Added support for module type as a constant value. We want to add all types we have shapes for to allow better type(x) optimization. This is only the start.
Onefile: During payload unpacking the memory mapped data was copied to an input buffer. Removing that avoids memory copying and reduces usage.
Onefile: Avoid repeated directory creations. Without it, the bootstrap was creating already existing directories up to the root over and over, making many unnecessary file system checks. Added in 1.5.5 already.
Anti-Bloat: Remove usage of IPython in trio package. Added in 1.5.6 already.
Onefile: Use resource for payload on Win32 rather than overlay. This integrates better with signatures, removing the need to check for original file size. Changed in 1.5.6 already.
Onefile: Avoid using zstd input buffer, but using the memory mapped contents directly avoiding to copy uncompressed payload data. Changed in 1.5.6 already.
Onefile: Avoid double slashes in expanded onefile temp spec paths, they are just ugly.
Anti-Bloat: Remove usage of pytest and IPython for some packages used by newer torch. Added in 1.5.7 already.
Anti-Bloat: Avoid triton to use setuptools. Added in 1.5.7 already.
Anti-Bloat: Avoid pytest in newer networkx package. Added in 1.5.7 already.
Prepare optimization for more built-in types with experimental code, but we need to disable it for now as it requires more completeness in code generation to cover them all. We did some, e.g. module type, but many more will be missing still.
Prepare optimization of class selection at compile time, by having a helper function rather than a dedicated node. This work is not complete though, and cannot be activated yet.
Windows: Cache short path name resolutions. Esp. for reporting, we now do a lot more of these than before, and this avoids they can become too time consuming.
Faster constant value handling for float value checks by avoiding module lookups per value.
Minimize size for hello world distribution such that no unused extension modules are included, by excluding even more modules and using modules from automatic inclusion of standard library.
Anti-Bloat: Catch pytest namespaces py and _pytest sooner, to point to the actual uses more directly.
Anti-Bloat: Usage of doctest equals usage of “unittest” so cover it too, to point to the actual uses more directly.
Ever more spelling fixes in code and tests were identified and fixed.
Make sure side effect nodes indicate properly that they are raising, allowing exceptions to fully bubble up. This should lead to more dead code being recognized as such.

Organizational

GitHub: Added marketplace action designed to cross platform build with Nuitka on GitHub directly. Usable with both standard and commercial Nuitka versions, and pronouncing it as officially supported.

Check out out at Nuitka-Action repository.
Windows: When MSVC doesn’t have WindowsSDK, just don’t use it, and proceed, to e.g. allow fallback to WinLibs gcc.
User Manual: The code to update benchmark numbers as giving was actually wrong. Fixed in 1.5.1 already.
UI: Make it clear that partially supported versions are considered experimental, not unsupported. Fixed in 1.5.2 already.
Plugins: Do not list deprecated plugins with plugin-list, they do not have any effect, but listing them, makes people use them still. Fixed in 1.5.4 already.
Plugins: Make sure all plugins have descriptions. Some didn’t have any yet, and sometimes the wording was improved. Fixed in 1.5.4 already.
UI: Accept y as a shortcut for yes in prompts. Added in 1.5.5 already.
Reports: Make sure the DLL dependencies for Linux are in a stable order. Added in 1.5.6 already.
Plugins: Check for latest fixes in PySide6. Added in 1.5.6 already.
Windows XP: For Python3.4 make using Python2 scons work again, we cannot have 3.5 or higher there. Added in 1.5.6 already.
Quality: Updated to latest PyLint. With Python 3.11 the older one, was not really working, and it was about time. Due to its many changes, we included it in the hotfix, so those can still be done. Changed in 1.5.7 already.
Release: Avoid broken requires.txt in source distribution. This apparently breaks poetry. Changed in 1.5.7 already.
GitHub: Enhanced issue template for more clarity, esp. to avoid unnecessary options, e.g. using --onefile for issues that show up with --standalone already, to report factory branch issues rather on Discord, and give a quick tip for a likely reproducer if a package fails to import.
User Manual: Added instructions on how to add a DLL or executable to a standalone distribution.
User Manual: Example paths in the table for path specs, meant for Windows were not properly escaping the backslashes and therefore rendered incorrectly.
Visual Code: Python3.11 is now the default configuration for C code editing.
Developer Manual: Updated descriptions for adding test suite. While added the Python 3.11 test suite, these instructions were further improved.
Debugging: Make it easier to fully deactivate free lists. Now only need to set max size to 0 and the free list will not be used.
Debugging: Added more assertions, added corrections to feature disables, check args after function calls for validity, check more types to be as expected.
Plugins: Enhanced plugin error messages generally, with --debug exceptions become warning messages with the original exception being raised instead, making debugging during development much easier.
UI: Make it clear what not using ccache actually means. Not everybody is familiar with the design of Nuitka there or what the tool can actually do.
UI: Do not warn about not found distributions but merely inform of them.

Since Nuitka is fully compatible with these, no need to consider those a warning, for some packages they also are given really a lot.
UI: Catch user error of wrong cases plugin names

This now points out the proper name rather than denying the existence outright. We do not want to accept wrong case names silently.

Cleanups

Use proper API for setting PyConfig values during interpreter initialization. There is otherwise always the risk of crashes, should these values change during runtime. Fixed in 1.5.2 already.
For our reformulations have a helper function that build release statements for multiple variables at once. This removed a bunch of repetitve code from re-formulations.
Move the pyi-file parser code out of the module nodes and to source handling, where it is more closely related.

Tests

Adding a nuitka-watch tool, which is still experimental and for use with the Nuitka-Watch repository.
Refined macOS standalone exceptions further to cover more normal usages of files on that OS and for frameworks that applications typically use from the system.
Detect and consider onefile mode if given in project options as well.
Was not really applying import check in programs tests. Added in 1.5.6 already.
Added coverage of testing the signing of Windows binaries with the commercial plugin.
Added coverage of version information to hello world onefile test, so we can use it for Virus tools checks.
Added tests to cover PyQt6 and PySide6 plugin availability, we so far only had that for PyQt5, which is of course not relevant, and totally different code anyway.
Cleanup distutils tests case to use common test case scanning. We now decide version skips based on names, and had to get away from number suffixes, so they are now in the middle.

Summary

The class bodies optimization has made some progress in this release, going to a re-formulation of the metaclass selection, so as to allow its future optimization. We are not yet at “compiled objects”, but this is a promising road. We need to make some optimization improvements for inlining constant value calls, then this can become really important, but by itself these changes do not yield a lot of improvement.

For macOS again a bunch of time was spent to improve and complete the detection of DLL dependencies. More corner cases are covered now and more packages just work fine as a result.

The most important is to become Python3.11 compatible, even if attribute lookups, and other things, and not yet optimized. We will get to that in future releases. For now, compatibility is the first step to take.

For GitHub users, the Nuitka-Action will be interesting. But it’s still in develop. We keep adding missing options of Nuitka for a while it seems, but for most people it should be usable already.

The new nuitka-watch ability, should allow us to detect breaking PyPI releases, that need a new tweak in Nuitka sooner. But it will probably grow in the coming releases to full value only. For now the tool itself is not yet finished.

From here, a few open ends in the CPython 3.11 test suite will have to be addressed, and maybe some of the performance tricks that it now will enable, e.g. with repeated attribute lookups.

Nuitka Release 1.5

2023-03-26T19:38:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains the long awaited 3.11 support, even if only on an experimental level. This means where 3.10 code is used, it is expected to work equally well, but the Python 3.11 specific new features have yet been done.

There is plenty of new features in Nuitka, e.g. much enhanced reports, Windows ARM native compilation support, and the usual slew of anti-bloat updates, and newly supported packages.

Bug Fixes

Standalone: Added implicit dependencies for charset_normalizer package. Fixed in 1.4.1 already.
Standalone: Added platform DLLs for sounddevice package. Fixed in 1.4.1 already.
Plugins: The info from Qt bindings about other Qt bindings being suppressed for import, was spawning multiple lines, breaking tests. Merged to a single line until we do text wrap for info messages as well. Fixed in 1.4.1 already.
Plugins: Fix removeDllDependencies was broken and could not longer be used to remove DLLs from inclusion. Fixed in 1.4.1 already.
Fix, assigning methods of lists and calling them that way could crash at runtime. The same was true of dict methods, but had never been observed. Fixed in 1.4.2 already.
Standalone: Added DLL dependencies for onnxruntime. Fixed in 1.4.2 already.
Standalone: Added implicit dependencies for textual package. Fixed in 1.4.2 already.
Fix, boolean tests of lists could be optimized to wrong result when list methods got recognized, due to not annotating the escape during that pass properly. Fixed in 1.4.3 already.
Standalone: Added missing implicit dependency of apsw. Fixed in 1.4.3 already.

Note

Currently apsw only works with manual workarounds and only in limited ways, there is an import level incompatible with __init__ being an extension module, that Nuitka does not yet handle.
Python3: Fix, for range arguments that fail to divide there difference, the code would have crashed. Fixed in 1.4.3 already.
Standalone: Fix, added support for newer pkg_resources with another vendored package. Fixed in 1.4.4 already.
Standalone: Fix, added support for newer shapely 2.0 versions. Fixed in 1.4.4 already.
Plugins: Fix, some yaml package configurations with DLLs by code didn’t work anymore, notably old shapely 1.7.x versions were affected. Fixed in 1.4.4 already.
Fix, for onefile final result the “–output-dir” option was ignored. Fixed in 1.4.4 already.
Standalone: Added mozilla-ca package data file. Fixed in 1.4.4 already.
Standalone: Fix, added missing implicit dependency for newer gevent. Fixed in 1.4.4 already.
Scons: Accept an installed Python 3.11 for Scons execution as well. Fixed in 1.4.4 already.
Python3.7: Some importlib.resource nodes asserted against use in 3.7, expecting it to be 3.8 or higher, but this interface is present in 3.7 already. Fixed in 1.4.5 already.
Standalone: Fix, Python DLLs installed to the Windows system folder were not included, causing the result to be not portable. Fixed in 1.4.5 already.
Python3.9+: Fix, metadata.resources files method joinpath is some contexts is expected to accept variable number of arguments. Fixed in 1.4.5 already.
Standalone: Workaround for customtkinter data files on non-Windows. Fixed in 1.4.5 already.
Standalone: Added support for overrides package. Fixed in 1.4.6 already.
Standalone: Added data files for strawberry package. Fixed in 1.4.7 already.
Fix, anti-bloat plugin caused crashes when attempting to warn about packages coming from --include-package by the user. Fixed in 1.4.7 already.
Windows: Fix, main program filenames with an extra dot apart from the .py suffix, had the part beyond that wrongly trimmed. Fixed in 1.4.7 already.
Fix, list methods didn’t properly annotated value escape during their optimization, which could lead to wrong optimization for boolean tests. Fixed in 1.4.7 already.
Standalone: Added support for imagej, scyjava, jpype packages. Fixed in 1.4.8 already.
Fix, using --include-package on extension module names was not working. Fixed in 1.4.8 already.
Standalone: Added support for tensorflow.keras namespace as well.
Distutils: Fix namespace packages were not including their contained modules properly with regards to __file__ properties, making relative file access impossible.
Onefile: On Windows the onefile binary did lock itself, which could fail with certain types of AV software. This is now avoided.
Accessing files using the top level metadata.resources files object was not working properly, this is now supported too.
MSYS2: Make sure mixing POSIX and Windows slashes causes no issues by hard-coding the onefile archive to use the subsystem slash rather than what MSYS prefers to use internally.
Standalone: Added missing dependencies of newer imageio.
Fix, side effect nodes didn’t annotate their non-exception raising nature properly, if that was the case.

New Features

Added experimental support for Python 3.11, for 3.10 language level code it should be fully usable, but the CPython311 test suite has not even been started to check newly added or changed features.
Windows: Support for native Python on Windows ARM64, which needs 3.11 or higher, but standalone and therefore onefile do not yet work, due to lack of any form of binary dependency analysis tool.

This platform is relatively new in Python and generally. For the time being standalone and onefile should be done with Intel based Python, they would also be ARM64 only, whereas 32/64 Bit binaries can be run on all Windows ARM platforms.
Reports: Write compilation report even in case of Nuitka being interrupted or crashing. This then includes the exception, and a status like completed or interrupted. At this time this happens only when --report= was specified, but in the future we will likely write one in case of Nuitka crashes.
Reports: Now the details of the used Python version, its flavor, the OS and the architecture are included. This is crucial information for analysis and can make --version output unnecessary.
Reports: License reports now handle UNKNOWN license by falling back to checking the classifiers, and therefore include the correct license e.g. with setuptools. Also in case no license text is found, do not create an empty block. Added in 1.4.4 already.
Reports: In case the distribution name and the contained package names differ, output the list of packages included from a distribution. Added in 1.4.4 already.
Reports: Include data file sizes in report. Added in 1.4.7 already.
Reports: Include memory usage into the compilation report as well.
macOS: Add support for downloading ccache on arm64 (M1/M2) too. Added in 1.4.4 already.
UI: Allow --output-filename for standalone mode again. Added in 1.4.3 already.
Standalone: Improved isolation with Python 3.8 or higher. Using new init mechanisms of Python, we now achieve that the scan for pyvenv.cfg on in current directory and above is not done, using it will be unwanted.
Python2: Expose __loader__ for modules and register with pkg_resources too which expects these to be present for custom resource handling.
Python3.9+: The metadata.resources files objects method iterdir was not implemented yet. Fixed in 1.4.5 already.
Python3.9+: The metadata.resources files objects method absolute was not implemented yet.
Added experimental ability to create virtualenv from an existing compilation report with new --create-environment-from-report option. It attempts to create a requirements file with the used packages and their versions. However, sometimes it seems not to be possible to due to conflicts.

Optimization

Onefile: Use memory mapping for calculating the checksum of files on all platforms. This is faster and simpler code. So far it had only be done this way on Windows, but other platforms also benefit a lot from it.
Onefile: Use memory mapping for accessing the payload rather than file operations. This avoids differences to macOS payload handling and is much faster too.
Anti-Bloat: Avoid using dask in joblib.

Note

Newer versions of joblib do not currently work yet due to their own form of multiprocessing spawn not being supported yet.
Anti-Bloat: Adapt for newer pandas package.
Anti-Bloat: Remove more IPython usages in newer tensorflow.
Use dedicated class bodies for Python2 and Python3, with the former has a static dict type shape, and with Python3 this needs to be traced in order to tell what the meta class put in there.
Compile time optimize dict in/not in and dict.has_key operations statically where the keys of a dict are known. As a result, the class declarations of Python3 no longer created code for both branches, the one with metaclass = in the class declaration and without. That means also a big scalability improvement.
For the Python3 class bodies, the usage of locals() was not recognized as not locally escaping all the variables, leading to variable traces where each class variable was marked as escaped for no good reason.
Added support for dict.fromkeys method, making the code generation understand and handle static methods as well.
Added support for os.listdir and os.path.basename. Added in 1.4.5 already for use in implementing the iterdir method, but they are also now optimized by themselves.
Added support for trusted constant values of the os module. These are curdir, pardir, sep, extsep, altsep, pathsep, linesep which may enable some minor compile time optimization to happen and completes this aspect of the os module.
Faster digit size checks during float code generation for better compile time performance.
Faster list operations due to using PyList_CheckExact everywhere this is applicable, this mostly makes debug operations faster, but also deep copying list values, or extending lists with iterables, etc.
Optimization: Collect module usages of the given module during its abstract execution. This avoids a full tree visit afterwards only to find them. It is much cheaper to collect them while we go over the tree. This enhances the scalability of large compilations by ca. 5%.
Optimization: Faster determination of loop variables. Rather than using a generic visitor, we use the children having generator codes to add traversal code that emits relevant variables to the user directly.
Cache extra search paths in order to avoid repeated directory operations as these are known to be slow at times.
Standalone: Do not include py.typed data files, these indicator files are for IDEs, but not needed at run time ever.
Make sure that the generic attribute code optimization is also effective in cases where a Python DLL is used. Previously this was only guaranteed to be used with static libpython.
Faster list constant usage

Small immutable constants get their own code that is much faster for small sizes.

Medium sized lists get code that just is hinted the size, but takes items from a source list, still a lot faster.

For repeated lists where all elements are the same, we use a dedicated helper for all sizes, that is even faster except for small ones with LTO enabled, where the C compiler may already do that effectively.
Added optimization for os.path.abspath and os.path.isabs which of course have not as much potential for compile time optimization, but we needed them for providing .absolute() for the meta path loader files implementation.
Faster class dictionary propagation decision. Instead of checking for trace types, let the trace object decide. Also abort immediately on first inhibit, rather than checking all variables. This improves Python2 compile time, and Python3 where this code is now starting to get used when the class dictionary is shown to have dict type.
Specialize type method __prepare__ which is used in the Python3 re-formation of class bodies to initialize the class dictionary. Where the metaclass is resolved, we can use this to decide that the standard empty dictionary is used statically, enabling class dictionary propagation for best scalability.

At this time this only happens with classes without bases, but we expect to soon do this with all compile time known base classes. At this time, these optimization to become effective, we need to optimize meta class selection from bases classes, as well as modification of base classes with __mro_entries__ methods.
The bool built-in on boolean values is now optimized away.

Since it’s used also for conditions being extracted, this is actually somewhat relevant, since it could keep code alive in side effects at least for no good reason and this allows a proper reduction.

Organizational

Project: Require the useful stuff for installation of Nuitka already. These are things we cannot inline really, but otherwise will frequently be warned about, e.g. zstandard for onefile and ordered-set for fast operation, but we do not require packages that might fail to install.
User Manual: Added section about virus scanners and how to avoid false reports.
User Manual: Enhanced description for plugin module loading, the old code was too complicated and actually working only for a mode of including plugin code that is discouraged.
User Manual: Fix section for standalone finding files on wrong level.
Windows: Using the console on Python 3.4 to 3.7 is not working very well with e.g. many Asian systems. Nuitka fails to setup the encoding for stdin and stdout or this platform. It can then produce exceptions on input or output of unicode data, that doesn’t overlap with UTF-8.

We now inform the user of these older Python with a warning and mnemonic, to either disable the console or to upgrade to Python 3.8 or higher, which normally won’t be much of an issue for most users.

Read more on the info page for detailed information. Added in 1.4.1 already.
Debugging: Fixup debugging reference count output with Python3.4. For Python 3.11 compatibility tests, actually it was useful to compare with a version that doesn’t have coroutines yet. Never tell me, supporting old versions is not good.
Deprecating support for Python 3.3, there is no apparent use of this version, and it has gained specific bugs, that are indeed not worth our time. Python 2.6 and Python 2.7 will continue to be supported probably indefinitely.
Recommend ordered-set for Python 3.7 to 3.9 as well, as not only for 3.10+ because on Windows, to install orderset MSVC needs to be installed, whereas ordered-set has a wheel for ready use.
Actually zstandard requirement is for a minimal version, added that to the requirement files.
Debugging: Lets not reexecute Nuitka in case if we are debugging it from Visual Code.
Debugging: Include the .pdb files in Windows standalone mode for proper C tracebacks should that be necessary.
UI: Detect the GitHub flavor of Python as well.
Quality: Check the clang-format version to avoid older ones with bugs that made it switch whitespace for one file. Using the one from Visual Code C extension is a good idea, since it will often be available. Running the checks on newer Ubuntu GitHub Actions runner to have the correct version available.
Quality: Updated the version of rstfmt and isort to the latest versions.
GitHub: Added commented out section for enabling ssh login, which we occasionally need to git bisect problems specific to GitHub Python flavor.
Plugins: Report problematic plugin name with module name or DLL name when these raise exceptions.
Use ordered-set package for Python3.7+ rather than only Python3.10+ because it doesn’t need any build dependency on Windows.
UI: When showing source changes, also display the module name with the changed code.
UI: Use function intended for user query when asking about downloads too.
UI: Do not report usage of ccache for linking from newer version, that is not relevant.
Onefile: Make sure we have proper error codes when reporting IO errors.
MSVC: Detect a version for developer prompts too. This version is needed for use in enabling version specific features.
Started UML diagrams with plantuml that will need to be completed before using them in then new and more visual parts of Nuitka documentation.
UI: Check icon conversion capability at start of compilation rather than error exiting at the very end informing the user about required imageio packages to convert to native icons.
Quality: Enhanced autoformat on Windows, which was susceptible to tools introducing Windows new lines before other steps were performed, that then could be confused, also enforcing use of UTF-8 encoding when working with Nuitka source code for formatting.

Cleanups

The delvewheel plugin was still using a zmq class name from its original implementation, adapted that.
Use common template for generator frames as well. This made them also work with 3.11, by avoiding duplication.
Applied code formatting to many more files in tests, etc.
Removed a few micro benchmarks that are instead to be covered by construct-based tests now.
Enhanced code generation for specialized in-place operations to avoid unused code for operations that do not have any shortcuts where the operation would be actual in-place of a reference count 1 object.
Better code generation for module variable in-place operations with proper indentation and no repeated calls.
Plugins: Use the namedtuple factory that we created for informational tuples from plugins as well.
Make details of download utils module more accessible for better reuse.
Remove last remaining Python 3.2 version check in C code, for us this is just Python3 with 3.2 being unsupported.
Cleanup, name generated call helper file properly, indicating that it is a generated file.

Tests

Made the CPython3.10 test suite largely executable with Python 3.11 and running that with CI now.
Allow measuring constructs without writing the code diff again. Was crashing when no filename was given.
Make Python3.11 test execution recognized by generally accepting partially supported versions to execute the tests with.
Handle also newfstat directory checks in file usage scan. This are used on newer Linux systems.
GitHub: In actions use --report for coverage and upload the reports as artifacts.
Use no-qt plugin to avoid warnings in matplotlib test rather than disabling the warnings about Qt bindings.
macOS: Detect if the machine can take runtime traces, which on Apple Silicon by default it cannot.
macOS: Cover all APIs for file tracing, rather than just one for extended coverage.
Fix, distutils test was not installing the built wheels, but source archive and therefore compiling that second time.
For the pyproject.toml using tests, Nuitka was always downloaded from PyPI rather than using the version under test.
Ignore ld info output about mismatching architecture libraries being ignored. Fixed in 1.4.1 already.

Summary

With this release an important new avenue for scalability has been started. While for Python2 class bodies were very often reduced to just that dictionary creation, with Python3 that was not the case, due to the many new complexities, and while this release makes a start, we will be able to continue this path towards much more scalable class creation codes. And while the performance does not really matter all that much for these, knowing these, will ultimately lead us to “compiled classes” as our own type, and “compiled objects” that may well perform much faster.

Already now, the enhancements to class creation codes will result in smaller binaries, but much more is expected the more this is completed.

The majority of the work was of course to become Python3.11 compatible, and unfortunately the attribute lookups are not as optimized as for 3.10 yet, which may cause disappointing results for performance initially. We will need to complete that before benchmarks will make much sense.

For the next release, full Python 3.11 support is planned. I believe it should be usable. Problems with 3.11 may get hotfixes, but ultimately the develop version is probably the one to recommend when using 3.11 with Nuitka, as there will be the whole set of fixes, since not everything will be ported back.

The new reports should be used in bug reporting soon. We foresee that for issue reports, these may well become mandatory. Together with the ability to create a virtualenv from the reports, this may make reproducing issues a breeze, but first tries on complex projects were also highlighting that it may not be as simple.

Python 3.11 and Nuitka experimental support

2023-03-13T13:05:25+00:00

In my all in with Nuitka post and my first post Python 3.11 and Nuitka and then progress post Python 3.11 and Nuitka Progress , I promised to give you more updates on Python 3.11 and in general.

So this is where 3.11 is at, and the TLDR is, experimental support has arrives with Nuitka 1.5 release, follow develop branch for best support, and 1.6 is expected to support new 3.11 features.

What is now

The 1.5 release passes the CPython3.10 test suite practically as good as with Python3.11 as with Python3.10, with only a handful of tests failing and these do not seem significant, and it is expected to be resolved later when making the CPython3.11 test suite working.

The 1.5 release now gives this kind of output.

Nuitka:WARNING: The Python version '3.11' is not officially supported by Nuitka '1.5', but an
Nuitka:WARNING: upcoming release will change that. In the mean time use Python version '3.10'
Nuitka:WARNING: instead or newer Nuitka.

Using develop should always be relatively good, it doesn’t often have regressions, but Python3.11 improvements will accumulate there until 1.6 release happens. Follow it there if you want. However, checking those standalone cases that can be done, as many packages are not available for 3.11 yet, I have not found a single issue.

What you can do?

Try your software with Nuitka and Python3.11 now. Very likely your code base is not using 3.11 specific features, or is it? If it is, of course you may have to wait until develop catches up with new features and changes in behavior.

In case you are wondering, how I can invest this much time into doing all of what I do, consider becoming a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All commonly essential packaging and performance features are entirely free, and I have put incredible amounts of works in this, and I need to now make a living off it, while I do not plan to make Nuitka annoying or unusable for non-commercial non-subscribers at all.

What was done?

Getting all of the test suite to work, is a big thing already. Also a bunch of performance degradations have been addressed. However right now, attribute lookups and updates e.g. are not as well optimized, and that despite and of course because Python 3.11 changed the core a lot in this area.

The Process

This was largely explained in my previous posts. I will just put where we are now and skip completed steps and avoid repeating it too much.

In the next phase, during 1.6 development the 3.11 test suite is used in the same way as the 3.10 test suite. Then we will get to support new features, new behaviors, newly allowed things, and achieve super compatibility with 3.11 as we always do for every CPython release. All the while doing this, the CPython3.10 test suite will be executed with 3.11 by my internal CI, immediately reporting when things change for the worse.

This phase is starting today actually.

When

It is very hard to predict what will be encountered in the test suite. It didn’t look like many things are there, but e.g. exception groups might be an invasive feature, otherwise I am not aware of too many things at this point. It sure feels close now.

These new features will be relatively unimportant to the masses of users who didn’t immediately change their code to use 3.11 only features.

The worst things with debugging is that I just never know how much time it will be. Often things are very quick to add to Nuitka, and sometimes they hurt a lot or cause regressions for other Python versions by mistake.

Benefits for older Python too

I mentioned stuff before, that I will not repeat only new stuff.

Most likely, attribute lookups will lead to adding the same JIT approach the Python 3.11 allows for now, and maybe that will be possible to backport to old Python as well. Not sure yet. For now, they are actually worse than with 3.10, while CPython made them faster.

Expected results

Not quite good for benchmarking at this time. From the comparisons I did, the compiled code of 3.10 and 3.11 seemed equally fast, allowing CPython to catch up. When Nuitka takes advantage of the core changes to dict and attributes more closely, hope is that will change.

So in a sense, using 3.11 with Nuitka over 3.10 actually doesn’t have much of a point yet.

I need to repeat this. People tend to expect that gains from Nuitka and enhancements of CPython stack up. The truth of the matter is, no they do not. CPython is now applying some tricks that Nuitka already did, some a decade ago. Not using its bytecode will then become less of a benefit, but that’s OK, this is not what Nuitka is about.

We need to get somewhere else entirely anyway, in terms of speed up. I will be talking about PGO and C types a lot in the coming year, that is at least the hope. The boost of 1.4 and 1.5 was only be the start. Once 3.11 support is sorted out, int will be getting dedicated code too, that’s where things will become interesting.

Final Words

So, this post is kind of too late. My excuse is that due to having Corona, I did kind of close down on some of the things, and actually started to do optimization that will lead towards more scalable class code. This is also already in 1.5 and important.

But now that I feel better, I actually forced myself to post this. I am getting better at this. Now back and starting the CPython3.11 test suite, I will get back to you once I have some progress with that. Unfortunately adding the suite is probably also a couple of days work, just technically before I encounter interesting stuff.

Python 3.11 and Nuitka Progress

2023-02-18T12:29:24+00:00

In my all in with Nuitka post and my first post Python 3.11 and Nuitka, I promised to give you more updates on Python 3.11 and in general. So this is where 3.11 is at, and the TLDR is, experimental support is coming with Nuitka 1.5 release.

What is now

Currently Nuitka 1.4 contains already some preparations for 3.11 support, but also from the feedback I had read, on how Nuitka has lost its performance lead over the releases, a huge amount of catching up on the quality and depth of integration, was put onto the plate, and so that became a large part of the 1.4 focus.

These preparations mostly addressed frames, unified separate code generation for them, and since with 3.11 generators frames were far too buggy, this had to be continued, and interestingly many frame issues were solved by some great unification right after 1.4 release for 1.5 develop.

And from here, most frame issues got debugged and fixed. This is the area where 3.11 had the biggest impact on Nuitka internals.

The 1.5 development release now gives this kind of output.

Nuitka:WARNING: The Python version '3.11' is not officially supported by
Nuitka:WARNING: Nuitka '1.5rc7', but an upcoming release will change that. In
Nuitka:WARNING: the mean time use Python version '3.10' instead or newer
Nuitka:WARNING: Nuitka.

Using develop should be good. But I expect to be better with every pre-release until 1.5 happens. Follow it there if you want.

What you can do?

Not a lot yet, as a mere mortal, what you can and should do is to consider becoming a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All commonly essential packaging and performance features are entirely free, and I have put incredible amounts of works in this, and I need to now make a living off it.

Working on 3.11 support, just so tings then continue to work, mostly on my own deserves financial support. Once a year, for weeks on end, I am re-implementing features that a much larger collective of developers came up with.

But don’t get me wrong. This may sounds like complaining here. Not at all. I love these things. I hate to be under some kind of time pressure though, but it seems that is coming to an end, so it’s all good now.

What was done?

So the “basic” tests of Nuitka are finally passing. That is by itself a huge step, and the now work on the CPython3.10 test suite, has been started to be executed with Python 3.11, and fixes are applied.

Doing that, still some bigger issues were still found, e.g. uncompiled frames as the parent of compiled frames, when inspected, did not automatically provide a f_back, these now have to created on the fly. This sort of things, which are not commonly observed by code.

Esp. the usual trouble makers, test_generators, test_coroutines, and test_asyncgen (with one test part as an exception) are passing. Honestly these are always the most scary to me. Debugging coroutines using asyncgen is a fairly large chunk of my time spent on Nuitka in recent years.

Compiled frames appear to be not entirely correctly seen for tracebacks and stacks when uncompiled code looks upward, but that seems negligible right now.

Up to test_inspect things appear to be working fine, and there is some kind of TODO list now, that I maintain on the roadmap document. I add TODOs there, e.g. because I had to disable attribute optimization with Python3.11 so that is going to harm performance and should be revisited before claiming full support.

As for inspect module, there will be more functions that need monkey patching, esp. related to frames, e.g. getframeinfo has no tolerance for the lack of bytecode in compiled function and their frames.

The Process

This was largely explained in my previous post. I will just put where we are now and skip completed steps and avoid repeating it too much.

The current phase is to run the CPython 3.10 test suite compiled with 3.11 and compare its results to running with plain 3.11, and expect it to be the same. It is normal for CPython 3.10 tests to fail with 3.11, and this does several things.

Test failures of 3.11 with 3.10 code actually give new test coverage. It has happened in previous new Python releases, that only then it became apparent that there were slight incompatibilities in exceptions, etc.

Also now a test may passes where it should not, i.e. 3.11 changed behavior and now Nuitka still follows 3.10 behavior. So maybe a new check is done or something like that, and Nuitka needs a version guarded change that also makes it behave the same with 3.11 version. But of course, when using 3.10 it has to retain the older behavior still.

The other thing, is when Python 3.11 passes, but Nuitka is not. Typically then Nuitka has some sort of change missing to the internal codes, often only a changed error message, exception type, etc. but sometimes this can be also a bigger thing.

During this phase it is hard to know where we stand. I do not really want to do repeated analysis. So e.g. when the exception for using a non-context manager in a with statement fails, I am not really interested to see how many other tests are failing because of that or other things. I am going to fix that first and only then I will look what the next failure is.

Since I will get to fix all the things, anyway, I tend to address most of the issues I find immediately and delay further executing of the test suite. Only when I understand the issue clearly enough and see that it will take a lot of time for a corner case, then I will add fixing it to the roadmap.

This I am doing e.g. with the single test part failure in test_asyncgen, because I suspect, that some change in 3.11 has happened, or that I will revisit the topic in the 3.11 test suite, which often has more dedicated tests to highlight changes. The one not good with Nuitka right now does not look important, nor easy to analyze.

In the next phase, the 3.11 test suite is used in the same way. Then we will get to support new features, new behaviors, newly allowed things, and achieve super compatibility with 3.11 as we always do for every CPython release. All the while doing this, the CPython3.10 test suite will be executed with 3.11 by my internal CI, immediately reporting when things change for the worse.

This phase has not even started however. But probably results of it will be in 1.6 I would assume now.

Intermediate Results

Once the Python 3.11 more or less well passed the 3.10 test suite without big issues, experimental support for it will be proclaimed and 1.5 shall be released. The warning from above will be given, but the error that 1.4 gave you will cease, and come back for 3.12 probably.

When

Very hard to predict. It feels close now. Supporting existing Nuitka is also a side tracking thing, that makes it unclear how much time I will have for it.

And the worst things with debugging is that I just never know how much time it will be. I have spent almost a day staring at debugging traces for the coroutine code, before these worked finally. And during that time it didn’t feel like progressing at all.

I think, look back at Python changes since 2.6, which was the first thing Nuitka supported, and still does btw, 3.5 and coroutines, 3.6 and asyncgen, and then 3.10 and match statements, the 3.11 release will probably have been the hardest.

Benefits for older Python too

I mentioned stuff before, that I will not repeat only new stuff. So the frame changes caused me to solve most of the issues by doing cleanups and refactoring that allowed for enhancements present in 1.4 and coming to 1.5 some more, covering generators as well.

Most likely, attribute lookups will gain the same JIT approach the Python 3.11 allows for now, and maybe that will be possible to backport to old Python as well. Not sure yet. For now, they are actually worse than with 3.10, while CPython made them faster. Not quite good for benchmarking at this time.

Expected results

We need to get somewhere else entirely anyway, in terms of speed up. I will be talking about PGO and C types a lot in the coming year, that is at least the hope. The boost of 1.4 will only be the start. Once 3.11 support is sorted out, int will be getting dedicated code too, that’s where things will become interesting.

Final Words

Look ma, I posted about something that is not complete. The temptation to just wait until I finish it was so huge. But I resisted successfully.

Nuitka Package Configuration Part 2

2023-02-13T09:01:00+00:00

This is the second part of a post series under the tag package_config that explains the Nuitka Package Configuration in more detail. To recap, Nuitka package configuration is the way Nuitka learns about hidden dependencies, needed DLLs, data files, and just generally avoids bloat in the compilation. The details are here on a dedicate page on the web site in Nuitka Package Configuration but reading on will be just fine.

Problem Package

Each post will feature one package that caused a particular problem. In this case, we are talking about the package customtkinter.

Problems are typically encountered in standalone mode only. Missing data files are typical issue there, and in this case, we already had a solution, but turns out, only working on Windows for a weird reason. We are going to look at that in some detail, and will see a workaround applied with the anti-bloat engine doing code modification on the fly.

Initial Symptom

The initial symptom reported was like this, on the Nuitka Discord server and much to my surprise, against the then current 1.4.4 release of Nuitka.

Traceback (most recent call last):
File "/tmp/onefile_32802_1675869955_205649/customtkinter/windows/widgets/theme/__init__.py", line 5, in <module customtkinter.windows.widgets.theme>
File "/tmp/onefile_32802_1675869955_205649/customtkinter/windows/widgets/theme/theme_manager.py", line 18, in load_theme
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/onefile_32802_1675869955_205649/customtkinter/windows/widgets/theme/../../../assets/themes/blue.json'

One thing to say here is that if you have this kind of issues, using onefile is definitely too early. The recommendation is to use onefile only after standalone works, because it is easier to debug.

Step 1 - Analysis

So first thing, I did was to normalize the path, so I can see if something funny is going on, with that.

>>> import os
>>> os.path.normpath("/tmp/onefile_32802_1675869955_205649/customtkinter/windows/widgets/theme/../../../assets/themes/blue.json")
'/tmp/onefile_32802_1675869955_205649/customtkinter/assets/themes/blue.json'

So this matches existing data files. Originally we had worked off this on Windows,

python3.10 -m nuitka --list-package-data=customtkinter
Nuitka-Tools:INFO: Checking package directory 'C:\Python310_64\lib\site-packages\customtkinter' ..
C:\Python310_64\lib\site-packages\customtkinter
C:\Python310_64\lib\site-packages\customtkinter\assets\fonts\CustomTkinter_shapes_font.otf
C:\Python310_64\lib\site-packages\customtkinter\assets\fonts\Roboto\Roboto-Medium.ttf
C:\Python310_64\lib\site-packages\customtkinter\assets\fonts\Roboto\Roboto-Regular.ttf
C:\Python310_64\lib\site-packages\customtkinter\assets\icons\CustomTkinter_icon_Windows.ico
C:\Python310_64\lib\site-packages\customtkinter\assets\themes\blue.json
C:\Python310_64\lib\site-packages\customtkinter\assets\themes\dark-blue.json
C:\Python310_64\lib\site-packages\customtkinter\assets\themes\green.json

Which confirms the data file, and had lead to adding configuration to include these data files.

- module-name: 'customtkinter'
  data-files:
    dirs:
      - 'assets'

This is the most simple for including data files. Because there are no needs to limit by file type, and not even a wish to know what file types might get added in the future. We add the folder and are good.

Since the original report was on Windows, it was tested there and assumed to be good on all platforms.

Step 2 - Debugging

So, the data files are indeed included, which was confirmed by asking to compile in standalone and to provide the listing. However, at run time they were not found.

This is where the platform differences come in, somewhat to my surprise, on Windows the customtkinter/windows/widgets/theme/../../../assets/themes/blue.json part of the path works, even though there is no windows directory. Since compiled code is contained in the executable, packages in standalone may not have a directory associated.

So while __file__ is set to that virtual directory, it does not exist. And usually that is all good, and directories get created of course when package data gets added. That is e.g. why customtkinter and then of course customtkinter/assets exist. But customtkinter/windows does not.

The platform difference, that I was not aware of that only on Windows, will a path being opened get normalized before being used.

>>> os.name
'nt'
>>> os.path.exists("README.rst")
True
>>> os.path.exists("doesnotexist")
False
>>> os.path.exists("doesnotexist\\..\\README.rst")

But neither on Linux nor macOS this is going to be possible. So that was the problem. Checkout their code, this is what is used:

with open(os.path.join(script_directory, "../../../assets", "themes", f"{theme_name_or_path}.json"), "r") as f:
    cls.theme = json.load(f)

This could have used pkg_resources or importlib.resources or many of the methods that Nuitka supports out of the box. But actually open and being __file__ relative is supposed to be fine.

So this does not work.

Step 2 - Devising a solution

So, there are actually a couple of ways to resolve this. One would be to just provide the empty directories that non-Windows is checking to exist. These would then have to be deployed. There is actually support for that in Nuitka Package Configuration.

And we have examples of that in our existing configuration, e.g. for Crypto.Util._raw_api.

- module-name: 'customtkinter.windows.widgets.theme.theme_manager'
  data-files:
    empty_dirs:
      - '.'

This is for code that insists on the package directory, but honestly, while it is easy enough, I don’t quite like this solution. The empty directory requires a dummy file, and ultimately it that is all that is used for is rather ugly.

So, what else we can do? Well, we can modify the code of course too. We have the anti-bloat engine and it is capable of achieving reduction of bloat. What if the we used it to modify the code to do a os.path.normpath and that is actually what we ended up doing. The most simple form of anti-bloat does plain string replacements. Here it is used.

- module-name: 'customtkinter.windows.widgets.theme.theme_manager'
  anti-bloat:
    - description: 'workaround for file path on Linux'
      replacements_plain:
        'script_directory, "../../../assets"': 'os.path.normpath(os.path.join(script_directory, "../../../assets"))'
      # Not necessary on Windows
      when: 'not win32'

The effect can be easily seen with the --show-source-changes option which outputs for modules the changes applied. Notice that we do not do it on Windows, because there it is not needed. The when clause allows us to specify such conditions.

customtkinter.windows.widgets.theme.theme_manager:
--- original
+++ modified

@@ -15,7 +15,7 @@

         script_directory = os.path.dirname(os.path.abspath(__file__))

         if theme_name_or_path in cls._built_in_themes:
-            with open(os.path.join(script_directory, "../../../assets", "themes", f"{theme_name_or_path}.json"), "r") as f:
+            with open(os.path.join(os.path.normpath(os.path.join(script_directory, "../../../assets")), "themes", f"{theme_name_or_path}.json"), "r") as f:

The normpath is limited to where it is needed, to keep the replaced expression minimal. And in this way, the problem is solved for in a pretty non-invasive way.

Source modification is of course always something that can break if the code changes, and should normally be avoided. But it appears this should be a very robust one.

Final remarks

I am hoping you will find this very helpful information and will join the effort to make packaging for Python work out of the box. Adding support for customtkinter seems trivial at first, and for Windows in fact is was.

Lessons learned. Data files working need to be confirmed on either Linux or macOS too, esp. in case of these relative paths.

We will have to have a repository of test cases for this kind of small programs, where you can help by checking for new version to still produce the same compilation report. Then it would be fairly easy to get the coverage on all platforms and in case of updated packages on PyPI.

Nuitka Release 1.4

2023-02-05T11:11:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of performance work, where specifically Python versions 3.7 or higher see regressions in relative performance to CPython fixed. Many cases of macros turned to functions have been found and resolved. For 3.10 specifically we take advantage of new opportunities for optimization. And generally avoiding DLL calls will benefit execution times on platform where the Python DLL is used, most prominently Windows.

Then this also adds new features, specifically custom reports. Also tools to aid with adding Nuitka Package Configuration input data, to list DLLs and data files.

With multidist we see a brand new ability to combine several programs into one, that will become very useful for packaging multiple binaries without the overhead of multiple distributions.

Bug Fixes

Standalone: Added implicit dependencies for dependency_injector package. Fixed in 1.3.1 already.
Fix, the generated metadata nodes for distribution queries had an error in their generated children handling that could cause crashes at compile time. Fixed in 1.3.2 already.
Standalone: Added implicit dependencies for passlib.apache package. Fixed in 1.3.2 already.
Windows: Fix, our shortcut to find DLLs by analyzing loaded DLLs stumbled in a case of a DLL loaded into the compiling Python that had no filename associated, while strange, we need to handle this as well. Fixed in 1.3.3 already.
Standalone: Also need to workaround more decorator tricks for networkx. Fixed in 1.3.3 already.
Scons: Fix, was not updating PATH environment variable anymore, which could lead to externally provided compilers and internal WinLibs gcc clashing on Windows, but should be a general problem. Fixed in 1.3.4 already.
Standalone: Added support for cefpython3 package. Fixed in 1.3.4 already.
Standalone: Added support for newer webview package versions. Fixed in 1.3.4 already.
Standalone: Fix, some extension modules set their __file__ to None during multi phase imports, which we then didn’t update anymore, however that is necessary. Fixed in 1.3.4 already.
Python3.10+: Fix, was not supporting match cases where an alternative had no condition associated. Fixed in 1.3.5 already.
Windows: Identify Windows ARM architecture Python properly. We do not yet support it, but we should report it properly and some package configurations are already taking it already into account. Fixed in 1.3.5 already.
Fix, the Nuitka meta path based loader, needs to expose a __module__ attribute because there is code out there, that identifies standard loaders through looking at this value, but crashes without it. Fixed in 1.3.5 already.
Fix, very old versions of the importlib_metadata backport were using themselves to load their __version__ attribute. Added a workaround for it, since in Nuitka it doesn’t work until after loading the module.
Fix, value escapes for attribute and subscript assignments sources were not properly annotated. This could cause incorrect code execution. Fixed in 1.3.6 already.
Fix, “pure” functions, which are currently only our complex call helper functions, were not visited in all cases. This lead to a crash in code generation after modules using them got demoted to bytecode. After use from cache, this didn’t happen again. Fixed in 1.3.6 already.
Standalone: Added more implicit dependencies of crypto packages. Fixed in 1.3.6 already.
Standalone: Added implicit dependencies of pygments.styles module. Fixed in 1.3.6 already.
Fix, was falsely encoding Ellipsis too soon during tree building. It is not quite like True and False. Fixed in 1.3.6 already.
Standalone: Fix, numpy on macOS didn’t work inside an application bundle anymore. Fixed in 1.3.7 already.
Python3.8+: Fix, need to follow change for extension module handling, otherwise some uses of os.add_dll_directory fail to work. Fixed in 1.3.8 already.
Standalone: Added missing implicit dependencies of sqlalchemy. Fixed in 1.3.8 already.
Python3.9+: Fix, resource reader files was not fully compatible and needed to register with importlib.resources.as_file to work well with it. Fixed in 1.3.8 already.
Fix, the version check for cv2 was not working with the opencv-python-headless variant. Package name and distribution name is not a 1:1 mapping for all things. Fixed in 1.3.8 already.
Standalone: Added DLLs needed for tls_client package.
Fix, imports of resolved names should be modified for runtime too. Where Nuitka recognizes aliases, as e.g. the requests module does, it only adding a dependency on the resolved name, but not requests itself. The import however was still done at runtime on requests which then didn’t work. This was only visible if only these aliases to other modules were used.
Onefile: Fix, do not send duplicate CTRL-C to child process. Our test only send it to the bootstrap process, rather than the process group, as it normally is working, therefore misleading us into sending it to the child even if not needed.
Onefile: When not using cached mode, on Windows the temporary folder used sometimes failed to delete after the executable stopped with CTRL-C. This is due to races in releasing of locks and process termination and AV tools, so we now retry for some time, to make sure it is always deleted.
Standalone: Fix, was not ignoring .dylib when scanning for data files unlike all other DLL suffixes.
Standalone: Added missing implicit dependency of mplcairo.
Standalone: The main binary name on non-Windows didn’t have a suffix .bin unlike in accelerated mode. However, this didn’t work well for packages which have binaries colliding with the package name. Therefore now the suffix is added in this case too.
macOS: Workaround bug in platform_utils.paths. It is guessing the wrong path for included data files with Nuitka.
Standalone: Added DLLs of sound_lib, selecting by OS and architecture.
Fix, for package metadata as from importlib.metadata.metadata for use at runtime we need to use both package name and distribution name to create it, or else it failed to work. Packages like opencv-python-headless can now with this too.
Standalone: Added support for tkinterweb on Windows. Other platforms will need work to be done later.

New Features

UI: Added new option to listing package data files. This is for use with analyzing standalone issues. And will output all files that are data files for a given package name.
```
python -m nuitka --list-package-data=tkinterweb
```
UI: Added new option to listing package DLL files. This is also for use with analyzing standalone issues.
```
python -m nuitka --list-package-dlls=tkinterweb
```
Reports: The usages of modules, successful or not, are now included in the compilation report. Checking out which ones are not-found might help recognition of issues.
Multidist: You can now experimentally create binaries with multiple entry points. At runtime one of multiple __main__ will be executed. The option to use is multiple --main=some_main.py arguments. If then the binary name is changed, on execution you get a different variant being executed.

Note

Using it with only one replaces the previous use of the positional argument given and is not using multidist at all.

Note

Multidist is compatible with onefile, standalone, and mere acceleration. It cannot be used for module mode obviously.

For deployment this can solve duplication.

Note

For wheels, we will probably change those with multiple entry points to compiling multidist executables, so we do avoid Python script entry points there. But this has not yet been done.
Onefile: Kill non-cooperating child processes on CTRL-C after a grace period, that can be controlled at compile time with --onefile-child-grace-time the hard way. This avoids hangs of processes that fail to properly shutdown.
Plugins: Add support for extra global search paths to mimic sys.path manipulations in the Yaml configuration with new global-sys-path import hack.
Reports: Include used distributions of compiled packages and their versions.
Reports: Added ability to generate custom reports with --report-template where the user can provide a Jinja2 template to make his own reports.
Anti-Bloat: Added support for checking python flags. There are no_asserts, no_docstrings and no_annotations now. These can be used to limit rules to be only applied when these optional modes are active.

Not all packages will work in these modes, but often can be enhanced to work with relatively little patching. This allows to limit these patches to only where they are necessary.

Optimization

Anti-Bloat: Avoid using sparse and through that Numba in the scipy package, reducing its distribution footprint. Part of 1.3.3 already.
Anti-Bloat: Avoid IPython and Numba in trimesh package. Part of 1.3.3 already.
Anti-Bloat: Avoid Numba in shap package. Part of 1.3.8 already.
Anti-bloat: Removed xgboost docstring dependencies, such that --python-flag=no_docstrings can be used with this package.
For guided deep copy frozenset and empty tuple need no copies

This also speeds up copies of non-empty tuples by avoiding that size checking branch in construction with Python 3.10 or higher.
For node construction, avoid keyword argument style calls of the base class, where there is only a single argument. They don’t really help readability, but cost compile time.
Determine guard mode of frames dynamically and avoid frame preservation checks where they are not needed.

For Python2 this is necessary, but not for Python3, so make the function avoid finding the parent frame for that version entirely, which should speed up compilation as well.

By not hard coding frame guard mode at creation time, and instead determine it at compile time, after optimization, so this now allows to use the “once” mode more often. This affects contractions and also classes on the module level right now. They do not need a cached frame, since their code is only executed once.

By avoiding that useless code, the C compiler also has a slightly better scalability, since the classes are all created in one function that then has less code.
The bytecode cache is now checking if the used modules or attempted to be used modules are available or not in just the same way. Previously it was very dependent on the file system to contain the same things, which was not giving cache hits even after only creating a new folder near a binary, since that affected importable modules. With the new check it should be much more directly hitting even across different virtual environments, but with same code.
Generate base classes or mixins for all kinds of expression, statements and statement sequences. The previous code had a dedicated variant for single child, to allow faster operation in a common case, but still a lot of hasattr/getattr/setattr on dynamic attribute names were done. This was making the tree traversal during optimization slower than necessary.

Another shortcoming was that for some nodes, some values are optional, where for others, they are not. Some values are a tuple actually, while most are nodes only. However, dealing with this generically was also slower than necessary.

The new code now enforces children types during creation and updated, it rejects unexpected None values for non-optional children, and it provides generated code to do this in the fastest way possible, although surely some more improvements will come here.

Also when abstract executing the tree, rather than generically visiting all children, this now just unrolls this, and there are even some modes added, where a node can indicate properties, e.g. auto_compute_handling = "final,no_raise" will tell the code generator that this expression never raises in the computation, and is final, i.e. doesn’t have any code to evaluate, because it cannot be optimized any further.

Also the way checkers previously worked, for every node creation, for every child update, a dictionary lookup had to be done. This is now hard coded for the few nodes that actually want to convert values on the fly and we might make a difference in the future for optional checkers, such that these are only run in debug mode.

These changes brought about much faster compilation, however the big elephant in the room will still be merging value traces, and scalability problems remain there.
Attribute node generation for method specs like dict.update, etc. now provide type shapes. From these type shapes, mixins for the result value type are picked automatically. Previously these shapes were added manually. In some cases, they were even missing. In a few cases, where the type is dependent on the Python version, we do not currently do this though, so this needs more work, but expanding the coverage got easier in this way.
Determining the used modules of a module requires a tree visit operations, that then asked for node types and used different APIs. This has been unified to be able to call a virtual method instead, which saves some compile time.
After scanning for a module, we then determined the module kind even after we previously knew it during the scan. Also, this was checking os.path.isdir which was making it relatively slow and wasting 5% compile time on the IO being done. The check got enhanced and most often replaced with using the knowledge from the original import scan eliminating this time.
Already most helper code of Nuitka was included from .c files, but compiled generators and compiled cells codes were not yet done like this, making life unnecessarily harder for the compiler and linker. This should also allow more optimization for some codes.
Cache the plugin decisions about recursion for a module name. When a module is imported multiple times plugins were each asked again and again, which is not a good thing to do.
Avoid usage of PyObject_RichCompareBool API, as we have our own comparison functions that are faster and faster to call without crossing of DLL barrier.
Python3.8+: Avoid usage of PyIndex_Check which has become an API in 3.8, and was as a result not inlined anymore with a DLL barrier was to be crossed, making all kinds of multiplication and subscript/index operations slower.
Replace PyNumber_Index API with our own code. As of 3.10 it enforces a conversion to long that for Nuitka is not a good thing to do in all places. But also due to DLL barrier it was potentially slow to call, and is used a lot, and we can drop the checks that are useless for Nuitka.
Python3.7+: Avoid the use of PyImport_GetModule for looking up imported modules from sys.modules, rather look it up from interpreter internals, also this was using subscript functions, when this is always a dictionary.
Avoid using PyImport_GetModuleDict and instead have our own API to get this quicker.
Faster exception match checks and sub type checks.

This solves a TODO about inlining the API function used, so we can be faster in a relatively common operation. For every exception handler, we had to do one API call there.
Faster subtype checks.

These are common in binary operations on non-identical types, but also needed for the exception checks, and object creation through class type calls. With our own PyType_IsSubType replacement these faster to use and avoid the API call.
Faster Python3 int value startup initialization.

On Python 3.9 or higher we can get small int values directly from the interpreter, and with 3.11 they are accessible as global values.

Also we no longer de-duplicate small int values through our cache, since there is no use in this, saving a bunch of startup time. And we can create the values with our own API replacement, that will work during startup already and save API calls as these can be relatively slow. And esp. for the small values, this benefits from not having to create them.
Faster Python3 bytes value startup initialization.

On Python 3.10 or higher, we can create these values ourselves without an API call, avoiding its overhead.

Also we no longer de-duplicate small bytes values through our cache, because that is already done by the API and our replacement, so this was just wasting time.
Faster slice object values with Python 3.10 or higher

On Python 3.10 or higher, we can create these values ourselves without an API call, avoiding its overhead.

These are important for Python3, because a[x:y] in the general case has to use a[slice(x,y)] on that version, making this somewhat relevant to performance in some cases.
Faster str built-in with API calls

For common cases, this avoids API calls. We mostly have this such that print style tests do not have this as API calls where we strive to remove all API calls for given programs.
Faster exception normalization.

For the common case, we have our own variant of PyErr_NormalizeException that will avoid the API call. It may still call the PyObject_IsSubclass API, for which we only have started replacement work, but this is already a step ahead in the right direction.
Faster object releases

For Python3.8 or higher when our code released objects, it was doing that with an API call, due to a macro change in Python headers. We revert that and do it still on our own which avoids the performance penalty.
Enable Python threading during extension module DLL loading

We now release the GIL for Python3.8 or higher when loading the DLL, following a change in that version.
Faster variable handling in trace collection. The code was doing checks for variable types, to decide what to do e.g. when control flow escapes for a variable. However, this is faster if solved with a virtual method in those variable classes, shifting the responsibility to inside there.
For call codes the need to check the return value was not perfectly annotated in all cases. This is now driven by the expression rather than passed, and will result in better code generated in some corner cases.

Organizational

Release: Make clear we require wheel and setuptools to install by adding a pyproject.toml that addresses a warning of pip. Part of 1.3.6 release already.
Debugging: When plugins evaluate when conditions that raise, output which it was exactly. Part of 1.3.3 already.
Anti-Bloat: Added a mnemonic and more clear message for the case of unwanted imports being encountered. Also do not warn about IPython itself using IPython packages, that must of course be considered normal. Now it also lists the module that does the unwanted usage immediately. Previously this was not as clear.
UI: More clear output for not yet supported Python version. Make it more clear in the message, what is the highest supported version, and what version is Nuitka and what is Python in this.
UI: Make sure data files have normalized paths. Specifically on Windows, otherwise a mix of slashes could appear. Part of 1.3.6 release already.
UI: Make it clear that disabling the console harms your debugging when we suggest the --disable-console for GUI packages. Otherwise using that, they just deprive themselves of ways to get error information.
UI: The ordering of scons ccache report was not enforced. Part of 1.3.7 release already.
Quality: Use proper temporary filename during autoformat, so as to avoid flicker in Visual Code, e.g. search results.
User Manual: Was still using old option name for --onefile-tempdir-spec that has since been made not OS specific, with even the OS specific name being removed.
Standalone: Do not include data files scanned with site-packages or __pycache__ folders. This should make it easier to use --include-data-file=./**.qml:. when you have a virtualenv living in the same folder.
Onefile: Added check for compression ability before starting the compilation to inform the user immediately.
Release: Mark macOS as supported in PyPI categories. This is of course true for a long time already.
Release: Mark Android as supported in PyPI categories as well. With some extra work, it can be used.
User Manual: Added section pointing to and explaining compilation reports. This has become extremely useful even if still somewhat work in progress.
User Manual: Added table with included custom reports, at this time only the license reports, which is very rough shape and needs contributors for good looks and content.

Cleanups

Plugins: Moved parts of the pywebview plugin that pertain to the DLLs and data files to package configuration.
Made the user query code a dedicated function, so it can be reused and more consistent across its uses in Nuitka. With a default that is proposed to a user, and a default that applies if used non-interactively. We will switch all prompts to using this.
Code generation for module, class and function frames is now unified, removing duplication while also becoming more flexible. For generators this work has been started, but is not yet completed.
Nodes exposing used modules now implement the same virtual method providing a list of them.
Make sure to pass tuple values rather than list values from the tree building stage and node optimization creating new nodes. This allows us to drop conversions previously done inside of nodes.

Tests

Do not enable deprecated plugins, the warnings about them break tests.
Ignore Qt binding warnings in tests, some are less supported than PySide6 or commercial PySide2.

Summary

The focus of this release was first a major restructuring of how children are handled in the node tree. The generated code opens up the possibility of many more scalability improvements in the coming releases. The pure iteration speed for the node tree will make compile times for the Python part even shorter in coming releases. Scalability will be a continuous focus for some releases.

Then the avoiding of API calls is a huge benefit for many platforms that are otherwise at a disadvantage. This is also only started. We will aim at getting more complex programs to do next to none of these, so far only some tests are working after program start without them, which is of course big progress. We will progress there with future releases as well.

Catching up on problems that previous migrations have not discovered is also a huge step forward to restoring the performance supremacy, that was not there anymore in extreme cases.

The Yaml package configuration work is showing its fruits. More people have been able to contribute changes for anti-bloat or missing dependencies than ever before.

Some part of the Python 3.11 work have positively influenced things, e.g. with the frame cleanup. THe focus of the next release cycle shall be to add support for it. Right now, generator frames need a cleanup to be finished, to also become better and working with 3.11 at the same time. Where possible, work to support 3.11 was also conducted as a cleanup action, or reduction of the technical debts.

All in all, it is fair to say that this release is a big leap forward in all kinds of ways.

Nuitka Package Configuration Kickoff

2023-01-20T07:51:00+00:00

The term “kickoff” refers to a series of posts about the Nuitka package configuration. The details are here on a dedicate page on the web site only. Nuitka Package Configuration.

This documentation is still very rough and bare of examples, but the goal is to make it more complete as package of this series of posts. When we have an instructive example, we will make a post.

This is an area of Nuitka, where help will be very easy and a wide variety of people will have the skills and desire to help, but the lack of documentation, makes it hard or impossible to channel the common knowledge.

Problem Package

Each post will feature one package that caused a particular problem. In this case, we are talking about the package tkinterweb.

Problems are typically encountered in standalone mode only. Missing data files and DLLs are usually only and issue there, but this one actually also had a problem with accelerated mode.

Initial Symptom

The compiled program gave the following error:

ModuleNotFoundError: The files required to run TkinterWeb could not be found. This usually occurs when bundling TkinterWeb into an app without forcing the application maker to include all nessessary files. See https://github.com/Andereoo/TkinterWeb/blob/main/tkinterweb/docs/FAQ.md for more information.

Error: Traceback (most recent call last):
    File "C:\...\200~1.0\tkinterweb\__init__.py", line 32, in <module tkinterweb>
ModuleNotFoundError: No module named 'bindings'

Note

The traceback has been redacted, removing user specific part and replacing it with ....

The strange looking part of the filename 200~1.0 in the traceback is because Nuitka on Windows is convincing the program that it runs in a short path. So the errors you get, are not pointing to the name of the binary inside the dist folder, but to a shorter path, that is however exactly the same folder. You wouldn’t believe how many parts of Python still don’t have long filenames handled properly. Actually Tcl and TkInter are among them.

Step 1 - data files

So, whenever a Python module fails to import, this can be caused by missing data files. In Nuitka land, these are not extension modules, DLLs, bytecode, etc. but everything else, and it has been seen that these can cause issues. So what to do here. This is easy, Nuitka has recently gained a command dedicated to this operation, making use of its internal code to list data files for a package.

Here we go

python3.10 -m nuitka --list-package-data=tkinterweb

Gives output:

Nuitka-Tools:INFO: Checking package directory 'C:\Python310\lib\site-packages\tkinterweb' ..
C:\Python310\lib\site-packages\tkinterweb
C:\Python310\lib\site-packages\tkinterweb\tkhtml\combobox-2.3.tm
C:\Python310\lib\site-packages\tkinterweb\tkhtml\pkgIndex.tcl
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Darwin\64-bit\pkgIndex.tcl
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Linux\32-bit\pkgIndex.tcl
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Linux\64-bit\pkgIndex.tcl
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Windows\32-bit\pkgIndex.tcl
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Windows\64-bit\pkgIndex.tcl

So there are data files, great. Lets just include them, and retry. There appears to be need for that anyway. Now we could use --include-package-data=tkinterweb for quickly trying it out, but since we are in here to make it generally useful, we will start modification of the Yaml file right away.

- module-name: 'tkinterweb'
  data-files:
    dirs:
      - 'tkhtml'

So, we retry, and nothing changes. Not enough, but on the other hand, it was obviously necessary anyway.

Step 2 - DLL files

The other big trouble maker, and easy to check nowadays, is missing DLLs. Extension modules can depend on DLLs. When they are linked against, this not usually and issue, and Nuitka and its plugins typically resolve all of those perfectly, but often DLLs are loading manually. Maybe this is what is going on here.

Also recently, Nuitka gained a command to check for DLLs.

Here we go

python3.10 bin/nuitka --list-package-dlls=tkinterweb

Gives output:

Nuitka-Tools:INFO: Checking package directory 'C:\Python310\lib\site-packages\tkinterweb' ..
C:\Python310\lib\site-packages\tkinterweb
C:\Python310\lib\site-packages\tkinterweb\tkhtml
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Darwin
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Darwin\64-bit
tkhtml\Darwin\64-bit\Tkhtml30.dylib
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Linux
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Linux\32-bit
tkhtml\Linux\32-bit\Tkhtml30.so
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Linux\64-bit
tkhtml\Linux\64-bit\Tkhtml30.so
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Windows
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Windows\32-bit
tkhtml\Windows\32-bit\Tkhtml30.dll
C:\Python310\lib\site-packages\tkinterweb\tkhtml\Windows\64-bit
tkhtml\Windows\64-bit\Tkhtml30.dll

Note

The output could be not containing folders that have no DLLs themselves, but well, such is life, we are going to improve that another time.

What this tells us, that in fact there are DLLs, and from the looks of it, there is no automatic anything in this. This also appears to be one of those PyPI packages that contain binaries for everything. Rather than building a wheel per architecture this contains some things, on all platforms. For a Python installation that is cool, but surely we do not want to deploy both the 32 and 64 bit DLLs where the compiled binary is only one of these. Do not even think of different OS, like including Linux DLLs on Windows.

So, luckily this is easy to handle. We can select for OS and architecture on Windows for a while already.

- module-name: 'tkinterweb'
  dlls:
    - from_filenames:
        relative_path: 'tkhtml/Windows/32-bit'
        prefixes:
          - 'Tkhtml'
      when: 'win32 and arch_x86'
    - from_filenames:
        relative_path: 'tkhtml/Windows/64-bit'
        prefixes:
          - 'Tkhtml'
      when: 'win32 and arch_amd64'

Note

Showing this here without the data files section for clarity, obviously the DLLs just get added, and Nuitka prevents you from having two blocks referencing the same module.

So, including DLLs is fairly easy. If the package directory is not where the DLL lives, you specify relative_path which is otherwise optional. This also influences where it is put in the distribution folder. Then when specifying the DLL, we do only give the prefix of the DLL. Choosing here to leave out the 30 part of Tkhtml30.dll just because it’s probably going to make our life easier down the road, should they update that version number, it would still automatically work.

Obviously for other platforms than Windows, the DLLs are not included now, but lets see if this works. And actually at the time of writing, this is a first. As you can see, on macOS (recognized from “Darwin”) only the x86_64 will work, and maybe we should check that out. For Linux and 32 bit, this shows what an old package this is.

So far, outside of Windows, we do not provide tags for arches.

Note

This is probably going to change now. At least on macOS this seems very much needed. Maybe also time to cleanup amd64 vs x86_64 which kind of is an inconsistency the technical community has.

Anyway, so more branches will be needed. There is no else in Nuitka package configuration. All from_filenames blocks are applied if the when matches.

And actually for data files are similar thing should be done, however, for the time being --noinclude-data-file can be your friend there. You can manually exclude them.

But low and behold, the DLLs are included. The data files are. Typically that is enough, but it still does not work.

Step 3 - Check the compilation report

So after following the easy steps to take, and still not working. We can check the compilation report. You should always compile with --report=compilation-report.xml which produces a very human readable compilation report, where you can check things easily.

It lists included DLLs and data files, and often also why it is included, and as of recently it learned to output also modules that were used by a module, and modules that were attempted to be used, but not found.

Note

This newly tracked information about failed attempts to use a module are the basis of largely enhanced bytecode caching (demoted e.g. because too large or standard library) in latest Nuitka.

Nuitka will tell use here about the issue from its perspective. So a module is not found at runtime, but what happened at compile time. Only the report can tell.

Lets quote the compilation report snippets.

<module name="__main__" kind="PythonMainModule" reason="Root module">
  <optimization-time pass="1" time="0.25" />
  <optimization-time pass="2" time="0.01" />
  <module_usages>
    <module_usage name="tkinterweb" finding="absolute" line="1" />
    <module_usage name="tkinter" finding="absolute" line="4" />
    <module_usage name="Tkinter" finding="not-found" line="6" />
  </module_usages>
</module>

This is the main module. Even without giving you the source code, you can see that the example code does import tkinterweb and tkinter. And due to this being probably very old code, the Python2/Python3 module name difference is present, so it imports the Python3 name successfully, but not the Python2 name.

How do we know this is a bug or not? The reality is, we do by context knowledge, there is not a single best way to decide if an import that is not found represents an issue in the compilation or not. But this looks good. I am showing it to you for educational purpose mostly.

Now lets find the module that raised the ModuleNotFoundError exception.

<module name="tkinterweb" kind="CompiledPythonPackage" reason="Instructed by user to follow to all non-standard library modules.">
  <plugin-influence name="dll-files" influence="condition-used" condition="win32 and arch_x86" tags_used="win32,arch_x86" result="false" />
  <plugin-influence name="dll-files" influence="condition-used" condition="win32 and arch_amd64" tags_used="win32,arch_amd64" result="true" />
  <optimization-time pass="1" time="0.07" />
  <optimization-time pass="2" time="0.03" />
  <module_usages>
    <module_usage name="os" finding="absolute" line="1" />
    <module_usage name="sys" finding="built-in" line="1" />
    <module_usage name="sys" finding="built-in" line="27" />
    <module_usage name="os" finding="absolute" line="27" />
    <module_usage name="ntpath" finding="absolute" line="28" />
    <module_usage name="bindings" finding="not-found" line="31" />
    <module_usage name="htmlwidgets" finding="not-found" line="32" />
    <module_usage name="utilities" finding="not-found" line="33" />
    <module_usage name="traceback" finding="absolute" line="35" />
    <module_usage name="sys" finding="built-in" line="41" />
    <module_usage name="tkinter" finding="absolute" line="44" />
    <module_usage name="tkinter" finding="absolute" line="45" />
    <module_usage name="tkinter.messagebox" finding="relative" line="45" />
    <module_usage name="Tkinter" finding="not-found" line="47" />
    <module_usage name="tkMessageBox" finding="not-found" line="48" />
    <module_usage name="tkinter" finding="absolute" line="67" />
    <module_usage name="Tkinter" finding="not-found" line="69" />
  </module_usages>
</module>

At the top, you can see the plugin-influence. This is where the plugin records that it influenced. It records what conditions were checked, and the result. Actually further down, we got this.

<included_dll name="Tkhtml30.dll" dest_path="tkinterweb\tkhtml\Windows\64-bit\Tkhtml30.dll" source_path="C:\Python310_64\lib\site-packages\tkinterweb\tkhtml/Windows/64-bit\Tkhtml30.dll" package="tkinterweb" ignored="no" reason="Yaml config of 'tkinterweb'" />

But that is now why we are here. You can also see the imports being done. They are given with line numbers and the one we care about is this snippet.

<module_usage name="bindings" finding="not-found" line="31" />

So Nuitka didn’t find it at compile time. And a quick check with Python on the prompt would reveal that this name is not importable. So now we switch to the source code of the trouble making module. There is no tool for that yet, typically just do this manually:

>>> import tkinterweb
>>> tkinterweb
<module 'tkinterweb' from 'C:\\Python310\\lib\\site-packages\\tkinterweb\\__init__.py'>

This is a clickable link in my Visual Code terminal, and after I click it and go to the line, what we see is:

import sys, os
sys.path.append(os.path.dirname(os.path.realpath(__file__)))

try:
    from bindings import TkinterWeb
    from htmlwidgets import HtmlFrame, HtmlLabel
    from utilities import Notebook
except (ImportError, ModuleNotFoundError):
    # Useless code goes here.
    ...

What strikes immediately is that Visual Code agrees, and displays the imported names a color used for modules that it couldn’t resolve. And actually the first like on top there is revealing rather rare code. This package is extending the global import path with its package contents. In this way, what would be tkinterweb.bindings is available as bindings after the module has been imported at runtime.

Expanding the PYTHONPATH is therefore our next step. Since I am using bash, I can prefix the call to Nuitka with PYTHONPATH='C:\\Python310_64\\lib\\site-packages\\tkinterweb' and low and behold, it works with this. Compilation takes longer and includes more modules, and the initial message is gone.

So, how to resolve this. Nuitka has gained a feature dedicated to this. It will be nice if this was automatically resolved at compile time, which is well could, note has been taken that there is value in tracking expanding sys.path at compile time.

There is another section called import-hacks and it too recently gained a new feature dedicated to this.

- module-name: 'tkinterweb'
  import-hacks:
    - global-sys-path:
        # This package forces itself into "sys.path" and expects absolute
        # imports to be available.
        - ''

We can here provide a list of relative paths, that are added when a package is imported to the search path of Nuitka. With this we can drop the PYTHONPATH which while being a nice workaround, required using absolute paths of the install, never easy to handle.

With this it now works fully automatically. One issue remains. The compiled program does not need the sys.path trick at runtime. And for isolation purposes, sys.path ought to be empty, so what we do we do with this here?

Step 4 - Cleanup the code

In order to get rid of that code, we can use the anti-bloat mechanism. It is very powerful and can do all sorts of things, but today we got a simple task for it.

This is the troubling line.

sys.path.append(os.path.dirname(os.path.realpath(__file__)))

There are many ways to change this, it’s always good be at less invasive as possible, so we do not want to append. We could prefix that line with if False:, but that typically only works well for single liners. What we can do rather generally is something like this:

sys.path.append(os.path.dirname(os.path.realpath(__file__)))
# -> we want this instead:
(os.path.dirname(os.path.realpath(__file__)))

Notice that just not calling will be good enough and extremely likely robust against all kinds of formatting changes, multiple lines, etc. and probably also very applicable should be encounter similar ones.

So this can be expressed with the following yaml snippet.

- module-name: 'tkinterweb'
  anti-bloat:
    - description: 'remove "sys.path" hack'
      replacements_plain:
        'sys.path.append': ''

And to know what effect it had and to see the wonders if anti-bloat in general, you can use --show-source-changes and output the diffs done on module source changes.

--- original
+++ modified

@@ -25,7 +25,7 @@

 """

 import sys, os
-sys.path.append(os.path.dirname(os.path.realpath(__file__)))
+(os.path.dirname(os.path.realpath(__file__)))

 try:
     from bindings import TkinterWeb

So, now this is perfect. Just need to add more OS specific branches, maybe also for the data files include more selectively, then this is perfect.

Final remarks

I am hoping you will find this very helpful information and will join the effort to make packaging for Python work out of the box. Adding support for tkinterweb was a little more complex than your typical package. The OS specific DLLs in different places are relatively unusual, although it has been seen before and will be gain.

This is a simpler example, that is way less complex, with all defaults just working.

- module-name: 'lightgbm.libpath'
  dlls:
    - from_filenames:
        prefixes:
          - 'lib_lightgbm'

Please review the guidelines for contributing, and esp. make sure to install the commit hook as described, or run bin/autoformat-nuitka-source --yaml at least, so the CI will not complain about formatting and we will have consistent files.

The last hot fixes of 1.3 already have user provided packaging enhancements that add dependencies and anti-bloat. We might discuss those in the next installment.

Nuitka Release 1.3

2022-12-28T09:58:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of performance work, that should specifically be useful on Windows, but also generally. A bit of scalability work has been applied, and as usual many bug fixes and small improvements, many of which have been in hotfixes.

Bug Fixes

macOS: Framework build of PySide6 were not properly supporting the use of WebEngine. This requires including frameworks and resources in new ways, and actually some duplication of files, making the bundle big, but this seems to be unavoidable to keep the signature intact.
Standalone: Added workaround for dotenv. Do not insist on compiled package directories that may not be there in case of no data files. Fixed in 1.2.1 already.
Python3.8+: Fix, the ctypes.CDLL node attributes the winmode argument to Python2, which is wrong, it was actually added with 3.8. Fixed in 1.2.1 already.
Windows: Attempt to detect corrupt object file in MSVC linking. These might be produced by cl.exe crashes or clcache bugs. When these are reported by the linker, it now suggests to use the --clean-cache=ccache which will remove it, otherwise there would be no way to cure it. Added in 1.2.1 already.
Standalone: Added data files for folium package. Added in 1.2.1 already.
Standalone: Added data files for branca package. Added in 1.2.1 already.
Fix, some forms try that had exiting finally branches were tracing values only assigned in the try block incorrectly. Fixed in 1.2.2 already.
Alpine: Fix, Also include libstdc++ for Alpine to not use the system one which is required by its other binaries, much like we already do for Anaconda. Fixed in 1.2.2 already.
Standalone: Added support for latest pytorch. One of our workarounds no longer applies. Fixed in 1.2.2 already.
Standalone: Added support for webcam on Windows with opencv-python. Fixed in 1.2.3 already.
Standalone: Added support for pytorch_lightning, it was not finding metadata for rich package. Fixed in 1.2.4 already.

For the release we found that pytorch_lightning may not find rich installed. Need to guard version() checks in our package configuration.
Standalone: Added data files for dash package. Fixed in 1.2.4 already.
Windows: Retry replace clcache entry after a delay, this works around Virus scanners giving access denied while they are checking the file. Naturally you ought to disable those for your build space, but new users often don’t have this. Fixed in 1.2.4 already.
Standalone: Added support for scipy 1.9.2 changes. Fixed in 1.2.4 already.
Catch corrupt object file outputs from gcc as well and suggest to clean cache as well. This has been observed to happen at least on Windows and should help resolve the ccache situation there.
Windows: In case clcache fails to acquire the global lock, simply ignore that. This happens sporadically and barely is a real locking issue, since that would require two compilations at the same time and for that it largely works.
Compatibility: Classes should have the f_locals set to the actual mapping used in their frame. This makes Nuitka usable with the multidispatch package which tries to find methods there while the class is building.
Anaconda: Fix, newer Anaconda versions have TCL and Tk in new places, breaking the tk-inter automatic detection. This was fixed in 1.2.6 already.
Windows 7: Fix, onefile was not working anymore, a new API usage was not done in a compatible fashion. Fixed in 1.2.6 already.
Standalone: Added data files for lark package. Fixed in 1.2.6 already.
Fix, pkgutil.iter_modules without arguments was given wrong compiled package names. Fixed in 1.2.6 already.
Standalone: Added support for newer clr DLLs changes. Fixed in 1.2.7 already.
Standalone: Added workarounds for tensorflow.compat namespace not being available. Fixed in 1.2.7 already.
Standalone: Added support for tkextrafont. Fixed in 1.2.7 already.
Python3: Fix, locals dict test code testing if a variable was present in a mapping could leak references. Fixed in 1.2.7 already.
Standalone: Added support for timm package. Fixed in 1.2.7 already.
Plugins: Add tls to list of sensible plugins. This enables at least pyqt6 plugin to do networking with SSL encryption.
Standalone: Added implicit dependencies of sklearn.cluster.
FreeBSD: Fix, fcopyfile is no longer available on newest OS version, and include files for sendfile have changed.
MSYS2: Add back support for MSYS Posix variant. Now onefile works there too.
Fix, when picking up data files from command line and plugins, different exclusions were applied. This has been unified to get better coverage for avoiding to include DLLs and the like as data files. DLLs are not data files and must be dealt with differently after all.

New Features

UI: Added new option for cache disabling --disable-cache that accepts all and cache names like ccache, bytecode and on Windows, dll-dependencies with selective values.

Note

The clcache is implied in ccache for simplicity.
UI: With the same values as --disable-cache Nuitka may now also be called with --clean-cache in a compilation or without a filename argument, and then it will erase those caches current data before making a compilation.
macOS: Added --macos-app-mode option for application bundles that should run in the background (background) or are only a UI element (ui-element).
Plugins: In the Nuitka Package Configuration files, the when allows now to check if a plugin is active. This allowed us to limit console warnings to only packages whose plugin was activated.
Plugins: Can now mark a plugin as a GUI toolkit responsible with the consequence that other toolkit detector plugins are all disabled, so when using tk-inter no longer will you be asked about PySide6 plugin, as that is not what you are using apparently.
Plugins: Generalized the GUI toolkit detection to include tk-inter as well, so it will now point out that wx and the Qt bindings should be removed for best results, if they are included in the compilation.
Plugins: Added ability to provide data files for macOS Resources folder of application bundles.
macOS: Fix, Qt WebEngine was not working for framework using Python builds, like the ones from PyPI. This adds support for both PySide2 and PySide6 to distribute those as well.
MSYS2: When asking a CPython installation to compress from the POSIX Python, it crashed on the main filename being not the same.
Scons: Fix, need to preserve environment attached modes when switching to WinLibs gcc on Windows. This was observed with MSYS2, but might have effects in other cases too.

Optimization

Python3.10+: When creating dictionaries, lists, and tuples, we use the newly exposed dictionary free list. This can speedup code that repeatedly allocates and releases dictionaries by a lot.
Python3.6+: Added fast path to dictionary copy. Compact dictionaries have their keys and values copied directly. This is inspired by a Python3.10 change, but it is applicable to older Python as well, and so we did.
Python3.9+: Faster compiled object creation, esp. on Python platforms that use a DLLs for libpython, which is a given on Windows. This makes up for core changes that went unnoticed so far and should regain relative speedups to standard Python.
Python3.10+: Faster float operations, we use the newly exposed float free list. This can speed up all kinds of float operations that are not doable in-place by a lot.
Python3.8+: On Windows, faster object tracking is now available, this previously had to go through a DLL call, that is now removed in this way as it was for non-Windows only so far.
Python3.7+: On non-Windows, faster object tracking is now used, this was regressed when adding support for this version, becoming equally bad as all of Windows at the time. However, we now managed to restore it.
Optimization: Faster deep copy of mutable tuples and list constants, these were already faster, but e.g. went up from 137% gain factor to 201% on Python3.10 as a result. We now use guided a deep copy, which then has the information, what types it is going to copy, removing the need to check through a dictionary.
Optimization: Also have own allocator function for fixed size objects. This accelerates allocation of compiled cells, dictionaries, some iterators, and lists objects.
More efficient code for object initialization, avoiding one DLL calls to set up our compiled objects.
Have our own PyObject_Size variant, that will be slightly faster and avoids DLL usage for len and size hints, e.g. in container creations.
Avoid using non-optimal malloc related macros and functions of Python, and instead of the fasted form generally. This avoids Python DLL calls that on Windows can be particularly slow.
Scalability: Generated child mixins are now used for the generated package metadata hard import nodes calls, and for all instances of single child tuple containers. These are more efficient for creation and traversal of the tree, directly improving the Python compile time.
Scalability: Slightly more efficient compile time constant property detections. For frozenset there was not need to check for hashable values, and some branches could be replaced with e.g. defining our own EllipsisType for use in short paths.
Windows: When using MSVC and LTO, the linking stage was done with only one thread, we now use the proper options to use all cores. This is controlled by --jobs much like C compilation already is. For large programs this will give big savings in overall execution time. Added in 1.2.7 already.
Anti-Bloat: Remove the use of pytest for dash package compilation.
Anti-Bloat: Remove the use of IPython for dotenv, pyvista, python_utils, and trimesh package compilation.
Anti-Bloat: Remove IPython usage in rdkit improving compile time for standalone by a lot. Fixed in 1.2.7 already.
Anti-Bloat: Avoid keras testing framework when using that package.

Organizational

Plugins: The numpy plugin functionality was moved to Nuitka package configuration, and as a result, the plugin is now deprecated and devoid of functionality. On non-Windows, this removes unused duplications of the numpy.core DLLs.
User Manual: Added information about macOS entitlements and Windows console. These features are supported very well by Nuitka, but needed documentation.
UI: Remove alternative options from --help output. These are there often only for historic reasons, e.g. when an option was renamed. They should not bother users reading them.
Plugins: Expose the mnemonics option to plugin warnings function, and use it for pyside2 and pyqt5 plugins.
Quality: Detect trailing/leading spaces in Nuitka package configuration description values during their automatic check.
UI: Detect the CPython official flavor on Windows by comparing to registry paths and consider real prefixes, when being used in virtualenv more often, e.g. when checking for CPython on Apple.
UI: Enhanced --version output to include the C compiler selection. It is doing that respecting your other options, e.g. --clang, etc. so it will be helpful in debugging setup issues.

UI: Some error messages were using %r rather than '%s' to output file paths, but that escaped backslashes on Windows, making them look worse, so we changed away from this.
UI: Document more clearly what --output-dir actually controls.
macOS: Added options hint that the Foundation module requires bundle mode to be usable.
UI: Allow using both --follow-imports and --nofollow-imports on command line rather than erroring out. Simply use what was given last, this allows overriding what was given in project options tests should the need arise.
Reports: Include plugin reasons for pre and post load modules provided. This solves a TODO and makes it easier to debug plugins.
UI: Handle --include-package-data before compilation, removing the ability to use pattern. This makes it easier to recognize mistakes without a long compilation and plugins can know them this way too.
GitHub: Migration workflows to using newer actions for Python and checkout. Also use newer Ubuntu LTS for Linux test runner.
UI: Catch user error of running Nuitka with the pythonw binary on Windows.
UI: Make it clear that MSYS2 defaults to --mingw64 mode. It had been like this, but the --help output didn’t say so.
GitHub: Updated contribution guidelines for better readability.
GitHub: Use organisation URLs everywhere, some were still pointing to the personal rather than the organisation one. While these are redirected, it is not good to have them like this.
Mastodon: Added link to https://fosstodon.org/@kayhayen to the PyPI package and User Manual.

Cleanups

Nodes for hard import calls of package meta data now have their base classes fully automatically created, replacing what was previously manual code. This aims at making them more consistent and easier to add.
When adding the new Scons file for C compiler version output, more values that are needed for both onefile and backend compilation were moved to centralized code, simplifying these somewhat again.
Remove unused main_module tag. It cannot happen that a module name matches, and still thinks of itself as __main__ during compilation, so that idea was unnecessary.
Generate the dictionary copy variants from template code rather than having manual duplications. For dict.copy(), for deep copy (needed e.g. when there are escaping mutable keyword argument constant values in say a function call), and for **kw value preparation in the called function (checking argument types), we have had diverged copies, that are now unified in a single Jinja2 template for optimization.
Plugins: Also allow providing generators for providing extra DLLs much like we already do for data files.
Naming of basic tests now makes sure to use a Test suffix, so in Visual Code selector they are more distinct from Nuitka code modules.
Rather than populating empty dictionaries, helper code now uses factory functions to create them, passing keys and values, and allowing values to be optional, removing noisy if branches at call side.
Removed remaining PyDev annotations, we don’t need those anymore for a long time already.
Cleanup, avoid lists objects for ctypes defined functions and their arglist, actually tuples are sufficient and naturally better to use.
Spelling cleanups were resumed, as an ongoing action.

Tests

Added construct test that demonstrates the mutable constant argument passing for lists to see the performance gains in this area too.
Made construct runner --diff output usable for interactive usage.
Repaired Nuitka Speedcenter, but it’s not yet too usable for general consumption. More work will be needed there, esp. to make comparisons more accessible for the general public.

Summary

The major achievement of this release was the removal of the long lived numpy plug-in, replacing it with package based configuration, that is even more optimal and works perfectly across all platforms on both important package installation flavors.

This release has a lot of consolidation efforts, but also as a result of 3.11 investigations, addresses a lot of issues, that have crept in over time with Python3 releases since 3.7, each time, something had not been noticed. There will more need for investigation of relative performance losses, but this should address the most crucial ones, and also takes advantage of optimization that had become with 3.10 already.

There is also some initial results from cleanups with the composite node tree structure, and how it is managed. Generated “child(ren) having” mixins, allow for faster traversal of the node tree.

Some technical things also improved in Scons. Using multiple cores in LTO with MSVC with help this a lot, although for big compilations --lto=no probably has to be recommended still.

More anti-bloat work on more packages rounds up the work.

For macOS specifically, the WebEngine support is crucial to some users, and the new --macos-app-mode with more GUI friendly default resolves long standing problems in this area.

And for MSYS2 and FreeBSD, support has been re-activated, so now 4 OSes work extremely well (others too likely), and on those, most Python flavors work well.

The performance and scalability improvements are going to be crucial. It’s a pity that 3.11 is not yet supported, but we will be getting there.

Python 3.11 and Nuitka

2022-12-10T11:29:24+00:00

In my all in with Nuitka post, I promised to give you an update on the current status of Python 3.11 support and Nuitka, and this is the report on my busy bee activities there.

What is now

Current Nuitka 1.2 and likely the coming 1.3 release, will react pretty hefty to attempts to compile with Python3.11, so this command:

python3.11 -m nuitka tests/basics/Asserts.py

Gives the following output:

Nuitka:WARNING: The version '3.11' is not currently supported. Expect
Nuitka:WARNING: problems.
FATAL: The Python version '3.11' is not supported by '1.2.7', but an upcoming release will add it.

What you can do?

As a mere mortal, what you can and should do is to consider becoming a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All essential packaging and performance features are free, and I have put incredible amounts of works in this, and I need to now make a living off it.

And working on 3.11 support, just so tings then continue to work, mostly on my own deserves financial support. Once a year, for weeks on end, I am re-implementing features that a collective of developers came up with.

Otherwise, unless you are CPython core developer responsible for the changes, I am afraid, not a lot. Having to know both Nuitka and CPython details is a tall ask. I might be the only one to do this for a while. I will however strive to get more people involved each time.

That is not to say this does not happen. Not too long ago, I merged proposed changes that will help Nuitka with 3.12 compatibility. This would be nice if it was more frequent, and the fact that it is not, is probably my fault entirely.

What was done?

In the summer, with release candidates, there was some refactoring done that aimed at making Nuitka compile, at least with less errors. One of the changes that was addressed at the time was the reduction of exception state to only a value.

In Python during an exception, you typically have a exception “type”, “value”, and a “traceback”. And these used to be sort of independent objects, which then through a process called “normalization” got aligned. E.g. when raising an exception, you would simply put a ZeroDivisionError as a type, and a string as the value, and only on normalization, which was attempted to be done as late as possible, the actual exception object was created.

For 3.11 now, in some parts (but not all of course), this got reduced, and making Nuitka cope with that was a bigger task, and even one that had then introduced a regression to pre-3.11, i.e. affecting other versions. I had to make a hotfix because of this for the release that included these changes. This is of course, due to also cleaning up the code that was changed, to reduce e.g. duplications that were considered harmless in the past.

Now, during the 1.3 releases cycle, many more changes have been addressed. The CPython core is using now “interpreter frames”, which are lightweight non-objects, and does only create a frame when an exception is needed. Nuitka on the other hand used to cache frame objects, that it also created only when needed. Adapting the interactions with the frame stack appears somewhat solved, but actually poses some bugs when dealing with generators that raise exceptions, and is currently being analysed.

The Process

When the goal is getting 3.11 supported (or any later Python, this will repeat forever), in a first step it just doesn’t compile on the C level anymore. First, all kinds of changes need to be addressed that show when compiling on the C level. Or extra time spent to disable some things to migrate only later (which I tend to not do), but the unfortunate truth is, that at that point, running with 3.11 to find out things, is not yet an option.

Once compiled 3.11 started being executable, it was hard crashing (worst case) and asserting (good case) like hell. Many assumptions made proved to be untrue, or things that used to be true, are no more true. That is pretty normal when you mingle with internal details of course.

Right now reference leaks are being seen with generators, and so are crashes. This is probably the same issue, where some link between objects, some reference to an object is taken, and what is most likely, in other instance, it is lost, which then causes corrupted objects. These are among the hardest things to debug in Python, often the symptom only shows later. Many times development in this phase is about adding assertions. This is where we are now.

The next phase then is to run the CPython 3.10 test suite compiled with 3.11 and compare its results to running with plain 3.11, and expect it to be the same. It is normal for CPython 3.10 tests to fail with 3.11, and this does two things.

Either the test passes where it should not, then 3.11 added extra checks, e.g. making it illegal to provide certain types of inputs that were legal in the past, or it fails where 3.11 does not, then it highlights a behavior change or a mistake in the migration. Another case of course is it crashing. Then some bug in e.g. frame handling or exception handling might cause that.

During this phase it is esp. initially very hard to assess where we stand. Often one bug affects many tests with different symptoms. We we just look for things that are easy to analyse.

In the next phase, the 3.11 test suite is used in the same way. This will often uncover new features, new behaviors, newly allowed things. For example Nuitka might reject an input, but that is correct for 3.10, but not for 3.11 anymore, this this will be added.

This phase has not even started.

One interesting case is new test coverage. Often is has happened in the past that something was incompatible with 3.10 already, but that has never been seen. Of course, in this instance, the modification will be backported, this is often revealed when running the 3.11 tests compiled with 3.11 and comparing the result with pure 3.10.

And this is also not even started. Once the test suite is adapted for Nuitka’s needs (avoiding random outputs, to make tests comparable e.g.) this could be started, and would enhance 3.10 compatibility already with its findings.

Intermediate Results

Once the Python3.11 more or less well passed the 3.10 test suite without big issues, experimental support for it will be added. The warning from above will be given, but the error will cease.

But only once the CPython 3.11 test suite is completely passing, it will be added as supported officially. Again, not all tests have to pass perfectly for this, cosmetic things are not counted of course.

When

Very hard to predict. Supporting existing Nuitka is also a side tracking thing, that makes it unclear how much time I will have for it. But the worst things with debugging is that I just never know how much time it will be. In past releases, the time has varied. I do not know at this time, what will have to be done for support of 3.11, it clearly is one of the harder ones. But I am hoping these core changes were the bulk of the work.

Benefits for older Python too

When I noticed that the dictionary implementation had heavily changed, and Nuitka’s “illegal” friendship (accessing internal details of it) caused it to not compile, step one was to compare current details with 3.10 and find that there are a bunch of things that were added over time. For example one fast branch that was added to 3.10, Nuitka now has it too, for 3.6 or higher. So we are also back-porting some speed improvements.

Also the allocator of 3.11 changed, and as a result, the allocator for all older Python will be way faster to use, esp. with platforms that use DLLs.

There are other benefits, e.g. 3.10 is exposing free lists (re-usable objects without going through allocation) it is using, allowing Nuitka to accelerate dictionary object creation esp. on Windows, and so on.

Expected results

People tend to expect that gains from Nuitka and enhancements of CPython stack up. The truth of the matter is, no they do not. CPython is applying some tricks that Nuitka already did. Not using its bytecode will then become less of a benefit, but that’s OK, this is not what Nuitka is about.

We need to get somewhere else entirely anyway, in terms of speed up. I will be talking about PGO and C types a lot in the coming year, that is at least the hope. That is where the “all in” part kicks in.

Final Words

Look ma, I posted about something that is not complete. I am getting better.

All in with Nuitka

2022-12-09T08:22:42+00:00

After more than 10 years of working on the Python compiler Nuitka, and at the age of 50, I have finally decided to take the plunge. My projects “Nuitka” and “Nuitka commercial” are now my day job, and I quit my work in the Air Traffic Community, after working on ARTAS (a software to create an Air Situation Picture from radar and other sensors) for 20 years.

Why now?

That is very good question. I feel I could have done this a long time ago, and my heart was torn for a while, and also I prepared this for a while now. Listing back to interviews from 4 years ago, it seems though as if it had not been on the radar (pun intended).

So then I launched Nuitka commercial around 1.5 years ago, first as vapor ware, with early bird customers, and meanwhile it has matured to a point, where it is the obvious solution to many corporate users, with big guys picking it up and small business as well. From startups, big name technology giants, universities, mid sized companies, and from individuals, I have all kinds of support.

I liked my old job very much, but it was getting old, and I felt like Nuitka needs me to fulfil my mission. And my wife needed my help with driving her to work, which was getting the load to high, something had to give. And it seems nothing related to what is called Nuitka (my wife, and the compiler share the name) was going to be that.

What will change?

Already, ever since I launched Nuitka commercial, Nuitka has become way more of a out of the box experience. It just works, and covers more packages than ever before, and steps have been taken, to make it possible to extend this even further with your help. Sure there are still shortcoming.

But Nuitka has become very polished, and shows this very much that I also love details and doing things correctly. I have conducted Nuitka from day one as an attempt to create a software that would be usable to the enterprise with a million lines of code, that they cannot check for unsupported language features, so Nuitka is just fully compatible, to a level of absurdity.

Having conducted Nuitka on a level of where a corporate business can throw it’s code at Nuitka and have it just work, and reliably, this will only improve now.

Naturally bugs have happened once in a while, but I was always more careful. You know, like if this was for safety critical software, on which lives might depend. Hard to not be affected by that stance.

So lets talk about other ways I plan to expand.

Nuitka Package Configuration

Extending Nuitka with tweaks for package has become increasingly simpler. Many previous plugins, e.g. for torch, tensorflow, and most recently numpy, and definitely also the Qt binding ones in the future, have been or will be converted to Yaml configuration applied by generic plugins that deal with DLLs, data-files, tweaks (called anti-bloat) etc.

Adding stuff there is simple, and as of recently, there is even documentation for it. This is going to be explained in a series of posts detailing Nuitka Package Configuration, under the package_config tag.

I must say it’s so much fun, I love working with it. But the goal is for this to have you join it, and PRs are indeed starting to come in more frequently it seems from first glance, but I need to make this more known.

Streaming

I have experimented with streaming in the past, and I absolutely love it. Without my day job, effectively working two jobs at the same time, I expect to have the time to do more of them in December and then on a regular basis.

Python 3.11

There will be a separate post on this issue later this week. I am scrambling to add support for it and it feels very far already. Many basic tests already work, but more debugging will be needed. I hope Nuitka 1.3 will add support for it, at least on the usual experimental level.

How you can help?

Please become a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All essential packaging and performance features are free, and I have put incredible amounts of works in this, and I need to now make a living off it.

Right now, I am far from being able to sustain my income from it, and I am taking a risk with this effort, hoping for support from you.

Nuitka commercial makes creating a Windows Service a blast. It has a PySide2 that is compatible with Nuitka (I did make PySide6 compatible myself, and backported it to PySide2), so if you need to stay on old software. And for IP protection, it is just incredibly important to not ship bytecode at all, and with Nuitka commercial, it protects your data and files at runtime too.

I am bad at marketing, but good at software, which is one more reason to purchase it. You know you love these kinds of people. Was that good marketing? Also, there are share buttons on this page. Please try them out and let people know about this.

In future postings, I will make it a regular thing to ask for help with some of the technical tasks. Stay tuned, there might be something that you can help with.

Becoming more active

I am the kind of person that would much rather do good stuff than to talk about it, like when it finally supports Python 3.11, normal old me would just mention it maybe on Twitter once, and have it in the changelog only.

I need to get away from that. I need to resume TWN (This week in Nuitka) and have all plans to do it. Obviously I can get caught up in things, and may just always do it later when there is more or better news.

Nuitka Release 1.2

2022-11-26T09:23:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new compatibility features and a few new optimization, while again consolidating what we have. Scalability should be better in many cases.

Bug Fixes

Standalone: Added implicit dependency of thinc backend. Fixed in 1.1.1 already.
Python3.10: Fix, match statements with unnamed star matches could give incorrect results. Fixed in 1.1.1 already.
match x: case [*_, y]: ... # y had wrong value here.

Python3.9+: Fix, file reader objects must convert to str objects. Fixed in 1.1.1 already.

# This was the `repr` rather than a path value, but it must be usable
# like that too.
str(importlib.resources.files("package_name").joinpath("lala"))

Standalone: Added data file of echopype package. Fixed in 1.1.1 already.
Anti-Bloat: Remove non-sense warning of compiled pyscf. Fixed in 1.1.1 already.
macOS: Fix, in LTO mode using incbin can fail, switch to source mode for constants resources. Fixed in 1.1.2 already.
Standalone: Add support for sv_ttk module. Fixed in 1.1.2 already.
macOS: Fix, was no longer correcting libpython path, this was a regression preventing CPython for creating properly portable binary. Fixed in 1.1.2 already.
macOS: Fix, main binary was not included in signing command. Fixed in 1.1.3 already.
Standalone: Added implicit dependency of orjson. Due to zoneinfo not being automatically included anymore, this was having a segfault. Fixed in 1.1.3 already.
Standalone: Added support for new shapely. Fixed in 1.1.4 already.
macOS: Ignore extension module of non-matching architecture. Some wheels contain extension modules for only x86_64 arch, and others contain them only for arm64, preventing the standalone build. Fixed in 1.1.4 already.
Standalone: Added missing sklearn dependencies. Fixed in 1.1.4 already.
Fix, packages available through relative import paths could be confused with the same ones imported by absolute paths. This should be very hard to trigger, by normal users, but was seen during development. Fixed in 1.1.4 already.
Standalone: Apply import hacks for pywin32 modules only on Windows, otherwise it can break e.g. macOS compilation. Fixed in 1.1.4 already.
Windows: More robust DLL dependency caching, otherwise e.g. a Windows update can break things. Also consider plugin contribution, and Nuitka version, to be absolutely sure, much like we already do for bytecode caching. Fixed in 1.1.4 already.
Standalone: Fix, seaborn needs the same workaround as scipy for corruption with MSVC. Fixed in 1.1.4 already.
UI: Fix, the options-nanny was no longer functional and therefore failed to warn about non working options and package usages. Fixed in 1.1.5 already.
macOS: Do not use extension modules of non-matching architecture. Fixed in 1.1.5 already.
Windows: Fix, resolving symlinks could fail for spaces in paths. Fixed in 1.1.6 already.
Standalone: Added missing DLL for lightgbm module. Fixed in 1.1.6 already.
Compatibility: Respect super module variable. It is now possible to have a module level change of super but still get compatible behavior with Nuitka. Fixed in 1.1.6 already.
Compatibility: Make sure we respect super overloads in the builtin module. Fixed in 1.1.6 already.
Fix, the anti-bloat replacement code for numpy.testing was missing a required function. Fixed in 1.1.6 already.
Fix, importlib.import_module static optimization was mishandling a module name of . with a package name given. Fixed in 1.1.6 already.
macOS: Fix, some extension modules use wrong suffixes in self references, we need to not complain about this kind of error. Fixed in 1.1.6 already.
Fix, do not make ctypes.wintypes a hard import on non-Windows. Nuitka asserted against it failing, where some code handles it failing on non-Windows platforms. Fixed in 1.1.6 already.
Standalone: Added data files for vedo package. Fixed in 1.1.7 already.
Plugins: Fix, the gi plugin did always set GI_TYPELIB_PATH even if already present from user code. Also it did not handle errors to detect its value during compile time. Fixed in 1.1.7 already.
Standalone: Added missing dependencies for sqlalchemy to have all SQL backends working. Fixed in 1.1.7 already.
Added support Nixpkgs’s default non-writable HOME directory. Fixed in 1.1.8 already.
Fix, distribution metadata name and package name need not align, need to preserve the original looked up name from importlib.metadata.distribution call. Fixed in 1.1.8 already.
Windows: Fix, catch usage of unsupported CLCACHE_MEMCACHED mode with MSVC compilation. It is just unsupported.
Windows: Fix, file version was spoiled from product version if it was the only version given.
Windows: The default for file description in version information was not as intended.
Plugins: Workaround for PyQt5 as contained in Anaconda providing wrong paths from the build machine.
macOS: After signing a binary with a certificate, compiling the next one was crashing on a warning about initially creating an ad-hoc binary.
Fix, detect case of non-writable cache path, make explaining error exit rather than crashing attempting to write to the cache.
macOS: Added support for pyobjc in version 9.0 or higher.

New Features

Python3.11: For now prevent the execution with 3.11 and give a warning to the user for a not yet supported version. This can be overridden with --experimental=python311 but at this times will not compile anything yet due to required and at this time missing core changes.
macOS: Added option --macos-sign-notarization that signs with runtime signature, but requires a developer certificate from Apple. As its name implies, this is for use with notarization for their App store.
DLLs used via delvewheel were so far only handled in the zmq plugin, but this has been generalized to cover any package using it. With that, e.g. shapely just works. This probably helps many other packages as well.
Added __compiled__ and __compiled_constant__ attributes to compiled functions.

With this, it can be decided per function what it is and bridges like pyobjc can use it to create better code on their side for constant value returning functions.
Added support_info check to Nuitka package format. Make it clear that pyobjc is only supported after 9.0 by erroring out if it has a too low version. It will not work at all before that version added support in upstream. Also using this to make it clear that opencv-python is best supported in version 4.6 or higher. It seems e.g. that video capture is not working with 4.5 at this time.
Added --report-template which can be used to provide Jinja2 templates to create custom reports, and refer to built-in reports, at this time e.g. a license reports.

Optimization

Trust the absence of a few selected hard modules and convert those to static raises of import errors.
For conditional nodes where only one branch exits, and the other does not, no merging of the trace collection should happen. This should enhance the scalability and leads to more static optimization being done after these kinds of branches on variables assigned in such branches.
```
if condition1:
   a = 1
else:
   raise KeyError

if condition2:
   b = 1

# Here, "a" is known to be assigned, but before it was only "maybe"
# assigned, like "b" would have to be since, the branch may or may
# not have been taken.
```

Do not merge tried blocks that are aborting with an exception handler that is not aborting. This is very similar to the change for conditional statements, again there is a control flow branch, that may have to be merged with an optional part, but sometimes that part is not optional from the perspective of the code following.

try:
   ... # potentially raising, but not aborting code
   return something() # this aborts
except Exception:
   a = 1

try:
   ... # potentially raising, but not aborting code
except Exception:
   b = 1

# Here, "a" is known to be assigned, but before it was only "maybe"
# assigned, like "b" would have to be since, the branch may or may
# not have been taken.

Exception matches were annotating a control flow escape and an exception exit, even when it is known that no error is possible to be happening that comparison.
```
try:
   ...
except ImportError: # an exception match is done here, that cannot raise
   ...
```
Trust importlib.metadata.PackageNotFoundError to exist, with this some more metadata usages are statically optimized. Added in 1.1.4 already.
Handle constant values from trusted imports as trusted values. So far, trusted import values were on equal footing to regular variables, which on the module level could mean that no optimization was done, due to control flow escapes happening.
```
# Known to be False at compile time.
from typing import TYPE_CHECKING
...

if TYPE_CHECKING:
   from something import normally_unused_unless_type_checking
```
In this code example above, the static optimization was not done because the value may be changed on the outside. However, for trusted constants, this is no longer assumed to be happening. So far only if typing.TYPE_CHECKING: using code had been optimized.
macOS: Use sections for main binary constants binary blob rather than C source code (which we started in a recent hotfix due to LTO issues with incbin) and onefile payload. The latter enables notarization of the onefile binary as well and makes it faster to unpack as well.
Windows: Do not include DLLs from SxS. For PyPI packages these are generally unused, and self compiled modules won’t be SxS installations either. We can add it back where it turns out needed. This avoids including comctl32 and similar DLLs, which ought to come from the OS, and might impede backward compatibility only.
Disabled C compilation of very large azure modules.
The per module usage information of other modules was only updated in first pass was used in later passes. But since they can get optimized away, we have to update every time, avoiding to still include unused modules.
Anti-Bloat: Fight the use of dask in xarray and pint, adding a mode controlling its use. This is however still incomplete and needs more work.
Fix, the anti-bloat configuration for rich.pretty introduced a SyntaxError that went unnoticed. In the future compilation will abort when this happens.
Standalone: Added support for including DLLs of llvmlite.binding package.
Anti-Bloat: Avoid using pywin32 through pkg_resources import. This seems rather pointless and follows an optimization done for the inline copy of Nuitka already, the ctypes code path works just fine, and this may well be the only reason why pywin32 is included, which is by itself relatively large.
Cache directory contents when scanning for modules. The sys.path and package directories were listed over and over, wasting time during the import analysis.
Optimization: Was not caching not found modules, but retrying every time for each usage, potentially wasting time during import analysis.
Anti-Bloat: Initial work to avoid pytest in patsy, it is however incomplete.

Organizational

User Manual: Explain how to create 64/32 bits binaries on Windows, with there being no option to control it, this can otherwise be a bit unobvious that you have to just use the matching Python binary.
UI: Added an example for a cached onefile temporary location spec to the help output of --onefile-tempdir-spec to make cached more easy to achieve in the proper way.
UI: Quote command line options with space in value better, no need to quote an affected command line option in its entirety, and it looks strange.
macOS: Catch user error of disabling the console without using the bundle mode, as it otherwise it has no effect.
macOS: Warn about not providing an icon with disabled console, otherwise the dock icon is empty, which just looks bad.
Debian: Also need to depend on glob2 packages which the yaml engine expects to use when searching for DLLs.
Debian: Pertain inline copies of modules in very old builds, there is e.g. no glob2 for older releases, but only recent Debian releases need very pure packages, our backport doesn’t have to do it right.
macOS: More reliable detection of Homebrew based Python. Rather than checking file system via its sitecustomize contents. The environment variables are only present to some usages.
Installations with pip did not include all license, README files, etc. which however was intended. Also the attempt to disable bytecode compilation for some inline copies was not effective yet.
Renamed pyzmq plugin to delvewheel as it is now absolutely generic and covers all uses of said packaging technique.
Caching: Use cache directory for cached downloads, rather than application directory, which is just wrong. This will cause all previously cached downloads to become unused and repeated.
Quality: Updated development requirements to latest black, isort, yamllint, and tqdm.
Visual Code: Added recommendation for extension for Debian packaging files.
Added warning for PyQt5 usage, since its support is very incomplete. Made the PyQt6 warning more concrete. It seems only Qt threading does not work, which is of course still significant. Instead PySide2 and PySide6 are recommended.
UI: Have dedicated options group for onefile, the spec for the temporary files is not a Windows option at all anymore. Also move the warnings group to the end, people need to see the inclusion related group first.
User Manual: Explain how to create 64/32 bits binaries on Windows, which is not too obvious.

Cleanups

Moved PySide plugins DLL search extra paths to the Yaml configuration. In this way it is not dependent on the plugin being active, avoiding cryptic errors on macOS when they are not found.
Plugins: Avoid duplicate link libraries due to casing. We are now normalizing the link library names, which avoids e.g. Shell32 and shell32 to be in the result.
Cleanups to prepare a PyLint update that so otherwise failed due to encountered issues.
Plugins: Pass so called build definitions for generically. Rather than having dedicated code for each, and plugins can now provide them and pass their index to the scons builds.
Onefile: Moved file handling code to common code reducing code duplication and heavily cleaning up the bootstrap code generally.
Onefile: The CRC32 checksum code was duplicated between constants blob and onefile, and has moved to shared code, with an actual interface to take the checksum.
Spelling cleanups resumed, e.g. this time more clearly distinguishing between run time and runtime, the first is when the program executes, but the latter can be an environment provided by a C compiler.

Tests

Tests: Added test that applies anti-bloat configuration to all found modules.
Tests: Tests: Avoid including unused nuitka.tools code in reflected test, which should make it faster. The compiler itself doesn’t use that code.

Summary

This release is again mainly a consolidation of previous release, as well as finishing off existing features. Optimization added in previous releases should have all regressions fixed now, again with a strong series of hotfixes.

New optimization was focused around findings with static optimization not being done, but still resulting in general improvements. Letting static optimization drive the effort is still paying off.

Scalability has seen improvements through some of the optimization, this time a lot less anti-bloat work has been done, and some things are only started. The focus will clearly now shift to making this a community effort. Expect postings that document the Yaml format we use.

For macOS specifically, with the sections work, onefile should launch faster, should be more compatible with signing, and can now be used in notarization, so for that platform, things are more round.

For Windows, a few issues with version information and onefile have been addressed. We should be able to use memory mapped view on this platform too, for faster unpacking of the payload, since it doesn’t have to go through the file anymore.

Nuitka Release 1.1

2022-10-16T10:56:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new compatibility features, while consolidating what we have. Scalability should be better in some cases.

Bug Fixes

Standalone: Enhanced dependency scan of dependent DLLs to forward the containing package, so it can be searched in as well. This fixed at least PySide on macOS. Fixed in 1.0.1 already.
macOS: Enhanced dependency detection to use normalized paths and therefore to be more stable. Fixed in 1.0.1 already.
Standalone: Added support for the networkx package which uses new support for a function decorator trying to copy function default values. Fixed in 1.0.1 already.
Standalone: Include data files for pandas.io.format package. This one has Jinja2 template files that will be needed when using this package.
Python3.10: Fix, could crash in case a class was not giving match arguments, but the user did attempt to match them. This happened e.g. with range objects. Fixed in 1.0.2 already.
Standalone: Added data files needed for pyenchant package. Fixed in 1.0.2 already.
Python3.10: Fix, matching sequence with as assignments in them didn’t check for sub-pattern given. Fixed in 1.0.2 already.
Standalone: Fix, do not attempt to list non-existent PATH entries on Windows, these can crash the dependency detection otherwise. Fixed in 1.0.2 already.
Standalone: Fix, on newer Linux, linux-vdso.so.1 appears in output of ldd in a way that suggests it may exist, which of course it does not, this is a kernel virtual DLL. Fixed in 1.0.3 already.
Fix, comparison expressions could give wrong results as a regression of the new release. Fixed in 1.0.3 already.
Fix, on older Python (before 3.6), it could crash on data files defined in the Yaml config. Fixed in 1.0.4 already.
Fix, binary operations could give wrong results as a regression of the new release. Fixed in 1.0.4 already.
Standalone: Added support for pyzbar package. Fixed in 1.0.5 already.
Standalone: Fix, empty directory structures were not working anymore due to a regression in the last release. Fixed in 1.0.5 already.
Windows: Fix, detected Pythons from Windows registry may of course fail to execute, because they were e.g. manually deleted. This would show e.g. in onefile compression. Fixed in 1.0.5 already.
Onefile: Fix, using a too old zstandard without finding another Python with a suitable one, lead to run time unpacking errors. Fixed in 1.0.6 already.
Fix, the inline copy of Jinja2 imported logging for no good reason, which lead to errors for users who have a module of the same name, that it was then using instead. Fixed in 1.0.6 already.
Fix, disable LTO mode for Anaconda Python, it is known to not work. Fixed in 1.0.6 already.
Linux: Fix, no need to insist on icon path for onefile anymore. Fixed in 1.0.6 already.
Standalone: Fix, the new version certifi was not working on Windows and 3.10 anymore. Fixed in 1.0.7 already.
Standalone: Added support for more rapidfuzz implicit dependencies. Fixed in 1.0.8 already.
Standalone: Added support for vibora. Fixed in 1.0.8 already.
Fix, must not expose module name objects to Python import hooks. Fixed in 1.0.8 already.
Fix, calls to bound methods of string values generated incorrect calls. Fixed in 1.0.8 already.
Fix, do not crash in version detection on gcc error exit querying of its version.
Standalone: Added back support for older versions of the pyzmq package.
Standalone: Ignore PATH elements that fail to be listed as a directory. It appears e.g. on Windows, folders can exist, despite being unusable in fact. These can then cause errors in DLL dependency scan. Also avoid having PATH set when executing dependency walker, it appears to use it even if not asked to.
Standalone: Added support for tzlocal package.
Python3.10: Fix, complex literals were not working for mappings in match statements.
Fix, bool built-in expressions were not properly annotating exception raises, where the value cannot raise on truth check.
Standalone: Added support for the agscheduler package. Plugins must be done manually still with explicit --include-module calls.
Standalone: Added support for using shapely in Anaconda as well.
Debian: Fix, versioned dependency for libzstd should also be in package, this should restore Nuitka package builds for Debian Jessie.
Standalone: Added support for vtk package.
Windows: Fix, avoid using pywin32 in our appdirs usage, it might be a broken installation and is optional to appdirs anyway, which then will fallback to using ctypes to make the lookups.
Standalone: Added support for more pandas versions.
Standalone: Adding support for mkl implicit DLL usage in Anaconda.
Standalone: Added support for jsonschema with Python 3.10.
Standalone: Added support for pyfiglet fonts data files.
Scons: Avoid gcc linker command line length limits for module mode too.
Standalone: Added data file of distributed.config.
Standalone: Add support for cv2 GUI on Linux, the Qt platform plugin is now included.
Fix, the anti-bloat configuration for numpy.testing tools exposed an incomplete suppress_warnings replacement that could lead to errors in some functions of numpy.
Standalone: Fix DLL dependency caching on Windows, need to consider DLL content of course too.
Standalone: Added missing dependency for torchvision.
Standalone: Added support for torchvision on Anaconda as well.
Standalone: Added support for panda3d.
Windows: Fix, need to make sure to use UTF-8 encoding for define values like company name. Otherwise the local system encoding is used, but the C compiler expects UTF-8 in wide literals. This may crash of give wrong results.
Standalone: Added facenet_torch data files.
Anaconda: Include libstdc++.so on Linux or else e.g. cv2 will not work with system library.
Windows: Fix, can have file version without a company name.

New Features

Python3.10: Added support for assignments in match alternatives | syntax.
Compatibility: Register Nuitka meta path based loader with pkg_resources such that checking resource presence with has_resource works too. This should also add support for using jinja2.PackageLoader, previously only jinja2.FileSystemLoader worked. Fixed in 1.0.1 already.
Compatibility: Make function __defaults__ attribute size changeable. For a long time, this was a relatively big issue for some packages, but now this is supported as well.
Compatibility: Added support for importlib.metadata.distribution and importlib_metadata.distribution functions as well importlib.metadata.metadata and importlib_metadata.metadata functions.
Onefile: Added support for including other binaries than the main executable in the payload. So far on non-Windows, we only made the main binary executable, hard coded, and nothing else. But Some things, e.g. Qt web engine, do require binaries to be used, and these no longer have the issue of missing x-bit on macOS and Linux now.
Standalone: Resolve executable path when called through symbolic link, which makes file resolutions work properly for it, for this type of installing it in %PATH%.
Python3.9+: Added support for importlib.resources.files with compiled modules.

It returns traversable objects, which can be used to opens files, checks them, etc. and this e.g. allows jsonschema to work with Python 3.10, despite bugs in CPython’s compatibility layer.
UI: Added interface method to specify filename patterns with package data inclusion option, making --include-package-data usable in many more cases, picking the only files or file types you want. You can now use --include-package-data=package_name=*.txt and select only a subset of package data files in this way. Before this, it included everything and --noinclude-data-files would have to be used.
macOS: Make runtime signing an experimental option.
Consistently allow when conditions for all package configuration elements, e.g. also DLLs.
Plugins: Added method to overload to work on standalone binary specifically. This makes it easier to only modify that specific binary.
Plugins: Added support for regular expressions in anti-bloat replacements, with new replacements_re code.

Optimization

Add support for os.path hard module imports along with specialized nodes for file tests os.path.exists, os.path.isfile, and os.path.isdir aiming at tracking used files, producing warnings about missing files in the future.
Standalone: Do not include concurrent standard library package automatically. This avoids the inclusion of multiprocessing which we essentially had reverted during last release cycle.
Standalone: Do not include zoneinfo standard library package automatically. It has many data files and is often not used (yet).
Standalone: Do not include asyncio standard library package automatically anymore.
Avoid compilation of large generated codes in the asyncua package.
Compile time optimize pkg_resources.iter_entry_points too, such that these can be used to resolve plugin modules, which helps with adding support for agscheduler package plugins. Note that these still need to be manually included with --include-module but now that works.
For known truth values of the right hand side of and or or conditions, reduce the expression as far as possible.
Added dedicated assignment node for hard imports, which then are propagated in classes as well, allowing for more static optimization for code on the class level.
Added linker options to make static --static-libpython work with clang on Linux as well.
macOS: Make sure libpython is loaded relative to the executable. This is needed for at least Anaconda Python.
macOS: Fix, need to search environment specific DLL paths and only then global paths, otherwise mixed Python versions will not work correctly.
Anti-Bloat: Remove IPython usage in rich package.
Anti-Bloat: Avoid doctest dependency when using pyrect.
Anti-Bloat: Some unittest removals from pytorch using libraries.
Keep the Scons report items sorted, or else it varies for the hashing of dependencies with Python versions before 3.6, causing cache misses without need.

Organizational

UI: Output the .cmd file created (if any) on Windows, e.g. when run in a virtualenv or for uninstalled Python versions, it will otherwise not run in accelerated mode, but previously the output suggested to run the executable directly.
UI: Enhanced command line option description of --include-plugin-directory which is frequently misunderstood. That option barely does what people want it to do. Point them to using the other options that are easy to use and will work.
UI: Specified needed Python version for use in --python-for-scons so users can know ahead of time what versions are suitable.
Reports: Added information about data files including, optimization times per module, active plugins.
Debugging: Repaired offline DLL dependency listing tool, such that it can be used during Windows DLL analysis.
Make --xml accept a filename for the node tree dump, and change it so it can be executed in addition to actual compilation. This way we need not be super-robust about keeping stdout clean, to not break XML parsing.
Plugins: Avoid useless warning about PySide2 plugin usage if another Qt plugin is actually selected.
UI: Catch error of directories being used as data files where plain files are expected and point out that other options must be used.
User Manual: Added section about accessing files in standalone mode too, so people can make sure it works properly.
Onefile: Using %TEMP% folder should not by itself prevent cached onefile mode, only really variable paths should. People may want to have this as some kind of temporary cache still.
UI: Catch user error of using elements, that resolve to absolute values in the middle of path specs, so using e.g. something/%PROGRAM% is now a mistake caught at compile time. These values can only be at the start of spec values naturally.
Quality: Updated to newer version of rstfmt.
UI: Nicer error message when a forbidden import is requested as an implicit import by a plugin.
Python3.11: Adapted to allocator and exception state changes, but more will be needed to compile at all.
Visual Code: Find clang-format from the recommended C++ extension of Visual Code, which makes it finally available on macOS easily too.
UI: Quote command line argument values as necessary when stating them in the logging. Otherwise they are not directly usable on the shell and also less readable.
Debian: Do not list fake modules as used debian packages codes, which could e.g. happen with the pre-load code of pkg_resources if that is from a Debian package. Fake packages should not be mentioned for these lists though.
Nuitka-Python: Added support to set link time flags coming from statically included packages.
For our isort trick of splitting files in two parts (mostly to setup import paths for nuitka package), make sure the second parts starts with a new line.
Added more usable form --output-filename to specify the output filename, the short form has become barely usable after we switched to enforcing no space separation for command line arguments.
UI: Check if output filename’s directory exists ahead of time, and error exit if not, otherwise compilation crashed only in the very end, trying to create the final result.
UI: When exiting with no error code, do not use red color or FATAL error annotation, that is not justified.
Quality: Make sure the Yaml auto-format does not change effective contents.
Quality: Added ability to limit autoformat by file type, which can be handy when e.g. only the yaml files should be scanned.
UI: Removed --windows-onefile-tempdir-spec alias of --onefile-tempdir-spec option, it is no longer Windows specific.

Cleanups

Prefer single quotes rather than double quotes in our package configuration Yaml files, otherwise esp. regular expressions with escapes become very confusing.
Move import hacks to general mechanism in Yaml package configuration files. This is for extra paths from package names or from directory paths relative to the package. This removes special purpose code from core code paths and allows their reuse.
Again more spelling cleanups have been done, to make the code cleaner to read and search.
Unified how plugins treat iteration over their value list, and how the when condition is applied for the various kinds of sections.
Output compilation command that failed during coverage taking, which makes it unnecessary to attempt to reconstruct what happened from test modes.

Tests

Added coverage for comparisons that need argument swaps.
Allow more time in onefile keyboard signal test, otherwise it can be a race on slow machines, e.g. emulated machines.
Tests: Added support for running a local web server.

Summary

This release is mainly a consolidation of previous release. Optimization added in previous release did in fact introduce regressions, that needed to be addressed and were cause for relatively many hotfixes.

The Yaml Nuitka Package Configuration feature is getting ever more powerful, but is not one bit more documented, such that the community as a whole is not yet capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.

New optimization was focused around compatibility with very few exceptions, where the non-automatic standard library work is standing out, and allows for smaller binaries in many cases.

Scalability has seen improvements through a few optimization, but mainly again with anti-bloat work being done. This is owed to the fact that consolidation was the name of the game.

For Anaconda specifically, a lot more software is covered, and generally, cv2 and torch related tools are now working better, but it seems DLL handling will remain problematic in many instances.

The compilation report contains much more information and is getting there is terms of completeness. At some point, we should ask for it in bug reports.

Nuitka Release 1.0

2022-08-16T13:45:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new features, while consolidating what we have with many bug fixes. Scalability should be dramatically better, as well as new optimization that will accelerate some code quite a bit. See the summary, how this release is paving the way forward.

Bug Fixes

Python3: Fix, bytes.decode with only errors argument given was not working. Fixed in 0.9.1 already.
MSYS2: Fix, the accelerate mode .cmd file was not working correctly. Fixed in 0.9.1 already.
Onefile: Fix, the bootstrap when waiting for the child, didn’t protect against signals that interrupt this call. This only affected users of the non-public --onefile-tempdir option on Linux, but with that becoming the default in 1.0, this was discovered. Fixed in 0.9.1 already.
Fix, pkg_resources compile time generated Distribution values could cause issues with code that put it into calls, or in tried blocks. Fixed in 0.9.1 already.
Standalone: Added implicit dependencies of Xlib package. Fixed in 0.9.1 already.
macOS: Fix, the package configuration for wx had become invalid when restructuring the Yaml with code and schema disagreeing on allowed values. Fixed in 0.9.1 already.
Fix: The str.format with a single positional argument didn’t generate proper code and failed to compile on the C level. Fixed in 0.9.1 already.
Fix, the type shape of str.count result was wrong. Fixed in 0.9.1 already.
UI: Fix, the warning about collision of just compiled package and original package in the same folder hiding the compiled package should not apply to packages without an __init__.py file, as those do not take precedence. Fixed in 0.9.2 already.
Debugging: Fix, the fallback to lldb from gdb when using the option --debugger was broken on anything but Windows. Fixed in 0.9.2 already.
Python3.8: The module importlib.metadata was not recognized before 3.9, but actually 3.8 already has it, causing the compile time resolution of package versions to not work there. Fixed in 0.9.3 already.
Standalone: Fix, at least on macOS we should also scan from parent folders of DLLs, since they may contain sub-directories in their names. This is mostly the case, when using frameworks. Fixed in 0.9.2 already.
Standalone: Added package configuration for PyQt5 to require onefile bundle mode on macOS, and recommend to disable console for PyQt6. This is same as we already do for PySide2 and PySide6. Fixed in 0.9.2 already.
Standalone: Removed stray macOS onefile bundle package configuration for pickle module which must have been added in error. Fixed in 0.9.2 already.
UI: Catch user error of attempting to compile the __init__.py rather than the package directory. Fixed in 0.9.2 already.
Fix, hard name import nodes failed to clone, causing issues in optimization phase. Fixed in 0.9.2 already.
Fix, avoid warnings given with gcc 11. Fixed in 0.9.2 already.
Fix, dictionary nodes where the operation itself has no effect, e.g. dict.copy were not properly annotating that their dictionary argument could still cause a raise and have side effects, triggering an assertion violation in Nuitka. Fixed in 0.9.2 already.
Standalone: Added pynput implicit dependencies on Linux. Fixed in 0.9.2 already.
Fix, boolean condition checks on variables converted immutable constant value assignments to boolean values, leading to incorrect code execution. Fixed in 0.9.2 already.
Python3.9: Fix, could crash on generic aliases with non-hashable values. Fixed in 0.9.3 already.
```
dict[str:any]
```
Python3: Fix, an iteration over sys.version_info was falsely optimized into a tuple, which is not always compatible. Fixed in 0.9.3 already.
Standalone: Added support for xgboost package. Fixed in 0.9.3 already.
Standalone: Added data file for text_unidecode package. Fixed in 0.9.4 already.
Standalone: Added data files for swagger_ui_bundle package. Fixed in 0.9.4 already.
Standalone: Added data files for connexion package. Fixed in 0.9.4 already.
Standalone: Added implicit dependencies for sklearn.utils and rapidfuzz. Fixed in 0.9.4 already.
Python3.10: Fix, the reformulation of match statements could create nodes that are used twice, causing code generation to assert. Fixed in 0.9.4 already.
Fix, module objects removed from sys.modules but still used could lack a reference to themselves, and therefore crash due to working on a released module variables dictionary. Fixed in 0.9.5 already.
Fix, the MSVC compiles code generated for SciPy 1.8 wrongly. Added a workaround for that code to avoid triggering it. Fixed in 0.9.6 already.
Fix, calls to str.format where the result is not used, could crash the compiler during code generation. Fixed in 0.9.6 already.
Standalone: For DLLs on macOS and Anaconda, also consider the lib directory of the root environment, as some DLLs are otherwise not found.
Fix, allow nonlocal and global for __class__ to be used on the class level.
Fix, xrange with large values didn’t work on all platforms. This affected at least Python2 on macOS, but potentially others as well.
Windows: When scanning for installed Pythons to e.g. run Scons or onefile compression, it was attempting to use installations that got deleted manually and could crash.
macOS: Fix, DLL conflicts are now resolved by checking the version information too, also all cases that previously errored out after a conflict was reported, will now work.
Fix, conditional expressions whose statically decided condition picking a branch will raise an exception could crash the compilation.
```
# Would previously crash Nuitka during optimization.
return 1/0 if os.name == "nt" else 1/0
```
Windows: Make sure we set C level standard file handles too

At least newer subprocess was affected by this, being unable to provide working handles to child processes that pass their current handles through, and also this should help DLL code to use it as level.
Standalone: Added support for pyqtgraph data files.
Standalone: Added support for dipy by anti-bloat removal of its testing framework that wants to do unsupported stuff.
UI: Could still give warnings about modules not being followed, where that was not true.
Fix, --include-module was not working for non-automatic standard library paths.

New Features

Onefile: Recognize a non-changing path from --onefile-tempdir-spec and then use cached mode. By default a temporary folder is used in the spec value, make it delete the files afterwards.

The cached mode is not necessarily faster, but it is not going to change files already there, leaving the binaries there intact. In the future it may also become faster to execute, but right now checking the validity of the file takes about as long as re-creating it, therefore no gain yet. The main point, is to not change where it runs from.
Standalone: Added option to exclude DLLs. You can npw use --noinclude-dlls to exclude DLLs by filename patterns.

The may e.g. come from Qt plugins, where you know, or experimented, that it is not going to be used in your specific application. Use with care, removing DLLs will lead to very hard to recognize errors.
Anaconda: Use CondaCC from environment variables for Linux and macOS, in case it is installed. This can be done with e.g. conda install gcc_linux-64 on Linux or conda install clang_osx-64 on macOS.
Added new option --nowarn-mnemonic to disable warnings that use mnemonics, there is currently not that many yet, but it’s going to expand. You can use this to acknowledge the ones you accept, and not get that warning with the information pointer anymore.
Added method for resolving DLL conflicts on macOS too. This is using version information and picks the newer one where possible.
Added option --user-package-configuration-file for user provided Yaml files, which can be used to provide package configuration to Nuitka, to e.g. add DLLs, data files, do some anti-bloat work, or add missing dependencies locally. The documentation for this does not yet exist though, but Nuitka contains a Yaml schema in the misc/nuitka-package-config-schema.json file.
Added nuitka-project-else to avoid repeating conditions in Nuitka project configuration, this can e.g. be used like this:
```
# nuitka-project-if: os.getenv("TEST_VARIANT", "pyside2") == "pyside2":
#   nuitka-project: --enable-plugin=no-qt
# nuitka-project-else:
#   nuitka-project: --enable-plugin=no-qt
#   nuitka-project: --noinclude-data-file=*.svg
```
Previously, the inverted condition had to be used in another nuitka-project-if which is no big deal, but less readable.

Added support for deep copying uncompiled functions. There is now a section in the User Manual that explains how to clone compiled functions. This allows a workaround like this:

def binder(func, name):
   try:
      result = func.clone()
   except AttributeError:
      result = types.FunctionType(func.__code__, func.__globals__, name=func.__name__, argdefs=func.__defaults__, closure=func.__closure__)
      result = functools.update_wrapper(result, func)
      result.__kwdefaults__ = func.__kwdefaults__

   result.__name__ = name
   return result

Plugins: Added explicit deprecation status of a plugin. We now have a few that do nothing, and are just there for compatibility with existing users, and this now informs the user properly rather than just saying it is not relevant.
Fix, some Python installations crash when attempting to import modules, such as os with a ModuleName object, because we limit string operations done, and e.g. refuse to do .startswith which of course, other loaders that your installation has added, might still use.
Windows: In case of not found DLLs, we can still examine the run time of the currently compiling Python process of Nuitka, and locate them that way, which helps for some Python configurations to support standalone, esp. to find CPython DLL in unusual spots.
Debian: Workaround for lib2to3 data files. These are from stdlib and therefore the patched code from Debian needs to be undone, to make these portable again.

Optimization

Scalability: Avoid merge traces of initial variable versions, which came into play when merging a variable used in only one branch. These are useless and only made other optimization slower or impossible.
Scalability: Also avoid merge traces of merge traces, instead flatten merge traces and avoid the duplication doing so. There were pathological cases, where this reduced optimization time for functions from infinite to instant.
For comparison helpers, switch comparison where possible, such that there are only 3 variants, rather than 6. Instead the boolean result is inverted, e.g. changing >= into not < effectively. Of course this can only be done for types, where we know that nothing special, i.e. no method overloads of __gte__ is going on.
For binary operations that are commutative with the selected types, in mixed type cases, swap the arguments during code generation, such that e.g. long_a + float_b is actually computed as float_b + long_a. This again avoids many helpers. It also can be done for * with integers and container types.
In cases, where a comparison (or one of the few binary operation where we consider it useful), is used in a boolean context, but we know it is impossible to raise an exception, a C boolean result type is used rather than a nuitka_bool which is now only used when necessary, because it can indicate the exception result.
Anti-Bloat: More anti-bloat work was done for popular packages, covering also uses of setuptools_scm, nose and nose2 package removals and warnings. There was also a focus on making mmvc, tensorflow and tifffile compile well, removing e.g. the uses of the tensorflow testing framework.
Faster comparison of int values with constant values, this uses helpers that work with C long values that represent a single “digit” of a value, or ones that use the full value space of C long.
Faster comparison of float values with constant values, this uses helpers that work with C float values, avoiding the useless Python level constant objects.
Python2: Comparison of int and long now has specialized helpers that avoids converting the int to a long through coercion. This takes advantage of code to compare C long values (which are at the core of Python2 int objects, with long objects.
For binary operation on mixed types, e.g. int * bytes the slot of the first function was still considered, and called to give a Py_NotImplemented return value for no good reason. This also applies to mixed operations of int, long, and float types, and for str and unicode values on Python2.
Added missing helper for ** operation with floats, this had been overlooked so far.
Added dedicated nodes for ctypes.CDLL which aims to allow us to detect used DLLs at compile time in the future, and to move closer to support its bindings more efficiently.
Added specialized nodes for dict.popitem as well. With this, now all of the dictionary methods are specialized.
Added specialized nodes for str.expandtabs, str.translate, str.ljust, str.rjust, str.center, str.zfill, and str.splitlines. While these are barely performance relevant, this completes all str methods, except removeprefix and removesuffix that are Python3.9 or higher.
Added type shape for result of str.index operation as well, this was missing so far.
Optimize str, bytes and dict method calls through variables.
Optimize calls through variables containing e.g. mutable constant values, these will be rare, because they all become exceptions.
Optimize calls through variables containing built-in values, unlocking optimization of such calls, where it is assigned to a local variable.
For generated attribute nodes, avoid local doing import statements on the function level. While these were easier to generate, they can only be slow at run time.
For the str built-in annotate its value as derived from str, which unfortunately does not allow much optimization, since that can still change many things, but it was still a missing attribute.
For variable value release nodes, specialize them by value type as well, enhancing the scalability, because e.g. parameter variable specific tests, need not be considered for all other variable types as well.

Organizational

Plugins: Major changes to the Yaml file content, cleaning up some of the DLL configuration to more easy to use.

The DLL configuration has two flavors, one from code and one from filename matching, and these got separated into distinct items in the Yaml configuration. Also how source and dest paths get provided got simplified, with a relative path now being used consistently and with sane defaults, deriving the destination path from where the module lives. Also what we called patterns, are actually prefixes, as there is still the platform specific DLL file naming appended.
Plugins: Move mode checks to dedicated plugin called options-nanny that is always enabled, giving also much cleaner Yaml configuration with a new section added specifically for these. It controls advice on the optional or required use of --disable-console and the like. Some packages, e.g. wx are known to crash on macOS when the console is enabled, so this advice is now done with saner configuration.
Plugins: Also for all Yaml configuration sub-items where is now a consistent when field, that allows checking Python version, OS, Nuitka modes such as standalone, and only apply configuration when matching this criterion, with that the anti-bloat options to allow certain bloat, should now have proper effect as well.
The use of AppImage on Linux is no more. The performance for startup was always slower, while having lost the main benefit of avoiding IO at startup, due to new cached mode, so now we always use the same bootstrap binary as on macOS and Windows.
UI: Do not display implicit reports reported by plugins by default anymore. These have become far too many, esp. with the recent stdlib work, and often do not add any value. The compilation report will become where to turn to find out why a module in included.
UI: Ask the user to install the ordered set package that will actually work for the specific Python version, rather than making him try one of two, where sometimes only one can work, esp. with Python 3.10 allowing only one.
GitHub: More clear wording in the issue template that python -m nuitka --version output is really required for support to given.
Attempt to use Anaconda ccache binary if installed on non-Windows. This is esp. handy on macOS, where it is harder to get it.
Windows: Avoid byte-compiling the inline copy of Scons that uses Python3 when installing for Python2.
Added experimental switches to disable certain optimization in order to try out their impact, e.g. on corruption bugs.
Reports: Added included DLLs for standalone mode to compilation report.
Reports: Added control tags influencing plugin decisions to the compilation report.
Plugins: Make the implicit-imports dependency section in the Yaml package configuration a list, for consistency with other blocks.
Plugins: Added checking of tags such from the package configuration, so that for things dependent on python version (e.g. python39_or_higher, before_python39), the usage of Anaconda (anaconda) or certain OS (e.g. macos), or modes (e.g. standalone), expressions in when can limit a configuration item.
Quality: Re-enabled string normalization from black, the issues with changes that are breaking to Python2 have been worked around.
User Manual: Describe using a minimal virtualenv as a possible help low memory situations as well.
Quality: The yaml auto-format now properly preserves comments, being based on ruamel.yaml.
Nuitka-Python: Added support for the Linux build with Nuitka-Python for our own CPython fork as well, previously only Windows was working, amd macOS will follow later.
The commit hook when installed from git bash was working, but doing so from cmd.exe didn’t find a proper path for shell from the git location.
Debugging: A lot of experimental toggles were added, that allow control over the use of certain optimization, e.g. use of dict, list, iterators, subscripts, etc. internals, to aid in debugging in situations where it’s not clear, if these are causing the issue or not.
Added support for Fedora 36, which requires some specific linker options, also recognize Fedora based distributions as such.
Removed long deprecated option --noinclude-matplotlib from numpy plugin, as it hasn’t had an effect for a long time now.
Visual Code: Added extension for editing Jinja2 templates. This one even detects that we are editing C or Python and properly highlights accordingly.

Cleanups

Standalone: Major cleanup of the dependency analysis for standalone. There is no longer a distinction between entry points (main binary, extension modules) and DLLs that they depend on. The OS specific parts got broken out into dedicated modules as well and decisions are now taken immediately.
Plugins: Split the Yaml package configuration files into 3 files. One contains now Python2 only stdlib configuration, and another one general stdlib.
Plugins: Also cleanup the zmq plugin, which was one the last holdouts of now removed plugin method, moving parts to the Yaml configuration. We therefore no longer have considerExtraDlls which used to work on the standalone folder, but instead only plugin code that provides included DLL or binary objects from getExtraDlls which gives Nuitka much needed control over DLL copying. This was a long lasting battle finally won, and will allow many new features to come.
UI: Avoid changing whitespace in warnings, where we have intended line breaks, e.g. in case of duplicate DLLs. Went over all warnings and made sure to either avoid new-lines or have them, depending on wanted output.
Iterator end check code now uses the same code as rich comparison expressions and can benefit from optimization being done there as well.
Solved TODO item about code generation time C types to specify if they have error checking or not, rather than hard coding it.
Production of binary helper function set was cleaned up massively, but still needs more work, comparison helper function set was also redesigned.
Changing the spelling of our container package to become more clear.
Used namedtuple objects for storing used DLL information for more clear code.
Added spellchecker ignores for all attribute and argument names of generated fixed attribute nodes.
In auto-format make sure the imports float to the top. That very much cleans up generated attribute nodes code, allowing also to combine the many ones it makes, but also cleans up some of our existing code.
The package configuration Yaml files are now sorted according to module names. This will help to avoid merge conflicts during hotfixes merge back to develop and automatically group related entries in a sane way.
Moved large amounts of code producing implicit imports to Yaml configuration files.
Changed the tensorflow plugin to Yaml based configuration, making it a deprecated do nothing plugin, that only remains there for a few releases, to not crash existing build scripts.
Lots of spelling cleanups, e.g. renaming nuitka.codegen to nuitka.code_generation for clarity.

Tests

Added generated test to cover bytes method. This would have found the issue with decode potentially.
Enhanced standalone test for ctypes on Linux to actually have something to test.

Summary

This release improves on many things at once. A lot of work has been put into polishing the Yaml configuration that now only lacks documentation and examples, such that the community as a whole should become capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.

Then a lot of new optimization has been done, to close the missing gaps with dict and str methods, but before completing list which is already a work in progress pull request, and bytes, we want to start and generate the node classes that form the link or basis of dedicated nodes. This will be an area to work on more.

The many improvements to existing code helpers, and them being able to pick target types for the arguments of comparisons and binary operations, is a pre-cursor to universal optimization of this kind. What is currently only done for constant values, will in the future be interesting for picking specific C types for use. That will then be a huge difference from what we are doing now, where most things still have to use PyObject * based types.

Scalability has again seen very real improvements, memory usage of Nuitka itself, as well as compile time inside Nuitka are down by a lot for some cases, very noticeably. There is never enough of this, but it appears, in many cases now, large compilations run much faster.

For macOS specifically, the new DLL dependency analysis, is much more capable or resolving conflicts all by itself. Many of the more complex packages with some variants of Python, specifically Anaconda will now be working a lot better.

And then, of course there is the big improvement for Onefile, that allows to use cached paths. This will make it more usable in the general case, e.g. where the firewall of Windows hate binaries that change their path each time they run.

Future directions will aim to make the compilation report more concise, and given reasons and dependencies as they are known on the inside more clearly, such that is can be a major tool for testing, bug reporting and analysis of the compilation result.

Nuitka Release 0.9

2022-07-02T08:17:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a many optimization improvements, and scalability improvements, while also adding new features, with also some important bug fixes.

Bug Fixes

Fix, hard module name lookups leaked a reference to that object. Fixed in 0.8.1 already.
Python2: Fix, str.decode with errors as the only argument wasn’t working. Fixed in 0.8.1 already.
Fix, could corrupt created uncompiled class objects __init__ functions in case of descriptors being used.
Standalone: Added support for newer torch. Fixed in 0.8.1 already.
Standalone: Added support for newer torchvision. Fixed in 0.8.1 already.
Fix, could compile time crash during initial parsing phase on constant dictionary literals with non-hashable keys.
```
{ {}:1, }
```
Fix, hard imported sub-modules of hard imports were falsely resolved to their parent. Fixed in 0.8.3 already.
```
importlib.resources.read_bytes() # gave importlib has no attribute...
```
Windows: Fix, outputs with --force-stdout-spec or --force-stderr-spec were created with the file system encoding on Python3, but they nee to be utf-8.
Fix, didn’t allow zero spaces in Nuitka project options, which is not expected.
Modules: Fix, the del __file__ in the top level module in module mode caused crashes at run time, when trying to restore the original __file__ value, after the loading CPython corrupted it.
Python2.6: Fixes for installations without pkg_resources.
Windows: Fix for very old Python 2.6 versions, these didn’t have a language assigned that could be used.
Security: Fix for CVE-2022-2054 where environment variables used for transfer of information between Nuitka restarting itself, could be used to execute arbitrary code at compile time.
Anaconda: Fix, the torch plugin was not working on Linux due to missing DLL dependencies.
Fix, static optimization of importlib.import_module with a package given, for an absolute import was optimized into the wrong import, package was not ignored as it should be.
Windows: Installed Python scan could crash on trying installation paths from registry that were manually removed in the mean time, but not through an uninstaller.
Standalone: Added missing implicit dependency for pyreadstat because parts of standard library it uses are no more automatically included.
Windows: Could still crash when no powershell is available with symlinks, handle this more gracefully.
Standalone: Added more missing Plotly dependencies, but more work will be needed to complete this.
Standalone: Add missing stdlib dependency on multiprocessing by concurrent.futures.process.
Standalone: Fix, implicit dependencies assigned to imageio on PIL plugins should actually be assigned to PIL.Image that actually loads them, so it works outside of imageio too.

New Features

UI: Added new option --user-package-configuration-file to allow users to provide extra Yaml configuration files for the Nuitka plugin mechanism to add hidden dependencies, anti-bloat, or data files, for packages. This will be useful for developing PRs to the standard file of Nuitka. Currently the schema is available, but it is not documented very well yet, so not really ready for end users just yet.
Standalone: Added new no-qt plugin as an easy way to prevent all of the Qt bindings from being included in a compilation.
Include module search path in compilation report.

Optimization

Faster dictionary iteration with our own replacement for PyDict_Next that avoids the DLL call overhead (in case of non-static libpython) and does less unnecessary checks.
Added optimization for str.count and str.format methods as well, this should help in some cases with compile time optimization.
The node for dict.update with only an iterable argument, but no keyword arguments, was in fact unused due to wrongly generated code. Also the form with no arguments wasn’t yet handled properly.
Scalability: Use specialized nodes for pair values, i.e. the representation of x = y in e.g. dictionary creations. With constant keys, and values, these avoid full constant value nodes, and therefore save memory and compile time for a lot of code.
Anti-Bloat: Added more scalability work to avoid including modules that make compilation unnecessarily big.
Python3.9+: Faster calls in case of mixed code, i.e. compiled code calling uncompiled code.
Removing duplicates and non-existent entries from modules search path should improve performance when locating modules.
Optimize calls through variables as well. These are needed for the package resource nodes to properly resolve at compile time from their hard imports to the called function.
Hard imported names should also be considered very trusted themselves, so they are e.g. also optimized in calls.
Anti-Bloat: Avoid more useless imports in Pandas, Numba, Plotly, and other packages, improving the scalability some more.
Added dedicated nodes for pkg_resources.require, pkg_resources.get_distribution, importlib.metadata.version, and importlib_metadata.version, so we can use compile time optimization to resolve their argument values where possible.
Avoid annotating control flow escape for all release statements. Sometimes we can tell that __del__ will not execute outside code ever, so this then avoids marking values as escaped, and taking the time to do so.
Calls of methods through variables on str, dict, bytes that have dedicated nodes are now also optimized through variables.
Boolean tests through variables now also are optimized when the original assignment is a compile time constant that is not mutable. This is only basic, but will allow tests on TYPE_CHECKING coming from a from typing import TYPE_CHECKING statement to be optimized, avoiding this overhead.

Cleanups

Changed to torch plugin to Yaml based configuration, making it obsolete, it only remains there for a few releases, to not crash existing build scripts.
Moved older package specific hacks to the Yaml file. Some of these were from hotfixes where the Yaml file wasn’t yet used by default, but now there is no need for them anymore.
Removed most of the pkg-resources plugin work. This is now done during optimization phase and rather than being based on source code matches, it uses actual value tracing, so it immediately covers many more cases.
Continued spelling improvements, renaming identifiers used in the source that the cspell based extension doesn’t like. This aims at producing more readable and searchable code.
Generated attribute nodes no longer do local imports of the operation nodes they refer to. This also avoids compile time penalties during optimization that are not necessary.
Windows: Avoid useless bytecode of inline copy used by Python3 when installing for Python2, this spams just a lot of errors.

Organizational

Removed MSI installers from the download page. The MSI installers are discontinued as Python has deprecated their support for them, as well as Windows 10 is making it harder for users to install them. Using the PyPI installation is recommended on Windows.
Merged our Yaml files into one and added schema description, for completion and checking in Visual Code while editing. Also check the schema in check-nuitka-with-yamllint which is now slightly misnamed. The schema is in no way final and will see improvements in future releases.
UI: Nicer progress bar layout that avoids flicker when optimizing modules.
UI: When linking, output the total number of object files used, to have that knowledge after the progress bar for C compilation is gone.
Quality: Auto-format the package configuration Yaml file for anti-bloat, implicit dependencies, etc.
GitHub: Point out the commit hook in the PR template.
UI: Nicer output in case of no commercial version is used.
Updated the WinLibs MinGW64 download used on Windows to the latest version based on gcc 11, the gcc 12 is not yet ready.
Git: Make sure we are not affected by core.autocrlf setting, as it interferes with auto-format enforcing Unix newlines.
Removed the MSI downloads. Windows 10 has made them harder to install and Python itself is discontinuing support for them, while often it was only used by beginners, for which it was not intended.
Anaconda: Make it more clear how to install static libpython with precise command.
UI: Warn about using module from Debian packages. These can be made non-portable to other OSes. Read more on the info page for detailed information.
Quality: The auto-format now floats imports to the top for consistency. With few exceptions, it was already done like this. But it makes things easier for generated code.

Tests

The reflected test was adapted to preserve PYTHONPATH now that module presence influences optimization.

Summary

This release marks a point, that will allow us to open up the compatibility work for implicit dependencies and anti-bloat stuff even further. The Yaml format will need documentation and potentially more refinement, but will open up a future, where latest packages can be supported with just updating this configuration.

The scalability improvements really make a difference for many libraries and are a welcome improvement on both memory usage and compile time. They are achieved by an accord of static optimization of

One optimization aimed at optimizing tuple unpacking, was not finished in time for this release, but will be subject of a future release. It has driven many other improvements though.

Generally, also from the UI, this is a huge step forward. With links to the website for complex topics being started, and the progress bar flicker being removed, the tool has yet again become more user friendly.

Next Nuitka Live Stream

2022-05-27T00:00:00+01:00

In the coming weekend, Saturday 04.06.2022, there will be the next live streams of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 10-14 CEST and potentially also one in the evening.

Last time

It’s actually been too long for me to remember what I did, but I do know that it took some time to get it right. It probably was about caching, but there are usually several topics.

I do however think that I need to use my CI machine with more processing power rather than my normal desktop, or maybe even both, because the screen capture, compiling, Visual Studio, and all, may make the Video otherwise stutter a bit too much.

This time

So, streaming is a huge success or Nuitka, however time is also scarce, and me starting new things, risks the release being bogged down by stabilization phases that are extended.

Not sure yet, what to do. I think, I want to look at adding the int as a C type and/or specializing code for mix of CLONG (C long value) and LONG (Python3 int object) and INT (Python2 int object). Right now adding 1 goes through a constant object, one that is prepared, and not time consuming to reference, but still overhead to use.

How to Join Nuitka Live

Join me

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

Nuitka Release 0.8

2022-05-26T10:54:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a massive amount of bug fixes, builds on existing features, and adds new ones.

Bug Fixes

Windows: Fix, need to ignore cases where shorting in path for external use during compilation gives an permission error. Fixed in 0.7.1 already.
Compatibility: Added workaround for scipy.stats function copying. Fixed in 0.7.1 already.
Windows: Fix, detect ARM64 arch of MSVC properly, such that we can give a proper mismatch for the Python architecture. Fixed in 0.7.1 already.
Standalone: Fix, using importlib.metadata module failed to include email from standard library parts no longer included by default. Fixed in 0.7.1 already.
macOS: Fix, the dependency parser was using normalized paths where original paths must be used. Fixed in 0.7.1 already.
Standalone: Fix, using shiboken6 module (mostly due to PySide6) failed to include argparse from the standard library from standard library parts no longer included by default. Fixed in 0.7.1 already.
Onefile: Fix, the detection of a usable Python for compression could crash. Fixed in 0.7.2 already.
Onefile: Adding the payload on Windows could run into locks still being held, need to wait in that case. Fixed in 0.7.2 already.
Fix, need to include pkg_resources as well, we need it for when we use Jinja2, which is more often now. For Python3 this was fixed in 0.7.3 already. Later a version to use with Python2 was added as well.
Release: The wheels built for Nuitka when installed through URLs were not version specific, but due to different inline copies per OS and Python version, they must not be reused. Therefore we now pretend to contain an extension module, which handles that. Fixed in 0.7.3 already.
Standalone: Fix, using urllib.requests module failed to include http.client from standard library parts no longer included by default. Fixed in 0.7.3 already. Later http.cookiejar was added too.
Standalone: Do not compress MSVC run time library when using upx plugin, that is not going to work. Fixed in 0.7.4 already.
Standalone: Fix, on Windows more files should be included for TkInter to work with all software. Fixed in 0.7.5 already.
Distutils: Added support for package_dir directive to specify where source code lives. Fixed in 0.7.6 already.
Standalone: Fix, using shelve module failed to include dbm from standard library parts no longer included by default. Fixed in 0.7.6 already.
Standalone: Added support for arcade data files. Fixed in 0.7.7 already.
Standalone: Fix, bytecode demotions should use relative filenames rather than original ones. Fixed in 0.7.7 already.
Standalone: Fix, must remove extension module objects from sys.modules before executing an extension module that will create it. This fixes cases of cyclic dependencies from modules loaded by the extension module.
Windows: In case of an exception, clcache was itself triggering one during its handling, hiding the real exception behind a TypeError.
Windows: Improved clcache locking behavior, avoiding a race. Also increase characters used for key from 2 to 3 chars, making collisions far more rare.
Standalone: Added support for persistent package.
Standalone: Added support for newer tensorflow package.
Module: Fix, need to restore the __file__ and __spec__ values of top level module. It is changed by CPython after import to an incompatible file name, and not our loader, preventing package resources to be found.
Standalone: Added support for Crpytodome.Cipher.PKCS1_v1_5.
Fix, pkgutil.iter_modules without arguments was not working properly with our meta path based loader.
Windows: Fix, could crash on when the Scons report was written due to directories in PATH that failed to encode.
Compatibility: Fix, positive divmod and modulo % with negative remainders of positive floats was not correct.
Fix, str.encode with only errors value, but default value for encoding was crashing the compilation.
Python3.10+: Fix, match statement sliced values must be lists, not tuples.
Standalone: Added support for newer glfw and OpenGL packages.
Python3: Fix, failed to read bytecode only stdlib files. This affect mostly Fedora Python which does this for encodings.
Python3.5+: Fix, two phase loading of modules could release it immediately.
Standalone: Added missing dependency for pydantic.
Fix, the str.split rejected default sep value with only maxsplit given as a keyword argument.
Standalone: Added missing dependency of wsgiref module.
Standalone: Added support for falcon module.
Standalone: Added support for eliot module.
Fix, need to mark assigned from variables as escaped. Without it, some aliased loop variables could be misunderstood and falsely statically optimized.
Standalone: Added support for newer uvicorn package.
Standalone: Added data files for the accessible_output2, babel, frozendict, and sound_lib package.
Standalone: Added support for newer sklearn package.
Standalone: Added support for tkinterdnd2 package.
Python3.7+: Fix, the error message wasn’t fully compatible for unsubscriptable type exception messages.
Standalone: Fix, idlelib from stdlib was always ignored.
Python3.4+: Fix, the __spec__.origin as produced by find_spec of our meta path based loader, didn’t have the correct origin attribute value.
Standalone: Disable QtPDF plugin of PySide 6.3.0, because it’s failing dependency checks. On macOS this was blocking, we will change it to detection if that is necessary in a future release.
Standalone: Added support for orderedmultidict.
Standalone: Added support for clr module.
Standalone: Added support for newer cv2 package.

New Features

Added new option --module-name-choice to select what value __name__ and __package__ are going to be. With --module-name-choice=runtime (default for --module mode), the created module uses the parent package to deduce the value of __package__, to be fully compatible. The value --module-name-choice=original (default for other modes) allows for more static optimization to happen.
Added support for get_resource_reader to our meta path based loader. This allows to avoid useless temporary files in case importlib.resources.path is used, due to a bad interaction with the fallback implementation used without it.
Added support for --force-stdout-spec and --force-stderr-spec on all platforms, this was previously limited to Windows.
Added support for requiring and suggesting modes. In part this was added to 0.7.3 already, and is used e.g. to enforce that on macOS the wx will only work as a GUI program and crash unless --disable-console is specified. These will warn the user or outright error the compilation if something is known to be needed or useful.
Debian: Detect version information for “Debian Sid”. Added in 0.7.4 already, and also improved how Debian/Ubuntu versions are output.
Added new option --noinclude-data-files to instruct Nuitka to not include data files matching patterns given. Also attached loggers and tags to included data file and include them in the compilation report.

Distutils: When using pyproject.toml without setup.py so far it was not possible to pass arguments. This is now possible by adding a section like this.

[tool.nuitka]
# options without an argument are passed as boolean value
show-scons = true

# options with single values, e.g. enable a plugin of Nuitka
enable-plugin = "pyside2"

# options with several values, e.g. avoiding including modules, accepts
# list argument.
nofollow-import-to = ["*.tests", "*.distutils"]

The option names are the same, but without leading dashes. Lists are only needed when passing multiple values with the same option.

macOS: Add support for specifying signing identity with --macos-sign-identity and access to protected resources --macos-app-protected-resource.
Included data files are now reported in the compilation report XML as well.
Accept absolute paths for searching paths of binaries. This allows e.g. the upx plugin to accept both a folder path and the full path including the binary name to work when you specify the binary location with --upx-binary making it more user friendly.
Python3.10: Added support for positional matching of classes, so far only keyword matching was working.
Added support for path spec values %CACHE_DIR, %COMPANY%, %PRODUCT%, %VERSION%, and %HOME in preparation of onefile once again being able to be cached and not unpacked repeatedly for each execution.
Standalone: Detect missing tk-inter plugin at run time. When TCL fails to load, it then outputs a more helpful error. This ought to be done for all plugins, where it’s not clear if they are needed.
Anti-Bloat: Added support for plain replacements in the anti-bloat.yml file. Before with replacement`, the new value had to be produced by an eval, which makes for less readable values due to extra quoting. for plain values.

Optimization

Python3.10+: Added support for ordered-set PyPI package to speed up compilation on these versions too, adding a warning if no accelerated form of OrderedSet is used, but believed to be usable.
Optimization: Added bytes.decode operations. This is only a start and we needed this for internal usage, more should follow later.
Much more anti-bloat work was added. Avoiding ipython, unittest, and sometimes even doctest usage for some more packages.
The ccache was not always used, sometimes it believed to catch a potential race, that we need to tell it to ignore. This will speed up re-compilation of the C side in many cases.
Do not compile the meta path based loader separate, which allows us to not expose functions and values only used by it. Also spares the C compiler one file.
Added various dedicated nodes for querying package resources data, e.g. pkgutil.get_data. This will make it easier to detect cases of missing data files in the future.
Added more hard imports, some of which help scalability in the compilation, because these are then known to exist in standalone mode, others are used for package resource specific operations.
Onefile: Releasing decompression buffers avoiding unnecessary high memory usage of bootstrap binary.
Standalone: Avoid proving directories with no DLLs (e.g. from packages) towards ldd, this should avoid exceeding command line limits.
For clcache remove writing of the stats file before Scons has completed, which avoids IO and locking churn.
Standalone: Avoid including wsgiref from stdlib by default.

Cleanups

Removed references to chrpath and dead code around it, it was still listed as a dependency, although we stopped using it a while ago.
Removed uses of anti-bloat in examples and tests, it is now enabled by default.
Made standard plugin file naming consistent, their name should be *Plugin.py.
Cleaned up tensorflow plugin. The source modification was moved to anti-bloat where it is easy to do. The implicit dependencies are now in the config file of implicit-imports plugin.
Massive cleanups of data file handling in plugins. Adding methods for producing the now required objects.
The Scons file handling got further cleaned up and unified, doing more things in common code.
Avoid #ifdefs usages with new helper function Nuitka_String_FromFormat that implies them for more readable code.
Split the allowance check from the encountering. Allow plugins and options all to say if an import should be followed, and only when that is decided, to complain about it. Previously the attempt was causing an error, even if another plugin were to decide against it later.
Python2.6: Avoid warnings from MSVC for out specialized long code. In testing it worked correctly, but this is more explicit and doesn’t rely on C implementation specific behavior, although it appears to be universal.

Organizational

UI: Warning tests are now wrapped to multiple lines if necessary. That makes it more accessible for larger messages that contain more guiding information.
Documented how to use local Nuitka checkout with pyproject.toml files, this makes debugging Nuitka straightforward in these setups.
Added instructions on how to pass extra C and linker flags and to the User Manual.
Made our auto-format usable for the Nuitka website code too.
Removed dependencies on chrpath and the now dead code that would use it, we are happy with patchelf.
Updated to latest versions of requirements for development, esp. black and pylint.
Renamed --macos-onefile-icon to --macos-app-icon because that is what it is really used for.
Unified how dependencies are installed in GitHub actions.
Updated man page contents for option name changes from last releases.
Updated the WinLibs MinGW64 download used on Windows to the latest version.
Updated the ccache binary used on Windows with MinGW64. This is in preparation of using it potentially for MSVC as well.
Updated Visual Code C config to use Python3.10 and MSVC 2022 include paths.

Tests

Better outputs from standalone library compilation test, esp. when finding a problem, present the script to reproduce it immediately.
Enhanced generated tests to cover str methods to use keyword arguments.
Added automatic execution of pyproject.toml driven test case.
Enhanced output in case of optimization test failures, dumping what value is there that has not become a compile time constant.

Summary

This release has seen a lot of consolidation. The plugins layer for data files is now all powerful, allowing much nicer handling of them by the plugins, they are better reported in normal output, and they are also part of the report file that Nuitka can create. You may now also inhibit their inclusion from the command line, if you decide otherwise.

The pyproject.toml now supporting Nuitka arguments is closing an important gap.

Generally many features got more polished, e.g. non-automatic inclusion of stdlib modules has most problems fixed up.

An important area of improvement, are the hard imports. These will be used to replace the source based resolution of package requirements with ones that are proper nodes in the tree. Getting these hard imports to still retain full compatibility with existing imports, that are more or less __import__ uses only, was revealing quite a bit of technical debt, that has been addressed with this release.

For onefile, the cached mode is being prepared with the variables added, but will only be in a later release.

Also a bunch of new or upgraded packages are working now, and the push for anti-bloat work has increased, making many compilations even more lean, but scalability is still an issue.

Nuitka Release 0.7

2022-02-21T13:10:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is massively improving macOS support, esp. for M1 and the latest OS releases, but it also has massive improvements for usability and bug fixes in all areas.

Bug Fixes

Fix, set creation wasn’t annotating its possible exception exit from hashing values and is not as free of side effects as list and tuple creations are. Fixed in 0.6.19.1 already.
Windows: Fix, --experimental option values got lost for the C compilation when switching from MSVC to MinGW64, making them have no effect. Fixed in 0.6.19.1 already.
Windows: Fix, Clang from MinGW64 doesn’t support LTO at this time, therefore default to no for it. Fixed in 0.6.19.1 already.
Debian: Fix, failed to detect Debian unstable as suitable for linking, it doesn’t have the release number. Fixed in 0.6.19.1 already.
Standalone: Added data files for pygsheets package. Fixed in 0.6.19.2 already.
Fix, paths from plugin related file paths need to be made absolute before used internally, otherwise the cache can fail to deduplicate them. Fixed in 0.6.19.2 already.
Python3: With gcc before version 5, e.g. on CentOS 7, where we switch to using g++ instead, the gcc version checks could crash. Fixed in 0.6.19.2 already.
Windows: Disable MinGW64 wildcard expansion for command line arguments. This was breaking command lines with arguments like --filename *.txt, which under cmd.exe are left alone by the shell, and are to be expanded by the program. Fixed in 0.6.19.2 already.
Standalone: Added missing implicit dependency needed for --follow-stdlib with Python for some uses of the locale module. Fixed in 0.6.19.2 already.
Standalone: Added workarounds for newest numpy that wants to set __code__ objects and required improvements for macOS library handling. Fixed in 0.6.19.3 already.
Windows: Caching of DLL dependencies for the main programs was not really working, requiring to detect them anew for every standalone compilation for no good reason. Fixed in 0.6.19.3 already.
Windows: Fix, CTRL-C from a terminal was not propagated to child processes on Windows. Fixed in 0.6.19.4 already.
Standalone: With certifi and Python3.10 the importlib.resource could trigger Virus scanner inflicted file access errors. Fixed in 0.6.19.4 already.
Python3.10: Reverted error back iteration past end of generator change for Python 3.10.2 or higher to become compatible with that too. Fixed in 0.6.19.5 already.
Standalone: Added support for anyio and by proxy for Solana. Fixed in 0.6.19.5 already.
Fix, compilation with resource mode incbin and --debugger was not working together. Fixed in 0.6.19.5 already.
Fix, format optimization of known str objects was not properly annotating an exception exit when being optimized away, causing consistency checks to complain. Fixed in 0.6.19.5 already.
Windows: Fix, clcache didn’t work for non-standard encoding source paths due to using th direct mode, where wrong filenames are output by MSVC. Fixed in 0.6.19.5 already.
Windows: Fix, ccache cannot handle source code paths for non-standard encoding source paths. Fixed in 0.6.19.5 already.
Python2.6: Fix, calls to iteritems and iterkeys on known dictionary values could give wrong values. Fixed in 0.6.19.5 already.
Fix, the value of __module__ if set by the metaclass was overwritten when creating types. Fixed in 0.6.19.6 already.
Plugins: Add support for the latest version of pkg_resources that has “vendored” even more packages. Fixed in 0.6.19.6 already.
Onefile: The onefile binary was locked during run time and could not be renamed, preventing in-place updates. This has been resolved and now on Windows, the standard trick for updating a running binary of renaming it, then placing the new file works.
Fix, wasn’t checking the zstandard version and as a result could crash if too old versions of it. This is now checked.
macOS: Large amounts of bug fixes for the dependency scanner. It got cleaned up and now handles many more cases correctly.
Windows: Fix, was not properly detecting wrong ClangCL architecture mismatch with the Python architecture. This could result in strange errors during C compilation in this setup.
Standalone: Added implicit dependencies for the asyncpg module.
Linux: Detect Debian or Ubuntu base and distribution name more reliably. This helps esp. with static libpython optimization being recognized automatically.

New Features

We now disallow options that take arguments to be provided without using =.
Previously --lto no worked just as well as --lto=no did. And that was the cause of problems when --lto first became a choice.

Recently similar, but worse problems were observed, where e.g. --include-module could swallow trailing other arguments when users forgot to specify the name by accident. Therefore this style of giving options is now explicitly rejected.
Compiled types of Nuitka now inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses PyObject_IsInstance work, e.g. pydantic.
macOS: Added signing of application bundles and standalone binaries for deployment to newer macOS platforms and esp. M1 where these are mandatory for execution.
macOS: Added support for selecting the single macOS target arch to create a binary for. The universal architecture is not yet supported though, but will be added in a future release.
Added support for compression in onefile mode through the use of an other Python installation, that has the zstandard module installed. With this it will work with 2.6 or higher, but require a 3.5 or higher Python with it installed in either PATH or on Windows in the registry alternatively.
Added UPX plugin to compress created extension modules and binaries and for standalone mode, the included DLLs. For onefile, the compression is not useful since it has the payload already compressed.
Added a more explicit way to list usable MSVC versions with --msvc=list rather than requiring an invalid value. Check values given in the same way that Scons will do.
Added support for --python-flag=-u which disabled outputs buffers, so that these outputs are written immediately.
Plugins: Always on plugins now can have command line options. We want this for the anti-bloat plugin that is enabled by default in this release.
Plugins: Added ability for plugin to provide fake dependencies for a module. We want the this for the multiprocessing plugin, that is now enabled by default in this release too.
Plugins: Added ability for plugins to modify DLLs after copy for standalone. We will be using this in the new upx plugin.
Added retry for file copies that fail due to still running program. This can happen on Windows with DLLs in standalone mode. For interactive compilation, this allows a retry to happen after prompting the user.
UI: Added ability to list MSVC versions with --msvc=list, and detect illegal values given to --msvc= before Scons sees them, it also crashes with a relative unhelpful error message.
UI: When linking, close the C compilation progress bar and state that that linking is going on. For some very large LTO compilations, it was otherwise at 100% and still taking a long time, confusing users.
Plugins: Added new plugin that is designed to handle DLL dependencies through a configuration file that can both handle filename patterns as well as code provided DLL locations.
Optimization: Exclude parts of the standard library by default. This allows for much smaller standalone distributions on modules, that can be expected to never be an implicit dependency of anything, e.g. argparse or pydoc.

Optimization

Standalone: Do not include encodings.bz2_codec and encodings.idna anymore, these are not file system encodings, but require extension modules.
Make sure we use proper (void) arguments for C functions without arguments, as for C functions, that makes a real difference, they are variable args functions and more expensive to call otherwise.
For standalone, default to using --python-flag=no_site to avoid the overhead that the typically unused site module incurs. It often includes large parts of the standard library, which we now want to be more selective about. There is new Python flag added called --python-flag=site that restores the inclusion of site module.
Standalone: Exclude non-critical codec modules from being technical, i.e. have to be available at program startup. This removes the need for e.g. bz2 related extension modules previously included.
In reformulations, use dictionary methods directly, we have since introduced dictionary specific methods, and avoid the unnecessary churn during optimization.
The complex call helper could trigger unnecessary passes in some cases. The pure functions were immediately optimized, but usages in other modules inside loops sometimes left them in incomplete states.
Windows: Avoid repeated hashing of the same files over and over for clcache.
Cache dependencies of bytecode demoted modules in first compile and reuse that information in subsequent compilations.
Linux: Added option for switching compression method for onefile created with AppImage. The default is also now gzip and not xz which has been observed to cause much slower startup for little size gains.
Standalone: For failed relative imports, during compiled time absolute imports were attempted still and included if successful, the imports would not be use them at run time, but lead to more modules being included than necessary.

Organizational

There is now a Discord server for Nuitka community where you can hang out with the developers and ask questions. It is mirrored with the Gitter community chat, but offers more features.
The anti-bloat is now on by default. It helps scalability by changing popular packages to not provide test frameworks, installation tools etc. in the resulting binary. This oftentimes reduces the compilation by thousands of modules.
Also the multiprocessing plugin is now on by default. Detecting its need automatically removes a source of problems for first time users, that didn’t know to enable it, but had all kinds of strange crashes from multiprocessing malfunctioning. This should enhance the out of the box experience by a lot.
With this release, the version numbering scheme will be changed. For a long time we have used 4 digits, where one is a leading zero. That was initially done to indicate that it’s not yet ready. However, that is just untrue these days. Therefore, we switch to 3 digits, and a first hotfix with now be 0.7.1 rather than 0.6.19.1, which is too long.

It has been observed that people disregard differences in the third digit, but actually for Nuitka these have oftentimes been very important updates. This change is to rectify it, and a new release will be 0.8, and there will be a 1.0 release after 0.9.
Added a new section to User Manual that explains how to manually load files, such that it is cleaner and compatible code. Using paths relative to current directory is not the right way, but there are nice helpers that make it very simple and correct with all kinds of contexts.
Report the MSVC version in Scons output during compilation. The 2022 version is required, but we support everything back to 2008, to work on very old systems as well. This will help identifying differences that arise from there.
Quality: Find Clang format from MSVC 2022 too. We use in auto format of Nuitka source code, but need to also search that as a new path.
Added a spellchecker extension for Visual Code, resulting in many spelling fixes in all kinds of documentation and code. This finds more things than codespell, but also has a lot of false alarms.
Check value of --onefile-tempdir-spec for typical user errors. It cannot be . as that would require to overwrite the onefile binary on Windows, and will generally behave very confusing. Warn about absolute or relative paths going outside of where the binary lives. Can be useful in controlled setups, but not generally. Also warn about using no variables, making non-unique paths.
macOS: Flavor detection was largely expanded. The Apple flavor is recognized on more systems. Homebrew was newly added, and we actually can detect CPython reliably as a first.
Added a tool from leo project to create better .pyi files for modules. We will make use of it in the future to enhance the files created by Nuitka to not only contain hidden dependencies, but optionally also module signatures.
Plugins: Clearer information from pyside2 that patched wheels might be mandatory and workarounds only patches cannot be done for older Python.
Added progress bars for DLL dependency detection and DLL copying for standalone. These both can end up using take a fair bit of time depending on project size, and it’s nice to know what’s going on.
macOS: Added support for using both --onefile and --macos-create-app-bundle as it is needed for PySide2 due to issues with signing code now.
Added warning when attempting to include extension modules in an accelerated compilation.
Modules: Catch the user error of following all imports when creating a module. This is very unlikely to produce usable results.
Start integrating Sourcery for improved Nuitka code. It will comment on PRs and automatically improve Nuitka code as we develop it.
Debugging: Added command line tool find-module that outputs how Nuitka locates a module in the Python environment it’s ran with. That removes the need to use Python prompt to dump __file__ of imported modules. Some modules even hide parts of their namespace actively during run-time, the tool will not be affected by that.

Cleanups

Refactored Python scan previously used for Scons usage on versions that need to run in with another Python to be more generally usable.
Use explicit nuitka.utils.Hashing module that allows the core to perform these operations with simpler code.
macOS: Use isPathBelow for checking if something is a system library for enhanced robustness and code clarity.
macOS: Make sure to use our proper error checking wrappers for command execution when using tools like otool or codesign.
Standalone: Avoid a temporary file with a script during technical import detection. These have been observed to potentially become corrupted, and this avoids any chance of that happening, while also being simpler code.
Avoid naming things shlib and call them extension instead. Inspired by the spell checker disliking the former term, which is also less precise.
Removed the dead architecture selection option for Windows, it was unused for a long time.
Moved Windows SxS handling of DLLs to a more general place where also macOS specific tasks are applied, to host standard modification of DLLs during their copying.

Tests

Better matching of relative filenames for search modes of the individual test suite runners.
Debugger outputs on segfaults were no longer visible and have been restored.

Summary

This release is tremendous progress for macOS. Finally biting the bullet and paying obscene amounts of money to rent an M1 machine, it was possible to enhance the support for this platform. Currently typical packages for macOS are being made compatible as well, so it can now be expected to perform equally well.

On the quality side, the spell checker has had some positive effects, finding typos and generally misspelled code, that codespell does not, due to it being very conservative.

The trend to enhance plugins has continued. The copying of DLLs is very nearly finalized. Making more plugins enabled by default is seeing a lot of progress, with 2 important ones addressed.

Work on the size of distributions has seen a lot of positive results, in that now standalone distributions are often very minimal, with many extension modules from standard library no longer being present.

Next Nuitka Live Stream

2022-01-30T00:00:00+00:00

Today, Sunday 30.01.2022, there will be the third live stream of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 9-12 CEST and from 18 CEST until probably at least 20 CEST, but it seems I tend to go overtime.

Last time

So the last two streams are on my Youtube, these are around 4h videos, done on the same day.

In the second stream I was working on Onefile compression for all Python versions, by making the location of another Python that for Scons we sort of did already, reusable. So now, you can run Nuitka with Python2.6 and compile your legacy code, while with say 3.6 also installed on the same system, you do the zstandard compression. That will be used for other things in the future as well. This is going to be part of the next release and currently on develop.

In the first stream, I also did a little bit of performance plans, but mostly only showing people what I have there in stock, not actually went there and I started work on a upx plugin, such that DLLs for standalone are compressed with that. That also required a bit of plugin interface changes and research. This one works mostly, but will need more live. Also I think I looked at reducing what is included with the follow standard library option, getting very minimal distributions out of it.

This time

All around, this last stream, was a huge success. I was a bit under the weather last weekend, but we go on now.

Not sure yet, what to do. I might be debugging issues I have with 2.6 and a recent optimization, that prevents factory branch from becoming the next pre-release, I might be looking at macOS still. And I might be looking at the caching of bytecode demoted modules, that is kind of ready to be used, but is not currently, which is a pity. And of course, the Python PGO may get a closer look, such that e.g. it works for standalone mode.

How to Join

Join me

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

First Nuitka Live Stream

2022-01-14T00:00:00+00:00

Tomorrow, Saturday 15.01.2022, there will be not one, but two live stream of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 9-12 CEST and from 17-20 CEST.

I want to do this on a regular basis, and I will be happy to do this even when nearly nobody shows up. It will be however your chance, and in previous GSoC and generally, I made very good experiences with joint sessions. So I hope you will help me pasting code snippets and Google results in chat, and discuss the optimization that we are going to add.

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

Nuitka Release 0.6.19

2022-01-12T10:51:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release adds support for 3.10 while also adding very many new optimization, and doing a lot of bug fixes.

Bug Fixes

Calls to importlib.import_module with expressions that need releases, i.e. are not constant values, could crash the compilation. Fixed in 0.6.18.1 already.
After a fix for the previous release, modules that fail to import are attempted again when another import is executed. However, during this initialization for top level module in --module mode, this was was done repeatedly, and could cause issues. Fixed in 0.6.18.1 already.
Standalone: Ignore warning given by patchelf on Linux with at least newer OpenSUSE. Fixed in 0.6.18.1 already.

Fix, need to avoid computing large values out of << operation as well. Fixed in 0.6.18.2 already.

# This large value was computed at run time and then if used, also
# converted to a string and potentially hashed, taking a long time.
1 << sys.maxint

Standalone: Ignore warning given by patchelf on Linux about a workaround being applied.
Fix, calls to importlib.import_module were not correctly creating code for dynamic argument values that need to be released, causing the compilation to report the error. Fixed in 0.6.18.1 already.
MSYS2: Fix, the console scripts are actually good for it as opposed to CPython, and the batch scripts should not be installed. Fixed in 0.6.18.2 already.
Setuptools: Added support older version of setuptools in meta build integration of Nuitka.
Fix, calls to importlib.import_module with 2 arguments that are dynamic, were not working at all. Fixed in 0.6.18.2 already.
Windows: Compiling with MinGW64 without ccache was not working due to issues in Scons. Fixed in 0.6.18.2 already.
Fix, the repr built-in was falsely annotated as producing a str value, but it can be also derived or unicode in Python2.
Fix, attribute nodes were not considering the value they are looking up on. Now that more values will know to have the attributes, that was causing errors. Fixed in 0.6.18.2 already.
Fix, left shifting can also produce large values and needs to be avoided in that case, similar to what we do for multiplications already. Fixed in 0.6.18.2 already.
UI: The new option --disable-ccache didn’t really have the intended effect. Fixed in 0.6.18.3 already.
UI: The progress bar was causing tearing and corrupted outputs, when outputs were made, now using proper tqdm API for doing it, this has been solved. Fixed in 0.6.18.4 already.
Fix, the constant value sys.version_info didn’t yet have support for its type to be also a compile time constant in e.g. tuples. Fixed in 0.6.18.4 already.
Onefile: Assertions were not disabled, and on Windows with MinGW64 this lead to including the C filenames of the zstd inline copy files and obviously less optimal code. Fixed in 0.6.18.4 already.
Standalone: Added support for bottle.ext loading extensions to resolve at compile time. Fixed in 0.6.18.5 already.
Standalone: Added support for seedir required data file. Fixed in 0.6.18.5 already.
MSYS2: Failed to link when using the static libpython, which is also now the default for MSYS2. Fixed in 0.6.18.5 already.
Python3.6+: Fix, the intended finalizer of compiled asyncgen was not present and in fact associated to help type. This could have caused corruption, but that was also very unlikely. Fixed in 0.6.18.5 already.
Python3: Fix, need to set __file__ before executing modules, as some modules, e.g. newer PyWin32 use them to locate things during their initialization already.
Standalone: Handle all PyWin32 modules that need the special DLLs and not just a few.
Fix, some .pth files create module namespaces with __path__ that does not exist, ignore these in module importing.
Python2.6-3.4: Fix, modules with an error could use their module name after it was released.
Distutils: When providing arguments, the method suggested in the docs is not compatible with all other systems, e.g. not setuptools_rust for which a two elements tuple form needs to be used for values. Added support for that and documented its use as well in the User Manual.
Python3.7+: Do no longer allow deleting cell values, this can lead to corruption and should be avoided, it seems unlikely outside of tests anyway.
Standalone: Added support for more ciphers and hashes with pycryptodome and pycryptodomex, while also only including Ciphers when needed.
Distutils: Was not including modules or packages only referenced in the entry point definition, but not in the list of packages. That is not compatible and has been fixed.
Fix, must not expose the constants blob from extension modules, as loading these into a compiled binary can cause issues in this case.
Standalone: Added support for including OpenGL and SSL libraries with PySide2 and PySide6 packages.
Windows: Fix, the cmd files created for uninstalled Python and accelerated programs to find the Python installation were not passing command line arguments.
Windows: Executing modules with --run was not working properly due to missing escaping of file paths.
Fix, parsing .pyi files that make relative imports was not resolving them correctly.
Python3: Fix, when disabling the console on Windows, make sure the file handles still work and are not None.
Windows: Fix, need to claim all OS versions of Windows as supported, otherwise e.g. high DPI features are not available.

New Features

Programs that are to be executed with the -m flag, can now be compiled with --python-flag=-m and will then behave in a compatible way, i.e. load the containing package first, and have a proper __package__ value at run time.
We now can write XML reports with information about the compilation. This is initially for use in PGO tests, to decide if the expected forms of inclusions have happened and should grow into a proper reporting tool over time. At this point, the report is not very useful yet.
Added support for Python 3.10, only match statements are not completely supported. Variations with | matches that also assign are not allowed currently.
Windows: Allow using --clang with --mingw64 to e.g. use the clang.exe that is contained in the Nuitka automatic download rather than gcc.exe.
Added support for Kivy. Works through a plugin that is automatically enabled and needs no other inputs, detecting everything from using Kivy at compile time.
Added initial support for Haiku OS, a clone of BeOS with a few differences in their Python installation.
Added experimental plugin trio that works around issues with that package.

Optimization

Also trust hard imports made on the module level in function level code, this unlocks many more static optimization e.g. with sys.version_info when the import and the use are not on the same level.
For the built-in type method calls with generic implementation, we now do faster method descriptor calls. These avoid creating a temporary PyCFunction object, that the normal call slot would, this should make these calls faster. Checking them for compiled function, etc. was only wasteful, so this makes it more direct.
Loop and normal merge traces were keeping assignments made before the loop or inside a branch, that was otherwise unused alive. This should enable more optimization for code with branches and loops. Also unused loop traces are now recognized and removed as well.
Avoiding merges of escaped traces with the unescaped trace, there is no point in them. This was actually happening a lot and should mean a scalability improvement and unlock new optimization as well.
Avoid escaping un-init traces. Unset values need not be considered as potentially modified as that cannot be done.
The str shape is now detected through variables, this enables many optimization on the function level.
Added many str operation nodes.

These are specifically all methods with no arguments, as these are very generic to add, introduced a base class for them, where we know they all have no effect or raise, as these functions are all guaranteed to succeed and can be served by a common base class.

This covers the str.capitalize, str.upper, str.lower, str.swapcase, str.title, str.isalnum, str.isalpha, str.isdigit, str.islower, str.isupper, str.isspace, and str.istitle functions.

For static optimization str.find and str.rfind were added, as they are e.g. used in a sys.version.find(...) style in the os module, helping to decide to not consider OS/2 only modules.

Then, support for str.index and str.rindex was added, as these are very similar to str.find forms, only that these may raise an exception.

Also add support for str.split and str.rsplit which will be used sometimes for code needed to be compile time computed, to e.g. detect imports.

Same goes for endswith and startswith, the later is e.g. popular with sys.platform checks, and can remove a lot of code from compilation with them now being decided at compile time.

Note

A few str methods are still missing, with time we will achieve all of them, but this will take time.
Added trust for sys.builtin_module_names as well. The os module is using it to make platform determinations.
When writing constant values, esp. tuple, list, or dict values, an encoding of “last value” has been added, avoiding the need to repeat the same value again, making many values more compact.
When starting Nuitka, it usually restarts itself with information collected in a mode without the site module loaded, and with hash randomization disabled, for deterministic behaviour. There is a option to prevent this from happening, where the goal is to avoid it, e.g. in testing, say for the coverage taking, but that meant to parse the options twice, which also loads a lot of code.

Now only a minimal amount of code is used, and the options are parsed only on the restart, and then an error is raised when it notices, it was not allowed to do so. This also makes code a lot cleaner.
Specialized comparison code for Python2 long and Python3 int code, making these operations much faster to use.
Specialized comparison code for Python2 unicode and Python3 str code, making these operations much faster to use, currently only == and != are fully accelerated, the other comparisons will follow.
Enable static libpython with Python3 Debian packages too. As with Python2, this will improve the performance of the created binary a lot and reduce size for standalone distribution.
Comparisons with in and not in also consider value traces and go through variables as well where possible. So far only the rich comparisons and is and is not did that.
Create fixed import nodes in re-formulations rather than __import__ nodes, avoiding later optimization doing that, and of course that’s simpler code too.
Python 3.10: Added support for union types as compiled time constants.
Modules are now fully optimized before considering which modules they are in turn using, this avoids temporary dependencies, that later turn out unused, and can shorten the compilation in some cases by a lot of time.
On platforms without a static link library, in LTO mode, and with gcc, we can use the -O3 mode, which doesn’t work for libpython, but that’s not used there. This also includes fake static libpython, as used by MinGW64 and Anaconda on Windows.
The anti-bloat plugin now also handles newer sklearn and knows more about the standard library, and its runners which it will exclude from compilation if use for it. Currently that is not the default, but it should become that.

Organizational

Migrated the Nuitka blog from Nikola to Sphinx based ABlog and made the whole site render with Sphinx, making it a lot more usable.
Added a small presentation about Nuitka on the Download page, to make sure people are aware of core features.
The gi plugin is now always on. The copying of the typelib when gi is imported is harmless and people can disable the plugin if that’s not needed.
The matplotlib plugin is new and also always on. It previously was part of the numpy plugin, which is doing too many unrelated things. Moving this one out is part of a plan to split it up and have it on by default without causing issues.
MSYS2: Detecting MinGW and POSIX flavors of this Python. For the MinGW flavor of MSYS2, the option --mingw64 is now the default, before it could attempt to use MSVC, which is not going to work for it. And also the Tcl and Tk installations of it are being detected automatically for the tk-inter plugin.
Added Windows version to Nuitka version output, so we have this for bug reports.
User Manual: Added example explaining how to access values from your code in Nuitka project options.
UI: For Python flavors where we expect a static libpython, the error message will now point out how to achieve it for each flavor.
UI: Disable progress bar when --show-scons is used, it makes capturing the output from the terminal only harder.
UI: Catch error of specifying both --msvc= and --mingw64 options.
Distutils: Improved error messages when using setuptools or build integration and failing to provide packages to compile.
Plugins: Removed now unused feature to rename modules on import, as it was only making the code more complex, while being no more needed after recently adding a place for meta path based importers to be accounted for.
Twitter: Use embedded Tweet in Credits, and regular follow button in User Manual.
Warnings about imports not done, are now only given when optimization can not remove the usage, and no options related to following have been given.
Added Windows version to --version output of Nuitka. This is to more clearly recognize Windows 10 from Windows 11 report, and also the odd Windows 7 report, where tool chain will be different.
In Visual Code, the default Python used is now 3.9 in the “Linux” C configuration. This matches Debian Bullseye.
Nicer outputs from check mode of the auto-format as run for CI testing, displays problematic files more clearly.
Remove broken links to old bug tracker that is no longer online from the Changelog.
UI: When hitting CTRL-C during initial technical import detection, no longer ask to submit a bug report with the exception stack, instead exit cleanly.
Windows: Enable LTO mode for MinGW64 and other gcc by default. We require a version that can do it, so take advantage of that.
For cases, where code generation of a module takes long, make sure its name is output when CTRL-C is hit.
Windows: Splash screen only works with MSVC, added error indicator for MinGW64 that states that and asks for porting help.

Cleanups

Generate all existing C code for generic builtin type method calls automatically, and use those for method attribute lookups, making it easier to add more.
Changed TkInter module to data file providing interface, yielding the 2 directories in question, with a filter for demos.
The importing code got a major overhaul and no longer works with relative filenames, or filenames combined with package names, and module names, but always only with module names and absolute filenames. This cleans up some of the oldest and most complex code in Nuitka, that had grown to address various requirements discovered over time.
Major cleanup of Jinja2 template organisation.

Renamed all C templates from .j2 to .c.j2 for clarity, this was not done fully consistent before. Also move all C templates to nuitka.codegen package data, it will be confusing to make a difference between ones used during compile time and for the static generation, and the lines are going to become blurry.

Added Jinja2 new macro CHECK_OBJECTS to avoid branches on argument count in the call code templates. More of these things should be added.

Cleanup of code that generates header declarations, there was some duplication going on, that made it hard to generate consistent code.
Removed nuitka.finalizatios.FinalizationBase, we only have one final visitor that does everything, and that of course makes a lot of sense for its performance.
Major cleanup of the Scons C compiler configuration setup. Moved things to the dedicate function, and harmonized it more.
Resolved deprecation warnings given by with --python-debug for Nuitka.

Tests

Started test suite for Python PGO, not yet completely working though, it’s not yet doing what is needed though.
Added generated test that exercises str methods in multiple variations.
Revived reflected test suite, that had been removed, because of Nuitka special needs. This one is not yet passing again though, due to a few details not yet being as compatible as needed.
Added test suite for CPython 3.10 and enable execution of tests with this version on GitHub actions.

Summary

This release is another big step forward.

The amount of optimization added is again very large, some of which yet again unlocks more static optimization of module imports, that previously would have to be considered implicit. Now analyzing these on the function level as well, we can start searching for cases, where it could be done, but is not done yet.

After starting with dict, method optimization has focused on str which is esp. important for static optimization of imports. The next goal will here be to cover list which are important for run time performance and currently not yet optimized. Future releases will progress there, and also add more types.

The C type specialization for Python3 has finally progressed, such that it is also covering the long and unicode and as such not limited to Python2 as much. The focus now needs to turn back to not working with PyObject * for these types, but e.g. with += 1 to make it directly work with CLONG rather than LONG for which structural changes in code generation will be needed.

For scalability, the anti-bloat work has not yet progressed as much as to be able to enable it by default. It needs to be more possible to disable it where it causes problems, e.g. when somebody really wants to include pytest and test frameworks generally, that’s something that needs to be doable. Compiling without anti-bloat plugin is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.

The support for Windows has been further refined, actually fixing a few important issues, esp. for the Qt bindings too.

This release adds support for 3.10 outside of very special match statements, bringing Nuitka back to where it works great with recent Python. Unfortunately orderedset is not available for it yet, which means it will be slower than 3.9 during compilation.

Overall, Nuitka is closing many open lines of action with this. The setuptools support has yet again improved and at this point should be very good.

Nuitka Release 0.6.18

2021-12-11T17:21:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a focus on new features of all kinds, and then also new kinds of performance improvements, some of which enable static optimization of what normally would be dynamic imports, while also polishing plugins and adding also many new features and a huge amount of organizational changes.

Bug Fixes

Python3.6+: Fixes to asyncgen, need to raise StopAsyncIteration rather than StopIteration in some situations to be fully compatible.
Onefile: Fix, LTO mode was always enabled for onefile compilation, but not all compilers support it yet, e.g. MinGW64 did not. Fixed in 0.6.17.1 already.
Fix, type calls with 3 arguments didn’t annotate their potential exception exit. Fixed in 0.6.17.2 already.
Fix, trusted module constants were not working properly in all cases. Fixed in 0.6.17.2 already.
Fix, pkg-resources exiting with error at compile time for unresolved requirements in compiled code, but these can of course still be optional, i.e. that code would never run. Instead give only a warning, and run time fail on these. Fixed in 0.6.17.2 already.
Standalone: Prevent the inclusion of drm libraries on Linux, they have to come from the target OS at run time. Fixed in 0.6.17.2 already.
Standalone: Added missing implicit dependency for ipcqueue module. Fixed in 0.6.17.3 already.
Fix, Qt webengine support for everything but PySide2 wasn’t working properly. Partially fixed in 0.6.17.3 already.
Windows: Fix, bootstrap splash screen code for Windows was missing in release packages. Fixed in 0.6.17.3 already.
Fix, could crash on known implicit data directories not present. Fixed in 0.6.17.3 already.
macOS: Disable download of ccache binary for M1 architecture and systems before macOS 10.14 as it doesn’t work on these. Fixed in 0.6.17.3 already.
Standalone: The pendulum.locals handling for Python 3.6 was regressed. Fixed in 0.6.17.4 already.
Onefile: Make sure the child process is cleaned up even after its successful exit. Fixed in 0.6.17.4 already.
Standalone: Added support for xmlschema. Fixed in 0.6.17.4 already.
Standalone: Added support for curses on Windows. Fixed in 0.6.17.4 already.
Standalone: Added support for coincurve module. Fixed in 0.6.17.5 already.
Python3.4+: Up until Python3.7 inclusive, a workaround for stream encoding (was ASCII), causing crashes on output of non-ASCII, other Python versions are not affected. Fixed in 0.6.17.5 already.
Python2: Workaround for LTO error messages from older gcc versions. Fixed in 0.6.17.5 already.
Standalone: Added support for win32print. Fixed in 0.6.17.6 already.
Fix, need to prevent usage of static libpython in module mode or else on some Python versions, linker errors can happen. Fixed in 0.6.17.6 already.
Standalone: Do not load site module early anymore. This might have caused issues in some configurations, but really only would be needed for loading inspect which doesn`t depend on it in standalone mode. Fixed in 0.6.17.6 already.
Fix, could crash with generator expressions in finally blocks of tried blocks that return. Fixed in 0.6.17.7 already.
```
try:
   return 9
finally:
   "".join(x for x in b"some_iterable")
```

Python3.5+: Compatibility of comparisons with types.CoroutineType and types.AsyncGeneratorType types was not yet implemented. Fixed in 0.6.17.7 already.

# These already worked:
assert isinstance(compiledCoroutine(), types.CoroutineType) is True
assert isinstance(compiledAsyncgen(), types.AsyncGeneratorType) is True

# These now work too:
assert type(compiledCoroutine()) == types.CoroutineType
assert type(compiledAsyncgen()) == types.AsyncGeneratorType

Standalone: Added support for ruamel.yaml. Fixed in 0.6.17.7 already.
Distutils: Fix, when building more than one package, things could go wrong. Fixed in 0.6.17.7 already.
Fix, for module mode filenames are used, and for packages, you can specify a directory, however, a trailing slash was not working. Fixed in 0.6.17.7 already.
Compatibility: Fix, when locating modules, a package directory and an extension module of the same name were not used according to priority. Fixed in 0.6.17.7 already.
Standalone: Added workaround importlib_resources insisting on Python source files to exist to be able to load datafiles. Fixed in 0.6.17.7 already.
Standalone: Properly detect usage of hard imports from standard library in --follow-stdlib mode.
Standalone: Added data files for openapi_spec_validator.
MSYS2: Fix, need to normalize compiler paths before comparing.
Anaconda: For accelerated binaries, the created .cmd file wasn’t containing all needed environment.
macOS: Set minimum OS version derived from the Python executable used, this should make it work on all supported platforms (of that Python).
Standalone: Added support for automatic inclusion of xmlschema package datafiles.
Standalone: Added support for automatic inclusion of eel package datafiles.
Standalone: Added support for h5py package.
Standalone: Added support for phonenumbers package.
Standalone: Added support for feedparser package, this currently depends on the anti-bloat plugin to be enabled, which will become enabled by default in the future.
Standalone: Added gi plugin for said package that copies typelib files and sets the search path for them in standalone mode.
Standalone: Added necessary datafiles for eel package.
Standalone: Added support for QtWebEngine to all Qt bindings and also make it work on Linux. Before only PySide2 on Windows was supported.
Python3: Fix, the all built-in was wrongly assuming that bytes values could not be false, but in fact they are if they contain \0 which is actually false. The same does not happen for string values, but that’s a difference to be considered.
Windows: The LTO was supposed to be used automatically on with MSVC 14.2 or higher, but that was regressed and has been repaired now.
Standalone: Extension modules contained in packages, depending on their mode of loading had the __package__ value set to a wrong value, which at least impacted new matplotlib detection of Qt backend.
Windows: The python setup.py install was installing binaries for no good reason.

New Features

Setuptools support. Documented bdist_nuitka and bdist_wheel integration and added support for Nuitka as a build package backend in pyproject.toml files. Using Nuitka to build your wheels is supposed to be easy now.
Added experimental support for Python 3.10, there are however still important issues with compatibility with the CPython 3.9 test suite with at least asyncgen and coroutines.
macOS: For app bundles, version information can be provided with the new option --macos-app-version.
Added Python vendor detection of Anaconda, pyenv, Apple Python, and pyenv and output the result in version output, this should make it easiert to analyse reported issues.
Plugins: Also handle the usage of __name__ for metadata version resolution of the pkg-resources standard plugin.
Plugins: The data-files standard plugin now reads configuration from a Yaml file that data-files.yml making it more accessible for contributions.
Windows: Allow enforcing usage of MSVC with --msvc=latest. This allows you to prevent accidental usage of MinGW64 on Windows, when MSVC is intended, but achieves that without fixing the version to use.
Windows: Added support for LTO with MinGW64 on Windows, this was previously limited to the MSVC compiler only.
Windows: Added support for using --debugger with the downloaded MinGW64 provided gdb.exe.

Note

It doesn`t work when executed from a Git bash prompt, but e.g. from a standard command prompt.
Added new experimental flag for compiled types to inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses PyObject_IsInstance work, providing support for packages like pydantic.
Plugins: The Qt binding plugins now resolve pyqtgraph selection of binding by hard coding QT_LIB. This will allow to resolve its own dynamic imports depending on that variable at compile time. At this time, the compile time analysis is not covering all cases yet, but we hope to get there.
macOS: Provide minOS for standalone builds, derived from the setting of the Python used to create it.
UI: Added new option --disable-ccache to prevent Nuitka from injecting ccache (Clang, gcc) and clcache (MSVC) for caching the C results of the compilation.
Plugins: Added experimental support for PyQt6. While using PySide2 or PySide6 is very much recommended with Nuitka, this allows its use.
UI: Added option --low-memory to allow the user to specify that the compilation should attempt to use less memory where possible, this increases compile times, but might enable compilation on some weaker machines.

Optimization

Added dedicated attribute nodes for attribute values that match names of dictionary operations. These are optimized into dedicate nodes for methods of dictionaries should their expression have an exact dictionary shape. These in turn optimize calls on them statically into dictionary operations. This is done for all methods of dict for both Python2 and Python3, namely get, items, iteritems, itervalues, iterkeys, viewvalues, viewkeys, pop, setdefault, has_key, clear, copy, update.

The new operation nodes also add compile time optimization for being used on constant values where possible.
Also added dedicated attribute nodes for string operations. For operations, currently only part of the methods are done. These are currently only join, strip, lstrip, rstrip, partition, rpartition. Besides performance, this subset was enough to cover compile time evaluation of module name computation for importlib.import_module as done by SWIG bindings, allowing these implicit dependencies to be discovered at compile time without any help, marking a significant improvement for standalone usage.
Annotate type shape for dictionary in/not in nodes, this was missing unlike in the generic in/not in nodes.
Faster processing of “expression only” statement nodes. These are nodes, where a value is computed, but then not used, it still needs to be accounted for though, representing the value release.
```
something() # ignores return value, means statement only node
```
Windows: Enabled LTO by default with MinGW64, which makes it produce much faster results. It now yield faster binaries than MSVC 2019 with pystone.
Windows: Added support for C level PGO (Profile Guided Optimization) with MSVC and MinGW64, allowing extra speed boosts from the C compilation on Windows as well.
Standalone: Better handling of requests.packages and six.moves. The old handling could duplicate their code. Now uses a new mechanism to resolve metapath based importer effects at compile time.
Avoid useless exception checks in our dictionary helpers, as these could only occur when working with dictionary overloads, which we know to not be the case.
For nodes, have dedicated child mixin classes for nodes with a single child value and for nodes with a tuple of children, so that these common kind of nodes operate faster and don’t have to check at run time what type they are during access.
Actually make use of the egg cache. Nuitka was unpacking eggs in every compilation, but in wheel installs, these can be quite common and should be faster.
Star arguments annotated their type shape, but the methods to check for dictionary exactly were not affected by this preventing optimization in some cases.
Added anti-bloat configuration for main programs present in the modules of the standard library, these can be removed from the compilation and should lower dependencies detected.
Using static libpython with pyenv automatically. This should give both smaller (standalone mode) and faster results as is the case when using this feature..
Plugins: Added improvements to the anti-bloat plugin for gevent to avoid including its testing framework.
Python3.9+: Faster calls into uncompiled functions from compiled code using newly introduced API of that version.
Statically optimize importlib.import_module calls with constant args into fixed name imports.
Added support for sys.version_info to be used as a compile time constant. This should enable many checks to be done at compile time.
Added hard import and static optimization for typing.TYPE_CHECKING.
Also compute named import lookup through variables, expanding their use to more cases, e.g. like this:
```
import sys
...
if sys.version_info.major >= 3:
   ...
```
Also optimize compile time comparisons through variable names if possible, i.e. the value cannot have changed.
Faster calls of uncompiled code with Python3.9 or higher avoiding DLL call overhead.

Organizational

Commercial: There are Buy Now buttons available now for the direct purchase of the Nuitka Commercial offering. Finally Credit Card, Google Pay, and Apple Pay are all possible. This is using Stripe. Get in touch with me if you want to use bank transfer, which is of course still best for me.
The main script runners for Python2 have been renamed to nuitka2 and nuitka2-run, which is consistent with what we do for Python3, and avoids issues where bin folder ends up in sys.path and prevents the loading of nuitka package.
Windows: Added support for Visual Studio 2022 by updating the inline copy of Scons used for Windows to version 4.3.0, on non Windows, the other ones will keep being used.
Windows: Requiring latest MinGW64 with version 11.2 as released by WinLibs, because this is known to allow LTO, where previous releases were missing needed binaries.
Reject standalone mode usage with Apple Python, as it works only with the other supported Pythons, avoiding pitfalls in attempting to distribute it.
Move hosting of documentation to Sphinx, added Changelog and some early parts of API documentation there too. This gives much more readable results than what we have done so far with Nikola. More things will move there.
User Manual: Add description how to access code attributes in nuitka-project style options.
User Manual: Added commands used to generate performance numbers for Python.
User Manual: List other Python’s for which static linking is supposed to work.
Improved help for --include-package with a hint how to exclude some of the subpackages.
Started using Jinja2 in code templates with a few types, adding basic infrastructure to do that. This will be expanded in the future.
Updated plugin documentation with more recent information.
Added Python flavor as detected to the --version output for improved bug reports.
Linux: Added distribution name to --version output for improved bug reports.
Always enable the gevent plugin, we want to achieve this for all plugins, and this is only a step in that direction.
Added project URLs for PyPI, so people looking at it from there have some immediate places to checkout.
Debian: Use common code for included PDF files, which have page styles and automatic corrections for rst2pdf applied.
Updated to latest black, isort, pylint versions.
The binary names for Python2 changed from nuitka and nuitka-run to nuitka2 and nuitka2-run. This harmonizes it with Python2 and avoids issues, where the bin folder in sys.path can cause issues with re-execution of Nuitka finding those to import.

Note

You ought to be using python -m nuitka style of calling Nuitka anyway, as it gives you best control over what Python is used to run Nuitka, you can pick python2 there if you want it to run with that, even with full path. Check the relevant section in the User Manual too.
Added support for Fedora 34 and Fedora 35.

Cleanups

In a change of mind --enable-plugin has become the only form to enable a plugin used in documentation and tests.
Massive cleanup of numpy and Qt binding plugins, e.g. pyside2. Data files and DLLs are now provided through proper declarative objects rather than copied manually. The handling of PyQt5 from the plugin should have improved as a side effect.
Massive cleanups of all documentation in ReST format. Plenty of formatting errors were resolved. Many typos were identified and globally fixed. Spellings e.g. of “Developer Manual” are now enforced with automatic replacements. Also missing or wrong quotes were turned to proper methods. Also enforce code language for shell scripts to be the same everywhere.
Removed last usages of getPythonFlags() and made the function private, replacing their use with dedicated function to check for individual flags.
Avoid string comparison with nuitka.utils.getOS() and instead add accessors that are more readable, e.g. nuitka.utils.isMacOS() and put them to use where it makes sense.
Replaced usages of string tests in list of python flags specified, with functions that check for a specific name with a speaking function name.
Added mixin for expressions that have no side effect outside of their value, providing common method implementation more consistently.
Remove code geared to using old PyLint and on Python2, we no longer use that. Also removed annotations only used for overriding Python2 builtins from Nuitka code.
The PDF specific annotations were moved into being applied only in the PDF building step, avoiding errors for raw PDF directives.
Apply Visual Code auto-format to our Yaml files. This is unfortunately not and automatic formatting yet.
Introduce dedicated nuitka.utils.Json module, as we intend to expand its usage, e.g. for caching.
Replacing remaining usages of print functions with uses of nuitka.Tracing instead.
Massive cleanup of the gevent plugin, user proper method to execute code after module load, rather than source patching without need. The plugin no longer messes with inclusions that other code already provides for standalone.
Using own helper to update sys module attributes, to avoid errors from old C compilers, and also cleaning up using code to not have to cast on string constants.
More consistent naming of plugin classes, and enforce a relationship of detector class names to the names of detected plugins. The new naming consistency is now enforced.

Tests

Added CPython 3.10 test suite, it needs more work though.
Added generated test that exercises dictionary methods in multiple variations.
Test suite names were specified wrongly in a few of them.

Summary

This release is again a huge step forward. It refines on PGO and LTO for C level to work with all relevant compilers. Internally Python level PGO is prepared, but only a future release will feature it. With that, scalability improvements as well as even more performance improvements will be unlocked.

The amount of optimization added this time is even bigger, some of which unlocks static optimization of module imports, that previously would have to be considered implicit. This work will need one extra step, namely to also trace hard imports on the function level, then this will be an extremely powerful tool to solve these kinds of issues in the future. The next release will have this and go even further in this area.

With the dictionary methods, and some string methods, also a whole new kind of optimization has been started. These will make working with dict containers faster, but obviously a lot of ground is to cover there still, e.g. list values are a natural target not yet started. Future releases will progress here.

Type specialization for Python3 has not progressed though, and will have to be featured in a future releases though.

For scalability, the anti-bloat work has continued, and this should be the last release, where this is not on by default. Compiling without it is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.

The support for macOS has been refined, with version information being possible to add, and adding information to the binary about which OSes are supported, as well as rejecting Apple Python, which is only a trap if you want to deploy to other OS versions. More work will be needed to support pyenv or even Homebrew there too, for now CPython is still the recommended platform to use.

This release achieves major compatibility improvements. And of course, the experimental support for 3.10 is not the least. The next release will strive to complete the support for it fully, but this should be usable at least, for now please stay on 3.9 if you can.

Nuitka Release 0.6.17

2021-11-11T13:40:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a focus on performance improvements, while also polishing plugins and adding many new features.

Bug Fixes

Fix, plugins were not catching being used on packages not installed. Fixed in 0.6.16.2 already.
macOS: Fix weaknesses in the otool parsing to determine DLL dependency parsing. Fixed in 0.6.16.2 already.
Linux: Allow onefile program args with spaces contained to be properly passed. Fixed in 0.6.16.3 already.
Windows: Avoid using less portable C function for %PID% formatting, which restores compilation on Windows 7 with old toolchains. Fixed in 0.6.16.3 already.
Standalone: Added support for fstrings package. Fixed in 0.6.16.3 already.
Compatibility: Fix, need to import .pth files after site module, not before. This was causing crashes on CentOS7 with Python2. Fixed in 0.6.16.3 already.
Compatibility: Fix, when extension modules failed to load, in some cases the ImportError was lost to a KeyError. Fixed in 0.6.16.3 already.
Fix, linker resource modes code and linker were not working anymore, but are needed with LTO mode at least. Fixed in 0.6.16.3 already.
Standalone: Bytecode modules with null bytes in standard library, typically from disk corruption, were not handled properly. Fixed in 0.6.16.3 already.
Fix, failed .throw() into generators could cause corruption. Fixed in 0.6.16.4 already.
Python2: Fix, the bytecode compilation didn’t respect the --python-flag=no_asserts mode. Fixed in 0.6.16.4 already.
Fix, calls were not annotating their arguments as escaped, causing corruption of mutable in static optimization. Fixed in 0.6.16.5 already.
Fix, some sequence objects, e.g. numpy.array actually implement in-place add operations that need to be called. Fixed in 0.6.16.5 already.
Windows: Fix, onefile binaries were not working after being signed. This now works.
Standalone: Added missing implicit dependency for sklearn.
Compatibility: Modules giving SyntaxError from source were not properly handled, giving run time ImportError. Now they are giving SyntaxError.
Fix, the LTO mode has issues with incbin usage on older gcc, so use linker mode when it is enabled.
Python3: Fix, locals dict codes were not properly checking errors that the mapping might raise when setting values.
Fix, modules named entry were causing compile time errors in the C stage.
macOS: Never include files from OS private frameworks in standalone mode.
Fix, the python flag --python-flag=no_warning wasn’t working on all platforms.
Compatibility: Fix, the main code of the site module wasn’t executing, so that its added builtins were not there. Of course, you ought to use --python-flag=no_site to not have it in the normal case.
Python2: Added code path to handle edited standard library source code which then has no valid bytecode file.
Anaconda: In module mode, the CondaCC wasn’t recognized as form of gcc.
Fix, bytecode modules could shadow compiled modules of the same name.
Onefile: Fix, expansion of %PID% wasn’t working properly on non-Windows, making temp paths less unique. The time stamp is not necessarily enough.
Fix, multiprocessing error exits from slave processes were not reporting tracebacks.
Standalone: Added xcbglintegrations to the list of sensible Qt plugins to include by default, otherwise rendering will be inferior.
Standalone: Added platformthemes to the list of sensible Qt plugins to include by default, otherwise file dialogs on non-Windows would be inferior.
Fix, created .pyi files were not ordered deterministically.
Standalone: Added support for win32file.
Fix, namespace packages were not using run time values for their __path__ value.
Python3.7+: Fix, was leaking AttributeError exceptions during name imports.
Fix, standard library detection could fail for relative paths.

New Features

Added experimental support for C level PGO (Profile Guided Optimization), which runs your program and then uses feedback from the execution. At this time only gcc is supported, and only C compiler is collecting feedback. Check the User Manual for a table with current results.
macOS: Added experimental support for creating application bundles. For these, icons can be specified and console can be disabled. But at this time, onefile and accelerated mode are not yet usable with it, only standalone mode works.
Plugins: Add support for pkg_resources.require calls to be resolved at compile time. These are not working at run time, but this avoids the issue very nicely.
Plugins: Massive improvements to the anti-bloat plugin, it can now make numpy, scipy, skimage, pywt, and matplotlib use much less packages and has better error handling.
Plugins: Added anti-bloat ability ability to append code to a module, which might get used in the future by other plugins that need some sort of post load changes to be applied.
Plugins: Added ability to replace code of functions at parse time, and use this in anti-bloat plugin to replace functions that do unnecessary stuff with variants that often just do nothing. This is illustrated here.
```
gevent._util:
description: "remove gevent release framework"
change_function:
   "prereleaser_middle": "'(lambda data: None)'"
   "postreleaser_before": "'(lambda data: None)'"
```
This example is removing gevent code that loads dependencies used for their CI release process, that need not be part of normal programs.
Added ability to persist source code changes done by plugins in the Python installation. This is considered experimental and needs write access to the Python installation, so this is best done in a Virtualenv and it may confuse plugins.
Added support for multiprocessing.tracker and spawn mode for all platforms. For non-default modes outside of Windows, you need to --enable-plugin=multiprocessing to use these.
Plugins: Allow multiple entry points to be provided by one or several plugins for the same modules. These are now merged into one automatically.
Standalone: Fix for numpy not working when compiling with --python-flag=no_docstrings.
Fix, method calls were not respecting descriptors provided by types with non-generic attribute lookups.
Windows: Add support for using self-compiled Python3 from the build folder too.
Added support for Nuitka-Python 2.7, which will be our faster Python fork.
Colorized output for error outputs encountered in Scons, these are now yellow for better recognition.

Optimization

Faster threading code was used for Python3.8 or higher, and this has been extended to 3.7 on Windows, but we won’t be able to have it other platforms and not on earlier Python3 versions.
Faster calls esp. with keyword arguments. Call with keywords no longer create dictionaries if the call target supports that, and with 3.8 or higher, non-compiled code that allows vectorcall is taken advantage of.
Faster class creation that avoids creation of argument tuples and dictionaries.
Faster attribute check code in case of non-present attributes.
Faster unbound method calls, unlike bound methods calls these were not optimized as well yet.

Type shapes for star arguments are now known and used in optimization.

def f(*args, **kwargs):
   type(args) # Statically known to be tuple
   type(kwargs) # Statically known to be dict

Python2: Faster old-style class creation. These are classes that do not explicitly inherit from object.
Python2: Faster string comparisons for Python by specializing for the str type as well.
Python3: Added specialization for bytes comparisons too. These are naturally very much the same as str comparisons in Python2.
Added specialization for list comparisons too. We had them for tuples only so far.
Faster method calls when called from Python core, our tp_call slot wasn’t as good as it can be.
Optimization: Faster deep copies of constants. This can speed up constant calls with mutable types. Before it was checking the type too often to be fast.
Allow using static linking with Debian Python giving much better performance with the system Python. This is actually a huge improvement as it makes things much faster. So far it’s only automatically enabled for Python2, but it seems to work for Python3 on Debian too. Needs more tweaking in the future.
Optimization: Added functools module to the list of hard imports in preparation of optimizing functools.partial to work better with compiled functions.
Python2: Demote to xrange when iterating over range calls, even for small ranges, they are always faster. Previously this was only done for values with at least 256 values.
Enable LTO automatically for Debian Python, this also allows more optimization.
Enable LTO automatically for Anaconda with CondaCC on non-Windows, also allowing more optimization.

Organizational

Added section in the User Manual on how to deal with memory issues and C compiler bugs. This is a frequent topic and should serve as a pointer for this kind of issue.
The --lto option was changed to require an argument, so that it can also be disabled. The default is auto which is the old behavior where it’s enabled if possible.
Changed --no-progress to --no-progressbar in order to make it more clear what it’s about. Previously it was possible to relate it to --show-progress.
No longer require specific versions of dependencies in our requirements.txt and relegate those to only being in requirements-devel.txt such that by default Nuitka doesn’t collide with user requirements on those same packages which absolutely all the time don’t really make a difference.
Added ability to check all unpushed changes with pylint with a new ./bin/check-nuitka-with-pylint --unpushed option. Before it was only possible to make the check (quickly) with --diff, but that stopped working after commits are made.
Revived support for vmprof based analysis of compiled programs, but it requires a fork of it now.
Make Windows specific compiler options visible on all platforms. There is no point in them being errors, instead warnings are given when they are specified on non-Windows.
Added project variable Commercial for use in Nuitka project syntax.
Consistent use of metavars for nicer help output should make it more readable.
Avoid ast tree dumps in case of KeyboardInterrupt exceptions, they are just very noisy. Also not annotate where Nuitka was in optimization when a plugin is asking to sysexit.

Cleanups

Encoding names for UTF-8 in calls to .encode() were used inconsistent with and without dashes in the source code, added cleanup to auto-format that picks the one blessed.
Cleanup taking of run time traces of DLLs used in preparation for using it in main code eventually, moving it to a dedicated module.
Avoid special names for Nuitka options in test runner, this only adds a level of confusion. Needs more work in future release.
Unify implementation to create modules into single function. We had 3 forms, one in recursion, one for main module, and one for plugin generated code. This makes it much easier to understand and use in plugins.
Further reduced code duplication between the two Scons files, but more work will be needed there.
Escaped variables are still known to be assigned/unassigned rather than unknown, allowing for many optimizations to still work on them., esp. for immutable value
Enhanced auto-format for rest documents, bullet list spacing is now consistent and spelling of organizational is unified automatically.
Moved icon conversion functionality to separate module, so it can be reused for other platforms more easily.

Tests

Removed reflected test, because of Nuitka special needs to restart with variable Python flags. This could be reverted though, since Nuitka no longer needs anything outside inline copies, and therefore no longer loads from site packages.
Use anti-bloat plugin in standalone tests of Numpy, Pandas and tests to reduce their compile times, these have become much more manageable now.
Enhanced checks for used files to use proper below path checks for their ignoring.
Remove reflected test, compiling Nuitka with Nuitka has gotten too difficult.
Verify constants integrity at program end in debug mode again, so we catch corruption of them in tests.

Summary

This release is one of the most important ones in a long time. The PGO and LTO, and static libpython work make a big different for performance of created binaries.

The amount of optimization added is also huge, calls are much faster now, and object creations too. These avoiding to go through actual dictionaries and tuples in most cases when compiled code interacts gives very significant gains. This can be seen in the increase of pystone performance.

The new type specializations allow many operations to be much faster. More work will follow in this area and important types, str and int do not have specialized comparisons for Python3, holding it back somewhat to where our Python2 performance is for these things.

For scalability, the anti-bloat work is extremely valuable, and this plugin should become active by default in the future, for now it must be strongly recommended. It needs more control over what parts you want to deactivate from it, in case of it causing problems, then we can and should do it.

The support for macOS has been enhanced a lot, and will become perfect in the next release (currently develop). The bundle mode is needed for all kinds of GUI programs to not need a console. This platform is becoming as well supported as the others now.

Generally this release marks a huge step forward. We hope to add Python level PGO in the coming releases, for type knowledge retrofitted without any annotations used. Benchmarks will become more fun clearly.

Nuitka Release 0.6.16

2021-09-09T11:01:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly polishing and new features. Optimization looked only at threading performance, and LTO improvements on Windows.

Bug Fixes

Fix, the pkg-resources failed to resolve versions for importlib.metadata from its standard library at compile time. Fixed in 0.6.15.1 already.
Standalone: Fix, --include-module was not including the module if it was an extension modules, but only for Python modules. Fixed in 0.6.15.1 already.
Standalone: Added missing implicit dependencies for gi.overrides. Fixed in 0.6.15.1 already.
Python3.9: Fix, could crash when using generic aliases in certain configurations. Fixed in 0.6.15.2 already.
Fix, the tensorflow plugin needed an update due to changed API. Fixed in 0.6.15.3 already.
When error exiting Nuitka, it now closes any open progress bar for cleaner display.
Standalone: Added missing dependency for skimage.
Standalone: The numpy plugin now automatically includes Qt backend if any of the Qt binding plugins is active.

New Features

Python3.5+: Added support for onefile compression. This is using zstd which is known to give very good compression with very high decompression, much better than e.g. zlib.
macOS: Added onefile support.
FreeBSD: Added onefile support.
Linux: Added method to use tempdir onefile support as used on other platforms as an alternative to AppImage based.
Added support for recursive addition of files from directories with patterns.
Attaching the payload to onefile now has a progress bar too.
Windows: Prelimary support for the yet unfinished Nuitka-Python that allows static linking and higher performance on Windows, esp. with Nuitka.
Windows: In acceleration mode, for uninstalled Python, now a CMD file is created rather than copying the DLL to the binary directory. That avoids conflicts with architectures and of course useless file copies.
New abilities for plugin anti-bloat allow to make it an error when certain modules are imported. Added more specific options for usual trouble makes, esp. setuptools, pytest are causing an explosion for some programs, while being unused code. This makes it now easier to oversee this.
It’s now possible to override appdirs decision for where cache files live with an environment variable NUITKA_CACHE_DIR.
The -o option now also works with onefile mode, it previously rejected anything but acceleration mode. Fixed in 0.6.15.3 already.
Plugins: It’s now possible for multiple plugins to provide pre or post load code for the same module.
Added indications for compilation modes standalone and onefile to the __compiled__ attribute.
Plugins: Give nicer error message in case of colliding command line options.

Optimization

Faster threading code is now using for Python3.8 or higher and not only 3.9, giving a performance boost, esp. on Windows.
Using --lto is now the default with MSVC 2019 or higher. This will given smaller and faster binaries. It has been available for some time, but not been the default yet.

Cleanups

Using different progress bar titles for C compilation of Python code and C compilation of onefile bootstrap.
Moved platform specific detections, for FreeBSD/OpenBSD/macOS out of the Scons file and to common Nuitka code, sometimes eliminating duplications with one version being more correct than the other.
Massive cleanup of datafile plugin, using pattern descriptions, so more code duplication can be removed.
More cleanup of the scons files, sharing more common code.

Organizational

Under the name Nuitka-Python we are now also developing a fork of CPython with enhancements, you can follow and joint it at https://github.com/Nuitka/Nuitka-Python but at this time it is not yet ready for prime time.
Onefile under Windows now only is temporary file mode. Until we figure out how to solve the problems with locking and caching, the mode where it installs to the AppData of the user is no longer available.
Renamed the plugin responsible for PyQt5 support to match the names of others. Note however, that at this time, PySide2 or PySide6 are to be recommended.
Make it clear that PySide 6.1.2 is actually going to be the supported version of PySide6.
Use MSVC in GitHub actions.

Summary

This release had a massive focus on expanding existing features, esp. for onefile, and plugins API, such that we can now configure anti-bloat with yaml, have really nice datafile handling options, and have onefile on all OSes practically.

Nuitka Release 0.6.15

2021-06-05T17:18:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release polished previous work with bug fixes, but there are also important new things that help make Nuitka more usable, with one important performance improvement.

Bug Fixes

Fix, hard imports were not automatically used in code generation leading to errors when used. Fixed in 0.6.14.2 already.
Windows: Fix, clcache was disabled by mistake. Fixed in 0.6.14.2 already.
Standalone: Added data files for jsonschema to be copied automatically.
Standalone: Support for pendulum wasn’t working anymore with the last release due to plugin interface issues.
Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already.
Fix, the ccache path wasn’t working if it contained spaces. Fixed in 0.6.14.5 already.
Onefile: For Linux and ARM using proper download off appimage. Fixed in 0.6.14.5 already.
Standalone: Added support for pyreadstat. Fixed in 0.6.14.5 already.
Standalone: Added missing dependencies for pandas. Fixed in 0.6.14.6 already.
Standalone: Some preloaded packages from .pth do not have a __path__, these can and must be ignored.
Onefile: On Linux, the sys.argv[0] was not the original file as advertised.
Standalone: Do not consider .mesh and .frag files as DLls in the Qt bindings when including the qml support. This was causing errors on Linux, but was generally wasteful.
Fix, project options could be injected twice, which could lead to errors with options that were only allowed once, e.g. --linux-onefile-icon.
Windows: When updating the resources in created binaries, treat all kinds of OSError with information output.
Onefile: Remove onefile target binary path at startup as well, so it cannot cause confusion after error exit.
Onefile: In case of error exit from AppImage, preserve its outputs and attempt to detect if there was a locking issue.
Standalone: Scan package folders on Linux for DLLs too. This is necessary to properly handle PyQt5 in case of Qt installed in the system as well.
Standalone: On Linux, standard QML files were not found.
Standalone: Enforce C locale when detecting DLLs on Linux, otherwise whitelisting messages didn’t work properly on newer Linux.
Standalone: Added support for tzdata package data files.
Standalone: Added support for exchangelib.
Python3.9: Fix, type subscripts could cause optimization errors.
UI: Project options didn’t properly handle quoting of arguments, these are normally removed by the shell.
Linux: The default icon for Python in the system is now found with more version specific names and should work on more systems.
Standalone: Do not include libstdc++ as it should come from the system rather.

New Features

Added plugin anti-bloat plugin, intended to fight bloat. For now it can make including certain modules an error, a warning, or force ImportError, e.g. --noinclude-setuptools-mode=nofollow is very much recommended to limit compilation size.
The pkg-resources builtin now covers metadata and importlib_metadata packages for compile time version resolution as well.
Added support for PySide2 on Python version before 3.6, because the patched code needs no workarounds. Fixed in 0.6.14.3 already.
Windows: Convert images to icon files on the fly. So now you can specify multiple PNG files, and Nuitka will create an icon out of that automatically.
macOS: Automatically download ccache binary if not present.
Plugins: New interface to query the main script path. This allows plugins to look at its directory.
UI: Output the versions of Nuitka and Python during compilation.
UI: Added option to control static linking. So far this had been enabled only automatically for cases where we are certain, but this allows to force enable or disable it. Now an info is given, if Nuitka thinks it might be possible to enable it, but doesn’t do it automatically.
UI: Added --no-onefile to disable --onefile from project options.

Optimization

Much enhanced GIL interaction with Python3.9 giving a big speed boost and better threading behaviour.
Faster conversion of iterables to list, if size can be know, allocation ahead saves a lot of effort.
Added support for GenericAlias objects as compile time constants.

Organizational

Enhanced GitHub issue raising instructions.
Apply rstfmt to all documentation and make it part of the commit hook.
Make sure to check Scons files as well. This would have caught the code used to disable clcache temporarily.
Do not mention Travis in PR template anymore, we have stopped using it.
Updated requirements to latest versions.
Bump requirements for development to 3.7 at least, toosl like black do not work with 3.6 anymore.
Started work on Nuitka Python, a CPython fork intended for enhanced performance and standalone support with Nuitka.

Cleanups

Determine system prefix without virtualenv outside of Scons, such that plugins can share the code. There was duplication with the numpy plugin, and this will only be more complete using all approaches. This also removes a lot of noise from the scons file moving it to shared code.
The Qt plugins now collect QML files with cleaner code.

Tests

Nicer error message if a wrong search mode is given.
Windows: Added timeout for determining run time traces with dependency walker, sometimes this hangs.
Added test to cover the zip importer.
Making use of project options in onefile tests, making it easier to execute them manually.

Summary

For Windows, it’s now easier than ever to create an icon for your deployment, because you can use PNG files, and need not produce ICO files anymore, with Nuitka doing that for you.

The onefile for Linux had some more or less severe problems that got addressed, esp. also when it came to QML applications with PySide.

On the side, we are preparing to greatly improve the caching of Nuitka, starting with retaining modules that were demoted to bytecode. There are changes in this release, to support that, but it’s not yet complete. We expect that scalability will then be possible to improve even further.

Generally this is mostly a maintenance release, which outside of the threading performance improvement has very little to offer for faster execution, but that actually does a lot. Unfortunately right now it’s limited to 3.9, but some of the newer Python’s will also be supported in later releases.

Nuitka Release 0.6.14

2021-05-01T09:19:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has few, but important bug fixes. The main focus was on expanding standalone support, esp. for PySide2, but also and in general with plugins added that workaround pkg_resources usage for version information.

Also an important new features was added, e.g. the project configuration in the main file should prove to be very useful.

Bug Fixes

Compatibility: Fix, modules that failed to import, should be retried on next import.

So far we only ever executed the module body once, but that is not how it’s supposed to be. Instead, only if it’s in sys.modules that should happen, which is the case after successful import.
Compatibility: Fix, constant False values in right hand side of and/or conditions were generating wrong code if the left side was of known bool shape too.
Standalone: Fix, add styles Qt plugins to list of sensible plugins.

Otherwise no mouse hover events are generated on some platforms.
Compatibility: Fix, relative from imports beyond level 1 were not loadingg modules from packages if necessary. Fixed in 0.6.13.3 already.
Standalone: The crypto DLL check for Qt bindings was wrong. Fixed in 0.6.13.2 already.
Standalone: Added experimental support for PySide6, but for good results, 6.1 will be needed.
Standalone: Added support for newer matplotlib. Fixed in 0.6.12.1 already.
Standalone: Reverted changes related to pkg_resources that were causing regressions. Fixed in 0.6.13.1 already.
Standalone: Adding missing implicit dependency for cytoolz package. Fixed in 0.6.13.1 already.
Standalone: Matching for package names to not suggest recompile for was broken and didn’t match. Fixed in 0.6.13.1 already.

New Features

Added support for project options.

When found in the filename provided, Nuitka will inject options to the commandline, such that it becomes possible to do a complex project with only using

python -m nuitka filename.py

# Compilation mode, support OS specific.
# nuitka-project-if: {OS} in ("Windows", "Linux"):
#    nuitka-project: --onefile
# nuitka-project-if: {OS} not in ("Windows", "Linux"):
#    nuitka-project: --standalone

# The PySide2 plugin covers qt-plugins
# nuitka-project: --enable-plugin=pyside2
# nuitka-project: --include-qt-plugins=sensible,qml

# The pkg-resources plugin is not yet automatic
# nuitka-project: --enable-plugin=pkg-resources

# Nuitka Commercial only features follow:

# Protect the constants from being readable.
# nuitka-project: --enable-plugin=data-hiding

# Include datafiles for Qt into the binary directory.
# nuitka-project: --enable-plugin=datafile-inclusion
# nuitka-project: --qt-datadir={MAIN_DIRECTORY}
# nuitka-project: --qt-datafile-pattern=*.js
# nuitka-project: --qt-datafile-pattern=*.qml
# nuitka-project: --qt-datafile-pattern=*.svg
# nuitka-project: --qt-datafile-pattern=*.png

Refer to the User Manual for a table of directives and the variables allowed to be used.

Added option to include whole data directory structures in standalone.

The new option --include-data-dir was added and is mostly required for onefile mode, but recommended for standalone too.
Added pkg-resources plugin.

This one can resolve code like this at compile time without any need for pip metadata to be present or used.
```
pkg_resources.get_distribution("module_name").version
pkg_resources.get_distribution("module_name").parsed_version
```
Standalone: Also process early imports in optimization.

Otherwise plugins cannot work on standard library modules. This makes it possible to handle them as well.

Optimization

Faster binary operations.

Applying lessons learnt during the enhancements for in-place operations that initially gave worse results than some manual code, we apply the same tricks for all binary operations, which speeds them up by significant margins, e.g. 30% for float addition, 25% for Python int addition, and still 6% for Python int addition.
More direct optimization of unary operations on constant value.

Without this, -1 was not directly a constant value, but had to go through the unary - operation, which it still does, but now it’s done at tree building time.
More direct optimization for not in branches.

Invertible comparisons, i.e. is/is not and in/not in do not have do be done during optimization. This mainly avoids noise during optimization from such unimportant steps.
More direct optimization for constant slices.

These are used in Python3 for all subscripts, e.g. a[1:2] will use slice(1,2) effectively. For Python2 they are used less often, but still. This also avoids a lot of noise during optimization, mostly on Python3
Scons: Avoid writing database to disk entirely.

This saves a bit of disk churn and makes it unnecessary to specify the location such that it doesn’t collide between Python versions.
For optimization passes, use previous max total as minimum for next pass. That will usually be a more accurate result, rather than starting from 1 again. Part of 0.6.13.1 already.
Enhancements to the branch merging improve the scalability of Nuitka somewhat, although the merging itself is still not very scalable, there are some modules that are very slow to optimize still.
Use orderset if available over the inline copy for OrderedSet which is much faster and improves Nuitka compile times.
Make pkgutil a hard import too, this is in preparation of more optimization for its functions.

Organizational

Upstream patches for PySide6 have been contributed and merged into the development branch dev. Full support should be available once this is released as part of 6.1 which is waiting for Qt 6.1 naturally.
Patches for PySide2 are available to commercial customers, see PySide2 support page.
Formatted all documents with rstfmt and made that part of the commit hook for Nuitka. It now works for all documents we have.
Updated inline copy of tqdm to 4.59.0 which ought to address spurious errors given.
User Manual: Remove --show-progress from the tutoral. The default progress bar is then disabled, and is actually much nicer to use.
Developer Manual: Added description of how context managers should be named.
Cleanup language for some warnings and outputs.

It was still using obsolete “recursion” language rather than talking about “following imports”, which is the new one.

Cleanups

Remove dead code related to constants marshal, the data composer has replaced this.
Avoid internal API usage for loading extension modules on Linux, there is a function in sys module to get the ld flags.

Tests

Fix, the only mode wasn’t working properly.
Use new project options feature for specific options in basic tests allowing to remove them from the test runner.

Summary

For PySide2 things became more perfect, but it takes upstream patches unfortunately such that only PySide6.1 will be working out of the box outside of the commercial offering. We will also attempt to provide workarounds, but there are some things that cannot be done that way.

This release added some more scalability to the optimization process, however there will be more work needed to make efficient branch merges.

For onefile, a feature to include whole directories had been missing, and could not easily be achieved with the existing options. This further rounds this up, now what’s considered missing is compression and macOS support, both of which should be coming in a future release.

For the performance side of things, the binary operator work can actually yield pretty good gains, with double digit improvements, but this covers only so much. Much more C types and better type tracing would be needed, but there was no progress on this front. Future releases will have to revisit the type tracing to make sure, we know more about loop variables, etc. so we can achieve the near C speed we are looking for, at least in the field of int performance.

This release has largely been driven by the Nuitka Commercial offering and needs for compatibility with more code, which is of course always a good thing.

Nuitka Release 0.6.13

2021-04-03T10:05:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release follows up with yet again massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

Windows: Icon group entries were not still not working properly in some cases, leading to no icon or too small icons being displayed. Fixed in 0.6.12.2 already.
Windows: Icons and version information were copied from the standalone executable to the onefile executable, but that failed due to race situations, sometimes reproducible. Instead we now apply things to both independently. Fixed in 0.6.12.2 already.
Standalone: Fixup scanning for DLLs with ldconfig on Linux and newer versions making unexpected outputs. Fixed in 0.6.12.2 already.
UI: When there is no standard input provided, prompts were crashing with EOFError when --assume-yes-for-downloads is not given, change that to defaulting to "no" instead. Fixed in 0.6.12.2 already.
Windows: Detect empty strings for company name, product name, product and file versions rather than crashing on them later. Them being empty rather than not there can cause a lot of issues in other places. Fixed in 0.6.12.2 already.
Scons: Pass on exceptions during execution in worker threads and abort compilation immediately. Fixed in 0.6.12.2 already.
Python3.9: Still not fully compatible with typing subclasses, the enhanced check is now closely matching the CPython code. Fixed in 0.6.12.2 already.
Linux: Nicer error message for missing libfuse requirement.
Compatibility: Lookups on dictionaries with None value giving a KeyError exception, but with no value, which is not what CPython does.
Python3.9: Fix, future annotations were crashing in debug mode. Fixed in 0.6.12.3 already.
Standalone: Corrections to the glfw were made. Fixed in 0.6.12.3 already.
Standalone: Added missing ìmplicit dependency for py.test. Fixed in 0.6.12.3 already.
Standalone: Adding missing implicit dependency for pyreadstat.
Windows: Added workaround for common clcache locking problems. Since we use it only inside a single Scons process, we can avoiding using Windows mutexes, and use a process level lock instead.
Plugins: Fix plugin for support for eventlet. Fixed in 0.6.12.3 already.
Standalone: Added support for latest zmq on Windows.
Scons: the --quiet flag was not fully honored yet, with Scons still making a few outputs.
Standalone: Added support for alternative DLL name for newer PyGTK3 on Windows. Fixed in 0.6.12.4 already.
Plugins: Fix plugin for support for gevent. Fixed in 0.6.12.4 already.
Standalone: Added yet another missing implicit dependency for pandas.
Plugins: Fix, the qt-plugins plugin could stumble over .mesh files.
Windows: Fix, dependency walker wasn’t properly working with unicode %PATH% which could e.g. happen with a virtualenv in a home directory that requires them.
Python3: Fixed a few Python debug mode warnings about unclosed files that have sneaked into the codebase.

New Features

Added new options --windows-force-stdout-spec and --windows-force-stderr-spec to force output to files. The paths provided at compile time can resolve symbolic paths, and are intended to e.g. place these files near the executable. Check the User Manual for a table of the currently supported values. At this time, the feature is limited to Windows, where the need arose first, but it will be ported to other supported OSes eventually. These are most useful for programs run as --windows-disable-console or with --enable-plugin=windows-service.

Note

These options have since been renamed to --force-stdout and --force-stderr and have been made to work on all OSes.
Windows: Added option --windows-onefile-tempdir-spec (since renamed to --onefile-tempdir-spec) to provide the temporary directory used with --windows-onefile-tempdir in onefile mode, you can now select your own pattern, and e.g. hardcode a base directory of your choice rather than %TEMP.
Added experimental support for PySide2 with workarounds for compiled methods not being accepted by its core. There are known issues with PySide2 still, but it’s working fine for some people now. Upstream patches will have to be created to remove the need for workarounds and full support.

Optimization

Use binary operation code for their in-place variants too, giving substantial performance improvements in all cases that were not dealt with manually already, but were covered in standard binary operations. Until now only some string, some float operations were caught sped up, most often due to findings of Nuitka being terribly slower, e.g. not reusing string memory for inplace concatenation, but now all operations have code that avoids a generic code path, that is also very slow on Windows due calling to using the embedded Python via API being slow.
For mixed type operations, there was only one direction provided, which caused fallbacks to slower forms, e.g. with long and float values leading to inconsistent results, such that a - 1 and 1 - a being accelerated or not.
Added C boolean optimization for a few operations that didn’t have it, as these allow to avoid doing full computation of what the object result would have to do. This is not exhausted fully yet.
Python3: Faster +/-/+=/-= binary and in-place operations with int values providing specialized code helpers that are much faster, esp. in cases where no new storage is allocated for in-place results and where not a lot of digits are involved.
Python2: The Python3 int code is the Python2 long type and benefits from the optimization of +/-/+=/-= code as well, but of course its use is relatively rare.
Improved __future__ imports to become hard imports, so more efficient code is generated for them.
Counting of instances had a run time impact by providing a __del__ that was still needed to be executed and limits garbage collection on types with older Python versions.
UI: Avoid loading tqdm module before it’s actually used if at all (it may get disabled by the user), speeding up the start of Nuitka.
Make sure to optimize internal helpers only once and immediately, avoiding extra global passes that were slowing down Python level compilation by of large programs by a lot.
Make sure to recognize the case where a module optimization can provide no immediate change, but only after a next run, avoiding extra global passes originating from these, that were slowing down compilation of large programs by a lot. Together with the other change, this can improve scalability by a lot.
Plugins: Remove implicit dependencies for pkg_resources.extern and use aliases instead. Using one of the packages, was causing all that might be used, to be considered as used, with some being relatively large. This was kind of a mistake in how we supported this so far.
Plugins: Revamped the eventlet plugin, include needed DNS modules as bytecode rather than as source code, scanning them with pkgutil rather than filesystem, with much cleaner code in the plugin. The plugin is also now enabled by default.

Organizational

Removed support for pefile dependency walker choice and inline copy of the code. It was never as good giving incomplete results, and after later improvements, slower, and therefore has lost the original benefit over using Dependency Walker that is faster and more correct.
Added example for onefile on Windows with the version information and with the temporary directory mode.
Describe difference in file access with onefile on Windows, where sys.argv[0] and os.path.dirname(__file__) will be different things.
Added inline copy of tqdm to make sure it’s available for progress bar output for 2.7 or higher. Recommend having it in the Debian package.
Added inline copy of colorama for use on Windows, where on some terminals it will give better results with the progress bar.
Stop using old PyLint for Python2, while it would be nice to catch errors, the burden of false alarms seems to high now.
UI: Added even more checks on options that make no sense, made sure to do this only after a possible restart in proper environment, so warnings are not duplicated.
For Linux onefile, keep appimage outputs in case of an error, that should help debugging it in case of issues.
UI: Added traces for plugin provided implicit dependencies leading to inclusions.
Added inline copy of zstd C code for use in decompression for the Windows onefile bootstrap, not yet used though.
Added checks to options that accept package names for obvious mistakes, such that --include-package-data --mingw64 will not swallow an option, as that is clearly not a package name, that would hide that option, while also not having any intended effect.
Added ignore list for decision to recompile extension modules with available source too. For now, Nuitka will not propose to recompile Cython modules that are very likely not used by the program anyway, and also not for lxml until it’s clear if there’s any benefit in that. More will be added in the future, this is mostly for cases, where Cython causes incompatibilities.
Added support for using abstract base classes in plugins. These are not considered for loading, and allow nicer implementation of shared code, e.g. between PyQt5 and PySide2 plugins, but allow e.g. to enforce the provision of certain overloads.
User Manual: Remove the instruction to install clcache, since it’s an inline copy, this makes no sense anymore and that was obsolete.
Updated PyLint to latest versions, and our requirements in general.

Cleanups

Started removal of PyLint annotations used for old Python2 only. This will be a continuous action to remove these.
Jinja2 based static code generation for operations was cleaned up, to avoid code for static mismatches in the result C, avoiding language constructs like if (1 && 0) with sometimes larger branches, replacing it with Jinja2 branches of the {% if ... %} form.
Jinja2 based Python2 int code was pioniering the use of macros, but this was expanded to allow kinds of types for binary operations, allow their reuse for in-place operation, with these macros making returns via goto exits rather than return statements in a function. Landing pads for these exits can then assign target values for in-place different from what those for binary operation result return do.
Changed the interfacing of plugins with DLL dependency detection, cleaning up the interactions considerably with more unified code, and faster executing due to cached plugin decisions.
Integrate manually provided slot function for unicode and str into the standard static code generation. Previously parts were generated and parts could be generated, but also provided with manual code. The later is now all gone.
Use a less verbose progress bar format with less useless infos, making it less likely to overflow.
Cleanup how payload data is accessed in Windows onefile bootstrap, preparing the addition of decompression, doing the reading from the file in only one dedicated function.
When Jinja2 generated exceptions in the static code, it is now done via proper Jinja2 macros rather than Python code, and these now avoid useless Python version branches where possible, e.g. because a type like bytes is already Python version specific, with the goal to get rid of PyErr_Format usage in our generated static code. That goal is future work though.
Move safe strings helpers (cannot overflow) to a dedicated file, and remove the partial duplication on the Windows onefile bootstrap code.
The Jinja2 static code generation was enhanced to track the usage of labels used as goto targets, so that error exits, and value typed exits from operations code no longer emitted when not used, and therefore labels that are not used are not present.
For implicit dependencies, the parsing of the .pyi file of a module no longer emits a dependency on the module itself. Also from plugins, these are now filtered away.

Tests

Detect if onefile mode has required downloads and if there is user consent, otherwise skip onefile tests in the test runner.
Need to also allow accesses to files via short paths on Windows if these are allowed long paths.
The standalone tests on Windows didn’t actually take run time traces and therefore were ineffective.
Added standalone test for glfw coverage.
construct-based tests for in-place operations are now using a value for the first time, and then a couple more times, allowing for real in-place usage, so we are sure we measure correctly if that’s happening.

Summary

Where the big change of the last release were optimization changes to reduce the global passes, this release addresses remaining causes for extra passes, that could cause these to still happen. That makes sure, Nuitka scalability is very much enhanced in this field again.

The new features for forced outputs are at this time Windows only and make a huge difference when it comes to providing a way to debug Windows Services or programs in general without a console, i.e. a GUI program. These will need even more specifiers, e.g. to cover program directory, rather than exe filename only, but it’s a very good start.

On the tooling side, not a lot has happened, with the clcache fix, it seems that locking issues on Windows are gone.

The plugin changes from previous releases had left a few of them in a state where they were not working, but this should be restored. Interaction with the plugins is being refined constantly, and this releases improved again on their interfaces. It will be a while until this becomes stable.

Adding support for PySide2 is a headline feature actually, but not as perfect as we are used to in other fields. More work will be needed, also in part with upstream changes, to get this to be fully supported.

For the performance side of things, the in-place work and the binary operations work has taken proof of concept stuff done for Python2 and applied it more universally to Python3. Until we cover all long operations, esp. * seems extremely important and is lacking, this cannot be considered complete, but it gives amazing speedups in some cases now.

Future releases will revisit the type tracing to make sure, we know more about loop variables, to apply specific code helpers more often, so we can achieve the near C speed we are looking for in the field of int performance.

Nuitka Release 0.6.12

2021-02-16T09:24:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is yet again a massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

Windows: Icon group entries were not working properly in some cases, leading to no icon or too small icons being displayed.
Standalone: The PyQt implicit dependencies were broken. Fixed in 0.6.11.1 already.
Standalone: The datafile collector plugin was broken. Fixed in 0.6.11.3 already.
Standalone: Added support for newer forms of matplotlib which need a different file layout and config file format. Fixed in 0.6.11.1 already.
Plugins: If there was an error during loading of the module or plugin, it could still be attempted for use. Fixed in 0.6.11.1 already.
Disable notes given by gcc, these were treated as errors. Fixed in 0.6.11.1 already.
Windows: Fix, spaces in gcc installation paths were not working. Partially fixed in 0.6.11.4 already.
Linux: Fix, missing onefile icon error message was not complete. Fixed in 0.6.11.4 already.
Standalone: Workaround zmq problem on Windows by duplicating a DLL in both expected places. Fixed in 0.6.11.4 already.
Standalone: The dill-compat plugin wasn’t working anymore. Fixed in 0.6.11.4 already.
Windows: Fix mistaken usage of sizeof for wide character buffers. This caused Windows onefile mode in temporary directory. Fixed in 0.6.11.4 already.
Windows: Fix, checking subfolder natured crashed with different drives on Windows. Fixed in 0.6.11.4 already.
Windows: Fix, usage from MSVC prompt was no longer working, detect used SDK properly. Fixed in 0.6.11.4 already.
Windows: Fix, old clcache installation uses pth files that prevented our inline copy from working, workaround was added.
Windows: Also specify stack size to be used when compiling with gcc or clang.
Fix, claim of Python 3.9 support also in PyPI metadata was missing. Fixed in 0.6.11.5 already.
Python3.9: Subscripting type for annotations wasn’t yet implemented.
Python3.9: Better matching of types for metaclass selection, generic aliases were not yet working, breaking some forms of type annotations in base classes.
Windows: Allow selecting --msvc-version when a MSVC prompt is currently activated.
Windows: Do not fallback to using gcc when --msvc-version has been specified. Instead it’s an error if that fails to work.

Python3.6+: Added support for del () statements, these have no effect, but were crashing Nuitka.

del a  # standard form
del a, b  # same as del a; del b
del (a, b)  # braces are allowed
del ()  # allowed for consistency, but wasn't working.

Standalone: Added support for glfw through a dedicated plugin.
macOS: Added support for Python3 from system and CPython official download for latest OS version.

New Features

UI: With tqdm installed alongside Nuitka, experimental progress bars are enabled. Do not use --show-progress or --verbose as these might have to disable it.
Plugins: Added APIs for final processing of the result and onefile post-processing.
Onefile: On Windows, the Python process terminates with KeyboardInterrupt when the user sends CTRL-break, CTRL-C, shutdown or logoff signals.
Onefile: On Windows, in case of the launching process terminating unexpectedly, e.g. due to Taskmanager killing it, or a os.sigkill resulting in that, the Python process still terminates with KeyboardInterrupt.
Windows: Now can select icons by index from files with multiple icons.

Optimization

Avoid global passes caused by module specific optimization. The variable completeness os now traced per module and function scope, allowing a sooner usage. Unused temporary variables and closure variables are remove immediately. Recognizing possible auto releases of parameter variables is also instantly.

This should bring down current passes from 5-6 global passes to only 2 global passes in the normal case, reducing frontend compile times in some cases massively.
Better unary node handling. Dedicated nodes per operation allow for more compact memory usage and faster optimization.
Detect flow control and value escape for the repr of node based on type shape.
Enhanced optimization of caught exception references, these never raise or have escapes of control flow.
Exception matching operations are more accurately annotated, and may be recognized to not raise in more cases.
Added optimization for the issubclass built-in.
Removed scons caching as used on Windows entirely. We should either be using clcache or ccache automatically now.
Make sure to use __slots__ for all node classes. In some cases, mixins were preventing the feature from being it. We now enforce their correct specification of slots, which makes sure we can’t miss it anymore. This should again gain more speed and save memory at frontend compile time.
Scons: Enhanced gcc version detection with improved caching behavior, this avoids querying the same gcc binary twice.

Organizational

The description of Nuitka on PyPI was absent for a while. Added back by adding long description of the project derived from the README file.
Avoid terms blacklist, whilelist and slave in the Nuitka code preferring blocklist, ignorelist and child instead, which are actually more clear anyway. We follow a general trend to do this.
Configured the inline copy of Scons so pylance has an easier time to find it.
The git commit hook had stopped applying diffs with newest git, improved that.
Updated description for adding new CPython test suite.
Using https URLs for downloading dependency walker, for it to be more secure.
The commit hook can now be disabled, it’s in the Developer Manual how to do it.

Cleanups

Moved unary operations to their own module, the operators module was getting too crowded.
The scons files for Python C backend and Windows onefile got cleaned up some more and moved more common code to shared modules.
When calling external tools, make sure to provide null input where possible.
Unified calling install_name_tool into a single method for adding rpath and name changes both at the same time.
Unified how tools like readelf, ldconfig etc. are called and error exits and outputs checked to make sure we don’t miss anything as easily.

Tests

Adapted for some openSUSE specific path usages in standalone tests.
Basic tests for onefile operation and with termination signal sent were added.

Summary

The big changes in this release are the optimization changes to reduce the global passes and the memory savings from other optimization. These should again make Nuitka more scalable with large projects, but there definitely is work remaining.

Adding nice stopping behaviour for the Onefile mode on Windows is seemingly a first, and it wasn’t easy, but it will make it more reliable to users.

Also tooling of gcc and MSVC on Windows got a lot more robust, covering more cases, and macOS support has been renewed and should be a lot better now.

The progress bar is a nice touch and improves the overall feel of the compilation process, but obviously we need to aim at getting faster overall still. For projects using large dependencies, e.g. Pandas the compilation is still far too slow and that will need work on caching frontend results, and better optimization and C code generation for the backend.

Nuitka Release 0.6.11

2021-01-25T10:17:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is a massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

Fix, the .pyi file parser didn’t handle relative imports. Fixed in 0.6.10.1 already.
Windows: Fix, multiprocessing plugin was not working reliable following of imports from the additional entry point. Fixed in 0.6.10.1 already.
Pipenv: Workaround parsing issue with our setup.py to allow installation from GitHub. Fixed in 0.6.10.1 already.
Merging of branches in optimization could give nondeterministic results leading to more iterations than necessary. Fixed in 0.6.10.1 already.
Windows: Avoid profile powershell when attempting to resolve symlinks. Fixed in 0.6.10.1 already.
Windows: Fix, always check for stdin, stdout, and stderr presence. This was so far restricted to gui mode applications, but it seems to be necessary in other situations too. Fixed in 0.6.10.1 already.
Python2: Fix, --trace-execution was not working for standalone mode but can be useful for debugging. Fixed in 0.6.10.1 already.
Windows: Onefile could run into path length limits. Fixed in 0.6.10.3 already.
Windows: The winlib gcc download link became broken and was updated. Fixed in 0.6.10.3 already.
Plugins: The “__main__” module was not triggering all plugin hooks, but it needs to for completeness.
Standalone: Fix, symlinked Python installations on Windows were not working, with dependency walker being unable to look into these. Fixed in 0.6.10.4 already.
Standalone: Fix support for numpy on Windows and macOS, the plugin failed to copy important DLLs. Fixed in 0.6.10.4 already.
Python3: For versions before 3.7, the symlink resolution also needs to be done, but wasn’t handling the bytes output yet. Fixed in 0.6.10.4 already.
Fix, folder based inclusion would both pick up namespace folders and modules of the same name, crashing the compilation due to conflicts. Fixed in 0.6.10.4 already.
Fix, the --lto wasn’t used for clang on non-Windows yet.
Fix, the order of locals dict releases wasn’t enforced, which could lead to differences that break caching of C files potentially. Fixed in 0.6.10.5 already.
Fix, hash nodes didn’t consider if their argument was raising, even if the type of the argument was str and therefore the operation should not. Fixed in 0.6.10.5 already.
Fix, need to copy type shape and escape description for the replacement inverted comparisons when used with not, otherwise the compilation can crash as these are expected to be present at all times. Fixed in 0.6.10.5 already.
Fix, some complex constant values could be confused, e.g. -0j and 0j. These corner cases were not properly considered in the constant loading code, only for float so far.
Standalone: Fix, bytecode only standard library modules were not working. This is at least used with Fedora 33.
Linux: Fix, extension modules compiled with --lto were not working.
Windows: Retry if updating resources fails due to Virus checkers keeping files locked.
Plugins: Pre- and postload code of modules should not be allowed to cause ImportError, as these will be invisible to the other parts of optimization, instead make them unraisable error traces.
Standalone: Adding missing import for SciPy 1.6 support.
Windows: Fix, only export required symbols when using MinGW64 in module mode.

New Features

Python3.9: Added official support for this version.
Onefile: Added command line options to include data files. These are --include-package-data which will copy all non-DLLs and non-Python files of package names matching the pattern given. And --include-data-file takes source and relative target file paths and copies them. For onefile this is the only way to include files, for standalone mode they are mostly a convenience function.
Onefile: Added mode where the file is unpacked to a temporary folder before running instead of doing it to appdata.
Onefile: Added linux specific options --linux-onefile-icon to allow provision of an icon to use in onefile mode on Linux, so far this was only available as the hard coded path to a Python icon, which also didn’t exist on all platforms.
UI: Major logging cleanup. Everything is now using our tracing classes and even error exits go through there and are therefore colored if possible.
Plugins: Make it easier to integrate commercial plugins, now only an environment variable needs to point to them.
UI: Enhanced option parsing gives notes. This complains about options that conflict or that are implied in others. Trying to catch more usage errors sooner.
Plugins: Ignore exceptions in buggy plugin code, only warn about them unless in debug mode, where they still crash Nuitka.
Scons: More complete scons report files, includes list values as well and more modes used.
Windows: The clcache is now included and no longer used from the system.
Output for clcache and ccache results got improved.
Enhanced support for clang, on Windows if present near a gcc.exe like it is the case for some WinLibs downloads, it will be used. To use it provide --mingw64 --clang both. Without the first one, it will mean clangcl.exe which uses the MSVC compiler as a host.

Optimization

Some modules had very slow load times, e.g. if they used many list objects due to linear searches for memory deduplication of objects. We now have dictionaries of practically all constant objects loaded, making these more instant.
Use less memory at compile time due using __slots__ for all node types, finally figured out, how to achieve this with multiple inheritance.
Use hedley for compiler macros like unlikely as they know best how to do these.
Special case the merging of 2 branches avoiding generic code and being much faster.
Hard imports have better code generated, and are being optimized into for the few standard library modules and builtin modules we handle, they also now annotate the type shape to be module.
No longer annotate hard module import attribute lookups as control flow escapes. Not present attributes are changed into static raises. Trust for values is configured for a few values, and experimental.
Avoid preloaded packages for modules that have no side effects and are in the standard library, typically .pth files will use e.g. os but that’s not needed to be preserved.
Use incbin for including binary data through inline assembly of the C compiler. This covers many more platforms than our previous linker option hacks, and the fallback to generated C code. In fact everything but Windows uses this now.

Organizational

Windows: For Scons we now require a Python 3.5 or higher to be installed to use it.
Windows: Removed support for gcc older than version 8. This specifically affects CondaCC and older MinGW64 installations. Since Nuitka can now download the MinGW64 10, there is no point in having these and they cause issues.
We took over the maintenance of clcache as Nuitka/clcache which is not yet ready for public consumption, but should become the new source of clache in the future.
Include an inline copy of clcache in Nuitka and use it on Windows for MSVC and ClangCL.
Removed compatibility older aliases of follow option, --recurse-* and require --follow-* options to be used instead.
For pylint checking, the tool now supports a --diff mode where only the changed files get checked. This is much faster and allows to do it more often before commit.
Check the versions of isort and black when doing the auto-format to avoid using outdated versions.
Handling missing pylint more gracefully when checking source code quality.
Make sure to use the codespell tool with Python3 and make sure to error exit when spelling problems were found, so we can use this in GitHub actions too.
Removed Travis config, we now only use GitHub actions.
Removed landscape config, it doesn’t really exist anymore.
Bumped all PyPI dependnecies to their latest versions.
Recommend ccache on Debian, as we now consider the absence of ccache something to warn about.
Plugins: The DLLs asked for by plugins that are not found are no longer warned about.
Allow our checker and format tools to run on outside of tree code. We are using that for Nuitka/clcache.
Added support for Fedora 33 and openSUSE 15.3, as well as Ubuntu Groovy.
Windows: Check if Windows SDK is installed for MSVC and ClangCL.
Windows: Enhanced wording in case no compiler was found. No longer tell people how to manually install MinGW64, that is no longer necessary and pywin32 is not needed to detect MSVC, so it’s not installed if not found.
Detect “embeddable Python” by missing include files, and reject it with proper error message.
Added onefile and standalone as a use case to the manual and put also the DLL and data files problems as typically issues.

Cleanups

Avoid decimal and string comparisons for Python versions checks, these were lazy and are going to break once 3.10 surfaces. In testing we now use tuples, in Nuitka core hexacimal values much like CPython itself does.
Stop using subnode child getters and setters, and instead only use subnode attributes. This was gradually changed so far, but in this release all remaining uses have migrated. This should also make the optimization stage go faster.
Change node constructors to not use a decorator to resolve conflicts with builtin names, rather handle these with manual call changes, the decorator only made it difficult to read and less performant.
Move safe string helpers to their own dedicated helper file, allowing for reuse in plugin code that doesn’t want to use all of Nuitka C helpers.
Added utils code for inline copy imports, as we use that for quite a few things now.
Further restructured the Scons files to use more common code.
Plugins: The module name objects now reject many str specific APIs that ought to not be used, and the code got changed to use these instead, leading to cleaner and more correct usages.
Using named tuples to specify included data files and entry points.
Use pkgutil in plugins to scan for modules rather than listing directories.

Tests

New option to display executed commands during comparisons.
Added test suite for onefile testing.

Summary

This release has seen Python3.9 and Onefile both being completed. The later needs compression added on Windows, but that can be added in a coming release, for now it’s fully functional.

The focus clearly has been on massive cleanups, some of which will affect compile time performance. There is relatively little new optimization otherwise.

The adoption of clcache enables a very fast caching, as it’s now loaded directly into the Scons process, avoiding a separate process fork.

Generally a lot of polishing has been applied with many cleanups lowering the technical debt. It will be interesting to see where the hard module imports can lead us in terms of more optimization. Static optimization of the Python version comparisons and checks is needed to lower the amount of imports to be processed.

Important fixes are also included, e.g. the constants loading performance was too slow in some cases. The multiprocessing on Windows and numpy plugins were regressed and finally everything ought to be back to working fine.

Future work will have to aim at enhanced scalability. In some cases, Nuitka still takes too much time to compile if projects like Pandas include virtually everything installed as an option for it to use.

Nuitka Release 0.6.10

2020-12-22T15:38:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes with many new features, e.g. onefile support, as well as many new optimization and bug fixes.

Bug Fixes

Fix, was memory leaking arguments of all complex call helper functions. Fixed in 0.6.9.6 already.
Plugins: Fix, the dill-compat code needs to follow API change. Fixed in 0.6.9.7 already.
Windows: Fixup for multiprocessing module and complex call helpers that could crash the program. Fixed in 0.6.9.7 already.
Fix, the frame caching could leak memory when using caching for functions and generators used in multiple threads.
Python3: Fix, importing an extension module below a compiled module was not possible in accelerated mode.
Python3: Fix, keyword arguments for open built-in were not fully compatible.
Fix, the scons python check should also not accept directories, otherwise strange misleading error will occur later.
Windows: When Python is installed through a symbolic link, MinGW64 and Scons were having issues, added a workaround to resolve it even on Python2.
Compatibility: Added support for co_freevars in code objects, e.g. newer matplotlib needs this.
Standalone: Add needed data files for gooey. Fixed in 0.6.9.4 already.
Scons: Fix, was not respecting --quiet option when running Scons. Fixed in 0.6.9.3 already.
Scons: Fix, wasn’t automatically detecting Scons from promised paths. Fixed in 0.6.9.2 already.
Scons: Fix, the clcache output parsing wasn’t robust enough. Fixed in 0.6.9.1 already.
Python3.8: Ignore all non-strings provided in doc-string fashion, they are not to be considered.
Fix, getattr, setattr and hasattr could not be used in finally clauses anymore. Fixed in 0.6.9.1 already.
Windows: For Python3 enhanced compatibility for Windows no console mode, they need a sys.stdin or else e.g. input will not be compatible and raise RuntimeError.

New Features

Added experimental support for Python 3.9, in such a way that the CPython3.8 test suite passes now, the 3.9 suite needs investigation still, so we might be missing new features.
Added experimental support for Onefile mode with --onefile that uses AppImage on Linux and our own bootstrap binary on Windows. Other platforms are not supported at this time. With this, the standalone folder is packed into a single binary. The Windows variant currently doesn’t yet do any compression yet, but the Linux one does.
Windows: Added downloading of ccache.exe, esp. as the other sources so far recommended were not working properly after updates. This is taken from the official project and should be good.
Windows: Added downloading of matching MinGW64 C compiler, if no other was found, or that was has the wrong architecture, e.g. 32 bits where we need 64 bits.
Windows: Added ability to copy icon resources from an existing binary with new option --windows-icon-from-exe.
Windows: Added ability to provide multiple icon files for use with different desktop resolutions with new option --windows-icon-from-ico that got renamed to disambiguate from other icon options.
Windows: Added support for requesting UAC admin right with new option --windows-uac-admin.
Windows: Added support for requesting “uiaccess” rights with yet another new option --windows-uac-uiaccess.
Windows: Added ability to specify version info to the binary. New options --windows-company-name, --windows-product-name, --windows-file-version, --windows-product-version, and --windows-file-description have been added. Some of these have defaults.
Enhanced support for using the Win32 compiler of MinGW64, but it’s not perfect yet and not recommended.
Windows: Added support for LTO mode for MSVC as well, this seems to allow more optimization.
Plugins: The numpy plugin now handles matplotlib3 config files correctly.

Optimization

Use less C variables in dictionary created, not one per key/value pair. This improved scalability of C compilation.
Use common code for module variable access, leading to more compact code and enhanced scalability of C compilation.
Use error exit during dictionary creation to release the dictionary, list, tuple, and set in case of an error occurring while they are still under construction. That avoids releases of it in error exists, reducing the generated code size by a lot. This improves scalability of C compilation for generating these.
Annotate no exception raise for local variables of classes with know dict shape, to avoid useless error exits.
Annotate no exception exit for staticmethod and classmethod as they do not check their arguments at all. This makes code generated for classes with these methods much more compact, mainly improving their scalability in C compilation.
In code generation, prefer bool over nuitka_bool which allows to annotate exception result, leading to more compact code. Also cleanup so that code generation always go through the C type objects, rather than doing cases locally, adding a C type for bool.
Use common code for C code handling const None return only, to cases where there is any immutable constant value returned, avoid code generation for this common case. Currently mutable constants are not handled, this may be added in the future.
Annotate no exception for exception type checks in handlers for Python2 and no exception if the value has exception type shape for Python3. The exception type shape was newly added. This avoids useless exception handlers in most cases, where the provided exception is just a built-in exception name.
Improve speed of often used compile time methods on nodes representing constant values, by making their implementation type specific to improve frontend compile time speed, we check e.g. mutable and hashable a lot.
Provide truth value for variable references, enhancing loop optimization and merge value tracing, to also decide this correctly for values only read, and then changed through attribute, e.g. append on lists. This allows many more static optimization.
Use staticmethod for methods in Nuitka nodes to achieve faster frontend compile times where possible.
Use dedicated helper code for calls with single argument, avoiding the need have a call site local C array of size one, just to pass a pointer to it.
Added handling of hash slot, to predict hashable keys for dictionary and sets.
Share more slot provision for built-in type shapes from mixin classes, to get them more universally provided, even for special types, where their consideration is unusual.
Trace “user provided” flag only for constants where it really matters, i.e. for containers and generally potentially large values, but not for every number or boolean value.
Added lowering of bytearray constant values to bytes value iteration, while handling constant values for this optimization with dedicated code for improved frontend compilation speed.
The dict built-in now annotates the dictionary type shape of its result.
The wrapping side-effects node now passes on the type shape of the wrapped value, allowing for optimization of these too.
Split slice nodes into variants with 1, 2 or 3 arguments, to avoid the overhead of determining which case we have, as well as to save a bit of memory, since these are more frequently used on Python3 for subscript operations. Also annotate their type shape, allowing more optimization.
Faster dictionary lookups, esp. in cases where errors occur, because we were manually recreating a KeyError that is already provided by the dict implementation. This should also be faster, as it avoids a CPython API call overhead on the DLL and they can provide a reference or not for the returned value, simplifying using code.
Faster dictionary containment checks, with our own dedicated helper, we can use code that won’t create an exception when an item is not present at all.
Faster hash lookups with our own helper, separating cases where we want an exception for non-hashable values or not. These should also be faster to call.
Avoid acquiring thread state in exception handling that checks if a StopIteration occurred, to improved speed on Python3, where is involves locking, but this needs to be applied way more often.
Make sure checks to debug mode and full compatibility mode are done with the variables introduced, to avoid losing performance due to calls for Nuitka compile time enhancements. This was so far only done partially.
Split constant references into two base classes, only one of them tracking if the value was provided by the user. This saves compile time memory and avoids the overhead to check if sizes are exceeded in cases they cannot possibly be so.
The truth value of container creations is now statically known, because the empty container creation is no longer a possibility for these nodes, allowing more optimization for them.
Optimize the bool built-in with no arguments directory, allow to simplify the node for single argument form to avoid checks if an argument was given.
Added iteration handles for xrange values, and make them faster to create by being tied to the node type, avoiding shared types, instead using the mixin approach. This is in preparation to using them for standard iterator tracing as well. So far they are only used for any and all decision.
Added detection if a iterator next can raise, using existing iterator checking which allows to remove needless checks and exception traces. Adding a code variant for calls to next that cannot fail, while tuning the code used for next and unpacking next, to use faster exception checking in the C code. This will speed up unpacking performance for some forms of unpacking from known sizes.
Make sure to use the fastest tuple API possible in all of Nuitka, many place e.g. used PyTuple_Size, and one was in a performance critical part, e.g. in code that used when compiled functions as called as a method.
Added optimized variant for _PyList_Extend for slightly faster unpacking code.
Added optimized variant for PyList_Append for faster list contractions code.
Avoid using RemoveFileSpec and instead provide our own code for that task, slightly reducing file size and avoiding to use the Shlapi link library.

Tests

Made reflected test use common cleanup of test folder, which is more robust against Windows locking issues.
Only output changed CPython output after the forced update of cached value was done, avoiding duplicate or outdated outputs.
Avoid complaining about exceptions for in-place operations in case they are lowered to non-inplace operations and then raise unsupported, not worth the effort to retain original operator.
Added generated test for subscript operations, also expanding coverage in generated tests by making sure, conditional paths are both taken by varying the cond value.
Use our own code helper to check if an object has an attribute, which is faster, because it avoids creating exceptions in the first place, instead of removing them afterwards.

Cleanups

Make sure that code generation always go through the C type objects rather than local elif casing of the type. This required cleaning up many of the methods and making code more abstract.
Added base class for C types without reference counting, so they can share the code that ignores their handling.
Remove getConstant for constant value nodes, use the more general getCompileTimeConstant instead, and provide quick methods that test for empty tuple or dict, to use for checking concrete values, e.g. with call operations.
Unified container creation into always using a factory function, to be sure that existing container creations are not empty.
Stop using @calledWithBuiltinArgumentNamesDecorator where possible, and instead make explicit wrapping or use correct names. This was used to allow e.g. an argument named list to be passed from built-in optimization, but that can be done in a cleaner fashion. Also aligned no attributes and the argument names, there was inconsistency there.
Name mangling was done differently for attribute names and normal names and with non-shared code, and later than necessary, removing this as a step from variable closure taking after initial tree build.
As part of the icon changes, now handled in Python code, we stop using the rc binary and handle all resources ourselves, allowing to remove that code from the Scons side of things.
Moved file comparison code of standalone mode into file utils function for use in plugins as well.
Unified how path concatenation is done in Nuitka helper code, there were more or less complete variants, this is making sure, the most capable form is used in all cases.
Massive cleanup to our scons file, by moving out util code that only scons uses, hacks we apply to speed up scons, and more to separate modules with dedicated interfaces.
When using enumerate we now provide start value of 1 where it is appropriate, e.g. when counting source code lines, rather than adding count+1 on every usage, making code more readable.

Organizational

Do not recommend Anaconda on Windows anymore, it seems barely possible to get anything installed on it with a fresh download, due to the resolver literally working for days without finishing, and then reporting conflicts, it would only we usable when starting with Miniconda, but that seems less interesting to users, also gcc 5.2 is way too old these days.
The commit hook should be reinstalled, since it got improved and adapted for newer git versions.
Added link to donations to funding document, following a GitHub standard.
Bumped requirements for development to the latest versions, esp. newer isort.
Added a rough description of tests to do to add a new CPython test suite, to allow others to take this task in the future.
Updated the git hook so that Windows and newest git works.
Make it more clear in the documentation that Microsoft Appstore Python is not supported.

Summary

This is the big release in terms of scalability. The optimization in this release mostly focused on getting things that cause increased compile times sorted out. A very important fix avoids loop optimization to leak into global passes of all modules unnecessarily, but just as important, generated code now is much better for the C compiler to consume in observed problematic cases.

More optimization changes are geared towards reducing Nuitka frontend compile time, which could also be a lot in some cases, ending up specializing more constant nodes and how they expose themselves to optimization.

Other optimization came from supporting Python 3.9 and things come across during the implementation of that feature, e.g. to be able to make differences with unpacking error messages, we provide more code to handle it ourselves, and to manually optimize how to interact with e.g. list objects.

For Windows, the automatic download of ccache and a matching MinGW64 if none was found, is a new step, that should lower the barrier of entry for people who have no clue what a C compiler is. More changes are bound to come in this field with future releases, e.g. making a minimum version requirement for gcc on Windows that excludes unfit C compilers.

All in all, this release should be taken as a major cleanup, resolving many technical debts of Nuitka and preparing more optimization to come.

Nuitka Release 0.6.9

2020-10-15T10:34:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains important bug fixes for regressions of the 0.6.8 series which had relatively many problems. Not all of these could be addressed as hotfixes, and other issues were even very involved, causing many changes to be necessary.

There are also many general improvements and performance work for tracing and loops, but the full potential of this will not be unlocked with this release yet.

Bug Fixes

Fix, loop optimization sometimes didn’t determinate, effectively making Nuitka run forever, with no indication why. This has been fixed and a mechanism to give up after too many attempts has been added.
Fix, closure taking object allowed a brief period where the garbage collector was exposed to uninitialized objects. Fixed in 0.6.8.1 already.
Python3.6+: Fix corruption for exceptions thrown into asyncgen. Fixed in 0.6.8.1 already.
Fix, deleting variables detected as C type bool could raise an UnboundLocalError that was wrong. Fixed in 0.6.8.1 already.
Python3.8.3+: Fix, future annotations parsing was using hard coded values that were changed in CPython, leading to errors.
Windows: Avoid encoding issues for Python3 on more systems, by going from wide characters to unicode strings more directly, avoiding an encoding as UTF-8 in the middle. Fixed in 0.6.8.2 already.
Windows: Do not crash when warning about uninstalled MSVC using Python3. This is a Scons bug that we fixed. Fixed in 0.6.8.3 already.
Standalone: The output of dependency walker should be considered as “latin1” rather than UTF-8. Fixed in 0.6.8.3 already.
Standalone: Added missing hidden dependencies for flask. Fixed in 0.6.8.1 already.
Standalone: Fixed win32com.client on Windows. Fixed in 0.6.8.1 already.
Standalone: Use pkgutil to scan encoding modules, properly ignoring the same files as Python does in case of garbage files being there. Fixed in 0.6.8.2 already.
Plugins: Enabling a plugin after the filename to compile was given, didn’t allow for arguments to the passed, causing problems. Fixed in 0.6.8.3 already.
Standalone: The certifi data file is now supported for all modules using it and not only some.
Standalone: The bytecode for the standard library had filenames pointing to the original installation attached. While these were not used, but replaced at run time, they increased the size of the binary, and leaked information.
Standalone: The path of sys.executable was not None, but pointing to the original executable, which could also point to some temporary virtualenv directory and therefore not exist, also it was leaking information about the original install.
Windows: With the MSVC compiler, elimination of duplicate strings was not active, causing even unused strings to be present in the binary, some of which contained file paths of the Nuitka installation.
Standalone: Added support for pyglet.
Plugins: The command line handling for Pmw plugin was using wrong defaults, making it include more code than necessary, and to crash if it was not there.

New Features

Windows: Added support for using Python 2.7 through a symlink too. This was already working for Python3, but a scons problem prevented this from working.
Caching of compiled C files is now checked with ccache and clcache, and added automatically where possible, plus a report of the success is made. This can accelerate the re-compile very much, even if you have to go through Nuitka compilation itself, which is not (yet) cached.
Added new --quiet option that will disable informational traces that are going to become more.
The Clang from MSVC installation is now picked up for both 32 and 64 bits and follows the new location in latest Visual Studio 2019.
Windows: The ccache from Anaconda is now supported as well as the one from msys64.

Optimization

The value tracing has become more correct with loops and in general less often inhibits optimization. Escaping of value traces is now a separate trace state allowing for more appropriate handling of actual unknowns.
Memory used for value tracing has been lowered by removing unnecessary states for traces, that we don’t use anymore.
Windows: Prevent scons from scanning for MSVC when asked to use MinGW64. This avoids a performance loss doing something that will then end up being unused.
Windows: Use function level linking with MSVC, this will allow for smaller binaries to be created, that don’t have to include unused helper functions.

Cleanups

The scons file now uses Nuitka utils functions and is itself split up into several modules for enhanced readability.
Plugin interfaces for providing extra entry points have been cleaned up and now named tuples are used. Backward compatibility is maintained though.

Organizational

The use of the logging module was replaced with more of our custom tracing and we now have the ability to write the optimization log to a separate file.
Old style plugin options are now detected and reported as a usage error rather than unknown plugin.
Changed submodules to use git over https, so as to not require ssh which requires a key registered and causes problems with firewalls too.
More correct Debian copyright file, made formatting of emails in source code consistent.
Added repository for Ubuntu focal.

Summary

The main focus of this release has been bug fixes with only a little performance work due to the large amount of regressions and other findings from the last release.

The new constants loading for removes a major scalability problem. The checked and now consistently possible use of ccache and clcache allows for much quicker recompilation. Nuitka itself can still be slow in some cases, but should have seen some improvements too. Scalability will have to remain a focus for the next releases too.

The other focus, was to make the binaries contain no original path location, which is interesting for standalone mode. Nuitka should be very good in this area now.

For optimization, the new loop code is again better. But it was also very time consuming, to redo it, yet again. This has prevented other optimization to be added.

And then for correctness, the locals scope work, while very invasive, was necessary, to handle the usage of locals inside of contractions, but also will be instrumental for function inlining to become generally available.

So, ultimately, this release is a necessary intermediate step. Upcoming releases will be able to focus more clearly on run time performance again as well as on scalability for generated C code.

Nuitka Release 0.6.8

2020-05-18T09:11:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains important general improvements and performance improvements and enhanced optimization as well as many bug fixes that enhance the Python 3.8 compatibility.

Bug Fixes

Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.
Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.
Fix, the fromlist of imports could loose references, potentially leading to corruption of contained strings.
Python3.8: Fix, positional only arguments were not enforced to actually be that way.
Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.
Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.
Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle Unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.
Windows: Fixup dependency walker in unicode input directories.
Standalone: Use frozen module loader only at libpython initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility.
Standalone: Fix for pydantic support.
Standalone: Added missing hidden dependency of uvicorn.
Fix, the parser for .pyi files couldn’t handle multiline imports.
Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.
Fixup static linking with libpython.a that contains main.o by making our colliding symbols for Py_GetArgcArgv weak.
Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.
Distutils: Fix build_nuitka for modules under nested namespaces.
OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.
macOS: Fixup for standalone mode library scan.
Fix, the logging of --show-modules was broken.
Windows: Enable /bigobj mode for MSVC for large compilations to work.
Windows: Fixup crash in warning with pefile dependency manager.
Windows: Fixup win32com standalone detection of other Python version win32com is in system PATH.
Fix, the python flag for static hashes didn’t have the intended effect.
Fix, generators may be resurrected in the cause of their destruction, and then must not be released.
Fix, method objects didn’t implement the methods __reduce__ and __reduce_ex__ necessary for pickling them.
Windows: Fix, using a Python installation through a symlink was not working.
Windows: Fix, icon paths that were relative were not working anymore.
Python3.8: Detect duplicate keywords yielded from star arguments.
Fix, methods could not be pickled.
Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.
Fix, frames need to traverse their attached locals to be released in some cases.

New Features

Plugin command line handling now allows for proper optparse options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.

Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.
Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.
Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.
Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.
Fix, the setattr built-in was leaking a reference to the None value.

Optimization

Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn’t see through iterators yet, but this will come too.
Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.
Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.
For in-place operations, there is the infrastructure to generate them for improved performance, but so far it’s only used for Python2 int, and not for the many types normal operations are supported.
Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.
Complex constants didn’t annotate their type shape, preventing compile time optimization for them.
Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.
Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.
Avoid using a preparing a constant tuple value for calls with only constant arguments.
Avoid using PyErr_Format where it’s not necessary by adding specialized helpers for common cases.
Detect del statements that will raise an exception and replace with that.
Exception matching is boolean shape, allowing for faster code generation.
Disable recursion checks outside of full compat mode.
Avoid large blocks for conditional statements that only need to enclose the condition evaluation.
Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.

Organizational

Updated Developer Manual with changes that happened, removing the obsolete language choice section.
Added 3.8 support mentions in even more places.
The mailing list has been deleted. We now prefer Gitter chat and GitHub issues for discussions.
Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.
Visual Code environments for Py38 and Py27 were added for easier switch.
Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.
Make it clear that --full-compat should not be used in help output.
Added instructions for MSVC runtimes and standalone compilation to support Windows 7.
More complete listing of copyright holders for Debian.
Updated to newer black and PyLint.
Enhanced gcc version check, properly works with gcc 10 and higher.

Tests

Pylint cleanups for some of the tests.
Added test for loading of user plugins.
Removed useless outputs for search mode skipping non-matches.

Cleanups

Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.
Pylint cleanup also foreign code like oset and odict.
In preparation of deprecating the alternative, --enable-plugin has become the only form used in documentation and tests.
Avoid numeric pylint symbols more often.
Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.
The “cannot get here” code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.
Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.

Summary

This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.

Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float or C long can be added, add to that iterator type shape and value analsis, and an actual jump in performance can be expected.

Nuitka Release 0.6.7

2020-01-25T08:42:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains bug fixes and improvements to the packaging, for the RPM side as well as for Debian, to cover Python3 only systems as they are now becoming more common.

Bug Fixes

Compatibility: The value of __module__ for extension modules was not dependent into which package the module was loaded, it now is.
Anaconda: Enhanced detection of Anaconda for Python 3.6 and higher.
CentOS6: Detect gcc version to allow saving on macro memory usage, very old gcc didn’t have that.
Include Python3 for all Fedora versions where it works as well as for openSUSE versions 15 and higher.
Windows: Using short path names to interact with Scons avoids problems with Unicode paths in all cases.
macOS: The usage of install_name_tool could sometimes fail due to length limits, we now increase it at link time.
macOS: Do not link against libpython for module mode. This prevented extension modules from actually being usable.
Python3.6: Follow coroutine fixes in our asyncgen implementation as well.
Fix, our version number handling could overflow with minor versions past 10, so we limited it for now.

New Features

Added support for Python 3.8, the experimental was already there and pretty good, but now added the last obscure features too.
Plugins can now provide C code to be included in the compilation.
Distutils: Added targets build_nuitka and install_nuitka to complement bdist_nuitka, so we support software other than wheels, e.g. RPM packaging that compiles with Nuitka.
Added support for lldb the Clang debugger with the --debugger mode.

Optimization

Make the file prefix map actually work for gcc and clang, and compile files inside the build folder, unless we are running in debugger mode, so we use ccache caching across different compilations for at least the static parts.
Avoid compilation of __frozen.c in accelerated mode, it’s not used.
Prefer using the inline copy of scons over systems scons. The later will only be slower. Use the fallback to external scons only from the Debian packages, since there we consider it forbidden to include software as a duplicate.

Organizational

Added recommended plugins for Visual Code, replacing the list in the Developer Manual.
Added repository for Fedora 30 for download.
Added repository for CentOS 8 for download.
Updated inline copy of Scons used for Python3 to 3.1.2, which is said to be faster for large compilations.
Removed Eclipse setup from the manual, it’s only infererior at this point and we do not use it ourselves.
Debian: Stop recommending PyQt5 in the package, we no longer use it for built-in GUI that was removed.
Debian: Bumped the standards version and modernized the packaging, solving a few warnings during the build.

Cleanups

Scons: Avoid to add Unix only include paths on Windows.
Scons: Have the static source code in a dedicated folder for clarity.

Tests

Added tests to GitHub Actions, for the supported Python versions for all of Linux, macOS and Windows, covering the later publicly for the first time. We use Anaconda on macOS for the tests now, rather than Homebrew.
Enable IO encoding to make sure we use UTF-8 for more test suites that actually need it in case of problems.
Comparing module outputs now handles segfaults by running in the debugger too.

Summary

This release adds full support for Python 3.8 finally, which took us a while, and it cleans up a lot on the packaging side. There aren’t that many important bug fixes, but it’s still nice to this cleaned up.

We have important actual optimization in the pipeline that will apply specialization to target types and for comparison operations. We expect to see actual performance improvements in the next release again.

Nuitka Release 0.6.6

2020-01-06T08:34:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains huge amounts of crucial bug fixes all across the board. There is also new optimization and many organizational improvements.

Bug Fixes

Fix, the top level module must not be bytecode. Otherwise we end up violating the requirement for an entry point on the C level.
Fix, avoid optimizing calls with default values used. This is not yet working and needed to be disabled for now.
Python3: Fix, missing keyword only arguments were not enforced to be provided keyword only, and were not giving the compatible error message when missing.
Windows: Find win32com DLLs too, even if they live in subfolders of site-packages, and otherwise not found. They are used by other DLLs that are found.
Standalone: Fixup for problem with standard library module in most recent Anaconda versions.
Scons: Fix, was using CXXFLAGS and CPPFLAGS even for the C compiler, which is wrong, and could lead to compilation errors.
Windows: Make --clang limited to clang-cl.exe as using it inside a MinGW64 is not currently supported.
Standalone: Added support for using lib2to2.pgen.
Standalone: Added paths used by openSUSE to the Tcl/Tk plugin.
Python3.6+: Fix, the __main__ package was None, but should be "" which allows relative imports from itself.
Python2: Fix, compile time optimization of floor division was using normal division.
Python3: Fix, some run time operations with known type shapes, were falsely reporting error message with unicode or long, which is of course not compatible.
Fix, was caching parent package, but these could be replaced e.g. due to bytecode demotion later, causing crashes during their optimization.
Fix, the value of __compiled__ could be corrupted when being deleted, which some modules wrappers do.
Fix, the value of __package__ could be corrupted when being deleted.
Scons: Make sure we can always output the compiler output, even if it has a broken encoding. This should resolve MSVC issues on non-English systems, e.g. German or Chinese.
Standalone: Support for newest sklearn was added.
macOS: Added resolver for run time variables in otool output, that gets PyQt5 to work on it again.
Fix, floor division of run time calculations with float values should not result in int, but float values instead.
Standalone: Enhanced support for boto3 data files.
Standalone: Added support for osgeo and gdal.
Windows: Fix, there were issues with spurious errors attaching the constants blob to the binary due to incorrect C types provided.
Distutils: Fix, need to allow / as separator for package names too.
Python3.6+: Fix reference losses in asyncgen when throwing exceptions into them.
Standalone: Added support for dill.
Standalone: Added support for scikit-image and skimage.
Standalone: Added support for weasyprint.
Standalone: Added support for dask.
Standalone: Added support for pendulum.
Standalone: Added support for pytz and pytzdata.
Fix, --python-flags=no_docstrings no longer implies disabling the assertions.

New Features

Added experimental support for Python 3.8, there is only very few things missing for full support.
Distutils: Added support for packages that are in a namespace and not just top level.
Distutils: Added support for single modules, not only packages, by supporting py_modules as well.
Distutils: Added support for distinct namespaces.
Windows: Compare Python and C compiler architecture for MSVC too, and catch the most common user error of mixing 32 and 64 bits.
Scons: Output variables used from the outside, so the debugging is easier.
Windows: Detect if clang installed inside MSVC automatically and use it if requested via --clang option. This is only the 32 bits variant, but currently the easy way to use it on Windows with Nuitka.

Optimization

Loop variables were analysed, but results were only available on the inside of the loop, preventing many optimization in these cases.
Added optimization for the abs built-in, which is also a numerical operator.
Added optimization for the all built-in, adding a new concept of iteration handle, for efficient checking that avoids looking at very large sequences, of which properties can still be known.
```
all(range(1, 100000))  # no need to look at all of them
```
Added support for optimizing ImportError construction with keyword-only arguments. Previously only used without these were optimized.
```
raise ImportError(path="lala", name="lele")  # now optimized
```
Added manual specialization for single argument calls, solving a TODO, as these will be very frequent.
Memory: Use single child form of node class where possible, the general class now raises an error if used with used with only one child name, this will use less memory at compile time.
Memory: Avoid list for non-local declarations in every function, these are very rare, only have it if absolutely necessary.
Generate more compact code for potential NameError exceptions being raised. These are very frequent, so this improves scalability with large files.
Python2: Annotate comparison of None with int and str types as not raising an exception.
Shared empty body functions and generators.

One shared implementation for all empty functions removes that burden from the C compiler, and from the CPU instruction cache. All the shared C code does is to release its arguments, or to return an empty generator function in case of generator.
Memory: Added support for automatic releases of parameter variables from the node tree. These are normally released in a try finally block, however, this is now handled during code generation for much more compact C code generated.
Added specialization for int and long operations %, <<, >>, |, &, ^, **, @.
Added dedicated nodes for representing and optimizing based on shapes for all binary operations.
Disable gcc macro tracing unless in debug mode, to save memory during the C compilation.
Restored Python2 fast path for int with unknown object types, restoring performance for these.

Cleanups

Use dedicated ModuleName type that makes the tests that check if a given module name is inside a namespace as methods. This was hard to get right and as a result, adopting this fixed a few bugs and or inconsistent results.
Expand the use of nuitka.PostProcessing to cover all actions needed to get a runnable binary. This includes using install_name_tool on macOS standalone, as well copying the Python DLL for acceleration mode, cleaning the x bit for module mode. Previously only a part of these lived there.
Avoid including the definitions of dynamically created helper functions in the C code, instead just statically declare the ones expected to be there. This resolves Visual Code complaining about it, and should make life also easier for the compiler and caches like ccache.
Create more helper code in closer form to what clang-format does, so they are easier to compare to the static forms. We often create hard coded variants for few arguments of call functions, and generate them for many argument variations.
Moved setter/getter methods for Nuitka nodes consistently to the start of the node class definitions.
Generate C code much closer to what clang-format would change it to be.
Unified calling install_name_tool on macOS into one function that takes care of all the things, including e.g. making the file writable.
Debug output from scons should be more consistent and complete now.
Sort files for compilation in scons for better reproducible results.
Create code objects version independent, avoiding python version checks by pre-processor, hiding new stuff behind macros, that ignore things on older Python versions.

Tests

Added many more built-in tests for increased coverage of the newly covered ones, some of them being generic tests that allow to test all built-ins with typical uses.
Many tests have become more PyLint clean as a result of work with Visual Code and it complaining about them.
Added test to check PyPI health of top 50 packages. This is a major GSoC 2019 result.
Output the standalone directory contents for Windows too in case of a failure.
Added generated tests to fully cover operations on different type shapes and their errors as well as results for typical values.
Added support for testing against installed version of Nuitka.
Cleanup up tests, merging those for only Python 3.2 with 3.3 as we no longer support that version anyway.
Execute the Python3 tests for macOS on Travis too.

Organizational

The donation sponsored machine called donatix had to be replaced due to hardware breakage. It was replaced with a Raspberry-Pi 4.
Enhanced plugin documentation.
Added description of the git workflow to the Developer Manual.
Added checker script check-nuitka-with-codespell that reports typos in the source code for easier use of codespell with Nuitka.
Use newest PyLint and clang-format.
Also check plugin documentation files for ReST errors.
Much enhanced support for Visual Code configuration.
Trigger module code is now written into the build directory in debug mode, to aid debugging.
Added deep check function that descends into tuples to check their elements too.

Summary

This release comes after a long time of 4 months without a release, and has accumulated massive amounts of changes. The work on CPython 3.8 is not yet complete, and the performance work has yet to show actual fruit, but has also progressed on all fronts. Connecting the dots and pieces seems not far away.

Nuitka Release 0.6.5

2019-07-30T21:52:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains many bug fixes all across the board. There is also new optimization and many organizational improvements.

Bug Fixes

Python3.4+: Fixed issues with modules that exited with an exception, that could lead to a crash, dealing with their __spec__ value.
Python3.4+: The __loader__ method is_package had the wrong signature.
Python3.6+: Fix for async with being broken with uncompiled generators.
Python3.5+: Fix for coroutines that got their awaited object closed behind their back, they were complaining with RuntimeError should they be closed themselves.
Fix, constant values None in a bool target that could not be optimized away, lead to failure during code generation.
```
if x() and None:
    ...
```
Standalone: Added support for sha224, sha384, sha512 in crypto package.
Windows: The icon wasn’t properly attached with MinGW64 anymore, this was a regression.
Windows: For compiler outputs, also attempt preferred locale to interpret outputs, so we have a better chance to not crash over MSVC error messages that are not UTF-8 compatible.
macOS: Handle filename collisions for generated code too, Nuitka now treats all filesystems for all OS as case insensitive for this purpose.
Compatibility: Added support for tolerant del in class exception handlers.
```
class C:

    try:
        ...
    except Exception as e:
        del e

        # At exception handler exit, "e" is deleted if still assigned
```
We already were compatible for functions and modules here, but due to the special nature of class variables really living in dictionaries, this was delayed. But after some other changes, it was now possible to solve this TODO.
Standalone: Added support for Python3 variant of Pmw.
Fix, the NumPy plugin now handles more installation types.
Fix, the qt plugin now handles multiple library paths.
Fix, need libm for some Anaconda variants too.
Fix, left over bytecode from plugins could crash the plugin loader.
Fix, pkgutil.iter_packages is now working for loaded packages.

New Features

Python3.8: Followed some of the changes and works with beta2 as a Python 3.7, but none of the new features are implemented yet.
Added support for Torch, Tensorflow, Gevent, Sklearn, with a new Nuitka plugin.
Added support for “hinted” compilation, where the used modules are determined through a test run.
Added support for including TCL on Linux too.

Optimization

Added support for the any built-in. This handles a wide range of type shapes and constant values at compile time, while also having optimized C code.
Generate code for some CLONG operations in preparation of eventual per expression C type selection, it then will allow to avoid objects in many instances.
Windows: Avoid creating link libraries for MinGW64 as these have become unnecessary is the mean time.
Packages: Do not export entry points for all included packages, only for the main package name it is importable as.

Organizational

Added support for Visual Studio 2019 as a C compiler backend.
Improved plugin documentation describing how to create plugins for Nuitka even better.
The is now a mode for running the tests called all which will execute all the tests and report their errors, and only fail at the very end. This doesn’t avoid wasting CPU cycles to report that e.g. all tests are broken, but it allows to know all errors before fixing some.
Added repository for Fedora 30 for download.
Added repository for openSUSE 15.1 for download.
Ask people to compile hello world program in the GitHub issue template, because many times, they have setup problems only.
Visual Studio Code is now the recommended IDE and has integrated configuration to make it immediately useful.
Updated internal copy of Scons to 3.1.0 as it incorporates many of our patches.
Changed wordings for optimization to use “lowering” as the only term to describe an optimization that simplifies.

Cleanups

Plugins: Major refactoring of Nuitka plugin API.
Plugins: To locate module kind, use core Nuitka code that handles more cases.
The test suite runners are also now auto-formatted and checked with PyLint.
The Scons file is now PyLint clean too.
Avoid build_definitions.h to be included everywhere, in that it’s only used in the main program part. This makes C linter hate us much less for using a non-existent file.

Tests

Run the tests using Travis on macOS for Python2 too.
More standalone tests have been properly whitelisting to cover openSSL usage from local system.
Disabled PySide2 test, it’s not useful to fail and ignore it.
Tests: Fixups for coverage testing mode.
Tests: Temporarily disable some checks for constants code in reflected tests as it only exposes marshal not being deterministic.

Summary

This release is huge again. Main points are compatibility fixes, esp. on the coroutine side. These have become apparently very compatible now and we might eventually focus on making them better.

Again, GSoC 2019 is also showing effects, and will definitely continue to do soin the next release.

Many use cases have been improved, and on an organizational level, the adoption of Visual Studio Code seems an huge improvement to have a well configured IDE out of the box too.

In upcoming releases, more built-ins will be optimized, and hopefully the specialization of operations will hit more and more code with more of the infrastructure getting there.

Intro

2019-07-12T18:00:00+01:00

This post compares the pytest results of dateutil to its nuitka-built .whl counterpart.

Dateutil standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the dateutil package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

The pytest results were very similar:

Regular pytests: ============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============
Nuitka wheel pytests: ============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Steps to Reproduce

Clone dateutil and nuitka into a new folder
Inside the dateutil folder, issue python -m pip install -r requirements-dev.txt to install its requirements.
Issue python -m pytest --disable-warnings, this runs the regular pytest for dateutil.
Change into the nuitka folder and issue python setup.py develop.
Change back into dateutil and issue python setup.py bdist_nuitka to build the dateutil wheel using nuitka. The newly built wheel should be found in the dist folder.
Use pip to uninstall the existing dateutil, then issue python -m pip install followed by the newly built .whl filename.
Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for dateutil.

Uncompile Python

dateutil regular pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2074 items

dateutil\test\test_easter.py ........................................... [  2%]
........................................................................ [  5%]
................................................                         [  7%]
dateutil\test\test_import_star.py .                                      [  7%]
dateutil\test\test_imports.py .......................                    [  9%]
dateutil\test\test_internals.py ....                                     [  9%]
dateutil\test\test_isoparser.py ........................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
.......x...x............................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 35%]
.....................xx                                                  [ 36%]
dateutil\test\test_parser.py ........................................... [ 38%]
........................................................................ [ 42%]
........................................................................ [ 45%]
.................................xxxxxxxxxxxxxsss......                  [ 48%]
dateutil\test\test_relativedelta.py .................................... [ 49%]
.............................................                            [ 52%]
dateutil\test\test_rrule.py ............................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
................................................................x....... [ 78%]
..............                                                           [ 79%]
dateutil\test\test_tz.py ............................s...........sssssss [ 81%]
sssssssssssssssssssssssssssssssssssssssss..s............................ [ 84%]
xxx..s......................................s........................... [ 88%]
s....................................................................... [ 91%]
..........s....................................s.s....................ss [ 95%]
sssssssssssss....s..........s........................................... [ 98%]
.............                                                            [ 99%]
dateutil\test\test_utils.py .......                                      [ 99%]
dateutil\test\property\test_isoparse_prop.py .                           [ 99%]
dateutil\test\property\test_parser_prop.py ..                            [ 99%]
docs\exercises\solutions\mlk_day_rrule_solution.py .                     [100%]

============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============

Compiled with Nuitka

nuitka wheel pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2073 items

test\test_easter.py .................................................... [  2%]
........................................................................ [  5%]
.......................................                                  [  7%]
test\test_import_star.py .                                               [  7%]
test\test_imports.py .......................                             [  9%]
test\test_internals.py ....                                              [  9%]
test\test_isoparser.py ................................................. [ 11%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
......................................................................x. [ 25%]
..x..................................................................... [ 28%]
........................................................................ [ 32%]
........................................................................ [ 35%]
............xx                                                           [ 36%]
test\test_parser.py .................................................... [ 39%]
........................................................................ [ 42%]
........................................................................ [ 46%]
........................xxxxxxxxxxxxxsss......                           [ 48%]
test\test_relativedelta.py ............................................. [ 50%]
....................................                                     [ 52%]
test\test_rrule.py ..................................................... [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.......................................................x................ [ 79%]
.....                                                                    [ 79%]
test\test_tz.py ............................s...........ssssssssssssssss [ 81%]
ssssssssssssssssssssssssssssssss..s............................xxx..s... [ 85%]
...................................s...........................s........ [ 88%]
........................................................................ [ 92%]
.s....................................s.s....................sssssssssss [ 95%]
ssss....s..........s.................................................... [ 99%]
....                                                                     [ 99%]
test\test_utils.py .......                                               [ 99%]
test\property\test_isoparse_prop.py .                                    [ 99%]
test\property\test_parser_prop.py ..                                     [100%]

============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Intro

2019-06-19T16:00:00+01:00

This post compares the pytest results of urllib3 to its nuitka-built .whl counterpart.

Urllib3 standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the urllib3 package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

At first, the urllib3 nuitka-wheel pytest was crashing because of the unsafe assumption that imports will always exist (which is not the case if exceptions are thrown). Issue 413 was filed to record and fix this bug.

After the fixes, the pytests were ran again and the results were very similar:

Regular pytests: ====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======
Nuitka wheel pytests: ====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

The extra passes are suspicious and require more investigation into why they happen. To make that easy, we are going to fully automate the process and compare outputs with verbose pytest modes.

Steps to Reproduce

Clone urllib3 and nuitka into a new folder
Inside the urllib3 folder, issue python -m pip install -r dev-requirements.txt to install its requirements.
Issue python -m pytest --disable-warnings, this runs the regular pytest for urllib3.
Change into the nuitka folder and issue python setup.py develop.
Change back into urllib3 and issue python setup.py bdist_nuitka to build the urllib3 wheel using nuitka. The newly built wheel should be found in the dist folder.
Use pip to uninstall the existing urllib3, then issue python -m pip install followed by the newly built .whl filename.
Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for urllib3.

Uncompile Python

urllib3 regular pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py ..                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py ..............................F.F.F..                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
.................................................Uncaught exception, closing connection.
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======

Compiled with Nuitka

nuitka wheel pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py .F                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py .....................................                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

Nuitka Release 0.6.4

2019-06-10T19:28:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains many bug fixes all across the board. There is also new optimization and many organizational improvements.

Bug Fixes

When linking very large programs or packages, with gcc compiler, Scons can produce commands that are too large for the OS. This happens sooner on the Windows OS, but also on Linux. We now have a workaround that avoids long command lines by using @sources.tmp syntax.
Standalone: Remove temporary module after its use, instead of keeping it in sys.modules where e.g. Quart code tripped over its __file__ value that is illegal on Windows.
Fixed non-usage of our enhanced detection of gcc version for compilers if given as a full path.
Fixed non-detection of gnu-cc as a form of gcc compiler.
Python3.4: The __spec__ value corrections for compiled modules was not taking into account that there was a __spec__ value, which can happen if something is wrapping imported modules.
Standalone: Added implicit dependencies for passlib.
Windows: Added workaround for OS command line length limit in compilation with MinGW64.
Python2: Revive the enum plugin, there are backports of the buggy code it tries to patch up.
Windows: Fixup handling of SxS with non zero language id, these occur e.g. in Anaconda.
Plugins: Handle multiple PyQt plugin paths, e.g. on openSUSE this is done, also enhanced finding that path with Anaconda on Windows.
Plugins: For multiprocessing on Windows, allow the .exe suffix to not be present, which can happen when ran from command line.
Windows: Better version checks for DLLs on Python3, the ctypes helper code needs more definitions to work properly.
Standalone: Added support for both pycryptodome and pycryptodomex.
Fix, the chr built-in was not giving fully compatible error on non number input.
Fix, the id built-in doesn’t raise an exception, but said otherwise.
Python3: Proper C identifiers for names that fit into latin-1, but are not ascii encodings.

New Features

Windows: Catch most common user error of using compiler from one architecture against Python from another. We now check those and compare it, and if they do not match, inform the user directly. Previously the compilation could fail, or the linking, with cryptic errors.
Distutils: Using setuptools and its runners works now too, not merely only pure distutils.
Distutils: Added more ways to pass Nuitka specific options via distutils.
Python3.8: Initial compatibility changes to get basic tests to work.

Organizational

Nuitka is participating in the GSoC 2019 with 2 students, Batakrishna and Tommy.
Point people creating PRs to using the pre-commit hook in the template. Due to making the style issues automatic, we can hope to encounter less noise and resulting merge problems.
Many improvements to the pre-commit hook were done, hopefully completing its development.
Updated to latest pylint, black, and isort versions, also added codespell to check for typos in the source code, but that is not automated yet.
Added description of how to use experimental flags for your PRs.
Removed mirroring from Bitbucket and Gitlab, as we increasingly use the GitHub organisation features.
Added support for Ubuntu Disco, removed support for Ubuntu Artful packages.

Optimization

Windows: Attach data blobs as Windows resource files directly for programs and avoid using C data files for modules or MinGW64, which can be slow.
Specialization of helper codes for + is being done for more types and more thoroughly and fully automatic with Jinja2 templating code. This does replace previously manual code.
Added specialization of helper codes for * operation which is entirely new.
Added specialization of helper codes for - operation which is entirely new.
Dedicated nodes for specialized operations now allow to save memory and all use type shape based analysis to predict result types and exception control flow.
Better code generation for boolean type values, removing error checks when possible.
Better static analysis for even more type operations.

Cleanups

Fixed many kinds of typos in the code base with codespell.
Apply automatic formatting to more test runner code, these were previously not done.
Avoid using shutil.copytree which fails to work when directory already exists, instead provide nuitka.util.FileOperations.copyTree and use that exclusively.

Tests

Added new mode of operation to test runners, only that executes just one test and stops, useful during development.
Added new mechanism for standalone tests to expression modules that need to be importable, or else to skip the test by a special comment in the file, instead of by coded checks in the test runner.
Added also for more complex cases, another form of special comment, that can be any expression, that decides if the test makes sense.
Cover also setuptools in our distutils tests and made the execution more robust against variable behavior of distutils and setuptools.
Added standalone test for Urllib3.
Added standalone test for rsa.
Added standalone test for Pmw.
Added standalone test for passlib.

Summary

Again this release is a sign of increasing adoption of Nuitka. The GSoC 2019 is also showing effects, definitely will in the next release.

This release has a lot of new optimization, called specialization, but for it to really used, in many instances, we need to get away from working on C types for variables only, and get to them beig used for expressions more often. Otherwise much of the new special code is not used for most code.

The focus of this release has been again to open up development further and to incorporate findings from users. The number of fixes or new use cases working is astounding.

In upcoming releases, new built-ins will be optimized, and specialization of operations will hit more and more code now that the infrastructure for it is in place.

Nuitka this week #13

2019-05-12T11:07:00+01:00

Communication vs. Coding

Communication was a lot more, just not these postings. Nuitka has ventured into important realms.

First, active developers have joined Nuitka. Second, as a consequence of the later, Nuitka indeed was able to participate with a insanely powerful offering of 4 mentors.

This meant, that a lot of technical or project management debt hat to be dealt with, and there was no time to make this kind of posting. This is probably good news on all fronts, except that I feel they are missing, and am glad to be able to resume them.

Google Summer of Code 2019

My Experience

This was the most crazy ride and fun. Became the admin of an active sub-organisation under the PSF umbrella for GSoC 2019. So shortly after the announcement of the project lists, students flooded in, and starting working on things right away as early as February.

They were working hard to prove themselves and get to know Nuitka, asking many questions, with us having lots of email interviews, and even a lot of video calls, where I personally talked to people about ideas.

This happened simultaneously to more experienced developers joining the project as well, making very important contributions.

This basically meant, that many days 90%-100% of my Nuitka time was for communication or working off technical debts I knew there were, or for preparing things, I want to have with multiple people working on the code base. This will be detailed in other sections.

Accepted Students

All of this has paid of in that we now have 2 really good students to work on 2 Nuitka projects over the summer. Let’s welcome Batakrishna and Tommy in separate posts. They will introduce themselves and what they are going to do shortly, in separate posts.

Lets just say, this is fantastic news. We had so many good applications and its a shame, but we knew right away, not everybody who deserved it could be picked. But this is going to be a good chance for us to get to be open and welcoming to new people.

Optimization Work

Core Stuff

So, the 0.6.3 release (btw. on Windows, be sure to use the 0.6.3.1 hotfix), which was made as a consolidation effort to get the good work of mostly other people out, didn’t contain much optimization work for the core, as that is still my thing.

However, this changed a lot. An idea that came to my mind for how to do the massive amounts of specialized helpers needed beyond + and +=, with which I had started for prior releases. And that is to use Jinja2 based templates for C, to generate the code.

This was an important idea. Took a while, but soon the manual code for + was already replaced with generated code, fixing a few bugs by the way, and from there, the generation was expanded to cover * as well.

Currently, support for the 3 (!) different kinds of additions (TrueDiv and FloorDir as well as Python2 default division, dubbed OldDiv in Nuitka was added along with - .

The reason, + and * were done first, is that they have special treatment for sequences, using sq_concat and sq_repeat, where the other operations will be more straightforward, e.g. nb_subtract (-) has a lot types supporting it and that makes those the easy cases.

I am saving a deeper explanation of 3 things we will need for the next time. Basically we need optimization of these things at compile time, and that is getting there, and code to use in the backend, and that is getting there, and a third thing, that is to use optimization knowledge to apply the special code as much as possible, and that is not yet fully there.

Faster Windows Dependencies

This is going to excite Windows users. After Orsiris de Jong had done a replacement for dependency walker that is faster, this had remained in an experimental status, just due to lack of time.

Recently however, I felt there is more time, after GSoC student selection has happened, and that I could finally work a bit on open issues like this. And when I wrote a dedicated tool, to analyse dependences with either technology to compare the results, I found that dendency walker founds a lot more things.

That was a turn down, but turns out, nothing it finds is stuff that should not be on the white list. In fact, it’s all core Windows things, and from the System32 folder. That made me question, why we take anything from there (except maybe PythonXY.dll) at all, and after that change the performance changed dramatically.

The dependency walker now finishes a file in milliseconds. Actually the pefile is now slow (surely it ought to be compiled), and takes some seconds, for a file. That is amazing, and has lead to me to remove the parallel usage, and since pefile allows for perfect caching, and is Free Software, we will probably keep it.

This will address a widespread complaint of many Windows users of the standalone mode. This is now a relatively unnoticable part of the overall experience.

Currently I need to finish off some remaining problems with it, before putting it out in the wild. Getting this into a release will solve many newcomer issues.

Nuitka Organisation

Esp. for Google Summer of Code, Nuitka has sought and found mentors, some of which are highly experienced for the task. I will let them decide and write their own introduction, but I feel really blessed by them helping me out in my desperate calls for help. Without them, neither could Nuitka participate, nor could it even overachieve as much as it does.

Therefore I welcomed Kamran and Vaibhav into the organisation and they are excited to work the the 2 accepted students, that are also added.

Plugin Work

On the jorj branch there is a lot of work from Jorj that aims at adding support for more of the beasty stuff with hidden dependency and plugin needs.

He is also working at run time tracing of your program to be translated to automatic imports of just that. I am going to highlight this later, once I manage to cherry-pick the release ready parts from it for the next release.

But this definitely awesome stuff, and going to make Nuitka very easy to use for some people, even with stranger software.

Opening Up

This is also a teaser. But we did so much work for the 0.6.3 release to make sure information is there, and things are accessible and changeable by everyone, or even the pre-commit hook that I am very proud of.

But I shall save this for next week, otherwise it will be too much new information.

Twitter

I continue to be active there, although often I fall prey to of not wanting to talk about unfinished things. On Twitter of all things.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.3

2019-04-07T08:21:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This has a focus on organizational improvements. With more and more people joining Nuitka, normal developers as well as many GSoC 2019 students, the main focus was to open up the development tools and processes, and to improve documentation.

That said, an impressive amount of bug fixes was contributed, but optimization was on hold.

Bug Fixes

Windows: Added support for running compiled binaries in Unicode path names.
Standalone: Added support for crytodomex and pycparser packages.
Standalone: Added support for OpenSSL support in PyQt on Windows.
Standalone: Added support for OpenGL support with QML in PyQt on Windows.
Standalone: Added support for SciPy and extended the NumPy plugin to also handle it.
UI: The option --plugin-list still needed a positional argument to work.
Make sure sys.base_prefix is set correctly too.
Python3: Also make sure sys.exec_prefix and sys.base_exec_prefix are set correctly.
Standalone: Added platform plugins for PyQt to the default list of sensible plugins to include.
Fix detection of standard library paths that include .. path elements.

Optimization

Avoid using static C++ runtime library when using MinGW64.

New Features

Plugins: A plugin may now also generate data files on the fly for a given module.
Added support for FreeBSD/PowerPC arch which still uses gcc and not clang.

Organizational

Nuitka is participating in the GSoC 2019.
Added documentation on how to create or use Nuitka plugins.
Added more API doc to functions that were missing them as part of the ongoing effort to complete it.
Updated to latest PyLint 2.3.1 for checking the code.
Scons: Using newer Scons inline copy with Python 2.7 as, the old one remains only used with Python 2.6, making it easier to know the relevant code.
Auto-format was very much enhanced and handles C and ReST files too now. For Python code it does pylint comment formatting, import statement sorting, and blackening.
Added script misc/install-git-hooks.py that adds a commit hook that runs auto-format on commit. Currently it commits unstaged content and therefore is not yet ready for prime time.
Moved adapted CPython test suites to GitHub repository under Nuitka Organisation.
Moved Nuitka-website repository to GitHub repository under Nuitka Organisation.
Moved Nuitka-speedcenter repository to GitHub repository under Nuitka Organisation.
There is now a Gitter chat for Nuitka community.
Many typo and spelling corrections on all the documentation.
Added short installation guide for Nuitka on Windows.

Cleanups

Moved commandline parsing helper functions from common code helpers to the main program where of course their only usage is.
Moved post-processing of the created standalone binary from main control to the freezer code.
Avoid using chmod binary to remove executable bit from created extension modules.
Windows: Avoid using rt.exe and mt.exe to deal with copying the manifest from the python.exe to created binaries. Instead use new code that extracts and adds Windows resources.
Fixed many ResourceWarnings on Python3 by improved ways of handling files.
Fixed deprecation warnings related to not using collections.abc.
The runners in bin directory are now formatted with black too.

Tests

Detect Windows permission errors for two step execution of Nuitka as well, leading to retries should they occur.
The salt value for CPython cached results was improved to take more things into account.
Tests: Added more trick assignments and generally added more tests that were so far missing.

Summary

With the many organizational changes in place, my normal work is expected to resume for after and yield quicker improvements now.

It is also important that people are now enabled to contribute to the Nuitka web site and the Nuitka speedcenter. Hope is to see more improvements on this otherwise neglected areas.

And generally, it’s great to see that a community of people is now looking at this release in excitement and pride. Thanks to everybody who contributed!

Nuitka Release 0.6.2

2019-02-20T11:03:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a huge focus on organizational things. Nuitka is growing in terms of contributors and supported platforms.

Bug Fixes

Fix, the Python flag --python-flag=-O was removing doc strings, but that should only be done with --python-flag=-OO which was added too.
Fix, accelerated binaries failed to load packages from the virtualenv (not venv) that they were created and ran with, due to not propagating sys.prefix.
Standalone: Do not include plat-* directories as frozen code, and also on some platforms they can also contain code that fails to import without error.
Standalone: Added missing implicit dependency needed for newer NumPy versions.

New Features

Added support for Alpine Linux.
Added support for MSYS2 based Python on Windows.
Added support for Python flag --python flag=-OO, which allows to remove doc strings.
Added experimental support for pefile based dependency scans on Windows, thanks to Orsiris for this contribution.
Added plugin for proper Tkinter standalone support on Windows, thanks to Jorj for this contribution.
There is now a __compiled__ attribute for each module that Nuitka has compiled. Should be like this now, and contains Nuitka version information for you to use, similar to what sys.version_info gives as a namedtuple for your checks.
```
__nuitka_version__(major=0, minor=6, micro=2, releaselevel="release")
```

Optimization

Experimental code for variant types for int and long values, that can be plain C value, as well as the PyObject *. This is not yet completed though.
Minor refinements of specialized code variants reducing them more often the actual needed code.

Organizational

The Nuitka GitHub Organisation that was created a while ago and owns the Nuitka repo now, has gained members. Check out https://github.com/orgs/Nuitka/people for their list. This is an exciting transformation for Nuitka.
Nuitka is participating in the GSoC 2019 under the PSF umbrella. We hope to grow even further. Thanks to the mentors who volunteered for this important task. Check out the GSoC 2019 page and thanks to the students that are already helping out.
Added Nuitka internal API documentation that will receive more love in the future. It got some for this release, but a lot is missing.
The Nuitka code has been black-ened and is formatted with an automatic tool now all the way, which makes contributors lives easier.
Added documentation for questions received as part of the GSoC applications and ideas work.
Some proof reading pull requests were merged for the documentation, thanks to everybody who addresses these kinds of errors. Sometimes typos, sometimes broken links, etc.
Updated inline copy of Scons used for Python3 to 3.0.4, which hopefully means more bugs are fixed.

Summary

This release is a sign of increasing adoption of Nuitka. The GSoC 2019 is showing early effects, as is more developers joining the effort. These are great times for Nuitka.

This release has not much on the optimization side that is user visible, but the work that has begun is capable of producing glorious benchmarks once it will be finished.

The focus on this and coming releases is definitely to open up the Nuitka development now that people are coming in as permanent or temporary contributors in (relatively) high numbers.

Nuitka this week #12

2019-02-08T14:09:00+00:00

Communication vs. Coding

Over the holiday season I sort of neglected these postings, but there are other fields, where I have put my focus, but I think these postings are now going to resume.

A quick update in a while in an eternally fluent situation, and not too polished.

Optimization Work

So, the 0.6.2 optimization work has been not a lot yet. I started some work on int/long C type and it looked good.

Week of Bugfixing

So I did this, and it turned out rather long. I continued fixing things and finishing up open things to the point that it now is clean. I hate to be falling behind. I am touching on a few more interesting topics.

Python flag -O and -OO

Nuitka was supporting -O but doing what should only the done for -OO which I only learned of now. So this is going to be supported both now.

Virtualenv vs. venv

Accelerated binaries were not running with full sys.path in the virtualenv, because site module uses sys.prefix and that was not propagated,but it now is.

Google Summer of Code for Nuitka

So the GSoC 2019 page is shaping up, the pull request to list Nuitka on the PSF project pages has been merged. More tweaking will be needed to get into perfect shape, and that has been my main communication time effort.

Finally somebody had stepped up for mentor, which means a lot to me. Now we need to see if students are willing to pick us.

And because I publish this too late. Already a bunch of stundents are showing interest and are hacking on Nuitka, which keeps me even more busy, and makes me extremely happy.

Nuitka Organisation

The Nuitka organisation on GitHub was introduced for my a while ago, and I had transferred the ownership of the Nuitka repository to there. Having moved the issue tracking to there, I was going more all in on it.

Recently more people have submitted PRs and with incredible quality and willingness to support it even after merge of their PR. No dry by contribution, but people looking to actually improve Nuitka together with me.

Therefore I welcome Orsiris and Jorj on board and am really happy about it.

Windows Dependencies from pefile

So Orsiris de Jong implemented a replacement for the dependency walker based code to scan DLLs for Windows standalone using the pefile module, which is likely way better at this. For starters it’s experimental in the next release, but I expect it to soon become the default.

Tkinter Plugin

And Jorj X. McKie implemented a Windows Tkinter plug-in that deals with copying of the TCL installation and integrating it with Nuitka standalone for distribution.

That is very nice and seems to affect a lot of people using that GUI style it seems.

Numpy Plugin

And Jorj X. McKie also implemented a NumPy plug-in that deals with the various acceleration methods there are, e.g. MKL, Blas, I have no idea. It copies the needed DLLs along and makes sure that numpy stays fast in standalone mode.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.1

2019-01-08T10:31:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes after a relatively long time, and contains important new optimization work, and even more bug fixes.

Bug Fixes

Fix, the options --[no]follow-import-to=package_name was supposed to not follow into the given package, but the check was executed too broadly, so that e.g. package_name2 was also affected. Fixed in 0.6.0.1 already.
Fix, wasn’t detecting multiple recursions into the same package in module mode, when attempting to compile a whole sub-package. Fixed in 0.6.0.1 already.
Fix, constant values are used as C boolean values still for some of the cases. Fixed in 0.6.0.1 already.
Fix, referencing a function cannot raise an exception, but that was not annotated. Fixed in 0.6.0.2 already.
macOS: Use standard include of C bool type instead of rolling our own, which was not compatible with newest Clang. Fixed in 0.6.0.3 already.
Python3: Fix, the bytes built-in type actually does have a __float__ slot. Fixed in 0.6.0.4 already.
Python3.7: Types that are also sequences still need to call the method __class_getitem__ for consideration. Fixed in 0.6.0.4 already.
Python3.7: Error exits from program exit could get lost on Windows due to __spec__ handling not preserving errors. Fixed in 0.6.0.4 already.
Windows: Negative exit codes from Nuitka, e.g. due to a triggered assertion in debug mode were not working. Fixed in 0.6.0.4 already.
Fix, conditional and expressions were mis-optimized when not used to not execute the right hand side still. Fixed in 0.6.0.4 already.
Python3.6: Fix, generators, coroutines, and asyncgen were not properly supporting annotations for local variables. Fixed in 0.6.0.5 already.
Python3.7: Fix, class declarations had memory leaks that were untestable before 3.7.1 fixed reference count issues in CPython. Fixed in 0.6.0.6 already.
Python3.7: Fix, asyncgen expressions can be created in normal functions without an immediate awaiting of the iterator. This new feature was not correctly supported.
Fix, star imports on the module level should disable built-in name optimization except for the most critical ones, otherwise e.g. names like all or pow can become wrong. Previous workarounds for pow were not good enough.
Fix, the scons for Python3 failed to properly report build errors due to a regression of the Scons version used for it. This would mask build errors on Windows.
Python3.4: Fix, packages didn’t indicate that they are packages in their __spec__ value, causing issues with importlib_resources module.
Python3.4: The __spec__ values of compiled modules didn’t have compatible origin and has_location values preventing importlib_resources module from working to load data files.
Fix, packages created from .pth files were also considered when checking for sub-packages of a module.
Standalone: Handle cases of conflicting DLLs better. On Windows pick the newest file version if different, and otherwise just report and pick randomly because we cannot really decide which ought to be loaded.
Standalone: Warn about collisions of DLLs on non-Windows only as this can happen with wheels apparently.
Standalone: For Windows Python extension modules .pyd files, remove the SxS configuration for cases where it causes problems, not needed.
Fix: The exec statement on file handles was not using the proper filename when compiling, therefore breaking e.g. inspect.getsource on functions defined there.
Standalone: Added support for OpenGL platform plugins to be included automatically.
Standalone: Added missing implicit dependency for zmq module.
Python3.7: Fix, using the -X utf8 flag on the calling interpreter, aka --python-flag=utf8_mode was not preserved in the compiled binary in all cases.

Optimization

Enabled C target type void which will catch creating unused stuff more immediately and give better code for expression only statements.
Enabled in-place optimization for module variables, avoiding write back to the module dict for unchanged values, accelerating these operations.
Compile time memory savings for the yield node of Python2, no need to track if it is in an exception handler, not relevant there.
Using the single child node for the yield nodes gives memory savings at compile time for these, while also making them operate faster.
More kinds of in-place operations are now optimized, e.g. int += int and the bytes ones were specialized to perform real in-place extension where possible.
Loop variables no longer loose type information, but instead collect the set of possible type shapes allowing optimization for them.

Organizational

Corrected download link for Arch AUR link of develop package.
Added repository for Ubuntu Cosmic (18.10) for download.
Added repository for Fedora 29 for download.
Describe the exact format used for clang-format in the Developer Manual.
Added description how to use CondaCC on Windows to the User Manual.

Cleanups

The operations used for async for, async with, and await were all doing a look-up of an awaitable, and then executing the yield from that awaitable as one thing. Now this is split into two parts, with a new ExpressionYieldFromAwaitable as a dedicated node.
The yield node types, now 3 share a base class and common computation for now, enhancing the one for awaitiable, which was not fully annotating everything that can happen.
In code generation avoid statement blocks that are not needed, because there are no local C variables declared, and properly indent them.

Tests

Fixups for the manual Valgrind runner and the UI changes.
Test runner detects lock issue of clcache on Windows and considers it a permission problem that causes a retry.

Summary

This addresses even more corner cases not working correctly, the out of the box experience should be even better now.

The push towards C level performance for integer operation was held up by the realization that loop SSA was not yet there really, and that it had to be implemented, which of course now makes a huge difference for the cases where e.g. bool are being used. There is no C type for int used yet, which limits the impact of optimization to only taking shortcuts for the supported types. These are useful and faster of course, but only building blocks for what is to come.

Most of the effort went into specialized helpers that e.g. add a float and and int value in a dedicated fashion, as well as comparison operations, so we can fully operate some minimal examples with specialized code. This is too limited still, and must be applied to ever more operations.

What’s more is that the benchmarking situation has not improved. Work will be needed in this domain to make improvements more demonstrable. It may well end up being the focus for the next release to improve Nuitka speedcenter to give more fine grained insights across minor changes of Nuitka and graphs with more history.

Nuitka this week #11

2018-12-10T07:09:00+00:00

Communication vs. Coding

I continue to force myself to report more publicly, and it feels good. This time things are in a stablizing period, and I feel I have a consistent message.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Optimization Work

So, the 0.6.1 optimization work has been a lot. And it’s containing improvements on every level. I think I will detail the levels in another section.

Levels of Optimization

First level is of course node level optimization. Here 0.6.1 adds many things, from better handling of closure variables not all as unknown every time control flow escapes, to some operations + and comparisons on known built-in type shapes to now be able to statically tell that they do not raise. The opposite (does definitely raise) is prepared, but not yet used.

This allows for type shapes to be longer known. Now a+b+c can be known, but previously only a+b was sort of known, and little used information.

The next level is picking the C target type. Here seeing more operations and understanding more variables allows to more often pick the C bool or C void types over the PyObject * C type. For 0.6.1 I have observed that esp. more indicator variables make it to that stage, generating way more efficient C code (for that indicator variable) for those in many instances, esp. with loops, as these no longer loose type shape information as badly as they did.

The, another level is when it is treated as an object, but known to be int, there are way more helpers used for +/+= and a whole new set of them for comparisons, that in these cases of full or partial type knowledge operate faster.

And even if e.g. only one type is known, this still allows to not make a lot of tests about it, and to avoid attempted shortcuts that cannot work. For 0.6.1 the + and += are pretty well covered for these, but some variants are not yet tuned to take all type knowledge advantage.

These will be also the building block, once the C type layer picks types like “C int or PyObject * known to be int” with indicator flags which values are currently valid to use, then these specialized calls still make sense.

The most attrative level, “C int” has not been reached for 0.6.1 but for my loop example and Python3, I can say that now would be a nice time to start it, as type shape knowledge is all there. This was totally not the case for 0.6.0, but it seems that this step will have to be postponed to another release, maybe 0.6.2, maybe even later.

Week of Bugfixing

But something that bothers me is seeing the issue tracker pile up on actionable items, where I just have not taken action. So as announced on Twitter already, I am having and continue to have bug fixing time. I am acting on issues that are relatively old and easy to act on, or where I have no hope of this happening by anybody else anymore.

I have listed some interesting examples below. But basically these are small, relatively unimportant, yet somewhat import for some use cases things.

Exec on Filehandles

So when doing exec on a filehandle, Nuitka was at runtime reading the source, then compiling it, but forgetting about the filename. This makes things like inspect.getsource() fail on functions from there, and ugly tracebacks not pointing to the filename. This was one of the things which I had understood, but not did the actual work yet.

pkgutil.iter_modules

And another one, which seemed just not done, but turned out to be rather complex, this one needs to populate a sys.path_importer_cache for imported modules, and then to report the child modules. There was no object to carry that information, so now instances of the meta path based importer are associated for every import.

Turns out for Python3, my simplistic type building calling type manually here does not work, as __init__ and iter_modules do not become anything but static methods ever. Needs a real type.

Plus, I had to disable it for now, because mixed packages, like the one we do with multiprocessing where only part is compiled (the one required) and part is pure Python from disk still, stopped to work. The iter_modules it seems will have to cover that case too.

So no luck, postponing this until next week of bug fixes. Frustrating a bit, but such is life.

When to release

There are still some issues that I want to get to. Specifically the OpenGL plugins which has been research ever since, and nobody stepped up, but it’s rather trivial. And the Tcl/Tk for Windows. People have provided sufficient instructions for a plugin that I am going to write this week.

Once I feel the issue tracker is clean, I will release. As a matter of experience, it is then going to grow a lot again.

Google Summer of Code for Nuitka

Finally somebody has stepped up, which means a lot to me. Now to the actual work!

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #10

2018-11-21T06:06:00+00:00

Communication vs. Coding

Recently it was a bit more tough to make that decision. First, there was much going privately, with me ill, then child ill, and ill again, and myself, and that made me have a much harder time to communicate about incomplete things.

Even now, I am torn between fixing issues for 0.6.1 and doing this, but I know that it will take at least one week, so I am missing the point, if I wait for it more.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Hotfixes

There has been another hotfix, 0.6.0.6 and there ought to be one 0.6.0.7, at least on factory there is a bunch of stuff for it, but I didn’t actually do it yet. I was wandering between there will be a release anyway, and the feeling that some of the material may cause regressions, so I might skip on that really.

So for the most fixes, I suspect, develop is going to be the way until next week.

Google Summer of Code for Nuitka

Nobody has stepped up, which means it will not happen unfortunately. This would be your last chance to step up. I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Pythran and Nuitka

As suggested by @wuoulf (Wolf Vollprecht) we had a meeting at the side of the PyCon DE 2018 conference in Karlsruhe, abusing the C++ regular table as a forum for that, which was a very nice experience.

First of all, Wolf is so much more knowledgeable about AnaConda and could point out to me, very important stuff, not the least, that AnaConda contains its own compiler, which I have successfully used since, to first add easier installation instructions for Windows, and second, to successfully statically link with LTO on Linux amd64. Both of which are important for me.

But for Pythran which is limited Python, specialized to translate Numpy API to C++, we showed each other, Nuitka and Pythran details, and somehow in my mind a plan formed how Nuitka could use the Pythran tricks long term, and mid term, how it could include a plugin that will allow to integrate with Pythran compilation.

This was a huge success.

Performance Work

Adding specialized object operations

See last week, this has seen more completion. Both + and += are more or less covered for the selected subset. The CPython test suites were initially not finding uses, but with more and more optimization phase improvements, it challenges code generation with missing ones, and then I added them more and more.

Controlflow Descriptions

Shapes were added for the + and < operation so far, but didn’t influence anything else really but code generation, but of course they should also impact optimization phase.

So the query for type shape has been enhanced to return not only a type shape saying that int+float -> float, but also now an object that describes impact on control flow of the program. This can then say e.g. that this doesn’t execute arbitrary code, and that it does not modify input values, things used in the code generation to avoid error checks, and in the optimization to not have to mark things as unknown.

Preparations for comparison operations

So optimization now also has proper type shape functions for the < and the warnings when they fail to know what to do for concrete types. This allows to actually remove checks, but so far this wasn’t exposed for neither + or for <. Doing this eliminates the exception check for the operation part, where previously it was done if anything in the expression could raise.

Specializing the rich comparisons helper codes is the next step, but so far I didn’t quite get to it yet, but it has been started.

Comparison Conditions

Preparing < optimization for the loop, I noticed that not was optimized for in to become not in, and also is to become is not, etc. but for comparisons, where we can not the result is of bool shape, we can now also switch not < to >= and not = to != of course.

And since our reformulation of while a < b ends up having a statement like if not a < b: break as part of its re-formulation, that is again one step closer to optimizing my example loop.

Local variable escaping

Much to my shock, I noticed that the code which is responsible to handle escaping control flow (i.e. unknown code is executed), was not only doing what it was supposed to do, i.e. mark closure variables as unknown, but more or less did it for all local variables with Python3.

Fixing that allows for a lot more optimization obviously, and makes my test find missing ones, and even bugs in existing ones, that were previously hidden. A good thing to notice this regression (was better once), now that I am looking at concrete examples.

One noticeable sign was that more of my tests failed with warnings about missing code helpers. And another that in my while loop with int increase, it now seems as if Python3 is good. For Python2, the “int or long” shape will need dedicated helpers. That is because ìnt + int becomes either int or long there, where Python3 only has long but renamed it int.

Benchmarks Missing

Speedcenter got repaired, but I need to add the loop examples I am using as test cases before next release, so I can show what Nuitka 0.6.1 will have achieved or at least have improved somewhat already.

But currently these examples only serve as input for general improvements that then take a lot of time, and don’t have immediate impact on their own.

Still would be good to see where Nuitka is standing after each one.

Static Linking

So static linking works now, provided it’s not a pyenv crappy libpython.a but one that can actually work. I got this to work on Linux and using the Conda CC, even LTO will work with it. Interestingly then linking is noticely slow, and I bet ccache and the likes won’t help with that.

I am interested to see what this means for performance impact. But it will allow to address issues, where embedded CPython run time is plain slower than the one that lives in the python binary. For acceleration this is great news.

Conda CC

Using Conda CC by default as a fallback in –mingw mode on Windows is something that was easy to add. So when no other gcc is found, and MSVC is not tried in this mode, and the right directory is added to PATH automatically, with Anaconda, things should now be smoother. It has also its own libpython.a, not sure yet if it’s a static link library, that would be fantastic, but unlike standard MinGW64 we do not have to roll our own at least.

I will try with –lto eventually though and see what it does. But I think static linking on Windows is not supported by CPython, but I am not entirely sure of that.

Annotations Future Feature

Found a 3.7 feature that is not covered by the test suite, the __future__ flag annotations wasn’t working as expected. In this, strings are to be used for __annotations__ where they show up (many are ignored simply) and that requires an unparse function, going from parsed ast (presumably it’s still syntax checked) back to the string, but that was only very hard to get at, and with evil hackery.

For 3.8 a bug fix is promised that will give us the string immediately, but for now my hack must suffice.

MSI files

Following the 3.7.1 release, there are MSI files again, as the regression of 3.7.0 to build them has been fixed in that release. The MSI files will work with 3.7.0 also, just the building was broken.

Overall

So 0.6.1 is in still in full swing in terms of optimization. I think I need to make a release soon, simply because there is too much unreleased, but useful stuff already.

I might have to postpone my goal of C int performance for one example loop until next release. No harm in that. There already are plenty of performance improvements across the board.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #9

2018-10-20T06:17:00+01:00

Communication vs. Coding

My new communication strategy is a full success, engagement with Nuitka is on an all time high.

But the recent weeks more than ever highlighted why I have to force myself to do it. I do not like to talk about unfinished stuff. And right now, there is really a lot of it, almost only it. Also I was ill, and otherwise busy, so this is now late by a week.

But I am keeping it up, and will give an update, despite the feeling that it would be better to just finish a few of those things and then talk about it, but then it will take forever and leave you in the dark. And that is not what is supposed to be.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Hotfixes

So the 0.6.0 release was a huge success, but it definitely wasn’t perfect, and hotfixes were necessary. The latest one 0.6.0.5 was done just yesterday and actually contains one for an important mis-optimization being done, and you ought to update to it from any prior 0.6.0 release.

There are also a few remaining compatibility issues fixed for 3.7 and generally using the latest hotfix is always a good idea.

Kind of what one has to expect from a 0 release, this one also had more expose than usual is seems.

Google Summer of Code for Nuitka

I need more people to work on Nuitka. One way of doing this could be to participate in Google Summer of Code under the Python umbrella. To make that possible, I need you to volunteer as a mentor. So please, please, do.

I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Website Overhaul

I updated the website to recent Nikola and dropped the tag cloud that I was using. Should have cleaner and better looks. Also integrated privacy aware sharing links, where two clicks are necessary to share a page or article like this one on Twitter, Facebook, etc.

Also the download page saw some structural updates and polishing. It should easier to overview now.

Performance Work

Adding specialized object operations

The feedback for performance and the work on 0.6.1 are fully ongoing, and there are many major points that are ongoing. I want to briefly cover each one of them now, but many of them will only have full effect, once everything is in place, which each one is very critical.

So, with the type tracing, objects have known types, and short of using a C type, knowing e.g. that an object is an int, and the other one too, doing + for them can take a lot of advantage avoiding unrelated checks and code paths, even if still using PyObject * at the end of the day.

And even we are only knowing it’s not an int, but say one value is a tuple and the other an unknown, that allows to remove checks for int shortcuts as they can no longer apply. These are tiny optimizations then, but still worthwhile.

To further this, first the inplace operations for a couple of more or less randomly selected types, list, tuple, int, long, str, unicode, bytes, and float, have been looked at and have gotten their own special object based helpers if one or both types are known to be of that kind.

Finding missing specialized object code generation

A report has been added, that will tell when such an operation could have been used, but was not available. This uncovered where typical stuff goes non optimized, a nice principle to see what is actually happening.

So adding list and str would now give a warning, although of course, the optimization phase ought to catch the static raise that is and never let it get there, so this report also addresses missing optimization in an earlier phase.

Optimizing plain object operations too

So the in-place operations were then covered, so this was extended to mere + operations too, the ones that are not in-place. Sometimes, esp. for immutable types, there was already code for that, e.g. int doesn’t really do it, in other cases, list + list code for a quicker concat was added.

And again a report for where it’s missing was added and basic coverage for most of the types. However, in some instances, the optimization doesn’t use the full knowledge yet. But where it does, it will shove off quite a few cycles.

Lack of type knowledge

To apply these things effectively, optimization and value tracing need to know types in the first place. I have found two obstacles for that. One are branch merges. If a branch or both assign to the same type or original type, well the type is changed. Previously it became “unknown” which is treated as object for code generation, and allows nothing really. But now that is better on develop now, and was actually a trivial missing thing.

The other area is loops. Loops put values to unknown when entering loop body, and again when leaving. Essentially making type tracing not effective where it is needed the most to achieve actual performance. Also this was limiting the knowledge for all function to one type to not happening for these kinds of variables that were assigned inside a loop at all.

Took me a while, but I figured out how to build type tracing for loops that works. It currently is still unfinished in my private repo, but passes all tests, I would just like to make it use dedicated interfaces, and clean it up.

I will most likely have that for 0.6.1 too and that should expand the cases where types are known in code generation by a fair amount.

The effect of that will be that more often C code generation will actually see types. Currently e.g. a boolean variable that is assigned in a loop, cannot use the C target type in code generation. Once loop code is merged, it will however take advantage there too. And only then I think adding “C int” as a C type makes sense at all.

Performance regressions vs. CPython

Then another area is performance regressions. So one thing I did early on in the 0.6.1 cycle was using the “module var C target type” to get in-place working for those too. Doing string concatenations on module variables could be slower by an order of magnitude, as could be other operations.

I still need to do it for closure variables too. Then Nuitka will do at least as many of them perfectly as CPython does. It also will be better at it them, because e.g. it doesn’t have to delete from the module dictionary first, due to it never taking a reference, and same applies to the cell. Should be faster for that too.

But strings in-place on these if not optimized, it will look very ugly in terms of worse performance, so 0.6.0 was still pretty bad for some users. This will however hopefully be addressed in 0.6.1 then.

In-place unicode still being bad

Another field was in-place string add for the already optimized case, it was still slower than CPython, and I finally found out what causes this. And that is the using of libpython where PyUnicode_Append is far worse than in the python binary that you normally use, I have see that at least for 3.5 and higher CPython. Analysis showed that e.g. MiniConda had the issue to a much smaller extent, and was being much faster anyway, but probably just has better libpython compilation flags.

So what to do. Ultimately that was to be solved by including a clone of that function, dubbed UNICODE_APPEND that behaves the same, and can even shove off a couple of cycles, by indicating the Python error status without extra checks, and specializing it for the pure unicode += unicode case that we see most often, same for UNICODE_CONCAT for mere +.

Right now the benchmarks to show it do not exist yet. Again something that typically wants me to delay stuff. But as you can imagine, tracking down these hard issues, writing that much code to replace the unicode resizing, is hard enough by itself.

But I hope to convince myself that this will allow to show that for compiled code, things are going to be faster only now.

Benchmarks Missing

In fact, speedcenter as a whole is currently broken, mostly due to Nikola changes that I am trying to work around, but it will take more time apparently and isn’t finished as I write this.

Type shapes in optimization

Another optimization end, is the type shapes of the + operation itself. Right now what is being done is that the shape is derived from the shape of the left argument with the right shape to be considered by it. These also have reports now, for cases where they are missing. So saying e.g. that int + float results in float and these kinds of things, are stuff being encoded there right now.

This is necessary step to e.g. know that int + int -> int_or_long, to make effective loop variable optimization.

Without these, and again, that is a lot of code to write, there is no way to hope for wide spread type knowledge in code generation.

Control flow escape

Something missing there, is to also make it known that + unlike it currently is now, should not in all cases lead to “control flow escape” with the consequence of removing all stuff, and expecting an exception possible, but instead to let the int type also make known that + int ont it not only gives an int_or_long result shape, but also while doing so, that it will never raise an exception (bare MemoryError), and therefore allow more optimization to happen and less and therefore faster code generated.

Until this is done, what is actually going to happen is that while the + result is known, Nuitka will assume control flow escape.

And speaking of that, I think this puts too many variables to a too unknown state. You can to distrust all values, but not the types in this case, so that could be better, but right now it is not. Something else to look into.

Overall

So 0.6.1 is in full swing in terms of optimization. All these ends need a completion, and then I can expect to use advantage of things in a loop, and ultimately to generate C performance code for one example of loop. esp. if we add a C int target type, which currently isn’t yet started, because I think it would barely be used yet.

But we are getting there and I wouldn’t even say we are making small steps, this is all just work to be completed, nothing fundamental about it. But it may take more than one release for sure.

Mind you, there is not only +, there is also -, *, %, and many more operators, all of them will require work. Granted, loop variables tend to use + more often, but any un-optimized operation will immediately loose a lot of type knowledge.

Improved Annotations

There are two kinds of annotations, ones for classes and modules, which actually are stored in a __annotations__ variable, and everything else is mostly just ignored.

So Nuitka got the criterion wrong, and did one thing for functions, and the other for everything else. So that annotations in generators, coroutines and asyncgen ended up with wrong, crashing, and slower code, due to it updating the module __annotations__, so that one is important too if you have to do those.

Release or not

To release or not. There is at least one bug about star imports that affects numpy that is solved in develop, and wasn’t back ported, and I was thinking it only applies to develop, but in fact does to stable. It makes me want to release even before all these optimization things happen and are polished, and I might well decide to go with that.

Maybe I only add the closure in-place stuff and the polish the loop SSA stuff, and then call it a release. It already will solve a lot of performance issues that exist right now, while staging the ground for more.

Standalone Improvements

Standalone work is also improving. Using pyi files got more apt, and a few things were added, all of which make sense to be used by people.

But I also have a backlog of issues there however. I will schedule one sprint for those I guess, where I focus on these. I am neglecting those somewhat recently.

Caching Examined

For the static code, I now noticed that it’s compiled for each target name, due to the build directory being part of the object file for debug. For gcc 8 there is an option to allow pointing at the original static C file location, and then ccache is more effective, because object files will be the same.

That’s actually pretty bad, as most of my machines are on gcc-6 and makes me think that libnuitka.a is really more of an requirement than ever. I might take some time to get this sorted out.

Python3 deprecation warnings

So Nuitka supports the no_warnings Python flag, and for a long time I have been annoyed at how it was not working for Python3 in some cases. The code was manually settign filters, but these would get overridden by CPython test suites testing warnings. And the code said that there is no CPython C-API to control it, which is just plain wrong.

So I changed that and it became possible to remove lots of ignore_stderr annotations in CPython test suites, and more importantly, I can stop adding them for when running older/newer CPython version with a suite.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #8

2018-10-02T06:05:00+01:00

Public / Private CI / Workflow

Note

I wrote this as part of a discussion recently, and I think it makes sense to share it here. This is a lot text though, feel free to skip forward.

Indeed I have a private repo, where I push and only private CI picks up. Based on Buildbot, I run many more compilations, basically around the clock on all of my computers, to find regressions from new optimization or codegen changes, and well UI changes too.

Public CI offerings like Travis are not aimed at allowing this many compilations. It will be a while before public cloud infrastructure will be donated to Nuitka, although I see it happening some time in the future. This leaves developers with the burden to run tests on their own hardware, and never enough. Casual contributors will never be able to do it themselves.

My scope is running the CPython test suites on Windows and Linux. These are the adapted 26, 27, 32, 33, 34, 35, 36, 37 suites, and also to get even more errors covered, they are ran with mismatching Python versions, so a lot of exceptions are raised. Often running the 36 tests with 37 and vice versa will extend the coverage, because of the exceptions being raise.

On Windows I compile with and without debug mode, x86 and x64, and it’s kind of getting too much. For Linux I have 2 laptops in use, and an ARM CuBox bought from your donations, there it’s working better, esp. due to ccache being used everywhere, although recent investigations show room for improvement there as well.

For memory usage I still compile mercurial and observe the memory it used in addition to comparing the mercurial tests to expected outputs its test suite gives. It’s a sad day when Mercurial tests find changes in behavior, and luckily that has been rare. Running the Mercurial test suite gives some confidence in the thing not corrupting data it works with without knowing.

Caching the CPython outputs of tests to compare against is something I am going to make operational these days, trying to make things ever faster. There is no point to re-run tests with Python, just to get at its output, which will typically not change at all.

But for the time being, ccache.exe and clcache.exe seem to have done wonders for Windows too, but I will want to investigate some more to avoid unnecessary cache misses.

Workflow

As for my workflow with Nuitka, I often tend to let some commits settle in my private repo only until they become trusted. Other times I will make bigger changes and put them out to factory immediately, because it will be hard to split up the changes later, so putting them out makes it easier.

I am more conservative with factory right after telling people to try something there. But also I break it on purpose, just trying out something. I really consider it a private branch for interacting with me or public CI. I do not recommend to use it, and it’s like a permanent pull request of mine that is not ever going to be finished.

Then on occasions I am making a sorting of all commits on factory and split it into some things that become hotfixes, some things that become current pre-release, and other things that will remain in that proving ground. That is why I typically make hotfix and pre-release at the same times. The git flow suggests doing that and it’s easy, so why not. As a bonus, develop is then practically stable at nearly all times too, with hardly any regressions.

I do however normally not take things as hotfixes that are on develop already, I hate the duplication of commits. Hotfixes must be small and risk free, and easy to put out, when there is any risk, it definitely will be on develop. Nuitka stable typically covers nearly all grounds already. No panic needed to add missing stuff and break others.

Hunting bugs with bisect

For me the git bisect is very important. My private commit history is basically a total mess and worthless, but on factory I am making very nice organized commits that I will frequently amend, even for the random PyLint cleanup. This allows me when e.g. one test suddenly says “segfault” on Windows to easily find the change that triggers it, look at C code difference, and spot the bug introduced, then amend the commit and be done with it.

It’s amazing how much time this can save. My goal is to always have a workable state which is supposed to pass all tests. Obviously I cannot prove it for every commit, but when I know it to not be the case, I tend to make rebases. At times I have been tempted and followed up on backward amending develop and even stable.

I am doing that to be sure to have that bisect ability, but fortunately it’s rare that kind of bug occurs, and I try not to do it.

Experimental Changes

As with recent changes, I sometimes make changes with the isExperimental() marker, activating breaking changes only gradually. The C bool type code generation has been there for months in a barely useful form, until it became more polished, and always guarded with a switch, until one day for 0.6 finally I changed it, and made the necessary fixes retroactively before that switch, to make it work while that was still in factory.

Then I will remove the experimental code. I feel it’s very important and even ideal to be able to always compare outputs to a fully working solution. I am willing to postpone some cleanups until later date as a price, but when then something in my mind tells me again “This cannot possibly have ever worked”… a command line flag away, I have the answer to compare, plus, that includes extra changes happened in the meantime, they don’t add noise to diff outputs of generated C code for example.

Then looking at that diff, I can tell where the unwanted effect is, and fix all the things, and that way find bugs much faster.

Even better, if I decide to make a cleanup action as part of making a change more viable to execute, then I get to execute it on stable grounds, covered by the full test suite. I can complete that cleanup, e.g. using variable identifier objects instead of mere strings was needed to make “heap generators” more workable. But I was able to put that one to active before “heap generators” was ever fully workable, and complete it, and actually reap some of its benefits already.

Hardware

Obviously this takes a lot of hardware and CPU to be able to compile this much Python code on a regular basis. And I really wish I could add one of the new AMD Threadripper 2 to the mix. Anybody donating one to me? Yes I know, I am only dreaming. But it would really help the cause.

Milestone Release

So the 0.6 is out, and already a hotfix that addresses mostly use cases of people that didn’t work. More people seemed to have tried out 0.6.0 and as a result 0.6.0.1 is going to cover a few corner cases. So far I have not encountered a single regression of 0.6.0, but instead it contained ones for 0.5.33 which did have one that was not easy to fix.

So that went really smooth.

UI rework

The UI needs more work still. Specifically that packages do not automatically include all stuff below them and have to be specified by file path instead of by name, is really annoying to me.

But I had delayed 0.6 for some UI work, and the quirks are to remain some. I will work on these things eventually.

Benchmarks

So I updated the website to state that PyStone is now 312% faster, from a number that was very old. I since then ran it with an updated version for Python3, and it’s much less there. That is pretty sad.

I will be looking into that for 0.6.1 release, or I will have to update the wording to provide 2 numbers there, because it seems for Python3 performance with Nuitka it might be misleading.

Something with unicode strings and in-place operations is driving me crazy. Nuitka is apparently slower for that, and I can’t point where that is happening exactly. It seems internally unicode objects are maybe put into a different state from some operations, which then making in-place extending in realloc fail more often, but I cannot know yet.

Inplace Operations

So more work has been put into those, adding more specialization, and esp. also applying them for module variables as well. CPython can do that, and actually is giving itself a hard time about it, and Nuitka should be doing this much clever with its more static knowledge.

But I cannot tell you how much scratching my head was wasted debugging that. I was totally stupid about how I approached that, looking from the final solution, it was always easy. Just not for me apparently.

New use cases

Talked about those above. So the top level logging module of your own was working fine in accelerated mode, but for standalone it failed and used the one from standard library instead. That kind of shadowing happened because Nuitka was going from module objects to their names and back to objects, which are bad in case of duplicates. That is fixed for develop, and one of those risk cases, where it cannot be a hotfix because it touched too much.

Then pure Python3 packages need not have __init__.py and so far that was best working for sub-packages, but after 0.6.0.1 hotfix, now it will also work for the main module you compile to be that empty.

Tcl/Tk Standalone

So instructions have been provided how to properly make that work for Python standalone on Windows. I have yet to live up to my promise and make Nuitka automatically include the necessary files. I hope to do it for 0.6.1 though.

Caching Examined

So I am looking at ccache on Linux right now, and found e.g. that it was reporting that gcc --version was called a lot at startup of Scons and then g++ --version once. The later is particularly stupid, because we are not going to use g++ normally, except if gcc is really old and does not support C11. So in case a good one was found, lets disable that version query and not do it.

And for the gcc version output, monkey patching scons to a version of getting that output that caches the result, removes those unnecessary forks.

So ccache is being called less frequently, and actually these --version outputs appears to actually take measurable time. It’s not dramatic, but ccache was apparently getting locks, and that’s worth avoiding by itself.

That said, the goal is for ccache and clcache to make them both report their effectiveness of cache usage after the end of a test suite run. That way I am hoping to notice and be able to know, if caching is used to its full effect.

Twitter

I continue to be very active there. I put out a poll about the comment system, and disabling Disqus comments as a result, I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Plans

Working on the 0.6.1 release, attacking more in-place add operations as a first goal, and now turning to binary operations, I am trying to shape how using different helper functions to different object types looks like. And to gain performance without C types. But ultimately the same issue will arise there, what to do with mixed input types.

My desire is for in-place operations to fully catch up with CPython, as these can easily loose a lot of performance. Closure variables and their cells are another target to pick on, and I feel they ought to be next after module ones are now working, because also their solution ought to be very similar. Then showing that depending on target storage, local, closure, or module, is then faster in all cases would be a goal for the 0.6.1 release.

This feels not too far away, but we will see. I am considering next weekend for release.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.0

2018-09-27T18:24:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release adds massive improvements for optimization and a couple of bug fixes.

It also indicates reaching the mile stone of doing actual type inference, even if only very limited.

And with the new version numbers, lots of UI changes go along. The options to control recursion into modules have all been renamed, some now have different defaults, and finally the filenames output have changed.

Bug Fixes

Python3.5: Fix, the awaiting flag was not removed for exceptions thrown into a coroutine, so next time it appeared to be awaiting instead of finished.
Python3: Classes in generators that were using built-in functions crashed the compilation with C errors.
Some regressions for XML outputs from previous changes were fixed.
Fix, hasattr was not raising an exception if used with non-string attributes.
For really large compilations, MSVC linker could choke on the input file, line length limits, which is now fixed for the inline copy of Scons.
Standalone: Follow changed hidden dependency of PyQt5 to PyQt5.sip for newer versions
Standalone: Include certificate file using by requests module in some cases as a data file.

Optimization

Enabled C target type nuitka_bool for variables that are stored with boolean shape only, and generate C code for those
Using C target type nuitka_bool many more expressions are now handled better in conditions.
Enhanced is and is not to be C source type aware, so they can be much faster for them.
Use C target type for bool built-in giving more efficient code for some source values.
Annotate the not result to have boolean type shape, allowing for more compile time optimization with it.
Restored previously lost optimization of loop break handling StopIteration which makes loops much faster again.
Restore lost optimization of subscripts with constant integer values making them faster again.
Optimize in-place operations for cases where left, right, or both sides have known type shapes for some values. Initially only a few variants were added, but there is more to come.
When adjacent parts of an f-string become known string constants, join them at compile time.
When there is only one remaining part in an f-string, use that directly as the result.
Optimize empty f-strings directly into empty strings constant during the tree building phase.
Added specialized attribute check for use in re-formulations that doesn’t expose exceptions.
Remove locals sync operation in scopes without local variables, e.g. classes or modules, making exec and the like slightly leaner there.
Remove try nodes that did only re-raise exceptions.
The del of variables is now driven fully by C types and generates more compatible code.
Removed useless double exception exits annotated for expressions of conditions and added code that allows conditions to adapt themselves to the target shape bool during optimization.

New Features

Added support for using .egg files in PYTHONPATH, one of the more rare uses, where Nuitka wasn’t yet compatible.
Output binaries in standalone mode with platform suffix, on non-Windows that means no suffix. In accelerated mode on non-Windows, use .bin as a suffix to avoid collision with files that have no suffix.
Windows: It’s now possible to use clang-cl.exe for CC with Nuitka as a third compiler on Windows, but it requires an existing MSVC install to be used for resource compilation and linking.
Windows: Added support for using ccache.exe and clcache.exe, so that object files can now be cached for re-compilation.
For debug mode, report missing in-place helpers. These kinds of reports are to become more universal and are aimed at recognizing missed optimization chances in Nuitka. This features is still in its infancy. Subsequent releases will add more like these.

Organizational

Disabled comments on the web site, we are going to use Twitter instead, once the site is migrated to an updated Nikola.
The static C code is now formatted with clang-format to make it easier for contributors to understand.
Moved the construct runner to top level binary and use it from there, with future changes coming that should make it generally useful outside of Nuitka.
Enhanced the issue template to tell people how to get the develop version of Nuitka to try it out.
Added documentation for how use the object caching on Windows to the User Manual.
Removed the included GUI, originally intended for debugging, but XML outputs are more powerful anyway, and it had been in disrepair for a long time.
Removed long deprecated options, e.g. --exe which has long been the default and is no more accepted.
Renamed options to include plugin files to --include-plugin-directory and --include-plugin-files for more clarity.
Renamed options for recursion control to e.g. --follow-imports to better express what they actually do.
Removed --python-version support for switching the version during compilation. This has only worked for very specific circumstances and has been deprecated for a while.
Removed --code-gen-no-statement-lines support for not having line numbers updated at run time. This has long been hidden and probably would never gain all that much, while causing a lot of incompatibilty.

Cleanups

Moved command line arguments to dedicated module, adding checks was becoming too difficult.
Moved rich comparison helpers to a dedicated C file.
Dedicated binary and unary node bases for clearer distinction and more efficient memory usage of unuary nodes. Unary operations also no longer have in-place operation as an issue.
Major cleanup of variable accesses, split up into multiple phases and all including module variables being performed through C types, with no special cases anymore.
Partial cleanups of C type classes with code duplications, there is much more to resolve though.
Windows: The way exec was performed is discouraged in the subprocess documentation, so use a variant that cannot block instead.
Code proving information about built-in names and values was using not very portable constructs, and is now written in a way that PyPy would also like.

Tests

Avoid using 2to3 for basic operators test, removing test of some Python2 only stuff, that is covered elsewhere.
Added ability to cache output of CPython when comparing to it. This is to allow CI tests to not execute the same code over and over, just to get the same value to compare with. This is not enabled yet.

Summary

This release marks a point, from which on performance improvements are likely in every coming release. The C target types are a major milestone. More C target types are in the work, e.g. void is coming for expressions that are done, but not used, that is scheduled for the next release.

Although there will be a need to also adapt optimization to take full advantage of it, progress should be quick from here. There is a lot of ground to cover, with more C types to come, and all of them needing specialized helpers. But as soon as e.g. int, str are covered, many more programs are going to benefiting from this.

Nuitka this week #7

2018-09-22T11:05:00+01:00

Nuitka Design Philosophy

Note

I wrote this as part of a discussion recently, and I think it makes sense to share my take on Nuitka and design. This is a lot text though, feel free to skip forward.

The issue with Nuitka and design mainly for me is that the requirements for many parts were and are largely unknown to me, until I actually start to do it.

My goto generators approach worked out as originally designed, and that felt really cool for once, but the whole “C type” thing was a total unknown to me, until it all magically took form.

But rather I know it will evolve further if I go from “bool” (complete and coming for 0.6.0) via “void” (should be complete already, but enabling will happen only for 0.6.1 likely) to “int”, not sure how long that will take.

I really think Nuitka, unlike other software that I have designed, is more of a prototype project that gradually turns more and more into the real thing.

I have literally spent years to inject proper design in steps into the optimization phase, what I call SSA, value tracing, and it is very much there now. I am probably going to spend similar amounts of time, to execute on applying type inference results to the code generation.

So I turned that into something working with code strings to something working with variable declaration objects knowing their type for the goto generators, aiming at C types generally. All the while carrying the full weight of passing every compatibility test there is.

Then e.g. suddenly cleaning up module variables to no longer have their special branch, but a pseudo C type, that makes them like everything else. Great. But when I first introduced the new thing, I postponed that, because I could sooner apply its benefits to some things and get experience from it.

While doing partial solutions, the design sometimes horribly degrades, but only until some features can carry the full weight, and/or have been explored to have their final form.

Making a whole Nuitka design upfront and then executing it, would instead give a very high probability of failing in the real world. I am therefore applying the more agile approach, where I make things work first. And then continue to work while I clean it up.

For every feature I added, I actively go out, and change the thing, that made it hard or even fail. Always. I think Nuitka is largely developed by cleanups and refactoring. Goto generators were a fine example of that, solving many of the issues by injecting variable declarations objects into code generation, made it easy to indicate storage (heap or object or stack) right there.

That is not to say that Nuitka didn’t have the typical compiler design. Like parsing inputs, optimizing a tree internally, producing outputs. But that grand top level design only tells you the obvious things really and is stolen anyway from knowing similar projects like gcc.

There always were of course obvious designs for Nuitka, but that really never was what anybody would consider to make a Python compiler hard. But for actual compatibility of CPython, so many details were going to require examination with no solutions known ahead of time.

I guess, I am an extreme programmer, or agile, or however they call it these days. At least for Nuitka. In my professional life, I have designed software for ATC on the drawing board, then in paper, and then in code, the design just worked, and got operational right after completion, which is rare I can tell you.

But maybe that is what keeps me exciting about Nuitka. How I need to go beyond my abilities and stable ground to achieve it.

But the complexity of Nuitka is so dramatically higher than anything I ever did. It is doing a complicated, i.e. detail rich work, and then it also is doing hard jobs where many things have to play together. And the wish to have something working before it is completed, if it ever is, makes things very different from projects I typically did.

So the first version of Nuitka already had a use, and when I publicly showed it first, was capable of handling most complex programs, and the desire was to evolve gradually.

I think I have described this elsewhere, but for large parts of the well or bad designed solutions of Nuitka, there is reliable ways of demonstrating it works correctly. Far better than I have ever encountered. i believe it’s the main reason I managed to get this off the ground is that. Having a test “oracle” is what makes Nuitka special, i.e. comparing to existing implementations.

Like a calculator can be tested comparing it to one of the many already perfect ones out there. That again makes Nuitka relatively easy despite the many details to get right, there is often an easy way to tell correct from wrong.

So for me, Nuitka is on the design level, something that goes through many iterations, discovery, prototyping, and is actually really exciting in that.

Compilers typically are boring. But for Nuitka that is totally not the case, because Python is not made for it. Well, that*s technically untrue, lets say not for optimizing compilers, not for type inference, etc.

UI rework

Following up on discussion on the mailing list, the user interface of Nuitka will become more clear with --include-* options and --[no]follow-import* options that better express what is going to happen.

Also the default for following with extension modules is now precisely what you say, as going beyond what you intend to deliver makes no sense in the normal case.

Goto Generators

Now release as 0.5.33 and there has been little regressions so far, but the one found is only in the pre-release of 0.6.0 so use that instead if you encounter a C compilation error.

Benchmarks

The performance regressions fixed for 0.6.0 impact pystone by a lot, loops were slower, so were subscripts with constant integer indexes. It is a pity these were introduced in previous releases during refactorings without noticing.

We should strive to have benchmarks with trends. Right now Nuitka speedcenter cannot do it. Focus should definitely go to this. Like I said, after 0.6.0 release, this will be a priority, to make them more useful.

Twitter

I continue to be active there. I just put out a poll about the comment system, and disabling Disqus comments I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Egg files in PYTHONPATH

This is a relatively old issue that now got addressed. Basically these should be loaded from for compilation. Nuitka now unpacks them to a cache folder so it can read source code from them, so this apparently rare use case works now, yet again improving compatibility.

Will be there for 0.6.0 release.

Certifi

Seems request module sometimes uses that. Nuitka now includes that data file starting with 0.6.0 release.

Compatibility with pkg_resources

It seems that getting “distributions” and taking versions from there, is really a thing, and Nuitka fails pkg_resources requirement checks in standalone mode at least, and that is of course sad.

I am currently researching how to fix that, not sure yet how to do it. But some forms of Python installs are apparently very affected by it. I try looking into its data gathering, maybe compiled modules can be registered there too. It seems to be based on file system scans of its own makings, but there is always a monkey patch possible to make it better.

Plans

Still working on the 0.6.0 release, cleaning up open ends only. Release tests seem to be pretty good looking. The UI changes and stuff are a good time to be done now, but delay things, and there is a bunch of small things that are low hanging fruits while I wait for test results.

But since it fixes so many performance things, it really ought to be out any day now.

Also the in-place operations stuff, I added it to 0.6.0 too, just because it feels very nice, and improves some operations by a lot too. Initially I had made a cut for 0.6.1 already, but that is no more.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.5.33

2018-09-14T19:54:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a bunch of fixes, most of which were previously released as part of hotfixes, and important new optimization for generators.

Bug Fixes

Fix, nested functions with local classes using outside function closure variables were not registering their usage, which could lead to errors at C compile time. Fixed in 0.5.32.1 already.
Fix, usage of built-in calls in a class level could crash the compiler if a class variable was updated with its result. Fixed in 0.5.32.1 already.
Python 3.7: The handling of non-type bases classes was not fully compatible and wrong usages were giving AttributeError instead of TypeError. Fixed in 0.5.32.2 already.
Python 3.5: Fix, await expressions didn’t annotate their exception exit. Fixed in 0.5.32.2 already.
Python3: The enum module usages with __new__ in derived classes were not working, due to our automatic staticmethod decoration. Turns out, that was only needed for Python2 and can be removed, making enum work all the way. Fixed in 0.5.32.3 already.
Fix, recursion into __main__ was done and could lead to compiler crashes if the main module was named like that. This is not prevented. Fixed in 0.5.32.3 already.
Python3: The name for list contraction’s frames was wrong all along and not just changed for 3.7, so drop that version check on it. Fixed in 0.5.32.3 already.
Fix, the hashing of code objects has creating a key that could produce more overlaps for the hash than necessary. Using a C1 on line 29 and C on line 129, was considered the same. And that is what actually happened. Fixed in 0.5.32.3 already.
macOS: Various fixes for newer Xcode versions to work as well. Fixed in 0.5.32.4 already.
Python3: Fix, the default __annotations__ was the empty dict and could be modified, leading to severe corruption potentially. Fixed in 0.5.32.4 already.
Python3: When an exception is thrown into a generator that currently does a yield from is not to be normalized.
Python3: Some exception handling cases of yield from were leaking references to objects. Fixed in 0.5.32.5 already.
Python3: Nested namespace packages were not working unless the directory continued to exist on disk. Fixed in 0.5.32.5 already.
Standalone: Do not include icuuc.dll which is a system DLL. Fixed in 0.5.32.5 already.
Standalone: Added hidden dependency of newer version of sip. Fixed in 0.5.32.5 already.
Standalone: Do not copy file permissions of DLLs and extension modules as that makes deleting and modifying them only harder. Fixed in 0.5.32.6 already.
Windows: The multiprocessing plugin was not always properly patching the run time for all module loads, made it more robust. Fixed in 0.5.32.6 already.
Standalone: Do not preserve permissions of copied DLLs, which can cause issues with read-only files on Windows when later trying to overwrite or remove files.
Python3.4: Make sure to disconnect finished generators from their frames to avoid potential data corruption. Fixed in 0.5.32.6 already.
Python3.5: Make sure to disconnect finished coroutines from their frames to avoid potential data corruption. Fixed in 0.5.32.6 already.
Python3.6: Make sure to disconnect finished asyncgen from their frames to avoid potential data corruption. Fixed in 0.5.32.6 already.
Python3.5: Explicit frame closes of frames owned by coroutines could corrupt data. Fixed in 0.5.32.7 already.
Python3.6: Explicit frame closes of frames owned by asyncgen could corrupt data. Fixed in 0.5.32.7 already.
Python 3.4: Fix threaded imports by properly handling _initializing in compiled modules __spec__ attributes. Before it happen that another thread attempts to use an unfinished module. Fixed in 0.5.32.8 already.
Fix, the options --include-module and --include-package were present but not visible in the help output. Fixed in 0.5.32.8 already.
Windows: The multiprocessing plugin failed to properly pass compiled functions. Fixed in 0.5.32.8 already.
Python3: Fix, optimization for in-place operations on mapping values are not allowed and had to be disabled. Fixed in 0.5.32.8 already.
Python 3.5: Fixed exception handling with coroutines and asyncgen throw to not corrupt exception objects.
Python 3.7: Added more checks to class creations that were missing for full compatibility.
Python3: Smarter hashing of unicode values avoids increased memory usage from cached converted forms in debug mode.

Organizational

The issue tracker on GitHub is now the one that should be used with Nuitka, winning due to easier issue templating and integration with pull requests.
Document the threading model and exception model to use for MinGW64.
Removed the enum plug-in which is no longer useful after the improvements to the staticmethod handling for Python3.
Added Python 3.7 testing for Travis.
Make it clear in the documentation that pyenv is not supported.
The version output includes more information now, OS and architecture, so issue reports should contain that now.
On PyPI we didn’t yet indicated Python 3.7 as supported, which it of course is.

New Features

Added support for MiniConda Python.

Optimization

Using goto based generators that return from execution and resume based on heap storage. This makes tests using generators twice as fast and they no longer use a full C stack of 2MB, but only 1K instead.
Conditional a if cond else b, a and b, a or b expressions of which the result value is are now transformed into conditional statements allowing to apply further optimizations to the right and left side expressions as well.
Replace unused function creations with side effects from their default values with just those, removing more unused code.
Put all statement related code and declarations for it in a dedicated C block, making things slightly more easy for the C compiler to reuse the stack space.
Avoid linking against libpython in module mode on everything but Windows where it is really needed. No longer check for static Python, not needed anymore.
More compact function, generator, and asyncgen creation code for the normal cases, avoid qualname if identical to name for all of them.
Python2 class dictionaries are now indeed directly optimized, giving more compact code.
Module exception exits and thus its frames have become optional allowing to avoid some code for some special modules.
Uncompiled generator integration was backported to 3.4 as well, improving compatibility and speed there as well.

Cleanups

Frame object and their cache declarations are now handled by the way of allocated variable descriptions, avoid special handling for them.
The interface to “forget” a temporary variable has been replaced with a new method that skips a number for it. This is done to keep expression use the same indexes for all their child expressions, but this is more explicit.
Instead of passing around C variables names for temporary values, we now have full descriptions, with C type, code name, storage location, and the init value to use. This makes the information more immediately available where it is needed.
Variable declarations are now created when needed and stored in dedicated variable storage objects, which then in can generate the code as necessary.
Module code generation has been enhanced to be closer to the pattern used by functions, generators, etc.
There is now only one spot that creates variable declaration, instead of previous code duplications.
Code objects are now attached to functions, generators, coroutines, and asyncgen bodies, and not anymore to the creation of these objects. This allows for simpler code generation.
Removed fiber implementations, no more needed.

Tests

Finally the asyncgen tests can be enabled in the CPython 3.6 test suite as the corrupting crash has been identified.
Cover ever more cases of spurious permission problems on Windows.
Added the ability to specify specific modules a comparison test should recurse to, making some CPython tests follow into modules where actual test code lives.

Summary

This release is huge in many ways.

First, finishing “goto generators” clears an old scalability problem of Nuitka that needed to be addressed. No more do generators/coroutines/asyncgen consume too much memory, but instead they become as lightweight as they ought to be.

Second, the use of variable declarations carying type information all through the code generation, is an important pre-condition for “C types” work to resume and become possible, what will be 0.6.0 and the next release.

Third, the improved generator performance will be removing a lot of cases, where Nuitka wasn’t as fast, as its current state not using “C types” yet, should allow. It is now consistently faster than CPython for everything related to generators.

Fourth, the fibers were a burden for the debugging and linking of Nuitka on various platforms, as they provided deprecated interfaces or not. As they are now gone, Nuitka ought to definitely work on any platform where Python works.

From here on, C types work can take it, and produce the results we are waiting for in the next major release cycle that is about to start.

Also the amount of fixes for this release has been incredibly high. Lots of old bugs esp. for coroutines and asyncgen have been fixed, this is not only faster, but way more correct. Mainly due to the easier debugging and interface to the context code, bugs were far easier to avoid and/or find.

Nuitka this week #6

2018-09-12T12:05:00+01:00

Holiday

In my 2 weeks holiday, I indeed focused on a really big thing, and got more done that I had hoped for. For C types, nuitka_bool, which is a tri-state boolean with true, false and unassigned, can be used for some variables, and executes some operations without going through objects anymore.

bool

Condition codes are no longer special. They all need a boolean value from the expression used as a condition, and there was a special paths for some popular expressions for conditions, but of course not all. That is now a universal thing, conditional statement/expressions will now simply ask to provide a temp variable of value nuitka_bool and then code generation handles it.

For where it is used, code gets a lot lighter, and of course faster, although I didn’t measure it yet. Going to Py_True/Py_False and comparing with it, wasn’t that optimal, and it’s nice this is now so much cleaner as a side effect of that C bool work.

This seems to be so good, that actually it’s the default for this to be used in 0.6.0, and that itself is a major break through. Not so much for actual performance, but for structure. Other C types are going to follow soon and will give massive performance gains.

void

And what was really good, is that not only did I get bool to work almost perfectly, I also started work on the void C target type and finished that after my return from holiday last weekend, which lead to new optimization that I am putting in the 0.5.33 release that is coming soon, even before the void code generation is out.

The void C type cannot read values back, and unused values should not be used, so this gives errors for cases where that becomes obvious.

a or b

Consider this expression. The or expression, that one is going to producing a value, which is then released, but not used otherwise. New optimzation creates a conditional statement out of it, which takes a as the condition and if not true, then evaluates b but ignores it.

if not a:
    b

The void evaluation of b can then do further optimization for it.

Void code generation can therefore highlight missed opportunities for this kid of optimization, and found a couple of these. That is why I was going for it, and I feel it pays off. Code generation checking optimization here, is a really nice synergy between the two.

Plus I got all the tests to work with it, and solved the missing optimizations it found very easily. And instead of allocating an object now, not assigning is often creating more obvious code. And that too allowed me to find a couple of bugs by C compiler warnings.

Obviously I will want to run a compile all the world test before making it the default, which is why this will probably become part of 0.6.1 to be the default.

module_var

Previously variable codes were making a hard distinction for module variables and make them use their own helper codes. Now this is encapsulated in a normal C type class like nuitka_bool, or the one for PyObject * variables, and integrates smoothly, and even got better. A sign things are going smooth.

Goto Generators

Still not released. I delayed it after my holiday, and due to the heap generator change, after stabilizing the C types work, I want to first finish a tests/library/compile_python_module.py resume run, which will for a Anaconda3 compile all the code found in there.

Right now it’s still doing that, and even found a few bugs. The heap storage can still cause issues, as can changes to cloning nodes, which happens for try nodes and their finally blocks.

This should finish these days. I looked at performance numbers and found that develop is indeed only faster, and factory due to even more optimization will be yet faster, and often noteworthy.

Benchmarks

The Speedcenter of Nuitka is what I use right now, but it’s only showing the state of 3 branches and compared to CPython, not as much historical information. Also the organization of tests is poor. At least there is tags for what improved.

After release of Nuitka 0.6.0 I will show more numbers, and I will start to focus on making it easier to understand. Therefore no link right now, google if you are so keen. ;-)

Twitter

During the holiday sprint, and even after, I am going to Tweet a lot about what is going on for Nuitka. So follow me on twitter if you like, I will post important stuff as it happens there:

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Poll on Executable Names

So I put e.g. poll up on Twitter, which is now over. But it made me implement a new scheme, due to popular consensus

Hotfixes

Even more hotfixes. I even did 2 during my holiday, however packages built only later.

Threaded imports on 3.4 or higher of modules were not using the locking they should use. Multiprocessing on Windows with Python3 had even more problems, and the --include-package and --include-module were present, but not working.

That last one was actually very strange. I had added a new option group for them, but not added it to the parser. Result: Option works. Just does not show up in help output. Really?

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very like one you can help with.

Nuitka definitely needs more people to work on it.

Plans

Working down the release backlog. Things should be out. I am already working on what should become 0.6.1, but it’s not yet 0.5.33 released. Not a big deal, but 0.6.0 has 2 really important fixes for performance regressions that have happened in the past. One is for loops, making that faster is probably like the most important one. The other for constant indexing, probably also very important. Very much measurable in pystone at least.

In the mean time, I am preparing to get int working as a target C type, so e.g. comparisons of such values could be done in pure C, or relatively pure C.

Also, I noticed that e.g. in-place operations can be way more optimized and did stuff for 0.6.1 already in this domain. That is unrelated to C type work, but kind of follows a similar route maybe. How to compare mixed types we know of, or one type only. That kind of things needs ideas and experiments.

Having int supported should help getting some functions to C speeds, or at least much closer to it. That will make noticeable effects in many of the benchmarks. More C types will then follow one by one.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #5

2018-08-24T12:55:00+01:00

Goto Generators

Finished. Done. Finally.

Benchmarking was exciting. One program benchmark I had run in the past, was twice as fast than before, showing that the new implementation is indeed much faster, which is fantastic news.

Creating generator expressions and using them both got substantially faster and that is great.

It took me a fair amount of time to debug coroutines and asyncgen based on the new goto implementation. But the result is really good, and a fair amount of old bugs have been fixed. There always had been a segfault with asyncgen test that now has been eradicated.

One major observation is now, with only one C stack, debugging got a lot easier before, where context switches left much of the program state not reachable.

Benchmarks

Posted this one Twitter already:

Nuitka Speedcenter Builtin sum with generator

That one construct test has been a problem child, where Nuitka was slower than CPython 2.x, and very little faster than 3.x, and now with goto generators finally has become consistently faster.

I will explain what you see there in the next issue. The short version is that there is code, in which for one run, one line is used, and in another the other line is used, and then the “construct” is measure that way, by making the delta of the two. That construct performance is then compared between Python and Nuitka.

So if e.g. Nuitka is already better at looping, that won’t influence the number of making that sum call with a generator expression.

The alternative line uses the generator expression, to make sure the construction time is not counted. To measure that, there is another construct test, that just creates it.

Nuitka Speedcenter Generator Expression Creation

This one shows that stable Nuitka was already faster at creating them, but that the develop version got even faster again. As creating generator objects became more lightweight, that is also news.

There are constructs for many parts of Python, to shed a light on how Nuitka fares for that particular one.

Holiday

In my 2 weeks holiday, I will try and focus on the next big thing, C types, something also started in the past, and where recent changes as part of the heap storage, should make it really a lot easier to get it finished. In fact I don’t know right now, why my bool experimental work shouldn’t just prove to be workable.

I am not going to post a TWN issue next week, mostly because my home servers won’t be running, and the static site is rendered on one of them. Of course that would be movable, but I won’t bother.

I am going to post a lot on Twitter though.

Static Compilation

There is a GitHub issue where I describe how pyenv on macOS ought to be possible to use, and indeed, a brave soul has confirmed and even provided the concrete commands. All it takes now is somebody to fit this into the existing caching mechanism of Nuitka and to make sure the static library is properly patched to work with these commands.

Now is anyone of you going to create the code that will solve it for good?

Twitter

Follow me on twitter if you like, I will post important stuff as it happens there:

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Hotfixes

And there have been yet again more hotfixes. Some are about coroutine and asyncgen corruptions for closes of frames. Multiprocessing plugin on Windows will work in all cases now.

Noteworthy was that the “0.5.32.6” was having a git merge problem on the cherry-pick that git didn’t tell me about, leading to crashes. That made it necessary to push an update right after. I was confused that I didn’t get a conflict, because there was one. But I am to blame for not checking the actual diff.

Bug Tracker

The next release will make GitHub the official tracker for Nuitka issues. I am working down the issues on the old tracker. The web site already pointed users there for a while, and I was set on this for some time, but yesterday I focused on taking action.

Basically what won me over is the easier templating of issues and pull requests that would have been possible with Roundup, but never happened. Also the OpenID integration that bugs.python.org has, never became available to me in a ready usable form.

Issue Backlog

Finishing goto “generators allowed” for around 10 issues to be closed alone, and I went over things, and checked out some stale issues, to see if they are dealt with, or pinging authors. I spent like half a day on this, bring down the issue count by a lot. Tedious work, but must be done too.

Also my inbox got a fair amount of cleanup, lots of issues pile up there, and from time to time, I do this, to get things straight. I raised issues for 2 things, that I won’t be doing immediately.

But actually as issues go, there really very little problematic stuff open right now, and nothing important really. I would almost call it issue clean.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very like one you can help with.

Nuitka definitely needs more people to work on it.

Plans

The goto generator work could be released, but I want to make the compile all the world test before I do so. It is running right now, but I will not complete before I leave. Also I do not want to get regression reports in my holiday, and goto generators along with heap storage, mean there could be some.

I am going to work on C types now. There is a few closing down actions on what I observed doing goto generators. There are a few easy ways to get even slightly better performance, definitely smaller code out of generators. Not sure if I go there first, or for the C types work directly. I often like to get these kind of observations dealt with more immediately, but I don’t want to spend too much quality time on it.

Donations

As I have been asked this, yes, you can donate to Nuitka if you wish to further its development. Go here:

Donate to Nuitka

Nuitka this week #4

2018-08-17T14:55:00+01:00

Goto Generators

This continues TWN #3 where I explained what is all about.

Good news is, at the time Python2 generators were largely working with the new ways, in the mean time not only did all of the Python 2.7 test suite pass with goto generators, also did the Python 3.4 test suite, i.e. also the yield from is working with it.

The way it was done is to set m_yieldfrom in generators, and then to enter a state, where the code will only be resumed, when that sub-generator that currently it is yielding from, is finished. That makes it very much like normal yield. In fact, code generation is hardly different there.

Since the whole purpose is to get rid of make/get/setcontext, the next stop is coroutines. They have async for, async with and await but at the end of the day, the implementation comes down to yield from really with only a lot of sugar applied.

Right now, I am debugging “goto coroutines”. It’s hard to tell when it will be finished, and then asyncgen will be waiting still.

This is easily the largest change in a long time, esp. due to the heap storage changes that I already discussed. One this is finished, I expect to turn towards C types with relative ease.

Tox Plugin

Anthony Shaw took on Tox and Nuitka and created a plugin that allows using Nuitka. I am still wrapping my head around these things. It’s only a proof of concept yet. I will give it more coverage in the future.

Twitter

Follow me on twitter if you like, I will:

Follow @kayhayen

Hotfixes

So there have even more hotfixes. One addresses memory leaks found with the yield from while I was adding tests. Usually if I encounter an old issue that has a small fix, that is what I do, push out a hotfix using the git flow model. Also nested namespace packages for Python3, those are the ones without a __init__.py were not working after the original directory was removed, and that got fixed.

And right now, I have hotfixes for frames close method, which apparently was never updated to work properly for coroutines and asyncgen. That is going to be in the next hotfix.

Plans

So the heap storage seems pretty complete now, and goto generators are on the final stretch. As always, things feel right around the corner. But it’s unclear how much longer I will have to debug. I am pretty sure the bare work of doing asyncgen is going to be low. Debugging that too then, that is the hard part.

A new release seems justified, but I kind of do not want to make it without that major new code used. Because apparently during the debugging, I tend to find issues that need hotfixes, so I will wait for the goto generator work to finish.

Nuitka this week #3

2018-08-11T11:58:00+01:00

New Series Rationale

This is working out well so far. I think driving more attention at the things that are going on can only be good. Also to explain will always help. It also kind of motivates me a bit.

Twitter

Also as part of my communications offensive, I am using my Twitter account more regularly. I used to highlight important fixes, or occasionally releases of some importance there. I will continue to do only important stuff there, but with more regularity.

And I noticed in the past, even when I do not post, followers makes me happy. So here you go:

Follow @kayhayen

Goto Generators

This continues TWN #2 where I promised to speak more of it, and this is the main focus of my work on Nuitka right now.

Brief summary, context switches were how this was initially implemented. The main reason being that for C++ there never was going to be a way to save and restore state in the middle of an expression that involves constructors and destructors.

Fast forward some years, and C-ish entered the picture. No objects are used anymore, and Nuitka is purely C11 now, which has convenience of C++, but no objects. Instead goto is used a lot already. So every time an exception occurs, a goto is done, every time a branch is done, a loop exit or continue, you get it, another goto.

But so far, all Python level variables of a frame live on that C stack still, and the context switch is done with functions that swap stack. That is fast, but the imporant drawback is that it takes more memory. How deep of a stack will we need? And we can use really many, if you imagine a pool of 1000 coroutines, that quickly become impossible to deal with.

So, the new way of doing this basically goes like this:

def g():
    yield 1
    yield 2

This was some far becoming something along this lines:

PyObject *impl_g( NuitkaGenerator *generator )
{
     YIELD( const_int_1 );
     YIELD( const_int_2 );

     PyErr_SetException( StopIteration );
     return NULL;
}

The YIELD in there was basically doing the switching of the stacks and for the C code, it looked like a normal function call.

In the new approach, this is done:

PyObject *impl_g( NuitkaGenerator *generator )
{
     switch( generator->m_resume_point )
     {
          case 1: goto resume_1;
          case 2: goto resume_2;
     }

     generator->m_yielded = const_int_1;
     generator->resume_point = 1
     return NULL;
     resume_1:

     generator->m_yielded = const_int_2;
     generator->resume_point = 2
     return NULL;
     resume_2:

     PyErr_SetException( StopIteration );
     return NULL;
}

As you can see, the function has an initial dispatcher. Resume point 0 means we are starting at the top. Then every yield results in a function return with an updated resume point.

I experimented with this actually a long time ago, and experimental code was the result that remained in Nuitka. The problem left to solve was to store the variables that would normally live on the stack, in a heap storage. That is what I am currently working on.

This leads me to “heap storage”, which is what I am currently working on and will report on next week. Once that is there, goto generators can work, and will become the norm. Until then, I am refactoring a lot to get accesses to variable go through proper objects that know their storage locations and types.

Hotfixes

So there have been 2 more hotfixes. One was to make the enum and __new__ compatibility available that I talked about last week in TWN #2 <./nuitka-this-week-2.html#python3-enumerators> coupled with a new minor things.

And then another one, actually important, where Python3 __annotations__ by default was the empty dictionary, but then could be modified, corrupting the Nuitka internally used one severely.

Right now I have on factory another fix for nested namespace packages in Python3 and that might become another hotfix soon.

As you know, I am following the git flow model, where it’s easy to push out small fixes, and just those, on top of the last release. I tend to decide based on importance. However, I feel that with the important fixes in the hotfixes now, it’s probably time to make a full release, to be sure everybody gets those.

Plans

Finishing heap storage is my top priority right now and I hope to complete the refactorings necessary in the coming week. I will also talk about how it also enables C types work next week.

Until next week then!

Nuitka this week #2

2018-08-02T07:08:54+01:00

New Series Rationale

As discussed last week in TWN #1 this is a new series that I am using to highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

Python 3.7

I made the first release with official 3.7 support, huge milestone in terms of catching up. Generic classes posed a few puzzles, and need more refinements for error handling, but good code works now.

The class creation got a bit more complex, yet again, which will make it even hard to know the exact base classes to be used. But eventually we will manage to overcome this and statically optimize that.

MSI 3.7 files for Nuitka

Building the MSI files for Nuitka ran into a 3.7.0 regression of CPython failing to build them, that I reported and seems to be valid bug of theirs.

So they will be missing for some longer time. Actually I wasn’t so sure if they are all that useful, or working as expected for the runners, but with the -m nuitka mode of execution, that ought to be a non-issue. so it would be nice to keep them for those who use them for deployment internally.

Planned Mode

I have a change here. This is going to be a draft post until I publish it, so I might the link, or mention it on the list, but I do not think I will wait for feedback, where there is not going to be all that much.

So I am shooting this off the web site.

Goto Generators

This is an exciting field of work, that I have been busy with this week. I will briefly describe the issue at hand.

So generators in Python are more generally called coroutines in other places, and basically that is code shaking hands, executing resuming in one, handing back a piece of data back and forth.

In Python, the way of doing this is yield and more recently yield from as a convienant way for of doing it in a loop in Python3. I still recall the days when that was a statement. Then communication was one way only. Actually when I was still privately doing Nuitka based on then Python 2.5 and was then puzzled for Python 2.6, when I learned in Nuitka about it becoming an expression.

The way this is implemented in Python, is that execution of a frame is simply suspended, and another frame stack bytecode is activated. This switching is of course very fast potentially, the state is already fully preserved on the stack of the virtual machine, which is owned by the frame. For Nuitka, when it still was C++, it wasn’t going to be possible to interrupt execution without preserving the stack. So what I did was very similar, and I started to use makecontext/setcontext to implement what I call fibers.

Basically that is C level stack switching, but with a huge issue. Python does not grow stacks, but can need a lot of stack space below. Therefore 1MB or even 2MB per generator was allocated, to be able to make deep function calls if needed.

So using a lot of generators on 32 bits could easily hit a 2GB limit. And now with Python3.5 coroutines people use more and more of them, and hit memory issues.

So, goto generators, now that C is possible, are an entirely new solution. With it, Nuitka will use one stack only. Generator code will become re-entrant, store values between entries on the heap, and continue execution at goto destinations dispatched by a switch according to last exit of the generator.

So I am now making changes to cleanup the way variable declarations and accesses for the C variables are being made. More on that next week though. For now I am very exited about the many cleanups that stem from it. The code generation used to have many special bells and whistles, and they were generalized into one thing now, making for cleaner and easier to understand Nuitka code.

Python3 Enumerators

On interesting thing, is that an incompatibility related to __new__ will go away now.

The automatic staticmethod that we had to hack into it, because the Python core will do it for uncompiled functions only, had to be done while declaring the class. So it was visible and causing issues with at least the Python enum module, which wants to call your __new__ manually. Because why would it not?!

But turns out, for Python3 the staticmethod is not needed anymore. So this is now only done for Python2, where it is needed, and things work smoothly with this kind of code now too. This is currently in my factory testing and will probably become part of a hotfix if it turns out good.

Hotfixes

Immediately after the release, some rarely run test, where I compiled all the code on my machine, found 2 older bugs, obscure ones arguably, that I made into a hotfix, also because the test runner was having a regression with 3.7, which prevented some package builds. So that was 0.5.32.1 release.

And then I received a bug report about await where a self test of Nuitka fails and reports an optimization error. Very nice, the new exceptions that automatically dump involved nodes as XML made it immediately clear from the report, what is going on, even without having to reproduce anything. I bundled a 3.7 improvement for error cases in class creation with it. So that was the 0.5.32.2 release.

Plans

Finishing goto generators is my top priority, but I am also going over minor issues with the 3.7 test suite, fixing test cases there, and as with e.g. the enum issue, even known issues this now finds.

Until next week.

Nuitka Release 0.5.32

2018-07-29T22:46:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains substantial new optimization, bug fixes, and already the full support for Python 3.7. Among the fixes, the enhanced coroutine work for compatibility with uncompiled ones is most important.

Bug Fixes

Fix, was optimizing write backs of attribute in-place assignments falsely.
Fix, generator stop future was not properly supported. It is now the default for Python 3.7 which showed some of the flaws.
Python3.5: The __qualname__ of coroutines and asyncgen was wrong.
Python3.5: Fix, for dictionary unpackings to calls, check the keys if they are string values, and raise an exception if not.
Python3.6: Fix, need to check assignment unpacking for too short sequences, we were giving IndexError instead of ValueError for these. Also the error messages need to consider if they should refer to “at least” in their wording.
Fix, outline nodes were cloned more than necessary, which would corrupt the code generation if they later got removed, leading to a crash.
Python3.5: Compiled coroutines awaiting uncompiled coroutines was not working properly for finishing the uncompiled ones. Also the other way around was raising a RuntimeError when trying to pass an exception to them when they were already finished. This should resolve issues with asyncio module.
Fix, side effects of a detected exception raise, when they had an exception detected inside of them, lead to an infinite loop in optimization. They are now optimized in-place, avoiding an extra step later on.

New Features

Support for Python 3.7 with only some corner cases not supported yet.

Optimization

Delay creation of StopIteration exception in generator code for as long as possible. This gives more compact code for generations, which now pass the return values via compiled generator attribute for Python 3.3 or higher.
Python3: More immediate re-formulation of classes with no bases. Avoids noise during optimization.
Python2: For class dictionaries that are only assigned from values without side effects, they are not converted to temporary variable usages, allowing the normal SSA based optimization to work on them. This leads to constant values for class dictionaries of simple classes.
Explicit cleanup of nodes, variables, and local scopes that become unused, has been added, allowing for breaking of cyclic dependencies that prevented memory release.

Tests

Adapted 3.5 tests to work with 3.7 coroutine changes.
Added CPython 3.7 test suite.

Cleanups

Removed remaining code that was there for 3.2 support. All uses of version comparisons with 3.2 have been adapted. For us, Python3 now means 3.3, and we will not work with 3.2 at all. This removed a fair bit of complexity for some things, but not all that much.
Have dedicated file for import released helpers, so they are easier to find if necessary. Also do not have code for importing a name in the header file anymore, not performance relevant.
Disable Python warnings when running scons. These are particularly given when using a Python debug binary, which is happening when Nuitka is run with --python-debug option and the inline copy of Scons is used.
Have a factory function for all conditional statement nodes created. This solved a TODO and handles the creation of statement sequences for the branches as necessary.
Split class reformulation into two files, one for Python2 and one for Python3 variant. They share no code really, and are too confusing in a single file, for the huge code bodies.
Locals scopes now have a registry, where functions and classes register their locals type, and then it is created from that.
Have a dedicated helper function for single argument calls in static code that does not require an array of objects as an argument.

Organizational

There are now requirements-devel.txt and requirements.txt files aimed at usage with scons and by users, but they are not used in installation.

Summary

This releases has this important step to add conversion of locals dictionary usages to temporary variables. It is not yet done everywhere it is possible, and the resulting temporary variables are not yet propagated in the all the cases, where it clearly is possible. Upcoming releases ought to achieve that most Python2 classes will become to use a direct dictionary creation.

Adding support for Python 3.7 is of course also a huge step. And also this happened fairly quickly and soon after its release. The generic classes it adds were the only real major new feature. It breaking the internals for exception handling was what was holding back initially, but past that, it was really easy.

Expect more optimization to come in the next releases, aiming at both the ability to predict Python3 metaclasses __prepare__ results, and at more optimization applied to variables after they became temporary variables.

Nuitka this week #1

2018-07-27T07:08:54+01:00

New Series Rationale

I think I tend to prefer coding over communication too much. I think I need to make more transparent what I am doing. Also things, will be getting exciting continuously for a while now.

I used to status report posts, many years ago, every 3 months or so, and that was nice for me also to get an idea of what changed, but I stopped. What did not happen, was to successfully engage other people to contribute.

This time I am getting more intense. I will aim to do roughly weekly or bi-weekly reports, where I highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

Planned Mode

I will do it this fashion. I will write a post to the mailing list, right about wednesday every week or so. I need to pick a day. I am working from home that day, saving me commute time. I will invest that time into this.

The writing will not be too high quality at times. Bare with me there. Then I will check feedback from the list, if any. Hope is for it to point out the things where I am not correct, missing, or even engage right away.

Topics are going to be random, albeit repeating. I will try and make links to previous issues where applicable. Therefore also the TOC, which makes for link targets in the pages.

Locals Dict

When I am speaking of locals dict, I am talking of class scopes (and functions with exec statements). These started to use actual dictionary a while ago, which was a severe setback to optimization.

Right now, so for this week, after a first prototype was making the replacement of local dict assignment and references for Python2, and kind of worked through my buildbots, flawlessly, I immediately noticed that it would require some refactoring to not depend on the locals scopes to be only in one of the trace collections. Thinking of future inlining, maybe part of a locals scope was going to be in multiple functions, that ought to not be affected.

Therefore I made a global registry of locals scopes, and working on those, I checked its variables, if they can be forward propagated, and do this not per module, but after all the modules have been done. This is kind of a setback for the idea of module specific optimization (cacheable later on) vs. program optimization, but since that is not yet a thing, it can remain this way for now.

Once I did that, I was interested to see the effect, but to my horror, I noticed, that memory was not released for the locals dict nodes. It was way too involved with cyclic dependencies, which are bad. So that was problematic of course. Compilation to keep nodes in memory for both tracing the usage as a locals dict and temporary variables, wasn’t going to help scaling at all.

Solution is finalization

Nodes need Finalization

So replaced nodes reference a parent, and then the locals scope references variables, and trace collections referencing variables, which reference locals scopes, and accesses referencing traces, and so on. The garbage collector can handle some of this, but seems I was getting past that.

For a solution, I started to add a finalize method, which released the links for locals scopes, when they are fully propagated, on the next run.

Adding a finalize to all nodes, ought to make sure, memory is released soon, and might even find bugs, as nodes become unusable after they are supposedly unused. Obviously, there will currently be cases, where nodes becomes unused, but they are not finalized yet. Also, often this is more manual, because part of the node is to be released, but one child is re-used. That is messy.

Impact on Memory Usage

The result was a bit disappointing. Yes, memory usage of mercurial compilation went back again, but mostly to what it had been. Some classes are now having their locals dict forward propagated, but the effect is not always a single dictionary making yet. Right now, function definitions, are not forward at all propagated. This is a task I want to take on before next release though, but maybe not, there is other things too. But I am assuming that will make most class dictionaries created without using any variables at all anymore, which should make it really lean.

Type Hints Question

Then, asking about type hints, I got the usual question about Nuitka going to use it. And my stance is unchanged. They are just hints, not reliable. Need to behave the same if users do it wrong. Suggested to create decorated which make type hints enforced. But I expect nobody takes this on though. I need to make it a GitHub issue of Nuitka, although technically it is pure CPython work and ought to be done independently. Right now Nuitka is not yet there anyway yet, to take full advantage.

Python 3.7

Then, for Python 3.7, I have long gotten the 3.6 test suite to pass. I raised 2 bugs with CPython, one of which lead to update of a failing test. Nuitka had with large delay, caught of with what del __annotations__ was doing in a class. Only with the recent work for proper locals dict code generation, we could enforce a name to be local, and have proper code generation, that allows for it to be unset.

This was of course a bit of work. But the optimization behind was always kind of necessary to get right. But now, that I got this, think of my amazement when for 3.7 they reverted to the old behavior, where annotiatons then corrupt the module annotations

The other bug is a reference counting bug, where Nuitka tests were failing with CPython 3.7, and turns out, there is a bug in the dictionary implementation of 3.7, but it only corrupts counts reported, not actual objects, so it’s harmless, but means for 3.7.0 the reference count tests are disabled.

Working through the 3.7 suite, I am cherry picking commits, that e.g. allow the repr of compiled functions to contain <compiled_function ...> and the like. Nothing huge yet. There is now a subscript of type, and foremost the async syntax became way more liberal, so it is more complex for Nuitka to make out if it is a coroutine due to something happening inside a generator declared inside of it. Also cr_origin was added to coroutines, but that is mostly it.

Coroutine Compatibility

A bigger thing was that I debugged coroutines and their interaction with uncompiled and compiled coroutines awaiting one another, and turns out, there was a lot to improve.

The next release will be much better compatible with asyncio module and its futures, esp with exceptions to cancel tasks passed along. That required to clone a lot of CPython generator code, due to how ugly they mess with bytecode instruction pointers in yield from on an uncompiled coroutine, as they don’t work with send method unlike everything else has to.

PyLint Troubles

For PyLint, the 2.0.0 release found new things, but unfortunately for 2.0.1 there is a lot of regressions that I had to report. I fixed the versions of first PyLint, and now also Astroid, so Travis cannot suddenly start to fail due to a PyLint release finding new warnings.

Currently, if you make a PR on GitHub, a PyLint update will break it. And also the cron job on Travis that checks master.

As somebody pointed out, I am now using requires.io <https://requires.io/github/kayhayen/Nuitka/requirements/?branch=factory> to check for Nuitka dependencies. But since 1.9.2 is still needed for Python2, that kind of is bound to give alarms for now.

TODO solving

I have a habit of doing off tasks, when I am with my notebook in some place, and don’t know what to work on. So I have some 2 hours recently like this, and used it to look at TODO and resolve them.

I did a bunch of cleanups for static code helpers. There was one in my mind about calling a function with a single argument. That fast call required a local array with one element to put the arg into. That makes using code ugly.

Issues Encountered

So the enum module of Python3 hates compiled classes and their staticmethod around __new__. Since it manually unwraps __new__ and then calls it itself, it then finds that a staticmethod object cannot be called. It’s purpose is to sit in the class dictionary to give a descriptor that removes the self arg from the call.

I am contemplating submitting an upstream patch for CPython here. The hard coded check for PyFunction on the __new__ value is hard to emulate.

So I am putting the staticmethod into the dictionary passed already. But the undecorated function should be there for full compatibility.

If I were to make compiled function type that is both a staticmethod alike and a function, maybe I can work around it. But it’s ugly and a burden. But it would need no change. And maybe there is more core wanting to call __new__ manually

Plans

I intend to make a release, probably this weekend. It might not contain full 3.7 compatibility yet, although I am aiming at that.

Then I want to turn to “goto generators”, a scalability improvement of generators and coroutines that I will talk about next week then.

Until next week.

Nuitka Release 0.5.31

2018-07-10T15:37:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is massive in terms of fixes, but also adds a lot of refinement to code generation, and more importantly adds experimental support for Python 3.7, while enhancing support for Pyt5 in standalone mode by a lot.

Bug Fixes

Standalone: Added missing dependencies for PyQt5.Qt module.
Plugins: Added support for PyQt5.Qt module and its qml plugins.
Plugins: The sensible plugin list for PyQt now includes that platforms plugins on Windows too, as they are kind of mandatory.
Python3: Fix, for uninstalled Python versions wheels that linked against the Python3 library as opposed to Python3X, it was not found.
Standalone: Prefer DLLs used by main program binary over ones used by wheels.
Standalone: For DLLs added by Nuitka plugins, add the package directory to the search path for dependencies where they might live.
Fix, the vars built-in didn’t annotate its exception exit.
Python3: Fix, the bytes and complex built-ins needs to be treated as a slot too.
Fix, consider if del variable must be assigned, in which case no exception exit should be created. This prevented Tkinter compilation.
Python3.6: Added support for the following language construct:
```
d = {"metaclass": M}


class C(**d):
    pass
```
Python3.5: Added support for cyclic imports. Now a from import with a name can really cause an import to happen, not just a module attribute lookup.
Fix, hasattr was never raising exceptions.
Fix, bytearray constant values were considered to be non-iterable.
Python3.6: Fix, now it is possible to del __annotations__ in a class and behave compatible. Previously in this case we were falling back to the module variable for annotations used after that which is wrong.
Fix, some built-in type conversions are allowed to return derived types, but Nuitka assumed the exact type, this affected bytes, int, long, unicode.
Standalone: Fix, the _socket module was insisted on to be found, but can be compiled in.

New Features

Added experimental support for Python 3.7, more work will be needed though for full support. Basic tests are working, but there are are at least more coroutine changes to follow.
Added support for building extension modules against statically linked Python. This aims at supporting manylinux containers, which are supposed to be used for creating widely usable binary wheels for Linux. Programs won’t work with statically linked Python though.
Added options to allow ignoring the Windows cache for DLL dependencies or force an update.
Allow passing options from distutils to Nuitka compilation via setup options.
Added option to disable the DLL dependency cache on Windows as it may become wrong after installing new software.
Added experimental ability to provide extra options for Nuitka to setuptools.
Python3: Remove frame preservation and restoration of exceptions. This is not needed, but leaked over from Python2 code.

Optimization

Apply value tracing to local dict variables too, enhancing the optimization for class bodies and function with exec statements by a lot.
Better optimization for “must not have value”, wasn’t considering merge traces of uninitialized values, for which this is also the case.
Use 10% less memory at compile time due to specialized base classes for statements with a single child only allowing __slots__ usage by not having multiple inheritance for those.
More immediately optimize branches with known truth values, so that merges are avoided and do not prevent trace based optimization before the pass after the next one. In some cases, optimization based on traces could fail to be done if there was no next pass caused by other things.
Much faster handling for functions with a lot of eval and exec calls.
Static optimization of type with known type shapes, the value is predicted at compile time.
Optimize containers for all compile time constants into constant nodes. This also enables further compile time checks using them, e.g. with isinstance or in checks.
Standalone: Using threads when determining DLL dependencies. This will speed up the un-cached case on Windows by a fair bit.
Also remove unused assignments for mutable constant values.
Python3: Also optimize calls to bytes built-in, this was so far not done.
Statically optimize iteration over constant values that are not iterable into errors.
Removed Fortran, Java, LaTex, PDF, etc. stuff from the inline copies of Scons for faster startup and leaner code. Also updated to 3.0.1 which is no important difference over 3.0.0 for Nuitka however.
Make sure to always release temporary objects before checking for error exits. When done the other way around, more C code than necessary will be created, releasing them in both normal case and error case after the check.
Also remove unused assignments in case the value is a mutable constant.

Cleanups

Don’t store “version” numbers of variable traces for code generation, instead directly use the references to the value traces instead, avoiding later lookups.
Added dedicated module for complex built-in nodes.
Moved C helpers for integer and complex types to dedicated files, solving the TODOs around them.
Removed some Python 3.2 only codes.

Organizational

For better bug reports, the --version output now contains also the Python version information and the binary path being used.
Started using specialized exceptions for some types of errors, which will output the involved data for better debugging without having to reproduce anything. This does e.g. output XML dumps of problematic nodes.
When encountering a problem (compiler crash) in optimization, output the source code line that is causing the issue.
Added support for Fedora 28 RPM builds.
Remove more instances of mentions of 3.2 as supported or usable.
Renovated the graphing code and made it more useful.

Summary

This release marks important progress, as the locals dictionary tracing is a huge step ahead in terms of correctness and proper optimization. The actual resulting dictionary is not yet optimized, but that ought to follow soon now.

The initial support of 3.7 is important. Right now it apparently works pretty well as a 3.6 replacement already, but definitely a lot more work will be needed to fully catch up.

For standalone, this accumulated a lot of improvements related to the plugin side of Nuitka. Thanks to those involved in making this better. On Windows things ought to be much faster now, due to parallel usage of dependency walker.

Nuitka Release 0.5.30

2018-05-01T11:53:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has improvements in all areas. Many bug fixes are accompanied with optimization changes towards value tracing.

Bug Fixes

Fix, the new setuptools runners were not used by pip breaking the use of Nuitka from PyPI.
Fix, imports of six.moves could crash the compiler for built-in names. Fixed in 0.5.29.2 already.
Windows: Make the nuitka-run not a symlink as these work really bad on that platform, instead make it a full copy just like we did for nuitka3-run already. Fixed in 0.5.29.2 already.
Python3.5: In module mode, types.coroutine was monkey patched into an endless recursion if including more than one module, e.g. for a package. Fixed in 0.5.29.3 already.
Python3.5: Dictionary unpackings with both star arguments and non star arguments could leak memory. Fixed in 0.5.29.3 already.
```
c = {a: 1, **d}
```
Fix, distutils usage was not working for Python2 anymore, due to using super for what are old style classes on that version.

Fix, some method calls to C function members could leak references.

class C:
    for_call = functools.partial

    def m(self):
        self.for_call()  # This leaked a reference to the descriptor.

Python3.5: The bases classes should be treated as an unpacking too.

class C(*D):  # Allowed syntax that was not supported.
    pass

Windows: Added back batch files to run Nuitka from the command line. Fixed in 0.5.29.5 already.

New Features

Added option --include-package to force inclusion of a whole package with the submodules in a compilation result.
Added options --include-module to force inclusion of a single module in a compilation result.
The multiprocessing plug-in got adapted to Python 3.4 changes and will now also work in accelerated mode on Windows.
It is now possible to specify the Qt plugin directories with e.g. --enable-plugin-=qt_plugins=imageformats and have only those included. This should avoid dependency creep for shared libraries.
Plugins can now make the decision about recursing to a module or not.
Plugins now can get their own options passed.

Optimization

The re-raising of exceptions has gotten its own special node type. This aims at more readability (XML output) and avoiding the overhead of checking potential attributes during optimization.
Changed built-in int, long, and float to using a slot mechanism that also analyses the type shape and detects and warns about errors at compile time.
Changed the variable tracing to value tracing. This meant to cleanup all the places that were using it to find the variable.
Enable must have / must not value value optimization for all kinds of variables including module and closure variables. This often avoids error exits and leads to smaller and faster generated code.

Tests

Added burn test with local install of pip distribution to virtualenv before making any PyPI upload. It seems pip got its specific error sources too.
Avoid calling 2to3 and prefer <python> -m lib2to3 instead, as it seems at least Debian Testing stopped to provide the binary by default. For Python 2.6 and 3.2 we continue to rely on it, as the don’t support that mode of operation.
The PyLint checks have been made more robust and even more Python3 portable.
Added PyLint to Travis builds, so PRs are automatically checked too.
Added test for distutils usage with Nuitka that should prevent regressions for this new feature and to document how it can be used.
Make coverage taking work on Windows and provide the full information needed, the rendering stage is not there working yet though.
Expanded the trick assignment test cases to cover more slots to find bugs introduced with more aggressive optimization of closure variables.
New test to cover multiprocessing usage.
Generating more code tests out of doctests for increased coverage of Nuitka.

Cleanups

Stop using --python-version in tests where they still remained.
Split the forms of int and long into two different nodes, they share nothing except the name. Create the constants for the zero arg variant more immediately.
Split the output comparison part into a dedicated testing module so it can be re-used, e.g. when doing distutils tests.
Removed dead code from variable closure taking.
Have a dedicated module for the metaclass of nodes in the tree, so it is easier to find, and doesn’t clutter the node base classes module as much.
Have a dedicated node for reraise statements instead of checking for all the arguments to be non-present.

Organizational

There is now a pull request template for GitHub when used.
Deprecating the --python-version argument which should be replaced by using -m nuitka with the correct Python version. Outputs have been updated to recommend this one instead.
Make automatic import sorting and auto-format tools properly executable on Windows without them changing new lines.
The documentation was updated to prefer the call method with -m nuitka and manually providing the Python binary to use.

Summary

This release continued the distutils integration adding first tests, but more features and documentation will be needed.

Also, for the locals dictionary work, the variable tracing was made generic, but not yet put to use. If we use this to also trace dictionary keys, we can expect a lot of improvements for class code again.

The locals dictionary tracing will be the focus before resuming the work on C types, where the ultimate performance boost lies. However, currently, not the full compatibility has been achieved even with currently using dictionaries for classes, and we would like to be able to statically optimize those better anyway.

Nuitka Release 0.5.29

2018-03-28T00:45:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes with a lot of improvements across the board. A lot of focus has been givevn to the packaging side of Nuitka, but also there is a lot of compatibility work.

Bug Fixes

Windows: When using Scons for Python3 and Scons for Python2 on the same build directory, a warning would be given about the need to migrate. Make the Scons cache directory use the Python ABI version as a key too, to avoid these issues. Fixed in 0.5.28.1 already.
Windows: Fixup for Python3 and Scons no more generating the MinGW64 import library for Python anymore properly. Was only working if cached from a previous install of Nuitka. Fixed in 0.5.28.1 already.
Plugins: Made the data files plugin mandatory and added support for the scrapy package needs.
Fix, added implicit dependencies for pkg_resources.external package. Fixed in 0.5.28.1 already.
Fix, an import of x.y where this was not a package didn’t cause the package x to be included.
Standalone: Added support for six.moves and requests.packages meta imports, these cause hidden implicit imports, that are now properly handled.
Standalone: Patch the __file__ value for technical bytecode modules loaded during Python library initialization in a more compatible way.
Standalone: Extension modules when loaded might actually raise legit errors, e.g. ImportError of another module, don’t make those into SystemError anymore.
Python3.2: The __package__ of sub-packages was wrong, which could cause issues when doing relative imports in that sub-package.
Python3: Contractions in a finally clause could crash the compiler.
Fix, unused closure variables could lead to a crash in they were passed to a nested function.
Linux: Standalone dependency analysis could enter an endless recursion in case of cyclic dependencies.
Python3.6: Async generation expressions need to return a None value too.
Python3.4: Fix, __spec__ is a package attribute and not a built-in value.

New Features

It is now possible to run Nuitka with some_python_you_choose -m nuitka ... and therefore know exactly which Python installation is going to be used. It does of course need Nuitka installed for this to work. This mechanism is going to replace the --python-version mechanism in the future.
There are dedicated runners for Python3, simply use nuitka3 or nuitka3-run to execute Nuitka if your code is Python3 code.
Added warning for implicit exception raises due to mismatch in unpacking length. These are statically detected, but so far were not warned about.
Added cache for depends.exe results. This speeds up standalone mode again as some of these calls were really slow.
The import tracer is more robust against recursion and works with Python3 now.
Added an option to assume yes for downloading questions. The currently only enables the download of depends.exe and is intended for CI servers.
There is now a report file for scons, which records the values used to run things, this could be useful for debugging.
Nuitka now registers with distutils and can be used with bdist_wheel directly, but this lacks documentation and tests. Many improvements in the distutils build.

Optimization

Forward propagate compile time constants even if they are only potential usages. This is actually the case where this makes the most sense, as it might remove its use entirely from the branches that do not use it.
Avoid extra copy of finally code. The cloning operation takes time and memory, and this shaved of 0.3% of Nuitka memory usage, as these can also become dangling.
Class dictionaries are now proper dictionaries in optimization, using some dedicated code for name lookups that are transformed to dedicated locals dictionary or mapping (Python3) accesses. This currently does not fully optimize, but will in coming releases, and saves about 25% of memory compared to the old code.
Treating module attributes __package__, __loader__, __file__, and __spec__ with dedicated nodes, that allow or forbid optimization dependent on usage.
Python3.6: Async generator expressions were not working fully, become more compatible.
Fix, using super inside a contraction could crash the compiler.
Fix, also accept __new__ as properly decorated in case it’s a classmethod too.
Fix, removed obsolete --nofreeze-stdlib which only complicated using the --recurse-stdlib which should be used instead.

Organizational

The nuitka Python package is now installed into the public namespace and used from there. There are distinct copies to be installed for both Python2 and Python3 on platforms where it is supported.
Using twine for upload to PyPI now as recommended on their site.
Running pylint on Windows became practical again.
Added RPM packages for Fedora 26 and 27, these used to fail due to packaging issues.
Added RPM packages for openSUSE Leap 42.2, 42.3 and 15.0 which were simply missing.
Added RPM packages for SLE 15.
Added support for PyLint 1.8 and its new warnings.
The RPM packages no longer contain nuitka-run3, it will be replaced by the new nuitka3-run which is in all packages.
The runners used for installation are now easy install created, but patched to avoid overhead at run time.
Added repository for Ubuntu Artful (17.10) for download, removed support for Ubuntu Yakkety, Vivid and Zesty (no more supported by them).
Removed support for Debian Wheezy and Ubuntu Precise (they are too old for modern packaging used).
There is now a issue template for GitHub when used.

Tests

Windows: Standalone tests were referencing an old path to depends.exe that wasn’t populated on new installs.
Refinements for CPython test suites to become more stable in results. Some tests occasionally fail to clean up, or might do nondeterministic outputs, or are not relevant at all.
The tests don’t use the runners, but more often do -m nuitka to become executable without having to find the proper runner. This improves usage during the RPM builds and generally.
Travis: Do not test development versions of CPython, even for stable release, they break too often.

Summary

This release consolidates a lot of what we already had, adding hopeful stuff for distutils integration. This will need tests and documentation though, but should make Nuitka really easy to use. A few features are still missing to make it generally reliable in that mode, but they are going to come.

Also the locals dictionary work is kind of incomplete without a proper generic tracing of not only local variables, but also dictionary keys. With that work in place, a lot of improvements will happen.

Nuitka Release 0.5.28

2017-10-18T11:00:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a focus on compatibility work and contains bug fixes and work to enhance the usability of Nuitka by integrating with distutils. The major improvement is that contractions no longer use pseudo functions to achieve their own local scope, but that there is now a dedicated structure for that representing an in-lined function.

Bug Fixes

Python3.6: Fix, async for was not yet implemented for async generators.
Fix, functions with keyword arguments where the value was determined to be a static raise could crash the compiler.
Detect using MinGW64 32 bits C compiler being used with 64 bits Python with better error message.
Fix, when extracting side effects of a static raise, extract them more recursively to catch expressions that themselves have no code generation being used. This fixes at least static raises in keyword arguments of a function call.
Compatibility: Added support for proper operation of pkgutil.get_data by implementing get_data in our meta path based loader.
Compatibility: Added __spec__ module attribute was previously missing, present on Python3.4 and higher.
Compatibility: Made __loader__ module attribute set when the module is loading already.
Standalone: Resolve the @rpath and @loader_path from otool on macOS manually to actual paths, which adds support for libraries compiled with that.
Fix, nested functions calling super could crash the compiler.
Fix, could not use --recurse-directory with arguments that had a trailing slash.
Fix, using --recurse-directory on packages that are not in the search crashed the compiler.
Compatibility: Python2 set and dict contractions were using extra frames like Python3 does, but those are not needed.
Standalone: Fix, the way PYTHONHOME was set on Windows had no effect, which allowed the compiled binary to access the original installation still.
Standalone: Added some newly discovered missing hidden dependencies of extension modules.
Compatibility: The name mangling of private names (e.g. __var) in classes was applied to variable names, and function declarations, but not to classes yet.
Python3.6: Fix, added support for list contractions with await expressions in async generators.
Python3.6: Fix, async for was not working in async generators yet.
Fix, for module tracebacks, we output the module name <module name> instead of merely <module>, but if the module was in a package, that was not indicated. Now it is <module package.name>.
Windows: The cache directory could be unicode which then failed to pass as an argument to scons. We now encode such names as UTF-8 and decode in Scons afterwards, solving the problem in a generic way.
Standalone: Need to recursively resolve shared libraries with ldd, otherwise not all could be included.
Standalone: Make sure sys.path has no references to CPython compile time paths, or else things may work on the compiling machine, but not on another.
Standalone: Added various missing dependencies.
Standalone: Wasn’t considering the DLLs directory for standard library extensions for freezing, which would leave out these.
Compatibility: For __future__ imports the __import__ function was called more than once.

Optimization

Contractions are now all properly inlined and allow for optimization as if they were fully local. This should give better code in some cases.
Classes are now all building their locals dictionary inline to the using scope, allowing for more compact code.
The dictionary API was not used in module template code, although it helps to generate more compact code.

New Features

Experimental support for building platform dependent wheel distribution.
```
python setup.py --command-packages=nuitka.distutils clean -a bdist_nuitka
```
Use with caution, this is incomplete work.
Experimental support for running tests against compiled installation with nose and py.test.
When specifying what to recurse to, now patterns can be used, e.g. like this --recurse-not-to=*.tests which will skip all tests in submodules from compilation.
By setting NUITKA_PACKAGE_packagename=/some/path the __path__ of packages can be extended automatically in order to allow and load uncompiled sources from another location. This can be e.g. a tests sub-package or other plugins.
By default when creating a module, now also a module.pyi file is created that contains all imported modules. This should be deployed alongside the extension module, so that standalone mode creation can benefit from knowing the dependencies of compiled code.
Added option --plugin-list that was mentioned in the help output, but still missing so far.
The import tracing of the hints module has achieved experimental status and can be used to test compatibility with regards to import behavior.

Cleanups

Rename tree and codegen Helper modules to unique names, making them easier to work with.
Share the code that decides to not warn for standard library paths with more warnings.
Use the bool enum definition of Python2 which is more elegant than ours.
Move quality tools, auto-format, isort, etc. to the nuitka.tools.quality namespace.
Move output comparison tool to the nuitka.tools.testing namespace.
Made frame code generation capable of using nested frames, allowing the real inline of classes and contraction bodies, instead of “direct” calls to pseudo functions being used.
Proper base classes for functions that are entry points, and functions that are merely a local expression using return statements.

Tests

The search mode with pattern, was not working anymore.
Resume hash values now consider the Python version too.
Added test that covers using test runners like nose and py.test with Nuitka compiled extension modules.

Organizational

Added support for Scons 3.0 and running Scons with Python3.5 or higher. The option to specify the Python to use for scons has been renamed to reflect that it may also be a Python3 now. Only for Python3.2 to Python3.4 we now need another Python installation.
Made recursion the default for --recurse-directory with packages. Before you also had to tell it to recurse into that package or else it would only include the top level package, but nothing below.
Updated the man pages, correct mentions of its C++ to C and don’t use now deprecated options.
Updated the help output which still said that standalone mode implies recursion into standard library, which is no longer true and even not recommended.
Added option to disable the output of .pyi file when creating an extension module.
Removed Ubuntu Wily package download, no longer supported by Ubuntu.

Summary

This release was done to get the fixes and new features out for testing. There is work started that should make generators use an explicit extra stack via pointer, and restore instruction state via goto dispatchers at function entry, but that is not complete.

This feature, dubbed “goto generators” will remove the need for fibers (which is itself a lot of code), reduce the memory footprint at run time for anything that uses a lot of generators, or coroutines.

Integrating with distutils is also a new thing, and once completed will make use of Nuitka for existing projects automatic and trivial to do. There is a lot missing for that goal, but we will get there.

Also, documenting how to run tests against compiled code, if that test code lives inside of that package, will make a huge difference, as that will make it easier for people to torture Nuitka with their own test cases.

And then of course, nested frames now mean that every function could be inlined, which was previously not possible due to collisions of frames. This will pave the route for better optimization in those cases in future releases.

The experimental features will require more work, but should make it easier to use Nuitka for existing projects. Future releases will make integrating Nuitka dead simple, or that is the hope.

And last but not least, now that Scons works with Python3, chances are that Nuitka will more often work out the of the box. The older Python3 versions that still retain the issue are not very widespread.

Nuitka Release 0.5.27

2017-07-23T17:42:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes a lot of bug fixes and improvements.

Bug Fixes

Fix, need to add recursed modules immediately to the working set, or else they might first be processed in second pass, where global names that are locally assigned, are optimized to the built-in names although that should not happen. Fixed in 0.5.26.1 already.
Fix, the accelerated call of methods could crash for some special types. This had been a regress of 0.5.25, but only happens with custom extension types. Fixed in 0.5.26.1 already.
Python3.5: For async def functions parameter variables could fail to properly work with in-place assignments to them. Fixed in 0.5.26.4 already.
Compatibility: Decorators that overload type checks didn’t pass the checks for compiled types. Now isinstance and as a result inspect module work fine for them.
Compatibility: Fix, imports from __init__ were crashing the compiler. You are not supposed to do them, because they duplicate the package code, but they work.
Compatibility: Fix, the super built-in on module level was crashing the compiler.
Standalone: For Linux, BSD and macOS extension modules and shared libraries using their own $ORIGIN to find loaded DLLs resulted in those not being included in the distribution.
Standalone: Added more missing implicit dependencies.
Standalone: Fix, implicit imports now also can be optional, as e.g. _tkinter if not installed. Only include those if available.
The --recompile-c-only was only working with C compiler as a backend, but not in the C++ compatibility fallback, where files get renamed. This prevented that edit and test debug approach with at least MSVC.
Plugins: The PyLint plug-in didn’t consider the symbolic name import-error but only the code F0401.
Implicit exception raises in conditional expressions would crash the compiler.

New Features

Added support for Visual Studio 2017.
Added option --python2-for-scons to specify the Python2 execute to use for calling Scons. This should allow using Anaconda Python for that task.

Optimization

References to known unassigned variables are now statically optimized to exception raises and warned about if the according option is enabled.
Non-hashable keys in dictionaries are now statically optimized to exception raises and warned about if the according option is enabled.
Enable forward propagation for classes too, resulting in some classes to create only static dictionaries. Currently this never happens for Python3, but it will, once we can statically optimize __prepare__ too.
Enable inlining of class dictionary creations if they are mere return statements of the created dictionary. Currently this never happens for Python3, see above for why.
Python2: Selecting the metaclass is now visible in the tree and can be statically optimized.
For executables, we now also use a free list for traceback objects, which also makes exception cases slightly faster.
Generator expressions no longer require the use of a function call with a .0 argument value to carry the iterator value, instead their creation is directly inlined.
Remove “pass through” frames for Python2 list contractions, they are no longer needed. Minimal gain for generated code, but more lightweight at compile time.
When compiling Windows x64 with MinGW64 a link library needs to be created for linking against the Python DLL. This one is now cached and re-used if already done.
Use common code for NameError and UnboundLocalError exception code raises. In some cases it was creating the full string at compile time, in others at run time. Since the later is more efficient in terms of code size, we now use that everywhere, saving a bit of binary size.
Make sure to release unused functions from a module. This saves memory and can be decided after a full pass.
Avoid using OrderedDict in a couple of places, where they are not needed, but can be replaced with a later sorting, e.g. temporary variables by name, to achieve deterministic output. This saves memory at compile time.
Add specialized return nodes for the most frequent constant values, which are None, True, and False. Also a general one, for constant value return, which avoids the constant references. This saves quite a bit of memory and makes traversal of the tree a lot faster, due to not having any child nodes for the new forms of return statements.
Previously the empty dictionary constant reference was specialized to save memory. Now we also specialize empty set, list, and tuple constants to the same end. Also the hack to make is not say that {} is {} was made more general, mutable constant references and now known to never alias.
The source references can be marked internal, which means that they should never be visible to the user, but that was tracked as a flag to each of the many source references attached to each node in the tree. Making a special class for internal references avoids storing this in the object, but instead it’s now a class property.
The nodes for named variable reference, assignment, and deletion got split into separate nodes, one to be used before the actual variable can be determined during tree building, and one for use later on. This makes their API clearer and saves a tiny bit of memory at compile time.
Also eliminated target variable references, which were pseudo children of assignments and deletion nodes for variable names, that didn’t really do much, but consume processing time and memory.
Added optimization for calls to staticmethod and classmethod built-in methods along with type shapes.
Added optimization for open built-in on Python3, also adding the type shape file for the result.
Added optimization for bytearray built-in and constant values. These mutable constants can now be compile time computed as well.
Added optimization for frozenset built-in and constant values. These mutable constants can now be compile time computed as well.
Added optimization for divmod built-in.
Treat all built-in constant types, e.g. type itself as a constant. So far we did this only for constant values types, but of course this applies to all types, giving slightly more compact code for their uses.
Detect static raises if iterating over non-iterables and warn about them if the option is enabled.
Split of locals node into different types, one which needs the updated value, and one which just makes a copy. Properly track if a functions needs an updated locals dict, and if it doesn’t, don’t use that. This gives more efficient code for Python2 classes, and exec using functions in Python2.
Build all constant values without use of the pickle module which has a lot more overhead than marshal, instead use that for too large long values, non-UTF-8 unicode values, nan float, etc.
Detect the linker arch for all Linux platforms using objdump instead of only a hand few hard coded ones.

Cleanups

The use of INCREASE_REFCOUNT got fully eliminated.
Use functions not vulenerable for buffer overflow. This is generally good and avoids warnings given on OpenBSD during linking.
Variable closure for classes is different from all functions, don’t handle the difference in the base class, but for class nodes only.
Make sure mayBeNone doesn’t return None which means normally “unclear”, but False instead, since it’s always clear for those cases.
Comparison nodes were using the general comparison node as a base class, but now a proper base class was added instead, allowing for cleaner code.
Valgrind test runners got changed to using proper tool namespace for their code and share it.
Made construct case generation code common testing code for reuse in the speedcenter web site. The code also has minor beauty bugs which will then become fixable.
Use appdirs package to determine place to store the downloaded copy of depends.exe.
The code still mentioned C++ in a lot of places, in comments or identifiers, which might be confusing readers of the code.
Code objects now carry all information necessary for their creation, and no longer need to access their parent to determine flag values. That parent is subject to change in the future.
Our import sorting wrapper automatically detects imports that could be local and makes them so, removing a few existing ones and preventing further ones on the future.
Cleanups and annotations to become Python3 PyLint clean as well. This found e.g. that source code references only had __cmp__ and need rich comparison to be fully portable.

Tests

The test runner for construct tests got cleaned up and the constructs now avoid using xrange so as to not need conversion for Python3 execution as much.
The main test runner got cleaned up and uses common code making it more versatile and robust.
Do not run test in debugger if CPython also segfaulted executing the test, then it’s not a Nuitka issue, so we can ignore that.
Improve the way the Python to test with is found in the main test runner, prefer the running interpreter, then PATH and registry on Windows, this will find the interesting version more often.
Added support for “Landscape.io” to ignore the inline copies of code, they are not under our control.
The test runner for Valgrind got merged with the usage for constructs and uses common code now.
Construct generation is now common code, intended for sharing it with the Speedcenter web site generation.
Rebased Python 3.6 test suite to 3.6.1 as that is the Python generally used now.

Organizational

Added inline copy of appdirs package from PyPI.
Added credits for RedBaron and isort.
The --experimental flag is now creating a list of indications and more than one can be used that way.
The PyLint runner can also work with Python3 pylint.
The Nuitka Speedcenter got more fine tuning and produces more tags to more easily identify trends in results. This needs to become more visible though.
The MSI files are also built on AppVeyor, where their building will not depend on me booting Windows. Getting these artifacts as downloads will be the next step.

Summary

This release improves many areas. The variable closure taking is now fully transparent due to different node types, the memory usage dropped again, a few obvious missing static optimizations were added, and many built-ins were completed.

This release again improves the scalability of Nuitka, which again uses less memory than before, although not an as big jump as before.

This does not extend or use special C code generation for bool or any type yet, which still needs design decisions to proceed and will come in a later release.

Nuitka Release 0.5.26

2017-06-11T09:10:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes after a long time and contains large amounts of changes in all areas. The driving goal was to prepare generating C specific code, which is still not the case, but this is very likely going to change soon. However this release improves all aspects.

Bug Fixes

Compatibility: Fix, for star imports didn’t check the values from the __all__ iterable, if they were string values which could cause problems at run time.
```
# Module level
__all__ = (1,)

# ...
# other module:
from module import *
```
Fix, for star imports, also didn’t check for values from __all__ if they actually exist in the original values.
Corner cases of imports should work a lot more precise, as the level of compatibility for calls to __import__ went from absurd to insane.
Windows: Fixed detection of uninstalled Python versions (not for all users and DLL is not in system directory). This of course only affected the accelerated mode, not standalone mode.
Windows: Scan directories for .pyd files for used DLLs as well. This should make the PyQt5 wheel work.
Python3.5: Fix, coroutines could have different code objects for the object and the frame using by it.
Fix, slices with built-in names crashed the compiler.
```
something[id:len:range]
```
Fix, the C11 via C++ compatibility uses symlinks tp C++ filenames where possible instead of making a copy from the C source. However, even on Linux that may not be allowed, e.g. on a DOS file system. Added fallback to using full copy in that case.
Python3.5: Fix coroutines to close the “yield from” where an exception is thrown into them.
Python3: Fix, list contractions should have their own frame too.
Linux: Copy the “rpath” of compiling Python binary to the created binary. This will make compiled binaries using uninstalled Python versions transparently find the Python shared library.
Standalone: Add the “rpath” of the compiling Python binary to the search path when checking for DLL dependencies on Linux. This fixes standalone support for Travis and Anaconda on Linux.
Scons: When calling scons, also try to locate a Python2 binary to overcome a potential Python3 virtualenv in which Nuitka is running.
Standalone: Ignore more Windows only encodings on non-Windows.

New Features

Support for Python 3.6 with only few corner cases not supported yet.
Added options --python-arch to pick 32 or 64 bits Python target of the --python-version argument.
Added support for more kinds of virtualenv configurations.
Uninstalled Python versions such as Anaconda will work fine in accelerated mode, except on Windows.

Optimization

The node tree children are no longer stored in a separate dictionary, but in the instance dictionary as attributes, making the tree more lightweight and in principle faster to access. This also saved about 6% of the memory usage.
The memory usage of Nuitka for the Python part has fallen by roughly 40% due to the use of new style classes, and slots where that is possible (some classes use multiple inheritance, where they don’t work), and generally by reducing useless members e.g. in source code references. This of course also will make things compiled faster (the C compilation of course is not affected by this.)
The code generation for frames was creating the dictionary for the raised exception by making a dictionary and then adding all variables, each tested to be set. This was a lot of code for each frame specific, and has been replaced by a generic “attach” mechanism which merely stores the values, and only takes a reference. When asked for frame locals, it only then builds the dictionary. So this is now only done, when that is absolutely necessary, which it normally never is. This of course makes the C code much less verbose, and actual handling of exceptions much more efficient.
For imports, we now detect for built-in modules, that their import cannot fail, and if name lookups can fail. This leads to less code generated for error handling of these. The following code now e.g. fully detects that no ImportError or AttributeError will occur.
```
try:
    from __builtin__ import len
except ImportError:
    from builtins import len
```
Added more type shapes for built-in type calls. These will improve type tracing.
Compiled frames now have a free list mechanism that should speed up frames that recurse and frames that exit with exceptions. In case of an exception, the frame ownership is immediately transferred to the exception making it easier to deal with.
The free list implementations have been merged into a new common one that can be used via macro expansion. It is now type agnostic and be slightly more efficient too.
Also optimize “true” division and “floor division”, not only the default division of Python2.
Removed the need for statement context during code generation making it less memory intensive and faster.

Cleanups

Now always uses the __import__ built-in node for all kinds of imports and directly optimizes and recursion into other modules based on that kind of node, instead of a static variant. This removes duplication and some incompatibility regarding defaults usage when doing the actual imports at run time.
Split the expression node bases and mixin classes to a dedicated module, moving methods that only belong to expressions outside of the node base, making for a cleaner class hierarchy.
Cleaned up the class structure of nodes, added base classes for typical compositions, e.g. expression with and without children, computation based on built-in, etc. while also checking proper ordering of base classes in the metaclass.
Moved directory and file operations to dedicated module, making also sure it is more generally used. This makes it easier to make more error resilient deletions of directories on e.g. Windows, where locks tend to live for short times beyond program ends, requiring second attempts.
Code generation for existing supported types, PyObject *, PyObject **, and struct Nuitka_CellObject * is now done via a C type class hierarchy instead of elif sequences.
Closure taking is now always done immediately correctly and references are take for closure variables still needed, making sure the tree is correct and needs no finalization.
When doing variable traces, initialize more traces immediately so it can be more reliable.
Code to setup a function for local variables and clean it up has been made common code instead of many similar copies.
The code was treating the f_executing frame member as if it were a counter with increases and decreases. Turn it into a mere boolean value and hide its usage behind helper functions.
The “maybe local variables” are no more. They were replaced by a new locals dict access node with a fallback to a module or closure variable should the dictionary not contain the name. This avoids many ugly checks to not do certain things for that kind of variable.
We now detect “exec” and “unqualified exec” as well as “star import” ahead of time as flags of the function to be created. We no longer need to mark functions as we go.
Handle “true”, “floor” and normal division properly by applying future flags to decide which one to use.
We now use symbolic identifiers in all PyLint annotations.
The release scripts started to move into nuitka.tools.release so they get PyLint checks, auto-format and proper code reuse.
The use of INCREASE_REFCOUNT_X was removed, it got replaced with proper Py_XINCREF usages.
The use of INCREASE_REFCOUNT got reduced further, e.g. no generated code uses it anymore, and only a few compiled types do. The function was once required before “C-ish” lifted the need to do everything in one single function call.

Tests

More robust deletion of directories, temporary stages used by CPython test suites, and standalone directories during test execution.
Moved tests common code into nuitka.tools.testing namespace and use it from there. The code now is allowed to use nuitka.utils and therefore often better implementations.
Made standalone binaries robust against GTK theme access, checking the Python binary (some site.py files do that),

Organizational

Added repository for Ubuntu Zesty (17.04) for download.
Added support for testing with Travis to complement the internal Buildbot based infrastructure and have pull requests on GitHub automatically tested before merge.
The factory branch is now also on GitHub.
Removed MSI for Python3.4 32 bits. It seems impossible to co-install this one with the 64 bits variant. All other versions are provided for both bit sizes still.

Summary

This release marks huge progress. The node tree is now absolutely clean, the variable closure taking is fully represented, and code generation is prepared to add another type, e.g. for bool for which work has already started.

On a practical level, the scalability of the release will have increased very much, as this uses so much less memory, generates simpler C code, while at the same time getting faster for the exception cases.

Coming releases will expand on the work of this release.

Frame objects should be allowed to be nested inside a function for better re-formulations of classes and contractions of all kinds, as well as real inline of functions, even if they could raise.

The memory savings could be even larger, if we stopped doing multiple inheritance for more node types. The __slots__ were and the child API change could potentially make things not only more compact, but faster to use too.

And also once special C code generation for bool is done, it will set the stage for more types to follow (int, float, etc). Only this will finally start to give the C type speed we are looking for.

Until then, this release marks a huge cleanup and progress to what we already had, as well as preparing the big jump in speed.

Nuitka Release 0.5.25

2017-01-25T06:27:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a huge amount of bug fixes, lots of optimization gains, and many new features. It also presents many organizational improvements, and many cleanups.

Bug Fixes

Python3.5: Coroutine methods using super were crashing the compiler. Fixed in 0.5.24.2 already.
Python3.3: Generator return values were not properly transmitted in case of tuple or StopIteration values.
Python3.5: Better interoperability between compiled coroutines and uncompiled generator coroutines.
Python3.5: Added support to compile in Python debug mode under Windows too.
Generators with arguments were using two code objects, one with, and one without the CO_NOFREE flag, one for the generator object creating function, and one for the generator object.
Python3.5: The duplicate code objects for generators with arguments lead to interoperability issues with between such compiled generator coroutines and compiled coroutines. Fixed in 0.5.24.2 already.
Standalone: On some Linux variants, e.g. Debian Stretch and Gentoo, the linker needs more flags to really compile to a binary with RPATH.
Compatibility: For set literal values, insertion order is wrong on some versions of Python, we now detect the bug and emulate it if necessary, previous Nuitka was always correct, but incompatible.
```
{1, 1.0}.pop()  # the only element of the set should be 1
```
Windows: Make the batch files detect where they live at run time, instead of during setup.py, making it possible to use them for all cases.
Standalone: Added package paths to DLL scan for depends.exe, as with wheels there now sometimes live important DLLs too.
Fix, the clang mode was regressed and didn’t work anymore, breaking the macOS support entirely.
Compatibility: For imports, we were passing for locals argument a real dictionary with actual values. That is not what CPython does, so stopped doing it.
Fix, for raised exceptions not passing the validity tests, they could be used after free, causing crashes.
Fix, the environment CC wasn’t working unless also specifying CXX.
Windows: The value of __file__ in module mode was wrong, and didn’t point to the compiled module.
Windows: Better support for --python-debug for installations that have both variants, it is now possible to switch to the right variant.

New Features

Added parsing for shebang to Nuitka. When compiling an executable, now Nuitka will check of the #! portion indicates a different Python version and ask the user to clarify with --python-version in case of a mismatch.
Added support for Python flag --python-flag=-O, which allows to disable assertions and remove doc strings.

Optimization

Faster method calls, combining attribute lookup and method call into one, where order of evaluation with arguments doesn’t matter. This gives really huge relative speedups for method calls with no arguments.
Faster attribute lookup in general for object descendants, which is all new style classes, and all built-in types.
Added dedicated xrange built-in implementation for Python2 and range for Python3. This makes those faster while also solving ordering problems when creating constants of these types.
Faster sum again, using quick iteration interface and specialized quick iteration code for typical standard type containers, tuple and list.
Compiled generators were making sure StopIteration was set after their iteration, although most users were only going to clear it. Now only the send method, which really needs that does it. This speed up the closing of generators quite a bit.
Compiled generators were preparing a throw into non-started compilers, to be checked for immediately after their start. This is now handled in a generic way for all generators, saving code and execution time in the normal case.
Compiled generators were applying checks only useful for manual send calls even during iteration, slowing them down.
Compiled generators could duplicate code objects due to handling a flag for closure variables differently.
For compiled frames, the f_trace is not writable, but was taking and releasing references to what must be None, which is not useful.
Not passing locals to import calls make it less code and faster too.

Organizational

This release also prepares Python 3.6 support, it includes full language support on the level of CPython 3.6.0 with the sole exception of the new generator coroutines.
The improved mode is now the default, and full compatibility is now the option, used by test suites. For syntax errors, improved mode is always used, and for test suites, now only the error message is compared, but not call stack or caret positioning anymore.
Removed long deprecated option “–no-optimization”. Code generation too frequently depends on not seeing unoptimized code. This has been hidden and broken long enough to finally remove it.
Added support for Python3.5 numbers to Speedcenter. There are now also tags for speedcenter, indicating how well “develop” branch fares in comparison to the stable branch.
With a new tool, source code and Developer Manual contents can be kept in sync, so that descriptions can be quoted there. Eventually a full Sphinx documentation might become available, but for now this makes it workable.
Added repository for Ubuntu Yakkety (16.10) for download.
Added repository for Fedora 25 for download.

Cleanups

Moved the tools to compare CPython output, to sort import statements (isort) to auto-format the source code (Redbaron usage), and to check with PyLint to a common new nuitka.tools package, runnable with __main__ modules and dedicated runners in bin directory.
The tools now share code to find source files, or have it for the first time, and other things, e.g. finding needed binaries on Windows installations.
No longer patch traceback objects dealloc function. Should not be needed anymore, and most probably was only bug hiding.
Moved handling of ast nodes related to import handling to the proper reformulation module.
Moved statement generation code to helpers module, making it accessible without cyclic dependencies that require local imports.
Removed deprecated method for getting constant code objects in favor of the new way of doing it. Both methods were still used, making it harder to analyse.
Removed useless temporary variable initializations from complex call helper internal functions. They worked around code generation issues that have long been solved.
The ABI flags are no longer passed to Scons together with the version.

Tests

Windows: Added support to detect and to switch debug Python where available to also be able to execute reference counting tests.
Added the CPython 3.3 test suite, after cleaning up the worst bits of it, and added the brandnew 3.6 test suite with a minimal set of changes.
Use the original 3.4 test suite instead of the one that comes from Debian as it has patched quite a few issues that never made it upstream, and might cause crashes.
More construct tests, making a difference between old style classes, which have instances and new style classes, with their objects.
It is now possible to run a test program with Python3 and Valgrind.

Summary

The quick iteration is a precursor to generally faster iteration over unknown object iterables. Expanding this to general code generation, and not just the sum built-in, might yield significant gains for normal code in the future, once we do code generation based on type inference.

The faster method calls complete work that was already prepared in this domain and also will be expanded to more types than compiled functions. More work will be needed to round this up.

Adding support for 3.6.0 in the early stages of its release, made sure we pretty much have support for it ready right after release. This is always a huge amount of work, and it’s good to catch up.

This release is again a significant improvement in performance, and is very important to clean up open ends. Now the focus of coming releases will now be on both structural optimization, e.g. taking advantage of the iterator tracing, and specialized code generation, e.g. for those iterations really necessary to use quick iteration code.

Nuitka Release 0.5.24

2016-11-15T09:43:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is again focusing on optimization, this time very heavily on the generator performance, which was found to be much slower than CPython for some cases. Also there is the usual compatibility work and improvements for Pure C support.

Bug Fixes

Windows: The 3.5.2 coroutine new protocol implementation was using the wrapper from CPython, but it’s not part of the ABI on Windows. Have our own instead. Fixed in 0.5.23.1 already.
Windows: Fixed second compilation with MSVC failing. The files renamed to be C++ files already existed, crashing the compilation. Fixed in 0.5.23.1 already.
Mac OS: Fixed creating extension modules with .so suffix. This is now properly determined by looking at the importer details, leading to correct suffix on all platforms. Fixed in 0.5.23.1 already.
Debian: Don’t depend on a C++ compiler primarily anymore, the C compiler from GNU or clang will do too. Fixed in 0.5.23.1 already.
Pure C: Adapted scons compiler detecting to properly consider C11 compilers from the environment, and more gracefully report things.

Optimization

Python2: Generators were saving and restoring exceptions, updating the variables sys.exc_type for every context switch, making it really slow, as these are 3 dictionary updates, normally not needed. Now it’s only doing it if it means a change.
Sped up creating generators, coroutines and coroutines by attaching the closure variable storage directly to the object, using one variable size allocation, instead of two, once of which was a standard malloc. This makes creating them easier and avoids maintaining the closure pointer entirely.
Using dedicated compiled cell implementation similar to PyCellObject but fully under our control. This allowed for smaller code generated, while still giving a slight performance improvement.
Added free list implementation to cache generator, coroutines, and function objects, avoiding the need to create and delete this kind of objects in a loop.
Added support for the built-in sum, making slight optimizations to be much faster when iterating over lists and tuples, as well as fast long sum for Python2, and much faster bool sums too. This is using a prototype version of a “qiter” concept.
Provide type shape for xrange calls that are not constant too, allowing for better optimization related to those.

Tests

Added workarounds for locks being held by Virus Scanners on Windows to our test runner.
Enhanced constructs that test generator expressions to more clearly show the actual construct cost.
Added construct tests for the sum built-in on various types of int containers, making sure we can do all of those really fast.

Summary

This release improves very heavily on generators in Nuitka. The memory allocator is used more cleverly, and free lists all around save a lot of interactions with it. More work lies ahead in this field, as these are not yet as fast as they should be. However, at least Nuitka should be faster than CPython for these kind of usages now.

Also, proper pure C in the Scons is relatively important to cover more of the rarer use cases, where the C compiler is too old.

The most important part is actually how sum optimization is staging a new kind of approach for code generation. This could become the standard code for iterators in loops eventually, making for loops even faster. This will be for future releases to expand.

Nuitka Release 0.5.23

2016-10-03T16:55:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is focusing on optimization, the most significant part for the users being enhanced scalability due to memory usage, but also break through structural improvements for static analysis of iterators and the debut of type shapes and value shapes, giving way to “shape tracing”.

Bug Fixes

Fix support Python 3.5.2 coroutine changes. The checks got added for improved mode for older 3.5.x, the new protocol is only supported when run with that version or higher.
Fix, was falsely optimizing away unused iterations for non-iterable compile time constants.
```
iter(1)  # needs to raise.
```
Python3: Fix, eval must not attempt to strip memoryviews. The was preventing it from working with that type.
Fix, calling type without any arguments was crashing the compiler. Also the exception raised for anything but 1 or 3 arguments was claiming that only 3 arguments were allowed, which is not the compatible thing.
Python3.5: Fix, follow enhanced error checking for complex call handling of star arguments.
Compatibility: The from x import x, y re-formulation was doing two __import__ calls instead of re-using the module value.

Optimization

Uses only about 66% of the memory compared to last release, which is very important step for scalability independent of re-loading. This was achieved by making sure to break loop traces and their reference cycle when they become unused.
Properly detect the len of multiplications at compile time from newly introduces value shapes, so that this is e.g. statically optimized.
```
print(len("*" * 10000000000))
```
Due to newly introduced type shapes, len and iter now properly detect more often if values will raise or not, and warn about detected raises.
```
iter(len(something))  # Will always raise
```
Due to newly introduced “iterator tracing”, we can now properly detect if the length of an unpacking matches its source or not. This allows to remove the check of the generic re-formulations of unpackings at compile time.
```
a, b = b, a  # Will never raise due to unpacking
a, b = b, a, c  # Will always raise, 3 items cannot unpack to 2
```
Added support for optimization of the xrange built-in for Python2.
Python2: Added support for xrange iterable constant values, pre-building those constants ahead of time.
Python3: Added support and range iterable constant values, pre-building those constants ahead of time. This brings optimization support for Python3 ranges to what was available for Python2 already.
Avoid having a special node variange for range with no arguments, but create the exception raising node directly.
Specialized constant value nodes are using less generic implementations to query e.g. their length or iteration capabilities, which should speed up many checks on them.
Added support for the format built-in.
Python3: Added support for the ascii built-in.

Organizational

The movement to pure C got the final big push. All C++ only idoms of C++ were removed, and everything works with C11 compilers. A C++03 compiler can be used as a fallback, in case of MSVC or too old gcc for instance.
Using pure C, MinGW64 6x is now working properly. The latest version had problems with hypot related changes in the C++ standard library. Using C11 solves that.
This release also prepares Python 3.6 support, it includes full language support on the level of CPython 3.6.0b1.
The CPython 3.6 test suite was run with Python 3.5 to ensure bug level compatibility, and had a few findings of incompatibilities.

Cleanups

The last holdouts of classes in Nuitka were removed, and many idioms of C++ were stopped using.
Moved range related helper functions to a dedicated include file.
Using str is not bytes to detect Python3 str handling or actual bytes type existence.
Trace collections were using a mix-in that was merged with the base class that every user of it was having.

Tests

Added more static optimization tests, a lot more has become feasible to decide at run time, and is now done. These are to detect regressions in that domain.
The CPython 3.6 test suite is now also run with CPython 3.5 which found some incompatibilities.

Summary

This release marks a huge step forward. We are having the structure for type inference now. This will expand in coming releases to cover more cases, and there are many low hanging fruits for optimization. Specialized codes for variable versions of certain known shapes seems feasible now.

Then there is also the move towards pure C. This will make the backend compilation lighter, but due to using C11, we will not suffer any loss of convenience compared to “C-ish”. The plan is to use continue to use C++ for compilation for compilers not capable of supporting C11.

The amount of static analysis done in Nuitka is now going to quickly expand, with more and more constructs predicted to raise errors or simplified. This will be an ongoing activity, as many types of expressions need to be enhanced, and only one missing will not let it optimize as well.

Also, it seems about time to add dedicated code for specific types to be as fast as C code. This opens up vast possibilities for acceleration and will lead us to zero overhead C bindings eventually. But initially the drive is towards enhanced import analysis, to become able to know the precise module expected to be imported, and derive type information from this.

The coming work will attack to start whole program optimization, as well as enhanced local value shape analysis, as well specialized type code generation, which will make Nuitka improve speed.

Nuitka Release 0.5.22

2016-08-17T14:35:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly an intermediate release on the way to the large goal of having per module compilation that is cacheable and requires far less memory for large programs. This is currently in progress, but required many changes that are in this release, more will be needed.

It also contains a bunch of bug fixes and enhancements that are worth to be released, and the next changes are going to be more invasive.

Bug Fixes

Compatibility: Classes with decorated __new__ functions could miss out on the staticmethod decorator that is implicit. It’s now applied always, unless of course it’s already done manually. This corrects an issue found with Pandas. Fixed in 0.5.22.1 already.
Standalone: For at least Python 3.4 or higher, it could happen that the locale needed was not importable. Fixed in 0.5.22.1 already.
Compatibility: Do not falsely assume that not expressions cannot raise on boolean expressions, since those arguments might raise during creation. This could lead to wrong optimization. Fixed in 0.5.22.2 already.
Standalone: Do not include system specific C libraries in the distribution created. This would lead to problems for some configurations on Linux in cases the glibc is no longer compatible with newer or older kernels. Fixed in 0.5.22.2 already.
The --recurse-directory option didn’t check with decision mechanisms for module inclusion, making it impossible to avoid some things.

Optimization

Introduced specialized constant classes for empty dictionaries and other special constants, e.g. “True” and “False”, so that they can have more hard coded properties and save memory by sharing constant values.
The “technical” sharing of a variable is only consider for variables that had some sharing going in the first place, speeing things up quite a bit for that still critical check.
Memory savings coming from enhanced trace storage are already visible at about 1%. That is not as much as the reloading will mean, but still helpful to use less overall.

Cleanups

The global variable registry was removed. It was in the way of unloading and reloading modules easily. Instead variables are now attached to their owner and referenced by other users. When they are released, these variables are released.
Global variable traces were removed. Instead each variable has a list of the traces attached to it. For non-shared variables, this allows to sooner tell attributes of those variables, allowing for sooner optimization of them.
No longer trace all initial users of a variable, just merely if there were such and if it constitutes sharing syntactically too. Not only does this save memory, it avoids useless references of the variable to functions that stop using it due to optimization.
Create constant nodes via a factory function to avoid non-special instances where variants exist that would be faster to use.
Moved the C string functions to a proper nuitka.utils.CStrings package as we use it for better code names of functions and modules.
Made functions and explicit child node of modules, which makes their use more generic, esp. for re-loading modules.
Have a dedicated function for building frame nodes, making it easier to see where they are created.

Summary

This release is the result of a couple of months work, and somewhat means that proper re-loading of cached results is becoming in sight. The reloading of modules still fails for some things, and more changes will be needed, but with that out of the way, Nuitka’s footprint is about to drop and making it then absolutely scalable. Something considered very important before starting to trace more information about values.

This next thing big ought to be one thing that structurally holds Nuitka back from generating C level performance code with say integer operations.

Nuitka Release 0.5.21

2016-04-25T12:54:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release focused on scalability work. Making Nuitka more usable in the common case, and covering more standalone use cases.

Bug Fixes

Windows: Support for newer MinGW64 was broken by a workaround for older MinGW64 versions.
Compatibility: Added support for the (unofficial) C-Python API Py_GetArgcArgv that was causing prctl module to fail loading on ARM platforms.
Compatibility: The proper error message template for complex call arguments is now detected as compile time. There are changes coming, that are already in some pre-releases of CPython.
Standalone: Wasn’t properly ignoring Tools and other directories in the standard library.

New Features

Windows: Detect the MinGW compiler arch and compare it to the Python arch. In case of a mismatch, the compiler is not used. Otherwise compilation or linking gives hard to understand errors. This also rules out MinGW32 as a compiler that can be used, as its arch doesn’t match MinGW64 32 bits variant.
Compile modules in two passes with the option to specify which modules will be considered for a second pass at all (compiled without program optimization) or even become bytecode.
The developer mode installation of Nuitka in develop mode with the command pip install -e nuitka_git_checkout_dir is now supported too.

Optimization

Popular modules known to not be performance relevant are no longer C compiled, e.g. numpy.distutils and many others frequently imported (from some other module), but mostly not used and definitely not performance relevant.

Cleanups

The progress tracing and the memory tracing and now more clearly separate and therefore more readable.
Moved RPM related files to new rpm directory.
Moved documentation related files to doc directory.
Converted import sorting helper script to Python and made it run fast.

Organizational

The Buildbot infrastructure for Nuitka was updated to Buildbot 0.8.12 and is now maintained up to date with Ansible.
Upgraded the Nuitka bug tracker to Roundup 1.5.1 to which I had previously contributed security fixes already active.
Added SSL certificates from Let’s Encrypt for the web server.

Summary

This release advances the scalability of Nuitka somewhat. The two pass approach does not yet carry all possible fruits. Caching of single pass compiled modules should follow for it to become consistently fast.

More work will be needed to achieve fast and scalable compilation, and that is going to remain the focus for some time.

Nuitka Release 0.5.20

2016-03-21T09:34:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly about catching up with issues. Most address standalone problems with special modules, but there are also some general compatibility corrections, as well as important fixes for Python3.5 and coroutines and to improve compatibility with special Python variants like Anaconda under the Windows system.

Bug Fixes

Standalone Python3.5: The _decimal module at least is using a __name__ that doesn’t match the name at load time, causing programs that use it to crash.
Compatibility: For Python3.3 the __loader__ attribute is now set in all cases, and it needs to have a __module__ attribute. This makes inspection as done by e.g. flask working.
Standalone: Added missing hidden dependencies for Tkinter module, adding support for this to work properly.
Windows: Detecting the Python DLL and EXE used at compile time and preserving this information use during backend compilation. This should make sure we use the proper ones, and avoids hacks for specific Python variants, enhancing the support for Anaconda, WinPython, and CPython installations.
Windows: The --python-debug flag now properly detects if the run time is supporting things and error exits if it’s not available. For a CPython3.5 installation, it will switch between debug and non-debug Python binaries and DLLs.
Standalone: Added plug-in for the Pwm package to properly combine it into a single file, suitable for distribution.
Standalone: Packages from standard library, e.g. xml now have proper __path__ as a list and not as a string value, which breaks code of e.g. PyXML.
Standalone: Added missing dependency of twisted.protocols.tls.
Python3.5: When finalizing coroutines that were not finished, a corruption of its reference count could happen under some circumstances.
Standalone: Added missing DLL dependency of the uuid module at run time, which uses ctypes to load it.

New Features

Added support for Anaconda Python on this Linux. Both accelerated and standalone mode work now.
Added support for standalone mode on FreeBSD.
The plug-in framework was expanded with new features to allow addressing some specific issues.

Cleanups

Moved memory related stuff to dedicated utils package nuitka.utils.MemoryUsage as part of an effort to have more topical modules.
Plugins how have a dedicated module through which the core accesses the API, which was partially cleaned up.
No more “early” and “late” import detections for standalone mode. We now scan everything at the start.

Summary

This release focused on expanding plugins. These were then used to enhance the success of standalone compatibility. Eventually this should lead to a finished and documented plug-in API, which will open up the Nuitka core to easier hacks and more user contribution for these topics.

Nuitka Release 0.5.19

2016-02-02T09:04:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings optimization improvements for dictionary using code. This is now lowering subscripts to dictionary accesses where possible and adds new code generation for known dictionary values. Besides this there is the usual range of bug fixes.

Bug Fixes

Fix, attribute assignments or deletions where the assigned value or the attribute source was statically raising crashed the compiler.
Fix, the order of evaluation during optimization was considered in the wrong order for attribute assignments source and value.
Windows: Fix, when g++ is the path, it was not used automatically, but now it is.
Windows: Detect the 32 bits variant of MinGW64 too.
Python3.4: The finalize of compiled generators could corrupt reference counts for shared generator objects. Fixed in 0.5.18.1 already.
Python3.5: The finalize of compiled coroutines could corrupt reference counts for shared generator objects.

Optimization

When a variable is known to have dictionary shape (assigned from a constant value, result of dict built-in, or a general dictionary creation), or the branch merge thereof, we lower subscripts from expecting mapping nodes to dictionary specific nodes. These generate more efficient code, and some are then known to not raise an exception.
```
def someFunction(a, b):
    value = {a: b}
    value["c"] = 1
    return value
```
The above function is not yet fully optimized (dictionary key/value tracing is not yet finished), however it at least knows that no exception can raise from assigning value["c"] anymore and creates more efficient code for the typical result = {} functions.
The use of “logical” sharing during optimization has been replaced with checks for actual sharing. So closure variables that were written to in dead code no longer inhibit optimization of the then no more shared local variable.
Global variable traces are now faster to decide definite writes without need to check traces for this each time.

Cleanups

No more using “logical sharing” allowed to remove that function entirely.
Using “technical sharing” less often for decisions during optimization and instead rely more often on proper variable registry.
Connected variables with their global variable trace statically avoid the need to check in variable registry for it.
Removed old and mostly unused “assume unclear locals” indications, we use global variable traces for this now.

Summary

This release aimed at dictionary tracing. As a first step, the value assign is now traced to have a dictionary shape, and this this then used to lower the operations which used to be normal subscript operations to mapping, but now can be more specific.

Making use of the dictionary values knowledge, tracing keys and values is not yet inside the scope, but expected to follow. We got the first signs of type inference here, but to really take advantage, more specific shape tracing will be needed.

Nuitka Progress in 2015

2016-01-29T08:08:54+00:00

For quite a bit, there have been no status posts, not for lack of news, but a lot has happened indeed. I just seem to post a lot more to the mailing list than I do here. Especially about unfinished stuff, which is essentially for a project like Nuitka everything that’s going on.

Like I previously said, I am shy to make public postings about unfinished stuff and that’s going to continue. But I am breaking it, to keep you up to date with where Nuitka has been going lately.

And with release focuses, I have been making some actual changes that I think are worth talking about.

SSA (Single State Assignment Form)

The SSA using release has been made last summer. Recent releases have lifted more and more restrictions on where and now it is applied and made sure the internal status is consistent and true. And that trend is going to continue even more.

For shared variables (closure variables and module variables), Nuitka is still too conservative to make optimization. Code does annotate value escapes, but it’s not yet trusting it. The next releases will focus on lifting that kind of restriction, and for quality of result, that will mean making a huge jump ahead once that works, so module variables used locally a lot will become even faster to use then and subject to static optimization too.

Function Inlining

When doing my talk to EuroPython 2015, I was demoing it that, and indeed, what a break through. The circumstances under which it is done are still far too limited though. Essentially that ability is there, but will not normally be noticeable yet due to other optimization, e.g. functions are most often module variables and not local to the using function.

More code generation improvements will be needed to be able to inline functions that might raise an exception. Also the “cost” of inlining a function is also very much an unsolved issue. It will become the focus again, once the SSA use as indicated above expands to module variables, as then inlining other things than local functions will be possible too.

So there is a lot of things to do for this to really make a difference to your programs. But it’s still great to have that part solved so far.

Scalability

Parameter Parsing

Recent releases have replaced some of the oldest code of Nuitka, the one that generated special argument parsing for each function individually, now replaced with generic code, that surprisingly is often even faster, although quick entry points were tough to beat.

That gives the C backend compiler a much easier time. Previously 3 C functions were created per Python level function, two of which could get really big with many arguments, and these are no more.

Variable Error Messages

Something similar was going on with variable error messages. Each had their exception value pre-computed and created at module load time. Most of these are of course unused. This has been replaced with code that generates it on the fly, resulting in a lot less constants code.

Code Objects

And another thing was to look after code objects, of which there often were two for each Python level function. The one used or the frame during run time and the one used in the function object, differered often, sometimes by small things like flags or local variable names.

That of course was just the result of not passing that along, but created cached objects with hopefully the same options, but that not being true.

Resolving that, and sharing the code object used for creation and then the frame is gives less complex C code too.

Optimization

The scalability of Nuitka also depends much on generated code size. With the optimization become more clever, less code is generated, and that trend will continue as more structural optimization are applied.

Every time e.g. an exception is identified to not happen, this removes the corresponding error exits from the C code, which then makes it easier for the C compiler. Also more specialized code as we now have or dictionaries, is often less complex to it.

Compatibility

Important things have happened here. Full compatibility mode is planned to not be the default anymore in upcoming releases, but that will only mean to not be stupid compatible, but to e.g. have more complete error messages than CPython, more correct line numbers, or for version differences, the best Python version behaviour.

The stable release has full support for Python 3.5, including the new async and await functions. So recent releases can pronounce it as fully supported which was quite a feat.

I am not sure, if you can fully appreciate the catch up game needed to play here. CPython clearly implements a lot of features, that I have to emulate too. That’s going to repeat for every major release.

The good news is that the function type of Nuitka is now specialized to the generators and classes, and that was a massive cleanup of its core that was due anyway. The generators have no more their own function creation stuff and that has been helpful with a lot of other stuff.

Another focus driven from Python3, is to get ahead with type shape tracing and type inference of dictionary, and value tracing. To fully support Python3 classes, we need to work on something that is a dictionary a-like, and that will only ever be efficient if we have that. Good news is that the next release is making progress there too.

Performance

Graphs and Benchmarks

I also presented this weak point to EuroPython 2015 and my plan on how to resolve it. Unfortunately, nothing really happened here. My plan is still to use what the PyPy people have developed as vmprof.

So that is not progressing, and I could need help with that definitely. Get in contact if you think you can.

Standalone

The standalone mode of Nuitka was pretty good, and continued to improve further, but I don’t care much.

Other Stuff

EuroPython 2015

This was a blast. Meeting people who knew Nuitka but not me was a regular occurrence. And many people well appreciate my work. It felt much different than the years before.

I was able to present Nuitka’s function in-lining indeed there, and this high goal that I set myself, quite impressed people.

Also I made many new contacts, largely with the scientific community. I hope to find work with data scientists in the coming years. More amd more it looks like my day job should be closer to Nuitka and my expertise in Python.

Funding

Nuitka receives the occasional donation and those make me very happy. As there is no support from organization like the PSF, I am all on my own there.

This year I want to travel to Europython 2016. It would be sweet if aside of my free time it wouldn’t also cost me money. So please consider donating some more, as these kind of events are really helpul to Nuitka.

Collaborators

Nuitka is making more and more break through progress. And you can be a part of it. Now.

You can join and should do so now, just follow this link or become part of the mailing list (since closed) and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Videos

There is a Youtube channel of mine with all the videos of Nuitka so far and I have been preparing myself with proper equipment to make Videos of Nuitka, but so far nothing has come out of that.

I do however really want to change that. Let’s see if it happens.

Twitter

I have started to use my Twitter account on occasions. You are welcome to follow me there. I will highlight interesting stuff there.

Future

So, there is multiple things going on:

Type Inference

With SSA in place, Nuitka starts to recognize types, and treat things that work something assigned from {} or dict built-in with special nodes and code.

That’s going to be a lot of work. For float and list there are very important use cases, where the code can be much better. But dict is the hardest case, and to get the structure of shape tracing right, we are going there first.
Shape Analyisis

The plan for types, is not to use them, but the more general shapes, things that will be more prevalent than actual type information in a program. In fact the precise knowledge will be rare, but more often, we will just have a set of operations performed on a variable, and be able to guess from there.

Shape analysis will begin though with concrete types like dict. The reason is that some re-formulations like Python3 classes should not use locals, but dictionary accesses throughout for full compatibility. Tracing that correctly to be effectively the same code quality will allow to make that change.
Plug-ins

Something I wish I could have shown at EuroPython was plug-ins to Nuitka. It has become more complete, and some demo plug-ins for say Qt plugins or multiprocessing, are starting to work, but it’s not progressing recently. The API will need work and of course documentation. Hope is for this to expand Nuitka’s reach and appeal to get more contributors.

It would be sweet, if there were any takers, aiming to complete these things.
Nested frames

One result of in-lining will be nested frames still present for exceptions to be properly annotated, or locals giving different sets of locals and so on.

Some cleanup of these will be needed for code generation and SSA to be able to attach variables to some sort of container, and for a function to be able to reference different sets of these.

Let me know, if you are willing to help. I really need that help to make things happen faster. Nuitka will become more and more important only. And with your help, things will be there sooner.

Release Focus

One thing I have started recently, is to make changes to Nuitka focused to just one goal, and to only deal with the rare bug in other fields, but not much else at all. So instead of across the board improvements in just about everything, I have e.g. in the last release added type inference for dictionaries and special nodes and their code generation for dictionary operations.

This progresses Nuitka in one field. And the next release then e.g. will only focus on making the performance comparison tool, and not continue much in other fields.

That way, more “flow” is possible and more visible progress too. As an example of this, these are the focuses of last releases.

Full Python 3.5 on a clean base with generators redone so that coroutines fit in nicely.
Scalability of C compilation with argument parsing redone
Next release soon: Shape analysis of subscript usages and optimization to exact dictionaries
Next release thereafter: Comparison benchmarking (vmprof, resolving C level function identifiers easier)

Other focuses will also happen, but that’s too far ahead. Mostly like some usability improvements will be the focus of a release some day. Focus is for things that are too complex to attack as a side project, and therefore never happen although surely possible.

Digging into Python3.5 coroutines and their semantics was hard enough, and the structural changes needed to integrate them properly with not too much special casing, but rather removing existing special cases (generator functions) was just too much work to ever happen while also doing other stuff.

Summary

So I am very excited about Nuitka. It feels like the puzzle is coming together finally, with type inference becoming a real thing. And should dictionaries be sorted out, the real important types, say float for scientific use cases, or int, list for others, will be easy to make.

With this, and then harder import association (knowing what other modules are), and module level SSA tracing that can be trusted, we can finally expect Nuitka to be generally fast and deserve to be called a compiler.

That will take a while, but it’s likely to happen in 2016. Let’s see if I will get the funding to go to EuroPython 2016, that would be great.

Nuitka Release 0.5.18

2016-01-16T11:41:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release mainly has a scalability focus. While there are few compatibility improvements, the larger goal has been to make Nuitka compilation and the final C compilation faster.

Bug Fixes

Compatibility: The nested arguments functions can now be called using their keyword arguments.

def someFunction(a, (b, c)):
    return a, b, c


someFunction(a=1, **{".1": (2, 3)})

Compatibility: Generators with Python3.4 or higher now also have a __del__ attribute, and therefore properly participate in finalization. This should improve their interactions with garbage collection reference cycles, although no issues had been observed so far.
Windows: Was outputting command line arguments debug information at program start. Fixed in 0.5.17.1 already.

Optimization

Code generated for parameter parsing is now a lot less verbose. Python level loops and conditionals to generate code for each variable has been replaced with C level generic code. This will speed up the backend compilation by a lot.
Function calls with constant arguments were speed up specifically, as their call is now fully prepared, and yet using less code. Variable arguments are also faster, and all defaulted arguments are also much faster. Method calls are not affected by these improvements though.
Nested argument functions now have a quick call entry point as well, making them faster to call too.
The slice built-in, and internal creation of slices (e.g. in re-formulations of Python3 slices as subscripts) cannot raise.
Standalone: Avoid inclusion of bytecode of unittest.test, sqlite3.test, distutils.test, and ensurepip. These are not needed, but simply bloat the amount of bytecode used on e.g. macOS.
Speed up compilation with Nuitka itself by avoid to copying and constructing variable lists as much as possible using an always accurate variable registry.

Cleanups

Nested argument functions of Python2 are now re-formulated into a wrapping function that directly calls the actual function body with the unpacking of nested arguments done in nodes explicitly. This allows for better optimization and checks of these steps and potential in-lining of these functions too.
Unified slice object creation and built-in slice nodes, these were two distinct nodes before.
The code generation for all statement kinds is now done via dispatching from a dictionary instead of long elif chains.
Named nodes more often consistently, e.g. all loop related nodes start with Loop now, making them easier to group.
Parameter specifications got simplified to work without variables where it is possible.

Organizational

Nuitka is now available on the social code platforms gitlab as well.

Summary

Long standing weaknesses have been addressed in this release, also quite a few structural cleanups have been performed, e.g. strengthening the role of the variable registry to always be accurate, is groundlaying to further improvement of optimization.

However, this release cycle was mostly dedicated to performance of the actual compilation, and more accurate information was needed to e.g. not search for information that should be instant.

Upcoming releases will focus on usability issues and further optimization, it was nice however to see speedups of created code even from these scalability improvements.

Nuitka Release 0.5.17

2015-12-28T21:56:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is a major feature release, as it adds full support for Python3.5 and its coroutines. In addition, in order to properly support coroutines, the generator implementation got enhanced. On top of that, there is the usual range of corrections.

Bug Fixes

Windows: Command line arguments that are unicode strings were not properly working.
Compatibility: Fix, only the code object attached to exceptions contained all variable names, but not the one of the function object.
Python3: Support for virtualenv on Windows was using non-portable code and therefore failing.
The tree displayed with --display-tree duplicated all functions and did not resolve source lines for functions. It also displayed unused functions, which is not helpful.
Generators with parameters leaked C level memory for each instance of them leading to memory bloat for long running programs that use a lot of generators. Fixed in 0.5.16.1 already.
Don’t drop positional arguments when called with --run, also make it an error if they are present without that option.

New Features

Added full support for Python3.5, coroutines work now too.

Optimization

Optimized frame access of generators to not use both a local frame variable and the frame object stored in the generator object itself. This gave about 1% speed up to setting them up.
Avoid having multiple code objects for functions that can raise and have local variables. Previously one code object would be used to create the function (with parameter variable names only) and when raising an exception, another one would be used (with all local variable names). Creating them both at start-up was wasteful and also needed two tuples to be created, thus more constants setup code.
The entry point for generators is now shared code instead of being generated for each one over and over. This should make things more cache local and also results in less generated C code.
When creating frame codes, avoid working with strings, but use proper emission for less memory churn during code generation.

Organizational

Updated the key for the Debian/Ubuntu repositories to remain valid for 2 more years.
Added support for Fedora 23.
MinGW32 is no more supported, use MinGW64 in the 32 bits variant, which has less issues.

Cleanups

Detecting function type ahead of times, allows to handle generators different from normal functions immediately.
Massive removal of code duplication between normal functions and generator functions. The later are now normal functions creating generator objects, which makes them much more lightweight.
The return statement in generators is now immediately set to the proper node as opposed to doing this in variable closure phase only. We can now use the ahead knowledge of the function type.
The nonlocal statement is now immediately checked for syntax errors as opposed to doing that only in variable closure phase.
The name of contraction making functions is no longer skewed to empty, but the real thing instead. The code name is solved differently now.
The local_locals mode for function node was removed, it was always true ever since Python2 list contractions stop using pseudo functions.
The outline nodes allowed to provide a body when creating them, although creating that body required using the outline node already to create temporary variables. Removed that argument.
Removed PyLint false positive annotations no more needed for PyLint 1.5 and solved some TODOs.
Code objects are now mostly created from specs (not yet complete) which are attached and shared between statement frames and function creations nodes, in order to have less guess work to do.

Tests

Added the CPython3.5 test suite.
Updated generated doctests to fix typos and use common code in all CPython test suites.

Summary

This release continues to address technical debt. Adding support for Python3.5 was the major driving force, while at the same time removing obstacles to the changes that were needed for coroutine support.

With Python3.5 sorted out, it will be time to focus on general optimization again, but there is more technical debt related to classes, so the cleanup has to continue.

Nuitka Release 0.5.16

2015-11-11T17:39:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is a maintenance release, largely intended to put out improved support for new platforms and minor corrections. It should improve the speed for standalone mode, and compilation in general for some use cases, but this is mostly to clean up open ends.

Bug Fixes

Fix, the len built-in could give false values for dictionary and set creations with the same element.
```
# This was falsely optimized to 2 even if "a is b and a == b" was true.
len({a, b})
```
Python: Fix, the gi_running attribute of generators is no longer an int, but bool instead.
Python3: Fix, the int built-in with two arguments, value and base, raised UnicodeDecodeError instead of ValueError for illegal bytes given as value.
Python3: Using tokenize.open to read source code, instead of reading manually and decoding from tokenize.detect_encoding, this handles corner cases more compatible.
Fix, the PyLint warnings plug-in could crash in some cases, make sure it’s more robust.
Windows: Fix, the combination of Anaconda Python, MinGW 64 bits and mere acceleration was not working.
Standalone: Preserve not only namespace packages created by .pth files, but also make the imports done by them. This makes it more compatible with uses of it in Fedora 22.
Standalone: The extension modules could be duplicated, turned this into an error and cache finding them during compile time and during early import resolution to avoid duplication.
Standalone: Handle “not found” from ldd output, on some systems not all the libraries wanted are accessible for every library.
Python3.5: Fixed support for namespace packages, these were not yet working for that version yet.
Python3.5: Fixes lack of support for unpacking in normal tuple, list, and set creations.
```
[*a]  # this has become legal in 3.5 and now works too.
```
Now also gives compatible SyntaxError for earlier versions. Python2 was good already.
Python3.5: Fix, need to reduce compiled functions to __qualname__ value, rather than just __name__ or else pickling methods doesn’t work.
Python3.5: Fix, added gi_yieldfrom attribute to generator objects.
Windows: Fixed harmless warnings for Visual Studio 2015 in --debug mode.

Optimization

Re-formulate exec and eval to default to globals() as the default for the locals dictionary in modules.
The try node was making a description of nodes moved to the outside when shrinking its scope, which was using a lot of time, just to not be output, now these can be postponed.
Refactored how freezing of bytecode works. Uncompiled modules are now explicit nodes too, and in the registry. We only have one or the other of it, avoiding to compile both.

Tests

When strace or dtruss are not found, given proper error message, so people know what to do.
The doc tests extracted and then generated for CPython3 test suites were not printing the expressions of the doc test, leading to largely decreased test coverage here.
The CPython 3.4 test suite is now also using common runner code, and avoids ignoring all Nuitka warnings, instead more white listing was added.
Started to run CPython 3.5 test suite almost completely, but coroutines are blocking some parts of that, so these tests that use this feature are currently skipped.
Removed more CPython tests that access the network and are generally useless to testing Nuitka.
When comparing outputs, normalize typical temporary file names used on posix systems.
Coverage tests have made some progress, and some changes were made due to its results.
Added test to cover too complex code module of idna module.
Added Python3.5 only test for unpacking variants.

Cleanups

Prepare plug-in interface to allow suppression of import warnings to access the node doing it, making the import node is accessible.
Have dedicated class function body object, which is a specialization of the function body node base class. This allowed removing class specific code from that class.
The use of “win_target” as a scons parameter was useless. Make more consistent use of it as a flag indicator in the scons file.
Compiled types were mixing uses of compiled_ prefixes, something with a space, sometimes with an underscore.

Organizational

Improved support for Python3.5 missing compatibility with new language features.
Updated the Developer Manual with changes that SSA is now a fact.
Added Python3.5 Windows MSI downloads.
Added repository for Ubuntu Wily (15.10) for download. Removed Ubuntu Utopic package download, no longer supported by Ubuntu.
Added repository with RPM packages for Fedora 22.

Summary

So this release is mostly to lower the technical debt incurred that holds it back from supporting making more interesting changes. Upcoming releases may have continue that trend for some time.

This release is mostly about catching up with Python3.5, to make sure we did not miss anything important. The new function body variants will make it easier to implement coroutines, and help with optimization and compatibility problems that remain for Python3 classes.

Ultimately it will be nice to require a lot less checks for when function in-line is going to be acceptable. Also code generation will need a continued push to use the new structure in preparation for making type specific code generation a reality.

Nuitka on Podcast init

2015-10-25T11:02:56+00:00

So, if you want to get to know Nuitka or me, there is a really good interview from the popular podcast “__init__” online.

Kay Hayen on Nuitka

I think it’s very good at explaining things, putting it into perspective, history, future, and generally getting to know what kind of person I am.

So, this is probably as good as it ever gets. So please share, like and go ahead and spread it in this social media craziness that you use all day.

Yours,

Kay

Nuitka Release 0.5.15

2015-10-15T08:44:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release enables SSA based optimization, the huge leap, not so much in terms of actual performance increase, but for now making the things possible that will allow it.

This has been in the making literally for years. Over and over, there was just “one more thing” needed. But now it’s there.

The release includes much stuff, and there is a perspective on the open tasks in the summary, but first out to the many details.

Bug Fixes

Standalone: Added implicit import for reportlab package configuration dynamic import. Fixed in 0.5.14.1 already.
Standalone: Fix, compilation of the ctypes module could happen for some import patterns, and then prevented the distribution to contain all necessary libraries. Now it is made sure to not include compiled and frozen form both. Fixed in 0.5.14.1 already.
Fix, compilation for conditional statements where the boolean check on the condition cannot raise, could fail compilation. Fixed in 0.5.14.2 already.
Fix, the __import__ built-in was making static optimization assuming compile time constants to be strings, which in the error case they are not, which was crashing the compiler.
```
__import__(("some.module",))  # tuples don't work
```
This error became only apparent, because now in some cases, Nuitka forward propagates values.
Windows: Fix, when installing Python2 only for the user, the detection of it via registry failed as it was only searching system key. This was a github pull request. Fixed in 0.5.14.3 already.
Some modules have extremely complex expressions requiring too deep recursion to work on all platforms. These modules are now included entirely as bytecode fallback.
The standard library may contain broken code due to installation mistakes. We have to ignore their SyntaxError.
Fix, pickling compiled methods was failing with the wrong kind of error, because they should not implement __reduce__, but only __deepcopy__.
Fix, when running under wine, the check for scons binary was fooled by existence of /usr/bin/scons.

New Features

Added experimental support for Python3.5, coroutines don’t work yet, but it works perfectly as a 3.4 replacement.
Added experimental Nuitka plug-in framework, and use it for the packaging of Qt plugins in standalone mode. The API is not yet stable nor polished.
New option --debugger that makes --run execute directly in gdb and gives a stack trace on crash.
New option --profile executes compiled binary and outputs measured performance with vmprof. This is work in progress and not functional yet.
Started work on --internal-graph to render the SSA state into diagrams. This is work in progress and not too functional yet.
Plug-in framework added. Not yet ready for users. Working PyQt4 and PyQt5 plug-in support. Experimental Windows multiprocessing support. Experimental PyLint warnings disable support. More to come.
Added support for Anaconda accelerated mode on macOS by modifying the rpath to the Python DLL.
Added experimental support for multiprocessing on Windows, which needs monkey patching of the module to support compiled methods.

Optimization

The SSA analysis is now enabled by default, eliminating variables that are not shared, and can be forward propagated. This is currently limited mostly to compile time constants, but things won’t remain that way.
Code generation for many constructs now takes into account if a specific operation can raise or not. If e.g. an attribute look-up is known to not raise, then that is now decided by the node the looked is done to, and then more often can determine this, or even directly the value.
Calls to C-API that we know cannot raise, no longer check, but merely assert the result.
For attribute look-up and other operations that might be known to not raise, we now only assert that it succeeds.
Built-in loop-ups cannot fail, merely assert that.
Creation of built-in exceptions never raises, merely assert that too.
More Python operation slots now have their own computations and some of these gained overloads for more compile time constant optimization.
When taking an iterator cannot raise, this is now detected more often.
The try/finally construct is now represented by duplicating the final block into all kinds of handlers (break, continue, return, or except) and optimized separately. This allows for SSA to trace values more correctly.
The hash built-in now has dedicated node and code generation too. This is mostly intended to represent the side effects of dictionary look-up, but gives more compact and faster code too.
Type type built-in cannot raise and has no side effect.
Speed improvement for in-place float operations for += and *=, as these will be common cases.

Tests

Made the construct-based testing executable with Python3.
Removed warnings using the new PyLint warnings plug-in for the reflected test. Nuitka now uses the PyLint annotations to not warn. Also do not go into PyQt for reflected test, not needed. Many Python3 improvements for cases where there are differences to report.
The optimization tests no longer use 2to3 anymore, made the tests portable to all versions.
Checked more in-place operations for speed.

Organizational

Many improvements to the coverage taking. We can hope to see public data from this, some improvements were triggered from this already, but full runs of the test suite with coverage data collection are yet to be done.

Summary

The release includes many important new directions. Coverage analysis will be important to remain certain of test coverage of Nuitka itself. This is mostly done, but needs more work to complete.

Then the graphing surely will help us to debug and understand code examples. So instead of tracing, and reading stuff, we should visualize things, to more clearly see, how things evolve under optimization iteration, and where exactly one thing goes wrong. This will be improved as it proves necessary to do just that. So far, this has been rare. Expect this to become end user capable with time. If only to allow you to understand why Nuitka won’t optimize code of yours, and what change of Nuitka it will need to improve.

The comparative performance benchmarking is clearly the most important thing to have for users. It deserves to be the top priority. Thanks to the PyPy tool vmprof, we may already be there on the data taking side, but the presenting and correlation part, is still open and a fair bit of work. It will be most important to empower users to make competent performance bug reports, now that Nuitka enters the phase, where these things matter.

As this is a lot of ground to cover. More than ever. We can make this compiler, but only if you help, it will arrive in your life time.

Nuitka Progress in Summer 2015

2015-10-05T08:08:54+01:00

A long time has passed again without me speaking about what’s going on with Nuitka, and that although definitely a lot has happened. I would contend it’s even because so much is going on.

I also am shy to make public postings about unfinished stuff it seems, but it’s long overdue, so much important and great stuff has happened. We are in the middle of big things with the compiler and there is a lot of great achievement.

SSA (Single State Assignment Form)

For a long, long time already, each release of Nuitka has worked towards increasing “SSA” usage in Nuitka.

Now it’s there. The current pre-release just uses it. There were many things to consider before enabling it, and always a next thing to be found that was needed. Often good changes to Nuitka, it was also annoying the hell out of me at times.

But basically now the forward propagation of variables is in place, with some limitations that are going to fall later.

So the current release, soon to be replaced, still doesn’t optimize this code as well as possible:

def f():
    a = 1
    return a

But starting with the next release, the value of a is forward propagated (also in way more complex situations), and that’s a serious milestone for the project.

Function Inlining

When submitting my talk to EuroPython 2015, I was putting a lot of pressure on me by promising to demo that. And I did. It was based on the SSA code that only now became completely reliable, but otherwise very few few other changes, and it just worked.

The example I used is this:

def f():
    def g(x, y):
        return x, y

    x = 2
    y = 1

    x, y = g(x, y)  # can be inlined

    return x, y

So, the function g is forward propagated to a direct call, as are x and y into the return statement after making the in-line, making this:

def f():
    return 2, 1

Currently function in-lining is not yet activated by default, for this I am waiting for a release cycle to carry the load of SSA in the wild. As you probably know I usually tend to be conservative and to not make too many changes at once.

And as this works for local functions only yet, it’s not too important yet either. This will generally become relevant once we have this working across modules and their globally defined functions or methods. This will be a while until Nuitka gets there.

Scalability

Having got Nuitka’s memory usage under control, it turned out that there are files that can trigger Python recursion RuntimeError exception when using the ast module to build the Nuitka internal tree. People really have code with many thousands of operations to a + operation.

So, Nuitka here learned to include whole modules as bytecode when it is too complex as there is no easy way to expand the stack on Windows at least. That is kind of a limitation of CPython itself I didn’t run into so far, and rather very annoying too.

Compatibility

Very few things are possible here anymore. For the tests, in full compatibility mode, even more often the less good line number is used.

Also the plug-in work is leading to improved compatibility with Qt plugins of PySide and PyQt. Or another example is the multiprocessing module that on Windows is now supposed to fork compiled code too.

Python 3.5

The next release has experimental support for Python 3.5, with the notable exception that async and await, these do not yet work. It passes the existing test suite for CPython3.4 successfully. Passing here means, to pass or fail in the same way as does the uncompiled Python. Failures are of course expected, as details change, and a nice way of having coverage for exception codes.

The new @ operator is now supported. As the stable release of Python3.5 was made recently, there is now some pressure on having full support of course.

I am not sure, if you can fully appreciate the catch up game to play here. It will take a compiled coroutine to support these things properly. And that poses lots of puzzles to solve. As usual I am binding these to internal cleanups so it becomes simpler.

In the case of Python3.5, the single function body node type that is used for generators, class bodies, and function, is bound to be replaced with a base class and detailing instances, instead of one thing for them all, then with coroutines added.

Importing Enhancements

A while ago, the import logic was basically re-written with compatibility much increased. Then quite some issues were fixed. I am not sure, but some of the fixes have apparently also been regressions at times, with the need for other fixes now.

So it may have worked for you in the past, but you might have to report new found issues.

It’s mainly the standalone community that encounters these issues, when just one of these imports doesn’t find the correct thing, but picking the wrong one will of course have seriously bad impacts on compile time analysis too. So once we do cross module optimization, this must be rock solid.

I think we have gotten a long way there, but we still need to tackle some more fine details.

Performance

Graphs and Benchmarks

I also presented this weak point to EuroPython 2015 and my plan on how to resolve it. And low and behold, turns out the PyPy people had already developed a tool that will be usable for the task and to present to the conference.

So basically I was capable of doing kind of a prototype of comparative benchmark during EuroPython 2015 already. I will need to complete this. My plan was to get code names of functions sorted out in a better way, to more easily match the Nuitka C function names with Python functions in an automatic way. That matching is the hard part.

So that is already progressing, but I could need help with that definitely.

Nuitka really has to catch up with benchmarks generally.. The work on automated performance graphs has made more progress, and they are supposed to show up on Nuitka Speedcenter each time, master, develop, or factory git branches change.

Note

There currently is no structure to these graphs. There is no explanations or comments, and there is no trend indicators. All of which makes it basically useless to everybody except me. And even harder for me than necessary.

As a glimpse of what is possible with in-lined functions, look at this:

Lambda call construct case

But we also need to put real programs and use cases to test. This may need your help. Let me know if you want to. It takes work on taking the data, and merging them into one view, linking it with the source code ideally. That will be the tool you can just use on your own code.

Standalone

The standalone mode of Nuitka was pretty good, and continued to improve further, now largely with the help of plug-ins.

I now know that PyGTK is an issue and will need a plug-in to work. Once the plug-in interface is public, I hope for more outside contributions here.

Other Stuff

Funding

Nuitka receives the occasional donation and those make me very happy. As there is no support from organization like the PSF, I am all on my own there.

This year I traveled to Europython 2015, I needed a new desktop computer after burning the old one through with CI tests, the website has running costs, and so on. That is pretty hefty money. It would be sweet if aside of my free time it wouldn’t also cost me money.

EuroPython 2015

This was a blast. Meeting people who knew Nuitka but not me was a regular occurrence. And many people well appreciate my work. It felt much different than the years before.

I was able to present Nuitka’s function in-lining indeed there, and this high goal that I set myself, quite impressed people. My talk went very well, I am going to post a link separately in another post.

Collaborators

Nuitka is making break through progress. And you can be a part of it. Now.

Future

So, there is multiple things going on:

Function in-lining

For locally declared functions, it should become possible to avoid their creation, and make direct calls instead of ones that use function objects and expensive parameter handling.
Nested frames

One result of in-lining will be nested frames still present for exceptions to be properly annotated, or locals giving different sets of locals and so on.

Some cleanup of these will be needed for code generation and SSA to be able to attach variables to some sort of container, and for a function to be able to reference different sets of these.
Type Inference

With SSA in place, we really can start to recognize types, and treat things that work something assigned from [] different, and with code special to these.

That’s going to be a lot of work. For float and list there are very important use cases, where the code can be much better.
Shape Analyisis

My plan for types, is not to use them, but the more general shapes, things that will be more prevalent than actual type information in a program. In fact the precise knowledge will be rare, but more often, we will just have a set of operations performed on a variable, and be able to guess from there.
Python 3.5 new features

The coroutines are a new type, and currently it’s unclear how deep this is tied into the core of things, i.e. if a compile coroutine can be a premier citizen immediately, or if that needs more work. I hope it just takes for the code object to have the proper flag. But there could be stupid type checks, we shall see.
Plug-ins

Something I wish I could have shown at EuroPython was plug-ins to Nuitka. It is recently becoming more complete, and some demo plug-ins for say Qt plugins, or multiprocessing, are starting to work. The API will need work and of course documentation. Hope is for this to expand Nuitka’s reach and appeal to get more contributors.

Let me know, if you are willing to help. I really need that help to make things happen faster. Nuitka will become more and more important only.

Nuitka Release 0.5.14

2015-08-28T07:03:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is an intermediate step towards value propagation, which is not considered ready for stable release yet. The major point is the elimination of the try/finally expressions, as they are problems to SSA. The try/finally statement change is delayed.

There are also a lot of bug fixes, and enhancements to code generation, as well as major cleanups of code base.

Bug Fixes

Python3: Added support assignments trailing star assignment.
```
*a, b = 1, 2
```
This raised ValueError before.
Python3: Properly detect illegal double star assignments.
```
*a, *b = c
```
Python3: Properly detect the syntax error to star assign from non-tuple/list.
```
*a = 1
```
Python3.4: Fixed a crash of the binary when copying dictionaries with split tables received as star arguments.
Python3: Fixed reference loss, when using raise a from b where b was an exception instance. Fixed in 0.5.13.8 already.
Windows: Fix, the flag --disable-windows-console was not properly handled for MinGW32 run time resulting in a crash.
Python2.7.10: Was not recognizing this as a 2.7.x variant and therefore not applying minor version compatibility levels properly.
Fix, when choosing to have frozen source references, code objects were not use the same value as __file__ did for its filename.
Fix, when re-executing itself to drop the site module, make sure we find the same file again, and not according to the PYTHONPATH changes coming from it. Fixed in 0.5.13.4 already.
Enhanced code generation for del variable statements, where it’s clear that the value must be assigned.
When pressing CTRL-C, the stack traces from both Nuitka and Scons were given, we now avoid the one from Scons.
Fix, the dump from --xml no longer contains functions that have become unused during analysis.
Standalone: Creating or running programs from inside Unicode paths was not working on Windows. Fixed in 0.5.13.7 already.
Namespace package support was not yet complete, importing the parent of a package was still failing. Fixed in 0.5.13.7 already.
Python2.6: Compatibility for exception check messages enhanced with newest minor releases.
Compatibility: The NameError in classes needs to say global name and not just name too.
Python3: Fixed creation of XML representation, now done without lxml as it doesn’t support needed features on that version. Fixed in 0.5.13.5 already.
Python2: Fix, when creating code for the largest negative constant to still fit into int, that was only working in the main module. Fixed in 0.5.13.5 already.
Compatibility: The print statement raised an assertion on unicode objects that could not be encoded with ascii codec.

New Features

Added support for Windows 10.
Followed changes for Python 3.5 beta 2. Still only usable as a Python 3.4 replacement, no new features.
Using a self compiled Python running from the source tree is now supported.
Added support for Anaconda Python distribution. As it doesn’t install the Python DLL, we copy it along for acceleration mode.
Added support for Visual Studio 2015. Fixed in 0.5.13.3 already.
Added support for self compiled Python versions running from build tree, this is intended to help debug things on Windows.

Optimization

Function in-lining is now present in the code, but still disabled, because it needs more changes in other areas, before we can generally do it.
Trivial outlines, result of re-formulations or function in-lining, are now in-lined, in case they just return an expression.
The re-formulation for or and and has been giving up, eliminating the use of a try/finally expression, at the cost of dedicated boolean nodes and code generation for these.

This saves around 8% of compile time memory for Nuitka, and allows for faster and more complete optimization, and gets rid of a complicated structure for analysis.
When a frame is used in an exception, its locals are detached. This was done more often than necessary and even for frames that are not necessary our own ones. This will speed up some exception cases.
When the default arguments, or the keyword default arguments (Python3) or the annotations (Python3) were raising an exception, the function definition is now replaced with the exception, saving a code generation. This happens frequently with Python2/Python3 compatible code guarded by version checks.
The SSA analysis for loops now properly traces “break” statement situations and merges the post-loop situation from all of them. This significantly allows for and improves optimization of code following the loop.
The SSA analysis of try/finally statements has been greatly enhanced. The handler for finally is now optimized for exception raise and no exception raise individually, as well as for break, continue and return in the tried code. The SSA analysis for after the statement is now the result of merging these different cases, should they not abort.
The code generation for del statements is now taking advantage should there be definite knowledge of previous value. This speed them up slightly.
The SSA analysis of del statements now properly decided if the statement can raise or not, allowing for more optimization.
For list contractions, the re-formulation was enhanced using the new outline construct instead of a pseudo function, leading to better analysis and code generation.
Comparison chains are now re-formulated into outlines too, allowing for better analysis of them.
Exceptions raised in function creations, e.g. in default values, are now propagated, eliminating the function’s code. This happens most often with Python2/Python3 in branches. On the other hand, function creations that cannot are also annotated now.
Closure variables that become unreferenced outside of the function become normal variables leading to better tracing and code generation for them.
Function creations cannot raise except their defaults, keyword defaults or annotations do.
Built-in references can now be converted to strings at compile time, e.g. when printed.

Organizational

Removed gitorious mirror of the git repository, they shut down.
Make it more clear in the documentation that Python2 is needed at compile time to create Python3 executables.

Cleanups

Moved more parts of code generation to their own modules, and used registry for code generation for more expression kinds.
Unified try/except and try/finally into a single construct that handles both through try/except/break/continue/return semantics. Finally is now solved via duplicating the handler into cases necessary.

No longer are nodes annotated with information if they need to publish the exception or not, this is now all done with the dedicated nodes.
The try/finally expressions have been replaced with outline function bodies, that instead of side effect statements, are more like functions with return values, allowing for easier analysis and dedicated code generation of much lower complexity.
No more “tolerant” flag for release nodes, we now decide this fully based on SSA information.
Added helper for assertions that code flow does not reach certain positions, e.g. a function must return or raise, aborting statements do not continue and so on.
To keep cloning of code parts as simple as possible, the limited use of makeCloneAt has been changed to a new makeClone which produces identical copies, which is what we always do. And a generic cloning based on “details” has been added, requiring to make constructor arguments and details complete and consistent.
The re-formulation code helpers have been improved to be more convenient at creating nodes.
The old nuitka.codegen module Generator was still used for many things. These now all got moved to appropriate code generation modules, and their users got updated, also moving some code generator functions in the process.
The module nuitka.codegen.CodeTemplates got replaces with direct uses of the proper topic module from nuitka.codegen.templates, with some more added, and their names harmonized to be more easily recognizable.
Added more assertions to the generated code, to aid bug finding.
The auto-format now sorts pylint markups for increased consistency.
Releases no longer have a tolerant flag, this was not needed anymore as we use SSA.
Handle CTRL-C in scons code preventing per job messages that are not helpful and avoid tracebacks from scons, also remove more unused tools like rpm from out in-line copy.

Tests

Added the CPython3.4 test suite.
The CPython3.2, CPython3.3, and CPython3.4 test suite now run with Python2 giving the same errors. Previously there were a few specific errors, some with line numbers, some with different SyntaxError be raised, due to different order of checks.

This increases the coverage of the exception raising tests somewhat.
Also the CPython3.x test suites now all pass with debug Python, as does the CPython 2.6 test suite with 2.6 now.
Added tests to cover all forms of unpacking assignments supported in Python3, to be sure there are no other errors unknown to us.
Started to document the reference count tests, and to make it more robust against SSA optimization. This will take some time and is work in progress.
Made the compile library test robust against modules that raise a syntax error, checking that Nuitka does the same.
Refined more tests to be directly executable with Python3, this is an ongoing effort.

Summary

This release is clearly major. It represents a huge step forward for Nuitka as it improves nearly every aspect of code generation and analysis. Removing the try/finally expression nodes proved to be necessary in order to even have the correct SSA in their cases. Very important optimization was blocked by it.

Going forward, the try/finally statements will be removed and dead variable elimination will happen, which then will give function inlining. This is expected to happen in one of the next releases.

This release is a consolidation of 8 hotfix releases, and many refactorings needed towards the next big step, which might also break things, and for that reason is going to get its own release cycle.

Nuitka Release 0.5.13

2015-05-02T09:45:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains the first use of SSA for value propagation and massive amounts of bug fixes and optimization. Some of the bugs that were delivered as hotfixes, were only revealed when doing the value propagation as they still could apply to real code.

Bug Fixes

Fix, relative imports in packages were not working with absolute imports enabled via future flags. Fixed in 0.5.12.1 already.
Loops were not properly degrading knowledge from inside the loop at loop exit, and therefore this could have lead missing checks and releases in code generation for cases, for del statements in the loop body. Fixed in 0.5.12.1 already.
The or and and re-formulation could trigger false assertions, due to early releases for compatibility. Fixed in 0.5.12.1 already.
Fix, optimizion of calls of constant objects (always an exception), crashed the compiler.Fixed in 0.5.12.2 already.
Standalone: Added support for site.py installations with a leading def or class statement, which is defeating our attempt to patch __file__ for it.
Compatibility: In full compatibility mode, the tracebacks of or and and expressions are now as wrong as they are in CPython. Does not apply to --improved mode.
Standalone: Added missing dependency on QtGui by QtWidgets for PyQt5.
macOS: Improved parsing of otool output to avoid duplicate entries, which can also be entirely wrong in the case of Qt plugins at least.
Avoid relative paths for main program with file reference mode original, as it otherwise changes as the file moves.
MinGW: The created modules depended on MinGW to be in PATH for their usage. This is no longer necessary, as we now link these libraries statically for modules too.
Windows: For modules, the option --run to immediately load the modules had been broken for a while.
Standalone: Ignore Windows DLLs that were attempted to be loaded, but then failed to load. This happens e.g. when both PySide and PyQt are installed, and could cause the dreaded conflicting DLLs message. The DLL loaded in error is now ignored, which avoids this.
MinGW: The resource file used might be empty, in which case it doesn’t get created, avoiding an error due to that.
MinGW: Modules can now be created again. The run time relative code uses an API that is WinXP only, and MinGW failed to find it without guidance.

Optimization

Make direct calls out of called function creations. Initially this applies to lambda functions only, but it’s expected to become common place in coming releases. This is now 20x faster than CPython.
```
# Nuitka avoids creating a function object, parsing function arguments:
(lambda x: x)(something)
```
Propagate assignments from non-mutable constants forward based on SSA information. This is the first step of using SSA for real compile time optimization.
Specialized the creation of call nodes at creation, avoiding to have all kinds be the most flexible form (keyword and plain arguments), but instead only what kind of call they really are. This saves lots of memory, and makes the tree faster to visit.
Added support for optimizing the slice built-in with compile time constant arguments to constants. The re-formulation for slices in Python3 uses these a lot. And the lack of this optimization prevented a bunch of optimization in this area. For Python2 the built-in is optimized too, but not as important probably.
Added support for optimizing isinstance calls with compile time constant arguments. This avoids static exception raises in the exec re-formulation which tests for file type, and then optimization couldn’t tell that a str is not a file instance. Now it can.
Lower in-place operations on immutable types to normal operations. This will allow to compile time compute these more accurately.
The re-formulation of loops puts the loop condition as a conditional statement with break. The not that needs to apply was only added in later optimization, leading to unnecessary compile time efforts.
Removed per variable trace visit from optimization, removing useless code and compile time overhead. We are going to optimize things by making decision in assignment and reference nodes based on forward looking statements using the last trace collection.

New Features

Added experimental support for Python 3.5, which seems to be passing the test suites just fine. The new @ matrix multiplicator operators are not yet supported though.
Added support for patching source on the fly. This is used to work around a (now fixed) issue with numexpr.cpuinfo making type checks with the is operation, about the only thing we cannot detect.

Organizational

Added repository for Ubuntu Vivid (15.04) for download. Removed Ubuntu Saucy and Ubuntu Raring package downloads, these are no longer supported by Ubuntu.
Added repository for Debian Stretch, after Jessie release.
Make it more clear in the documentation that in order to compile Python3, a Python2 is needed to execute Scons, but that the end result is a Python3 binary.
The PyLint checker tool now can operate on directories given on the command line, and whitelists an error that is Windows only.

Cleanups

Split up standalone code further, moving depends.exe handling to a separate module.
Reduced code complexity of scons interface.
Cleaned up where trace collection is being done. It was partially still done inside the collection itself instead in the owner.
In case of conflicting DLLs for standalone mode, these are now output with nicer formatting, that makes it easy to recognize what is going on.
Moved code to fetch depends.exe to dedicated module, so it’s not as much in the way of standalone code.

Tests

Made BuiltinsTest directly executable with Python3.
Added construct test to demonstrate the speed up of direct lambda calls.
The deletion of @test for the CPython test suite is more robust now, esp. on Windows, the symbolic links are now handled.
Added test to cover or usage with in-place assignment.
Cover local relative import from . with absolute_import future flag enabled.
Again, more basic tests are now directly executable with Python3.

Summary

This release is major due to amount of ground covered. The reduction in memory usage of Nuitka itself (the C++ compiler will still use much memory) is very massive and an important aspect of scalability too.

Then the SSA changes are truly the first sign of major improvements to come. In their current form, without eliminating dead assignments, the full advantage is not taken yet, but the next releases will do this, and that’s a major milestone to Nuitka.

The other optimization mostly stem from looking at things closer, and trying to work towards function in-lining, for which we are making a lot of progress now.

Nuitka Progress in Spring 2015

2015-04-30T16:08:54+01:00

It’s absolutely time to speak about what’s going on with Nuitka, there have been a few releases, and big things are going to happen now. The ones I have always talked of, it’s happening now.

I absolutely prefer to talk of things when they are completed, that is why I am shy to make these kinds of postings, but this time, I think it’s warranted. The next couple of releases are going to be very different.

SSA (Single State Assignment Form)

For a long, long time already, each release of Nuitka has worked towards increasing “SSA” usage in Nuitka.

The component that works on this, is now called “trace collection”, and does the major driving part for optimization. It collects “variable traces” and puts them together into “global” forms as well.

Based on these traces, optimizations can be made. Having SSA or not, is (to me) the difference between Nuitka as a mere compiler, and Nuitka as an optimizing compiler.

The major news is that factory versions of Nuitka now do this in serious ways, propagating values forward, and we also are close to eliminating dead assignments, some of which become dead by being having been forward propagated.

So we can now finally see that big step, jump really, happening, and Nuitka does now do some pretty good static optimization, at least locally.

Still, right now, this trivial code assigns to a local variable, then reads from it to return. But not for much longer.

def f():
    a = 1
    return a

This is going to instantly give performance gains, and more importantly, will enable analysis, that leads to avoiding e.g. the creation of function objects for local functions, becoming able to in-line, etc.

This is major excitement to me. And I cannot wait to have the releases that do this.

Scalability

The focus has also been lately, to reduce Nuitka’s own memory usage. It has gone down by a large factor, often by avoiding cyclic dependencies in the data structures, that the garbage collector of Python failed to deal with properly.

The scalability of Nuitka also depends much on generated code size. With the optimization become more clever, less code needs to be generated, and that will help a lot. On some platforms, MSVC most notably, it can be really slow, but it’s noteworthy that Nuitka works not just with 2008 edition, but with the latest MSVC, which appears to be better.

Compatibility

There was not a whole lot to gain in the compatibility domain anymore. Nothing important certainly. But there are import changes.

Python 3.5

The next release has changes to compile and run the Python3.4 test suite successfully. Passing here means, to pass/fail in the same way as does the uncompiled Python. Failures are of course expected, and a nice way of having coverage for exception codes.

The new @ operator is not supported yet. I will wait with that for things to stabilize. It’s currently only an alpha release.

However, Nuitka has probably never been this close to supporting a new Python version at release time. And since 3.4 was such a heavy drain, and still not perfectly handled (super still works like it’s 3.3 e.g.), I wanted to know what is coming a bit sooner.

Cells for Closure

We now provide a __closure__ value for compiled functions too. These are not writable in Python, so it’s only a view. Having moved storage into the compiled function object, that was easy.

Importing Enhancements

The the past couple of releases, the import logic was basically re-written with compatibility much increased. The handling of file case during import, multiple occurrences in the path, and absolute import future flags for relative imports has been added.

It’s mainly the standalone community that will have issues, when just one of these imports doesn’t find the correct thing, but picking the wrong one will of course have seriously bad impacts on compile time analysis too. So once we do cross module optimization, this must be rock solid.

I think we have gotten there, tackling these finer details now too.

Performance

Graphs and Benchmarks

Nuitka, users don’t know what to expect regarding the speed of their code after compilation through Nuitka, neither now nor after type inference (possibly hard to guess). Nuitka does a bunch of optimizations for some constructs pretty heavily, but weak at others. But how much does that affect real code?

There may well be no significant gain at all for many people, while there is a number for PyStone that suggests higher. The current and future versions possibly do speed up but the point is that you cannot tell if it is even worth for someone to try.

Nuitka really has to catch up here. The work on automated performance graphs has some made progress, and they are supposed to show up on Nuitka Speedcenter each time, master, develop or factory git branches change.

Note

However, as a glimpse of what will happen when we in-line functions, take a look at the case, where we already eliminate parameter parsing only, and make tremendous speedups:

Lambda call construct case

Right now (the graph gets automatic updates with each change), what you should see, is that develop branch is 20 times faster than CPython for that very specific bit of code. That is where we want to be, except that with actually in-line, this will of course be even better.

It’s artificial, but once we can forward propagate local function creations, it will apply there too. The puzzle completes.

But we also need to put real programs and use cases to test. This may need your help. Let me know if you want to.

Standalone

The standalone mode of Nuitka is pretty good, and as usual it continued to improve only.

Nothing all that important going on there, except the work on a plug-in framework, which is under development, and being used to handle e.g. PyQt plug-ins, or known issues with certain packages.

The importing improvements already mentioned, have now allowed to cover many more libraries successfully than before.

Other Stuff

Debian Stable

Nuitka is now part of Debian stable, aka Jessie. Debian and Python are the two things closest to my heart in the tech field. You can imagine that being an upstream worthy of inclusion into Debian stable is an import milestone to Nuitka for me.

Funding

Nuitka receives the occasional donation and those make me very happy. As there is no support from organization like the PSF, I am all on my own there.

This year I likely will travel to Europython 2015, and would ask you to support me with that, it’s going to be expensive.

EuroPython 2015

I have plans to present Nuitka’s function in-lining there, real stuff, on a fully and functional compiler that works as a drop-in replacement.

Not 100% sure if I can make it by the time, but things look good. Actually so far I felt ahead of the plan, but as you know, this can easily change at any point. But Nuitka stands on very stable grounds code wise.

Collaborators

Things are coming along nicely. When I started out, I was fully aware that the project is something that I can do on my own if necessary, and that has not changed. Things are going slower than necessary though, but that’s probably very typical.

But you can join and should do so now, just follow this link or become part of the mailing list (since closed) and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Nuitka is about to make break through progress. And you can be a part of it. Now.

Future

So, there is multiple things going on:

More SSA usage

The next releases are going to be all about getting this done.

Once we take it to that next level, Nuitka will be able to speed up some things by much more than the factor it basically has provided for 2 years now, and it’s probably going to happen long before EuroPython 2015.
Function in-lining

For locally declared functions, it should become possible to avoid their creation, and make direct calls instead of ones that use function objects and expensive parameter handling.

The next step there of course is to not only bind the arguments to the function signature, but then also to in-line and potentially specialize the function code. It’s my goal to have that at EuroPython 2015 in a form ready to show off.

When these 2 things come to term, Nuitka will have made really huge steps ahead and laid the ground for success.

From then on, a boatload of work remains. The infrastructure in place, still there is going to be plenty of work to optimize more and more things conretely, and to e.g. do type inference, and generate different codes for booleans, ints or float values.

Let me know, if you are willing to help. I really need that help to make things happen faster. Nuitka will become more and more important only.

Nuitka Release 0.5.12

2015-04-07T17:55:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains massive amounts of corrections for long standing issues in the import recursion mechanism, as well as for standalone issues now visible after the __file__ and __path__ values have changed to become run time dependent values.

Bug Fixes

Fix, the __path__ attribute for packages was still the original filename’s directory, even in file reference mode was runtime.
The use of runtime as default file reference mode for executables, even if not in standalone mode, was making acceleration harder than necessary. Changed to original for that case. Fixed in 0.5.11.1 already.
The constant value for the smallest int that is not yet a long is created using 1 due to C compiler limitations, but 1 was not yet initialized properly, if this was a global constant, i.e. used in multiple modules. Fixed in 0.5.11.2 already.
Standalone: Recent fixes around __path__ revealed issues with PyWin32, where modules from win32com.shell were not properly recursed to. Fixed in 0.5.11.2 already.
The importing of modules with the same name as a built-in module inside a package falsely assumed these were the built-ins which need not exist, and then didn’t recurse into them. This affected standalone mode the most, as the module was then missing entirely.
```
# Inside "x.y" module:
import x.y.exceptions
```
Similarly, the importing of modules with the same name as standard library modules could go wrong.
```
# Inside "x.y" module:
import x.y.types
```
Importing modules on Windows and macOS was not properly checking the checking the case, making it associate wrong modules from files with mismatching case.
Standalone: Importing with from __future__ import absolute_import would prefer relative imports still.
Python3: Code generation for try/return expr/finally could loose exceptions when expr raised an exception, leading to a RuntimeError for NULL return value. The real exception was lost.
Lambda expressions that were directly called with star arguments caused the compiler to crash.
```
(lambda *args: args)(*args)  # was crashing Nuitka
```

Optimization

Focusing on compile time memory usage, cyclic dependencies of trace merges that prevented them from being released, even when replaced were removed.
More memory efficient updating of global SSA traces, reducing memory usage during optimization by ca. 50%.
Code paths that cannot and therefore must not happen are now more clearly indicated to the backend compiler, allowing for slightly better code to be generated by it, as it can tell that certain code flows need not be merged.

New Features

Standalone: On systems, where .pth files inject Python packages at launch, these are now detected, and taking into account. Previously Nuitka did not recognize them, due to lack of __init__.py files. These are mostly pip installations of e.g. zope.interface.
Added option --explain-imports to debug the import resolution code of Nuitka.
Added options --show-memory to display the amount of memory used in total and how it’s spread across the different node types during compilation.
The option --trace-execution now also covers early program initialisation before any Python code runs, to ease finding bugs in this domain as well.

Organizational

Changed default for file reference mode to original unless standalone or module mode are used. For mere acceleration, breaking the reading of data files from __file__ is useless.
Added check that the in-line copy of scons is not run with Python3, which is not supported. Nuitka works fine with Python3, but a Python2 is required to execute scons.
Discover more kinds of Python2 installations on Linux/macOS installations.
Added instructions for macOS to the download page.

Cleanups

Moved oset and odict modules which provide ordered sets and dictionaries into a new package nuitka.container to clean up the top level scope.
Moved SyntaxErrors to nuitka.tree package, where it is used to format error messages.
Moved nuitka.Utils package to nuitka.utils.Utils creating a whole package for utils, so as to better structure them for their purpose.

Summary

This release is a major maintenance release. Support for namespace modules injected by *.pth is a major step for new compatibility. The import logic improvements expand the ability of standalone mode widely. Many more use cases will now work out of the box, and less errors will be found on case insensitive systems.

There is aside of memory issues, no new optimization though as many of these improvements could not be delivered as hotfixes (too invasive code changes), and should be out to the users as a stable release. Real optimization changes have been postponed to be next release.

Nuitka Release 0.5.11

2015-03-19T20:54:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

The last release represented a significant change and introduced a few regressions, which got addressed with hot fix releases. But it also had a focus on cleaning up open optimization issues that were postponed in the last release.

New Features

The filenames of source files as found in the __file__ attribute are now made relative for all modes, not just standalone mode.

This makes it possible to put data files along side compiled modules in a deployment.

Bug Fixes

Local functions that reference themselves were not released. They now are.
```
def someFunction():
    def f():
        f()  # referencing 'f' in 'f' caused the garbage collection to fail.
```
Recent changes to code generation attached closure variable values to the function object, so now they can be properly visited. Fixed in 0.5.10.1 already.
Python2.6: The complex constants with real or imaginary parts -0.0 were collapsed with constants of value 0.0. This became more evident after we started to optimize the complex built-in. Fixed in 0.5.10.1 already.
```
complex(0.0, 0.0)
complex(-0.0, -0.0)  # Could be confused with the above.
```
Complex call helpers could leak references to their arguments. This was a regression. Fixed in 0.5.10.1 already.
Parameter variables offered as closure variables were not properly released, only the cell object was, but not the value. This was a regression. Fixed in 0.5.10.1 already.
Compatibility: The exception type given when accessing local variable values not initialized in a closure taking function, needs to be NameError and UnboundLocalError for accesses in the providing function. Fixed in 0.5.10.1 already.
Fix support for “venv” on systems, where the system Python uses symbolic links too. This is the case on at least on Mageia Linux. Fixed in 0.5.10.2 already.
Python3.4: On systems where long and Py_ssize_t are different (e.g. Win64) iterators could be corrupted if used by uncompiled Python code. Fixed in 0.5.10.2 already.
Fix, generator objects didn’t release weak references to them properly. Fixed in 0.5.10.2 already.
Compatibility: The __closure__ attributes of functions was so far not supported, and rarely missing. Recent changes made it easy to expose, so now it was added.
macOS: A linker warning about deprecated linker option -s was solved by removing the option.
Compatibility: Nuitka was enforcing that the __doc__ attribute to be a string object, and gave a misleading error message. This check must not be done though, __doc__ can be any type in Python.

Optimization

Variables that need not be shared, because the uses in closure taking functions were eliminated, no longer use cell objects.
The try/except and try/finally statements now both have actual merging for SSA, allowing for better optimization of code behind it.
```
def f():

    try:
        a = something()
    except:
        return 2

    # Since the above exception handling cannot continue the code flow,
    # we do not have to invalidate the trace of "a", and e.g. do not have
    # to generate code to check if it's assigned.
    return a
```
Since try/finally is used in almost all re-formulations of complex Python constructs this is improving SSA application widely. The uses of try/except in user code will no longer degrade optimization and code generation efficiency as much as they did.
The try/except statement now reduces the scope of tried block if possible. When no statement raised, already the handling was removed, but leading and trailing statements that cannot raise, were not considered.
```
def f():

    try:
        b = 1
        a = something()
        c = 1
    except:
        return 2
```
This is now optimized to.
```
def f():

    b = 1
    try:
        a = something()
    except:
        return 2
    c = 1
```
The impact may on execution speed may be marginal, but it is definitely going to improve the branch merging to be added later. Note that c can only be optimized, because the exception handler is aborting, otherwise it would change behaviour.
The creation of code objects for standalone mode and now all code objects was creating a distinct filename object for every function in a module, despite them being same content. This was wasteful for module loading. Now it’s done only once.

Also, when having multiple modules, the code to build the run time filename used for code objects, was calling import logic, and doing lookups to find os.path.join again and again. These are now cached, speeding up the use of many modules as well.

Cleanups

Nuitka used to have “variable usage profiles” and still used them to decide if a global variable is written to, in which case, it stays away from doing optimization of it to built-in lookups, and later calls.

The have been replaced by “global variable traces”, which collect the traces to a variable across all modules and functions. While this is now only a replacement, and getting rid of old code, and basing on SSA, later it will also allow to become more correct and more optimized.
The standalone now queries its hidden dependencies from a plugin framework, which will become an interface to Nuitka internals in the future.

Testing

The use of deep hashing of constants allows us to check if constants become mutated during the run-time of a program. This allows to discover corruption should we encounter it.
The tests of CPython are now also run with Python in debug mode, but only on Linux, enhancing reference leak coverage.
The CPython test parts which had been disabled due to reference cycles involving compiled functions, or usage of __closure__ attribute, were reactivated.

Organizational

Since Google Code has shutdown, it has been removed from the Nuitka git mirrors.

Summary

This release brings exciting new optimization with the focus on the try constructs, now being done more optimal. It is also a maintenance release, bringing out compatibility improvements, and important bug fixes, and important usability features for the deployment of modules and packages, that further expand the use cases of Nuitka.

The git flow had to be applied this time to get out fixes for regression bug fixes, that the big change of the last release brought, so this is also to consolidate these and the other corrections into a full release before making more invasive changes.

The cleanups are leading the way to expanded SSA applied to global variable and shared variable values as well. Already the built-in detect is now based on global SSA information, which was an important step ahead.

Nuitka Release 0.5.10

2015-03-06T07:07:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a focus on code generation optimization. Doing major changes away from “C++-ish” code to “C-ish” code, many constructs are now faster or got looked at and optimized.

Bug Fixes

Compatibility: The variable name in locals for the iterator provided to the generator expression should be .0, now it is.
Generators could leak frames until program exit, these are now properly freed immediately.

Optimization

Faster exception save and restore functions that might be in-lined by the backend C compiler.
Faster error checks for many operations, where these errors are expected, e.g. instance attribute lookups.
Do not create traceback and locals dictionary for frame when StopIteration or GeneratorExit are raised. These tracebacks were wasted, as they were immediately released afterwards.
Closure variables to functions and parameters of generator functions are now attached to the function and generator objects.
The creation of functions with closure taking was accelerated.
The creation and destruction of generator objects was accelerated.
The re-formulation for in-place assignments got simplified and got faster doing so.
In-place operations of str were always copying the string, even if was not necessary.
```
a += b  # Was not re-using the storage of "a" in case of strings
```
Python2: Additions of int for Python2 are now even faster.
Access to local variable values got slightly accelerated at the expense of closure variables.
Added support for optimizing the complex built-in.
Removing unused temporary and local variables as a result of optimization, these previously still allocated storage.

Cleanup

The use of C++ classes for variable objects was removed. Closure variables are now attached as PyCellObject to the function objects owning them.
The use of C++ context classes for closure taking and generator parameters has been replaced with attaching values directly to functions and generator objects.
The indentation of code template instantiations spanning multiple was not in all cases proper. We were using emission objects that handle it new lines in code and mere list objects, that don’t handle them in mixed forms. Now only the emission objects are used.
Some templates with C++ helper functions that had no variables got changed to be properly formatted templates.
The internal API for handling of exceptions is now more consistent and used more efficiently.
The printing helpers got cleaned up and moved to static code, removing any need for forward declaration.
The use of INCREASE_REFCOUNT_X was removed, it got replaced with proper Py_XINCREF usages. The function was once required before “C-ish” lifted the need to do everything in one function call.
The use of INCREASE_REFCOUNT got reduced. See above for why that is any good. The idea is that Py_INCREF must be good enough, and that we want to avoid the C function it was, even if in-lined.
The assertObject function that checks if an object is not NULL and has positive reference count, i.e. is sane, got turned into a preprocessor macro.
Deep hashes of constant values created in --debug mode, which cover also mutable values, and attempt to depend on actual content. These are checked at program exit for corruption. This may help uncover bugs.

Organizational

Speedcenter has been enhanced with better graphing and has more benchmarks now. More work will be needed to make it useful.
Updates to the Developer Manual, reflecting the current near finished state of “C-ish” code generation.

Tests

New reference count tests to cover generator expressions and their usage got added.
Many new construct-based tests got added, these will be used for performance graphing, and serve as micro benchmarks now.
Again, more basic tests are directly executable with Python3.

Summary

This is the next evolution of “C-ish” coming to pass. The use of C++ has for all practical purposes vanished. It will remain an ongoing activity to clear that up and become real C. The C++ classes were a huge road block to many things, that now will become simpler. One example of these were in-place operations, which now can be dealt with easily.

Also, lots of polishing and tweaking was done while adding construct benchmarks that were made to check the impact of these changes. Here, generators probably stand out the most, as some of the missed optimization got revealed and then addressed.

Their speed increases will be visible to some programs that depend a lot on generators.

This release is clearly major in that the most important issues got addressed, future releases will provide more tuning and completeness, but structurally the “C-ish” migration has succeeded, and now we can reap the benefits in the coming releases. More work will be needed for all in-place operations to be accelerated.

More work will be needed to complete this, but it’s good that this is coming to an end, so we can focus on SSA based optimization for the major gains to be had.

Nuitka progress 2014

2015-03-02T07:08:54+00:00

Again, not much has happened publicly to Nuitka, except for some releases, so it’s time to make a kind of status post, about the really exciting news there is, also looking back at 2014 for Nuitka, and forward of course.

I meant to post this basically since last year, but never got around to it, therefore the 2014 in the title.

SSA (Single State Assignment Form)

For a long, long time already, each release of Nuitka has worked towards enabling “SSA” usage in Nuitka. There is a component called “constraint collection”, which is tasked with driving the optimization, and collecting variable traces.

Based on these traces, optimizations can be made. Having SSA or not, is (to me) the difference between Nuitka as a compiler, and Nuitka as an optimizing compiler.

The news is, SSA has carried the day, and is used throughout code generation for some time now, and gave minor improvements. It has been applied to the temporary and local variable values.

And currently, work is on the way to expand it to module and shared variables, which can get invalidated quite easily, as soon as unknown code is executed. An issue there is to identify all those spots reliably.

And this spring, we are finally going to see the big jump that is happening, once Nuitka starts to use that information to propagate things.

Still, right now, this code assigns to a local variable, then reads from it to return. But not much longer.

def f():
    a = 1
    return a

This is going to instantly give gains, and more importantly, will enable analysis, that leads to avoiding e.g. the creation of function objects for local functions, being able to in-line, etc.

Improved Code Generation

Previously, under the title “C-ish”, Nuitka moved away from C++ based code generation to less C++ based code generated, and more C-ish code. This trend continues, and has lead to removing more code generation improvements.

The important change recently was to remove the usage of the blocking holdouts, the C++ classes used for local variables are closure taking, and release, and move those to be done manually.

This enabled special code generation for in-place operations, which are the most significant improvements of the upcoming release. These were held back on, as with C++ destructors doing the release, it’s practically impossible to deal with values suddenly becoming illegal. Transfer of object ownership needs to be more fluid than could be presented to C++ objects.

Currently, this allows to speed up string in-place operations, which very importantly then, can avoid to memcpy large values potentially. And this is about catching up to CPython in this regard. After that, we will likely be able to expand it to cases where CPython could never do it, e.g. also int objects

Scalability

The scalability of Nuitka depends much on generated code size. With it being less stupid, the generated code is now not only faster, but definitely smaller, and with more optimization, it will only become more practical.

Removing the many C++ classes already gave the backend compiler an easier time. But we need to do more, to e.g. have generic parameter parsing, instead of specialized per function, and module exclusive constants should not be pre-created, but in the module body, when they are used.

Compatibility

There is not a whole lot to gain in the compatibility domain anymore. Nothing important certainly. But there are these minor things.

Cells for Closure

However, since we now use PyCell objects for closure, we could start and provide a real __closure__ value, that could even be writable. We could start supporting that easily.

Local Variable Storage

Currently, local variables use stack storage. Were we to use function object or frame object attached storage, we could provide frame locals that actually work. This may be as simple as to put those in an array on the stack and use the pointer to it.

Suddenly locals would become writable. I am not saying this is useful, just that it’s possible to do this.

Performance

Graphs and Benchmarks

The work on automated performance graphs has made progress, and they are supposed to show up on Nuitka Speedcenter each time, master, develop or factory git branches change.

At least it’s updated to latest Nikola, and uses PyGal for the graphics now, so it’s easier to expand. The plan here, is to integrate with special pages from a Wiki, making it easy to provide comments.

Standalone

The standalone mode of Nuitka is pretty good, and as usual it continued to improve only.

The major improvements came from handling case collisions between modules and packages. One can have Module.py and module/__init__.py and they both are expected to be different, even on Windows, where filenames are case insensitive.

So, giving up on implib and similar, we finally have our own code to scan in a compatible way the file system, and make these determinations, whereas library code exposing functionality, doesn’t handling all things in really the proper way.

Other Stuff

Funding

Nuitka receives some, bit not quite enough donations. There is no support from organizations like e.g. the PSF, and it seems I better not hold my breath for it. I will travel to Europython 2015, and would ask you to support me with that, it’s going to be expensive.

In 2014, with donations, I bought a “Cubox i4-Pro”, which is an ARM based machine with 4 cores, and 2GB RAM. Works from flash, and with the eSATA disk attached, it works nice for continuous integration, which helps me a lot to deliver extremely high quality releases. It’s pretty nice, except that when using all 4 cores, it gets too hot. So “systemd” to the rescue and just limited the Buildbot slave’s service to use 3 cores of CPU maximum and now it runs stable.

Also with donations I bought a Terrabyte SSD, which I use on the desktop to speed up hosting the virtual machines, and my work in general.

And probably more important, the host of “nuitka.net” became a real machine with real hardware last year, and lots more RAM, so I can spare myself of optimizing e.g. MySQL for low memory usage. The monthly fee of that is substantial, but supported from your donations. Thanks a lot!

Collaborators

Nuitka is about to make break through progress. And you can be a part of it. Now.

Future

So, there is multiple things going on:

More “C-ish” code generation

The next release is going to be more “C-ish” than before, and we can start to actually migrate to really “C” language. You can help out if you want to, this is fairly standard cleanups. Just pop up on the mailing list and say so.

This prong of action is coming to a logical end. The “C-ish” project, while not planned from the outset, turns out to be a full success. Initially, I would not have started Nuitka, should I have faced the full complexity of code generation that there is now. So it was good to start with “C++”, but it’s a better Nuitka now.
More SSA usage

The previous releases consolidated on SSA. A few missing optimizations were found, because SSA didn’t realize things, which were then highlighted by code generation being too good, e.g. not using exception variables.

We seem to have an SSA that can be fully trusted now, and while it can be substantially improved (e.g. the try/finally removes all knowledge, although it only needs to do a partial removing of knowledge for the finally block, not for afterwards at all), it will already allow for many nice things to happen.

Once we take it to that next level, Nuitka will be able to speed up some things by much more than the factor it basically has provided for 2 years now, and it’s probably going to happen before summer, or so I hope.
Value propagation

Starting out with simple cases, Nuitka will forward propagate variable values, and start to eliminate variable usages entirely, where they are not needed.

That will make many things much more compact, and faster at run time. We will then try and build “gates” for statements that they cannot pass, so we can e.g. optimize constant things outside of loops, that kind of thing.

When these 3 things come to term, Nuitka will make a huge step ahead. I look forward to demoing function call in-lining, or at least avoiding the argument parsing at EuroPython 2015, making direct calls, which will be way faster than normal calls.

From then on, a boatload of work remains. The infrastructure in place, still there is going to be plenty of work to optimize more and more things conretely.

Let me know, if you are willing to help. I really need that help to make things happen faster.

Nuitka Release 0.5.9

2015-01-30T07:37:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly a maintenance release, bringing out minor compatibility improvements, and some standalone improvements. Also new options to control the recursion into modules are added.

Bug Fixes

Compatibility: Checks for iterators were using PyIter_Check which is buggy when running outside of Python core, because it’s comparing pointers we don’t see. Replaced with HAS_ITERNEXT helper which compares against the pointer as extracting for a real non-iterator object.
```
class Iterable:
    def __init__(self):
        self.consumed = 2

    def __iter__(self):
        return Iterable()


iter(Iterable())  # This is suppose to raise, but didn't with Nuitka
```
Python3: Errors when creating class dictionaries raised by the __prepare__ dictionary (e.g. enum classes with wrong identifiers) were not immediately raised, but only by the type call.

This was not observable, but might have caused issues potentially.
Standalone macOS: Shared libraries and extension modules didn’t have their DLL load paths updated, but only the main binary. This is not sufficient for more complex programs.
Standalone Linux: Shared libraries copied into the .dist folder were read-only and executing chrpath could potentially then fail. This has not been observed, but is a conclusion of macOS fix.
Standalone: When freezing standard library, the path of Nuitka and the current directory remained in the search path, which could lead to looking at the wrong files.

Organizational

The getattr built-in is now optimized for compile time constants if possible, even in the presence of a default argument. This is more a cleanup than actually useful yet.
The calling of PyCFunction from normal Python extension modules got accelerated, especially for the no or single argument cases where Nuitka now avoids building the tuple.

New Features

Added the option --recurse-pattern to include modules per filename, which for Python3 is the only way to not have them in a package automatically.
Added the option --generate-c++-only to only generate the C++ source code without starting the compiler.

Mostly used for debugging and testing coverage. In the later case we do not want the C++ compiler to create any binary, but only to measure what would have been used.

Organizational

Renamed the debug option --c++-only to --recompile-c++-only to make its purpose more clear and there now is --generate-c++-only too.

Tests

Added support for taking coverage of Nuitka in a test run on a given input file.
Added support for taking coverage for all Nuitka test runners, migrating them all to common code for searching.
Added uniform way of reporting skipped tests, not generally used yet.

Summary

This release marks progress towards having coverage testing. Recent releases had made it clear that not all code of Nuitka is actually used at least once in our release tests. We aim at identifying these.

Another direction was to catch cases, where Nuitka leaks exceptions or is subject to leaked exceptions, which revealed previously unnoticed errors.

Important changes have been delayed, e.g. the closure variables will not yet use C++ objects to share storage, but proper PyCellObject for improved compatibility, and to approach a more “C-ish” status. These is unfinished code that does this. And the forward propagation of values is not enabled yet again either.

So this is an interim step to get the bug fixes and improvements accumulated out. Expect more actual changes in the next releases.

Nuitka Release 0.5.8

2015-01-16T05:04:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has mainly a focus on cleanups and compatibility improvements. It also advances standalone support, and a few optimization improvements, but it mostly is a maintenance release, attacking long standing issues.

Bug Fixes

Compatibility Windows macOS: Fix importing on case insensitive systems.

It was not always working properly, if there was both a package Something and something, by merit of having files Something/__init__.py and something.py.
Standalone: The search path was preferring system directories and therefore could have conflicting DLLs.
Fix, the optimization of getattr with predictable result was crashing the compilation. This was a regression, fixed in 0.5.7.1 already.
Compatibility: The name mangling inside classes also needs to be applied to global variables.
Fix, providing clang++ for CXX was mistakenly thinking of it as a g++ and making version checks on it.
Python3: Declaring __class__ global is now a SyntaxError before Python3.4.
Standalone Python3: Making use of module state in extension modules was not working properly.

New Features

The filenames of source files as found in the __file__ attribute are now made relative in standalone mode.

This should make it more apparent if things outside of the distribution folder are used, at the cost of tracebacks. Expect the default ability to copy the source code along in an upcoming release.
Added experimental standalone mode support for PyQt5. At least headless mode should be working, plugins (needed for anything graphical) are not yet copied and will need more work.

Cleanup

No longer using imp.find_module anymore. To solve the casing issues we needed to make our own module finding implementation finally.
The name mangling was handled during code generation only. Moved to tree building instead.
More code generation cleanups. The compatible line numbers are now attached during tree building and therefore better preserved, as well as that code no longer polluting code generation as much.

Organizational

No more packages for openSUSE 12.1/12.2/12.3 and Fedora 17/18/19 as requested by the openSUSE Build Service.
Added RPM packages for Fedora 21 and CentOS 7 on openSUSE Build Service.

Tests

Lots of test refinements for the CPython test suites to be run continuously in Buildbot for both Windows and Linux.

Summary

This release brings about two major changes, each with the risk to break things.

One is that we finally started to have our own import logic, which has the risk to cause breakage, but apparently currently rather improved compatibility. The case issues were not fixable with standard library code.

The second one is that the __file__ attributes for standalone mode is now no longer pointing to the original install and therefore will expose missing stuff sooner. This will have to be followed up with code to scan for missing “data” files later on.

For SSA based optimization, there are cleanups in here, esp. the one removing the name mangling, allowing to remove special code for class variables. This makes the SSA tree more reliable. Hope is that the big step (forward propagation through variables) can be made in one of the next releases.

Article about Nuitka Standalone Mode

2015-01-06T12:46:50+00:00

There is a really well written article about Nuitka written by Tom Sheffler.

Note

The article has since become unavailable unfortunately.

It inspired me to finally become clean with __file__ attributes in standalone mode. Currently it points to where your source was when things were compiled. In the future (in standalone mode, for accelerated mode that continues to be good), it will point into the .dist folder, so that the SWIG workaround may become no longer necessary.

Thanks Tom for sharing your information, and good article.

Yours,

Kay

Nuitka Release 0.5.7

2015-01-02T09:49:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is brings a newly supported platform, bug fixes, and again lots of cleanups.

Bug Fixes

Fix, creation of dictionary and set literals with non-hashable indexes did not raise an exception.
```
{[]: None}  # This is now a TypeError
```

Optimization

Calls to the dict built-in with only keyword arguments are now optimized to mere dictionary creations. This is new for the case of non-constant arguments only of course.
```
dict(a=b, c=d)
# equivalent to
{"a": b, "c": d}
```
Slice del with indexable arguments are now using optimized code that avoids Python objects too. This was already done for slice look-ups.
Added support for bytearray built-in.

Organizational

Added support for OpenBSD with fiber implementation from library, as it has no context support.

Cleanups

Moved slicing solutions for Python3 to the re-formulation stage. So far the slice nodes were used, but only at code generation time, there was made a distinction between Python2 and Python3 for them. Now these nodes are purely Python2 and slice objects are used universally for Python3.

Tests

The test runners now have common code to scan for the first file to compile, an implementation of the search mode. This will allow to introduce the ability to search for pattern matches, etc.
More tests are directly executable with Python3.
Added recurse_none mode to test comparison, making using extra options for that purpose unnecessary.

Summary

This solves long standing issues with slicing and subscript not being properly distinguished in the Nuitka code. It also contains major bug fixes that really problematic. Due to the involved nature of these fixes they are made in this new release.

Nuitka Release 0.5.6

2014-12-19T07:54:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings bug fixes, important new optimization, newly supported platforms, and important compatibility improvements. Progress on all fronts.

Bug Fixes

Closure taking of global variables in member functions of classes that had a class variable of the same name was binding to the class variable as opposed to the module variable.
Overwriting compiled function’s __doc__ attribute more than once could corrupt the old value, leading to crashes. Fixed in 0.5.5.2 already.

Compatibility Python2: The exec statement execfile were changing locals() was given as an argument.

def function():
    a = 1

    exec code in locals()  # Cannot change local "a".
    exec code in None  # Can change local "a"
    exec code

Previously Nuitka treated all 3 variants the same.

Compatibility: Empty branches with a condition were reduced to only the condition, but they need in fact to also check the truth value:
```
if condition:
    pass
# must be treated as
bool(condition)
# and not (bug)
condition
```
Detection of Windows virtualenv was not working properly. Fixed in 0.5.5.2 already.
Large enough constants structures are now unstreamed via marshal module, avoiding large codes being generated with no point. Fixed in 0.5.5.2 already.
Windows: Pressing CTRL-C gave two stack traces, one from the re-execution of Nuitka which was rather pointless. Fixed in 0.5.5.1 already.
Windows: Searching for virtualenv environments didn’t terminate in all cases. Fixed in 0.5.5.1 already.
During installation from PyPI with Python3 versions, there were errors given for the Python2 only scons files. Fixed in 0.5.5.3 already.
Fix, the arguments of yield from expressions could be leaked.

Fix, closure taking of a class variable could have in a sub class where the module variable was meant.

var = 1


class C:
    var = 2

    class D:
        def f():
            # was C.var, now correctly addressed top level var
            return var

Fix, setting CXX environment variable because the installed gcc has too low version, wasn’t affecting the version check at all.
Fix, on Debian/Ubuntu with hardening-wrapper installed the version check was always failing, because these report a shortened version number to Scons.

Optimization

Local variables that must be assigned also have no side effects, making use of SSA. This allows for a host of optimization to be applied to them as well, often yielding simpler access/assign code, and discovering in more cases that frames are not necessary.
Micro optimization to dict built-in for simpler code generation.

Organizational

Added support for ARM “hard float” architecture.
Added package for Ubuntu 14.10 for download.
Added package for openSUSE 13.2 for download.
Donations were used to buy a Cubox-i4 Pro. It got Debian Jessie installed on it, and will be used to run an even larger amount of tests.
Made it more clear in the user documentation that the .exe suffix is used for all platforms, and why.
Generally updated information in User Manual and Developer Manual about the optimization status.
Using Nikola 7.1 with external filters instead of our own, outdated branch for the web site.

Cleanups

PyLint clean for the first time ever. We now have a Buildbot driven test that this stays that way.
Massive indentation cleanup of keyword argument calls. We have a rule to align the keywords, but as this was done manually, it could easily get out of touch. Now with a auto-format tool based on RedBaron, it’s correct. Also, spacing around arguments is now automatically corrected. More to come.
For exec statements, the coping back to local variables is now an explicit node in the tree, leader to cleaner code generation, as it now uses normal variable assignment code generation.
The MaybeLocalVariables became explicit about which variable they might be, and contribute to its SSA trace as well, which was incomplete before.
Removed some cases of code duplication that were marked as TODO items. This often resulted in cleanups.
Do not use replaceWith on child nodes, that potentially were re-used during their computation.

Summary

The release is mainly the result of consolidation work. While the previous release contained many important enhancements, this is another important step towards full SSA, closing one loop whole (class variables and exec functions), as well as applying it to local variables, largely extending its use.

The amount of cleanups is tremendous, in huge part due to infrastructure problems that prevented release repeatedly. This reduces the technological debt very much.

More importantly, it would appear that now eliminating local and temporary variables that are not necessary is only a small step away. But as usual, while this may be easy to implement now, it will uncover more bugs in existing code, that we need to address before we continue.

Nuitka Release 0.5.5

2014-10-05T23:15:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is finally making full use of SSA analysis knowledge for code generation, leading to many enhancements over previous releases.

It also adds support for Python3.4, which has been longer in the making, due to many rather subtle issues. In fact, even more work will be needed to fully solve remaining minor issues, but these should affect no real code.

And then there is much improved support for using standalone mode together with virtualenv. This combination was not previously supported, but should work now.

New Features

Added support for Python3.4

This means support for clear method of frames to close generators, dynamic __qualname__, affected by global statements, tuples as yield from arguments, improved error messages, additional checks, and many more detail changes.

Optimization

Using SSA knowledge, local variable assignments now no longer need to check if they need to release previous values, they know definitely for the most cases.
```
def f():
    a = 1  # This used to check if old value of "a" needs a release
    ...
```
Using SSA knowledge, local variable references now no longer need to check for raising exceptions, let alone produce exceptions for cases, where that cannot be.
```
def f():
    a = 1
    return a  # This used to check if "a" is assigned
```
Using SSA knowledge, local variable references now are known if they can raise the UnboundLocalError exception or not. This allows to eliminate frame usages for many cases. Including the above example.
Using less memory for keeping variable information.
Also using less memory for constant nodes.

Bug Fixes

The standalone freezing code was reading Python source as UTF-8 and not using the code that handles the Python encoding properly. On some platforms there are files in standard library that are not encoded like that.
The fiber implementation for Linux amd64 was not working with glibc from RHEL 5. Fixed to use now multiple int to pass pointers as necessary. Also use uintptr_t instead of intprt_t to transport pointers, which may be more optimal.
Line numbers for exceptions were corrupted by with statements due to setting line numbers even for statements marked as internal.
Partial support for win32com by adding support for its hidden __path__ change.
Python3: Finally figured out proper chaining of exceptions, given proper context messages for exception raised during the handling of exceptions.
Corrected C++ memory leak for each closure variable taken, each time a function object was created.
Python3: Raising exceptions with tracebacks already attached, wasn’t using always them, but producing new ones instead.
Some constants could cause errors, as they cannot be handled with the marshal module as expected, e.g. (int,).
Standalone: Make sure to propagate sys.path to the Python instance used to check for standard library import dependencies. This is important for virtualenv environments, which need site.py to set the path, which is not executed in that mode.
Windows: Added support for different path layout there, so using virtualenv should work there too.
The code object flag “optimized” (fast locals as opposed to locals dictionary) for functions was set wrongly to value for the parent, but for frames inside it, one with the correct value. This lead to more code objects than necessary and false co_flags values attached to the function.
Options passed to nuitka-python could get lost.
```
nuitka-python program.py argument1 argument2 ...
```
The above is supposed to compile program.py, execute it immediately and pass the arguments to it. But when Nuitka decides to restart itself, it would forget these options. It does so to e.g. disable hash randomization as it would affect code generation.
Raising tuples exception as exceptions was not compatible (Python2) or reference leaking (Python3).

Tests

Running 2to3 is now avoided for tests that are already running on both Python2 and Python3.
Made XML based optimization tests work with Python3 too. Previously these were only working on Python2.
Added support for ignoring messages that come from linking against self compiled Pythons.
Added test case for threaded generators that tortures the fiber layer a bit and exposed issues on RHEL 5.
Made reference count test of compiled functions generic. No more code duplication, and automatic detection of shared stuff. Also a more clear interface for disabling test cases.
Added Python2 specific reference counting tests, so the other cases can be executed with Python3 directly, making debugging them less tedious.

Cleanups

Really important removal of “variable references”. They didn’t solve any problem anymore, but their complexity was not helpful either. This allowed to make SSA usable finally, and removed a lot of code.
Removed special code generation for parameter variables, and their dedicated classes, no more needed, as every variable access code is now optimized like this.
Stop using C++ class methods at all. Now only the destructor of local variables is actually supposed to do anything, and their are no methods anymore. The unused var_name got removed, setVariableValue is now done manually.
Moved assertions for the fiber layer to a common place in the header, so they are executed on all platforms in debug mode.
As usual, also a bunch of cleanups for PyLint were applied.
The locals built-in code now uses code generation for accessing local variable values instead having its own stuff.

Organizational

The Python version 3.4 is now officially supported. There are a few problems open, that will be addressed in future releases, none of which will affect normal people though.
Major cleanup of Nuitka options.
- Windows specific stuff is now in a dedicated option group. This includes options for icon, disabling console, etc.
- There is now a dedicated group for controlling backend compiler choices and options.
Also pickup g++44 automatically, which makes using Nuitka on CentOS5 more automatic.

Summary

This release represents a very important step ahead. Using SSA for real stuff will allow us to build the trust necessary to take the next steps. Using the SSA information, we could start implementing more optimizations.

Nuitka shaping up

2014-10-02T07:08:54+01:00

Not much has happened publicly to Nuitka, so it’s time to make a kind of status post, about the exciting news there is.

SSA (Single State Assignment Form)

Based on these traces, optimizations could be made. Having SSA or not, is (to me) the difference between Nuitka as a compiler, and Nuitka as an optimizing compiler.

The news is, SSA is shaping up, and will be used in the next release. Not yet to drive variable based optimization (reserved for a release after it), but to aid the code generation to avoid useless checks.

Improved Code Generation

The more important change is from the SSA derived knowledge. Now Nuitka knows that a variable must be assigned, cannot be assigned, may be assigned, based on its SSA traces.

Lets check out an example:

def f():
    a = 1
    return a

Nevermind, that obviously the variable a can be removed, and this could be transformed to statically return 1. That is the next step (and easy if SSA is working properly), now we are looking at what changed now.

This is code as generated now, with current 0.5.5pre5:

tmp_assign_source_1 = const_int_pos_1;
assert( var_a.object == NULL );
var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );

tmp_return_value = var_a.object;

Py_INCREF( tmp_return_value );
goto function_return_exit;

There are some things, wrong with it still. For one, var_a is still a C++ object, which we directly access. But the good thing is, we can assert that it starts out uninitialized, before we overwrite it. The stable release as of now, 0.5.4, generates code like this:

tmp_assign_source_1 = const_int_pos_1;
if (var_a.object == NULL)
{
    var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );
}
else
{
    PyObject *old = var_a.object;
    var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );
    Py_DECREF( old );
}
static PyFrameObject *cache_frame_function = NULL;
MAKE_OR_REUSE_FRAME( cache_frame_function, codeobj_4e03e5698a52dd694c5c263550d71551, module___main__ );
PyFrameObject *frame_function = cache_frame_function;

// Push the new frame as the currently active one.
pushFrameStack( frame_function );

// Mark the frame object as in use, ref count 1 will be up for reuse.
Py_INCREF( frame_function );
assert( Py_REFCNT( frame_function ) == 2 ); // Frame stack

// Framed code:
tmp_return_value = var_a.object;

if ( tmp_return_value == NULL )
{

    exception_type = INCREASE_REFCOUNT( PyExc_UnboundLocalError );
    exception_value = UNSTREAM_STRING( &constant_bin[ 0 ], 47, 0 );
    exception_tb = NULL;

    frame_function->f_lineno = 4;
    goto frame_exception_exit_1;
}

Py_INCREF( tmp_return_value );
goto frame_return_exit_1;

As you can see, the assignment to var_a.object was checking if it were NULL, and if were not (which we now statically know), would release the old value. Next up, before returning, the value of var_a.object needed to be checked, if it were NULL, in which case, we would need to create a Python exception, and in order to do so, we need to create a frame object, that even if cached, consumes time, and code size.

So, that is the major change to code generation. The SSA information is now used in it, and doing so, has found a bunch of issues, in how it is built, in e.g. nested branches, that kind of stuff.

The removal of local variables as C++ classes, and them managed as temporary variables, is going to happen in a future release, reducing code complexity further. Were a a temporary variable, already, the Py_INCREF which implies a later Py_DECREF on the constant 1 could be totally avoided.

Scalability

The scalability of Nuitka hinges much of generated code size. With it being less stupid, the generated code is now not only faster, but definitely smaller, and with more optimization, it will only become more practical.

Compatibility

Python2 exec statements

A recent change in CPython 2.7.8+ which is supposed to become 2.7.9 one day, highlighted an issue with exec statements in Nuitka. These were considered to be fully compatible, but apparently are not totally.

def f():
    exec a in b, c
    exec (a, b, c)

The above two are supposed to be identical. So far this was rectified at run time of CPython, but apparently the parser is now tasked with it, so Nuitka now sees exec a in b, c for both lines. Which is good.

However, as it stands, Nuitka handles exec in locals() the same as exec in None for plain functions (OK to classes and modules), which is totally a bug.

I have been working on an enhanced re-formulation (it needs to be tracked if the value was None, and then the sync back to locals from the provided dictionary ought to be done. But the change breaks execfile in classes, which was implemented piggy-backing on exec, and now requires locals to be a dictionary, and immediately written to.

Anyway, consider exec as well working already. The non-working cases are really corner cases, obviously nobody came across so far.

Python3 classes

Incidentally, that execfile issue will be solved as soon as a bug is fixed, that was exposed by new abilities of Python3 metaclasses. They were first observed in Python3.4 enum classes.

class MyEnum(enum):
    red = 1
    blue = 2
    red = 3  # error

Currently, Nuitka is delaying the building of the dictionary (absent execfile built-in), and that is not allowed, in fact, immediate writes to the mapping giving by __prepare__ of the metaclass will be required, in which case, the enum class can raise an error for the second assignment to red.

So that area now hinges on code generation to learn different local variable codes for classes, centered around the notion of using the locals dictionary immediately.

Python3.4

The next release is no longer warning you if you use Python3.4, as many of the remaining problems have been sorted out. Many small things were found, and in some cases these highlighted general Python3 problems.

Nuitka for Python3 is not yet all that much in the focus in terms of performance, but correctness will have become much better, with most prominently, exception context being now correct most often.

The main focus of Nuitka is Python2, but to Nuitka the incompatibility of Python3 is largely not all that much an issue. The re-formulations to lower level operations for just about everything means that for the largest part there is not much trouble in supporting a mostly only slightly different version of Python.

The gain is mostly in that new tests are added in new releases, and these sometimes find things that affect Nuitka in all versions, or at least some others. And this could be a mere reference leak.

Consider this:

try:
    raise (TypeError, ValueError)
except TypeError:
    pass

So, that is working with Python2, but comes from a Python3 test. Python2 is supposed to unwrap the tuple and take the first argument and raise that. It didn’t do that so far. Granted, obscure feature, but still an incompatibility. For Python3, a TypeError should be raised complaining that tuple is not derived from BaseException.

Turned out, that also, in that case, a reference leak occurs, in that the wrong exception was not released, and therefore memory leaked. Should that happen a lot during a programs live, it will potentially become an issue, as it keeps frames on the traceback also alive.

So this lead to a compatibility fix and a reference leak fix. And it was found by the Python3.4 suite, checking that exception objects are properly released, and that the proper kind of exception is raised in the no longer supported case.

Performance

Graphs and Benchmarks

I had been working on automated performance graphs, and they are supposed to show up on Nuitka Speedcenter already, but currently it’s broken and outdated.

Sad state of affairs. Reasons include that I found it too ugly to publish unless updated to latest Nikola, for which I didn’t take the time. I intend to fix it, potentially before the release though.

Incremental Assignments

Consider the following code:

a += "bbb"

If a is a str, and if (and only if), it’s the only reference being held, then CPython, reuses the object, instead of creating a new object and copying a over. Well, Nuitka doesn’t do this. This is despite the problem being known for quite some time.

With SSA in place, and “C-ish” code generation complete, this will be solved, but I am not going to solve this before.

Standalone

The standalone mode of Nuitka is pretty good, and in the pre-release it was again improved. For instance, virtualenv and standalone should work now, and more modules are supported.

However, there are known issues with win32com and a few other packages, which need to be debugged. Mostly these are modules doing nasty things that make Nuitka not automatically detect imports.

This has as usual only so much priority from me. I am working on this on some occasions, as kind of interesting puzzles to solve. Most of the time, it just works though, with wxpython being the most notable exception. I am going to work on that though.

The standalone compilation exhibits scalability problems of Nuitka the most, and while it has been getting better, the recent and future improvements will lead to smaller code, which in turn means not only smaller executables, but also faster compilation. Again, wxpython is a major offender there, due to its many constants, global variables, etc. in the bindings, while Qt, PySide, and GTK are apparently already good.

Other Stuff

Funding

Nuitka doesn’t receive enough donations. There is no support from organizations like e.g. the PSF, which recently backed several projects by doubling donations given to them.

I remember talking to a PSF board member during Europython 2013 about this, and the reaction was fully in line with the Europython 2012 feedback towards me from the dictator. They wouldn’t help Nuitka in any way before it is successful.

I have never officially applied for help with funding though with them. I am going to choose to take pride in that, I suppose.

Collaborators

My quest to find collaborators to Nuitka is largely failing. Aside from the standalone mode, there have been too little contributions. Hope is that it will change in the future, once the significant speed gains arrive. And it might be my fault for not asking for help more, and to arrange myself with that state of things.

Not being endorsed by the Python establishment is clearly limiting the visibility of the project.

Anyway, things are coming along nicely. When I started out, I was fully aware that the project is something that I can do on my own if necessary, and that has not changed. Things are going slower than necessary though, but that’s probably very typical.

But you can join now, just follow this link or become part of the mailing list (since closed) and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Future

So, there is multiple things going on:

More “C-ish” code generation

The next release is going to be more “C-ish” than before, generating less complex code than before, and removes the previous optimizations, which were a lot of code, to e.g. detect parameter variables without del statements.

This prong of action will have to continue, as it unblocks further changes that lead to more compatibility and correctness.
More SSA usage

The next release did and will find bugs in the SSA tracing of Nuitka. It is on purpose only using it, to add assert statements to things it now no longer does. These will trigger in tests or cause crashes, which then can be fixed.

We better know that SSA is flawless in its tracking, before we use it to make optimizations, which then have no chance to assert anything at all anymore.

Once we take it to that next level, Nuitka will be able to speed up some things by more than the factor it basically has provided for 2 years now, and it’s probably going to happen this year.
More compatibility

The new exec code makes the dictionary synchronization explicit, and e.g. now it is optimized away to even check for its need, if we are in a module or a class, or if it can be known.

That means faster exec, but more importantly, a better understood exec, with improved ability to do SSA traces for them. Being able to in-line them, or to know the limit of their impact, as it will help to know more invariants for that code.

When these 3 things come to term, Nuitka will be a huge, huge step ahead towards being truly a static optimizing compiler (so far it is mostly only peep hole optimization, and byte code avoidance). I still think of this as happening this year.

Nuitka Release 0.5.4

2014-08-11T10:29:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is aiming at preparatory changes to enable optimization based on SSA analysis, introducing a variable registry, so that variables no longer trace their references to themselves.

Otherwise, MinGW64 support has been added, and lots of bug fixes were made to improve the compatibility.

Optimization

Using new variable registry, now properly detecting actual need for sharing variables. Optimization may discover that it is unnecessary to share a variable, and then it no longer is. This also allows --debug without it reporting unused variable warnings on Python3.
Scons startup has been accelerated, removing scans for unused tools, and avoiding making more than one gcc version check.

Bug Fixes

Compatibility: In case of unknown encodings, Nuitka was not giving the name of the problematic encoding in the error message. Fixed in 0.5.3.3 already.
Submodules with the same name as built-in modules were wrongly shadowed. Fixed in 0.5.3.2 already.
Python3: Added implementations of is_package to the meta path based loader.
Python3.4: Added find_spec implementation to the meta path based loader for increased compatibility.
Python3: Corrections for --debug to work with Python3 and MSVC compiler more often.
Fixed crash with --show-scons when no compiler was found. Fixed in 0.5.3.5 already.
Standalone: Need to blacklist lib2to3 from standard library as well. Fixed in 0.5.3.4 already.
Python3: Adapted to changes in SyntaxError on newer Python releases, there is now a msg that can override reason.
Standalone Windows: Preserve sys.executable as it might be used to fork binaries.
Windows: The caching of Scons was not arch specific, and files could be used again, even if changing the arch from x86 to x86_64 or back.
Windows: On 32 bit Python it can happen that with large number of generators running concurrently (>1500), one cannot be started anymore. Raising an MemoryError now.

Organizational

Added support for MinGW64. Currently needs to be run with PATH environment properly set up.
Updated internal version of Scons to 2.3.2, which breaks support for VS 2008, but adds support for VS 2013 and VS 2012. The VS 2013 is now the recommended compiler.
Added RPM package and repository for RHEL 7.
The output of --show-scons now includes the used compiler, including the MSVC version.
Added option --msvc to select the MSVC compiler version to use, which overrides automatic selection of the latest.
Added option -python-flag=no_warnings to disable user and deprecation warnings at run time.
Repository for Ubuntu Raring was removed, no more supported by Ubuntu.

Cleanups

Made technical and logical sharing decisions separate functions and implement them in a dedicated variable registry.
The Scons file has seen a major cleanup.

Summary

This release is mostly a maintenance release. The Scons integrations has been heavily visited, as has been Python3 and esp. Python3.4 compatibility, and results from the now possible debug test runs.

Standalone should be even more practical now, and MinGW64 is an option for those cases, where MSVC is too slow.

Nuitka Release 0.5.3

2014-07-07T23:25:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly a follow up, resolving points that have become possible to resolve after completing the C-ish evolution of Nuitka. So this is more of a service release.

New Features

Improved mode --improved now sets error lines more properly than CPython does in many cases.
The -python-flag=-S mode now preserves PYTHONPATH and therefore became usable with virtualenv.

Optimization

Line numbers of frames no longer get set unless an exception occurs, speeding up the normal path of execution.
For standalone mode, using --python-flag-S is now always possible and yields less module usage, resulting in smaller binaries and faster compilation.

Bug Fixes

Corrected an issue for frames being optimized away where in fact they are still necessary. Fixed in 0.5.2.1 already.
Fixed handling of exception tests as side effects. These could be remainders of optimization, but didn’t have code generation. Fixed in 0.5.2.1 already.
Previously Nuitka only ever used the statement line as the line number for all the expression, even if it spawned multiple lines. Usually nothing important, and often even more correct, but sometimes not. Now the line number is most often the same as CPython in full compatibility mode, or better, see above.
Python3.4: Standalone mode for Windows is working now.
Standalone: Undo changes to PYTHONPATH or PYTHONHOME allowing potentially forked CPython programs to run properly.
Standalone: Fixed import error when using PyQt and Python3.

New Tests

For our testing approach, the improved line number handling means we can undo lots of changes that are no more necessary.
The compile library test has been extended to cover a third potential location where modules may live, covering the matplotlib module as a result.

Cleanups

In Python2, the list contractions used to be re-formulated to be function calls that have no frame stack entry of their own right. This required some special handling, in e.g. closure taking, and determining variable sharing across functions.

This now got cleaned up to be properly in-lined in a try/finally expression.
The line number handling got simplified by pushing it into error exits only, removing the need to micro manage a line number stack which got removed.
Use intptr_t over unsigned long to store fiber code pointers, increasing portability.

Organizational

Providing own Debian/Ubuntu repositories for all relevant distributions.
Windows MSI files for Python 3.4 were added.
Hosting of the web site was moved to metal server with more RAM and performance.

Summary

This release brings about structural simplification that is both a follow-up to C-ish, as well as results from a failed attempt to remove static “variable references” and be fully SSA based. It incorporates changes aimed at making this next step in Nuitka evolution smaller.

Nuitka Release 0.5.2

2014-06-23T08:49:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is a major release, with huge changes to code generation that improve performance in a significant way. It is a the result of a long development period, and therefore contains a huge jump ahead.

New Features

Added experimental support for Python 3.4, which is still work in progress.
Added support for virtualenv on macOS.
Added support for virtualenv on Windows.
Added support for macOS X standalone mode.
The code generation uses no header files anymore, therefore adding a module doesn’t invalidate all compiled object files from caches anymore.
Constants code creation is now distributed, and constants referenced in a module are declared locally. This means that changing a module doesn’t affect the validity of other modules object files from caches anymore.

Optimization

C-ish code generation uses less C++ classes and generates more C-like code. Explicit temporary objects are now used for statement temporary variables.
The constants creation code is no more in a single file, but distributed across all modules, with only shared values created in a single file. This means improved scalability. There are remaining bad modules, but more often, standalone mode is now fast.
Exception handling no longer uses C++ exception, therefore has become much faster.
Loops that only break are eliminated.
Dead code after loops that do not break is now removed.
The try/finally and try/except constructs are now eliminated, where that is possible.
The try/finally part of the re-formulation for print statements is now only done when printing to a file, avoiding useless node tree bloat.
Tuples and lists are now generated with faster code.
Locals and global variables are now access with more direct code.
Added support for the anonymous code type built-in.
Added support for compile built-in.
Generators that statically return immediately, e.g. due to optimization results, are no longer using frame objects.
The complex call helpers use no pseudo frames anymore. Previous code generation required to have them, but with C-ish code generation that is no more necessary, speeding up those kind of calls.
Modules with only code that cannot raise, need not have a frame created for them. This avoids useless code size bloat because of them. Previously the frame stack entry was mandatory.

Bug Fixes

Windows: The resource files were cached by Scons and re-used, even if the input changed. The could lead to corrupted incremental builds. Fixed in 0.5.1.1 already.
Windows: For functions with too many local variables, the MSVC failed with an error “C1026: parser stack overflow, program too complex”. The rewritten code generation doesn’t burden the compiler as much.
Compatibility: The timing deletion of nested call arguments was different from C++. This shortcoming has been addressed in the rewritten code generation.
Compatibility: The __future__ flags and CO_FREECELL were not present in frame flags. These were then not always properly inherited to eval and exec in all cases.
Compatibility: Compiled frames for Python3 had f_restricted attribute, which is Python2 only. Removed it.
Compatibility: The SyntaxError of having a continue in a finally clause is now properly raised.
Python2: The exec statement with no locals argument provided, was preventing list contractions to take closure variables.
Python2: Having the ASCII encoding declared in a module wasn’t working.
Standalone: Included the idna encoding as well.
Standalone: For virtualenv, the file orig-prefix.txt needs to be present, now it’s copied into the “dist” directory as well. Fixed in 0.5.1.1 already.
Windows: Handle cases, where Python and user program are installed on different volumes.
Compatibility: Can now finally use execfile as an expression. One of our oldest issues, no 5, is finally fixed after all this time thanks to C-ish code generation.
Compatibility: The order or call arguments deletion is now finally compatible. This too is thanks to C-ish code generation.
Compatibility: Code object flags are now more compatible for Python3.
Standalone: Removing “rpath” settings of shared libraries and extension modules included. This makes standalone binaries more robust on Fedora 20.
Python2: Wasn’t falsely rejecting unicode strings as values for int and long variants with base argument provided.
Windows: For Python3.2 and 64 bits, global variable accesses could give false NameError exceptions. Fixed in 0.5.1.6 already.
Compatibility: Many exec and eval details have become more correctly, the argument handling is more compatible, and e.g. future flags are now passed along properly.
Compatibility: Using open with no arguments is now giving the same error.

Organizational

Replying to email from the issue tracker works now.
Added option name alias --xml for --dump-xml.
Added option name alias --python-dbg for --python-debug, which actually might make it a bit more clear that it is about using the CPython debug run time.
Remove option --dump-tree, it had been broken for a long time and unused in favor of XML dumps.
New digital art folder with 3D version of Nuitka logo. Thanks to Juan Carlos for creating it.
Using “README.rst” instead of “README.txt” to make it look better on web pages.
More complete whitelisting of missing imports in standard library. These should give no warnings anymore.
Updated the Nuitka GUI to the latest version, with enhanced features.
The builds of releases and update of the downloads page is now driven by Buildbot. Page will be automatically updated as updated binaries arrive.

Cleanups

Temporary keeper variables and the nodes to handle them are now unified with normal temporary variables, greatly simplifying variable handling on that level.
Less code is coming from templates, more is actually derived from the node tree instead.
Releasing the references to temporary variables is now always explicit in the node tree.
The publishing and preservation of exceptions in frames was turned into explicit nodes.
Exception handling is now done with a single handle that checks with branches on the exception. This eliminates exception handler nodes.
The dir built-in with no arguments is now re-formulated to locals or globals with their .keys() attribute taken.
Dramatic amounts of cleanups to code generation specialities, that got done right for the new C-ish code generation.

New Tests

Warnings from MSVC are now error exits for --debug mode too, expanding the coverage of these tests.
The outputs with python-dbg can now also be compared, allowing to expand test coverage for reference counts.
Many of the basic tests are now executable with Python3 directly. This allows for easier debug.
The library compilation test is now also executed with Python3.

Summary

This release would deserve more than a minor number increase. The C-ish code generation, is a huge body of work. In many ways, it lays ground to taking benefit of SSA results, that previously would not have been possible. In other ways, it’s incomplete in not yet taking full advantage yet.

The release contains so many improvements, that are not yet fully realized, but as a compiler, it also reflects a stable and improved state.

The important changes are about making SSA even more viable. Many of the problematic cases, e.g. exception handlers, have been stream lined. A whole class of variables, temporary keepers, has been eliminated. This is big news in this domain.

For the standalone users, there are lots of refinements. There is esp. a lot of work to create code that doesn’t show scalability issues. While some remain, the most important problems have been dealt with. Others are still in the pipeline.

More work will be needed to take full advantage. This has been explained in a separate post in greater detail.

State of Nuitka

2014-06-17T05:35:48+01:00

For quite some time, publicly, very little has happened with my Python compiler Nuitka. But that doesn’t mean, there hasn’t been progress. In fact it is tremendous. I would like to have a post that kind of summarizes, what happened.

The last release, 0.5.1 was more of a maintenance release than making real changes. It turns out, that the bigger changes got delayed by a feature that I have described as “C-ish”. Let me outline, what this means.

C-ish vs. C++-ish

When I started working on Nuitka, the big question was if it is possible to create a sufficiently compatible compiler. The use of C++11 then, together with some templates made it easy to cover a wide, wide part of the language, and to fully integrate with CPython for compatibility.

The main goal was to get it going to work correctly. As time went on, execution order demanded to do away with variadic templates, raw strings were not all that perfect at all, and so C++-03 was good enough at one point.

And then, as Nuitka became less and less template based, and shoving more things into the node tree, and re-formulations, making this where the knowledge resided. It became more and more obvious that C++ has two problems. One in the way I used it. One inherent in the language typical implementations:

C++ exceptions are god damn slow
Everything should be a in a single statement.

The later was my choice. Initially it made it easy to pass on references and put the releasing C++ class around every expression as necessary. Identifier classes were allowing for code generation to avoid taking references where necessary, and it was not all that bad. Yet limiting.

This led to a couple of issues.

The order of call arguments release for e.g. f(g(h())) was not completely compatible, with the way how CPython does it. C++ destructors for objects living in a single statement didn’t give sufficient control, and make the order and timing of finalization not compatible.
The generated C++ code complexity became large. The compilation of the generated C++ in some cases was huge. To the point, that e.g. “code too complex” was giving by compilers like MSVC for some modules.
Cases of in-place assignments were discovered, where CPython outperforms Nuitka by a large margin. But these don’t fit into that style of code generation.

So, at some point, the pain had built up. Code generation was already lighter than in the beginning. For example, initially with statements had dedicated code templates to them. This, and many other things, are long gone.

I took a deep dive, and rewrote the whole code generation, to be much more “C-ish” than “C++-ish”. A huge undertaking that would take months.

Where previously, code didn’t have to handle return error codes (a C++ exception was thrown), now everything needed a return value name, and error check.
Where classes were previously conviently made sure things happened at function or scope exit, manual handling needed to be added.
The handling of continue, break, and return was previously done with exceptions being thrown, if they were to pass a try/finally handler. Now these are done with stacks of exit handlers, where goto statements are used to produce the correct behaviour.

Rewriting Code Generation

Redoing code generation, over months, while ultimately, slowly, arriving at a point where Nuitka would be doing this, it already did before, was kind of frustrating.

Of course, the performance benefit would be there, but it would not be all that much, except for exception raising and handling. There it would be huge. Ultimately for PyStone, a couple of extra percents were gained.

This really was a point, where I felt, that Nuitka will make it or break. And for a long time, I honestly wasn’t so sure, that I pull through. But I did.

Current Situation

The current pre-release is release quality. You should try it out, it’s great.

There are many changes to Standalone mode. Due to changes in how constants are now created in the modules that uses them, instead of everything globally, the parallel compilation now works great. What previously took an hour with MSVC (the problem child, gcc was always relatively good), now takes minutes only.
The support for virtualenv’s of all kinds seems to work on Windows, Linux, and macOS, which all seem to have different kinds of codes.
The support for macOS is now there. Thanks to a virtual server month donated to Jarrad Hope, I was able to iron issues out.
The final release will also work with standalone binaries created on Fedora 20 which got hard code rpaths removed on the factory git branch.

And yet, I am not yet releasing. In fact, I would like to ask you to give it a roll, and integrate test feedback.

Although more tests than ever are executed and pass, (e.g. the Mercurial test suite is now run each time I make a commit, and fully identically passes or fails the test suite with the current Mercurial code), there can never be enough.

The changes I made are the most intense ever, and definitely have potential for regressions. I am used to providing very high quality releases.

Also, I am working on the Buildbot instances to automate the production of performance graphs, which get updated fully automatically. I am working on updating the downloads page automatically for each release that gets made.

And generally, I am trying to improve my work flow, to make it easier to push out releases with less effort. Buildbot should drive the release process more completely. I am using the git flow to provide hot-fixes, and this should be even less painful in the future.

Open Points

With this release, presenting great progress, many things remain in an unfinished state.

The support for Python3.4 is not complete. Most things work, but some need more work. Specifically the changes to __class__ variable closure taking, need another major refactoring, this time on variable handling.

Currently there are variables, closure variables, temp variables, and then temp variable references. The way they work is different. One way they work different, prevents a temp variable closure reference to carry a name, in that case -_class__, which would be needed for Python3.4, where that is suddenly necessary.

With this done, the SSA code will be even easier to write, as temp variables and named variables will finally be fully unified.
The use of C++ classes is largely reduced now. But a few still remain, namely for local variables, closure variables, and temp variables that are explicit variables. They still use C++ classes, although changing that seems quite possible now, because at least for temporary variables, the class doesn’t do anything in terms of code anymore.

Removing these classes may well gain more performance.
Now that code generation can more easily make a difference, and SSA apparently is becoming reliable, it could be used to know that values must be value and to optimize checks away.

Currently every variable access checks for “NULL”, when it’s part of an assign trace. Some optimizations exist for parameter variables without del on them, that do not use SSA.

This could be expanded and made general, allowing for much less code to be generated (specifically avoiding error code, and release code for variables that cannot give an error).
The SSA has been found unreliable in some instances, due to bugs that I believe I found. We could attempt and forward propagate variable assignments to where they are used, eliminating variables, etc.

This is a place, where a lot of performance can be gained. We really want to be there. And “C-ish” now makes this ever more attractive, despite the large delay in time it has caused.
The in-place assignment code for strings, where CPython can be way faster than current Nuitka, it bears a risk of getting it wrong. It is therefore pushed to a future release.

Other Things

For the website, I am relocating the virtual machine to a dedicated server rented for an increased price. This will allow to add a few more dynamic features, as the virtual machine was always too limited in RAM. It’s more expensive, but I feel a better investment of my time.

As mentioned before, I am not going to conferences this year. Enjoy Europython, and consider having a Lightning talk about Nuitka. I will be there next year again.

Call for Help

Please test the latest release of Nuitka.
Please consider making a donation to support my work on Nuitka. I have continuous monthly costs of it, so it would be sweet if it’s with all my time spent working on it, at least not a financial cost to me.
Please join the mailing list (since closed), and offer your help with tasks. Nuitka can seriously take more people developing, testing, reviewing, and quality checking it.

Final Words

So, there is this “C-ish” release 0.5.2 cooking. You are invited to help. Big, improvements are coming to Nuitka. Even after this next huge release, very important work is still open, but hope is to have this complete over the summer.

Yours,

Kay

Nuitka Release 0.5.1

2014-03-08T04:21:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings corrections and major improvements to how standalone mode performs. Much of it was contributed via patches and bug reports.

Bug Fixes

There was a crash when using next on a non-iterable. Fixed in 0.5.0.1 already.
Module names with special characters not allowed in C identifiers were not fully supported. Fixed in 0.5.0.1 already.
Name mangling for classes with leading underscores was not removing them from resulting attribute names. This broke at __slots__ with private attributes for such classes. Fixed in 0.5.0.1 already.
Standalone on Windows might need “cp430” encoding. Fixed in 0.5.0.2 already.
Standalone mode didn’t work with lxml.etree due to lack of hard coded dependencies. When a shared library imports things, Nuitka cannot detect it easily.
Wasn’t working on macOS 64 bits due to using Linux 64 bits specific code. Fixed in 0.5.0.2 already.
On MinGW the constants blob was not properly linked on some installations, this is now done differently (see below).

New Features

Memory usages are now traced with --show-progress allowing us to trace where things go wrong.

Optimization

Standalone mode now includes standard library as bytecode by default. This is workaround scalability issues with many constants from many modules. Future releases are going to undo it.
On Windows the constants blob is now stored as a resource, avoiding compilation via C code for MSVC as well. MinGW was changed to use the same code.

New Tests

Expanded test coverage for “standalone mode” demonstrating usage of “hex” encoding, PySide, and PyGtk packages.

Summary

This release is mostly an interim maintenance release for standalone. Major changes that provide optimization beyond that, termed “C-ish code generation” are delayed for future releases.

This release makes standalone practical which is an important point. Instead of hour long compilation, even for small programs, we are down to less than a minute.

The solution of the scalability issues with many constants from many modules will be top priority going forward. Since they are about how even single use constants are created all in one place, this will be easy, but as large changes are happening in “C-ish code generation”, we are waiting for these to complete.

Nuitka Release 0.5.0

2014-01-04T10:05:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release breaks interface compatibility, therefore the major version number change. Also “standalone mode” has seen significant improvements on both Windows, and Linux. Should work much better now.

But consider that this part of Nuitka is still in its infancy. As it is not the top priority of mine for Nuitka, which primarily is intended as an super compatible accelerator of Python, it will continue to evolve nearby.

There is also many new optimization based on structural improvements in the direction of actual SSA.

Bug Fixes

The “standalone mode” was not working on all Redhat, Fedora, and openSUSE platforms and gave warnings with older compilers. Fixed in 0.4.7.1 already.
The “standalone mode” was not including all useful encodings. Fixed in 0.4.7.2 already.
The “standalone mode” was defaulting to --python-flag=-S which disables the parsing of “site” module. That unfortunately made it necessary to reach some modules without modifying PYTHONPATH which conflicts with the “out-of-the-box” experience.
The “standalone mode” is now handling packages properly and generally working on Windows as well.
The syntax error of having an all catching except clause and then a more specific one wasn’t causing a SyntaxError with Nuitka.
```
try:
    something()
except:
    somehandling()
except TypeError:
    notallowed()
```
A corruption bug was identified, when re-raising exceptions, the top entry of the traceback was modified after usage. Depending on malloc this was potentially causing an endless loop when using it for output.

New Features

Windows: The “standalone” mode now properly detects used DLLs using Dependency Walker which it offers to download and extra for you.

It is used as a replacement to ldd on Linux when building the binary, and as a replacement of strace on Linux when running the tests to check that nothing is loaded from the outside.

Optimization

When iterating over list, set, this is now automatically lowered to tuples avoiding the mutable container types.

So the following code is now equivalent:
```
for x in [a, b, c]:
    ...

# same as
for x in (a, b, c):
    ...
```
For constants, this is even more effective, because for mutable constants, no more is it necessary to make a copy.
Python2: The iteration of large range is now automatically lowered to xrange which is faster to loop over, and more memory efficient.
Added support for the xrange built-in.
The statement only expression optimization got generalized and now is capable of removing useless parts of operations, not only the whole thing when it has not side effects.
```
[a, b]

# same as
a
b
```
This works for all container types.

Another example is type built-in operation with single argument. When the result is not used, it need not be called.
```
type(a)

# same as
a
```
And another example is and is not have no effect of their own as well, therefore:
```
a is b

# same as
a
b
```
Added proper handling of conditional expression branches in SSA based optimization. So far these branches were ignored, which only acceptable for temporary variables as created by tree building, but not other variable types. This is preparatory for introducing SSA for local variables.

Organizational

The option --exe is now ignored and creating an executable is the default behavior of nuitka, a new option --module allows to produce extension modules.
The binary nuitka-python was removed, and is replaced by nuitka-run with now only implies --execute on top of what nuitka is.
Using dedicated Buildbot for continuous integration testing and release creation as well.
The Downloads now offers MSI files for Win64 as well.
Discontinued the support for cross compilation to Win32. That was too limited and the design choice is to have a running CPython instance of matching architecture at Nuitka compile time.

New Tests

Expanded test coverage for “standalone mode” demonstrating usage of “hex” encoding, and PySide package.

Summary

The “executable by default” interface change improves on the already high ease of use. The new optimization do not give all that much in terms of numbers, but are all signs of structural improvements, and it is steadily approaching the point, where the really interesting stuff will happen.

The progress for standalone mode is of course significant. It is still not quite there yet, but it is making quick progress now. This will attract a lot of attention hopefully.

As for optimization, the focus for it has shifted to making exception handlers work optimal by default (publish the exception to sys.exc_info() and create traceback only when necessary) and be based on standard branches. Removing special handling of exception handlers, will be the next big step. This release includes some correctness fixes stemming from that work already.

Nuitka Standalone Mode is Work in Progress

2013-12-27T10:48:22+00:00

Many of you who turn to my easy to use, highly compatible Python compiler Nuitka, do this mostly because they seek to solve the deployment problem that Python suffers from.

Be this, because you want to use a newer Python2.7 on RHEL5 without installing anything at all there. Or because Windows is difficult to tackle otherwise.

For the longest time, Nuitka had not offered anything in this domain, focusing solely on being an accelerator. Lately, I have taken up the challenge and polished initial solutions submitted by contributors.

This first showed up in the 0.4.7 release, but turned out relatively weak. While first examples were working on Linux, it was not working at all on Windows (anymore). And basically there was a huge lack of tests.

Actually I didn’t mean for it to be released with that feature, but as parts of it seemed to work, I did so. But truth to be sad, that feature is not nearly as polished in that release as you would like it to.

In current development releases, of what is going to become 0.5.0 really soon now, it’s much better already. More things actually work. But it appears, there will be more ground to cover, and this is a lot of stuff to sort out.

So, this is mostly about asking you two things. Give that development release a try and report issues you have with it. And help me.

And have patience. I am developing Nuitka as an accelerator on a “no known bugs” basis. That means, once I know of a bug, I will fix it. OK, some issues in fact take longer, but then it really is not important at all, but difficult at the time. For standalone mode, I can’t do it that way, or I would have to neglect the acceleration parts, which I totally don’t want to do.

Because while you maybe are only interested in a packaging solution, many others would like to have that orders of magnitude speedup that I have been aiming for and that feels near now. This is about making Python a viable language for more uses than it currently is.

So why do it in the first place. For one, I am hoping that it helps people to not turn away from Python. And second, and more important, I am hoping that by making it more useful, more people will join me. (Oh, and thirdly, it’s also a nice puzzle to solve. I seem to enjoy that.)

Ultimately both modes will be needed, standalone, and acceleration. And it seems like I am working to provide both. For standalone, more often, than seeking to avoid bugs as far as possible, I am going to rely on your participation.

So join Nuitka. Now. Download from here. Join the mailing list (since closed). And help if you can. And yes you can.

Nuitka Release 0.4.7

2013-12-05T01:45:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release includes important new features, lots of polishing cleanups, and some important performance improvements as well.

Bug Fixes

The RPM packages didn’t build due to missing in-line copy of Scons. Fixed in 0.4.6.1 already.
The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already.
The Windows installer was not including Scons. Fixed in 0.4.6.3 already.
Windows: The immediate execution as performed by nuitka --execute was not preserving the exit code.
Python3.3: Packages without __init.py__ were not properly embedding the name-space package as well.
Python3: Fix, modules and packages didn’t add themselves to sys.modules which they should, happened only for programs.
Python3.3: Packages should set __package to their own name, not the one of their parents.
Python3.3: The __qualname__ of nested classes was corrected.
For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons.

New Features

The “standalone” mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already.

To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized.
New option --python-flag allows to specify flags to the compiler that the “python” binary normally would. So far -S and -v are supported, with sane aliases no_site and trace_imports.

The recommended use of --python-flag=-S is to avoid dependency creep in standalone mode compilations, because the site module often imports many useless things that often don’t apply to target systems.

Optimization

Faster frame stack handling for functions without try/except (or try/finally in Python3). This gives a speed boost to “PyStone” of ca. 2.5% overall.
Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to “PyStone” of ca. 0.5% overall.
Python2: Much quicker calls of __getattr__ and __setattr__ as this is now using the quicker call method avoiding temporary tuples.
Don’t treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies.
Create unicode constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up.
Directly create int and long constants outside of 2**31 and 2**32-1, but only limited according to actual platform values. Affects only program start-up.
When creating set values, no longer use a temporary tuple value, but use a properly generated helper functions instead. This makes creating sets much faster.
Directly create set constants instead of un-streaming them. Affects only program start-up.
For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement.
Module variables are now accessed even faster, the gain for “PyStone” is only 0.1% and mostly the result of leaner code.

Organizational

The “standalone mode” code (formerly known as “portable mode” has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement.
Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported.
Added package for openSUSE 13.1 for download.
Nuitka is now part of Arch and can be installed with pacman -S nuitka.
Using dedicated Buildbot for continuous integration testing. Not yet public.
Windows: In order to speed up repeated compilation on a platform without ccache, added Scons level caching in the build directory.
Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ccache and Scons as well.

Tests

The programs tests cases now fail if module or directory recursion is not working, being executed in another directory.
Added test runner for packages, with initial test case for package with recursion and sub-packages.
Made some test cases more strict by reducing PYTHONPATH provision.
Detect use of extra flags in tests that don’t get consumed avoiding ineffective flags.
Use --execute on Windows as well, the issue that prevented it has been solved after all.

Cleanups

The generated code uses const_, var_, par_ prefixes in the generated code and centralized the decision about these into single place.
Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation.
The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious.
Massive general cleanups, many of which came from new contributor Juan Carlos Paco.
Moved standalone and freezer related codes to dedicated package nuitka.freezer to not pollute the nuitka package name space.
The code generation use variable identifiers and their accesses was cleaned up.
Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them.
Moved main program, function object, set related code generation to dedicated modules.

Summary

This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation.

The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead.

Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future.

Nuitka in Arch Linux

2013-11-30T14:17:59+00:00

Got this message:

Just thought you wanted to know that Nuitka is now also an official
Arch Linux package and can be installed simply with:

pacman -S nuitka

So, that is another sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Many thanks to Alexander Rødseth and Panagiotis Mavrogiorgos for maintaining the Nuitka packages inside Arch and on AUR.

Nuitka and GUIs

2013-11-26T09:35:53+00:00

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

Nuitka is a typical compiler, just for Python, and aims to be integrating with build tools, and as such is so much a command line tool, it hurts.

Yet, there are GUIs. Yes, plural, there are even two.

Many of you are maybe using Ninja IDE already. I have developed most of early Nuitka in vim, and later, mostly because of Nuitka, switched back to Emacs. I had switched from Emacs at one point 12 years ago, but then went back a couple of years ago.

But recently +Juan Carlos convinced me to give Ninja a try. Because not only is he the creator the Nuitka GUI, but also the creator of the Ninja IDE plugin for Nuitka.

So on Windows, I now use Ninja, and I like it, a lot.

I must admit, I am not a heavy user of both Nuitka GUIs, rather being a command line guy I suppose, but that is just great. I would like to thank Juan Carlos for these contributions.

They are one sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Nuitka Release 0.4.6

2013-10-28T08:52:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups.

Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot.

Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues.

And then there is performance. As can be seen in the performance graph, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types.

And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix.

Bug Fixes

Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. Fixed in 0.4.5.1 already.
Support Python being installed to a path that contains spaces. Fixed in 0.4.5.2 already.
Windows: User provided constants larger than 65k didn’t work with MSVC. Fixed in 0.4.5.3 already.
Windows: The option --windows-disable-console was not effective with MSVC. Fixed in 0.4.5.3 already.
Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn’t honor the target architecture. Fixed in 0.4.5.3 already.
When creating Python modules, these were marked as executable (“x” bit), which they are of course not. Fixed in 0.4.5.3 already.
Python3.3: On architectures where Py_ssize_t is not the same as long this could lead to errors. Fixed in 0.4.5.3 already.
Code that was using nested mutable constants and changed the nested ones was not executing correctly.
Python2: Due to list contractions being re-formulated as functions, del was rejected for the variables assigned in the contraction.
```
[expr(x) for x in iterable()]

del x  # Should work, was gave an unjustified SyntaxError.
```

New Features

Compiled types when used in Python comparison now work. Code like this will work:
```
def f():
    pass


assert type(f) == types.FunctionType
```
This of course also works for in operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka.
Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH.
Windows: New options --mingw64 to force compilation with MinGW.

Optimization

Rich comparisons (==, <, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall.
Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython.
The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs.

Cleanups

Replaced “value friend” based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of or and and expressions.
Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using “variable usage” profiles for this anymore.
The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows.
Stop using os.system and subprocess.call(..., shell = True) as it is not really portable at all, use subprocess.call(..., shell = False) instead.
As usual lots of cleanups related to line length issues and PyLint.

Organizational

Added support for Gentoo Linux.
Added support for self compiled Python versions with and without debug enabled.
Added use of Nuitka fonts for headers in manuals.
Does not install in-line copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages.

Summary

While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later.

To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever.

So give this a roll, it’s worth it. And feel free to join the mailing list (since closed) or make a donation to support Nuitka.

Nuitka Release 0.4.5

2013-08-20T17:50:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release incorporates very many bug fixes, most of which were already part of hot fixes, usability improvements, documentation improvements, new logo, simpler Python3 on Windows, warnings for recursion options, and so on. So it’s mostly a consolidation release.

Bug Fixes

When targeting Python 3.x, Nuitka was using “python” to run Scons to run it under Python 2.x, which is not good enough on systems, where that is already Python3. Improved to only do the guessing where necessary (i.e. when using the in-line copy of Scons) and then to prefer “python2”. Fixed in 0.4.4.1 already.
When using Nuitka created binaries inside a “virtualenv”, created programs would instantly crash. The attempt to load and patch inspect module was not making sure that site module was already imported, but inside the “virtualenv”, it cannot be found unless. Fixed in 0.4.4.1 already.
The option --recurse-directory to include plugin directories was broken. Fixed in 0.4.4.2 already.
Python3: Files with “BOM” marker causes the compiler to crash. Fixed in 0.4.4.2 already.
Windows: The generated code for try/return/finally was working with gcc (and therefore MinGW), but not with MSVC, causing crashes. Fixed in 0.4.4.2 already.
The option --recurse-all did not recurse to package __init__.py files in case from x.y import z syntax was used. Fixed in 0.4.4.2 already.
Python3 on macOS: Corrected link time error. Fixed in 0.4.4.2 already.
Python3.3 on Windows: Fixed crash with too many arguments to a kwonly argument using function. Fixed in 0.4.4.2 already.
Python3.3 on Windows: Using “yield from” resulted in a link time error. Fixed in 0.4.4.2 already.
Windows: Added back XML manifest, found a case where it is needed to prevent clashes with binary modules.
Windows: Generators only worked in the main Python threads. Some unusual threading modules therefore failed.
Using sys.prefix to find the Python installation instead of hard coded paths.

New Features

Windows: Python3 finds Python2 installation to run Scons automatically now.

Nuitka itself runs under Python3 just fine, but in order to build the generated C++ code into binaries, it uses Scons which still needs Python2.

Nuitka will now find the Python2 installation searching Windows registry instead of requiring hard coded paths.
Windows: Python2 and Python3 find their headers now even if Python is not installed to specific paths.

The installation path now is passed on to Scons which then uses it.
Better error checking for --recurse-to and --recurse-not-to arguments, tell the user not to use directory paths.
Added a warning for --recurse-to arguments that end up having no effect to the final result.

Cleanups

Import mechanism got cleaned up, stopped using “PyImport_ExtendInittab”. It does not handle packages, and the sys.meta_path based importer is now well proven.
Moved some of the constraint collection code mess into proper places. It still remains a mess.

Organizational

Added LICENSE.txt file with Apache License 2.0 text to make it more immediately obvious which license Nuitka is under.
Added section about Nuitka license to the User Manual.
Added Nuitka Logo to the distribution.
Use Nuitka Logo as the bitmap in the Windows installer.
Use Nuitka Logo in the documentation (User Manual and Developer Manual).
Enhanced documentation to number page numbers starting after table of contents, removed header/footer from cover pages.

Summary

This release is mostly the result of improvements made based on the surge of users after Europython 2013. Some people went to extents and reported their experience very detailed, and so I could aim at making e.g. their misconceptions about how recursion options work, more obvious through warnings and errors.

This release is not addressing performance improvements. The next release will be able to focus on that. I am taking my claim of full compatibility very serious, so any time it’s broken, it’s the highest priority to restore it.

My Europython 2013 report

2013-08-15T08:15:17+01:00

Back from Europython 2013 in Florence for a while now. I had a presentation about my Python compiler Nuitka. You can watch the Video on Youtube. I believe it’s absolutely worth your time. I was not doing a lot of “why” at all now, only “what”, and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good and at ease. The state presented is very good and progress - there was a lot, so that was great. The slides of the talk may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight then, and also much more excited, at least initially. Background: I was surprised then that Guido van Rossum was attending the talk, plus I was kind of not in the state of this year, the confidence that people should believe, that things will work after that, was not yet there. I knew it, but standing in front of a croud and saying, look I am attempting what you consider impossible, or even have failed at, is not easy:

There are also the slides of that talk which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven things. Let’s see what this year does to it. I would hope for real type inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out of the box experience, and how it just works. That’s right. I put a lot of focus on stability, correctness, and ease of use. In fact, since my return, I have mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays), it’s make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a lot of users, and not as many contributions or donations as I would hope for, but that’s kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important exception are scientific people, doing number crunching. And I made some interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which were unfortunately regressions of 0.4.4, and these got fixed in two hotfix releases. Currently 0.4.4.2 is released, and contains a huge amount of fixes, mostly stuff found by the surge new users.

But I deserved that, having said that I didn’t have to do hotfixes for a while now.

Florence

Such a lovely place, and so lovely people. It’s a tourist place, true, but it’s very open minded. The food is extremely good. And going from the conference hotel straight over to the river to meet up and have a few beers was very comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it’s moving on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large amount of non-satisfaction, was that the community is fed up with the leadership. Questioned about lack of Python3 adoption, a PSF guy in his presentation said they were 3 years into a 5 years plan, and got interrupted by laughter.

It may be about time, somebody actually gets up, removes the GIL and has a Python2.8 that is compatible. Obviously that’s much more useful. It could be a Python3.x with compatibility names of modules, print statement, and dict.iteritems added, probably also making some of the unicode mess more harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython top versions, to add things, that would allow Nuitka to work better with it, that could be a staging ground, where such changes are nurtured. On the other hand, Nuitka is too irrelevant yet to drive this, and it would detract me away from it.

Gains

So Nuitka got a lot more exposure. And one guy watching the talk even created a really nice logo for it. I will make it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And reinforcement that what I am doing is actually useful to some people already.

Donations

My travel to Florence was in part funded by donations. Thanks a lot to all of you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there via train.

Conclusion

The Europython 2012 and 2013 both are among the times of my life.

Nuitka Release 0.4.4

2013-06-27T00:16:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release marks the point, where Nuitka for the first time supports all major current Python versions and all major features. It adds Python 3.3 support and it adds support for threading. And then there is a massive amount of fixes that improve compatibility even further.

Aside of that, there is major performance work. One side is the optimization of call performance (to CPython non-compiled functions) and to compiled functions, both. This gave a serious improvement to performance.

Then of course, we are making other, long term performance progress, as in “–experimental” mode, the SSA code starts to optimize unused code away. That code is not yet ready for prime time yet, but the trace structure will hold.

New Features

Python3.3 support.

The test suite of CPython3.3 passes now too. The yield from is now supported, but the improved argument parsing error messages are not implemented yet.
Tracing user provided constants, now Nuitka warns about too large constants produced during optimization.
Line numbers of expressions are now updates as evaluation progresses. This almost corrects.

Now only expression parts that cannot raise, do not update, which can still cause difference, but much less often, and then definitely useless.
Experimental support for threads.

Threading appears to work just fine in the most cases. It’s not as optimal as I wanted it to be, but that’s going to change with time.

Optimization

Previous corrections for ==, !=, and <=, caused a performance regression for these operations in case of handling identical objects.

For built-in objects of sane types (not float), these operations are now accelerated again. The overreaching acceleration of >= was still there (bug, see below) and has been adapted too.
Calling non-compiled Python functions from compiled functions was slower than in CPython. It is now just as fast.
Calling compiled functions without keyword arguments has been accelerated with a dedicated entry point that may call the implementation directly and avoid parameter parsing almost entirely.
Making calls to compiled and non-compiled Python functions no longer requires to build a temporary tuple and therefore is much faster.
Parameter parsing code is now more compact, and re-uses error raises, or creates them on the fly, instead of hard coding it. Saves binary size and should be more cache friendly.

Bug Fixes

Corrected false optimization of a >= a on C++ level.

When it’s not done during Nuitka compile time optimization, the rich comparison helper still contained short cuts for >=. This is now the same for all the comparison operators.
Calling a function with default values, not providing it, and not providing a value for a value without default, was not properly detecting the error, and instead causing a run time crash.
```
def f(a, b=2):
    pass


f(b=2)
```
This now properly raises the TypeError exception.
Constants created with + could become larger than the normally enforced limits. Not as likely to become huge, but still potentially an issue.
The vars built-in, when used on something without __dict__ attribute, was giving AttributeError instead of TypeError.
When re-cursing to modules at compile time, script directory and current directory were used last, while at run time, it was the other way around, which caused overloaded standard library modules to not be embedded.

Thanks for the patch to James Michael DuPont.
Super without arguments was not raising the correct RuntimeError exception in functions that cannot be methods, but UnboundLocalError instead.
```
def f():
    super()  # Error, cannot refer to first argument of f
```
Generators no longer use raise StopIteration for return statements, because that one is not properly handled in try/except clauses, where it’s not supposed to trigger, while try/finally should be honored.
Exception error message when throwing non-exceptions into generators was not compatible.
The use of return with value in generators is a SyntaxError before Python3.3, but that was not raised.
Variable names of the “__var” style need to be mangled. This was only done for classes, but not for functions contained in classes, there they are now mangled too.
Python3: Exceptions raised with causes were not properly chaining.
Python3: Specifying the file encoding corrupted line numbers, making them all of by one.

Cleanups

For containers (tuple, list, set, dict) defined on the source code level, Nuitka immediately created constant references from them.

For function calls, class creations, slice objects, this code is now re-used, and its dictionaries and tuples, may now become constants immediately, reducing noise in optimization steps.
The parameter parsing code got cleaned up. There were a lot of relics from previously explored paths. And error raises were part of the templates, but now are external code.
Global variable management moved to module objects and out of “Variables” module.
Make sure, nodes in the tree are not shared by accident.

This helped to find a case of duplicate use in the complex call helpers functions. Code generation will now notice this kind of duplication in debug mode.
The complex call helper functions were manually taking variable closure, which made these functions inconsistent to other functions, e.g. no variable version was allocated to assignments.

Removing the manual setting of variables allowed a huge reduction of code volume, as it became more generic code.
Converting user provided constants to create containers into constants immediately, to avoid noise from doing this in optimization.
The site module is now imported explicitly in the __main__ module, so it can be handled by the recursion code as well. This will help portable mode.
Many line length 80 changes, improved comments.

New Tests

The CPython3.3 test suite was added, and run with both Python3.2 and Python3.3, finding new bugs.
The doctest to code generation didn’t successfully handle all tests, most notably, “test_generators.py” was giving a SyntaxError and therefore not actually active. Correcting that improved the coverage of generator testing.

Organizational

The portable code is still delayed.

Support for Python3.3 was a higher priority, but the intention is to get it into shape for Europython still.

Added notes about it being disabled it in the User Manual documentation.

Summary

This release is in preparation for Europython 2013. Wanted to get this much out, as it changes the status slides quite a bit, and all of that was mostly done in my Cyprus holiday a while ago.

The portable code has not seen progress. The idea here is to get this into a development version later.

Nuitka Release 0.4.3

2013-05-18T10:17:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka.

New Features

Support for new compiler: Microsoft Visual C++.

You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows.
Support for NetBSD.

Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic “fibers” implementation.
Support for Python3 under Windows too.

Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to C:\Python26 or C:\Python27 for Nuitka to be able to find it.
Enhanced Python 3.3 compatibility.

The support the newest version of Python has been extended, improving compatibility for many minor corner cases.
Added warning when a user compiles a module and executes it immediately when that references __name__.

Because very likely the intention was to create an executable. And esp. if there is code like this:
```
if __name__ == "__main__":
    main()
```
In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command
```
nuitka --execute module
```
is behavioral more like
```
python -c "import module"
```
and that was a trap for new users.
All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don’t have to implement for specific architectures anymore.

Bug Fixes

Dictionary creation was not fully compatible.

As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn’t observe this before, and it’s likely the new dictionary implementation that finds it.

Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing.
Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was.
Using a __future__ import inside a function was giving an assertion, instead of the proper syntax error.
Python3: Do not set the attributes sys.exc_type, sys.exc_value, sys.exc_traceback.
Python3: Annotations of function worked only as long as their definition was not referring to local variables.

Optimization

Calls with no positional arguments are now using the faster call methods.

The generated C++ code was using the () constant at call site, when doing calls that use no positional arguments, which is of course useless.
For Windows now uses OS “Fibers” for Nuitka “Fibers”.

Using threads for fibers was causing only overhead and with this API, MSVC had less issues too.

Organizational

Accepting Donations via Paypal, please support funding travels, website, etc.
The User Manual has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated.
The Changelog has been improved for consistency, wordings, and styles.
Nuitka is now available on the social code platforms as well
- Bitbucket (since removed)
- GitHub
- Gitorious (since discontinued)
- Google Code (since discontinued)
Removed clean-up.sh, which is practically useless, as tests now clean up after themselves reasonably, and with git clean -dfx working better.
Removed “create-environment.sh” script, which was only setting the PATH variable, which is not necessary.
Added check-with-pylint --emacs option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint.
Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated.

Cleanups

Removed useless manifest and resource file creation under Windows.

Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it.
PyLint massive cleanups and annotations bringing down the number of warnings by a lot.
Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes.
Do not track needed tuple, list, and dict creation code variants in context, but e.g. in nuitka.codegen.TupleCodes module instead.
Introduced an “internal” module to host the complex call helper functions, instead of just adding it to any module that first uses it.

New Tests

Added basic tests for order evaluation, where there currently were None.
Added support for “2to3” execution under Windows too, so we can run tests for Python3 installations too.

Summary

The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now.

Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization.

Going to Europython 2013

2013-04-16T09:58:34+01:00

I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.

Things to talk about

Status

Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.

What I would like to get done until then:

Win64

One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.
Threading

And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.

Since it’s not totally a priority right now, one of the next releases will support it, likely before the conference.

But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.

Last years questions

Builtins

For instance, writing to built-ins, what will/did happen.

Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.

This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it’s not effective in others (len, str).

Good news is that we got contributed a “compiled built-ins” code, where we now will be able to see such writes. Now it’s only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.

The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.

Note

Currently Nuitka does nothing of this, but it will.

Debugger - pdb

The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.

Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.

Threading

I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don’t like them either, so my experience was non-existing.

But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.

C++ to Python gaps

I consider all of these more or less solved.

Well maybe except recently arose issues with MSVC for “function calls”. It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.

But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.

Performance

There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).

And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.

Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.

But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it’s the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.

Demos

The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.

And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.

If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.

Future Work

Generally the talk will be more directed at the future, although this is kind of a dark area now. That’s its nature I guess.

SSA

The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say “Python is too dynamic (to be statically compiled)” is that Python values may escape to unknown code that changes them very often.

I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.

Compiled Modules

And I believe with “compiled modules” potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.

I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn’t have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.

Compiled Locals

Who knows, we might even have a “compiled locals” as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.

But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don’t know if it might use it.

Whole Program Analysis

Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.

Perspective

Last time, I spent a lot of time on justification, “why a new project?”, “why not work with the others?”, what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.

Funding

And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It’s probably fair to allow people to make donations if they feel they want to further the project, but don’t know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikely though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.

Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these “times of my life”. Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.

Nuitka needs you - a call for help

2013-04-10T07:51:13+01:00

Update 3

And the first point has been done too, Python3.3 is now fully supported, including yield from syntax.

Update 2

The third point, Windows 64 support has been done as well. Turns out, that I do in fact own only Win64 systems, and with MSVC support in place, only a few portability fixes were needed.

Help with the other point, “re-formulation of yield”, would still be more than welcome, and no signs of progress there yet. So stop thinking “you could”, enter telling people “you did” mode.

Update

The second point, Fibers implementation for Windows, has been done. Thanks for the help you people provide. The next release will contain it.

Hello everybody,

the Python compiler Nuitka has come an ever longer way, and currently I have quite a bunch of issues, that I believe could well need your help. These are all issues of some important and significance, yet self contained jobs that you may enjoy.

Note

You can check the page What is Nuitka? for clarification of what it is now and what it wants to be.

Python 3.3 - reformulation of yield from needed.

As you can see, covering all the CPython 2.6, 2.7, and 3.2 language features is already something. Also CPython 3.3 is now working on a basic level. Other projects are far, far away from that.

Many language constructs, such as the with statement are re-formulated into other constructs. This makes it possible to work with a simple core for optimization, and to reduce the complexity a lot. For the with statement case, it’s changed to try/finally and try/except statements, together with a few temporary variables. Check the Nuitka source of it:

ReformulationWithStatements.py

There is also descriptions of all these re-formulations in the Developer Manual so you can see how this is done.

Now check PEP 380 for the details of it, and your task would be to come up with a re-formulation of yield from statements to the semantically equivalent.

The benefit is, you are working on a totally self-contained level. The re-formulation only needs to use ast node information, and turn that into Nuitka nodes. The problem will be fully solved this way.

And once we have this, we can pronounce CPython3.3 as fully supported. So if you would like to see this happen, please join.
Windows - MSVC support needs low level code help

Update

This is now done.

The support for MSVC is already mostly there. If you execute the environment setting script, and use Nuitka under Windows, it will work. The only problem is that for the generator functions, Nuitka is using a concept called “Fiber”, which is basically just switching C stacks, so called co-routines.

It works great for Linux and UNIX. For Windows, Fibers are currently using threads, which wouldn’t be as much of a performance problem, because they are very lightweight, it is slightly wasteful only.

But, the bad thing, is with these threads switching and C++ exceptions used to represent Python exceptions, no combination of MSVC options seems to carry the day and keep the exception in the current thread. They are caught in the wrong thread, and make generators misbehave, crash.

This is the only thing that holds back proper MSVC support at this time, and it has been like this for a while. With MinGW, the issue does not exist, but MinGW is a stranger to many, and MSVC will be more efficient code generation, so we would like to have that.

So what it takes is somebody familiar with Windows and its ABI, esp. register usage to write code that swaps the registers around, so we can have multiple C stacks without threads for it too. This may involve messing around with exception jump back pointers as well.

It needs somebody who can handle this and does it for Nuitka, and then it would be well supported. It may involve writing some assembler code. The job is also very well contained, but also requires very specific knowledge, so please join me here. If you think you can do this, do it.
Windows 64 bit

I have no access to Win64. I bought many Windows versions, but not the Win64 variants so far. I understand that there are CPython builds for it, and probably a MinGW matching it, as well as a matching newer MSVC, that may work better for C++ exceptions than the one CPython 2.x is tied to.

Doing that, you would mostly only have to play around with SingleExe.scons, which abstracts the details of it all. One issue you will encounter is that Scons needs a Python2, and other small things. This is mostly only about porting Nuitka, and could be done by anybody with access to Win64 and willing to fiddle around a bit.

So these are 3 different ways you can help Nuitka. Creating the Python compiler is a huge effort. And if you help it, we will sooner arrive at its goals. Join the Nuitka mailing list (since closed) or contact me directly if you choose to help with any of these issues.

Wait, actually there is another way you can help. Please spread this news to people that you believe might be willing to help. And allow me to plug this, I am doing Nuitka in my spare time. If you feel, you cannot help, but still would like to support it, you can still make donations to allow me to travel to conferences and spend more time on it myself.

Anyway, it would be great if these 3 things got sorted out. Come and join me for this grand vision of the Python compiler.

Yours,

Kay

Support for portable (standalone) programs

2013-04-07T13:52:44+01:00

This post is about a feature often requested, but so far not available feature of Nuitka. Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

In forums, and in Google, people are looking at a Python compiler, also as a way of deployment. It should offer what py2exe does, allow installation independent of Python.

Well, for a long time it didn’t. But thanks to recent contributions, it’s upcoming for the next release, Nuitka 0.4.3, and it’s in the current pre-releases.

It works by adding --portable to the command line. So this should work for you:

nuitka-python --recurse-all --portable your-program.py

Right now, it will create a folder “_python” with DLLs, and “_python.zip” with standard library modules used along to the “your-program.exe”. Copy these to another machine, without a Python installation, and it will (should) work. Making that statement fully true may need refinements, as some DLL dependencies might not be defined yet.

Note

We may improve it in the future to meld everything into one executable for even easier deployment.

You are more than welcome to experiment with it. To do so, download Nuitka from the download page and give it a roll.

Note

Of course, Nuitka is not about replacing “py2exe” primarily, it’s only a side effect of what we do. Our major goal is of course to accelerate Python, but surely nobody minds achieving the two things at the same time.

And while the post is labeled “Windows”, this feature also works for Linux at least too. It’s just that the lack of Python installations on client systems is far more widespread on this platform.

To me, as this is from a contributor, it’s another sign of Nuitka gaining adoption for real usage. My personal “py2exe” experience is practically not existing, I have never used it. And I will only merge the improvements into the Nuitka project as provided by others. My focus for the time to come is of course the compile time and run time optimization.

Nuitka Release 0.4.2

2013-03-29T14:32:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the performance diagrams got expanded.

New Features

Support for FreeBSD.

Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some “Linuxisms” in the build process.
New option for warning about compile time detected exception raises.

Nuitka can now warn about exceptions that will be raised at run time.
Basic Python3.3 support.

The test suite of CPython3.2 passes and fails in a compatible way. New feature yield from is not yet supported, and the improved argument parsing error messages are not implemented yet.

Bug Fixes

Nuitka already supported compilation of “main directories”, i.e. directories with a “__main__.py” file inside. The resulting binary name was “__main__.exe” though, but now it is “directory.exe”
```
# ls directory
__main__.py

# nuitka --exe directory
# ls
directory directory.exe
```
This makes this usage more obvious, and fixes an older issue for this feature.
Evaluation order of binary operators was not enforced.

Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems.
Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.
Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.
```
global a  # Not in a function, but on module level. Pointless but legal!
a = 1
```
Effectively these statements can be ignored.
Future imports are only legal when they are at the start of the file.

This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error.
Raising exceptions from context was leaking references.
```
raise ValueError() from None
```
Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments.
Importing the module that became __main__ through the module name, didn’t recurse to it.

This also gives a warning. PyBench does it, and then stumbles over the non-found “pybench” module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don’t share data.
Compiled method repr leaked references when printed.

When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point.
The super built-in leaked references to given object.

This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run.
The raise statement didn’t enforce type of third argument.

This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it’s None (which is also legal), no reference to None is leaked.
Python3 built-in exceptions were strings instead of exceptions.

A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn’t help their usage of course, fixed.
The -nan and nan both exist and make a difference.

A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka.
Wrong optimization of a == a, a != a, a <= a on C++ level.

While it’s not done during Nuitka optimization, the rich comparison helpers still contained short cuts for ==, !=, and <=.
The sys.executable for nuitka-python --python-version 3.2 was still python.

When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.
Keyword only functions with default values were losing references to defaults.
```
def f(*, a=X()):
    pass


f()
f()  # Can crash, X() should already be released.
```
This is now corrected. Of course, a Python3 only issue.
Pressing CTRL-C didn’t generate KeyboardInterrupt in compiled code.

Nuitka never executes “pending calls”. It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.
For with statements with return, break, or continue to leave their body, the __exit__ was not called.
```
with a:  # This called a.__enter__().
    return 2  # This didn't call a.__exit__(None, None, None).
```
This is of course quite huge, and unfortunately wasn’t covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing.
Starting with Python3, absolute imports are now the default.

This was already present for Python3.3, and it turns out that all of Python3 does it.

Optimization

Constants are now much less often created with pickle module, but created directly.

This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.
```
a = ("something_special",)
b = "something_special"

assert a[0] is b  # Now true
```
This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the “==” operator will be able to shortcut dramatically in cases of identical objects.

Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.
Continue statements might be optimized away.

A terminal continue in a loop, was not optimized away:
```
while 1:
    something
    continue  # Now optimized away
```
The trailing continue has no effect and can therefore be removed.
```
while 1:
    something
```
Loops with only break statements are optimized away.
```
while 1:
    break
```
A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate “if … : break” checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.

New Tests

Added tests for the found issues.
Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered.
Running the “CPython3.2” test suite with Python3.3 based Nuitka works and found a few minor issues.

Organizational

The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and “ownssh” for contributing an RPM spec file.

The page got improved with logos for the distributions.
Added “ownssh” as contributor.
Revamped the User Manual in terms of layout, structure, and content.

Summary

This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.

Nuitka RPMs (RHEL, CentOS, F17, F18, openSUSE)

2013-03-16T18:44:45+00:00

Thanks to a contribution, Nuitka now has a proper RPM spec file. And using openSUSE Build Service, we can now enjoy RPMs for all major distributions.

Check out the revamped download page as well.

Note

At the time of this writing, the “stable” Nuitka 0.4.1 link points to a 0.4.2 pre-release. That is only until 0.4.2 final will be released.

Thanks for making this possible, very much appreciated.

Nuitka Release 0.4.1

2013-03-05T22:13:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree.

Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes.

New Features

Python 3.3 experimental support
- Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface.
- Added support for __qualname__ to classes and functions.
- Small compatibility changes. Some exceptions changed, absolute imports are now default, etc.
- For comparison tests, the hash randomization is disabled.
Python 3.2 support has been expanded.

The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code.

Bug Fixes

Default values were not “is” identical.
```
def defaultKeepsIdentity(arg="str_value"):
    print arg is "str_value"


defaultKeepsIdentity()
```
This now prints “True” as it does with CPython. The solution is actually a general code optimization, see below.
Usage of unicode built-in with more than one argument could corrupt the encoding argument string.

An implementation error of the unicode was releasing references to arguments converted to default encoding, which could corrupt it.
Assigning Python3 function annotations could cause a segmentation fault.

Optimization

Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization.

# Now optimizing:
raise TypeError, 1 / 0
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
(1 / 0).something
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
function(a, 1 / 0).something
# into (minus normalization), notice the side effects of first checking
# function and a as names to be defined, these may be removed only if
# they can be demonstrated to have no effect.
function
a
raise ZeroDivisionError, "integer division or modulo by zero"

There is more examples, where the raise propagation is new, but you get the idea.

Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things.
```
# This was already optimized, as it's a compile time constant.
a if ("a",) else b
a if True else b

# These are now optimized, as their truth value is known.
a if (c,) else b
a if not (c,) else b
```
This is simply taking advantage of infrastructure that now exists. Each node kind can overload “getTruthValue” and benefit from it. Help would be welcome to review which ones can be added.
Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused.
Code generation for constants now shares element values used in tuples.

The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up.

Cleanups

Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection.
The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information.
The class names of nodes were stripped from the “CPython” prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it’s also not important enough to dominate the class name.
The re-formulations performed in tree building have moved out of the “Building” module, into names “ReformulationClasses” e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it’s much easier to find the content of interest now.
Added new re-formulation of print statements. The conversion to strings is now made explicit in the node tree.

New Tests

Added test to cover default value identity.

Organizational

The upload of Nuitka to PyPI has been repaired and now properly displays project information again.

Summary

The quicker release is mostly a consolidation effort, without much actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization.

Nuitka Release 0.4.0

2013-02-25T17:39:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings massive progress on all fronts. The big highlight is of course: Full Python3.2 support. With this release, the test suite of CPython3.2 is considered passing when compiled with Nuitka.

Then lots of work on optimization and infrastructure. The major goal of this release was to get in shape for actual optimization. This is also why for the first time, it is tested that some things are indeed compile time optimized to spot regressions easier. And we are having performance diagrams, even if weak ones:

New Features

Python3.2 is now fully supported.
Fully correct metaclass = semantics now correctly supported. It had been working somewhat previously, but now all the corner cases are covered too.
- Keyword only parameters.
- Annotations of functions return value and their arguments.
- Exception causes, chaining, automatic deletion of exception handlers as values.
- Added support for starred assigns.
- Unicode variable names are also supported, although it’s of course ugly, to find a way to translate these to C++ ones.

Bug Fixes

Checking compiled code with instance(some_function, types.FunctionType) as “zope.interfaces” does, was causing compatibility problems. Now this kind of check passes for compiled functions too.
The frame of modules had an empty locals dictionary, which is not compatible to CPython which puts the globals dictionary there too.
For nested exceptions and interactions with generator objects, the exceptions in sys.exc_info() were not always fully compatible. They now are.
The range builtin was not raising exceptions if given arguments appeared to not have side effects, but were still illegal, e.g. range([], 1, -1) was optimized away if the value was not used.
Don’t crash on imported modules with syntax errors. Instead, the attempted recursion is simply not done.
Doing a del on __defaults and __module__ of compiled functions was crashing. This was noticed by a Python3 test for __kwdefaults__ that exposed this compiled functions weakness.
Wasn’t detecting duplicate arguments, if one of them was not a plain arguments. Star arguments could collide with normal ones.
The __doc__ of classes is now only set, where it was in fact specified. Otherwise it only polluted the name space of locals().
When return from the tried statements of a try/finally block, was overridden, by the final block, a reference was leaked. Example code:
```
try:
    return 1
finally:
    return 2
```
Raising exception instances with value, was leaking references, and not raising the TypeError error it is supposed to do.
When raising with multiple arguments, the evaluation order of them was not enforced, it now is. This fixes a reference leak when raising exceptions, where building the exception was raising an exception.

Optimization

Optimizing attribute access to compile time constants for the first time. The old registry had no actual user yet.
Optimizing subscript and slices for all compile time constants beyond constant values, made easy by using inheritance.
Built-in references now convert to strings directly, e.g. when used in a print statement. Needed for the testing approach “compiled file contains only prints with constant value”.
Optimizing calls to constant nodes directly into exceptions.
Optimizing built-in bool for arguments with known truth value. This would be creations of tuples, lists, and dictionaries.
Optimizing a is b and a is not b based on aliasing interface, which at this time effectively is limited to telling that a is a is true and a is not a is false, but this will expand.
Added support for optimizing hasattr, getattr, and setattr built-ins as well. The hasattr was needed for the class re-formulation of Python3 anyway.
Optimizing getattr with string argument and no default to simple attribute access.
Added support for optimizing isinstance built-in.
Was handling “BreakException” and “ContinueException” in all loops that used break or continue instead of only where necessary.
When catching “ReturnValueException”, was raising an exception where a normal return was sufficient. Raising them now only where needed, which also means, function need not catch them ever.

Cleanups

The handling of classes for Python2 and Python3 have been re-formulated in Python more completely.
- The calling of the determined “metaclass” is now in the node tree, so this call may possible to in-line in the future. This eliminated some static C++ code.
- Passing of values into dictionary creation function is no longer using hard coded special parameters, but temporary variables can now have closure references, making this normal and visible to the optimization.
- Class dictionary creation functions are therefore no longer as special as they used to be.
- There is no class creation node anymore, it’s merely a call to type or the metaclass detected.
Re-formulated complex calls through helper functions that process the star list and dict arguments and do merges, checks, etc.
- Moves much C++ code into the node tree visibility.
- Will allow optimization to eliminate checks and to compile time merge, once in-line functions and loop unrolling are supported.
Added “return None” to function bodies without a an aborting statement at the end, and removed the hard coded fallback from function templates. Makes it explicit in the node tree and available for optimization.
Merged C++ classes for frame exception keeper with frame guards.
- The exception is now saved in the compiled frame object, making it potentially more compatible to start with.
- Aligned module and function frame guard usage, now using the same class.
- There is now a clear difference in the frame guard classes. One is for generators and one is for functions, allowing to implement their different exception behavior there.
The optimization registries for calls, subscripts, slices, and attributes have been replaced with attaching them to nodes.
- The ensuing circular dependency has been resolved by more local imports for created nodes.
- The package “nuitka.transform.optimization.registries” is no more.
- New per node methods “computeNodeCall”, “computeNodeSubscript”, etc. dispatch the optimization process to the nodes directly.
Use the standard frame guard code generation for modules too.
- Added a variant “once”, that avoids caching of frames entirely.
The variable closure taking has been cleaned up.
- Stages are now properly numbered.
- Python3 only stage is not executed for Python2 anymore.
- Added comments explaining things a bit better.
- Now an early step done directly after building a tree.
The special code generation used for unpacking from iterators and catching “StopIteration” was cleaned up.
- Now uses template, Generator functions, and proper identifiers.
The return statements in generators are now re-formulated into raise StopIteration for generators, because that’s what they really are. Allowed to remove special handling of return nodes in generators.
The specialty of CPython2.6 yielding non-None values of lambda generators, was so far implemented in code generation. This was moved to tree building as a re-formulation, making it subject to normal optimization.
Mangling of attribute names in functions contained in classes, has been moved into the early tree building. So far it was done during code generation, making it invisible to the optimization stages.
Removed tags attribute from node classes. This was once intended to make up for non-inheritance of similar node kinds, but since we have function references, the structure got so clean, it’s no more needed.
Introduced new package nuitka.tree, where the building of node trees, and operations on them live, as well as recursion and variable closure.
Removed nuitka.transform and move its former children nuitka.optimization and nuitka.finalization one level up. The deeply nested structure turned out to have no advantage.
Checks for Python version was sometimes “> 300”, where of course “>= 300” is the only thing that makes sense.
Split out helper code for exception raising from the handling of exception objects.

New Tests

The complete CPython3.2 test suite was adapted (no __code__, no __closure__, etc.) and is now passing, but only without “–debug”, because otherwise some of the generated C++ triggers (harmless) warnings.
Added new test suite designed to prove that expressions that are known to be compile time constant are indeed so. This works using the XML output done with --dump-xml and then searching it to only have print statements with constant values.
Added new basic CPython3.2 test “Functions32” and “ParameterErrors32” to cover keyword only parameter handling.
Added tests to cover generator object and exception interactions.
Added tests to cover try/finally and return in one or both branches correctly handling the references.
Added tests to cover evaluation order of arguments when raising exceptions.

Organizational

Changed my email from GMX over to Gmail, the old one will still continue to work. Updated the copyright notices accordingly.
Uploaded Nuitka to PyPI as well.

Summary

This release marks a milestone. The support of Python3 is here. The re-formulation of complex calls, and the code generation improvements are quite huge. More re-formulation could be done for argument parsing, but generally this is now mostly complete.

The 0.3.x series had a lot releases. Many of which brought progress with re-formulations that aimed at making optimization easier or possible. Sometimes small things like making “return None” explicit. Sometimes bigger things, like making class creations normal functions, or getting rid of or and and. All of this was important ground work, to make sure, that optimization doesn’t deal with complex stuff.

So, the 0.4.x series begins with this. The focus from now on can be almost purely optimization. This release contains already some of it, with frames being optimized away, with the assignment keepers from the or and and re-formulation being optimized away. This will be about achieving goals from the “ctypes” plan as discussed in the Developer Manual.

Also the performance page will be expanded with more benchmarks and diagrams as I go forward. I have finally given up on “codespeed”, and do my own diagrams.

Static Compilation - That is the point

2012-12-02T12:23:32+00:00

In a recent post, Stefan Behnel questioned the point of static compilation and suggests that in order to be useful, a static compiler needs to add something on top.

This is going to be a rebuttal.

Compatibility, I mean it

First of all, let me start out, by saying that Nuitka is intended to be the fully optimizing compiler of Python. The optimizing part is not yet true. Right now, it’s probably a correct compiler of Python. Correct in the sense that it’s compatible to CPython as far as possible.

As examples of what I mean with compatibility:

Nuitka will hold references to local variables of frames, when an exception is raised, and will release them only once the next exception is raised.
Nuitka will give the same error messages for all kinds of errors. For example, the parameter parsing of functions will be the same.
Nuitka provides all language constructs no matter how absurd or unused they are.

Compatibility != Slower

While generally Nuitka will have a hard time to be faster and compatible to CPython, I don’t have much concern about that. Using guards between optimistic, and less optimistic variants of code, there is no doubt in my head, that for programs, lots of code will only need very minimal type annotation and still receive respectable speedups.

Of course, at this point, this is only speculation. But I somehow gather that the sentiment is that incompatible and fast need to go along. I totally dispute that.

Language Extensions

Now, the “in addition” stuff, that Stefan is talking about. I don’t see the point at all. It is quite obvious that everything you can say with cdef syntax, could also be said with a more Pythonic syntax. And if it were missing, it could be added. And where it’s a semantic change, it should be frowned upon.

For the Nuitka project, I never considered an own parser of Python. No matter how easy it would be, to roll your own, and I understand that Cython did that, it’s always going to be wrong and it’s work. Work that has no point. The CPython implementation exhibits and maintains the module ast that works just fine.

For Python, if this were so really useful, such language extensions should be added to Python itself. If there were missing meaningful things, I contend they would best be added there, not in a fork of it. Note how ctypes and cffi have been added. When I created bindings for Ada code (C exports) to Python, it was so easy to do that in pure Python with ctypes. I so much enjoyed that.

So, slow bindings are in my view really easy to create with plain Python now. Somebody ought to make a “.h” to ctypes/cffi declarations converter, once they are really faster to use (possibly due to Nuitka). For Nuitka it should be possible to accelerate these into direct calls and accesses. At which point, mixing generated C code and C include statements, will just be what it is, a source of bugs that Nuitka won’t have.

Note

Further down, I will give examples of why I think that cdef is inferior to plain Python, even from a practical point of view.

Lack of Interpreter is bad

Static compilation vs. interpretation as a discussion has little merits to me. I find it totally obvious that you don’t need static compilation, but 2 other things:

You may need interpretation.
And may need speed.

To me static code analysis and compilation are means to achieve that speed, but not intended to remove interpretation, e.g. plugins need to still work, no matter how deep the go.

For Cython syntax there is no interpreter, is there? That makes it loose an important point. So it has to have another reason for using it, and that would be speed and probably convenience. Now suppose Nuitka takes over with these benefits, what would it be left with? Right. Nothing. At all. Well, of course legacy users.

The orinal sin fall of PyRex - that is now Cython - is nothing that Nuitka should repeat.

No Lock-in

The Cython language is so underspecified, I doubt anybody could make a compatible implementation. Should you choose to use it, you will become locked in. That means, if Cython breaks or won’t work to begin with, you are stuck.

That situation I totally despise. It seems an unnecessary risk to take. Not only, if your program does not work, you can’t just try another compiler. You also will never really know, if it’s either your fault or Cython’s fault until you do know, whose fault it is. Find yourself playing with removing, adding, or re-phrasing cdef statements, until things work.

Common. I would rather use PyPy or anything else, that can be checked with CPython. Should I ever encounter a bug with it, I can try CPython, and narrow down with it. Or I can try Jython, IronPython, or low and behold, Nuitka.

I think, this totally makes it obvious, that static compilation of a non-Python language has no point to people with Python code.

What I will always admit, is that Cython is (currently) the best way to create fast bindings, because Nuitka is not ready yet. But from my point of view, Cython has no point long term if a viable replacement that is Pythonic exists.

Python alone is a point

So, if you leave out static compilation vs. interpretation and JIT compilation, what would be the difference between PyPy and Nuitka? Well, obviously PyPy people are a lot cooler and cleverer. Their design is really inspiring and impressive. My design and whole approach to Nuitka is totally boring in comparison.

But from a practical standpoint, is there any difference? What is the difference between Jython and PyPy? The target VM it is. PyPy’s or Java’s. And performance it is, of course.

So, with Python implementations all being similar, and just differing in targets, and performances, do they all have no point? I believe taken to the logical conclusion, that is what Stefan suggests. I of course think that PyPy, Nuitka, and Jython have have as much of a point, as CPython does.

Type Annotations done right

And just for fun. This is making up a use cases of type annotations:

plong = long if python_version < 3 else int


@hints.signature(plong, plong)
def some_function(a):
    return a ** 2

Notice how plong depends on an expression, that may become known during compile time or not. Should that turn out to be not possible, Nuitka can always generate code for both branches and branch when called.

Or more complex and useful like this:

def guess_signature(func):
    types = [None]

    emit = types.append
    for arg in inspect.getargnames(func):
        if arg == "l":
            emit(list)
        elif arg == "f":
            emit(float)
        elif arg == "i":
            emit(int)
        else:
            hints.warning("Unknown type %s" % arg)
            emit(None)

    return hints.signature(*types)


def many_hints(func):
    # Won't raise exception.
    hints.doesnot_raise(func)

    # Signature to be inferred by conventions
    guess_signature(func)(func)

    # No side effects
    hints.pure(func)


@many_hints
def some_func1(f):
    return f + 2.0


@many_hints
def some_func2(i):
    return i + 2


@many_hints
def some_func3(l):
    return i + [2]

This is just a rough sketch, but hopefully you get the idea. Do this with Cython, can you?

The hints can be put into decorators, which may be discovered as inlinable, which then see more inlines. For this to work best, the loop over the compile time constant code object, needs to be unrolled, but that appears quite possible.

The signatures can therefore be done fully automatic. One could use prefix notation to indicate types.

Another way would put fixed types for certain variable names. In Nuitka code, “node”, “code”, “context”, etc. have always the same types. I suspect many programs are the same, and it would be sweet, if you could plug something in and check such types throughout all of the package.

And then, what do you do then? Well, you can inspect these hints at run time as well, they work with CPython as well (though they won’t make things faster, only will that find errors in your program), they will even work with PyPy, or at least not harm it. It will nicely JIT them away I suppose.

Your IDE will like the code. syntax highlighting, auto indent will work. With every Python IDE. PyLint will find the bugs I made in that code up there. And Nuitka will compile it and benefit from the hints.

My point here really is, that cdef is not flexible, not standard, not portable. It should die. It totally is anti-Pythonic to me.

Elsewhere

In Java land, people compile to machine code as well. They probably also - like stupid me - didn’t understand that static compilation would have no point. Why do they do it? Why am I using compiled binaries done with their compiler then?

And why didn’t they take the chance to introduce ubercool cdef a-likes while doing it? They probably just didn’t know better, did they?

No seriously. A compiler is just a compiler. It takes a source code in a language and turns it into a package to execute. That may be a required or an optional step. I prefer optional for development turn around. It should try and make code execute as fast as it can. But it should not change the language. With Cython I have to compile. With Nuitka I could.

In fact, I would be hard pressed to find another example of a compiler that extends the interpreted language compiled, just so there is a point in having it.

Conclusion

Nuitka has a point. On top of that I enjoy doing it. It’s great to have the time to do this thing in the correct way.

So far, things worked out pretty well. My earlier experimentations with type inference had shown some promise. The “value friends” thing, and the whole plan, appears relatively sound, but likely is in need of an update. I will work on it in december. Up to now, and even right now I worked on re-formulations, that should have made it possible to get more release ready effects from this.

When I say correct way, I mean this. When I noticed that type inference was harder than it should be, I could take the time and re-architecture things so that it will be simpler. To me that is fun. This being my spare time allows me to do things this efficiently. That’s not an excuse, it’s a fact that explains my approach. It doesn’t mean it makes less sense, not at all.

As for language compatibility, there is more progress with Python3. I am currently changing the class re-formulations for Python2 and Python3 (they need totally different ones due to metaclass changes) and then “test_desc.py” should pass with it too, which will be a huge achievement in that domain. I will do a post on that later.

Then infrastructure, should complete the valgrind based benchmark automatism. Numbers will become more important from now on. It starts to make sense to observe them. This is not entirely as fun. But with improving numbers, it will be good to show off.

And of course, I am going to document some more. The testing strategy of Nuitka is worth a look, because it’s totally different from everything else people normally do.

Anyway. I am not a big fan of controversy. I respect Cython for all it achieved. I do want to go where it fails to achieve. I should not have to justify that, it’s actually quite obvious, isn’t it?

Yours,

Kay

Nuitka Release 0.3.25

2012-11-11T16:29:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings about changes on all fronts, bug fixes, new features. Also very importantly Nuitka no longer uses C++11 for its code, but mere C++03. There is new re-formulation work, and re-factoring of functions.

But the most important part is this: Mercurial unit tests are working. Nearly. With the usual disclaimer of me being wrong, all remaining errors are errors of the test, or minor things. Hope is that these unit tests can be added as release tests to Nuitka. And once that is done, the next big Python application can come.

Bug Fixes

Local variables were released when an exception was raised that escaped the local function. They should only be released, after another exception was raised somewhere.
Identifiers of nested tuples and lists could collide.
```
a = ((1, 2), 3)
b = ((1,), 2, 3)
```
Both tuples had the same name previously, not the end of the tuple is marked too. Fixed in 0.3.24.1 already.
The __name__ when used read-only in modules in packages was optimized to a string value that didn’t contain the package name.
Exceptions set when entering compiled functions were unset at function exit.

New Features

Compiled frames support. Before, Nuitka was creating frames with the standard CPython C/API functions, and tried its best to cache them. This involved some difficulties, but as it turns out, it is actually possible to instead provide a compatible type of our own, that we have full control over.

This will become the base of enhanced compatibility. Keeping references to local variables attached to exception tracebacks is something we may be able to solve now.
Enhanced Python3 support, added support for nonlocal declarations and many small corrections for it.
Writable __defaults__ attribute for compiled functions, actually changes the default value used at call time. Not supported is changing the amount of default parameters.

Cleanups

Keep the functions along with the module and added “FunctionRef” node kind to point to them.
Reformulated or and and operators with the conditional expression construct which makes the “short-circuit” branch.
Access self in methods from the compiled function object instead of pointer to context object, making it possible to access the function object.
Removed “OverflowCheck” module and its usage, avoids one useless scan per function to determine the need for “locals dictionary”.
Make “compileTree” of “MainControl” module to only do what the name says and moved the rest out, making the top level control clearer.
Don’t export module entry points when building executable and not modules. These exports cause MinGW and MSVC compilers to create export libraries.

Optimization

More efficient code for conditional expressions in conditions:
```
if a if b else c:
    ...
```
See above, this code is now the typical pattern for each or and and, so this was much needed now.

Organizational

The remaining uses of C++11 have been removed. Code generated with Nuitka and complementary C++ code now compile with standard C++03 compilers. This lowers the Nuitka requirements and enables at least g++ 4.4 to work with Nuitka.
The usages of the GNU extension operation a ?: b have replaced with standard C++ constructs. This is needed to support MSVC which doesn’t have this.
Added examples for the typical use cases to the User Manual.
The “compare_with_cpython” script has gained an option to immediately remove the Nuitka outputs (build directory and binary) if successful. Also the temporary files are now put under “/var/tmp” if available.
Debian package improvements, registering with doc-base the User Manual so it is easier to discover. Also suggest mingw32 package which provides the cross compiler to Windows.
Partial support for MSVC (Visual Studio 2008 to be exact, the version that works with CPython2.6 and CPython2.7).

All basic tests that do not use generators are working now, but those will currently cause crashes.
Renamed the --g++-only option to --c++-only.

The old name is no longer correct after clang and MSVC have gained support, and it could be misunderstood to influence compiler selection, rather than causing the C++ source code to not be updated, so manual changes will the used.
Catch exceptions for continue, break, and return only where needed for try/finally and loop constructs.

New Tests

Added CPython3.2 test suite as “tests/CPython32” from 3.2.3 and run it with CPython2.7 to check that Nuitka gives compatible error messages. It is not expected to pass yet on Python3.2, but work will be done towards this goal.
Make CPython2.7 test suite runner also execute the generated “doctest” modules.
Enabled tests for default parameters and their reference counts.

Summary

This release marks an important point. The compiled frames are exciting new technology, that will allow even better integration with CPython, while improving speed. Lowering the requirements to C++03 means, we will become usable on Android and with MSVC, which will make adoption of Nuitka on Windows easier for many.

Structurally the outstanding part is the function as references cleanup. This was a blocker for value propagation, because now functions references can be copied, whereas previously this was duplicating the whole function body, which didn’t work, and wasn’t acceptable. Now, work can resume in this domain.

Also very exciting when it comes to optimization is the remove of special code for or and and operators, as these are now only mere conditional expressions. Again, this will make value propagation easier with two special cases less.

And then of course, with Mercurial unit tests running compiled with Nuitka, an important milestone has been hit.

For a while now, the focus will be on completing Python3 support, XML based optimization regression tests, benchmarks, and other open ends. Once that is done, and more certainty about Mercurial tests support, I may call it a 0.4 and start with local type inference for actual speed gains.

Letting go of C++11

2012-10-27T08:59:07+01:00

How Nuitka came to use C++11 and then stop it.

This post is about Nuitka the Python compiler started out using C++0x which is now C++11, and then chose to stop it.

In the Beginning

Very early on, when I considered how to generate code from the node tree, in a way, that mistakes should practically be impossible to make, I made the fundamental decision, that every Python expression, which produces temporary variables, should become an expression in the generated code too.

Note

That is my choice, I think it keeps code generation more simple, and easier to understand. There may come a separate post about how that played out.

That decision meant some trouble. Certain things were not easy, but generally, it was achievable for g++ relatively quickly, and then lots of helper functions would be needed. Think of MAKE_TUPLE and MAKE_DICT, but also other stuff needed that. Calling a Python built-in with variable number of parameters e.g. could be implemented that way easily.

Other nice things were enum classes, and generally good stuff. It was really quick to get Nuitka code generation off the ground this way.

Reality Strikes

But then, as time went on, I found that the order of evaluation was becoming an issue. It became apparent that for more and more things, I needed to reverse it, so it works. Porting to ARM, it then became clear, that it needs to be the other way around for that platform. And checking out clang, which is also a C++11 compiler, I noticed, this one yet uses a different one.

So, for normal functions, I found a solution that involves the pre-processor to reverse or not, both function definition and call sites, and then it is already correct.

This of course, doesn’t work for C++11 variadic functions. So, there came a point, where I had to realize, that each of its uses was more or less causing evaluation order bugs. So that most of their uses were already removed. And so I basically knew they couldn’t stay that way.

Other Features

Also, things I initially assumed, e.g. that lambda functions of C++11 may prove useful, or even “auto”, didn’t turn out to be true. There seemingly is a wealth of new features, besides variadic templates that I didn’t see how Nuitka would benefit from it at all.

New Wishes

Then, at Europython, I realized, that Android is still stuck with g++-4.4 and as such, that an important target platform will be unavailable to me. This platform will become even more important, as I intend to buy an device now.

Biting it

So what I did, was to remove all variadic functions and instead generate code for them as necessary. I just need to trace the used argument counts, and then provide those, simple enough.

Also, other things like deleted copy constructors, and so on, I had to give up on these a bit.

This change was probably suited to remove subtle evaluation order problems, although I don’t recall seeing them.

The Present

The current stable release still requires C++11, but the next release will work on g++-4.4 and compiles fine with MSVC from Visual Studio 2008, although at this time, there is still the issue of generators not working yet, but I believe that ought to be solvable.

The new requirement is only C++03, which means, there is a good chance that supporting Android will become feasible. I know there is interest from App developers, because there, even the relatively unimportant 2x speedup, that Nuitka might give for some code, may matter.

Conclusion

So that is a detour, I have taken, expanding the base of Nuitka even further. I felt, this was important enough to write down the history part of it.

Nuitka Release 0.3.24

2012-08-18T21:30:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains progress on many fronts, except performance.

The extended coverage from running the CPython 2.7 and CPython 3.2 (partially) test suites shows in a couple of bug fixes and general improvements in compatibility.

Then there is a promised new feature that allows to compile whole packages.

Also there is more Python3 compatibility, the CPython 3.2 test suite now succeeds up to “test_builtin.py”, where it finds that str doesn’t support the new parameters it has gained, future releases will improve on this.

And then of course, more re-formulation work, in this case, class definitions are now mere simple functions. This and later function references, is the important and only progress towards type inference.

Bug Fixes

The compiled method type can now be used with copy module. That means, instances with methods can now be copied too. Fixed in 0.3.23.1 already.
The assert statement as of Python2.7 creates the AssertionError object from a given value immediately, instead of delayed as it was with Python2.6. This makes a difference for the form with 2 arguments, and if the value is a tuple. Fixed in 0.3.23.1 already.
Sets written like this didn’t work unless they were predicted at compile time:
```
{value}
```
This apparently rarely used Python2.7 syntax didn’t have code generation yet and crashed the compiler. Fixed in 0.3.23.1 already.
For Python2, the default encoding for source files is ascii, and it is now enforced by Nuitka as well, with the same SyntaxError.
Corner cases of exec statements with nested functions now give proper SyntaxError exceptions under Python2.
The exec statement with a tuple of length 1 as argument, now also gives a TypeError exception under Python2.
For Python2, the del of a closure variable is a SyntaxError.

New Features

Added support creating compiled packages. If you give Nuitka a directory with an “__init__.py” file, it will compile that package into a “.so” file. Adding the package contents with --recurse-dir allows to compile complete packages now. Later there will be a cleaner interface likely, where the later is automatic.
Added support for providing directories as main programs. It’s OK if they contain a “__main__.py” file, then it’s used instead, otherwise give compatible error message.
Added support for optimizing the super built-in. It was already working correctly, but not optimized on CPython2. But for CPython3, the variant without any arguments required dedicated code.
Added support for optimizing the unicode built-in under Python2. It was already working, but will become the basis for the str built-in of Python3 in future releases.
For Python3, lots of compatibility work has been done. The Unicode issues appear to be ironed out now. The del of closure variables is allowed and supported now. Built-ins like ord and chr work more correctly and attributes are now interned strings, so that monkey patching classes works.

Organizational

Migrated “bin/benchmark.sh” to Python as “misc/run-valgrind.py” and made it a bit more portable that way. Prefers “/var/tmp” if it exists and creates temporary files in a secure manner. Triggered by the Debian “insecure temp file” bug.
Migrated “bin/make-dependency-graph.sh” to Python as “misc/make-dependency-graph.py” and made a more portable and powerful that way.

The filtering is done a more robust way. Also it creates temporary files in a secure manner, also triggered by the Debian “insecure temp file” bug.

And it creates SVG files and no longer PostScript as the first one is more easily rendered these days.
Removed the “misc/gist” git sub-module, which was previously used by “misc/make-doc.py” to generate HTML from User Manual and Developer Manual.

These are now done with Nikola, which is much better at it and it integrates with the web site.
Lots of formatting improvements to the change log, and manuals:
- Marking identifiers with better suited ReStructured Text markup.
- Added links to the bug tracker all Issues.
- Unified wordings, quotation, across the documents.

Cleanups

The creation of the class dictionaries is now done with normal function bodies, that only needed to learn how to throw an exception when directly called, instead of returning NULL.

Also the assignment of __module__ and __doc__ in these has become visible in the node tree, allowing their proper optimization.

These re-formulation changes allowed to remove all sorts of special treatment of class code in the code generation phase, making things a lot simpler.
There was still a declaration of PRINT_ITEMS and uses of it, but no definition of it.
Code generation for “main” module and “other” modules are now merged, and no longer special.
The use of raw strings was found unnecessary and potentially still buggy and has been removed. The dependence on C++11 is getting less and less.

New Tests

Updated CPython2.6 test suite “tests/CPython26” to 2.6.8, adding tests for recent bug fixes in CPython. No changes to Nuitka were needed in order to pass, which is always good news.
Added CPython2.7 test suite as “tests/CPython27” from 2.7.3, making it public for the first time. Previously a private copy of some age, with many no longer needed changes had been used by me. Now it is up to par with what was done before for “tests/CPython26”, so this pending action is finally done.
Added test to cover Python2 syntax error of having a function with closure variables nested inside a function that is an overflow function.
Added test “BuiltinSuper” to cover super usage details.
Added test to cover del on nested scope as syntax error.
Added test to cover exec with a tuple argument of length 1.
Added test to cover barry_as_FLUFL future import to work.
Removed “Unicode” from known error cases for CPython3.2, it’s now working.

Summary

This release brought forward the most important remaining re-formulation changes needed for Nuitka. Removing class bodies, makes optimization yet again simpler. Still, making function references, so they can be copied, is missing for value propagation to progress.

Generally, as usual, a focus has been laid on correctness. This is also the first time, I am release with a known bug though, one which I believe now, may be the root cause of the mercurial tests not yet passing.

The solution will be involved and take a bit of time. It will be about “compiled frames” and be a (invasive) solution. It likely will make Nuitka faster too. But this release includes lots of tiny improvements, for Python3 and also for Python2. So I wanted to get this out now.

As usual, please check it out, and let me know how you fare.

Nuitka Release 0.3.23

2012-06-30T23:40:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is the one that completes the Nuitka “sun rise phase”.

All of Nuitka is now released under Apache License 2.0 which is a very liberal license, and compatible with basically all Free Software licenses there are. It’s only asking to allow integration, of what you send back, and patent grants for the code.

In the first phase of Nuitka development, I wanted to keep control over Nuitka, so it wouldn’t repeat mistakes of other projects. This is no longer a concern for me, it’s not going to happen anymore.

I would like to thank Debian Legal team, for originally bringing to my attention, that this license will be better suited, than any copyright assignment could be.

Bug Fixes

The compiled functions could not be used with multiprocessing or copy.copy. Fixed in 0.3.22.1 already.
In-place operations for slices with not both bounds specified crashed the compiler. Fixed in 0.3.22.1 already.
Cyclic imports could trigger an endless loop, because module import expressions became the parent of the imported module object. Fixed in 0.3.22.2 already.
Modules named proc or func could not be compiled to modules or embedded due to a collision with identifiers of CPython2.7 includes. Fixed in 0.3.22.2 already.

New Features

The function copying fix also makes pickling of compiled functions available. As it is the case for non-compiled functions in CPython, no code objects are stored, only names of module level variables.

Organizational

Using the Apache License 2.0 for all of Nuitka now.
Speedcenter has been re-activated, but is not yet having a lot of benchmarks yet, subject to change.

Update

We have given up on this version of speedcenter meanwhile, and generate static pages with graphs instead. We can this still speedcenter.

New Tests

Changed the “CPython26” tests to no longer disable the parts that relied on copying of functions to work as that is now supported.
Extended in-place assignment tests to cover error cases of we had issues with.
Extended compile library test to also try and compile the path where numpy lives. This is apparently another path, where Debian installs some modules, and compiling this would have revealed issues sooner.

Summary

The release contains bug fixes, and the huge step of changing the license. It is made in preparation to PyCON EU.

Nuitka Speedcenter is back

2012-06-30T16:39:00+01:00

Once a long time ago, I was benchmarking Nuitka more often. Check “What is Nuitka?” in case you don’t know what it is.

Problem

And I was considering the use of codespeed, and had some data online. But ultimately, it got discontinued. This has 3 reasons:

Moved the website to a dedicated machine, which broke the previous install.
Controlling which data is used for display was hard and not satisfactory.
1. For example, I didn’t want to have to commit and push, just to let the benchmarks run.
2. And I wanted to be able to re-run benchmarks with newer compiler, even newer Python, but old Nuitka. Using g++ 4.6 over g++ 4.5 should not impact the data.
3. It turned out to be a nightmare to migrate to newer codespeed versions. I found myself starting from empty database - over and over.
4. Many things were not supported.
  
  For example, I would want to collect all PyBench results, but only publish those who are expressive. That seemed difficult to achieve.
Benchmarks of Nuitka are not yet useful
1. Nuitka was not yet doing type inference
2. Most of the work was aimed at correctness, and effectively was often degrading performance even if only temporary. Seeing it wouldn’t have been too motivating.

Solution

I have simply created a small wrapper:

Small script to run benchmarks and collect data.

It checks out Nuitka in all versions in a playground, and then runs defined benchmarks, with valgrind, etc. taking exe sizes, etc.
Data is stored in local sqlite databases.

I have a database per machine, i.e. a distributed repository, where I collect information. That works for me, and will allow me to compare different kinds of machines.

The advantage is that I have no risk of data loss anymore, and no issues and difficulty with poor interfaces to replace existing data.
Data is merged on one machine, and then pushed.

That allows me to inspect the changes before publishing them. It allows me to play with local commits, branches, with information that will go away. I can then push when I choose to.

That integrates better with my work flow. It allows me to retro-fit benchmarks results on the machine and to be tool independent.

In principle, I could publish the data in other forms as well, and I likely will. Making tables of e.g. PyBench results seems like one application. Recently, I have worked with Nikola, and could also imagine to integrate Codespeed graph functionality (which is apparently all I want) to there.

Yours,

Kay

Nuitka Release 0.3.22

2012-05-13T13:17:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimization.

It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to compile itself with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it’s quite fine now.

Bug Fixes

List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions.
Calling a compiled function in an exception handler cleared the exception on return, it no longer does that.
Reference counter handling with generator throw method is now correct.
A module “builtins” conflicted with the handling of the Python builtins module. Those now use different identifiers.

New Features

New metaclass syntax for the class statement works, and the old __metaclass__ attribute is properly ignored.

# Metaclass syntax in Python3, illegal in Python2
class X(metaclass=Y):
    pass

# Metaclass syntax in Python2, no effect in Python3
class X:
    __metaclass__ = Y

Note

The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn’ it. Surely, the support for __metaclass__ could still live.

# For Python2/3 compatible source, we create a base class that has the
# metaclass used and doesn't require making a choice.

CPythonNodeMetaClassBase = NodeCheckMetaClass("CPythonNodeMetaClassBase", (object,), {})

The --dump-xml option works with Nuitka running under Python3. This was not previously supported.
Python3 now also has compatible parameter errors and compatible exception error messages.
Python3 has changed scope rules for list contractions (assignments don’t affect outside values) and this is now respected as well.
Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well.

Optimization

Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used.

This avoids overhead for the very simple functions. And example of this can be seen here:
```
def simple():
    return 7
```

Optimize len built-in for non-constant, but known length values.

An example can be seen here:

# The range isn't constructed at compile time, but we still know its
# length.
len(range(10000000))

# The string isn't constructed at compile time, but we still know its
# length.
len("*" * 1000)

# The tuple isn't constructed, instead it's known length is used, and
# side effects are maintained.
len((a(), b()))

This new optimization applies to all kinds of container creations and the range built-in initially.

Optimize conditions for non-constant, but known truth values.

At this time, known truth values of non-constants means range built-in calls with know size and container creations.

An example can be seen here:
```
if (a,):
    print "In Branch"
```
It’s clear, that the tuple will be true, we just need to maintain the side effect, which we do.

Optimize or and and operators for known truth values.

See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can:

# The access and call to "something()" cannot possibly happen
0 and something()

# Can be replaced with "something()", as "1" is true. If it had a side
# effect, it would be maintained.
1 and something()

# The access and call to "something()" cannot possibly happen, the value
# is already decided, it's "1".
1 or something()

# Can be replaced with "something()", as "0" is false. If it had a side
# effect, it would be maintained.
0 or something()

Optimize print arguments to become strings.

The arguments to print statements are now converted to strings at compile time if possible.
```
print 1
```
becomes:
```
print "1"
```
Combine print arguments to single ones.

When multiple strings are printed, these are now combined.
```
print "1+1=", 1 + 1
```
becomes:
```
print "1+1= 2"
```

Organizational

Enhanced Python3 support, enabling support for most basic tests.
Check files with PyLint in deterministic (alphabetical) order.

Cleanups

Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module.
The try/except/else has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not.
Side effects are now a dedicated node, used in several optimization to maintain the effect of an expression with known value.

New Tests

Expanded and adapted basic tests to work for Python3 as well.
Added reference count tests for generator functions throw, send, and close methods.
Cover calling a function with try/except in an exception handler twice. No test was previously doing that.

Summary

This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away.

There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense.

Nuitka Release 0.3.21

2012-04-13T10:19:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it will be easy to start tracking them.

Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc.

Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete.

Bug Fixes

The built-in next could causes a program crash when iterating past the end of an iterator. Fixed in 0.3.20.1 already.
The set constants could cause a compiler error, as that type was not considered in the “mutable” check yet. Fixed in 0.3.20.2 already.
Performance regression. Optimize expression for exception types caught as well again, this was lost in last release.
Functions that contain exec, are supposed to have a writable locals. But when removing that exec statement as part of optimization, this property of the function could get lost.
The so called “overflow functions” are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an exec or a star import.
The syntax error for return outside of a function, was not given, instead the code returned at run time. Fixed to raise a SyntaxError at compile time.

Optimization

Avoid tuple objects to be created when catching multiple exception types, instead call exception match check function multiple times.
Removal of dead code following break, continue, return, and raise. Code that follows these statements, or conditional statements, where all branches end with it.

Note

These may not actually occur often in actual code, but future optimization may produce them more frequently, and their removal may in turn make other possible optimization.
Detect module variables as “read only” after all writes have been detected to not be executed as removed. Previously the “read only indicator” was determined only once and then stayed the same.
Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value.
Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still.

Note

An exception is where len = len actually does have an impact, because that variable becomes assignable. The “compile itself” test of Nuitka found that to happen with long from the nuitka.__past__ module.
Created Python3 variant of quick unicode string access, there was no such thing in the CPython C/API, but we make the distinction in the source code, so it makes sense to have it.
Created an optimized implementation for the built-in iter with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython C/API.
For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take.
Don’t use a “context” object for generator functions (and generator expressions) that don’t need one. And even if it does to store e.g. the given parameter values, avoid to have a “common context” if there is no closure taken. This avoids useless malloc calls and speeds up repeated generator object creation.

Organizational

Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated.
The --experimental option is no longer available outside of checkouts of git, and even there not on stable branches (master, hotfix/...). It only pollutes --help output as stable releases have no experimental code options, not even development version will make a difference.
The binary “bin/Nuitka.py” has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway.
The --python-version option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values.
Added --keep-pythonpath alias for --execute-with-pythonpath option, probably easier to remember.
Support --debug with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn’t report anything new.
The contents environment variable CXX determines the default C++ compiler when set, so that checking with CXX=g++-4.7 nuitka-python ... has become supported.
The check-with-pylint script now has a real command line option to control the display of TODO items.

Cleanups

Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead.
```
a = b = c
```
```
_tmp = c
b = _tmp
a = _tmp
```
In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block.

All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed.
Changed unpacking assignments to use temporary variables. Code like this:
```
a, b = c
```
Is handled more like this:
```
_tmp_iter = iter(c)
_tmp1 = next(_tmp_iter)
_tmp2 = next(_tmp_iter)
if not finished(_tmp_iter):
    raise ValueError("too many values to unpack")
a = _tmp1
b = _tmp2
```
In reality, not really next is used, as it wouldn’t raise the correct exception for unpacking, and the finished check is more condensed into it.

Generally this cleanup allowed that the AssignTargetTuple and associated code generation was removed, and in the future value propagation may optimize these next and iter calls away where possible. At this time, this is not done yet.
Exception handlers assign caught exception value through assignment statement.

Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well.
Exception handlers now explicitly catch more than one type.

Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it’s all.
Contractions are now functions as well.

Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation.
Global is handled during tree building.

Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node.

It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that global statement, allowing to remove the node class.
Read only module variable detection integrated to constraint collection.

The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed.

New Tests

Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment.
Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn’t even know it can be done, example:
```
try:
    raise ValueError(1, 2)
except ValueError as (a, b):
    print "Unpacking caught exception and unpacked", a, b
```
Will assign a=1 and b=2.
Added test to cover return statements on module level and class level, they both must give syntax errors.
Cover exceptions from accessing unassigned global names.
Added syntax test to show that star imports do not allow other names to be imported at the same time as well.
Python3 is now also running the compile itself test successfully.

Summary

The progress made towards value propagation and type inference is very significant, and makes those appears as if they are achievable.

Nuitka Release 0.3.20

2012-02-27T22:48:00+00:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This time there are a few bug fixes and some really major cleanups, lots of new optimization and preparations for more. And then there is a new compiler clang and a new platform supported. macOS X appears to work mostly, thanks for the patches from Pete Hunt.

Bug Fixes

The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. Fixed in release 0.3.19.1 already.
The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. Fixed in release 0.3.19.1 already.
The from x import * attempted to respect __all__ but failed to do so. Fixed in release 0.3.19.2 already.
The from x import * didn’t give a SyntaxError when used on Python3. Fixed in release 0.3.19.2 already.
The syntax error messages for “global for function argument name” and “duplicate function argument name” are now identical as well.
Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed.

New Features

Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent.
Enhanced Python3 support for syntax errors, these are now also compatible.
Support for macOS X was added.
Support for using the clang compiler was added, it can be enforced via --clang option. Currently this option is mainly intended to allow testing the “macOS X” support as good as possible under Linux.

Optimization

Enhanced all optimization that previously worked on “constants” to work on “compile time constants” instead. A “compile time constant” can currently also be any form of a built-in name or exception reference. It is intended to expand this in the future.
Added support for built-ins bin, oct, and hex, which also can be computed at compile time, if their arguments are compile time constant.
Added support for the iter built-in in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster.
Added support for the next built-in, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well.
Added support for the open built-in in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files.
Optimize the __debug__ built-in constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided.
Optimize the Ellipsis built-in constant as well. It falls in the same category as True, False, None, i.e. names of built-in constants that a singletons.
Added support for anonymous built-in references, i.e. built-ins which have names that are not normally accessible. An example is type(None) which is not accessible from anywhere. Other examples of such names are compiled_method_or_function.

Having these as represented internally, and flagged as “compile time constants”, allows the compiler to make more compile time optimization and to generate more efficient C++ code for it that won’t e.g. call the type built-in with None as an argument.
All built-in names used in the program are now converted to “built-in name references” in a first step. Unsupported built-ins like e.g. zip, for which Nuitka has no own code or understanding yet, remained as “module variables”, which made access to them slow, and difficult to recognize.
Added optimization for module attributes __file__, __doc__ and __package__ if they are read only. It’s the same as was done for __name__ so far only.
Added optimization for slices and subscripts of “compile time constant” values. These will play a more important role, once value propagation makes them more frequent.

Organizational

Created a “change log” from the previous release announcements. It’s as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it’s so easy to generate the PDF on that yourself.
The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to Wordpress as a draft posting or updates it, until it’s release time. Simple, sweet and elegant.

Cleanups

Split out the nuitka.nodes.Nodes module into many topic nodes, so that there are now nuitka.nodes.BoolNodes or nuitka.nodes.LoopNodes to host nodes of similar kinds, so that it is now cleaner.
Split del statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same:
```
del a, b.c, d
```
```
del a
del b.c
del d
```
So that’s now represented in the node tree. And even more complex looking cases, like this one, also the same:
```
del a, (b.c, d)
```
This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all.
Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions.

Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the bases for classes and the defaults for functions and make optional.
Refactored the decorator handling to the tree building stage, presenting them as function calls on “function body expression” or class body expression”.

This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimization, as they practically are now just function calls.
```
@some_classdecorator
class C:
    @staticmethod
    def f():
        pass
```
It’s just a different form of writing things. Nothing requires the implementation of decorators, it’s just functions calls with function bodies before the assignment.

The following is only similar:
```
class C:
    def f():
        pass

    f = staticmethod(f)


C = some_classdecorator(C)
```
It’s only similar, because the assignment to an intermediate value of C and f is not done, and if an exception was raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already.
The in-place assignments statements are now handled using temporary variable blocks

Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal look-ups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future.
The for loop used to have has a “source” expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimization. Moved it to tree building stage instead, where optimization can work on it then.
Tree building now generally allows statement sequences to be None everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist.
Moved the optimization for __name__ to compute node of variable references, where it doesn’t need anything complex to replace with the constant value if it’s only read.
Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults.
Made the built-in code more reusable.

New Tests

Added some more diagnostic tests about complex assignment and del statements.
Added syntax test for star import on function level, that must fail on Python3.
Added syntax test for duplicate argument name.
Added syntax test for global on a function argument name.

Summary

The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much more consistency there.

Yet, even more work will be needed in the same direction. There may e.g. be work required to cover with statements as well. And assignments will become no more complex than unpacking from a temporary variable.

For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only minuscule progress. The remaining tasks appear all more or less difficult work that I don’t want to touch now.

There are still remaining steps, but we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about.

Nuitka Blog - Posts tagged compiler

Nuitka Release 2.8

Bug Fixes

Package Support

New Features

Optimization

Anti-Bloat

Organizational

Tests

Cleanups

Summary

Nuitka Release 2.7

Bug Fixes

Package Support

New Features

Optimization

Anti-Bloat

Organizational

Tests

Cleanups

Summary

Nuitka this week #16

Nuitka Onefile Gets More Flexible: --onefile-cache-mode and {PROGRAM_DIR}

Taking Control of Onefile Unpacking: --onefile-cache-mode

Relative Paths with {PROGRAM_DIR}

Nuitka Commercial and Open Source

Give it a Try!

Nuitka Release 2.6

Bug Fixes

Package Support

New Features

Optimization

Anti-Bloat

Organizational

Tests

Cleanups

Summary

Nuitka Release 2.5

Bug Fixes

Package Support

New Features

Optimization

Anti-Bloat

Organizational

Tests

Cleanups

Summary

Nuitka Release 2.4

Nuitka Release 2.3

Nuitka Release 2.2

Bug Fixes

New Features

Optimization

Organizational

Tests

Cleanups

Summary

Nuitka Release 2.1

Bug Fixes

New Features

Optimization

Organizational

Cleanups

Tests

Summary

Nuitka Release 2.0

Bug Fixes

New Features

Optimization

Organizational

Cleanups

Tests

Summary

Nuitka this week #15

Nuitka Package Configuration Part 3

Problem Package

Initial Symptom

Step 1 - Analysis of initial crashing

Step 2 - Dynamic Backend dependency

Final remarks

Nuitka Onefile Gets More Flexible: `--onefile-cache-mode` and `{PROGRAM_DIR}`

Taking Control of Onefile Unpacking: `--onefile-cache-mode`

Relative Paths with `{PROGRAM_DIR}`