05 February 2024
Nuitka Release 2.0
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release had focus on new features and new optimization. There is a really large amount of compatibility with things newly added, but also massive amounts of new features, and esp. for macOS and Windows, lot of platform specified new abilities and corrections. Fix, workaround for private functions as Qt slots not having names mangled. Fixed in 1.9.1 already. Fix, when using Nuitka with Fix, for Standalone: Added data files for Standalone: Added DLL for Standalone: Fix, the specs expanded at run time in some causes could contain random characters. Fixed in 1.9.2 already. Fix, Standalone: Added data files for Standalone: Added more metadata requirements for Plugins: Fix, could crash when including packages from the command line, if they had yaml configuration that requires checking the using module, e.g. anti-bloat work. Fixed in 1.9.3 already. Standalone: Added support for Android: Fix, cannot exclude Standalone: Add needed DLLs for Windows: Fix, newer Windows: Fix, could crash when working with junctions that switch drives. Fixed in 1.9.3 already. Fix, was crashing with poetry installed environments. Fixed in 1.9.3 already. Standalone: Added support for newer Fix, could crash in report creation on modules excluded that were asked via command line for inclusion. Fixed in 1.9.3 already. Anti-Bloat: Fix for newer Arch: Added support for their OS release file location too. Fixed in 1.9.4 already. Windows: Fix, MinGW64 doesn’t accept chinese module names a C source files. Use short paths for these instead. Fixed in 1.9.4 already. Standalone: Added missing DLL for Fix, properly skip directories with non-module top level names when trying to find top level packages of distributions. Fixed in 1.9.4 already. Fix, avoid memory leak bug in triggered by Python3.11+: Fix, didn’t detect non-keywords on star dict calls in some cases. Fixed in 1.9.4 already. Fix, avoid crashes due to unrecognized installers on macOS and Windows, some packages that are built via legacy fallbacks of certain pip versions do not leave any indication of their origin at all. Fixed in 1.9.4 already. Windows: Fix, need to indicate that the program is long path aware or else it cannot work with the paths. Fixed in 1.9.4 already. Debian: The Compatibility: Fix, need to also have macOS: Fix, PyQt5 standalone can fail due to Compatibility: Make Standalone: Added data file for Windows: Fix warnings given by MinGW64 in debug mode for onefile compilation. Fixed in 1.9.6 already. Python2: The handling of DLL permission changes was not robust against using unicode filenames. Fixed in 1.9.7 already. Python2: Fix, could crash on Debian packages when detecting their installer. Fixed in 1.9.7 already. Standalone: Added required data file for Reports: Fix, in case of build crashes during optimization, the bug report creation could be crashing because the module is not in the list of done modules yet. Fixed in 1.9.7 already. Python2: Fix, Accelerated: Fix, tensorflow configuration removing Plugins: Fix, the Standalone: Added necessary data file for Standalone: Fix, was not properly handling standard library overloading module names for decisions. Inclusion and compilation mode were made as if the module was part of the standard library, rather than user code. This is now properly checking if it’s also an actual standard library module. Plugins: Fix, crashing on missing absence message with no UPX binary was found. Windows: Fix, couldn’t load extension modules from UNC paths, so standalone distributions failed to launch from network drives. This now works again and was a regression from adding support for symlinks on Windows. Standalone: Added support for non-legacy Standalone: Added required Nuitka-Python: Fix, do not demote to non-LTO for “too many” modules there in the default auto mode, it doesn’t work without it. Fix, macOS: Fix, the standard Compatibility: Add back PySide6 workaround for overloading names like Standalone: Added Python2: Fix, code objects must be made from Standalone: Added missing data files for Standalone: Added missing DLL for Compatibility: Fix, PySide2 and PySide6 signal disconnection without arguments were not working yet. Standalone: Added support for Scons: Fix, need to Avoid picking up Fix, version of Fix, icon options for platforms were conflated, so what should be windows only icon could get used on other platforms as well. Fix, could not create compiled methods from compiled methods. Also now errors out for invalid types given properly. Plugins: Added support for module decisions, these are Plugins: Added support for Standalone: Added module decision to disable Torch JIT. This is generally the right idea, but the decision is still asked for since some packages and programs want to do Torch Tracing, and that is then disabled as well. This makes a bunch of transformers programs work. Standalone: Added module decision to disable Numba JIT. This makes New option macOS: Added new value macOS: Added support for Windows: Added support for using junctions in the Python environment, these are used e.g. when installing via Added option Reports: Added output filename to report, mainly intended for automatically locating the compilation result independent of options used. Plugins: Now provides a checksum for yaml files, but not yet verifies them at runtime, to ask the user to run the checker tool to update it when they make modifications. Windows: Detect when we create too large compiled executables. There is a limit of 2GB that you might e.g. violate by attempting to embed very large files. This doesn’t cover onefile yet. Watch: The tool can now create PRs with the changes in Nuitka-Watch for merging, this is for using it in the CI. Watch: Scanning for Python versions now requires Watch: Added ability to create branch and PR from watch run results. Plugins: Added Plugins: Added Nuitka-Action: Nuitka options that can be given multiple times, cannot be specified multiple times in your workflow. As a workaround, Nuitka now allows in Actions, to use new lines as separator. This is best done with this kind of quoting a multiline string. The Nuitka Package Configuration Windows: Added support for compiling in case sensitive folders. When this option is enabled, using The Scalability: Avoid variables that are not shared to be treated as if they were, marking their type shape as Escaped variables did provide a type shape Loop type shape analysis now succeeds in detecting the types for this code example, which is sort of a break-through for future performance enhancements in generated code. Python3.11+: Use Avoid late specialization for Added successful detection of static libpython for self compiled Python Linux and macOS. This makes it work with Standalone: Avoid including macOS: Make static libpython default with CPython for more compact standalone distribution and faster binaries. Remove non-existent entries from Anti-Bloat: Avoid using Anti-Bloat: Avoid using Anti-Bloat: Avoid using Anti-Bloat: Avoid using Anti-Bloat: Avoid using Anti-Bloat: Avoid Anti-Bloat: Avoid Avoid importing Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Using Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Avoid more Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Avoid UI: Change template paths over from The old spec values are migrated transparently and continue to work, but get a warning when used. The new code detects unknown variable names and more formatting issues than before. Using only the The main benefit and reason of doing this, is that Windows As a result, compile time and run time variables now clash, there is e.g. Project: Added Code of Conduct. Adapted from the one used in the Linux kernel. UI: Warnings given by Nuitka used to be in red color, changed those to be yellow for consistency. User Manual: Added pointer for Nuitka-Action Nuitka-Action for users interested in using Nuitka in GitHub workflows. Added User Manual: Added information on how to debug fork bombs from created binaries. Debugging: The output of Quality: Warnings from UI: Enhanced formatting of info traces, drop the UI: Changed UI: Allow using Quality: Enhanced schema for our package configuration yaml files to detect suffixes with leading dots, that is not wanted. These now fail checks, but we also tolerate them now. Quality: Check module names used in the package configuration yaml files for validity, this catches e.g. trailing dots. Quality: Make sure to really prefer Scons: Allow disabling to enforce no warnings for C compilation Currently only for gcc, where we need it until loop tracing is better, we can now use macOS: Make Standalone: Added support for using self-compiled Python versions that are not installed on Linux and macOS. This avoids having to do Release: Added ability to simple re-date hotfixes. Previously the version bump commit needed to be dropped, now a fixup commit is easy to generate. Release: Man pages are no longer built during package builds, but are available statically in the git, which should make it easier. Release: Disable verbose output in package installation of Nuitka, it never was any use, and just makes things hard to read. UI: Check user yaml file present immediately. Otherwise it was crashing when parsing yaml files first time with less comprehensible exceptions. Added in 1.9.7 already. Quality: Updated to latest Debian: Remove references to PDF documentation that no longer exists. Quality: Do not crash when collecting modified files due to deleted files. UI: Detect the Alpine flavor of Python as well. UI: Detect UI: Detect self compiled uninstalled Python as a dedicated flavor. For the Nuitka-Action part of the available options is now generated from Nuitka option definitions itself, adding some previously missing options as a result. As a result, adding The warnings for onefile only options without onefile mode provided have been moved to common code, and in some cases were having wrong texts corrected. Use enum definitions in the Nuitka Package Configuration schema rather than manual The User Manual was proof read and had a bunch of wordings improved. Cleanup, avoid “unused but set variable” warning from the C compiler for hard some forms of hard imports. Prefer Changed parts of the C codes that When locating the standalone binary created, use a compilation report and resolve the path specified there. This allows macOS app bundles to be used in these tests as well. Made the PyQt tests executable on macOS too adding necessary options. Added reference test case for unpacking into a list, this was not covered but under suspect of reference leaking which turns out to be wrong. Much enhanced usage of This release deserves the 2.0 marker, as it is ground breaking in many ways. The loop type analysis stands out on the optimization front. This will open an avenue for much optimized code at least for some benchmark examples this summer. The new features for package configuration, demonstrate abilities to avoid plugins for Nuitka, where those previously would have been used. The new The scope of supported Python configurations got expanded a bit, and the the usual slew of anti-bloat work and new packages supported, makes Nuitka an ever more round package. The improved user dialog with less noisy messages and slightly better coloring, continues a trend, where Nuitka becomes more and more easy to use.Bug Fixes
pdm it was not detected as using pip packages. Fixed in 1.9.1 already.pydantic our lazy loader parser didn’t handle all cases properly yet. Fixed in 1.9.1 already.pyocd package. Fixed in 1.9.1 already.cmsis_pack_manager package. Fixed in 1.9.1 already.{"a":b, ...}.get("b") could crash at runtime. Fixed in 1.9.2 already.pyproj package. Fixed in 1.9.2 already.transformers package. Fixed in 1.9.2 already.delphifmx package. Fixed in 1.9.4 already.libz on that platform, it’s not a full Linux OS. Fixed in 1.9.3 already.bitsandbytes package. Fixed in 1.9.3 already.joblib was not working anymore. Fixed in 1.9.3 already.chromadb package. Fixed in 1.9.3 already.streamlit, it was causing SyntaxError for the compilation. Fixed in 1.9.4 already.libusb_package package. Fixed in 1.9.4 already.rich package. Fixed in 1.9.4 already.extern namespace might not exist in the pkg_resources module, make the code work with versions that remove it and use the proper external package names then. Fixed in 1.9.6 already..exists method in our files reader objects. Fixed in 1.9.5 already.libqpdf too.dill-compat plugin support module mode too, previously this only worked for executables only. Fixed in 1.9.6 already.curl_cffi package. Fixed in 1.9.6 already.astor package. Fixed in 1.9.7 already.unittest.mock was not yet available, code attempting to use it was crashing the compilation. Fixed in 1.9.7 already.site usage needs to apply only to standalone mode. Fixed in 1.9.7 already.get_dist_name Nuitka Package Configuration function could crash in some rare configurations. Fixed in 1.9.7 already.pygame package. Added in 1.9.7 already.pillow in imageio package.easyOCR data file.python setup.py install could fail. Apparently it tries to lookup Nuitka during installation, which then could fail, due to hacks we due to make sure wheels are platform dependent. That hack is of course not really needed for install, since no collision is going to happen there.matplotlib plugin that uses native UI was not included yet, and it was also not working due to bindings requiring uncompiled functions, which is now worked around.update with slots.geopandas data files.str exactly, unicode however was used in some configurations after recent improvements to the run time path handling.boto, the predecessor of boto3 as well.tensorflow factorization module.toga.clang from PATH on Windows with --clang provided, as only our winlibs version is really working.setuptools when included (which we try to avoid very much) was None which breaks some users of it, now it’s the correct version so checks of e.g. setuptools_scm can succeed.New Features
parameters provided by the user which can be used to influence the Nuitka per package configuration with a new get_parameter function. We are using these to control important choices in the user, sometimes warning it to make that decision, if the default can be considered problematic.variables in Nuitka package configurations. We can now query at compile time, values from installed packages and use them, e.g. to know what backend is to be used.numba work in some cases, but not all. Some packages go very deep with JIT integration, but simpler uses will now compile.--include-onefile-external-data allows you to specify file patterns that you included by other data files others, but to put those files not inside, but on the outside of the onefile binary. This makes it easier to create deployments fully within Nuitka project configuration, and to change your mind back and forth without adding/removing the data file option.auto for detecting signing identity, if only one is available in the system.--copyright and --trademark information to be in app bundles as well, this was previously Windows only.scoop. Added in 1.9.2 already.--cf-protection to select the control flow protection mode for the GCC compiler and deviate from default values of some environments to less strict values.pipenv to be installed in them to be found.overridden-environment-variables feature to package configuration. These are environment variable changes that only last during the import of that module and are undone later.force-environment-variables feature to package configuration. These are environment variable changes done on module import that are not undone.include-data-dir: |
a=b
c=d
no-auto-follow now applies recursively, i.e. that a top level package can have it, and not every sub-package that uses a package but should not be automatically followed, does have to say this. With this e.g. networkx configuration became simpler, and yet covered automatically older versions as well, and future changes too.os.path.normcase can make filenames not found, so with a few cleanups, for lazy code that wasn’t really using the APIs designed for comparisons and filename suffix testing, this works now better.__compiled__ value has a new attribute containing_dir that allows to find where a module, accelerate executable, a standalone dist folder, a macOS app bundle, or the onefile binary lives in a consistent fashion. This allows esp. better use than sys.argv[0] which points deep into the .app bundle, and can be used cross platform.Optimization
tshape_unknown in the first micro pass. These micro passes are not visible, but basically constitute a full visit of the module tree over and over, until no more optimization is changing it. This can lead to quicker resolution, as that unknown type shape effectively disallowed all optimization for variables and reduce the number of necessary micro passes by one.tshape_unknown and while a lot of optimization looks for value knowledge, and gets by the escaped nature of the value, sometimes, this was seriously inhibiting some of the type based optimization.# Initial the value of "i" is "NUITKA_NINT_UNASSIGNED" in its
# indicator part. The C compiler will remove that assignment
# as it's only checked in the assignment coming up.
i = 0
# Assignment from a constant, produces a value where both the C
# and the object value are value. This is indicated by a value
# of "NUITKA_NINT_BOTH_VALID". The code generation will assign
# both the object member from a prepared value, and the clong
# member to 0.
# For the conditional check, "NUITKA_NINT_CLONG_VALID" will
# always be set, and therefore function will resort to comparing
# that clong member against 9 simply, that will always be very
# fast. Depending on how well the C compiler can tell if an overflow
# can even occur, such that an object might get created, it can even
# optimize that statically. In this case it probably could, but we
# do not rely on that to be fast.
while i < 9: # RICH_COMPARE_LT_CBOOL_NINT_CLONG
# Here, we might change the type of the object. In Python2,
# this can change from ``int`` to ``long``, and our type
# analysis tells us that. We can consider another thing,
# not "NINT", but "NINTLONG" or so, to special case that
# code. We ignore Python2 here, but multiple possible types
# will be an issue, e.g. list or tuple, float or complex.
# So this calls a function, that returns a value of type
# "NINT" (actually it will become an in-place operation
# but lets ignore that too).
# That function is "BINARY_OPERATION_ADD_NINT_NINT_CLONG"(i, 1)
# and it is going to check if the CLONG is valid, add the one,
# and set to result to a new int. It will reset the
# "NUITKA_NINT_OBJECT_VALID" flag, since the object will not be
# bothered to create.
i = i + 1
# Since "NUITKA_INT_OBJECT_VALID" not given, need to create the
# PyObject and return it.
return i
tomllib from standard library for our distutils integration into pyproject based builds.None returns in generators and do it during tree building already, to remove noise.pyenv as well..pyx files when scanning for data files, these are code files too, in this case source files that are definitely unused most of the time.sys.path, avoiding many file system lookups during import scans.triton in torch package in more cases. Added in 1.9.2 already.pytest in knetworkx package in more cases. Added in 1.9.2 already.IPython in distributed package. Added in 1.9.3 already.dask in skimage. Added in 1.9.3 already.triton in the bitsandbytes package. Added in 1.9.3 already.IPython in tf_keras package as well. Added in 1.9.6 already.unittest in mock.mock module. Added in 1.9.7 already.setuptools_scm during compilation when using the tqdm inline copy, this also avoids a warning on Ubuntu. Added in 1.9.7 already.doctest in skimage in their tifffile inline copy as well. Added in 1.9.7 already.h5py.tests with older h5py as well. Added in 1.9.7 already.distributed.utils_test is also considered using pytest.IPython in the pip package.site module for older tensorflow versions too.unittest usages in tensorflow packages.nose in skimage package.nose in networkx package.nose in pywt package.Organizational
%VAR% to {VAR}.{PID} value for process ID, is now making it temporary value for onefile, that was previously a bug.CMD.EXE does expand those values before Nuitka sees them as even with quoting %TEMP% is the current one on the building machine, a recipe for disaster. As some people still use that, and e.g. os.system or subprocess with shell=True will use it too, this is just not sustainable for a good user experience.{VERSION} (program version information given) and {Version} (Nuitka version), and we should clean that up too..gitignore to build folder that just causes these folders to be ignored by git.--experimental=--report-refcounts that we use to show leaks of compiled time objects at program exit, now counts and reports on functions, generator objects and compiled cells as well.yamllint not disabled are errors. These were only output, but didn’t cause the autoformat to error exit yet.:INFO part that shouts, and reserve that for errors and warnings. Also format info messages to make sure they fit into the line.--show-source-changes to accept module pattern to make it easier to only see the ones currently being worked on. To get the old behavior of showing everything, use * as a pattern.~ in data files source path for command line options and expand it properly.clang-format from Visual Code and MSVC for formatting C code, otherwise a system installed one could be used that gives slightly different outputs.--experimental=allow-c-warnings options to make --debug work for some known currently unavoidable warnings.--macos-create-app-bundle imply standalone mode, it’s not working or useful for accelerated mode anyway.make install and can ease debugging with changes made in Python core itself. Added in 1.9.6 already.rstfmt, black and isort versions.manylinux Pythons as a Python flavors as well.Cleanups
--include-onefile-external-data was automatic this time.oneOf types.os.getenv over os.environ.get for readability.clang-format had a hard time with to something more normal.Tests
virtualenv in the distutils test cases. We make more sure to delete them even in case of issues. We disable warnings during Nuitka package installation. The code to execute a case was factored out and became more clear. We now handle errors in execution with stating what case actually failed, this was a bit hard to tell previously. Also do not install Nuitka when a pyproject case is used, since the build tool installs Nuitka itself.Summary
variables and parameters made it unnecessary to have them, and still add compile time variable use and user decisions and information, without them.