Nuitka Blog - Posted in 2020

Nuitka Release 0.6.10

2020-12-22T15:38:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes with many new features, e.g. onefile support, as well as many new optimization and bug fixes.

Bug Fixes

Fix, was memory leaking arguments of all complex call helper functions. Fixed in 0.6.9.6 already.
Plugins: Fix, the dill-compat code needs to follow API change. Fixed in 0.6.9.7 already.
Windows: Fixup for multiprocessing module and complex call helpers that could crash the program. Fixed in 0.6.9.7 already.
Fix, the frame caching could leak memory when using caching for functions and generators used in multiple threads.
Python3: Fix, importing an extension module below a compiled module was not possible in accelerated mode.
Python3: Fix, keyword arguments for open built-in were not fully compatible.
Fix, the scons python check should also not accept directories, otherwise strange misleading error will occur later.
Windows: When Python is installed through a symbolic link, MinGW64 and Scons were having issues, added a workaround to resolve it even on Python2.
Compatibility: Added support for co_freevars in code objects, e.g. newer matplotlib needs this.
Standalone: Add needed data files for gooey. Fixed in 0.6.9.4 already.
Scons: Fix, was not respecting --quiet option when running Scons. Fixed in 0.6.9.3 already.
Scons: Fix, wasn’t automatically detecting Scons from promised paths. Fixed in 0.6.9.2 already.
Scons: Fix, the clcache output parsing wasn’t robust enough. Fixed in 0.6.9.1 already.
Python3.8: Ignore all non-strings provided in doc-string fashion, they are not to be considered.
Fix, getattr, setattr and hasattr could not be used in finally clauses anymore. Fixed in 0.6.9.1 already.
Windows: For Python3 enhanced compatibility for Windows no console mode, they need a sys.stdin or else e.g. input will not be compatible and raise RuntimeError.

New Features

Added experimental support for Python 3.9, in such a way that the CPython3.8 test suite passes now, the 3.9 suite needs investigation still, so we might be missing new features.
Added experimental support for Onefile mode with --onefile that uses AppImage on Linux and our own bootstrap binary on Windows. Other platforms are not supported at this time. With this, the standalone folder is packed into a single binary. The Windows variant currently doesn’t yet do any compression yet, but the Linux one does.
Windows: Added downloading of ccache.exe, esp. as the other sources so far recommended were not working properly after updates. This is taken from the official project and should be good.
Windows: Added downloading of matching MinGW64 C compiler, if no other was found, or that was has the wrong architecture, e.g. 32 bits where we need 64 bits.
Windows: Added ability to copy icon resources from an existing binary with new option --windows-icon-from-exe.
Windows: Added ability to provide multiple icon files for use with different desktop resolutions with new option --windows-icon-from-ico that got renamed to disambiguate from other icon options.
Windows: Added support for requesting UAC admin right with new option --windows-uac-admin.
Windows: Added support for requesting “uiaccess” rights with yet another new option --windows-uac-uiaccess.
Windows: Added ability to specify version info to the binary. New options --windows-company-name, --windows-product-name, --windows-file-version, --windows-product-version, and --windows-file-description have been added. Some of these have defaults.
Enhanced support for using the Win32 compiler of MinGW64, but it’s not perfect yet and not recommended.
Windows: Added support for LTO mode for MSVC as well, this seems to allow more optimization.
Plugins: The numpy plugin now handles matplotlib3 config files correctly.

Optimization

Use less C variables in dictionary created, not one per key/value pair. This improved scalability of C compilation.
Use common code for module variable access, leading to more compact code and enhanced scalability of C compilation.
Use error exit during dictionary creation to release the dictionary, list, tuple, and set in case of an error occurring while they are still under construction. That avoids releases of it in error exists, reducing the generated code size by a lot. This improves scalability of C compilation for generating these.
Annotate no exception raise for local variables of classes with know dict shape, to avoid useless error exits.
Annotate no exception exit for staticmethod and classmethod as they do not check their arguments at all. This makes code generated for classes with these methods much more compact, mainly improving their scalability in C compilation.
In code generation, prefer bool over nuitka_bool which allows to annotate exception result, leading to more compact code. Also cleanup so that code generation always go through the C type objects, rather than doing cases locally, adding a C type for bool.
Use common code for C code handling const None return only, to cases where there is any immutable constant value returned, avoid code generation for this common case. Currently mutable constants are not handled, this may be added in the future.
Annotate no exception for exception type checks in handlers for Python2 and no exception if the value has exception type shape for Python3. The exception type shape was newly added. This avoids useless exception handlers in most cases, where the provided exception is just a built-in exception name.
Improve speed of often used compile time methods on nodes representing constant values, by making their implementation type specific to improve frontend compile time speed, we check e.g. mutable and hashable a lot.
Provide truth value for variable references, enhancing loop optimization and merge value tracing, to also decide this correctly for values only read, and then changed through attribute, e.g. append on lists. This allows many more static optimization.
Use staticmethod for methods in Nuitka nodes to achieve faster frontend compile times where possible.
Use dedicated helper code for calls with single argument, avoiding the need have a call site local C array of size one, just to pass a pointer to it.
Added handling of hash slot, to predict hashable keys for dictionary and sets.
Share more slot provision for built-in type shapes from mixin classes, to get them more universally provided, even for special types, where their consideration is unusual.
Trace “user provided” flag only for constants where it really matters, i.e. for containers and generally potentially large values, but not for every number or boolean value.
Added lowering of bytearray constant values to bytes value iteration, while handling constant values for this optimization with dedicated code for improved frontend compilation speed.
The dict built-in now annotates the dictionary type shape of its result.
The wrapping side-effects node now passes on the type shape of the wrapped value, allowing for optimization of these too.
Split slice nodes into variants with 1, 2 or 3 arguments, to avoid the overhead of determining which case we have, as well as to save a bit of memory, since these are more frequently used on Python3 for subscript operations. Also annotate their type shape, allowing more optimization.
Faster dictionary lookups, esp. in cases where errors occur, because we were manually recreating a KeyError that is already provided by the dict implementation. This should also be faster, as it avoids a CPython API call overhead on the DLL and they can provide a reference or not for the returned value, simplifying using code.
Faster dictionary containment checks, with our own dedicated helper, we can use code that won’t create an exception when an item is not present at all.
Faster hash lookups with our own helper, separating cases where we want an exception for non-hashable values or not. These should also be faster to call.
Avoid acquiring thread state in exception handling that checks if a StopIteration occurred, to improved speed on Python3, where is involves locking, but this needs to be applied way more often.
Make sure checks to debug mode and full compatibility mode are done with the variables introduced, to avoid losing performance due to calls for Nuitka compile time enhancements. This was so far only done partially.
Split constant references into two base classes, only one of them tracking if the value was provided by the user. This saves compile time memory and avoids the overhead to check if sizes are exceeded in cases they cannot possibly be so.
The truth value of container creations is now statically known, because the empty container creation is no longer a possibility for these nodes, allowing more optimization for them.
Optimize the bool built-in with no arguments directory, allow to simplify the node for single argument form to avoid checks if an argument was given.
Added iteration handles for xrange values, and make them faster to create by being tied to the node type, avoiding shared types, instead using the mixin approach. This is in preparation to using them for standard iterator tracing as well. So far they are only used for any and all decision.
Added detection if a iterator next can raise, using existing iterator checking which allows to remove needless checks and exception traces. Adding a code variant for calls to next that cannot fail, while tuning the code used for next and unpacking next, to use faster exception checking in the C code. This will speed up unpacking performance for some forms of unpacking from known sizes.
Make sure to use the fastest tuple API possible in all of Nuitka, many place e.g. used PyTuple_Size, and one was in a performance critical part, e.g. in code that used when compiled functions as called as a method.
Added optimized variant for _PyList_Extend for slightly faster unpacking code.
Added optimized variant for PyList_Append for faster list contractions code.
Avoid using RemoveFileSpec and instead provide our own code for that task, slightly reducing file size and avoiding to use the Shlapi link library.

Tests

Made reflected test use common cleanup of test folder, which is more robust against Windows locking issues.
Only output changed CPython output after the forced update of cached value was done, avoiding duplicate or outdated outputs.
Avoid complaining about exceptions for in-place operations in case they are lowered to non-inplace operations and then raise unsupported, not worth the effort to retain original operator.
Added generated test for subscript operations, also expanding coverage in generated tests by making sure, conditional paths are both taken by varying the cond value.
Use our own code helper to check if an object has an attribute, which is faster, because it avoids creating exceptions in the first place, instead of removing them afterwards.

Cleanups

Make sure that code generation always go through the C type objects rather than local elif casing of the type. This required cleaning up many of the methods and making code more abstract.
Added base class for C types without reference counting, so they can share the code that ignores their handling.
Remove getConstant for constant value nodes, use the more general getCompileTimeConstant instead, and provide quick methods that test for empty tuple or dict, to use for checking concrete values, e.g. with call operations.
Unified container creation into always using a factory function, to be sure that existing container creations are not empty.
Stop using @calledWithBuiltinArgumentNamesDecorator where possible, and instead make explicit wrapping or use correct names. This was used to allow e.g. an argument named list to be passed from built-in optimization, but that can be done in a cleaner fashion. Also aligned no attributes and the argument names, there was inconsistency there.
Name mangling was done differently for attribute names and normal names and with non-shared code, and later than necessary, removing this as a step from variable closure taking after initial tree build.
As part of the icon changes, now handled in Python code, we stop using the rc binary and handle all resources ourselves, allowing to remove that code from the Scons side of things.
Moved file comparison code of standalone mode into file utils function for use in plugins as well.
Unified how path concatenation is done in Nuitka helper code, there were more or less complete variants, this is making sure, the most capable form is used in all cases.
Massive cleanup to our scons file, by moving out util code that only scons uses, hacks we apply to speed up scons, and more to separate modules with dedicated interfaces.
When using enumerate we now provide start value of 1 where it is appropriate, e.g. when counting source code lines, rather than adding count+1 on every usage, making code more readable.

Organizational

Do not recommend Anaconda on Windows anymore, it seems barely possible to get anything installed on it with a fresh download, due to the resolver literally working for days without finishing, and then reporting conflicts, it would only we usable when starting with Miniconda, but that seems less interesting to users, also gcc 5.2 is way too old these days.
The commit hook should be reinstalled, since it got improved and adapted for newer git versions.
Added link to donations to funding document, following a GitHub standard.
Bumped requirements for development to the latest versions, esp. newer isort.
Added a rough description of tests to do to add a new CPython test suite, to allow others to take this task in the future.
Updated the git hook so that Windows and newest git works.
Make it more clear in the documentation that Microsoft Appstore Python is not supported.

Summary

This is the big release in terms of scalability. The optimization in this release mostly focused on getting things that cause increased compile times sorted out. A very important fix avoids loop optimization to leak into global passes of all modules unnecessarily, but just as important, generated code now is much better for the C compiler to consume in observed problematic cases.

More optimization changes are geared towards reducing Nuitka frontend compile time, which could also be a lot in some cases, ending up specializing more constant nodes and how they expose themselves to optimization.

Other optimization came from supporting Python 3.9 and things come across during the implementation of that feature, e.g. to be able to make differences with unpacking error messages, we provide more code to handle it ourselves, and to manually optimize how to interact with e.g. list objects.

For Windows, the automatic download of ccache and a matching MinGW64 if none was found, is a new step, that should lower the barrier of entry for people who have no clue what a C compiler is. More changes are bound to come in this field with future releases, e.g. making a minimum version requirement for gcc on Windows that excludes unfit C compilers.

All in all, this release should be taken as a major cleanup, resolving many technical debts of Nuitka and preparing more optimization to come.

Nuitka Release 0.6.9

2020-10-15T10:34:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains important bug fixes for regressions of the 0.6.8 series which had relatively many problems. Not all of these could be addressed as hotfixes, and other issues were even very involved, causing many changes to be necessary.

There are also many general improvements and performance work for tracing and loops, but the full potential of this will not be unlocked with this release yet.

Bug Fixes

Fix, loop optimization sometimes didn’t determinate, effectively making Nuitka run forever, with no indication why. This has been fixed and a mechanism to give up after too many attempts has been added.
Fix, closure taking object allowed a brief period where the garbage collector was exposed to uninitialized objects. Fixed in 0.6.8.1 already.
Python3.6+: Fix corruption for exceptions thrown into asyncgen. Fixed in 0.6.8.1 already.
Fix, deleting variables detected as C type bool could raise an UnboundLocalError that was wrong. Fixed in 0.6.8.1 already.
Python3.8.3+: Fix, future annotations parsing was using hard coded values that were changed in CPython, leading to errors.
Windows: Avoid encoding issues for Python3 on more systems, by going from wide characters to unicode strings more directly, avoiding an encoding as UTF-8 in the middle. Fixed in 0.6.8.2 already.
Windows: Do not crash when warning about uninstalled MSVC using Python3. This is a Scons bug that we fixed. Fixed in 0.6.8.3 already.
Standalone: The output of dependency walker should be considered as “latin1” rather than UTF-8. Fixed in 0.6.8.3 already.
Standalone: Added missing hidden dependencies for flask. Fixed in 0.6.8.1 already.
Standalone: Fixed win32com.client on Windows. Fixed in 0.6.8.1 already.
Standalone: Use pkgutil to scan encoding modules, properly ignoring the same files as Python does in case of garbage files being there. Fixed in 0.6.8.2 already.
Plugins: Enabling a plugin after the filename to compile was given, didn’t allow for arguments to the passed, causing problems. Fixed in 0.6.8.3 already.
Standalone: The certifi data file is now supported for all modules using it and not only some.
Standalone: The bytecode for the standard library had filenames pointing to the original installation attached. While these were not used, but replaced at run time, they increased the size of the binary, and leaked information.
Standalone: The path of sys.executable was not None, but pointing to the original executable, which could also point to some temporary virtualenv directory and therefore not exist, also it was leaking information about the original install.
Windows: With the MSVC compiler, elimination of duplicate strings was not active, causing even unused strings to be present in the binary, some of which contained file paths of the Nuitka installation.
Standalone: Added support for pyglet.
Plugins: The command line handling for Pmw plugin was using wrong defaults, making it include more code than necessary, and to crash if it was not there.

New Features

Windows: Added support for using Python 2.7 through a symlink too. This was already working for Python3, but a scons problem prevented this from working.
Caching of compiled C files is now checked with ccache and clcache, and added automatically where possible, plus a report of the success is made. This can accelerate the re-compile very much, even if you have to go through Nuitka compilation itself, which is not (yet) cached.
Added new --quiet option that will disable informational traces that are going to become more.
The Clang from MSVC installation is now picked up for both 32 and 64 bits and follows the new location in latest Visual Studio 2019.
Windows: The ccache from Anaconda is now supported as well as the one from msys64.

Optimization

The value tracing has become more correct with loops and in general less often inhibits optimization. Escaping of value traces is now a separate trace state allowing for more appropriate handling of actual unknowns.
Memory used for value tracing has been lowered by removing unnecessary states for traces, that we don’t use anymore.
Windows: Prevent scons from scanning for MSVC when asked to use MinGW64. This avoids a performance loss doing something that will then end up being unused.
Windows: Use function level linking with MSVC, this will allow for smaller binaries to be created, that don’t have to include unused helper functions.

Cleanups

The scons file now uses Nuitka utils functions and is itself split up into several modules for enhanced readability.
Plugin interfaces for providing extra entry points have been cleaned up and now named tuples are used. Backward compatibility is maintained though.

Organizational

The use of the logging module was replaced with more of our custom tracing and we now have the ability to write the optimization log to a separate file.
Old style plugin options are now detected and reported as a usage error rather than unknown plugin.
Changed submodules to use git over https, so as to not require ssh which requires a key registered and causes problems with firewalls too.
More correct Debian copyright file, made formatting of emails in source code consistent.
Added repository for Ubuntu focal.

Summary

The main focus of this release has been bug fixes with only a little performance work due to the large amount of regressions and other findings from the last release.

The new constants loading for removes a major scalability problem. The checked and now consistently possible use of ccache and clcache allows for much quicker recompilation. Nuitka itself can still be slow in some cases, but should have seen some improvements too. Scalability will have to remain a focus for the next releases too.

The other focus, was to make the binaries contain no original path location, which is interesting for standalone mode. Nuitka should be very good in this area now.

For optimization, the new loop code is again better. But it was also very time consuming, to redo it, yet again. This has prevented other optimization to be added.

And then for correctness, the locals scope work, while very invasive, was necessary, to handle the usage of locals inside of contractions, but also will be instrumental for function inlining to become generally available.

So, ultimately, this release is a necessary intermediate step. Upcoming releases will be able to focus more clearly on run time performance again as well as on scalability for generated C code.

Nuitka Release 0.6.8

2020-05-18T09:11:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains important general improvements and performance improvements and enhanced optimization as well as many bug fixes that enhance the Python 3.8 compatibility.

Bug Fixes

Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.
Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.
Fix, the fromlist of imports could loose references, potentially leading to corruption of contained strings.
Python3.8: Fix, positional only arguments were not enforced to actually be that way.
Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.
Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.
Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle Unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.
Windows: Fixup dependency walker in unicode input directories.
Standalone: Use frozen module loader only at libpython initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility.
Standalone: Fix for pydantic support.
Standalone: Added missing hidden dependency of uvicorn.
Fix, the parser for .pyi files couldn’t handle multiline imports.
Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.
Fixup static linking with libpython.a that contains main.o by making our colliding symbols for Py_GetArgcArgv weak.
Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.
Distutils: Fix build_nuitka for modules under nested namespaces.
OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.
macOS: Fixup for standalone mode library scan.
Fix, the logging of --show-modules was broken.
Windows: Enable /bigobj mode for MSVC for large compilations to work.
Windows: Fixup crash in warning with pefile dependency manager.
Windows: Fixup win32com standalone detection of other Python version win32com is in system PATH.
Fix, the python flag for static hashes didn’t have the intended effect.
Fix, generators may be resurrected in the cause of their destruction, and then must not be released.
Fix, method objects didn’t implement the methods __reduce__ and __reduce_ex__ necessary for pickling them.
Windows: Fix, using a Python installation through a symlink was not working.
Windows: Fix, icon paths that were relative were not working anymore.
Python3.8: Detect duplicate keywords yielded from star arguments.
Fix, methods could not be pickled.
Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.
Fix, frames need to traverse their attached locals to be released in some cases.

New Features

Plugin command line handling now allows for proper optparse options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.

Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.
Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.
Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.
Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.
Fix, the setattr built-in was leaking a reference to the None value.

Optimization

Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn’t see through iterators yet, but this will come too.
Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.
Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.
For in-place operations, there is the infrastructure to generate them for improved performance, but so far it’s only used for Python2 int, and not for the many types normal operations are supported.
Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.
Complex constants didn’t annotate their type shape, preventing compile time optimization for them.
Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.
Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.
Avoid using a preparing a constant tuple value for calls with only constant arguments.
Avoid using PyErr_Format where it’s not necessary by adding specialized helpers for common cases.
Detect del statements that will raise an exception and replace with that.
Exception matching is boolean shape, allowing for faster code generation.
Disable recursion checks outside of full compat mode.
Avoid large blocks for conditional statements that only need to enclose the condition evaluation.
Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.

Organizational

Updated Developer Manual with changes that happened, removing the obsolete language choice section.
Added 3.8 support mentions in even more places.
The mailing list has been deleted. We now prefer Gitter chat and GitHub issues for discussions.
Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.
Visual Code environments for Py38 and Py27 were added for easier switch.
Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.
Make it clear that --full-compat should not be used in help output.
Added instructions for MSVC runtimes and standalone compilation to support Windows 7.
More complete listing of copyright holders for Debian.
Updated to newer black and PyLint.
Enhanced gcc version check, properly works with gcc 10 and higher.

Tests

Pylint cleanups for some of the tests.
Added test for loading of user plugins.
Removed useless outputs for search mode skipping non-matches.

Cleanups

Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.
Pylint cleanup also foreign code like oset and odict.
In preparation of deprecating the alternative, --enable-plugin has become the only form used in documentation and tests.
Avoid numeric pylint symbols more often.
Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.
The “cannot get here” code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.
Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.

Summary

This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.

Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float or C long can be added, add to that iterator type shape and value analsis, and an actual jump in performance can be expected.

Nuitka Release 0.6.7

2020-01-25T08:42:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains bug fixes and improvements to the packaging, for the RPM side as well as for Debian, to cover Python3 only systems as they are now becoming more common.

Bug Fixes

Compatibility: The value of __module__ for extension modules was not dependent into which package the module was loaded, it now is.
Anaconda: Enhanced detection of Anaconda for Python 3.6 and higher.
CentOS6: Detect gcc version to allow saving on macro memory usage, very old gcc didn’t have that.
Include Python3 for all Fedora versions where it works as well as for openSUSE versions 15 and higher.
Windows: Using short path names to interact with Scons avoids problems with Unicode paths in all cases.
macOS: The usage of install_name_tool could sometimes fail due to length limits, we now increase it at link time.
macOS: Do not link against libpython for module mode. This prevented extension modules from actually being usable.
Python3.6: Follow coroutine fixes in our asyncgen implementation as well.
Fix, our version number handling could overflow with minor versions past 10, so we limited it for now.

New Features

Added support for Python 3.8, the experimental was already there and pretty good, but now added the last obscure features too.
Plugins can now provide C code to be included in the compilation.
Distutils: Added targets build_nuitka and install_nuitka to complement bdist_nuitka, so we support software other than wheels, e.g. RPM packaging that compiles with Nuitka.
Added support for lldb the Clang debugger with the --debugger mode.

Optimization

Make the file prefix map actually work for gcc and clang, and compile files inside the build folder, unless we are running in debugger mode, so we use ccache caching across different compilations for at least the static parts.
Avoid compilation of __frozen.c in accelerated mode, it’s not used.
Prefer using the inline copy of scons over systems scons. The later will only be slower. Use the fallback to external scons only from the Debian packages, since there we consider it forbidden to include software as a duplicate.

Organizational

Added recommended plugins for Visual Code, replacing the list in the Developer Manual.
Added repository for Fedora 30 for download.
Added repository for CentOS 8 for download.
Updated inline copy of Scons used for Python3 to 3.1.2, which is said to be faster for large compilations.
Removed Eclipse setup from the manual, it’s only infererior at this point and we do not use it ourselves.
Debian: Stop recommending PyQt5 in the package, we no longer use it for built-in GUI that was removed.
Debian: Bumped the standards version and modernized the packaging, solving a few warnings during the build.

Cleanups

Scons: Avoid to add Unix only include paths on Windows.
Scons: Have the static source code in a dedicated folder for clarity.

Tests

Added tests to GitHub Actions, for the supported Python versions for all of Linux, macOS and Windows, covering the later publicly for the first time. We use Anaconda on macOS for the tests now, rather than Homebrew.
Enable IO encoding to make sure we use UTF-8 for more test suites that actually need it in case of problems.
Comparing module outputs now handles segfaults by running in the debugger too.

Summary

This release adds full support for Python 3.8 finally, which took us a while, and it cleans up a lot on the packaging side. There aren’t that many important bug fixes, but it’s still nice to this cleaned up.

We have important actual optimization in the pipeline that will apply specialization to target types and for comparison operations. We expect to see actual performance improvements in the next release again.

Nuitka Release 0.6.6

2020-01-06T08:34:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains huge amounts of crucial bug fixes all across the board. There is also new optimization and many organizational improvements.

Bug Fixes

Fix, the top level module must not be bytecode. Otherwise we end up violating the requirement for an entry point on the C level.
Fix, avoid optimizing calls with default values used. This is not yet working and needed to be disabled for now.
Python3: Fix, missing keyword only arguments were not enforced to be provided keyword only, and were not giving the compatible error message when missing.
Windows: Find win32com DLLs too, even if they live in subfolders of site-packages, and otherwise not found. They are used by other DLLs that are found.
Standalone: Fixup for problem with standard library module in most recent Anaconda versions.
Scons: Fix, was using CXXFLAGS and CPPFLAGS even for the C compiler, which is wrong, and could lead to compilation errors.
Windows: Make --clang limited to clang-cl.exe as using it inside a MinGW64 is not currently supported.
Standalone: Added support for using lib2to2.pgen.
Standalone: Added paths used by openSUSE to the Tcl/Tk plugin.
Python3.6+: Fix, the __main__ package was None, but should be "" which allows relative imports from itself.
Python2: Fix, compile time optimization of floor division was using normal division.
Python3: Fix, some run time operations with known type shapes, were falsely reporting error message with unicode or long, which is of course not compatible.
Fix, was caching parent package, but these could be replaced e.g. due to bytecode demotion later, causing crashes during their optimization.
Fix, the value of __compiled__ could be corrupted when being deleted, which some modules wrappers do.
Fix, the value of __package__ could be corrupted when being deleted.
Scons: Make sure we can always output the compiler output, even if it has a broken encoding. This should resolve MSVC issues on non-English systems, e.g. German or Chinese.
Standalone: Support for newest sklearn was added.
macOS: Added resolver for run time variables in otool output, that gets PyQt5 to work on it again.
Fix, floor division of run time calculations with float values should not result in int, but float values instead.
Standalone: Enhanced support for boto3 data files.
Standalone: Added support for osgeo and gdal.
Windows: Fix, there were issues with spurious errors attaching the constants blob to the binary due to incorrect C types provided.
Distutils: Fix, need to allow / as separator for package names too.
Python3.6+: Fix reference losses in asyncgen when throwing exceptions into them.
Standalone: Added support for dill.
Standalone: Added support for scikit-image and skimage.
Standalone: Added support for weasyprint.
Standalone: Added support for dask.
Standalone: Added support for pendulum.
Standalone: Added support for pytz and pytzdata.
Fix, --python-flags=no_docstrings no longer implies disabling the assertions.

New Features

Added experimental support for Python 3.8, there is only very few things missing for full support.
Distutils: Added support for packages that are in a namespace and not just top level.
Distutils: Added support for single modules, not only packages, by supporting py_modules as well.
Distutils: Added support for distinct namespaces.
Windows: Compare Python and C compiler architecture for MSVC too, and catch the most common user error of mixing 32 and 64 bits.
Scons: Output variables used from the outside, so the debugging is easier.
Windows: Detect if clang installed inside MSVC automatically and use it if requested via --clang option. This is only the 32 bits variant, but currently the easy way to use it on Windows with Nuitka.

Optimization

Loop variables were analysed, but results were only available on the inside of the loop, preventing many optimization in these cases.
Added optimization for the abs built-in, which is also a numerical operator.
Added optimization for the all built-in, adding a new concept of iteration handle, for efficient checking that avoids looking at very large sequences, of which properties can still be known.
```
all(range(1, 100000))  # no need to look at all of them
```
Added support for optimizing ImportError construction with keyword-only arguments. Previously only used without these were optimized.
```
raise ImportError(path="lala", name="lele")  # now optimized
```
Added manual specialization for single argument calls, solving a TODO, as these will be very frequent.
Memory: Use single child form of node class where possible, the general class now raises an error if used with used with only one child name, this will use less memory at compile time.
Memory: Avoid list for non-local declarations in every function, these are very rare, only have it if absolutely necessary.
Generate more compact code for potential NameError exceptions being raised. These are very frequent, so this improves scalability with large files.
Python2: Annotate comparison of None with int and str types as not raising an exception.
Shared empty body functions and generators.

One shared implementation for all empty functions removes that burden from the C compiler, and from the CPU instruction cache. All the shared C code does is to release its arguments, or to return an empty generator function in case of generator.
Memory: Added support for automatic releases of parameter variables from the node tree. These are normally released in a try finally block, however, this is now handled during code generation for much more compact C code generated.
Added specialization for int and long operations %, <<, >>, |, &, ^, **, @.
Added dedicated nodes for representing and optimizing based on shapes for all binary operations.
Disable gcc macro tracing unless in debug mode, to save memory during the C compilation.
Restored Python2 fast path for int with unknown object types, restoring performance for these.

Cleanups

Use dedicated ModuleName type that makes the tests that check if a given module name is inside a namespace as methods. This was hard to get right and as a result, adopting this fixed a few bugs and or inconsistent results.
Expand the use of nuitka.PostProcessing to cover all actions needed to get a runnable binary. This includes using install_name_tool on macOS standalone, as well copying the Python DLL for acceleration mode, cleaning the x bit for module mode. Previously only a part of these lived there.
Avoid including the definitions of dynamically created helper functions in the C code, instead just statically declare the ones expected to be there. This resolves Visual Code complaining about it, and should make life also easier for the compiler and caches like ccache.
Create more helper code in closer form to what clang-format does, so they are easier to compare to the static forms. We often create hard coded variants for few arguments of call functions, and generate them for many argument variations.
Moved setter/getter methods for Nuitka nodes consistently to the start of the node class definitions.
Generate C code much closer to what clang-format would change it to be.
Unified calling install_name_tool on macOS into one function that takes care of all the things, including e.g. making the file writable.
Debug output from scons should be more consistent and complete now.
Sort files for compilation in scons for better reproducible results.
Create code objects version independent, avoiding python version checks by pre-processor, hiding new stuff behind macros, that ignore things on older Python versions.

Tests

Added many more built-in tests for increased coverage of the newly covered ones, some of them being generic tests that allow to test all built-ins with typical uses.
Many tests have become more PyLint clean as a result of work with Visual Code and it complaining about them.
Added test to check PyPI health of top 50 packages. This is a major GSoC 2019 result.
Output the standalone directory contents for Windows too in case of a failure.
Added generated tests to fully cover operations on different type shapes and their errors as well as results for typical values.
Added support for testing against installed version of Nuitka.
Cleanup up tests, merging those for only Python 3.2 with 3.3 as we no longer support that version anyway.
Execute the Python3 tests for macOS on Travis too.

Organizational

The donation sponsored machine called donatix had to be replaced due to hardware breakage. It was replaced with a Raspberry-Pi 4.
Enhanced plugin documentation.
Added description of the git workflow to the Developer Manual.
Added checker script check-nuitka-with-codespell that reports typos in the source code for easier use of codespell with Nuitka.
Use newest PyLint and clang-format.
Also check plugin documentation files for ReST errors.
Much enhanced support for Visual Code configuration.
Trigger module code is now written into the build directory in debug mode, to aid debugging.
Added deep check function that descends into tuples to check their elements too.

Summary

This release comes after a long time of 4 months without a release, and has accumulated massive amounts of changes. The work on CPython 3.8 is not yet complete, and the performance work has yet to show actual fruit, but has also progressed on all fronts. Connecting the dots and pieces seems not far away.