Nuitka Blog - Posted in 2022

Nuitka Release 1.3

2022-12-28T09:58:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of performance work, that should specifically be useful on Windows, but also generally. A bit of scalability work has been applied, and as usual many bug fixes and small improvements, many of which have been in hotfixes.

Bug Fixes

macOS: Framework build of PySide6 were not properly supporting the use of WebEngine. This requires including frameworks and resources in new ways, and actually some duplication of files, making the bundle big, but this seems to be unavoidable to keep the signature intact.
Standalone: Added workaround for dotenv. Do not insist on compiled package directories that may not be there in case of no data files. Fixed in 1.2.1 already.
Python3.8+: Fix, the ctypes.CDLL node attributes the winmode argument to Python2, which is wrong, it was actually added with 3.8. Fixed in 1.2.1 already.
Windows: Attempt to detect corrupt object file in MSVC linking. These might be produced by cl.exe crashes or clcache bugs. When these are reported by the linker, it now suggests to use the --clean-cache=ccache which will remove it, otherwise there would be no way to cure it. Added in 1.2.1 already.
Standalone: Added data files for folium package. Added in 1.2.1 already.
Standalone: Added data files for branca package. Added in 1.2.1 already.
Fix, some forms try that had exiting finally branches were tracing values only assigned in the try block incorrectly. Fixed in 1.2.2 already.
Alpine: Fix, Also include libstdc++ for Alpine to not use the system one which is required by its other binaries, much like we already do for Anaconda. Fixed in 1.2.2 already.
Standalone: Added support for latest pytorch. One of our workarounds no longer applies. Fixed in 1.2.2 already.
Standalone: Added support for webcam on Windows with opencv-python. Fixed in 1.2.3 already.
Standalone: Added support for pytorch_lightning, it was not finding metadata for rich package. Fixed in 1.2.4 already.

For the release we found that pytorch_lightning may not find rich installed. Need to guard version() checks in our package configuration.
Standalone: Added data files for dash package. Fixed in 1.2.4 already.
Windows: Retry replace clcache entry after a delay, this works around Virus scanners giving access denied while they are checking the file. Naturally you ought to disable those for your build space, but new users often don’t have this. Fixed in 1.2.4 already.
Standalone: Added support for scipy 1.9.2 changes. Fixed in 1.2.4 already.
Catch corrupt object file outputs from gcc as well and suggest to clean cache as well. This has been observed to happen at least on Windows and should help resolve the ccache situation there.
Windows: In case clcache fails to acquire the global lock, simply ignore that. This happens sporadically and barely is a real locking issue, since that would require two compilations at the same time and for that it largely works.
Compatibility: Classes should have the f_locals set to the actual mapping used in their frame. This makes Nuitka usable with the multidispatch package which tries to find methods there while the class is building.
Anaconda: Fix, newer Anaconda versions have TCL and Tk in new places, breaking the tk-inter automatic detection. This was fixed in 1.2.6 already.
Windows 7: Fix, onefile was not working anymore, a new API usage was not done in a compatible fashion. Fixed in 1.2.6 already.
Standalone: Added data files for lark package. Fixed in 1.2.6 already.
Fix, pkgutil.iter_modules without arguments was given wrong compiled package names. Fixed in 1.2.6 already.
Standalone: Added support for newer clr DLLs changes. Fixed in 1.2.7 already.
Standalone: Added workarounds for tensorflow.compat namespace not being available. Fixed in 1.2.7 already.
Standalone: Added support for tkextrafont. Fixed in 1.2.7 already.
Python3: Fix, locals dict test code testing if a variable was present in a mapping could leak references. Fixed in 1.2.7 already.
Standalone: Added support for timm package. Fixed in 1.2.7 already.
Plugins: Add tls to list of sensible plugins. This enables at least pyqt6 plugin to do networking with SSL encryption.
Standalone: Added implicit dependencies of sklearn.cluster.
FreeBSD: Fix, fcopyfile is no longer available on newest OS version, and include files for sendfile have changed.
MSYS2: Add back support for MSYS Posix variant. Now onefile works there too.
Fix, when picking up data files from command line and plugins, different exclusions were applied. This has been unified to get better coverage for avoiding to include DLLs and the like as data files. DLLs are not data files and must be dealt with differently after all.

New Features

UI: Added new option for cache disabling --disable-cache that accepts all and cache names like ccache, bytecode and on Windows, dll-dependencies with selective values.

Note

The clcache is implied in ccache for simplicity.
UI: With the same values as --disable-cache Nuitka may now also be called with --clean-cache in a compilation or without a filename argument, and then it will erase those caches current data before making a compilation.
macOS: Added --macos-app-mode option for application bundles that should run in the background (background) or are only a UI element (ui-element).
Plugins: In the Nuitka Package Configuration files, the when allows now to check if a plugin is active. This allowed us to limit console warnings to only packages whose plugin was activated.
Plugins: Can now mark a plugin as a GUI toolkit responsible with the consequence that other toolkit detector plugins are all disabled, so when using tk-inter no longer will you be asked about PySide6 plugin, as that is not what you are using apparently.
Plugins: Generalized the GUI toolkit detection to include tk-inter as well, so it will now point out that wx and the Qt bindings should be removed for best results, if they are included in the compilation.
Plugins: Added ability to provide data files for macOS Resources folder of application bundles.
macOS: Fix, Qt WebEngine was not working for framework using Python builds, like the ones from PyPI. This adds support for both PySide2 and PySide6 to distribute those as well.
MSYS2: When asking a CPython installation to compress from the POSIX Python, it crashed on the main filename being not the same.
Scons: Fix, need to preserve environment attached modes when switching to WinLibs gcc on Windows. This was observed with MSYS2, but might have effects in other cases too.

Optimization

Python3.10+: When creating dictionaries, lists, and tuples, we use the newly exposed dictionary free list. This can speedup code that repeatedly allocates and releases dictionaries by a lot.
Python3.6+: Added fast path to dictionary copy. Compact dictionaries have their keys and values copied directly. This is inspired by a Python3.10 change, but it is applicable to older Python as well, and so we did.
Python3.9+: Faster compiled object creation, esp. on Python platforms that use a DLLs for libpython, which is a given on Windows. This makes up for core changes that went unnoticed so far and should regain relative speedups to standard Python.
Python3.10+: Faster float operations, we use the newly exposed float free list. This can speed up all kinds of float operations that are not doable in-place by a lot.
Python3.8+: On Windows, faster object tracking is now available, this previously had to go through a DLL call, that is now removed in this way as it was for non-Windows only so far.
Python3.7+: On non-Windows, faster object tracking is now used, this was regressed when adding support for this version, becoming equally bad as all of Windows at the time. However, we now managed to restore it.
Optimization: Faster deep copy of mutable tuples and list constants, these were already faster, but e.g. went up from 137% gain factor to 201% on Python3.10 as a result. We now use guided a deep copy, which then has the information, what types it is going to copy, removing the need to check through a dictionary.
Optimization: Also have own allocator function for fixed size objects. This accelerates allocation of compiled cells, dictionaries, some iterators, and lists objects.
More efficient code for object initialization, avoiding one DLL calls to set up our compiled objects.
Have our own PyObject_Size variant, that will be slightly faster and avoids DLL usage for len and size hints, e.g. in container creations.
Avoid using non-optimal malloc related macros and functions of Python, and instead of the fasted form generally. This avoids Python DLL calls that on Windows can be particularly slow.
Scalability: Generated child mixins are now used for the generated package metadata hard import nodes calls, and for all instances of single child tuple containers. These are more efficient for creation and traversal of the tree, directly improving the Python compile time.
Scalability: Slightly more efficient compile time constant property detections. For frozenset there was not need to check for hashable values, and some branches could be replaced with e.g. defining our own EllipsisType for use in short paths.
Windows: When using MSVC and LTO, the linking stage was done with only one thread, we now use the proper options to use all cores. This is controlled by --jobs much like C compilation already is. For large programs this will give big savings in overall execution time. Added in 1.2.7 already.
Anti-Bloat: Remove the use of pytest for dash package compilation.
Anti-Bloat: Remove the use of IPython for dotenv, pyvista, python_utils, and trimesh package compilation.
Anti-Bloat: Remove IPython usage in rdkit improving compile time for standalone by a lot. Fixed in 1.2.7 already.
Anti-Bloat: Avoid keras testing framework when using that package.

Organizational

Plugins: The numpy plugin functionality was moved to Nuitka package configuration, and as a result, the plugin is now deprecated and devoid of functionality. On non-Windows, this removes unused duplications of the numpy.core DLLs.
User Manual: Added information about macOS entitlements and Windows console. These features are supported very well by Nuitka, but needed documentation.
UI: Remove alternative options from --help output. These are there often only for historic reasons, e.g. when an option was renamed. They should not bother users reading them.
Plugins: Expose the mnemonics option to plugin warnings function, and use it for pyside2 and pyqt5 plugins.
Quality: Detect trailing/leading spaces in Nuitka package configuration description values during their automatic check.
UI: Detect the CPython official flavor on Windows by comparing to registry paths and consider real prefixes, when being used in virtualenv more often, e.g. when checking for CPython on Apple.
UI: Enhanced --version output to include the C compiler selection. It is doing that respecting your other options, e.g. --clang, etc. so it will be helpful in debugging setup issues.

UI: Some error messages were using %r rather than '%s' to output file paths, but that escaped backslashes on Windows, making them look worse, so we changed away from this.
UI: Document more clearly what --output-dir actually controls.
macOS: Added options hint that the Foundation module requires bundle mode to be usable.
UI: Allow using both --follow-imports and --nofollow-imports on command line rather than erroring out. Simply use what was given last, this allows overriding what was given in project options tests should the need arise.
Reports: Include plugin reasons for pre and post load modules provided. This solves a TODO and makes it easier to debug plugins.
UI: Handle --include-package-data before compilation, removing the ability to use pattern. This makes it easier to recognize mistakes without a long compilation and plugins can know them this way too.
GitHub: Migration workflows to using newer actions for Python and checkout. Also use newer Ubuntu LTS for Linux test runner.
UI: Catch user error of running Nuitka with the pythonw binary on Windows.
UI: Make it clear that MSYS2 defaults to --mingw64 mode. It had been like this, but the --help output didn’t say so.
GitHub: Updated contribution guidelines for better readability.
GitHub: Use organisation URLs everywhere, some were still pointing to the personal rather than the organisation one. While these are redirected, it is not good to have them like this.
Mastodon: Added link to https://fosstodon.org/@kayhayen to the PyPI package and User Manual.

Cleanups

Nodes for hard import calls of package meta data now have their base classes fully automatically created, replacing what was previously manual code. This aims at making them more consistent and easier to add.
When adding the new Scons file for C compiler version output, more values that are needed for both onefile and backend compilation were moved to centralized code, simplifying these somewhat again.
Remove unused main_module tag. It cannot happen that a module name matches, and still thinks of itself as __main__ during compilation, so that idea was unnecessary.
Generate the dictionary copy variants from template code rather than having manual duplications. For dict.copy(), for deep copy (needed e.g. when there are escaping mutable keyword argument constant values in say a function call), and for **kw value preparation in the called function (checking argument types), we have had diverged copies, that are now unified in a single Jinja2 template for optimization.
Plugins: Also allow providing generators for providing extra DLLs much like we already do for data files.
Naming of basic tests now makes sure to use a Test suffix, so in Visual Code selector they are more distinct from Nuitka code modules.
Rather than populating empty dictionaries, helper code now uses factory functions to create them, passing keys and values, and allowing values to be optional, removing noisy if branches at call side.
Removed remaining PyDev annotations, we don’t need those anymore for a long time already.
Cleanup, avoid lists objects for ctypes defined functions and their arglist, actually tuples are sufficient and naturally better to use.
Spelling cleanups were resumed, as an ongoing action.

Tests

Added construct test that demonstrates the mutable constant argument passing for lists to see the performance gains in this area too.
Made construct runner --diff output usable for interactive usage.
Repaired Nuitka Speedcenter, but it’s not yet too usable for general consumption. More work will be needed there, esp. to make comparisons more accessible for the general public.

Summary

The major achievement of this release was the removal of the long lived numpy plug-in, replacing it with package based configuration, that is even more optimal and works perfectly across all platforms on both important package installation flavors.

This release has a lot of consolidation efforts, but also as a result of 3.11 investigations, addresses a lot of issues, that have crept in over time with Python3 releases since 3.7, each time, something had not been noticed. There will more need for investigation of relative performance losses, but this should address the most crucial ones, and also takes advantage of optimization that had become with 3.10 already.

There is also some initial results from cleanups with the composite node tree structure, and how it is managed. Generated “child(ren) having” mixins, allow for faster traversal of the node tree.

Some technical things also improved in Scons. Using multiple cores in LTO with MSVC with help this a lot, although for big compilations --lto=no probably has to be recommended still.

More anti-bloat work on more packages rounds up the work.

For macOS specifically, the WebEngine support is crucial to some users, and the new --macos-app-mode with more GUI friendly default resolves long standing problems in this area.

And for MSYS2 and FreeBSD, support has been re-activated, so now 4 OSes work extremely well (others too likely), and on those, most Python flavors work well.

The performance and scalability improvements are going to be crucial. It’s a pity that 3.11 is not yet supported, but we will be getting there.

Python 3.11 and Nuitka

2022-12-10T11:29:24+01:00

In my all in with Nuitka post, I promised to give you an update on the current status of Python 3.11 support and Nuitka, and this is the report on my busy bee activities there.

What is now

Current Nuitka 1.2 and likely the coming 1.3 release, will react pretty hefty to attempts to compile with Python3.11, so this command:

python3.11 -m nuitka tests/basics/Asserts.py

Gives the following output:

Nuitka:WARNING: The version '3.11' is not currently supported. Expect
Nuitka:WARNING: problems.
FATAL: The Python version '3.11' is not supported by '1.2.7', but an upcoming release will add it.

What you can do?

As a mere mortal, what you can and should do is to consider becoming a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All essential packaging and performance features are free, and I have put incredible amounts of works in this, and I need to now make a living off it.

And working on 3.11 support, just so tings then continue to work, mostly on my own deserves financial support. Once a year, for weeks on end, I am re-implementing features that a collective of developers came up with.

Otherwise, unless you are CPython core developer responsible for the changes, I am afraid, not a lot. Having to know both Nuitka and CPython details is a tall ask. I might be the only one to do this for a while. I will however strive to get more people involved each time.

That is not to say this does not happen. Not too long ago, I merged proposed changes that will help Nuitka with 3.12 compatibility. This would be nice if it was more frequent, and the fact that it is not, is probably my fault entirely.

What was done?

In the summer, with release candidates, there was some refactoring done that aimed at making Nuitka compile, at least with less errors. One of the changes that was addressed at the time was the reduction of exception state to only a value.

In Python during an exception, you typically have a exception “type”, “value”, and a “traceback”. And these used to be sort of independent objects, which then through a process called “normalization” got aligned. E.g. when raising an exception, you would simply put a ZeroDivisionError as a type, and a string as the value, and only on normalization, which was attempted to be done as late as possible, the actual exception object was created.

For 3.11 now, in some parts (but not all of course), this got reduced, and making Nuitka cope with that was a bigger task, and even one that had then introduced a regression to pre-3.11, i.e. affecting other versions. I had to make a hotfix because of this for the release that included these changes. This is of course, due to also cleaning up the code that was changed, to reduce e.g. duplications that were considered harmless in the past.

Now, during the 1.3 releases cycle, many more changes have been addressed. The CPython core is using now “interpreter frames”, which are lightweight non-objects, and does only create a frame when an exception is needed. Nuitka on the other hand used to cache frame objects, that it also created only when needed. Adapting the interactions with the frame stack appears somewhat solved, but actually poses some bugs when dealing with generators that raise exceptions, and is currently being analysed.

The Process

When the goal is getting 3.11 supported (or any later Python, this will repeat forever), in a first step it just doesn’t compile on the C level anymore. First, all kinds of changes need to be addressed that show when compiling on the C level. Or extra time spent to disable some things to migrate only later (which I tend to not do), but the unfortunate truth is, that at that point, running with 3.11 to find out things, is not yet an option.

Once compiled 3.11 started being executable, it was hard crashing (worst case) and asserting (good case) like hell. Many assumptions made proved to be untrue, or things that used to be true, are no more true. That is pretty normal when you mingle with internal details of course.

Right now reference leaks are being seen with generators, and so are crashes. This is probably the same issue, where some link between objects, some reference to an object is taken, and what is most likely, in other instance, it is lost, which then causes corrupted objects. These are among the hardest things to debug in Python, often the symptom only shows later. Many times development in this phase is about adding assertions. This is where we are now.

The next phase then is to run the CPython 3.10 test suite compiled with 3.11 and compare its results to running with plain 3.11, and expect it to be the same. It is normal for CPython 3.10 tests to fail with 3.11, and this does two things.

Either the test passes where it should not, then 3.11 added extra checks, e.g. making it illegal to provide certain types of inputs that were legal in the past, or it fails where 3.11 does not, then it highlights a behavior change or a mistake in the migration. Another case of course is it crashing. Then some bug in e.g. frame handling or exception handling might cause that.

During this phase it is esp. initially very hard to assess where we stand. Often one bug affects many tests with different symptoms. We we just look for things that are easy to analyse.

In the next phase, the 3.11 test suite is used in the same way. This will often uncover new features, new behaviors, newly allowed things. For example Nuitka might reject an input, but that is correct for 3.10, but not for 3.11 anymore, this this will be added.

This phase has not even started.

One interesting case is new test coverage. Often is has happened in the past that something was incompatible with 3.10 already, but that has never been seen. Of course, in this instance, the modification will be backported, this is often revealed when running the 3.11 tests compiled with 3.11 and comparing the result with pure 3.10.

And this is also not even started. Once the test suite is adapted for Nuitka’s needs (avoiding random outputs, to make tests comparable e.g.) this could be started, and would enhance 3.10 compatibility already with its findings.

Intermediate Results

Once the Python3.11 more or less well passed the 3.10 test suite without big issues, experimental support for it will be added. The warning from above will be given, but the error will cease.

But only once the CPython 3.11 test suite is completely passing, it will be added as supported officially. Again, not all tests have to pass perfectly for this, cosmetic things are not counted of course.

When

Very hard to predict. Supporting existing Nuitka is also a side tracking thing, that makes it unclear how much time I will have for it. But the worst things with debugging is that I just never know how much time it will be. In past releases, the time has varied. I do not know at this time, what will have to be done for support of 3.11, it clearly is one of the harder ones. But I am hoping these core changes were the bulk of the work.

Benefits for older Python too

When I noticed that the dictionary implementation had heavily changed, and Nuitka’s “illegal” friendship (accessing internal details of it) caused it to not compile, step one was to compare current details with 3.10 and find that there are a bunch of things that were added over time. For example one fast branch that was added to 3.10, Nuitka now has it too, for 3.6 or higher. So we are also back-porting some speed improvements.

Also the allocator of 3.11 changed, and as a result, the allocator for all older Python will be way faster to use, esp. with platforms that use DLLs.

There are other benefits, e.g. 3.10 is exposing free lists (re-usable objects without going through allocation) it is using, allowing Nuitka to accelerate dictionary object creation esp. on Windows, and so on.

Expected results

People tend to expect that gains from Nuitka and enhancements of CPython stack up. The truth of the matter is, no they do not. CPython is applying some tricks that Nuitka already did. Not using its bytecode will then become less of a benefit, but that’s OK, this is not what Nuitka is about.

We need to get somewhere else entirely anyway, in terms of speed up. I will be talking about PGO and C types a lot in the coming year, that is at least the hope. That is where the “all in” part kicks in.

Final Words

Look ma, I posted about something that is not complete. I am getting better.

All in with Nuitka

2022-12-09T08:22:42+01:00

After more than 10 years of working on the Python compiler Nuitka, and at the age of 50, I have finally decided to take the plunge. My projects “Nuitka” and “Nuitka commercial” are now my day job, and I quit my work in the Air Traffic Community, after working on ARTAS (a software to create an Air Situation Picture from radar and other sensors) for 20 years.

Why now?

That is very good question. I feel I could have done this a long time ago, and my heart was torn for a while, and also I prepared this for a while now. Listing back to interviews from 4 years ago, it seems though as if it had not been on the radar (pun intended).

So then I launched Nuitka commercial around 1.5 years ago, first as vapor ware, with early bird customers, and meanwhile it has matured to a point, where it is the obvious solution to many corporate users, with big guys picking it up and small business as well. From startups, big name technology giants, universities, mid sized companies, and from individuals, I have all kinds of support.

I liked my old job very much, but it was getting old, and I felt like Nuitka needs me to fulfil my mission. And my wife needed my help with driving her to work, which was getting the load to high, something had to give. And it seems nothing related to what is called Nuitka (my wife, and the compiler share the name) was going to be that.

What will change?

Already, ever since I launched Nuitka commercial, Nuitka has become way more of a out of the box experience. It just works, and covers more packages than ever before, and steps have been taken, to make it possible to extend this even further with your help. Sure there are still shortcoming.

But Nuitka has become very polished, and shows this very much that I also love details and doing things correctly. I have conducted Nuitka from day one as an attempt to create a software that would be usable to the enterprise with a million lines of code, that they cannot check for unsupported language features, so Nuitka is just fully compatible, to a level of absurdity.

Having conducted Nuitka on a level of where a corporate business can throw it’s code at Nuitka and have it just work, and reliably, this will only improve now.

Naturally bugs have happened once in a while, but I was always more careful. You know, like if this was for safety critical software, on which lives might depend. Hard to not be affected by that stance.

So lets talk about other ways I plan to expand.

Nuitka Package Configuration

Extending Nuitka with tweaks for package has become increasingly simpler. Many previous plugins, e.g. for torch, tensorflow, and most recently numpy, and definitely also the Qt binding ones in the future, have been or will be converted to Yaml configuration applied by generic plugins that deal with DLLs, data-files, tweaks (called anti-bloat) etc.

Adding stuff there is simple, and as of recently, there is even documentation for it. This is going to be explained in a series of posts detailing Nuitka Package Configuration, under the package_config tag.

I must say it’s so much fun, I love working with it. But the goal is for this to have you join it, and PRs are indeed starting to come in more frequently it seems from first glance, but I need to make this more known.

Streaming

I have experimented with streaming in the past, and I absolutely love it. Without my day job, effectively working two jobs at the same time, I expect to have the time to do more of them in December and then on a regular basis.

Python 3.11

There will be a separate post on this issue later this week. I am scrambling to add support for it and it feels very far already. Many basic tests already work, but more debugging will be needed. I hope Nuitka 1.3 will add support for it, at least on the usual experimental level.

How you can help?

Please become a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All essential packaging and performance features are free, and I have put incredible amounts of works in this, and I need to now make a living off it.

Right now, I am far from being able to sustain my income from it, and I am taking a risk with this effort, hoping for support from you.

Nuitka commercial makes creating a Windows Service a blast. It has a PySide2 that is compatible with Nuitka (I did make PySide6 compatible myself, and backported it to PySide2), so if you need to stay on old software. And for IP protection, it is just incredibly important to not ship bytecode at all, and with Nuitka commercial, it protects your data and files at runtime too.

I am bad at marketing, but good at software, which is one more reason to purchase it. You know you love these kinds of people. Was that good marketing? Also, there are share buttons on this page. Please try them out and let people know about this.

In future postings, I will make it a regular thing to ask for help with some of the technical tasks. Stay tuned, there might be something that you can help with.

Becoming more active

I am the kind of person that would much rather do good stuff than to talk about it, like when it finally supports Python 3.11, normal old me would just mention it maybe on Twitter once, and have it in the changelog only.

I need to get away from that. I need to resume TWN (This week in Nuitka) and have all plans to do it. Obviously I can get caught up in things, and may just always do it later when there is more or better news.

Nuitka Release 1.2

2022-11-26T09:23:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new compatibility features and a few new optimization, while again consolidating what we have. Scalability should be better in many cases.

Bug Fixes

Standalone: Added implicit dependency of thinc backend. Fixed in 1.1.1 already.
Python3.10: Fix, match statements with unnamed star matches could give incorrect results. Fixed in 1.1.1 already.
match x: case [*_, y]: ... # y had wrong value here.

Python3.9+: Fix, file reader objects must convert to str objects. Fixed in 1.1.1 already.

# This was the `repr` rather than a path value, but it must be usable
# like that too.
str(importlib.resources.files("package_name").joinpath("lala"))

Standalone: Added data file of echopype package. Fixed in 1.1.1 already.
Anti-Bloat: Remove non-sense warning of compiled pyscf. Fixed in 1.1.1 already.
macOS: Fix, in LTO mode using incbin can fail, switch to source mode for constants resources. Fixed in 1.1.2 already.
Standalone: Add support for sv_ttk module. Fixed in 1.1.2 already.
macOS: Fix, was no longer correcting libpython path, this was a regression preventing CPython for creating properly portable binary. Fixed in 1.1.2 already.
macOS: Fix, main binary was not included in signing command. Fixed in 1.1.3 already.
Standalone: Added implicit dependency of orjson. Due to zoneinfo not being automatically included anymore, this was having a segfault. Fixed in 1.1.3 already.
Standalone: Added support for new shapely. Fixed in 1.1.4 already.
macOS: Ignore extension module of non-matching architecture. Some wheels contain extension modules for only x86_64 arch, and others contain them only for arm64, preventing the standalone build. Fixed in 1.1.4 already.
Standalone: Added missing sklearn dependencies. Fixed in 1.1.4 already.
Fix, packages available through relative import paths could be confused with the same ones imported by absolute paths. This should be very hard to trigger, by normal users, but was seen during development. Fixed in 1.1.4 already.
Standalone: Apply import hacks for pywin32 modules only on Windows, otherwise it can break e.g. macOS compilation. Fixed in 1.1.4 already.
Windows: More robust DLL dependency caching, otherwise e.g. a Windows update can break things. Also consider plugin contribution, and Nuitka version, to be absolutely sure, much like we already do for bytecode caching. Fixed in 1.1.4 already.
Standalone: Fix, seaborn needs the same workaround as scipy for corruption with MSVC. Fixed in 1.1.4 already.
UI: Fix, the options-nanny was no longer functional and therefore failed to warn about non working options and package usages. Fixed in 1.1.5 already.
macOS: Do not use extension modules of non-matching architecture. Fixed in 1.1.5 already.
Windows: Fix, resolving symlinks could fail for spaces in paths. Fixed in 1.1.6 already.
Standalone: Added missing DLL for lightgbm module. Fixed in 1.1.6 already.
Compatibility: Respect super module variable. It is now possible to have a module level change of super but still get compatible behavior with Nuitka. Fixed in 1.1.6 already.
Compatibility: Make sure we respect super overloads in the builtin module. Fixed in 1.1.6 already.
Fix, the anti-bloat replacement code for numpy.testing was missing a required function. Fixed in 1.1.6 already.
Fix, importlib.import_module static optimization was mishandling a module name of . with a package name given. Fixed in 1.1.6 already.
macOS: Fix, some extension modules use wrong suffixes in self references, we need to not complain about this kind of error. Fixed in 1.1.6 already.
Fix, do not make ctypes.wintypes a hard import on non-Windows. Nuitka asserted against it failing, where some code handles it failing on non-Windows platforms. Fixed in 1.1.6 already.
Standalone: Added data files for vedo package. Fixed in 1.1.7 already.
Plugins: Fix, the gi plugin did always set GI_TYPELIB_PATH even if already present from user code. Also it did not handle errors to detect its value during compile time. Fixed in 1.1.7 already.
Standalone: Added missing dependencies for sqlalchemy to have all SQL backends working. Fixed in 1.1.7 already.
Added support Nixpkgs’s default non-writable HOME directory. Fixed in 1.1.8 already.
Fix, distribution metadata name and package name need not align, need to preserve the original looked up name from importlib.metadata.distribution call. Fixed in 1.1.8 already.
Windows: Fix, catch usage of unsupported CLCACHE_MEMCACHED mode with MSVC compilation. It is just unsupported.
Windows: Fix, file version was spoiled from product version if it was the only version given.
Windows: The default for file description in version information was not as intended.
Plugins: Workaround for PyQt5 as contained in Anaconda providing wrong paths from the build machine.
macOS: After signing a binary with a certificate, compiling the next one was crashing on a warning about initially creating an ad-hoc binary.
Fix, detect case of non-writable cache path, make explaining error exit rather than crashing attempting to write to the cache.
macOS: Added support for pyobjc in version 9.0 or higher.

New Features

Python3.11: For now prevent the execution with 3.11 and give a warning to the user for a not yet supported version. This can be overridden with --experimental=python311 but at this times will not compile anything yet due to required and at this time missing core changes.
macOS: Added option --macos-sign-notarization that signs with runtime signature, but requires a developer certificate from Apple. As its name implies, this is for use with notarization for their App store.
DLLs used via delvewheel were so far only handled in the zmq plugin, but this has been generalized to cover any package using it. With that, e.g. shapely just works. This probably helps many other packages as well.
Added __compiled__ and __compiled_constant__ attributes to compiled functions.

With this, it can be decided per function what it is and bridges like pyobjc can use it to create better code on their side for constant value returning functions.
Added support_info check to Nuitka package format. Make it clear that pyobjc is only supported after 9.0 by erroring out if it has a too low version. It will not work at all before that version added support in upstream. Also using this to make it clear that opencv-python is best supported in version 4.6 or higher. It seems e.g. that video capture is not working with 4.5 at this time.
Added --report-template which can be used to provide Jinja2 templates to create custom reports, and refer to built-in reports, at this time e.g. a license reports.

Optimization

Trust the absence of a few selected hard modules and convert those to static raises of import errors.
For conditional nodes where only one branch exits, and the other does not, no merging of the trace collection should happen. This should enhance the scalability and leads to more static optimization being done after these kinds of branches on variables assigned in such branches.
```
if condition1:
   a = 1
else:
   raise KeyError

if condition2:
   b = 1

# Here, "a" is known to be assigned, but before it was only "maybe"
# assigned, like "b" would have to be since, the branch may or may
# not have been taken.
```

Do not merge tried blocks that are aborting with an exception handler that is not aborting. This is very similar to the change for conditional statements, again there is a control flow branch, that may have to be merged with an optional part, but sometimes that part is not optional from the perspective of the code following.

try:
   ... # potentially raising, but not aborting code
   return something() # this aborts
except Exception:
   a = 1

try:
   ... # potentially raising, but not aborting code
except Exception:
   b = 1

# Here, "a" is known to be assigned, but before it was only "maybe"
# assigned, like "b" would have to be since, the branch may or may
# not have been taken.

Exception matches were annotating a control flow escape and an exception exit, even when it is known that no error is possible to be happening that comparison.
```
try:
   ...
except ImportError: # an exception match is done here, that cannot raise
   ...
```
Trust importlib.metadata.PackageNotFoundError to exist, with this some more metadata usages are statically optimized. Added in 1.1.4 already.
Handle constant values from trusted imports as trusted values. So far, trusted import values were on equal footing to regular variables, which on the module level could mean that no optimization was done, due to control flow escapes happening.
```
# Known to be False at compile time.
from typing import TYPE_CHECKING
...

if TYPE_CHECKING:
   from something import normally_unused_unless_type_checking
```
In this code example above, the static optimization was not done because the value may be changed on the outside. However, for trusted constants, this is no longer assumed to be happening. So far only if typing.TYPE_CHECKING: using code had been optimized.
macOS: Use sections for main binary constants binary blob rather than C source code (which we started in a recent hotfix due to LTO issues with incbin) and onefile payload. The latter enables notarization of the onefile binary as well and makes it faster to unpack as well.
Windows: Do not include DLLs from SxS. For PyPI packages these are generally unused, and self compiled modules won’t be SxS installations either. We can add it back where it turns out needed. This avoids including comctl32 and similar DLLs, which ought to come from the OS, and might impede backward compatibility only.
Disabled C compilation of very large azure modules.
The per module usage information of other modules was only updated in first pass was used in later passes. But since they can get optimized away, we have to update every time, avoiding to still include unused modules.
Anti-Bloat: Fight the use of dask in xarray and pint, adding a mode controlling its use. This is however still incomplete and needs more work.
Fix, the anti-bloat configuration for rich.pretty introduced a SyntaxError that went unnoticed. In the future compilation will abort when this happens.
Standalone: Added support for including DLLs of llvmlite.binding package.
Anti-Bloat: Avoid using pywin32 through pkg_resources import. This seems rather pointless and follows an optimization done for the inline copy of Nuitka already, the ctypes code path works just fine, and this may well be the only reason why pywin32 is included, which is by itself relatively large.
Cache directory contents when scanning for modules. The sys.path and package directories were listed over and over, wasting time during the import analysis.
Optimization: Was not caching not found modules, but retrying every time for each usage, potentially wasting time during import analysis.
Anti-Bloat: Initial work to avoid pytest in patsy, it is however incomplete.

Organizational

User Manual: Explain how to create 64/32 bits binaries on Windows, with there being no option to control it, this can otherwise be a bit unobvious that you have to just use the matching Python binary.
UI: Added an example for a cached onefile temporary location spec to the help output of --onefile-tempdir-spec to make cached more easy to achieve in the proper way.
UI: Quote command line options with space in value better, no need to quote an affected command line option in its entirety, and it looks strange.
macOS: Catch user error of disabling the console without using the bundle mode, as it otherwise it has no effect.
macOS: Warn about not providing an icon with disabled console, otherwise the dock icon is empty, which just looks bad.
Debian: Also need to depend on glob2 packages which the yaml engine expects to use when searching for DLLs.
Debian: Pertain inline copies of modules in very old builds, there is e.g. no glob2 for older releases, but only recent Debian releases need very pure packages, our backport doesn’t have to do it right.
macOS: More reliable detection of Homebrew based Python. Rather than checking file system via its sitecustomize contents. The environment variables are only present to some usages.
Installations with pip did not include all license, README files, etc. which however was intended. Also the attempt to disable bytecode compilation for some inline copies was not effective yet.
Renamed pyzmq plugin to delvewheel as it is now absolutely generic and covers all uses of said packaging technique.
Caching: Use cache directory for cached downloads, rather than application directory, which is just wrong. This will cause all previously cached downloads to become unused and repeated.
Quality: Updated development requirements to latest black, isort, yamllint, and tqdm.
Visual Code: Added recommendation for extension for Debian packaging files.
Added warning for PyQt5 usage, since its support is very incomplete. Made the PyQt6 warning more concrete. It seems only Qt threading does not work, which is of course still significant. Instead PySide2 and PySide6 are recommended.
UI: Have dedicated options group for onefile, the spec for the temporary files is not a Windows option at all anymore. Also move the warnings group to the end, people need to see the inclusion related group first.
User Manual: Explain how to create 64/32 bits binaries on Windows, which is not too obvious.

Cleanups

Moved PySide plugins DLL search extra paths to the Yaml configuration. In this way it is not dependent on the plugin being active, avoiding cryptic errors on macOS when they are not found.
Plugins: Avoid duplicate link libraries due to casing. We are now normalizing the link library names, which avoids e.g. Shell32 and shell32 to be in the result.
Cleanups to prepare a PyLint update that so otherwise failed due to encountered issues.
Plugins: Pass so called build definitions for generically. Rather than having dedicated code for each, and plugins can now provide them and pass their index to the scons builds.
Onefile: Moved file handling code to common code reducing code duplication and heavily cleaning up the bootstrap code generally.
Onefile: The CRC32 checksum code was duplicated between constants blob and onefile, and has moved to shared code, with an actual interface to take the checksum.
Spelling cleanups resumed, e.g. this time more clearly distinguishing between run time and runtime, the first is when the program executes, but the latter can be an environment provided by a C compiler.

Tests

Tests: Added test that applies anti-bloat configuration to all found modules.
Tests: Tests: Avoid including unused nuitka.tools code in reflected test, which should make it faster. The compiler itself doesn’t use that code.

Summary

This release is again mainly a consolidation of previous release, as well as finishing off existing features. Optimization added in previous releases should have all regressions fixed now, again with a strong series of hotfixes.

New optimization was focused around findings with static optimization not being done, but still resulting in general improvements. Letting static optimization drive the effort is still paying off.

Scalability has seen improvements through some of the optimization, this time a lot less anti-bloat work has been done, and some things are only started. The focus will clearly now shift to making this a community effort. Expect postings that document the Yaml format we use.

For macOS specifically, with the sections work, onefile should launch faster, should be more compatible with signing, and can now be used in notarization, so for that platform, things are more round.

For Windows, a few issues with version information and onefile have been addressed. We should be able to use memory mapped view on this platform too, for faster unpacking of the payload, since it doesn’t have to go through the file anymore.

Nuitka Release 1.1

2022-10-16T10:56:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new compatibility features, while consolidating what we have. Scalability should be better in some cases.

Bug Fixes

Standalone: Enhanced dependency scan of dependent DLLs to forward the containing package, so it can be searched in as well. This fixed at least PySide on macOS. Fixed in 1.0.1 already.
macOS: Enhanced dependency detection to use normalized paths and therefore to be more stable. Fixed in 1.0.1 already.
Standalone: Added support for the networkx package which uses new support for a function decorator trying to copy function default values. Fixed in 1.0.1 already.
Standalone: Include data files for pandas.io.format package. This one has Jinja2 template files that will be needed when using this package.
Python3.10: Fix, could crash in case a class was not giving match arguments, but the user did attempt to match them. This happened e.g. with range objects. Fixed in 1.0.2 already.
Standalone: Added data files needed for pyenchant package. Fixed in 1.0.2 already.
Python3.10: Fix, matching sequence with as assignments in them didn’t check for sub-pattern given. Fixed in 1.0.2 already.
Standalone: Fix, do not attempt to list non-existent PATH entries on Windows, these can crash the dependency detection otherwise. Fixed in 1.0.2 already.
Standalone: Fix, on newer Linux, linux-vdso.so.1 appears in output of ldd in a way that suggests it may exist, which of course it does not, this is a kernel virtual DLL. Fixed in 1.0.3 already.
Fix, comparison expressions could give wrong results as a regression of the new release. Fixed in 1.0.3 already.
Fix, on older Python (before 3.6), it could crash on data files defined in the Yaml config. Fixed in 1.0.4 already.
Fix, binary operations could give wrong results as a regression of the new release. Fixed in 1.0.4 already.
Standalone: Added support for pyzbar package. Fixed in 1.0.5 already.
Standalone: Fix, empty directory structures were not working anymore due to a regression in the last release. Fixed in 1.0.5 already.
Windows: Fix, detected Pythons from Windows registry may of course fail to execute, because they were e.g. manually deleted. This would show e.g. in onefile compression. Fixed in 1.0.5 already.
Onefile: Fix, using a too old zstandard without finding another Python with a suitable one, lead to run time unpacking errors. Fixed in 1.0.6 already.
Fix, the inline copy of Jinja2 imported logging for no good reason, which lead to errors for users who have a module of the same name, that it was then using instead. Fixed in 1.0.6 already.
Fix, disable LTO mode for Anaconda Python, it is known to not work. Fixed in 1.0.6 already.
Linux: Fix, no need to insist on icon path for onefile anymore. Fixed in 1.0.6 already.
Standalone: Fix, the new version certifi was not working on Windows and 3.10 anymore. Fixed in 1.0.7 already.
Standalone: Added support for more rapidfuzz implicit dependencies. Fixed in 1.0.8 already.
Standalone: Added support for vibora. Fixed in 1.0.8 already.
Fix, must not expose module name objects to Python import hooks. Fixed in 1.0.8 already.
Fix, calls to bound methods of string values generated incorrect calls. Fixed in 1.0.8 already.
Fix, do not crash in version detection on gcc error exit querying of its version.
Standalone: Added back support for older versions of the pyzmq package.
Standalone: Ignore PATH elements that fail to be listed as a directory. It appears e.g. on Windows, folders can exist, despite being unusable in fact. These can then cause errors in DLL dependency scan. Also avoid having PATH set when executing dependency walker, it appears to use it even if not asked to.
Standalone: Added support for tzlocal package.
Python3.10: Fix, complex literals were not working for mappings in match statements.
Fix, bool built-in expressions were not properly annotating exception raises, where the value cannot raise on truth check.
Standalone: Added support for the agscheduler package. Plugins must be done manually still with explicit --include-module calls.
Standalone: Added support for using shapely in Anaconda as well.
Debian: Fix, versioned dependency for libzstd should also be in package, this should restore Nuitka package builds for Debian Jessie.
Standalone: Added support for vtk package.
Windows: Fix, avoid using pywin32 in our appdirs usage, it might be a broken installation and is optional to appdirs anyway, which then will fallback to using ctypes to make the lookups.
Standalone: Added support for more pandas versions.
Standalone: Adding support for mkl implicit DLL usage in Anaconda.
Standalone: Added support for jsonschema with Python 3.10.
Standalone: Added support for pyfiglet fonts data files.
Scons: Avoid gcc linker command line length limits for module mode too.
Standalone: Added data file of distributed.config.
Standalone: Add support for cv2 GUI on Linux, the Qt platform plugin is now included.
Fix, the anti-bloat configuration for numpy.testing tools exposed an incomplete suppress_warnings replacement that could lead to errors in some functions of numpy.
Standalone: Fix DLL dependency caching on Windows, need to consider DLL content of course too.
Standalone: Added missing dependency for torchvision.
Standalone: Added support for torchvision on Anaconda as well.
Standalone: Added support for panda3d.
Windows: Fix, need to make sure to use UTF-8 encoding for define values like company name. Otherwise the local system encoding is used, but the C compiler expects UTF-8 in wide literals. This may crash of give wrong results.
Standalone: Added facenet_torch data files.
Anaconda: Include libstdc++.so on Linux or else e.g. cv2 will not work with system library.
Windows: Fix, can have file version without a company name.

New Features

Python3.10: Added support for assignments in match alternatives | syntax.
Compatibility: Register Nuitka meta path based loader with pkg_resources such that checking resource presence with has_resource works too. This should also add support for using jinja2.PackageLoader, previously only jinja2.FileSystemLoader worked. Fixed in 1.0.1 already.
Compatibility: Make function __defaults__ attribute size changeable. For a long time, this was a relatively big issue for some packages, but now this is supported as well.
Compatibility: Added support for importlib.metadata.distribution and importlib_metadata.distribution functions as well importlib.metadata.metadata and importlib_metadata.metadata functions.
Onefile: Added support for including other binaries than the main executable in the payload. So far on non-Windows, we only made the main binary executable, hard coded, and nothing else. But Some things, e.g. Qt web engine, do require binaries to be used, and these no longer have the issue of missing x-bit on macOS and Linux now.
Standalone: Resolve executable path when called through symbolic link, which makes file resolutions work properly for it, for this type of installing it in %PATH%.
Python3.9+: Added support for importlib.resources.files with compiled modules.

It returns traversable objects, which can be used to opens files, checks them, etc. and this e.g. allows jsonschema to work with Python 3.10, despite bugs in CPython’s compatibility layer.
UI: Added interface method to specify filename patterns with package data inclusion option, making --include-package-data usable in many more cases, picking the only files or file types you want. You can now use --include-package-data=package_name=*.txt and select only a subset of package data files in this way. Before this, it included everything and --noinclude-data-files would have to be used.
macOS: Make runtime signing an experimental option.
Consistently allow when conditions for all package configuration elements, e.g. also DLLs.
Plugins: Added method to overload to work on standalone binary specifically. This makes it easier to only modify that specific binary.
Plugins: Added support for regular expressions in anti-bloat replacements, with new replacements_re code.

Optimization

Add support for os.path hard module imports along with specialized nodes for file tests os.path.exists, os.path.isfile, and os.path.isdir aiming at tracking used files, producing warnings about missing files in the future.
Standalone: Do not include concurrent standard library package automatically. This avoids the inclusion of multiprocessing which we essentially had reverted during last release cycle.
Standalone: Do not include zoneinfo standard library package automatically. It has many data files and is often not used (yet).
Standalone: Do not include asyncio standard library package automatically anymore.
Avoid compilation of large generated codes in the asyncua package.
Compile time optimize pkg_resources.iter_entry_points too, such that these can be used to resolve plugin modules, which helps with adding support for agscheduler package plugins. Note that these still need to be manually included with --include-module but now that works.
For known truth values of the right hand side of and or or conditions, reduce the expression as far as possible.
Added dedicated assignment node for hard imports, which then are propagated in classes as well, allowing for more static optimization for code on the class level.
Added linker options to make static --static-libpython work with clang on Linux as well.
macOS: Make sure libpython is loaded relative to the executable. This is needed for at least Anaconda Python.
macOS: Fix, need to search environment specific DLL paths and only then global paths, otherwise mixed Python versions will not work correctly.
Anti-Bloat: Remove IPython usage in rich package.
Anti-Bloat: Avoid doctest dependency when using pyrect.
Anti-Bloat: Some unittest removals from pytorch using libraries.
Keep the Scons report items sorted, or else it varies for the hashing of dependencies with Python versions before 3.6, causing cache misses without need.

Organizational

UI: Output the .cmd file created (if any) on Windows, e.g. when run in a virtualenv or for uninstalled Python versions, it will otherwise not run in accelerated mode, but previously the output suggested to run the executable directly.
UI: Enhanced command line option description of --include-plugin-directory which is frequently misunderstood. That option barely does what people want it to do. Point them to using the other options that are easy to use and will work.
UI: Specified needed Python version for use in --python-for-scons so users can know ahead of time what versions are suitable.
Reports: Added information about data files including, optimization times per module, active plugins.
Debugging: Repaired offline DLL dependency listing tool, such that it can be used during Windows DLL analysis.
Make --xml accept a filename for the node tree dump, and change it so it can be executed in addition to actual compilation. This way we need not be super-robust about keeping stdout clean, to not break XML parsing.
Plugins: Avoid useless warning about PySide2 plugin usage if another Qt plugin is actually selected.
UI: Catch error of directories being used as data files where plain files are expected and point out that other options must be used.
User Manual: Added section about accessing files in standalone mode too, so people can make sure it works properly.
Onefile: Using %TEMP% folder should not by itself prevent cached onefile mode, only really variable paths should. People may want to have this as some kind of temporary cache still.
UI: Catch user error of using elements, that resolve to absolute values in the middle of path specs, so using e.g. something/%PROGRAM% is now a mistake caught at compile time. These values can only be at the start of spec values naturally.
Quality: Updated to newer version of rstfmt.
UI: Nicer error message when a forbidden import is requested as an implicit import by a plugin.
Python3.11: Adapted to allocator and exception state changes, but more will be needed to compile at all.
Visual Code: Find clang-format from the recommended C++ extension of Visual Code, which makes it finally available on macOS easily too.
UI: Quote command line argument values as necessary when stating them in the logging. Otherwise they are not directly usable on the shell and also less readable.
Debian: Do not list fake modules as used debian packages codes, which could e.g. happen with the pre-load code of pkg_resources if that is from a Debian package. Fake packages should not be mentioned for these lists though.
Nuitka-Python: Added support to set link time flags coming from statically included packages.
For our isort trick of splitting files in two parts (mostly to setup import paths for nuitka package), make sure the second parts starts with a new line.
Added more usable form --output-filename to specify the output filename, the short form has become barely usable after we switched to enforcing no space separation for command line arguments.
UI: Check if output filename’s directory exists ahead of time, and error exit if not, otherwise compilation crashed only in the very end, trying to create the final result.
UI: When exiting with no error code, do not use red color or FATAL error annotation, that is not justified.
Quality: Make sure the Yaml auto-format does not change effective contents.
Quality: Added ability to limit autoformat by file type, which can be handy when e.g. only the yaml files should be scanned.
UI: Removed --windows-onefile-tempdir-spec alias of --onefile-tempdir-spec option, it is no longer Windows specific.

Cleanups

Prefer single quotes rather than double quotes in our package configuration Yaml files, otherwise esp. regular expressions with escapes become very confusing.
Move import hacks to general mechanism in Yaml package configuration files. This is for extra paths from package names or from directory paths relative to the package. This removes special purpose code from core code paths and allows their reuse.
Again more spelling cleanups have been done, to make the code cleaner to read and search.
Unified how plugins treat iteration over their value list, and how the when condition is applied for the various kinds of sections.
Output compilation command that failed during coverage taking, which makes it unnecessary to attempt to reconstruct what happened from test modes.

Tests

Added coverage for comparisons that need argument swaps.
Allow more time in onefile keyboard signal test, otherwise it can be a race on slow machines, e.g. emulated machines.
Tests: Added support for running a local web server.

Summary

This release is mainly a consolidation of previous release. Optimization added in previous release did in fact introduce regressions, that needed to be addressed and were cause for relatively many hotfixes.

The Yaml Nuitka Package Configuration feature is getting ever more powerful, but is not one bit more documented, such that the community as a whole is not yet capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.

New optimization was focused around compatibility with very few exceptions, where the non-automatic standard library work is standing out, and allows for smaller binaries in many cases.

Scalability has seen improvements through a few optimization, but mainly again with anti-bloat work being done. This is owed to the fact that consolidation was the name of the game.

For Anaconda specifically, a lot more software is covered, and generally, cv2 and torch related tools are now working better, but it seems DLL handling will remain problematic in many instances.

The compilation report contains much more information and is getting there is terms of completeness. At some point, we should ask for it in bug reports.

Nuitka Release 1.0

2022-08-16T13:45:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains a large amount of new features, while consolidating what we have with many bug fixes. Scalability should be dramatically better, as well as new optimization that will accelerate some code quite a bit. See the summary, how this release is paving the way forward.

Bug Fixes

Python3: Fix, bytes.decode with only errors argument given was not working. Fixed in 0.9.1 already.
MSYS2: Fix, the accelerate mode .cmd file was not working correctly. Fixed in 0.9.1 already.
Onefile: Fix, the bootstrap when waiting for the child, didn’t protect against signals that interrupt this call. This only affected users of the non-public --onefile-tempdir option on Linux, but with that becoming the default in 1.0, this was discovered. Fixed in 0.9.1 already.
Fix, pkg_resources compile time generated Distribution values could cause issues with code that put it into calls, or in tried blocks. Fixed in 0.9.1 already.
Standalone: Added implicit dependencies of Xlib package. Fixed in 0.9.1 already.
macOS: Fix, the package configuration for wx had become invalid when restructuring the Yaml with code and schema disagreeing on allowed values. Fixed in 0.9.1 already.
Fix: The str.format with a single positional argument didn’t generate proper code and failed to compile on the C level. Fixed in 0.9.1 already.
Fix, the type shape of str.count result was wrong. Fixed in 0.9.1 already.
UI: Fix, the warning about collision of just compiled package and original package in the same folder hiding the compiled package should not apply to packages without an __init__.py file, as those do not take precedence. Fixed in 0.9.2 already.
Debugging: Fix, the fallback to lldb from gdb when using the option --debugger was broken on anything but Windows. Fixed in 0.9.2 already.
Python3.8: The module importlib.metadata was not recognized before 3.9, but actually 3.8 already has it, causing the compile time resolution of package versions to not work there. Fixed in 0.9.3 already.
Standalone: Fix, at least on macOS we should also scan from parent folders of DLLs, since they may contain sub-directories in their names. This is mostly the case, when using frameworks. Fixed in 0.9.2 already.
Standalone: Added package configuration for PyQt5 to require onefile bundle mode on macOS, and recommend to disable console for PyQt6. This is same as we already do for PySide2 and PySide6. Fixed in 0.9.2 already.
Standalone: Removed stray macOS onefile bundle package configuration for pickle module which must have been added in error. Fixed in 0.9.2 already.
UI: Catch user error of attempting to compile the __init__.py rather than the package directory. Fixed in 0.9.2 already.
Fix, hard name import nodes failed to clone, causing issues in optimization phase. Fixed in 0.9.2 already.
Fix, avoid warnings given with gcc 11. Fixed in 0.9.2 already.
Fix, dictionary nodes where the operation itself has no effect, e.g. dict.copy were not properly annotating that their dictionary argument could still cause a raise and have side effects, triggering an assertion violation in Nuitka. Fixed in 0.9.2 already.
Standalone: Added pynput implicit dependencies on Linux. Fixed in 0.9.2 already.
Fix, boolean condition checks on variables converted immutable constant value assignments to boolean values, leading to incorrect code execution. Fixed in 0.9.2 already.
Python3.9: Fix, could crash on generic aliases with non-hashable values. Fixed in 0.9.3 already.
```
dict[str:any]
```
Python3: Fix, an iteration over sys.version_info was falsely optimized into a tuple, which is not always compatible. Fixed in 0.9.3 already.
Standalone: Added support for xgboost package. Fixed in 0.9.3 already.
Standalone: Added data file for text_unidecode package. Fixed in 0.9.4 already.
Standalone: Added data files for swagger_ui_bundle package. Fixed in 0.9.4 already.
Standalone: Added data files for connexion package. Fixed in 0.9.4 already.
Standalone: Added implicit dependencies for sklearn.utils and rapidfuzz. Fixed in 0.9.4 already.
Python3.10: Fix, the reformulation of match statements could create nodes that are used twice, causing code generation to assert. Fixed in 0.9.4 already.
Fix, module objects removed from sys.modules but still used could lack a reference to themselves, and therefore crash due to working on a released module variables dictionary. Fixed in 0.9.5 already.
Fix, the MSVC compiles code generated for SciPy 1.8 wrongly. Added a workaround for that code to avoid triggering it. Fixed in 0.9.6 already.
Fix, calls to str.format where the result is not used, could crash the compiler during code generation. Fixed in 0.9.6 already.
Standalone: For DLLs on macOS and Anaconda, also consider the lib directory of the root environment, as some DLLs are otherwise not found.
Fix, allow nonlocal and global for __class__ to be used on the class level.
Fix, xrange with large values didn’t work on all platforms. This affected at least Python2 on macOS, but potentially others as well.
Windows: When scanning for installed Pythons to e.g. run Scons or onefile compression, it was attempting to use installations that got deleted manually and could crash.
macOS: Fix, DLL conflicts are now resolved by checking the version information too, also all cases that previously errored out after a conflict was reported, will now work.
Fix, conditional expressions whose statically decided condition picking a branch will raise an exception could crash the compilation.
```
# Would previously crash Nuitka during optimization.
return 1/0 if os.name == "nt" else 1/0
```
Windows: Make sure we set C level standard file handles too

At least newer subprocess was affected by this, being unable to provide working handles to child processes that pass their current handles through, and also this should help DLL code to use it as level.
Standalone: Added support for pyqtgraph data files.
Standalone: Added support for dipy by anti-bloat removal of its testing framework that wants to do unsupported stuff.
UI: Could still give warnings about modules not being followed, where that was not true.
Fix, --include-module was not working for non-automatic standard library paths.

New Features

Onefile: Recognize a non-changing path from --onefile-tempdir-spec and then use cached mode. By default a temporary folder is used in the spec value, make it delete the files afterwards.

The cached mode is not necessarily faster, but it is not going to change files already there, leaving the binaries there intact. In the future it may also become faster to execute, but right now checking the validity of the file takes about as long as re-creating it, therefore no gain yet. The main point, is to not change where it runs from.
Standalone: Added option to exclude DLLs. You can npw use --noinclude-dlls to exclude DLLs by filename patterns.

The may e.g. come from Qt plugins, where you know, or experimented, that it is not going to be used in your specific application. Use with care, removing DLLs will lead to very hard to recognize errors.
Anaconda: Use CondaCC from environment variables for Linux and macOS, in case it is installed. This can be done with e.g. conda install gcc_linux-64 on Linux or conda install clang_osx-64 on macOS.
Added new option --nowarn-mnemonic to disable warnings that use mnemonics, there is currently not that many yet, but it’s going to expand. You can use this to acknowledge the ones you accept, and not get that warning with the information pointer anymore.
Added method for resolving DLL conflicts on macOS too. This is using version information and picks the newer one where possible.
Added option --user-package-configuration-file for user provided Yaml files, which can be used to provide package configuration to Nuitka, to e.g. add DLLs, data files, do some anti-bloat work, or add missing dependencies locally. The documentation for this does not yet exist though, but Nuitka contains a Yaml schema in the misc/nuitka-package-config-schema.json file.
Added nuitka-project-else to avoid repeating conditions in Nuitka project configuration, this can e.g. be used like this:
```
# nuitka-project-if: os.getenv("TEST_VARIANT", "pyside2") == "pyside2":
#   nuitka-project: --enable-plugin=no-qt
# nuitka-project-else:
#   nuitka-project: --enable-plugin=no-qt
#   nuitka-project: --noinclude-data-file=*.svg
```
Previously, the inverted condition had to be used in another nuitka-project-if which is no big deal, but less readable.

Added support for deep copying uncompiled functions. There is now a section in the User Manual that explains how to clone compiled functions. This allows a workaround like this:

def binder(func, name):
   try:
      result = func.clone()
   except AttributeError:
      result = types.FunctionType(func.__code__, func.__globals__, name=func.__name__, argdefs=func.__defaults__, closure=func.__closure__)
      result = functools.update_wrapper(result, func)
      result.__kwdefaults__ = func.__kwdefaults__

   result.__name__ = name
   return result

Plugins: Added explicit deprecation status of a plugin. We now have a few that do nothing, and are just there for compatibility with existing users, and this now informs the user properly rather than just saying it is not relevant.
Fix, some Python installations crash when attempting to import modules, such as os with a ModuleName object, because we limit string operations done, and e.g. refuse to do .startswith which of course, other loaders that your installation has added, might still use.
Windows: In case of not found DLLs, we can still examine the run time of the currently compiling Python process of Nuitka, and locate them that way, which helps for some Python configurations to support standalone, esp. to find CPython DLL in unusual spots.
Debian: Workaround for lib2to3 data files. These are from stdlib and therefore the patched code from Debian needs to be undone, to make these portable again.

Optimization

Scalability: Avoid merge traces of initial variable versions, which came into play when merging a variable used in only one branch. These are useless and only made other optimization slower or impossible.
Scalability: Also avoid merge traces of merge traces, instead flatten merge traces and avoid the duplication doing so. There were pathological cases, where this reduced optimization time for functions from infinite to instant.
For comparison helpers, switch comparison where possible, such that there are only 3 variants, rather than 6. Instead the boolean result is inverted, e.g. changing >= into not < effectively. Of course this can only be done for types, where we know that nothing special, i.e. no method overloads of __gte__ is going on.
For binary operations that are commutative with the selected types, in mixed type cases, swap the arguments during code generation, such that e.g. long_a + float_b is actually computed as float_b + long_a. This again avoids many helpers. It also can be done for * with integers and container types.
In cases, where a comparison (or one of the few binary operation where we consider it useful), is used in a boolean context, but we know it is impossible to raise an exception, a C boolean result type is used rather than a nuitka_bool which is now only used when necessary, because it can indicate the exception result.
Anti-Bloat: More anti-bloat work was done for popular packages, covering also uses of setuptools_scm, nose and nose2 package removals and warnings. There was also a focus on making mmvc, tensorflow and tifffile compile well, removing e.g. the uses of the tensorflow testing framework.
Faster comparison of int values with constant values, this uses helpers that work with C long values that represent a single “digit” of a value, or ones that use the full value space of C long.
Faster comparison of float values with constant values, this uses helpers that work with C float values, avoiding the useless Python level constant objects.
Python2: Comparison of int and long now has specialized helpers that avoids converting the int to a long through coercion. This takes advantage of code to compare C long values (which are at the core of Python2 int objects, with long objects.
For binary operation on mixed types, e.g. int * bytes the slot of the first function was still considered, and called to give a Py_NotImplemented return value for no good reason. This also applies to mixed operations of int, long, and float types, and for str and unicode values on Python2.
Added missing helper for ** operation with floats, this had been overlooked so far.
Added dedicated nodes for ctypes.CDLL which aims to allow us to detect used DLLs at compile time in the future, and to move closer to support its bindings more efficiently.
Added specialized nodes for dict.popitem as well. With this, now all of the dictionary methods are specialized.
Added specialized nodes for str.expandtabs, str.translate, str.ljust, str.rjust, str.center, str.zfill, and str.splitlines. While these are barely performance relevant, this completes all str methods, except removeprefix and removesuffix that are Python3.9 or higher.
Added type shape for result of str.index operation as well, this was missing so far.
Optimize str, bytes and dict method calls through variables.
Optimize calls through variables containing e.g. mutable constant values, these will be rare, because they all become exceptions.
Optimize calls through variables containing built-in values, unlocking optimization of such calls, where it is assigned to a local variable.
For generated attribute nodes, avoid local doing import statements on the function level. While these were easier to generate, they can only be slow at run time.
For the str built-in annotate its value as derived from str, which unfortunately does not allow much optimization, since that can still change many things, but it was still a missing attribute.
For variable value release nodes, specialize them by value type as well, enhancing the scalability, because e.g. parameter variable specific tests, need not be considered for all other variable types as well.

Organizational

Plugins: Major changes to the Yaml file content, cleaning up some of the DLL configuration to more easy to use.

The DLL configuration has two flavors, one from code and one from filename matching, and these got separated into distinct items in the Yaml configuration. Also how source and dest paths get provided got simplified, with a relative path now being used consistently and with sane defaults, deriving the destination path from where the module lives. Also what we called patterns, are actually prefixes, as there is still the platform specific DLL file naming appended.
Plugins: Move mode checks to dedicated plugin called options-nanny that is always enabled, giving also much cleaner Yaml configuration with a new section added specifically for these. It controls advice on the optional or required use of --disable-console and the like. Some packages, e.g. wx are known to crash on macOS when the console is enabled, so this advice is now done with saner configuration.
Plugins: Also for all Yaml configuration sub-items where is now a consistent when field, that allows checking Python version, OS, Nuitka modes such as standalone, and only apply configuration when matching this criterion, with that the anti-bloat options to allow certain bloat, should now have proper effect as well.
The use of AppImage on Linux is no more. The performance for startup was always slower, while having lost the main benefit of avoiding IO at startup, due to new cached mode, so now we always use the same bootstrap binary as on macOS and Windows.
UI: Do not display implicit reports reported by plugins by default anymore. These have become far too many, esp. with the recent stdlib work, and often do not add any value. The compilation report will become where to turn to find out why a module in included.
UI: Ask the user to install the ordered set package that will actually work for the specific Python version, rather than making him try one of two, where sometimes only one can work, esp. with Python 3.10 allowing only one.
GitHub: More clear wording in the issue template that python -m nuitka --version output is really required for support to given.
Attempt to use Anaconda ccache binary if installed on non-Windows. This is esp. handy on macOS, where it is harder to get it.
Windows: Avoid byte-compiling the inline copy of Scons that uses Python3 when installing for Python2.
Added experimental switches to disable certain optimization in order to try out their impact, e.g. on corruption bugs.
Reports: Added included DLLs for standalone mode to compilation report.
Reports: Added control tags influencing plugin decisions to the compilation report.
Plugins: Make the implicit-imports dependency section in the Yaml package configuration a list, for consistency with other blocks.
Plugins: Added checking of tags such from the package configuration, so that for things dependent on python version (e.g. python39_or_higher, before_python39), the usage of Anaconda (anaconda) or certain OS (e.g. macos), or modes (e.g. standalone), expressions in when can limit a configuration item.
Quality: Re-enabled string normalization from black, the issues with changes that are breaking to Python2 have been worked around.
User Manual: Describe using a minimal virtualenv as a possible help low memory situations as well.
Quality: The yaml auto-format now properly preserves comments, being based on ruamel.yaml.
Nuitka-Python: Added support for the Linux build with Nuitka-Python for our own CPython fork as well, previously only Windows was working, amd macOS will follow later.
The commit hook when installed from git bash was working, but doing so from cmd.exe didn’t find a proper path for shell from the git location.
Debugging: A lot of experimental toggles were added, that allow control over the use of certain optimization, e.g. use of dict, list, iterators, subscripts, etc. internals, to aid in debugging in situations where it’s not clear, if these are causing the issue or not.
Added support for Fedora 36, which requires some specific linker options, also recognize Fedora based distributions as such.
Removed long deprecated option --noinclude-matplotlib from numpy plugin, as it hasn’t had an effect for a long time now.
Visual Code: Added extension for editing Jinja2 templates. This one even detects that we are editing C or Python and properly highlights accordingly.

Cleanups

Standalone: Major cleanup of the dependency analysis for standalone. There is no longer a distinction between entry points (main binary, extension modules) and DLLs that they depend on. The OS specific parts got broken out into dedicated modules as well and decisions are now taken immediately.
Plugins: Split the Yaml package configuration files into 3 files. One contains now Python2 only stdlib configuration, and another one general stdlib.
Plugins: Also cleanup the zmq plugin, which was one the last holdouts of now removed plugin method, moving parts to the Yaml configuration. We therefore no longer have considerExtraDlls which used to work on the standalone folder, but instead only plugin code that provides included DLL or binary objects from getExtraDlls which gives Nuitka much needed control over DLL copying. This was a long lasting battle finally won, and will allow many new features to come.
UI: Avoid changing whitespace in warnings, where we have intended line breaks, e.g. in case of duplicate DLLs. Went over all warnings and made sure to either avoid new-lines or have them, depending on wanted output.
Iterator end check code now uses the same code as rich comparison expressions and can benefit from optimization being done there as well.
Solved TODO item about code generation time C types to specify if they have error checking or not, rather than hard coding it.
Production of binary helper function set was cleaned up massively, but still needs more work, comparison helper function set was also redesigned.
Changing the spelling of our container package to become more clear.
Used namedtuple objects for storing used DLL information for more clear code.
Added spellchecker ignores for all attribute and argument names of generated fixed attribute nodes.
In auto-format make sure the imports float to the top. That very much cleans up generated attribute nodes code, allowing also to combine the many ones it makes, but also cleans up some of our existing code.
The package configuration Yaml files are now sorted according to module names. This will help to avoid merge conflicts during hotfixes merge back to develop and automatically group related entries in a sane way.
Moved large amounts of code producing implicit imports to Yaml configuration files.
Changed the tensorflow plugin to Yaml based configuration, making it a deprecated do nothing plugin, that only remains there for a few releases, to not crash existing build scripts.
Lots of spelling cleanups, e.g. renaming nuitka.codegen to nuitka.code_generation for clarity.

Tests

Added generated test to cover bytes method. This would have found the issue with decode potentially.
Enhanced standalone test for ctypes on Linux to actually have something to test.

Summary

This release improves on many things at once. A lot of work has been put into polishing the Yaml configuration that now only lacks documentation and examples, such that the community as a whole should become capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.

Then a lot of new optimization has been done, to close the missing gaps with dict and str methods, but before completing list which is already a work in progress pull request, and bytes, we want to start and generate the node classes that form the link or basis of dedicated nodes. This will be an area to work on more.

The many improvements to existing code helpers, and them being able to pick target types for the arguments of comparisons and binary operations, is a pre-cursor to universal optimization of this kind. What is currently only done for constant values, will in the future be interesting for picking specific C types for use. That will then be a huge difference from what we are doing now, where most things still have to use PyObject * based types.

Scalability has again seen very real improvements, memory usage of Nuitka itself, as well as compile time inside Nuitka are down by a lot for some cases, very noticeably. There is never enough of this, but it appears, in many cases now, large compilations run much faster.

For macOS specifically, the new DLL dependency analysis, is much more capable or resolving conflicts all by itself. Many of the more complex packages with some variants of Python, specifically Anaconda will now be working a lot better.

And then, of course there is the big improvement for Onefile, that allows to use cached paths. This will make it more usable in the general case, e.g. where the firewall of Windows hate binaries that change their path each time they run.

Future directions will aim to make the compilation report more concise, and given reasons and dependencies as they are known on the inside more clearly, such that is can be a major tool for testing, bug reporting and analysis of the compilation result.

Nuitka Release 0.9

2022-07-02T08:17:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a many optimization improvements, and scalability improvements, while also adding new features, with also some important bug fixes.

Bug Fixes

Fix, hard module name lookups leaked a reference to that object. Fixed in 0.8.1 already.
Python2: Fix, str.decode with errors as the only argument wasn’t working. Fixed in 0.8.1 already.
Fix, could corrupt created uncompiled class objects __init__ functions in case of descriptors being used.
Standalone: Added support for newer torch. Fixed in 0.8.1 already.
Standalone: Added support for newer torchvision. Fixed in 0.8.1 already.
Fix, could compile time crash during initial parsing phase on constant dictionary literals with non-hashable keys.
```
{ {}:1, }
```
Fix, hard imported sub-modules of hard imports were falsely resolved to their parent. Fixed in 0.8.3 already.
```
importlib.resources.read_bytes() # gave importlib has no attribute...
```
Windows: Fix, outputs with --force-stdout-spec or --force-stderr-spec were created with the file system encoding on Python3, but they nee to be utf-8.
Fix, didn’t allow zero spaces in Nuitka project options, which is not expected.
Modules: Fix, the del __file__ in the top level module in module mode caused crashes at run time, when trying to restore the original __file__ value, after the loading CPython corrupted it.
Python2.6: Fixes for installations without pkg_resources.
Windows: Fix for very old Python 2.6 versions, these didn’t have a language assigned that could be used.
Security: Fix for CVE-2022-2054 where environment variables used for transfer of information between Nuitka restarting itself, could be used to execute arbitrary code at compile time.
Anaconda: Fix, the torch plugin was not working on Linux due to missing DLL dependencies.
Fix, static optimization of importlib.import_module with a package given, for an absolute import was optimized into the wrong import, package was not ignored as it should be.
Windows: Installed Python scan could crash on trying installation paths from registry that were manually removed in the mean time, but not through an uninstaller.
Standalone: Added missing implicit dependency for pyreadstat because parts of standard library it uses are no more automatically included.
Windows: Could still crash when no powershell is available with symlinks, handle this more gracefully.
Standalone: Added more missing Plotly dependencies, but more work will be needed to complete this.
Standalone: Add missing stdlib dependency on multiprocessing by concurrent.futures.process.
Standalone: Fix, implicit dependencies assigned to imageio on PIL plugins should actually be assigned to PIL.Image that actually loads them, so it works outside of imageio too.

New Features

UI: Added new option --user-package-configuration-file to allow users to provide extra Yaml configuration files for the Nuitka plugin mechanism to add hidden dependencies, anti-bloat, or data files, for packages. This will be useful for developing PRs to the standard file of Nuitka. Currently the schema is available, but it is not documented very well yet, so not really ready for end users just yet.
Standalone: Added new no-qt plugin as an easy way to prevent all of the Qt bindings from being included in a compilation.
Include module search path in compilation report.

Optimization

Faster dictionary iteration with our own replacement for PyDict_Next that avoids the DLL call overhead (in case of non-static libpython) and does less unnecessary checks.
Added optimization for str.count and str.format methods as well, this should help in some cases with compile time optimization.
The node for dict.update with only an iterable argument, but no keyword arguments, was in fact unused due to wrongly generated code. Also the form with no arguments wasn’t yet handled properly.
Scalability: Use specialized nodes for pair values, i.e. the representation of x = y in e.g. dictionary creations. With constant keys, and values, these avoid full constant value nodes, and therefore save memory and compile time for a lot of code.
Anti-Bloat: Added more scalability work to avoid including modules that make compilation unnecessarily big.
Python3.9+: Faster calls in case of mixed code, i.e. compiled code calling uncompiled code.
Removing duplicates and non-existent entries from modules search path should improve performance when locating modules.
Optimize calls through variables as well. These are needed for the package resource nodes to properly resolve at compile time from their hard imports to the called function.
Hard imported names should also be considered very trusted themselves, so they are e.g. also optimized in calls.
Anti-Bloat: Avoid more useless imports in Pandas, Numba, Plotly, and other packages, improving the scalability some more.
Added dedicated nodes for pkg_resources.require, pkg_resources.get_distribution, importlib.metadata.version, and importlib_metadata.version, so we can use compile time optimization to resolve their argument values where possible.
Avoid annotating control flow escape for all release statements. Sometimes we can tell that __del__ will not execute outside code ever, so this then avoids marking values as escaped, and taking the time to do so.
Calls of methods through variables on str, dict, bytes that have dedicated nodes are now also optimized through variables.
Boolean tests through variables now also are optimized when the original assignment is a compile time constant that is not mutable. This is only basic, but will allow tests on TYPE_CHECKING coming from a from typing import TYPE_CHECKING statement to be optimized, avoiding this overhead.

Cleanups

Changed to torch plugin to Yaml based configuration, making it obsolete, it only remains there for a few releases, to not crash existing build scripts.
Moved older package specific hacks to the Yaml file. Some of these were from hotfixes where the Yaml file wasn’t yet used by default, but now there is no need for them anymore.
Removed most of the pkg-resources plugin work. This is now done during optimization phase and rather than being based on source code matches, it uses actual value tracing, so it immediately covers many more cases.
Continued spelling improvements, renaming identifiers used in the source that the cspell based extension doesn’t like. This aims at producing more readable and searchable code.
Generated attribute nodes no longer do local imports of the operation nodes they refer to. This also avoids compile time penalties during optimization that are not necessary.
Windows: Avoid useless bytecode of inline copy used by Python3 when installing for Python2, this spams just a lot of errors.

Organizational

Removed MSI installers from the download page. The MSI installers are discontinued as Python has deprecated their support for them, as well as Windows 10 is making it harder for users to install them. Using the PyPI installation is recommended on Windows.
Merged our Yaml files into one and added schema description, for completion and checking in Visual Code while editing. Also check the schema in check-nuitka-with-yamllint which is now slightly misnamed. The schema is in no way final and will see improvements in future releases.
UI: Nicer progress bar layout that avoids flicker when optimizing modules.
UI: When linking, output the total number of object files used, to have that knowledge after the progress bar for C compilation is gone.
Quality: Auto-format the package configuration Yaml file for anti-bloat, implicit dependencies, etc.
GitHub: Point out the commit hook in the PR template.
UI: Nicer output in case of no commercial version is used.
Updated the WinLibs MinGW64 download used on Windows to the latest version based on gcc 11, the gcc 12 is not yet ready.
Git: Make sure we are not affected by core.autocrlf setting, as it interferes with auto-format enforcing Unix newlines.
Removed the MSI downloads. Windows 10 has made them harder to install and Python itself is discontinuing support for them, while often it was only used by beginners, for which it was not intended.
Anaconda: Make it more clear how to install static libpython with precise command.
UI: Warn about using module from Debian packages. These can be made non-portable to other OSes. Read more on the info page for detailed information.
Quality: The auto-format now floats imports to the top for consistency. With few exceptions, it was already done like this. But it makes things easier for generated code.

Tests

The reflected test was adapted to preserve PYTHONPATH now that module presence influences optimization.

Summary

This release marks a point, that will allow us to open up the compatibility work for implicit dependencies and anti-bloat stuff even further. The Yaml format will need documentation and potentially more refinement, but will open up a future, where latest packages can be supported with just updating this configuration.

The scalability improvements really make a difference for many libraries and are a welcome improvement on both memory usage and compile time. They are achieved by an accord of static optimization of

One optimization aimed at optimizing tuple unpacking, was not finished in time for this release, but will be subject of a future release. It has driven many other improvements though.

Generally, also from the UI, this is a huge step forward. With links to the website for complex topics being started, and the progress bar flicker being removed, the tool has yet again become more user friendly.

Next Nuitka Live Stream

2022-05-27T00:00:00+02:00

In the coming weekend, Saturday 04.06.2022, there will be the next live streams of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 10-14 CEST and potentially also one in the evening.

Last time

It’s actually been too long for me to remember what I did, but I do know that it took some time to get it right. It probably was about caching, but there are usually several topics.

I do however think that I need to use my CI machine with more processing power rather than my normal desktop, or maybe even both, because the screen capture, compiling, Visual Studio, and all, may make the Video otherwise stutter a bit too much.

This time

So, streaming is a huge success or Nuitka, however time is also scarce, and me starting new things, risks the release being bogged down by stabilization phases that are extended.

Not sure yet, what to do. I think, I want to look at adding the int as a C type and/or specializing code for mix of CLONG (C long value) and LONG (Python3 int object) and INT (Python2 int object). Right now adding 1 goes through a constant object, one that is prepared, and not time consuming to reference, but still overhead to use.

How to Join Nuitka Live

There is a dedicated page on the web site which has the details. Spoiler, it’s free and I have no plans for anything that involves a subscription of any kind. Of course, talking of subscription, do also checkout the Nuitka commercial offering. That is a subscription with adds that protect your IP even more than regular Nuitka.

Join me

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

Nuitka Release 0.8

2022-05-26T10:54:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release has a massive amount of bug fixes, builds on existing features, and adds new ones.

Bug Fixes

Windows: Fix, need to ignore cases where shorting in path for external use during compilation gives an permission error. Fixed in 0.7.1 already.
Compatibility: Added workaround for scipy.stats function copying. Fixed in 0.7.1 already.
Windows: Fix, detect ARM64 arch of MSVC properly, such that we can give a proper mismatch for the Python architecture. Fixed in 0.7.1 already.
Standalone: Fix, using importlib.metadata module failed to include email from standard library parts no longer included by default. Fixed in 0.7.1 already.
macOS: Fix, the dependency parser was using normalized paths where original paths must be used. Fixed in 0.7.1 already.
Standalone: Fix, using shiboken6 module (mostly due to PySide6) failed to include argparse from the standard library from standard library parts no longer included by default. Fixed in 0.7.1 already.
Onefile: Fix, the detection of a usable Python for compression could crash. Fixed in 0.7.2 already.
Onefile: Adding the payload on Windows could run into locks still being held, need to wait in that case. Fixed in 0.7.2 already.
Fix, need to include pkg_resources as well, we need it for when we use Jinja2, which is more often now. For Python3 this was fixed in 0.7.3 already. Later a version to use with Python2 was added as well.
Release: The wheels built for Nuitka when installed through URLs were not version specific, but due to different inline copies per OS and Python version, they must not be reused. Therefore we now pretend to contain an extension module, which handles that. Fixed in 0.7.3 already.
Standalone: Fix, using urllib.requests module failed to include http.client from standard library parts no longer included by default. Fixed in 0.7.3 already. Later http.cookiejar was added too.
Standalone: Do not compress MSVC run time library when using upx plugin, that is not going to work. Fixed in 0.7.4 already.
Standalone: Fix, on Windows more files should be included for TkInter to work with all software. Fixed in 0.7.5 already.
Distutils: Added support for package_dir directive to specify where source code lives. Fixed in 0.7.6 already.
Standalone: Fix, using shelve module failed to include dbm from standard library parts no longer included by default. Fixed in 0.7.6 already.
Standalone: Added support for arcade data files. Fixed in 0.7.7 already.
Standalone: Fix, bytecode demotions should use relative filenames rather than original ones. Fixed in 0.7.7 already.
Standalone: Fix, must remove extension module objects from sys.modules before executing an extension module that will create it. This fixes cases of cyclic dependencies from modules loaded by the extension module.
Windows: In case of an exception, clcache was itself triggering one during its handling, hiding the real exception behind a TypeError.
Windows: Improved clcache locking behavior, avoiding a race. Also increase characters used for key from 2 to 3 chars, making collisions far more rare.
Standalone: Added support for persistent package.
Standalone: Added support for newer tensorflow package.
Module: Fix, need to restore the __file__ and __spec__ values of top level module. It is changed by CPython after import to an incompatible file name, and not our loader, preventing package resources to be found.
Standalone: Added support for Crpytodome.Cipher.PKCS1_v1_5.
Fix, pkgutil.iter_modules without arguments was not working properly with our meta path based loader.
Windows: Fix, could crash on when the Scons report was written due to directories in PATH that failed to encode.
Compatibility: Fix, positive divmod and modulo % with negative remainders of positive floats was not correct.
Fix, str.encode with only errors value, but default value for encoding was crashing the compilation.
Python3.10+: Fix, match statement sliced values must be lists, not tuples.
Standalone: Added support for newer glfw and OpenGL packages.
Python3: Fix, failed to read bytecode only stdlib files. This affect mostly Fedora Python which does this for encodings.
Python3.5+: Fix, two phase loading of modules could release it immediately.
Standalone: Added missing dependency for pydantic.
Fix, the str.split rejected default sep value with only maxsplit given as a keyword argument.
Standalone: Added missing dependency of wsgiref module.
Standalone: Added support for falcon module.
Standalone: Added support for eliot module.
Fix, need to mark assigned from variables as escaped. Without it, some aliased loop variables could be misunderstood and falsely statically optimized.
Standalone: Added support for newer uvicorn package.
Standalone: Added data files for the accessible_output2, babel, frozendict, and sound_lib package.
Standalone: Added support for newer sklearn package.
Standalone: Added support for tkinterdnd2 package.
Python3.7+: Fix, the error message wasn’t fully compatible for unsubscriptable type exception messages.
Standalone: Fix, idlelib from stdlib was always ignored.
Python3.4+: Fix, the __spec__.origin as produced by find_spec of our meta path based loader, didn’t have the correct origin attribute value.
Standalone: Disable QtPDF plugin of PySide 6.3.0, because it’s failing dependency checks. On macOS this was blocking, we will change it to detection if that is necessary in a future release.
Standalone: Added support for orderedmultidict.
Standalone: Added support for clr module.
Standalone: Added support for newer cv2 package.

New Features

Added new option --module-name-choice to select what value __name__ and __package__ are going to be. With --module-name-choice=runtime (default for --module mode), the created module uses the parent package to deduce the value of __package__, to be fully compatible. The value --module-name-choice=original (default for other modes) allows for more static optimization to happen.
Added support for get_resource_reader to our meta path based loader. This allows to avoid useless temporary files in case importlib.resources.path is used, due to a bad interaction with the fallback implementation used without it.
Added support for --force-stdout-spec and --force-stderr-spec on all platforms, this was previously limited to Windows.
Added support for requiring and suggesting modes. In part this was added to 0.7.3 already, and is used e.g. to enforce that on macOS the wx will only work as a GUI program and crash unless --disable-console is specified. These will warn the user or outright error the compilation if something is known to be needed or useful.
Debian: Detect version information for “Debian Sid”. Added in 0.7.4 already, and also improved how Debian/Ubuntu versions are output.
Added new option --noinclude-data-files to instruct Nuitka to not include data files matching patterns given. Also attached loggers and tags to included data file and include them in the compilation report.

Distutils: When using pyproject.toml without setup.py so far it was not possible to pass arguments. This is now possible by adding a section like this.

[tool.nuitka]
# options without an argument are passed as boolean value
show-scons = true

# options with single values, e.g. enable a plugin of Nuitka
enable-plugin = "pyside2"

# options with several values, e.g. avoiding including modules, accepts
# list argument.
nofollow-import-to = ["*.tests", "*.distutils"]

The option names are the same, but without leading dashes. Lists are only needed when passing multiple values with the same option.

macOS: Add support for specifying signing identity with --macos-sign-identity and access to protected resources --macos-app-protected-resource.
Included data files are now reported in the compilation report XML as well.
Accept absolute paths for searching paths of binaries. This allows e.g. the upx plugin to accept both a folder path and the full path including the binary name to work when you specify the binary location with --upx-binary making it more user friendly.
Python3.10: Added support for positional matching of classes, so far only keyword matching was working.
Added support for path spec values %CACHE_DIR, %COMPANY%, %PRODUCT%, %VERSION%, and %HOME in preparation of onefile once again being able to be cached and not unpacked repeatedly for each execution.
Standalone: Detect missing tk-inter plugin at run time. When TCL fails to load, it then outputs a more helpful error. This ought to be done for all plugins, where it’s not clear if they are needed.
Anti-Bloat: Added support for plain replacements in the anti-bloat.yml file. Before with replacement`, the new value had to be produced by an eval, which makes for less readable values due to extra quoting. for plain values.

Optimization

Python3.10+: Added support for ordered-set PyPI package to speed up compilation on these versions too, adding a warning if no accelerated form of OrderedSet is used, but believed to be usable.
Optimization: Added bytes.decode operations. This is only a start and we needed this for internal usage, more should follow later.
Much more anti-bloat work was added. Avoiding ipython, unittest, and sometimes even doctest usage for some more packages.
The ccache was not always used, sometimes it believed to catch a potential race, that we need to tell it to ignore. This will speed up re-compilation of the C side in many cases.
Do not compile the meta path based loader separate, which allows us to not expose functions and values only used by it. Also spares the C compiler one file.
Added various dedicated nodes for querying package resources data, e.g. pkgutil.get_data. This will make it easier to detect cases of missing data files in the future.
Added more hard imports, some of which help scalability in the compilation, because these are then known to exist in standalone mode, others are used for package resource specific operations.
Onefile: Releasing decompression buffers avoiding unnecessary high memory usage of bootstrap binary.
Standalone: Avoid proving directories with no DLLs (e.g. from packages) towards ldd, this should avoid exceeding command line limits.
For clcache remove writing of the stats file before Scons has completed, which avoids IO and locking churn.
Standalone: Avoid including wsgiref from stdlib by default.

Cleanups

Removed references to chrpath and dead code around it, it was still listed as a dependency, although we stopped using it a while ago.
Removed uses of anti-bloat in examples and tests, it is now enabled by default.
Made standard plugin file naming consistent, their name should be *Plugin.py.
Cleaned up tensorflow plugin. The source modification was moved to anti-bloat where it is easy to do. The implicit dependencies are now in the config file of implicit-imports plugin.
Massive cleanups of data file handling in plugins. Adding methods for producing the now required objects.
The Scons file handling got further cleaned up and unified, doing more things in common code.
Avoid #ifdefs usages with new helper function Nuitka_String_FromFormat that implies them for more readable code.
Split the allowance check from the encountering. Allow plugins and options all to say if an import should be followed, and only when that is decided, to complain about it. Previously the attempt was causing an error, even if another plugin were to decide against it later.
Python2.6: Avoid warnings from MSVC for out specialized long code. In testing it worked correctly, but this is more explicit and doesn’t rely on C implementation specific behavior, although it appears to be universal.

Organizational

UI: Warning tests are now wrapped to multiple lines if necessary. That makes it more accessible for larger messages that contain more guiding information.
Documented how to use local Nuitka checkout with pyproject.toml files, this makes debugging Nuitka straightforward in these setups.
Added instructions on how to pass extra C and linker flags and to the User Manual.
Made our auto-format usable for the Nuitka website code too.
Removed dependencies on chrpath and the now dead code that would use it, we are happy with patchelf.
Updated to latest versions of requirements for development, esp. black and pylint.
Renamed --macos-onefile-icon to --macos-app-icon because that is what it is really used for.
Unified how dependencies are installed in GitHub actions.
Updated man page contents for option name changes from last releases.
Updated the WinLibs MinGW64 download used on Windows to the latest version.
Updated the ccache binary used on Windows with MinGW64. This is in preparation of using it potentially for MSVC as well.
Updated Visual Code C config to use Python3.10 and MSVC 2022 include paths.

Tests

Better outputs from standalone library compilation test, esp. when finding a problem, present the script to reproduce it immediately.
Enhanced generated tests to cover str methods to use keyword arguments.
Added automatic execution of pyproject.toml driven test case.
Enhanced output in case of optimization test failures, dumping what value is there that has not become a compile time constant.

Summary

This release has seen a lot of consolidation. The plugins layer for data files is now all powerful, allowing much nicer handling of them by the plugins, they are better reported in normal output, and they are also part of the report file that Nuitka can create. You may now also inhibit their inclusion from the command line, if you decide otherwise.

The pyproject.toml now supporting Nuitka arguments is closing an important gap.

Generally many features got more polished, e.g. non-automatic inclusion of stdlib modules has most problems fixed up.

An important area of improvement, are the hard imports. These will be used to replace the source based resolution of package requirements with ones that are proper nodes in the tree. Getting these hard imports to still retain full compatibility with existing imports, that are more or less __import__ uses only, was revealing quite a bit of technical debt, that has been addressed with this release.

For onefile, the cached mode is being prepared with the variables added, but will only be in a later release.

Also a bunch of new or upgraded packages are working now, and the push for anti-bloat work has increased, making many compilations even more lean, but scalability is still an issue.

Compile Python on Windows

2022-05-15T07:46:50+02:00

Looking to create an executable from Python script? Let me show you the full steps to achieve it on Windows.

Steps to create a Windows executable from a Python script using Nuitka

Step 1: Add Python to Windows Path

The simple way to add Python to the PATH do this is to check the box during installation of CPython. You just download python and install or modify Python by checking the box in the installer:

This box is not enabled by default. You can also manually add the Python installation path to PATH environment variable.

Note

You do not strictly have to execute this step, you can also replace python with just the absolute path, e.g. C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe but that can become inconvenient.

Step 2: Open a Windows Prompt

This can be cmd.exe or Windows Terminal, or from an IDE like Visual Code or PyCharm. And then type python to verify the correct installation, and exit to leave the Python prompt again.

Step 3: Install the Nuitka Python Compiler package

Now install Nuitka with the following command.

python -m pip install Nuitka

Step 4: Run your Program

Now run your program from the terminal. Convince yourself that everything is working.

python fancy-program.py

Note

If it’s a GUI program, make sure it has a .pyw suffix. That is going to make Python know it’s one.

Step 5: Create the Executable using Nuitka

python -m nuitka --onefile fancy-program.py

In case of a terminal program, add one of many options that Nuitka has to adapt for platform specifics, e.g. program icon, and so on.

python -m nuitka --onefile --windows-disable-console fancy-program.py

This will create fancy-program.exe.

Step 6: Run the Executable

Your executable should appear right near fancy-program.py and opening the explorer or running fancy-program.exe from the Terminal should be good.

fancy-program.exe

Nuitka Release 0.7

2022-02-21T13:10:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is massively improving macOS support, esp. for M1 and the latest OS releases, but it also has massive improvements for usability and bug fixes in all areas.

Bug Fixes

Fix, set creation wasn’t annotating its possible exception exit from hashing values and is not as free of side effects as list and tuple creations are. Fixed in 0.6.19.1 already.
Windows: Fix, --experimental option values got lost for the C compilation when switching from MSVC to MinGW64, making them have no effect. Fixed in 0.6.19.1 already.
Windows: Fix, Clang from MinGW64 doesn’t support LTO at this time, therefore default to no for it. Fixed in 0.6.19.1 already.
Debian: Fix, failed to detect Debian unstable as suitable for linking, it doesn’t have the release number. Fixed in 0.6.19.1 already.
Standalone: Added data files for pygsheets package. Fixed in 0.6.19.2 already.
Fix, paths from plugin related file paths need to be made absolute before used internally, otherwise the cache can fail to deduplicate them. Fixed in 0.6.19.2 already.
Python3: With gcc before version 5, e.g. on CentOS 7, where we switch to using g++ instead, the gcc version checks could crash. Fixed in 0.6.19.2 already.
Windows: Disable MinGW64 wildcard expansion for command line arguments. This was breaking command lines with arguments like --filename *.txt, which under cmd.exe are left alone by the shell, and are to be expanded by the program. Fixed in 0.6.19.2 already.
Standalone: Added missing implicit dependency needed for --follow-stdlib with Python for some uses of the locale module. Fixed in 0.6.19.2 already.
Standalone: Added workarounds for newest numpy that wants to set __code__ objects and required improvements for macOS library handling. Fixed in 0.6.19.3 already.
Windows: Caching of DLL dependencies for the main programs was not really working, requiring to detect them anew for every standalone compilation for no good reason. Fixed in 0.6.19.3 already.
Windows: Fix, CTRL-C from a terminal was not propagated to child processes on Windows. Fixed in 0.6.19.4 already.
Standalone: With certifi and Python3.10 the importlib.resource could trigger Virus scanner inflicted file access errors. Fixed in 0.6.19.4 already.
Python3.10: Reverted error back iteration past end of generator change for Python 3.10.2 or higher to become compatible with that too. Fixed in 0.6.19.5 already.
Standalone: Added support for anyio and by proxy for Solana. Fixed in 0.6.19.5 already.
Fix, compilation with resource mode incbin and --debugger was not working together. Fixed in 0.6.19.5 already.
Fix, format optimization of known str objects was not properly annotating an exception exit when being optimized away, causing consistency checks to complain. Fixed in 0.6.19.5 already.
Windows: Fix, clcache didn’t work for non-standard encoding source paths due to using th direct mode, where wrong filenames are output by MSVC. Fixed in 0.6.19.5 already.
Windows: Fix, ccache cannot handle source code paths for non-standard encoding source paths. Fixed in 0.6.19.5 already.
Python2.6: Fix, calls to iteritems and iterkeys on known dictionary values could give wrong values. Fixed in 0.6.19.5 already.
Fix, the value of __module__ if set by the metaclass was overwritten when creating types. Fixed in 0.6.19.6 already.
Plugins: Add support for the latest version of pkg_resources that has “vendored” even more packages. Fixed in 0.6.19.6 already.
Onefile: The onefile binary was locked during run time and could not be renamed, preventing in-place updates. This has been resolved and now on Windows, the standard trick for updating a running binary of renaming it, then placing the new file works.
Fix, wasn’t checking the zstandard version and as a result could crash if too old versions of it. This is now checked.
macOS: Large amounts of bug fixes for the dependency scanner. It got cleaned up and now handles many more cases correctly.
Windows: Fix, was not properly detecting wrong ClangCL architecture mismatch with the Python architecture. This could result in strange errors during C compilation in this setup.
Standalone: Added implicit dependencies for the asyncpg module.
Linux: Detect Debian or Ubuntu base and distribution name more reliably. This helps esp. with static libpython optimization being recognized automatically.

New Features

We now disallow options that take arguments to be provided without using =.
Previously --lto no worked just as well as --lto=no did. And that was the cause of problems when --lto first became a choice.

Recently similar, but worse problems were observed, where e.g. --include-module could swallow trailing other arguments when users forgot to specify the name by accident. Therefore this style of giving options is now explicitly rejected.
Compiled types of Nuitka now inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses PyObject_IsInstance work, e.g. pydantic.
macOS: Added signing of application bundles and standalone binaries for deployment to newer macOS platforms and esp. M1 where these are mandatory for execution.
macOS: Added support for selecting the single macOS target arch to create a binary for. The universal architecture is not yet supported though, but will be added in a future release.
Added support for compression in onefile mode through the use of an other Python installation, that has the zstandard module installed. With this it will work with 2.6 or higher, but require a 3.5 or higher Python with it installed in either PATH or on Windows in the registry alternatively.
Added UPX plugin to compress created extension modules and binaries and for standalone mode, the included DLLs. For onefile, the compression is not useful since it has the payload already compressed.
Added a more explicit way to list usable MSVC versions with --msvc=list rather than requiring an invalid value. Check values given in the same way that Scons will do.
Added support for --python-flag=-u which disabled outputs buffers, so that these outputs are written immediately.
Plugins: Always on plugins now can have command line options. We want this for the anti-bloat plugin that is enabled by default in this release.
Plugins: Added ability for plugin to provide fake dependencies for a module. We want the this for the multiprocessing plugin, that is now enabled by default in this release too.
Plugins: Added ability for plugins to modify DLLs after copy for standalone. We will be using this in the new upx plugin.
Added retry for file copies that fail due to still running program. This can happen on Windows with DLLs in standalone mode. For interactive compilation, this allows a retry to happen after prompting the user.
UI: Added ability to list MSVC versions with --msvc=list, and detect illegal values given to --msvc= before Scons sees them, it also crashes with a relative unhelpful error message.
UI: When linking, close the C compilation progress bar and state that that linking is going on. For some very large LTO compilations, it was otherwise at 100% and still taking a long time, confusing users.
Plugins: Added new plugin that is designed to handle DLL dependencies through a configuration file that can both handle filename patterns as well as code provided DLL locations.
Optimization: Exclude parts of the standard library by default. This allows for much smaller standalone distributions on modules, that can be expected to never be an implicit dependency of anything, e.g. argparse or pydoc.

Optimization

Standalone: Do not include encodings.bz2_codec and encodings.idna anymore, these are not file system encodings, but require extension modules.
Make sure we use proper (void) arguments for C functions without arguments, as for C functions, that makes a real difference, they are variable args functions and more expensive to call otherwise.
For standalone, default to using --python-flag=no_site to avoid the overhead that the typically unused site module incurs. It often includes large parts of the standard library, which we now want to be more selective about. There is new Python flag added called --python-flag=site that restores the inclusion of site module.
Standalone: Exclude non-critical codec modules from being technical, i.e. have to be available at program startup. This removes the need for e.g. bz2 related extension modules previously included.
In reformulations, use dictionary methods directly, we have since introduced dictionary specific methods, and avoid the unnecessary churn during optimization.
The complex call helper could trigger unnecessary passes in some cases. The pure functions were immediately optimized, but usages in other modules inside loops sometimes left them in incomplete states.
Windows: Avoid repeated hashing of the same files over and over for clcache.
Cache dependencies of bytecode demoted modules in first compile and reuse that information in subsequent compilations.
Linux: Added option for switching compression method for onefile created with AppImage. The default is also now gzip and not xz which has been observed to cause much slower startup for little size gains.
Standalone: For failed relative imports, during compiled time absolute imports were attempted still and included if successful, the imports would not be use them at run time, but lead to more modules being included than necessary.

Organizational

There is now a Discord server for Nuitka community where you can hang out with the developers and ask questions. It is mirrored with the Gitter community chat, but offers more features.
The anti-bloat is now on by default. It helps scalability by changing popular packages to not provide test frameworks, installation tools etc. in the resulting binary. This oftentimes reduces the compilation by thousands of modules.
Also the multiprocessing plugin is now on by default. Detecting its need automatically removes a source of problems for first time users, that didn’t know to enable it, but had all kinds of strange crashes from multiprocessing malfunctioning. This should enhance the out of the box experience by a lot.
With this release, the version numbering scheme will be changed. For a long time we have used 4 digits, where one is a leading zero. That was initially done to indicate that it’s not yet ready. However, that is just untrue these days. Therefore, we switch to 3 digits, and a first hotfix with now be 0.7.1 rather than 0.6.19.1, which is too long.

It has been observed that people disregard differences in the third digit, but actually for Nuitka these have oftentimes been very important updates. This change is to rectify it, and a new release will be 0.8, and there will be a 1.0 release after 0.9.
Added a new section to User Manual that explains how to manually load files, such that it is cleaner and compatible code. Using paths relative to current directory is not the right way, but there are nice helpers that make it very simple and correct with all kinds of contexts.
Report the MSVC version in Scons output during compilation. The 2022 version is required, but we support everything back to 2008, to work on very old systems as well. This will help identifying differences that arise from there.
Quality: Find Clang format from MSVC 2022 too. We use in auto format of Nuitka source code, but need to also search that as a new path.
Added a spellchecker extension for Visual Code, resulting in many spelling fixes in all kinds of documentation and code. This finds more things than codespell, but also has a lot of false alarms.
Check value of --onefile-tempdir-spec for typical user errors. It cannot be . as that would require to overwrite the onefile binary on Windows, and will generally behave very confusing. Warn about absolute or relative paths going outside of where the binary lives. Can be useful in controlled setups, but not generally. Also warn about using no variables, making non-unique paths.
macOS: Flavor detection was largely expanded. The Apple flavor is recognized on more systems. Homebrew was newly added, and we actually can detect CPython reliably as a first.
Added a tool from leo project to create better .pyi files for modules. We will make use of it in the future to enhance the files created by Nuitka to not only contain hidden dependencies, but optionally also module signatures.
Plugins: Clearer information from pyside2 that patched wheels might be mandatory and workarounds only patches cannot be done for older Python.
Added progress bars for DLL dependency detection and DLL copying for standalone. These both can end up using take a fair bit of time depending on project size, and it’s nice to know what’s going on.
macOS: Added support for using both --onefile and --macos-create-app-bundle as it is needed for PySide2 due to issues with signing code now.
Added warning when attempting to include extension modules in an accelerated compilation.
Modules: Catch the user error of following all imports when creating a module. This is very unlikely to produce usable results.
Start integrating Sourcery for improved Nuitka code. It will comment on PRs and automatically improve Nuitka code as we develop it.
Debugging: Added command line tool find-module that outputs how Nuitka locates a module in the Python environment it’s ran with. That removes the need to use Python prompt to dump __file__ of imported modules. Some modules even hide parts of their namespace actively during run-time, the tool will not be affected by that.

Cleanups

Refactored Python scan previously used for Scons usage on versions that need to run in with another Python to be more generally usable.
Use explicit nuitka.utils.Hashing module that allows the core to perform these operations with simpler code.
macOS: Use isPathBelow for checking if something is a system library for enhanced robustness and code clarity.
macOS: Make sure to use our proper error checking wrappers for command execution when using tools like otool or codesign.
Standalone: Avoid a temporary file with a script during technical import detection. These have been observed to potentially become corrupted, and this avoids any chance of that happening, while also being simpler code.
Avoid naming things shlib and call them extension instead. Inspired by the spell checker disliking the former term, which is also less precise.
Removed the dead architecture selection option for Windows, it was unused for a long time.
Moved Windows SxS handling of DLLs to a more general place where also macOS specific tasks are applied, to host standard modification of DLLs during their copying.

Tests

Better matching of relative filenames for search modes of the individual test suite runners.
Debugger outputs on segfaults were no longer visible and have been restored.

Summary

This release is tremendous progress for macOS. Finally biting the bullet and paying obscene amounts of money to rent an M1 machine, it was possible to enhance the support for this platform. Currently typical packages for macOS are being made compatible as well, so it can now be expected to perform equally well.

On the quality side, the spell checker has had some positive effects, finding typos and generally misspelled code, that codespell does not, due to it being very conservative.

The trend to enhance plugins has continued. The copying of DLLs is very nearly finalized. Making more plugins enabled by default is seeing a lot of progress, with 2 important ones addressed.

Work on the size of distributions has seen a lot of positive results, in that now standalone distributions are often very minimal, with many extension modules from standard library no longer being present.

Next Nuitka Live Stream

2022-01-30T00:00:00+01:00

Today, Sunday 30.01.2022, there will be the third live stream of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 9-12 CEST and from 18 CEST until probably at least 20 CEST, but it seems I tend to go overtime.

Last time

So the last two streams are on my Youtube, these are around 4h videos, done on the same day.

In the second stream I was working on Onefile compression for all Python versions, by making the location of another Python that for Scons we sort of did already, reusable. So now, you can run Nuitka with Python2.6 and compile your legacy code, while with say 3.6 also installed on the same system, you do the zstandard compression. That will be used for other things in the future as well. This is going to be part of the next release and currently on develop.

In the first stream, I also did a little bit of performance plans, but mostly only showing people what I have there in stock, not actually went there and I started work on a upx plugin, such that DLLs for standalone are compressed with that. That also required a bit of plugin interface changes and research. This one works mostly, but will need more live. Also I think I looked at reducing what is included with the follow standard library option, getting very minimal distributions out of it.

This time

All around, this last stream, was a huge success. I was a bit under the weather last weekend, but we go on now.

Not sure yet, what to do. I might be debugging issues I have with 2.6 and a recent optimization, that prevents factory branch from becoming the next pre-release, I might be looking at macOS still. And I might be looking at the caching of bytecode demoted modules, that is kind of ready to be used, but is not currently, which is a pity. And of course, the Python PGO may get a closer look, such that e.g. it works for standalone mode.

How to Join

Join me

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

First Nuitka Live Stream

2022-01-14T00:00:00+01:00

Tomorrow, Saturday 15.01.2022, there will be not one, but two live stream of me coding on Nuitka, and talking and chatting with visitors in the Discord channel created specifically for this. I will go from 9-12 CEST and from 17-20 CEST.

I want to do this on a regular basis, and I will be happy to do this even when nearly nobody shows up. It will be however your chance, and in previous GSoC and generally, I made very good experiences with joint sessions. So I hope you will help me pasting code snippets and Google results in chat, and discuss the optimization that we are going to add.

Come and join me there. Instructions here. You know you want to do it. I know I want you to do to it!

Yours,

Kay

Nuitka Release 0.6.19

2022-01-12T10:51:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release adds support for 3.10 while also adding very many new optimization, and doing a lot of bug fixes.

Bug Fixes

Calls to importlib.import_module with expressions that need releases, i.e. are not constant values, could crash the compilation. Fixed in 0.6.18.1 already.
After a fix for the previous release, modules that fail to import are attempted again when another import is executed. However, during this initialization for top level module in --module mode, this was was done repeatedly, and could cause issues. Fixed in 0.6.18.1 already.
Standalone: Ignore warning given by patchelf on Linux with at least newer OpenSUSE. Fixed in 0.6.18.1 already.

Fix, need to avoid computing large values out of << operation as well. Fixed in 0.6.18.2 already.

# This large value was computed at run time and then if used, also
# converted to a string and potentially hashed, taking a long time.
1 << sys.maxint

Standalone: Ignore warning given by patchelf on Linux about a workaround being applied.
Fix, calls to importlib.import_module were not correctly creating code for dynamic argument values that need to be released, causing the compilation to report the error. Fixed in 0.6.18.1 already.
MSYS2: Fix, the console scripts are actually good for it as opposed to CPython, and the batch scripts should not be installed. Fixed in 0.6.18.2 already.
Setuptools: Added support older version of setuptools in meta build integration of Nuitka.
Fix, calls to importlib.import_module with 2 arguments that are dynamic, were not working at all. Fixed in 0.6.18.2 already.
Windows: Compiling with MinGW64 without ccache was not working due to issues in Scons. Fixed in 0.6.18.2 already.
Fix, the repr built-in was falsely annotated as producing a str value, but it can be also derived or unicode in Python2.
Fix, attribute nodes were not considering the value they are looking up on. Now that more values will know to have the attributes, that was causing errors. Fixed in 0.6.18.2 already.
Fix, left shifting can also produce large values and needs to be avoided in that case, similar to what we do for multiplications already. Fixed in 0.6.18.2 already.
UI: The new option --disable-ccache didn’t really have the intended effect. Fixed in 0.6.18.3 already.
UI: The progress bar was causing tearing and corrupted outputs, when outputs were made, now using proper tqdm API for doing it, this has been solved. Fixed in 0.6.18.4 already.
Fix, the constant value sys.version_info didn’t yet have support for its type to be also a compile time constant in e.g. tuples. Fixed in 0.6.18.4 already.
Onefile: Assertions were not disabled, and on Windows with MinGW64 this lead to including the C filenames of the zstd inline copy files and obviously less optimal code. Fixed in 0.6.18.4 already.
Standalone: Added support for bottle.ext loading extensions to resolve at compile time. Fixed in 0.6.18.5 already.
Standalone: Added support for seedir required data file. Fixed in 0.6.18.5 already.
MSYS2: Failed to link when using the static libpython, which is also now the default for MSYS2. Fixed in 0.6.18.5 already.
Python3.6+: Fix, the intended finalizer of compiled asyncgen was not present and in fact associated to help type. This could have caused corruption, but that was also very unlikely. Fixed in 0.6.18.5 already.
Python3: Fix, need to set __file__ before executing modules, as some modules, e.g. newer PyWin32 use them to locate things during their initialization already.
Standalone: Handle all PyWin32 modules that need the special DLLs and not just a few.
Fix, some .pth files create module namespaces with __path__ that does not exist, ignore these in module importing.
Python2.6-3.4: Fix, modules with an error could use their module name after it was released.
Distutils: When providing arguments, the method suggested in the docs is not compatible with all other systems, e.g. not setuptools_rust for which a two elements tuple form needs to be used for values. Added support for that and documented its use as well in the User Manual.
Python3.7+: Do no longer allow deleting cell values, this can lead to corruption and should be avoided, it seems unlikely outside of tests anyway.
Standalone: Added support for more ciphers and hashes with pycryptodome and pycryptodomex, while also only including Ciphers when needed.
Distutils: Was not including modules or packages only referenced in the entry point definition, but not in the list of packages. That is not compatible and has been fixed.
Fix, must not expose the constants blob from extension modules, as loading these into a compiled binary can cause issues in this case.
Standalone: Added support for including OpenGL and SSL libraries with PySide2 and PySide6 packages.
Windows: Fix, the cmd files created for uninstalled Python and accelerated programs to find the Python installation were not passing command line arguments.
Windows: Executing modules with --run was not working properly due to missing escaping of file paths.
Fix, parsing .pyi files that make relative imports was not resolving them correctly.
Python3: Fix, when disabling the console on Windows, make sure the file handles still work and are not None.
Windows: Fix, need to claim all OS versions of Windows as supported, otherwise e.g. high DPI features are not available.

New Features

Programs that are to be executed with the -m flag, can now be compiled with --python-flag=-m and will then behave in a compatible way, i.e. load the containing package first, and have a proper __package__ value at run time.
We now can write XML reports with information about the compilation. This is initially for use in PGO tests, to decide if the expected forms of inclusions have happened and should grow into a proper reporting tool over time. At this point, the report is not very useful yet.
Added support for Python 3.10, only match statements are not completely supported. Variations with | matches that also assign are not allowed currently.
Windows: Allow using --clang with --mingw64 to e.g. use the clang.exe that is contained in the Nuitka automatic download rather than gcc.exe.
Added support for Kivy. Works through a plugin that is automatically enabled and needs no other inputs, detecting everything from using Kivy at compile time.
Added initial support for Haiku OS, a clone of BeOS with a few differences in their Python installation.
Added experimental plugin trio that works around issues with that package.

Optimization

Also trust hard imports made on the module level in function level code, this unlocks many more static optimization e.g. with sys.version_info when the import and the use are not on the same level.
For the built-in type method calls with generic implementation, we now do faster method descriptor calls. These avoid creating a temporary PyCFunction object, that the normal call slot would, this should make these calls faster. Checking them for compiled function, etc. was only wasteful, so this makes it more direct.
Loop and normal merge traces were keeping assignments made before the loop or inside a branch, that was otherwise unused alive. This should enable more optimization for code with branches and loops. Also unused loop traces are now recognized and removed as well.
Avoiding merges of escaped traces with the unescaped trace, there is no point in them. This was actually happening a lot and should mean a scalability improvement and unlock new optimization as well.
Avoid escaping un-init traces. Unset values need not be considered as potentially modified as that cannot be done.
The str shape is now detected through variables, this enables many optimization on the function level.
Added many str operation nodes.

These are specifically all methods with no arguments, as these are very generic to add, introduced a base class for them, where we know they all have no effect or raise, as these functions are all guaranteed to succeed and can be served by a common base class.

This covers the str.capitalize, str.upper, str.lower, str.swapcase, str.title, str.isalnum, str.isalpha, str.isdigit, str.islower, str.isupper, str.isspace, and str.istitle functions.

For static optimization str.find and str.rfind were added, as they are e.g. used in a sys.version.find(...) style in the os module, helping to decide to not consider OS/2 only modules.

Then, support for str.index and str.rindex was added, as these are very similar to str.find forms, only that these may raise an exception.

Also add support for str.split and str.rsplit which will be used sometimes for code needed to be compile time computed, to e.g. detect imports.

Same goes for endswith and startswith, the later is e.g. popular with sys.platform checks, and can remove a lot of code from compilation with them now being decided at compile time.

Note

A few str methods are still missing, with time we will achieve all of them, but this will take time.
Added trust for sys.builtin_module_names as well. The os module is using it to make platform determinations.
When writing constant values, esp. tuple, list, or dict values, an encoding of “last value” has been added, avoiding the need to repeat the same value again, making many values more compact.
When starting Nuitka, it usually restarts itself with information collected in a mode without the site module loaded, and with hash randomization disabled, for deterministic behaviour. There is a option to prevent this from happening, where the goal is to avoid it, e.g. in testing, say for the coverage taking, but that meant to parse the options twice, which also loads a lot of code.

Now only a minimal amount of code is used, and the options are parsed only on the restart, and then an error is raised when it notices, it was not allowed to do so. This also makes code a lot cleaner.
Specialized comparison code for Python2 long and Python3 int code, making these operations much faster to use.
Specialized comparison code for Python2 unicode and Python3 str code, making these operations much faster to use, currently only == and != are fully accelerated, the other comparisons will follow.
Enable static libpython with Python3 Debian packages too. As with Python2, this will improve the performance of the created binary a lot and reduce size for standalone distribution.
Comparisons with in and not in also consider value traces and go through variables as well where possible. So far only the rich comparisons and is and is not did that.
Create fixed import nodes in re-formulations rather than __import__ nodes, avoiding later optimization doing that, and of course that’s simpler code too.
Python 3.10: Added support for union types as compiled time constants.
Modules are now fully optimized before considering which modules they are in turn using, this avoids temporary dependencies, that later turn out unused, and can shorten the compilation in some cases by a lot of time.
On platforms without a static link library, in LTO mode, and with gcc, we can use the -O3 mode, which doesn’t work for libpython, but that’s not used there. This also includes fake static libpython, as used by MinGW64 and Anaconda on Windows.
The anti-bloat plugin now also handles newer sklearn and knows more about the standard library, and its runners which it will exclude from compilation if use for it. Currently that is not the default, but it should become that.

Organizational

Migrated the Nuitka blog from Nikola to Sphinx based ABlog and made the whole site render with Sphinx, making it a lot more usable.
Added a small presentation about Nuitka on the Download page, to make sure people are aware of core features.
The gi plugin is now always on. The copying of the typelib when gi is imported is harmless and people can disable the plugin if that’s not needed.
The matplotlib plugin is new and also always on. It previously was part of the numpy plugin, which is doing too many unrelated things. Moving this one out is part of a plan to split it up and have it on by default without causing issues.
MSYS2: Detecting MinGW and POSIX flavors of this Python. For the MinGW flavor of MSYS2, the option --mingw64 is now the default, before it could attempt to use MSVC, which is not going to work for it. And also the Tcl and Tk installations of it are being detected automatically for the tk-inter plugin.
Added Windows version to Nuitka version output, so we have this for bug reports.
User Manual: Added example explaining how to access values from your code in Nuitka project options.
UI: For Python flavors where we expect a static libpython, the error message will now point out how to achieve it for each flavor.
UI: Disable progress bar when --show-scons is used, it makes capturing the output from the terminal only harder.
UI: Catch error of specifying both --msvc= and --mingw64 options.
Distutils: Improved error messages when using setuptools or build integration and failing to provide packages to compile.
Plugins: Removed now unused feature to rename modules on import, as it was only making the code more complex, while being no more needed after recently adding a place for meta path based importers to be accounted for.
Twitter: Use embedded Tweet in Credits, and regular follow button in User Manual.
Warnings about imports not done, are now only given when optimization can not remove the usage, and no options related to following have been given.
Added Windows version to --version output of Nuitka. This is to more clearly recognize Windows 10 from Windows 11 report, and also the odd Windows 7 report, where tool chain will be different.
In Visual Code, the default Python used is now 3.9 in the “Linux” C configuration. This matches Debian Bullseye.
Nicer outputs from check mode of the auto-format as run for CI testing, displays problematic files more clearly.
Remove broken links to old bug tracker that is no longer online from the Changelog.
UI: When hitting CTRL-C during initial technical import detection, no longer ask to submit a bug report with the exception stack, instead exit cleanly.
Windows: Enable LTO mode for MinGW64 and other gcc by default. We require a version that can do it, so take advantage of that.
For cases, where code generation of a module takes long, make sure its name is output when CTRL-C is hit.
Windows: Splash screen only works with MSVC, added error indicator for MinGW64 that states that and asks for porting help.

Cleanups

Generate all existing C code for generic builtin type method calls automatically, and use those for method attribute lookups, making it easier to add more.
Changed TkInter module to data file providing interface, yielding the 2 directories in question, with a filter for demos.
The importing code got a major overhaul and no longer works with relative filenames, or filenames combined with package names, and module names, but always only with module names and absolute filenames. This cleans up some of the oldest and most complex code in Nuitka, that had grown to address various requirements discovered over time.
Major cleanup of Jinja2 template organisation.

Renamed all C templates from .j2 to .c.j2 for clarity, this was not done fully consistent before. Also move all C templates to nuitka.codegen package data, it will be confusing to make a difference between ones used during compile time and for the static generation, and the lines are going to become blurry.

Added Jinja2 new macro CHECK_OBJECTS to avoid branches on argument count in the call code templates. More of these things should be added.

Cleanup of code that generates header declarations, there was some duplication going on, that made it hard to generate consistent code.
Removed nuitka.finalizatios.FinalizationBase, we only have one final visitor that does everything, and that of course makes a lot of sense for its performance.
Major cleanup of the Scons C compiler configuration setup. Moved things to the dedicate function, and harmonized it more.
Resolved deprecation warnings given by with --python-debug for Nuitka.

Tests

Started test suite for Python PGO, not yet completely working though, it’s not yet doing what is needed though.
Added generated test that exercises str methods in multiple variations.
Revived reflected test suite, that had been removed, because of Nuitka special needs. This one is not yet passing again though, due to a few details not yet being as compatible as needed.
Added test suite for CPython 3.10 and enable execution of tests with this version on GitHub actions.

Summary

This release is another big step forward.

The amount of optimization added is again very large, some of which yet again unlocks more static optimization of module imports, that previously would have to be considered implicit. Now analyzing these on the function level as well, we can start searching for cases, where it could be done, but is not done yet.

After starting with dict, method optimization has focused on str which is esp. important for static optimization of imports. The next goal will here be to cover list which are important for run time performance and currently not yet optimized. Future releases will progress there, and also add more types.

The C type specialization for Python3 has finally progressed, such that it is also covering the long and unicode and as such not limited to Python2 as much. The focus now needs to turn back to not working with PyObject * for these types, but e.g. with += 1 to make it directly work with CLONG rather than LONG for which structural changes in code generation will be needed.

For scalability, the anti-bloat work has not yet progressed as much as to be able to enable it by default. It needs to be more possible to disable it where it causes problems, e.g. when somebody really wants to include pytest and test frameworks generally, that’s something that needs to be doable. Compiling without anti-bloat plugin is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.

The support for Windows has been further refined, actually fixing a few important issues, esp. for the Qt bindings too.

This release adds support for 3.10 outside of very special match statements, bringing Nuitka back to where it works great with recent Python. Unfortunately orderedset is not available for it yet, which means it will be slower than 3.9 during compilation.

Overall, Nuitka is closing many open lines of action with this. The setuptools support has yet again improved and at this point should be very good.