Nuitka Release 0.6.3

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This has a focus on organisational improvements. With more and more people joining Nuitka, normal developers as well as many GSoC 2019 students, the main focus was to open up the development tools and processes, and to improve documentation.

That said, an impressive amount of bug fixes was contributed, but optimization was on hold.

Bug Fixes

  • Windows: Added support for running compiled binaries in unicode path names.

  • Standalone: Added support for crytodomex and pycparser packages.

  • Standalone: Added support for OpenSSL support in PyQt on Windows.

  • Standalone: Added support for OpenGL support with QML in PyQt on Windows.

  • Standalone: Added support for SciPy and extended the NumPy plugin to also handle it.

  • UI: The option --plugin-list still needed a positional argument to work.

  • Make sure sys.base_prefix is set correctly too.

  • Python3: Also make sure sys.exec_prefix and sys.base_exec_prefix are set correctly.

  • Standalone: Added platform plugins for PyQt to the default list of sensible plugins to include.

  • Fix detection of standard library paths that include .. path elements.


  • Avoid static C++ runtime library when using MinGW64.

New Features

  • Plugins: A plugin may now also generate data files on the fly for a given module.

  • Added support for FreeBSD/PowerPC arch which still uses gcc and not clang.


  • Nuitka is participating in the GSoC 2019.

  • Added documentation on how to create or use Nuitka plugins.

  • Added more API doc to functions that were missing them as part of the ongoing effort to complete it.

  • Updated to latest PyLint 2.3.1 for checking the code.

  • Scons: Using newer Scons inline copy with Python 2.7 as, the old one remains only used with Python 2.6, making it easier to know the relevant code.

  • Autoformat was very much enhanced and handles C and ReST files too now. For Python code it does pylint comment formatting, import statement sorting, and blackening.

  • Added script misc/ that adds a commit hook that runs autoformat on commit. Currently it commits unstaged content and therefore is not yet ready for prime time.

  • Moved adapted CPython test suites to Github repository under Nuitka Organisation.

  • Moved Nuitka-website repository to Github repository under Nuitka Organisation.

  • Moved Nuitka-speedcenter repository to Github repository under Nuitka Organisation.

  • There is now a Gitter chat for Nuitka community.

  • Many typo and spelling corrections on all the documentation.

  • Added short installation guide for Nuitka on Windows.


  • Moved commandline parsing helper functions from common code helpers to the main program where of course their only usage is.

  • Moved post processing of the created standalone binary from main control to the freezer code.

  • Avoid using chmod binary to remove executable bit from created extension modules.

  • Windows: Avoid using rt.exe and mt.exe to deal with copying the manifest from the python.exe to created binaries. Instead use new code that extracts and adds Windows resources.

  • Fixed many ResourceWarnings on Python3 by improved ways of handling files.

  • Fixed deprecation warnings related to not using

  • The runners in bin directory are now formatted with black too.


  • Detect Windows permission errors for two step execution of Nuitka as well, leading to retries should they occur.

  • The salt value for CPython cached results was improved to take more things into account.

  • Tests: Added more trick assignments and generally added more tests that were so far missing.


With the many organisational changes in place, my normal work is expected to resume for after and yield quicker improvements now.

It is also important that people are now enabled to contribute to the Nuitka web site and the Nuitka speedcenter. Hope is to see more improvements on this otherwise neglected areas.

And generally, it's great to see that a community of people is now looking at this release in excitement and pride. Thanks to everybody who contributed!

Nuitka Release 0.6.2

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release has a huge focus on organizational things. Nuitka is growing in terms of contributors and supported platforms.

Bug Fixes

  • Fix, the Python flag --python-flag=-O was removing doc strings, but that should only be done with --python-flag=-OO which was added too.

  • Fix, accelerated binaries failed to load packages from the virtualenv (not venv) that they were created and ran with, due to not propagating sys.prefix.

  • Standalone: Do not include plat-* directories as frozen code, and also on some platforms they can also contain code that fails to import without error.

  • Standalone: Added missing implicit dependency needed for newer NumPy versions.

New Features

  • Added support for Alpine Linux.

  • Added support for MSYS2 based Python on Windows.

  • Added support for Python flag --python flag=-OO, which allows to remove doc strings.

  • Added experimental support for pefile based dependency scans on Windows, thanks to Orsiris for this contribution.

  • Added plugin for proper Tkinter standalone support on Windows, thanks to Jorj for this contribution.

  • There is now a __compiled__ attribute for each module that Nuitka has compiled. Should be like this now, and contains Nuitka version information for you to use, similar to what sys.version_info gives as a namedtuple for your checks.

    __nuitka_version__(major=0, minor=6, micro=2, releaselevel="release")


  • Experimental code for variant types for int and long values, that can be plain C value, as well as the PyObject *. This is not yet completed though.

  • Minor refinements of specialized code variants reducing them more often the actual needed code.


  • The Nuitka Github Organisation that was created a while ago and owns the Nuitka repo now, has gained members. Check out for their list. This is an exciting transformation for Nuitka.

  • Nuitka is participating in the GSoC 2019 under the PSF umbrella. We hope to grow even further. Thanks to the mentors who volunteered for this important task. Check out the GSoC 2019 page and thanks to the students that are already helping out.

  • Added Nuitka internal API documentation that will receive more love in the future. It got some for this release, but a lot is missing.

  • The Nuitka code has been black-ened and is formatted with an automatic tool now all the way, which makes contributors lives easier.

  • Added documentation for questions received as part of the GSoC applications and ideas work.

  • Some proof reading pull requests were merged for the documentation, thanks to everybody who addresses these kinds of errors. Sometimes typos, sometimes broken links, etc.

  • Updated inline copy of Scons used for Python3 to 3.0.4, which hopefully means more bugs are fixed.


This release is a sign of increasing adoption of Nuitka. The GSoC 2019 is showing early effects, as is more developers joining the effort. These are great times for Nuitka.

This release has not much on the optimization side that is user visible, but the work that has begun is capable of producing glorious benchmarks once it will be finished.

The focus on this and coming releases is definitely to open up the Nuitka development now that people are coming in as permanent or temporary contributors in (relatively) high numbers.

Nuitka this week #12

Communication vs. Coding

Over the holiday season I sort of neglected these postings, but there are other fields, where I have put my focus, but I think these postings are now going to resume.

A quick update in a while in an eternally fluent situation, and not too polished.

Optimization Work

So, the 0.6.2 optimization work has been not a lot yet. I started some work on int/long C type and it looked good.

Week of Bugfixing

So I did this, and it turned out rather long. I continued fixing things and finishing up open things to the point that it now is clean. I hate to be falling behind. I am touching on a few more interesting topics.

Python flag -O and -OO

Nuitka was supporting -O but doing what should only the done for -OO which I only learned of now. So this is going to be supported both now.

Virtualenv vs. venv

Accelerated binaries were not running with full sys.path in the virtualenv, because site module uses sys.prefix and that was not propagated,but it now is.

Google Summer of Code for Nuitka

So the GSoC 2019 page is shaping up, the pull request to list Nuitka on the PSF project pages has been merged. More tweaking will be needed to get into perfect shape, and that has been my main communication time effort.

Finally somebody had stepped up for mentor, which means a lot to me. Now we need to see if students are willing to pick us.

And because I publish this too late. Already a bunch of stundents are showing interest and are hacking on Nuitka, which keeps me even more busy, and makes me extremely happy.

Nuitka Organisation

The Nuitka organisation on Github was introduced for my a while ago, and I had transferred the ownership of the Nuitka repository to there. Having moved the issue tracking to there, I was going more all in on it.

Recently more people have submitted PRs and with incredible quality and willingness to support it even after merge of their PR. No dry by contribution, but people looking to actually improve Nuitka together with me.

Therefore I welcome Orsiris and Jorj on board and am really happy about it.

Windows Dependencies from pefile

So Orsiris de Jong implemented a replacement for the dependency walker based code to scan DLLs for Windows standalone using the pefile module, which is likely way better at this. For starters it's experimental in the next release, but I expect it to soon become the default.

Tkinter Plugin

And Jorj X. McKie implemented a Windows Tkinter plug-in that deals with copying of the TCL installation and integrating it with Nuitka standalone for distribution.

That is very nice and seems to affect a lot of people using that GUI style it seems.

Numpy Plugin

And Jorj X. McKie also implemented a NumPy plug-in that deals with the various acceleration methods there are, e.g. MKL, Blas, I have no idea. It copies the needed DLLs along and makes sure that numpy stays fast in standalone mode.


I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.1

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes after a relatively long time, and contains important new optimization work, and even more bug fixes.

Bug Fixes

  • Fix, the options --[no]follow-import-to=package_name was supposed to not follow into the given package, but the check was executed too broadly, so that e.g. package_name2 was also affected. Fixed in already.

  • Fix, wasn't detecting multiple recursions into the same package in module mode, when attempting to compile a whole sub-package. Fixed in already.

  • Fix, constant values are used as C boolean values still for some of the cases. Fixed in already.

  • Fix, referencing a function cannot raise an exception, but that was not annotated. Fixed in already.

  • macOS: Use standard include of C bool type instead of rolling our own, which was not compatible with newest Clang. Fixed in already.

  • Python3: Fix, the bytes built-in type actually does have a __float__ slot. Fixed in already.

  • Python3.7: Types that are also sequences still need to call the method __class_getitem__ for consideration. Fixed in already.

  • Python3.7: Error exits from program exit could get lost on Windows due to __spec__ handling not preserving errors. Fixed in already.

  • Windows: Negative exit codes from Nuitka, e.g. due to a triggered assertion in debug mode were not working. Fixed in already.

  • Fix, conditional and expressions were mis-optimized when not used to not execute the right hand side still. Fixed in already.

  • Python3.6: Fix, generators, coroutines, and asyncgen were not properly supporting annotations for local variables. Fixed in already.

  • Python3.7: Fix, class declarations had memory leaks that were untestable before 3.7.1 fixed reference count issues in CPython. Fixed in already.

  • Python3.7: Fix, asyncgen expressions can be created in normal functions without an immediate awaiting of the iterator. This new feature was not correctly supported.

  • Fix, star imports on the module level should disable built-in name optimization except for the most critical ones, otherwise e.g. names like all or pow can become wrong. Previous workarounds for pow were not good enough.

  • Fix, the scons for Python3 failed to properly report build errors due to a regression of the Scons version used for it. This would mask build errors on Windows.

  • Python3.4: Fix, packages didn't indicate that they are packages in their __spec__ value, causing issues with importlib_resources module.

  • Python3.4: The __spec__ values of compiled modules didn't have compatible origin and has_location values preventing importlib_resources module from working to load data files.

  • Fix, packages created from .pth files were also considered when checking for sub-packages of a module.

  • Standalone: Handle cases of conflicting DLLs better. On Windows pick the newest file version if different, and otherwise just report and pick randomly because we cannot really decide which ought to be loaded.

  • Standalone: Warn about collisions of DLLs on non-Windows only as this can happen with wheels apparently.

  • Standalone: For Windows Python extension modules .pyd files, remove the SxS configuration for cases where it causes problems, not needed.

  • Fix: The exec statement on file handles was not using the proper filename when compiling, therefore breaking e.g. inspect.getsource on functions defined there.

  • Standalone: Added support for OpenGL platform plugins to be included automatically.

  • Standalone: Added missing implicit dependency for zmq module.

  • Python3.7: Fix, using the -X utf8 flag on the calling interpreter, aka --python-flag=utf8_mode was not preserved in the compiled binary in all cases.

New Optimization

  • Enabled C target type void which will catch creating unused stuff more immediately and give better code for expression only statements.

  • Enabled in-place optimization for module variables, avoiding write back to the module dict for unchanged values, accelerating these operations.

  • Compile time memory savings for the yield node of Python2, no need to track if it is in an exception handler, not relevant there.

  • Using the single child node for the yield nodes gives memory savings at compile time for these, while also making them operate faster.

  • More kinds of in-place operations are now optimized, e.g. int += int and the bytes ones were specialized to perform real in-place extension where possible.

  • Loop variables no longer loose type information, but instead collect the set of possible type shapes allowing optimization for them.


  • Corrected download link for Arch AUR link of develop package.

  • Added repository for Ubuntu Cosmic (18.10) for download.

  • Added repository for Fedora 29 for download.

  • Describe the exact format used for clang-format in the Developer Manual.

  • Added description how to use CondaCC on Windows to the User Manual.


  • The operations used for async for, async with, and await were all doing a look-up of an awaitable, and then executing the yield from that awaitable as one thing. Now this is split into two parts, with a new ExpressionYieldFromAwaitable as a dedicated node.

  • The yield node types, now 3 share a base class and common computation for now, enhancing the one for awaitiable, which was not fully annotating everything that can happen.

  • In code generation avoid statement blocks that are not needed, because there are no local C variables declared, and properly indent them.


  • Fixups for the manual Valgrind runner and the UI changes.

  • Test runner detects lock issue of clcache on Windows and considers it a permission problem that causes a retry.


This addresses even more corner cases not working correctly, the out of the box experience should be even better now.

The push towards C level performance for integer operation was held up by the realization that loop SSA was not yet there really, and that it had to be implemented, which of course now makes a huge difference for the cases where e.g. bool are being used. There is no C type for int used yet, which limits the impact of optimization to only taking shortcuts for the supported types. These are useful and faster of course, but only building blocks for what is to come.

Most of the effort went into specialized helpers that e.g. add a float and and int value in a dedicated fashion, as well as comparison operations, so we can fully operate some minimal examples with specialized code. This is too limited still, and must be applied to ever more operations.

What's more is that the benchmarking situation has not improved. Work will be needed in this domain to make improvements more demonstrable. It may well end up being the focus for the next release to improve Nuitka speedcenter to give more fine grained insights across minor changes of Nuitka and graphs with more history.

Nuitka this week #11

Communication vs. Coding

I continue to force myself to report more publicly, and it feels good. This time things are in a stablizing period, and I feel I have a consistent message.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Optimization Work

So, the 0.6.1 optimization work has been a lot. And it's containing improvements on every level. I think I will detail the levels in another section.

Levels of Optimization

First level is of course node level optimization. Here 0.6.1 adds many things, from better handling of closure variables not all as unknown every time control flow escapes, to some operations + and comparisons on known built-in type shapes to now be able to statically tell that they do not raise. The opposite (does definitely raise) is prepared, but not yet used.

This allows for type shapes to be longer known. Now a+b+c can be known, but previously only a+b was sort of known, and little used information.

The next level is picking the C target type. Here seeing more operations and understanding more variables allows to more often pick the C bool or C void types over the PyObject * C type. For 0.6.1 I have observed that esp. more indicator variables make it to that stage, generating way more efficient C code (for that indicator variable) for those in many instances, esp. with loops, as these no longer loose type shape information as badly as they did.

The, another level is when it is treated as an object, but known to be int, there are way more helpers used for +/+= and a whole new set of them for comparisons, that in these cases of full or partial type knowledge operate faster.

And even if e.g. only one type is known, this still allows to not make a lot of tests about it, and to avoid attempted shortcuts that cannot work. For 0.6.1 the + and += are pretty well covered for these, but some variants are not yet tuned to take all type knowledge advantage.

These will be also the building block, once the C type layer picks types like "C int or PyObject * known to be int" with indicator flags which values are currently valid to use, then these specialized calls still make sense.

The most attrative level, "C int" has not been reached for 0.6.1 but for my loop example and Python3, I can say that now would be a nice time to start it, as type shape knowledge is all there. This was totally not the case for 0.6.0, but it seems that this step will have to be postponed to another release, maybe 0.6.2, maybe even later.

Week of Bugfixing

But something that bothers me is seeing the issue tracker pile up on actionable items, where I just have not taken action. So as announced on Twitter already, I am having and continue to have bug fixing time. I am acting on issues that are relatively old and easy to act on, or where I have no hope of this happening by anybody else anymore.

I have listed some interesting examples below. But basically these are small, relatively unimportant, yet somewhat import for some use cases things.

Exec on Filehandles

So when doing exec on a filehandle, Nuitka was at runtime reading the source, then compiling it, but forgetting about the filename. This makes things like inspect.getsource() fail on functions from there, and ugly tracebacks not pointing to the filename. This was one of the things which I had understood, but not did the actual work yet.


And another one, which seemed just not done, but turned out to be rather complex, this one needs to populate a sys.path_importer_cache for imported modules, and then to report the child modules. There was no object to carry that information, so now instances of the meta path based importer are associated for every import.

Turns out for Python3, my simplistic type building calling type manually here does not work, as __init__ and iter_modules do not become anything but static methods ever. Needs a real type.

Plus, I had to disable it for now, because mixed packages, like the one we do with multiprocessing" where only part is compiled (the one required) and part is pure Python from disk still, stopped to work. The ``iter_modules it seems will have to cover that case too.

So no luck, postponing this until next week of bug fixes. Frustrating a bit, but such is life.

When to release

There are still some issues that I want to get to. Specicially the OpenGL plugins which has been research ever since, and nobody stepped up, but it's rather trivial. And the Tcl/Tk for Windows. People have provided sufficient instructions for a plugin that I am going to write this week.

Once I feel the issue tracker is clean, I will release. As a matter of experience, it is then going to grow a lot again.

Google Summer of Code for Nuitka

Finally somebody has stepped up, which means a lot to me. Now to the actual work!


I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #10

Communication vs. Coding

Recently it was a bit more tough to make that decision. First, there was much going privately, with me ill, then child ill, and ill again, and myself, and that made me have a much harder time to communicate about incomplete things.

Even now, I am torn between fixing issues for 0.6.1 and doing this, but I know that it will take at least one week, so I am missing the point, if I wait for it more.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.


There has been another hotfix, and there ought to be one, at least on factory there is a bunch of stuff for it, but I didn't actually do it yet. I was wandering between there will be a release anyway, and the feeling that some of the material may cause regressions, so I might skip on that really.

So for the most fixes, I suspect, develop is going to be the way until next week.

Google Summer of Code for Nuitka

Nobody has stepped up, which means it will not happen unfortunately. This would be your last chance to step up. I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Pythran and Nuitka

As suggested by @wuoulf (Wolf Vollprecht) we had a meeting at the side of the PyCon DE 2018 conference in Karlsruhe, abusing the C++ regular table as a forum for that, which was a very nice experience.

First of all, Wolf is so much more knowledgeable about AnaConda and could point out to me, very important stuff, not the least, that AnaConda contains its own compiler, which I have successfully used since, to first add easier installation instructions for Windows, and second, to successfully statically link with LTO on Linux amd64. Both of which are important for me.

But for Pythran which is limited Python, specialized to translate Numpy API to C++, we showed each other, Nuitka and Pythran details, and somehow in my mind a plan formed how Nuitka could use the Pythran tricks long term, and mid term, how it could include a plugin that will allow to integrate with Pythran compilation.

This was a huge success.

Performance Work

Adding specialized object operations

See last week, this has seen more completion. Both + and += are more or less covered for the selected subset. The CPython test suites were initially not finding uses, but with more and more optimization phase improvements, it challenges code generation with missing ones, and then I added them more and more.

Controlflow Descriptions

Shapes were added for the + and < operation so far, but didn't influence anything else really but code generation, but of course they should also impact optimization phase.

So the query for type shape has been enhanced to return not only a type shape saying that int+float -> float, but also now an object that describes impact on control flow of the program. This can then say e.g. that this doesn't execute arbitrary code, and that it does not modify input values, things used in the code generation to avoid error checks, and in the optimization to not have to mark things as unknown.

Preparations for comparison operations

So optimization now also has proper type shape functions for the < and the warnings when they fail to know what to do for concrete types. This allows to actually remove checks, but so far this wasn't exposed for neither + or for <. Doing this eliminates the exception check for the operation part, where previously it was done if anything in the expression could raise.

Specializing the rich comparisons helper codes is the next step, but so far I didn't quite get to it yet, but it has been started.

Comparison Conditions

Preparing < optimization for the loop, I noticed that not was optimized for in to become not in, and also is to become is not, etc. but for comparisons, where we can not the result is of bool shape, we can now also switch not < to >= and not = to != of course.

And since our reformulation of while a < b ends up having a statement like if not a < b: break as part of its re-formulation, that is again one step closer to optimizing my example loop.

Local variable escaping

Much to my shock, I noticed that the code which is responsible to handle escaping control flow (i.e. unknown code is executed), was not only doing what it was supposed to do, i.e. mark closure variables as unknown, but more or less did it for all local variables with Python3.

Fixing that allows for a lot more optimization obviously, and makes my test find missing ones, and even bugs in existing ones, that were previously hidden. A good thing to notice this regression (was better once), now that I am looking at concrete examples.

One noticeable sign was that more of my tests failed with warnings about missing code helpers. And another that in my while loop with int increase, it now seems as if Python3 is good. For Python2, the "int or long" shape will need dedicated helpers. That is because ìnt + int becomes either int or long there, where Python3 only has long but renamed it int.

Benchmarks Missing

Speedcenter got repaired, but I need to add the loop examples I am using as test cases before next release, so I can show what Nuitka 0.6.1 will have achieved or at least have improved somewhat already.

But currently these examples only serve as input for general improvements that then take a lot of time, and don't have immediate impact on their own.

Still would be good to see where Nuitka is standing after each one.

Static Linking

So static linking works now, provided it's not a pyenv crappy libpython.a but one that can actually work. I got this to work on Linux and using the Conda CC, even LTO will work with it. Interestingly then linking is noticely slow, and I bet ccache and the likes won't help with that.

I am interested to see what this means for performance impact. But it will allow to address issues, where embedded CPython run time is plain slower than the one that lives in the python binary. For acceleration this is great news.

Conda CC

Using Conda CC by default as a fallback in --mingw mode on Windows is something that was easy to add. So when no other gcc is found, and MSVC is not tried in this mode, and the right directory is added to PATH automatically, with Anaconda, things should now be smoother. It has also its own libpython.a, not sure yet if it's a static link library, that would be fantastic, but unlike standard MinGW64 we do not have to roll our own at least.

I will try with --lto eventually though and see what it does. But I think static linking on Windows is not supported by CPython, but I am not entirely sure of that.

Annotations Future Feature

Found a 3.7 feature that is not covered by the test suite, the __future__ flag annotations wasn't working as expected. In this, strings are to be used for __annotations__ where they show up (many are ignored simply) and that requires an unparse function, going from parsed ast (presumably it's still syntax checked) back to the string, but that was only very hard to get at, and with evil hackery.

For 3.8 a bug fix is promised that will give us the string immediately, but for now my hack must suffice.

MSI files

Following the 3.7.1 release, there are MSI files again, as the regression of 3.7.0 to build them has been fixed in that release. The MSI files will work with 3.7.0 also, just the building was broken.


So 0.6.1 is in still in full swing in terms of optimization. I think I need to make a release soon, simply because there is too much unreleased, but useful stuff already.

I might have to postpone my goal of C int performance for one example loop until next release. No harm in that. There already are plenty of performance improvements across the board.


I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #9

Communication vs. Coding

My new communication strategy is a full success, engagement with Nuitka is on an all time high.

But the recent weeks more than ever highlighted why I have to force myself to do it. I do not like to talk about unfinished stuff. And right now, there is really a lot of it, almost only it. Also I was ill, and otherwise busy, so this is now late by a week.

But I am keeping it up, and will give an update, despite the feeling that it would be better to just finish a few of those things and then talk about it, but then it will take forever and leave you in the dark. And that is not what is supposed to be.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.


So the 0.6.0 release was a huge success, but it definitely wasn't perfect, and hotfixes were necessary. The latest one was done just yesterday and actually contains one for an important mis-optimization being done, and you ought to update to it from any prior 0.6.0 release.

There are also a few remaining compatibility issues fixed for 3.7 and generally using the latest hotfix is always a good idea.

Kind of what one has to expect from a 0 release, this one also had more expose than usual is seems.

Google Summer of Code for Nuitka

I need more people to work on Nuitka. One way of doing this could be to participate in Google Summer of Code under the Python umbrella. To make that possible, I need you to volunteer as a mentor. So please, please, do.

I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Website Overhaul

I updated the website to recent Nikola and dropped the tag cloud that I was using. Should have cleaner and better looks. Also integrated privacy aware sharing links, where two clicks are necessary to share a page or article like this one on Twitter, Facebook, etc.

Also the download page saw some structural updates and polishing. It should easier to overview now.

Performance Work

Adding specialized object operations

The feedback for performance and the work on 0.6.1 are fully ongoing, and there are many major points that are ongoing. I want to briefly cover each one of them now, but many of them will only have full effect, once everything is in place, which each one is very critical.

So, with the type tracing, objects have known types, and short of using a C type, knowing e.g. that an object is an int, and the other one too, doing + for them can take a lot of advantage avoiding unrelated checks and code paths, even if still using PyObject * at the end of the day.

And even we are only knowing it's not an int, but say one value is a tuple and the other an unknown, that allows to remove checks for int shortcuts as they can no longer apply. These are tiny optimizations then, but still worthwhile.

To further this, first the inplace operations for a couple of more or less randomly selected types, list, tuple, int, long, str, unicode, bytes, and float, have been looked at and have gotten their own special object based helpers if one or both types are known to be of that kind.

Finding missing specialized object code generation

A report has been added, that will tell when such an operation could have been used, but was not available. This uncovered where typical stuff goes non optimized, a nice principle to see what is actually happening.

So adding list and str would now give a warning, although of course, the optimization phase ought to catch the static raise that is and never let it get there, so this report also addresses missing optimization in an earlier phase.

Optimizing plain object operations too

So the in-place operations were then covered, so this was extended to mere + operations too, the ones that are not in-place. Sometimes, esp. for immutable types, there was already code for that, e.g. int doesn't really do it, in other cases, list + list code for a quicker concat was added.

And again a report for where it's missing was added and basic coverage for most of the types. However, in some instances, the optimization doesn't use the full knowledge yet. But where it does, it will shove off quite a few cycles.

Lack of type knowledge

To apply these things effectively, optimization and value tracing need to know types in the first place. I have found two obstacles for that. One are branch merges. If a branch or both assign to the same type or original type, well the type is changed. Previously it became "unknown" which is treated as object for code generation, and allows nothing really. But now that is better on develop now, and was actually a trivial missing thing.

The other area is loops. Loops put values to unknown when entering loop body, and again when leaving. Essentially making type tracing not effective where it is needed the most to achieve actual performance. Also this was limiting the knowledge for all function to one type to not happening for these kinds of variables that were assigned inside a loop at all.

Took me a while, but I figured out how to build type tracing for loops that works. It currently is still unfinished in my private repo, but passes all tests, I would just like to make it use dedicated interfaces, and clean it up.

I will most likely have that for 0.6.1 too and that should expand the cases where types are known in code generation by a fair amount.

The effect of that will be that more often C code generation will actually see types. Currently e.g. a boolean variable that is assigned in a loop, cannot use the C target type in code generation. Once loop code is merged, it will however take advantage there too. And only then I think adding "C int" as a C type makes sense at all.

Performance regressions vs. CPython

Then another area is performance regressions. So one thing I did early on in the 0.6.1 cycle was using the "module var C target type" to get in-place working for those too. Doing string concatenations on module variables could be slower by an order of magnitude, as could be other operations.

I still need to do it for closure variables too. Then Nuitka will do at least as many of them perfectly as CPython does. It also will be better at it them, because e.g. it doesn't have to delete from the module dictionary first, due to it never taking a reference, and same applies to the cell. Should be faster for that too.

But strings in-place on these if not optimized, it will look very ugly in terms of worse performance, so 0.6.0 was still pretty bad for some users. This will however hopefully be addressed in 0.6.1 then.

In-place unicode still being bad

Another field was in-place string add for the already optimized case, it was still slower than CPython, and I finally found out what causes this. And that is the using of libpython where PyUnicode_Append is far worse than in the python binary that you normally use, I have see that at least for 3.5 and higher CPython. Analysis showed that e.g. MiniConda had the issue to a much smaller extent, and was being much faster anyway, but probably just has better libpython compilation flags.

So what to do. Ultimately that was to be solved by including a clone of that function, dubbed UNICODE_APPEND that behaves the same, and can even shove off a couple of cycles, by indicating the Python error status without extra checks, and specializing it for the pure unicode += unicode case that we see most often, same for UNICODE_CONCAT for mere +.

Right now the benchmarks to show it do not exist yet. Again something that typically wants me to delay stuff. But as you can imagine, tracking down these hard issues, writing that much code to replace the unicode resizing, is hard enough by itself.

But I hope to convince myself that this will allow to show that for compiled code, things are going to be faster only now.

Benchmarks Missing

In fact, speedcenter as a whole is currently broken, mostly due to Nikola changes that I am trying to work around, but it will take more time apparently and isn't finished as I write this.

Type shapes in optimization

Another optimization end, is the type shapes of the + operation itself. Right now what is being done is that the shape is derived from the shape of the left argument with the right shape to be considered by it. These also have reports now, for cases where they are missing. So saying e.g. that int + float results in float and these kinds of things, are stuff being encoded there right now.

This is necessary step to e.g. know that int + int -> int_or_long, to make effective loop variable optimization.

Without these, and again, that is a lot of code to write, there is no way to hope for wide spread type knowledge in code generation.

Control flow escape

Something missing there, is to also make it known that + unlike it currently is now, should not in all cases lead to "control flow escape" with the consequence of removing all stuff, and expecting an exception possible, but instead to let the int type also make known that + int ont it not only gives an int_or_long result shape, but also while doing so, that it will never raise an exception (bare MemoryError), and therefore allow more optimization to happen and less and therefore faster code generated.

Until this is done, what is actually going to happen is that while the + result is known, Nuitka will assume control flow escape.

And speaking of that, I think this puts too many variables to a too unknown state. You can to distrust all values, but not the types in this case, so that could be better, but right now it is not. Something else to look into.


So 0.6.1 is in full swing in terms of optimization. All these ends need a completion, and then I can expect to use advantage of things in a loop, and ultimately to generate C performance code for one example of loop. esp. if we add a C int target type, which currently isn't yet started, because I think it would barely be used yet.

But we are getting there and I wouldn't even say we are making small steps, this is all just work to be completed, nothing fundamental about it. But it may take more than one release for sure.

Mind you, there is not only +, there is also -, *, %, and many more operators, all of them will require work. Granted, loop variables tend to use + more often, but any un-optimized operation will immediately loose a lot of type knowledge.

Improved Annotations

There are two kinds of annotations, ones for classes and modules, which actually are stored in a __annotations__ variable, and everything else is mostly just ignored.

So Nuitka got the criterion wrong, and did one thing for functions, and the other for everything else. So that annotations in generators, coroutines and asyncgen ended up with wrong, crashing, and slower code, due to it updating the module __annotations__, so that one is important too if you have to do those.

Release or not

To release or not. There is at least one bug about star imports that affects numpy that is solved in develop, and wasn't back ported, and I was thinking it only applies to develop, but in fact does to stable. It makes me want to release even before all these optimization things happen and are polished, and I might well decide to go with that.

Maybe I only add the closure in-place stuff and the polish the loop SSA stuff, and then call it a release. It already will solve a lot of performance issues that exist right now, while staging the ground for more.

Standalone Improvements

Standalone work is also improving. Using pyi files got more apt, and a few things were added, all of which make sense to be used by people.

But I also have a backlog of issues there however. I will schedule one sprint for those I guess, where I focus on these. I am neglecting those somewhat recently.

Caching Examined

For the static code, I now noticed that it's compiled for each target name, due to the build directory being part of the object file for debug. For gcc 8 there is an option to allow pointing at the original static C file location, and then ccache is more effective, because object files will be the same.

That's actually pretty bad, as most of my machines are on gcc-6 and makes me think that libnuitka.a is really more of an requirement than ever. I might take some time to get this sorted out.

Python3 deprecation warnings

So Nuitka supports the no_warnings Python flag, and for a long time I have been annoyed at how it was not working for Python3 in some cases. The code was manually settign filters, but these would get overridden by CPython test suites testing warnings. And the code said that there is no CPython C-API to control it, which is just plain wrong.

So I changed that and it became possible to remove lots of ignore_stderr annotations in CPython test suites, and more importantly, I can stop adding them for when running older/newer CPython version with a suite.


I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #8

Public / Private CI / Workflow


I wrote this as part of a discussion recently, and I think it makes sense to share it here. This is a lot text though, feel free to skip forward.

Indeed I have a private repo, where I push and only private CI picks up. Based on Buildbot, I run many more compilations, basically around the clock on all of my computers, to find regressions from new optimization or codegen changes, and well UI changes too.

Public CI offerings like Travis are not aimed at allowing this many compilations. It will be a while before public cloud infrastructure will be donated to Nuitka, although I see it happening some time in the future. This leaves developers with the burden to run tests on their own hardware, and never enough. Casual contributors will never be able to do it themselves.

My scope is running the CPython test suites on Windows and Linux. These are the adapted 26, 27, 32, 33, 34, 35, 36, 37 suites, and also to get even more errors covered, they are ran with mismatching Python versions, so a lot of exceptions are raised. Often running the 36 tests with 37 and vice versa will extend the coverage, because of the exceptions being raise.

On Windows I compile with and without debug mode, x86 and x64, and it's kind of getting too much. For Linux I have 2 laptops in use, and an ARM CuBox bought from your donations, there it's working better, esp. due to ccache being used everywhere, although recent investigations show room for improvement there as well.

For memory usage I still compile mercurial and observe the memory it used in addition to comparing the mercurial tests to expected outputs its test suite gives. It's a sad day when Mercurial tests find changes in behavior, and luckily that has been rare. Running the Mercurial test suite gives some confidence in the thing not corrupting data it works with without knowing.

Caching the CPython outputs of tests to compare against is something I am going to make operational these days, trying to make things ever faster. There is no point to re-run tests with Python, just to get at its output, which will typically not change at all.

But for the time being, ccache.exe and clcache.exe seem to have done wonders for Windows too, but I will want to investigate some more to avoid unnecessary cache misses.


As for my workflow with Nuitka, I often tend to let some commits settle in my private repo only until they become trusted. Other times I will make bigger changes and put them out to factory immediately, because it will be hard to split up the changes later, so putting them out makes it easier.

I am more conservative with factory right after telling people to try something there. But also I break it on purpose, just trying out something. I really consider it a private branch for interacting with me or public CI. I do not recommend to use it, and it's like a permanent pull request of mine that is not ever going to be finished.

Then on occasions I am making a sorting of all commits on factory and split it into some things that become hotfixes, some things that become current pre-release, and other things that will remain in that proving ground. That is why I typically make hotfix and pre-release at the same times. The git flow suggests doing that and it's easy, so why not. As a bonus, develop is then practically stable at nearly all times too, with hardly any regressions.

I do however normally not take things as hotfixes that are on develop already, I hate the duplication of commits. Hotfixes must be small and risk free, and easy to put out, when there is any risk, it definitely will be on develop. Nuitka stable typically covers nearly all grounds already. No panic needed to add missing stuff and break others.

Hunting bugs with bisect

For me the git bisect is very important. My private commit history is basically a total mess and worthless, but on factory I am making very nice organized commits that I will frequently amend, even for the random PyLint cleanup. This allows me when e.g. one test suddenly says "segfault" on Windows to easily find the change that triggers it, look at C code difference, and spot the bug introduced, then amend the commit and be done with it.

It's amazing how much time this can save. My goal is to always have a workable state which is supposed to pass all tests. Obviously I cannot prove it for every commit, but when I know it to not be the case, I tend to make rebases. At times I have been tempted and followed up on backward amending develop and even stable.

I am doing that to be sure to have that bisect ability, but fortunately it's rare that kind of bug occurs, and I try not to do it.

Experimental Changes

As with recent changes, I sometimes make changes with the isExperimental() marker, activating breaking changes only gradually. The C bool type code generation has been there for months in a barely useful form, until it became more polished, and always guarded with a switch, until one day for 0.6 finally I changed it, and made the necessary fixes retroactively before that switch, to make it work while that was still in factory.

Then I will remove the experimental code. I feel it's very important and even ideal to be able to always compare outputs to a fully working solution. I am willing to postpone some cleanups until later date as a price, but when then something in my mind tells me again "This cannot possibly have ever worked"... a command line flag away, I have the answer to compare, plus, that includes extra changes happened in the meantime, they don't add noise to diff outputs of generated C code for example.

Then looking at that diff, I can tell where the unwanted effect is, and fix all the things, and that way find bugs much faster.

Even better, if I decide to make a cleanup action as part of making a change more viable to execute, then I get to execute it on stable grounds, covered by the full test suite. I can complete that cleanup, e.g. using variable identifier objects instead of mere strings was needed to make "heap generators" more workable. But I was able to put that one to active before "heap generators" was ever fully workable, and complete it, and actually reap some of its benefits already.


Obviously this takes a lot of hardware and CPU to be able to compile this much Python code on a regular basis. And I really wish I could add one of the new AMD Threadripper 2 to the mix. Anybody donating one to me? Yes I know, I am only dreaming. But it would really help the cause.

Milestone Release

So the 0.6 is out, and already a hotfix that addresses mostly use cases of people that didn't work. More people seemed to have tried out 0.6.0 and as a result is going to cover a few corner cases. So far I have not encountered a single regression of 0.6.0, but instead it contained ones for 0.5.33 which did have one that was not easy to fix.

So that went really smooth.

UI rework

The UI needs more work still. Specifically that packages do not automatically include all stuff below them and have to be specified by file path instead of by name, is really annoying to me.

But I had delayed 0.6 for some UI work, and the quirks are to remain some. I will work on these things eventually.


So I updated the website to state that PyStone is now 312% faster, from a number that was very old. I since then ran it with an updated version for Python3, and it's much less there. That is pretty sad.

I will be looking into that for 0.6.1 release, or I will have to update the wording to provide 2 numbers there, because it seems for Python3 performance with Nuitka it might be misleading.

Something with unicode strings and in-place operations is driving me crazy. Nuitka is apparently slower for that, and I can't point where that is happening exactly. It seems internally unicode objects are maybe put into a different state from some operations, which then making in-place extending in realloc fail more often, but I cannot know yet.

Inplace Operations

So more work has been put into those, adding more specialization, and esp. also applying them for module variables as well. CPython can do that, and actually is giving itself a hard time about it, and Nuitka should be doing this much clever with its more static knowledge.

But I cannot tell you how much scratching my head was wasted debugging that. I was totally stupid about how I approached that, looking from the final solution, it was always easy. Just not for me apparently.

New use cases

Talked about those above. So the top level logging module of your own was working fine in accelerated mode, but for standalone it failed and used the one from standard library instead. That kind of shadowing happened because Nuitka was going from module objects to their names and back to objects, which are bad in case of duplicates. That is fixed for develop, and one of those risk cases, where it cannot be a hotfix because it touched too much.

Then pure Python3 packages need not have and so far that was best working for sub-packages, but after hotfix, now it will also work for the main module you compile to be that empty.

Tcl/Tk Standalone

So instructions have been provided how to properly make that work for Python standalone on Windows. I have yet to live up to my promise and make Nuitka automatically include the necessary files. I hope to do it for 0.6.1 though.

Caching Examined

So I am looking at ccache on Linux right now, and found e.g. that it was reporting that gcc --version was called a lot at startup of Scons and then g++ --version once. The later is particularly stupid, because we are not going to use g++ normally, except if gcc is really old and does not support C11. So in case a good one was found, lets disable that version query and not do it.

And for the gcc version output, monkey patching scons to a version of getting that output that caches the result, removes those unnecessary forks.

So ccache is being called less frequently, and actually these --version outputs appears to actually take measurable time. It's not dramatic, but ccache was apparently getting locks, and that's worth avoiding by itself.

That said, the goal is for ccache and clcache to make them both report their effectiveness of cache usage after the end of a test suite run. That way I am hoping to notice and be able to know, if caching is used to its full effect.


I continue to be very active there. I put out a poll about the comment system, and disabling Disqus comments as a result, I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.


Working on the 0.6.1 release, attacking more in-place add operations as a first goal, and now turning to binary operations, I am trying to shape how using different helper functions to different object types looks like. And to gain performance without C types. But ultimately the same issue will arise there, what to do with mixed input types.

My desire is for in-place operations to fully catch up with CPython, as these can easily loose a lot of performance. Closure variables and their cells are another target to pick on, and I feel they ought to be next after module ones are now working, because also their solution ought to be very similar. Then showing that depending on target storage, local, closure, or module, is then faster in all cases would be a goal for the 0.6.1 release.

This feels not too far away, but we will see. I am considering next weekend for release.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.0

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release adds massive improvements for optimization and a couple of bug fixes.

It also indicates reaching the mile stone of doing actual type inference, even if only very limited.

And with the new version numbers, lots of UI changes go along. The options to control recursion into modules have all been renamed, some now have different defaults, and finally the filenames output have changed.

Bug Fixes

  • Python3.5: Fix, the awaiting flag was not removed for exceptions thrown into a coroutine, so next time it appeared to be awaiting instead of finished.

  • Python3: Classes in generators that were using built-in functions crashed the compilation with C errors.

  • Some regressions for XML outputs from previous changes were fixed.

  • Fix, hasattr was not raising an exception if used with non-string attributes.

  • For really large compilations, MSVC linker could choke on the input file, line length limits, which is now fixed for the inline copy of Scons.

  • Standalone: Follow changed hidden dependency of PyQt5 to PyQt5.sip for newer versions

  • Standalone: Include certificate file using by requests module in some cases as a data file.

New Optimization

  • Enabled C target type nuitka_bool for variables that are stored with boolean shape only, and generate C code for those

  • Using C target type nuitka_bool many more expressions are now handled better in conditions.

  • Enhanced is and is not to be C source type aware, so they can be much faster for them.

  • Use C target type for bool built-in giving more efficient code for some source values.

  • Annotate the not result to have boolean type shape, allowing for more compile time optimization with it.

  • Restored previously lost optimization of loop break handling StopIteration which makes loops much faster again.

  • Restore lost optimization of subscripts with constant integer values making them faster again.

  • Optimize in-place operations for cases where left, right, or both sides have known type shapes for some values. Initially only a few variants were added, but there is more to come.

  • When adjacent parts of an f-string become known string constants, join them at compile time.

  • When there is only one remaining part in an f-string, use that directly as the result.

  • Optimize empty f-strings directly into empty strings constant during the tree building phase.

  • Added specialized attribute check for use in re-formulations that doesn't expose exceptions.

  • Remove locals sync operation in scopes without local variables, e.g. classes or modules, making exec and the like slightly leaner there.

  • Remove try nodes that did only re-raise exceptions.

  • The del of variables is now driven fully by C types and generates more compatible code.

  • Removed useless double exception exits annotated for expressions of conditions and added code that allows conditions to adapt themselves to the target shape bool during optimization.

New Features

  • Added support for using .egg files in PYTHONPATH, one of the more rare uses, where Nuitka wasn't yet compatible.

  • Output binaries in standalone mode with platform suffix, on non-Windows that means no suffix. In accelerated mode on non-Windows, use .bin as a suffix to avoid collision with files that have no suffix.

  • Windows: It's now possible to use clang-cl.exe for CC with Nuitka as a third compiler on Windows, but it requires an existing MSVC install to be used for resource compilation and linking.

  • Windows: Added support for using ccache.exe and clcache.exe, so that object files can now be cached for re-compilation.

  • For debug mode, report missing in-place helpers. These kinds of reports are to become more universal and are aimed at recognizing missed optimization chances in Nuitka. This features is still in its infancy. Subsequent releases will add more like these.


  • Disabled comments on the web site, we are going to use Twitter instead, once the site is migrated to an updated Nikola.

  • The static C code is now formatted with clang-format to make it easier for contributors to understand.

  • Moved the construct runner to top level binary and use it from there, with future changes coming that should make it generally useful outside of Nuitka.

  • Enhanced the issue template to tell people how to get the develop version of Nuitka to try it out.

  • Added documentation for how use the object caching on Windows to the User Manual.

  • Removed the included GUI, originally intended for debugging, but XML outputs are more powerful anyway, and it had been in disrepair for a long time.

  • Removed long deprecated options, e.g. --exe which has long been the default and is no more accepted.

  • Renamed options to include plugin files to --include-plugin-directory and --include-plugin-files for more clarity.

  • Renamed options for recursion control to e.g. --follow-imports to better express what they actually do.

  • Removed --python-version support for switching the version during compilation. This has only worked for very specific circumstances and has been deprecated for a while.

  • Removed --code-gen-no-statement-lines support for not having line numbers updated at run time. This has long been hidden and probably would never gain all that much, while causing a lot of incompatibilty.


  • Moved command line arguments to dedicated module, adding checks was becoming too difficult.

  • Moved rich comparison helpers to a dedicated C file.

  • Dedicated binary and unary node bases for clearer distinction and more efficient memory usage of unuary nodes. Unary operations also no longer have in-place operation as an issue.

  • Major cleanup of variable accesses, split up into multiple phases and all including module variables being performed through C types, with no special cases anymore.

  • Partial cleanups of C type classes with code duplications, there is much more to resolve though.

  • Windows: The way exec was performed is discouraged in the subprocess documentation, so use a variant that cannot block instead.

  • Code proving information about built-in names and values was using not very portable constructs, and is now written in a way that PyPy would also like.


  • Avoid using 2to3 for basic operators test, removing test of some Python2 only stuff, that is covered elsewhere.

  • Added ability to cache output of CPython when comparing to it. This is to allow CI tests to not execute the same code over and over, just to get the same value to compare with. This is not enabled yet.


This release marks a point, from which on performance improvements are likely in every coming release. The C target types are a major milestone. More C target types are in the work, e.g. void is coming for expressions that are done, but not used, that is scheduled for the next release.

Although there will be a need to also adapt optimization to take full advantage of it, progress should be quick from here. There is a lot of ground to cover, with more C types to come, and all of them needing specialized helpers. But as soon as e.g. int, str are covered, many more programs are going to benefiting from this.

Nuitka this week #7

Nuitka Design Philosophy


I wrote this as part of a discussion recently, and I think it makes sense to share my take on Nuitka and design. This is a lot text though, feel free to skip forward.

The issue with Nuitka and design mainly for me is that the requirements for many parts were and are largely unknown to me, until I actually start to do it.

My goto generators approach worked out as originally designed, and that felt really cool for once, but the whole "C type" thing was a total unknown to me, until it all magically took form.

But rather I know it will evolve further if I go from "bool" (complete and coming for 0.6.0) via "void" (should be complete already, but enabling will happen only for 0.6.1 likely) to "int", not sure how long that will take.

I really think Nuitka, unlike other software that I have designed, is more of a prototype project that gradually turns more and more into the real thing.

I have literally spent years to inject proper design in steps into the optimization phase, what I call SSA, value tracing, and it is very much there now. I am probably going to spend similar amounts of time, to execute on applying type inference results to the code generation.

So I turned that into something working with code strings to something working with variable declaration objects knowing their type for the goto generators, aiming at C types generally. All the while carrying the full weight of passing every compatibility test there is.

Then e.g. suddenly cleaning up module variables to no longer have their special branch, but a pseudo C type, that makes them like everything else. Great. But when I first introduced the new thing, I postponed that, because I could sooner apply its benefits to some things and get experience from it.

While doing partial solutions, the design sometimes horribly degrades, but only until some features can carry the full weight, and/or have been explored to have their final form.

Making a whole Nuitka design upfront and then executing it, would instead give a very high probability of failing in the real world. I am therefore applying the more agile approach, where I make things work first. And then continue to work while I clean it up.

For every feature I added, I actively go out, and change the thing, that made it hard or even fail. Always. I think Nuitka is largely developed by cleanups and refactoring. Goto generators were a fine example of that, solving many of the issues by injecting variable declarations objects into code generation, made it easy to indicate storage (heap or object or stack) right there.

That is not to say that Nuitka didn't have the typical compiler design. Like parsing inputs, optimizing a tree internally, producing outputs. But that grand top level design only tells you the obvious things really and is stolen anyway from knowing similar projects like gcc.

There always were of course obvious designs for Nuitka, but that really never was what anybody would consider to make a Python compiler hard. But for actual compatibility of CPython, so many details were going to require examination with no solutions known ahead of time.

I guess, I am an extreme programmer, or agile, or however they call it these days. At least for Nuitka. In my professional life, I have designed software for ATC on the drawing board, then in paper, and then in code, the design just worked, and got operational right after completion, which is rare I can tell you.

But maybe that is what keeps me exciting about Nuitka. How I need to go beyond my abilities and stable ground to achieve it.

But the complexity of Nuitka is so dramatically higher than anything I ever did. It is doing a complicated, i.e. detail rich work, and then it also is doing hard jobs where many things have to play together. And the wish to have something working before it is completed, if it ever is, makes things very different from projects I typically did.

So the first version of Nuitka already had a use, and when I publicly showed it first, was capable of handling most complex programs, and the desire was to evolve gradually.

I think I have described this elsewhere, but for large parts of the well or bad designed solutions of Nuitka, there is reliable ways of demonstrating it works correctly. Far better than I have ever encountered. i believe it's the main reason I managed to get this off the ground is that. Having a test "oracle" is what makes Nuitka special, i.e. comparing to existing implementations.

Like a calculator can be tested comparing it to one of the many already perfect ones out there. That again makes Nuitka relatively easy despite the many details to get right, there is often an easy way to tell correct from wrong.

So for me, Nuitka is on the design level, something that goes through many iterations, discovery, prototyping, and is actually really exciting in that.

Compilers typically are boring. But for Nuitka that is totally not the case, because Python is not made for it. Well, that*s technically untrue, lets say not for optimizing compilers, not for type inference, etc.

UI rework

Following up on discussion on the mailing list, the user interface of Nuitka will become more clear with --include-* options and --[no]follow-import* options that better express what is going to happen.

Also the default for following with extension modules is now precisely what you say, as going beyond what you intend to deliver makes no sense in the normal case.

Goto Generators

Now release as 0.5.33 and there has been little regressions so far, but the one found is only in the pre-release of 0.6.0 so use that instead if you encounter a C compilation error.


The performance regressions fixed for 0.6.0 impact pystone by a lot, loops were slower, so were subscripts with constant integer indexes. It is a pity these were introduced in previous releases during refactorings without noticing.

We should strive to have benchmarks with trends. Right now Nuitka speedcenter cannot do it. Focus should definitely go to this. Like I said, after 0.6.0 release, this will be a priority, to make them more useful.


I continue to be active there. I just put out a poll about the comment system, and disabling Disqus comments I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Egg files in PYTHONPATH

This is a relatively old issue that now got addressed. Basically these should be loaded from for compilation. Nuitka now unpacks them to a cache folder so it can read source code from them, so this apparently rare use case works now, yet again improving compatibility.

Will be there for 0.6.0 release.


Seems request module sometimes uses that. Nuitka now includes that data file starting with 0.6.0 release.

Compatibility with pkg_resources

It seems that getting "distributions" and taking versions from there, is really a thing, and Nuitka fails pkg_resources requirement checks in standalone mode at least, and that is of course sad.

I am currently researching how to fix that, not sure yet how to do it. But some forms of Python installs are apparently very affected by it. I try looking into its data gathering, maybe compiled modules can be registered there too. It seems to be based on file system scans of its own makings, but there is always a monkey patch possible to make it better.


Still working on the 0.6.0 release, cleaning up open ends only. Release tests seem to be pretty good looking. The UI changes and stuff are a good time to be done now, but delay things, and there is a bunch of small things that are low hanging fruits while I wait for test results.

But since it fixes so many performance things, it really ought to be out any day now.

Also the in-place operations stuff, I added it to 0.6.0 too, just because it feels very nice, and improves some operations by a lot too. Initially I had made a cut for 0.6.1 already, but that is no more.


If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka