Nuitka Release 0.3.13

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is mostly the result of working towards compilation of a real programs (Mercurial) and to merge and finalize the frame stack work. Now Nuitka has a correct frame stack at all times, and supports func_code and gi_code objects, something previously thought to be impossible.

Actually now it's only the "bytecode" objects that won't be there. And not attributes of func_code are meaningful yet, but in theory can be supported.

Due to the use of the "git flow" for Nuitka, most of the bugs listed here were already fixed in on the stable release before this release. This time there were 5 such hot fix releases, sometimes fixing multiple bugs.

Bug fixes

  • In case of syntax errors in the main program, an exception stack was giving that included Nuitka code. Changed to make the same output as CPython does. Fixed in 0.3.12a already.

  • The star import (from x import *) didn't work for submodules. Providing * as the import list to the respective code allowed to drop the complex lookups we were doing before, and to simply trust CPython C/API to do it correctly. Fixed in 0.3.12 already.

  • The absolute import is not the default of CPython 2.7 it seems. A local posix package shadows the standard library one. Fixed in 0.3.12 already.

  • In --deep mode, a module may contain a syntax error. This is e.g. true of "PyQt" with port_v3 included. These files contain Python3 syntax and fail to be imported in Python2, but that is not to be considered an error. These modules are now skipped with a warning. Fixed in 0.3.12b already.

  • The code to import modules wasn't using the __import__ built-in, which prevented __import__ overriding code to work. Changed import to use the built-in. Fixed in 0.3.12c already.

  • The code generated for the __import__ built-in with constant values was doing relative imports only. It needs to attempt relative and absolute imports. Fixed in 0.3.12c already.

  • The code of packages in "" believed it was outside of the package, giving problems for package local imports. Fixed in 0.3.12d already.

  • It appears that "Scons", which Nuitka uses internally and transparent to you, to execute the compilation and linking tasks, was sometimes not building the binaries or shared libraries, due to a false caching. As a workaround, these are now erased before doing the build. Fixed in 0.3.12d already.

  • The use of in and not in in comparison chains (e.g. a < b < c is one), wasn't supported yet. The use of these in comparison chains a in b in c is very strange.

    Only in the it was ever used I believe. Anyway, it's supported now, solving this TODO and reducing the difference. Fixed in 0.3.12e already.

  • The order of evaluation for in and not in operators wasn't enforced in a portable way. Now it is correct on "ARM" too. Fixed in 0.3.12e already.

New Optimization

  • The built-ins GeneratorExit and StopIteration are optimized to their Python C/API names where possible as well.


  • The __file__ attribute of modules was the relative filename, but for absolute filenames these become a horrible mess at least on Linux.

  • Added assertion helpers for sane frame and code objects and use them.

  • Make use of assertObject in more places.

  • Instead of using os.path.sep all over, added a helper Utils.joinpath that hides this and using os.path.join. This gives more readable code.

  • Added traces to the "unfreezer" guarded by a define. Helpful in analyzing import problems.

  • Some PyLint cleanups removing dead code, unused variables, useless pass statement, etc.

New Tests

  • New tests to cover SyntaxError and IndentationError from --deep imports and in main program.

  • New test to cover evaluation order of in and not in comparisons.

  • New test to cover package local imports made by the "" of the package.


  • Drop "" in favor of the new "", because the later is more portable.

  • The logging output is now nicer, and for failed recursions, outputs the line that is having the problem.


The frame stack work and the func_code are big for compatibility.

The func_code was also needed for "hg" to work. For Mercurial to pass all of its test suite, more work will be needed, esp. the inspect module needs to be run-time patched to accept compiled functions and generators too.

Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization.

Putting a conference T-shirt to good use

This is my 7rd old son in his "Halloween" outfit. It is the first time, he was allowed to do it and so he was very excited.

The link to Python is the T-shirt "Don't Panic - Use Python". I got that from PyCON DE, and the family loves it.

Michael wears Python T-shirt for Halloween.

Michael in his Halloween outfit.

Nuitka Release 0.3.12

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new release of Nuitka many bug fixes, and substantial improvements especially in the organizational area. There is a new "User Manual" (PDF), with much improved content, a sys.meta_path based import mechanism for --deep mode, git flow goodness.

This release is generally also the result of working towards compilation of a real programs (Mercurial) and to get things work more nicely on Windows by default. Thanks go to Liu Zhenhai for helping me with this goal.

Due to the use of the "git flow", most of the bugs listed here were already fixed in on the stable release before this release. And there were many of these.

Bug fixes

  • The order of evaluation for base classes and class dictionaries was not enforced.

    Apparently nothing in the CPython test suite did that, I only noticed during debugging that Nuitka gave a different error than CPython did, for a class that had an undefined base class, because both class body and base classes were giving an error. Fixed in 0.3.11a already.

  • Method objects didn't hold a reference to the used class.

    The effect was only noticed when --python-debug was used, i.e. the debug version of Python linked, because then the garbage collector makes searches. Fixed in 0.3.11b already.

  • Set sys.executable on Linux as well. On Debian it is otherwise /usr/bin/python which might be a different version of Python entirely. Fixed in 0.3.11c already.

  • Embedded modules inside a package could hide package variables of the same name. Learned during PyCON DE about this corner case. Fixed in 0.3.11d already.

  • Packages could be duplicated internally. This had no effect on generated code other than appearing twice in the list if frozen modules. Fixed in 0.3.11d already.

  • When embedding modules from outside current directory, the look-up failed. The embedding only ever worked for the compile itself and programs test cases, because they are all in the current directory then. Fixed in 0.3.11e already.

  • The check for ARM target broke Windows support in the Scons file. Fixed in 0.3.11f already.

  • The star import from external modules failed with an error in --deep mode. Fixed in 0.3.11g already.

  • Modules with a parent package could cause a problem under some circumstances. Fixed in 0.3.11h already.

  • One call variant, with both list and dict star arguments and keyword arguments, but no positional parameters, didn't have the required C++ helper function implemented. Fixed in 0.3.11h already.

  • The detection of the CPU core count was broken on my hexacore at least. Gave 36 instead of 6, which is a problem for large programs. Fixed in 0.3.11h already.

  • The in-line copy of Scons didn't really work on Windows, which was sad, because we added it to simplify installation on Windows precisely because of this.

  • Cleaning up the build directory from old sources and object files wasn't portable to Windows and therefore wasn't effective there.

  • From imports where part of the imported were found modules and parts were not, didn't work. Solved by the feature branch meta_path_import that was merged for this release.

  • Newer MinGW gave warnings about the default visibility not being possible to apply to class members. Fixed by not setting this default visibility anymore on Windows.

  • The sys.executable gave warnings on Windows because of backslashes in the path. Using a raw string to prevent such problems.

  • The standard library path was hard coded. Changed to run time detection.


  • Version checks on Python runtime now use a new define PYTHON_VERSION that makes it easier. I don't like PY_VERSION_HEX, because it is so unreadable. Makes some of the checks a lot more safe.

  • The sys.meta_path based import from the meta_path_import feature branch allowed the cleanup the way importing is done. It's a lot less code now.

  • Removed some unused code. We will aim at making Nuitka the tool to detect dead code really.

  • Moved nuitka.Nodes to nuitka.nodes.Nodes, that is what the package is intended for, the split will come later.

New Tests

  • New tests for import variants that previously didn't work: Mixed imports. Imports from a package one level up. Modules hidden by a package variable, etc.

  • Added test of function call variant that had no test previously. Only found it when compiling "hg". Amazing how nothing in my tests, CPython tests, etc. used it.

  • Added test to cover the partial success of import statements.

  • Added test to cover evaluation order of class definitions.


  • Migrated the "README.txt" from org-mode to ReStructured Text, which allows for a more readable document, and to generate a nice "User Manual" in PDF form.

  • The amount of information in "README.txt" was increased, with many more subjects are now covered, e.g. "git flow" and how to join Nuitka development. It's also impressive to see what code blocks and syntax highlighting can do for readability.

  • The Nuitka git repository has seen multiple hot fixes.

    These allowed to publish bug fixes immediately after they were made, and avoided the need for a new release just to get these out. This really saves me a lot of time too, because I can postpone releasing the new version until it makes sense because of other things.

  • Then there was a feature branch meta_path_import that lived until being merged to develop to improve the import code, which is now released on master as stable. Getting that feature right took a while.

  • And there is the feature branch minimize_CPython26_tests_diff which has some success already in documenting the required changes to the "CPython26" test suite and in reducing the amount of differences, while doing it. We have a frame stack working there, albeit in too ugly code form.

  • The release archives are now built using setuptools. You can now also download a zip file, which is probably more Windows friendly. The intention is to work on that to make produce a Nuitka install that won't rely on any environment variables at all. Right now won't even allow any other options than sdist to be given.

  • Ported "" to "", i.e. ported it to Python. This way, we can execute it easily on Windows too, where it currently still fails. Replacing diff, rm -rf, etc. is a challenge, but it reduces the dependency on MSYS tools on Windows.

  • The compilation of standard library is disabled by default, but site or dist packages are now embedded. To include even standard library, there is a --really-deep option that has to be given in addition to --deep, which forces this.


Again, huge progress. The improved import mechanism is very beautiful. It appears that little is missing to compile real world programs like "hg" with Nuitka. The next release cycle will focus on that and continue to improve the Windows support which appears to have some issues.

Cat update

The new cat has become indispensable for just about anything. She follows throughout the hous, and is just too lovely, here two photos of her in typical spots.

When I code for Nuitka, she often lays between keyboard and monitors, sleeping, getting caressed by me.

Cat Muska laying in her usual place between monitor and keyboard.

And here is another one, hiding in her cat tree.

Cat muska in her cat tree box.

My 7yr old fotographer

This photo was shot by my 7yr old son. It's truly artful, if not perfect. I have not applied any correction. He has my old camera, and knows how to use it, more and more.

Artful photo of his mom taken by my son Michael

Michael shot of his mom

PyCON DE 2011 - My Report

The PyCON DE 2011 is just over, sprints are still happening over the weekend, but my wife wouldn't allow me to stay away for that long, so it's not for me this time. Maybe next time.

Right now I feel very happy and excited that I went there. What a great experience this was.

It was the first German PyCON and clearly it was overdue as it was now the merger of many already grown up communities. A huge number of talks over 3 days in 3 parallel tracks, with 3 keynotes, was an outstanding program. And very well run. Strict time management, every detail was well prepared.

I can only admire the professional preparation and setup. I wanted to say thank you deeply. I didn't consider it possible to be this good. Clearly not a first time.

I enjoyed the talks, most often in the technical track, but other tracks would have been very interesting too. The parallelism was making me do hard decisions.


The food was great too. I esp. liked the Asian day, but there was also Italian and French, and what many liked very much is that there was a Vegan food offer too. I do not live vegan style, but I appreciate good food and the vegan food often is that.

Social Event

The social event was a visit to a "Variete" (music hall, French origin), where I am sure, there will be images posted, I currently found this one , that my wife will find interesting too.


The quality of the organization team, the city "Leipzig", where we also got to have a guided city tour of fantastic enthusiasms, was very high. I knew Leipzig from earlier visits and liked it before, but this time it seemed everybody was even friendlier.


The convention place "Kubus" was very well chosen, absolutely ideal. It's got good equipment, and that large room setup, where you can make a split with movable walls, and have 3 big screens. The acoustics were pretty damn good there.

My own Presentation

As to my own presentation, it was well received, although I sort of regret that I agreed to have only 30m instead of original plan of 60m. I had so much to say.

I ended up with getting my manifesto part out, but that one pretty well. And it's OK I guess, because nobody really listens that long anyway. And my major points came across that way.

That focus on my Nuitka "manifesto" was probably a good idea. The talk will be available online as a video, I will link it then. The PDF that I presented only a small part of, is linked here. I believe it went pretty well.

I will use that content from the PDF in updated documentation (currently ongoing in PDF is work to use REST and document a lot more). The presentation was created with "rst2pdf", which I find is a fantastic tool.


Cython / lxml

Then contacts!

Early on I already made contacts with interesting people, e.g. with Dr.Stefan Behnel, author of lxml and core Cython developer. I him offered a beer for using his software in the best of Free Software traditions. He doesn't drink these, but a large mango juice counts too or so I assume.

We also talked about Cython and Nuitka, and the common history we had as well. For some time, I attempted to change Cython, but that failed to get the developers support at the time. Not wanting to deviate from PyRex clearly isn't the state anymore, but that was then.

We also had a evening session of showing each other the good and bad parts, comparing was quite fun. And it was quite interesting to the both of us. I believe we made friends and will only benefit another.

We discussed my goals, and I think we came to the conclusion that they are in fact different enough from Cythons. Although I go away with the sense, that of course Stefan believes, it would be better if I joined Cython. Naturally.

But that's not going to happen. I think i have a cleaner and better implementation now, closer to my goals with a realistic chance to succeed. To me it would be a step back to fix language parsing issues and incompatibilities of Cython, with the danger that my goals will not be shared.

As an example of these things, I would mention function call errors, where e.g. Cython gives different and sometimes worse error messages than CPython, and I designed the code so that it does things in that same order than CPython does.

It do not want to give different error messages, and who knows, somebody may check for the exception text and expect CPython output. In this case, I will rather accept a worse performance, than an incompatibility.

Eliminating function parameter parsing for the whole program as far as possible is going to be more worthwhile anyway.

But in my mind, Cython is something I can and do recommend. For as long as I am not able to declare Nuitka "useful" yet. That statement may come within a year though. In my mind, in many fields Nuitka is already superior.


Another interesting contact I made, was with the author of PyHasse. It's Rainer Bruggemann, who is a really nice and witty guy. He introduced me to how he applies graph theory to multi-parameter optimization problems.

We agreed that we will try and work together on this project. Hopefully it will come to pass. One thing I personally wanted, was to get into contact with people who understand or are part of the scientific community.

I can see what NumPy is. But I may never know myself what it really is, unless I find proxies, and make these kind of contacts. The same thing is true of Django, or e.g. Mercurial. I am positive though that with time, and such conferences, my knowledge of these will only grow.

We said that we will try and see how far we can go. In the worst case, Nuitka will not yet be useful, but I will have a clearer image what is needed.


I saw the presentation from Jan Dittberner and met him later too, asking him questions, and generally discussing Debian packaging of Nuitka. He encouraged me to contact the Debian Python Team, and so I will.

I used the chance to make contact with a Debian guy, who made a presentation on how to package Python modules for Debian. He gave me hints on how to solve that "find files near me" issue that plagues Nuitka just as much as other software. Really kind and helpful guy and clearly I admire Debian Developers, keep up the good work.


I also made contacts with lots of other people. Python is diverse and it was fun to get to know, many people with similar and entirely different backgrounds.

The mood was extremely constructive. Nuitka was well received, but that's not why I say it. There is that general sense of respect around that German community, you can feel how pretty much everybody is well established and doesn't have to disprove the others.


One keynotes speaker had a part about how trolling and hate is bad for a community, but that's not the German Python community.

Another keynote speaker (Paul Everitt) had a part about how Zope, which was kind of his project, failed in many ways. He seemed to be quite disappointed about that, which triggered me to point out, that he should start his story with Apache, and not see the "failure to integrate" as a failure.

If there had not been Apache failing, there wouldn't have been Zope, and then not Django, etc. that's kind of normal and actually good. He agreed and pointed out how Apache was created from another project that had failed to integrate people.

You either fork a projects code, or ideas. The fork still should credit and appreciate the predecessor/origin.

In my mind, Cython failed to integrate me. Which triggered me to come up with Nuitka, and as I will point out over time (there ought to be postings and there probably will be), some better approaches.

So not integrating me is not necessarily a failure. If it were not for Cython, there would not be Nuitka. The original projects will regret the fork/remake, but they probably shouldn't. Competition is good.

Lets repeat that

I believe the PyCON DE 2011 was a huge success. I will most likely go again to update people on Nuitka. It's already clear there will be a PyCON DE 2012 I understand. And I am aiming for a slot at PyCON EU 2012 next year too. I wanted to go in 2011, but need to not put it in my early booked holiday again.

But you know what Murphy says about that.

Kay Hayen

Nuitka git-flow

Hello there,

this is to let you know that I have switched Nuitka to the "git flow" development model. That means, now there is a supported stable version, and a develop branch, together with feature branches.


Git flow example for Nuitka release 0.3.12

Git flow example for Nuitka release 0.3.12


In case, you want and can improve the source visually or otherwise, please go ahead. I am using it for a presentation next week too, and would be glad if you could make it more pretty. My artistic skills are not the same as my programmer skills. :-)

So there is now always at least these 2 branches:

and then there may be feature branches, like this one currently:

These will only have certain life, until they are completed, then they are merge into "develop" and become part of the next release. This may or may not happen, depending on how things go.

Benefits of the new model

  • Hotfixes, typically bug fixes, can be made simultaneously on stable and develop branch. The git-flow package takes care of the merging to both.

  • Because that's so easy now, a stable version can be provided and supported for a longer time.

  • Features can be published while under development. My idea is that feature branches should basically work, but the bar will be lower. People can have a look at them, or start their own and make me integrate them.

Uses of Feature Branch

For example, in the new feature branch, a couple of boring things are happening. Support for frame stack will reduce the diff, as will some work to match CPython's choices for exception line numbers. Completing will take a while, but should not block a release. So this is best done in the feature branch, esp. as nothing is going to really depend on it.

General Picture

As you can see from this diagram, I am working mostly on documentation things. The new and improved README on develop, which is closer to a User Manual in PDF form, and other organization things, may get a release before the PyCon DE next week. The README also describes this process.

Hope is that with this approach, I will improve transparency (you can see earlier what i am working on, because there is now a place where things may break (develop) or may not yet be integrated or completed fully (feature branches) and yet be public.

The overhead appears to minimal thanks to "git-flow". Developing hotfixes is actually easier, when done on the stable branch, because problems cannot originate from the current development work that may or may not be all that perfect yet.

Kay Hayen

Nuitka Release 0.3.11

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new release of Nuitka with some bug fixes and portability work.

This release is generally cleaning up things, and makes Nuitka portable to ARM Linux. I used to host the Nuitka homepage on that machine, but now that it's no longer so, I can run heavy compile jobs on it. To my surprise, it found many portability problems. So I chose to fix that first, the result being that Nuitka now works on ARM Linux too.

Bug fixes

  • The order of slice expressions was not correct on x86 as well, and I found that with new tests only. So the porting to ARM revealed a bug category, I previously didn't consider.

  • The use of linux2 in the Scons file is potentially incompatible with Linux 3.0, although it seems that at least on Debian the sys.platform was changed back to linux2. Anyway, it's probably best to allow just anything that starts with linux these days.

  • The print statement worked like a print function, i.e. it first evaluated all printed expressions, and did the output only then. That is incompatible in case of exceptions, where partial outputs need to be done, and so that got fixed.

New Optimization

  • Function calls now each have a dedicated helper function, avoiding in some cases unnecessary work. We will may build further on this and in-line PyObject_Call differently for the special cases.


  • Moved many C++ helper declarations and in-line implementations to dedicated header files for better organisation.

  • Some dependencies were removed and consolidated to make the dependency graph sane.

  • Multiple decorators were in reverse order in the node tree. The code generation reversed it back, so no bug, yet that was a distorted tree.

    Finding this came from the ARM work, because the "reversal" was in fact just the argument evaluation order of C++ under x86/x64, but on ARM that broke. Correcting it highlighted this issue.

  • The deletion of slices, was not using Py_ssize for indexes, disallowing some kinds of optimization, so that was harmonized.

  • The function call code generation got a general overhaul. It is now more consistent, has more helpers available, and creates more readable code.

  • PyLint is again happier than ever.

New Tests

  • There is a new basic test OrderChecks that covers the order of expression evaluation. These problems were otherwise very hard to detect, and in some cases not previously covered at all.

  • Executing Nuitka with Python3 (it won't produce correct Python3 C/API code) is now part of the release tests, so non-portable code of Nuitka gets caught.


  • Support for ARM Linux. I will make a separate posting on the challenges of this. Suffice to say now, that C++ leaves way too much things unspecified.

  • The Nuitka git repository now uses "git flow". The new git policy will be detailed in another separate posting.

  • There is an unstable develop branch in which the development occurs. For this release ca. 40 commits were done to this branch, before merging it. I am also doing more fine grained commits now.

  • Unlike previously, there is master branch for the stable release.

  • There is a script "" (Update: meanwhile it was renamed to "") to produce a dependency graphs of Nuitka. I detected a couple of strange things through this.

  • The Python3 __pycache__ directories get removed too by the cleanup script.


We only have "PyStone" now, and on a new machine, so the numbers cannot be compared to previous releases:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

So this a speedup factor of 258%, last time on another machine it was 240%. Yet it only proves that the generated and compiled are more efficient than bytecode, but Nuitka doesn't yet do the relevant optimization. Only once it does, the factor will be significantly higher.


Overall, there is quite some progress. Nuitka is a lot cleaner now, which will help us later only. I wanted to get this out, mostly because of the bug fixes, and of course just in case somebody attempts to use it on ARM.

Going to PyCon DE

Hello everybody,

I am going to the German Python conference in Leipzip and I am going to have a presentation (link in German) there.

Of course it's about Nuitka (see What is Nuitka? ) and I hope it will get a lot of attendance. I am naturally very happy to have the opportunity to present it finally. I had wanted to visit PyCon EU, but the date was never known, and then my early booking holiday overlapped with it, so it was not an option.

Now giving this presentation will of course be exciting to me. I gave presentations as part of the day job many times, but this time it's obviously a different. It's also a first chance to meet the others as I never was at a Python conference before, and that will be interesting in itself.

Presenting Nuitka will of course be easy for me. Now I need to plan what I want to be able to present for a demo. Running a big thing like Mercurial would be nice, but I honestly don't know, if that will even be difficult, or if it will take a lot of work. Also the amount of documentation available for Nuitka should increase as part of this. Designs, etc. could be made into diagrams, so people who want to join will have it easier.

Lots of possibilities, and then there is only going to be one reality. Lets hope it's good. :-)