My Europython 2013 report

Back from Europython 2013 in Florence for a while now. I had a presentation about my Python compiler Nuitka. You can watch the Video on Youtube. I believe it's absolutely worth your time. I was not doing a lot of "why" at all now, only "what", and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good and at ease. The state presented is very good and progress - there was a lot, so that was great. The slides of the talk may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight then, and also much more excited, at least initially. Background: I was surprised then that Guido van Rossum was attending the talk, plus I was kind of not in the state of this year, the confidence that people should believe, that things will work after that, was not yet there. I knew it, but standing in front of a croud and saying, look I am attempting what you consider impossible, or even have failed at, is not easy:

There are also the slides of that talk which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven things. Let's see what this year does to it. I would hope for real type inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out of the box experience, and how it just works. That's right. I put a lot of focus on stability, correctness, and ease of use. In fact, since my return, I have mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays), it's make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a lot of users, and not as many contributions or donations as I would hope for, but that's kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important exception are scientific people, doing number crunching. And I made some interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which were unfortunately regressions of 0.4.4, and these got fixed in two hotfix releases. Currently is released, and contains a huge amount of fixes, mostly stuff found by the surge new users.

But I deserved that, having said that I didn't have to do hotfixes for a while now.


Such a lovely place, and so lovely people. It's a tourist place, true, but it's very open minded. The food is extremely good. And going from the conference hotel straight over to the river to meet up and have a few beers was very comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it's moving on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large amount of non-satisfaction, was that the community is fed up with the leadership. Questioned about lack of Python3 adoption, a PSF guy in his presentation said they were 3 years into a 5 years plan, and got interrupted by laughter.

It may be about time, somebody actually gets up, removes the GIL and has a Python2.8 that is compatible. Obviously that's much more useful. It could be a Python3.x with compatibility names of modules, print statement, and dict.iteritems added, probably also making some of the unicode mess more harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython top versions, to add things, that would allow Nuitka to work better with it, that could be a staging ground, where such changes are nurtured. On the other hand, Nuitka is too irrelevant yet to drive this, and it would detract me away from it.


So Nuitka got a lot more exposure. And one guy watching the talk even created a really nice logo for it. I will make it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And reinforcement that what I am doing is actually useful to some people already.


My travel to Florence was in part funded by donations. Thanks a lot to all of you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there via train.


The Europython 2012 and 2013 both are among the times of my life.

Nuitka Release 0.4.4

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release marks the point, where Nuitka for the first time supports all major current Python versions and all major features. It adds Python 3.3 support and it adds support for threading. And then there is a massive amount of fixes that improve compatibility even further.

Aside of that, there is major performance work. One side is the optimization of call performance (to CPython non-compiled functions) and to compiled functions, both. This gave a serious improvement to performance.

Then of course, we are making other, long term performance progress, as in "--experimental" mode, the SSA code starts to optimize unused code away. That code is not yet ready for prime time yet, but the trace structure will hold.

New Features

  • Python3.3 support.

    The test suite of CPython3.3 passes now too. The yield from is now supported, but the improved argument parsing error messages are not implemented yet.

  • Tracing user provided constants, now Nuitka warns about too large constants produced during optimization.

  • Line numbers of expressions are now updates as evaluation progresses. This almost corrects.

    Finally improves Issue#9. Now only expression parts that cannot raise, do not update, which can still cause difference, but much less often, and then definitely useless.

  • Experimental support for threads.

    Threading appears to work just fine in the most cases. It's not as optimal as I wanted it to be, but that's going to change with time.

New Optimization

  • Previous corrections for ==, !=, and <=, caused a performance regression for these operations in case of handling identical objects.

    For built-in objects of sane types (not float), these operations are now accelerated again. The overreaching acceleration of >= was still there (bug, see below) and has been adapted too.

  • Calling non-compiled Python functions from compiled functions was slower than in CPython. It is now just as fast.

  • Calling compiled functions without keyword arguments has been accelerated with a dedicated entry point that may call the implementation directly and avoid parameter parsing almost entirely.

  • Making calls to compiled and non-compiled Python functions no longer requires to build a temporary tuple and therefore is much faster.

  • Parameter parsing code is now more compact, and re-uses error raises, or creates them on the fly, instead of hard coding it. Saves binary size and should be more cache friendly.

Bug Fixes

  • Corrected false optimization of a >= a on C++ level.

    When it's not done during Nuitka compile time optimization, the rich comparison helper still contained short cuts for >=. This is now the same for all the comparison operators.

  • Calling a function with default values, not providing it, and not providing a value for a value without default, was not properly detecting the error, and instead causing a run time crash.

    def f(a, b=2):

    This now properly raises the TypeError exception.

  • Constants created with + could become larger than the normally enforced limits. Not as likely to become huge, but still potentially an issue.

  • The vars built-in, when used on something without __dict__ attribute, was giving AttributeError instead of TypeError.

  • When re-cursing to modules at compile time, script directory and current directory were used last, while at run time, it was the other way around, which caused overloaded standard library modules to not be embedded. Corrects Issue#94.

    Thanks for the patch to James Michael DuPont.

  • Super without arguments was not raising the correct RuntimeError exception in functions that cannot be methods, but UnboundLocalError instead.

    def f():
        super()  # Error, cannot refer to first argument of f
  • Generators no longer use raise StopIteration for return statements, because that one is not properly handled in try/except clauses, where it's not supposed to trigger, while try/finally should be honored.

  • Exception error message when throwing non-exceptions into generators was not compatible.

  • The use of return with value in generators is a SyntaxError before Python3.3, but that was not raised.

  • Variable names of the "__var" style need to be mangled. This was only done for classes, but not for functions contained in classes, there they are now mangled too.

  • Python3: Exceptions raised with causes were not properly chaining.

  • Python3: Specifying the file encoding corrupted line numbers, making them all of by one.


  • For containers (tuple, list, set, dict) defined on the source code level, Nuitka immediately created constant references from them.

    For function calls, class creations, slice objects, this code is now re-used, and its dictionaries and tuples, may now become constants immediately, reducing noise in optimization steps.

  • The parameter parsing code got cleaned up. There were a lot of relics from previously explored paths. And error raises were part of the templates, but now are external code.

  • Global variable management moved to module objects and out of "Variables" module.

  • Make sure, nodes in the tree are not shared by accident.

    This helped to find a case of duplicate use in the complex call helpers functions. Code generation will now notice this kind of duplication in debug mode.

  • The complex call helper functions were manually taking variable closure, which made these functions inconsistent to other functions, e.g. no variable version was allocated to assignments.

    Removing the manual setting of variables allowed a huge reduction of code volume, as it became more generic code.

  • Converting user provided constants to create containers into constants immediately, to avoid noise from doing this in optimization.

  • The site module is now imported explicitly in the __main__ module, so it can be handled by the recursion code as well. This will help portable mode.

  • Many line length 80 changes, improved comments.

New Tests

  • The CPython3.3 test suite was added, and run with both Python3.2 and Python3.3, finding new bugs.

  • The doctest to code generation didn't successfully handle all tests, most notably, "" was giving a SyntaxError and therefore not actually active. Correcting that improved the coverage of generator testing.


  • The portable code is still delayed.

    Support for Python3.3 was a higher priority, but the intention is to get it into shape for Europython still.

    Added notes about it being disabled it in the "User Manual" documentation.


This release is in preparation for Europython 2013. Wanted to get this much out, as it changes the status slides quite a bit, and all of that was mostly done in my Cyprus holiday a while ago.

The portable code has not seen progress. The idea here is to get this into a development version later.

Nuitka Release 0.4.3

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka.

New Features

  • Support for new compiler: Microsoft Visual C++.

    You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows.

  • Support for NetBSD.

    Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic "fibers" implementation.

  • Support for Python3 under Windows too.

    Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to C:\Python26 or C:\Python27 for Nuitka to be able to find it.

  • Enhanced Python 3.3 compatibility.

    The support the newest version of Python has been extended, improving compatibility for many minor corner cases.

  • Added warning when a user compiles a module and executes it immediately when that references __name__.

    Because very likely the intention was to create an executable. And esp. if there is code like this:

    if __name__ == "__main__":

    In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command

    nuitka --execute module

    is behavioral more like

    python -c "import module"

    and that was a trap for new users.

  • All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don't have to implement for specific architectures anymore.

Bug Fixes

  • Dictionary creation was not fully compatible.

    As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn't observe this before, and it's likely the new dictionary implementation that finds it.

    Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing.

  • Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was.

  • Using a __future__ import inside a function was giving an assertion, instead of the proper syntax error.

  • Python3: Do not set the attributes sys.exc_type, sys.exc_value, sys.exc_traceback.

  • Python3: Annotations of function worked only as long as their definition was not referring to local variables.

New Optimization

  • Calls with no positional arguments are now using the faster call methods.

    The generated C++ code was using the () constant at call site, when doing calls that use no positional arguments, which is of course useless.

  • For Windows now uses OS "Fibers" for Nuitka "Fibers".

    Using threads for fibers was causing only overhead and with this API, MSVC had less issues too.


  • Accepting Donations via Paypal, please support funding travels, website, etc.

  • The "User Manual" has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated.

  • The Changelog has been improved for consistency, wordings, and styles.

  • Nuitka is now available on the social code platforms as well

  • Removed "", which is practically useless, as tests now clean up after themselves reasonably, and with git clean -dfx working better.

  • Removed "" script, which was only setting the PATH variable, which is not necessary.

  • Added check-with-pylint --emacs option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint.

  • Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated.


  • Removed useless manifest and resource file creation under Windows.

    Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it.

  • PyLint massive cleanups and annotations bringing down the number of warnings by a lot.

  • Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes.

  • Do not track needed tuple, list, and dict creation code variants in context, but e.g. in nuitka.codegen.TupleCodes module instead.

  • Introduced an "internal" module to host the complex call helper functions, instead of just adding it to any module that first uses it.

New Tests

  • Added basic tests for order evaluation, where there currently were None.

  • Added support for "2to3" execution under Windows too, so we can run tests for Python3 installations too.


The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now.

Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization.

Nuitka on Github, Bitbucket and Gitorious

This is to let you know, that I have finally carved in and created accounts on all those "social" platforms, that deal with code. So far I had resisted their terms (you pay us when we get sued), but I have given up on that stance, it's likely irrelevant, as Nuitka won't get sued, and if it is, those are not the issues.

And I don't want to feel too old school it seems. And who knows, maybe this helps Nuitka to get more contributions.

So these are the landing pages:

  • Bitbucket

    I kind of like the looks of it the most. Also it's very popular for Python people due to its Mercurial ties. I also liked that it imported my Nuitka repository easily.

  • Github

    Is not being on Github allowed these days? Anyway, from ease of use, the interface looks more powerful.

  • Gitorious

    The interface and looks definitely are worst. But it's Free Software, and an open platform, so it has to be in the list.

Let me know what you think. I am going to push 'develop' at pre-release and 'master' at release time, automatically from the release scripts into all these. Hopefully that way, they turn out to be time savers no wasters.

And yes, I couldn't decide on one. Since I fancy Free Software, Gitorious probably would have won, but it would also miss the point of expanding the reach. And for as long as I don't use their issue tracking or Wikis, it appears to be not necessary to make that decision.


The preferred for use, and most up to date repository will remain the one from Downloads.

So fork away, like it, and stuff. Awaiting your pull requests...

Going to Europython 2013

I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.

Things to talk about


Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.

What I would like to get done until then:

  • Win64

    One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.

  • Threading

    And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.

    Since it's not totally a priority right now, one of the next releases will support it, likely before the conference.

But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.

Last years questions


For instance, writing to built-ins, what will/did happen.

Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.

This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it's not effective in others (len, str).

Good news is that we got contributed a "compiled built-ins" code, where we now will be able to see such writes. Now it's only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.

The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.


Currently Nuitka does nothing of this, but it will.

Debugger - pdb

The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.

Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.


I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don't like them either, so my experience was non-existing.

But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.

C++ to Python gaps

I consider all of these more or less solved.

Well maybe except recently arose issues with MSVC for "function calls". It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.

But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.


There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).

And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.

Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.

But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it's the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.


The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.

And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.

If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.

Future Work

Generally the talk will be more directed at the future, although this is kind of a dark area now. That's its nature I guess.


The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say "Python is too dynamic (to be statically compiled)" is that Python values may escape to unknown code that changes them very often.

I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.

Compiled Modules

And I believe with "compiled modules" potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.

I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn't have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.

Compiled Locals

Who knows, we might even have a "compiled locals" as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.

But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don't know if it might use it.

Whole Program Analysis

Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.


Last time, I spent a lot of time on justification, "why a new project?", "why not work with the others?", what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.


And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It's probably fair to allow people to make donations if they feel they want to further the project, but don't know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikely though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.


Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these "times of my life". Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.

Support for MSVC upcoming

The good news are lately coming in high density, although I barely have the opportunity to work on Nuitka. Check the page What is Nuitka? for clarification of what it is now and what it wants to be.

It seems my recent call for help was at least partially effective. One the major points has been addressed. In the next release, Visual Studio 2008 and 2010 will both work for Nuitka, right now it's only in git develop branch.

That is really nice, as it will make Nuitka on Windows a whole lot more accessible. Combine that with portable support and it's up to solving problems with distributing on Windows it seems.

The other two things listed there have received no feedback yet. So if you want to help with those, feel free to do so. Nuitka needs you.

Nuitka needs you - a call for help

Update 3

And the first point has been done too, Python3.3 is now fully supported, including yield from syntax.

Update 2

The third point, Windows 64 support has been done as well. Turns out, that I do in fact own only Win64 systems, and with MSVC support in place, only a few portability fixes were needed.

Help with the other point, "re-formulation of yield", would still be more than welcome, and no signs of progress there yet. So stop thinking "you could", enter telling people "you did" mode.


The second point, Fibers implementation for Windows, has been done. Thanks for the help you people provide. The next release will contain it.

Hello everybody,

the Python compiler Nuitka has come an ever longer way, and currently I have quite a bunch of issues, that I believe could well need your help. These are all issues of some important and significance, yet self contained jobs that you may enjoy.


You can check the page What is Nuitka? for clarification of what it is now and what it wants to be.

  • Python 3.3 - reformulation of yield from needed.

    As you can see, covering all the CPython 2.6, 2.7, and 3.2 language features is already something. Also CPython 3.3 is now working on a basic level. Other projects are far, far away from that.

    Many language constructs, such as the with statement are re-formulated into other constructs. This makes it possible to work with a simple core for optimization, and to reduce the complexity a lot. For the with statement case, it's changed to try/finally and try/except statements, together with a few temporary variables. Check the Nuitka source of it:

    There is also descriptions of all these re-formulations in the developer manual so you can see how this is done.

    Now check PEP 380 for the details of it, and your task would be to come up with a re-formulation of yield from statements to the semantically equivalent.

    The benefit is, you are working on a totally self-contained level. The re-formulation only needs to use ast node information, and turn that into Nuitka nodes. The problem will be fully solved this way.

    And once we have this, we can pronounce CPython3.3 as fully supported. So if you would like to see this happen, please join.

  • Windows - MSVC support needs low level code help


    This is now done.

    The support for MSVC is already mostly there. If you execute the environment setting script, and use Nuitka under Windows, it will work. The only problem is that for the generator functions, Nuitka is using a concept called "Fiber", which is basically just switching C stacks, so called co-routines.

    It works great for Linux and UNIX. For Windows, Fibers are currently using threads, which wouldn't be as much of a performance problem, because they are very lightweight, it is slightly wasteful only.

    But, the bad thing, is with these threads switching and C++ exceptions used to represent Python exceptions, no combination of MSVC options seems to carry the day and keep the exception in the current thread. They are caught in the wrong thread, and make generators misbehave, crash.

    This is the only thing that holds back proper MSVC support at this time, and it has been like this for a while. With MinGW, the issue does not exist, but MinGW is a stranger to many, and MSVC will be more efficient code generation, so we would like to have that.

    So what it takes is somebody familiar with Windows and its ABI, esp. register usage to write code that swaps the registers around, so we can have multiple C stacks without threads for it too. This may involve messing around with exception jump back pointers as well.

    It needs somebody who can handle this and does it for Nuitka, and then it would be well supported. It may involve writing some assembler code. The job is also very well contained, but also requires very specific knowledge, so please join me here. If you think you can do this, do it.

  • Windows 64 bit

    I have no access to Win64. I bought many Windows versions, but not the Win64 variants so far. I understand that there are CPython builds for it, and probably a MinGW matching it, as well as a matching newer MSVC, that may work better for C++ exceptions than the one CPython 2.x is tied to.

    Doing that, you would mostly only have to play around with SingleExe.scons, which abstracts the details of it all. One issue you will encounter is that Scons needs a Python2, and other small things. This is mostly only about porting Nuitka, and could be done by anybody with access to Win64 and willing to fiddle around a bit.

So these are 3 different ways you can help Nuitka. Creating the Python compiler is a huge effort. And if you help it, we will sooner arrive at its goals. Join the Nuitka mailing list (since closed) or contact me directly if you choose to help with any of these issues.

Wait, actually there is another way you can help. Please spread this news to people that you believe might be willing to help. And allow me to plug this, I am doing Nuitka in my spare time. If you feel, you cannot help, but still would like to support it, you can still make donations to allow me to travel to conferences and spend more time on it myself.

Anyway, it would be great if these 3 things got sorted out. Come and join me for this grand vision of the Python compiler.


Support for portable (standalone) programs

This post is about a feature often requested, but so far not available feature of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

In forums, and in Google, people are looking at a Python compiler, also as a way of deployment. It should offer what py2exe does, allow installation independent of Python.

Well, for a long time it didn't. But thanks to recent contributions, it's upcoming for the next release, Nuitka 0.4.3, and it's in the current pre-releases.

It works by adding --portable to the command line. So this should work for you:

nuitka-python --recurse-all --portable

Right now, it will create a folder "_python" with DLLs, and "" with standard library modules used along to the "your-program.exe". Copy these to another machine, without a Python installation, and it will (should) work. Making that statement fully true may need refinements, as some DLL dependencies might not be defined yet.


We may improve it in the future to meld everything into one executable for even easier deployment.

You are more than welcome to experiment with it. To do so, download Nuitka from the download page and give it a roll.


Of course, Nuitka is not about replacing "py2exe" primarily, it's only a side effect of what we do. Our major goal is of course to accelerate Python, but surely nobody minds achieving the two things at the same time.

And while the post is labeled "Windows", this feature also works for Linux at least too. It's just that the lack of Python installations on client systems is far more widespread on this platform.

To me, as this is from a contributor, it's another sign of Nuitka gaining adoption for real usage. My personal "py2exe" experience is practically not existing, I have never used it. And I will only merge the improvements into the Nuitka project as provided by others. My focus for the time to come is of course the compile time and run time optimization.

NetBSD support upcoming

My first real UNIX ever was a NetBSD. That was now about 22 years ago. I am still sentimental about it. I had installed it last about 8 years ago. And I still like it. Back in the days, it was the first UNIX to encounter for me, running on Amiga hardware, first of a friend, then on my own.

Recently, there had been support for Nuitka on FreeBSD added. A lot of people use it on the web, and some want to use Nuitka to improve their Python performance, so this is kind of relevant.

There were issues resolved, but in the end, something was with Clang on FreeBSD 8, that I could for the life of it, not resolve remotely. So I attempted to install it myself. Using "virt-install", these things are a breeze now. I had already done it with CentOS6 before to test the RPM repositories of Nuitka. That "virt-install" is a wonderful thing by itself, making virtualisation somewhat useful. It's only a pity, that I can't just install other qemu support architectures. I would love to checkout Nuitka on PowerPC.


If you could check out Nuitka on other Linux architectures than x86_64, x86, or arm, that would be great.

This is the report of getting NetBSD supported. It was a quite an interesting story that I would like to share with you.

Naivly I was assuming, that it would be just for fun, and that Nuitka will work right away. Little did I know.

On FreeBSD 9 the minimal install medium was chosen, and entered its ports collection, installed git, cloned Nuitka, and ran the tests, successfully right away. Now that is unfair, in the Nuitka there were tons of "Linuxism" already removed. In fact, it had to work, and on the newest FreeBSD (version 9.1) and then it did. Great!


If you would like to add Nuitka to FreeBSD's ports, please do so. It should be really easy now.

On NetBSD, things were unfortunately a little different. I also chose minimal system. After going through "pkg source" boot strap and git install, I cloned Nuitka, and then tried to start it. First off, it couldn't locate "python" at all. I am using /usr/bin/env python already. But Python2 was on the system. I ended up creating the "python" link myself. What I should have done according to "#netbsd" is to install the software, and indeed, python2.7 install gives an installation of Nuitka that is executable.

Next up, you need to know about "Fibers" in Nuitka. These are used for C co-routines, used to implement Python generators. They have an interface, that is very close to makecontext/swapcontext routines in C.

For ARM and x86_64 Linux we have optimized code, that switches faster, but other platforms, including x86 Linux, use the generic implementation, also because it normally is very fast already.

Now you have to know that since 2001 the interface is deprecated and shall not be used. And next up, is that on NetBSD, makecontext gave a segfault only. So I ran to "#netbsd" and asked.

Now that was a very friendly experience. Of course, I had to give a rationale for using an obsolete interface. It's not quite obvious, why threads wouldn't be a better choice. And maybe they are, but they definitely have more overhead associated, and if they never run at the same time, why use them.

Ultimately it helped to point out, that for a user of 22 years, an interface that is only obsolete for 11 years, is not quite as horrifying as for others.

And they helped me through it. And it turns out, funny thing. For the context to setup, you are allocating a stack to use for the C routine, and you "get" the current context, then you make a new one. All the examples have a certain order of doing it. And my code did it the other way around. No system but NetBSD noticed.

On FreeBSD and Linux, it didn't matter. But it seems, that the needed getcontext call was overwriting my stack pointer now with the current stack. And makecontext deeply hated that, a lot. It was preparing that stack to be used, while it was in usage. Doesn't sound like a good task to give to it, right? My fault truly, because every example on every man page, on all systems, was doing it differently. But then they were also all using arrays from the local stack, so quite obviously that was not real code.

So that was fixed, and all good? No! Next thing was it crashed when free happened in Python on a compiled frame object, in a later part of a test that heavily uses generators. Turns out, malloc information was corrupted. I had to suspect the generic "Fiber" code, but that took me a while to figure out.

And how could my simple malloc and free do that, and make it happen. When I knew that a context would not longer be used (the generator has finished, the generator object deleted, etc), I would look at the context handle stack pointer and free it.

But that pointer changed. Something totally unexpected (by me obviously), but it also explains the earlier problem. For all systems, I had used so far, this pointer was not being changed, and remained the same. So I could free it from there. It worked fine, but not on NetBSD. And it wasn't correct anywhere.

It seems NetBSD is doing something clever, since instead of saving the stack pointer register in a separate area, it saves it to that place originally specified. It's quite obviously an improvement, in that you save the pointer.

It's only bad, that now to make up for this savings, I have added the pointer in a separate field, which won't be changed, so I can free it again. If one needs it again, and that's not unlikely, you have to remember it elsewhere. So maybe that idea is not that clever. But it surely was wrong by me to assume that the provided value would not be touched.

So, these are 2 bugs it found. The wrong order of calls. And the usage of a pointer, that may have been changed. This can only help with other systems, or possibly architectures under Linux.

While this is all description of nasty problems, it's also the report of the solution, and it was big fun. I would also like to compliment "#netbsd" for being very helpful and friendly with my porting of Nuitka. I highly enjoyed doing so. It was a lot of fun. I know that it's probably on a very tiny amount of people that uses both NetBSD and Nuitka, but still.

If this Nuitka project were about market share, it wouldn't exist. And I can work for market share on another day.

Nuitka Release 0.4.2

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the performance diagrams got expanded.

New Features

  • Support for FreeBSD.

    Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process.

  • New option for warning about compile time detected exception raises.

    Nuitka can now warn about exceptions that will be raised at run time.

  • Basic Python3.3 support.

    The test suite of CPython3.2 passes and fails in a compatible way. New feature yield from is not yet supported, and the improved argument parsing error messages are not implemented yet.

Bug Fixes

  • Nuitka already supported compilation of "main directories", i.e. directories with a "" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe"

    # ls directory
    # nuitka --exe directory
    # ls
    directory directory.exe

    This makes this usage more obvious, and fixes the older issue Issue#49 for this feature.

  • Evaluation order of binary operators was not enforced.

    Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects Issue#61.

  • Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.

  • Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.

    global a  # Not in a function, but on module level. Pointless but legal!
    a = 1

    Effectively these statements can be ignored. Corrects part of Issue#65.

  • Future imports are only legal when they are at the start of the file.

    This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of Issue#65.

  • Raising exceptions from context was leaking references.

    raise ValueError() from None

    Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects Issue#76.

  • Importing the module that became __main__ through the module name, didn't recurse to it.

    This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don't share data. Corrects Issue#68.

  • Compiled method repr leaked references when printed.

    When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point. Corrects Issue#81.

  • The super built-in leaked references to given object.

    This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run. Corrects Issue#81.

  • The raise statement didn't enforce type of third argument.

    This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked.

  • Python3 built-in exceptions were strings instead of exceptions.

    A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects Issue#82.

  • The -nan and nan both exist and make a difference.

    A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka. Corrects Issue#75.

  • Wrong optimization of a == a, a != a, a <= a on C++ level.

    While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for ==, !=, and <=.

  • The sys.executable for nuitka-python --python-version 3.2 was still python.

    When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.

  • Keyword only functions with default values were losing references to defaults.

    def f(*, a=X()):
    f()  # Can crash, X() should already be released.

    This is now corrected. Of course, a Python3 only issue.

  • Pressing CTRL-C didn't generate KeyboardInterrupt in compiled code.

    Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.

  • For with statements with return, break, or continue to leave their body, the __exit__ was not called.

    with a:  # This called a.__enter__().
        return 2  # This didn't call a.__exit__(None, None, None).

    This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing. Corrects Issue#59.

  • Starting with Python3, absolute imports are now the default.

    This was already present for Python3.3, and it turns out that all of Python3 does it.

New Optimization

  • Constants are now much less often created with pickle module, but created directly.

    This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.

    a = ("something_special",)
    b = "something_special"
    assert a[0] is b  # Now true

    This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects.

    Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.

  • Continue statements might be optimized away.

    A terminal continue in a loop, was not optimized away:

    while 1:
        continue  # Now optimized away

    The trailing continue has no effect and can therefore be removed.

    while 1:
  • Loops with only break statements are optimized away.

    while 1:

    A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.

New Tests

  • Added tests for the found issues.

  • Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered.

  • Running the "CPython3.2" test suite with Python3.3 based Nuitka works and found a few minor issues.


  • The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file.

    The page got improved with logos for the distributions.

  • Added "ownssh" as contributor.

  • Revamped the "User Manual" in terms of layout, structure, and content.


This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.