Nuitka Blog - Posted in 2013

Re: About Python 3

2013-12-30T20:56:32+01:00

In Alex Gaynor’s post there is just about everything right. I still want to add my take on it.

Python3 is a fork of the Python community, that has left the user base largely behind. After developing features very conservatively for a long time (from __future__ import division and stuff), where features were first available but not active by default, that whole “Python 3000” thing came up.

Misguided, after having maintained CPython on a level of excellence for such a long time, there was a sense of “finally we get to make free choices”. I can understand that. And having witnessed Gnome 2.0.0, and KDE 4.0.0, I even start to believe that there is some kind of underlying law. Something that makes people want to start over.

That “5 years plan” Alex mentioned has failed, and everybody knows that without solving the GIL within the next 5 years, i.e. another incompatible Python change, it will likely become obsolete.

In terms of Python2.8, call it that way. Or have a 3.5 that has print statement, and then all kinds of incompatible changes to Python3 to make it behave sane bytes(7) needs to do what str(7) once it. That would be about from __past__ import .... I suppose.

I also had another idea, having a “python2” built-in module that carries a CPython2 interpreter and proxy objects for instances, where they talk to another with incompatible types (new bytes/unicode to old str).

Due to my work on that Python compiler that should have existed 20 years ago, I am familiar enough, to actually be able and create that kind of CPython2/CPython3 hybrid.

But the main problems with all of that are:

CPython is developed by volunteers, and as such, they are neither obliged to do what we need, nor will they bend to any amount of posting. We need to get our acts together or it’s not happening, and we are people who care about Python relevancy.

There may not be nearly enough overlap of people capable and caring. And that would be a tremendous amount of work. Highly speculative, in the sense, that it may not see fruits.
And then, when you got that sorted out, expect a hostile reaction from the core developers and/or PSF. Whatever it is, that you develop, you won’t be able to call it Python 2.8, as they are going to hate it.
And finally, as a course of action, that may even bring the complete downfall of Python as a community. The controversy associated is not for everybody, and we don’t need a libreoffice/openoffice situation in Python, do we.

It’s a pity, because clearly, for Nuitka there would be a bunch of patches, that I would like to make, that would make my life far easier, and even interpreted Python faster.

Nuitka Standalone Mode is Work in Progress

2013-12-27T10:48:22+01:00

Many of you who turn to my easy to use, highly compatible Python compiler Nuitka, do this mostly because they seek to solve the deployment problem that Python suffers from.

Be this, because you want to use a newer Python2.7 on RHEL5 without installing anything at all there. Or because Windows is difficult to tackle otherwise.

For the longest time, Nuitka had not offered anything in this domain, focusing solely on being an accelerator. Lately, I have taken up the challenge and polished initial solutions submitted by contributors.

This first showed up in the 0.4.7 release, but turned out relatively weak. While first examples were working on Linux, it was not working at all on Windows (anymore). And basically there was a huge lack of tests.

Actually I didn’t mean for it to be released with that feature, but as parts of it seemed to work, I did so. But truth to be sad, that feature is not nearly as polished in that release as you would like it to.

In current development releases, of what is going to become 0.5.0 really soon now, it’s much better already. More things actually work. But it appears, there will be more ground to cover, and this is a lot of stuff to sort out.

So, this is mostly about asking you two things. Give that development release a try and report issues you have with it. And help me.

And have patience. I am developing Nuitka as an accelerator on a “no known bugs” basis. That means, once I know of a bug, I will fix it. OK, some issues in fact take longer, but then it really is not important at all, but difficult at the time. For standalone mode, I can’t do it that way, or I would have to neglect the acceleration parts, which I totally don’t want to do.

Because while you maybe are only interested in a packaging solution, many others would like to have that orders of magnitude speedup that I have been aiming for and that feels near now. This is about making Python a viable language for more uses than it currently is.

So why do it in the first place. For one, I am hoping that it helps people to not turn away from Python. And second, and more important, I am hoping that by making it more useful, more people will join me. (Oh, and thirdly, it’s also a nice puzzle to solve. I seem to enjoy that.)

Ultimately both modes will be needed, standalone, and acceleration. And it seems like I am working to provide both. For standalone, more often, than seeking to avoid bugs as far as possible, I am going to rely on your participation.

So join Nuitka. Now. Download from here. Join the mailing list (since closed). And help if you can. And yes you can.

Nuitka Release 0.4.7

2013-12-05T01:45:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release includes important new features, lots of polishing cleanups, and some important performance improvements as well.

Bug Fixes

The RPM packages didn’t build due to missing in-line copy of Scons. Fixed in 0.4.6.1 already.
The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already.
The Windows installer was not including Scons. Fixed in 0.4.6.3 already.
Windows: The immediate execution as performed by nuitka --execute was not preserving the exit code.
Python3.3: Packages without __init.py__ were not properly embedding the name-space package as well.
Python3: Fix, modules and packages didn’t add themselves to sys.modules which they should, happened only for programs.
Python3.3: Packages should set __package to their own name, not the one of their parents.
Python3.3: The __qualname__ of nested classes was corrected.
For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons.

New Features

The “standalone” mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already.

To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized.
New option --python-flag allows to specify flags to the compiler that the “python” binary normally would. So far -S and -v are supported, with sane aliases no_site and trace_imports.

The recommended use of --python-flag=-S is to avoid dependency creep in standalone mode compilations, because the site module often imports many useless things that often don’t apply to target systems.

Optimization

Faster frame stack handling for functions without try/except (or try/finally in Python3). This gives a speed boost to “PyStone” of ca. 2.5% overall.
Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to “PyStone” of ca. 0.5% overall.
Python2: Much quicker calls of __getattr__ and __setattr__ as this is now using the quicker call method avoiding temporary tuples.
Don’t treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies.
Create unicode constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up.
Directly create int and long constants outside of 2**31 and 2**32-1, but only limited according to actual platform values. Affects only program start-up.
When creating set values, no longer use a temporary tuple value, but use a properly generated helper functions instead. This makes creating sets much faster.
Directly create set constants instead of un-streaming them. Affects only program start-up.
For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement.
Module variables are now accessed even faster, the gain for “PyStone” is only 0.1% and mostly the result of leaner code.

Organizational

The “standalone mode” code (formerly known as “portable mode” has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement.
Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported.
Added package for openSUSE 13.1 for download.
Nuitka is now part of Arch and can be installed with pacman -S nuitka.
Using dedicated Buildbot for continuous integration testing. Not yet public.
Windows: In order to speed up repeated compilation on a platform without ccache, added Scons level caching in the build directory.
Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ccache and Scons as well.

Tests

The programs tests cases now fail if module or directory recursion is not working, being executed in another directory.
Added test runner for packages, with initial test case for package with recursion and sub-packages.
Made some test cases more strict by reducing PYTHONPATH provision.
Detect use of extra flags in tests that don’t get consumed avoiding ineffective flags.
Use --execute on Windows as well, the issue that prevented it has been solved after all.

Cleanups

The generated code uses const_, var_, par_ prefixes in the generated code and centralized the decision about these into single place.
Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation.
The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious.
Massive general cleanups, many of which came from new contributor Juan Carlos Paco.
Moved standalone and freezer related codes to dedicated package nuitka.freezer to not pollute the nuitka package name space.
The code generation use variable identifiers and their accesses was cleaned up.
Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them.
Moved main program, function object, set related code generation to dedicated modules.

Summary

This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation.

The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead.

Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future.

Nuitka in Arch Linux

2013-11-30T14:17:59+01:00

Got this message:

Just thought you wanted to know that Nuitka is now also an official
Arch Linux package and can be installed simply with:

pacman -S nuitka

So, that is another sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Many thanks to Alexander Rødseth and Panagiotis Mavrogiorgos for maintaining the Nuitka packages inside Arch and on AUR.

Nuitka and GUIs

2013-11-26T09:35:53+01:00

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

Nuitka is a typical compiler, just for Python, and aims to be integrating with build tools, and as such is so much a command line tool, it hurts.

Yet, there are GUIs. Yes, plural, there are even two.

Many of you are maybe using Ninja IDE already. I have developed most of early Nuitka in vim, and later, mostly because of Nuitka, switched back to Emacs. I had switched from Emacs at one point 12 years ago, but then went back a couple of years ago.

But recently +Juan Carlos convinced me to give Ninja a try. Because not only is he the creator the Nuitka GUI, but also the creator of the Ninja IDE plugin for Nuitka.

So on Windows, I now use Ninja, and I like it, a lot.

I must admit, I am not a heavy user of both Nuitka GUIs, rather being a command line guy I suppose, but that is just great. I would like to thank Juan Carlos for these contributions.

They are one sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Nuitka Release 0.4.6

2013-10-28T08:52:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups.

Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot.

Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues.

And then there is performance. As can be seen in the performance graph, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types.

And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix.

Bug Fixes

Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. Fixed in 0.4.5.1 already.
Support Python being installed to a path that contains spaces. Fixed in 0.4.5.2 already.
Windows: User provided constants larger than 65k didn’t work with MSVC. Fixed in 0.4.5.3 already.
Windows: The option --windows-disable-console was not effective with MSVC. Fixed in 0.4.5.3 already.
Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn’t honor the target architecture. Fixed in 0.4.5.3 already.
When creating Python modules, these were marked as executable (“x” bit), which they are of course not. Fixed in 0.4.5.3 already.
Python3.3: On architectures where Py_ssize_t is not the same as long this could lead to errors. Fixed in 0.4.5.3 already.
Code that was using nested mutable constants and changed the nested ones was not executing correctly.
Python2: Due to list contractions being re-formulated as functions, del was rejected for the variables assigned in the contraction.
```
[expr(x) for x in iterable()]

del x  # Should work, was gave an unjustified SyntaxError.
```

New Features

Compiled types when used in Python comparison now work. Code like this will work:
```
def f():
    pass


assert type(f) == types.FunctionType
```
This of course also works for in operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka.
Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH.
Windows: New options --mingw64 to force compilation with MinGW.

Optimization

Rich comparisons (==, <, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall.
Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython.
The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs.

Cleanups

Replaced “value friend” based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of or and and expressions.
Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using “variable usage” profiles for this anymore.
The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows.
Stop using os.system and subprocess.call(..., shell = True) as it is not really portable at all, use subprocess.call(..., shell = False) instead.
As usual lots of cleanups related to line length issues and PyLint.

Organizational

Added support for Gentoo Linux.
Added support for self compiled Python versions with and without debug enabled.
Added use of Nuitka fonts for headers in manuals.
Does not install in-line copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages.

Summary

While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later.

To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever.

So give this a roll, it’s worth it. And feel free to join the mailing list (since closed) or make a donation to support Nuitka.

Nuitka, Python research and Physics

2013-10-04T09:54:45+02:00

Once upon a time ago physics had multiple forces. Each with very distinct traits. These were the early days. Then as time went on people started to realize that weak nuclear force and electromagnetic force were the same sort. That was after discovering that electricity and magnetism were the same, and that is the pattern. It’s called symmetry, equivalence, and so on.

The Nuitka project is effectively a Python research that goes through the same steps. There are e.g. currently local variables, closure variables (shared from containing function), module variables, temp holder, temp keeper, and temp variables.

Often when I added one more this was driven by code generation needs or deficiencies at the time. These are of course way too many, and all are good at one thing only.

So the main task Nuitka is faced with now to e.g. generalize that “temp keeper” variables are temporary variables local to one statement only, surely to be released afterwards, and make that particular feature e.g. work for all variables across the board. Right now, code generation uses “cheats” where it puts a C++ block and local declarations for “temp keeper” variables.

It would be good, if it could be a try with del statement on the “temp keeper variable” in its finally part and yet, get the same code generated. Difficult, yes, but not impossible, and definitely what I am aiming at.

So that is what Nuitka will be dealing with in the next releases. Once done, maybe there will be a “gravity”, i.e. the one thing not yet harmonized, but for good reasons. Potentially limited due to lack of understanding, potentially because there is a good reason. And so not all we be unified or maybe e.g. module variables will be a bit more special than local variables, although modules are just functions with variables writable everywhere.

Good stuff is coming, hold on. Join the project, or support it financially if you wish.

Nuitka still is too much of a one man show. Should I be asked to name one single weakness of it - that would be it.

Nuitka Release 0.4.5

2013-08-20T17:50:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release incorporates very many bug fixes, most of which were already part of hot fixes, usability improvements, documentation improvements, new logo, simpler Python3 on Windows, warnings for recursion options, and so on. So it’s mostly a consolidation release.

Bug Fixes

When targeting Python 3.x, Nuitka was using “python” to run Scons to run it under Python 2.x, which is not good enough on systems, where that is already Python3. Improved to only do the guessing where necessary (i.e. when using the in-line copy of Scons) and then to prefer “python2”. Fixed in 0.4.4.1 already.
When using Nuitka created binaries inside a “virtualenv”, created programs would instantly crash. The attempt to load and patch inspect module was not making sure that site module was already imported, but inside the “virtualenv”, it cannot be found unless. Fixed in 0.4.4.1 already.
The option --recurse-directory to include plugin directories was broken. Fixed in 0.4.4.2 already.
Python3: Files with “BOM” marker causes the compiler to crash. Fixed in 0.4.4.2 already.
Windows: The generated code for try/return/finally was working with gcc (and therefore MinGW), but not with MSVC, causing crashes. Fixed in 0.4.4.2 already.
The option --recurse-all did not recurse to package __init__.py files in case from x.y import z syntax was used. Fixed in 0.4.4.2 already.
Python3 on macOS: Corrected link time error. Fixed in 0.4.4.2 already.
Python3.3 on Windows: Fixed crash with too many arguments to a kwonly argument using function. Fixed in 0.4.4.2 already.
Python3.3 on Windows: Using “yield from” resulted in a link time error. Fixed in 0.4.4.2 already.
Windows: Added back XML manifest, found a case where it is needed to prevent clashes with binary modules.
Windows: Generators only worked in the main Python threads. Some unusual threading modules therefore failed.
Using sys.prefix to find the Python installation instead of hard coded paths.

New Features

Windows: Python3 finds Python2 installation to run Scons automatically now.

Nuitka itself runs under Python3 just fine, but in order to build the generated C++ code into binaries, it uses Scons which still needs Python2.

Nuitka will now find the Python2 installation searching Windows registry instead of requiring hard coded paths.
Windows: Python2 and Python3 find their headers now even if Python is not installed to specific paths.

The installation path now is passed on to Scons which then uses it.
Better error checking for --recurse-to and --recurse-not-to arguments, tell the user not to use directory paths.
Added a warning for --recurse-to arguments that end up having no effect to the final result.

Cleanups

Import mechanism got cleaned up, stopped using “PyImport_ExtendInittab”. It does not handle packages, and the sys.meta_path based importer is now well proven.
Moved some of the constraint collection code mess into proper places. It still remains a mess.

Organizational

Added LICENSE.txt file with Apache License 2.0 text to make it more immediately obvious which license Nuitka is under.
Added section about Nuitka license to the User Manual.
Added Nuitka Logo to the distribution.
Use Nuitka Logo as the bitmap in the Windows installer.
Use Nuitka Logo in the documentation (User Manual and Developer Manual).
Enhanced documentation to number page numbers starting after table of contents, removed header/footer from cover pages.

Summary

This release is mostly the result of improvements made based on the surge of users after Europython 2013. Some people went to extents and reported their experience very detailed, and so I could aim at making e.g. their misconceptions about how recursion options work, more obvious through warnings and errors.

This release is not addressing performance improvements. The next release will be able to focus on that. I am taking my claim of full compatibility very serious, so any time it’s broken, it’s the highest priority to restore it.

My Europython 2013 report

2013-08-15T08:15:17+02:00

Back from Europython 2013 in Florence for a while now. I had a presentation about my Python compiler Nuitka. You can watch the Video on Youtube. I believe it’s absolutely worth your time. I was not doing a lot of “why” at all now, only “what”, and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good and at ease. The state presented is very good and progress - there was a lot, so that was great. The slides of the talk may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight then, and also much more excited, at least initially. Background: I was surprised then that Guido van Rossum was attending the talk, plus I was kind of not in the state of this year, the confidence that people should believe, that things will work after that, was not yet there. I knew it, but standing in front of a croud and saying, look I am attempting what you consider impossible, or even have failed at, is not easy:

There are also the slides of that talk which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven things. Let’s see what this year does to it. I would hope for real type inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out of the box experience, and how it just works. That’s right. I put a lot of focus on stability, correctness, and ease of use. In fact, since my return, I have mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays), it’s make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a lot of users, and not as many contributions or donations as I would hope for, but that’s kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important exception are scientific people, doing number crunching. And I made some interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which were unfortunately regressions of 0.4.4, and these got fixed in two hotfix releases. Currently 0.4.4.2 is released, and contains a huge amount of fixes, mostly stuff found by the surge new users.

But I deserved that, having said that I didn’t have to do hotfixes for a while now.

Florence

Such a lovely place, and so lovely people. It’s a tourist place, true, but it’s very open minded. The food is extremely good. And going from the conference hotel straight over to the river to meet up and have a few beers was very comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it’s moving on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large amount of non-satisfaction, was that the community is fed up with the leadership. Questioned about lack of Python3 adoption, a PSF guy in his presentation said they were 3 years into a 5 years plan, and got interrupted by laughter.

It may be about time, somebody actually gets up, removes the GIL and has a Python2.8 that is compatible. Obviously that’s much more useful. It could be a Python3.x with compatibility names of modules, print statement, and dict.iteritems added, probably also making some of the unicode mess more harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython top versions, to add things, that would allow Nuitka to work better with it, that could be a staging ground, where such changes are nurtured. On the other hand, Nuitka is too irrelevant yet to drive this, and it would detract me away from it.

Gains

So Nuitka got a lot more exposure. And one guy watching the talk even created a really nice logo for it. I will make it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And reinforcement that what I am doing is actually useful to some people already.

Donations

My travel to Florence was in part funded by donations. Thanks a lot to all of you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there via train.

Conclusion

The Europython 2012 and 2013 both are among the times of my life.

Changes in Python FAQ

2013-08-15T08:15:17+02:00

The Python FAQ changed a while ago:

Nuitka is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language.

Nuitka Release 0.4.4

2013-06-27T00:16:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release marks the point, where Nuitka for the first time supports all major current Python versions and all major features. It adds Python 3.3 support and it adds support for threading. And then there is a massive amount of fixes that improve compatibility even further.

Aside of that, there is major performance work. One side is the optimization of call performance (to CPython non-compiled functions) and to compiled functions, both. This gave a serious improvement to performance.

Then of course, we are making other, long term performance progress, as in “–experimental” mode, the SSA code starts to optimize unused code away. That code is not yet ready for prime time yet, but the trace structure will hold.

New Features

Python3.3 support.

The test suite of CPython3.3 passes now too. The yield from is now supported, but the improved argument parsing error messages are not implemented yet.
Tracing user provided constants, now Nuitka warns about too large constants produced during optimization.
Line numbers of expressions are now updates as evaluation progresses. This almost corrects.

Now only expression parts that cannot raise, do not update, which can still cause difference, but much less often, and then definitely useless.
Experimental support for threads.

Threading appears to work just fine in the most cases. It’s not as optimal as I wanted it to be, but that’s going to change with time.

Optimization

Previous corrections for ==, !=, and <=, caused a performance regression for these operations in case of handling identical objects.

For built-in objects of sane types (not float), these operations are now accelerated again. The overreaching acceleration of >= was still there (bug, see below) and has been adapted too.
Calling non-compiled Python functions from compiled functions was slower than in CPython. It is now just as fast.
Calling compiled functions without keyword arguments has been accelerated with a dedicated entry point that may call the implementation directly and avoid parameter parsing almost entirely.
Making calls to compiled and non-compiled Python functions no longer requires to build a temporary tuple and therefore is much faster.
Parameter parsing code is now more compact, and re-uses error raises, or creates them on the fly, instead of hard coding it. Saves binary size and should be more cache friendly.

Bug Fixes

Corrected false optimization of a >= a on C++ level.

When it’s not done during Nuitka compile time optimization, the rich comparison helper still contained short cuts for >=. This is now the same for all the comparison operators.
Calling a function with default values, not providing it, and not providing a value for a value without default, was not properly detecting the error, and instead causing a run time crash.
```
def f(a, b=2):
    pass


f(b=2)
```
This now properly raises the TypeError exception.
Constants created with + could become larger than the normally enforced limits. Not as likely to become huge, but still potentially an issue.
The vars built-in, when used on something without __dict__ attribute, was giving AttributeError instead of TypeError.
When re-cursing to modules at compile time, script directory and current directory were used last, while at run time, it was the other way around, which caused overloaded standard library modules to not be embedded.

Thanks for the patch to James Michael DuPont.
Super without arguments was not raising the correct RuntimeError exception in functions that cannot be methods, but UnboundLocalError instead.
```
def f():
    super()  # Error, cannot refer to first argument of f
```
Generators no longer use raise StopIteration for return statements, because that one is not properly handled in try/except clauses, where it’s not supposed to trigger, while try/finally should be honored.
Exception error message when throwing non-exceptions into generators was not compatible.
The use of return with value in generators is a SyntaxError before Python3.3, but that was not raised.
Variable names of the “__var” style need to be mangled. This was only done for classes, but not for functions contained in classes, there they are now mangled too.
Python3: Exceptions raised with causes were not properly chaining.
Python3: Specifying the file encoding corrupted line numbers, making them all of by one.

Cleanups

For containers (tuple, list, set, dict) defined on the source code level, Nuitka immediately created constant references from them.

For function calls, class creations, slice objects, this code is now re-used, and its dictionaries and tuples, may now become constants immediately, reducing noise in optimization steps.
The parameter parsing code got cleaned up. There were a lot of relics from previously explored paths. And error raises were part of the templates, but now are external code.
Global variable management moved to module objects and out of “Variables” module.
Make sure, nodes in the tree are not shared by accident.

This helped to find a case of duplicate use in the complex call helpers functions. Code generation will now notice this kind of duplication in debug mode.
The complex call helper functions were manually taking variable closure, which made these functions inconsistent to other functions, e.g. no variable version was allocated to assignments.

Removing the manual setting of variables allowed a huge reduction of code volume, as it became more generic code.
Converting user provided constants to create containers into constants immediately, to avoid noise from doing this in optimization.
The site module is now imported explicitly in the __main__ module, so it can be handled by the recursion code as well. This will help portable mode.
Many line length 80 changes, improved comments.

New Tests

The CPython3.3 test suite was added, and run with both Python3.2 and Python3.3, finding new bugs.
The doctest to code generation didn’t successfully handle all tests, most notably, “test_generators.py” was giving a SyntaxError and therefore not actually active. Correcting that improved the coverage of generator testing.

Organizational

The portable code is still delayed.

Support for Python3.3 was a higher priority, but the intention is to get it into shape for Europython still.

Added notes about it being disabled it in the User Manual documentation.

Summary

This release is in preparation for Europython 2013. Wanted to get this much out, as it changes the status slides quite a bit, and all of that was mostly done in my Cyprus holiday a while ago.

The portable code has not seen progress. The idea here is to get this into a development version later.

Nuitka Release 0.4.3

2013-05-18T10:17:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka.

New Features

Support for new compiler: Microsoft Visual C++.

You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows.
Support for NetBSD.

Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic “fibers” implementation.
Support for Python3 under Windows too.

Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to C:\Python26 or C:\Python27 for Nuitka to be able to find it.
Enhanced Python 3.3 compatibility.

The support the newest version of Python has been extended, improving compatibility for many minor corner cases.
Added warning when a user compiles a module and executes it immediately when that references __name__.

Because very likely the intention was to create an executable. And esp. if there is code like this:
```
if __name__ == "__main__":
    main()
```
In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command
```
nuitka --execute module
```
is behavioral more like
```
python -c "import module"
```
and that was a trap for new users.
All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don’t have to implement for specific architectures anymore.

Bug Fixes

Dictionary creation was not fully compatible.

As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn’t observe this before, and it’s likely the new dictionary implementation that finds it.

Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing.
Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was.
Using a __future__ import inside a function was giving an assertion, instead of the proper syntax error.
Python3: Do not set the attributes sys.exc_type, sys.exc_value, sys.exc_traceback.
Python3: Annotations of function worked only as long as their definition was not referring to local variables.

Optimization

Calls with no positional arguments are now using the faster call methods.

The generated C++ code was using the () constant at call site, when doing calls that use no positional arguments, which is of course useless.
For Windows now uses OS “Fibers” for Nuitka “Fibers”.

Using threads for fibers was causing only overhead and with this API, MSVC had less issues too.

Organizational

Accepting Donations via Paypal, please support funding travels, website, etc.
The User Manual has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated.
The Changelog has been improved for consistency, wordings, and styles.
Nuitka is now available on the social code platforms as well
- Bitbucket (since removed)
- GitHub
- Gitorious (since discontinued)
- Google Code (since discontinued)
Removed clean-up.sh, which is practically useless, as tests now clean up after themselves reasonably, and with git clean -dfx working better.
Removed “create-environment.sh” script, which was only setting the PATH variable, which is not necessary.
Added check-with-pylint --emacs option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint.
Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated.

Cleanups

Removed useless manifest and resource file creation under Windows.

Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it.
PyLint massive cleanups and annotations bringing down the number of warnings by a lot.
Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes.
Do not track needed tuple, list, and dict creation code variants in context, but e.g. in nuitka.codegen.TupleCodes module instead.
Introduced an “internal” module to host the complex call helper functions, instead of just adding it to any module that first uses it.

New Tests

Added basic tests for order evaluation, where there currently were None.
Added support for “2to3” execution under Windows too, so we can run tests for Python3 installations too.

Summary

The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now.

Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization.

Nuitka on GitHub, Bitbucket and Gitorious

2013-04-26T07:17:16+02:00

This is to let you know, that I have finally carved in and created accounts on all those “social” platforms, that deal with code. So far I had resisted their terms (you pay us when we get sued), but I have given up on that stance, it’s likely irrelevant, as Nuitka won’t get sued, and if it is, those are not the issues.

And I don’t want to feel too old school it seems. And who knows, maybe this helps Nuitka to get more contributions.

So these are the landing pages:

Bitbucket (since deleted)

I kind of like the looks of it the most. Also it’s very popular for Python people due to its Mercurial ties. I also liked that it imported my Nuitka repository easily.
GitHub

Is not being on GitHub allowed these days? Anyway, from ease of use, the interface looks more powerful.
Gitorious (since discontinued)

The interface and looks definitely are worst. But it’s Free Software, and an open platform, so it has to be in the list.

Let me know what you think. I am going to push ‘develop’ at pre-release and ‘master’ at release time, automatically from the release scripts into all these. Hopefully that way, they turn out to be time savers no wasters.

And yes, I couldn’t decide on one. Since I fancy Free Software, Gitorious probably would have won, but it would also miss the point of expanding the reach. And for as long as I don’t use their issue tracking or Wikis, it appears to be not necessary to make that decision.

Note

The preferred for use, and most up to date repository will remain the one from Downloads.

So fork away, like it, and stuff. Awaiting your pull requests…

Going to Europython 2013

2013-04-16T09:58:34+02:00

I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.

Things to talk about

Status

Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.

What I would like to get done until then:

Win64

One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.
Threading

And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.

Since it’s not totally a priority right now, one of the next releases will support it, likely before the conference.

But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.

Last years questions

Builtins

For instance, writing to built-ins, what will/did happen.

Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.

This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it’s not effective in others (len, str).

Good news is that we got contributed a “compiled built-ins” code, where we now will be able to see such writes. Now it’s only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.

The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.

Note

Currently Nuitka does nothing of this, but it will.

Debugger - pdb

The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.

Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.

Threading

I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don’t like them either, so my experience was non-existing.

But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.

C++ to Python gaps

I consider all of these more or less solved.

Well maybe except recently arose issues with MSVC for “function calls”. It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.

But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.

Performance

There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).

And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.

Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.

But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it’s the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.

Demos

The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.

And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.

If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.

Future Work

Generally the talk will be more directed at the future, although this is kind of a dark area now. That’s its nature I guess.

SSA

The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say “Python is too dynamic (to be statically compiled)” is that Python values may escape to unknown code that changes them very often.

I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.

Compiled Modules

And I believe with “compiled modules” potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.

I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn’t have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.

Compiled Locals

Who knows, we might even have a “compiled locals” as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.

But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don’t know if it might use it.

Whole Program Analysis

Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.

Perspective

Last time, I spent a lot of time on justification, “why a new project?”, “why not work with the others?”, what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.

Funding

And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It’s probably fair to allow people to make donations if they feel they want to further the project, but don’t know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikely though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.

Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these “times of my life”. Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.

Support for MSVC upcoming

2013-04-10T20:30:11+02:00

The good news are lately coming in high density, although I barely have the opportunity to work on Nuitka. Check the page What is Nuitka? for clarification of what it is now and what it wants to be.

It seems my recent call for help was at least partially effective. One the major points has been addressed. In the next release, Visual Studio 2008 and 2010 will both work for Nuitka, right now it’s only in git develop branch.

That is really nice, as it will make Nuitka on Windows a whole lot more accessible. Combine that with portable support and it’s up to solving problems with distributing on Windows it seems.

The other two things listed there have received no feedback yet. So if you want to help with those, feel free to do so. Nuitka needs you.

Nuitka needs you - a call for help

2013-04-10T07:51:13+02:00

Update 3

And the first point has been done too, Python3.3 is now fully supported, including yield from syntax.

Update 2

The third point, Windows 64 support has been done as well. Turns out, that I do in fact own only Win64 systems, and with MSVC support in place, only a few portability fixes were needed.

Help with the other point, “re-formulation of yield”, would still be more than welcome, and no signs of progress there yet. So stop thinking “you could”, enter telling people “you did” mode.

Update

The second point, Fibers implementation for Windows, has been done. Thanks for the help you people provide. The next release will contain it.

Hello everybody,

the Python compiler Nuitka has come an ever longer way, and currently I have quite a bunch of issues, that I believe could well need your help. These are all issues of some important and significance, yet self contained jobs that you may enjoy.

Note

You can check the page What is Nuitka? for clarification of what it is now and what it wants to be.

Python 3.3 - reformulation of yield from needed.

As you can see, covering all the CPython 2.6, 2.7, and 3.2 language features is already something. Also CPython 3.3 is now working on a basic level. Other projects are far, far away from that.

Many language constructs, such as the with statement are re-formulated into other constructs. This makes it possible to work with a simple core for optimization, and to reduce the complexity a lot. For the with statement case, it’s changed to try/finally and try/except statements, together with a few temporary variables. Check the Nuitka source of it:

ReformulationWithStatements.py

There is also descriptions of all these re-formulations in the Developer Manual so you can see how this is done.

Now check PEP 380 for the details of it, and your task would be to come up with a re-formulation of yield from statements to the semantically equivalent.

The benefit is, you are working on a totally self-contained level. The re-formulation only needs to use ast node information, and turn that into Nuitka nodes. The problem will be fully solved this way.

And once we have this, we can pronounce CPython3.3 as fully supported. So if you would like to see this happen, please join.
Windows - MSVC support needs low level code help

Update

This is now done.

The support for MSVC is already mostly there. If you execute the environment setting script, and use Nuitka under Windows, it will work. The only problem is that for the generator functions, Nuitka is using a concept called “Fiber”, which is basically just switching C stacks, so called co-routines.

It works great for Linux and UNIX. For Windows, Fibers are currently using threads, which wouldn’t be as much of a performance problem, because they are very lightweight, it is slightly wasteful only.

But, the bad thing, is with these threads switching and C++ exceptions used to represent Python exceptions, no combination of MSVC options seems to carry the day and keep the exception in the current thread. They are caught in the wrong thread, and make generators misbehave, crash.

This is the only thing that holds back proper MSVC support at this time, and it has been like this for a while. With MinGW, the issue does not exist, but MinGW is a stranger to many, and MSVC will be more efficient code generation, so we would like to have that.

So what it takes is somebody familiar with Windows and its ABI, esp. register usage to write code that swaps the registers around, so we can have multiple C stacks without threads for it too. This may involve messing around with exception jump back pointers as well.

It needs somebody who can handle this and does it for Nuitka, and then it would be well supported. It may involve writing some assembler code. The job is also very well contained, but also requires very specific knowledge, so please join me here. If you think you can do this, do it.
Windows 64 bit

I have no access to Win64. I bought many Windows versions, but not the Win64 variants so far. I understand that there are CPython builds for it, and probably a MinGW matching it, as well as a matching newer MSVC, that may work better for C++ exceptions than the one CPython 2.x is tied to.

Doing that, you would mostly only have to play around with SingleExe.scons, which abstracts the details of it all. One issue you will encounter is that Scons needs a Python2, and other small things. This is mostly only about porting Nuitka, and could be done by anybody with access to Win64 and willing to fiddle around a bit.

So these are 3 different ways you can help Nuitka. Creating the Python compiler is a huge effort. And if you help it, we will sooner arrive at its goals. Join the Nuitka mailing list (since closed) or contact me directly if you choose to help with any of these issues.

Wait, actually there is another way you can help. Please spread this news to people that you believe might be willing to help. And allow me to plug this, I am doing Nuitka in my spare time. If you feel, you cannot help, but still would like to support it, you can still make donations to allow me to travel to conferences and spend more time on it myself.

Anyway, it would be great if these 3 things got sorted out. Come and join me for this grand vision of the Python compiler.

Yours,

Kay

Support for portable (standalone) programs

2013-04-07T13:52:44+02:00

This post is about a feature often requested, but so far not available feature of Nuitka. Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

In forums, and in Google, people are looking at a Python compiler, also as a way of deployment. It should offer what py2exe does, allow installation independent of Python.

Well, for a long time it didn’t. But thanks to recent contributions, it’s upcoming for the next release, Nuitka 0.4.3, and it’s in the current pre-releases.

It works by adding --portable to the command line. So this should work for you:

nuitka-python --recurse-all --portable your-program.py

Right now, it will create a folder “_python” with DLLs, and “_python.zip” with standard library modules used along to the “your-program.exe”. Copy these to another machine, without a Python installation, and it will (should) work. Making that statement fully true may need refinements, as some DLL dependencies might not be defined yet.

Note

We may improve it in the future to meld everything into one executable for even easier deployment.

You are more than welcome to experiment with it. To do so, download Nuitka from the download page and give it a roll.

Note

Of course, Nuitka is not about replacing “py2exe” primarily, it’s only a side effect of what we do. Our major goal is of course to accelerate Python, but surely nobody minds achieving the two things at the same time.

And while the post is labeled “Windows”, this feature also works for Linux at least too. It’s just that the lack of Python installations on client systems is far more widespread on this platform.

To me, as this is from a contributor, it’s another sign of Nuitka gaining adoption for real usage. My personal “py2exe” experience is practically not existing, I have never used it. And I will only merge the improvements into the Nuitka project as provided by others. My focus for the time to come is of course the compile time and run time optimization.

NetBSD support upcoming

2013-04-06T12:58:37+02:00

My first real UNIX ever was a NetBSD. That was now about 22 years ago. I am still sentimental about it. I had installed it last about 8 years ago. And I still like it. Back in the days, it was the first UNIX to encounter for me, running on Amiga hardware, first of a friend, then on my own.

Recently, there had been support for Nuitka on FreeBSD added. A lot of people use it on the web, and some want to use Nuitka to improve their Python performance, so this is kind of relevant.

There were issues resolved, but in the end, something was with Clang on FreeBSD 8, that I could for the life of it, not resolve remotely. So I attempted to install it myself. Using “virt-install”, these things are a breeze now. I had already done it with CentOS6 before to test the RPM repositories of Nuitka. That “virt-install” is a wonderful thing by itself, making virtualisation somewhat useful. It’s only a pity, that I can’t just install other qemu support architectures. I would love to checkout Nuitka on PowerPC.

Note

If you could check out Nuitka on other Linux architectures than x86_64, x86, or arm, that would be great.

This is the report of getting NetBSD supported. It was a quite an interesting story that I would like to share with you.

Naivly I was assuming, that it would be just for fun, and that Nuitka will work right away. Little did I know.

On FreeBSD 9 the minimal install medium was chosen, and entered its ports collection, installed git, cloned Nuitka, and ran the tests, successfully right away. Now that is unfair, in the Nuitka there were tons of “Linuxism” already removed. In fact, it had to work, and on the newest FreeBSD (version 9.1) and then it did. Great!

Note

If you would like to add Nuitka to FreeBSD’s ports, please do so. It should be really easy now.

On NetBSD, things were unfortunately a little different. I also chose minimal system. After going through “pkg source” boot strap and git install, I cloned Nuitka, and then tried to start it. First off, it couldn’t locate “python” at all. I am using /usr/bin/env python already. But Python2 was on the system. I ended up creating the “python” link myself. What I should have done according to “#netbsd” is to install the software, and indeed, python2.7 setup.py install gives an installation of Nuitka that is executable.

Next up, you need to know about “Fibers” in Nuitka. These are used for C co-routines, used to implement Python generators. They have an interface, that is very close to makecontext/swapcontext routines in C.

For ARM and x86_64 Linux we have optimized code, that switches faster, but other platforms, including x86 Linux, use the generic implementation, also because it normally is very fast already.

Now you have to know that since 2001 the interface is deprecated and shall not be used. And next up, is that on NetBSD, makecontext gave a segfault only. So I ran to “#netbsd” and asked.

Now that was a very friendly experience. Of course, I had to give a rationale for using an obsolete interface. It’s not quite obvious, why threads wouldn’t be a better choice. And maybe they are, but they definitely have more overhead associated, and if they never run at the same time, why use them.

Ultimately it helped to point out, that for a user of 22 years, an interface that is only obsolete for 11 years, is not quite as horrifying as for others.

And they helped me through it. And it turns out, funny thing. For the context to setup, you are allocating a stack to use for the C routine, and you “get” the current context, then you make a new one. All the examples have a certain order of doing it. And my code did it the other way around. No system but NetBSD noticed.

On FreeBSD and Linux, it didn’t matter. But it seems, that the needed getcontext call was overwriting my stack pointer now with the current stack. And makecontext deeply hated that, a lot. It was preparing that stack to be used, while it was in usage. Doesn’t sound like a good task to give to it, right? My fault truly, because every example on every man page, on all systems, was doing it differently. But then they were also all using arrays from the local stack, so quite obviously that was not real code.

So that was fixed, and all good? No! Next thing was it crashed when free happened in Python on a compiled frame object, in a later part of a test that heavily uses generators. Turns out, malloc information was corrupted. I had to suspect the generic “Fiber” code, but that took me a while to figure out.

And how could my simple malloc and free do that, and make it happen. When I knew that a context would not longer be used (the generator has finished, the generator object deleted, etc), I would look at the context handle stack pointer and free it.

But that pointer changed. Something totally unexpected (by me obviously), but it also explains the earlier problem. For all systems, I had used so far, this pointer was not being changed, and remained the same. So I could free it from there. It worked fine, but not on NetBSD. And it wasn’t correct anywhere.

It seems NetBSD is doing something clever, since instead of saving the stack pointer register in a separate area, it saves it to that place originally specified. It’s quite obviously an improvement, in that you save the pointer.

It’s only bad, that now to make up for this savings, I have added the pointer in a separate field, which won’t be changed, so I can free it again. If one needs it again, and that’s not unlikely, you have to remember it elsewhere. So maybe that idea is not that clever. But it surely was wrong by me to assume that the provided value would not be touched.

So, these are 2 bugs it found. The wrong order of calls. And the usage of a pointer, that may have been changed. This can only help with other systems, or possibly architectures under Linux.

While this is all description of nasty problems, it’s also the report of the solution, and it was big fun. I would also like to compliment “#netbsd” for being very helpful and friendly with my porting of Nuitka. I highly enjoyed doing so. It was a lot of fun. I know that it’s probably on a very tiny amount of people that uses both NetBSD and Nuitka, but still.

If this Nuitka project were about market share, it wouldn’t exist. And I can work for market share on another day.

Nuitka Release 0.4.2

2013-03-29T14:32:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the performance diagrams got expanded.

New Features

Support for FreeBSD.

Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some “Linuxisms” in the build process.
New option for warning about compile time detected exception raises.

Nuitka can now warn about exceptions that will be raised at run time.
Basic Python3.3 support.

The test suite of CPython3.2 passes and fails in a compatible way. New feature yield from is not yet supported, and the improved argument parsing error messages are not implemented yet.

Bug Fixes

Nuitka already supported compilation of “main directories”, i.e. directories with a “__main__.py” file inside. The resulting binary name was “__main__.exe” though, but now it is “directory.exe”
```
# ls directory
__main__.py

# nuitka --exe directory
# ls
directory directory.exe
```
This makes this usage more obvious, and fixes an older issue for this feature.
Evaluation order of binary operators was not enforced.

Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems.
Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.
Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.
```
global a  # Not in a function, but on module level. Pointless but legal!
a = 1
```
Effectively these statements can be ignored.
Future imports are only legal when they are at the start of the file.

This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error.
Raising exceptions from context was leaking references.
```
raise ValueError() from None
```
Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments.
Importing the module that became __main__ through the module name, didn’t recurse to it.

This also gives a warning. PyBench does it, and then stumbles over the non-found “pybench” module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don’t share data.
Compiled method repr leaked references when printed.

When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point.
The super built-in leaked references to given object.

This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run.
The raise statement didn’t enforce type of third argument.

This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it’s None (which is also legal), no reference to None is leaked.
Python3 built-in exceptions were strings instead of exceptions.

A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn’t help their usage of course, fixed.
The -nan and nan both exist and make a difference.

A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka.
Wrong optimization of a == a, a != a, a <= a on C++ level.

While it’s not done during Nuitka optimization, the rich comparison helpers still contained short cuts for ==, !=, and <=.
The sys.executable for nuitka-python --python-version 3.2 was still python.

When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.
Keyword only functions with default values were losing references to defaults.
```
def f(*, a=X()):
    pass


f()
f()  # Can crash, X() should already be released.
```
This is now corrected. Of course, a Python3 only issue.
Pressing CTRL-C didn’t generate KeyboardInterrupt in compiled code.

Nuitka never executes “pending calls”. It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.
For with statements with return, break, or continue to leave their body, the __exit__ was not called.
```
with a:  # This called a.__enter__().
    return 2  # This didn't call a.__exit__(None, None, None).
```
This is of course quite huge, and unfortunately wasn’t covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing.
Starting with Python3, absolute imports are now the default.

This was already present for Python3.3, and it turns out that all of Python3 does it.

Optimization

Constants are now much less often created with pickle module, but created directly.

This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.
```
a = ("something_special",)
b = "something_special"

assert a[0] is b  # Now true
```
This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the “==” operator will be able to shortcut dramatically in cases of identical objects.

Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.
Continue statements might be optimized away.

A terminal continue in a loop, was not optimized away:
```
while 1:
    something
    continue  # Now optimized away
```
The trailing continue has no effect and can therefore be removed.
```
while 1:
    something
```
Loops with only break statements are optimized away.
```
while 1:
    break
```
A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate “if … : break” checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.

New Tests

Added tests for the found issues.
Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered.
Running the “CPython3.2” test suite with Python3.3 based Nuitka works and found a few minor issues.

Organizational

The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and “ownssh” for contributing an RPM spec file.

The page got improved with logos for the distributions.
Added “ownssh” as contributor.
Revamped the User Manual in terms of layout, structure, and content.

Summary

This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.

Nuitka RPMs (RHEL, CentOS, F17, F18, openSUSE)

2013-03-16T18:44:45+01:00

Thanks to a contribution, Nuitka now has a proper RPM spec file. And using openSUSE Build Service, we can now enjoy RPMs for all major distributions.

Check out the revamped download page as well.

Note

At the time of this writing, the “stable” Nuitka 0.4.1 link points to a 0.4.2 pre-release. That is only until 0.4.2 final will be released.

Thanks for making this possible, very much appreciated.

Nuitka Release 0.4.1

2013-03-05T22:13:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree.

Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes.

New Features

Python 3.3 experimental support
- Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface.
- Added support for __qualname__ to classes and functions.
- Small compatibility changes. Some exceptions changed, absolute imports are now default, etc.
- For comparison tests, the hash randomization is disabled.
Python 3.2 support has been expanded.

The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code.

Bug Fixes

Default values were not “is” identical.
```
def defaultKeepsIdentity(arg="str_value"):
    print arg is "str_value"


defaultKeepsIdentity()
```
This now prints “True” as it does with CPython. The solution is actually a general code optimization, see below.
Usage of unicode built-in with more than one argument could corrupt the encoding argument string.

An implementation error of the unicode was releasing references to arguments converted to default encoding, which could corrupt it.
Assigning Python3 function annotations could cause a segmentation fault.

Optimization

Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization.

# Now optimizing:
raise TypeError, 1 / 0
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
(1 / 0).something
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
function(a, 1 / 0).something
# into (minus normalization), notice the side effects of first checking
# function and a as names to be defined, these may be removed only if
# they can be demonstrated to have no effect.
function
a
raise ZeroDivisionError, "integer division or modulo by zero"

There is more examples, where the raise propagation is new, but you get the idea.

Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things.
```
# This was already optimized, as it's a compile time constant.
a if ("a",) else b
a if True else b

# These are now optimized, as their truth value is known.
a if (c,) else b
a if not (c,) else b
```
This is simply taking advantage of infrastructure that now exists. Each node kind can overload “getTruthValue” and benefit from it. Help would be welcome to review which ones can be added.
Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused.
Code generation for constants now shares element values used in tuples.

The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up.

Cleanups

Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection.
The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information.
The class names of nodes were stripped from the “CPython” prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it’s also not important enough to dominate the class name.
The re-formulations performed in tree building have moved out of the “Building” module, into names “ReformulationClasses” e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it’s much easier to find the content of interest now.
Added new re-formulation of print statements. The conversion to strings is now made explicit in the node tree.

New Tests

Added test to cover default value identity.

Organizational

The upload of Nuitka to PyPI has been repaired and now properly displays project information again.

Summary

The quicker release is mostly a consolidation effort, without much actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization.

PyStone Comparison Nuitka, Cython, and CPython

2013-03-04T08:57:10+01:00

As you all know, Nuitka (see “what is Nuitka?” ) has recently completed a milestone. Always short on time, I am not doing a whole lot of benchmarking yet, and focus on development. But here is an interesting submission from Dave Kierans (CTO of iPowow! Ltd):

➜  ~  python pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 10.2972
This machine benchmarks at 97113.5 pystones/second
➜  ~  cython --embed pystone.py;gcc pystone.c -I/usr/include/python2.6 -L /usr/lib/ -lpython2.6 -o ./pystone.cython;./pystone.cython 1000000
Pystone(1.1) time for 1000000 passes = 8.20789
This machine benchmarks at 121834 pystones/second
➜  ~  nuitka-python pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 4.06196
This machine benchmarks at 246187 pystones/second

This is nice result for Nuitka, even if we all know that pystone is not a really good benchmark at all, and that its results definitely do not translate to other software. It definitely makes me feel good. With the 0.4.x series kicked off, it will be exciting to see, where these numbers can indeed go, once Nuitka actually applies standard compiler techniques.

Nuitka Release 0.4.0

2013-02-25T17:39:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings massive progress on all fronts. The big highlight is of course: Full Python3.2 support. With this release, the test suite of CPython3.2 is considered passing when compiled with Nuitka.

Then lots of work on optimization and infrastructure. The major goal of this release was to get in shape for actual optimization. This is also why for the first time, it is tested that some things are indeed compile time optimized to spot regressions easier. And we are having performance diagrams, even if weak ones:

New Features

Python3.2 is now fully supported.
Fully correct metaclass = semantics now correctly supported. It had been working somewhat previously, but now all the corner cases are covered too.
- Keyword only parameters.
- Annotations of functions return value and their arguments.
- Exception causes, chaining, automatic deletion of exception handlers as values.
- Added support for starred assigns.
- Unicode variable names are also supported, although it’s of course ugly, to find a way to translate these to C++ ones.

Bug Fixes

Checking compiled code with instance(some_function, types.FunctionType) as “zope.interfaces” does, was causing compatibility problems. Now this kind of check passes for compiled functions too.
The frame of modules had an empty locals dictionary, which is not compatible to CPython which puts the globals dictionary there too.
For nested exceptions and interactions with generator objects, the exceptions in sys.exc_info() were not always fully compatible. They now are.
The range builtin was not raising exceptions if given arguments appeared to not have side effects, but were still illegal, e.g. range([], 1, -1) was optimized away if the value was not used.
Don’t crash on imported modules with syntax errors. Instead, the attempted recursion is simply not done.
Doing a del on __defaults and __module__ of compiled functions was crashing. This was noticed by a Python3 test for __kwdefaults__ that exposed this compiled functions weakness.
Wasn’t detecting duplicate arguments, if one of them was not a plain arguments. Star arguments could collide with normal ones.
The __doc__ of classes is now only set, where it was in fact specified. Otherwise it only polluted the name space of locals().
When return from the tried statements of a try/finally block, was overridden, by the final block, a reference was leaked. Example code:
```
try:
    return 1
finally:
    return 2
```
Raising exception instances with value, was leaking references, and not raising the TypeError error it is supposed to do.
When raising with multiple arguments, the evaluation order of them was not enforced, it now is. This fixes a reference leak when raising exceptions, where building the exception was raising an exception.

Optimization

Optimizing attribute access to compile time constants for the first time. The old registry had no actual user yet.
Optimizing subscript and slices for all compile time constants beyond constant values, made easy by using inheritance.
Built-in references now convert to strings directly, e.g. when used in a print statement. Needed for the testing approach “compiled file contains only prints with constant value”.
Optimizing calls to constant nodes directly into exceptions.
Optimizing built-in bool for arguments with known truth value. This would be creations of tuples, lists, and dictionaries.
Optimizing a is b and a is not b based on aliasing interface, which at this time effectively is limited to telling that a is a is true and a is not a is false, but this will expand.
Added support for optimizing hasattr, getattr, and setattr built-ins as well. The hasattr was needed for the class re-formulation of Python3 anyway.
Optimizing getattr with string argument and no default to simple attribute access.
Added support for optimizing isinstance built-in.
Was handling “BreakException” and “ContinueException” in all loops that used break or continue instead of only where necessary.
When catching “ReturnValueException”, was raising an exception where a normal return was sufficient. Raising them now only where needed, which also means, function need not catch them ever.

Cleanups

The handling of classes for Python2 and Python3 have been re-formulated in Python more completely.
- The calling of the determined “metaclass” is now in the node tree, so this call may possible to in-line in the future. This eliminated some static C++ code.
- Passing of values into dictionary creation function is no longer using hard coded special parameters, but temporary variables can now have closure references, making this normal and visible to the optimization.
- Class dictionary creation functions are therefore no longer as special as they used to be.
- There is no class creation node anymore, it’s merely a call to type or the metaclass detected.
Re-formulated complex calls through helper functions that process the star list and dict arguments and do merges, checks, etc.
- Moves much C++ code into the node tree visibility.
- Will allow optimization to eliminate checks and to compile time merge, once in-line functions and loop unrolling are supported.
Added “return None” to function bodies without a an aborting statement at the end, and removed the hard coded fallback from function templates. Makes it explicit in the node tree and available for optimization.
Merged C++ classes for frame exception keeper with frame guards.
- The exception is now saved in the compiled frame object, making it potentially more compatible to start with.
- Aligned module and function frame guard usage, now using the same class.
- There is now a clear difference in the frame guard classes. One is for generators and one is for functions, allowing to implement their different exception behavior there.
The optimization registries for calls, subscripts, slices, and attributes have been replaced with attaching them to nodes.
- The ensuing circular dependency has been resolved by more local imports for created nodes.
- The package “nuitka.transform.optimization.registries” is no more.
- New per node methods “computeNodeCall”, “computeNodeSubscript”, etc. dispatch the optimization process to the nodes directly.
Use the standard frame guard code generation for modules too.
- Added a variant “once”, that avoids caching of frames entirely.
The variable closure taking has been cleaned up.
- Stages are now properly numbered.
- Python3 only stage is not executed for Python2 anymore.
- Added comments explaining things a bit better.
- Now an early step done directly after building a tree.
The special code generation used for unpacking from iterators and catching “StopIteration” was cleaned up.
- Now uses template, Generator functions, and proper identifiers.
The return statements in generators are now re-formulated into raise StopIteration for generators, because that’s what they really are. Allowed to remove special handling of return nodes in generators.
The specialty of CPython2.6 yielding non-None values of lambda generators, was so far implemented in code generation. This was moved to tree building as a re-formulation, making it subject to normal optimization.
Mangling of attribute names in functions contained in classes, has been moved into the early tree building. So far it was done during code generation, making it invisible to the optimization stages.
Removed tags attribute from node classes. This was once intended to make up for non-inheritance of similar node kinds, but since we have function references, the structure got so clean, it’s no more needed.
Introduced new package nuitka.tree, where the building of node trees, and operations on them live, as well as recursion and variable closure.
Removed nuitka.transform and move its former children nuitka.optimization and nuitka.finalization one level up. The deeply nested structure turned out to have no advantage.
Checks for Python version was sometimes “> 300”, where of course “>= 300” is the only thing that makes sense.
Split out helper code for exception raising from the handling of exception objects.

New Tests

The complete CPython3.2 test suite was adapted (no __code__, no __closure__, etc.) and is now passing, but only without “–debug”, because otherwise some of the generated C++ triggers (harmless) warnings.
Added new test suite designed to prove that expressions that are known to be compile time constant are indeed so. This works using the XML output done with --dump-xml and then searching it to only have print statements with constant values.
Added new basic CPython3.2 test “Functions32” and “ParameterErrors32” to cover keyword only parameter handling.
Added tests to cover generator object and exception interactions.
Added tests to cover try/finally and return in one or both branches correctly handling the references.
Added tests to cover evaluation order of arguments when raising exceptions.

Organizational

Changed my email from GMX over to Gmail, the old one will still continue to work. Updated the copyright notices accordingly.
Uploaded Nuitka to PyPI as well.

Summary

This release marks a milestone. The support of Python3 is here. The re-formulation of complex calls, and the code generation improvements are quite huge. More re-formulation could be done for argument parsing, but generally this is now mostly complete.

The 0.3.x series had a lot releases. Many of which brought progress with re-formulations that aimed at making optimization easier or possible. Sometimes small things like making “return None” explicit. Sometimes bigger things, like making class creations normal functions, or getting rid of or and and. All of this was important ground work, to make sure, that optimization doesn’t deal with complex stuff.

So, the 0.4.x series begins with this. The focus from now on can be almost purely optimization. This release contains already some of it, with frames being optimized away, with the assignment keepers from the or and and re-formulation being optimized away. This will be about achieving goals from the “ctypes” plan as discussed in the Developer Manual.

Also the performance page will be expanded with more benchmarks and diagrams as I go forward. I have finally given up on “codespeed”, and do my own diagrams.

Python3 support by Nuitka is upcoming

2013-02-09T08:38:37+01:00

This is kind of semi-interesting news for you Python3 lovers. My “Python compiler Nuitka” has been supporting the language somewhat, but in the next release it’s going to be complete.

Support for annotations, unicode variable names, keyword only arguments, starred assignments, exception chaining plus causes, and full support for new meta classes all have been added. So Python3.2 is covered now too and passing the test suite on about the same level as with Python2.6 and Python2.7 already.

I readied this while working on optimization, which is also seeing some progress, that I will report on another day. Plus it’s “available now on PyPI” too. At the time of this writing, there is pre-releases there, but after next release, there will only stable releases published.

My quick take on the Python3 that I saw:

Keyword Only Arguments

Now seriously. I didn’t miss that. And I am going to hate the extra effort it causes to implement argument parsing. And when I looked into default handling, I was just shocked:

def f( some_arg = function1(), *, some_key_arg = function2() )

What would you expect the evaluation order to be for defaults? I raised a CPython “bug report” about it. And I am kind of shocked this could be wrong.

Annotations

def f( arg1 : c_types.c_int, arg2 : c_types.c_int ) -> c_types.c_long
   return arg1+arg2

That looks pretty promising. I had known about that already, and I guess, whatever “hints” idea, we come up with Nuitka, it should at least allow this notation. Unfortunately, evaluation order of annotations is not that great either. I would expected them to come first, but they come last. Coming last, they kind of come too late to accept do anything with defaults.

Unicode Variables

Now great. While it’s one step closer to “you can’t re-enter any identifier without copy&paste”, it is also a whole lot more consistent, if you come e.g. from German or French and need only a few extra letters. And there has to be at least one advantage to making everything unicode, right?

Exception Chaining

Gotta love that. Exception handlers are often subject to bit rot. They will themselves contain errors, when they are actually used, hiding the important thing that happened. No more. That’s great.

The exception causes on the other hand, might be useful, but I didn’t see them a lot yet. That’s probably because they are too new.

Starred Assigns

I was wondering already, why that didn’t work before.

a, *b = something()

Metaclass prepare

That’s great stuff. Wish somebody had thought about meta classes that way from the outset.

nonlocal

Yeah, big stuff. I love it. Also being able to write closure variables is great for consistency.

Conclusion

Nothing ground breaking. Nothing that makes me give up on Python2.7 yet, but a bunch of surprises. And for Nuitka, this also found bugs and corner cases, where things were not yet properly tested. The Python3 test suite, inspired the finding of some obscure compatibility problems for sure.

Overall, it’s possible to think of Python2 and Python3 as still the same language with only minor semantic differences. These can then be dealt with re-formulations into the simpler Python that Nuitka optimization deals with internally.

I for my part, am still missing the print statement. But well, now Python3 finds them when I accidentally check them in. So that’s kind of a feature for as long as I develop in Python2.6/2.7 language.

Now if one of you would help out with Python3.3, that would be great. It would be about the new dictionary implementation mostly. And maybe finding a proper re-formulation for “yield from” too.