Nuitka this week #2

New Series Rationale

As discussed last week in TWN #1 this is a new series that I am using to highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

Python 3.7

I made the first release with official 3.7 support, huge milestone in terms of catching up. Generic classes posed a few puzzles, and need more refinements for error handling, but good code works now.

The class creation got a bit more complex, yet again, which will make it even hard to know the exact base classes to be used. But eventually we will manage to overcome this and statically optimize that.

MSI 3.7 files for Nuitka

Building the MSI files for Nuitka ran into a 3.7.0 regression of CPython failing to build them, that I reported and seems to be valid bug of theirs.

So they will be missing for some longer time. Actually I wasn't so sure if they are all that useful, or working as expected for the runners, but with the -m nuitka mode of execution, that ought to be a non-issue. so it would be nice to keep them for those who use them for deployment internally.

Planned Mode

I have a change here. This is going to be a draft post until I publish it, so I might the link, or mention it on the list, but I do not think I will wait for feedback, where there is not going to be all that much.

So I am shooting this off the web site.

Goto Generators

This is an exciting field of work, that I have been busy with this week. I will briefly describe the issue at hand.

So generators in Python are more generally called coroutines in other places, and basically that is code shaking hands, executing resuming in one, handing back a piece of data back and forth.

In Python, the way of doing this is yield and more recently yield from as a convience way for of doing it in a loop in Python3. I still recall the days when that was a statement. Then communication was one way only. Actually when I was still privately doing Nuitka based on then Python 2.5 and was then puzzled for Python 2.6, when I learned in Nuitka about it becoming an expression.

The way this is implemented in Python, is that execution of a frame is simply suspended, and another frame stack bytecode is activated. This switching is of course very fast potentially, the state is already fully preserved on the stack of the virtual machine, which is owned by the frame. For Nuitka, when it still was C++, it wasn't going to be possible to interrupt execution without preserving the stack. So what I did was very similar, and I started to use makecontext/setcontext to implement what I call fibers.

Basically that is C level stack switching, but with a huge issue. Python does not grow stacks, but can need a lot of stack space below. Therefore 1MB or even 2MB per generator was allocated, to be able to make deep function calls if needed.

So using a lot of generators on 32 bits could easily hit a 2GB limit. And now with Python3.5 coroutines people use more and more of them, and hit memory issues.

So, goto generators, now that C is possible, are an entirely new solution. With it, Nuitka will use one stack only. Generator code will become re-entrant, store values between entries on the heap, and continue execution at goto destinations dispatched by a switch according to last exit of the generator.

So I am now making changes to cleanup the way variable declarations and accesses for the C variables are being made. More on that next week though. For now I am very exited about the many cleanups that stem from it. The code generation used to have many special bells and whistles, and they were generalized into one thing now, making for cleaner and easier to understand Nuitka code.

Python3 Enumerators

On interesting thing, is that an incompatiblity related to __new__ will go away now.

The automatic staticmethod that we had to hack into it, because the Python core will do it for uncompiled functions only, had to be done while declaring the class. So it was visible and causing issues with at least the Python enum module, which wants to call your __new__ manually. Because why would it not?!

But turns out, for Python3 the staticmethod is not needed anymore. So this is now only done for Python2, where it is needed, and things work smoothly with this kind of code now too. This is currently in my factory testing and will probably become part of a hotfix if it turns out good.

Hotfixes

Immediately after the release, some rarely run test, where I compiled all the code on my machine, found 2 older bugs, obscure ones arguably, that I made into a hotfix, also because the test runner was having a regression with 3.7, which prevented some package builds. So that was 0.5.32.1 release.

And then I received a bug report about await where a self test of Nuitka fails and reports an optimization error. Very nice, the new exceptions that automatically dump involved nodes as XML made it immediately clear from the report, what is going on, even without having to reproduce anything. I bundled a 3.7 improvement for error cases in class creation with it. So that was the 0.5.32.2 release.

Plans

Finishing goto generators is my top priority, but I am also going over minor issues with the 3.7 test suite, fixing test cases there, and as with e.g. the enum issue, even known issues this now finds.

Until next week.