Nuitka Blog - Posts tagged europython

Nuitka Progress in 2015

2016-01-29T08:08:54+01:00

For quite a bit, there have been no status posts, not for lack of news, but a lot has happened indeed. I just seem to post a lot more to the mailing list than I do here. Especially about unfinished stuff, which is essentially for a project like Nuitka everything that’s going on.

Like I previously said, I am shy to make public postings about unfinished stuff and that’s going to continue. But I am breaking it, to keep you up to date with where Nuitka has been going lately.

And with release focuses, I have been making some actual changes that I think are worth talking about.

SSA (Single State Assignment Form)

The SSA using release has been made last summer. Recent releases have lifted more and more restrictions on where and now it is applied and made sure the internal status is consistent and true. And that trend is going to continue even more.

For shared variables (closure variables and module variables), Nuitka is still too conservative to make optimization. Code does annotate value escapes, but it’s not yet trusting it. The next releases will focus on lifting that kind of restriction, and for quality of result, that will mean making a huge jump ahead once that works, so module variables used locally a lot will become even faster to use then and subject to static optimization too.

Function Inlining

When doing my talk to EuroPython 2015, I was demoing it that, and indeed, what a break through. The circumstances under which it is done are still far too limited though. Essentially that ability is there, but will not normally be noticeable yet due to other optimization, e.g. functions are most often module variables and not local to the using function.

More code generation improvements will be needed to be able to inline functions that might raise an exception. Also the “cost” of inlining a function is also very much an unsolved issue. It will become the focus again, once the SSA use as indicated above expands to module variables, as then inlining other things than local functions will be possible too.

So there is a lot of things to do for this to really make a difference to your programs. But it’s still great to have that part solved so far.

Scalability

Parameter Parsing

Recent releases have replaced some of the oldest code of Nuitka, the one that generated special argument parsing for each function individually, now replaced with generic code, that surprisingly is often even faster, although quick entry points were tough to beat.

That gives the C backend compiler a much easier time. Previously 3 C functions were created per Python level function, two of which could get really big with many arguments, and these are no more.

Variable Error Messages

Something similar was going on with variable error messages. Each had their exception value pre-computed and created at module load time. Most of these are of course unused. This has been replaced with code that generates it on the fly, resulting in a lot less constants code.

Code Objects

And another thing was to look after code objects, of which there often were two for each Python level function. The one used or the frame during run time and the one used in the function object, differered often, sometimes by small things like flags or local variable names.

That of course was just the result of not passing that along, but created cached objects with hopefully the same options, but that not being true.

Resolving that, and sharing the code object used for creation and then the frame is gives less complex C code too.

Optimization

The scalability of Nuitka also depends much on generated code size. With the optimization become more clever, less code is generated, and that trend will continue as more structural optimization are applied.

Every time e.g. an exception is identified to not happen, this removes the corresponding error exits from the C code, which then makes it easier for the C compiler. Also more specialized code as we now have or dictionaries, is often less complex to it.

Compatibility

Important things have happened here. Full compatibility mode is planned to not be the default anymore in upcoming releases, but that will only mean to not be stupid compatible, but to e.g. have more complete error messages than CPython, more correct line numbers, or for version differences, the best Python version behaviour.

The stable release has full support for Python 3.5, including the new async and await functions. So recent releases can pronounce it as fully supported which was quite a feat.

I am not sure, if you can fully appreciate the catch up game needed to play here. CPython clearly implements a lot of features, that I have to emulate too. That’s going to repeat for every major release.

The good news is that the function type of Nuitka is now specialized to the generators and classes, and that was a massive cleanup of its core that was due anyway. The generators have no more their own function creation stuff and that has been helpful with a lot of other stuff.

Another focus driven from Python3, is to get ahead with type shape tracing and type inference of dictionary, and value tracing. To fully support Python3 classes, we need to work on something that is a dictionary a-like, and that will only ever be efficient if we have that. Good news is that the next release is making progress there too.

Performance

Graphs and Benchmarks

I also presented this weak point to EuroPython 2015 and my plan on how to resolve it. Unfortunately, nothing really happened here. My plan is still to use what the PyPy people have developed as vmprof.

So that is not progressing, and I could need help with that definitely. Get in contact if you think you can.

Standalone

The standalone mode of Nuitka was pretty good, and continued to improve further, but I don’t care much.

Other Stuff

EuroPython 2015

This was a blast. Meeting people who knew Nuitka but not me was a regular occurrence. And many people well appreciate my work. It felt much different than the years before.

I was able to present Nuitka’s function in-lining indeed there, and this high goal that I set myself, quite impressed people.

Also I made many new contacts, largely with the scientific community. I hope to find work with data scientists in the coming years. More amd more it looks like my day job should be closer to Nuitka and my expertise in Python.

Funding

Nuitka receives the occasional donation and those make me very happy. As there is no support from organization like the PSF, I am all on my own there.

This year I want to travel to Europython 2016. It would be sweet if aside of my free time it wouldn’t also cost me money. So please consider donating some more, as these kind of events are really helpul to Nuitka.

Collaborators

Nuitka is making more and more break through progress. And you can be a part of it. Now.

You can join and should do so now, just follow this link or become part of the mailing list (since closed) and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Videos

There is a Youtube channel of mine with all the videos of Nuitka so far and I have been preparing myself with proper equipment to make Videos of Nuitka, but so far nothing has come out of that.

I do however really want to change that. Let’s see if it happens.

Twitter

I have started to use my Twitter account on occasions. You are welcome to follow me there. I will highlight interesting stuff there.

Future

So, there is multiple things going on:

Type Inference

With SSA in place, Nuitka starts to recognize types, and treat things that work something assigned from {} or dict built-in with special nodes and code.

That’s going to be a lot of work. For float and list there are very important use cases, where the code can be much better. But dict is the hardest case, and to get the structure of shape tracing right, we are going there first.
Shape Analyisis

The plan for types, is not to use them, but the more general shapes, things that will be more prevalent than actual type information in a program. In fact the precise knowledge will be rare, but more often, we will just have a set of operations performed on a variable, and be able to guess from there.

Shape analysis will begin though with concrete types like dict. The reason is that some re-formulations like Python3 classes should not use locals, but dictionary accesses throughout for full compatibility. Tracing that correctly to be effectively the same code quality will allow to make that change.
Plug-ins

Something I wish I could have shown at EuroPython was plug-ins to Nuitka. It has become more complete, and some demo plug-ins for say Qt plugins or multiprocessing, are starting to work, but it’s not progressing recently. The API will need work and of course documentation. Hope is for this to expand Nuitka’s reach and appeal to get more contributors.

It would be sweet, if there were any takers, aiming to complete these things.
Nested frames

One result of in-lining will be nested frames still present for exceptions to be properly annotated, or locals giving different sets of locals and so on.

Some cleanup of these will be needed for code generation and SSA to be able to attach variables to some sort of container, and for a function to be able to reference different sets of these.

Let me know, if you are willing to help. I really need that help to make things happen faster. Nuitka will become more and more important only. And with your help, things will be there sooner.

Release Focus

One thing I have started recently, is to make changes to Nuitka focused to just one goal, and to only deal with the rare bug in other fields, but not much else at all. So instead of across the board improvements in just about everything, I have e.g. in the last release added type inference for dictionaries and special nodes and their code generation for dictionary operations.

This progresses Nuitka in one field. And the next release then e.g. will only focus on making the performance comparison tool, and not continue much in other fields.

That way, more “flow” is possible and more visible progress too. As an example of this, these are the focuses of last releases.

Full Python 3.5 on a clean base with generators redone so that coroutines fit in nicely.
Scalability of C compilation with argument parsing redone
Next release soon: Shape analysis of subscript usages and optimization to exact dictionaries
Next release thereafter: Comparison benchmarking (vmprof, resolving C level function identifiers easier)

Other focuses will also happen, but that’s too far ahead. Mostly like some usability improvements will be the focus of a release some day. Focus is for things that are too complex to attack as a side project, and therefore never happen although surely possible.

Digging into Python3.5 coroutines and their semantics was hard enough, and the structural changes needed to integrate them properly with not too much special casing, but rather removing existing special cases (generator functions) was just too much work to ever happen while also doing other stuff.

Summary

So I am very excited about Nuitka. It feels like the puzzle is coming together finally, with type inference becoming a real thing. And should dictionaries be sorted out, the real important types, say float for scientific use cases, or int, list for others, will be easy to make.

With this, and then harder import association (knowing what other modules are), and module level SSA tracing that can be trusted, we can finally expect Nuitka to be generally fast and deserve to be called a compiler.

That will take a while, but it’s likely to happen in 2016. Let’s see if I will get the funding to go to EuroPython 2016, that would be great.

Not going to FOSDEM 2014

2014-01-18T20:48:53+01:00

So I submitted a talk for FOSDEM “Python has a compiler now”. And it didn’t get accepted. Quite a surprise to me, but likely mostly logical. These are the accepted talks (link since became unavailable) and well, it surely didn’t fit in, right.

My intent was to introduce Nuitka to the larger community. Not going to happen now, but maybe next year.

Clearly triggered by the rejection, I am questioning Europython 2014 in Berlin as a useful destination too. To me it seems, that code writing is the best way to create the community at this time anyway.

My Europython 2013 report

2013-08-15T08:15:17+02:00

Back from Europython 2013 in Florence for a while now. I had a presentation about my Python compiler Nuitka. You can watch the Video on Youtube. I believe it’s absolutely worth your time. I was not doing a lot of “why” at all now, only “what”, and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good and at ease. The state presented is very good and progress - there was a lot, so that was great. The slides of the talk may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight then, and also much more excited, at least initially. Background: I was surprised then that Guido van Rossum was attending the talk, plus I was kind of not in the state of this year, the confidence that people should believe, that things will work after that, was not yet there. I knew it, but standing in front of a croud and saying, look I am attempting what you consider impossible, or even have failed at, is not easy:

There are also the slides of that talk which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven things. Let’s see what this year does to it. I would hope for real type inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out of the box experience, and how it just works. That’s right. I put a lot of focus on stability, correctness, and ease of use. In fact, since my return, I have mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays), it’s make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a lot of users, and not as many contributions or donations as I would hope for, but that’s kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important exception are scientific people, doing number crunching. And I made some interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which were unfortunately regressions of 0.4.4, and these got fixed in two hotfix releases. Currently 0.4.4.2 is released, and contains a huge amount of fixes, mostly stuff found by the surge new users.

But I deserved that, having said that I didn’t have to do hotfixes for a while now.

Florence

Such a lovely place, and so lovely people. It’s a tourist place, true, but it’s very open minded. The food is extremely good. And going from the conference hotel straight over to the river to meet up and have a few beers was very comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it’s moving on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large amount of non-satisfaction, was that the community is fed up with the leadership. Questioned about lack of Python3 adoption, a PSF guy in his presentation said they were 3 years into a 5 years plan, and got interrupted by laughter.

It may be about time, somebody actually gets up, removes the GIL and has a Python2.8 that is compatible. Obviously that’s much more useful. It could be a Python3.x with compatibility names of modules, print statement, and dict.iteritems added, probably also making some of the unicode mess more harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython top versions, to add things, that would allow Nuitka to work better with it, that could be a staging ground, where such changes are nurtured. On the other hand, Nuitka is too irrelevant yet to drive this, and it would detract me away from it.

Gains

So Nuitka got a lot more exposure. And one guy watching the talk even created a really nice logo for it. I will make it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And reinforcement that what I am doing is actually useful to some people already.

Donations

My travel to Florence was in part funded by donations. Thanks a lot to all of you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there via train.

Conclusion

The Europython 2012 and 2013 both are among the times of my life.

Going to Europython 2013

2013-04-16T09:58:34+02:00

I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.

Things to talk about

Status

Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.

What I would like to get done until then:

Win64

One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.
Threading

And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.

Since it’s not totally a priority right now, one of the next releases will support it, likely before the conference.

But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.

Last years questions

Builtins

For instance, writing to built-ins, what will/did happen.

Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.

This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it’s not effective in others (len, str).

Good news is that we got contributed a “compiled built-ins” code, where we now will be able to see such writes. Now it’s only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.

The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.

Note

Currently Nuitka does nothing of this, but it will.

Debugger - pdb

The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.

Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.

Threading

I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don’t like them either, so my experience was non-existing.

But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.

C++ to Python gaps

I consider all of these more or less solved.

Well maybe except recently arose issues with MSVC for “function calls”. It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.

But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.

Performance

There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).

And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.

Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.

But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it’s the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.

Demos

The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.

And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.

If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.

Future Work

Generally the talk will be more directed at the future, although this is kind of a dark area now. That’s its nature I guess.

SSA

The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say “Python is too dynamic (to be statically compiled)” is that Python values may escape to unknown code that changes them very often.

I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.

Compiled Modules

And I believe with “compiled modules” potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.

I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn’t have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.

Compiled Locals

Who knows, we might even have a “compiled locals” as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.

But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don’t know if it might use it.

Whole Program Analysis

Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.

Perspective

Last time, I spent a lot of time on justification, “why a new project?”, “why not work with the others?”, what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.

Funding

And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It’s probably fair to allow people to make donations if they feel they want to further the project, but don’t know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikely though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.

Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these “times of my life”. Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.