12 September 2018

Nuitka this week #6

Holiday

In my 2 weeks holiday, I indeed focused on a really big thing, and got more done that I had hoped for. For C types, nuitka_bool, which is a tri-state boolean with true, false and unassigned, can be used for some variables, and executes some operations without going through objects anymore.

bool

Condition codes are no longer special. They all need a boolean value from the expression used as a condition, and there was a special paths for some popular expressions for conditions, but of course not all. That is now a universal thing, conditional statement/expressions will now simply ask to provide a temp variable of value nuitka_bool and then code generation handles it.

For where it is used, code gets a lot lighter, and of course faster, although I didn’t measure it yet. Going to Py_True/Py_False and comparing with it, wasn’t that optimal, and it’s nice this is now so much cleaner as a side effect of that C bool work.

This seems to be so good, that actually it’s the default for this to be used in 0.6.0, and that itself is a major break through. Not so much for actual performance, but for structure. Other C types are going to follow soon and will give massive performance gains.

void

And what was really good, is that not only did I get bool to work almost perfectly, I also started work on the void C target type and finished that after my return from holiday last weekend, which lead to new optimization that I am putting in the 0.5.33 release that is coming soon, even before the void code generation is out.

The void C type cannot read values back, and unused values should not be used, so this gives errors for cases where that becomes obvious.

a or b

Consider this expression. The or expression, that one is going to producing a value, which is then released, but not used otherwise. New optimzation creates a conditional statement out of it, which takes a as the condition and if not true, then evaluates b but ignores it.

if not a:
    b

The void evaluation of b can then do further optimization for it.

Void code generation can therefore highlight missed opportunities for this kid of optimization, and found a couple of these. That is why I was going for it, and I feel it pays off. Code generation checking optimization here, is a really nice synergy between the two.

Plus I got all the tests to work with it, and solved the missing optimizations it found very easily. And instead of allocating an object now, not assigning is often creating more obvious code. And that too allowed me to find a couple of bugs by C compiler warnings.

Obviously I will want to run a compile all the world test before making it the default, which is why this will probably become part of 0.6.1 to be the default.

module_var

Previously variable codes were making a hard distinction for module variables and make them use their own helper codes. Now this is encapsulated in a normal C type class like nuitka_bool, or the one for PyObject * variables, and integrates smoothly, and even got better. A sign things are going smooth.

Goto Generators

Still not released. I delayed it after my holiday, and due to the heap generator change, after stabilizing the C types work, I want to first finish a tests/library/compile_python_module.py resume run, which will for a Anaconda3 compile all the code found in there.

Right now it’s still doing that, and even found a few bugs. The heap storage can still cause issues, as can changes to cloning nodes, which happens for try nodes and their finally blocks.

This should finish these days. I looked at performance numbers and found that develop is indeed only faster, and factory due to even more optimization will be yet faster, and often noteworthy.

Benchmarks

The Speedcenter of Nuitka is what I use right now, but it’s only showing the state of 3 branches and compared to CPython, not as much historical information. Also the organization of tests is poor. At least there is tags for what improved.

After release of Nuitka 0.6.0 I will show more numbers, and I will start to focus on making it easier to understand. Therefore no link right now, google if you are so keen. ;-)

Twitter

During the holiday sprint, and even after, I am going to Tweet a lot about what is going on for Nuitka. So follow me on twitter if you like, I will post important stuff as it happens there:

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Poll on Executable Names

So I put e.g. poll up on Twitter, which is now over. But it made me implement a new scheme, due to popular consensus

Hotfixes

Even more hotfixes. I even did 2 during my holiday, however packages built only later.

Threaded imports on 3.4 or higher of modules were not using the locking they should use. Multiprocessing on Windows with Python3 had even more problems, and the --include-package and --include-module were present, but not working.

That last one was actually very strange. I had added a new option group for them, but not added it to the parser. Result: Option works. Just does not show up in help output. Really?

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very like one you can help with.

Nuitka definitely needs more people to work on it.

Plans

Working down the release backlog. Things should be out. I am already working on what should become 0.6.1, but it’s not yet 0.5.33 released. Not a big deal, but 0.6.0 has 2 really important fixes for performance regressions that have happened in the past. One is for loops, making that faster is probably like the most important one. The other for constant indexing, probably also very important. Very much measurable in pystone at least.

In the mean time, I am preparing to get int working as a target C type, so e.g. comparisons of such values could be done in pure C, or relatively pure C.

Also, I noticed that e.g. in-place operations can be way more optimized and did stuff for 0.6.1 already in this domain. That is unrelated to C type work, but kind of follows a similar route maybe. How to compare mixed types we know of, or one type only. That kind of things needs ideas and experiments.

Having int supported should help getting some functions to C speeds, or at least much closer to it. That will make noticeable effects in many of the benchmarks. More C types will then follow one by one.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka