Nuitka Blog - Posted in 2012

Static Compilation - That is the point

2012-12-02T12:23:32+01:00

In a recent post, Stefan Behnel questioned the point of static compilation and suggests that in order to be useful, a static compiler needs to add something on top.

This is going to be a rebuttal.

Compatibility, I mean it

First of all, let me start out, by saying that Nuitka is intended to be the fully optimizing compiler of Python. The optimizing part is not yet true. Right now, it’s probably a correct compiler of Python. Correct in the sense that it’s compatible to CPython as far as possible.

As examples of what I mean with compatibility:

Nuitka will hold references to local variables of frames, when an exception is raised, and will release them only once the next exception is raised.
Nuitka will give the same error messages for all kinds of errors. For example, the parameter parsing of functions will be the same.
Nuitka provides all language constructs no matter how absurd or unused they are.

Compatibility != Slower

While generally Nuitka will have a hard time to be faster and compatible to CPython, I don’t have much concern about that. Using guards between optimistic, and less optimistic variants of code, there is no doubt in my head, that for programs, lots of code will only need very minimal type annotation and still receive respectable speedups.

Of course, at this point, this is only speculation. But I somehow gather that the sentiment is that incompatible and fast need to go along. I totally dispute that.

Language Extensions

Now, the “in addition” stuff, that Stefan is talking about. I don’t see the point at all. It is quite obvious that everything you can say with cdef syntax, could also be said with a more Pythonic syntax. And if it were missing, it could be added. And where it’s a semantic change, it should be frowned upon.

For the Nuitka project, I never considered an own parser of Python. No matter how easy it would be, to roll your own, and I understand that Cython did that, it’s always going to be wrong and it’s work. Work that has no point. The CPython implementation exhibits and maintains the module ast that works just fine.

For Python, if this were so really useful, such language extensions should be added to Python itself. If there were missing meaningful things, I contend they would best be added there, not in a fork of it. Note how ctypes and cffi have been added. When I created bindings for Ada code (C exports) to Python, it was so easy to do that in pure Python with ctypes. I so much enjoyed that.

So, slow bindings are in my view really easy to create with plain Python now. Somebody ought to make a “.h” to ctypes/cffi declarations converter, once they are really faster to use (possibly due to Nuitka). For Nuitka it should be possible to accelerate these into direct calls and accesses. At which point, mixing generated C code and C include statements, will just be what it is, a source of bugs that Nuitka won’t have.

Note

Further down, I will give examples of why I think that cdef is inferior to plain Python, even from a practical point of view.

Lack of Interpreter is bad

Static compilation vs. interpretation as a discussion has little merits to me. I find it totally obvious that you don’t need static compilation, but 2 other things:

You may need interpretation.
And may need speed.

To me static code analysis and compilation are means to achieve that speed, but not intended to remove interpretation, e.g. plugins need to still work, no matter how deep the go.

For Cython syntax there is no interpreter, is there? That makes it loose an important point. So it has to have another reason for using it, and that would be speed and probably convenience. Now suppose Nuitka takes over with these benefits, what would it be left with? Right. Nothing. At all. Well, of course legacy users.

The orinal sin fall of PyRex - that is now Cython - is nothing that Nuitka should repeat.

No Lock-in

The Cython language is so underspecified, I doubt anybody could make a compatible implementation. Should you choose to use it, you will become locked in. That means, if Cython breaks or won’t work to begin with, you are stuck.

That situation I totally despise. It seems an unnecessary risk to take. Not only, if your program does not work, you can’t just try another compiler. You also will never really know, if it’s either your fault or Cython’s fault until you do know, whose fault it is. Find yourself playing with removing, adding, or re-phrasing cdef statements, until things work.

Common. I would rather use PyPy or anything else, that can be checked with CPython. Should I ever encounter a bug with it, I can try CPython, and narrow down with it. Or I can try Jython, IronPython, or low and behold, Nuitka.

I think, this totally makes it obvious, that static compilation of a non-Python language has no point to people with Python code.

What I will always admit, is that Cython is (currently) the best way to create fast bindings, because Nuitka is not ready yet. But from my point of view, Cython has no point long term if a viable replacement that is Pythonic exists.

Python alone is a point

So, if you leave out static compilation vs. interpretation and JIT compilation, what would be the difference between PyPy and Nuitka? Well, obviously PyPy people are a lot cooler and cleverer. Their design is really inspiring and impressive. My design and whole approach to Nuitka is totally boring in comparison.

But from a practical standpoint, is there any difference? What is the difference between Jython and PyPy? The target VM it is. PyPy’s or Java’s. And performance it is, of course.

So, with Python implementations all being similar, and just differing in targets, and performances, do they all have no point? I believe taken to the logical conclusion, that is what Stefan suggests. I of course think that PyPy, Nuitka, and Jython have have as much of a point, as CPython does.

Type Annotations done right

And just for fun. This is making up a use cases of type annotations:

plong = long if python_version < 3 else int


@hints.signature(plong, plong)
def some_function(a):
    return a ** 2

Notice how plong depends on an expression, that may become known during compile time or not. Should that turn out to be not possible, Nuitka can always generate code for both branches and branch when called.

Or more complex and useful like this:

def guess_signature(func):
    types = [None]

    emit = types.append
    for arg in inspect.getargnames(func):
        if arg == "l":
            emit(list)
        elif arg == "f":
            emit(float)
        elif arg == "i":
            emit(int)
        else:
            hints.warning("Unknown type %s" % arg)
            emit(None)

    return hints.signature(*types)


def many_hints(func):
    # Won't raise exception.
    hints.doesnot_raise(func)

    # Signature to be inferred by conventions
    guess_signature(func)(func)

    # No side effects
    hints.pure(func)


@many_hints
def some_func1(f):
    return f + 2.0


@many_hints
def some_func2(i):
    return i + 2


@many_hints
def some_func3(l):
    return i + [2]

This is just a rough sketch, but hopefully you get the idea. Do this with Cython, can you?

The hints can be put into decorators, which may be discovered as inlinable, which then see more inlines. For this to work best, the loop over the compile time constant code object, needs to be unrolled, but that appears quite possible.

The signatures can therefore be done fully automatic. One could use prefix notation to indicate types.

Another way would put fixed types for certain variable names. In Nuitka code, “node”, “code”, “context”, etc. have always the same types. I suspect many programs are the same, and it would be sweet, if you could plug something in and check such types throughout all of the package.

And then, what do you do then? Well, you can inspect these hints at run time as well, they work with CPython as well (though they won’t make things faster, only will that find errors in your program), they will even work with PyPy, or at least not harm it. It will nicely JIT them away I suppose.

Your IDE will like the code. syntax highlighting, auto indent will work. With every Python IDE. PyLint will find the bugs I made in that code up there. And Nuitka will compile it and benefit from the hints.

My point here really is, that cdef is not flexible, not standard, not portable. It should die. It totally is anti-Pythonic to me.

Elsewhere

In Java land, people compile to machine code as well. They probably also - like stupid me - didn’t understand that static compilation would have no point. Why do they do it? Why am I using compiled binaries done with their compiler then?

And why didn’t they take the chance to introduce ubercool cdef a-likes while doing it? They probably just didn’t know better, did they?

No seriously. A compiler is just a compiler. It takes a source code in a language and turns it into a package to execute. That may be a required or an optional step. I prefer optional for development turn around. It should try and make code execute as fast as it can. But it should not change the language. With Cython I have to compile. With Nuitka I could.

In fact, I would be hard pressed to find another example of a compiler that extends the interpreted language compiled, just so there is a point in having it.

Conclusion

Nuitka has a point. On top of that I enjoy doing it. It’s great to have the time to do this thing in the correct way.

So far, things worked out pretty well. My earlier experimentations with type inference had shown some promise. The “value friends” thing, and the whole plan, appears relatively sound, but likely is in need of an update. I will work on it in december. Up to now, and even right now I worked on re-formulations, that should have made it possible to get more release ready effects from this.

When I say correct way, I mean this. When I noticed that type inference was harder than it should be, I could take the time and re-architecture things so that it will be simpler. To me that is fun. This being my spare time allows me to do things this efficiently. That’s not an excuse, it’s a fact that explains my approach. It doesn’t mean it makes less sense, not at all.

As for language compatibility, there is more progress with Python3. I am currently changing the class re-formulations for Python2 and Python3 (they need totally different ones due to metaclass changes) and then “test_desc.py” should pass with it too, which will be a huge achievement in that domain. I will do a post on that later.

Then infrastructure, should complete the valgrind based benchmark automatism. Numbers will become more important from now on. It starts to make sense to observe them. This is not entirely as fun. But with improving numbers, it will be good to show off.

And of course, I am going to document some more. The testing strategy of Nuitka is worth a look, because it’s totally different from everything else people normally do.

Anyway. I am not a big fan of controversy. I respect Cython for all it achieved. I do want to go where it fails to achieve. I should not have to justify that, it’s actually quite obvious, isn’t it?

Yours,

Kay

Nuitka Release 0.3.25

2012-11-11T16:29:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings about changes on all fronts, bug fixes, new features. Also very importantly Nuitka no longer uses C++11 for its code, but mere C++03. There is new re-formulation work, and re-factoring of functions.

But the most important part is this: Mercurial unit tests are working. Nearly. With the usual disclaimer of me being wrong, all remaining errors are errors of the test, or minor things. Hope is that these unit tests can be added as release tests to Nuitka. And once that is done, the next big Python application can come.

Bug Fixes

Local variables were released when an exception was raised that escaped the local function. They should only be released, after another exception was raised somewhere.
Identifiers of nested tuples and lists could collide.
```
a = ((1, 2), 3)
b = ((1,), 2, 3)
```
Both tuples had the same name previously, not the end of the tuple is marked too. Fixed in 0.3.24.1 already.
The __name__ when used read-only in modules in packages was optimized to a string value that didn’t contain the package name.
Exceptions set when entering compiled functions were unset at function exit.

New Features

Compiled frames support. Before, Nuitka was creating frames with the standard CPython C/API functions, and tried its best to cache them. This involved some difficulties, but as it turns out, it is actually possible to instead provide a compatible type of our own, that we have full control over.

This will become the base of enhanced compatibility. Keeping references to local variables attached to exception tracebacks is something we may be able to solve now.
Enhanced Python3 support, added support for nonlocal declarations and many small corrections for it.
Writable __defaults__ attribute for compiled functions, actually changes the default value used at call time. Not supported is changing the amount of default parameters.

Cleanups

Keep the functions along with the module and added “FunctionRef” node kind to point to them.
Reformulated or and and operators with the conditional expression construct which makes the “short-circuit” branch.
Access self in methods from the compiled function object instead of pointer to context object, making it possible to access the function object.
Removed “OverflowCheck” module and its usage, avoids one useless scan per function to determine the need for “locals dictionary”.
Make “compileTree” of “MainControl” module to only do what the name says and moved the rest out, making the top level control clearer.
Don’t export module entry points when building executable and not modules. These exports cause MinGW and MSVC compilers to create export libraries.

Optimization

More efficient code for conditional expressions in conditions:
```
if a if b else c:
    ...
```
See above, this code is now the typical pattern for each or and and, so this was much needed now.

Organizational

The remaining uses of C++11 have been removed. Code generated with Nuitka and complementary C++ code now compile with standard C++03 compilers. This lowers the Nuitka requirements and enables at least g++ 4.4 to work with Nuitka.
The usages of the GNU extension operation a ?: b have replaced with standard C++ constructs. This is needed to support MSVC which doesn’t have this.
Added examples for the typical use cases to the User Manual.
The “compare_with_cpython” script has gained an option to immediately remove the Nuitka outputs (build directory and binary) if successful. Also the temporary files are now put under “/var/tmp” if available.
Debian package improvements, registering with doc-base the User Manual so it is easier to discover. Also suggest mingw32 package which provides the cross compiler to Windows.
Partial support for MSVC (Visual Studio 2008 to be exact, the version that works with CPython2.6 and CPython2.7).

All basic tests that do not use generators are working now, but those will currently cause crashes.
Renamed the --g++-only option to --c++-only.

The old name is no longer correct after clang and MSVC have gained support, and it could be misunderstood to influence compiler selection, rather than causing the C++ source code to not be updated, so manual changes will the used.
Catch exceptions for continue, break, and return only where needed for try/finally and loop constructs.

New Tests

Added CPython3.2 test suite as “tests/CPython32” from 3.2.3 and run it with CPython2.7 to check that Nuitka gives compatible error messages. It is not expected to pass yet on Python3.2, but work will be done towards this goal.
Make CPython2.7 test suite runner also execute the generated “doctest” modules.
Enabled tests for default parameters and their reference counts.

Summary

This release marks an important point. The compiled frames are exciting new technology, that will allow even better integration with CPython, while improving speed. Lowering the requirements to C++03 means, we will become usable on Android and with MSVC, which will make adoption of Nuitka on Windows easier for many.

Structurally the outstanding part is the function as references cleanup. This was a blocker for value propagation, because now functions references can be copied, whereas previously this was duplicating the whole function body, which didn’t work, and wasn’t acceptable. Now, work can resume in this domain.

Also very exciting when it comes to optimization is the remove of special code for or and and operators, as these are now only mere conditional expressions. Again, this will make value propagation easier with two special cases less.

And then of course, with Mercurial unit tests running compiled with Nuitka, an important milestone has been hit.

For a while now, the focus will be on completing Python3 support, XML based optimization regression tests, benchmarks, and other open ends. Once that is done, and more certainty about Mercurial tests support, I may call it a 0.4 and start with local type inference for actual speed gains.

Letting go of C++11

2012-10-27T08:59:07+02:00

How Nuitka came to use C++11 and then stop it.

This post is about Nuitka the Python compiler started out using C++0x which is now C++11, and then chose to stop it.

In the Beginning

Very early on, when I considered how to generate code from the node tree, in a way, that mistakes should practically be impossible to make, I made the fundamental decision, that every Python expression, which produces temporary variables, should become an expression in the generated code too.

Note

That is my choice, I think it keeps code generation more simple, and easier to understand. There may come a separate post about how that played out.

That decision meant some trouble. Certain things were not easy, but generally, it was achievable for g++ relatively quickly, and then lots of helper functions would be needed. Think of MAKE_TUPLE and MAKE_DICT, but also other stuff needed that. Calling a Python built-in with variable number of parameters e.g. could be implemented that way easily.

Other nice things were enum classes, and generally good stuff. It was really quick to get Nuitka code generation off the ground this way.

Reality Strikes

But then, as time went on, I found that the order of evaluation was becoming an issue. It became apparent that for more and more things, I needed to reverse it, so it works. Porting to ARM, it then became clear, that it needs to be the other way around for that platform. And checking out clang, which is also a C++11 compiler, I noticed, this one yet uses a different one.

So, for normal functions, I found a solution that involves the pre-processor to reverse or not, both function definition and call sites, and then it is already correct.

This of course, doesn’t work for C++11 variadic functions. So, there came a point, where I had to realize, that each of its uses was more or less causing evaluation order bugs. So that most of their uses were already removed. And so I basically knew they couldn’t stay that way.

Other Features

Also, things I initially assumed, e.g. that lambda functions of C++11 may prove useful, or even “auto”, didn’t turn out to be true. There seemingly is a wealth of new features, besides variadic templates that I didn’t see how Nuitka would benefit from it at all.

New Wishes

Then, at Europython, I realized, that Android is still stuck with g++-4.4 and as such, that an important target platform will be unavailable to me. This platform will become even more important, as I intend to buy an device now.

Biting it

So what I did, was to remove all variadic functions and instead generate code for them as necessary. I just need to trace the used argument counts, and then provide those, simple enough.

Also, other things like deleted copy constructors, and so on, I had to give up on these a bit.

This change was probably suited to remove subtle evaluation order problems, although I don’t recall seeing them.

The Present

The current stable release still requires C++11, but the next release will work on g++-4.4 and compiles fine with MSVC from Visual Studio 2008, although at this time, there is still the issue of generators not working yet, but I believe that ought to be solvable.

The new requirement is only C++03, which means, there is a good chance that supporting Android will become feasible. I know there is interest from App developers, because there, even the relatively unimportant 2x speedup, that Nuitka might give for some code, may matter.

Conclusion

So that is a detour, I have taken, expanding the base of Nuitka even further. I felt, this was important enough to write down the history part of it.

Nuitka Release 0.3.24

2012-08-18T21:30:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release contains progress on many fronts, except performance.

The extended coverage from running the CPython 2.7 and CPython 3.2 (partially) test suites shows in a couple of bug fixes and general improvements in compatibility.

Then there is a promised new feature that allows to compile whole packages.

Also there is more Python3 compatibility, the CPython 3.2 test suite now succeeds up to “test_builtin.py”, where it finds that str doesn’t support the new parameters it has gained, future releases will improve on this.

And then of course, more re-formulation work, in this case, class definitions are now mere simple functions. This and later function references, is the important and only progress towards type inference.

Bug Fixes

The compiled method type can now be used with copy module. That means, instances with methods can now be copied too. Fixed in 0.3.23.1 already.
The assert statement as of Python2.7 creates the AssertionError object from a given value immediately, instead of delayed as it was with Python2.6. This makes a difference for the form with 2 arguments, and if the value is a tuple. Fixed in 0.3.23.1 already.
Sets written like this didn’t work unless they were predicted at compile time:
```
{value}
```
This apparently rarely used Python2.7 syntax didn’t have code generation yet and crashed the compiler. Fixed in 0.3.23.1 already.
For Python2, the default encoding for source files is ascii, and it is now enforced by Nuitka as well, with the same SyntaxError.
Corner cases of exec statements with nested functions now give proper SyntaxError exceptions under Python2.
The exec statement with a tuple of length 1 as argument, now also gives a TypeError exception under Python2.
For Python2, the del of a closure variable is a SyntaxError.

New Features

Added support creating compiled packages. If you give Nuitka a directory with an “__init__.py” file, it will compile that package into a “.so” file. Adding the package contents with --recurse-dir allows to compile complete packages now. Later there will be a cleaner interface likely, where the later is automatic.
Added support for providing directories as main programs. It’s OK if they contain a “__main__.py” file, then it’s used instead, otherwise give compatible error message.
Added support for optimizing the super built-in. It was already working correctly, but not optimized on CPython2. But for CPython3, the variant without any arguments required dedicated code.
Added support for optimizing the unicode built-in under Python2. It was already working, but will become the basis for the str built-in of Python3 in future releases.
For Python3, lots of compatibility work has been done. The Unicode issues appear to be ironed out now. The del of closure variables is allowed and supported now. Built-ins like ord and chr work more correctly and attributes are now interned strings, so that monkey patching classes works.

Organizational

Migrated “bin/benchmark.sh” to Python as “misc/run-valgrind.py” and made it a bit more portable that way. Prefers “/var/tmp” if it exists and creates temporary files in a secure manner. Triggered by the Debian “insecure temp file” bug.
Migrated “bin/make-dependency-graph.sh” to Python as “misc/make-dependency-graph.py” and made a more portable and powerful that way.

The filtering is done a more robust way. Also it creates temporary files in a secure manner, also triggered by the Debian “insecure temp file” bug.

And it creates SVG files and no longer PostScript as the first one is more easily rendered these days.
Removed the “misc/gist” git sub-module, which was previously used by “misc/make-doc.py” to generate HTML from User Manual and Developer Manual.

These are now done with Nikola, which is much better at it and it integrates with the web site.
Lots of formatting improvements to the change log, and manuals:
- Marking identifiers with better suited ReStructured Text markup.
- Added links to the bug tracker all Issues.
- Unified wordings, quotation, across the documents.

Cleanups

The creation of the class dictionaries is now done with normal function bodies, that only needed to learn how to throw an exception when directly called, instead of returning NULL.

Also the assignment of __module__ and __doc__ in these has become visible in the node tree, allowing their proper optimization.

These re-formulation changes allowed to remove all sorts of special treatment of class code in the code generation phase, making things a lot simpler.
There was still a declaration of PRINT_ITEMS and uses of it, but no definition of it.
Code generation for “main” module and “other” modules are now merged, and no longer special.
The use of raw strings was found unnecessary and potentially still buggy and has been removed. The dependence on C++11 is getting less and less.

New Tests

Updated CPython2.6 test suite “tests/CPython26” to 2.6.8, adding tests for recent bug fixes in CPython. No changes to Nuitka were needed in order to pass, which is always good news.
Added CPython2.7 test suite as “tests/CPython27” from 2.7.3, making it public for the first time. Previously a private copy of some age, with many no longer needed changes had been used by me. Now it is up to par with what was done before for “tests/CPython26”, so this pending action is finally done.
Added test to cover Python2 syntax error of having a function with closure variables nested inside a function that is an overflow function.
Added test “BuiltinSuper” to cover super usage details.
Added test to cover del on nested scope as syntax error.
Added test to cover exec with a tuple argument of length 1.
Added test to cover barry_as_FLUFL future import to work.
Removed “Unicode” from known error cases for CPython3.2, it’s now working.

Summary

This release brought forward the most important remaining re-formulation changes needed for Nuitka. Removing class bodies, makes optimization yet again simpler. Still, making function references, so they can be copied, is missing for value propagation to progress.

Generally, as usual, a focus has been laid on correctness. This is also the first time, I am release with a known bug though, one which I believe now, may be the root cause of the mercurial tests not yet passing.

The solution will be involved and take a bit of time. It will be about “compiled frames” and be a (invasive) solution. It likely will make Nuitka faster too. But this release includes lots of tiny improvements, for Python3 and also for Python2. So I wanted to get this out now.

As usual, please check it out, and let me know how you fare.

Python 3 wonders - Breaking str

2012-08-15T21:42:28+02:00

That just killed some hope inside of me, breaking code that uses str ought to be forbidden.

Update

Turns out, this only a bug, and not intentional. And the bug is only in the doc string, so it’s being fixed, and there is no inconsistency then any more.

Python 3:

Python 3.2.3 (default, Jun 25 2012, 23:10:56)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print( str.__doc__ )
str(string[, encoding[, errors]]) -> str

Create a new string object from the given encoded string.
encoding defaults to the current default string encoding.
errors can be 'strict', 'replace' or 'ignore' and defaults to 'strict'.
>>> str( string = "a" )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'string' is an invalid keyword argument for this function

Python 2:

Python 2.7.3 (default, Jul 13 2012, 17:48:29)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print str.__doc__
str(object) -> string

Return a nice string representation of the object.
If the argument is a string, the return value is the same object.
>>> str( object = "a" )
'a'

I do understand that it’s in fact just the old unicode built-in. In fact, I made it work like that for Nuitka just now. But there is a difference, for Python2, it was well behaved.

Python 2:

>>> print unicode.__doc__
unicode(string [, encoding[, errors]]) -> object

Create a new Unicode object from the given encoded string.
encoding defaults to the current default string encoding.
errors can be 'strict', 'replace' or 'ignore' and defaults to 'strict'.
>>> unicode( string = "a" )
u'a'

Is Python3 supposed to be more clean or what? I think it is not happening.

Yours,

Kay Hayen

Python 3 wonders - Barry BDFL

2012-07-23T06:56:41+02:00

While writing Nuitka I get to see an absurd amount of CPython code. For a while now, it’s also CPython3.2 that I look at. Checking out __future__ handling, I was surprised the other day though, this really works:

# Python 3.2.3 (default, Jun 25 2012, 23:10:56)
# [GCC 4.7.1] on linux2
>>> from __future__ import barry_as_FLUFL
>>> 1 <> 2
True
>>> 1 != 2
  File "<stdin>", line 1
    1 != 2
       ^
SyntaxError: invalid syntax

It’s new in CPython3, and this the code that makes it possible, from the Python parser:

if (type == NOTEQUAL) {
    if (!(ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                    strcmp(str, "!=")) {
        PyObject_FREE(str);
        err_ret->error = E_SYNTAX;
        break;
    }
    else if ((ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                    strcmp(str, "<>")) {
        PyObject_FREE(str);
        err_ret->text = "with Barry as BDFL, use '<>' "
                        "instead of '!='";
        err_ret->error = E_SYNTAX;
        break;
    }
}

Now who would think bad of that, would you? The fun aspect is, that Nuitka will easily supports it. By re-using the Python parser, it works out of the box, I only needed to add the flag value.

For fun, I tried to add a test that confirms - and then notice:

It doesn’t really work for CPython3.2 already.

The flag is only used for eval and exec and not on the same level, so it’s only inherited.

And “2to3” kindly removes that flag silently. It probably should raise an error.

So yeah, oddities in Python3!

Python Assert Quiz

2012-07-15T17:07:00+02:00

At Europython conference, in my presentation, I talked about re-formulations of Python into simpler Python. It is my intention to turn this into a series of Python quiz questions that you will hopefully enjoy.

Update

Due to comments feedback, I made it more clear that “-O” affects of course both cases, and due to work on getting the recent CPython2.7 test suite to work, I noticed, how the re-formulation for Quiz Question 2 needed a version dependent solution.

And I thought this one was easy. :-)

Quiz Question 1

Say you have the following code:

assert x == y

How can you achieve the same thing, without using the assert statement at all. The behavior is required to be absolutely the same.

The answer is in the next paragraph, so stop reading if you want to find out yourself.

Solution 1

The correct answer is that assertions are the same as a raise exception in a conditional statement.

if not x == y:
    raise AssertionError

The thing where this makes a difference, is “-O”, which will discard assertions, but I consider it rarely used. To be really compatible with that, it should be:

if __debug__ and not x == y:
    raise AssertionError

Quiz Question 2

But wait, there is slightly more to it. Say you have the following code:

assert x == y, arg

How can you achieve the same thing, without using the assert statement at all. The behavior is required to be absolutely the same.

The answer is in the next paragraph, so stop reading if you want to find out yourself.

Solution 2

This is actually version dependent, due to recent optimizations of CPython.

For version 2.6 it is as follows:

The extra value to assert, simply becomes an extra value to raise, which indicates, delayed creation of the AssertionError exception.
```
if not x == y:
    raise AssertionError, arg
```
For version 2.7 and higher it is as follows:

The extra value to assert, simply becomes the argument to creating the AssertionError exception.
```
if not x == y:
    raise AssertionError(arg)
```

So, even in the more complex case, you end up with a conditional raise.

The only thing where this makes a difference, is “-O”, which will discard assertions, but I consider it rarely used. To be really compatible with that, it should be:

if __debug__ and not x == y:
   raise AssertionError ....

Surprised? Well, yes, there really is nothing to assert statements. I am using this for my Python compiler Nuitka which benefits from having not to deal with assert as anything special at all. See also the respective section in the Developer Manual which explains this and other things.

Nuitka and Debian changes

2012-07-05T14:52:00+02:00

Hello everybody,

this is just a quick note to Nuitka users that are Debian users. For Debian, the freeze has arrived. This is a time of great joy for me. I love Debian so deeply, I believe I am now for at least 10 years on it, of 15 years on Linux.

As you may know, Nuitka has been in Debian for some time now. In discussion with my great sponsor Yaroslav Halchenko, we needed to consider if Nuitka is ready to be maintained in Debian Stable.

And for that I have to admit, it is just not ready. You will be able to download packages from the Downloads and install them. They will still be in Debian Unstable, but they won’t be in Debian Testing and the next Debian Stable.

Nuitka Release 0.3.23

2012-06-30T23:40:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is the one that completes the Nuitka “sun rise phase”.

All of Nuitka is now released under Apache License 2.0 which is a very liberal license, and compatible with basically all Free Software licenses there are. It’s only asking to allow integration, of what you send back, and patent grants for the code.

In the first phase of Nuitka development, I wanted to keep control over Nuitka, so it wouldn’t repeat mistakes of other projects. This is no longer a concern for me, it’s not going to happen anymore.

I would like to thank Debian Legal team, for originally bringing to my attention, that this license will be better suited, than any copyright assignment could be.

Bug Fixes

The compiled functions could not be used with multiprocessing or copy.copy. Fixed in 0.3.22.1 already.
In-place operations for slices with not both bounds specified crashed the compiler. Fixed in 0.3.22.1 already.
Cyclic imports could trigger an endless loop, because module import expressions became the parent of the imported module object. Fixed in 0.3.22.2 already.
Modules named proc or func could not be compiled to modules or embedded due to a collision with identifiers of CPython2.7 includes. Fixed in 0.3.22.2 already.

New Features

The function copying fix also makes pickling of compiled functions available. As it is the case for non-compiled functions in CPython, no code objects are stored, only names of module level variables.

Organizational

Using the Apache License 2.0 for all of Nuitka now.
Speedcenter has been re-activated, but is not yet having a lot of benchmarks yet, subject to change.

Update

We have given up on this version of speedcenter meanwhile, and generate static pages with graphs instead. We can this still speedcenter.

New Tests

Changed the “CPython26” tests to no longer disable the parts that relied on copying of functions to work as that is now supported.
Extended in-place assignment tests to cover error cases of we had issues with.
Extended compile library test to also try and compile the path where numpy lives. This is apparently another path, where Debian installs some modules, and compiling this would have revealed issues sooner.

Summary

The release contains bug fixes, and the huge step of changing the license. It is made in preparation to PyCON EU.

Nuitka Speedcenter is back

2012-06-30T16:39:00+02:00

Once a long time ago, I was benchmarking Nuitka more often. Check “What is Nuitka?” in case you don’t know what it is.

Problem

And I was considering the use of codespeed, and had some data online. But ultimately, it got discontinued. This has 3 reasons:

Moved the website to a dedicated machine, which broke the previous install.
Controlling which data is used for display was hard and not satisfactory.
1. For example, I didn’t want to have to commit and push, just to let the benchmarks run.
2. And I wanted to be able to re-run benchmarks with newer compiler, even newer Python, but old Nuitka. Using g++ 4.6 over g++ 4.5 should not impact the data.
3. It turned out to be a nightmare to migrate to newer codespeed versions. I found myself starting from empty database - over and over.
4. Many things were not supported.
  
  For example, I would want to collect all PyBench results, but only publish those who are expressive. That seemed difficult to achieve.
Benchmarks of Nuitka are not yet useful
1. Nuitka was not yet doing type inference
2. Most of the work was aimed at correctness, and effectively was often degrading performance even if only temporary. Seeing it wouldn’t have been too motivating.

Solution

I have simply created a small wrapper:

Small script to run benchmarks and collect data.

It checks out Nuitka in all versions in a playground, and then runs defined benchmarks, with valgrind, etc. taking exe sizes, etc.
Data is stored in local sqlite databases.

I have a database per machine, i.e. a distributed repository, where I collect information. That works for me, and will allow me to compare different kinds of machines.

The advantage is that I have no risk of data loss anymore, and no issues and difficulty with poor interfaces to replace existing data.
Data is merged on one machine, and then pushed.

That allows me to inspect the changes before publishing them. It allows me to play with local commits, branches, with information that will go away. I can then push when I choose to.

That integrates better with my work flow. It allows me to retro-fit benchmarks results on the machine and to be tool independent.

In principle, I could publish the data in other forms as well, and I likely will. Making tables of e.g. PyBench results seems like one application. Recently, I have worked with Nikola, and could also imagine to integrate Codespeed graph functionality (which is apparently all I want) to there.

Yours,

Kay

Nikola Speed Improvements

2012-06-12T13:15:00+02:00

At end the end of May, I switched my website from a W3 Total Cache enhanced wordpress installation to static generated with Nikola, and this is what Google Webmaster tool says about it.

The numbers are average time to download pages in milliseconds.

Nikola for nuitka.net

2012-05-26T13:06:00+02:00

Some time ago, I asked about static site generators and tried a few, but nothing was as ReST friendly as I would like to. To me, ReST is to text publishing, what Python is to programming.

Well, the search has found a late hit. I had installed a work flow, where I was using the XMLRPC interface, but it was some effort, and somewhat limited.

Then Roberto Alsina announced Nikola that was just what I wanted.

I did migrate to it during the last days. Looking at the content in ReST form, was a big ease, and I already found a bit of obsolete content, and updated it. And the last release announcements, they now render somewhat prettier.

I am not using all the features yet, notable not using the image galleries yet, and the download page is not yet generated again, and so on. But this is a good base.

And while I loved it, this was the result:

apt-get remove wordpress

Nuitka Release 0.3.22

2012-05-13T13:17:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimization.

It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to compile itself with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it’s quite fine now.

Bug Fixes

List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions.
Calling a compiled function in an exception handler cleared the exception on return, it no longer does that.
Reference counter handling with generator throw method is now correct.
A module “builtins” conflicted with the handling of the Python builtins module. Those now use different identifiers.

New Features

New metaclass syntax for the class statement works, and the old __metaclass__ attribute is properly ignored.

# Metaclass syntax in Python3, illegal in Python2
class X(metaclass=Y):
    pass

# Metaclass syntax in Python2, no effect in Python3
class X:
    __metaclass__ = Y

Note

The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn’ it. Surely, the support for __metaclass__ could still live.

# For Python2/3 compatible source, we create a base class that has the
# metaclass used and doesn't require making a choice.

CPythonNodeMetaClassBase = NodeCheckMetaClass("CPythonNodeMetaClassBase", (object,), {})

The --dump-xml option works with Nuitka running under Python3. This was not previously supported.
Python3 now also has compatible parameter errors and compatible exception error messages.
Python3 has changed scope rules for list contractions (assignments don’t affect outside values) and this is now respected as well.
Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well.

Optimization

Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used.

This avoids overhead for the very simple functions. And example of this can be seen here:
```
def simple():
    return 7
```

Optimize len built-in for non-constant, but known length values.

An example can be seen here:

# The range isn't constructed at compile time, but we still know its
# length.
len(range(10000000))

# The string isn't constructed at compile time, but we still know its
# length.
len("*" * 1000)

# The tuple isn't constructed, instead it's known length is used, and
# side effects are maintained.
len((a(), b()))

This new optimization applies to all kinds of container creations and the range built-in initially.

Optimize conditions for non-constant, but known truth values.

At this time, known truth values of non-constants means range built-in calls with know size and container creations.

An example can be seen here:
```
if (a,):
    print "In Branch"
```
It’s clear, that the tuple will be true, we just need to maintain the side effect, which we do.

Optimize or and and operators for known truth values.

See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can:

# The access and call to "something()" cannot possibly happen
0 and something()

# Can be replaced with "something()", as "1" is true. If it had a side
# effect, it would be maintained.
1 and something()

# The access and call to "something()" cannot possibly happen, the value
# is already decided, it's "1".
1 or something()

# Can be replaced with "something()", as "0" is false. If it had a side
# effect, it would be maintained.
0 or something()

Optimize print arguments to become strings.

The arguments to print statements are now converted to strings at compile time if possible.
```
print 1
```
becomes:
```
print "1"
```
Combine print arguments to single ones.

When multiple strings are printed, these are now combined.
```
print "1+1=", 1 + 1
```
becomes:
```
print "1+1= 2"
```

Organizational

Enhanced Python3 support, enabling support for most basic tests.
Check files with PyLint in deterministic (alphabetical) order.

Cleanups

Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module.
The try/except/else has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not.
Side effects are now a dedicated node, used in several optimization to maintain the effect of an expression with known value.

New Tests

Expanded and adapted basic tests to work for Python3 as well.
Added reference count tests for generator functions throw, send, and close methods.
Cover calling a function with try/except in an exception handler twice. No test was previously doing that.

Summary

This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away.

There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense.

Nuitka Release 0.3.21

2012-04-13T10:19:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it will be easy to start tracking them.

Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc.

Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete.

Bug Fixes

The built-in next could causes a program crash when iterating past the end of an iterator. Fixed in 0.3.20.1 already.
The set constants could cause a compiler error, as that type was not considered in the “mutable” check yet. Fixed in 0.3.20.2 already.
Performance regression. Optimize expression for exception types caught as well again, this was lost in last release.
Functions that contain exec, are supposed to have a writable locals. But when removing that exec statement as part of optimization, this property of the function could get lost.
The so called “overflow functions” are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an exec or a star import.
The syntax error for return outside of a function, was not given, instead the code returned at run time. Fixed to raise a SyntaxError at compile time.

Optimization

Avoid tuple objects to be created when catching multiple exception types, instead call exception match check function multiple times.
Removal of dead code following break, continue, return, and raise. Code that follows these statements, or conditional statements, where all branches end with it.

Note

These may not actually occur often in actual code, but future optimization may produce them more frequently, and their removal may in turn make other possible optimization.
Detect module variables as “read only” after all writes have been detected to not be executed as removed. Previously the “read only indicator” was determined only once and then stayed the same.
Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value.
Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still.

Note

An exception is where len = len actually does have an impact, because that variable becomes assignable. The “compile itself” test of Nuitka found that to happen with long from the nuitka.__past__ module.
Created Python3 variant of quick unicode string access, there was no such thing in the CPython C/API, but we make the distinction in the source code, so it makes sense to have it.
Created an optimized implementation for the built-in iter with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython C/API.
For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take.
Don’t use a “context” object for generator functions (and generator expressions) that don’t need one. And even if it does to store e.g. the given parameter values, avoid to have a “common context” if there is no closure taken. This avoids useless malloc calls and speeds up repeated generator object creation.

Organizational

Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated.
The --experimental option is no longer available outside of checkouts of git, and even there not on stable branches (master, hotfix/...). It only pollutes --help output as stable releases have no experimental code options, not even development version will make a difference.
The binary “bin/Nuitka.py” has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway.
The --python-version option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values.
Added --keep-pythonpath alias for --execute-with-pythonpath option, probably easier to remember.
Support --debug with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn’t report anything new.
The contents environment variable CXX determines the default C++ compiler when set, so that checking with CXX=g++-4.7 nuitka-python ... has become supported.
The check-with-pylint script now has a real command line option to control the display of TODO items.

Cleanups

Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead.
```
a = b = c
```
```
_tmp = c
b = _tmp
a = _tmp
```
In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block.

All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed.
Changed unpacking assignments to use temporary variables. Code like this:
```
a, b = c
```
Is handled more like this:
```
_tmp_iter = iter(c)
_tmp1 = next(_tmp_iter)
_tmp2 = next(_tmp_iter)
if not finished(_tmp_iter):
    raise ValueError("too many values to unpack")
a = _tmp1
b = _tmp2
```
In reality, not really next is used, as it wouldn’t raise the correct exception for unpacking, and the finished check is more condensed into it.

Generally this cleanup allowed that the AssignTargetTuple and associated code generation was removed, and in the future value propagation may optimize these next and iter calls away where possible. At this time, this is not done yet.
Exception handlers assign caught exception value through assignment statement.

Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well.
Exception handlers now explicitly catch more than one type.

Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it’s all.
Contractions are now functions as well.

Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation.
Global is handled during tree building.

Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node.

It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that global statement, allowing to remove the node class.
Read only module variable detection integrated to constraint collection.

The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed.

New Tests

Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment.
Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn’t even know it can be done, example:
```
try:
    raise ValueError(1, 2)
except ValueError as (a, b):
    print "Unpacking caught exception and unpacked", a, b
```
Will assign a=1 and b=2.
Added test to cover return statements on module level and class level, they both must give syntax errors.
Cover exceptions from accessing unassigned global names.
Added syntax test to show that star imports do not allow other names to be imported at the same time as well.
Python3 is now also running the compile itself test successfully.

Summary

The progress made towards value propagation and type inference is very significant, and makes those appears as if they are achievable.

Nuitka Release 0.3.20

2012-02-27T22:48:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This time there are a few bug fixes and some really major cleanups, lots of new optimization and preparations for more. And then there is a new compiler clang and a new platform supported. macOS X appears to work mostly, thanks for the patches from Pete Hunt.

Bug Fixes

The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. Fixed in release 0.3.19.1 already.
The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. Fixed in release 0.3.19.1 already.
The from x import * attempted to respect __all__ but failed to do so. Fixed in release 0.3.19.2 already.
The from x import * didn’t give a SyntaxError when used on Python3. Fixed in release 0.3.19.2 already.
The syntax error messages for “global for function argument name” and “duplicate function argument name” are now identical as well.
Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed.

New Features

Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent.
Enhanced Python3 support for syntax errors, these are now also compatible.
Support for macOS X was added.
Support for using the clang compiler was added, it can be enforced via --clang option. Currently this option is mainly intended to allow testing the “macOS X” support as good as possible under Linux.

Optimization

Enhanced all optimization that previously worked on “constants” to work on “compile time constants” instead. A “compile time constant” can currently also be any form of a built-in name or exception reference. It is intended to expand this in the future.
Added support for built-ins bin, oct, and hex, which also can be computed at compile time, if their arguments are compile time constant.
Added support for the iter built-in in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster.
Added support for the next built-in, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well.
Added support for the open built-in in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files.
Optimize the __debug__ built-in constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided.
Optimize the Ellipsis built-in constant as well. It falls in the same category as True, False, None, i.e. names of built-in constants that a singletons.
Added support for anonymous built-in references, i.e. built-ins which have names that are not normally accessible. An example is type(None) which is not accessible from anywhere. Other examples of such names are compiled_method_or_function.

Having these as represented internally, and flagged as “compile time constants”, allows the compiler to make more compile time optimization and to generate more efficient C++ code for it that won’t e.g. call the type built-in with None as an argument.
All built-in names used in the program are now converted to “built-in name references” in a first step. Unsupported built-ins like e.g. zip, for which Nuitka has no own code or understanding yet, remained as “module variables”, which made access to them slow, and difficult to recognize.
Added optimization for module attributes __file__, __doc__ and __package__ if they are read only. It’s the same as was done for __name__ so far only.
Added optimization for slices and subscripts of “compile time constant” values. These will play a more important role, once value propagation makes them more frequent.

Organizational

Created a “change log” from the previous release announcements. It’s as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it’s so easy to generate the PDF on that yourself.
The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to Wordpress as a draft posting or updates it, until it’s release time. Simple, sweet and elegant.

Cleanups

Split out the nuitka.nodes.Nodes module into many topic nodes, so that there are now nuitka.nodes.BoolNodes or nuitka.nodes.LoopNodes to host nodes of similar kinds, so that it is now cleaner.
Split del statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same:
```
del a, b.c, d
```
```
del a
del b.c
del d
```
So that’s now represented in the node tree. And even more complex looking cases, like this one, also the same:
```
del a, (b.c, d)
```
This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all.
Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions.

Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the bases for classes and the defaults for functions and make optional.
Refactored the decorator handling to the tree building stage, presenting them as function calls on “function body expression” or class body expression”.

This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimization, as they practically are now just function calls.
```
@some_classdecorator
class C:
    @staticmethod
    def f():
        pass
```
It’s just a different form of writing things. Nothing requires the implementation of decorators, it’s just functions calls with function bodies before the assignment.

The following is only similar:
```
class C:
    def f():
        pass

    f = staticmethod(f)


C = some_classdecorator(C)
```
It’s only similar, because the assignment to an intermediate value of C and f is not done, and if an exception was raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already.
The in-place assignments statements are now handled using temporary variable blocks

Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal look-ups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future.
The for loop used to have has a “source” expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimization. Moved it to tree building stage instead, where optimization can work on it then.
Tree building now generally allows statement sequences to be None everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist.
Moved the optimization for __name__ to compute node of variable references, where it doesn’t need anything complex to replace with the constant value if it’s only read.
Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults.
Made the built-in code more reusable.

New Tests

Added some more diagnostic tests about complex assignment and del statements.
Added syntax test for star import on function level, that must fail on Python3.
Added syntax test for duplicate argument name.
Added syntax test for global on a function argument name.

Summary

The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much more consistency there.

Yet, even more work will be needed in the same direction. There may e.g. be work required to cover with statements as well. And assignments will become no more complex than unpacking from a temporary variable.

For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only minuscule progress. The remaining tasks appear all more or less difficult work that I don’t want to touch now.

There are still remaining steps, but we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about.

Award winning cat

2012-02-20T12:51:00+01:00

The sweetness phase is over. At least in that instant. :-)

Ubuntu Packages for Nuitka

2012-01-31T08:25:00+01:00

Hello there,

as I previously said, I will make these a separate announcement. Thanks to the NeuroDebian project build infrastructure, there is Nuitka packages for the Ubuntu versions back to Maverick, Natty, and Oneiric.

Please see the page What is Nuitka? for clarification of what it is now and what it wants to be.

You may use the apt sources to track Nuitka or download the packages from the page where I offer the latest version here.

Note: In fact, it’s 0.3.19.1 now, which includes a hotfix release made after 0.3.19. The same is in Debian Unstable right now, waiting to propagate into Debian Testing, where there is still 0.3.18 living at this time.

I would appreciate feedback on these.

Thanks in advance, Kay

Nuitka Release 0.3.19

2012-01-26T20:49:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things in this are justifying a new release.

Bug Fixes

The man pages of nuitka and nuitka-python had no special layout for the option groups and broken whitespace for --recurse-to option. Also --g++-only was only partially bold. Released as 0.3.18.1 hot fix already.
The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hot fix already.
Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of / instead of using a joinpath call. Released as 0.3.18.3 already.
A call to the range built-in with no arguments would crash the compiler, see Released as 0.3.18.4 already.
Compatibility Fix: When rich comparison operators returned false value other False, for comparison chains, these would not be used, but False instead, see .
The support for __import__ didn’t cover keyword arguments, these were simply ignored. Fixed, but no warning is given yet.

New Features

A new option has been added, one can now specify --recurse-directory and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements.
Added support for the exec built-in of Python3, this enables us to run one more basic test, GlobalStatement.py with Python3. The test ExecEval.py nearly works now.

Optimization

The no arguments range() call now optimized into the static CPython exception it raises.
Parts of comparison chains with constant arguments are now optimized away.

Cleanups

Simplified the CPythonExpressionComparison node, it now always has only 2 operands.

If there are more, the so called “comparison chain”, it’s done via and with assignments to temporary variables, which are expressed by a new node type CPythonExpressionTempVariableRef. This allowed to remove expression_temps from C++ code templates and generation, reducing the overall complexity.
When executing a module (--execute but not --exe), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment.
The calls to the variadic MAKE_TUPLE were replaced with calls the MAKE_TUPLExx (where xx is the number of arguments), that are generated on a as-needed basis. This gives more readable code, because no EVAL_ORDERED_xx is needed at call site anymore.
Many node classes have moved to new modules in nuitka.nodes and grouped by theme. That makes them more accessible.
The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the sys.abiflags and works with Python3.
The replacing of .py in filenames was made more robust. No longer is str.replace used, but instead proper means to assure that having .py as other parts of the filenames won’t be a trouble.
Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements.
As always, some PyLint work, and some minor TODO were solved.

Organizational

Added more information to the Developer Manual, e.g. documenting the tree changes for assert to become a conditional statement with a raise statement, etc.
The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise.
Added support to specify the binary under test with a NUITKA environment, so the test framework can run with installed version of Nuitka too.
Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for os.execl not propagating exit codes under Windows.
For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the PATH or even installed to execute the binary.

New Tests

The basic, programs, syntax, and reflected were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations.
Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains.
The Referencing.py test was made portable to Python3.
Cover no arguments range() exception as well.
Added test to demonstrate that --recurse-directory actually works. This is using an __import__ that cannot be predicted at run time (yet).
The created source package is now tested on pbuilder chroots to be pass installation and the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn’t break without notice.

Summary

This releases contains many changes. The “temporary variable ref” and “assignment expression” work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. in-place assignments can be reduced to assignments to temporary variables and conditional statements.

While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with “value propagation” working.

These are the real focus. The old comparison chain handling was a big wart. Working, but no way understood by any form of analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimization to see through them.

Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of in-place assignments. With an “assignment expression” to a “temporary variable ref”, these become the same as user code using such a variable. There will be more of these to find.

So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the __import__ related one, were not suitable for hot fix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already, also worth to get out.

Static Site Generator

2012-01-26T07:32:00+01:00

Hello,

somehow triggered by reading about Mynth, and the re-post of the last release announcement on my feed, just because I added a missing category to the post, I am wondering what it takes to achieve the following:

Edit the pages site as ReST (reStructured text) document with links inside

Edit posts as single ReST files

Have it look decent and provide feeds, proper caching headers, etc.

Generate a static site from with, with more control over what kind of updates makes it into the feed.

I am currently very annoyed, because:

I would normally generate code examples now with “rst2html” and replace the existing Wordpress plugin that renders in JavaScript. Except I cannot without re-posting all these articles, or removing the “python” tag from them.

The documentation of Nuitka is in ReST and I even intend to expand it with UML diagrams generated from text. Making the cross to HTML content is difficult.

Some pages only replicate content from the User or Developer Manual, keeping those updated, in sync, etc. is a dull thing to do.

Extending Wordpress in Python is not feasible. And in php, I doubt I want to master.

Other stuff in favor of it, is that Wordpress or any CMS needs more memory and more time to process than static pages of course. I can’t right now let anyone fork and edit the pages in a practical manner. So while for User Manual and Developer Manual of Nuitka I will get corrections, for the website it’s not happening.

Is Mynth an appropriate intermediate step? Do I really want to learn one of these strange templating languages or even consider software that has no Debian package. I never considered “virtualenv” and “pip” much of options to run software on a website. I guess I am more tied to Debian than to Python still.

Over the last months I have occasionally looked at Pyramids, but it seems overly complex, if all you want is to simplify the authoring. I have seen that e.g. Trac supports ReST as an exception, but that feels too little for a hefty software like that.

I may end up with using ReST to generate HTML which is then uploaded via XMLRPC to Wordpress. Won’t give me the “static” benefits. Won’t solve the post update problem. But at least will let me edit more easily.

Yours,

Kay

Nuitka Release 0.3.18

2012-01-14T11:34:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via apt-get install nuitka.

The release cycle was too short to have much focus. It merely includes fixes, which were available as hot fixes, and some additional optimization and node tree cleanups, as well as source cleanups. But not much else.

Bug Fixes

Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. Released as 0.3.17a hot fix already.
Nuitka was considering directories to contain packages that had no “__init__.py” which could lead to errors when it couldn’t find the package later in the compilation process. Released as 0.3.17a hot fix already.
When providing locals() to exec statements, this was not making the locals() writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn’t consider the actual value locals(). Released as 0.3.17b hot fix already.
Compatibility Fix: When no defaults are given, CPython uses None for func.func_defaults, but Nuitka had been using None.

Optimization

If the condition of assert statements can be predicted, these are now optimized in a static raise or removed.
For built-in name references, there is now dedicated code to look them up, that doesn’t check the module level at all. Currently these are used in only a few cases though.
Cleaner code is generated for the simple case of print statements. This is not only faster code, it’s also more readable.

Cleanups

Removed the CPythonStatementAssert node.

It’s not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with AssertionError and the assertion argument.

This allowed to remove code and complexity from the subsequent steps of Nuitka, and enabled existing optimization to work on assert statements as well.
Moved built-in exception names and built-in names to a new module nuitka.Builtins instead of having in other places. This was previously a bit spread-out and misplaced.
Added cumulative tags to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
New node for built-in name lookups

This allowed to remove tricks played with adding module variable lookups for staticmethod when adding them for __new__ or module variable lookups for str when predicting the result of type('a'), which was unlikely to cause a problem, but an important TODO item still.

Organizational

The “Download” page is now finally updated for releases automatically.

Up to this release, I had to manually edit that page, but now mastered the art of upload via XMLRCP and a Python script, so that don’t loose as much time with editing, checking it, etc.
The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages.
Made sure the test runners worth with bare python2.6 as well.

New Tests

Added some tests intended for type inference development.

Summary

This releases contains not as much changes as others, mostly because it’s the intended base for a Debian upload.

The exec fix was detected by continued work on the branch feature/minimize_CPython26_tests_diff branch, but that work is now complete.

It is being made pretty (many git rebase iterations) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made.

But of course, the real excitement is the “type inference” work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient.

So when I learn something. e.g. assert is not special, I apply it to the develop branch immediately, to keep the differences as small as possible, and to immediately benefit from such improvements.

Nuitka Release 0.3.17

2012-01-09T07:59:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organizational work, and yet again improved compatibility and cleanups. Also huge is the advance in making --deep go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one.

The release cycle focused on improving the quality of the test scripts, the packaging, and generally to prepare the work on “type inference” in a new feature branch.

I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via 2to3 conversion) without trouble. There is still work to do, exceptions don’t seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day.

And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository.

Bug Fixes

Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hot fix already.
Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hot fix already.
Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hot fix already.
Fix, add the missing staticmethod decorator to __new__ methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hot fix already.

New Features

Enhanced compatibility again, provide enough co_varnames in the code objects, so that slicing them up to code_object.co_argcount will work. They are needed by inspect module and might be used by some decorators as well.
New options to control the recursion:

--recurse-none (do not warn about not-done recursions) --recurse-all (recurse to all otherwise warned modules) --recurse-to (confirm to recurse to those modules) --recurse-not-to (confirm to not recurse to those modules)

Optimization

The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case.
Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs.
Use the pickle “protocol 2” on CPython2 except for unicode strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of pickle was not really necessary anymore.

Organizational

Added a Developer Manual to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.
Improved the --help output to use metavar where applicable. This makes it more readable for some options.
Instead of error message, give help output when no module or program file name was given. This makes Nuitka help out more convenient.
Consistently use #!/usr/bin/env python for all scripts, this was previously only done for some of them.
Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for TODO items optional.
All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it’s also already the default for Python.
The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
Make the tests run without any dependence on PATH to contain the executables of Nuitka. This makes it easier to use.
Add license texts to 3rd-party file that were missing them, apply licensecheck results to cleanup Nuitka. Also removed own copyright statement from in-line copy of Scons, it had been added by accident only.
Release the tests that I own as well as the Debian packaging I created under “Apache License 2.0” which is very liberal, meaning every project will be able to use it.
Don’t require copyright assignment for contributions anymore, instead only “Apache License 2.0”, the future Nuitka license, so that the code won’t be a problem when changing the license of all of Nuitka to that license.
Give contributors listed in the User Manual an exception to the GPL terms until Nuitka is licensed under “Apache License 2.0” as well.
Added an --experimental option which can be used to control experimental features, like the one currently being added on branch feature/ctypes_annotation, where “type inference” is currently only activated when that option is given. For this stable release, it does nothing.
Check the static C++ files of Nuitka with cppcheck as well. Didn’t find anything.
Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit.

Cleanups

Changed not boolean operation to become a normal operator. Changed and and or boolean operators to a new base class, and making their interface more similar to that of operations.
Added cumulative tags to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
Enhanced the “visitor” interface to provide more kinds of callbacks, enhanced the way “each scope” visiting is achieved by generalizing is as “child has not tag ‘closure_taker’” and that for every “node that has tag ‘closure_taker’”.
Moved SyntaxHighlighting module to nuitka.gui package where it belongs.
More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that were white listed.
Consistently use nuitka in test scripts, as there isn’t a Nuitka.py on all platforms. The later is scheduled for removal.
Some more PyLint cleanups.

New Tests

Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail.
The basic tests can now be run with PYTHON=python3.2 and use 2to3 conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script check-release.
Include the syntax tests in release tests as well.
Changed many existing tests so that they can run under CPython3 too. Of course this is via 2to3 conversion.
Don’t fail if the CPython test suites are not there.

Currently they remain largely unpublished, and as such are mostly only available to me (exception, feature/minimize_CPython26_tests_diff branch references the CPython2.6 tests repository, but that remains work in progress).
For the compile itself test: Make the presence of the Scons in-line copy optional, the Debian package doesn’t contain it.
Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python.
Before releasing, test that the created Debian package builds fine in a minimal Debian unstable chroot, and passes all the tests included in the package (basics, syntax, programs, reflected). Also many other Debian packaging improvements.

Summary

The “git flow” was used again in this release cycle and proved to be useful not only for hot fix, but also for creating the branch feature/ctypes_annotation and rebasing it often while things are still flowing.

The few hot fixes didn’t require a new release, but the many organizational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control.

The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn’t regress by accident, so running with CPython3.2 is now part of the normal release tests.

What’s still missing is more “hg” completeness. Only the co_varnames work for inspect was going in that direction, and this has slowed down. It was more important to make Nuitka’s recursion more accessible with the new options, so that was done first.

And of course, the real excitement is the “type inference” work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of ctypes native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly.