Release Nuitka 0.2.3    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This new release is marking a closing in on feature parity to CPython 2.6 which is an important mile stone. Once this is reached, a "Nuitka 0.3.x" series will strive for performance.

Bug Fixes

  • Generator functions no longer leak references when started, but not finished.
  • Yield can in fact be used as an expression and returns values that the generator user send() to it.

Reduced Differences / New Features

  • Generator functions already worked quite fine, but now they have the throw(), send() and close() methods.

  • Yield is now an expression as is ought to be, it returns values put in by send() on the generator user.

  • Support for extended slices:

    x = d[:42, ..., :24:, 24, 100]
    d[:42, ..., :24:, 24, 100] = "Strange"
    del d[:42, ..., :24:, 24, 100]
    

Tests Work

  • The "test_contextlib" is now working perfectly due to the generator functions having a correct throw(). Added that test back, so context managers are now fully covered.
  • Added a basic test for "overflow functions" has been added, these are the ones which have an unknown number of locals due to the use of language constructs exec or from bla import * on the function level. This one currently only highlights the failure to support it.
  • Reverted removals of extended slice syntax from some parts of the CPython test suite.

Cleanups

  • The compiled generator types are using the new C++0x type safe enums feature.
  • Resolved a circular dependency between TreeBuilding and TreeTransforming modules.

Comments

Python Scope Quiz    Posted:


Quiz Question

Say you have the following module code:

a = 1

class some_class():
    a = a

def some_function():
    a = a

some_class()
some_function()

What is going to happen? Well, think about it, the solution is in the next paragraph.

Solution

The correct answer is that the call "some_function()" is going to give you a "UnboundLocalError" exception.

This is because it in functions unlike in classes (or modules) do look ahead for assigned to variable names. Python allocates a slot for local variables of functions and that is a property that doesn't change - unless you say "global" of course. This slot is used for every access to the variable name, which forbids you to make it local.

Surprised? I was too! I only learned it while doing my Python compiler Nuitka and I made it a separate posting, because it really surprised me how different function body and class body work.

Comments

Release Nuitka 0.2.2    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is some significant progress, a lot of important things were addressed.

Bug Fixes

  • Scope analysis is now done during the tree building instead of sometimes during code generation, this fixed a few issues that didn't show up in tests previously.
  • Reference leaks of generator expressions that were not fishing, but then deleted are not more.
  • Inlining of exec is more correct now.
  • More accurate exception lines when iterator creation executes compiled code, e.g. in a for loop
  • The list of base classes of a class was evaluated in the context of the class, now it is done in the context of the containing scope.
  • The first iterated of a generator expression was evaluated in its own context, now it is done in the context of the containing scope.

Reduced Differences

  • With the enhanced scope analysis, UnboundLocalError is now correctly supported.
  • Generator expressions (but not yet functions) have a throw(), send() and close() method.
  • Exec can now write to local function namespace even if None is provided at run time.
  • Relative imports inside packages are now correctly resolved at compile time when using --deep.

Cleanups

  • The compiled function type got further enhanced and cleaned up.
  • The compiled generator expression function type lead to a massive cleanup of the code for generator expressions.
  • Cleaned up namespaces, was still using old names, or "Py*" which is reserved to core CPython.
  • Overhaul of the code responsible for eval and exec, it has been split, and it pushed the detection defaults to the C++ compiler which means, we can do it at run time or compile time, depending on circumstances.
  • Made PyTemporaryObject safer to use, disabling copy constructor it should be also a relief to the C++ compiler if it doesn't have to eliminate all its uses.
  • The way delayed work is handled in TreeBuilding step has been changed to use closured functions, should be more readable.
  • Some more code templates have been created, making the code generation more readable in some parts. More to come.

New Features

  • As I start to consider announcing Nuitka, I moved the version logic so that the version can now be queried with --version.

New Optimization

  • Name lookups for None, True and False and now always detected as constants, eliminating many useless module variable lookups.

New Tests

  • More complete test of generator expressions.
  • Added test program for packages with relative imports inside the package.
  • The built-in dir() in a function was not having fully deterministic output list, now it does.

Summary

Overall, the amount of differences between CPython and Nuitka is heading towards zero. Also most of the improvements done in this release were very straightforward cleanups and not much work was required, mostly things are about cleanups and then it becomes easily right. The new type for the compiled generator expressions was simple to create, esp. as I could check what CPython does in its source code.

For optimization purposes, I decided that generator expressions and generator functions will be separate compiled types, as most of their behavior will not be shared. I believe optimizing generator expressions to run well is an important enough goal to warrant that they have their own implementation. Now that this is done, I will repeat it with generator functions.

Generator functions already work quite fine, but like generator expressions did before this release, they can leak references if not finished , and they don't have the throw() method, which seems very important to the correct operation of contextlib. So I will introduce a decicated type for these too, possibly in the next release.

Comments

Family Photo    Posted:


As promised I will post some photos of the family occasionally.

This is an artwork that I created with GIMP:

Photo of my family

It's a good family, and it's mine. :-)

Yours, Kay Hayen

Comments

Release Nuitka 0.2.1    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

The march goes on, this is another minor release with a bunch of substantial improvements:

Bug Fixes

  • Packages now also can be embedded with the --deep option too, before they could not be imported from the executable.
  • Inlined exec with their own future statements leaked these to the surrounding code.

Reduced Differences

  • The future print function import is now supported too.

Cleanups

  • Independence of the compiled function type. When I started it was merely PyCFunction and then a copy of it patched at run time, using increasingly less code from CPython. Now it's nothing at all anymore.
  • This lead to major cleanup of run time compiled function creation code, no more methoddefs, PyCObject holding context, etc.
  • PyLint was used to find the more important style issues and potential bugs, also helping to identify some dead code.

Summary

The major difference now is the lack of a throw method for generator functions. I will try to address that in a 0.2.2 release if possible. The plan is that the 0.2.x series will complete these tasks, and 0.3 could aim at some basic optimization finally.

Comments

Release Nuitka 0.2    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

Good day, this is a major step ahead, improvements everywhere.

Bug fixes

  • Migrated the Python parser from the deprecated and problematic compiler module to the ast module which fixes the d[a,] = b parser problem. A pity it was not available at the time I started, but the migration was relatively painless now.
  • I found and fixed wrong encoding of binary data into C++ literals. Now Nuitka uses C++0x raw strings, and these problems are gone.
  • The decoding of constants was done with the marshal module, but that appears to not deeply care enough about unicode encoding it seems. Using cPickle now, which seems less efficient, but is more correct.
  • Another difference is gone: The continue and break inside loops do no longer prevent the execution of finally blocks inside the loop.

Organizational

  • I now maintain the "README.txt" in org-mode, and intend to use it as the issue tracker, but I am still a beginner at that.

    Update

    Turned out I never master it, and used ReStructured Text instead.

  • There is a public git repository for you to track Nuitka releases. Make your changes and then git pull --rebase. If you encounter conflicts in things you consider useful, please submit the patches and a pull offer. When you make your clones of Nuitka public, use nuitka-unofficial or not the name Nuitka at all.

  • There is a now a mailing list available too.

Reduced Differences

  • Did you know you could write lambda : (yield something) and it gives you a lambda that creates a generator that produces that one value? Well, now Nuitka has support for lambda generator functions.
  • The from __future__ import division statement works as expected now, leading to some newly passing CPython tests.
  • Same for from __future__ import unicode_literals statement, these work as expected now, removing many differences in the CPython tests that use this already.

New Features

  • The Python binary provided and Nuitka.py are now capable of accepting parameters for the program executed, in order to make it even more of a drop-in replacement to python.
  • Inlining of exec statements with constant expressions. These are now compiled at compile time, not at run time anymore. I observed that an increasing number of CPython tests use exec to do things in isolation or to avoid warnings, and many more these tests will now be more effective. I intend to do the same with eval expressions too, probably in a minor release.

Summary

So give it a whirl. I consider it to be substantially better than before, and the list of differences to CPython is getting small enough, plus there is already a fair bit of polish to it. Just watch out that it needs gcc-4.5 or higher now.

Comments

New git repository to sync with Nuitka releases    Posted:


For you git fans out there I have just added a new method to download Nuitka from something I call the release git repository:

git clone http://git.nuitka.net/Nuitka.git

Then do your own modifications, and do:

git pull --rebase

each time there is a new release. You will be led through the merge process as usual. To reduce your differences, feel free to send me the patches you create with:

git format-patch <commit-id>

and I will incorporate useful stuff.

Please note

If you publish your own git repository, please be so kind and name it "Nuitka-unofficial" or similar, or else it might be mistaken with the real thing, and drop me a line about it, just so I know.

Kay Hayen

Comments

Minor Release Nuitka 0.1.1    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

I just have just updated Nuitka to version 0.1.1 which is a bug fix release to 0.1, which corrects many of the small things:

  • Updated the CPython test suite to 2.6.6rc and minimized much of existing differences in the course.
  • Compiles standalone executable that includes modules (with --deep option), but packages are not yet included successfully.
  • Reference leaks with exceptions are no more.
  • sys.exc_info() works now mostly as expected (it's not a stack of exceptions).
  • More readable generated code, better organisation of C++ template code.
  • Restored debug option --g++-only.

The biggest thing probably is the progress with exception tracebacks objects in exception handlers, which were not there before (always None). Having these in place will make it much more compatible. Also with manually raised exceptions and assertions, tracebacks will now be more correct to the line.

On a bad news, I discovered that the compiler module that I use to create the AST from Python source code, is not only deprecated, but also broken. I created the CPython bug about it, basically it cannot distinguish some code of the form d[1,] = None from d[1] = None. This will require a migration of the ast module, which should not be too challenging, but will take some time.

I am aiming at it for a 0.2 release. Generating wrong code (Nuitka sees d[1] = None in both cases) is a show blocker and needs a solution.

So, yeah. It's better, it's there, but still experimental. You will find its latest version here. Please try it out and let me know what you think in the comments section.

Comments

Releasing Nuitka to the World    Posted:


Obviously this is very exciting step for me. I am releasing Nuitka today. Finally. For a long time I knew I would, but actually doing it, is a different beast. Reaching my goals for release turned out to be less far away than I hope, so instead of end of August, I can already release it now.

Currently it's not more than 4% faster than CPython. No surprise there, if all you did, is removing the bytecode interpretation so far. It's not impressive at all. It's not even a reason to use it. But it's also only a start. Clearly, once I get into optimizing the code generation of Nuitka, it will only get better, and then probably in sometimes dramatic steps. But I see this as a long term goal.

I want to have infrastructure in the code place, before doing lots of possible optimizations that just make Nuitka unmaintainable. And I will want to have a look at what others did so far in the domain of type inference and how to apply that for my project.

I look forward to the reactions about getting this far. The supported language volume is amazing, and I have a set of nice tricks used. For example the way generator functions are done is a clever hack.

Where to go from here? Well, I guess, I am going to judge it by the feedback I receive. I personally see "constant propagation" as a laudable first low hanging fruit, that could be solved.

Consider this readable code on the module level:

meters_per_nautical_mile = 1852

def convertMetersToNauticalMiles( meters ):
    return meters / meters_per_nautical_mile
def convertNauticalMilesToMeters( miles ):
    return miles * meters_per_nautical_mile

Now imagine you are using this very frequently in code. Quickly you determine that the following will be much faster:

def convertMetersToNauticalMiles( meters ):
    return meters / 1852
def convertNauticalMilesToMeters( miles ):
    return miles * 1852

Still good? Well, probably next step you are going to inline the function calls entirely. For optimization, you are making your code less readable. I do not all appreciate that. My first goal is there to make the more readable code perform as well or better as the less readable variant.

But yes, lets see what happens. Oh, and you will find its latest version here.

Kay Hayen

Comments

Starting to blog, who and why?    Posted:


My name is Kay Hayen, I was born in 1972 in the very northern part of Germany very close to the North Sea.

I am privately a developer of Free Software, and currently preparing the release of my Python compiler. The name of the project will be "Nuitka" chosen to honor my Russian wife Anna. We are a happy family with 2 sons, born 2004 and 2007.

I am starting this blog mostly to report about the new compiler. I don't think I will share much about my private life here, but I sure will share a couple of photos with you occasionally. But mostly I will be excited to report on the compiler progress.

So far I didn't feel like I had anything to say, but the compiler is approaching its release later this month, and I will need a forum to talk about it.

Kay Hayen

Comments