Try Finally Python Quiz    Posted:


When working on my Python compiler Nuitka, I often come across ridiculous language details of the Python language, and turn these into quizzes, for which I finally added a dedicated quiz tag.

Anyway, who can predict, what these will do to you:

def f():
    try:
        return 1
    finally:
        return 2

Will it return 1 or 2 ?

def f():
    try:
        1/0
    finally:
        return 2

Will this raise an ZeroDivisionError or return 2 ?

def f():
    while 1
        try:
            continue
        finally:
            break

Is this an endless loop or does it return?

def f():
    while 1
        try:
            break
        finally:
            continue

What about that? This one holds an inconsistency.

No solutions yet this time.

Comments

Nuitka Release 0.5.1    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release brings corrections and major improvements to how standalone mode performs. Much of it was contributed via patches and bug reports.

Bug Fixes

  • There was a crash when using next on a non-iterable. Fixed in 0.5.0.1 already.
  • Module names with special characters not allowed in C identifiers were not fully supported. Issue#118. Fixed in 0.5.0.1 already.
  • Name mangling for classes with leading underscores was not removing them from resulting attribute names. This broke at __slots__ with private attributes for such classes. Issue#119. Fixed in 0.5.0.1 already.
  • Standalone on Windows might need "cp430" encoding. Issue#120. Fixed in 0.5.0.2 already.
  • Standalone mode didn't work with lxml.etree due to lack of hard coded dependencies. When a shared library imports things, Nuitka cannot detect it easily.
  • Wasn't working on MacOS 64 bits due to using Linux 64 bits specific code. Issue#123. Fixed in 0.5.0.2 already.
  • On MinGW the constants blob was not properly linked on some installations, this is now done differently (see below).

New Features

  • Memory usages are now traced with --show-progress allowing us to trace where things go wrong.

New Optimization

  • Standalone mode now includes standard library as bytecode by default. This is workaround scalability issues with many constants from many modules. Future releases are going to undo it.
  • On Windows the constants blob is now stored as a resource, avoiding compilation via C code for MSVC as well. MinGW was changed to use the same code.

New Tests

  • Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, PySide, and PyGtk packages.

Summary

This release is mostly an interim maintenance release for standalone. Major changes that provide optimization beyond that, termed "C-ish code generation" are delayed for future releases.

This release makes standalone practical which is an important point. Instead of hour long compilation, even for small programs, we are down to less than a minute.

The solution of the scalability issues with many constants from many modules will be top priority going forward. Since they are about how even single use constants are created all in one place, this will be easy, but as large changes are happening in "C-ish code generation", we are waiting for these to complete.

Comments

Not going to FOSDEM 2014    Posted:


So I submitted a talk for FOSDEM "Python has a compiler now". And it didn't get accepted. Quite a surprise to me, but likely mostly logical. These are the accepted talks and well, it surely didn't fit in, right.

My intent was to introduce Nuitka to the larger community. Not going to happen now, but maybe next year.

Clearly triggered by the rejection, I am questioning Europython 2014 in Berlin as a useful destination too. To me it seems, that code writing is the best way to create the community at this time anyway.

Comments

Nuitka Release 0.5.0    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release breaks interface compatibility, therefore the major version number change. Also "standalone mode" has seen significant improvements on both Windows, and Linux. Should work much better now.

But consider that this part of Nuitka is still in its infancy. As it is not the top priority of mine for Nuitka, which primarily is intended as an super compatible accelerator of Python, it will continue to evolve nearby.

There is also many new optimization based on structural improvements in the direction of actual SSA.

Bug Fixes

  • The "standalone mode" was not working on all Redhat, Fedora, and openSUSE platforms and gave warnings with older compilers. Fixed in 0.4.7.1 already.

  • The "standalone mode" was not including all useful encodings. Issue#116. Fixed in 0.4.7.2 already.

  • The "standalone mode" was defaulting to --python-flag=-S which disables the parsing of "site" module. That unfortunately made it necessary to reach some modules without modifying PYTHONPATH which conflicts with the "out-of-the-box" experience.

  • The "standalone mode" is now handling packages properly and generally working on Windows as well.

  • The syntax error of having an all catching except clause and then a more specific one wasn't causing a SyntaxError with Nuitka.

    try:
        something()
    except:
        somehandling():
    except TypeError:
        notallowed()
    
  • A corruption bug was identified, when re-raising exceptions, the top entry of the traceback was modified after usage. Depending on malloc this was potentially causing an endless loop when using it for output.

New Features

  • Windows: The "standalone" mode now properly detects used DLLs using Dependency Walker which it offers to download and extra for you.

    It is used as a replacement to ldd on Linux when building the binary, and as a replacement of strace on Linux when running the tests to check that nothing is loaded from the outside.

New Optimization

  • When iterating over list, set, this is now automatically lowered to tuples avoiding the mutable container types.

    So the following code is now equivalent:

    for x in [ a, b, c ]:
       ...
    
    # same as
    for x in (a, b, c):
       ...
    

    For constants, this is even more effective, because for mutable constants, no more is it necessary to make a copy.

  • Python2: The iteration of large range is now automatically lowered to xrange which is faster to loop over, and more memory efficient.

  • Added support for the xrange built-in.

  • The statement only expression optimization got generalized and now is capable of removing useless parts of operations, not only the whole thing when it has not side effects.

    [a,b]
    
    # same as
    a
    b
    

    This works for all container types.

    Another example is type built-in operation with single argument. When the result is not used, it need not be called.

    type(a)
    
    # same as
    a
    

    And another example is and is not have no effect of their own as well, therefore:

    a is b
    
    # same as
    a
    b
    
  • Added proper handling of conditional expression branches in SSA based optimization. So far these branches were ignored, which only acceptable for temporary variables as created by tree building, but not other variable types. This is preparatory for introducing SSA for local variables.

Organizational

  • The option --exe is now ignored and creating an executable is the default behavior of nuitka, a new option --module allows to produce extension modules.
  • The binary nuitka-python was removed, and is replaced by nuitka-run with now only implies --execute on top of what nuitka is.
  • Using dedicated Buildbot for continuous integration testing and release creation as well.
  • The Downloads now offers MSI files for Win64 as well.
  • Discontinued the support for cross compilation to Win32. That was too limited and the design choice is to have a running CPython instance of matching architecture at Nuitka compile time.

New Tests

  • Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, and PySide package.

Summary

The "executable by default" interface change improves on the already high ease of use. The new optimization do not give all that much in terms of numbers, but are all signs of structural improvements, and it is steadily approaching the point, where the really interesting stuff will happen.

The progress for standalone mode is of course significant. It is still not quite there yet, but it is making quick progress now. This will attract a lot of attention hopefully.

As for optimization, the focus for it has shifted to making exception handlers work optimal by default (publish the exception to sys.exc_info() and create traceback only when necessary) and be based on standard branches. Removing special handling of exception handlers, will be the next big step. This release includes some correctness fixes stemming from that work already.

Comments

Re: About Python 3    Posted:


In Alex Gaynor's post there is just about everything right. I still want to add my take on it.

Python3 is a fork of the Python community, that has left the user base largely behind. After developing features very conservatively for a long time (from __future__ import division and stuff), where features were first available but not active by default, that whole "Python 3000" thing came up.

Misguided, after having maintained CPython on a level of excellence for such a long time, there was a sense of "finally we get to make free choices". I can understand that. And having witnessed Gnome 2.0.0, and KDE 4.0.0, I even start to believe that there is some kind of underlying law. Something that makes people want to start over.

That "5 years plan" Alex mentioned has failed, and everybody knows that without solving the GIL within the next 5 years, i.e. another incompatible Python change, it will likely become obsolete.

In terms of Python2.8, call it that way. Or have a 3.5 that has print statement, and then all kinds of incompatible changes to Python3 to make it behave sane bytes(7) needs to do what str(7) once it. That would be about from __past__ import .... I suppose.

I also had another idea, having a "python2" built-in module that carries a CPython2 interpreter and proxy objects for instances, where they talk to another with incompatible types (new bytes/unicode to old str).

Due to my work on that Python compiler that should have existed 20 years ago, I am familiar enough, to actually be able and create that kind of CPython2/CPython3 hybrid.

But the main problems with all of that are:

  1. CPython is developed by volunteers, and as such, they are neither obliged to do what we need, nor will they bend to any amount of posting. We need to get our acts together or it's not happening, and we are people who care about Python relevancy.

    There may not be nearly enough overlap of people capable and caring. And that would be a tremendous amount of work. Highly speculative, in the sense, that it may not see fruits.

  2. And then, when you got that sorted out, expect a hostile reaction from the core developers and/or PSF. Whatever it is, that you develop, you won't be able to call it Python 2.8, as they are going to hate it.

  3. And finally, as a course of action, that may even bring the complete downfall of Python as a community. The controversy associated is not for everybody, and we don't need a libreoffice/openoffice situation in Python, do we.

It's a pity, because clearly, for Nuitka there would be a bunch of patches, that I would like to make, that would make my life far easier, and even interpreted Python faster.

Comments

Nuitka Standalone Mode is Work in Progress    Posted:


Many of you who turn to my easy to use, highly compatible Python compiler Nuitka, do this mostly because they seek to solve the deployment problem that Python suffers from.

Be this, because you want to use a newer Python2.7 on RHEL5 without installing anything at all there. Or because Windows is difficult to tackle otherwise.

For the longest time, Nuitka had not offered anything in this domain, focusing solely on being an accelerator. Lately, I have taken up the challenge and polished initial solutions submitted by contributors.

This first showed up in the 0.4.7 release, but turned out relatively weak. While first examples were working on Linux, it was not working at all on Windows (anymore). And basically there was a huge lack of tests.

Actually I didn't mean for it to be released with that feature, but as parts of it seemed to work, I did so. But truth to be sad, that feature is not nearly as polished in that release as you would like it to.

In current development releases, of what is going to become 0.5.0 really soon now, it's much better already. More things actually work. But it appears, there will be more ground to cover, and this is a lot of stuff to sort out.

So, this is mostly about asking you two things. Give that development release a try and report issues you have with it. And help me.

And have patience. I am developing Nuitka as an accelerator on a "no known bugs" basis. That means, once I know of a bug, I will fix it. OK, some issues in fact take longer, but then it really is not important at all, but difficult at the time. For standalone mode, I can't do it that way, or I would have to neglect the acceleration parts, which I totally don't want to do.

Because while you maybe are only interested in a packaging solution, many others would like to have that orders of magnitude speedup that I have been aiming for and that feels near now. This is about making Python a viable language for more uses than it currently is.

So why do it in the first place. For one, I am hoping that it helps people to not turn away from Python. And second, and more important, I am hoping that by making it more useful, more people will join me. (Oh, and thirdly, it's also a nice puzzle to solve. I seem to enjoy that.)

Ultimately both modes will be needed, standalone, and acceleration. And it seems like I am working to provide both. For standalone, more often, than seeking to avoid bugs as far as possible, I am going to rely on your participation.

So join Nuitka. Now. Download from here. Join the mailing list. And help if you can. And yes you can.

Comments

Nuitka Release 0.4.7    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes important new features, lots of polishing cleanups, and some important performance improvements as well.

Bug Fixes

  • The RPM packages didn't build due to missing inline copy of Scons. Fixed in 0.4.6.1 already.
  • The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already.
  • The Windows installer was not including Scons. Fixed in 0.4.6.3 already.
  • Windows: The immediate execution as performed by nuitka --execute was not preserving the exit code. Issue#26.
  • Python3.3: Packages without __init.py__ were not properly embedding the name-space package as well.
  • Python3: Fix, modules and packages didn't add themselves to sys.modules which they should, happened only for programs.
  • Python3.3: Packages should set __package to their own name, not the one of their parents.
  • Python3.3: The __qualname__ of nested classes was corrected.
  • For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons. Issue#115.

New Features

  • The "standalone" mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already.

    To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized.

  • New option --python-flag allows to specify flags to the compiler that the "python" binary normally would. So far -S and -v are supported, with sane aliases no_site and trace_imports.

    The recommended use of --python-flag=-S is to avoid dependency creep in standalone mode compilations, because the site module often imports many useless things that often don't apply to target systems.

New Optimization

  • Faster frame stack handling for functions without try/except (or try/finally in Python3). This gives a speed boost to "PyStone" of ca. 2.5% overall.
  • Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to "PyStone" of ca. 0.5% overall.
  • Python2: Much quicker calls of __getattr__ and __setattr__ as this is now using the quicker call method avoiding temporary tuples.
  • Don't treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies.
  • Create unicode constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up.
  • Directly create int and long constants outside of 2**31 and 2**32-1, but only limited according to actual platform values. Affects only program start-up.
  • When creating set values, no longer use a temporary tuple value, but use a properly generated helper functions instead. This makes creating sets much faster.
  • Directly create set constants instead of un-streaming them. Affects only program start-up.
  • For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement.
  • Module variables are now accessed even faster, the gain for "PyStone" is only 0.1% and mostly the result of leaner code.

Organizational

  • The "standalone mode" code (formerly known as "portable mode" has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement.
  • Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported.
  • Added package for openSUSE 13.1 for download.
  • Nuitka is now part of Arch and can be installed with pacman -S nuitka.
  • Using dedicated Buildbot for continuous integration testing. Not yet public.
  • Windows: In order to speed up repeated compilation on a platform without ccache, added Scons level caching in the build directory.
  • Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ccache and Scons as well.

Tests

  • The programs tests cases now fail if module or directory recursion is not working, being executed in another directory.
  • Added test runner for packages, with initial test case for package with recursion and sub-packages.
  • Made some test cases more strict by reducing PYTHONPATH provision.
  • Detect use of extra flags in tests that don't get consumed avoiding ineffective flags.
  • Use --execute on Windows as well, the issue that prevented it has been solved after all.

Cleanups

  • The generated code uses const_, var_, par_ prefixes in the generated code and centralized the decision about these into single place.
  • Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation.
  • The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious.
  • Massive general cleanups, many of which came from new contributor Juan Carlos Paco.
  • Moved standalone and freezer related codes to dedicated package nuitka.freezer to not pollute the nuitka package name space.
  • The code generation use variable identifiers and their accesses was cleaned up.
  • Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them.
  • Moved main program, function object, set related code generation to dedicated modules.

Summary

This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation.

The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead.

Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future.

Comments

Nuitka in Arch Linux    Posted:


Got this message:

Just thought you wanted to know that Nuitka is now also an official
Arch Linux package and can be installed simply with:

pacman -S nuitka

So, that is another sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Many thanks to Alexander Rødseth and Panagiotis Mavrogiorgos for maintaining the Nuitka packages inside Arch and on AUR.

Comments

Nuitka and GUIs    Posted:


Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

Nuitka is a typical compiler, just for Python, and aims to be integrating with build tools, and as such is so much a command line tool, it hurts.

Yet, there are GUIs. Yes, plural, there are even two.

Many of you are maybe using Ninja IDE already. I have developed most of early Nuitka in vim, and later, mostly because of Nuitka, switched back to Emacs. I had switched from Emacs at one point 12 years ago, but then went back a couple of years ago.

But recently +Juan Carlos convinced me to give Ninja a try. Because not only is he the creator the Nuitka GUI, but also the creator of the Ninja IDE plugin for Nuitka.

So on Windows, I now use Ninja, and I like it, a lot.

I must admit, I am not a heavy user of both Nuitka GUIs, rather being a command line guy I suppose, but that is just great. I would like to thank Juan Carlos for these contributions.

They are one sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Comments

Nuitka Release 0.4.6    Posted:


This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups.

Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot.

Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues.

And then there is performance. As can be seen in the performance graph, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types.

And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix.

Bug Fixes

  • Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. Issue#106. Fixed in 0.4.5.1 already.

  • Support Python being installed to a path that contains spaces. Issue#106. Fixed in 0.4.5.2 already.

  • Windows: User provided constants larger than 65k didn't work with MSVC. Issue#108. Fixed in 0.4.5.3 already.

  • Windows: The option --windows-disable-console was not effective with MSVC. Issue#107. Fixed in 0.4.5.3 already.

  • Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn't honor the target architecture. Issue#99. Fixed in 0.4.5.3 already.

  • When creating Python modules, these were marked as executable ("x" bit), which they are of course not. Fixed in 0.4.5.3 already.

  • Python3.3: On architectures where Py_ssize_t is not the same as long this could lead to errors. Fixed in 0.4.5.3 already.

  • Code that was using nested mutable constants and changed the nested ones was not executing correctly. Issue#112.

  • Python2: Due to list contractions being re-formulated as functions, del was rejected for the variables assigned in the contraction. Issue#111.

    [ expr(x) for x in iterable() ]
    
    del x # Should work, was gave an unjustified SyntaxError.
    

New Features

  • Compiled types when used in Python comparison now work. Code like this will work:

    def f():
       pass
    
    assert type(f) == types.FunctionType
    

    This of course also works for in operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka.

  • Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH.

  • Windows: Now options --mingw to force compilation with MinGW.

New Optimization

  • Rich comparisons (==, <, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall.
  • Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython.
  • The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs.

Cleanups

  • Replaced "value friend" based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of or and and expressions.
  • Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using "variable usage" profiles for this anymore.
  • The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows.
  • Stop using os.system and subprocess.call(..., shell = True) as it is not really portable at all, use subprocess.call(..., shell = False) instead.
  • As usual lots of cleanups related to line length issues and PyLint.

Organizational

  • Added support for Gentoo Linux.
  • Added support for self compiled Python versions with and without debug enabled. Issue#110
  • Added use of Nuitka fonts for headers in manuals.
  • Does not install inline copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages.

Summary

While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later.

To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever.

So give this a roll, it's worth it. And feel free to join the mailing list or make a donation to support Nuitka.

Comments