Re: About Python 3

In Alex Gaynor's post there is just about everything right. I still want to add my take on it.

Python3 is a fork of the Python community, that has left the user base largely behind. After developing features very conservatively for a long time (from __future__ import division and stuff), where features were first available but not active by default, that whole "Python 3000" thing came up.

Misguided, after having maintained CPython on a level of excellence for such a long time, there was a sense of "finally we get to make free choices". I can understand that. And having witnessed Gnome 2.0.0, and KDE 4.0.0, I even start to believe that there is some kind of underlying law. Something that makes people want to start over.

That "5 years plan" Alex mentioned has failed, and everybody knows that without solving the GIL within the next 5 years, i.e. another incompatible Python change, it will likely become obsolete.

In terms of Python2.8, call it that way. Or have a 3.5 that has print statement, and then all kinds of incompatible changes to Python3 to make it behave sane bytes(7) needs to do what str(7) once it. That would be about from __past__ import .... I suppose.

I also had another idea, having a "python2" built-in module that carries a CPython2 interpreter and proxy objects for instances, where they talk to another with incompatible types (new bytes/unicode to old str).

Due to my work on that Python compiler that should have existed 20 years ago, I am familiar enough, to actually be able and create that kind of CPython2/CPython3 hybrid.

But the main problems with all of that are:

  1. CPython is developed by volunteers, and as such, they are neither obliged to do what we need, nor will they bend to any amount of posting. We need to get our acts together or it's not happening, and we are people who care about Python relevancy.

    There may not be nearly enough overlap of people capable and caring. And that would be a tremendous amount of work. Highly speculative, in the sense, that it may not see fruits.

  2. And then, when you got that sorted out, expect a hostile reaction from the core developers and/or PSF. Whatever it is, that you develop, you won't be able to call it Python 2.8, as they are going to hate it.

  3. And finally, as a course of action, that may even bring the complete downfall of Python as a community. The controversy associated is not for everybody, and we don't need a libreoffice/openoffice situation in Python, do we.

It's a pity, because clearly, for Nuitka there would be a bunch of patches, that I would like to make, that would make my life far easier, and even interpreted Python faster.

Nuitka Standalone Mode is Work in Progress

Many of you who turn to my easy to use, highly compatible Python compiler Nuitka, do this mostly because they seek to solve the deployment problem that Python suffers from.

Be this, because you want to use a newer Python2.7 on RHEL5 without installing anything at all there. Or because Windows is difficult to tackle otherwise.

For the longest time, Nuitka had not offered anything in this domain, focusing solely on being an accelerator. Lately, I have taken up the challenge and polished initial solutions submitted by contributors.

This first showed up in the 0.4.7 release, but turned out relatively weak. While first examples were working on Linux, it was not working at all on Windows (anymore). And basically there was a huge lack of tests.

Actually I didn't mean for it to be released with that feature, but as parts of it seemed to work, I did so. But truth to be sad, that feature is not nearly as polished in that release as you would like it to.

In current development releases, of what is going to become 0.5.0 really soon now, it's much better already. More things actually work. But it appears, there will be more ground to cover, and this is a lot of stuff to sort out.

So, this is mostly about asking you two things. Give that development release a try and report issues you have with it. And help me.

And have patience. I am developing Nuitka as an accelerator on a "no known bugs" basis. That means, once I know of a bug, I will fix it. OK, some issues in fact take longer, but then it really is not important at all, but difficult at the time. For standalone mode, I can't do it that way, or I would have to neglect the acceleration parts, which I totally don't want to do.

Because while you maybe are only interested in a packaging solution, many others would like to have that orders of magnitude speedup that I have been aiming for and that feels near now. This is about making Python a viable language for more uses than it currently is.

So why do it in the first place. For one, I am hoping that it helps people to not turn away from Python. And second, and more important, I am hoping that by making it more useful, more people will join me. (Oh, and thirdly, it's also a nice puzzle to solve. I seem to enjoy that.)

Ultimately both modes will be needed, standalone, and acceleration. And it seems like I am working to provide both. For standalone, more often, than seeking to avoid bugs as far as possible, I am going to rely on your participation.

So join Nuitka. Now. Download from here. Join the mailing list. And help if you can. And yes you can.

Nuitka Release 0.4.7

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes important new features, lots of polishing cleanups, and some important performance improvements as well.

Bug Fixes

  • The RPM packages didn't build due to missing in-line copy of Scons. Fixed in 0.4.6.1 already.
  • The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already.
  • The Windows installer was not including Scons. Fixed in 0.4.6.3 already.
  • Windows: The immediate execution as performed by nuitka --execute was not preserving the exit code. Issue#26.
  • Python3.3: Packages without __init.py__ were not properly embedding the name-space package as well.
  • Python3: Fix, modules and packages didn't add themselves to sys.modules which they should, happened only for programs.
  • Python3.3: Packages should set __package to their own name, not the one of their parents.
  • Python3.3: The __qualname__ of nested classes was corrected.
  • For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons. Issue#115.

New Features

  • The "standalone" mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already.

    To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized.

  • New option --python-flag allows to specify flags to the compiler that the "python" binary normally would. So far -S and -v are supported, with sane aliases no_site and trace_imports.

    The recommended use of --python-flag=-S is to avoid dependency creep in standalone mode compilations, because the site module often imports many useless things that often don't apply to target systems.

New Optimization

  • Faster frame stack handling for functions without try/except (or try/finally in Python3). This gives a speed boost to "PyStone" of ca. 2.5% overall.
  • Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to "PyStone" of ca. 0.5% overall.
  • Python2: Much quicker calls of __getattr__ and __setattr__ as this is now using the quicker call method avoiding temporary tuples.
  • Don't treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies.
  • Create unicode constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up.
  • Directly create int and long constants outside of 2**31 and 2**32-1, but only limited according to actual platform values. Affects only program start-up.
  • When creating set values, no longer use a temporary tuple value, but use a properly generated helper functions instead. This makes creating sets much faster.
  • Directly create set constants instead of un-streaming them. Affects only program start-up.
  • For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement.
  • Module variables are now accessed even faster, the gain for "PyStone" is only 0.1% and mostly the result of leaner code.

Organizational

  • The "standalone mode" code (formerly known as "portable mode" has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement.
  • Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported.
  • Added package for openSUSE 13.1 for download.
  • Nuitka is now part of Arch and can be installed with pacman -S nuitka.
  • Using dedicated Buildbot for continuous integration testing. Not yet public.
  • Windows: In order to speed up repeated compilation on a platform without ccache, added Scons level caching in the build directory.
  • Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ccache and Scons as well.

Tests

  • The programs tests cases now fail if module or directory recursion is not working, being executed in another directory.
  • Added test runner for packages, with initial test case for package with recursion and sub-packages.
  • Made some test cases more strict by reducing PYTHONPATH provision.
  • Detect use of extra flags in tests that don't get consumed avoiding ineffective flags.
  • Use --execute on Windows as well, the issue that prevented it has been solved after all.

Cleanups

  • The generated code uses const_, var_, par_ prefixes in the generated code and centralized the decision about these into single place.
  • Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation.
  • The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious.
  • Massive general cleanups, many of which came from new contributor Juan Carlos Paco.
  • Moved standalone and freezer related codes to dedicated package nuitka.freezer to not pollute the nuitka package name space.
  • The code generation use variable identifiers and their accesses was cleaned up.
  • Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them.
  • Moved main program, function object, set related code generation to dedicated modules.

Summary

This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation.

The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead.

Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future.

Nuitka in Arch Linux

Got this message:

Just thought you wanted to know that Nuitka is now also an official
Arch Linux package and can be installed simply with:

pacman -S nuitka

So, that is another sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Many thanks to Alexander Rødseth and Panagiotis Mavrogiorgos for maintaining the Nuitka packages inside Arch and on AUR.

Nuitka and GUIs

Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

Nuitka is a typical compiler, just for Python, and aims to be integrating with build tools, and as such is so much a command line tool, it hurts.

Yet, there are GUIs. Yes, plural, there are even two.

Many of you are maybe using Ninja IDE already. I have developed most of early Nuitka in vim, and later, mostly because of Nuitka, switched back to Emacs. I had switched from Emacs at one point 12 years ago, but then went back a couple of years ago.

But recently +Juan Carlos convinced me to give Ninja a try. Because not only is he the creator the Nuitka GUI, but also the creator of the Ninja IDE plugin for Nuitka.

So on Windows, I now use Ninja, and I like it, a lot.

I must admit, I am not a heavy user of both Nuitka GUIs, rather being a command line guy I suppose, but that is just great. I would like to thank Juan Carlos for these contributions.

They are one sign of Nuitka getting more external recognition. It is showing that the eco system around Nuitka is indeed warming up.

Nuitka Release 0.4.6

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups.

Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot.

Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues.

And then there is performance. As can be seen in the performance graph, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types.

And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix.

Bug Fixes

  • Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. Issue#106. Fixed in 0.4.5.1 already.

  • Support Python being installed to a path that contains spaces. Issue#106. Fixed in 0.4.5.2 already.

  • Windows: User provided constants larger than 65k didn't work with MSVC. Issue#108. Fixed in 0.4.5.3 already.

  • Windows: The option --windows-disable-console was not effective with MSVC. Issue#107. Fixed in 0.4.5.3 already.

  • Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn't honor the target architecture. Issue#99. Fixed in 0.4.5.3 already.

  • When creating Python modules, these were marked as executable ("x" bit), which they are of course not. Fixed in 0.4.5.3 already.

  • Python3.3: On architectures where Py_ssize_t is not the same as long this could lead to errors. Fixed in 0.4.5.3 already.

  • Code that was using nested mutable constants and changed the nested ones was not executing correctly. Issue#112.

  • Python2: Due to list contractions being re-formulated as functions, del was rejected for the variables assigned in the contraction. Issue#111.

    [ expr(x) for x in iterable() ]
    
    del x # Should work, was gave an unjustified SyntaxError.
    

New Features

  • Compiled types when used in Python comparison now work. Code like this will work:

    def f():
       pass
    
    assert type(f) == types.FunctionType
    

    This of course also works for in operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka.

  • Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH.

  • Windows: Now options --mingw to force compilation with MinGW.

New Optimization

  • Rich comparisons (==, <, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall.
  • Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython.
  • The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs.

Cleanups

  • Replaced "value friend" based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of or and and expressions.
  • Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using "variable usage" profiles for this anymore.
  • The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows.
  • Stop using os.system and subprocess.call(..., shell = True) as it is not really portable at all, use subprocess.call(..., shell = False) instead.
  • As usual lots of cleanups related to line length issues and PyLint.

Organizational

  • Added support for Gentoo Linux.
  • Added support for self compiled Python versions with and without debug enabled. Issue#110
  • Added use of Nuitka fonts for headers in manuals.
  • Does not install in-line copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages.

Summary

While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later.

To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever.

So give this a roll, it's worth it. And feel free to join the mailing list or make a donation to support Nuitka.

Nuitka, Python research and Physics

Once upon a time ago physics had multiple forces. Each with very distinct traits. These were the early days. Then as time went on people started to realize that weak nuclear force and electromagnetic force were the same sort. That was after discovering that electricity and magnetism were the same, and that is the pattern. It's called symmetry, equivalence, and so on.

The Nuitka project is effectively a Python research that goes through the same steps. There are e.g. currently local variables, closure variables (shared from containing function), module variables, temp holder, temp keeper, and temp variables.

Often when I added one more this was driven by code generation needs or deficiencies at the time. These are of course way too many, and all are good at one thing only.

So the main task Nuitka is faced with now to e.g. generalize that "temp keeper" variables are temporary variables local to one statement only, surely to be released afterwards, and make that particular feature e.g. work for all variables across the board. Right now, code generation uses "cheats" where it puts a C++ block and local declarations for "temp keeper" variables.

It would be good, if it could be a try with del statement on the "temp keeper variable" in its finally part and yet, get the same code generated. Difficult, yes, but not impossible, and definitely what I am aiming at.

So that is what Nuitka will be dealing with in the next releases. Once done, maybe there will be a "gravity", i.e. the one thing not yet harmonized, but for good reasons. Potentially limited due to lack of understanding, potentially because there is a good reason. And so not all we be unified or maybe e.g. module variables will be a bit more special than local variables, although modules are just functions with variables writable everywhere.

Good stuff is coming, hold on. Join the project, or support it financially if you wish.

Nuitka still is too much of a one man show. Should I be asked to name one single weakness of it - that would be it.

Nuitka Release 0.4.5

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release incorporates very many bug fixes, most of which were already part of hot fixes, usability improvements, documentation improvements, new logo, simpler Python3 on Windows, warnings for recursion options, and so on. So it's mostly a consolidation release.

Bug Fixes

  • When targeting Python 3.x, Nuitka was using "python" to run Scons to run it under Python 2.x, which is not good enough on systems, where that is already Python3. Improved to only do the guessing where necessary (i.e. when using the in-line copy of Scons) and then to prefer "python2". Issue#95. Fixed in 0.4.4.1 already.
  • When using Nuitka created binaries inside a "virtualenv", created programs would instantly crash. The attempt to load and patch inspect module was not making sure that site module was already imported, but inside the "virtualenv", it cannot be found unless. Issue#96. Fixed in 0.4.4.1 already.
  • The option --recurse-directory to include plugin directories was broken. Issue#97. Fixed in 0.4.4.2 already.
  • Python3: Files with "BOM" marker causes the compiler to crash. Issue#98. Fixed in 0.4.4.2 already.
  • Windows: The generated code for try/return/finally was working with gcc (and therefore MinGW), but not with MSVC, causing crashes. Issue#102. Fixed in 0.4.4.2 already.
  • The option --recurse-all did not recurse to package __init__.py files in case from x.y import z syntax was used. Issue#100. Fixed in 0.4.4.2 already.
  • Python3 on MacOS: Corrected link time error. Fixed in 0.4.4.2 already.
  • Python3.3 on Windows: Fixed crash with too many arguments to a kwonly argument using function. Fixed in 0.4.4.2 already.
  • Python3.3 on Windows: Using "yield from" resulted in a link time error. Fixed in 0.4.4.2 already.
  • Windows: Added back XML manifest, found a case where it is needed to prevent clashes with binary modules.
  • Windows: Generators only worked in the main Python threads. Some unusual threading modules therefore failed.
  • Using sys.prefix to find the Python installation instead of hard coded paths. Issue#103.

New Features

  • Windows: Python3 finds Python2 installation to run Scons automatically now.

    Nuitka itself runs under Python3 just fine, but in order to build the generated C++ code into binaries, it uses Scons which still needs Python2.

    Nuitka will now find the Python2 installation searching Windows registry instead of requiring hard coded paths.

  • Windows: Python2 and Python3 find their headers now even if Python is not installed to specific paths.

    The installation path now is passed on to Scons which then uses it.

  • Better error checking for --recurse-to and --recurse-not-to arguments, tell the user not to use directory paths.

  • Added a warning for --recurse-to arguments that end up having no effect to the final result.

Cleanups

  • Import mechanism got cleaned up, stopped using "PyImport_ExtendInittab". It does not handle packages, and the sys.meta_path based importer is now well proven.
  • Moved some of the constraint collection code mess into proper places. It still remains a mess.

Organizational

  • Added LICENSE.txt file with Apache License 2.0 text to make it more immediately obvious which license Nuitka is under.
  • Added section about Nuitka license to the "User Manual".
  • Added Nuitka Logo to the distribution.
  • Use Nuitka Logo as the bitmap in the Windows installer.
  • Use Nuitka Logo in the documentation ("User Manual" and "Developer Manual").
  • Enhanced documentation to number page numbers starting after table of contents, removed header/footer from cover pages.

Summary

This release is mostly the result of improvements made based on the surge of users after Europython 2013. Some people went to extents and reported their experience very detailed, and so I could aim at making e.g. their misconceptions about how recursion options work, more obvious through warnings and errors.

This release is not addressing performance improvements. The next release will be able to focus on that. I am taking my claim of full compatibility very serious, so any time it's broken, it's the highest priority to restore it.

My Europython 2013 report

Back from Europython 2013 in Florence for a while now. I had a presentation about my Python compiler Nuitka. You can watch the Video on Youtube. I believe it's absolutely worth your time. I was not doing a lot of "why" at all now, only "what", and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good and at ease. The state presented is very good and progress - there was a lot, so that was great. The slides of the talk may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight then, and also much more excited, at least initially. Background: I was surprised then that Guido van Rossum was attending the talk, plus I was kind of not in the state of this year, the confidence that people should believe, that things will work after that, was not yet there. I knew it, but standing in front of a croud and saying, look I am attemting what you consider impossible, or even have failed at, is not easy:

There are also the slides of that talk which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven things. Let's see what this year does to it. I would hope for real type inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out of the box experience, and how it just works. That's right. I put a lot of focus on stability, correctness, and ease of use. In fact, since my return, I have mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays), it's make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a lot of users, and not as many contributions or donations as I would hope for, but that's kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important exception are scientific people, doing number crunching. And I made some interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which were unfortunately regressions of 0.4.4, and these got fixed in two hotfix releases. Currently 0.4.4.2 is released, and contains a huge amount of fixes, mostly stuff found by the surge new users.

But I deserved that, having said that I didn't have to do hotfixes for a while now.

Florence

Such a lovely place, and so lovely people. It's a tourist place, true, but it's very open minded. The food is extremely good. And going from the conference hotel straight over to the river to meet up and have a few beers was very comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it's moving on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large amount of non-satisfaction, was that the community is fed up with the leadership. Questioned about lack of Python3 adoption, a PSF guy in his presentation said they were 3 years into a 5 years plan, and got interrupted by laughter.

It may be about time, somebody actually gets up, removes the GIL and has a Python2.8 that is compatible. Obviously that's much more useful. It could be a Python3.x with compatibility names of modules, print statement, and dict.iteritems added, probably also making some of the unicode mess more harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython top versions, to add things, that would allow Nuitka to work better with it, that could be a staging ground, where such changes are nurtured. On the other hand, Nuitka is too irrelevant yet to drive this, and it would detract me away from it.

Gains

So Nuitka got a lot more exposure. And one guy watching the talk even created a really nice logo for it. I will make it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And reinforcement that what I am doing is actually useful to some people already.

Donations

My travel to Florence was in part funded by donations. Thanks a lot to all of you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there via train.

Conclusion

The Europython 2012 and 2013 both are among the times of my life.