Nuitka Blog - Posted in 2014

Nuitka Release 0.5.6

2014-12-19T07:54:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings bug fixes, important new optimization, newly supported platforms, and important compatibility improvements. Progress on all fronts.

Bug Fixes

Closure taking of global variables in member functions of classes that had a class variable of the same name was binding to the class variable as opposed to the module variable.
Overwriting compiled function’s __doc__ attribute more than once could corrupt the old value, leading to crashes. Fixed in 0.5.5.2 already.

Compatibility Python2: The exec statement execfile were changing locals() was given as an argument.

def function():
    a = 1

    exec code in locals()  # Cannot change local "a".
    exec code in None  # Can change local "a"
    exec code

Previously Nuitka treated all 3 variants the same.

Compatibility: Empty branches with a condition were reduced to only the condition, but they need in fact to also check the truth value:
```
if condition:
    pass
# must be treated as
bool(condition)
# and not (bug)
condition
```
Detection of Windows virtualenv was not working properly. Fixed in 0.5.5.2 already.
Large enough constants structures are now unstreamed via marshal module, avoiding large codes being generated with no point. Fixed in 0.5.5.2 already.
Windows: Pressing CTRL-C gave two stack traces, one from the re-execution of Nuitka which was rather pointless. Fixed in 0.5.5.1 already.
Windows: Searching for virtualenv environments didn’t terminate in all cases. Fixed in 0.5.5.1 already.
During installation from PyPI with Python3 versions, there were errors given for the Python2 only scons files. Fixed in 0.5.5.3 already.
Fix, the arguments of yield from expressions could be leaked.

Fix, closure taking of a class variable could have in a sub class where the module variable was meant.

var = 1


class C:
    var = 2

    class D:
        def f():
            # was C.var, now correctly addressed top level var
            return var

Fix, setting CXX environment variable because the installed gcc has too low version, wasn’t affecting the version check at all.
Fix, on Debian/Ubuntu with hardening-wrapper installed the version check was always failing, because these report a shortened version number to Scons.

Optimization

Local variables that must be assigned also have no side effects, making use of SSA. This allows for a host of optimization to be applied to them as well, often yielding simpler access/assign code, and discovering in more cases that frames are not necessary.
Micro optimization to dict built-in for simpler code generation.

Organizational

Added support for ARM “hard float” architecture.
Added package for Ubuntu 14.10 for download.
Added package for openSUSE 13.2 for download.
Donations were used to buy a Cubox-i4 Pro. It got Debian Jessie installed on it, and will be used to run an even larger amount of tests.
Made it more clear in the user documentation that the .exe suffix is used for all platforms, and why.
Generally updated information in User Manual and Developer Manual about the optimization status.
Using Nikola 7.1 with external filters instead of our own, outdated branch for the web site.

Cleanups

PyLint clean for the first time ever. We now have a Buildbot driven test that this stays that way.
Massive indentation cleanup of keyword argument calls. We have a rule to align the keywords, but as this was done manually, it could easily get out of touch. Now with a auto-format tool based on RedBaron, it’s correct. Also, spacing around arguments is now automatically corrected. More to come.
For exec statements, the coping back to local variables is now an explicit node in the tree, leader to cleaner code generation, as it now uses normal variable assignment code generation.
The MaybeLocalVariables became explicit about which variable they might be, and contribute to its SSA trace as well, which was incomplete before.
Removed some cases of code duplication that were marked as TODO items. This often resulted in cleanups.
Do not use replaceWith on child nodes, that potentially were re-used during their computation.

Summary

The release is mainly the result of consolidation work. While the previous release contained many important enhancements, this is another important step towards full SSA, closing one loop whole (class variables and exec functions), as well as applying it to local variables, largely extending its use.

The amount of cleanups is tremendous, in huge part due to infrastructure problems that prevented release repeatedly. This reduces the technological debt very much.

More importantly, it would appear that now eliminating local and temporary variables that are not necessary is only a small step away. But as usual, while this may be easy to implement now, it will uncover more bugs in existing code, that we need to address before we continue.

Nuitka Release 0.5.5

2014-10-05T23:15:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is finally making full use of SSA analysis knowledge for code generation, leading to many enhancements over previous releases.

It also adds support for Python3.4, which has been longer in the making, due to many rather subtle issues. In fact, even more work will be needed to fully solve remaining minor issues, but these should affect no real code.

And then there is much improved support for using standalone mode together with virtualenv. This combination was not previously supported, but should work now.

New Features

Added support for Python3.4

This means support for clear method of frames to close generators, dynamic __qualname__, affected by global statements, tuples as yield from arguments, improved error messages, additional checks, and many more detail changes.

Optimization

Using SSA knowledge, local variable assignments now no longer need to check if they need to release previous values, they know definitely for the most cases.
```
def f():
    a = 1  # This used to check if old value of "a" needs a release
    ...
```
Using SSA knowledge, local variable references now no longer need to check for raising exceptions, let alone produce exceptions for cases, where that cannot be.
```
def f():
    a = 1
    return a  # This used to check if "a" is assigned
```
Using SSA knowledge, local variable references now are known if they can raise the UnboundLocalError exception or not. This allows to eliminate frame usages for many cases. Including the above example.
Using less memory for keeping variable information.
Also using less memory for constant nodes.

Bug Fixes

The standalone freezing code was reading Python source as UTF-8 and not using the code that handles the Python encoding properly. On some platforms there are files in standard library that are not encoded like that.
The fiber implementation for Linux amd64 was not working with glibc from RHEL 5. Fixed to use now multiple int to pass pointers as necessary. Also use uintptr_t instead of intprt_t to transport pointers, which may be more optimal.
Line numbers for exceptions were corrupted by with statements due to setting line numbers even for statements marked as internal.
Partial support for win32com by adding support for its hidden __path__ change.
Python3: Finally figured out proper chaining of exceptions, given proper context messages for exception raised during the handling of exceptions.
Corrected C++ memory leak for each closure variable taken, each time a function object was created.
Python3: Raising exceptions with tracebacks already attached, wasn’t using always them, but producing new ones instead.
Some constants could cause errors, as they cannot be handled with the marshal module as expected, e.g. (int,).
Standalone: Make sure to propagate sys.path to the Python instance used to check for standard library import dependencies. This is important for virtualenv environments, which need site.py to set the path, which is not executed in that mode.
Windows: Added support for different path layout there, so using virtualenv should work there too.
The code object flag “optimized” (fast locals as opposed to locals dictionary) for functions was set wrongly to value for the parent, but for frames inside it, one with the correct value. This lead to more code objects than necessary and false co_flags values attached to the function.
Options passed to nuitka-python could get lost.
```
nuitka-python program.py argument1 argument2 ...
```
The above is supposed to compile program.py, execute it immediately and pass the arguments to it. But when Nuitka decides to restart itself, it would forget these options. It does so to e.g. disable hash randomization as it would affect code generation.
Raising tuples exception as exceptions was not compatible (Python2) or reference leaking (Python3).

Tests

Running 2to3 is now avoided for tests that are already running on both Python2 and Python3.
Made XML based optimization tests work with Python3 too. Previously these were only working on Python2.
Added support for ignoring messages that come from linking against self compiled Pythons.
Added test case for threaded generators that tortures the fiber layer a bit and exposed issues on RHEL 5.
Made reference count test of compiled functions generic. No more code duplication, and automatic detection of shared stuff. Also a more clear interface for disabling test cases.
Added Python2 specific reference counting tests, so the other cases can be executed with Python3 directly, making debugging them less tedious.

Cleanups

Really important removal of “variable references”. They didn’t solve any problem anymore, but their complexity was not helpful either. This allowed to make SSA usable finally, and removed a lot of code.
Removed special code generation for parameter variables, and their dedicated classes, no more needed, as every variable access code is now optimized like this.
Stop using C++ class methods at all. Now only the destructor of local variables is actually supposed to do anything, and their are no methods anymore. The unused var_name got removed, setVariableValue is now done manually.
Moved assertions for the fiber layer to a common place in the header, so they are executed on all platforms in debug mode.
As usual, also a bunch of cleanups for PyLint were applied.
The locals built-in code now uses code generation for accessing local variable values instead having its own stuff.

Organizational

The Python version 3.4 is now officially supported. There are a few problems open, that will be addressed in future releases, none of which will affect normal people though.
Major cleanup of Nuitka options.
- Windows specific stuff is now in a dedicated option group. This includes options for icon, disabling console, etc.
- There is now a dedicated group for controlling backend compiler choices and options.
Also pickup g++44 automatically, which makes using Nuitka on CentOS5 more automatic.

Summary

This release represents a very important step ahead. Using SSA for real stuff will allow us to build the trust necessary to take the next steps. Using the SSA information, we could start implementing more optimizations.

Nuitka shaping up

2014-10-02T07:08:54+02:00

Not much has happened publicly to Nuitka, so it’s time to make a kind of status post, about the exciting news there is.

SSA (Single State Assignment Form)

For a long, long time already, each release of Nuitka has worked towards enabling “SSA” usage in Nuitka. There is a component called “constraint collection”, which is tasked with driving the optimization, and collecting variable traces.

Based on these traces, optimizations could be made. Having SSA or not, is (to me) the difference between Nuitka as a compiler, and Nuitka as an optimizing compiler.

The news is, SSA is shaping up, and will be used in the next release. Not yet to drive variable based optimization (reserved for a release after it), but to aid the code generation to avoid useless checks.

Improved Code Generation

Previously, under the title “C-ish”, Nuitka moved away from C++ based code generation to less C++ based code generated, and more C-ish code. This trend continues, and has lead to removing even more code cleanups.

The more important change is from the SSA derived knowledge. Now Nuitka knows that a variable must be assigned, cannot be assigned, may be assigned, based on its SSA traces.

Lets check out an example:

def f():
    a = 1
    return a

Nevermind, that obviously the variable a can be removed, and this could be transformed to statically return 1. That is the next step (and easy if SSA is working properly), now we are looking at what changed now.

This is code as generated now, with current 0.5.5pre5:

tmp_assign_source_1 = const_int_pos_1;
assert( var_a.object == NULL );
var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );

tmp_return_value = var_a.object;

Py_INCREF( tmp_return_value );
goto function_return_exit;

There are some things, wrong with it still. For one, var_a is still a C++ object, which we directly access. But the good thing is, we can assert that it starts out uninitialized, before we overwrite it. The stable release as of now, 0.5.4, generates code like this:

tmp_assign_source_1 = const_int_pos_1;
if (var_a.object == NULL)
{
    var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );
}
else
{
    PyObject *old = var_a.object;
    var_a.object = INCREASE_REFCOUNT( tmp_assign_source_1 );
    Py_DECREF( old );
}
static PyFrameObject *cache_frame_function = NULL;
MAKE_OR_REUSE_FRAME( cache_frame_function, codeobj_4e03e5698a52dd694c5c263550d71551, module___main__ );
PyFrameObject *frame_function = cache_frame_function;

// Push the new frame as the currently active one.
pushFrameStack( frame_function );

// Mark the frame object as in use, ref count 1 will be up for reuse.
Py_INCREF( frame_function );
assert( Py_REFCNT( frame_function ) == 2 ); // Frame stack

// Framed code:
tmp_return_value = var_a.object;

if ( tmp_return_value == NULL )
{

    exception_type = INCREASE_REFCOUNT( PyExc_UnboundLocalError );
    exception_value = UNSTREAM_STRING( &constant_bin[ 0 ], 47, 0 );
    exception_tb = NULL;

    frame_function->f_lineno = 4;
    goto frame_exception_exit_1;
}

Py_INCREF( tmp_return_value );
goto frame_return_exit_1;

As you can see, the assignment to var_a.object was checking if it were NULL, and if were not (which we now statically know), would release the old value. Next up, before returning, the value of var_a.object needed to be checked, if it were NULL, in which case, we would need to create a Python exception, and in order to do so, we need to create a frame object, that even if cached, consumes time, and code size.

So, that is the major change to code generation. The SSA information is now used in it, and doing so, has found a bunch of issues, in how it is built, in e.g. nested branches, that kind of stuff.

The removal of local variables as C++ classes, and them managed as temporary variables, is going to happen in a future release, reducing code complexity further. Were a a temporary variable, already, the Py_INCREF which implies a later Py_DECREF on the constant 1 could be totally avoided.

Scalability

The scalability of Nuitka hinges much of generated code size. With it being less stupid, the generated code is now not only faster, but definitely smaller, and with more optimization, it will only become more practical.

Compatibility

Python2 exec statements

A recent change in CPython 2.7.8+ which is supposed to become 2.7.9 one day, highlighted an issue with exec statements in Nuitka. These were considered to be fully compatible, but apparently are not totally.

def f():
    exec a in b, c
    exec (a, b, c)

The above two are supposed to be identical. So far this was rectified at run time of CPython, but apparently the parser is now tasked with it, so Nuitka now sees exec a in b, c for both lines. Which is good.

However, as it stands, Nuitka handles exec in locals() the same as exec in None for plain functions (OK to classes and modules), which is totally a bug.

I have been working on an enhanced re-formulation (it needs to be tracked if the value was None, and then the sync back to locals from the provided dictionary ought to be done. But the change breaks execfile in classes, which was implemented piggy-backing on exec, and now requires locals to be a dictionary, and immediately written to.

Anyway, consider exec as well working already. The non-working cases are really corner cases, obviously nobody came across so far.

Python3 classes

Incidentally, that execfile issue will be solved as soon as a bug is fixed, that was exposed by new abilities of Python3 metaclasses. They were first observed in Python3.4 enum classes.

class MyEnum(enum):
    red = 1
    blue = 2
    red = 3  # error

Currently, Nuitka is delaying the building of the dictionary (absent execfile built-in), and that is not allowed, in fact, immediate writes to the mapping giving by __prepare__ of the metaclass will be required, in which case, the enum class can raise an error for the second assignment to red.

So that area now hinges on code generation to learn different local variable codes for classes, centered around the notion of using the locals dictionary immediately.

Python3.4

The next release is no longer warning you if you use Python3.4, as many of the remaining problems have been sorted out. Many small things were found, and in some cases these highlighted general Python3 problems.

Nuitka for Python3 is not yet all that much in the focus in terms of performance, but correctness will have become much better, with most prominently, exception context being now correct most often.

The main focus of Nuitka is Python2, but to Nuitka the incompatibility of Python3 is largely not all that much an issue. The re-formulations to lower level operations for just about everything means that for the largest part there is not much trouble in supporting a mostly only slightly different version of Python.

The gain is mostly in that new tests are added in new releases, and these sometimes find things that affect Nuitka in all versions, or at least some others. And this could be a mere reference leak.

Consider this:

try:
    raise (TypeError, ValueError)
except TypeError:
    pass

So, that is working with Python2, but comes from a Python3 test. Python2 is supposed to unwrap the tuple and take the first argument and raise that. It didn’t do that so far. Granted, obscure feature, but still an incompatibility. For Python3, a TypeError should be raised complaining that tuple is not derived from BaseException.

Turned out, that also, in that case, a reference leak occurs, in that the wrong exception was not released, and therefore memory leaked. Should that happen a lot during a programs live, it will potentially become an issue, as it keeps frames on the traceback also alive.

So this lead to a compatibility fix and a reference leak fix. And it was found by the Python3.4 suite, checking that exception objects are properly released, and that the proper kind of exception is raised in the no longer supported case.

Performance

Graphs and Benchmarks

I had been working on automated performance graphs, and they are supposed to show up on Nuitka Speedcenter already, but currently it’s broken and outdated.

Sad state of affairs. Reasons include that I found it too ugly to publish unless updated to latest Nikola, for which I didn’t take the time. I intend to fix it, potentially before the release though.

Incremental Assignments

Consider the following code:

a += "bbb"

If a is a str, and if (and only if), it’s the only reference being held, then CPython, reuses the object, instead of creating a new object and copying a over. Well, Nuitka doesn’t do this. This is despite the problem being known for quite some time.

With SSA in place, and “C-ish” code generation complete, this will be solved, but I am not going to solve this before.

Standalone

The standalone mode of Nuitka is pretty good, and in the pre-release it was again improved. For instance, virtualenv and standalone should work now, and more modules are supported.

However, there are known issues with win32com and a few other packages, which need to be debugged. Mostly these are modules doing nasty things that make Nuitka not automatically detect imports.

This has as usual only so much priority from me. I am working on this on some occasions, as kind of interesting puzzles to solve. Most of the time, it just works though, with wxpython being the most notable exception. I am going to work on that though.

The standalone compilation exhibits scalability problems of Nuitka the most, and while it has been getting better, the recent and future improvements will lead to smaller code, which in turn means not only smaller executables, but also faster compilation. Again, wxpython is a major offender there, due to its many constants, global variables, etc. in the bindings, while Qt, PySide, and GTK are apparently already good.

Other Stuff

Funding

Nuitka doesn’t receive enough donations. There is no support from organizations like e.g. the PSF, which recently backed several projects by doubling donations given to them.

I remember talking to a PSF board member during Europython 2013 about this, and the reaction was fully in line with the Europython 2012 feedback towards me from the dictator. They wouldn’t help Nuitka in any way before it is successful.

I have never officially applied for help with funding though with them. I am going to choose to take pride in that, I suppose.

Collaborators

My quest to find collaborators to Nuitka is largely failing. Aside from the standalone mode, there have been too little contributions. Hope is that it will change in the future, once the significant speed gains arrive. And it might be my fault for not asking for help more, and to arrange myself with that state of things.

Not being endorsed by the Python establishment is clearly limiting the visibility of the project.

Anyway, things are coming along nicely. When I started out, I was fully aware that the project is something that I can do on my own if necessary, and that has not changed. Things are going slower than necessary though, but that’s probably very typical.

But you can join now, just follow this link or become part of the mailing list (since closed) and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Future

So, there is multiple things going on:

More “C-ish” code generation

The next release is going to be more “C-ish” than before, generating less complex code than before, and removes the previous optimizations, which were a lot of code, to e.g. detect parameter variables without del statements.

This prong of action will have to continue, as it unblocks further changes that lead to more compatibility and correctness.
More SSA usage

The next release did and will find bugs in the SSA tracing of Nuitka. It is on purpose only using it, to add assert statements to things it now no longer does. These will trigger in tests or cause crashes, which then can be fixed.

We better know that SSA is flawless in its tracking, before we use it to make optimizations, which then have no chance to assert anything at all anymore.

Once we take it to that next level, Nuitka will be able to speed up some things by more than the factor it basically has provided for 2 years now, and it’s probably going to happen this year.
More compatibility

The new exec code makes the dictionary synchronization explicit, and e.g. now it is optimized away to even check for its need, if we are in a module or a class, or if it can be known.

That means faster exec, but more importantly, a better understood exec, with improved ability to do SSA traces for them. Being able to in-line them, or to know the limit of their impact, as it will help to know more invariants for that code.

When these 3 things come to term, Nuitka will be a huge, huge step ahead towards being truly a static optimizing compiler (so far it is mostly only peep hole optimization, and byte code avoidance). I still think of this as happening this year.

Nuitka Release 0.5.4

2014-08-11T10:29:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is aiming at preparatory changes to enable optimization based on SSA analysis, introducing a variable registry, so that variables no longer trace their references to themselves.

Otherwise, MinGW64 support has been added, and lots of bug fixes were made to improve the compatibility.

Optimization

Using new variable registry, now properly detecting actual need for sharing variables. Optimization may discover that it is unnecessary to share a variable, and then it no longer is. This also allows --debug without it reporting unused variable warnings on Python3.
Scons startup has been accelerated, removing scans for unused tools, and avoiding making more than one gcc version check.

Bug Fixes

Compatibility: In case of unknown encodings, Nuitka was not giving the name of the problematic encoding in the error message. Fixed in 0.5.3.3 already.
Submodules with the same name as built-in modules were wrongly shadowed. Fixed in 0.5.3.2 already.
Python3: Added implementations of is_package to the meta path based loader.
Python3.4: Added find_spec implementation to the meta path based loader for increased compatibility.
Python3: Corrections for --debug to work with Python3 and MSVC compiler more often.
Fixed crash with --show-scons when no compiler was found. Fixed in 0.5.3.5 already.
Standalone: Need to blacklist lib2to3 from standard library as well. Fixed in 0.5.3.4 already.
Python3: Adapted to changes in SyntaxError on newer Python releases, there is now a msg that can override reason.
Standalone Windows: Preserve sys.executable as it might be used to fork binaries.
Windows: The caching of Scons was not arch specific, and files could be used again, even if changing the arch from x86 to x86_64 or back.
Windows: On 32 bit Python it can happen that with large number of generators running concurrently (>1500), one cannot be started anymore. Raising an MemoryError now.

Organizational

Added support for MinGW64. Currently needs to be run with PATH environment properly set up.
Updated internal version of Scons to 2.3.2, which breaks support for VS 2008, but adds support for VS 2013 and VS 2012. The VS 2013 is now the recommended compiler.
Added RPM package and repository for RHEL 7.
The output of --show-scons now includes the used compiler, including the MSVC version.
Added option --msvc to select the MSVC compiler version to use, which overrides automatic selection of the latest.
Added option -python-flag=no_warnings to disable user and deprecation warnings at run time.
Repository for Ubuntu Raring was removed, no more supported by Ubuntu.

Cleanups

Made technical and logical sharing decisions separate functions and implement them in a dedicated variable registry.
The Scons file has seen a major cleanup.

Summary

This release is mostly a maintenance release. The Scons integrations has been heavily visited, as has been Python3 and esp. Python3.4 compatibility, and results from the now possible debug test runs.

Standalone should be even more practical now, and MinGW64 is an option for those cases, where MSVC is too slow.

Nuitka Release 0.5.3

2014-07-07T23:25:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly a follow up, resolving points that have become possible to resolve after completing the C-ish evolution of Nuitka. So this is more of a service release.

New Features

Improved mode --improved now sets error lines more properly than CPython does in many cases.
The -python-flag=-S mode now preserves PYTHONPATH and therefore became usable with virtualenv.

Optimization

Line numbers of frames no longer get set unless an exception occurs, speeding up the normal path of execution.
For standalone mode, using --python-flag-S is now always possible and yields less module usage, resulting in smaller binaries and faster compilation.

Bug Fixes

Corrected an issue for frames being optimized away where in fact they are still necessary. Fixed in 0.5.2.1 already.
Fixed handling of exception tests as side effects. These could be remainders of optimization, but didn’t have code generation. Fixed in 0.5.2.1 already.
Previously Nuitka only ever used the statement line as the line number for all the expression, even if it spawned multiple lines. Usually nothing important, and often even more correct, but sometimes not. Now the line number is most often the same as CPython in full compatibility mode, or better, see above.
Python3.4: Standalone mode for Windows is working now.
Standalone: Undo changes to PYTHONPATH or PYTHONHOME allowing potentially forked CPython programs to run properly.
Standalone: Fixed import error when using PyQt and Python3.

New Tests

For our testing approach, the improved line number handling means we can undo lots of changes that are no more necessary.
The compile library test has been extended to cover a third potential location where modules may live, covering the matplotlib module as a result.

Cleanups

In Python2, the list contractions used to be re-formulated to be function calls that have no frame stack entry of their own right. This required some special handling, in e.g. closure taking, and determining variable sharing across functions.

This now got cleaned up to be properly in-lined in a try/finally expression.
The line number handling got simplified by pushing it into error exits only, removing the need to micro manage a line number stack which got removed.
Use intptr_t over unsigned long to store fiber code pointers, increasing portability.

Organizational

Providing own Debian/Ubuntu repositories for all relevant distributions.
Windows MSI files for Python 3.4 were added.
Hosting of the web site was moved to metal server with more RAM and performance.

Summary

This release brings about structural simplification that is both a follow-up to C-ish, as well as results from a failed attempt to remove static “variable references” and be fully SSA based. It incorporates changes aimed at making this next step in Nuitka evolution smaller.

Nuitka Release 0.5.2

2014-06-23T08:49:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is a major release, with huge changes to code generation that improve performance in a significant way. It is a the result of a long development period, and therefore contains a huge jump ahead.

New Features

Added experimental support for Python 3.4, which is still work in progress.
Added support for virtualenv on macOS.
Added support for virtualenv on Windows.
Added support for macOS X standalone mode.
The code generation uses no header files anymore, therefore adding a module doesn’t invalidate all compiled object files from caches anymore.
Constants code creation is now distributed, and constants referenced in a module are declared locally. This means that changing a module doesn’t affect the validity of other modules object files from caches anymore.

Optimization

C-ish code generation uses less C++ classes and generates more C-like code. Explicit temporary objects are now used for statement temporary variables.
The constants creation code is no more in a single file, but distributed across all modules, with only shared values created in a single file. This means improved scalability. There are remaining bad modules, but more often, standalone mode is now fast.
Exception handling no longer uses C++ exception, therefore has become much faster.
Loops that only break are eliminated.
Dead code after loops that do not break is now removed.
The try/finally and try/except constructs are now eliminated, where that is possible.
The try/finally part of the re-formulation for print statements is now only done when printing to a file, avoiding useless node tree bloat.
Tuples and lists are now generated with faster code.
Locals and global variables are now access with more direct code.
Added support for the anonymous code type built-in.
Added support for compile built-in.
Generators that statically return immediately, e.g. due to optimization results, are no longer using frame objects.
The complex call helpers use no pseudo frames anymore. Previous code generation required to have them, but with C-ish code generation that is no more necessary, speeding up those kind of calls.
Modules with only code that cannot raise, need not have a frame created for them. This avoids useless code size bloat because of them. Previously the frame stack entry was mandatory.

Bug Fixes

Windows: The resource files were cached by Scons and re-used, even if the input changed. The could lead to corrupted incremental builds. Fixed in 0.5.1.1 already.
Windows: For functions with too many local variables, the MSVC failed with an error “C1026: parser stack overflow, program too complex”. The rewritten code generation doesn’t burden the compiler as much.
Compatibility: The timing deletion of nested call arguments was different from C++. This shortcoming has been addressed in the rewritten code generation.
Compatibility: The __future__ flags and CO_FREECELL were not present in frame flags. These were then not always properly inherited to eval and exec in all cases.
Compatibility: Compiled frames for Python3 had f_restricted attribute, which is Python2 only. Removed it.
Compatibility: The SyntaxError of having a continue in a finally clause is now properly raised.
Python2: The exec statement with no locals argument provided, was preventing list contractions to take closure variables.
Python2: Having the ASCII encoding declared in a module wasn’t working.
Standalone: Included the idna encoding as well.
Standalone: For virtualenv, the file orig-prefix.txt needs to be present, now it’s copied into the “dist” directory as well. Fixed in 0.5.1.1 already.
Windows: Handle cases, where Python and user program are installed on different volumes.
Compatibility: Can now finally use execfile as an expression. One of our oldest issues, no 5, is finally fixed after all this time thanks to C-ish code generation.
Compatibility: The order or call arguments deletion is now finally compatible. This too is thanks to C-ish code generation.
Compatibility: Code object flags are now more compatible for Python3.
Standalone: Removing “rpath” settings of shared libraries and extension modules included. This makes standalone binaries more robust on Fedora 20.
Python2: Wasn’t falsely rejecting unicode strings as values for int and long variants with base argument provided.
Windows: For Python3.2 and 64 bits, global variable accesses could give false NameError exceptions. Fixed in 0.5.1.6 already.
Compatibility: Many exec and eval details have become more correctly, the argument handling is more compatible, and e.g. future flags are now passed along properly.
Compatibility: Using open with no arguments is now giving the same error.

Organizational

Replying to email from the issue tracker works now.
Added option name alias --xml for --dump-xml.
Added option name alias --python-dbg for --python-debug, which actually might make it a bit more clear that it is about using the CPython debug run time.
Remove option --dump-tree, it had been broken for a long time and unused in favor of XML dumps.
New digital art folder with 3D version of Nuitka logo. Thanks to Juan Carlos for creating it.
Using “README.rst” instead of “README.txt” to make it look better on web pages.
More complete whitelisting of missing imports in standard library. These should give no warnings anymore.
Updated the Nuitka GUI to the latest version, with enhanced features.
The builds of releases and update of the downloads page is now driven by Buildbot. Page will be automatically updated as updated binaries arrive.

Cleanups

Temporary keeper variables and the nodes to handle them are now unified with normal temporary variables, greatly simplifying variable handling on that level.
Less code is coming from templates, more is actually derived from the node tree instead.
Releasing the references to temporary variables is now always explicit in the node tree.
The publishing and preservation of exceptions in frames was turned into explicit nodes.
Exception handling is now done with a single handle that checks with branches on the exception. This eliminates exception handler nodes.
The dir built-in with no arguments is now re-formulated to locals or globals with their .keys() attribute taken.
Dramatic amounts of cleanups to code generation specialities, that got done right for the new C-ish code generation.

New Tests

Warnings from MSVC are now error exits for --debug mode too, expanding the coverage of these tests.
The outputs with python-dbg can now also be compared, allowing to expand test coverage for reference counts.
Many of the basic tests are now executable with Python3 directly. This allows for easier debug.
The library compilation test is now also executed with Python3.

Summary

This release would deserve more than a minor number increase. The C-ish code generation, is a huge body of work. In many ways, it lays ground to taking benefit of SSA results, that previously would not have been possible. In other ways, it’s incomplete in not yet taking full advantage yet.

The release contains so many improvements, that are not yet fully realized, but as a compiler, it also reflects a stable and improved state.

The important changes are about making SSA even more viable. Many of the problematic cases, e.g. exception handlers, have been stream lined. A whole class of variables, temporary keepers, has been eliminated. This is big news in this domain.

For the standalone users, there are lots of refinements. There is esp. a lot of work to create code that doesn’t show scalability issues. While some remain, the most important problems have been dealt with. Others are still in the pipeline.

More work will be needed to take full advantage. This has been explained in a separate post in greater detail.

Yup, another Python Quiz

2014-06-18T07:46:50+02:00

Using the following source code as a test happily in my Python compiler Nuitka for some years now.

# Testing dict optimization with all constants for compatibility.
print(
    "Dictionary entirely from constant args",
    dict(q="Guido", w="van", e="Rossum", r="invented", t="Python", y=""),
)

Quiz Question

Lately, when adding Python 3.4 support, this and other code changed. So lets do this manually:

PYTHONHASHSEED=0 python3.3

Python 3.3.5 (default, Mar 22 2014, 13:24:53)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dict(
...                q='Guido',
...                w='van',
...                e='Rossum',
...                r='invented',
...                t='Python',
...                y=''
...             )
{'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
>>> {'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
{'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
>>> {'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
{'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
>>> {'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}
{'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van', 'y': ''}

See, the dictionary is stable, once it gets reordered, due to hash values, but then it stays fixed. Which is pretty OK, and using a fixed hash value, it’s deterministic. Random hashing is not good for comparison testing, so I disable it for tests.

Now things get interesting, repeat with 3.4:

PYTHONHASHSEED=0 python3.4

Python 3.4.1rc1 (default, May  5 2014, 14:28:34)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dict(
...                q='Guido',
...                w='van',
...                e='Rossum',
...                r='invented',
...                t='Python',
...                y=''
...             )
{'y': '', 'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
>>> {'y': '', 'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
{'r': 'invented', 'q': 'Guido', 'y': '', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
>>> {'y': '', 'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
{'r': 'invented', 'q': 'Guido', 'y': '', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
>>> {'y': '', 'q': 'Guido', 'r': 'invented', 'e': 'Rossum', 't': 'Python', 'w': 'van'}
{'r': 'invented', 'q': 'Guido', 'y': '', 'e': 'Rossum', 't': 'Python', 'w': 'van'}

Nuitka builds this as the argument dictionary, before it is
passed to dict. Since it's all compile time constants, we can do that, right, and
we can use the result instead. So see this:

Look at how the result of “dict” is not reproducing itself, when used as a constant. I am only feeding the dict result to the interpreter, and it changes.

So the quizz this time is, why does this happen. What change in CPython3.4 makes this occur. Obviously it has to do with dictionary sizes.

Solution

I had a theory, but I couldn’t confirm it looking at all of CPython sources “ceval.c” and “dictobject.c” differences between the two versions.

I am suspecting a difference between presized and non-presized dictionaries, or that change to dictionary grow. When dict is being called, the amount of keys is know though, as well as when building the constant. So this ought to not play any role.

Hm, actually. I don’t know the solution yet. :-)

State of Nuitka

2014-06-17T05:35:48+02:00

For quite some time, publicly, very little has happened with my Python compiler Nuitka. But that doesn’t mean, there hasn’t been progress. In fact it is tremendous. I would like to have a post that kind of summarizes, what happened.

The last release, 0.5.1 was more of a maintenance release than making real changes. It turns out, that the bigger changes got delayed by a feature that I have described as “C-ish”. Let me outline, what this means.

C-ish vs. C++-ish

When I started working on Nuitka, the big question was if it is possible to create a sufficiently compatible compiler. The use of C++11 then, together with some templates made it easy to cover a wide, wide part of the language, and to fully integrate with CPython for compatibility.

The main goal was to get it going to work correctly. As time went on, execution order demanded to do away with variadic templates, raw strings were not all that perfect at all, and so C++-03 was good enough at one point.

And then, as Nuitka became less and less template based, and shoving more things into the node tree, and re-formulations, making this where the knowledge resided. It became more and more obvious that C++ has two problems. One in the way I used it. One inherent in the language typical implementations:

C++ exceptions are god damn slow
Everything should be a in a single statement.

The later was my choice. Initially it made it easy to pass on references and put the releasing C++ class around every expression as necessary. Identifier classes were allowing for code generation to avoid taking references where necessary, and it was not all that bad. Yet limiting.

This led to a couple of issues.

The order of call arguments release for e.g. f(g(h())) was not completely compatible, with the way how CPython does it. C++ destructors for objects living in a single statement didn’t give sufficient control, and make the order and timing of finalization not compatible.
The generated C++ code complexity became large. The compilation of the generated C++ in some cases was huge. To the point, that e.g. “code too complex” was giving by compilers like MSVC for some modules.
Cases of in-place assignments were discovered, where CPython outperforms Nuitka by a large margin. But these don’t fit into that style of code generation.

So, at some point, the pain had built up. Code generation was already lighter than in the beginning. For example, initially with statements had dedicated code templates to them. This, and many other things, are long gone.

I took a deep dive, and rewrote the whole code generation, to be much more “C-ish” than “C++-ish”. A huge undertaking that would take months.

Where previously, code didn’t have to handle return error codes (a C++ exception was thrown), now everything needed a return value name, and error check.
Where classes were previously conviently made sure things happened at function or scope exit, manual handling needed to be added.
The handling of continue, break, and return was previously done with exceptions being thrown, if they were to pass a try/finally handler. Now these are done with stacks of exit handlers, where goto statements are used to produce the correct behaviour.

Rewriting Code Generation

Redoing code generation, over months, while ultimately, slowly, arriving at a point where Nuitka would be doing this, it already did before, was kind of frustrating.

Of course, the performance benefit would be there, but it would not be all that much, except for exception raising and handling. There it would be huge. Ultimately for PyStone, a couple of extra percents were gained.

This really was a point, where I felt, that Nuitka will make it or break. And for a long time, I honestly wasn’t so sure, that I pull through. But I did.

Current Situation

The current pre-release is release quality. You should try it out, it’s great.

There are many changes to Standalone mode. Due to changes in how constants are now created in the modules that uses them, instead of everything globally, the parallel compilation now works great. What previously took an hour with MSVC (the problem child, gcc was always relatively good), now takes minutes only.
The support for virtualenv’s of all kinds seems to work on Windows, Linux, and macOS, which all seem to have different kinds of codes.
The support for macOS is now there. Thanks to a virtual server month donated to Jarrad Hope, I was able to iron issues out.
The final release will also work with standalone binaries created on Fedora 20 which got hard code rpaths removed on the factory git branch.

And yet, I am not yet releasing. In fact, I would like to ask you to give it a roll, and integrate test feedback.

Although more tests than ever are executed and pass, (e.g. the Mercurial test suite is now run each time I make a commit, and fully identically passes or fails the test suite with the current Mercurial code), there can never be enough.

The changes I made are the most intense ever, and definitely have potential for regressions. I am used to providing very high quality releases.

Also, I am working on the Buildbot instances to automate the production of performance graphs, which get updated fully automatically. I am working on updating the downloads page automatically for each release that gets made.

And generally, I am trying to improve my work flow, to make it easier to push out releases with less effort. Buildbot should drive the release process more completely. I am using the git flow to provide hot-fixes, and this should be even less painful in the future.

Open Points

With this release, presenting great progress, many things remain in an unfinished state.

The support for Python3.4 is not complete. Most things work, but some need more work. Specifically the changes to __class__ variable closure taking, need another major refactoring, this time on variable handling.

Currently there are variables, closure variables, temp variables, and then temp variable references. The way they work is different. One way they work different, prevents a temp variable closure reference to carry a name, in that case -_class__, which would be needed for Python3.4, where that is suddenly necessary.

With this done, the SSA code will be even easier to write, as temp variables and named variables will finally be fully unified.
The use of C++ classes is largely reduced now. But a few still remain, namely for local variables, closure variables, and temp variables that are explicit variables. They still use C++ classes, although changing that seems quite possible now, because at least for temporary variables, the class doesn’t do anything in terms of code anymore.

Removing these classes may well gain more performance.
Now that code generation can more easily make a difference, and SSA apparently is becoming reliable, it could be used to know that values must be value and to optimize checks away.

Currently every variable access checks for “NULL”, when it’s part of an assign trace. Some optimizations exist for parameter variables without del on them, that do not use SSA.

This could be expanded and made general, allowing for much less code to be generated (specifically avoiding error code, and release code for variables that cannot give an error).
The SSA has been found unreliable in some instances, due to bugs that I believe I found. We could attempt and forward propagate variable assignments to where they are used, eliminating variables, etc.

This is a place, where a lot of performance can be gained. We really want to be there. And “C-ish” now makes this ever more attractive, despite the large delay in time it has caused.
The in-place assignment code for strings, where CPython can be way faster than current Nuitka, it bears a risk of getting it wrong. It is therefore pushed to a future release.

Other Things

For the website, I am relocating the virtual machine to a dedicated server rented for an increased price. This will allow to add a few more dynamic features, as the virtual machine was always too limited in RAM. It’s more expensive, but I feel a better investment of my time.

As mentioned before, I am not going to conferences this year. Enjoy Europython, and consider having a Lightning talk about Nuitka. I will be there next year again.

Call for Help

Please test the latest release of Nuitka.
Please consider making a donation to support my work on Nuitka. I have continuous monthly costs of it, so it would be sweet if it’s with all my time spent working on it, at least not a financial cost to me.
Please join the mailing list (since closed), and offer your help with tasks. Nuitka can seriously take more people developing, testing, reviewing, and quality checking it.

Final Words

So, there is this “C-ish” release 0.5.2 cooking. You are invited to help. Big, improvements are coming to Nuitka. Even after this next huge release, very important work is still open, but hope is to have this complete over the summer.

Yours,

Kay

Try Finally Python Quiz

2014-04-24T22:08:56+02:00

When working on my Python compiler Nuitka, I often come across ridiculous language details of the Python language, and turn these into quizzes, for which I finally added a dedicated quiz tag.

Anyway, who can predict, what these will do to you:

def f():
    try:
        return 1
    finally:
        return 2

Will it return 1 or 2 ?

def f():
    try:
        1 / 0
    finally:
        return 2

Will this raise an ZeroDivisionError or return 2 ?

def f():
    while 1:
        try:
            continue
        finally:
            break

Is this an endless loop or does it return?

def f():
    while 1:
        try:
            break
        finally:
            continue

What about that? This one holds an inconsistency.

No solutions yet this time.

Nuitka Release 0.5.1

2014-03-08T04:21:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release brings corrections and major improvements to how standalone mode performs. Much of it was contributed via patches and bug reports.

Bug Fixes

There was a crash when using next on a non-iterable. Fixed in 0.5.0.1 already.
Module names with special characters not allowed in C identifiers were not fully supported. Fixed in 0.5.0.1 already.
Name mangling for classes with leading underscores was not removing them from resulting attribute names. This broke at __slots__ with private attributes for such classes. Fixed in 0.5.0.1 already.
Standalone on Windows might need “cp430” encoding. Fixed in 0.5.0.2 already.
Standalone mode didn’t work with lxml.etree due to lack of hard coded dependencies. When a shared library imports things, Nuitka cannot detect it easily.
Wasn’t working on macOS 64 bits due to using Linux 64 bits specific code. Fixed in 0.5.0.2 already.
On MinGW the constants blob was not properly linked on some installations, this is now done differently (see below).

New Features

Memory usages are now traced with --show-progress allowing us to trace where things go wrong.

Optimization

Standalone mode now includes standard library as bytecode by default. This is workaround scalability issues with many constants from many modules. Future releases are going to undo it.
On Windows the constants blob is now stored as a resource, avoiding compilation via C code for MSVC as well. MinGW was changed to use the same code.

New Tests

Expanded test coverage for “standalone mode” demonstrating usage of “hex” encoding, PySide, and PyGtk packages.

Summary

This release is mostly an interim maintenance release for standalone. Major changes that provide optimization beyond that, termed “C-ish code generation” are delayed for future releases.

This release makes standalone practical which is an important point. Instead of hour long compilation, even for small programs, we are down to less than a minute.

The solution of the scalability issues with many constants from many modules will be top priority going forward. Since they are about how even single use constants are created all in one place, this will be easy, but as large changes are happening in “C-ish code generation”, we are waiting for these to complete.

Not going to FOSDEM 2014

2014-01-18T20:48:53+01:00

So I submitted a talk for FOSDEM “Python has a compiler now”. And it didn’t get accepted. Quite a surprise to me, but likely mostly logical. These are the accepted talks (link since became unavailable) and well, it surely didn’t fit in, right.

My intent was to introduce Nuitka to the larger community. Not going to happen now, but maybe next year.

Clearly triggered by the rejection, I am questioning Europython 2014 in Berlin as a useful destination too. To me it seems, that code writing is the best way to create the community at this time anyway.

Nuitka Release 0.5.0

2014-01-04T10:05:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release breaks interface compatibility, therefore the major version number change. Also “standalone mode” has seen significant improvements on both Windows, and Linux. Should work much better now.

But consider that this part of Nuitka is still in its infancy. As it is not the top priority of mine for Nuitka, which primarily is intended as an super compatible accelerator of Python, it will continue to evolve nearby.

There is also many new optimization based on structural improvements in the direction of actual SSA.

Bug Fixes

The “standalone mode” was not working on all Redhat, Fedora, and openSUSE platforms and gave warnings with older compilers. Fixed in 0.4.7.1 already.
The “standalone mode” was not including all useful encodings. Fixed in 0.4.7.2 already.
The “standalone mode” was defaulting to --python-flag=-S which disables the parsing of “site” module. That unfortunately made it necessary to reach some modules without modifying PYTHONPATH which conflicts with the “out-of-the-box” experience.
The “standalone mode” is now handling packages properly and generally working on Windows as well.
The syntax error of having an all catching except clause and then a more specific one wasn’t causing a SyntaxError with Nuitka.
```
try:
    something()
except:
    somehandling()
except TypeError:
    notallowed()
```
A corruption bug was identified, when re-raising exceptions, the top entry of the traceback was modified after usage. Depending on malloc this was potentially causing an endless loop when using it for output.

New Features

Windows: The “standalone” mode now properly detects used DLLs using Dependency Walker which it offers to download and extra for you.

It is used as a replacement to ldd on Linux when building the binary, and as a replacement of strace on Linux when running the tests to check that nothing is loaded from the outside.

Optimization

When iterating over list, set, this is now automatically lowered to tuples avoiding the mutable container types.

So the following code is now equivalent:
```
for x in [a, b, c]:
    ...

# same as
for x in (a, b, c):
    ...
```
For constants, this is even more effective, because for mutable constants, no more is it necessary to make a copy.
Python2: The iteration of large range is now automatically lowered to xrange which is faster to loop over, and more memory efficient.
Added support for the xrange built-in.
The statement only expression optimization got generalized and now is capable of removing useless parts of operations, not only the whole thing when it has not side effects.
```
[a, b]

# same as
a
b
```
This works for all container types.

Another example is type built-in operation with single argument. When the result is not used, it need not be called.
```
type(a)

# same as
a
```
And another example is and is not have no effect of their own as well, therefore:
```
a is b

# same as
a
b
```
Added proper handling of conditional expression branches in SSA based optimization. So far these branches were ignored, which only acceptable for temporary variables as created by tree building, but not other variable types. This is preparatory for introducing SSA for local variables.

Organizational

The option --exe is now ignored and creating an executable is the default behavior of nuitka, a new option --module allows to produce extension modules.
The binary nuitka-python was removed, and is replaced by nuitka-run with now only implies --execute on top of what nuitka is.
Using dedicated Buildbot for continuous integration testing and release creation as well.
The Downloads now offers MSI files for Win64 as well.
Discontinued the support for cross compilation to Win32. That was too limited and the design choice is to have a running CPython instance of matching architecture at Nuitka compile time.

New Tests

Expanded test coverage for “standalone mode” demonstrating usage of “hex” encoding, and PySide package.

Summary

The “executable by default” interface change improves on the already high ease of use. The new optimization do not give all that much in terms of numbers, but are all signs of structural improvements, and it is steadily approaching the point, where the really interesting stuff will happen.

The progress for standalone mode is of course significant. It is still not quite there yet, but it is making quick progress now. This will attract a lot of attention hopefully.

As for optimization, the focus for it has shifted to making exception handlers work optimal by default (publish the exception to sys.exc_info() and create traceback only when necessary) and be based on standard branches. Removing special handling of exception handlers, will be the next big step. This release includes some correctness fixes stemming from that work already.