Nuitka Blog - Posted in 2011

Nuitka Release 0.3.16

2011-12-18T17:24:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups.

The release cycle focused on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via 2to3 conversion) without trouble. I don’t know when, but it seems that it’s going to work one day.

Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It’s still not there, but it’s making progress.

Bug Fixes

Fixed a packaging problem for Linux and x64 platform, the new swapFiber.S file for the fiber management was not included. Released as 0.3.15a hot fix already.
Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hot fix already.
Fixed an issue with __import__ and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hot fix already.
Fixed eval() and locals() to be supported in generator expressions and contractions too. Released as 0.3.15d hot fix already.
Fixed the Windows batch files nuitka.bat and nuitka-python.bat to not output the rem statements with the copyright header. Released as 0.3.15d hot fix already.
Fixed re-raise with raise, but without a current exception set. Released as 0.3.15e hot fix already.
Fixed vars() call on the module level, needs to be treated as globals(). Released as 0.3.15e hot fix already.
Fix handling of broken new lines in source files. Read the source code in “universal line ending mode”. Released as 0.3.15f hot fix already.
Fixed handling of constant module attribute __name__ being replaced. Don’t replace local variables of the same name too. Released as 0.3.15g hot fix already.
Fixed assigning to True, False or None. There was this old TODO, and some code has compatibility craft that does it. Released as 0.3.15g hot fix already.
Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hot fix already.
Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not.
Fixed non-execution of cleanup codes that e.g. flush sys.stdout, by adding Py_Finalize().
Fix throw() method of generator expression objects to not check arguments properly.
Fix missing fallback to subscript operations for slicing with non-indexable objects.
Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did.
Fix, the future spec was not properly preserving the future division flag.

Optimization

The optimization scales now much better, because per-module optimization only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time.
The creation of dictionaries from constants is now also optimized.

New Features

As a new feature functions now have the func_defaults and __defaults__ attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though.
The names True, False and None are now converted to constants only when they are read-only module variables.
The PYTHONPATH variable is now cleared when immediately executing a compiled binary unless --execute-with-pythonpath is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required.

Organizational

The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
The in-line copy of Scons is not checked with PyLint anymore. We of course don’t care.
Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious.
The basic tests can now be run with PYTHON=python3.2 and use 2to3 conversion in that case.

Cleanups

Moved tags to a separate module, make optimization emit only documented tags, checked against the list of allowed ones.
The Debian package has seen lots of improvements, to make it “lintian clean”, even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc.
Use os.path.join in more of the test code to achieve more Windows portability for them.
Some more PyLint cleanups.

New Tests

There is now a dedicated test for things that crashed Nuitka previously.
Added a program test where the imported module does a sys.exit() and make sure it really doesn’t continue after the SystemExit exception that creates.
Cover the type of __builtins__ in the main program and in imported modules in tests too. It’s funny and differs between module and dict in CPython2.
Cover a final print statement without newline in the test. Must still receive a newline, which only happens when Py_Finalize() is called.
Added test with functions that makes a raise without an exception set.
Cover the calling of vars() on module level too.
Cover the use of eval in contractions and generator expressions too.
Cover func_defaults and __default__ attributes for a function too.
Added test function with two raise in an exception handler, so that one becomes dead code and removed without the crash.

Summary

The “git flow” was really great in this release cycle. There were many hot fix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects.

This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority, correctness is.

The work on Python3 is a bit strange. I don’t need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can’t really stop to work on it. It has that appeal of small fixups here and there, and then something else works too.

Python3 work is like when I was first struggling with Nuitka to pass the CPython2 unit tests for a first time. It’s fun. And then it finds real actual bugs that apply to CPython2 too. Not doing Py_Finalize (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the earlier Python3 is supported, the more benefit from increased test covered.

What’s missing is more “hg” completeness. I think only the raise without exception set and the func_defaults issue were going into its direction, but it won’t be enough yet.

Nuitka Release 0.3.15

2011-12-01T20:24:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new stable release of Nuitka. This time again many organizational improvements, some bug fixes, much improved compatibility and cleanups.

This release cycle focused on packaging Nuitka for easier consumption, i.e. automatic packaging, making automatic uploads, improvement documentation, and generally cleaning things up, so that Nuitka becomes more compatible and ultimately capable to run the “hg” test suite. It’s not there yet, but this is a huge jump for usability of Nuitka and its compatibility, again.

Then lots of changes that make Nuitka approach Python3 support, the generated C++ for at least one large example is compiling with this new release. It won’t link, but there will be later releases.

And there is a lot of cleanup going on, geared towards compatibility with line numbers in the frame object.

Bug Fixes

The main module was using __main__ in tracebacks, but it must be <module>. Released as 0.3.14a hot fix already.
Workaround for “execfile cannot be used as an expression”. It wasn’t possible to use execfile in an expression, only as a statement.

But then there is crazy enough code in e.g. mercurial that uses it in a lambda function, which made the issue more prominent. The fix now allows it to be an expression, except on the class level, which wasn’t seen yet.
The in-line copy of Scons was not complete enough to work for “Windows” or with --windows-target for cross compile. Fixed.
Cached frames didn’t release the “back” frame, therefore holding variables of these longer than CPython does, which could cause ordering problems. Fixed for increased compatibility.
Handle “yield outside of function” syntax error in compiled source correctly. This one was giving a Nuitka backtrace, now it gives a SyntaxError as it needs to.
Made syntax/indentation error output absolutely identical to CPython.
Using the frame objects f_lineno may fix endless amounts bugs related to traceback line numbers.

New Features

Guesses the location of the MinGW compiler under Windows to default install location, so it need not be added to PATH environment variable. Removes the need to modify PATH environment just for Nuitka to find it.
Added support for “lambda generators”. You don’t want to know what it is. Lets just say, it was the last absurd language feature out there, plus that didn’t work. It now works perfect.

Organizational

You can now download a Windows installer and a Debian package that works on Debian Testing, current Ubuntu and Mint Linux.
New release scripts give us the ability to have hot fix releases as download packages immediately. That means the “git flow” makes even more beneficial to the users.
Including the generated “README.pdf” in the distribution archives, so it can be read instead of “README.txt”. The text file is fairly readable, due to the use of ReStructured Text, but the PDF is even nicer to read, due to e.g. syntax highlighting of the examples.
Renamed the main binaries to nuitka and nuitka-python, so that there is no dependency on case sensitive file systems.
For Windows there are batch files nuitka.bat and nuitka-python.bat to make Nuitka directly executable without finding the Python.exe, which the batch files can tell from their own location.
There are now man pages of nuitka and nuitka-python with examples for the most common use cases. They are of course included in the Debian package.
Don’t strip the binary when executing it to analyse compiled binary with valgrind. It will give better information that way, without changing the code.

Optimization

Implemented swapcontext alike (swapFiber) for x64 to achieve 8 times speedup for Generators. It doesn’t do useless syscalls to preserve signal masks. Now Nuitka is faster at frame switching than CPython on x64, which is already good by design.

Cleanups

Using the frame objects to store current line of execution avoids the need to store it away in helper code at all. It ought to also help a lot with threading support, and makes Nuitka even more compatible, because now line numbers will be correct even outside tracebacks, but for mere stack frame dumps.
Moved the for_return detection from code generation to tree building where it belongs. Yield statements used as return statements need slightly different code for Python2.6 difference. That solved an old TODO.
Much Python3 portability work. Sometimes even improving existing code, the Python compiler code had picked up a few points, where the latest Nuitka didn’t work with Python3 anymore, when put to actual compile.

The test covered only syntax, but e.g. meta classes need different code in CPython3, and that’s now supported. Also helper code was made portable in more places, but not yet fully. This will need more work.
Cleaned up uses of debug defines, so they are now more consistent and in one place.
Some more PyLint cleanups.

New Tests

The tests are now executed by Python scripts and cover stderr output too. Before we only checked stdout. This unveiled a bunch of issues Nuitka had, but went unnoticed so far, and triggered e.g. the frame line number improvements.
Separate syntax tests.
The scripts to run the tests now are all in pure Python. This means, no more MinGW shell is needed to execute the tests.

Summary

The Debian package, Windows installer, etc. are now automatically updated and uploaded. From here on, there can be such packages for the hot fix releases too.

The exception tracebacks are now correct by design, and better covered.

The generator performance work showed that the approach taken by Nuitka is in fact fast. It was fast on ARM already, but it’s nice to see that it’s now also fast on x64. Programs using generators will be affected a lot by this.

Overall, this release brings Nuitka closer to usability. Better binary names, man pages, improved documentation, issue tracker, etc. all there now. I am in fact now looking for a sponsor for the Debian package to upload it into Debian directly.

Update

The upload to Debian happened for 0.3.18 and was done by Yaroslav Halchenko.

What’s missing is more “hg” completeness. The frame release issue helped it, but inspect.getargs() doesn’t work yet, and is a topic for a future release. Won’t be easy, as func_defaults will be an invasive change too.

Nuitka Debian Package and Windows Installer

2011-11-13T22:20:00+01:00

There is now a Windows installer and a Debian package of Nuitka available on the Download page. Please try it out and give me feedback.

Specifically I do know that the Debian package won’t work on Debian Squeeze, but only on Debian Wheezy (Testing) maybe it does on Ubuntu as well, please report. If you have anything to criticize about the package, let me know. There is no apt source now, I hope to get it into Debian proper directly.

UPDATE: After Stani’s report that Ubuntu has an older Scons, I lowered the dependency and updated the package on the page. It may now work on Ubuntu as well.

And then, the Windows installer still requires you to install MinGW and add it to your path, but it’s clearly a huge step ahead. It’s created with distutils, and that works very well. If you have the skills to enhance it, so e.g. the PATH variable is changed, or it will launch a MinGW install if not present, contact me and help.

UPDATE: And the idea that I got while writing a reply to “swong” is also now implemented. The new Nuitka on Windows simply guesses the PATH to MinGW to be the default path C:\MinGW or at least \MinGW and from there, it ought to just run after you installed it. Of course you can still set your own PATH environment and make the pick yourself.

Yours,

Kay Hayen

Nuitka Release 0.3.14

2011-11-07T20:52:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new stable release of Nuitka. This time it contains mostly organizational improvements, some bug fixes, improved compatibility and cleanups.

It is again the result of working towards compilation of a real program (Mercurial). This time, I have added support for proper handling of compiled types by the inspect module.

Bug Fixes

Fix for “Missing checks in parameter parsing with star list, star dict and positional arguments”. There was whole in the checks for argument counts, now the correct error is given. Fixed in 0.3.13a already.
The simple slice operations with 2 values, not extended with 3 values, were not applying the correct order for evaluation. Fixed in 0.3.13a already.
The simple slice operations couldn’t handle None as the value for lower or upper index. Fixed in 0.3.11a already.
The in-place simple slice operations evaluated the slice index expressions twice, which could cause problems if they had side effects. Fixed in 0.3.11a already.

New Features

Run time patching the inspect module so it accepts compiled functions, compiled methods, and compiled generator objects. The test_inspect test of CPython is nearly working unchanged with this.
The generator functions didn’t have CO_GENERATOR set in their code object, setting it made compatible with CPython in this regard too. The inspect module will therefore return correct value for inspect.isgeneratorfunction() too.

Optimization

Slice indexes that are None are now constant propagated as well.
Slightly more efficient code generation for dual star arg functions, removing useless checks.

Cleanups

Moved the Scons, static C++ files, and assembler files to new package nuitka.build where also now SconsInterface module lives.
Moved the Qt dialog files to nuitka.gui
Moved the “unfreezer” code to its own static C++ file.
Some PyLint cleanups.

New Tests

New test Recursion to cover recursive functions.
New test Inspection to cover the patching of inspect module.
Cover execfile on the class level as well in ExecEval test.
Cover evaluation order of simple slices in OrderCheck too.

Organizational

There is a new issue tracker available (since migrated and removed)

Please register and report issues you encounter with Nuitka. I have put all the known issues there and started to use it recently. It’s Roundup based like https://bugs.python.org is, so people will find it familiar.
The setup.py is now apparently functional. The source releases for download are made it with, and it appears the binary distributions work too. We may now build a windows installer. It’s currently in testing, we will make it available when finished.

Summary

The new source organisation makes packaging Nuitka really easy now. From here, we can likely provide “binary” package of Nuitka soon. A windows installer will be nice.

The patching of inspect works wonders for compatibility for those programs that insist on checking types, instead of doing duck typing. The function call problem, was an issue found by the Mercurial test suite.

For the “hg.exe” to pass all of its test suite, more work may be needed, this is the overall goal I am currently striving for. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization.

Nuitka Release 0.3.13

2011-11-01T15:07:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This release is mostly the result of working towards compilation of a real programs (Mercurial) and to merge and finalize the frame stack work. Now Nuitka has a correct frame stack at all times, and supports func_code and gi_code objects, something previously thought to be impossible.

Actually now it’s only the “bytecode” objects that won’t be there. And not attributes of func_code are meaningful yet, but in theory can be supported.

Due to the use of the “git flow” for Nuitka, most of the bugs listed here were already fixed in on the stable release before this release. This time there were 5 such hot fix releases, sometimes fixing multiple bugs.

Bug Fixes

In case of syntax errors in the main program, an exception stack was giving that included Nuitka code. Changed to make the same output as CPython does. Fixed in 0.3.12a already.
The star import (from x import *) didn’t work for submodules. Providing * as the import list to the respective code allowed to drop the complex lookups we were doing before, and to simply trust CPython C/API to do it correctly. Fixed in 0.3.12 already.
The absolute import is not the default of CPython 2.7 it seems. A local posix package shadows the standard library one. Fixed in 0.3.12 already.
In --deep mode, a module may contain a syntax error. This is e.g. true of “PyQt” with port_v3 included. These files contain Python3 syntax and fail to be imported in Python2, but that is not to be considered an error. These modules are now skipped with a warning. Fixed in 0.3.12b already.
The code to import modules wasn’t using the __import__ built-in, which prevented __import__ overriding code to work. Changed import to use the built-in. Fixed in 0.3.12c already.
The code generated for the __import__ built-in with constant values was doing relative imports only. It needs to attempt relative and absolute imports. Fixed in 0.3.12c already.
The code of packages in “__init__.py” believed it was outside of the package, giving problems for package local imports. Fixed in 0.3.12d already.
It appears that “Scons”, which Nuitka uses internally and transparent to you, to execute the compilation and linking tasks, was sometimes not building the binaries or shared libraries, due to a false caching. As a workaround, these are now erased before doing the build. Fixed in 0.3.12d already.
The use of in and not in in comparison chains (e.g. a < b < c is one), wasn’t supported yet. The use of these in comparison chains a in b in c is very strange.

Only in the test_grammar.py it was ever used I believe. Anyway, it’s supported now, solving this TODO and reducing the difference. Fixed in 0.3.12e already.
The order of evaluation for in and not in operators wasn’t enforced in a portable way. Now it is correct on “ARM” too. Fixed in 0.3.12e already.

Optimization

The built-ins GeneratorExit and StopIteration are optimized to their Python C/API names where possible as well.

Cleanups

The __file__ attribute of modules was the relative filename, but for absolute filenames these become a horrible mess at least on Linux.
Added assertion helpers for sane frame and code objects and use them.
Make use of assertObject in more places.
Instead of using os.path.sep all over, added a helper Utils.joinpath that hides this and using os.path.join. This gives more readable code.
Added traces to the “unfreezer” guarded by a define. Helpful in analyzing import problems.
Some PyLint cleanups removing dead code, unused variables, useless pass statement, etc.

New Tests

New tests to cover SyntaxError and IndentationError from --deep imports and in main program.
New test to cover evaluation order of in and not in comparisons.
New test to cover package local imports made by the “__init__.py” of the package.

Organizational

Drop “compile_itself.sh” in favor of the new “compile_itself.py”, because the later is more portable.
The logging output is now nicer, and for failed recursions, outputs the line that is having the problem.

Summary

The frame stack work and the func_code are big for compatibility.

The func_code was also needed for “hg” to work. For Mercurial to pass all of its test suite, more work will be needed, esp. the inspect module needs to be run-time patched to accept compiled functions and generators too.

Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization.

Putting a conference T-shirt to good use

2011-11-01T09:11:00+01:00

This is my 7rd old son in his “Halloween” outfit. It is the first time, he was allowed to do it and so he was very excited.

The link to Python is the T-shirt “Don’t Panic - Use Python”. I got that from PyCON DE, and the family loves it.

Michael in his Halloween outfit.

Nuitka Release 0.3.12

2011-10-24T20:43:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new release of Nuitka many bug fixes, and substantial improvements especially in the organizational area. There is a new User Manual (PDF), with much improved content, a sys.meta_path based import mechanism for --deep mode, git flow goodness.

This release is generally also the result of working towards compilation of a real programs (Mercurial) and to get things work more nicely on Windows by default. Thanks go to Liu Zhenhai for helping me with this goal.

Due to the use of the “git flow”, most of the bugs listed here were already fixed in on the stable release before this release. And there were many of these.

Bug Fixes

The order of evaluation for base classes and class dictionaries was not enforced.

Apparently nothing in the CPython test suite did that, I only noticed during debugging that Nuitka gave a different error than CPython did, for a class that had an undefined base class, because both class body and base classes were giving an error. Fixed in 0.3.11a already.
Method objects didn’t hold a reference to the used class.

The effect was only noticed when --python-debug was used, i.e. the debug version of Python linked, because then the garbage collector makes searches. Fixed in 0.3.11b already.
Set sys.executable on Linux as well. On Debian it is otherwise /usr/bin/python which might be a different version of Python entirely. Fixed in 0.3.11c already.
Embedded modules inside a package could hide package variables of the same name. Learned during PyCON DE about this corner case. Fixed in 0.3.11d already.
Packages could be duplicated internally. This had no effect on generated code other than appearing twice in the list if frozen modules. Fixed in 0.3.11d already.
When embedding modules from outside current directory, the look-up failed. The embedding only ever worked for the compile itself and programs test cases, because they are all in the current directory then. Fixed in 0.3.11e already.
The check for ARM target broke Windows support in the Scons file. Fixed in 0.3.11f already.
The star import from external modules failed with an error in --deep mode. Fixed in 0.3.11g already.
Modules with a parent package could cause a problem under some circumstances. Fixed in 0.3.11h already.
One call variant, with both list and dict star arguments and keyword arguments, but no positional parameters, didn’t have the required C++ helper function implemented. Fixed in 0.3.11h already.
The detection of the CPU core count was broken on my hexacore at least. Gave 36 instead of 6, which is a problem for large programs. Fixed in 0.3.11h already.
The in-line copy of Scons didn’t really work on Windows, which was sad, because we added it to simplify installation on Windows precisely because of this.
Cleaning up the build directory from old sources and object files wasn’t portable to Windows and therefore wasn’t effective there.
From imports where part of the imported were found modules and parts were not, didn’t work. Solved by the feature branch meta_path_import that was merged for this release.
Newer MinGW gave warnings about the default visibility not being possible to apply to class members. Fixed by not setting this default visibility anymore on Windows.
The sys.executable gave warnings on Windows because of backslashes in the path. Using a raw string to prevent such problems.
The standard library path was hard coded. Changed to run time detection.

Cleanups

Version checks on Python runtime now use a new define PYTHON_VERSION that makes it easier. I don’t like PY_VERSION_HEX, because it is so unreadable. Makes some of the checks a lot more safe.
The sys.meta_path based import from the meta_path_import feature branch allowed the cleanup the way importing is done. It’s a lot less code now.
Removed some unused code. We will aim at making Nuitka the tool to detect dead code really.
Moved nuitka.Nodes to nuitka.nodes.Nodes, that is what the package is intended for, the split will come later.

New Tests

New tests for import variants that previously didn’t work: Mixed imports. Imports from a package one level up. Modules hidden by a package variable, etc.
Added test of function call variant that had no test previously. Only found it when compiling “hg”. Amazing how nothing in my tests, CPython tests, etc. used it.
Added test to cover the partial success of import statements.
Added test to cover evaluation order of class definitions.

Organizational

Migrated the “README.txt” from org-mode to ReStructured Text, which allows for a more readable document, and to generate a nice User Manual in PDF form.
The amount of information in “README.txt” was increased, with many more subjects are now covered, e.g. “git flow” and how to join Nuitka development. It’s also impressive to see what code blocks and syntax highlighting can do for readability.
The Nuitka git repository has seen multiple hot fixes.

These allowed to publish bug fixes immediately after they were made, and avoided the need for a new release just to get these out. This really saves me a lot of time too, because I can postpone releasing the new version until it makes sense because of other things.
Then there was a feature branch meta_path_import that lived until being merged to develop to improve the import code, which is now released as part of the main branch. Getting that feature right took a while.
And there is the feature branch minimize_CPython26_tests_diff which has some success already in documenting the required changes to the “CPython26” test suite and in reducing the amount of differences, while doing it. We have a frame stack working there, albeit in too ugly code form.
The release archives are now built using setuptools. You can now also download a zip file, which is probably more Windows friendly. The intention is to work on that to make setup.py produce a Nuitka install that won’t rely on any environment variables at all. Right now setup.py won’t even allow any other options than sdist to be given.
Ported “compile_itself.sh” to “compile_itself.py”, i.e. ported it to Python. This way, we can execute it easily on Windows too, where it currently still fails. Replacing diff, rm -rf, etc. is a challenge, but it reduces the dependency on MSYS tools on Windows.
The compilation of standard library is disabled by default, but site or dist packages are now embedded. To include even standard library, there is a --really-deep option that has to be given in addition to --deep, which forces this.

Summary

Again, huge progress. The improved import mechanism is very beautiful. It appears that little is missing to compile real world programs like “hg” with Nuitka. The next release cycle will focus on that and continue to improve the Windows support which appears to have some issues.

Nuitka - PyCON DE Video

2011-10-22T06:39:00+02:00

Hello everybody,

the Video of my presentation is online: https://pyvideo.org/pycon-de-2011/nuitka-der-python-compiler.html

Leaving it to Mike Müller to announce here the links to everything, when it’s all finished. Thanks for the good work, the video was very well done.

Update

The video is also available via Youtube.

Yours,

Kay

Cat update

2011-10-18T17:36:00+02:00

The new cat has become indispensable for just about anything. She follows throughout the hous, and is just too lovely, here two photos of her in typical spots.

When I code for Nuitka, she often lays between keyboard and monitors, sleeping, getting caressed by me.

And here is another one, hiding in her cat tree.

My 7yr old fotographer

2011-10-18T17:24:00+02:00

This photo was shot by my 7yr old son. It’s truly artful, if not perfect. I have not applied any correction. He has my old camera, and knows how to use it, more and more.

Michael shot of his mom

PyCON DE 2011 - My Report

2011-10-08T21:24:00+02:00

The PyCON DE 2011 is just over, sprints are still happening over the weekend, but my wife wouldn’t allow me to stay away for that long, so it’s not for me this time. Maybe next time.

Right now I feel very happy and excited that I went there. What a great experience this was.

It was the first German PyCON and clearly it was overdue as it was now the merger of many already grown up communities. A huge number of talks over 3 days in 3 parallel tracks, with 3 keynotes, was an outstanding program. And very well run. Strict time management, every detail was well prepared.

I can only admire the professional preparation and setup. I wanted to say thank you deeply. I didn’t consider it possible to be this good. Clearly not a first time.

I enjoyed the talks, most often in the technical track, but other tracks would have been very interesting too. The parallelism was making me do hard decisions.

Food

The food was great too. I esp. liked the Asian day, but there was also Italian and French, and what many liked very much is that there was a Vegan food offer too. I do not live vegan style, but I appreciate good food and the vegan food often is that.

Leipzig

The quality of the organization team, the city “Leipzig”, where we also got to have a guided city tour of fantastic enthusiasms, was very high. I knew Leipzig from earlier visits and liked it before, but this time it seemed everybody was even friendlier.

Site

The convention place “Kubus” was very well chosen, absolutely ideal. It’s got good equipment, and that large room setup, where you can make a split with movable walls, and have 3 big screens. The acoustics were pretty damn good there.

My own Presentation

As to my own presentation, it was well received, although I sort of regret that I agreed to have only 30m instead of original plan of 60m. I had so much to say.

I ended up with getting my manifesto part out, but that one pretty well. And it’s OK I guess, because nobody really listens that long anyway. And my major points came across that way.

That focus on my Nuitka “manifesto” was probably a good idea. The talk will be available online as a video, I will link it then. The PDF that I presented only a small part of, is linked here. I believe it went pretty well.

I will use that content from the PDF in updated documentation (currently ongoing in PDF is work to use REST and document a lot more). The presentation was created with “rst2pdf”, which I find is a fantastic tool.

Contacts

Cython / lxml

Then contacts!

Early on I already made contacts with interesting people, e.g. with Dr.Stefan Behnel, author of lxml and core Cython developer. I him offered a beer for using his software in the best of Free Software traditions. He doesn’t drink these, but a large mango juice counts too or so I assume.

We also talked about Cython and Nuitka, and the common history we had as well. For some time, I attempted to change Cython, but that failed to get the developers support at the time. Not wanting to deviate from PyRex clearly isn’t the state anymore, but that was then.

We also had a evening session of showing each other the good and bad parts, comparing was quite fun. And it was quite interesting to the both of us. I believe we made friends and will only benefit another.

We discussed my goals, and I think we came to the conclusion that they are in fact different enough from Cythons. Although I go away with the sense, that of course Stefan believes, it would be better if I joined Cython. Naturally.

But that’s not going to happen. I think i have a cleaner and better implementation now, closer to my goals with a realistic chance to succeed. To me it would be a step back to fix language parsing issues and incompatibilities of Cython, with the danger that my goals will not be shared.

As an example of these things, I would mention function call errors, where e.g. Cython gives different and sometimes worse error messages than CPython, and I designed the code so that it does things in that same order than CPython does.

It do not want to give different error messages, and who knows, somebody may check for the exception text and expect CPython output. In this case, I will rather accept a worse performance, than an incompatibility.

Eliminating function parameter parsing for the whole program as far as possible is going to be more worthwhile anyway.

But in my mind, Cython is something I can and do recommend. For as long as I am not able to declare Nuitka “useful” yet. That statement may come within a year though. In my mind, in many fields Nuitka is already superior.

PyHasse

Another interesting contact I made, was with the author of PyHasse. It’s Rainer Bruggemann, who is a really nice and witty guy. He introduced me to how he applies graph theory to multi-parameter optimization problems.

We agreed that we will try and work together on this project. Hopefully it will come to pass. One thing I personally wanted, was to get into contact with people who understand or are part of the scientific community.

I can see what NumPy is. But I may never know myself what it really is, unless I find proxies, and make these kind of contacts. The same thing is true of Django, or e.g. Mercurial. I am positive though that with time, and such conferences, my knowledge of these will only grow.

We said that we will try and see how far we can go. In the worst case, Nuitka will not yet be useful, but I will have a clearer image what is needed.

Debian

I saw the presentation from Jan Dittberner and met him later too, asking him questions, and generally discussing Debian packaging of Nuitka. He encouraged me to contact the Debian Python Team, and so I will.

I used the chance to make contact with a Debian guy, who made a presentation on how to package Python modules for Debian. He gave me hints on how to solve that “find files near me” issue that plagues Nuitka just as much as other software. Really kind and helpful guy and clearly I admire Debian Developers, keep up the good work.

General

I also made contacts with lots of other people. Python is diverse and it was fun to get to know, many people with similar and entirely different backgrounds.

The mood was extremely constructive. Nuitka was well received, but that’s not why I say it. There is that general sense of respect around that German community, you can feel how pretty much everybody is well established and doesn’t have to disprove the others.

Keynotes

One keynotes speaker had a part about how trolling and hate is bad for a community, but that’s not the German Python community.

Another keynote speaker (Paul Everitt) had a part about how Zope, which was kind of his project, failed in many ways. He seemed to be quite disappointed about that, which triggered me to point out, that he should start his story with Apache, and not see the “failure to integrate” as a failure.

If there had not been Apache failing, there wouldn’t have been Zope, and then not Django, etc. that’s kind of normal and actually good. He agreed and pointed out how Apache was created from another project that had failed to integrate people.

You either fork a projects code, or ideas. The fork still should credit and appreciate the predecessor/origin.

In my mind, Cython failed to integrate me. Which triggered me to come up with Nuitka, and as I will point out over time (there ought to be postings and there probably will be), some better approaches.

So not integrating me is not necessarily a failure. If it were not for Cython, there would not be Nuitka. The original projects will regret the fork/remake, but they probably shouldn’t. Competition is good.

Lets repeat that

I believe the PyCON DE 2011 was a huge success. I will most likely go again to update people on Nuitka. It’s already clear there will be a PyCON DE 2012 I understand. And I am aiming for a slot at PyCON EU 2012 next year too. I wanted to go in 2011, but need to not put it in my early booked holiday again.

But you know what Murphy says about that.

Yours,

Kay Hayen

Nuitka git-flow

2011-10-01T08:37:00+02:00

Hello there,

this is to let you know that I have switched Nuitka to the “git flow” development model. That means, now there is a supported stable version, and a develop branch, together with feature branches.

Example

Git flow example for Nuitka release 0.3.12

Note

In case, you want and can improve the source (link since removed) visually or otherwise, please go ahead. I am using it for a presentation next week too, and would be glad if you could make it more pretty. My artistic skills are not the same as my programmer skills. :-)

So there is now always at least these 2 branches:

Stable (master branch)
Unstable (develop branch)

and then there may be feature branches, like this one currently:

Feature branch minimize_CPython26_tests_diff

These will only have certain life, until they are completed, then they are merge into “develop” and become part of the next release. This may or may not happen, depending on how things go.

Benefits of the new model

Hotfixes, typically bug fixes, can be made simultaneously on stable and develop branch. The git-flow package takes care of the merging to both.
Because that’s so easy now, a stable version can be provided and supported for a longer time.
Features can be published while under development. My idea is that feature branches should basically work, but the bar will be lower. People can have a look at them, or start their own and make me integrate them.

Uses of Feature Branch

For example, in the new feature branch, a couple of boring things are happening. Support for frame stack will reduce the diff, as will some work to match CPython’s choices for exception line numbers. Completing will take a while, but should not block a release. So this is best done in the feature branch, esp. as nothing is going to really depend on it.

General Picture

As you can see from this diagram, I am working mostly on documentation things. The new and improved README on develop, which is closer to a User Manual in PDF form, and other organization things, may get a release before the PyCon DE next week. The README also describes this process.

Hope is that with this approach, I will improve transparency (you can see earlier what i am working on, because there is now a place where things may break (develop) or may not yet be integrated or completed fully (feature branches) and yet be public.

The overhead appears to minimal thanks to “git-flow”. Developing hotfixes is actually easier, when done on the stable branch, because problems cannot originate from the current development work that may or may not be all that perfect yet.

Yours,

Kay Hayen

Nuitka Release 0.3.11

2011-09-17T02:41:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new release of Nuitka with some bug fixes and portability work.

This release is generally cleaning up things, and makes Nuitka portable to ARM Linux. I used to host the Nuitka homepage on that machine, but now that it’s no longer so, I can run heavy compile jobs on it. To my surprise, it found many portability problems. So I chose to fix that first, the result being that Nuitka now works on ARM Linux too.

Bug Fixes

The order of slice expressions was not correct on x86 as well, and I found that with new tests only. So the porting to ARM revealed a bug category, I previously didn’t consider.
The use of linux2 in the Scons file is potentially incompatible with Linux 3.0, although it seems that at least on Debian the sys.platform was changed back to linux2. Anyway, it’s probably best to allow just anything that starts with linux these days.
The print statement worked like a print function, i.e. it first evaluated all printed expressions, and did the output only then. That is incompatible in case of exceptions, where partial outputs need to be done, and so that got fixed.

Optimization

Function calls now each have a dedicated helper function, avoiding in some cases unnecessary work. We will may build further on this and in-line PyObject_Call differently for the special cases.

Cleanups

Moved many C++ helper declarations and in-line implementations to dedicated header files for better organisation.
Some dependencies were removed and consolidated to make the dependency graph sane.
Multiple decorators were in reverse order in the node tree. The code generation reversed it back, so no bug, yet that was a distorted tree.

Finding this came from the ARM work, because the “reversal” was in fact just the argument evaluation order of C++ under x86/x64, but on ARM that broke. Correcting it highlighted this issue.
The deletion of slices, was not using Py_ssize for indexes, disallowing some kinds of optimization, so that was harmonized.
The function call code generation got a general overhaul. It is now more consistent, has more helpers available, and creates more readable code.
PyLint is again happier than ever.

New Tests

There is a new basic test OrderChecks that covers the order of expression evaluation. These problems were otherwise very hard to detect, and in some cases not previously covered at all.
Executing Nuitka with Python3 (it won’t produce correct Python3 C/API code) is now part of the release tests, so non-portable code of Nuitka gets caught.

Organizational

Support for ARM Linux. I will make a separate posting on the challenges of this. Suffice to say now, that C++ leaves way too much things unspecified.
The Nuitka git repository now uses “git flow”. The new git policy will be detailed in another separate posting.
There is an unstable develop branch in which the development occurs. For this release ca. 40 commits were done to this branch, before merging it. I am also doing more fine grained commits now.
Unlike previously, there is master branch for the stable release.
There is a script “make-dependency-graph.sh” (Update: meanwhile it was renamed to “make-dependency-graph.py”) to produce a dependency graphs of Nuitka. I detected a couple of strange things through this.
The Python3 __pycache__ directories get removed too by the cleanup script.

Numbers

We only have “PyStone” now, and on a new machine, so the numbers cannot be compared to previous releases:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

So this a speedup factor of 258%, last time on another machine it was 240%. Yet it only proves that the generated and compiled are more efficient than bytecode, but Nuitka doesn’t yet do the relevant optimization. Only once it does, the factor will be significantly higher.

Summary

Overall, there is quite some progress. Nuitka is a lot cleaner now, which will help us later only. I wanted to get this out, mostly because of the bug fixes, and of course just in case somebody attempts to use it on ARM.

Going to PyCon DE

2011-08-18T13:02:00+02:00

Hello everybody,

I am going to the German Python conference in Leipzip and I am going to have a presentation (link in German) there.

Of course it’s about Nuitka (see What is Nuitka? ) and I hope it will get a lot of attendance. I am naturally very happy to have the opportunity to present it finally. I had wanted to visit PyCon EU, but the date was never known, and then my early booking holiday overlapped with it, so it was not an option.

Now giving this presentation will of course be exciting to me. I gave presentations as part of the day job many times, but this time it’s obviously a different. It’s also a first chance to meet the others as I never was at a Python conference before, and that will be interesting in itself.

Presenting Nuitka will of course be easy for me. Now I need to plan what I want to be able to present for a demo. Running a big thing like Mercurial would be nice, but I honestly don’t know, if that will even be difficult, or if it will take a lot of work. Also the amount of documentation available for Nuitka should increase as part of this. Designs, etc. could be made into diagrams, so people who want to join will have it easier.

Lots of possibilities, and then there is only going to be one reality. Lets hope it’s good. :-)

Yours,

Kay

Nuitka has a new home - nuitka.net

2011-08-16T18:21:00+02:00

Hello everybody,

my new vServer is now online, the old site blog redirects to it now, and planets follow the new site already. The old site is redirecting with 301 error code for some more time, hopefully this will be good enough of a migration. It felt smooth from here, clearly, and the new site is much faster of course, because it’s not throttled through my poor DSL upstream.

I also registered it a domain name “nuitka.net” for it finally. Originally I wanted to give it a DynDNS name, but I already have 2, and the third was supposed to cost money, and I didn’t feel like using another service, or cheating them if you consider that an option, so that was it, I bought the domain name. Clearly it will also be easier for people to remember.

Moving things over was pretty painless. The Wordpress software has a pretty good export feature, which only didn’t manage the custom menu and appearance settings. But I guess it’s about content anyway, so probably some way that would work too, but it was easy enough to reproduce that by hand. I appreciate Wordpress even more now.

Also I did some tuning of the system, to use less memory. Only 512M are available, and so I run less Apache processes for less requests (memory leaks), disabled IPv6 (yes, hate me for it), reduced amounts of gettys, and so on. Nothing I am not familiar with, the ARM machine had 512M as well, and to me no reason to use the bigger package just because of that.

The main difference is the faster CPU, I seem to get 3Ghz Intel now, instead of my 1Ghz ARM, which together with faster internet speed, makes the site extremely fast.

Now U will dare to make the gitweb interface public as well. The git repository is already running there.

Note

This is obsolete information, we use GitHub for this now.

And I took the chance to sanitize the old posts somewhat. Changed the links to not use the old domain name anymore, and correct some broken ones too.

Only downside is that I currently haven’t got the “speedcenter” up and running again. After losing my hardware, the old data cannot be compared with new one, and then it doesn’t feel like a priority right now. I seem to be working instead on XML based regression tests of the optimizer: The output of “–dump-xml” should be compared for a large quantity of files, to discover regressions of the optimizer as soon as possible, this will enable me to make changes and not have to review the C++ as much, to find out if something is compiled correctly. This way I should detect it when known good cases degrade, and generally to demonstrate better, what actually did improve.

Yours,

Kay Hayen

PS: Oh, you people, who wonder, “but why are you not using Google/github/gitorious?”, my counter question: “Did you read the agreement?” I did. It basically says (from Google code):

13. INDEMNITY

You agree to hold harmless and indemnify Google, and its subsidiaries, affiliates, officers, agents, employees, advertisers, licensors, suppliers or partners, (collectively "Google and Partners") from and against any third-party claim arising from or in any way related to your use of Google services, violation of the Terms or any other actions connected with use of Google services, including any liability or expense arising from all claims, losses, damages (actual and consequential), suits, judgments, litigation costs and attorneys' fees, of every kind and nature. In such a case, Google will provide you with written notice of such claim, suit or action.

No thank you, instead I will run my own server, then I get to pay the attorneys of my discretion - in the admittedly unlikely event that somebody should sue me, because my Compiler violates some patent, or whatever.

Yours,

Kay

Nuitka Release 0.3.10

2011-07-31T17:12:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The focus was roundup and breaking new grounds with structural optimization enhancements.

Bug Fixes

Exceptions now correctly stack.

When you catch an exception, there always was the exception set, but calling a new function, and it catching the exception, the values of sys.exc_info() didn’t get reset after the function returned.

This was a small difference (of which there are nearly none left now) but one that might effect existing code, which affects code that calls functions in exception handling to check something about it.

So it’s good this is resolved now too. Also because it is difficult to understand, and now it’s just like CPython behaves, which means that we don’t have to document anything at all about it.
Using exec in generator functions got fixed up. I realized that this wouldn’t work while working on other things. It’s obscure yes, but it ought to work.
Lambda generator functions can now be nested and in generator functions. There were some problems here with the allocation of closure variables that got resolved.
List contractions could not be returned by lambda functions. Also a closure issue.
When using a mapping for globals to exec or eval that had a side effect on lookup, it was evident that the lookup was made twice. Correcting this also improves the performance for the normal case.

Optimization

Statically raised as well as predicted exceptions are propagated upwards, leading to code and block removal where possible, while maintaining the side effects.

This is brand new and doesn’t do everything possible yet. Most notable, the matching of raised exception to handlers is not yet performed.
Built-in exception name references and creation of instances of them are now optimized as well, which leads to faster exception raising/catching for these cases.
More kinds of calls to built-ins are handled, positional parameters are checked and more built-ins are covered.

Notable is that now checks are performed if you didn’t potentially overload e.g. the len with your own version in the module. Locally it was always detected already. So it’s now also safe.
All operations and comparisons are now simulated if possible and replaced with their result.
In the case of predictable true or false conditions, not taken branches are removed.
Empty branches are now removed from most constructs, leading to sometimes cleaner code generated.

Cleanups

Removed the lambda body node and replaced it with function body. This is a great win for the split into body and builder. Regular functions and lambda functions now only differ in how the created body is used.
Large cleanup of the operation/comparison code. There is now only use of a simulator function, which exists for every operator and comparison. This one is then used in a prediction call, shared with the built-in predictions.
Added a Tracing module to avoid future imports of print_function, which annoyed me many times by causing syntax failures for when I quickly added a print statement, not noting it must have the braces.
PyLint is happier than ever.

New Tests

Enhanced OverflowFunctions test to cover even deeper nesting of overflow functions taking closure from each level. While it’s not yet working, this makes clearer what will be needed. Even if this code is obscure, I would like to be that correct here.
Made Operators test to cover the ^ operator as well.
Added to ListContractions the case where a contraction is returned by a lambda function, but still needs to leak its loop variable.
Enhanced GeneratorExpressions test to cover lambda generators, which is really crazy code:
```
def y():
    yield ((yield 1), (yield 2))
```
Added to ExecEval a case where the exec is inside a generator, to cover that too.
Activated the testing of sys.exc_info() in ExceptionRaising test.

This was previously commented out, and now I added stuff to illustrate all of the behavior of CPython there.
Enhanced ComparisonChains test to demonstrate that the order of evaluations is done right and that side effects are maintained.
Added BuiltinOverload test to show that overloaded built-ins are actually called and not the optimized version. So code like this has to print 2 lines:
```
from __builtin__ import len as _len


def len(x):
    print x


return _len(x)

print len(range(9))
```

Organizational

Changed “README.txt” to no longer say that “Scons” is a requirement. Now that it’s included (patched up to work with ctypes on Windows), we don’t have to say that anymore.
Documented the status of optimization and added some more ideas.
There is now an option to dump the node tree after optimization as XML. Not currently use, but is for regression testing, to identify where new optimization and changes have an impact. This make it more feasible to be sure that Nuitka is only becoming better.
Executable with Python3 again, although it won’t do anything, the necessary code changes were done.

Summary

It’s nice to see, that I some long standing issues were resolved, and that structural optimization has become almost a reality.

The difficult parts of exception propagation are all in place, now it’s only details. With that we can eliminate and predict even more of the stupid code of “pybench” at compile time, achieving more infinite speedups.

The new cat

2011-07-31T12:49:00+02:00

This is the latest addition to the family, our beautiful, young and lovely cat:

Muska on her first day with us.

Her name is Muska, and she is with us for a week now. This is an image from her first day in our house.

Nuitka Release 0.3.9

2011-04-24T11:19:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is about the new release of Nuitka which some bug fixes and offers a good speed improvement.

This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The main focus was on faster function calls, faster class attributes (not instance), faster unpacking, and more built-ins detected and more thoroughly optimizing them.

Bug Fixes

Exceptions raised inside with statements had references to the exception and traceback leaked.
On Windows the binaries sys.executable pointed to the binary itself instead of the Python interpreter. Changed, because some code uses sys.executable to know how to start Python scripts.
There is a bug (fixed in their repository) related to C++ raw strings and C++ “trigraphs” that affects Nuitka, added a workaround that makes Nuitka not emit “trigraphs” at all.
The check for mutable constants was erroneous for tuples, which could lead to assuming a tuple with only mutable elements to be not mutable, which is of course wrong.

Optimization

This time there are so many new optimization, it makes sense to group them by the subject.

Exceptions

The code to add a traceback is now our own, which made it possible to use frames that do not contain line numbers and a code object capable of lookups.
Raising exceptions or adding to tracebacks has been made way faster by reusing a cached frame objects for the task.
The class used for saving exceptions temporarily (e.g. used in try/finally code, or with statement) has been improved.

It now doesn’t make a copy of the exception with a C++ new call, but it simply stores the exception properties itself and creates the exception object only on demand, which is more efficient.
When catching exceptions, the addition of tracebacks is now done without exporting and re-importing the exception to Python, but directly on the exception objects traceback, this avoids a useless round trip.

Function Calls

Uses of PyObject_Call provide NULL as the dictionary, instead of an empty dictionary, which is slightly faster for function calls.
There are now dedicated variants for complex function calls with * and ** arguments in all forms.

These can take advantage of easier cases. For example, a merge with star arguments is only needed if there actually were any of these.
The check for non-string values in the ** arguments can now be completely short-cut for the case of a dictionary that has never had a string added. There is now code that detects this case and skips the check, eliminating it as a performance concern.

Parameter Parsing

Reversed the order in which parameters are checked.

Now the keyword dictionary is iterated first and only then the positional arguments after that is done. This iteration is not only much faster (avoiding repeated lookups for each possible parameter), it also can be more correct, in case the keyword argument is derived from a dictionary and its keys mutate it when being compared.
Comparing parameter names is now done with a fast path, in which the pointer values are compare first. This can avoid a call to the comparison at all, which has become very likely due to the interning of parameter name strings, see below.
Added a dedicated call to check for parameter equality with rich equality comparison, which doesn’t raise an exception.
Unpacking of tuples is now using dedicated variants of the normal unpacking code instead of rolling out everything themselves.

Attribute Access

The class type (in executables, not yet for extension modules) is changed to a faster variant of our own making that doesn’t consider the restricted mode a possibility. This avoids very expensive calls, and makes accessing class attributes in compiled code and in non-compiled code faster.
Access to attributes (but not of instances) got in-lined and therefore much faster. Due to other optimization, a specific step to intern the string used for attribute access is not necessary with Nuitka at all anymore. This made access to attributes about 50% faster which is big of course.

Constants

The bug for mutable tuples also caused non-mutable tuples to be considered as mutable, which lead to less efficient code.
The constant creation with the g++ bug worked around, can now use raw strings to create string constants, without resorting to un-pickling them as a work around. This allows us to use PyString_FromStringAndSize to create strings again, which is obviously faster, and had not been done, because of the confusion caused by the g++ bug.
For string constants that are usable as attributes (i.e. match the identifier regular expression), these are now interned, directly after creation. With this, the check for identical value of pointers for parameters has a bigger chance to succeed, and this saves some memory too.
For empty containers (set, dict, list, tuple) the constants created are now are not unstreamed, but created with the dedicated API calls, saving a bit of code and being less ugly.
For mutable empty constant access (set, dict, list) the values are no longer made by copying the constant, but instead with the API functions to create new ones. This makes code like a = [] a tiny bit faster.
For slice indices the code generation now takes advantage of creating a C++ Py_ssize_t from constant value if possible. Before it was converting the integer constant at run time, which was of course wasteful even if not (very) slow.

Iteration

The creation of iterators got our own code. This avoids a function call and is otherwise only a small gain for anything but sequence iterators. These may be much faster to create now, as it avoids another call and repeated checks.
The next on iterator got our own code too, which has simpler code flow, because it avoids the double check in case of NULL returned.
The unpack check got similar code to the next iterator, it also has simpler code flow now and avoids double checks.

Built-ins

Added support for the list, tuple, dict, str, float and bool built-ins along with optimizing their use with constant parameter.
Added support for the int and long built-ins, based on a new “call spec” object, that detects parameter errors at compile time and raises appropriate exceptions as required, plus it deals with keyword arguments just as well.

So, to Nuitka it doesn’t matter now it you write int(value) or int(x = value) anymore. The base parameter of these built-ins is also supported.

The use of this call spec mechanism will the expanded, currently it is not applied to the built-ins that take only one parameter. This is a work in progress as is the whole built-ins business as not all the built-ins are covered yet.

Cleanups

In 0.3.8 per module global classes were introduced, but the IMPORT_MODULE kept using the old universal class, this got resolved and the old class is now fully gone.
Using assertObject in more cases, and in more places at all, catches errors earlier on.
Moved the addition to tracebacks into the _PythonException class, where it works directly on the contained traceback. This is cleaner as it no longer requires to export exceptions to Python, just to add a traceback entry.
Some PyLint cleanups were done, reducing the number of reports a bit, but there is still a lot to do.
Added a DefaultValueIdentifier class that encapsulates the access to default values in the parameter parsing more cleanly.
The module CodeTemplatesListContractions was renamed to CodeTemplatesContractions to reflect the fact that it deals with all kinds of contractions (also set and dict contractions), not just list contractions.
Moved the with related template to its own module CodeTemplatesWith, so its easier to find.
The options handling for g++ based compilers was cleaned up, so that g++ 4.6 and MinGW are better supported now.
Documented more aspects of the Scons build file.
Some more generated code white space fixes.
Moved some helpers to dedicated files. There is now calling.hpp for function calls, an importing.cpp for import related stuff.
Moved the manifest generation to the scons file, which now produces ready to use executables.

New Tests

Added a improved version of “pybench” that can cope with the “0 ms” execution time that Nuitka has for some if its sub-tests.
Reference counting test for with statement was added.
Micro benchmarks to demonstrate try finally performance when an exception travels through it.
Micro benchmark for with statement that eats up exceptions raised inside the block.
Micro benchmarks for the read and write access to class attributes.
Enhanced Printing test to cover the trigraphs constant bug case. Output is required to make the error detectable.
Enhanced Constants test to cover repeated mutation of mutable tuple constants, this covers the bug mentioned.

Organizational

Added a credits section to the “README.txt” where I give credit to the people who contributed to Nuitka, and the projects it is using. I will make it a separate posting to cite these.
Documented the requirements on the compiler more clearly, document the fact that we require scons and which version of Python (2.6 or 2.7).
The is now a codespeed implementation up and running with historical data for up to Nuitka 0.3.8 runs of “PyStone” and with pybench. It will be updated for 0.3.9 once I have the infrastructure in place to do that automatically.
The cleanup script now also removes .so files.
The handling of options for g++ got improved, so it’s the same for g++ and MinGW compilers, plus adequate errors messages are given, if the compiler version is too low.
There is now a --unstripped option that just keeps the debug information in the file, but doesn’t keep the assertions.

This will be helpful when looking at generated assembler code from Nuitka to not have the distortions that --debug causes (reduced optimization level, assertions, etc.) and instead a clear view.

Looking where Nuitka stands

2011-04-16T11:52:00+02:00

In case you wonder, [what Nuitka is](/pages/overview.html), look here. Over the 0.3.x release cycle, I have mostly looked at its performance with “pystone”. I merely wanted to have a target to look at and enjoy the progress we have made there.

In the context of the Windows port then, Khalid Abu Bakr used the pybench on Windows and that got me interested. It’s a nice collection of micro benchmarks, which is quite obviously aimed for looking CPython implementations only. In that it’s quite good to check where Nuitka is good at, and where it can still take improvements for the milestone 2 stuff.

Enhancements to PyBench

The pybench refused to accept that Nuitka could use so little time on some tests, I needed to hack it to allow it.
Then it had “ZeroDivisionError” exceptions, because Nuitka can run fully predictable code not at all, thus with a time of 0ms, which gives interesting factors.
Also these are many results, we are going to care for regressions only, so there is an option now to output only tests with negative values.

The Interesting Parts

Nuitka currently has some fields where optimizations are already so effective as to render the whole benchmark pointless. Longterm, most of PyBench will not be looked at anymore, where the factor becomes “infinity”, there is little point in looking at it. We will likely just use it as a test that optimizations didn’t suddenly regress. Publishing the numbers will not be as interesting.
Then there are slow downs. These I take seriously, because of course I expect that Nuitka shall only be faster than CPython. Sometimes the implementation of Nuitka for some rarely used features is sub par though. I color coded these in red in the table below.
ComplexPythonFunctionCalls: These are twice as slow, which is an tribute to the fact, that the code in this domain is only as good as it needs to be. Of course function calls are very important, and this needs to be addressed.
TryRaiseExcept: This is much slower because of the cost of the raise statement, which is extremely high currently. For every raise, a frame object with a specific code object is created, so the traceback will point to the correct location. This is very inefficient, and wasteful. We need to be able to create code objects that can be used for all lines needed, and then we can re-use it and only have one frame object per function, which then can be re-used itself. There is already some work for that in [current git](/doc/download.html) (0.3.9 pre 2), but it’s not yet complete at all.
WithRaiseExcept: Same problem as TryRaiseExcept, the exception raising is too expensive.
Note also that -90% is in fact much worse that +90%, the “diff” numbers from pybench make improvements look much better than regressions do. You can also checkout the comparison on the new [benchmark pages](https://speedcenter.nuitka.net) that I am just creating, they are based on codespeed, which I will blog upon separately.

Look at this table of results as produced by pybench:

Benchmark Results

Test Name	min CPython	min Nuitka	diff
BuiltinFunctionCalls	76ms	54ms	+41.0%
BuiltinMethodLookup	57ms	47ms	+22.1%
CompareFloats	79ms	0ms	+inf%
CompareFloatsIntegers	75ms	0ms	+inf%
CompareIntegers	76ms	0ms	+inf%
CompareInternedStrings	68ms	32ms	+113.0%
CompareLongs	60ms	0ms	+inf%
CompareStrings	86ms	62ms	+38.2%
CompareUnicode	61ms	50ms	+21.9%
ComplexPythonFunctionCalls	86ms	179ms	-52.3%
ConcatStrings	98ms	99ms	-0.6%
ConcatUnicode	127ms	124ms	+2.3%
CreateInstances	76ms	52ms	+46.8%
CreateNewInstances	58ms	47ms	+22.1%
CreateStringsWithConcat	85ms	90ms	-6.5%
CreateUnicodeWithConcat	74ms	68ms	+9.5%
DictCreation	58ms	36ms	+60.9%
DictWithFloatKeys	67ms	44ms	+51.7%
DictWithIntegerKeys	64ms	30ms	+113.8%
DictWithStringKeys	60ms	26ms	+130.6%
ForLoops	47ms	15ms	+216.2%
IfThenElse	67ms	16ms	+322.5%
ListSlicing	69ms	70ms	-0.9%
NestedForLoops	72ms	25ms	+187.4%
NestedListComprehensions	87ms	42ms	+105.9%
NormalClassAttribute	62ms	77ms	-18.9%
NormalInstanceAttribute	56ms	24ms	+129.7%
PythonFunctionCalls	72ms	34ms	+116.1%
PythonMethodCalls	84ms	38ms	+120.0%
Recursion	97ms	56ms	+73.1%
SecondImport	61ms	47ms	+31.6%
SecondPackageImport	66ms	29ms	+125.4%
SecondSubmoduleImport	86ms	32ms	+172.0%
SimpleComplexArithmetic	74ms	62ms	+18.3%
SimpleDictManipulation	65ms	35ms	+89.7%
SimpleFloatArithmetic	77ms	56ms	+39.3%
SimpleIntFloatArithmetic	58ms	39ms	+48.3%
SimpleIntegerArithmetic	59ms	37ms	+57.7%
SimpleListComprehensions	75ms	33ms	+128.7%
SimpleListManipulation	57ms	27ms	+109.4%
SimpleLongArithmetic	68ms	57ms	+19.9%
SmallLists	69ms	41ms	+66.6%
SmallTuples	66ms	98ms	-32.2%
SpecialClassAttribute	63ms	49ms	+29.1%
SpecialInstanceAttribute	130ms	24ms	+434.5%
StringMappings	67ms	62ms	+8.5%
StringPredicates	69ms	59ms	+16.6%
StringSlicing	73ms	47ms	+54.8%
TryExcept	57ms	0ms	+3821207.1%
TryFinally	65ms	26ms	+153.4%
TryRaiseExcept	64ms	610ms	-89.5%
TupleSlicing	76ms	67ms	+12.7%
UnicodeMappings	88ms	91ms	-2.9%
UnicodePredicates	64ms	59ms	+8.8%
UnicodeProperties	69ms	63ms	+8.8%
UnicodeSlicing	80ms	68ms	+17.6%
WithFinally	84ms	26ms	+221.2%
WithRaiseExcept	67ms	1178ms	-94.3%

She’s a doctor now

2011-04-16T11:48:00+02:00

My wife has passed the final exams here in Germany finally. I am very proud of her for managing that. It’s with 2 kids born, a new house built, and lots of difficulties, like not living in a city with a university that offers medicin as a study course.

To celebrate, here is a picture of her from happy days (no photoshop unlike the last time):

Anna in front of a bush in Dithmarsia (my home state).

Nuitka Release 0.3.8

2011-04-03T00:30:00+02:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is to inform you about the new release of Nuitka with some real news and a slight performance increase. The significant news is added “Windows Support”. You can now hope to run Nuitka on Windows too and have it produce working executables against either the standard Python distribution or a MinGW compiled Python.

There are still some small things to iron out, and clearly documentation needs to be created, and esp. the DLL hell problem of msvcr90.dll vs. msvcrt.dll, is not yet fully resolved, but appears to be not as harmful, at least not on native Windows.

I am thanking Khalid Abu Bakr for making this possible. I was surprised to see this happen. I clearly didn’t make it easy. He found a good way around ucontext, identifier clashes, and a very tricky symbol problems where the CPython library under Windows exports less than under Linux. Thanks a whole lot.

Currently the Windows support is considered experimental and works with MinGW 4.5 or higher only.

Otherwise there have been the usual round of performance improvements and more cleanups. This release is otherwise milestone 2 work only, which will have to continue for some time more.

Bug Fixes

Lambda generators were not fully compatible, their simple form could yield an extra value. The behavior for Python 2.6 and 2.7 is also different and Nuitka now mimics both correctly, depending on the used Python version
The given parameter count cited in the error message in case of too many parameters, didn’t include the given keyword parameters in the error message.
There was an assert False right after warning about not found modules in the --deep mode, which was of course unnecessary.

Optimization

When unpacking variables in assignments, the temporary variables are now held in a new temporary class that is designed for the task specifically.

This avoids the taking of a reference just because the PyObjectTemporary destructor insisted on releasing one. The new class PyObjectTempHolder hands the existing reference over and releases only in case of exceptions.
When unpacking variable in for loops, the value from the iterator may be directly assigned, if it’s to a variable.

In general this would be possible for every assignment target that cannot raise, but the infrastructure cannot tell yet, which these would be. This will improve with more milestone 3 work.
Branches with only pass inside are removed, pass statements are removed before the code generation stage. This makes it easier to achieve and decide empty branches.
There is now a global variable class per module. It appears that it is indeed faster to roll out a class per module accessing the module * rather than having one class and use a module **, which is quite disappointing from the C++ compiler.
Also MAKE_LIST and MAKE_TUPLE have gained special cases for the 0 arguments case. Even when the size of the variadic template parameters should be known to the compiler, it seems, it wasn’t eliminating the branch, so this was a speedup measured with valgrind.
Empty tried branches are now replaced when possible with try/except statements, try/finally is simplified in this case. This gives a cleaner tree structure and less verbose C++ code which the compiler threw away, but was strange to have in the first place.
In conditions the or and and were evaluated with Python objects instead of with C++ bool, which was unnecessary overhead.
List contractions got more clever in how they assign from the iterator value.

It now uses a PyObjectTemporary if it’s assigned to multiple values, a PyObjectTempHolder if it’s only assigned once, to something that could raise, or a PyObject * if an exception cannot be raised. This avoids temporary references completely for the common case.

Cleanups

The if, for, and while statements had always empty else nodes which were then also in the generated C++ code as empty branches. No harm to performance, but this got cleaned up.
Some more generated code white space fixes.

New Tests

The CPython 2.7 test suite now also has the doctests extracted to static tests, which improves test coverage for Nuitka again.

This was previously only done for CPython 2.6 test suite, but the test suites are different enough to make this useful, e.g. to discover newly changed behavior like with the lambda generators.
Added Shed Skin 0.7.1 examples as benchmarks, so we can start to compare Nuitka performance in these tests. These will be the focus of numbers for the 0.4.x release series.
Added a micro benchmark to check unpacking behavior. Some of these are needed to prove that a change is an actual improvement, when its effect can go under in noise of in-line vs. no in-line behavior of the C++ compiler.
Added “pybench” benchmark which reveals that Nuitka is for some things much faster, but there are still fields to work on. This version needed changes to stand the speed of Nuitka. These will be subject of a later posting.

Organizational

There is now a “tests/benchmarks/micro” directory to contain tiny benchmarks that just look at a single aspect, but have no other meaning, e.g. the “PyStone” extracts fall into this category.
There is now a --windows-target option that attempts a cross-platform build on Linux to Windows executable. This is using “MingGW-cross-env” cross compilation tool chain. It’s not yet working fully correctly due to the DLL hell problem with the C runtime. I hope to get this right in subsequent releases.
The --execute option uses wine to execute the binary if it’s a cross-compile for windows.
Native windows build is recognized and handled with MinGW 4.5, the VC++ is not supported yet due to missing C++0x support.
The basic test suite ran with Windows so far only and some adaptations were necessary. Windows new lines are now ignored in difference check, and addresses under Windows are upper case, small things.

Numbers

python 2.6:

Pystone(1.1) time for 50000 passes = 0.65
This machine benchmarks at 76923.1 pystones/second

Nuitka 0.3.8 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.27
This machine benchmarks at 185185 pystones/second

This is a 140% speed increase of 0.3.8 compared to CPython, up from 132% compared to the previous release.

Nuitka Release 0.3.7

2011-03-19T14:05:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This is about the new release with focus on performance and cleanups. It indicates significant progress with the milestone this release series really is about as it adds a compiled_method type.

So far functions, generator function, generator expressions were compiled objects, but in the context of classes, functions were wrapped in CPython instancemethod objects. The new compiled_method is specifically designed for wrapping compiled_function and therefore more efficient at it.

Bug Fixes

When using Python or Nuitka.py to execute some script, the exit code in case of “file not found” was not the same as CPython. It should be 2, not 1.
The exit code of the created programs (--deep mode) in case of an uncaught exception was 0, now it an error exit with value 1, like CPython does it.
Exception tracebacks created inside with statements could contain duplicate lines, this was corrected.

Optimization

Global variable assignments now also use assign0 where no reference exists.

The assignment code for module variables is actually faster if it needs not drop the reference, but clearly the code shouldn’t bother to take it on the outside just for that. This variant existed, but wasn’t used as much so far.
The instance method objects are now Nuitka’s own compiled type too. This should make things slightly faster by itself.
Our new compiled method objects support dedicated method parsing code, where self is passed directly, allowing to make calls taking a fast path in parameter parsing.

This avoids allocating/freeing a tuple object per method call, while reduced 3% ticks in “PyStone” benchmark, so that’s significant.
Solved a TODO of BUILTIN_RANGE to change it to pre-allocating the list in the final size as we normally do everywhere else. This was a tick reduction of 0.4% in “PyStone” benchmark, but the measurement method normalizes on loop speed, so it’s not visible in the numbers output.
Parameter variables cannot possibly be uninitialized at creation and most often they are never subject to a del statement. Adding dedicated C++ variable classes gave a big speedup, around 3% of “PyStone” benchmark ticks.
Some abstract object operations were re-implemented, which allows to avoid function calls e.g. in the ITERATOR_NEXT case, this gave a few percent on “PyStone” as well.

Cleanups

New package nuitka.codegen to contain all code generation related stuff, moved nuitka.templates to nuitka.codegen.templates as part of that.
Inside the nuitka.codegen package the MainControl module now longer reaches into Generator for simple things, but goes through CodeGeneration for everything now.
The Generator module uses almost no tree nodes anymore, but instead gets information passed in function calls. This allows for a cleanup of the interface towards CodeGeneration. Gives a cleaner view on the C++ code generation, and generally furthers the goal of other than C++ language backends.
More “PyLint” work, many of the reported warnings have been addressed, but it’s not yet happy.
Defaults for yield and return are None and these values are now already added (as constants) during tree building so that no such special cases need to be dealt with in CodeGeneration and future analysis steps.
Parameter parsing code has been unified even further, now the whole entry point is generated by one of the function in the new nuitka.codegen.ParameterParsing module.
Split variable, exception, built-in helper classes into separate header files.

New Tests

The exit codes of CPython execution and Nuitka compiled programs are now compared as well.
Errors messages of methods are now covered by the ParameterErrors test as well.

Organizational

A new script “benchmark.sh” (now called “run-valgrind.py”) script now starts “kcachegrind” to display the valgrind result directly.

One can now use it to execute a test and inspect valgrind information right away, then improve it. Very useful to discover methods for improvements, test them, then refine some more.
The “check-release.sh” script needs to unset NUITKA_EXTRA_OPTIONS or else the reflection test will trip over the changed output paths.

Numbers

python 2.6:

Pystone(1.1) time for 50000 passes = 0.65
This machine benchmarks at 76923.1 pystones/second

Nuitka 0.3.7 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.28
This machine benchmarks at 178571 pystones/second

This is a 132% speed of 0.3.7 compared to CPython, up from 109% compare to the previous release. This is a another small increase, that can be fully attributed to milestone 2 measures, i.e. not analysis, but purely more efficient C++ code generation and the new compiled method type.

One can now safely assume that it is at least twice as fast, but I will try and get the PyPy or Shedskin test suite to run as benchmarks to prove it.

No milestone 3 work in this release. I believe it’s best to finish with milestone 2 first, because these are quite universal gains that we should have covered.

Nuitka Release 0.3.6

2011-02-28T21:45:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

The major point this for this release is cleanup work, and generally bug fixes, esp. in the field of importing. This release cleans up many small open ends of Nuitka, closing quite a bunch of consistency TODO items, and then aims at cleaner structures internally, so optimization analysis shall become “easy”. It is a correctness and framework release, not a performance improvement at all.

Bug Fixes

Imports were not respecting the level yet. Code like this was not working, now it is:
```
from .. import something
```
Absolute and relative imports were e.g. both tried all the time, now if you specify absolute or relative imports, it will be attempted in the same way than CPython does. This can make a difference with compatibility.
Functions with a “locals dict” (using locals built-in or exec statement) were not 100% compatible in the way the locals dictionary was updated, this got fixed. It seems that directly updating a dict is not what CPython does at all, instead it only pushes things to the dictionary, when it believes it has to. Nuitka now does the same thing, making it faster and more compatible at the same time with these kind of corner cases.
Nested packages didn’t work, they do now. Nuitka itself is now successfully using nested packages (e.g. nuitka.transform.optimizations)

New Features

The --lto option becomes usable. It’s not measurably faster immediately, and it requires g++ 4.6 to be available, but then it at least creates smaller binaries and may provide more optimization in the future.

Optimization

Exceptions raised by pre-computed built-ins, unpacking, etc. are now transformed to raising the exception statically.

Cleanups

There is now a getVariableForClosure that a variable provider can use. Before that it guessed from getVariableForReference or getVariableForAssignment what might be the intention. This makes some corner cases easier.
Classes, functions and lambdas now also have separate builder and body nodes, which enabled to make getSameScopeNodes() really simple. Either something has children which are all in a new scope or it has them in the same scope.
Twisted workarounds like TransitiveProvider are no longer needed, because class builder and class body were separated.
New packages nuitka.transform.optimizations and nuitka.transform.finalizations, where the first was nuitka.optimizations before. There is also code in nuitka.transform that was previously in a dedicated module. This allowed to move a lot of displaced code.
TreeBuilding now has fast paths for all 3 forms, things that need a “provider”, “node”, and “source_ref”; things that need “node” and “source_ref”; things that need nothing at all, e.g. pass.
Variables now avoid building duplicated instances, but instead share one. Better for analysis of them.

New Tests

The Python 2.7 test suite is no longer run with Python 2.6 as it will just crash with the same exception all the time, there is no importlib in 2.6, but every test is using that through test_support.
Nested packages are now covered with tests too.
Imports of upper level packages are covered now too.

Organizational

Updated the “README.txt” with the current plan on optimization.

Numbers

python 2.6:

Pystone(1.1) time for 50000 passes = 0.65
This machine benchmarks at 76923.1 pystones/second

Nuitka 0.3.6 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.31
This machine benchmarks at 161290 pystones/second

This is 109% for 0.3.6, but no change from the previous release. No surprise, because no new effective new optimization means have been implemented. Stay tuned for future release for actual progress.

Nuitka Release 0.3.5

2011-01-22T14:52:00+01:00

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.

This new release of Nuitka is an overall improvement on many fronts, there is no real focus this time, likely due to the long time it was in the making.

The major points are more optimization work, largely enhanced import handling and another improvement on the performance side. But there are also many bug fixes, more test coverage, usability and compatibility.

Something esp. noteworthy to me and valued is that many important changes were performed or at least triggered by Nicolas Dumazet, who contributed a lot of high quality commits as you can see from the gitweb history. He appears to try and compile Mercurial and Nuitka, and this resulted in important contributions.

Bug Fixes

Nicolas found a reference counting bug with nested parameter calls. Where a function had parameters of the form a, (b,c) it could crash. This got fixed and covered with a reference count test.
Another reference count problem when accessing the locals dictionary was corrected.
Values 0.0 and -0.0 were treated as the same. They are not though, they have a different sign that should not get lost.
Nested contractions didn’t work correctly, when the contraction was to iterate over another contraction which needs a closure. The problem was addressing by splitting the building of a contraction from the body of the contraction, so that these are now 2 nodes, making it easy for the closure handling to get things right.
Global statements in function with local exec() would still use the value from the locals dictionary. Nuitka is now compatible to CPython with this too.
Nicolas fixed problems with modules of the same name inside different packages. We now use the full name including parent package names for code generation and look-ups.
The __module__ attribute of classes was only set after the class was created. Now it is already available in the class body.
The __doc__ attribute of classes was not set at all. Now it is.
The relative import inside nested packages now works correctly. With Nicolas moving all of Nuitka to a package, the compile itself exposed many weaknesses.
A local re-raise of an exception didn’t have the original line attached but the re-raise statement line.

New Features

Modules and packages have been unified. Packages can now also have code in “__init__.py” and then it will be executed when the package is imported.
Nicolas added the ability to create deep output directory structures without having to create them beforehand. This makes --output-dir=some/deep/path usable.
Parallel build by Scons was added as an option and enabled by default, which enhances scalability for --deep compilations a lot.
Nicolas enhanced the CPU count detection used for the parallel build. Turned out that multithreading.cpu_count() doesn’t give us the number of available cores, so he contributed code to determine that.
Support for upcoming g++ 4.6 has been added. The use of the new option --lto has been been prepared, but right now it appears that the C++ compiler will need more fixes, before we can this feature with Nuitka.
The --display-tree feature got an overhaul and now displays the node tree along with the source code. It puts the cursor on the line of the node you selected. Unfortunately I cannot get it to work two-way yet. I will ask for help with this in a separate posting as we can really use a “python-qt” expert it seems.
Added meaningful error messages in the “file not found” case. Previously I just didn’t care, but we sort of approach end user usability with this.

Optimization

Added optimization for the built-in range() which otherwise requires a module and builtin module lookup, then parameter parsing. Now this is much faster with Nuitka and small ranges (less than 256 values) are converted to constants directly, avoiding run time overhead entirely.
Code for re-raise statements now use a simple re-throw of the exception where possible, and only do the hard work where the re-throw is not inside an exception handler.
Constant folding of operations and comparisons is now performed if the operands are constants.
Values of some built-ins are pre-computed if the operands are constants.
The value of module attribute __name__ is replaced by a constant unless it is assigned to. This is the first sign of upcoming constant propagation, even if only a weak one.
Conditional statement and/or their branches are eliminated where constant conditions allow it.

Cleanups

Nicolas moved the Nuitka source code to its own nuitka package. That is going to make packaging it a lot easier and allows cleaner code.
Nicolas introduced a fast path in the tree building which often delegates (or should do that) to a function. This reduced a lot of the dispatching code and highlights more clearly where such is missing right now.
Together we worked on the line length issues of Nuitka. We agreed on a style and very long lines will vanish from Nuitka with time. Thanks for pushing me there.
Nicolas also did provide many style fixes and general improvements, e.g. using PyObjectTemporary in more places in the C++ code, or not using str.find where x in y is a better choice.
The node structure got cleaned up towards the direction that assignments always have an assignment as a child.

A function definition, or a class definition, are effectively assignments, and in order to not have to treat this as special cases everywhere, they need to have assignment targets as child nodes.

Without such changes, optimization will have to take too many things into account. This is not yet completed.
Nicolas merged some node tree building functions that previously handled deletion and assigning differently, giving us better code reuse.
The constants code generation was moved to a __constants.cpp where it doesn’t make __main__.cpp so much harder to read anymore.
The module declarations have been moved to their own header files.
Nicolas cleaned up the scripts used to test Nuitka big time, removing repetitive code and improving the logic. Very much appreciated.
Nicolas also documented a things in the Nuitka source code or got me to document things that looked strange, but have reasons behind it.
Nicolas solved the TODO related to built-in module accesses. These will now be way faster than before.
Nicolas also solved the TODO related to the performance of “locals dict” variable accesses.
Generator.py no longer contains classes. The Contexts objects are supposed to contain the state, and as such the generator objects never made much sense.
Also with the help of Scons community, I figured out how to avoid having object files inside the src directory of Nuitka. That should also help packaging, now all build products go to the .build directory as they should.
The vertical white space of the generated C++ got a few cleanups, trailing/leading new line is more consistent now, and there were some assertions added that it doesn’t happen.

New Tests

The CPython 2.6 tests are now also run by CPython 2.7 and the other way around and need to report the same test failure reports, which found a couple of issues.
Now the test suite is run with and without --debug mode.
Basic tests got extended to cover more topics and catch more issues.
Program tests got extended to cover code in packages.
Added more exec scope tests. Currently inlining of exec statements is disabled though, because it requires entirely different rules to be done right, it has been pushed back to the next release.

Organizational

The g++-nuitka script is no more. With the help of the Scons community, this is now performed inside the scons and only once instead of each time for every C++ file.
When using --debug, the generated C++ is compiled with -Wall and -Werror so that some form of bugs in the generated C++ code will be detected immediately. This found a few issues already.
There is a new git merge policy in place. Basically it says, that if you submit me a pull request, that I will deal with it before publishing anything new, so you can rely on the current git to provide you a good base to work on. I am doing more frequent pre-releases already and I would like to merge from your git.
The “README.txt” was updated to reflect current optimization status and plans. There is still a lot to do before constant propagation can work, but this explains things a bit better now. I hope to expand this more and more with time.
There is now a “misc/clean-up.sh” script that prints the commands to erase all the temporary files sticking around in the source tree.

That is for you if you like me, have other directories inside, ignored, that you don’t want to delete.
Then there is now a script that prints all source filenames, so you can more easily open them all in your editor.
And very important, there is now a “check-release.sh” script that performs all the tests I think should be done before making a release.
Pylint got more happy with the current Nuitka source. In some places, I added comments where rules should be granted exceptions.

Numbers

python 2.6:

Pystone(1.1) time for 50000 passes = 0.65
This machine benchmarks at 76923.1 pystones/second

Nuitka 0.3.5 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.31
This machine benchmarks at 161290 pystones/second

This is 109% for 0.3.5, up from 91% before.

Overall this release is primarily an improvement in the domain of compatibility and contains important bug and feature fixes to the users. The optimization framework only makes a first showing of with the framework to organize them. There is still work to do to migrate optimization previously present

It will take more time before we will see effect from these. I believe that even more cleanups of TreeBuilding, Nodes and CodeGeneration will be required, before everything is in place for the big jump in performance numbers. But still, passing 100% feels good. Time to rejoice.

Quiz Question

2011-01-02T13:37:00+01:00

Say you have the following code:

assert type(s) is str
x = float(s)
if x != x:
    print "Bad bad float!"

What value of “s” and then “x” can make the code complain? Do you see the really bad side of it?

The answer is in the next paragraph, so stop reading if you want to find out yourself.

Solution

The correct answer is that there is one float that is not equal to itself and that is float(“nan”). Which I find terrible. It is so bad, it spoils set, dict, and everything there is. Any container that has it inside is no longer equal to itself.

Surprised? I was too! I only learned it while doing my Python compiler Nuitka and I made it a separate posting, because it really surprised me how this could possibly happen. A builtin type that breaks fundamental assumptions like “x == x”.

Nuitka Blog - Posted in 2011

Nuitka Release 0.3.16

Bug Fixes

Optimization

New Features

Organizational

Cleanups

New Tests

Summary

Nuitka Release 0.3.15

Bug Fixes

New Features

Organizational

Optimization

Cleanups

New Tests

Summary

Nuitka Debian Package and Windows Installer

Nuitka Release 0.3.14

Bug Fixes

New Features

Optimization

Cleanups

New Tests

Organizational

Summary

Nuitka Release 0.3.13

Bug Fixes

Optimization

Cleanups

New Tests

Organizational

Summary

Putting a conference T-shirt to good use

Nuitka Release 0.3.12

Bug Fixes

Cleanups

New Tests

Organizational

Summary

Nuitka - PyCON DE Video

Cat update

My 7yr old fotographer

PyCON DE 2011 - My Report

Food

Social Event

Leipzig

Site

My own Presentation

Contacts

Cython / lxml

PyHasse

Debian

General

Keynotes

Lets repeat that

Nuitka git-flow

Example

Benefits of the new model

Uses of Feature Branch

General Picture

Nuitka Release 0.3.11

Bug Fixes

Optimization

Cleanups

New Tests

Organizational

Numbers

Summary

Going to PyCon DE

Nuitka has a new home - nuitka.net

Nuitka Release 0.3.10

Bug Fixes

Optimization

Cleanups

New Tests

Organizational

Summary

The new cat

Nuitka Release 0.3.9