Nuitka Home and Blog of Kay Hayen

Nuitka Release 0.3.22

kayhayen — Sun, 13 May 2012 11:17:54 +0000

This is to inform you about the new stable release of Nuitka. Please see the page “What is Nuitka?”for clarification of what it is now and what it wants to be.

This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimizations.

It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to compile itself with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it’s quite fine now.

Bug fixes

List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions.
Calling a compiled function in an exception handler cleared the exception on return, it no longer does that.
Reference counter handling with generator “throw” method is now correct.
A module “builtins” conflicted with the handling of the Python builtins module. Those now use different identifiers.

New Features

New “metaclass” syntax for the “class” statement works, and the old “__metaclass__” attribute is properly ignored.

# Metaclass syntax in Python3, illegal in Python2
class X(metaclass = Y ):
    pass

# Metaclass syntax in Python2, no effect in Python3
class X:
    __metaclass__ = Y

Note

The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn’ it. Surely, the support for “__metaclass__” could still live.

# For Python2/3 compatible source, we create a base class that has the metaclass # used and doesn't require making a choice.  CPythonNodeMetaClassBase = NodeCheckMetaClass( "CPythonNodeMetaClassBase", (object, ), {} )

The “–dump-xml” option works with Nuitka running under Python3. This was not previously supported.
Python3 now also has compatible parameter errors and compatible exception error messages.
Python3 has changed scope rules for list contractions (assignments don’t affect outside values) and this is now respected as well.
Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well.

New Optimizations

Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used.
This avoids overhead for the very simple functions. And example of this can be seen here:
```
def simple():
   return 7
```

Optimize “len” builtin for non-constant, but known length values.

An example can be seen here:

# The range isn't constructed at compile time, but we still know its length.
len( range( 10000000 ) )

# The string isn't constructed at compile time, but we still know its length.
len( "*" * 1000 )

# The tuple isn't constructed, instead it's known length is used, and side effects
# are maintained.
len( ( a(), b() ) )

This new optimizations applies to all kinds of container creations and the “range” builtin initially.

Optimize conditions for non-constant, but known truth values.
At this time, known truth values of non-constants means “range” builtin calls with know size and container creations.
An example can be seen here:
```
if ( a, ):
   print "In Branch"
```
It’s clear, that the tuple will be true, we just need to maintain the side effect, which we do.

Optimize “or” and “and” operators for known truth values.

See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can:

# The access and call to "something()" cannot possibly happen
0 and something()

# Can be replaced with "something()", as "1" is true. If it had a side effect, it
# would be maintained.
1 and something()

# The access and call to "something()" cannot possibly happen, the value is already
# decided, it's "1".
1 or something()

# Can be replaced with "something()", as "0" is false. If it had a side effect, it
# would be maintained.
0 or something()

Optimize print arguments to become strings.
The arguments to “print” are now converted to strings at compile time.
```
print 1
```
becomes:
```
print "1"
```
Combine print arguments to single ones.
When multiple strings are printed, these are now combined.
```
print "1+1=", 1+1
```
becomes:
```
print "1+1= 2"
```

Organizational

Enhanced Python3 support, enabling support for most basic tests.
Check files with PyLint in deterministic (alphabetical) order.

Cleanups

Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module.
The try/except/else has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not.
Side effects are now a dedicated node, used in several optimizations to maintain the effect of an expression with known value.

New Tests

Expanded and adapted basic tests to work for Python3 as well.
Added reference count tests for generator functions “throw”, “send”, and “close” methods.
Cover calling a function with “try/except” in an exception handler twice. No test was previously doing that.

Summary

This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away.

There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense.

Nuitka Release 0.3.21

kayhayen — Fri, 13 Apr 2012 08:19:56 +0000

This is to inform you about the new stable release of Nuitka. Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it should be easy to track them.

Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc.

Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete.

Bug fixes

The builtin “next” could causes a program crash when iterating past the end of an iterator. Issue#34. Fixed in 0.3.20.1 already.
The “set” constants could cause a compiler error, as that type was not considered in the “mutable” check yet. Fixed in 0.3.20.2 already.
Performance regression. Optimize expression for exception types caught as well again, this was lost in last release.
Functions that contain “exec”, are supposed to have a writable locals. But when removing that “exec” statement as part of optimizations, this property of the function could get lost.
The so called “overflow functions” are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an “exec” or a star import.
The syntax error for “return” outside of a function, was not given, instead the code returned at run time. Fixed to raise a “SyntaxError” at compile time.

New Optimizations

Avoid “tuple” objects to be created when catching multiple exception types, instead call exception match check function multiple times.
Removal of dead code following “break”, “continue”, “return”, and “raise”. Code that follows these statements, or conditional statements, where all branches end with it.
Note
These may not actually occur often in actual code, but future optimizations may produce them more frequently, and their removal may in turn make other possible optimizations.
Detect module variables as “read only” after all writes have been detected to not be executed as removed. Previously the “read only indicator” was determined only once and then stayed the same.
Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value.
Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still.
Note
An exception is where “len = len” actually does have an impact, because that variable becomes assignable. The “compile itself” test of Nuitka found that to happen with “long” from the “nuitka.__past__” module.
Created Python3 variant of quick “unicode” string access, there was no such thing in the API, but we make the distinction in the source code, so it makes sense to have it.
Created an optimized implementation for the builtin “iter” with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython API.
For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take.
Don’t use a “context” object for generator functions (and generator expressions) that don’t need one. And even if it does to store e.g. the given parameter values, avoid to have a “common context” if there is no closure taken. This avoids useless “malloc” calls and should speed up repeated generator object creation.

Organizational

Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated.
The “–experimental” option is no longer available outside of checkouts of git, and even there not on stable branches (“master”, “hotfix/…”). It only pollutes “–help” output as stable releases have no experimental code options, not even development version will make a difference.
The binary “bin/Nuitka.py” has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway.
The “–python-version” option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values.
Added “–keep-pythonpath” alias for “–execute-with-pythonpath” option, probably easier to remember.
Support “–debug” with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn’t report anything new.
The contents environment variable “CXX” determines the default C++ compiler when set, so that checking with “CXX=g++-4.7 nuitka-python …” has become supported.
The “check-with-pylint” script now has a real command line option to control the display of “TODO” items.

Cleanups

Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead.
```
a = b = c
```
```
_tmp = c
b = _tmp
a = _tmp
```
In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block.
All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed.
Changed unpacking assignments to use temporary variables. Code like this:
```
a, b = c
```
Is handled more like this:
```
_tmp_iter = iter( c )
_tmp1 = next( _tmp_iter )
_tmp2 = next( _tmp_iter )
if not finished( _tmp_iter ):
    raise ValueError( "too many values to unpack" )
a = _tmp1
b = _tmp2
```
In reality, not really “next” is used, as it wouldn’t raise the correct exception for unpacking, and the “finished” check is more condensed into it.
Generally this cleanup allowed that the “AssignTargetTuple” and associated code generation was removed, and in the future value propagation should optimize these “next” and “iter” calls away where possible. At this time, this is not done yet.
Exception handlers assign caught exception value through assignment statement.
Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well.
Exception handlers now explicitly catch more than one type.
Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it’s all.
Contractions are now functions as well.
Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation.
Global is handled during tree building.
Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node.
It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that “global” statement, allowing to remove the node class.
Read only module variable detection integrated to constraint collection.
The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed.

New Tests

Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment.
Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn’t even know it can be done, example:
```
try:
    raise ValueError(1,2)
except ValueError as (a,b):
    print "Unpacking caught exception and unpacked", a, b
```
Will assign “a=1″ and “b=2″.
Added test to cover return statements on module level and class level, they both must give syntax errors.
Cover exceptions from accessing unassigned global names.
Added syntax test to show that star imports do not allow other names to be imported at the same time as well.

Summary

From a user standpoint, Nuitka is still not very useful. But the progress made towards value propagation and type inference is very significant, and makes those appears as if they are achievable “relatively soon” now.

Nuitka Release 0.3.20

kayhayen — Mon, 27 Feb 2012 21:48:30 +0000

This is to inform you about the new stable release of Nuitka. Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

This time there are a few bug fixes and some really major cleanups, lots of new optimizations and preparations for more. And then there is a new compiler clang and a new platform supported. MacOS X appears to work mostly, thanks for the patches from Pete Hunt.

Bug fixes

The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. Issue#30. Fixed in release 0.3.19.1 already.
The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. Issue#31. Fixed in release 0.3.19.1 already.
The “from x import *” attempted to respect “__all__” but failed to do so. Issue#32. Fixed in release 0.3.19.2 already.
The “from x import *” didn’t give a “SyntaxError” when used on Python3. Fixed in release 0.3.19.2 already.
The syntax error messages for “global for function argument name” and “duplicate function argument name” are now identical as well.
Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed.

New Features

Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent.
Enhanced Python3 support for syntax errors, these are now also compatible.
Support for MacOS X was added.
Support for using the “clang” compiler was added, it can be enforced via “–clang” option. Currently this option is mainly intended to allow testing the “MacOS X” support as good as possible under Linux.

New Optimizations

Enhanced all optimizations that previously worked on “constants” to work on “compile time constants” instead. A “compile time constant” can currently also be any form of a builtin name or exception reference. It is intended to expand this in the future.
Added support for builtins “bin”, “oct”, and “hex”, which also can be computed at compile time, if their arguments are compile time constant.
Added support for the “iter” builtin in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster.
Added support for the “next” builtin, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well.
Added support the the “open” builtin in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files.
Optimize the “__debug__” builtin constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided.
Optimize the “Ellipsis” builtin constant as well. It falls in the same category as “True”, “False”, “None”, i.e. names of builtin constants that a singletons.
Added support for anonymous builtin references, i.e. builtins which have names that are not normally accessible. An example is “type(None)” which is not accessible from anywhere. Other examples of such names are “compiled_method_or_function”. Having these as represented internally, and flagged as “compile time constants”, allows the compiler to make more compile time optimizations and to generate more efficient C++ code for it that won’t e.g. call the “type” builtin with “None” as an argument.
All builtin names used in the program are now converted to “builtin name references” in a first step. Unsupported builtins like e.g. “zip”, for which Nuitka has no own code or understanding yet, remained as “module variables”, which made access to them slow, and difficult to recognize.
Added optimization for module attributes “__file__”, “__doc__” and “__package__” if they are read only. It’s the same as “__name__”.
Added optimization for slices and subscripts of “compile time constant” values. These will play a more important role, once value propagation makes them more frequent.

Organizational

Created a “change log” from the previous release announcements. It’s as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it’s so easy to generate the PDF on that yourself.
The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to WordPress as a draft posting or updates it, until it’s release time. Simple, sweet and elegant.

Cleanups

Split out the “nuitka.nodes.Nodes” module into many topic nodes, so that there are now “nuitka.nodes.BoolNodes” or “nuitka.nodes.LoopNodes” to host nodes of similar kinds, so that it is now cleaner.
Split “del” statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same:
```
del a, b.c, d
```
```
del a
del b.c
del d
```
So that’s now represented in the node tree. And even more complex looking cases, like this one, also the same:
```
del a, (b.c, d)
```
This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all.
Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions. Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the “bases” for classes and the “defaults” for functions and make optional.
Refactored the decorator handling to the tree building stage, presenting them as function calls on “function body expression” or class body expression”.
This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimizations, as they practically are now just function calls.
```
@some_classdecorator
class C:
    @staticmethod
    def f():
        pass
```
It’s just a different form of writing things. Nothing requires the implementation of decorators, it’s just functions calls with function bodies before the assignment.
The following is only similar:
```
class C:
    def f():
        pass

    f = staticmethod( f )

C = some_classdecorator( C )
```
It’s only similar, because the assignment to an intermediate value of “C” and “f” is not done, and should an exception be raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already.
The in-place assignments statements are now handled using temporary variable blocks
Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal lookups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future.
The for loop used to have has a “source” expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimizations. Moved it to tree building stage instead, where optimizations can work on it then.
Tree building now generally allows statement sequences to be “None” everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist.
Moved the optimization for “__name__” to compute node of variable references, where it doesn’t need anything complex to replace with the constant value if it’s only read.
Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults.
Made the builtin code more reusable.

New Tests

Added some more diagnostic tests about complex assignment and “del” statements.
Added syntax test for star import on function level, that should fail on Python3.
Added syntax test for duplicate argument name.
Added syntax test for global on a function argument name.

Summary

The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much more consistency there.

Yet, even more work will be needed in the same direction. There may e.g. be work required to cover “with” statements as well. And assignments should never be any more complex than unpacking from a temporary variable.

For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only miniscule progress. The remaining tasks appear all more or less difficult work that I don’t want to touch now.

But overall, it’s fantastic progress that was made. There are still remaining steps, but I believe, we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about.

Award winning cat

kayhayen — Mon, 20 Feb 2012 11:51:15 +0000

The sweetness phase is over. At least in that instant. :-)

Ubuntu Packages for Nuitka

kayhayen — Tue, 31 Jan 2012 07:25:43 +0000

Hello there,

as I previously said, I will make these a separate announcement. Thanks to the NeuroDebian project build infrastructure, there is Nuitka packages for the Ubuntu versions back to Maverick, Natty, and Oneiric.

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

You may use the apt sources to track Nuitka or download the packages from the page where I offer the latest version here.

Note: In fact, it’s 0.3.19.1 now, which includes a hotfix release made after 0.3.19. The same is in Debian Unstable right now, waiting to propagate into Debian Testing, where there is still 0.3.18 living at this time.

I would appreciate feedback on these.

Thanks in advance,
Kay

Nuitka Release 0.3.19

kayhayen — Thu, 26 Jan 2012 19:49:30 +0000

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things are in this.

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

Bug fixes

- The manpages of “nuitka” and “nuitka-python” had no special layout for the option groups and broken whitespace for “–recurse-to” option. Also “–g++-only” was only partially bold. Released as 0.3.18.1 hotfix already.

- The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hotfix already.

- Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of “/” instead of using a “joinpath” call. Released as 0.3.18.3 already.

- A call to the range built-in with no arguments would crash the compiler, see Issue#29. Released as 0.3.18.4 already.

- Compatibility Fix: When rich comparison operators returned false value other “False”, for comparison chains, these would not be used, but “False” instead, see .

The support for “__import__” didn’t cover keyword arguments, these were simply ignored. See Issue#28. Fixed, but no warning is given yet.

New Features

- A new option has been added, one can now specify “–recurse-directory” and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements.

- Added support for the “exec” built-in of Python3, this enables us to run one more basic test, “GlobalStatement.py” with Python3. The test “ExecEval.py” nearly works now.

New Optimizations

- The no arguments “range()” call now optimized into the static CPython exception it raises.

- Parts of comparison chains with constant arguments are now optimized away.

Cleanups

- Simplified the “CPythonExpressionComparison” node, it now always has only 2 operands. If there are more, the so called comparison chain, it’s done via “and” with assignments to temporary variables, which are expressed by a new node type “CPythonExpressionTempVariableRef”. This allowed to remove “expression_temps” from code templates and generation, reducing the overall complexity.

- When executing a module (“–execute” but not “–exe”), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment.

- The calls to the variadic “MAKE_TUPLE” were replaced with calls the MAKE_TUPLExx, that are generated on a as-needed basis. This gives more readable code, because no EVAL_ORDERED_xx is needed at call site anymore.

- Many node classes have moved to new modules in “nuitka.nodes” and grouped by theme. That should make them more accessible.

- The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the “sys.abiflags” and works with Python3.

- The replacing of “.py” in filenames was made more robust. No longer is “str.replace” used, but instead proper means to assure that having “.py” as other parts of the filenames won’t be a trouble.

- Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements.

- As always, some PyLint work, and some minor TODOs were solved.

Organizational

- Added more information to the “Developer Manual”, e.g. documenting the tree changes for “assert” to become a conditional statement with a raise statement, etc.

- The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise. I expect to make a separate announcement with the links to packages soonish.

- Added support to specify the binary under test with a “NUITKA” environment, so the test framework can run with installed version of Nuitka too.

- Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for “os.execl” not propagating exit codes under Windows. See Issue#26 for more information.

- For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the “PATH” or even installed to execute the binary.

New Tests

- The “basic”, “programs”, “syntax”, and “reflected” were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations.

- Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains.

- The “Referencing.py” test was made portable to Python3.

- Cover no arguments “range()” exception as well.

- Added test to demonstrate that “–recurse-directory” actually works. This is using an “__import__” that cannot be predicted at run time (yet).

- The created source package is now tested on pbuilder chroots to be installable and capable of the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn’t break without notice.

Numbers

There are no new numbers. Nuitka should be as fast as it was, 258% speedup:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

Summary

This releases contains many changes. The “temporary variable ref” and “assignment expression” work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. inplace assignments can be reduced to assignments to temporary variables.

While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with “value propagation” working.

These are the real focus. The old comparison chains were a wart. Working, but no way understood by any analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimizations to see through them.

Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of inplace assignments. With an “assignment expression” to a “temporary variable ref”, these become the same as user code using such a variable. There will be more of these to find.

So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the “__import__” related one, were not suitable for hotfix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already.

As always you will find its latest version here.

Yours,
Kay Hayen

Static Site Generator

kayhayen — Thu, 26 Jan 2012 06:32:46 +0000

Hello,

somehow triggered by reading about Mynth, and the re-post of the last release announcement on my feed, just because I added a missing category to the post, I am wondering what it takes to achieve the following:

Edit the pages site as ReST (reStructured text) document with links inside
Edit posts as single ReST files
Have it look decent and provide feeds, proper caching headers, etc.
Generate a static site from with, with more control over what kind of updates makes it into the feed.

I am currently very annoyed, because:

I would normally generate code examples now with “rst2man” and replace the existing WordPress plugin that renders in JavaScript. Except I cannot without re-posting all these articles, or removing the “python” tag from them.
The documentation of Nuitka is in ReST and I even intend to expand it with UML diagrams generated from text. Making the cross to HTML content is difficult.
Some pages only replicate content from the User or Developer Manual, keeping those updated, in sync, etc. is a dull thing to do.
Extending WordPress in Python is not feasible. And in php, I doubt I want to master.

Other stuff in favor of it, is that WordPress or any CMS needs more memory and more time to process than static pages of course. I can’t right now let anyone fork and edit the pages in a practical manner. So while for user manual and developer manual of Nuitka I will get corrections, for the website it’s not happening.

Is Mynth an appropiate intermediate step? Do I really want to learn one of these strange templating languages or even consider software that has no Debian package. I never considered “virtualenv” and “pip” much of options to run software on a website. I guess I am more tied to Debian than to Python still.

Over the last months I have occasionally looked at Pyramids, but it seems overly complex, if all you want is to simplify the authoring. I have seen that e.g. Trac supports ReST as an exception, but that feels too little for a hefty software like that.

I may end up with using ReST to generate HTML which is then uploaded via XMLRPC to WordPress. Won’t give me the “static” benefits. Won’t solve the post update problem. But at least will let me edit more easily.

Yours,
Kay

Nuitka Release 0.3.18

kayhayen — Sat, 14 Jan 2012 10:34:44 +0000

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via “apt-get install nuitka”.

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

The release cycle was too short to have much focus. It merely includes fixes, which were available as hotfixes, and some additional optimizations and node tree cleanups, as well as source cleanups. But

Bug fixes

- Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. Issue#16. Released as 0.3.17a hotfix already.

- Nuitka was considering directories to contain packages that had no “__init__.py” which could lead to errors when it couldn’t find the package later in the compilation process. Released as 0.3.17a hotfix already.

- When providing “locals()” to “exec” statements, this was not making the “locals()” writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn’t consider the actual value “locals()”. Released as 0.3.17b hotfix already.

- Compatibility Fix: When no defaults are given, CPython uses “None” for “func.func_defaults”, but Nuitka had been using “None”.

New Optimizations

- If the condition of assert statements can be predicted, these are now optimized in a static raise or removed.

- For built-in name references, there is now dedicated code to look them up, that doesn’t check the module level at all. Currently these are used in only a few cases though.

- Cleaner code is generated for the simple case of “print” statements. This is not only faster code, it’s also more readable.

Cleanups

- Removed the “CPythonStatementAssert” node. It’s not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with “AssertionError” and the assertion argument. This allowed to remove code and complexity from the subsequent stetps of Nuitka, and enabled existing optimizations to work on assert statements as well.

- Moved builtin exception names and builtin names to a new module “nuitka.Builtins” instead of having in other places. This was previously a bit spread-out and misplaced.

- Added cumulative “tags” to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.

- Enhanced the “visitor” interface to provide more kinds of callbacks, enhanced the way “each scope” visiting is achieved by generalizing is as “child has not tag ‘closure_taker’” and that for every “node that has tag ‘closure_taker’”.

- Moved “SyntaxHighlighting” module to “nuitka.gui” package where it belongs.

- More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that should be white listed.

- Consistently use “nuitka” in test scripts, as there isn’t a “Nuitka.py” on all platforms. The later is scheduled for removal.

- New node for builtin name loopups, which allowed to remove tricks played with adding module variable lookups for “staticmethod” when adding them for “__new__” or module variable lookups for “str” when predicting the result of “type( ‘a’ )”, which was unlikely to cause a problem, but an important TODO item still.

Organizational

- The “Download” page is now finally updated for releases automatically. This closes Issue#7 completely. Up to this release, I had to manually edit that page, but I now mastered the art of upload via XMLRCP and a Python script, so that I don’t loose as much time with editing, checking it, etc.

- Added a “Developer Manual” to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.

- The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages soonish.

- Made sure the test runners worth with bare “python2.6″ as well.

New Tests

- Added some tests intended for type inference development.

Numbers

There are no new numbers. Nuitka should be as fast as it was, 258% speedup:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

Summary

This releases contains not as much changes as others. It’s going to be the stable one for a while. In the mean time, we are going to work on the branch “feature/ctypes_annotation” and rebasing it often while things are still flowing.

The “exec” fix was detected by continued work on the branch “feature/minimize_CPython26_tests_diff” branch, but that work is now complete.

It is being made pretty (many git rebases) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made.

But of course, the real excitement is the the “type inference” work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient. So when I learn something. e.g. “assert” is not special, I apply it to the “develop” branch immediately, to keep the differences as small as possible.

Overall, this is a great day. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.

As always you will find its latest version here.

Yours,
Kay Hayen

Nuitka 0.3.17 Release

kayhayen — Mon, 09 Jan 2012 09:59:24 +0000

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organisational work, and yet again improved compatibility and cleanups. Also huge is the advance in making “–deep” go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one.

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

The release cycle had a focus on improving the quality of the test scripts, the packaging, and generally to prepare the work on “type inference” in a new feature branch. I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via “2to3″ conversion) without trouble. There is still work to do, exceptions don’t seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day.

And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository. I have entered Debian Mentors, and found a potential sponsor who may upload the packages in foreseeable time. There may be more review comments, but I expect that Nuitka may enter Debian proper.

Bug fixes

- Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hotfix already.

- Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hotfix already.

- Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hotfix already.

- Fix, add the missing “staticmethod” decorator to “__new__” methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hotfix already.

New Features

- Enhanced compatibility again, provide enough “co_varnames” in the code objects, so that slicing them up to “code_object.co_argcount” will work. They are needed by “inspect” module and might be used by some decorators as well.

- New options to control the recursion:

–recurse-none (do not warn about not-done recursions)
–recurse-all (recurse to all otherwise warned modules)
–recurse-to (confirm to recurse to those modules)
–recurse-not-to (confirm to not recurse to those modules)

New Optimizations

- The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case.

- Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs.

- Use the pickle “protocol 2″ on CPython2 except for “unicode” strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of “pickle” was not really necessary anymore.

Organizational

- Added a “Developer Manual” to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.

- Improved the “–help” output to use “metavar” where applicable. This makes it more readable for some options.

- Instead of error message, give help output when no module/program file name was given. This should make Nuitka help more convenient.

- Consistently use “#!/usr/bin/env python” for all scripts, this was previously only done for some of them.

- Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for “TODO” items optional.

- All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it’s also already the default for “Python”.

- The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.

- Make the tests run without any dependence on “PATH” to contain the executables of Nuitka. This should make it easier to use.

- Add license texts to 3rd party file that were missing them, apply “licensecheck” results to cleanup Nuitka. Also removed own copyright statement from inline copy of Scons, it had been added by accident only.

- Release the tests that I own as well as the Debian packaging I created under “Apache License 2.0″ which is very liberal, meaning every project should be able to use it.

- Don’t require copyright assignment for contributions anymore, instead only “Apache License 2.0″, the future Nuitka license, so that the code won’t be a problem when changing the license of all of Nuitka to that license.

- Give contributors listed in the user manual an exception to the GPL terms until Nuitka is licensed under “Apache License 2.0″ as well.

- Added an “–experimental” option which can be used to control experimental features, like the one currently being added on “feature/ctypes_annotation”, where “type inference” is currently only activated when that option is given. For this stable release, it does nothing.

- Check the static C++ files of Nuitka with “cppcheck” as well. Didn’t find anything.

- Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit.

Cleanups

- Changed “not” boolean operation to become a normal operator. Changed “and” and “or” boolean operators to a new base class, and making their interface more similar to that of operations.

- Added cumulative “tags” to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.

- Moved “SyntaxHighlighting” module to “nuitka.gui” package where it belongs.

- Consistently use “nuitka” in test scripts, as there isn’t a “Nuitka.py” on all platforms. The later is scheduled for removal.

- Some more PyLint cleanups.

New Tests

- Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail.

- The basic tests can now be run with “PYTHON=python3.2″ and use “2to3″ conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script “check-release”.

- Include the syntax tests in release tests as well.

- Changed many existing tests so that they can run under CPython3 too. Of course this is via “2to3″ conversion.

- Don’t fail if the CPython test suites are not there. Currently they remain largely unpublished, and as such are mostly only available to me (exception, “feature/minimize_CPython26_tests_diff” branch references the CPython2.6 tests repository, but that remains work in progress).

- For the compile itself test: Make the presence of the Scons inline copy optional, the Debian package doesn’t contain it.

- Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python.

- Before releasing, test that the created Debian package builds fine in a minimal Debian “unstable” chroot, and passes all the tests included in the package (“basics”, “syntax”, “programs”, “reflected”). Also many other Debian packaging improvements.

Numbers

There are no new numbers. Nuitka should be as fast as it was, 258% speedup:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

Summary

The “git flow” was used again in this release cycle and proved to be useful not only for hotfix, but also for creating the branch “feature/ctypes_annotation” and rebasing it often while things are still flowing. The few hotfixes didn’t require a new release, but the many organisational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control.

The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn’t regress by accident, so running with CPython3.2 is now part of the normal release tests.

What’s still missing is more “hg” completeness. Only the “co_varnames” work for “inspect” was going in that direction, and this has slowed down. It was more important to make Nuitka’s recursion more accessible with the new options, so that was done first.

And of course, the real excitement is the the “type inference” work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of “ctypes” native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly.

Overall, this feels very good. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.

As always you will find its latest version here.

Yours,
Kay Hayen

Nuitka Release 0.3.16

kayhayen — Sun, 18 Dec 2011 19:24:16 +0000

This is to inform you about the new stable release of Nuitka. This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups.

Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.

The release cycle had a focus on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via “2to3″ conversion) without trouble. I don’t know when, but it seems that it’s going to work one day.

Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It’s still not there, but I am getting more helpful now.

Bug fixes

- Fixed a packaging problem for Linux and x64 platform, the new “swapFiber.S” file for the fiber management was not included. Released as 0.3.15a hotfix already.

- Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hotfix already.

- Fixed an issue with “__import__” and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hotfix already.

- Fixed “eval()” and “locals()” to be supported in generator expressions and contractions too. Released as 0.3.15d hotfix already.

- Fixed the Windows batch files “nuitka.bat” and “nuitka-python.bat” to not output the “rem” statements with the copyright header. Released as 0.3.15d hotfix already.

- Fixed re-raise with “raise”, but without a current exception set. Released as 0.3.15e hotfix already.

- Fixed “vars()” call on the module level, needs to be treated as “globals()”. Released as 0.3.15e hotfix already.

- Fix handling of broken new lines in files. Read the source code in “universal line ending mode”. Released as 0.3.15f hotfix already.

- Fixed handling of constant module attribute “__name__” being replaced. Don’t replace local variables of the same name too. Released as 0.3.15g hotfix already.

- Fixed assigning to “True”, “False” or “None”. There was this old TODO, and some code has compatibility craft that does it. Released as 0.3.15g hotfix already.

- Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hotfix already.

- Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not.

- Fixed non-execution of cleanup codes that e.g. flush “sys.stdout”, by adding “Py_Finalize()”.

- Fix “throw()” method of generator expression objects to not check arguments properly.

- Fix missing fallback to subscript operations for slicing with non-indexable objects.

- Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did.

- Fix, the future spec was not properly preserving the future division flag.

New Optimizations

- The optimization scales now much better, because per-module optimizations only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time.

- The creation of dictionaries from constants is now also optimized.

- The names “True”, “False” and “None” are now converted to constants only when they are read-only module variables.

Organizational

- As a new feature functions now have the “func_defaults” and “__defaults__” attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though.

- The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.

- The “PYTHONPATH” variable is now cleared when immediately executing a compiled binary unless “–execute-with-pythonpath” is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required.

- The inline copy of Scons is not checked with PyLint anymore. We of course don’t care.

- Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious.

- The basic tests can now be run with “PYTHON=python3.2″ and use “2to3″ conversion in that case.

Cleanups

- Moved “tags” to a separate module, make optimizations emit only documented tags, checked against the list of allowed ones.

- The Debian package has seen lots of improvements, to make it “lintian clean”, even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc.

- Use “os.path.join” is more of the test code.

- Some more PyLint cleanups.

New Tests

- There is now a “Crasher” test, for tests that crashed Nuitka previously.

- Added a program test where the imported module does a “sys.exit()” and make sure it really doesn’t continue after the “SystemExit” exception that creates.

- Cover the type of “__builtins__” in the main program and in imported modules in tests too. It’s funny and differs between module and dict in CPython2.

- Cover a final print without newline in the test. Should still receive a newline, which only happens when “Py_Finalize()” is called.

- Added test with functions that makes a “raise” without an exception set.

- Cover the calling of “vars()” on module level too.

- Cover the use of eval in contractions and generator expressions too.

- Cover “func_defaults” and “__default__” attributes for a function too.

- Added test function with two “raise” in an exception handler, so that one becomes dead code and removed without the crash.

Numbers

There are no new numbers. Nuitka should be as fast as it was, 258% speedup:

python 2.6:

Pystone(1.1) time for 50000 passes = 0.48
This machine benchmarks at 104167 pystones/second

Nuitka 0.3.11 (driven by python 2.6):

Pystone(1.1) time for 50000 passes = 0.19
This machine benchmarks at 263158 pystones/second

Summary

The “git flow” was really great in this release cycle. There were many hotfix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects.

This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority.

The work on Python3 is a bit strange. I don’t need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can’t really stop to work on it. It has that appeal of small fixups here and there, and then something else works too. It’s like when I was struggling with Nuitka to pass the CPython2 unit tests the first time. It’s fun. And then it finds real actual bugs that apply to CPython2 too. Not doing “Py_Finalize” (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the early “Python3″ is supported, the more benefit from increased test covered.

What’s missing is more “hg” completeness. I think only the “raise” without exception set and the “func_defaults” issue were going into its direction, but it won’t be enough yet. Before that, the plan is to make Nuitka’s options to control recursion more accessible. Right now it requires you to embed all dist-packages or None. And there is no way to specify modules that should be included even if not directly used, which would be important for the hg plugins to use.

Overall, this feels very good. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.

As always you will find its latest version here.

Yours,
Kay Hayen