Nuitka Home

Nuitka Release 0.4.3

Sat, 18 May 2013 10:17:00 GMT

This is to inform you about the new stable release of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka.

New Features

Support for new compiler: Microsoft Visual C++.

You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows.
Support for NetBSD.

Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic "fibers" implementation.
Support for Python3 under Windows too.

Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to" C:Python26" or "C:Python27" for Nuitka to be able to find it.
Enhanced Python 3.3 compatibility.

The support the newest version of Python has been extended, improving compatibility for many minor corner cases.
Added warning when a user compiles a module and executes it immediately when that references __name__.

Because very likely the intention was to create an executable. And esp. if there is code like this:
```
if __name__ == "__main__":
   main()
```
In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command
```
nuitka --execute module
```
is behavioral more like

python -c "import module"

and that was a trap for new users.
All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don't have to implement for specific architectures anymore.

Bug Fixes

Dictionary creation was not fully compatible.

As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn't observe this before, and it's likely the new dictionary implementation that finds it.

Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing.
Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was.
Using a __future__ import inside a function was giving an assertion, instead of the proper syntax error.
Python3: Do not set the attributes sys.exc_type, sys.exc_value, sys.exc_traceback.
Python3: Annotations of function worked only as long as their definition was not referring to local variables.

New Optimization

Calls with no positional arguments are now using the faster call methods.

The generated C++ code was using the () constant at call site, when doing calls that use no positional arguments, which is of course useless.
For Windows now uses OS "Fibers" for Nuitka "Fibers".

Using threads for fibers was causing only overhead and with this API, MSVC had less issues too.

Organizational

Accepting Donations via Paypal, please support funding travels, website, etc.
The User Manual has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated.
The Changelog has been improved for consistency, wordings, and styles.
Nuitka is now available on the social code platforms as well
Removed "clean-up.sh", which is practically useless, as tests now clean up after themselves reasonably, and with git clean -dfx working better.
Removed "create-environment.sh" script, which was only setting the PATH variable, which is not necessary.
Added check-with-pylint --emacs option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint.
Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated.

Cleanups

Removed useless manifest and resource file creation under Windows.

Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it.
PyLint massive cleanups and annotations bringing down the number of warnings by a lot.
Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes.
Do not track needed tuple, list, and dict creation code variants in context, but e.g. in nuitka.codegen.TupleCodes module instead.
Introduced an "internal" module to host the complex call helper functions, instead of just adding it to any module that first uses it.

New Tests

Added basic tests for order evaluation, where there currently were None.
Added support for "2to3" execution under Windows too, so we can run tests for Python3 installations too.

Summary

The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now.

Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization.

Nuitka on Github, Bitbucket and Gitorious

Fri, 26 Apr 2013 07:17:16 GMT

This is to let you know, that I have finally carved in and created accounts on all those "social" platforms, that deal with code. So far I had resisted their terms (you pay us when we get sued), but I have given up on that stance, it's likely irrelevant, as Nuitka won't get sued, and if it is, those are not the issues.

And I don't want to feel too old school it seems. And who knows, maybe this helps Nuitka to get more contributions.

So these are the landing pages:

Bitbucket

I kind of like the looks of it the most. Also it's very popular for Python people due to its Mercurial ties. I also liked that it imported my Nuitka repository easily.
Github

Is not being on Github allowed these days? Anyway, from ease of use, the interface looks more powerful.
Gitorious

The interface and looks definitely are worst. But it's Free Software, and an open platform, so it has to be in the list.

Let me know what you think. I am going to push 'develop' at pre-release and 'master' at release time, automatically from the release scripts into all these. Hopefully that way, they turn out to be time savers no wasters.

And yes, I couldn't decide on one. Since I fancy Free Software, Gitorious probably would have won, but it would also miss the point of expanding the reach. And for as long as I don't use their issue tracking or Wikis, it appears to be not necessary to make that decision.

Note

The preferred for use, and most up to date repository will remain the one from Downloads.

So fork away, like it, and stuff. Awaiting your pull requests...

Going to Europython 2013

Tue, 16 Apr 2013 09:58:34 GMT

I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.

Contents

Things to talk about

Status

Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.

What I would like to get done until then:

Win64

One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.
Threading

And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.

Since it's not totally a priority right now, one of the next releases will support it, likely before the conference.

But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.

Last years questions

Builtins

For instance, writing to built-ins, what will/did happen.

Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.

This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it's not effective in others (len, str).

Good news is that we got contributed a "compiled built-ins" code, where we now will be able to see such writes. Now it's only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.

The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.

Note

Currently Nuitka does nothing of this, but it will.

Debugger - pdb

The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.

Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.

Threading

I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don't like them either, so my experience was non-existing.

But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.

C++ to Python gaps

I consider all of these more or less solved.

Well maybe except recently arised issues with MSVC for "function calls". It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.

But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.

Performance

There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).

And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.

Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.

But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it's the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.

Demos

The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.

And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.

If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.

Future Work

Generally the talk will be more directed at the future, although this is kind of a dark area now. That's its nature I guess.

SSA

The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say "Python is too dynamic (to be statically compiled)" is that Python values may escape to unknown code that changes them very often.

I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.

Compiled Modules

And I believe with "compiled modules" potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.

I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn't have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.

Compiled Locals

Who knows, we might even have a "compiled locals" as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.

But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don't know if it might use it.

Whole Program Analysis

Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.

Perspective

Last time, I spent a lot of time on justification, "why a new project?", "why not work with the others?", what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.

Funding

And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It's probably fair to allow people to make donations if they feel they want to further the project, but don't know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikley though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.

Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these "times of my life". Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.

Support for MSVC upcoming

Wed, 10 Apr 2013 20:30:11 GMT

The good news are lately coming in high density, although I barely have the opportunity to work on Nuitka. Check the page What is Nuitka? for clarification of what it is now and what it wants to be.

It seems my recent call for help was at least partially effective. One the major points has been addressed. In the next release, Visual Studio 2008 and 2010 will both work for Nuitka, right now it's only in git develop branch.

That is really nice, as it will make Nuitka on Windows a whole lot more accessible. Combine that with portable support and it's up to solving problems with distributing on Windows it seems.

The other two things listed there have received no feedback yet. So if you want to help with those, feel free to do so. Nuitka needs you.

Nuitka needs you - a call for help

Wed, 10 Apr 2013 07:51:13 GMT

Update 2

The third point, Windows 64 support has been done as well. Turns out, that I do in fact own only Win64 systems, and with MSVC support in place, only a few portability fixes were needed.

Help with the other point, "re-formulation of yield", would still be more than welcome, and no signs of progress there yet. So stop thinking "you could", enter telling people "you did" mode.

Update

The second point, Fibers implementation for Windows, has been done. Thanks for the help you people provide. The next release will contain it.

Hello everybody,

the Python compiler Nuitka has come an ever longer way, and currently I have quite a bunch of issues, that I believe could well need your help. These are all issues of some important and significance, yet self contained jobs that you may enjoy.

Note

You can check the page What is Nuitka? for clarification of what it is now and what it wants to be.

Python 3.3 - reformulation of yield from needed.

As you can see, covering all the CPython 2.6, 2.7, and 3.2 language features is already something. Also CPython 3.3 is now working on a basic level. Other projects are far, far away from that.

Many language constructs, such as the with statement are re-formulated into other constructs. This makes it possible to work with a simple core for optimization, and to reduce the complexity a lot. For the with statement case, it's changed to try/finally and try/except statements, together with a few temporary variables. Check the Nuitka source of it:

ReformulationWithStatements.py

There is also descriptions of all these re-formulations in the developer manual so you can see how this is done.

Now check PEP 380 for the details of it, and your task would be to come up with a re-formulation of yield from statements to the semantically equivalent.

The benefit is, you are working on a totally self-contained level. The re-formulation only needs to use ast node information, and turn that into Nuitka nodes. The problem will be fully solved this way.

And once we have this, we can pronounce CPython3.3 as fully supported. So if you would like to see this happen, please join.
Windows - MSVC support needs low level code help

update

This is now done.

The support for MSVC is already mostly there. If you execute the environment setting script, and use Nuitka under Windows, it will work. The only problem is that for the generator functions, Nuitka is using a concept called "Fiber", which is basically just switching C stacks, so called co-routines.

It works great for Linux and UNIX. For Windows, Fibers are currently using threads, which wouldn't be as much of a performance problem, because they are very lightweight, it is slightly wasteful only.

But, the bad thing, is with these threads switching and C++ exceptions used to represent Python exceptions, no combination of MSVC options seems to carry the day and keep the exception in the current thread. They are caught in the wrong thread, and make generators misbehave, crash.

This is the only thing that holds back proper MSVC support at this time, and it has been like this for a while. With MinGW, the issue does not exist, but MinGW is a stranger to many, and MSVC will be more efficient code generation, so we would like to have that.

So what it takes is somebody familiar with Windows and its ABI, esp. register usage to write code that swaps the registers around, so we can have multiple C stacks without threads for it too. This may involve messing around with exception jump back pointers as well.

It needs somebody who can handle this and does it for Nuitka, and then it would be well supported. It may involve writing some assembler code. The job is also very well contained, but also requires very specific knowledge, so please join me here. If you think you can do this, do it.
Windows 64 bit

I have no access to Win64. I bought many Windows versions, but not the Win64 variants so far. I understand that there are CPython builds for it, and probably a MinGW matching it, as well as a matching newer MSVC, that may work better for C++ exceptions than the one CPython 2.x is tied to.

Doing that, you would mostly only have to play around with SingleExe.scons, which abstracts the details of it all. One issue you will encounter is that Scons needs a Python2, and other small things. This is mostly only about porting Nuitka, and could be done by anybody with access to Win64 and willing to fiddle around a bit.

So these are 3 different ways you can help Nuitka. Creating the Python compiler is a huge effort. And if you help it, we will sooner arrive at its goals. Join the Nuitka mailing list or contact me directly if you choose to help with any of these issues.

Wait, actually there is another way you can help. Please spread this news to people that you believe might be willing to help. And allow me to plug this, I am doing Nuitka in my spare time. If you feel, you cannot help, but still would like to support it, you can still make donations to allow me to travel to conferences and spend more time on it myself.

Anyway, it would be great if these 3 things got sorted out. Come and join me for this grand vision of the Python compiler.

Yours, Kay

Support for portable (standalone) programs

Sun, 07 Apr 2013 13:52:44 GMT

This post is about a feature often requested, but so far not available feature of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

In forums, and in Google, people are looking at a Python compiler, also as a way of deployment. It should offer what py2exe does, allow installation independent of Python.

Well, for a long time it didn't. But thanks to recent contributions, it's upcoming for the next release, Nuitka 0.4.3, and it's in the current pre-releases.

It works by adding --portable to the command line. So this should work for you:

nuitka-python --recurse-all --portable your-program.py

Right now, it will create a folder "_python" with DLLs, and "_python.zip" with standard library modules used along to the "your-program.exe". Copy these to another machine, without a Python installation, and it will (should) work. Making that statement fully true may need refinements, as some DLL dependencies might not be defined yet.

Note

We may improve it in the future to meld everything into one executable for even easier deployment.

You are more than welcome to experiment with it. To do so, download Nuitka from the download page and give it a roll.

Note

Of course, Nuitka is not about replacing "py2exe" primarily, it's only a side effect of what we do. Our major goal is of course to accelerate Python, but surely nobody minds achieving the two things at the same time.

And while the post is labeled "Windows", this feature also works for Linux at least too. It's just that the lack of Python installations on client systems is far more widespread on this platform.

To me, as this is from a contributor, it's another sign of Nuitka gaining adoption for real usage. My personal "py2exe" experience is practically not existing, I have never used it. And I will only merge the improvements into the Nuitka project as provided by others. My focus for the time to come is of course the compile time and run time optimization.

NetBSD support upcoming

Sat, 06 Apr 2013 12:58:37 GMT

My first real UNIX ever was a NetBSD. That was now about 22 years ago. I am still sentimental about it. I had installed it last about 8 years ago. And I still like it. Back in the days, it was the first UNIX to encounter for me, running on Amiga hardware, first of a friend, then on my own.

Recently, there had been support for Nuitka on FreeBSD added. A lot of people use it on the web, and some want to use Nuitka to improve their Python performance, so this is kind of relevant.

There were issues resolved, but in the end, something was with Clang on FreeBSD 8, that I could for the life of it, not resolve remotely. So I attempted to install it myself. Using "virt-install", these things are a breeze now. I had already done it with CentOS6 before to test the RPM repositories of Nuitka. That "virt-install" is a wonderful thing by itself, making virtualisation somewhat useful. It's only a pity, that I can't just install other qemu support architectures. I would love to checkout Nuitka on PowerPC.

Note

If you could check out Nuitka on other Linux architectures than x86_64, x86, or arm, that would be great.

This is the report of getting NetBSD supported. It was a quite an interesting story that I would like to share with you.

Naivly I was assuming, that it would be just for fun, and that Nuitka will work right away. Little did I know.

On FreeBSD 9 the minimal install medium was chosen, and entered its ports collection, installed git, cloned Nuitka, and ran the tests, successfully right away. Now that is unfair, in the Nuitka there were tons of "Linuxism" already removed. In fact, it had to work, and on the newest FreeBSD (version 9.1) and then it did. Great!

Note

If you would like to add Nuitka to FreeBSD's ports, please do so. It should be really easy now.

On NetBSD, things were unfortunately a little different. I also chose minimal system. After going through "pkg source" boot strap and git install, I cloned Nuitka, and then tried to start it. First off, it couldn't locate "python" at all. I am using /usr/bin/env python already. But Python2 was on the system. I ended up creating the "python" link myself. What I should have done according to "#netbsd" is to install the software, and indeed, python2.7 setup.py install gives an installation of Nuitka that is executable.

Next up, you need to know about "Fibers" in Nuitka. These are used for C co-routines, used to implement Python generators. They have an interface, that is very close to makecontext/swapcontext routines in C.

For ARM and x86_64 Linux we have optimized code, that switches faster, but other platforms, including x86 Linux, use the generic implementation, also because it normally is very fast already.

Now you have to know that since 2001 the interface is deprecated and shall not be used. And next up, is that on NetBSD, makecontext gave a segfault only. So I ran to "#netbsd" and asked.

Now that was a very friendly experience. Of course, I had to give a rationale for using an obsolete interface. It's not quite obvious, why threads wouldn't be a better choice. And maybe they are, but they definitely have more overhead associated, and if they never run at the same time, why use them.

Ultimately it helped to point out, that for a user of 22 years, an interface that is only obsolete for 11 years, is not quite as horrifying as for others.

And they helped me through it. And it turns out, funny thing. For the context to setup, you are allocating a stack to use for the C routine, and you "get" the current context, then you make a new one. All the examples have a certain order of doing it. And my code did it the other way around. No system but NetBSD noticed.

On FreeBSD and Linux, it didn't matter. But it seems, that the needed getcontext call was overwriting my stack pointer now with the current stack. And makecontext deeply hated that, a lot. It was preparing that stack to be used, while it was in usage. Doesn't sound like a good task to give to it, right? My fault truly, because every example on every man page, on all systems, was doing it differently. But then they were also all using arrays from the local stack, so quite obviously that was not real code.

So that was fixed, and all good? No! Next thing was it crashed when free happened in Python on a compiled frame object, in a later part of a test that heavily uses generators. Turns out, malloc information was corrupted. I had to suspect the generic "Fiber" code, but that took me a while to figure out.

And how could my simple malloc and free do that, and make it happen. When I knew that a context would not longer be used (the generator has finished, the generator object deleted, etc), I would look at the context handle stack pointer and free it.

But that pointer changed. Something totally unexpected (by me obviously), but it also explains the earlier problem. For all systems, I had used so far, this pointer was not being changed, and remained the same. So I could free it from there. It worked fine, but not on NetBSD. And it wasn't correct anywhere.

It seems NetBSD is doing something clever, since instead of saving the stack pointer register in a separate area, it saves it to that place originally specified. It's quite obviously an improvement, in that you save the pointer.

It's only bad, that now to make up for this savings, I have added the pointer in a separate field, which won't be changed, so I can free it again. If one needs it again, and that's not unlikely, you have to remember it elsewhere. So maybe that idea is not that clever. But it surely was wrong by me to assume that the provided value would not be touched.

So, these are 2 bugs it found. The wrong order of calls. And the usage of a pointer, that may have been changed. This can only help with other systems, or possibly architecures under Linux.

While this is all description of nasty problems, it's also the report of the solution, and it was big fun. I would also like to compliment "#netbsd" for being very helpful and friendly with my porting of Nuitka. I highly enjoyed doing so. It was a lot of fun. I know that it's probably on a very tiny amount of people that uses both NetBSD and Nuitka, but still.

If this Nuitka project were about market share, it wouldn't exist. And I can work for market share on another day.

Nuitka Release 0.4.2

Fri, 29 Mar 2013 14:32:00 GMT

This is to inform you about the new stable release of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the performance diagrams got expanded.

New Features

Support for FreeBSD.

Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process.
New option for warning about compile time detected exception raises.

Nuitka can now warn about exceptions that will be raised at run time.
Basic Python3.3 support.

The test suite of CPython3.2 passes and fails in a compatible way. New feature "yield from" is not yet supported, and the improved argument parsing error messages are not implemented yet.

Bug Fixes

Nuitka already supported compilation of "main directories", i.e. directories with a "__main__.py" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe"
```
# ls directory
__main__.py

# nuitka --exe directory
# ls
directory directory.exe
```
This makes this usage more obvious, and fixes the older issue Issue#49 for this feature.
Evaluation order of binary operators was not enforced.

Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects Issue#61.
Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.
Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.
```
global a # Not in a function, but on module level. Pointless but legal!
a = 1
```
Effectively these statements can be ignored. Corrects part of Issue#65.
Future imports are only legal when they are at the start of the file. This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of Issue#65.
Raising exceptions from context was leaking references.
```
raise ValueError() from None
```
Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects Issue#76.
Importing the module that became __main__ through the module name, didn't recurse to it.

This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don't share data. Corrects Issue#68.
Compiled method repr leaked references when printed.

When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point. Corrects Issue#81.
The super built-in leaked references to given object.

This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run. Corrects Issue#81.
The raise statement didn't enforce type of third argument.

This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked.
Python3 built-in exceptions were strings instead of exceptions.

A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects Issue#82.
The -nan and nan both exist and make a difference.

A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka. Corrects Issue#75.
Wrong optimization of a == a, a != a, a < a on C++ level.

While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for "==", "!=", and "<".
The sys.executable for nuitka-python --python-version 3.2 was still python.

When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.
Keyword only functions with default values were loosing references to defaults.
```
def f( *, a = X() )
   pass

f()
f() # Can crash, X() should already be released.
```
This is now corrected. Of course, a Python3 only issue.
Pressing CTRL-C didn't generate KeyboardInterrupt in compiled code.

Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.
For with statements with return, break, or continue to leave their body, the __exit__ was not called.
```
with a:      # This called a.__enter__().
    return 2 # This didn't call a.__exit__( None, None, None ).
```
This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing. Corrects Issue#59.
Starting with Python3, absolute imports are now the default.

This was already present for Python3.3, and it turns out that all of Python3 does it.

New Optimization

Constants are now much less often created with pickle module, but created directly.

This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.
```
a = ( "something_special", )
b = "something_special"

assert a[0] is b # Now true
```
This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects.

Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.
Continue statements might be optimized away.

A terminal continue in a loop, was not optimized away:
```
while 1:
    something
    continue   # Now optimized away
```
The trailing continue has no effect and can therefore be removed.
```
while 1:
    something
```
Loops with only break statements are optimized away.
```
while 1:
    break
```
A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.

New Tests

Added tests for the found issues.
Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered.
Running the "CPython3.2" test suite with Python3.3 based Nuitka works and found a few minor issues.

Organizational

The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file.

The page got improved with logos for the distributions.
Added "ownssh" as contributor.
Revamped the User Manual in terms of layout, structure, and content.

Summary

This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.

Nuitka RPMs (RHEL, CentOS, F17, F18, openSUSE)

Sat, 16 Mar 2013 18:44:45 GMT

Thanks to contributor, Nuitka now has a proper RPM spec file. And using openSUSE Build Service, we can now enjoy RPMs for all major distributions.

Check out the revamped download page.

Note

At the time of this writing, the "stable" Nuitka 0.4.1 link points to a 0.4.2 pre-release. That is only until 0.4.2 final will be released.

Thanks for making this possible, very much appreciated.

Nuitka Release 0.4.1

Tue, 05 Mar 2013 22:13:00 GMT

This is to inform you about the new stable release of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.

This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree.

Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes.

New Features

Python 3.3 experimental support
- Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface.
- Added support for __qualname__ to classes and functions.
- Small compatibility changes. Some exceptions changed, absolute imports are now default, etc.
- For comparison tests, the hash randomization is disabled.
Python 3.2 support has been expanded.

The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code.

Bug fixes

Default values were not "is" identical.
```
def defaultKeepsIdentity( arg = "str_value" ):
    print arg is "str_value"

defaultKeepsIdentity()
```
This now prints "True" as it does with CPython. The solution is actually a general code optimization, see below. Issue#55
Usage of unicode built-in with more than one argument could corrupt the encoding argument string.

An implementation error of the unicode was releasing references to arguments converted to default encoding, which could corrupt it.
Assigning Python3 function annotations could cause a segmentation fault.

New Optimization

Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization.

# Now optimizing:
raise TypeError, 1/0
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
(1/0).something
# into (minus normalization):
raise ZeroDivisionError, "integer division or modulo by zero"

# Now optimizing:
function( a, 1/0 ).something
# into (minus normalization), notice the side effects of first checking
# function and a as names to be defined, these may be removed only if
# they can be demonstrated to have no effect.
function
a
raise ZeroDivisionError, "integer division or modulo by zero"

There is more examples, where the raise propagation is new, but you get the idea.

Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things.
```
# This was already optimized, as it's a compile time constant.
a if ( "a", ) else b
a if True else b

# These are now optimized, as their truth value is known.
a if ( c, ) else b
a if not (c, ) else b
```
This is simply taking advantage of infrastructure that now exists. Each node kind can overload "getTruthValue" and benefit from it. Help would be welcome to review which ones can be added.
Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused.
Code generation for constants now shares element values used in tuples.

The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up.

Cleanups

Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection.
The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information.
The class names of nodes were stripped from the "CPython" prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it's also not important enough to dominate the class name.
The re-formulations performed in tree building have moved out of the "Building" module, into names "ReformulationClasses" e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it's much easier to find the content of interest now.
Added new re-formulation of print statements. The conversion to strings is now made explicit in the node tree.

New Tests

Added test to cover default value identity.

Organizational

The upload of Nuitka to PyPI has been repaired and now properly displays project information again.

Summary

The quicker release is mostly a consolidation effort, without actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization.