This is to let you know, that I have finally carved in and created accounts on all those "social" platforms, that deal with code. So far I had resisted their terms (you pay us when we get sued), but I have given up on that stance, it's likely irrelevant, as Nuitka won't get sued, and if it is, those are not the issues.
And I don't want to feel too old school it seems. And who knows, maybe this helps Nuitka to get more contributions.
So these are the landing pages:
I kind of like the looks of it the most. Also it's very popular for Python people due to its Mercurial ties. I also liked that it imported my Nuitka repository easily.
Is not being on Github allowed these days? Anyway, from ease of use, the interface looks more powerful.
The interface and looks definitely are worst. But it's Free Software, and an open platform, so it has to be in the list.
Let me know what you think. I am going to push 'develop' at pre-release and 'master' at release time, automatically from the release scripts into all these. Hopefully that way, they turn out to be time savers no wasters.
And yes, I couldn't decide on one. Since I fancy Free Software, Gitorious probably would have won, but it would also miss the point of expanding the reach. And for as long as I don't use their issue tracking or Wikis, it appears to be not necessary to make that decision.
The preferred for use, and most up to date repository will remain the one from Downloads.
So fork away, like it, and stuff. Awaiting your pull requests...
I am going to the wonderful city of Florence, and Europython 2013 and make a presentation there. This time, I am not introducing Nuitka anymore, I did that in 2012, this time, I will try and dive into static optimization and try to convey the message why I believe it is possible.
Only briefly this time, since I will be able to say that all current Python versions are fully supported (surely if you help me with Python3.3 yield from), all major platforms now. One important milestone has been reached meanwhile, and work on far reaching compile time optimizations is happening. As this should be quite obvious stuff, I am going to keep that short.
What I would like to get done until then:
One remaining area of work to achieve Win64 support, which is almost complete. I epxect one of the next releases to support it.
And of course there is threading, which was the one real major weakness present last time, which appears only short of pronouncing break-through. At least one user is using Nuitka with many threads operationally already. I just have to re-activate the thread using tests CPython that I disabled. Seems I only have to convince myself of it too.
Since it's not totally a priority right now, one of the next releases will support it, likely before the conference.
But as you see. Completion all around is there or at least in sight. Kind of worked on this nearby.
For instance, writing to built-ins, what will/did happen.
Changing builtins can be done in two ways. One is to set the value on the module level, which is something that has always worked. The other is writing to builtins module.
This is something that is OK for Nuitka in some cases (__import__, open ) and handled by it at run time. And it's not effective in others (len, str).
Good news is that we got contributed a "compiled built-ins" code, where we now will be able to see such writes. Now it's only used to not check every time for changes, but to know them (pull vs. push). But we will also use it and trigger RuntimeError exceptions for things we cannot handle when we only learn of it at run time.
The other element to address is, it of course whole program analysis. When Nuitka sees the write to builtins.str, it may very well consider it. The distinction between initial and current builtin values, and the optimization of it, that will be interesting to cover.
Currently Nuitka does nothing of this, but it will.
The compiled binaries work the same as the normal Python code. So you will be able to simply use pdb on it instead.
Interacting with pdb is not totally out of reach, but kind of pointless mostly, unless you need to attach to long running operational programs. For now that use case is not supported though.
I learned a whole lot about threading. Also thanks to the kind people of Stackless Python, who explained things to me. I am still amazed at how little I did know of these things, and still went so far. In my industry, threads are considered not allowed, and I personally don't like them either, so my experience was non-existing.
But in the mean time, I managed to come up with ideas that appear to work, and if I implement the full design, it will even be more efficient than anything.
I consider all of these more or less solved.
Well maybe except recently arised issues with MSVC for "function calls". It appears that compiler highlights a weakness in one of my approaches. Nuitka so far only changed the order of declaration and call arguments around, which is kind of transparent.
But MSVC actively takes liberty to calculate function arguments as it sees fit. The fix for it, is now totally different and should be highly portable and even compliant to C++.
There are still slow exceptions. I would like to avoid raising C++ exceptions in the future, because they are so slow (Python exceptions are much faster).
And diagrams, I would like to have a whole lot more of these. Since I dropped speedcenter, I am making actual progress there. I hope to have enough to show at the conference, where the actual strength and weakness currently is.
Since I am getting closer to pronouncing Nuitka useful. I surely believe, I need to answer the performance question in larger detail. And of course, I need now a better idea, what impact measures have.
But as this is a lot of work, I doubt that I will be all that perfect by then, my goal is to have a comparison with Shedskin. No matter how unfair it is (Shedskin does only a small subset of Python), it's the most useful comparison in my eyes, as where Shedskin makes static type analysis, Nuitka also should do it, only hampered by guards at maximum.
The talk with start out with demonstration of Nuitka, something simple first, and then Mercurial as a more complex example, and then its test suite.
And I will show portable binaries. It seems to work quite nicely. Generally I expect to start out with demos, and explain from there, instead of having a demo only at the end.
If it all works out, this time, they will be prepared with recordmydesktop so I can publish them separately too.
Generally the talk will be more directed at the future, although this is kind of a dark area now. That's its nature I guess.
The talk will also be largely built SSA (static single assignment) and how it applies to Python. What everybody means, when they say "Python is too dynamic (to be statically compiled)" is that Python values may escape to unknown code that changes them very often.
I will have to talk about that, and how get out of that trap, basically guards, much like PyPy does it too. Escaped values and strange code are only one option of things to occur. Having code for both cases, sounds possible. I will talk about how to decide, which branches we shall have and which not.
And I believe with "compiled modules" potentially already in place, we can achieve very cheap guards in most cases. I can at least explain, why guards can be relatively cheap, where we need them.
I am kind of bugged by that idea. It kind of means to revisit an older milestone, but now an idea has surfaced, that I previously didn't have, and that I am very curious to learn the benefit of. Very quick and safe module variabls, are very tempting to have, and definitely make a difference for the Nuitka design.
Who knows, we might even have a "compiled locals" as well, which as a side effect, also allows total interactivity with the most absurd codes. So far, each local variable is a C++ object, and as this is compiled, and very fast.
But the frame locals is not in sync with it. Were it a special object, it could try and interact with these when control escapes to code that we don't know if it might use it.
Big words. Can Nuitka avoid module objects entirely. Can it inline functions, specialize them according to the only types used (think including os.path.dirname in the binary, but with the constraint that it only need to work on string objects as input, because the program is known to use it any different.
Last time, I spent a lot of time on justification, "why a new project?", "why not work with the others?", what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.
With this out of the way, I can now focus on inclusion, and success.
And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.
For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It's probably fair to allow people to make donations if they feel they want to further the project, but don't know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikley though. Too much fun.
And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.
Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.
I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these "times of my life". Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.
The good news are lately coming in high density, although I barely have the opportunity to work on Nuitka. Check the page What is Nuitka? for clarification of what it is now and what it wants to be.
It seems my recent call for help was at least partially effective. One the major points has been addressed. In the next release, Visual Studio 2008 and 2010 will both work for Nuitka, right now it's only in git develop branch.
That is really nice, as it will make Nuitka on Windows a whole lot more accessible. Combine that with portable support and it's up to solving problems with distributing on Windows it seems.
The other two things listed there have received no feedback yet. So if you want to help with those, feel free to do so. Nuitka needs you.
The third point, Windows 64 support has been done as well. Turns out, that I do in fact own only Win64 systems, and with MSVC support in place, only a few portability fixes were needed.
Help with the other point, "re-formulation of yield", would still be more than welcome, and no signs of progress there yet. So stop thinking "you could", enter telling people "you did" mode.
The second point, Fibers implementation for Windows, has been done. Thanks for the help you people provide. The next release will contain it.
the Python compiler Nuitka has come an ever longer way, and currently I have quite a bunch of issues, that I believe could well need your help. These are all issues of some important and significance, yet self contained jobs that you may enjoy.
You can check the page What is Nuitka? for clarification of what it is now and what it wants to be.
Python 3.3 - reformulation of yield from needed.
As you can see, covering all the CPython 2.6, 2.7, and 3.2 language features is already something. Also CPython 3.3 is now working on a basic level. Other projects are far, far away from that.
Many language constructs, such as the with statement are re-formulated into other constructs. This makes it possible to work with a simple core for optimization, and to reduce the complexity a lot. For the with statement case, it's changed to try/finally and try/except statements, together with a few temporary variables. Check the Nuitka source of it:
There is also descriptions of all these re-formulations in the developer manual so you can see how this is done.
Now check PEP 380 for the details of it, and your task would be to come up with a re-formulation of yield from statements to the semantically equivalent.
The benefit is, you are working on a totally self-contained level. The re-formulation only needs to use ast node information, and turn that into Nuitka nodes. The problem will be fully solved this way.
And once we have this, we can pronounce CPython3.3 as fully supported. So if you would like to see this happen, please join.
Windows - MSVC support needs low level code help
This is now done.
The support for MSVC is already mostly there. If you execute the environment setting script, and use Nuitka under Windows, it will work. The only problem is that for the generator functions, Nuitka is using a concept called "Fiber", which is basically just switching C stacks, so called co-routines.
It works great for Linux and UNIX. For Windows, Fibers are currently using threads, which wouldn't be as much of a performance problem, because they are very lightweight, it is slightly wasteful only.
But, the bad thing, is with these threads switching and C++ exceptions used to represent Python exceptions, no combination of MSVC options seems to carry the day and keep the exception in the current thread. They are caught in the wrong thread, and make generators misbehave, crash.
This is the only thing that holds back proper MSVC support at this time, and it has been like this for a while. With MinGW, the issue does not exist, but MinGW is a stranger to many, and MSVC will be more efficient code generation, so we would like to have that.
So what it takes is somebody familiar with Windows and its ABI, esp. register usage to write code that swaps the registers around, so we can have multiple C stacks without threads for it too. This may involve messing around with exception jump back pointers as well.
It needs somebody who can handle this and does it for Nuitka, and then it would be well supported. It may involve writing some assembler code. The job is also very well contained, but also requires very specific knowledge, so please join me here. If you think you can do this, do it.
Windows 64 bit
I have no access to Win64. I bought many Windows versions, but not the Win64 variants so far. I understand that there are CPython builds for it, and probably a MinGW matching it, as well as a matching newer MSVC, that may work better for C++ exceptions than the one CPython 2.x is tied to.
Doing that, you would mostly only have to play around with SingleExe.scons, which abstracts the details of it all. One issue you will encounter is that Scons needs a Python2, and other small things. This is mostly only about porting Nuitka, and could be done by anybody with access to Win64 and willing to fiddle around a bit.
So these are 3 different ways you can help Nuitka. Creating the Python compiler is a huge effort. And if you help it, we will sooner arrive at its goals. Join the Nuitka mailing list or contact me directly if you choose to help with any of these issues.
Wait, actually there is another way you can help. Please spread this news to people that you believe might be willing to help. And allow me to plug this, I am doing Nuitka in my spare time. If you feel, you cannot help, but still would like to support it, you can still make donations to allow me to travel to conferences and spend more time on it myself.
Anyway, it would be great if these 3 things got sorted out. Come and join me for this grand vision of the Python compiler.
This post is about a feature often requested, but so far not available feature of Nuitka. Please see the page "What is Nuitka?" for clarification of what it is now and what it wants to be.
In forums, and in Google, people are looking at a Python compiler, also as a way of deployment. It should offer what py2exe does, allow installation independent of Python.
Well, for a long time it didn't. But thanks to recent contributions, it's upcoming for the next release, Nuitka 0.4.3, and it's in the current pre-releases.
It works by adding --portable to the command line. So this should work for you:
nuitka-python --recurse-all --portable your-program.py
Right now, it will create a folder "_python" with DLLs, and "_python.zip" with standard library modules used along to the "your-program.exe". Copy these to another machine, without a Python installation, and it will (should) work. Making that statement fully true may need refinements, as some DLL dependencies might not be defined yet.
We may improve it in the future to meld everything into one executable for even easier deployment.
You are more than welcome to experiment with it. To do so, download Nuitka from the download page and give it a roll.
Of course, Nuitka is not about replacing "py2exe" primarily, it's only a side effect of what we do. Our major goal is of course to accelerate Python, but surely nobody minds achieving the two things at the same time.
And while the post is labeled "Windows", this feature also works for Linux at least too. It's just that the lack of Python installations on client systems is far more widespread on this platform.
To me, as this is from a contributor, it's another sign of Nuitka gaining adoption for real usage. My personal "py2exe" experience is practically not existing, I have never used it. And I will only merge the improvements into the Nuitka project as provided by others. My focus for the time to come is of course the compile time and run time optimization.
My first real UNIX ever was a NetBSD. That was now about 22 years ago. I am still sentimental about it. I had installed it last about 8 years ago. And I still like it. Back in the days, it was the first UNIX to encounter for me, running on Amiga hardware, first of a friend, then on my own.
Recently, there had been support for Nuitka on FreeBSD added. A lot of people use it on the web, and some want to use Nuitka to improve their Python performance, so this is kind of relevant.
There were issues resolved, but in the end, something was with Clang on FreeBSD 8, that I could for the life of it, not resolve remotely. So I attempted to install it myself. Using "virt-install", these things are a breeze now. I had already done it with CentOS6 before to test the RPM repositories of Nuitka. That "virt-install" is a wonderful thing by itself, making virtualisation somewhat useful. It's only a pity, that I can't just install other qemu support architectures. I would love to checkout Nuitka on PowerPC.
If you could check out Nuitka on other Linux architectures than x86_64, x86, or arm, that would be great.
This is the report of getting NetBSD supported. It was a quite an interesting story that I would like to share with you.
Naivly I was assuming, that it would be just for fun, and that Nuitka will work right away. Little did I know.
On FreeBSD 9 the minimal install medium was chosen, and entered its ports collection, installed git, cloned Nuitka, and ran the tests, successfully right away. Now that is unfair, in the Nuitka there were tons of "Linuxism" already removed. In fact, it had to work, and on the newest FreeBSD (version 9.1) and then it did. Great!
If you would like to add Nuitka to FreeBSD's ports, please do so. It should be really easy now.
On NetBSD, things were unfortunately a little different. I also chose minimal system. After going through "pkg source" boot strap and git install, I cloned Nuitka, and then tried to start it. First off, it couldn't locate "python" at all. I am using /usr/bin/env python already. But Python2 was on the system. I ended up creating the "python" link myself. What I should have done according to "#netbsd" is to install the software, and indeed, python2.7 setup.py install gives an installation of Nuitka that is executable.
Next up, you need to know about "Fibers" in Nuitka. These are used for C co-routines, used to implement Python generators. They have an interface, that is very close to makecontext/swapcontext routines in C.
For ARM and x86_64 Linux we have optimized code, that switches faster, but other platforms, including x86 Linux, use the generic implementation, also because it normally is very fast already.
Now you have to know that since 2001 the interface is deprecated and shall not be used. And next up, is that on NetBSD, makecontext gave a segfault only. So I ran to "#netbsd" and asked.
Now that was a very friendly experience. Of course, I had to give a rationale for using an obsolete interface. It's not quite obvious, why threads wouldn't be a better choice. And maybe they are, but they definitely have more overhead associated, and if they never run at the same time, why use them.
Ultimately it helped to point out, that for a user of 22 years, an interface that is only obsolete for 11 years, is not quite as horrifying as for others.
And they helped me through it. And it turns out, funny thing. For the context to setup, you are allocating a stack to use for the C routine, and you "get" the current context, then you make a new one. All the examples have a certain order of doing it. And my code did it the other way around. No system but NetBSD noticed.
On FreeBSD and Linux, it didn't matter. But it seems, that the needed getcontext call was overwriting my stack pointer now with the current stack. And makecontext deeply hated that, a lot. It was preparing that stack to be used, while it was in usage. Doesn't sound like a good task to give to it, right? My fault truly, because every example on every man page, on all systems, was doing it differently. But then they were also all using arrays from the local stack, so quite obviously that was not real code.
So that was fixed, and all good? No! Next thing was it crashed when free happened in Python on a compiled frame object, in a later part of a test that heavily uses generators. Turns out, malloc information was corrupted. I had to suspect the generic "Fiber" code, but that took me a while to figure out.
And how could my simple malloc and free do that, and make it happen. When I knew that a context would not longer be used (the generator has finished, the generator object deleted, etc), I would look at the context handle stack pointer and free it.
But that pointer changed. Something totally unexpected (by me obviously), but it also explains the earlier problem. For all systems, I had used so far, this pointer was not being changed, and remained the same. So I could free it from there. It worked fine, but not on NetBSD. And it wasn't correct anywhere.
It seems NetBSD is doing something clever, since instead of saving the stack pointer register in a separate area, it saves it to that place originally specified. It's quite obviously an improvement, in that you save the pointer.
It's only bad, that now to make up for this savings, I have added the pointer in a separate field, which won't be changed, so I can free it again. If one needs it again, and that's not unlikely, you have to remember it elsewhere. So maybe that idea is not that clever. But it surely was wrong by me to assume that the provided value would not be touched.
So, these are 2 bugs it found. The wrong order of calls. And the usage of a pointer, that may have been changed. This can only help with other systems, or possibly architecures under Linux.
While this is all description of nasty problems, it's also the report of the solution, and it was big fun. I would also like to compliment "#netbsd" for being very helpful and friendly with my porting of Nuitka. I highly enjoyed doing so. It was a lot of fun. I know that it's probably on a very tiny amount of people that uses both NetBSD and Nuitka, but still.
If this Nuitka project were about market share, it wouldn't exist. And I can work for market share on another day.
This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the performance diagrams got expanded.
Support for FreeBSD.
Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process.
New option for warning about compile time detected exception raises.
Nuitka can now warn about exceptions that will be raised at run time.
Basic Python3.3 support.
The test suite of CPython3.2 passes and fails in a compatible way. New feature "yield from" is not yet supported, and the improved argument parsing error messages are not implemented yet.
Nuitka already supported compilation of "main directories", i.e. directories with a "__main__.py" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe"
# ls directory __main__.py # nuitka --exe directory # ls directory directory.exe
This makes this usage more obvious, and fixes the older issue Issue#49 for this feature.
Evaluation order of binary operators was not enforced.
Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects Issue#61.
Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.
Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.
global a # Not in a function, but on module level. Pointless but legal! a = 1
Effectively these statements can be ignored. Corrects part of Issue#65.
Future imports are only legal when they are at the start of the file. This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of Issue#65.
Raising exceptions from context was leaking references.
raise ValueError() from None
Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects Issue#76.
Importing the module that became __main__ through the module name, didn't recurse to it.
This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don't share data. Corrects Issue#68.
Compiled method repr leaked references when printed.
When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point. Corrects Issue#81.
The super built-in leaked references to given object.
This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run. Corrects Issue#81.
The raise statement didn't enforce type of third argument.
This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked.
Python3 built-in exceptions were strings instead of exceptions.
A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects Issue#82.
The -nan and nan both exist and make a difference.
A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka. Corrects Issue#75.
Wrong optimization of a == a, a != a, a < a on C++ level.
While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for "==", "!=", and "<".
The sys.executable for nuitka-python --python-version 3.2 was still python.
When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.
Keyword only functions with default values were loosing references to defaults.
def f( *, a = X() ) pass f() f() # Can crash, X() should already be released.
This is now corrected. Of course, a Python3 only issue.
Pressing CTRL-C didn't generate KeyboardInterrupt in compiled code.
Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.
For with statements with return, break, or continue to leave their body, the __exit__ was not called.
with a: # This called a.__enter__(). return 2 # This didn't call a.__exit__( None, None, None ).
This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing. Corrects Issue#59.
Starting with Python3, absolute imports are now the default.
This was already present for Python3.3, and it turns out that all of Python3 does it.
Constants are now much less often created with pickle module, but created directly.
This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.
a = ( "something_special", ) b = "something_special" assert a is b # Now true
This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects.
Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.
Continue statements might be optimized away.
A terminal continue in a loop, was not optimized away:
while 1: something continue # Now optimized away
The trailing continue has no effect and can therefore be removed.
while 1: something
Loops with only break statements are optimized away.
while 1: break
A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.
The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file.
The page got improved with logos for the distributions.
Added "ownssh" as contributor.
Revamped the User Manual in terms of layout, structure, and content.
This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.
Thanks to contributor, Nuitka now has a proper RPM spec file. And using openSUSE Build Service, we can now enjoy RPMs for all major distributions.
Check out the revamped download page.
At the time of this writing, the "stable" Nuitka 0.4.1 link points to a 0.4.2 pre-release. That is only until 0.4.2 final will be released.
Thanks for making this possible, very much appreciated.
This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree.
Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes.
Python 3.3 experimental support
Python 3.2 support has been expanded.
The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code.
Default values were not "is" identical.
def defaultKeepsIdentity( arg = "str_value" ): print arg is "str_value" defaultKeepsIdentity()
This now prints "True" as it does with CPython. The solution is actually a general code optimization, see below. Issue#55
Usage of unicode built-in with more than one argument could corrupt the encoding argument string.
An implementation error of the unicode was releasing references to arguments converted to default encoding, which could corrupt it.
Assigning Python3 function annotations could cause a segmentation fault.
Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization.
# Now optimizing: raise TypeError, 1/0 # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: (1/0).something # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: function( a, 1/0 ).something # into (minus normalization), notice the side effects of first checking # function and a as names to be defined, these may be removed only if # they can be demonstrated to have no effect. function a raise ZeroDivisionError, "integer division or modulo by zero"
There is more examples, where the raise propagation is new, but you get the idea.
Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things.
# This was already optimized, as it's a compile time constant. a if ( "a", ) else b a if True else b # These are now optimized, as their truth value is known. a if ( c, ) else b a if not (c, ) else b
This is simply taking advantage of infrastructure that now exists. Each node kind can overload "getTruthValue" and benefit from it. Help would be welcome to review which ones can be added.
Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused.
Code generation for constants now shares element values used in tuples.
The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up.
The quicker release is mostly a consolidation effort, without actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization.
As you all know, Nuitka (see "what is Nuitka?" ) has recently completed a milestone. Always short on time, I am not doing a whole lot of benchmarking yet, and focus on development. But here is an interesting submission from Dave Kierans (CTO of iPowow! Ltd):
➜ ~ python pystone.py 1000000 Pystone(1.1) time for 1000000 passes = 10.2972 This machine benchmarks at 97113.5 pystones/second ➜ ~ cython --embed pystone.py;gcc pystone.c -I/usr/include/python2.6 -L /usr/lib/ -lpython2.6 -o ./pystone.cython;./pystone.cython 1000000 Pystone(1.1) time for 1000000 passes = 8.20789 This machine benchmarks at 121834 pystones/second ➜ ~ nuitka-python pystone.py 1000000 Pystone(1.1) time for 1000000 passes = 4.06196 This machine benchmarks at 246187 pystones/second
This is nice result for Nuitka, even if we all know that pystone is not a really good benchmark at all, and that its results definitely do not translate to other software. It definitely makes me feel good. With the 0.4.x series kicked off, it will be exciting to see, where these numbers can indeed go, once Nuitka actually applies standard compiler techniques.