This is to inform you about the new release of Nuitka many bug fixes, and substantial improvements especially in the organizational area. There is a new "User Manual" (PDF), with much improved content, a sys.meta_path based import mechanism for --deep mode, git flow goodness.
This release is generally also the result of working towards compilation of a real programs (Mercurial) and to get things work more nicely on Windows by default. Thanks go to Liu Zhenhai for helping me with this goal.
Due to the use of the "git flow", most of the bugs listed here were already fixed in on the stable release before this release. And there were many of these.
The order of evaluation for base classes and class dictionaries was not enforced.
Apparently nothing in the CPython test suite did that, I only noticed during debugging that Nuitka gave a different error than CPython did, for a class that had an undefined base class, because both class body and base classes were giving an error. Fixed in 0.3.11a already.
Method objects didn't hold a reference to the used class.
The effect was only noticed when --python-debug was used, i.e. the debug version of Python linked, because then the garbage collector makes searches. Fixed in 0.3.11b already.
Set sys.executable on Linux as well. On Debian it is otherwise /usr/bin/python which might be a different version of Python entirely. Fixed in 0.3.11c already.
Embedded modules inside a package could hide package variables of the same name. Learned during PyCON DE about this corner case. Fixed in 0.3.11d already.
Packages could be duplicated internally. This had no effect on generated code other than appearing twice in the list if frozen modules. Fixed in 0.3.11d already.
When embedding modules from outside current directory, the look-up failed. The embedding only ever worked for the compile itself and programs test cases, because they are all in the current directory then. Fixed in 0.3.11e already.
The check for ARM target broke Windows support in the Scons file. Fixed in 0.3.11f already.
The star import from external modules failed with an error in --deep mode. Fixed in 0.3.11g already.
Modules with a parent package could cause a problem under some circumstances. Fixed in 0.3.11h already.
One call variant, with both list and dict star arguments and keyword arguments, but no positional parameters, didn't have the required C++ helper function implemented. Fixed in 0.3.11h already.
The detection of the CPU core count was broken on my hexacore at least. Gave 36 instead of 6, which is a problem for large programs. Fixed in 0.3.11h already.
The in-line copy of Scons didn't really work on Windows, which was sad, because we added it to simplify installation on Windows precisely because of this.
Cleaning up the build directory from old sources and object files wasn't portable to Windows and therefore wasn't effective there.
From imports where part of the imported were found modules and parts were not, didn't work. Solved by the feature branch meta_path_import that was merged for this release.
Newer MinGW gave warnings about the default visibility not being possible to apply to class members. Fixed by not setting this default visibility anymore on Windows.
The sys.executable gave warnings on Windows because of backslashes in the path. Using a raw string to prevent such problems.
The standard library path was hard coded. Changed to run time detection.
- Version checks on Python runtime now use a new define PYTHON_VERSION that makes it easier. I don't like PY_VERSION_HEX, because it is so unreadable. Makes some of the checks a lot more safe.
- The sys.meta_path based import from the meta_path_import feature branch allowed the cleanup the way importing is done. It's a lot less code now.
- Removed some unused code. We will aim at making Nuitka the tool to detect dead code really.
- Moved nuitka.Nodes to nuitka.nodes.Nodes, that is what the package is intended for, the split will come later.
- New tests for import variants that previously didn't work: Mixed imports. Imports from a package one level up. Modules hidden by a package variable, etc.
- Added test of function call variant that had no test previously. Only found it when compiling "hg". Amazing how nothing in my tests, CPython tests, etc. used it.
- Added test to cover the partial success of import statements.
- Added test to cover evaluation order of class definitions.
Migrated the "README.txt" from org-mode to ReStructured Text, which allows for a more readable document, and to generate a nice "User Manual" in PDF form.
The amount of information in "README.txt" was increased, with many more subjects are now covered, e.g. "git flow" and how to join Nuitka development. It's also impressive to see what code blocks and syntax highlighting can do for readability.
The Nuitka git repository has seen multiple hot fixes.
These allowed to publish bug fixes immediately after they were made, and avoided the need for a new release just to get these out. This really saves me a lot of time too, because I can postpone releasing the new version until it makes sense because of other things.
Then there was a feature branch meta_path_import that lived until being merged to develop to improve the import code, which is now released on master as stable. Getting that feature right took a while.
And there is the feature branch minimize_CPython26_tests_diff which has some success already in documenting the required changes to the "CPython26" test suite and in reducing the amount of differences, while doing it. We have a frame stack working there, albeit in too ugly code form.
The release archives are now built using setuptools. You can now also download a zip file, which is probably more Windows friendly. The intention is to work on that to make setup.py produce a Nuitka install that won't rely on any environment variables at all. Right now setup.py won't even allow any other options than sdist to be given.
Ported "compile_itself.sh" to "compile_itself.py", i.e. ported it to Python. This way, we can execute it easily on Windows too, where it currently still fails. Replacing diff, rm -rf, etc. is a challenge, but it reduces the dependency on MSYS tools on Windows.
The compilation of standard library is disabled by default, but site or dist packages are now embedded. To include even standard library, there is a --really-deep option that has to be given in addition to --deep, which forces this.
Again, huge progress. The improved import mechanism is very beautiful. It appears that little is missing to compile real world programs like "hg" with Nuitka. The next release cycle will focus on that and continue to improve the Windows support which appears to have some issues.
the Video of my presentation is online: http://blip.tv/episode/5632565
Leaving it to Mike Müller to announce here the links to everything, when it's all finished. Thanks for the good work, the video was very well done.
The video is also available via Youtube.
The new cat has become indispensable for just about anything. She follows throughout the hous, and is just too lovely, here two photos of her in typical spots.
When I code for Nuitka, she often lays between keyboard and monitors, sleeping, getting caressed by me.
And here is another one, hiding in her cat tree.
The PyCON DE 2011 is just over, sprints are still happening over the weekend, but my wife wouldn't allow me to stay away for that long, so it's not for me this time. Maybe next time.
Right now I feel very happy and excited that I went there. What a great experience this was.
It was the first German PyCON and clearly it was overdue as it was now the merger of many already grown up communities. A huge number of talks over 3 days in 3 paralell tracks, with 3 keynotes, was an outstanding program. And very well run. Strict time management, every detail was well prepared.
I can only admire the professional preparation and setup. I wanted to say thank you deeply. I didn't consider it possible to be this good. Clearly not a first time.
I enjoyed the talks, most often in the technical track, but other tracks would have been very interesting too. The parallelism was making me do hard decisions.
The food was great too. I esp. liked the Asian day, but there was also Italian and French, and what many liked very much is that there was a Vegan food offer too. I do not live vegan style, but I appreciate good food and the vegan food often is that.
The quality of the organization team, the city "Leipzig", where we also got to have a guided city tour of fantastic enthusiasms, was very high. I knew Leipzig from earlier visits and liked it before, but this time it seemed everybody was even friendlier.
The convention place "Kubus" was very well chosen, absolutely ideal. It's got good equipment, and that large room setup, where you can make a split with movable walls, and have 3 big screens. The acoustics were pretty damn good there.
My own Presentation
As to my own presentation, it was well received, although I sort of regret that I agreed to have only 30m instead of original plan of 60m. I had so much to say.
I ended up with getting my manifesto part out, but that one pretty well. And it's OK I guess, because nobody really listens that long anyway. And my major points came across that way.
That focus on my Nuitka "manifesto" was probably a good idea. The talk will be available online as a video, I will link it then. The PDF that I presented only a small part of, is linked here. I believe it went pretty well.
I will use that content from the PDF in updated documentation (currently ongoing in PDF is work to use REST and document a lot more). The presentation was created with "rst2pdf", which I find is a fantastic tool.
Cython / lxml
Early on I already made contacts with interesting people, e.g. with Dr.Stefan Behnel, author of lxml and core Cython developer. I him offered a beer for using his software in the best of Free Software traditions. He doesn't drink these, but a large mango juice counts too or so I assume.
We also talked about Cython and Nuitka, and the common history we had as well. For some time, I attempted to change Cython, but that failed to get the developers support at the time. Not wanting to deviate from PyRex clearly isn't the state anymore, but that was then.
We also had a evening session of showing each other the good and bad parts, comparing was quite fun. And it was quite interesting to the both of us. I believe we made friends and will only benefit another.
We discussed my goals, and I think we came to the conclusion that they are in fact different enough from Cythons. Although I go away with the sense, that of course Stefan believes, it would be better if I joined Cython. Naturally.
But that's not going to happen. I think i have a cleaner and better implementation now, closer to my goals with a realistic chance to succeed. To me it would be a step back to fix language parsing issues and incompatabilities of Cython, with the danger that my goals will not be shared.
As an example of these things, I would mention function call errors, where e.g. Cython gives different and sometimes worse error messages than CPython, and I designed the code so that it does things in that same order than CPython does.
It do not want to give different error messages, and who knows, somebody may check for the exception text and expect CPython output. In this case, I will rather accept a worse performance, than an incompatibility.
Eliminating function parameter parsing for the whole program as far as possible is going to be more worthwhile anyway.
But in my mind, Cython is something I can and do recommend. For as long as I am not able to declare Nuitka "useful" yet. That statement may come within a year though. In my mind, in many fields Nuitka is already superior.
Another interesting contact I made, was with the author of PyHasse. It's Rainer Bruggemann, who is a really nice and witty guy. He introduced me to how he applies graph theory to multi-parameter optimization problems.
We agreed that we will try and work together on this project. Hopefully it will come to pass. One thing I personally wanted, was to get into contact with people who understand or are part of the scientific community.
I can see what NumPy is. But I may never know myself what it really is, unless I find proxies, and make these kind of contacts. The same thing is true of Django, or e.g. Mercurial. I am positive though that with time, and such conferences, my knowledge of these will only grow.
We said that we will try and see how far we can go. In the worst case, Nuitka will not yet be useful, but I will have a clearer image what is needed.
I saw the presentation from Jan Dittberner and met him later too, asking him questions, and generally discussing Debian packaging of Nuitka. He encouraged me to contact the Debian Python Team, and so I will.
I used the chance to make contact with a Debian guy, who made a presentation on how to package Python modules for Debian. He gave me hints on how to solve that "find files near me" issue that plagues Nuitka just as much as other software. Really kind and helpful guy and clearly I admire Debian Developers, keep up the good work.
I also made contacts with lots of other people. Python is diverse and it was fun to get to know, many people with similar and entirely different backgrounds.
The mood was extremely constructive. Nuitka was well received, but that's not why I say it. There is that general sense of respect around that German community, you can feel how pretty much everybody is well established and doesn't have to disprove the others.
One keynotes speaker had a part about how trolling and hate is bad for a community, but that's not the German Python community.
Another keynote speaker (Paul Everitt) had a part about how Zope, which was kind of his project, failed in many ways. He seemed to be quite disappointed about that, which triggered me to point out, that he should start his story with Apache, and not see the "failure to integrate" as a failure.
If there had not been Apache failing, there wouldn't have been Zope, and then not Django, etc. that's kind of normal and actually good. He agreed and pointed out how Apache was created from another project that had failed to integrate people.
You either fork a projects code, or ideas. The fork still should credit and appreciate the predecessor/origin.
In my mind, Cython failed to integrate me. Which triggered me to come up with Nuitka, and as I will point out over time (there ought to be postings and there probably will be), some better approaches.
So not integrating me is not necessarily a failure. If it were not for Cython, there would not be Nuitka. The original projects will regret the fork/remake, but they probably shouldn't. Competition is good.
Lets repeat that
I believe the PyCON DE 2011 was a huge success. I will most likely go again to update people on Nuitka. It's already clear there will be a PyCON DE 2012 I understand. And I am aiming for a slot at PyCON EU 2012 next year too. I wanted to go in 2011, but need to not put it in my early booked holiday again.
But you know what Murphy says about that.
Yours, Kay Hayen
In case, you want and can improve the source visually or otherwise, please go ahead. I am using it for a presentation next week too, and would be glad if you could make it more pretty. My artistic skills are not the same as my programmer skills. :-)
So there is now always at least these 2 branches:
and then there may be feature branches, like this one currently:
- Feature branch minimize_CPython26_tests_diff
These will only have certain life, until they are completed, then they are merge into "develop" and become part of the next release. This may or may not happen, depending on how things go.
Benefits of the new model
- Hotfixes, typically bug fixes, can be made simultaneously on stable and develop branch. The git-flow package takes care of the merging to both.
- Because that's so easy now, a stable version can be provided and supported for a longer time.
- Features can be published while under development. My idea is that feature branches should basically work, but the bar will be lower. People can have a look at them, or start their own and make me integrate them.
Uses of Feature Branch
For example, in the new feature branch, a couple of boring things are happening. Support for frame stack will reduce the diff, as will some work to match CPython's choices for exception line numbers. Completing will take a while, but should not block a release. So this is best done in the feature branch, esp. as nothing is going to really depend on it.
As you can see from this diagram, I am working mostly on documentation things. The new and improved README on develop, which is closer to a User Manual in PDF form, and other organization things, may get a release before the PyCon DE next week. The README also describes this process.
Hope is that with this approach, I will improve transparency (you can see earlier what i am working on, because there is now a place where things may break (develop) or may not yet be integrated or completed fully (feature branches) and yet be public.
The overhead appears to minimal thanks to "git-flow". Developing hotfixes is actually easier, when done on the stable branch, because problems cannot originate from the current development work that may or may not be all that perfect yet.
Yours, Kay Hayen
This is to inform you about the new release of Nuitka with some bug fixes and portability work.
This release is generally cleaning up things, and makes Nuitka portable to ARM Linux. I used to host the Nuitka homepage on that machine, but now that it's no longer so, I can run heavy compile jobs on it. To my surprise, it found many portability problems. So I chose to fix that first, the result being that Nuitka now works on ARM Linux too.
- The order of slice expressions was not correct on x86 as well, and I found that with new tests only. So the porting to ARM revealed a bug category, I previously didn't consider.
- The use of linux2 in the Scons file is potentially incompatible with Linux 3.0, although it seems that at least on Debian the sys.platform was changed back to linux2. Anyway, it's probably best to allow just anything that starts with linux these days.
- The print statement worked like a print function, i.e. it first evaluated all printed expressions, and did the output only then. That is incompatible in case of exceptions, where partial outputs need to be done, and so that got fixed.
- Function calls now each have a dedicated helper function, avoiding in some cases unnecessary work. We will may build further on this and in-line PyObject_Call differently for the special cases.
Moved many C++ helper declarations and in-line implementations to dedicated header files for better organisation.
Some dependencies were removed and consolidated to make the dependency graph sane.
Multiple decorators were in reverse order in the node tree. The code generation reversed it back, so no bug, yet that was a distorted tree.
Finding this came from the ARM work, because the "reversal" was in fact just the argument evaluation order of C++ under x86/x64, but on ARM that broke. Correcting it highlighted this issue.
The deletion of slices, was not using Py_ssize for indexes, disallowing some kinds of optimization, so that was harmonized.
The function call code generation got a general overhaul. It is now more consistent, has more helpers available, and creates more readable code.
PyLint is again happier than ever.
- There is a new basic test OrderChecks that covers the order of expression evaluation. These problems were otherwise very hard to detect, and in some cases not previously covered at all.
- Executing Nuitka with Python3 (it won't produce correct Python3 C/API code) is now part of the release tests, so non-portable code of Nuitka gets caught.
- Support for ARM Linux. I will make a separate posting on the challenges of this. Suffice to say now, that C++ leaves way too much things unspecified.
- The Nuitka git repository now uses "git flow". The new git policy will be detailed in another separate posting.
- There is an unstable develop branch in which the development occurs. For this release ca. 40 commits were done to this branch, before merging it. I am also doing more fine grained commits now.
- Unlike previously, there is master branch for the stable release.
- There is a script "make-dependency-graph.sh" (Update: meanwhile it was renamed to "make-dependency-graph.py") to produce a dependency graphs of Nuitka. I detected a couple of strange things through this.
- The Python3 __pycache__ directories get removed too by the cleanup script.
We only have "PyStone" now, and on a new machine, so the numbers cannot be compared to previous releases:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
So this a speedup factor of 258%, last time on another machine it was 240%. Yet it only proves that the generated and compiled are more efficient than bytecode, but Nuitka doesn't yet do the relevant optimization. Only once it does, the factor will be significantly higher.
Overall, there is quite some progress. Nuitka is a lot cleaner now, which will help us later only. I wanted to get this out, mostly because of the bug fixes, and of course just in case somebody attempts to use it on ARM.
I am going to the German Python conference in Leipzip and I am going to have a presentation (link in German) there.
Of course it's about Nuitka (see What is Nuitka? ) and I hope it will get a lot of attendance. I am naturally very happy to have the opportunity to present it finally. I had wanted to visit PyCon EU, but the date was never known, and then my early booking holiday overlapped with it, so it was not an option.
Now giving this presentation will of course be exciting to me. I gave presentations as part of the day job many times, but this time it's obviously a different. It's also a first chance to meet the others as I never was at a Python conference before, and that will be interesting in itself.
Presenting Nuitka will of course be easy for me. Now I need to plan what I want to be able to present for a demo. Running a big thing like Mercurial would be nice, but I honestly don't know, if that will even be difficult, or if it will take a lot of work. Also the amount of documentation available for Nuitka should increase as part of this. Designs, etc. could be made into diagrams, so people who want to join will have it easier.
Lots of possibilities, and then there is only going to be one reality. Lets hope it's good. :-)
my new vServer is now online, the old site blog redirects to it now, and planets follow the new site already. The old site is redirecting with 301 error code for some more time, hopefully this will be good enough of a migration. It felt smooth from here, clearly, and the new site is much faster of course, because it's not throttled through my poor DSL upstream.
I also registered it a domain name "nuitka.net" for it finally. Originally I wanted to give it a DynDNS name, but I already have 2, and the third was supposed to cost money, and I didn't feel like using another service, or cheating them if you consider that an option, so that was it, I bought the domain name. Clearly it will also be easier for people to remember.
Moving things over was pretty painless. The Wordpress software has a pretty good export feature, which only didn't manage the custom menu and appearance settings. But I guess it's about content anyway, so probably some way that would work too, but it was easy enough to reproduce that by hand. I appreciate Wordpress even more now.
Also I did some tuning of the system, to use less memory. Only 512M are available, and so I run less Apache processes for less requests (memory leaks), disabled IPv6 (yes, hate me for it), reduced amounts of gettys, and so on. Nothing I am not familiar with, the ARM machine had 512M as well, and to me no reason to use the bigger package just because of that.
The main difference is the faster CPU, I seem to get 3Ghz Intel now, instead of my 1Ghz ARM, which together with faster internet speed, makes the site extremely fast.
Now U will dare to make the gitweb interface public as well. The git repository is already running there.
And I took the chance to sanitize the old posts somewhat. Changed the links to not use the old domain name anymore, and correct some broken ones too.
Only downside is that I currently haven't got the "speedcenter" up and running again. After loosing my hardware, the old data cannot be compared with new one, and then it doesn't feel like a priority right now. I seem to be working instead on XML based regression tests of the optimizer: The output of "--dump-xml" should be compared for a large quantity of files, to discover regressions of the optimizer as soon as possible, this will enable me to make changes and not have to review the C++ as much, to find out if something is compiled correctly. This way I should detect it when known good cases degrade, and generally to demonstrate better, what actually did improve.
Yours, Kay Hayen
PS: Oh, you people, who wonder, "but why are you not using Google/github/gitorious?", my counter question: "Did you read the agreement?" I did. It basically says (from Google code):
13. INDEMNITY You agree to hold harmless and indemnify Google, and its subsidiaries, affiliates, officers, agents, employees, advertisers, licensors, suppliers or partners, (collectively "Google and Partners") from and against any third party claim arising from or in any way related to your use of Google services, violation of the Terms or any other actions connected with use of Google services, including any liability or expense arising from all claims, losses, damages (actual and consequential), suits, judgments, litigation costs and attorneys' fees, of every kind and nature. In such a case, Google will provide you with written notice of such claim, suit or action.
No thank you, instead I will run my own server, then I get to pay the attorneys of my discretion - in the admittedly unlikely event that somebody should sue me, because my Compiler violates some patent, or whatever.