The sweetness phase is over. At least in that instant. :-)
Hello there,
as I previously said, I will make these a separate announcement. Thanks to the NeuroDebian project build infrastructure, there is Nuitka packages for the Ubuntu versions back to Maverick, Natty, and Oneiric.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
You may use the apt sources to track Nuitka or download the packages from the page where I offer the latest version here.
Note: In fact, it’s 0.3.19.1 now, which includes a hotfix release made after 0.3.19. The same is in Debian Unstable right now, waiting to propagate into Debian Testing, where there is still 0.3.18 living at this time.
I would appreciate feedback on these.
Thanks in advance,
Kay
This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things are in this.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
- The manpages of “nuitka” and “nuitka-python” had no special layout for the option groups and broken whitespace for “–recurse-to” option. Also “–g++-only” was only partially bold. Released as 0.3.18.1 hotfix already.
- The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hotfix already.
- Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of “/” instead of using a “joinpath” call. Released as 0.3.18.3 already.
- A call to the range built-in with no arguments would crash the compiler, see Issue#29. Released as 0.3.18.4 already.
- Compatibility Fix: When rich comparison operators returned false value other “False”, for comparison chains, these would not be used, but “False” instead, see .
The support for “__import__” didn’t cover keyword arguments, these were simply ignored. See Issue#28. Fixed, but no warning is given yet.
- A new option has been added, one can now specify “–recurse-directory” and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements.
- Added support for the “exec” built-in of Python3, this enables us to run one more basic test, “GlobalStatement.py” with Python3. The test “ExecEval.py” nearly works now.
- The no arguments “range()” call now optimized into the static CPython exception it raises.
- Parts of comparison chains with constant arguments are now optimized away.
- Simplified the “CPythonExpressionComparison” node, it now always has only 2 operands. If there are more, the so called comparison chain, it’s done via “and” with assignments to temporary variables, which are expressed by a new node type “CPythonExpressionTempVariableRef”. This allowed to remove “expression_temps” from code templates and generation, reducing the overall complexity.
- When executing a module (“–execute” but not “–exe”), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment.
- The calls to the variadic “MAKE_TUPLE” were replaced with calls the MAKE_TUPLExx, that are generated on a as-needed basis. This gives more readable code, because no EVAL_ORDERED_xx is needed at call site anymore.
- Many node classes have moved to new modules in “nuitka.nodes” and grouped by theme. That should make them more accessible.
- The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the “sys.abiflags” and works with Python3.
- The replacing of “.py” in filenames was made more robust. No longer is “str.replace” used, but instead proper means to assure that having “.py” as other parts of the filenames won’t be a trouble.
- Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements.
- As always, some PyLint work, and some minor TODOs were solved.
- Added more information to the “Developer Manual”, e.g. documenting the tree changes for “assert” to become a conditional statement with a raise statement, etc.
- The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise. I expect to make a separate announcement with the links to packages soonish.
- Added support to specify the binary under test with a “NUITKA” environment, so the test framework can run with installed version of Nuitka too.
- Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for “os.execl” not propagating exit codes under Windows. See Issue#26 for more information.
- For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the “PATH” or even installed to execute the binary.
- The “basic”, “programs”, “syntax”, and “reflected” were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations.
- Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains.
- The “Referencing.py” test was made portable to Python3.
- Cover no arguments “range()” exception as well.
- Added test to demonstrate that “–recurse-directory” actually works. This is using an “__import__” that cannot be predicted at run time (yet).
- The created source package is now tested on pbuilder chroots to be installable and capable of the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn’t break without notice.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
This releases contains many changes. The “temporary variable ref” and “assignment expression” work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. inplace assignments can be reduced to assignments to temporary variables.
While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with “value propagation” working.
These are the real focus. The old comparison chains were a wart. Working, but no way understood by any analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimizations to see through them.
Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of inplace assignments. With an “assignment expression” to a “temporary variable ref”, these become the same as user code using such a variable. There will be more of these to find.
So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the “__import__” related one, were not suitable for hotfix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already.
As always you will find its latest version here.
Yours,
Kay Hayen
Hello,
somehow triggered by reading about Mynth, and the re-post of the last release announcement on my feed, just because I added a missing category to the post, I am wondering what it takes to achieve the following:
I am currently very annoyed, because:
Other stuff in favor of it, is that WordPress or any CMS needs more memory and more time to process than static pages of course. I can’t right now let anyone fork and edit the pages in a practical manner. So while for user manual and developer manual of Nuitka I will get corrections, for the website it’s not happening.
Is Mynth an appropiate intermediate step? Do I really want to learn one of these strange templating languages or even consider software that has no Debian package. I never considered “virtualenv” and “pip” much of options to run software on a website. I guess I am more tied to Debian than to Python still.
Over the last months I have occasionally looked at Pyramids, but it seems overly complex, if all you want is to simplify the authoring. I have seen that e.g. Trac supports ReST as an exception, but that feels too little for a hefty software like that.
I may end up with using ReST to generate HTML which is then uploaded via XMLRPC to WordPress. Won’t give me the “static” benefits. Won’t solve the post update problem. But at least will let me edit more easily.
Yours,
Kay
This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via “apt-get install nuitka”.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
The release cycle was too short to have much focus. It merely includes fixes, which were available as hotfixes, and some additional optimizations and node tree cleanups, as well as source cleanups. But
- Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. Issue#16. Released as 0.3.17a hotfix already.
- Nuitka was considering directories to contain packages that had no “__init__.py” which could lead to errors when it couldn’t find the package later in the compilation process. Released as 0.3.17a hotfix already.
- When providing “locals()” to “exec” statements, this was not making the “locals()” writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn’t consider the actual value “locals()”. Released as 0.3.17b hotfix already.
- Compatibility Fix: When no defaults are given, CPython uses “None” for “func.func_defaults”, but Nuitka had been using “None”.
- If the condition of assert statements can be predicted, these are now optimized in a static raise or removed.
- For built-in name references, there is now dedicated code to look them up, that doesn’t check the module level at all. Currently these are used in only a few cases though.
- Cleaner code is generated for the simple case of “print” statements. This is not only faster code, it’s also more readable.
- Removed the “CPythonStatementAssert” node. It’s not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with “AssertionError” and the assertion argument. This allowed to remove code and complexity from the subsequent stetps of Nuitka, and enabled existing optimizations to work on assert statements as well.
- Moved builtin exception names and builtin names to a new module “nuitka.Builtins” instead of having in other places. This was previously a bit spread-out and misplaced.
- Added cumulative “tags” to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
- Enhanced the “visitor” interface to provide more kinds of callbacks, enhanced the way “each scope” visiting is achieved by generalizing is as “child has not tag ‘closure_taker’” and that for every “node that has tag ‘closure_taker’”.
- Moved “SyntaxHighlighting” module to “nuitka.gui” package where it belongs.
- More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that should be white listed.
- Consistently use “nuitka” in test scripts, as there isn’t a “Nuitka.py” on all platforms. The later is scheduled for removal.
- New node for builtin name loopups, which allowed to remove tricks played with adding module variable lookups for “staticmethod” when adding them for “__new__” or module variable lookups for “str” when predicting the result of “type( ‘a’ )”, which was unlikely to cause a problem, but an important TODO item still.
- The “Download” page is now finally updated for releases automatically. This closes Issue#7 completely. Up to this release, I had to manually edit that page, but I now mastered the art of upload via XMLRCP and a Python script, so that I don’t loose as much time with editing, checking it, etc.
- Added a “Developer Manual” to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.
- The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages soonish.
- Made sure the test runners worth with bare “python2.6″ as well.
- Added some tests intended for type inference development.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
This releases contains not as much changes as others. It’s going to be the stable one for a while. In the mean time, we are going to work on the branch “feature/ctypes_annotation” and rebasing it often while things are still flowing.
The “exec” fix was detected by continued work on the branch “feature/minimize_CPython26_tests_diff” branch, but that work is now complete.
It is being made pretty (many git rebases) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made.
But of course, the real excitement is the the “type inference” work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient. So when I learn something. e.g. “assert” is not special, I apply it to the “develop” branch immediately, to keep the differences as small as possible.
Overall, this is a great day. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.
As always you will find its latest version here.
Yours,
Kay Hayen
This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organisational work, and yet again improved compatibility and cleanups. Also huge is the advance in making “–deep” go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
The release cycle had a focus on improving the quality of the test scripts, the packaging, and generally to prepare the work on “type inference” in a new feature branch. I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via “2to3″ conversion) without trouble. There is still work to do, exceptions don’t seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day.
And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository. I have entered Debian Mentors, and found a potential sponsor who may upload the packages in foreseeable time. There may be more review comments, but I expect that Nuitka may enter Debian proper.
- Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hotfix already.
- Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hotfix already.
- Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hotfix already.
- Fix, add the missing “staticmethod” decorator to “__new__” methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hotfix already.
- Enhanced compatibility again, provide enough “co_varnames” in the code objects, so that slicing them up to “code_object.co_argcount” will work. They are needed by “inspect” module and might be used by some decorators as well.
- New options to control the recursion:
- The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case.
- Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs.
- Use the pickle “protocol 2″ on CPython2 except for “unicode” strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of “pickle” was not really necessary anymore.
- Added a “Developer Manual” to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.
- Improved the “–help” output to use “metavar” where applicable. This makes it more readable for some options.
- Instead of error message, give help output when no module/program file name was given. This should make Nuitka help more convenient.
- Consistently use “#!/usr/bin/env python” for all scripts, this was previously only done for some of them.
- Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for “TODO” items optional.
- All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it’s also already the default for “Python”.
- The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
- Make the tests run without any dependence on “PATH” to contain the executables of Nuitka. This should make it easier to use.
- Add license texts to 3rd party file that were missing them, apply “licensecheck” results to cleanup Nuitka. Also removed own copyright statement from inline copy of Scons, it had been added by accident only.
- Release the tests that I own as well as the Debian packaging I created under “Apache License 2.0″ which is very liberal, meaning every project should be able to use it.
- Don’t require copyright assignment for contributions anymore, instead only “Apache License 2.0″, the future Nuitka license, so that the code won’t be a problem when changing the license of all of Nuitka to that license.
- Give contributors listed in the user manual an exception to the GPL terms until Nuitka is licensed under “Apache License 2.0″ as well.
- Added an “–experimental” option which can be used to control experimental features, like the one currently being added on “feature/ctypes_annotation”, where “type inference” is currently only activated when that option is given. For this stable release, it does nothing.
- Check the static C++ files of Nuitka with “cppcheck” as well. Didn’t find anything.
- Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit.
- Changed “not” boolean operation to become a normal operator. Changed “and” and “or” boolean operators to a new base class, and making their interface more similar to that of operations.
- Added cumulative “tags” to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
- Enhanced the “visitor” interface to provide more kinds of callbacks, enhanced the way “each scope” visiting is achieved by generalizing is as “child has not tag ‘closure_taker’” and that for every “node that has tag ‘closure_taker’”.
- Moved “SyntaxHighlighting” module to “nuitka.gui” package where it belongs.
- More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that should be white listed.
- Consistently use “nuitka” in test scripts, as there isn’t a “Nuitka.py” on all platforms. The later is scheduled for removal.
- Some more PyLint cleanups.
- Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail.
- The basic tests can now be run with “PYTHON=python3.2″ and use “2to3″ conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script “check-release”.
- Include the syntax tests in release tests as well.
- Changed many existing tests so that they can run under CPython3 too. Of course this is via “2to3″ conversion.
- Don’t fail if the CPython test suites are not there. Currently they remain largely unpublished, and as such are mostly only available to me (exception, “feature/minimize_CPython26_tests_diff” branch references the CPython2.6 tests repository, but that remains work in progress).
- For the compile itself test: Make the presence of the Scons inline copy optional, the Debian package doesn’t contain it.
- Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python.
- Before releasing, test that the created Debian package builds fine in a minimal Debian “unstable” chroot, and passes all the tests included in the package (“basics”, “syntax”, “programs”, “reflected”). Also many other Debian packaging improvements.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
The “git flow” was used again in this release cycle and proved to be useful not only for hotfix, but also for creating the branch “feature/ctypes_annotation” and rebasing it often while things are still flowing. The few hotfixes didn’t require a new release, but the many organisational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control.
The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn’t regress by accident, so running with CPython3.2 is now part of the normal release tests.
What’s still missing is more “hg” completeness. Only the “co_varnames” work for “inspect” was going in that direction, and this has slowed down. It was more important to make Nuitka’s recursion more accessible with the new options, so that was done first.
And of course, the real excitement is the the “type inference” work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of “ctypes” native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly.
Overall, this feels very good. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.
As always you will find its latest version here.
Yours,
Kay Hayen
This is to inform you about the new stable release of Nuitka. This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
The release cycle had a focus on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via “2to3″ conversion) without trouble. I don’t know when, but it seems that it’s going to work one day.
Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It’s still not there, but I am getting more helpful now.
- Fixed a packaging problem for Linux and x64 platform, the new “swapFiber.S” file for the fiber management was not included. Released as 0.3.15a hotfix already.
- Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hotfix already.
- Fixed an issue with “__import__” and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hotfix already.
- Fixed “eval()” and “locals()” to be supported in generator expressions and contractions too. Released as 0.3.15d hotfix already.
- Fixed the Windows batch files “nuitka.bat” and “nuitka-python.bat” to not output the “rem” statements with the copyright header. Released as 0.3.15d hotfix already.
- Fixed re-raise with “raise”, but without a current exception set. Released as 0.3.15e hotfix already.
- Fixed “vars()” call on the module level, needs to be treated as “globals()”. Released as 0.3.15e hotfix already.
- Fix handling of broken new lines in files. Read the source code in “universal line ending mode”. Released as 0.3.15f hotfix already.
- Fixed handling of constant module attribute “__name__” being replaced. Don’t replace local variables of the same name too. Released as 0.3.15g hotfix already.
- Fixed assigning to “True”, “False” or “None”. There was this old TODO, and some code has compatibility craft that does it. Released as 0.3.15g hotfix already.
- Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hotfix already.
- Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not.
- Fixed non-execution of cleanup codes that e.g. flush “sys.stdout”, by adding “Py_Finalize()”.
- Fix “throw()” method of generator expression objects to not check arguments properly.
- Fix missing fallback to subscript operations for slicing with non-indexable objects.
- Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did.
- Fix, the future spec was not properly preserving the future division flag.
- The optimization scales now much better, because per-module optimizations only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time.
- The creation of dictionaries from constants is now also optimized.
- The names “True”, “False” and “None” are now converted to constants only when they are read-only module variables.
- As a new feature functions now have the “func_defaults” and “__defaults__” attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though.
- The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
- The “PYTHONPATH” variable is now cleared when immediately executing a compiled binary unless “–execute-with-pythonpath” is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required.
- The inline copy of Scons is not checked with PyLint anymore. We of course don’t care.
- Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious.
- The basic tests can now be run with “PYTHON=python3.2″ and use “2to3″ conversion in that case.
- Moved “tags” to a separate module, make optimizations emit only documented tags, checked against the list of allowed ones.
- The Debian package has seen lots of improvements, to make it “lintian clean”, even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc.
- Use “os.path.join” is more of the test code.
- Some more PyLint cleanups.
- There is now a “Crasher” test, for tests that crashed Nuitka previously.
- Added a program test where the imported module does a “sys.exit()” and make sure it really doesn’t continue after the “SystemExit” exception that creates.
- Cover the type of “__builtins__” in the main program and in imported modules in tests too. It’s funny and differs between module and dict in CPython2.
- Cover a final print without newline in the test. Should still receive a newline, which only happens when “Py_Finalize()” is called.
- Added test with functions that makes a “raise” without an exception set.
- Cover the calling of “vars()” on module level too.
- Cover the use of eval in contractions and generator expressions too.
- Cover “func_defaults” and “__default__” attributes for a function too.
- Added test function with two “raise” in an exception handler, so that one becomes dead code and removed without the crash.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
The “git flow” was really great in this release cycle. There were many hotfix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects.
This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority.
The work on Python3 is a bit strange. I don’t need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can’t really stop to work on it. It has that appeal of small fixups here and there, and then something else works too. It’s like when I was struggling with Nuitka to pass the CPython2 unit tests the first time. It’s fun. And then it finds real actual bugs that apply to CPython2 too. Not doing “Py_Finalize” (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the early “Python3″ is supported, the more benefit from increased test covered.
What’s missing is more “hg” completeness. I think only the “raise” without exception set and the “func_defaults” issue were going into its direction, but it won’t be enough yet. Before that, the plan is to make Nuitka’s options to control recursion more accessible. Right now it requires you to embed all dist-packages or None. And there is no way to specify modules that should be included even if not directly used, which would be important for the hg plugins to use.
Overall, this feels very good. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.
As always you will find its latest version here.
Yours,
Kay Hayen
This is to inform you about the new stable release of Nuitka. This time again many organisational improvements, some bug fixes, much improved compatibility and cleanups.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
This release cycle had a focus on packaging Nuitka for easier consumption, i.e. automatic packaging, making automatic uploads, improvement documentation, and generally cleaning things up, so that Nuitka becomes more compatible and ultimately capable to run the “hg” test suite. It’s not there yet, but this is a huge jump for usability of Nuitka and its compatibility, again.
Then lots of changes that make Nuitka approach Python3 support, the generated C++ for at least one large example is compiling with this new release. It won’t link, but there will be later releases.
And there is a lot of cleanup going on, geared towards compatibility with line numbers in the frame object.
- The main module was using “__main__” in tracebacks, but it should be “<module>”. Released as 0.3.14a hotfix already.
- Workaround for “execfile cannot be used as an expression”. It wasn’t possible to use “execfile” in an expression, only as a statement. But then there is crazy enough code in e.g. mercurial that uses it in a lambda function, which made the issue more prominent. The fix now allows it to be an expression, except on the class level, which wasn’t seen yet.
- The inline copy of Scons was not complete enough to work for “Windows” or with “–windows-target” for cross compile. Fixed.
- Cached frames didn’t release the “back” frame, therefore holding variables of these longer than CPython does, which could cause ordering problems. Fixed for increased compatibility.
- Handle “yield outside of function” syntax error in compiled source correctly. This one was giving a Nuitka backtrace, now it gives a “SyntaxError” as it should.
- Made syntax/indentation error output absolutely identical to CPython.
- Using the frame objects “f_lineno” may fix endless amounts bugs related to traceback line numbers.
- Guesses the location of the MinGW compiler under Windows to default location, so it need not be added to “PATH” environment variable. Removes the need to modify “PATH” environment just for Nuitka to find it.
- Added support for “lambda generators”. You don’t want to know what it is. Lets just say, it was the last absurd language feature out there, that didn’t work. It now works perfect.
- You can now download a Windows installer and a Debian package that works on Debian Testing, current Ubuntu and Mint Linux.
- New release scripts give us the ability to have hotfix releases as download packages immediately. That means the “git flow” makes even more beneficial to the users.
- Including the generated “README.pdf” in the distribution archives, so it can be read instead of “README.txt”. The text file is fairly readable, due to the use of ReStructured Text, but the PDF is even nicer to read, due to e.g. syntax highlighting of the examples.
- Renamed the main binaries to “nuitka” and “nuitka-python”, so that there is no dependency on case sensitive file systems.
- For Windows there are batch files “nuitka.bat” and “nuitka-python.bat” to make Nuitka directly executable without finding the “Python.exe”, which the batch files can tell from their own location.
- There are now man pages of “nuitka” and “nuitka-python” with examples for the most common use cases. They are of course included in the Debian package.
- Don’t strip the binary when executing “benchmark.sh” to analyse compiled binary with “valgrind”. It will give better information that way, without changing the code.
- Implemented “swapcontext” alike (“swapFiber”) for x64 to achieve 8 times speedup for Generators. It doesn’t do useless syscalls to preserve signal masks. Now Nuitka is faster at frame switching than CPython on x64, which is already good by design.
- Using the frame objects to store current line of execution avoids the need to store it away in helper code at all. It ought to also help a lot with threading support, and makes Nuitka even more compatible, because now line numbers will be correct even outside tracebacks, but for mere stack frame dumps.
- Moved the “for_return” detection from code generation to tree building where it belongs. Yield statements used as return statements need slightly different code for Python2.6 difference. That solved an old TODO.
- Much Python3 portability work. Sometimes even improving existing code, the Python compiler code had picked up a few points, where the latest Nuitka didn’t work with Python3 anymore, when put to actual compile. The test covered only syntax, but e.g. metaclasses need different code in CPython3, and that’s now supported. Also helper code was made portable in more places, but not yet fully. This will need more work.
- Cleaned up uses of debug defines, so they are now more consistent and in one place.
- Some more PyLint cleanups.
- The tests are now executed by Python scripts and cover “stderr” output too. Before we only checked “stdout”. This unveiled a bunch of issues Nuitka had, but went unnoticed so far, and triggered e.g. the frame line number improvements.
- Separate syntax tests.
- The scripts to run the tests now are all in pure Python. This means, no more MinGW shell is needed to execute the tests.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
The Debian package, Windows installer, etc. are now automatically updated and uploaded. From here on, there can be such packages for the hotfix releases too.
The exception tracebacks are now correct by design, and better covered.
The generator performance work showed that the approach taken by Nuitka is in fact fast. It was fast on ARM already, but it’s nice to see that it’s now fast on x64. Programs using generators will be affected a lot by this.
Overall, this release brings Nuitka closer to usability. Better binary names, man pages, improved documentation, issue tracker, etc. all there now. I am in fact now looking for a sponsor for the Debian package to upload it into Debian directly.
What’s missing is more “hg” completeness. The frame release issue helped it, but “inspect.getargs()” doesn’t work yet, and is a topic for a future release. Won’t be easy, as “func_defaults” will be an invasive change too.
But I feel now certain, that the milestone “compatibility” may reach that level “compiled hg passes all tests” soonish. Then the focus will turn towards performance. Nuitka will then too compatible too believe, a good base.
As always you will find its latest version here.
Yours,
Kay Hayen
There is now a Windows installer and a Debian package of Nuitka available on the Download page. Please try it out and give me feedback.
Specifically I do know that the Debian package won’t work on Debian Squeeze, but only on Debian Wheezy (Testing) maybe it does on Ubuntu as well, please report. If you have anything to criticize about the package, let me know. There is no apt source now, I hope to get it into Debian proper directly.
UPDATE: After Stani’s report that Ubuntu has an older Scons, I lowered the dependency and updated the package on the Download page. It may now work on Ubuntu as well.
And then, the Windows installer still requires you to install MinGW and add it to your path, but it’s clearly a huge step ahead. It’s created with distutils, and that works very well. If you have the skills to enhance it, so e.g. the PATH variable is changed, or it will launch a MinGW install if not present, contact me and help.
UPDATE: And the idea that I got while writing a reply to “swong” is also now implemented. The new Nuitka on Windows simply guesses the PATH to MinGW to be the default path “C:\MinGW” or at least “\MinGW” and from there, it ought to just run after you installed it. Of course you can still set your own PATH environment and make the pick yourself.
Yours,
Kay Hayen
This is to inform you about the new stable release of Nuitka. This time it contains mostly organisational improvements, some bug fixes, improved compatibility and cleanups.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
This release is again the result of working towards compilation of a real program (Mercurial). This time, I have added support for proper handling of compiled types by the “inspect” module.
- Fix for “Missing checks in parameter parsing with star list, star dict and positional arguments”. There as whole in the checks for argument counts, now the correct error is given. Fixed in 0.3.13a already.
- The simple slice operations with 2 values, not extended with 3 values, were not applying the correct order for evaluation. Fixed in 0.3.13a already.
- The simple slice operations couldn’t handle “None” as the value for lower or upper index. Fixed in 0.3.11a already.
- The inplace simple slice operations evaluated the slice index expressions twice, which could cause problems if they had side effects. Fixed in 0.3.11a already.
- Run time patching the “inspect” module so it accepts compiled functions, compiled methods, and compiled generator objects. The “test_inspect” test of CPython is nearly working unchanged with this.
- The generator functions didn’t have “CO_GENERATOR” set in their code object, made compatible with CPython in this regard too. The inspect module will therefore return correct value for “inspect.isgeneratorfunction()” too.
- Slice indexes that are None are now constant propagated as well.
- Slightly more efficient code generation for dual star arg functions, removing useless checks.
- Moved the scons, static C++ files, and asm files to new package “nuitka.build” where also now “SconsInterface” module lives.
- Moved the Qt dialog files to “nuitka.gui”
- Moved the unfreezer code to its own static C++ file.
- Some pylint cleanups.
- New test “Recursion” to cover recursive functions.
- New test “Inspection” to cover the patching of “inspect” module.
- Cover “execfile” on the class level as well in “ExecEval” test.
- Cover evaluation order of simple slices in “OrderCheck” too.
- There is a new issue tracker available under http://bugs.nuitka.net
Please register and report issues you encounter with Nuitka. I have put all the known issues there and started to use it recently. It’s Roundup based like bugs.python.org is, so people will find it familiar.
- The “setup.py” is apparently functional. The source releases for download are made it with, and it appears the binary distributions work too. We may now build a windows installer. It’s currently in testing, we will make it available when finished.
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
The new source organisation makes packaging Nuitka really easy now. From here, we can likely provide “binary” package of Nuitka soon. A windows installer will be nice.
The patching of “inspect” works wonders for compatibility for those programs that insist on checking types, instead of doing duck typing. The function call problem, was an issue found by the Mercurial test suite.
For the “hg.exe” to pass all of its test suite, more work may be needed, this is the overall goal I am currently striving for. Once real world programs like “hg” work, we can use these as more meaningful benchmarks and resume work on optimization.
As always you will find its latest version here.
Yours,
Kay Hayen
