Ubuntu Packages for Nuitka

Hello there,

as I previously said, I will make these a separate announcement. Thanks to the NeuroDebian project build infrastructure, there is Nuitka packages for the Ubuntu versions back to Maverick, Natty, and Oneiric.

Please see the page What is Nuitka? for clarification of what it is now and what it wants to be.

You may use the apt sources to track Nuitka or download the packages from the page where I offer the latest version here.

Note: In fact, it's 0.3.19.1 now, which includes a hotfix release made after 0.3.19. The same is in Debian Unstable right now, waiting to propagate into Debian Testing, where there is still 0.3.18 living at this time.

I would appreciate feedback on these.

Thanks in advance, Kay

Nuitka Release 0.3.19

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things in this are justifying a new release.

Bug fixes

  • The man pages of nuitka and nuitka-python had no special layout for the option groups and broken whitespace for --recurse-to option. Also --g++-only was only partially bold. Released as 0.3.18.1 hot fix already.
  • The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hot fix already.
  • Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of / instead of using a joinpath call. Released as 0.3.18.3 already.
  • A call to the range built-in with no arguments would crash the compiler, see Issue#29. Released as 0.3.18.4 already.
  • Compatibility Fix: When rich comparison operators returned false value other False, for comparison chains, these would not be used, but False instead, see .
  • The support for __import__ didn't cover keyword arguments, these were simply ignored. See Issue#28. Fixed, but no warning is given yet.

New Features

  • A new option has been added, one can now specify --recurse-directory and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements.
  • Added support for the exec built-in of Python3, this enables us to run one more basic test, GlobalStatement.py with Python3. The test ExecEval.py nearly works now.

New Optimization

  • The no arguments range() call now optimized into the static CPython exception it raises.
  • Parts of comparison chains with constant arguments are now optimized away.

Cleanups

  • Simplified the CPythonExpressionComparison node, it now always has only 2 operands.

    If there are more, the so called "comparison chain", it's done via and with assignments to temporary variables, which are expressed by a new node type CPythonExpressionTempVariableRef. This allowed to remove expression_temps from C++ code templates and generation, reducing the overall complexity.

  • When executing a module (--execute but not --exe), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment.

  • The calls to the variadic MAKE_TUPLE were replaced with calls the MAKE_TUPLExx (where xx is the number of arguments), that are generated on a as-needed basis. This gives more readable code, because no EVAL_ORDERED_xx is needed at call site anymore.

  • Many node classes have moved to new modules in nuitka.nodes and grouped by theme. That makes them more accessible.

  • The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the sys.abiflags and works with Python3.

  • The replacing of .py in filenames was made more robust. No longer is str.replace used, but instead proper means to assure that having .py as other parts of the filenames won't be a trouble.

  • Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements.

  • As always, some PyLint work, and some minor TODO were solved.

Organizational

  • Added more information to the "Developer Manual", e.g. documenting the tree changes for assert to become a conditional statement with a raise statement, etc.
  • The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise.
  • Added support to specify the binary under test with a NUITKA environment, so the test framework can run with installed version of Nuitka too.
  • Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for os.execl not propagating exit codes under Windows. See Issue#26 for more information.
  • For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the PATH or even installed to execute the binary.

New Tests

  • The basic, programs, syntax, and reflected were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations.
  • Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains.
  • The Referencing.py test was made portable to Python3.
  • Cover no arguments range() exception as well.
  • Added test to demonstrate that --recurse-directory actually works. This is using an __import__ that cannot be predicted at run time (yet).
  • The created source package is now tested on pbuilder chroots to be pass installation and the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn't break without notice.

Summary

This releases contains many changes. The "temporary variable ref" and "assignment expression" work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. in-place assignments can be reduced to assignments to temporary variables and conditional statements.

While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with "value propagation" working.

These are the real focus. The old comparison chain handling was a big wart. Working, but no way understood by any form of analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimization to see through them.

Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of in-place assignments. With an "assignment expression" to a "temporary variable ref", these become the same as user code using such a variable. There will be more of these to find.

So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the __import__ related one, were not suitable for hot fix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already, also worth to get out.

Static Site Generator

Hello,

somehow triggered by reading about Mynth, and the re-post of the last release announcement on my feed, just because I added a missing category to the post, I am wondering what it takes to achieve the following:

  • Edit the pages site as ReST (reStructured text) document with links inside
  • Edit posts as single ReST files
  • Have it look decent and provide feeds, proper caching headers, etc.
  • Generate a static site from with, with more control over what kind of updates makes it into the feed.

I am currently very annoyed, because:

  • I would normally generate code examples now with "rst2html" and replace the existing Wordpress plugin that renders in JavaScript. Except I cannot without re-posting all these articles, or removing the "python" tag from them.
  • The documentation of Nuitka is in ReST and I even intend to expand it with UML diagrams generated from text. Making the cross to HTML content is difficult.
  • Some pages only replicate content from the User or Developer Manual, keeping those updated, in sync, etc. is a dull thing to do.
  • Extending Wordpress in Python is not feasible. And in php, I doubt I want to master.

Other stuff in favor of it, is that Wordpress or any CMS needs more memory and more time to process than static pages of course. I can't right now let anyone fork and edit the pages in a practical manner. So while for user manual and developer manual of Nuitka I will get corrections, for the website it's not happening.

Is Mynth an appropiate intermediate step? Do I really want to learn one of these strange templating languages or even consider software that has no Debian package. I never considered "virtualenv" and "pip" much of options to run software on a website. I guess I am more tied to Debian than to Python still.

Over the last months I have occasionally looked at Pyramids, but it seems overly complex, if all you want is to simplify the authoring. I have seen that e.g. Trac supports ReST as an exception, but that feels too little for a hefty software like that.

I may end up with using ReST to generate HTML which is then uploaded via XMLRPC to Wordpress. Won't give me the "static" benefits. Won't solve the post update problem. But at least will let me edit more easily.

Yours, Kay

Nuitka Release 0.3.18

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via apt-get install nuitka.

The release cycle was too short to have much focus. It merely includes fixes, which were available as hot fixes, and some additional optimization and node tree cleanups, as well as source cleanups. But not much else.

Bug fixes

  • Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. Issue#16. Released as 0.3.17a hot fix already.
  • Nuitka was considering directories to contain packages that had no "__init__.py" which could lead to errors when it couldn't find the package later in the compilation process. Released as 0.3.17a hot fix already.
  • When providing locals() to exec statements, this was not making the locals() writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn't consider the actual value locals(). Released as 0.3.17b hot fix already.
  • Compatibility Fix: When no defaults are given, CPython uses None for func.func_defaults, but Nuitka had been using None.

New Optimization

  • If the condition of assert statements can be predicted, these are now optimized in a static raise or removed.
  • For built-in name references, there is now dedicated code to look them up, that doesn't check the module level at all. Currently these are used in only a few cases though.
  • Cleaner code is generated for the simple case of print statements. This is not only faster code, it's also more readable.

Cleanups

  • Removed the CPythonStatementAssert node.

    It's not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with AssertionError and the assertion argument.

    This allowed to remove code and complexity from the subsequent steps of Nuitka, and enabled existing optimization to work on assert statements as well.

  • Moved built-in exception names and built-in names to a new module nuitka.Builtins instead of having in other places. This was previously a bit spread-out and misplaced.

  • Added cumulative tags to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.

  • New node for built-in name loopups, which allowed to remove tricks played with adding module variable lookups for staticmethod when adding them for __new__ or module variable lookups for str when predicting the result of type('a'), which was unlikely to cause a problem, but an important TODO item still.

Organizational

  • The "Download" page is now finally updated for releases automatically.

    This closes Issue#7 completely. Up to this release, I had to manually edit that page, but now mastered the art of upload via XMLRCP and a Python script, so that don't loose as much time with editing, checking it, etc.

  • The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages.

  • Made sure the test runners worth with bare python2.6 as well.

New Tests

  • Added some tests intended for type inference development.

Summary

This releases contains not as much changes as others, mostly because it's the intended base for a Debian upload.

The exec fix was detected by continued work on the branch feature/minimize_CPython26_tests_diff branch, but that work is now complete.

It is being made pretty (many git rebase iterations) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made.

But of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient.

So when I learn something. e.g. assert is not special, I apply it to the develop branch immediately, to keep the differences as small as possible, and to immediately benefit from such improvements.

Nuitka Release 0.3.17

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organisational work, and yet again improved compatibility and cleanups. Also huge is the advance in making --deep go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one.

The release cycle had a focus on improving the quality of the test scripts, the packaging, and generally to prepare the work on "type inference" in a new feature branch.

I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via 2to3 conversion) without trouble. There is still work to do, exceptions don't seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day.

And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository.

Bug fixes

  • Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hot fix already.
  • Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hot fix already.
  • Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hot fix already.
  • Fix, add the missing staticmethod decorator to __new__ methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hot fix already.

New Features

  • Enhanced compatibility again, provide enough co_varnames in the code objects, so that slicing them up to code_object.co_argcount will work. They are needed by inspect module and might be used by some decorators as well.

  • New options to control the recursion:

    --recurse-none (do not warn about not-done recursions) --recurse-all (recurse to all otherwise warned modules) --recurse-to (confirm to recurse to those modules) --recurse-not-to (confirm to not recurse to those modules)

New Optimization

  • The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case.
  • Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs.
  • Use the pickle "protocol 2" on CPython2 except for unicode strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of pickle was not really necessary anymore.

Organizational

  • Added a "Developer Manual" to the release. It's incomplete, but it details some of the existing stuff, coding rules, plans for "type inference", etc.
  • Improved the --help output to use metavar where applicable. This makes it more readable for some options.
  • Instead of error message, give help output when no module or program file name was given. This makes Nuitka help out more convenient.
  • Consistently use #!/usr/bin/env python for all scripts, this was previously only done for some of them.
  • Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for TODO items optional.
  • All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it's also already the default for Python.
  • The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
  • Make the tests run without any dependence on PATH to contain the executables of Nuitka. This makes it easier to use.
  • Add license texts to 3rd party file that were missing them, apply licensecheck results to cleanup Nuitka. Also removed own copyright statement from in-line copy of Scons, it had been added by accident only.
  • Release the tests that I own as well as the Debian packaging I created under "Apache License 2.0" which is very liberal, meaning every project will be able to use it.
  • Don't require copyright assignment for contributions anymore, instead only "Apache License 2.0", the future Nuitka license, so that the code won't be a problem when changing the license of all of Nuitka to that license.
  • Give contributors listed in the "User Manual" an exception to the GPL terms until Nuitka is licensed under "Apache License 2.0" as well.
  • Added an --experimental option which can be used to control experimental features, like the one currently being added on feature/ctypes_annotation, where "type inference" is currently only activated when that option is given. For this stable release, it does nothing.
  • Check the static C++ files of Nuitka with cppcheck as well. Didn't find anything.
  • Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit.

Cleanups

  • Changed not boolean operation to become a normal operator. Changed and and or boolean operators to a new base class, and making their interface more similar to that of operations.
  • Added cumulative tags to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
  • Enhanced the "visitor" interface to provide more kinds of callbacks, enhanced the way "each scope" visiting is achieved by generalizing is as "child has not tag 'closure_taker'" and that for every "node that has tag 'closure_taker'".
  • Moved SyntaxHighlighting module to nuitka.gui package where it belongs.
  • More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that were white listed.
  • Consistently use nuitka in test scripts, as there isn't a Nuitka.py on all platforms. The later is scheduled for removal.
  • Some more PyLint cleanups.

New Tests

  • Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail.

  • The basic tests can now be run with PYTHON=python3.2 and use 2to3 conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script check-release.

  • Include the syntax tests in release tests as well.

  • Changed many existing tests so that they can run under CPython3 too. Of course this is via 2to3 conversion.

  • Don't fail if the CPython test suites are not there.

    Currently they remain largely unpublished, and as such are mostly only available to me (exception, feature/minimize_CPython26_tests_diff branch references the CPython2.6 tests repository, but that remains work in progress).

  • For the compile itself test: Make the presence of the Scons in-line copy optional, the Debian package doesn't contain it.

  • Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python.

  • Before releasing, test that the created Debian package builds fine in a minimal Debian unstable chroot, and passes all the tests included in the package (basics, syntax, programs, reflected). Also many other Debian packaging improvements.

Summary

The "git flow" was used again in this release cycle and proved to be useful not only for hot fix, but also for creating the branch feature/ctypes_annotation and rebasing it often while things are still flowing.

The few hot fixes didn't require a new release, but the many organizational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control.

The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn't regress by accident, so running with CPython3.2 is now part of the normal release tests.

What's still missing is more "hg" completeness. Only the co_varnames work for inspect was going in that direction, and this has slowed down. It was more important to make Nuitka's recursion more accessible with the new options, so that was done first.

And of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of ctypes native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly.

Nuitka Release 0.3.16

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups.

The release cycle had a focus on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via 2to3 conversion) without trouble. I don't know when, but it seems that it's going to work one day.

Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It's still not there, but it's making progress.

Bug fixes

  • Fixed a packaging problem for Linux and x64 platform, the new swapFiber.S file for the fiber management was not included. Released as 0.3.15a hot fix already.
  • Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hot fix already.
  • Fixed an issue with __import__ and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hot fix already.
  • Fixed eval() and locals() to be supported in generator expressions and contractions too. Released as 0.3.15d hot fix already.
  • Fixed the Windows batch files nuitka.bat and nuitka-python.bat to not output the rem statements with the copyright header. Released as 0.3.15d hot fix already.
  • Fixed re-raise with raise, but without a current exception set. Released as 0.3.15e hot fix already.
  • Fixed vars() call on the module level, needs to be treated as globals(). Released as 0.3.15e hot fix already.
  • Fix handling of broken new lines in source files. Read the source code in "universal line ending mode". Released as 0.3.15f hot fix already.
  • Fixed handling of constant module attribute __name__ being replaced. Don't replace local variables of the same name too. Released as 0.3.15g hot fix already.
  • Fixed assigning to True, False or None. There was this old TODO, and some code has compatibility craft that does it. Released as 0.3.15g hot fix already.
  • Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hot fix already.
  • Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not.
  • Fixed non-execution of cleanup codes that e.g. flush sys.stdout, by adding Py_Finalize().
  • Fix throw() method of generator expression objects to not check arguments properly.
  • Fix missing fallback to subscript operations for slicing with non-indexable objects.
  • Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did.
  • Fix, the future spec was not properly preserving the future division flag.

New Optimization

  • The optimization scales now much better, because per-module optimization only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time.
  • The creation of dictionaries from constants is now also optimized.

New Features

  • As a new feature functions now have the func_defaults and __defaults__ attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though.
  • The names True, False and None are now converted to constants only when they are read-only module variables.
  • The PYTHONPATH variable is now cleared when immediately executing a compiled binary unless --execute-with-pythonpath is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required.

Organizational

  • The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together.
  • The in-line copy of Scons is not checked with PyLint anymore. We of course don't care.
  • Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious.
  • The basic tests can now be run with PYTHON=python3.2 and use 2to3 conversion in that case.

Cleanups

  • Moved tags to a separate module, make optimization emit only documented tags, checked against the list of allowed ones.
  • The Debian package has seen lots of improvements, to make it "lintian clean", even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc.
  • Use os.path.join in more of the test code to achieve more Windows portability for them.
  • Some more PyLint cleanups.

New Tests

  • There is now a Crasher test, for tests that crashed Nuitka previously.
  • Added a program test where the imported module does a sys.exit() and make sure it really doesn't continue after the SystemExit exception that creates.
  • Cover the type of __builtins__ in the main program and in imported modules in tests too. It's funny and differs between module and dict in CPython2.
  • Cover a final print statement without newline in the test. Must still receive a newline, which only happens when Py_Finalize() is called.
  • Added test with functions that makes a raise without an exception set.
  • Cover the calling of vars() on module level too.
  • Cover the use of eval in contractions and generator expressions too.
  • Cover func_defaults and __default__ attributes for a function too.
  • Added test function with two raise in an exception handler, so that one becomes dead code and removed without the crash.

Summary

The "git flow" was really great in this release cycle. There were many hot fix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects.

This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority, correctness is.

The work on Python3 is a bit strange. I don't need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can't really stop to work on it. It has that appeal of small fixups here and there, and then something else works too.

Python3 work is like when I was first struggling with Nuitka to pass the CPython2 unit tests for a first time. It's fun. And then it finds real actual bugs that apply to CPython2 too. Not doing Py_Finalize (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the earlier Python3 is supported, the more benefit from increased test covered.

What's missing is more "hg" completeness. I think only the raise without exception set and the func_defaults issue were going into its direction, but it won't be enough yet.

Nuitka Release 0.3.15

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new stable release of Nuitka. This time again many organizational improvements, some bug fixes, much improved compatibility and cleanups.

This release cycle had a focus on packaging Nuitka for easier consumption, i.e. automatic packaging, making automatic uploads, improvement documentation, and generally cleaning things up, so that Nuitka becomes more compatible and ultimately capable to run the "hg" test suite. It's not there yet, but this is a huge jump for usability of Nuitka and its compatibility, again.

Then lots of changes that make Nuitka approach Python3 support, the generated C++ for at least one large example is compiling with this new release. It won't link, but there will be later releases.

And there is a lot of cleanup going on, geared towards compatibility with line numbers in the frame object.

Bug fixes

  • The main module was using __main__ in tracebacks, but it must be <module>. Released as 0.3.14a hot fix already.

  • Workaround for "execfile cannot be used as an expression". It wasn't possible to use execfile in an expression, only as a statement.

    But then there is crazy enough code in e.g. mercurial that uses it in a lambda function, which made the issue more prominent. The fix now allows it to be an expression, except on the class level, which wasn't seen yet.

  • The in-line copy of Scons was not complete enough to work for "Windows" or with --windows-target for cross compile. Fixed.

  • Cached frames didn't release the "back" frame, therefore holding variables of these longer than CPython does, which could cause ordering problems. Fixed for increased compatibility.

  • Handle "yield outside of function" syntax error in compiled source correctly. This one was giving a Nuitka backtrace, now it gives a SyntaxError as it needs to.

  • Made syntax/indentation error output absolutely identical to CPython.

  • Using the frame objects f_lineno may fix endless amounts bugs related to traceback line numbers.

New Features

  • Guesses the location of the MinGW compiler under Windows to default install location, so it need not be added to PATH environment variable. Removes the need to modify PATH environment just for Nuitka to find it.
  • Added support for "lambda generators". You don't want to know what it is. Lets just say, it was the last absurd language feature out there, plus that didn't work. It now works perfect.

Organizational

  • You can now download a Windows installer and a Debian package that works on Debian Testing, current Ubuntu and Mint Linux.
  • New release scripts give us the ability to have hot fix releases as download packages immediately. That means the "git flow" makes even more beneficial to the users.
  • Including the generated "README.pdf" in the distribution archives, so it can be read instead of "README.txt". The text file is fairly readable, due to the use of ReStructured Text, but the PDF is even nicer to read, due to e.g. syntax highlighting of the examples.
  • Renamed the main binaries to nuitka and nuitka-python, so that there is no dependency on case sensitive file systems.
  • For Windows there are batch files nuitka.bat and nuitka-python.bat to make Nuitka directly executable without finding the Python.exe, which the batch files can tell from their own location.
  • There are now man pages of nuitka and nuitka-python with examples for the most common use cases. They are of course included in the Debian package.
  • Don't strip the binary when executing it to analyse compiled binary with valgrind. It will give better information that way, without changing the code.

New Optimization

  • Implemented swapcontext alike (swapFiber) for x64 to achieve 8 times speedup for Generators. It doesn't do useless syscalls to preserve signal masks. Now Nuitka is faster at frame switching than CPython on x64, which is already good by design.

Cleanups

  • Using the frame objects to store current line of execution avoids the need to store it away in helper code at all. It ought to also help a lot with threading support, and makes Nuitka even more compatible, because now line numbers will be correct even outside tracebacks, but for mere stack frame dumps.

  • Moved the for_return detection from code generation to tree building where it belongs. Yield statements used as return statements need slightly different code for Python2.6 difference. That solved an old TODO.

  • Much Python3 portability work. Sometimes even improving existing code, the Python compiler code had picked up a few points, where the latest Nuitka didn't work with Python3 anymore, when put to actual compile.

    The test covered only syntax, but e.g. meta classes need different code in CPython3, and that's now supported. Also helper code was made portable in more places, but not yet fully. This will need more work.

  • Cleaned up uses of debug defines, so they are now more consistent and in one place.

  • Some more PyLint cleanups.

New Tests

  • The tests are now executed by Python scripts and cover stderr output too. Before we only checked stdout. This unveiled a bunch of issues Nuitka had, but went unnoticed so far, and triggered e.g. the frame line number improvements.
  • Separate syntax tests.
  • The scripts to run the tests now are all in pure Python. This means, no more MinGW shell is needed to execute the tests.

Summary

The Debian package, Windows installer, etc. are now automatically updated and uploaded. From here on, there can be such packages for the hot fix releases too.

The exception tracebacks are now correct by design, and better covered.

The generator performance work showed that the approach taken by Nuitka is in fact fast. It was fast on ARM already, but it's nice to see that it's now also fast on x64. Programs using generators will be affected a lot by this.

Overall, this release brings Nuitka closer to usability. Better binary names, man pages, improved documentation, issue tracker, etc. all there now. I am in fact now looking for a sponsor for the Debian package to upload it into Debian directly.

Update

The upload to Debian happened for 0.3.18 and was done by Yaroslav Halchenko.

What's missing is more "hg" completeness. The frame release issue helped it, but inspect.getargs() doesn't work yet, and is a topic for a future release. Won't be easy, as func_defaults will be an invasive change too.

Nuitka Debian Package and Windows Installer

There is now a Windows installer and a Debian package of Nuitka available on the Download page. Please try it out and give me feedback.

Specifically I do know that the Debian package won't work on Debian Squeeze, but only on Debian Wheezy (Testing) maybe it does on Ubuntu as well, please report. If you have anything to criticize about the package, let me know. There is no apt source now, I hope to get it into Debian proper directly.

UPDATE: After Stani's report that Ubuntu has an older Scons, I lowered the dependency and updated the package on the Download page. It may now work on Ubuntu as well.

And then, the Windows installer still requires you to install MinGW and add it to your path, but it's clearly a huge step ahead. It's created with distutils, and that works very well. If you have the skills to enhance it, so e.g. the PATH variable is changed, or it will launch a MinGW install if not present, contact me and help.

UPDATE: And the idea that I got while writing a reply to "swong" is also now implemented. The new Nuitka on Windows simply guesses the PATH to MinGW to be the default path C:\MinGW or at least \MinGW and from there, it ought to just run after you installed it. Of course you can still set your own PATH environment and make the pick yourself.

Yours, Kay Hayen

Nuitka Release 0.3.14

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This is to inform you about the new stable release of Nuitka. This time it contains mostly organisational improvements, some bug fixes, improved compatibility and cleanups.

It is again the result of working towards compilation of a real program (Mercurial). This time, I have added support for proper handling of compiled types by the inspect module.

Bug fixes

  • Fix for "Missing checks in parameter parsing with star list, star dict and positional arguments". There was whole in the checks for argument counts, now the correct error is given. Fixed in 0.3.13a already.
  • The simple slice operations with 2 values, not extended with 3 values, were not applying the correct order for evaluation. Fixed in 0.3.13a already.
  • The simple slice operations couldn't handle None as the value for lower or upper index. Fixed in 0.3.11a already.
  • The in-place simple slice operations evaluated the slice index expressions twice, which could cause problems if they had side effects. Fixed in 0.3.11a already.

New Features

  • Run time patching the inspect module so it accepts compiled functions, compiled methods, and compiled generator objects. The test_inspect test of CPython is nearly working unchanged with this.
  • The generator functions didn't have CO_GENERATOR set in their code object, setting it made compatible with CPython in this regard too. The inspect module will therefore return correct value for inspect.isgeneratorfunction() too.

New Optimization

  • Slice indexes that are None are now constant propagated as well.
  • Slightly more efficient code generation for dual star arg functions, removing useless checks.

Cleanups

  • Moved the Scons, static C++ files, and assembler files to new package nuitka.build where also now SconsInterface module lives.
  • Moved the Qt dialog files to nuitka.gui
  • Moved the "unfreezer" code to its own static C++ file.
  • Some PyLint cleanups.

New Tests

  • New test Recursion to cover recursive functions.
  • New test Inspection to cover the patching of inspect module.
  • Cover execfile on the class level as well in ExecEval test.
  • Cover evaluation order of simple slices in OrderCheck too.

Organizational

  • There is a new issue tracker available under http://bugs.nuitka.net

    Please register and report issues you encounter with Nuitka. I have put all the known issues there and started to use it recently. It's Roundup based like http://bugs.python.org is, so people will find it familiar.

  • The setup.py is now apparently functional. The source releases for download are made it with, and it appears the binary distributions work too. We may now build a windows installer. It's currently in testing, we will make it available when finished.

Summary

The new source organisation makes packaging Nuitka really easy now. From here, we can likely provide "binary" package of Nuitka soon. A windows installer will be nice.

The patching of inspect works wonders for compatibility for those programs that insist on checking types, instead of doing duck typing. The function call problem, was an issue found by the Mercurial test suite.

For the "hg.exe" to pass all of its test suite, more work may be needed, this is the overall goal I am currently striving for. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization.

Nuitka Release 0.3.13

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is mostly the result of working towards compilation of a real programs (Mercurial) and to merge and finalize the frame stack work. Now Nuitka has a correct frame stack at all times, and supports func_code and gi_code objects, something previously thought to be impossible.

Actually now it's only the "bytecode" objects that won't be there. And not attributes of func_code are meaningful yet, but in theory can be supported.

Due to the use of the "git flow" for Nuitka, most of the bugs listed here were already fixed in on the stable release before this release. This time there were 5 such hot fix releases, sometimes fixing multiple bugs.

Bug fixes

  • In case of syntax errors in the main program, an exception stack was giving that included Nuitka code. Changed to make the same output as CPython does. Fixed in 0.3.12a already.

  • The star import (from x import *) didn't work for submodules. Providing * as the import list to the respective code allowed to drop the complex lookups we were doing before, and to simply trust CPython C/API to do it correctly. Fixed in 0.3.12 already.

  • The absolute import is not the default of CPython 2.7 it seems. A local posix package shadows the standard library one. Fixed in 0.3.12 already.

  • In --deep mode, a module may contain a syntax error. This is e.g. true of "PyQt" with port_v3 included. These files contain Python3 syntax and fail to be imported in Python2, but that is not to be considered an error. These modules are now skipped with a warning. Fixed in 0.3.12b already.

  • The code to import modules wasn't using the __import__ built-in, which prevented __import__ overriding code to work. Changed import to use the built-in. Fixed in 0.3.12c already.

  • The code generated for the __import__ built-in with constant values was doing relative imports only. It needs to attempt relative and absolut imports. Fixed in 0.3.12c already.

  • The code of packages in "__init__.py" believed it was outside of the package, giving problems for package local imports. Fixed in 0.3.12d already.

  • It appears that "Scons", which Nuitka uses internally and transparent to you, to execute the compilation and linking tasks, was sometimes not building the binaries or shared libraries, due to a false caching. As a workaround, these are now erased before doing the build. Fixed in 0.3.12d already.

  • The use of in and not in in comparison chains (e.g. a < b < c is one), wasn't supported yet. The use of these in comparison chains a in b in c is very strange.

    Only in the test_grammar.py it was ever used I believe. Anyway, it's supported now, solving this TODO and reducing the difference. Fixed in 0.3.12e already.

  • The order of evaluation for in and not in operators wasn't enforced in a portable way. Now it is correct on "ARM" too. Fixed in 0.3.12e already.

New Optimization

  • The built-ins GeneratorExit and StopIteration are optimized to their Python C/API names where possible as well.

Cleanups

  • The __file__ attribute of modules was the relative filename, but for absolute filenames these become a horrible mess at least on Linux.
  • Added assertion helpers for sane frame and code objects and use them.
  • Make use of assertObject in more places.
  • Instead of using os.path.sep all over, added a helper Utils.joinpath that hides this and using os.path.join. This gives more readable code.
  • Added traces to the "unfreezer" guarded by a define. Helpful in analyzing import problems.
  • Some PyLint cleanups removing dead code, unused variables, useless pass statement, etc.

New Tests

  • New tests to cover SyntaxError and IndentationError from --deep imports and in main program.
  • New test to cover evaluation order of in and not in comparisons.
  • New test to cover package local imports made by the "__init__.py" of the package.

Organizational

  • Drop "compile_itself.sh" in favor of the new "compile_itself.py", because the later is more portable.
  • The logging output is now nicer, and for failed recursions, outputs the line that is having the problem.

Summary

The frame stack work and the func_code are big for compatibility.

The func_code was also needed for "hg" to work. For Mercurial to pass all of its test suite, more work will be needed, esp. the inspect module needs to be run-time patched to accept compiled functions and generators too.

Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization.