Python Assert Quiz

At Europython conference, in my presentation, I talked about re-formulations of Python into simpler Python. It is my intention to turn this into a series of Python quiz questions that you will hopefully enjoy.

Update

Due to comments feedback, I made it more clear that "-O" affects of course both cases, and due to work on getting the recent CPython2.7 test suite to work, I noticed, how the re-formulation for Quiz Question 2 needed a version dependent solution.

And I thought this one was easy. :-)

Quiz Question 1

Say you have the following code:

assert x == y

How can you achieve the same thing, without using the assert statement at all. The behavior is required to be absolutely the same.

The answer is in the next paragraph, so stop reading if you want to find out yourself.

Solution 1

The correct answer is that assertions are the same as a raise exception in a conditional statement.

if not x == y:
    raise AssertionError

The thing where this makes a difference, is "-O", which will discard assertions, but I consider it rarely used. To be really compatible with that, it should be:

if __debug__ and not x == y:
    raise AssertionError

Quiz Question 2

But wait, there is slightly more to it. Say you have the following code:

assert x == y, arg

How can you achieve the same thing, without using the assert statement at all. The behavior is required to be absolutely the same.

The answer is in the next paragraph, so stop reading if you want to find out yourself.

Solution 2

This is actually version dependent, due to recent optimizations of CPython.

  • For version 2.6 it is as follows:

    The extra value to assert, simply becomes an extra value to raise, which indicates, delayed creation of the AssertionError exception.

    if not x == y:
        raise AssertionError, arg
    
  • For version 2.7 and higher it is as follows:

    The extra value to assert, simply becomes the argument to creating the AssertionError exception.

    if not x == y:
        raise AssertionError( arg )
    

So, even in the more complex case, you end up with a conditional raise.

The only thing where this makes a difference, is "-O", which will discard assertions, but I consider it rarely used. To be really compatible with that, it should be:

if __debug__ and not x == y:
   raise AssertionError ....

Surprised? Well, yes, there really is nothing to assert statements. I am using this for my Python compiler Nuitka which benefits from having not to deal with assert as anything special at all. See also the respective section in the Developer Manual which explains this and other things.

Nuitka and Debian changes

Hello everybody,

this is just a quick note to Nuitka users that are Debian users. For Debian, the freeze has arrived. This is a time of great joy for me. I love Debian so deeply, I believe I am now for at least 10 years on it, of 15 years on Linux.

As you may know, Nuitka has been in Debian for some time now. In discussion with my great sponsor Yaroslav Halchenko, we needed to consider if Nuitka is ready to be maintained in Debian Stable.

And for that I have to admit, it is just not ready. You will be able to download packages from the Downloads and install them. They will still be in Debian Unstable, but they won't be in Debian Testing and the next Debian Stable.

Nuitka Release 0.3.23

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is the one that completes the Nuitka "sun rise phase".

All of Nuitka is now released under Apache License 2.0 which is a very liberal license, and compatible with basically all Free Software licenses there are. It's only asking to allow integration, of what you send back, and patent grants for the code.

In the first phase of Nuitka development, I wanted to keep control over Nuitka, so it wouldn't repeat mistakes of other projects. This is no longer a concern for me, it's not going to happen anymore.

I would like to thank Debian Legal team, for originally bringing to my attention, that this license will be better suited, than any copyright assignment could be.

Bug fixes

  • The compiled functions could not be used with multiprocessing or copy.copy. Issue#19. Fixed in 0.3.22.1 already.
  • In-place operations for slices with not both bounds specified crashed the compiler. Issue#36. Fixed in 0.3.22.1 already.
  • Cyclic imports could trigger an endless loop, because module import expressions became the parent of the imported module object. Issue#37. Fixed in 0.3.22.2 already.
  • Modules named proc or func could not be compiled to modules or embedded due to a collision with identifiers of CPython2.7 includes. Issue#38. Fixed in 0.3.22.2 already.

New Features

  • The fix for Issue#19 also makes pickling of compiled functions available. As it is the case for non-compiled functions in CPython, no code objects are stored, only names of module level variables.

Organizational

  • Using the Apache License 2.0 for all of Nuitka now.

  • Speedcenter has been re-activated, but is not yet having a lot of benchmarks yet, subject to change.

    Update

    We have given up on speedcenter meanwhile, and generate static pages with graphs instead.

New Tests

  • Changed the "CPython26" tests to no longer disable the parts that relied on copying of functions to work, as Issue#19 is now supported.
  • Extended in-place assignment tests to cover error cases of Issue#36.
  • Extended compile library test to also try and compile the path where numpy lives. This is apparently another path, where Debian installs some modules, and compiling this would have revealed Issue#36 sooner.

Summary

The release contains bug fixes, and the huge step of changing the license. It is made in preparation to PyCON EU.

Nuitka Speedcenter is back

Once a long time ago, I was benchmarking Nuitka more often. Check "What is Nuitka?" in case you don't know what it is.

Problem

And I was considering the use of codespeed, and had some data online. But ultimately, it got discontinued. This has 3 reasons:

  • Moved the website to a dedicated machine, which broke the previous install.

  • Controlling which data is used for display was hard and not satisfactory.

    1. For example, I didn't want to have to commit and push, just to let the benchmarks run.

    2. And I wanted to be able to re-run benchmarks with newer compiler, even newer Python, but old Nuitka. Using g++ 4.6 over g++ 4.5 should not impact the data.

    3. It turned out to be a nightmare to migrate to newer codespeed versions. I found myself starting from empty database - over and over.

    4. Many things were not supported.

      For example, I would want to collect all PyBench results, but only publish those who are expressive. That seemed difficult to achieve.

  • Benchmarks of Nuitka are not yet useful

    1. Nuitka was not yet doing type inference
    2. Most of the work was aimed at correctness, and effectively was often degrading performance even if only temporary. Seeing it wouldn't have been too motivating.

Solution

I have simply created a small wrapper:

  1. Small script to run benchmarks and collect data.

    It checks out Nuitka in all versions in a playground, and then runs defined benchmarks, with valgrind, etc. taking exe sizes, etc.

  2. Data is stored in local sqlite databases.

    I have a database per machine, i.e. a distributed repository, where I collect information. That works for me, and will allow me to compare different kinds of machines.

    The advantage is that I have no risk of data loss anymore, and no issues and difficulty with poor interfaces to replace existing data.

  3. Data is merged on one machine, and then pushed.

    That allows me to inspect the changes before publishing them. It allows me to play with local commits, branches, with information that will go away. I can then push when I choose to.

That integrates better with my work flow. It allows me to retro-fit benchmarks results on the machine and to be tool independent.

In principle, I could publish the data in other forms as well, and I likely will. Making tables of e.g. PyBench results seems like one application. Recently, I have workd with Nikola, and could also imagine to integrate Codespeed graph functionality (which is apparently all I want) to there.

Yours, Kay

Nikola Speed Improvements

At end the end of May, I switched my website from a W3 Total Cache enhanced wordpress installation to static generated with Nikola, and this is what Google Webmaster tool says about it.

/posts/images/nikola-speed-improvements.png

The numbers are average time to download pages in milliseconds.

Nikola for nuitka.net

Some time ago, I asked about static site generators and tried a few, but nothing was as ReST friendly as I would like to. To me, ReST is to text publishing, what Python is to programming.

Well, the search has found a late hit. I had installed a work flow, where I was using the XMLRPC interface, but it was some effort, and somewhat limited.

Then Roberto Alsina announced Nikola that was just what I wanted.

I did migrate to it during the last days. Looking at the content in ReST form, was a big ease, and I already found a bit of obsolete content, and updated it. And the last release announcements, they now render somewhat prettier.

I am not using all the features yet, notable not using the image galleries yet, and the download page is not yet generated again, and so on. But this is a good base.

And while I loved it, this was the result:

apt-get remove wordpress

Nuitka Release 0.3.22

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimization.

It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to compile itself with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it's quite fine now.

Bug fixes

  • List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions.
  • Calling a compiled function in an exception handler cleared the exception on return, it no longer does that.
  • Reference counter handling with generator throw method is now correct.
  • A module "builtins" conflicted with the handling of the Python builtins module. Those now use different identifiers.

New Features

  • New metaclass syntax for the class statement works, and the old __metaclass__ attribute is properly ignored.

    # Metaclass syntax in Python3, illegal in Python2
    class X(metaclass = Y):
        pass
    
    # Metaclass syntax in Python2, no effect in Python3
    class X:
        __metaclass__ = Y
    

    Note

    The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn' it. Surely, the support for __metaclass__ could still live.

    # For Python2/3 compatible source, we create a base class that has the
    # metaclass used and doesn't require making a choice.
    
    CPythonNodeMetaClassBase = NodeCheckMetaClass("CPythonNodeMetaClassBase", (object,), {})
    
  • The --dump-xml option works with Nuitka running under Python3. This was not previously supported.

  • Python3 now also has compatible parameter errors and compatible exception error messages.

  • Python3 has changed scope rules for list contractions (assignments don't affect outside values) and this is now respected as well.

  • Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well.

New Optimization

  • Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used.

    This avoids overhead for the very simple functions. And example of this can be seen here:

    def simple():
       return 7
    
  • Optimize len built-in for non-constant, but known length values.

    An example can be seen here:

    # The range isn't constructed at compile time, but we still know its
    # length.
    len(range(10000000))
    
    # The string isn't constructed at compile time, but we still know its
    # length.
    len("*" * 1000)
    
    # The tuple isn't constructed, instead it's known length is used, and
    # side effects are maintained.
    len((a(), b()))
    

    This new optimization applies to all kinds of container creations and the range built-in initially.

  • Optimize conditions for non-constant, but known truth values.

    At this time, known truth values of non-constants means range built-in calls with know size and container creations.

    An example can be seen here:

    if (a,):
       print "In Branch"
    

    It's clear, that the tuple will be true, we just need to maintain the side effect, which we do.

  • Optimize or and and operators for known truth values.

    See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can:

    # The access and call to "something()" cannot possibly happen
    0 and something()
    
    # Can be replaced with "something()", as "1" is true. If it had a side effect, it
    # would be maintained.
    1 and something()
    
    # The access and call to "something()" cannot possibly happen, the value is already
    # decided, it's "1".
    1 or something()
    
    # Can be replaced with "something()", as "0" is false. If it had a side effect, it
    # would be maintained.
    0 or something()
    
  • Optimize print arguments to become strings.

    The arguments to print statements are now converted to strings at compile time if possible.

    print 1
    

    becomes:

    print "1"
    
  • Combine print arguments to single ones.

    When multiple strings are printed, these are now combined.

    print "1+1=", 1+1
    

    becomes:

    print "1+1= 2"
    

Organizational

  • Enhanced Python3 support, enabling support for most basic tests.
  • Check files with PyLint in deterministic (alphabetical) order.

Cleanups

  • Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module.
  • The try/except/else has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not.
  • Side effects are now a dedicated node, used in several optimization to maintain the effect of an expression with known value.

New Tests

  • Expanded and adapted basic tests to work for Python3 as well.
  • Added reference count tests for generator functions throw, send, and close methods.
  • Cover calling a function with try/except in an exception handler twice. No test was previously doing that.

Summary

This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away.

There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense.

Nuitka Release 0.3.21

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it will be easy to start tracking them.

Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc.

Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete.

Bug fixes

  • The built-in next could causes a program crash when iterating past the end of an iterator. Issue#34. Fixed in 0.3.20.1 already.
  • The set constants could cause a compiler error, as that type was not considered in the "mutable" check yet. Fixed in 0.3.20.2 already.
  • Performance regression. Optimize expression for exception types caught as well again, this was lost in last release.
  • Functions that contain exec, are supposed to have a writable locals. But when removing that exec statement as part of optimization, this property of the function could get lost.
  • The so called "overflow functions" are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an exec or a star import.
  • The syntax error for return outside of a function, was not given, instead the code returned at run time. Fixed to raise a SyntaxError at compile time.

New Optimization

  • Avoid tuple objects to be created when catching multiple exception types, instead call exception match check function multiple times.

  • Removal of dead code following break, continue, return, and raise. Code that follows these statements, or conditional statements, where all branches end with it.

    Note

    These may not actually occur often in actual code, but future optimization may produce them more frequently, and their removal may in turn make other possible optimization.

  • Detect module variables as "read only" after all writes have been detected to not be executed as removed. Previously the "read only indicator" was determined only once and then stayed the same.

  • Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value.

  • Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still.

    Note

    An exception is where len = len actually does have an impact, because that variable becomes assignable. The "compile itself" test of Nuitka found that to happen with long from the nuitka.__past__ module.

  • Created Python3 variant of quick unicode string access, there was no such thing in the CPython C/API, but we make the distinction in the source code, so it makes sense to have it.

  • Created an optimized implementation for the built-in iter with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython C/API.

  • For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take.

  • Don't use a "context" object for generator functions (and generator expressions) that don't need one. And even if it does to store e.g. the given parameter values, avoid to have a "common context" if there is no closure taken. This avoids useless malloc calls and speeds up repeated generator object creation.

Organizational

  • Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated.
  • The --experimental option is no longer available outside of checkouts of git, and even there not on stable branches (master, hotfix/...). It only pollutes --help output as stable releases have no experimental code options, not even development version will make a difference.
  • The binary "bin/Nuitka.py" has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway.
  • The --python-version option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values.
  • Added --keep-pythonpath alias for --execute-with-pythonpath option, probably easier to remember.
  • Support --debug with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn't report anything new.
  • The contents environment variable CXX determines the default C++ compiler when set, so that checking with CXX=g++-4.7 nuitka-python ... has become supported.
  • The check-with-pylint script now has a real command line option to control the display of TODO items.

Cleanups

  • Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead.

    a = b = c
    
    _tmp = c
    b = _tmp
    a = _tmp
    

    In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block.

    All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed.

  • Changed unpacking assignments to use temporary variables. Code like this:

    a, b = c
    

    Is handled more like this:

    _tmp_iter = iter(c)
    _tmp1 = next(_tmp_iter)
    _tmp2 = next(_tmp_iter)
    if not finished(_tmp_iter):
        raise ValueError("too many values to unpack")
    a = _tmp1
    b = _tmp2
    

    In reality, not really next is used, as it wouldn't raise the correct exception for unpacking, and the finished check is more condensed into it.

    Generally this cleanup allowed that the AssignTargetTuple and associated code generation was removed, and in the future value propagation may optimize these next and iter calls away where possible. At this time, this is not done yet.

  • Exception handlers assign caught exception value through assignment statement.

    Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well.

  • Exception handlers now explicitly catch more than one type.

    Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it's all.

  • Contractions are now functions as well.

    Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation.

  • Global is handled during tree building.

    Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node.

    It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that global statement, allowing to remove the node class.

  • Read only module variable detection integrated to constraint collection.

    The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed.

New Tests

  • Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment.

  • Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn't even know it can be done, example:

    try:
        raise ValueError(1,2)
    except ValueError as (a,b):
        print "Unpacking caught exception and unpacked", a, b
    

    Will assign a=1 and b=2.

  • Added test to cover return statements on module level and class level, they both must give syntax errors.

  • Cover exceptions from accessing unassigned global names.

  • Added syntax test to show that star imports do not allow other names to be imported at the same time as well.

  • Python3 is now also running the compile itself test successfully.

Summary

The progress made towards value propagation and type inference is very significant, and makes those appears as if they are achievable.

Nuitka Release 0.3.20

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This time there are a few bug fixes and some really major cleanups, lots of new optimization and preparations for more. And then there is a new compiler clang and a new platform supported. MacOS X appears to work mostly, thanks for the patches from Pete Hunt.

Bug fixes

  • The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. Issue#30. Fixed in release 0.3.19.1 already.
  • The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. Issue#31. Fixed in release 0.3.19.1 already.
  • The from x import * attempted to respect __all__ but failed to do so. Issue#32. Fixed in release 0.3.19.2 already.
  • The from x import * didn't give a SyntaxError when used on Python3. Fixed in release 0.3.19.2 already.
  • The syntax error messages for "global for function argument name" and "duplicate function argument name" are now identical as well.
  • Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed.

New Features

  • Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent.
  • Enhanced Python3 support for syntax errors, these are now also compatible.
  • Support for MacOS X was added.
  • Support for using the clang compiler was added, it can be enforced via --clang option. Currently this option is mainly intended to allow testing the "MacOS X" support as good as possible under Linux.

New Optimization

  • Enhanced all optimization that previously worked on "constants" to work on "compile time constants" instead. A "compile time constant" can currently also be any form of a built-in name or exception reference. It is intended to expand this in the future.

  • Added support for built-ins bin, oct, and hex, which also can be computed at compile time, if their arguments are compile time constant.

  • Added support for the iter built-in in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster.

  • Added support for the next built-in, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well.

  • Added support for the open built-in in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files.

  • Optimize the __debug__ built-in constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided.

  • Optimize the Ellipsis built-in constant as well. It falls in the same category as True, False, None, i.e. names of built-in constants that a singletons.

  • Added support for anonymous built-in references, i.e. built-ins which have names that are not normally accessible. An example is type(None) which is not accessible from anywhere. Other examples of such names are compiled_method_or_function.

    Having these as represented internally, and flagged as "compile time constants", allows the compiler to make more compile time optimization and to generate more efficient C++ code for it that won't e.g. call the type built-in with None as an argument.

  • All built-in names used in the program are now converted to "built-in name references" in a first step. Unsupported built-ins like e.g. zip, for which Nuitka has no own code or understanding yet, remained as "module variables", which made access to them slow, and difficult to recognize.

  • Added optimization for module attributes __file__, __doc__ and __package__ if they are read only. It's the same as __name__.

  • Added optimization for slices and subscripts of "compile time constant" values. These will play a more important role, once value propagation makes them more frequent.

Organizational

  • Created a "change log" from the previous release announcements. It's as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it's so easy to generate the PDF on that yourself.
  • The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to Wordpress as a draft posting or updates it, until it's release time. Simple, sweet and elegant.

Cleanups

  • Split out the nuitka.nodes.Nodes module into many topic nodes, so that there are now nuitka.nodes.BoolNodes or nuitka.nodes.LoopNodes to host nodes of similar kinds, so that it is now cleaner.

  • Split del statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same:

    del a, b.c, d
    
    del a
    del b.c
    del d
    

    So that's now represented in the node tree. And even more complex looking cases, like this one, also the same:

    del a, (b.c, d)
    

    This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all.

  • Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions.

    Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the bases for classes and the defaults for functions and make optional.

  • Refactored the decorator handling to the tree building stage, presenting them as function calls on "function body expression" or class body expression".

    This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimization, as they practically are now just function calls.

    @some_classdecorator
    class C:
        @staticmethod
        def f():
            pass
    

    It's just a different form of writing things. Nothing requires the implementation of decorators, it's just functions calls with function bodies before the assignment.

    The following is only similar:

    class C:
        def f():
            pass
    
        f = staticmethod(f)
    
    C = some_classdecorator(C)
    

    It's only similar, because the assignment to an intermediate value of C and f is not done, and if an exception was raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already.

  • The in-place assignments statements are now handled using temporary variable blocks

    Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal look-ups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future.

  • The for loop used to have has a "source" expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimization. Moved it to tree building stage instead, where optimization can work on it then.

  • Tree building now generally allows statement sequences to be None everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist.

  • Moved the optimization for __name__ to compute node of variable references, where it doesn't need anything complex to replace with the constant value if it's only read.

  • Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults.

  • Made the built-in code more reusable.

New Tests

  • Added some more diagnostic tests about complex assignment and del statements.
  • Added syntax test for star import on function level, that must fail on Python3.
  • Added syntax test for duplicate argument name.
  • Added syntax test for global on a function argument name.

Summary

The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much more consistency there.

Yet, even more work will be needed in the same direction. There may e.g. be work required to cover with statements as well. And assignments will become no more complex than unpacking from a temporary variable.

For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only miniscule progress. The remaining tasks appear all more or less difficult work that I don't want to touch now.

There are still remaining steps, but we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about.