bksahu's GSoC warp up

Hello everyone!

GSoC 2019 has almost come to an end! It's the time to wrap up this mega event started back in May 2019. Under the mentorship of Mentor Hayen, my learning experience has undergone a roller-coaster ride and it has not only boosted my growth as a developer but also as an individual. Over the last 3 months the followings are my major contributions to this project:

Built-ins Optimizations

  • "any": PR #246
    • nuitka.nodes.BuiltinAnyNodes node added to optimize the "any" built-in.

    • Developed an algorithm to predict the "any" for arguments having repetitive values at compile time. For example:

      any([0]*255) -> False

      any([False, False, True]) -> True

    • Extended support for range, set and dict built-ins.

    • Added the optimized C side support too

    • Added a method getMetaClassBase to make Python 2 and Python 3 compatible while working with metaclasses.

  • Issue reported and closed Issue #349
    • Created a new module nuitka.nodes.IterationHandles to work with iterables.
    • Added support of Iteration for non-mutable types.
  • "all": PR #407
    • Added nuitka.nodes.BuiltinAllNodes to optimize the "all" built-ins.

    • Developed an algorithm similar to "any" to predict the "all" arguments. For example:

      all([0, 0, 1]) -> False

      all([True]*100) -> True

    • Other similar optimizations are done like "any" built-in.

    • Additionally, added a new testing module tests.optimizations.CommonOptimizations to test the built-ins optimizations at the same place.

  • "abs": PR #419
    • Added new operation node ExpressionOperationAbs to optimize the abs built-in.
    • Manually added shapeSlotAbs to different shapes.
    • Finally pre-computed the compile time constant abs
  • "max" and "min": PR #442
    • This PR is work in progress and is half complete.

    • This is the first optimizations in which I used reformulations instead of added in a new node.

    • Pseudo-code of "min" reformulation:

      def _min(a, b, c, ...):
       tmp_arg1 = a
       tmp_arg2 = b
       tmp_arg3 = c
       ...
       result = tmp_arg1
       if keyfunc is None: # can be decided during re-formulation
           tmp_key_result = keyfunc(result)
           tmp_key_candidate = keyfunc(tmp_arg2)
           if tmp_key_candidate < tmp_key_result:
               result = tmp_arg2
               tmp_key_result = tmp_key_candidate
           tmp_key_candidate = keyfunc(tmp_arg3)
           if tmp_key_candidate < tmp_key_result:
               result = tmp_arg3
               tmp_key_result = tmp_key_candidate
           ...
       else:
           if tmp_arg2 < result:
               result = tmp_arg2
           if tmp_arg3 < result:
               result = tmp_arg3
           ...
        return result
      
    • Adding support for keyfunc is pending

  • "zip": PR #462
    • This built-in uses both types of optimizations that the previous built-ins optimizations used.
    • zip for Python 2 uses the reformulations.
    • Pseudo-code of "zip" reformulation:
    def _zip(a, b, c, ... ):
    # First assign, to preserve order of execution,
    # the arguments might be complex expressions.
    tmp_arg1 = a
    tmp_arg2 = b
    tmp_arg3 = c
    ...
    tmp_iter_1 = iter(tmp_arg1)
    tmp_iter_2 = iter(tmp_arg2)
    tmp_iter_3 = iter(tmp_arg3)
    ...
    # could be more
    tmp_result = []
    try:
        while 1:
            tmp_result.append(
                (
                     next(tmp_iter_1),
                     next(tmp_iter_2),
                     next(tmp_iter_3),
                     ...
                )
             )
       except StopIteration:
           pass
     return tmp_result
    
    • zip for Python 3 needs a new node that calls the zip because unlike zip in Python 2, zip in Python 3 returns a zipobject.

Test suite

  • Search mode "All": PR #378
    • In the test suite, I added a new search mode "all" that will test all the modules and return all the results at once. For example:
    $ ./tests/basics/run_all.py all
       Using concrete python 2.7.12 on x86_64
       Comparing output of 'Asserts.py' using '/usr/bin/python' with flags silent, expect_success, remove_output,     recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
       .
       .
       .
       .
       Total 0 error(s) found.
    
  • Search mode "Only": PR #333
    • Added "only" search mode to test only a single module. For example:
    $ ./tests/basics/run_all.py only BuiltinsTest.py
       Using concrete python 2.7.12 on x86_64
       Skipping Asserts.py
       Skipping Assignments.py
       Skipping BigConstants.py
       Skipping Branching.py
       Skipping BuiltinOverload.py
       Skipping BuiltinSuper.py
       Comparing output of 'BuiltinsTest.py' using '/usr/bin/python' with flags silent, expect_success, remove_output, recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
    
  • Reported and closed Issue #334: PR #336
    • Fixed the autoformat bug by reading and writing the files in bytes instead of string.

Documentation

And other minor doc fixes are added with their respective pull requests.

What I learned

  • Learned the software engineering principles and how to keep my work clean.
  • I also learned how to effectively use software designing principles like DRY and KISS.
  • Got exposed to Nuitka internals which helped me to better understand how compilers in general work.
  • Explored how CPython works internally.
  • Got some great advice from Mentor Hayen about starting my professional career in Software engineering.

Overall, it was a great experience to be a part of Nuitka :)

Yours,

Tommy's Google Summer of Code Summary

Intro

As Google Summer of Code (GSoC) is coming to an end, I am writing this blog post as a final summary describing all the work I have done as well as my experiences in this program.

Summary of My Work

  • #314 run_all.py new special-comment mechanism & Urllib3Using.py
    • Before GSoC started, I looked around for whatever work I could help with.
    • In this pull request, I added a checkRequirements function for the Nuitka standalone test suite.
    • This function checks for special-comments at the top of standalone tests in the format of # nuitka-skip-unless-expression: expression to be evaluated OR # nuitka-skip-unless-imports: module1,module2,... and will decide whether to skip a test depending on if its specified requirements are met.
    • In addition, standalone test Urllib3Using.py was created.
    • This pull request was soon merged and allowed me the lucky opportunity of GSoC 2019 with Nuitka :)
  • #339 Standalone tests for botocore & boto3 + fix to Urllib3Using.py
    • This PR was also created before the start of GSoC.
    • Standalone test Boto3Using.py was created using moto to mock AWS calls which did not turn out well.
    • Changed Urllib3Using.py with the addition of python version checks as a fix to Issue #373.
  • Urllib3 Wheel with Nuitka Pytest Results and Python-Dateutil Wheel with Nuitka Pytest Results
    • At the start of GSoC, I performed manual pytest comparison for PyPI packages urllib3 and dateutil.
    • The findings of my testing were documented in these postings.
    • Manual testing compares the pytest results of an installed nuitka wheel built using python setup.py bdist_nuitka to the regular pytest results of each package.
    • Testing is done to ensure that nuitka is building the wheel correctly.
    • If the pytests pass/fail in the same way, that means Nuitka built the wheel properly.
    • Else if the tests differ, then something is wrong.
    • Virtualenv is used to create a clean environment with no outside pollution.
    • Over the course of performing manual testing, I became familiar with the use of virtualenv, wheel, and pytest.
    • A bug was found with the package urllib3 bdist and I created Issue #413 to document the bug.
  • #440 Automating PyPI Wheel Pytest
    • After familiarizing myself with how virtualenv, wheel, and pytest work, I started to work on a script which would automate the pytest comparison for top PyPI packages.
    • The script first uses git to update each package if it is already existing in the local cache, else it will git clone that package into the local cache.
    • The script then uses calls to os.system to automate the creation of a virtualenv which is then used to install pytest and pip install the package's requirements (if any) for running pytest.
    • The script then handles each package depending on different needs before building a regular wheel with python setup.py bdist_wheel.
    • This wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into a string.
    • The script then resets the package to its original state and builds a nuitka-compiled wheel using python setup.py bdist_nuitka.
    • This compiled wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into another string.
    • The two strings containing pytest outputs are then compared to find differences.
    • If no differences are found, this means bdist_nuitka worked properly. Else Nuitka compilation did something wrong.
    • The above process is repeated for each suitable PyPI package from the PyPI top 50. (Some packages are left out if they do not contain a test suite or if they do not need to be tested)
    • At the end, a colored summary is given for all the packages tested.
    • This automation script is meant to be run regularly to inform developers of Nuitka regressions.
  • Issue #477 Unable to compile modules listed under unworthy_namespaces
    • Raised due to package pycparser failing in the automated test suite.
    • This issue will be addressed in the future.
  • Issue #479 bdist_nuitka fails for packages containing py_modules only
    • While I worked on #440, I found a bug with bdist_nuitka failing on PyPI packages containing py_modules only.
    • This bug occurs due to Nuitka making the assumption that a main package always exists for all packages. However, some packages contain only a main module and not a main package.
    • Applies to PyPI packages decorator, ipaddress, and pyparsing.
  • #483 Add support for py_modules_only compilation
    • This pull request changes bdist_nuitka.py and various other files to fix Issue #479.
    • Checks are added for the bdist_nuitka command to see if a main package exists. If there is not a main package, it will set its compile target to the main module instead.
    • This also addressed the case of a package with both a main package and a main module, in which case both are included inside the resulting wheel.
    • In addition, distutils examples py_modules_only and package_and_module were created and added for future testing.
    • During this PR, I found an import bug in Nuitka and hotfixed it with #487 Fixup_import_module.
  • #484 PyPI Standalone Tests
    • This pull request adds more standalone tests for each top PyPI package.
  • #495 Improve pypi automation
    • Improves the PyPI test suite created in #440 with functional improvements, readability improvements, and added documentation.

Things I Learned

Before GSoC, I was very uncomfortable with working inside a terminal. I was unfamiliar with many basic bash commands because I simply did not have any prior professional industrial experiences. I was also very unfamiliar with the Git flow, which is evident in the messy commit histories of my earliest pull requests.

As I continued throughout my GSoC journey, however, I became much more comfortable with working inside the terminal as well as using git as a version-control system (shoutout to my mentor Kay Hayen for helping me through all the annoying conflicts).

Although I am still no expert, I can confidently say that I am now far more proficient working with git and inside the terminal.

In addition, I became much more familiar with many of the most popular PyPI packages as well as the inner workings of python, which I believe will help me go very far in my career as a software developer.

Overall, the GSoC experience was truly astounding and I am more than thankful to my mentor Kay Hayen as well as Google for making this amazing program possible.

Yours, Tommy

Nuitka Release 0.6.5

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains many bug fixes all across the board. There is also new optimization and many organisational improvements.

Bug Fixes

  • Python3.4+: Fixed issues with modules that exited with an exception, that could lead to a crash, dealing with their __spec__ value.

  • Python3.4+: The __loader__ method is_package had the wrong signature.

  • Python3.6+: Fix for async with being broken with uncompiled generators.

  • Python3.5+: Fix for coroutines that got their awaited object closed behind their back, they were complaining with RuntimeError should they be closed themselves.

  • Fix, constant values None in a bool target that could not be optimized away, lead to failure during code generation.

    if x() and None:
      ...
    
  • Standalone: Added support for sha224, sha384, sha512 in crypto package.

  • Windows: The icon wasn't properly attached with MinGW64 anymore, this was a regression.

  • Windows: For compiler outputs, also attempt preferred locale to interpret outputs, so we have a better chance to not crash over MSVC error messages that are not UTF-8 compatible.

  • macOS: Handle filename collisions for generated code too, Nuitka now treats all filesystems for all OS as case insensitive for this purpose.

  • Compatibility: Added support for tolerant del in class exception handlers.

    class C:
    
      try:
        ...
      except Exception as e:
        del e
    
        # At exception handler exit, "e" is deleted if still assigned
    

    We already were compatible for functions and modules here, but due to the special nature of class variables really living in dictionaries, this was delayed. But after some other changes, it was now possible to solve this TODO.

  • Standalone: Added support for Python3 variant of Pmw.

  • Fix, the NumPy plugin now handles more installation types.

  • Fix, the qt plugin now handles multiple library paths.

  • Fix, need libm for some Anaconda variants too.

  • Fix, left over bytecode from plugins could crash the plugin loader.

  • Fix, pkgutil.iter_packages is now working for loaded packages.

New Features

  • Python3.8: Followed some of the changes and works with beta2 as a Python 3.7, but none of the new features are implemented yet.
  • Added support for Torch, Tensorflow, Gevent, Sklearn, with a new Nuitka plugin.
  • Added support for "hinted" compilation, where the used modules are determined through a test run.
  • Added support for including TCL on Linux too.

Optimization

  • Added support for the any built-in. This handles a wide range of type shapes and constant values at compile time, while also having optimized C code.
  • Generate code for some CLONG operations in preparation of eventual per expression C type selection, it then will allow to avoid objects in many instances.
  • Windows: Avoid creating link libraries for MinGW64 as these have become unnecessary is the mean time.
  • Packages: Do not export entry points for all included packages, only for the main package name it is importable as.

Organisational

  • Added support for Visual Studio 2019 as a C compiler backend.
  • Improved plugin documentation describing how to create plugins for Nuitka even better.
  • The is now a mode for running the tests called all which will execute all the tests and report their errors, and only fail at the very end. This doesn't avoid wasting CPU cycles to report that e.g. all tests are broken, but it allows to know all errors before fixing some.
  • Added repository for Fedora 30 for download.
  • Added repository for openSUSE 15.1 for download.
  • Ask people to compile hello world program in the Github issue template, because many times, they have setup problems only.
  • Visual Studio Code is now the recommended IDE and has integrated configuration to make it immediately useful.
  • Updated internal copy of Scons to 3.1.0 as it incorporates many of our patches.
  • Changed wordings for optimization to use "lowering" as the only term to describe an optimization that simplifies.

Cleanups

  • Plugins: Major refactoring of Nuitka plugin API.
  • Plugins: To locate module kind, use core Nuitka code that handles more cases.
  • The test suite runners are also now autoformatted and checked with PyLint.
  • The Scons file is now PyLint clean too.
  • Avoid build_definitions.h to be included everywhere, in that it's only used in the main program part. This makes C linter hate us much less for using a non-existent file.

Tests

  • Run the tests using Travis on macOS too.
  • More standalone tests have been properly whitelisting to cover openSSL usage from local system.
  • Disabled PySide2 test, it's not useful to fail and ignore it.
  • Tests: Fixups for coverage testing mode.
  • Tests: Temporarily disable some checks for constants code in reflected tests as it only exposes marshal not being deterministic.

Summary

This release is huge again. Main points are compatibility fixes, esp. on the coroutine side. These have become apparently very compatible now and we might eventually focus on making them better.

Again, GSoC 2019 is also showing effects, and will definitely continue to do soin the next release.

Many use cases have been improved, and on an organizational level, the adoption of Visual Studio Code seems an huge improvement to have a well configured IDE out of the box too.

In upcoming releases, more built-ins will be optimized, and hopefully the specialization of operations will hit more and more code with more of the

Python-Dateutil Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of dateutil to its nuitka-built .whl counterpart.

Dateutil standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the dateutil package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

The pytest results were very similar:

Regular pytests: ============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============
Nuitka wheel pytests: ============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Steps to Reproduce

  1. Clone dateutil and nuitka into a new folder
  2. Inside the dateutil folder, issue python -m pip install -r requirements-dev.txt to install its requirements.
  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for dateutil.
  4. Change into the nuitka folder and issue python setup.py develop.
  5. Change back into dateutil and issue python setup.py bdist_nuitka to build the dateutil wheel using nuitka. The newly built wheel should be found in the dist folder.
  6. Use pip to uninstall the existing dateutil, then issue python -m pip install followed by the newly built .whl filename.
  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for dateutil.

Uncompile Python

dateutil regular pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2074 items

dateutil\test\test_easter.py ........................................... [  2%]
........................................................................ [  5%]
................................................                         [  7%]
dateutil\test\test_import_star.py .                                      [  7%]
dateutil\test\test_imports.py .......................                    [  9%]
dateutil\test\test_internals.py ....                                     [  9%]
dateutil\test\test_isoparser.py ........................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
.......x...x............................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 35%]
.....................xx                                                  [ 36%]
dateutil\test\test_parser.py ........................................... [ 38%]
........................................................................ [ 42%]
........................................................................ [ 45%]
.................................xxxxxxxxxxxxxsss......                  [ 48%]
dateutil\test\test_relativedelta.py .................................... [ 49%]
.............................................                            [ 52%]
dateutil\test\test_rrule.py ............................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
................................................................x....... [ 78%]
..............                                                           [ 79%]
dateutil\test\test_tz.py ............................s...........sssssss [ 81%]
sssssssssssssssssssssssssssssssssssssssss..s............................ [ 84%]
xxx..s......................................s........................... [ 88%]
s....................................................................... [ 91%]
..........s....................................s.s....................ss [ 95%]
sssssssssssss....s..........s........................................... [ 98%]
.............                                                            [ 99%]
dateutil\test\test_utils.py .......                                      [ 99%]
dateutil\test\property\test_isoparse_prop.py .                           [ 99%]
dateutil\test\property\test_parser_prop.py ..                            [ 99%]
docs\exercises\solutions\mlk_day_rrule_solution.py .                     [100%]

============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============

Compiled with Nuitka

nuitka wheel pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2073 items

test\test_easter.py .................................................... [  2%]
........................................................................ [  5%]
.......................................                                  [  7%]
test\test_import_star.py .                                               [  7%]
test\test_imports.py .......................                             [  9%]
test\test_internals.py ....                                              [  9%]
test\test_isoparser.py ................................................. [ 11%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
......................................................................x. [ 25%]
..x..................................................................... [ 28%]
........................................................................ [ 32%]
........................................................................ [ 35%]
............xx                                                           [ 36%]
test\test_parser.py .................................................... [ 39%]
........................................................................ [ 42%]
........................................................................ [ 46%]
........................xxxxxxxxxxxxxsss......                           [ 48%]
test\test_relativedelta.py ............................................. [ 50%]
....................................                                     [ 52%]
test\test_rrule.py ..................................................... [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.......................................................x................ [ 79%]
.....                                                                    [ 79%]
test\test_tz.py ............................s...........ssssssssssssssss [ 81%]
ssssssssssssssssssssssssssssssss..s............................xxx..s... [ 85%]
...................................s...........................s........ [ 88%]
........................................................................ [ 92%]
.s....................................s.s....................sssssssssss [ 95%]
ssss....s..........s.................................................... [ 99%]
....                                                                     [ 99%]
test\test_utils.py .......                                               [ 99%]
test\property\test_isoparse_prop.py .                                    [ 99%]
test\property\test_parser_prop.py ..                                     [100%]

============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Urllib3 Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of urllib3 to its nuitka-built .whl counterpart.

Urllib3 standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the urllib3 package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

At first, the urllib3 nuitka-wheel pytest was crashing because of the unsafe assumption that imports will always exist (which is not the case if exceptions are thrown). Issue 413 was filed to record and fix this bug.

After the fixes, the pytests were ran again and the results were very similar:

Regular pytests: ====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======
Nuitka wheel pytests: ====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

The extra passes are suspicious and require more investigation into why they happen. To make that easy, we are going to fully automate the process and compare outputs with verbose pytest modes.

Steps to Reproduce

  1. Clone urllib3 and nuitka into a new folder
  2. Inside the urllib3 folder, issue python -m pip install -r dev-requirements.txt to install its requirements.
  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for urllib3.
  4. Change into the nuitka folder and issue python setup.py develop.
  5. Change back into urllib3 and issue python setup.py bdist_nuitka to build the urllib3 wheel using nuitka. The newly built wheel should be found in the dist folder.
  6. Use pip to uninstall the existing urllib3, then issue python -m pip install followed by the newly built .whl filename.
  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for urllib3.

Uncompile Python

urllib3 regular pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py ..                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py ..............................F.F.F..                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
.................................................Uncaught exception, closing connection.
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======

Compiled with Nuitka

nuitka wheel pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py .F                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py .....................................                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

Nuitka Release 0.6.4

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains many bug fixes all across the board. There is also new optimization and many organisational improvements.

Bug Fixes

  • When linking very large programs or packages, with gcc compiler, Scons can produce commands that are too large for the OS. This happens sooner on the Windows OS, but also on Linux. We now have a workaround that avoids long command lines by using @sources.tmp syntax.
  • Standalone: Remove temporary module after its use, instead of keeping it in sys.modules where e.g. Quart code tripped over its __file__ value that is illegal on Windows.
  • Fixed non-usage of our enhanced detection of gcc version for compilers if given as a full path.
  • Fixed non-detection of gnu-cc as a form of gcc compiler.
  • Python3.4: The __spec__ value corrections for compiled modules was not taking into account that there was a __spec__ value, which can happen if something is wrapping imported modules.
  • Standalone: Added implicit dependencies for passlib.
  • Windows: Added workaround for OS command line length limit in compilation with MinGW64.
  • Python2: Revive the enum plugin, there are backports of the buggy code it tries to patch up.
  • Windows: Fixup handling of SxS with non zero language id, these occur e.g. in Anaconda.
  • Plugins: Handle multiple PyQt plugin paths, e.g. on openSUSE this is done, also enhanced finding that path with Anaconda on Windows.
  • Plugins: For multiprocessing on Windows, allow the .exe suffix to not be present, which can happen when ran from command line.
  • Windows: Better version checks for DLLs on Python3, the ctypes helper code needs more definitions to work properly.
  • Standalone: Added support for both pycryptodome and pycryptodomex.
  • Fix, the chr built-in was not giving fully compatible error on non number input.
  • Fix, the id built-in doesn't raise an exception, but said otherwise.
  • Python3: Proper C identifiers for names that fit into latin-1, but are not ascii encodings.

New Features

  • Windows: Catch most common user error of using compiler from one architecture against Python from another. We now check those and compare it, and if they do not match, inform the user directly. Previously the compilation could fail, or the linking, with cryptic errors.
  • Distutils: Using setuptools and its runners works now too, not merely only pure distutils.
  • Distutils: Added more ways to pass Nuitka specific options via distutils.
  • Python3.8: Initial compatibility changes to get basic tests to work.

Organisational

  • Nuitka is participating in the GSoC 2019 with 2 students, Batakrishna and Tommy.
  • Point people creating PRs to using the pre-commit hook in the template. Due to making the style issues automatic, we can hope to encounter less noise and resulting merge problems.
  • Many improvements to the pre-commit hook were done, hopefully completing its development.
  • Updated to latest pylint, black, and isort versions, also added codespell to check for typos in the source code, but that is not automated yet.
  • Added description of how to use experimental flags for your PRs.
  • Removed mirroring from Bitbucket and Gitlab, as we increasingly use the Github organisation features.
  • Added support for Ubuntu Disco, removed support for Ubuntu Artful packages.

Optimization

  • Windows: Attach data blobs as Windows resource files directly for programs and avoid using C data files for modules or MinGW64, which can be slow.
  • Specialization of helper codes for + is being done for more types and more thoroughly and fully automatic with Jinja2 templating code. This does replace previously manual code.
  • Added specialization of helper codes for * operation which is entirely new.
  • Added specialization of helper codes for - operation which is entirely new.
  • Dedicated nodes for specialized operations now allow to save memory and all use type shape based analysis to predict result types and exception control flow.
  • Better code generation for boolean type values, removing error checks when possible.
  • Better static analysis for even more type operations.

Cleanups

  • Fixed many kinds of typos in the code base with codespell.
  • Apply automatic formatting to more test runner code, these were previously not done.
  • Avoid using shutil.copytree which fails to work when directory already exists, instead provide nuitka.util.FileOperations.copyTree and use that exclusively.

Tests

  • Added new mode of operation to test runners, only that executes just one test and stops, useful during development.
  • Added new mechanism for standalone tests to expression modules that need to be importable, or else to skip the test by a special comment in the file, instead of by coded checks in the test runner.
  • Added also for more complex cases, another form of special comment, that can be any expression, that decides if the test makes sense.
  • Cover also setuptools in our distutils tests and made the execution more robust against variable behavior of distutils and setuptools.
  • Added standalone test for Urllib3.
  • Added standalone test for rsa.
  • Added standalone test for Pmw.
  • Added standalone test for passlib.

Summary

Again this release is a sign of increasing adoption of Nuitka. The GSoC 2019 is also showing effects, definitely will in the next release.

This release has a lot of new optimization, called specialization, but for it to really used, in many instances, we need to get away from working on C types for variables only, and get to them beig used for expressions more often. Otherwise much of the new special code is not used for most code.

The focus of this release has been again to open up development further and to incorporate findings from users. The number of fixes or new use cases working is astounding.

In upcoming releases, new built-ins will be optimized, and specialization of operations will hit more and more code now that the infrastructure for it is in place.

Jorj X. McKie

Hello everyone,

I am Jorj X. McKie (this is not my real name!) and I am an enthusiastic supporter of Nuitka. For the time being, my focus area within this package is standalone compilation and maintenance of the plugin feature.

Apart from extending Nuitka's support to virtually all Python packages living out there, my driving motivation in this respect is keeping up and improving the usability of Nuitka.

You may want to look into the Nuitka Utilities repository, which contains many things I am working on.

images/jorjmckie-harald-lieder.jpg

Short Biography

  • Born 1951 as Harald Lieder near the city of Leipzig
  • Studied Mathematics and Physics at the Frankfurt/M Johann-Wolfgang-Goethe University from 10/1970 through 06/1976 (Diploma in Mathematics)
  • Professional career overview
    • insurance company (4 years): application and systems programming (IBM mainframe, Cobol, Assembler), data center lead
    • book wholesaler (5 years): systems programming and data center lead (IBM mainframe)
    • consulting company (until 2009): leading of or participating in numerous large and very large projects mostly around banking applications, banking mergers, data center optimization and quality assurance.
    • since 2009: retirement, living on Isla Margarita, Venezuela. Enjoying my beautiful house, pursuing my hobbies: mathematics, science theory, biological evolution theory, science fiction (my pseudonym comes from novels by one of my favorite authors Frank Herbert) and of course programming (maintainer of PyMuPDF, and Nuitka).

Read more…

Tommy's Google Summer of Code Intro

Hi all! :)

My name is Taofeng Li and I go by the nickname of Tommy, github handle tommyli3318. I'm a Computer Science major at the University of California, Irvine in the United States. I am very excited to announce that I'll be contributing to Nuitka this summer under the Google Summer of Code program!

My project is "Nuitka support for PyPI top 50", it ensures Nuitka’s compatibility with the 50 most popular PyPI packages by setting up automated testing for each package. These automated tests will serve to be very important tools for the development of Nuitka, as they will be used to assure Nuitka’s compatibility with the most used Python packages every time Nuitka receives a new update.

My mentors will be Vaibhav Tulsyan, Kamran Ghanaat, and Kay Hayen. My project abstract can be seen here and the full details of my project can be found here.

I am greatly looking forward to the many things that I'll learn this summer!

Batakrishna Sahu

Hello everyone!

My name is Batakrishna Sahu and I am from India. I'm very grateful to be one of the 2 selected students who will work with Nuitka during Google Summer of Code 2019. My project is on "Optimizing Python 2 and Python 3 built-ins for Nuitka" mentored by Vaibhav Tulsyan, Kamran Ghanaat, Kay Hayen.

So far, ​there are a total of 69 built-ins in Python 3 and ​76 built-in in Python 2 out of which 27 built-ins are yet to be optimized in Nuitka​. My aim is to optimize those missing built-ins in Nuitka. This will make Nuitka a lot faster while working with built-ins.

I'm really excited to gain new experience while working on this project this summer.

Yours,

Nuitka this week #13

Communication vs. Coding

Communication was a lot more, just not these postings. Nuitka has ventured into important realms.

First, active developers have joined Nuitka. Second, as a consequence of the later, Nuitka indeed was able to participate with a insanely powerful offering of 4 mentors.

This meant, that a lot of technical or project management debt hat to be dealt with, and there was no time to make this kind of posting. This is probably good news on all fronts, except that I feel they are missing, and am glad to be able to resume them.

Google Summer of Code 2019

My Experience

This was the most crazy ride and fun. Became the admin of an active sub-organisation under the PSF umbrella for GSoC 2019. So shortly after the announcement of the project lists, students flooded in, and starting working on things right away as early as February.

They were working hard to prove themselves and get to know Nuitka, asking many questions, with us having lots of email interviews, and even a lot of video calls, where I personally talked to people about ideas.

This happened simultaneously to more experienced developers joining the project as well, making very important contributions.

This basically meant, that many days 90%-100% of my Nuitka time was for communication or working off technical debts I knew there were, or for preparing things, I want to have with multiple people working on the code base. This will be detailed in other sections.

Accepted Students

All of this has paid of in that we now have 2 really good students to work on 2 Nuitka projects over the summer. Let's welcome Batakrishna and Tommy in separate posts. They will introduce themselves and what they are going to do shortly, in separate posts.

Lets just say, this is fantastic news. We had so many good applications and its a shame, but we knew right away, not everybody who deserved it could be picked. But this is going to be a good chance for us to get to be open and welcoming to new people.

Optimization Work

Core Stuff

So, the 0.6.3 release (btw. on Windows, be sure to use the 0.6.3.1 hotfix), which was made as a consolidation effort to get the good work of mostly other people out, didn't contain much optimization work for the core, as that is still my thing.

However, this changed a lot. An idea that came to my mind for how to do the massive amounts of specialized helpers needed beyond + and +=, with which I had started for prior releases. And that is to use Jinja2 based templates for C, to generate the code.

This was an important idea. Took a while, but soon the manual code for + was already replaced with generated code, fixing a few bugs by the way, and from there, the generation was expanded to cover * as well.

Currently, support for the 3 (!) different kinds of additions (TrueDiv and FloorDir as well as Python2 default division, dubbed OldDiv in Nuitka was added along with - .

The reason, + and * were done first, is that they have special treatment for sequences, using sq_concat and sq_repeat, where the other operations will be more straightforward, e.g. nb_subtract (-) has a lot types supporting it and that makes those the easy cases.

I am saving a deeper explanation of 3 things we will need for the next time. Basically we need optimization of these things at compile time, and that is getting there, and code to use in the backend, and that is getting there, and a third thing, that is to use optimization knowledge to apply the special code as much as possible, and that is not yet fully there.

Faster Windows Dependencies

This is going to excite Windows users. After Orsiris de Jong had done a replacement for dependency walker that is faster, this had remained in an experimental status, just due to lack of time.

Recently however, I felt there is more time, after GSoC student selection has happened, and that I could finally work a bit on open issues like this. And when I wrote a dedicated tool, to analyse dependences with either technology to compare the results, I found that dendency walker founds a lot more things.

That was a turn down, but turns out, nothing it finds is stuff that should not be on the white list. In fact, it's all core Windows things, and from the System32 folder. That made me question, why we take anything from there (except maybe PythonXY.dll) at all, and after that change the performance changed dramatically.

The dependency walker now finishes a file in milliseconds. Actually the pefile is now slow (surely it ought to be compiled), and takes some seconds, for a file. That is amazing, and has lead to me to remove the parallel usage, and since pefile allows for perfect caching, and is Free Software, we will probably keep it.

This will address a widespread complaint of many Windows users of the standalone mode. This is now a relatively unnoticable part of the overall experience.

Currently I need to finish off some remaining problems with it, before putting it out in the wild. Getting this into a release will solve many newcomer issues.

Nuitka Organisation

Esp. for Google Summer of Code, Nuitka has sought and found mentors, some of which are highly experienced for the task. I will let them decide and write their own introduction, but I feel really blessed by them helping me out in my desperate calls for help. Without them, neither could Nuitka participate, nor could it even overachieve as much as it does.

Therefore I welcomed Kamran and Vaibhav into the organisation and they are excited to work the the 2 accepted students, that are also added.

Plugin Work

On the jorj branch there is a lot of work from Jorj that aims at adding support for more of the beasty stuff with hidden dependency and plugin needs.

He is also working at run time tracing of your program to be translated to automatic imports of just that. I am going to highlight this later, once I manage to cherry-pick the release ready parts from it for the next release.

But this definitely awesome stuff, and going to make Nuitka very easy to use for some people, even with stranger software.

Opening Up

This is also a teaser. But we did so much work for the 0.6.3 release to make sure information is there, and things are accessible and changeable by everyone, or even the pre-commit hook that I am very proud of.

But I shall save this for next week, otherwise it will be too much new information.

Twitter

I continue to be active there, although often I fall prey to of not wanting to talk about unfinished things. On Twitter of all things.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka