Python-Dateutil Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of dateutil to its nuitka-built .whl counterpart.

Dateutil standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the dateutil package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

The pytest results were very similar:

Regular pytests: ============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============
Nuitka wheel pytests: ============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Steps to Reproduce

  1. Clone dateutil and nuitka into a new folder
  2. Inside the dateutil folder, issue python -m pip install -r requirements-dev.txt to install its requirements.
  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for dateutil.
  4. Change into the nuitka folder and issue python setup.py develop.
  5. Change back into dateutil and issue python setup.py bdist_nuitka to build the dateutil wheel using nuitka. The newly built wheel should be found in the dist folder.
  6. Use pip to uninstall the existing dateutil, then issue python -m pip install followed by the newly built .whl filename.
  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for dateutil.

Uncompile Python

dateutil regular pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2074 items

dateutil\test\test_easter.py ........................................... [  2%]
........................................................................ [  5%]
................................................                         [  7%]
dateutil\test\test_import_star.py .                                      [  7%]
dateutil\test\test_imports.py .......................                    [  9%]
dateutil\test\test_internals.py ....                                     [  9%]
dateutil\test\test_isoparser.py ........................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
.......x...x............................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 35%]
.....................xx                                                  [ 36%]
dateutil\test\test_parser.py ........................................... [ 38%]
........................................................................ [ 42%]
........................................................................ [ 45%]
.................................xxxxxxxxxxxxxsss......                  [ 48%]
dateutil\test\test_relativedelta.py .................................... [ 49%]
.............................................                            [ 52%]
dateutil\test\test_rrule.py ............................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
................................................................x....... [ 78%]
..............                                                           [ 79%]
dateutil\test\test_tz.py ............................s...........sssssss [ 81%]
sssssssssssssssssssssssssssssssssssssssss..s............................ [ 84%]
xxx..s......................................s........................... [ 88%]
s....................................................................... [ 91%]
..........s....................................s.s....................ss [ 95%]
sssssssssssss....s..........s........................................... [ 98%]
.............                                                            [ 99%]
dateutil\test\test_utils.py .......                                      [ 99%]
dateutil\test\property\test_isoparse_prop.py .                           [ 99%]
dateutil\test\property\test_parser_prop.py ..                            [ 99%]
docs\exercises\solutions\mlk_day_rrule_solution.py .                     [100%]

============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============

Compiled with Nuitka

nuitka wheel pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2073 items

test\test_easter.py .................................................... [  2%]
........................................................................ [  5%]
.......................................                                  [  7%]
test\test_import_star.py .                                               [  7%]
test\test_imports.py .......................                             [  9%]
test\test_internals.py ....                                              [  9%]
test\test_isoparser.py ................................................. [ 11%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
......................................................................x. [ 25%]
..x..................................................................... [ 28%]
........................................................................ [ 32%]
........................................................................ [ 35%]
............xx                                                           [ 36%]
test\test_parser.py .................................................... [ 39%]
........................................................................ [ 42%]
........................................................................ [ 46%]
........................xxxxxxxxxxxxxsss......                           [ 48%]
test\test_relativedelta.py ............................................. [ 50%]
....................................                                     [ 52%]
test\test_rrule.py ..................................................... [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.......................................................x................ [ 79%]
.....                                                                    [ 79%]
test\test_tz.py ............................s...........ssssssssssssssss [ 81%]
ssssssssssssssssssssssssssssssss..s............................xxx..s... [ 85%]
...................................s...........................s........ [ 88%]
........................................................................ [ 92%]
.s....................................s.s....................sssssssssss [ 95%]
ssss....s..........s.................................................... [ 99%]
....                                                                     [ 99%]
test\test_utils.py .......                                               [ 99%]
test\property\test_isoparse_prop.py .                                    [ 99%]
test\property\test_parser_prop.py ..                                     [100%]

============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Urllib3 Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of urllib3 to its nuitka-built .whl counterpart.

Urllib3 standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the urllib3 package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

At first, the urllib3 nuitka-wheel pytest was crashing because of the unsafe assumption that imports will always exist (which is not the case if exceptions are thrown). Issue 413 was filed to record and fix this bug.

After the fixes, the pytests were ran again and the results were very similar:

Regular pytests: ====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======
Nuitka wheel pytests: ====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

The extra passes are suspicious and require more investigation into why they happen. To make that easy, we are going to fully automate the process and compare outputs with verbose pytest modes.

Steps to Reproduce

  1. Clone urllib3 and nuitka into a new folder
  2. Inside the urllib3 folder, issue python -m pip install -r dev-requirements.txt to install its requirements.
  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for urllib3.
  4. Change into the nuitka folder and issue python setup.py develop.
  5. Change back into urllib3 and issue python setup.py bdist_nuitka to build the urllib3 wheel using nuitka. The newly built wheel should be found in the dist folder.
  6. Use pip to uninstall the existing urllib3, then issue python -m pip install followed by the newly built .whl filename.
  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for urllib3.

Uncompile Python

urllib3 regular pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py ..                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py ..............................F.F.F..                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
.................................................Uncaught exception, closing connection.
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======

Compiled with Nuitka

nuitka wheel pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py .F                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py .....................................                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

Nuitka Release 0.6.4

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains many bug fixes all across the board. There is also new optimization and many organisational improvements.

Bug Fixes

  • When linking very large programs or packages, with gcc compiler, Scons can produce commands that are too large for the OS. This happens sooner on the Windows OS, but also on Linux. We now have a workaround that avoids long command lines by using @sources.tmp syntax.
  • Standalone: Remove temporary module after its use, instead of keeping it in sys.modules where e.g. Quart code tripped over its __file__ value that is illegal on Windows.
  • Fixed non-usage of our enhanced detection of gcc version for compilers if given as a full path.
  • Fixed non-detection of gnu-cc as a form of gcc compiler.
  • Python3.4: The __spec__ value corrections for compiled modules was not taking into account that there was a __spec__ value, which can happen if something is wrapping imported modules.
  • Standalone: Added implicit dependencies for passlib.
  • Windows: Added workaround for OS command line length limit in compilation with MinGW64.
  • Python2: Revive the enum plugin, there are backports of the buggy code it tries to patch up.
  • Windows: Fixup handling of SxS with non zero language id, these occur e.g. in Anaconda.
  • Plugins: Handle multiple PyQt plugin paths, e.g. on openSUSE this is done, also enhanced finding that path with Anaconda on Windows.
  • Plugins: For multiprocessing on Windows, allow the .exe suffix to not be present, which can happen when ran from command line.
  • Windows: Better version checks for DLLs on Python3, the ctypes helper code needs more definitions to work properly.
  • Standalone: Added support for both pycryptodome and pycryptodomex.
  • Fix, the chr built-in was not giving fully compatible error on non number input.
  • Fix, the id built-in doesn't raise an exception, but said otherwise.
  • Python3: Proper C identifiers for names that fit into latin-1, but are not ascii encodings.

New Features

  • Windows: Catch most common user error of using compiler from one architecture against Python from another. We now check those and compare it, and if they do not match, inform the user directly. Previously the compilation could fail, or the linking, with cryptic errors.
  • Distutils: Using setuptools and its runners works now too, not merely only pure distutils.
  • Distutils: Added more ways to pass Nuitka specific options via distutils.
  • Python3.8: Initial compatibility changes to get basic tests to work.

Organisational

  • Nuitka is participating in the GSoC 2019 with 2 students, Batakrishna and Tommy.
  • Point people creating PRs to using the pre-commit hook in the template. Due to making the style issues automatic, we can hope to encounter less noise and resulting merge problems.
  • Many improvements to the pre-commit hook were done, hopefully completing its development.
  • Updated to latest pylint, black, and isort versions, also added codespell to check for typos in the source code, but that is not automated yet.
  • Added description of how to use experimental flags for your PRs.
  • Removed mirroring from Bitbucket and Gitlab, as we use the Github organisation features.
  • Added support for Ubuntu Disco, removed support for Ubuntu Artful packages.

Optimization

  • Windows: Attach data blobs as Windows resource files directly for programs and avoid using C data files for modules or MinGW64, which can be slow.
  • Specialization of helper codes for + is being done for more types and more thoroughly and fully automatic with Jinja2 templating code. This does replace previously manual code.
  • Added specialization of helper codes for * operation which is entirely new.
  • Added specialization of helper codes for - operation which is entirely new.
  • Dedicated nodes for specialized operations now allow to save memory and all use type shape based analysis to predict result types and exception control flow.
  • Better code generation for boolean type values, removing error checks when possible.
  • Better static analysis for even more type operations.

Cleanups

  • Fixed many kinds of typos in the code base with codespell.
  • Apply automatic formatting to more test runner code, these were previously not done.
  • Avoid using shutil.copytree which fails to work when directory already exists, instead provide nuitka.util.FileOperations.copyTree and use that exclusively.

Tests

  • Added new mode of operation to test runners, only that executes just one test and stops, useful during development.
  • Added new mechanism for standalone tests to expression modules that need to be importable, or else to skip the test by a special comment in the file, instead of by coded checks in the test runner.
  • Added also for more complex cases, another form of special comment, that can be any expression, that decides if the test makes sense.
  • Cover also setuptools in our distutils tests and made the execution more robust against variable behavior of distutils and setuptools.
  • Added standalone test for Urllib3.
  • Added standalone test for rsa.
  • Added standalone test for Pmw.
  • Added standalone test for passlib.

Summary

Again this release is a sign of increasing adoption of Nuitka. The GSoC 2019 is also showing effects, definitely will in the next release.

This release has a lot of new optimization, called specialization, but for it to really used, in many instances, we need to get away from working on C types for variables only, and get to them beig used for expressions more often. Otherwise much of the new special code is not used for most code.

The focus of this release has been again to open up development further and to incorporate findings from users. The number of fixes or new use cases working is astounding.

In upcoming releases, new built-ins will be optimized, and specialization of operations will hit more and more code now that the infrastructure for it is in place.

Jorj X. McKie

Hello everyone,

I am Jorj X. McKie (this is not my real name!) and I am an enthusiastic supporter of Nuitka. For the time being, my focus area within this package is standalone compilation and maintenance of the plugin feature.

Apart from extending Nuitka's support to virtually all Python packages living out there, my driving motivation in this respect is keeping up and improving the usability of Nuitka.

You may want to look into the Nuitka Utilities repository, which contains many things I am working on.

images/jorjmckie-harald-lieder.jpg

Short Biography

  • Born 1951 as Harald Lieder near the city of Leipzig
  • Studied Mathematics and Physics at the Frankfurt/M Johann-Wolfgang-Goethe University from 10/1970 through 06/1976 (Diploma in Mathematics)
  • Professional career overview
    • insurance company (4 years): application and systems programming (IBM mainframe, Cobol, Assembler), data center lead
    • book wholesaler (5 years): systems programming and data center lead (IBM mainframe)
    • consulting company (until 2009): leading of or participating in numerous large and very large projects mostly around banking applications, banking mergers, data center optimization and quality assurance.
    • since 2009: retirement, living on Isla Margarita, Venezuela. Enjoying my beautiful house, pursuing my hobbies: mathematics, science theory, biological evolution theory, science fiction (my pseudonym comes from novels by one of my favorite authors Frank Herbert) and of course programming (maintainer of PyMuPDF, and Nuitka).

Read more…

Tommy's Google Summer of Code Intro

Hi all! :)

My name is Taofeng Li and I go by the nickname of Tommy, github handle tommyli3318. I'm a Computer Science major at the University of California, Irvine in the United States. I am very excited to announce that I'll be contributing to Nuitka this summer under the Google Summer of Code program!

My project is "Nuitka support for PyPI top 50", it ensures Nuitka’s compatibility with the 50 most popular PyPI packages by setting up automated testing for each package. These automated tests will serve to be very important tools for the development of Nuitka, as they will be used to assure Nuitka’s compatibility with the most used Python packages every time Nuitka receives a new update.

My mentors will be Vaibhav Tulsyan, Kamran Ghanaat, and Kay Hayen. My project abstract can be seen here and the full details of my project can be found here.

I am greatly looking forward to the many things that I'll learn this summer!

Batakrishna Sahu

Hello everyone!

My name is Batakrishna Sahu and I am from India. I'm very grateful to be one of the 2 selected students who will work with Nuitka during Google Summer of Code 2019. My project is on "Optimizing Python 2 and Python 3 built-ins for Nuitka" mentored by Vaibhav Tulsyan, Kamran Ghanaat, Kay Hayen.

So far, ​there are a total of 69 built-ins in Python 3 and ​76 built-in in Python 2 out of which 27 built-ins are yet to be optimized in Nuitka​. My aim is to optimize those missing built-ins in Nuitka. This will make Nuitka a lot faster while working with built-ins.

I'm really excited to gain new experience while working on this project this summer.

Yours,
Batakrishna

Nuitka this week #13

Communication vs. Coding

Communication was a lot more, just not these postings. Nuitka has ventured into important realms.

First, active developers have joined Nuitka. Second, as a consequence of the later, Nuitka indeed was able to participate with a insanely powerful offering of 4 mentors.

This meant, that a lot of technical or project management debt hat to be dealt with, and there was no time to make this kind of posting. This is probably good news on all fronts, except that I feel they are missing, and am glad to be able to resume them.

Google Summer of Code 2019

My Experience

This was the most crazy ride and fun. Became the admin of an active sub-organisation under the PSF umbrella for GSoC 2019. So shortly after the announcement of the project lists, students flooded in, and starting working on things right away as early as February.

They were working hard to prove themselves and get to know Nuitka, asking many questions, with us having lots of email interviews, and even a lot of video calls, where I personally talked to people about ideas.

This happened simultaneously to more experienced developers joining the project as well, making very important contributions.

This basically meant, that many days 90%-100% of my Nuitka time was for communication or working off technical debts I knew there were, or for preparing things, I want to have with multiple people working on the code base. This will be detailed in other sections.

Accepted Students

All of this has paid of in that we now have 2 really good students to work on 2 Nuitka projects over the summer. Let's welcome Batakrishna and Tommy in separate posts. They will introduce themselves and what they are going to do shortly, in separate posts.

Lets just say, this is fantastic news. We had so many good applications and its a shame, but we knew right away, not everybody who deserved it could be picked. But this is going to be a good chance for us to get to be open and welcoming to new people.

Optimization Work

Core Stuff

So, the 0.6.3 release (btw. on Windows, be sure to use the 0.6.3.1 hotfix), which was made as a consolidation effort to get the good work of mostly other people out, didn't contain much optimization work for the core, as that is still my thing.

However, this changed a lot. An idea that came to my mind for how to do the massive amounts of specialized helpers needed beyond + and +=, with which I had started for prior releases. And that is to use Jinja2 based templates for C, to generate the code.

This was an important idea. Took a while, but soon the manual code for + was already replaced with generated code, fixing a few bugs by the way, and from there, the generation was expanded to cover * as well.

Currently, support for the 3 (!) different kinds of additions (TrueDiv and FloorDir as well as Python2 default division, dubbed OldDiv in Nuitka was added along with - .

The reason, + and * were done first, is that they have special treatment for sequences, using sq_concat and sq_repeat, where the other operations will be more straightforward, e.g. nb_subtract (-) has a lot types supporting it and that makes those the easy cases.

I am saving a deeper explanation of 3 things we will need for the next time. Basically we need optimization of these things at compile time, and that is getting there, and code to use in the backend, and that is getting there, and a third thing, that is to use optimization knowledge to apply the special code as much as possible, and that is not yet fully there.

Faster Windows Dependencies

This is going to excite Windows users. After Orsiris de Jong had done a replacement for dependency walker that is faster, this had remained in an experimental status, just due to lack of time.

Recently however, I felt there is more time, after GSoC student selection has happened, and that I could finally work a bit on open issues like this. And when I wrote a dedicated tool, to analyse dependences with either technology to compare the results, I found that dendency walker founds a lot more things.

That was a turn down, but turns out, nothing it finds is stuff that should not be on the white list. In fact, it's all core Windows things, and from the System32 folder. That made me question, why we take anything from there (except maybe PythonXY.dll) at all, and after that change the performance changed dramatically.

The dependency walker now finishes a file in milliseconds. Actually the pefile is now slow (surely it ought to be compiled), and takes some seconds, for a file. That is amazing, and has lead to me to remove the parallel usage, and since pefile allows for perfect caching, and is Free Software, we will probably keep it.

This will address a widespread complaint of many Windows users of the standalone mode. This is now a relatively unnoticable part of the overall experience.

Currently I need to finish off some remaining problems with it, before putting it out in the wild. Getting this into a release will solve many newcomer issues.

Nuitka Organisation

Esp. for Google Summer of Code, Nuitka has sought and found mentors, some of which are highly experienced for the task. I will let them decide and write their own introduction, but I feel really blessed by them helping me out in my desperate calls for help. Without them, neither could Nuitka participate, nor could it even overachieve as much as it does.

Therefore I welcomed Kamran and Vaibhav into the organisation and they are excited to work the the 2 accepted students, that are also added.

Plugin Work

On the jorj branch there is a lot of work from Jorj that aims at adding support for more of the beasty stuff with hidden dependency and plugin needs.

He is also working at run time tracing of your program to be translated to automatic imports of just that. I am going to highlight this later, once I manage to cherry-pick the release ready parts from it for the next release.

But this definitely awesome stuff, and going to make Nuitka very easy to use for some people, even with stranger software.

Opening Up

This is also a teaser. But we did so much work for the 0.6.3 release to make sure information is there, and things are accessible and changeable by everyone, or even the pre-commit hook that I am very proud of.

But I shall save this for next week, otherwise it will be too much new information.

Twitter

I continue to be active there, although often I fall prey to of not wanting to talk about unfinished things. On Twitter of all things.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka Release 0.6.3

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This has a focus on organisational improvements. With more and more people joining Nuitka, normal developers as well as many GSoC 2019 students, the main focus was to open up the development tools and processes, and to improve documentation.

That said, an impressive amount of bug fixes was contributed, but optimization was on hold.

Bug Fixes

  • Windows: Added support for running compiled binaries in unicode path names.
  • Standalone: Added support for crytodomex and pycparser packages.
  • Standalone: Added support for OpenSSL support in PyQt on Windows.
  • Standalone: Added support for OpenGL support with QML in PyQt on Windows.
  • Standalone: Added support for SciPy and extended the NumPy plugin to also handle it.
  • UI: The option --plugin-list still needed a positional argument to work.
  • Make sure sys.base_prefix is set correctly too.
  • Python3: Also make sure sys.exec_prefix and sys.base_exec_prefix are set correctly.
  • Standalone: Added platform plugins for PyQt to the default list of sensible plugins to include.
  • Fix detection of standard library paths that include .. path elements.

Optimization

  • Avoid static C++ runtime library when using MinGW64.

New Features

  • Plugins: A plugin may now also generate data files on the fly for a given module.
  • Added support for FreeBSD/PowerPC arch which still uses gcc and not clang.

Organisational

  • Nuitka is participating in the GSoC 2019.
  • Added documentation on how to create or use Nuitka plugins.
  • Added more API doc to functions that were missing them as part of the ongoing effort to complete it.
  • Updated to latest PyLint 2.3.1 for checking the code.
  • Scons: Using newer Scons inline copy with Python 2.7 as, the old one remains only used with Python 2.6, making it easier to know the relevant code.
  • Autoformat was very much enhanced and handles C and ReST files too now. For Python code it does pylint comment formatting, import statement sorting, and blackening.
  • Added script misc/install-git-hooks.py that adds a commit hook that runs autoformat on commit. Currently it commits unstaged content and therefore is not yet ready for prime time.
  • Moved adapted CPython test suites to Github repository under Nuitka Organisation.
  • Moved Nuitka-website repository to Github repository under Nuitka Organisation.
  • Moved Nuitka-speedcenter repository to Github repository under Nuitka Organisation.
  • There is now a Gitter chat for Nuitka community.
  • Many typo and spelling corrections on all the documentation.
  • Added short installation guide for Nuitka on Windows.

Cleanups

  • Moved commandline parsing helper functions from common code helpers to the main program where of course their only usage is.
  • Moved post processing of the created standalone binary from main control to the freezer code.
  • Avoid using chmod binary to remove executable bit from created extension modules.
  • Windows: Avoid using rt.exe and mt.exe to deal with copying the manifest from the python.exe to created binaries. Instead use new code that extracts and adds Windows resources.
  • Fixed many ResourceWarnings on Python3 by improved ways of handling files.
  • Fixed deprecation warnings related to not using collections.abc.
  • The runners in bin directory are now formatted with black too.

Tests

  • Detect Windows permission errors for two step execution of Nuitka as well, leading to retries should they occur.
  • The salt value for CPython cached results was improved to take more things into account.
  • Tests: Added more trick assignments and generally added more tests that were so far missing.

Summary

With the many organisational changes in place, my normal work is expected to resume for after and yield quicker improvements now.

It is also important that people are now enabled to contribute to the Nuitka web site and the Nuitka speedcenter. Hope is to see more improvements on this otherwise neglected areas.

And generally, it's great to see that a community of people is now looking at this release in excitement and pride. Thanks to everybody who contributed!

Nuitka Release 0.6.2

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release has a huge focus on organizational things. Nuitka is growing in terms of contributors and supported platforms.

Bug Fixes

  • Fix, the Python flag --python-flag=-O was removing doc strings, but that should only be done with --python-flag=-OO which was added too.
  • Fix, accelerated binaries failed to load packages from the virtualenv (not venv) that they were created and ran with, due to not propagating sys.prefix.
  • Standalone: Do not include plat-* directories as frozen code, and also on some platforms they can also contain code that fails to import without error.
  • Standalone: Added missing implicit dependency needed for newer NumPy versions.

New Features

  • Added support for Alpine Linux.

  • Added support for MSYS2 based Python on Windows.

  • Added support for Python flag --python flag=-OO, which allows to remove doc strings.

  • Added experimental support for pefile based dependency scans on Windows, thanks to Orsiris for this contribution.

  • Added plugin for proper Tkinter standalone support on Windows, thanks to Jorj for this contribution.

  • There is now a __compiled__ attribute for each module that Nuitka has compiled. Should be like this now, and contains Nuitka version information for you to use, similar to what sys.version_info gives as a namedtuple for your checks.

    __nuitka_version__(major=0, minor=6, micro=2, releaselevel='release')
    

Optimization

  • Experimental code for variant types for int and long values, that can be plain C value, as well as the PyObject *. This is not yet completed though.
  • Minor refinements of specialized code variants reducing them more often the actual needed code.

Organisational

  • The Nuitka Github Organisation that was created a while ago and owns the Nuitka repo now, has gained members. Check out https://github.com/orgs/Nuitka/people for their list. This is an exciting transformation for Nuitka.
  • Nuitka is participating in the GSoC 2019 under the PSF umbrella. We hope to grow even further. Thanks to the mentors who volunteered for this important task. Check out the GSoC 2019 page and thanks to the students that are already helping out.
  • Added Nuitka internal API documentation that will receive more love in the future. It got some for this release, but a lot is missing.
  • The Nuitka code has been black-ened and is formatted with an automatic tool now all the way, which makes contributors lives easier.
  • Added documentation for questions received as part of the GSoC applications and ideas work.
  • Some proof reading pull requests were merged for the documentation, thanks to everybody who addresses these kinds of errors. Sometimes typos, sometimes broken links, etc.
  • Updated inline copy of Scons used for Python3 to 3.0.4, which hopefully means more bugs are fixed.

Summary

This release is a sign of increasing adoption of Nuitka. The GSoC 2019 is showing early effects, as is more developers joining the effort. These are great times for Nuitka.

This release has not much on the optimization side that is user visible, but the work that has begun is capable of producing glorious benchmarks once it will be finished.

The focus on this and coming releases is definitely to open up the Nuitka development now that people are coming in as permanent or temporary contributors in (relatively) high numbers.

Nuitka this week #12

Communication vs. Coding

Over the holiday season I sort of neglected these postings, but there are other fields, where I have put my focus, but I think these postings are now going to resume.

A quick update in a while in an eternally fluent situation, and not too polished.

Optimization Work

So, the 0.6.2 optimization work has been not a lot yet. I started some work on int/long C type and it looked good.

Week of Bugfixing

So I did this, and it turned out rather long. I continued fixing things and finishing up open things to the point that it now is clean. I hate to be falling behind. I am touching on a few more interesting topics.

Python flag -O and -OO

Nuitka was supporting -O but doing what should only the done for -OO which I only learned of now. So this is going to be supported both now.

Virtualenv vs. venv

Accelerated binaries were not running with full sys.path in the virtualenv, because site module uses sys.prefix and that was not propagated,but it now is.

Google Summer of Code for Nuitka

So the GSoC 2019 page is shaping up, the pull request to list Nuitka on the PSF project pages has been merged. More tweaking will be needed to get into perfect shape, and that has been my main communication time effort.

Finally somebody had stepped up for mentor, which means a lot to me. Now we need to see if students are willing to pick us.

And because I publish this too late. Already a bunch of stundents are showing interest and are hacking on Nuitka, which keeps me even more busy, and makes me extremely happy.

Nuitka Organisation

The Nuitka organisation on Github was introduced for my a while ago, and I had transferred the ownership of the Nuitka repository to there. Having moved the issue tracking to there, I was going more all in on it.

Recently more people have submitted PRs and with incredible quality and willingness to support it even after merge of their PR. No dry by contribution, but people looking to actually improve Nuitka together with me.

Therefore I welcome Orsiris and Jorj on board and am really happy about it.

Windows Dependencies from pefile

So Orsiris de Jong implemented a replacement for the dependency walker based code to scan DLLs for Windows standalone using the pefile module, which is likely way better at this. For starters it's experimental in the next release, but I expect it to soon become the default.

Tkinter Plugin

And Jorj X. McKie implemented a Windows Tkinter plug-in that deals with copying of the TCL installation and integrating it with Nuitka standalone for distribution.

That is very nice and seems to affect a lot of people using that GUI style it seems.

Numpy Plugin

And Jorj X. McKie also implemented a NumPy plug-in that deals with the various acceleration methods there are, e.g. MKL, Blas, I have no idea. It copies the needed DLLs along and makes sure that numpy stays fast in standalone mode.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka