Nuitka Blog - Posts tagged NTW

Nuitka this week #16

2025-02-15T00:00:00+01:00

Hey Nuitka users! This started out as an idea of a weekly update, but that hasn’t happened, and so we will switch it over to just writing up when something interesting happens and then push it out relatively immediately when it happens.

Nuitka Onefile Gets More Flexible: `--onefile-cache-mode` and `{PROGRAM_DIR}`

We’ve got a couple of exciting updates to Nuitka’s onefile mode that give you more control and flexibility in how you deploy your applications. These enhancements stem from real-world needs and demonstrate Nuitka’s commitment to providing powerful and adaptable solutions.

Taking Control of Onefile Unpacking: `--onefile-cache-mode`

Onefile mode is fantastic for creating single-file executables, but the management of the unpacking directory where the application expands has sometimes been a bit… opaque. Previously, Nuitka would decide whether to clean up this directory based on whether the path used runtime-dependent variables. This made sense in theory, but in practice, it could lead to unexpected behavior and made debugging onefile issues harder.

Now, you have complete control! The new --onefile-cache-mode option lets you explicitly specify what happens to the unpacking directory:

--onefile-cache-mode=auto: This is the default behavior. Nuitka will remove the unpacking directory unless runtime-dependent values were used in the path specification. This is the same behavior as previous versions.
--onefile-cache-mode=cached: The unpacking directory is not removed and becomes a persistent, cached directory. This is useful for debugging, inspecting the unpacked files, or if you have a use case that benefits from persistent caching of the unpacked data. The files will remain available for subsequent runs.
--onefile-cache-mode=temporary: The unpacking directory is removed after the program exits.

This gives you the power to choose the behavior that best suits your needs. No more guessing!

Relative Paths with `{PROGRAM_DIR}`

Another common request, particularly from users deploying applications in more restricted environments, was the ability to specify the onefile unpacking directory relative to the executable itself. Previously, you were limited to absolute paths or paths relative to the user’s temporary directory space.

We’ve introduced a new variable, {PROGRAM_DIR}, that you can use in the --onefile-tempdir-spec option. This variable is dynamically replaced at runtime with the full path to the directory containing the onefile executable.

For example:

nuitka --onefile --onefile-tempdir-spec="{PROGRAM_DIR}/.myapp_data" my_program.py

This would create a directory named .myapp_data inside the same directory as the my_program.exe (or my_program on Linux/macOS) and unpack the application there. This is perfect for creating truly self-contained applications where all data and temporary files reside alongside the executable.

Nuitka Commercial and Open Source

These features, like many enhancements to Nuitka, originated from a request by a Nuitka commercial customer. This highlights the close relationship between the commercial offerings and the open-source core. While commercial support helps drive development and ensures the long-term sustainability of Nuitka, the vast majority of features are made freely available to all users.

Give it a Try!

This change will be in 2.7 and is currently

We encourage you to try out these new features and let us know what you think! As always, bug reports, feature requests, and contributions are welcome on GitHub.

Nuitka this week #15

2024-01-30T00:00:00+01:00

This is a weekly update, or at least it’s supposed to be of what’s going on in Nuitka land, for you to learn about ongoing developments and important changes to the project.

In this issue, I am first going to cover a bit of backlog from news update missed in the past, but also covering very exciting changes from this week.

Nuitka Action

A GitHub Action is a component used in GitHub workflows. These are yaml driven configurations that can cause GitHub to do automatic building of your software.

Many of the more professional users build their binaries as part of GitHub workflows, and Nuitka and Nuitka commercial are both used in that way a lot. Many times people do it on their own, i.e. install Nuitka by hand, and call it by hand, which is kind of not the preferred way for many people.

Enter the great Nuitka Action which was originally created by Jim Kring, who handed over the maintenance of it to the Nuitka organization that has further refined it. This was a great contribution that makes it everything easier for Nuitka users on GitHub if they want to use it.

- name: Build Executable
  uses: Nuitka/Nuitka-Action@main
  with:
    nuitka-version: main
    script-name: kasa_cli
    onefile: true

- name: Upload Artifacts
  uses: actions/upload-artifact@v3
  with:
    name: ${{ runner.os }} Build
    path: |
      build/*.exe
      build/*.bin
      build/*.app/**/*

Options of Nuitka are exposed as yaml attributes. The documentation of this mapping could be very much enhanced, but basically it’s just dropping the -- part from e.g. --onefile and for toggles, you say true.

Now one interesting limitation of GitHub Action, I have come across this week and that is that it’s not easily possible to specify an option twice. For some values in Nuitka, that however is necessary. Where module names are acceptable, a , separation os supported, but with file paths, we don’t do that, e.g. not with --include-data-dir (note --include-package-data is much better to use) but that is the one it came up for.

But now we support splitting by new-line from GitHub actions for everything that produces a list value as a Nuitka option. See below for a very nice example of how the | in Yaml makes that even easy to read.

- name: Build Executable
  uses: Nuitka/Nuitka-Action@main
  with:
    nuitka-version: main
    script-name: kasa_cli
    onefile: true
    include-data-dir: |
      source_path_dir1=dest_path_dir1
      source_path_dir2=dest_path_dir2
      source_path_dir3=dest_path_dir3

Note

This works with Nuitka 2.0 or higher.

The Nuitka-Action is permanently refined. Just today we updated its caching action to latest, and there is an ongoing activity to improve options. We started to generate options from Nuitka help output directly, so that it is easier to add support for new Options in Nuitka, and to generally make them more consistent.

Community

On the Discord server, you can get in touch with an ever more vibrant community of Nuitka users. You are welcome to join us on the Discord server for Nuitka community where you can hang out with the developers and ask questions. It’s not intended as an interactive manual. You are supposed to read the docs for yourself first. And issues are best reported to GitHub.

I am also now occasionally on the Python Discord server. Mostly when I get summoned to answer questions that my community thinks make sense, and have been awarded the community role there, which is pretty nice. I seem to make new connections there.

Optimization Work

For me this is extremely exciting, this has been on my nerves for a long time, and I didn’t have the time to figure it out. Now for the scalability work, I wanted to make sure the algorithm used for loop type analysis is actually going to be sustainable, before I optimize the implementation to scale better.

And low and behold, one of my oldest code examples, the one I mean to demonstrate C type performance from Python code with, has failed to get proper results for a long time now. But this changed this week and it’s part of the 2.0 release, making it my mind worth the bump itself. Checkout this annotated code.

# Initially the value of undefined "i" is "NUITKA_NINT_UNASSIGNED"
# in its indicator part. The C compiler will remove that assignment
# as it's only checked in the assignment coming up.

i = 0
# Assignment from a constant, produces a value where both the C
# and the object value are value. This is indicated by a value
# of "NUITKA_NINT_BOTH_VALID". The code generation will assign
# both the object member from a prepared value, and the clong
# member to 0.

# For the conditional check, "NUITKA_NINT_CLONG_VALID" will
# always be set, and therefore function will resort to comparing
# that clong member against 9 simply, that will always be very
# fast. Depending on how well the C compiler can tell if an overflow
# can even occur, such that an object might get created, it can even
# optimize that statically. In this case it probably could, but we
# do not rely on that to be fast.
while i < 9:  # RICH_COMPARE_LT_CBOOL_NINT_CLONG
    # Here, we might change the type of the object. In Python2,
    # this can change from ``int`` to ``long``, and our type
    # analysis tells us that. We can consider another thing,
    # not "NINT", but "NINTLONG" or so, to special case that
    # code. We ignore Python2 here, but multiple possible types
    # will be an issue, e.g. list or tuple, float or complex.
    # So this calls a function, that returns a value of type
    # "NINT" (actually it will become an in-place operation
    # but lets ignore that too).
    # That function is "BINARY_OPERATION_ADD_NINT_NINT_CLONG"(i, 1)
    # and it is going to check if the CLONG is valid, add the one,
    # and set to result to a new int. It will reset the
    # "NUITKA_NINT_OBJECT_VALID" flag, since the object will not be
    # bothered to create.
    i = i + 1

# Since "NUITKA_INT_OBJECT_VALID" not given, need to create the
# PyObject and return it.
return i

Now that the loop analysis works, I will be much happier to make the value trace collection faster. I will describe it when I do it. From here on for optimization, the C type NINT needs to be created and code generation for the branching helper functions be added, and then the should see this perform perfectly.

Functions like RICH_COMPARE_LT_CBOOL_NINT_CLONG will look like this. We do not yet have RICH_COMPARE_LT_CBOOL_LONG_CLONG which it will fall back to, but we did RICH_COMPARE_LT_CBOOL_INT_CLONG for Python2 a while ago, and we could expand that no problem.

extern bool RICH_COMPARE_LT_CBOOL_NINT_CLONG(nuitka_long *operand1, long operand2) {
   if (operand1->validity & NUITKA_LONG_VALUE_VALID) {
      return operand1->long_value < operand2;
   } else {
      return RICH_COMPARE_LT_CBOOL_LONG_CLONG(operand1->long_object, operand2);
   }
}

Once I get to that, performance will get a hot topic. From there then, adding sources of type information, be it profile guided compilation, be it type annotations, be it ever better compile time type inference, will start to make a lot more sense.

Nuitka 2.0

The 2.0 release has been made. I am going to announce it separately. I am usually waiting for a few days, to settle potentially regressions. This time older C compiler support needed a fixup, there is always something. And then I announce it when I feel that the regressions are gone and that new users will not encounter obvious breakage at all.

Technical Writer

When I first launched Nuitka commercial, I needed to get myself financially supported, dropping my day job after 20 years. I am willing to say that has happened.

Now as you all know, Nuitka is technically excellent. I cannot say the same thing of the documentation. Large parts of Nuitka commercial are still not even publicly described. The User Manual of Nuitka is good, but not nearly as good as it should be. The website is kind of unorganized. It’s pretty obvious that my talent is not there. I have received plenty of help over the years, but it’s not been enough to match Nuitka and Nuitka commercial outstanding quality.

So, what I did this week, after seeing that my financial projection for the coming years seems to allow it, is to attempt and hire people on a free lancing basis. The first step is a technical writer. She will know very little of Python and even the terminal, but she will know how to organize and improve the content of Nuitka.

It will take time for her to get there and this is very fresh.

Nuitka Website as a DevContainer

As a first necessary step to make it easier to contribute to the Nuitka documentation, the website repo, has gained DevContainer configuration. It will install a small Ubuntu via docker (or podman if you configured Visual Code like that), and run the pipenv environment and a daemon to open the website.

The docs for that are spotty right now, and the Technical Writer that is using that, is tasked to improve this right now.

It should become really easy that way to contribute enhancements to the documentation.

I have yet to figure out how to handle the release matching documentation vs. website documentation for user manual. But the idea is certainly that the Nuitka documentation is edited on the website.

Nuitka Package Configuration

So I added a post that explains variables, but the one for parameters, I still need to do that and also update the actual reference documentation.

Teasers

Future TWN still have a lot to talk about, we will speak about Nuitka-Python (our own Python fork with incredible capabilities), about Nuitka-Watch (our way of making sure Nuitka works with PyPI packages and hot-fixes to not regress), about compilation reports as a new feature, Windows AV stuff, onefile improvements, and so on and so on. I got interesting stuff for many weeks. Limiting myself for now or I will never publish this.

Twitter and Mastodon

I should be more active there, although often I fail due to not wanting to talk about unfinished things, so actually I do not post there as much.

And lets not forget, having followers make me happy. So do re-tweets. Esp. those, please do them.

Help Wanted

Nuitka definitely needs more people to work on it. I hope the technical writer will aid us in better laying how ways for you to help.

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka this week #14

2024-01-12T00:00:00+01:00

This is a weekly update, or at least it’s supposed to be of what’s going on in Nuitka land, for you to learn about ongoing developments and important changes to the project.

Communication vs. Coding

After GSoC 2019, it seems I dropped off with communication about Nuitka quite a lot, e.g. I stopped “Nuitka This Week”. The reasons are multi-facetted. I think part of the reason is that I was getting busy, part of it clearly also was Corona. But also a more dreadful change in my private life, where the real life Nuitka, my wife, became ill for a long time. Effectively it’s only become really better mid last year.

I think, this caused me to go full into code for Nuitka, and to launch Nuitka commercial, but generally to become more quiet. I have already relaxed this for a bit, e.g. about Python 3.11, I made a bunch of postings.

So this one is a bit general to start off, but I also provide fairly recent details about what I worked on for 2.0 as well.

Nuitka has evolved a lot

From a usability standpoint, ever since I went all in with Nuitka, but also before, the out of the box experience of Nuitka has become ever better. And even 2.0 will take it noticeable further. It’s the premier choice for Python deployment if you want efficiency. Its onefile mode is pretty great already and is continuously getting better.

It’s fair to say that Nuitka was great in 2019. I think in 2023 it became almost amazing for deployment. This is in large part due to working on the Yaml configuration and these things. In 2024 I hope to get it really smooth.

I actually made posts about the Yaml stuff, and I will resume the series shortly, basically it allows people to help improve the deployment side of Nuitka, e.g. missing DLLs and data files, hacks needed, etc. for packages, and it’s quickly becoming better and more complete.

Community

On the Discord server, I have been in touch with users of Nuitka a lot more. You are welcome to join us on the Discord server for Nuitka community where you can hang out with the developers and ask questions. It’s not intended as an interactive manual. You are supposed to read the docs for yourself first.

Optimization Work

I think, it’s in vain to explain what I did for performance in all that time. Mostly, some technical debts for Python3 were collected, extending to advantage in in speed of Nuitka over pure Python again. The advantage compared on Python2 was not as present, and still is not, on Python3, but for 3.10 it’s pretty good.

The major breakthroughs have not happened. But I will be talking about the plans, these sure are exciting. Lots of things are in place, some are not, but I hope to get there.

Current Evolutions

So many things on my mind right now, for one, I guess, 4 plugin changes that I have yet to document in new postings. Two are visible here in this code.

- module-name: 'toga.platform' # checksum: 4db91cac
  variables:
    setup_code: 'import toga.platform'
    declarations:
      'toga_backend_module_name': 'toga.platform.get_platform_factory().__name__'
  anti-bloat:
    - change_function:
        'get_platform_factory': "'importlib.import_module(%r)' % get_variable('toga_backend_module_name')"

First, the checksum. Nuitka is going to warn you about checking your user yaml files for correctness in the future. Since it often finds structural problems, very much needed, since yaml is whitespace sensitive, and you never know what it is: a list, a dict, etc. but the schema we created, can tell.

Second, variables are a new section, and in fact so new, they are not even documented. They can be used to query values from code at compile time. In this case we are using it to get at the backend to use, so we can tell it at runtime. Otherwise, it’s hidden to Nuitka, and could e.g. still be subject to a changed decision from environment variables, something we typically do not want.

For the third and fourth thing, we need another example. Torch can use a JIT to speed up some things, with a compilation very similar to what Nuitka does. That however needs a compiler and the source code on the target platform. Not an easy ask for all kinds of deployments. A new feature makes this easier than before.

- module-name: 'torch' # checksum: 84315db6
  parameters:
    - 'name': 'disable-jit'
      'values': 'value in ("yes", "no")'
  options:
    checks:
      - description: "Torch JIT is disabled by default, make a choice explicit with '--module-parameter=torch-disable-jit=yes|no'"
        support_info: 'parameter'
        when: 'standalone and get_parameter("disable-jit", None) is None'
  import-hacks:
    - force-environment-variables:
        'PYTORCH_JIT': '0'
      when: 'get_parameter("disable-jit", "no" if standalone else "yes") == "yes"'

So, what this does, is to make Nuitka accept parameters. The options part is designed to complain when the default value is used in standalone mode, kind of making the user acknowledge that it’s the intended value. For accelerated mode, we do not disable the JIT, since we can expect to be in the same environment with source code intact.

With get_parameter you get the option value, and can be conditional on it in the when block. That is the 3rd new thing.

The fourth new thing, is the forcing of environment variables. We have so far done this, including in plugins like tk-inter manually with post-load-code. The above is the same, effectively doing os.environ["PYTORCH_JIT"] = "0" if the JIT is to be disabled.

These changes are designed to avoid having to do plugins again. Historically for toga support, we should have created a new plugin, but now it’s not necessary anymore, since the Yaml mechanism can cover retrieval of compile time values from modules. And for torch and the JIT, a plugin would have been needed to provide the command line control for that decision.

This increased power of the Yaml will make it even less often the case that a plugin must be written. But of course docs will have to be added and maybe more places will need to work with variables to make that true even more often. More on that in the future.

Teasers

Future TWN will speak about Nuitka-Python (our own Python fork with incredible capabilities), about Nuitka-Watch (our way of making sure Nuitka works with PyPI packages and hot-fixes to not regress), about compilation reports as a new feature, Windows AV stuff, onefile improvements, and so on and so on. I got interesting stuff for many weeks. Limiting myself for now or I will never publish this.

Twitter and Mastodon

I should be more active there, although often I fail due to not wanting to talk about unfinished things, so actually I do not post there as much.

And lets not forget, having followers make me happy. So do re-tweets. Esp. those, please do them.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

All in with Nuitka

2022-12-09T08:22:42+01:00

After more than 10 years of working on the Python compiler Nuitka, and at the age of 50, I have finally decided to take the plunge. My projects “Nuitka” and “Nuitka commercial” are now my day job, and I quit my work in the Air Traffic Community, after working on ARTAS (a software to create an Air Situation Picture from radar and other sensors) for 20 years.

Why now?

That is very good question. I feel I could have done this a long time ago, and my heart was torn for a while, and also I prepared this for a while now. Listing back to interviews from 4 years ago, it seems though as if it had not been on the radar (pun intended).

So then I launched Nuitka commercial around 1.5 years ago, first as vapor ware, with early bird customers, and meanwhile it has matured to a point, where it is the obvious solution to many corporate users, with big guys picking it up and small business as well. From startups, big name technology giants, universities, mid sized companies, and from individuals, I have all kinds of support.

I liked my old job very much, but it was getting old, and I felt like Nuitka needs me to fulfil my mission. And my wife needed my help with driving her to work, which was getting the load to high, something had to give. And it seems nothing related to what is called Nuitka (my wife, and the compiler share the name) was going to be that.

What will change?

Already, ever since I launched Nuitka commercial, Nuitka has become way more of a out of the box experience. It just works, and covers more packages than ever before, and steps have been taken, to make it possible to extend this even further with your help. Sure there are still shortcoming.

But Nuitka has become very polished, and shows this very much that I also love details and doing things correctly. I have conducted Nuitka from day one as an attempt to create a software that would be usable to the enterprise with a million lines of code, that they cannot check for unsupported language features, so Nuitka is just fully compatible, to a level of absurdity.

Having conducted Nuitka on a level of where a corporate business can throw it’s code at Nuitka and have it just work, and reliably, this will only improve now.

Naturally bugs have happened once in a while, but I was always more careful. You know, like if this was for safety critical software, on which lives might depend. Hard to not be affected by that stance.

So lets talk about other ways I plan to expand.

Nuitka Package Configuration

Extending Nuitka with tweaks for package has become increasingly simpler. Many previous plugins, e.g. for torch, tensorflow, and most recently numpy, and definitely also the Qt binding ones in the future, have been or will be converted to Yaml configuration applied by generic plugins that deal with DLLs, data-files, tweaks (called anti-bloat) etc.

Adding stuff there is simple, and as of recently, there is even documentation for it. This is going to be explained in a series of posts detailing Nuitka Package Configuration, under the package_config tag.

I must say it’s so much fun, I love working with it. But the goal is for this to have you join it, and PRs are indeed starting to come in more frequently it seems from first glance, but I need to make this more known.

Streaming

I have experimented with streaming in the past, and I absolutely love it. Without my day job, effectively working two jobs at the same time, I expect to have the time to do more of them in December and then on a regular basis.

Python 3.11

There will be a separate post on this issue later this week. I am scrambling to add support for it and it feels very far already. Many basic tests already work, but more debugging will be needed. I hope Nuitka 1.3 will add support for it, at least on the usual experimental level.

How you can help?

Please become a subscriber of Nuitka commercial, even if you do not need the IP protection features it mostly has. All essential packaging and performance features are free, and I have put incredible amounts of works in this, and I need to now make a living off it.

Right now, I am far from being able to sustain my income from it, and I am taking a risk with this effort, hoping for support from you.

Nuitka commercial makes creating a Windows Service a blast. It has a PySide2 that is compatible with Nuitka (I did make PySide6 compatible myself, and backported it to PySide2), so if you need to stay on old software. And for IP protection, it is just incredibly important to not ship bytecode at all, and with Nuitka commercial, it protects your data and files at runtime too.

I am bad at marketing, but good at software, which is one more reason to purchase it. You know you love these kinds of people. Was that good marketing? Also, there are share buttons on this page. Please try them out and let people know about this.

In future postings, I will make it a regular thing to ask for help with some of the technical tasks. Stay tuned, there might be something that you can help with.

Becoming more active

I am the kind of person that would much rather do good stuff than to talk about it, like when it finally supports Python 3.11, normal old me would just mention it maybe on Twitter once, and have it in the changelog only.

I need to get away from that. I need to resume TWN (This week in Nuitka) and have all plans to do it. Obviously I can get caught up in things, and may just always do it later when there is more or better news.

Nuitka this week #13

2019-05-12T11:07:00+02:00

Communication vs. Coding

Communication was a lot more, just not these postings. Nuitka has ventured into important realms.

First, active developers have joined Nuitka. Second, as a consequence of the later, Nuitka indeed was able to participate with a insanely powerful offering of 4 mentors.

This meant, that a lot of technical or project management debt hat to be dealt with, and there was no time to make this kind of posting. This is probably good news on all fronts, except that I feel they are missing, and am glad to be able to resume them.

Google Summer of Code 2019

My Experience

This was the most crazy ride and fun. Became the admin of an active sub-organisation under the PSF umbrella for GSoC 2019. So shortly after the announcement of the project lists, students flooded in, and starting working on things right away as early as February.

They were working hard to prove themselves and get to know Nuitka, asking many questions, with us having lots of email interviews, and even a lot of video calls, where I personally talked to people about ideas.

This happened simultaneously to more experienced developers joining the project as well, making very important contributions.

This basically meant, that many days 90%-100% of my Nuitka time was for communication or working off technical debts I knew there were, or for preparing things, I want to have with multiple people working on the code base. This will be detailed in other sections.

Accepted Students

All of this has paid of in that we now have 2 really good students to work on 2 Nuitka projects over the summer. Let’s welcome Batakrishna and Tommy in separate posts. They will introduce themselves and what they are going to do shortly, in separate posts.

Lets just say, this is fantastic news. We had so many good applications and its a shame, but we knew right away, not everybody who deserved it could be picked. But this is going to be a good chance for us to get to be open and welcoming to new people.

Optimization Work

Core Stuff

So, the 0.6.3 release (btw. on Windows, be sure to use the 0.6.3.1 hotfix), which was made as a consolidation effort to get the good work of mostly other people out, didn’t contain much optimization work for the core, as that is still my thing.

However, this changed a lot. An idea that came to my mind for how to do the massive amounts of specialized helpers needed beyond + and +=, with which I had started for prior releases. And that is to use Jinja2 based templates for C, to generate the code.

This was an important idea. Took a while, but soon the manual code for + was already replaced with generated code, fixing a few bugs by the way, and from there, the generation was expanded to cover * as well.

Currently, support for the 3 (!) different kinds of additions (TrueDiv and FloorDir as well as Python2 default division, dubbed OldDiv in Nuitka was added along with - .

The reason, + and * were done first, is that they have special treatment for sequences, using sq_concat and sq_repeat, where the other operations will be more straightforward, e.g. nb_subtract (-) has a lot types supporting it and that makes those the easy cases.

I am saving a deeper explanation of 3 things we will need for the next time. Basically we need optimization of these things at compile time, and that is getting there, and code to use in the backend, and that is getting there, and a third thing, that is to use optimization knowledge to apply the special code as much as possible, and that is not yet fully there.

Faster Windows Dependencies

This is going to excite Windows users. After Orsiris de Jong had done a replacement for dependency walker that is faster, this had remained in an experimental status, just due to lack of time.

Recently however, I felt there is more time, after GSoC student selection has happened, and that I could finally work a bit on open issues like this. And when I wrote a dedicated tool, to analyse dependences with either technology to compare the results, I found that dendency walker founds a lot more things.

That was a turn down, but turns out, nothing it finds is stuff that should not be on the white list. In fact, it’s all core Windows things, and from the System32 folder. That made me question, why we take anything from there (except maybe PythonXY.dll) at all, and after that change the performance changed dramatically.

The dependency walker now finishes a file in milliseconds. Actually the pefile is now slow (surely it ought to be compiled), and takes some seconds, for a file. That is amazing, and has lead to me to remove the parallel usage, and since pefile allows for perfect caching, and is Free Software, we will probably keep it.

This will address a widespread complaint of many Windows users of the standalone mode. This is now a relatively unnoticable part of the overall experience.

Currently I need to finish off some remaining problems with it, before putting it out in the wild. Getting this into a release will solve many newcomer issues.

Nuitka Organisation

Esp. for Google Summer of Code, Nuitka has sought and found mentors, some of which are highly experienced for the task. I will let them decide and write their own introduction, but I feel really blessed by them helping me out in my desperate calls for help. Without them, neither could Nuitka participate, nor could it even overachieve as much as it does.

Therefore I welcomed Kamran and Vaibhav into the organisation and they are excited to work the the 2 accepted students, that are also added.

Plugin Work

On the jorj branch there is a lot of work from Jorj that aims at adding support for more of the beasty stuff with hidden dependency and plugin needs.

He is also working at run time tracing of your program to be translated to automatic imports of just that. I am going to highlight this later, once I manage to cherry-pick the release ready parts from it for the next release.

But this definitely awesome stuff, and going to make Nuitka very easy to use for some people, even with stranger software.

Opening Up

This is also a teaser. But we did so much work for the 0.6.3 release to make sure information is there, and things are accessible and changeable by everyone, or even the pre-commit hook that I am very proud of.

But I shall save this for next week, otherwise it will be too much new information.

Twitter

I continue to be active there, although often I fall prey to of not wanting to talk about unfinished things. On Twitter of all things.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #12

2019-02-08T14:09:00+01:00

Communication vs. Coding

Over the holiday season I sort of neglected these postings, but there are other fields, where I have put my focus, but I think these postings are now going to resume.

A quick update in a while in an eternally fluent situation, and not too polished.

Optimization Work

So, the 0.6.2 optimization work has been not a lot yet. I started some work on int/long C type and it looked good.

Week of Bugfixing

So I did this, and it turned out rather long. I continued fixing things and finishing up open things to the point that it now is clean. I hate to be falling behind. I am touching on a few more interesting topics.

Python flag -O and -OO

Nuitka was supporting -O but doing what should only the done for -OO which I only learned of now. So this is going to be supported both now.

Virtualenv vs. venv

Accelerated binaries were not running with full sys.path in the virtualenv, because site module uses sys.prefix and that was not propagated,but it now is.

Google Summer of Code for Nuitka

So the GSoC 2019 page is shaping up, the pull request to list Nuitka on the PSF project pages has been merged. More tweaking will be needed to get into perfect shape, and that has been my main communication time effort.

Finally somebody had stepped up for mentor, which means a lot to me. Now we need to see if students are willing to pick us.

And because I publish this too late. Already a bunch of stundents are showing interest and are hacking on Nuitka, which keeps me even more busy, and makes me extremely happy.

Nuitka Organisation

The Nuitka organisation on GitHub was introduced for my a while ago, and I had transferred the ownership of the Nuitka repository to there. Having moved the issue tracking to there, I was going more all in on it.

Recently more people have submitted PRs and with incredible quality and willingness to support it even after merge of their PR. No dry by contribution, but people looking to actually improve Nuitka together with me.

Therefore I welcome Orsiris and Jorj on board and am really happy about it.

Windows Dependencies from pefile

So Orsiris de Jong implemented a replacement for the dependency walker based code to scan DLLs for Windows standalone using the pefile module, which is likely way better at this. For starters it’s experimental in the next release, but I expect it to soon become the default.

Tkinter Plugin

And Jorj X. McKie implemented a Windows Tkinter plug-in that deals with copying of the TCL installation and integrating it with Nuitka standalone for distribution.

That is very nice and seems to affect a lot of people using that GUI style it seems.

Numpy Plugin

And Jorj X. McKie also implemented a NumPy plug-in that deals with the various acceleration methods there are, e.g. MKL, Blas, I have no idea. It copies the needed DLLs along and makes sure that numpy stays fast in standalone mode.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #11

2018-12-10T07:09:00+01:00

Communication vs. Coding

I continue to force myself to report more publicly, and it feels good. This time things are in a stablizing period, and I feel I have a consistent message.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Optimization Work

So, the 0.6.1 optimization work has been a lot. And it’s containing improvements on every level. I think I will detail the levels in another section.

Levels of Optimization

First level is of course node level optimization. Here 0.6.1 adds many things, from better handling of closure variables not all as unknown every time control flow escapes, to some operations + and comparisons on known built-in type shapes to now be able to statically tell that they do not raise. The opposite (does definitely raise) is prepared, but not yet used.

This allows for type shapes to be longer known. Now a+b+c can be known, but previously only a+b was sort of known, and little used information.

The next level is picking the C target type. Here seeing more operations and understanding more variables allows to more often pick the C bool or C void types over the PyObject * C type. For 0.6.1 I have observed that esp. more indicator variables make it to that stage, generating way more efficient C code (for that indicator variable) for those in many instances, esp. with loops, as these no longer loose type shape information as badly as they did.

The, another level is when it is treated as an object, but known to be int, there are way more helpers used for +/+= and a whole new set of them for comparisons, that in these cases of full or partial type knowledge operate faster.

And even if e.g. only one type is known, this still allows to not make a lot of tests about it, and to avoid attempted shortcuts that cannot work. For 0.6.1 the + and += are pretty well covered for these, but some variants are not yet tuned to take all type knowledge advantage.

These will be also the building block, once the C type layer picks types like “C int or PyObject * known to be int” with indicator flags which values are currently valid to use, then these specialized calls still make sense.

The most attrative level, “C int” has not been reached for 0.6.1 but for my loop example and Python3, I can say that now would be a nice time to start it, as type shape knowledge is all there. This was totally not the case for 0.6.0, but it seems that this step will have to be postponed to another release, maybe 0.6.2, maybe even later.

Week of Bugfixing

But something that bothers me is seeing the issue tracker pile up on actionable items, where I just have not taken action. So as announced on Twitter already, I am having and continue to have bug fixing time. I am acting on issues that are relatively old and easy to act on, or where I have no hope of this happening by anybody else anymore.

I have listed some interesting examples below. But basically these are small, relatively unimportant, yet somewhat import for some use cases things.

Exec on Filehandles

So when doing exec on a filehandle, Nuitka was at runtime reading the source, then compiling it, but forgetting about the filename. This makes things like inspect.getsource() fail on functions from there, and ugly tracebacks not pointing to the filename. This was one of the things which I had understood, but not did the actual work yet.

pkgutil.iter_modules

And another one, which seemed just not done, but turned out to be rather complex, this one needs to populate a sys.path_importer_cache for imported modules, and then to report the child modules. There was no object to carry that information, so now instances of the meta path based importer are associated for every import.

Turns out for Python3, my simplistic type building calling type manually here does not work, as __init__ and iter_modules do not become anything but static methods ever. Needs a real type.

Plus, I had to disable it for now, because mixed packages, like the one we do with multiprocessing where only part is compiled (the one required) and part is pure Python from disk still, stopped to work. The iter_modules it seems will have to cover that case too.

So no luck, postponing this until next week of bug fixes. Frustrating a bit, but such is life.

When to release

There are still some issues that I want to get to. Specifically the OpenGL plugins which has been research ever since, and nobody stepped up, but it’s rather trivial. And the Tcl/Tk for Windows. People have provided sufficient instructions for a plugin that I am going to write this week.

Once I feel the issue tracker is clean, I will release. As a matter of experience, it is then going to grow a lot again.

Google Summer of Code for Nuitka

Finally somebody has stepped up, which means a lot to me. Now to the actual work!

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #10

2018-11-21T06:06:00+01:00

Communication vs. Coding

Recently it was a bit more tough to make that decision. First, there was much going privately, with me ill, then child ill, and ill again, and myself, and that made me have a much harder time to communicate about incomplete things.

Even now, I am torn between fixing issues for 0.6.1 and doing this, but I know that it will take at least one week, so I am missing the point, if I wait for it more.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Hotfixes

There has been another hotfix, 0.6.0.6 and there ought to be one 0.6.0.7, at least on factory there is a bunch of stuff for it, but I didn’t actually do it yet. I was wandering between there will be a release anyway, and the feeling that some of the material may cause regressions, so I might skip on that really.

So for the most fixes, I suspect, develop is going to be the way until next week.

Google Summer of Code for Nuitka

Nobody has stepped up, which means it will not happen unfortunately. This would be your last chance to step up. I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Pythran and Nuitka

As suggested by @wuoulf (Wolf Vollprecht) we had a meeting at the side of the PyCon DE 2018 conference in Karlsruhe, abusing the C++ regular table as a forum for that, which was a very nice experience.

First of all, Wolf is so much more knowledgeable about AnaConda and could point out to me, very important stuff, not the least, that AnaConda contains its own compiler, which I have successfully used since, to first add easier installation instructions for Windows, and second, to successfully statically link with LTO on Linux amd64. Both of which are important for me.

But for Pythran which is limited Python, specialized to translate Numpy API to C++, we showed each other, Nuitka and Pythran details, and somehow in my mind a plan formed how Nuitka could use the Pythran tricks long term, and mid term, how it could include a plugin that will allow to integrate with Pythran compilation.

This was a huge success.

Performance Work

Adding specialized object operations

See last week, this has seen more completion. Both + and += are more or less covered for the selected subset. The CPython test suites were initially not finding uses, but with more and more optimization phase improvements, it challenges code generation with missing ones, and then I added them more and more.

Controlflow Descriptions

Shapes were added for the + and < operation so far, but didn’t influence anything else really but code generation, but of course they should also impact optimization phase.

So the query for type shape has been enhanced to return not only a type shape saying that int+float -> float, but also now an object that describes impact on control flow of the program. This can then say e.g. that this doesn’t execute arbitrary code, and that it does not modify input values, things used in the code generation to avoid error checks, and in the optimization to not have to mark things as unknown.

Preparations for comparison operations

So optimization now also has proper type shape functions for the < and the warnings when they fail to know what to do for concrete types. This allows to actually remove checks, but so far this wasn’t exposed for neither + or for <. Doing this eliminates the exception check for the operation part, where previously it was done if anything in the expression could raise.

Specializing the rich comparisons helper codes is the next step, but so far I didn’t quite get to it yet, but it has been started.

Comparison Conditions

Preparing < optimization for the loop, I noticed that not was optimized for in to become not in, and also is to become is not, etc. but for comparisons, where we can not the result is of bool shape, we can now also switch not < to >= and not = to != of course.

And since our reformulation of while a < b ends up having a statement like if not a < b: break as part of its re-formulation, that is again one step closer to optimizing my example loop.

Local variable escaping

Much to my shock, I noticed that the code which is responsible to handle escaping control flow (i.e. unknown code is executed), was not only doing what it was supposed to do, i.e. mark closure variables as unknown, but more or less did it for all local variables with Python3.

Fixing that allows for a lot more optimization obviously, and makes my test find missing ones, and even bugs in existing ones, that were previously hidden. A good thing to notice this regression (was better once), now that I am looking at concrete examples.

One noticeable sign was that more of my tests failed with warnings about missing code helpers. And another that in my while loop with int increase, it now seems as if Python3 is good. For Python2, the “int or long” shape will need dedicated helpers. That is because ìnt + int becomes either int or long there, where Python3 only has long but renamed it int.

Benchmarks Missing

Speedcenter got repaired, but I need to add the loop examples I am using as test cases before next release, so I can show what Nuitka 0.6.1 will have achieved or at least have improved somewhat already.

But currently these examples only serve as input for general improvements that then take a lot of time, and don’t have immediate impact on their own.

Still would be good to see where Nuitka is standing after each one.

Static Linking

So static linking works now, provided it’s not a pyenv crappy libpython.a but one that can actually work. I got this to work on Linux and using the Conda CC, even LTO will work with it. Interestingly then linking is noticely slow, and I bet ccache and the likes won’t help with that.

I am interested to see what this means for performance impact. But it will allow to address issues, where embedded CPython run time is plain slower than the one that lives in the python binary. For acceleration this is great news.

Conda CC

Using Conda CC by default as a fallback in –mingw mode on Windows is something that was easy to add. So when no other gcc is found, and MSVC is not tried in this mode, and the right directory is added to PATH automatically, with Anaconda, things should now be smoother. It has also its own libpython.a, not sure yet if it’s a static link library, that would be fantastic, but unlike standard MinGW64 we do not have to roll our own at least.

I will try with –lto eventually though and see what it does. But I think static linking on Windows is not supported by CPython, but I am not entirely sure of that.

Annotations Future Feature

Found a 3.7 feature that is not covered by the test suite, the __future__ flag annotations wasn’t working as expected. In this, strings are to be used for __annotations__ where they show up (many are ignored simply) and that requires an unparse function, going from parsed ast (presumably it’s still syntax checked) back to the string, but that was only very hard to get at, and with evil hackery.

For 3.8 a bug fix is promised that will give us the string immediately, but for now my hack must suffice.

MSI files

Following the 3.7.1 release, there are MSI files again, as the regression of 3.7.0 to build them has been fixed in that release. The MSI files will work with 3.7.0 also, just the building was broken.

Overall

So 0.6.1 is in still in full swing in terms of optimization. I think I need to make a release soon, simply because there is too much unreleased, but useful stuff already.

I might have to postpone my goal of C int performance for one example loop until next release. No harm in that. There already are plenty of performance improvements across the board.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #9

2018-10-20T06:17:00+02:00

Communication vs. Coding

My new communication strategy is a full success, engagement with Nuitka is on an all time high.

But the recent weeks more than ever highlighted why I have to force myself to do it. I do not like to talk about unfinished stuff. And right now, there is really a lot of it, almost only it. Also I was ill, and otherwise busy, so this is now late by a week.

But I am keeping it up, and will give an update, despite the feeling that it would be better to just finish a few of those things and then talk about it, but then it will take forever and leave you in the dark. And that is not what is supposed to be.

Bear in mind, that this is supposed to be a quick, not too polished, and straight from top of my head, even if really a lot of content. But I feel that esp. the optimization parts are worth reading.

Hotfixes

So the 0.6.0 release was a huge success, but it definitely wasn’t perfect, and hotfixes were necessary. The latest one 0.6.0.5 was done just yesterday and actually contains one for an important mis-optimization being done, and you ought to update to it from any prior 0.6.0 release.

There are also a few remaining compatibility issues fixed for 3.7 and generally using the latest hotfix is always a good idea.

Kind of what one has to expect from a 0 release, this one also had more expose than usual is seems.

Google Summer of Code for Nuitka

I need more people to work on Nuitka. One way of doing this could be to participate in Google Summer of Code under the Python umbrella. To make that possible, I need you to volunteer as a mentor. So please, please, do.

I know you will feel not qualified. But I just need a backup that will help a student around obstacles in case I go missing. Contact me and I will be very happy.

Website Overhaul

I updated the website to recent Nikola and dropped the tag cloud that I was using. Should have cleaner and better looks. Also integrated privacy aware sharing links, where two clicks are necessary to share a page or article like this one on Twitter, Facebook, etc.

Also the download page saw some structural updates and polishing. It should easier to overview now.

Performance Work

Adding specialized object operations

The feedback for performance and the work on 0.6.1 are fully ongoing, and there are many major points that are ongoing. I want to briefly cover each one of them now, but many of them will only have full effect, once everything is in place, which each one is very critical.

So, with the type tracing, objects have known types, and short of using a C type, knowing e.g. that an object is an int, and the other one too, doing + for them can take a lot of advantage avoiding unrelated checks and code paths, even if still using PyObject * at the end of the day.

And even we are only knowing it’s not an int, but say one value is a tuple and the other an unknown, that allows to remove checks for int shortcuts as they can no longer apply. These are tiny optimizations then, but still worthwhile.

To further this, first the inplace operations for a couple of more or less randomly selected types, list, tuple, int, long, str, unicode, bytes, and float, have been looked at and have gotten their own special object based helpers if one or both types are known to be of that kind.

Finding missing specialized object code generation

A report has been added, that will tell when such an operation could have been used, but was not available. This uncovered where typical stuff goes non optimized, a nice principle to see what is actually happening.

So adding list and str would now give a warning, although of course, the optimization phase ought to catch the static raise that is and never let it get there, so this report also addresses missing optimization in an earlier phase.

Optimizing plain object operations too

So the in-place operations were then covered, so this was extended to mere + operations too, the ones that are not in-place. Sometimes, esp. for immutable types, there was already code for that, e.g. int doesn’t really do it, in other cases, list + list code for a quicker concat was added.

And again a report for where it’s missing was added and basic coverage for most of the types. However, in some instances, the optimization doesn’t use the full knowledge yet. But where it does, it will shove off quite a few cycles.

Lack of type knowledge

To apply these things effectively, optimization and value tracing need to know types in the first place. I have found two obstacles for that. One are branch merges. If a branch or both assign to the same type or original type, well the type is changed. Previously it became “unknown” which is treated as object for code generation, and allows nothing really. But now that is better on develop now, and was actually a trivial missing thing.

The other area is loops. Loops put values to unknown when entering loop body, and again when leaving. Essentially making type tracing not effective where it is needed the most to achieve actual performance. Also this was limiting the knowledge for all function to one type to not happening for these kinds of variables that were assigned inside a loop at all.

Took me a while, but I figured out how to build type tracing for loops that works. It currently is still unfinished in my private repo, but passes all tests, I would just like to make it use dedicated interfaces, and clean it up.

I will most likely have that for 0.6.1 too and that should expand the cases where types are known in code generation by a fair amount.

The effect of that will be that more often C code generation will actually see types. Currently e.g. a boolean variable that is assigned in a loop, cannot use the C target type in code generation. Once loop code is merged, it will however take advantage there too. And only then I think adding “C int” as a C type makes sense at all.

Performance regressions vs. CPython

Then another area is performance regressions. So one thing I did early on in the 0.6.1 cycle was using the “module var C target type” to get in-place working for those too. Doing string concatenations on module variables could be slower by an order of magnitude, as could be other operations.

I still need to do it for closure variables too. Then Nuitka will do at least as many of them perfectly as CPython does. It also will be better at it them, because e.g. it doesn’t have to delete from the module dictionary first, due to it never taking a reference, and same applies to the cell. Should be faster for that too.

But strings in-place on these if not optimized, it will look very ugly in terms of worse performance, so 0.6.0 was still pretty bad for some users. This will however hopefully be addressed in 0.6.1 then.

In-place unicode still being bad

Another field was in-place string add for the already optimized case, it was still slower than CPython, and I finally found out what causes this. And that is the using of libpython where PyUnicode_Append is far worse than in the python binary that you normally use, I have see that at least for 3.5 and higher CPython. Analysis showed that e.g. MiniConda had the issue to a much smaller extent, and was being much faster anyway, but probably just has better libpython compilation flags.

So what to do. Ultimately that was to be solved by including a clone of that function, dubbed UNICODE_APPEND that behaves the same, and can even shove off a couple of cycles, by indicating the Python error status without extra checks, and specializing it for the pure unicode += unicode case that we see most often, same for UNICODE_CONCAT for mere +.

Right now the benchmarks to show it do not exist yet. Again something that typically wants me to delay stuff. But as you can imagine, tracking down these hard issues, writing that much code to replace the unicode resizing, is hard enough by itself.

But I hope to convince myself that this will allow to show that for compiled code, things are going to be faster only now.

Benchmarks Missing

In fact, speedcenter as a whole is currently broken, mostly due to Nikola changes that I am trying to work around, but it will take more time apparently and isn’t finished as I write this.

Type shapes in optimization

Another optimization end, is the type shapes of the + operation itself. Right now what is being done is that the shape is derived from the shape of the left argument with the right shape to be considered by it. These also have reports now, for cases where they are missing. So saying e.g. that int + float results in float and these kinds of things, are stuff being encoded there right now.

This is necessary step to e.g. know that int + int -> int_or_long, to make effective loop variable optimization.

Without these, and again, that is a lot of code to write, there is no way to hope for wide spread type knowledge in code generation.

Control flow escape

Something missing there, is to also make it known that + unlike it currently is now, should not in all cases lead to “control flow escape” with the consequence of removing all stuff, and expecting an exception possible, but instead to let the int type also make known that + int ont it not only gives an int_or_long result shape, but also while doing so, that it will never raise an exception (bare MemoryError), and therefore allow more optimization to happen and less and therefore faster code generated.

Until this is done, what is actually going to happen is that while the + result is known, Nuitka will assume control flow escape.

And speaking of that, I think this puts too many variables to a too unknown state. You can to distrust all values, but not the types in this case, so that could be better, but right now it is not. Something else to look into.

Overall

So 0.6.1 is in full swing in terms of optimization. All these ends need a completion, and then I can expect to use advantage of things in a loop, and ultimately to generate C performance code for one example of loop. esp. if we add a C int target type, which currently isn’t yet started, because I think it would barely be used yet.

But we are getting there and I wouldn’t even say we are making small steps, this is all just work to be completed, nothing fundamental about it. But it may take more than one release for sure.

Mind you, there is not only +, there is also -, *, %, and many more operators, all of them will require work. Granted, loop variables tend to use + more often, but any un-optimized operation will immediately loose a lot of type knowledge.

Improved Annotations

There are two kinds of annotations, ones for classes and modules, which actually are stored in a __annotations__ variable, and everything else is mostly just ignored.

So Nuitka got the criterion wrong, and did one thing for functions, and the other for everything else. So that annotations in generators, coroutines and asyncgen ended up with wrong, crashing, and slower code, due to it updating the module __annotations__, so that one is important too if you have to do those.

Release or not

To release or not. There is at least one bug about star imports that affects numpy that is solved in develop, and wasn’t back ported, and I was thinking it only applies to develop, but in fact does to stable. It makes me want to release even before all these optimization things happen and are polished, and I might well decide to go with that.

Maybe I only add the closure in-place stuff and the polish the loop SSA stuff, and then call it a release. It already will solve a lot of performance issues that exist right now, while staging the ground for more.

Standalone Improvements

Standalone work is also improving. Using pyi files got more apt, and a few things were added, all of which make sense to be used by people.

But I also have a backlog of issues there however. I will schedule one sprint for those I guess, where I focus on these. I am neglecting those somewhat recently.

Caching Examined

For the static code, I now noticed that it’s compiled for each target name, due to the build directory being part of the object file for debug. For gcc 8 there is an option to allow pointing at the original static C file location, and then ccache is more effective, because object files will be the same.

That’s actually pretty bad, as most of my machines are on gcc-6 and makes me think that libnuitka.a is really more of an requirement than ever. I might take some time to get this sorted out.

Python3 deprecation warnings

So Nuitka supports the no_warnings Python flag, and for a long time I have been annoyed at how it was not working for Python3 in some cases. The code was manually settign filters, but these would get overridden by CPython test suites testing warnings. And the code said that there is no CPython C-API to control it, which is just plain wrong.

So I changed that and it became possible to remove lots of ignore_stderr annotations in CPython test suites, and more importantly, I can stop adding them for when running older/newer CPython version with a suite.

Twitter

I continue to be very active there.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #8

2018-10-02T06:05:00+02:00

Public / Private CI / Workflow

Note

I wrote this as part of a discussion recently, and I think it makes sense to share it here. This is a lot text though, feel free to skip forward.

Indeed I have a private repo, where I push and only private CI picks up. Based on Buildbot, I run many more compilations, basically around the clock on all of my computers, to find regressions from new optimization or codegen changes, and well UI changes too.

Public CI offerings like Travis are not aimed at allowing this many compilations. It will be a while before public cloud infrastructure will be donated to Nuitka, although I see it happening some time in the future. This leaves developers with the burden to run tests on their own hardware, and never enough. Casual contributors will never be able to do it themselves.

My scope is running the CPython test suites on Windows and Linux. These are the adapted 26, 27, 32, 33, 34, 35, 36, 37 suites, and also to get even more errors covered, they are ran with mismatching Python versions, so a lot of exceptions are raised. Often running the 36 tests with 37 and vice versa will extend the coverage, because of the exceptions being raise.

On Windows I compile with and without debug mode, x86 and x64, and it’s kind of getting too much. For Linux I have 2 laptops in use, and an ARM CuBox bought from your donations, there it’s working better, esp. due to ccache being used everywhere, although recent investigations show room for improvement there as well.

For memory usage I still compile mercurial and observe the memory it used in addition to comparing the mercurial tests to expected outputs its test suite gives. It’s a sad day when Mercurial tests find changes in behavior, and luckily that has been rare. Running the Mercurial test suite gives some confidence in the thing not corrupting data it works with without knowing.

Caching the CPython outputs of tests to compare against is something I am going to make operational these days, trying to make things ever faster. There is no point to re-run tests with Python, just to get at its output, which will typically not change at all.

But for the time being, ccache.exe and clcache.exe seem to have done wonders for Windows too, but I will want to investigate some more to avoid unnecessary cache misses.

Workflow

As for my workflow with Nuitka, I often tend to let some commits settle in my private repo only until they become trusted. Other times I will make bigger changes and put them out to factory immediately, because it will be hard to split up the changes later, so putting them out makes it easier.

I am more conservative with factory right after telling people to try something there. But also I break it on purpose, just trying out something. I really consider it a private branch for interacting with me or public CI. I do not recommend to use it, and it’s like a permanent pull request of mine that is not ever going to be finished.

Then on occasions I am making a sorting of all commits on factory and split it into some things that become hotfixes, some things that become current pre-release, and other things that will remain in that proving ground. That is why I typically make hotfix and pre-release at the same times. The git flow suggests doing that and it’s easy, so why not. As a bonus, develop is then practically stable at nearly all times too, with hardly any regressions.

I do however normally not take things as hotfixes that are on develop already, I hate the duplication of commits. Hotfixes must be small and risk free, and easy to put out, when there is any risk, it definitely will be on develop. Nuitka stable typically covers nearly all grounds already. No panic needed to add missing stuff and break others.

Hunting bugs with bisect

For me the git bisect is very important. My private commit history is basically a total mess and worthless, but on factory I am making very nice organized commits that I will frequently amend, even for the random PyLint cleanup. This allows me when e.g. one test suddenly says “segfault” on Windows to easily find the change that triggers it, look at C code difference, and spot the bug introduced, then amend the commit and be done with it.

It’s amazing how much time this can save. My goal is to always have a workable state which is supposed to pass all tests. Obviously I cannot prove it for every commit, but when I know it to not be the case, I tend to make rebases. At times I have been tempted and followed up on backward amending develop and even stable.

I am doing that to be sure to have that bisect ability, but fortunately it’s rare that kind of bug occurs, and I try not to do it.

Experimental Changes

As with recent changes, I sometimes make changes with the isExperimental() marker, activating breaking changes only gradually. The C bool type code generation has been there for months in a barely useful form, until it became more polished, and always guarded with a switch, until one day for 0.6 finally I changed it, and made the necessary fixes retroactively before that switch, to make it work while that was still in factory.

Then I will remove the experimental code. I feel it’s very important and even ideal to be able to always compare outputs to a fully working solution. I am willing to postpone some cleanups until later date as a price, but when then something in my mind tells me again “This cannot possibly have ever worked”… a command line flag away, I have the answer to compare, plus, that includes extra changes happened in the meantime, they don’t add noise to diff outputs of generated C code for example.

Then looking at that diff, I can tell where the unwanted effect is, and fix all the things, and that way find bugs much faster.

Even better, if I decide to make a cleanup action as part of making a change more viable to execute, then I get to execute it on stable grounds, covered by the full test suite. I can complete that cleanup, e.g. using variable identifier objects instead of mere strings was needed to make “heap generators” more workable. But I was able to put that one to active before “heap generators” was ever fully workable, and complete it, and actually reap some of its benefits already.

Hardware

Obviously this takes a lot of hardware and CPU to be able to compile this much Python code on a regular basis. And I really wish I could add one of the new AMD Threadripper 2 to the mix. Anybody donating one to me? Yes I know, I am only dreaming. But it would really help the cause.

Milestone Release

So the 0.6 is out, and already a hotfix that addresses mostly use cases of people that didn’t work. More people seemed to have tried out 0.6.0 and as a result 0.6.0.1 is going to cover a few corner cases. So far I have not encountered a single regression of 0.6.0, but instead it contained ones for 0.5.33 which did have one that was not easy to fix.

So that went really smooth.

UI rework

The UI needs more work still. Specifically that packages do not automatically include all stuff below them and have to be specified by file path instead of by name, is really annoying to me.

But I had delayed 0.6 for some UI work, and the quirks are to remain some. I will work on these things eventually.

Benchmarks

So I updated the website to state that PyStone is now 312% faster, from a number that was very old. I since then ran it with an updated version for Python3, and it’s much less there. That is pretty sad.

I will be looking into that for 0.6.1 release, or I will have to update the wording to provide 2 numbers there, because it seems for Python3 performance with Nuitka it might be misleading.

Something with unicode strings and in-place operations is driving me crazy. Nuitka is apparently slower for that, and I can’t point where that is happening exactly. It seems internally unicode objects are maybe put into a different state from some operations, which then making in-place extending in realloc fail more often, but I cannot know yet.

Inplace Operations

So more work has been put into those, adding more specialization, and esp. also applying them for module variables as well. CPython can do that, and actually is giving itself a hard time about it, and Nuitka should be doing this much clever with its more static knowledge.

But I cannot tell you how much scratching my head was wasted debugging that. I was totally stupid about how I approached that, looking from the final solution, it was always easy. Just not for me apparently.

New use cases

Talked about those above. So the top level logging module of your own was working fine in accelerated mode, but for standalone it failed and used the one from standard library instead. That kind of shadowing happened because Nuitka was going from module objects to their names and back to objects, which are bad in case of duplicates. That is fixed for develop, and one of those risk cases, where it cannot be a hotfix because it touched too much.

Then pure Python3 packages need not have __init__.py and so far that was best working for sub-packages, but after 0.6.0.1 hotfix, now it will also work for the main module you compile to be that empty.

Tcl/Tk Standalone

So instructions have been provided how to properly make that work for Python standalone on Windows. I have yet to live up to my promise and make Nuitka automatically include the necessary files. I hope to do it for 0.6.1 though.

Caching Examined

So I am looking at ccache on Linux right now, and found e.g. that it was reporting that gcc --version was called a lot at startup of Scons and then g++ --version once. The later is particularly stupid, because we are not going to use g++ normally, except if gcc is really old and does not support C11. So in case a good one was found, lets disable that version query and not do it.

And for the gcc version output, monkey patching scons to a version of getting that output that caches the result, removes those unnecessary forks.

So ccache is being called less frequently, and actually these --version outputs appears to actually take measurable time. It’s not dramatic, but ccache was apparently getting locks, and that’s worth avoiding by itself.

That said, the goal is for ccache and clcache to make them both report their effectiveness of cache usage after the end of a test suite run. That way I am hoping to notice and be able to know, if caching is used to its full effect.

Twitter

I continue to be very active there. I put out a poll about the comment system, and disabling Disqus comments as a result, I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Adding Twitter more prominently to the web site is something that is also going to happen.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Plans

Working on the 0.6.1 release, attacking more in-place add operations as a first goal, and now turning to binary operations, I am trying to shape how using different helper functions to different object types looks like. And to gain performance without C types. But ultimately the same issue will arise there, what to do with mixed input types.

My desire is for in-place operations to fully catch up with CPython, as these can easily loose a lot of performance. Closure variables and their cells are another target to pick on, and I feel they ought to be next after module ones are now working, because also their solution ought to be very similar. Then showing that depending on target storage, local, closure, or module, is then faster in all cases would be a goal for the 0.6.1 release.

This feels not too far away, but we will see. I am considering next weekend for release.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #7

2018-09-22T11:05:00+02:00

Nuitka Design Philosophy

Note

I wrote this as part of a discussion recently, and I think it makes sense to share my take on Nuitka and design. This is a lot text though, feel free to skip forward.

The issue with Nuitka and design mainly for me is that the requirements for many parts were and are largely unknown to me, until I actually start to do it.

My goto generators approach worked out as originally designed, and that felt really cool for once, but the whole “C type” thing was a total unknown to me, until it all magically took form.

But rather I know it will evolve further if I go from “bool” (complete and coming for 0.6.0) via “void” (should be complete already, but enabling will happen only for 0.6.1 likely) to “int”, not sure how long that will take.

I really think Nuitka, unlike other software that I have designed, is more of a prototype project that gradually turns more and more into the real thing.

I have literally spent years to inject proper design in steps into the optimization phase, what I call SSA, value tracing, and it is very much there now. I am probably going to spend similar amounts of time, to execute on applying type inference results to the code generation.

So I turned that into something working with code strings to something working with variable declaration objects knowing their type for the goto generators, aiming at C types generally. All the while carrying the full weight of passing every compatibility test there is.

Then e.g. suddenly cleaning up module variables to no longer have their special branch, but a pseudo C type, that makes them like everything else. Great. But when I first introduced the new thing, I postponed that, because I could sooner apply its benefits to some things and get experience from it.

While doing partial solutions, the design sometimes horribly degrades, but only until some features can carry the full weight, and/or have been explored to have their final form.

Making a whole Nuitka design upfront and then executing it, would instead give a very high probability of failing in the real world. I am therefore applying the more agile approach, where I make things work first. And then continue to work while I clean it up.

For every feature I added, I actively go out, and change the thing, that made it hard or even fail. Always. I think Nuitka is largely developed by cleanups and refactoring. Goto generators were a fine example of that, solving many of the issues by injecting variable declarations objects into code generation, made it easy to indicate storage (heap or object or stack) right there.

That is not to say that Nuitka didn’t have the typical compiler design. Like parsing inputs, optimizing a tree internally, producing outputs. But that grand top level design only tells you the obvious things really and is stolen anyway from knowing similar projects like gcc.

There always were of course obvious designs for Nuitka, but that really never was what anybody would consider to make a Python compiler hard. But for actual compatibility of CPython, so many details were going to require examination with no solutions known ahead of time.

I guess, I am an extreme programmer, or agile, or however they call it these days. At least for Nuitka. In my professional life, I have designed software for ATC on the drawing board, then in paper, and then in code, the design just worked, and got operational right after completion, which is rare I can tell you.

But maybe that is what keeps me exciting about Nuitka. How I need to go beyond my abilities and stable ground to achieve it.

But the complexity of Nuitka is so dramatically higher than anything I ever did. It is doing a complicated, i.e. detail rich work, and then it also is doing hard jobs where many things have to play together. And the wish to have something working before it is completed, if it ever is, makes things very different from projects I typically did.

So the first version of Nuitka already had a use, and when I publicly showed it first, was capable of handling most complex programs, and the desire was to evolve gradually.

I think I have described this elsewhere, but for large parts of the well or bad designed solutions of Nuitka, there is reliable ways of demonstrating it works correctly. Far better than I have ever encountered. i believe it’s the main reason I managed to get this off the ground is that. Having a test “oracle” is what makes Nuitka special, i.e. comparing to existing implementations.

Like a calculator can be tested comparing it to one of the many already perfect ones out there. That again makes Nuitka relatively easy despite the many details to get right, there is often an easy way to tell correct from wrong.

So for me, Nuitka is on the design level, something that goes through many iterations, discovery, prototyping, and is actually really exciting in that.

Compilers typically are boring. But for Nuitka that is totally not the case, because Python is not made for it. Well, that*s technically untrue, lets say not for optimizing compilers, not for type inference, etc.

UI rework

Following up on discussion on the mailing list, the user interface of Nuitka will become more clear with --include-* options and --[no]follow-import* options that better express what is going to happen.

Also the default for following with extension modules is now precisely what you say, as going beyond what you intend to deliver makes no sense in the normal case.

Goto Generators

Now release as 0.5.33 and there has been little regressions so far, but the one found is only in the pre-release of 0.6.0 so use that instead if you encounter a C compilation error.

Benchmarks

The performance regressions fixed for 0.6.0 impact pystone by a lot, loops were slower, so were subscripts with constant integer indexes. It is a pity these were introduced in previous releases during refactorings without noticing.

We should strive to have benchmarks with trends. Right now Nuitka speedcenter cannot do it. Focus should definitely go to this. Like I said, after 0.6.0 release, this will be a priority, to make them more useful.

Twitter

I continue to be active there. I just put out a poll about the comment system, and disabling Disqus comments I will focus on Twitter for web site comments too now.

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Nuitka definitely needs more people to work on it.

Egg files in PYTHONPATH

This is a relatively old issue that now got addressed. Basically these should be loaded from for compilation. Nuitka now unpacks them to a cache folder so it can read source code from them, so this apparently rare use case works now, yet again improving compatibility.

Will be there for 0.6.0 release.

Certifi

Seems request module sometimes uses that. Nuitka now includes that data file starting with 0.6.0 release.

Compatibility with pkg_resources

It seems that getting “distributions” and taking versions from there, is really a thing, and Nuitka fails pkg_resources requirement checks in standalone mode at least, and that is of course sad.

I am currently researching how to fix that, not sure yet how to do it. But some forms of Python installs are apparently very affected by it. I try looking into its data gathering, maybe compiled modules can be registered there too. It seems to be based on file system scans of its own makings, but there is always a monkey patch possible to make it better.

Plans

Still working on the 0.6.0 release, cleaning up open ends only. Release tests seem to be pretty good looking. The UI changes and stuff are a good time to be done now, but delay things, and there is a bunch of small things that are low hanging fruits while I wait for test results.

But since it fixes so many performance things, it really ought to be out any day now.

Also the in-place operations stuff, I added it to 0.6.0 too, just because it feels very nice, and improves some operations by a lot too. Initially I had made a cut for 0.6.1 already, but that is no more.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #6

2018-09-12T12:05:00+02:00

Holiday

In my 2 weeks holiday, I indeed focused on a really big thing, and got more done that I had hoped for. For C types, nuitka_bool, which is a tri-state boolean with true, false and unassigned, can be used for some variables, and executes some operations without going through objects anymore.

bool

Condition codes are no longer special. They all need a boolean value from the expression used as a condition, and there was a special paths for some popular expressions for conditions, but of course not all. That is now a universal thing, conditional statement/expressions will now simply ask to provide a temp variable of value nuitka_bool and then code generation handles it.

For where it is used, code gets a lot lighter, and of course faster, although I didn’t measure it yet. Going to Py_True/Py_False and comparing with it, wasn’t that optimal, and it’s nice this is now so much cleaner as a side effect of that C bool work.

This seems to be so good, that actually it’s the default for this to be used in 0.6.0, and that itself is a major break through. Not so much for actual performance, but for structure. Other C types are going to follow soon and will give massive performance gains.

void

And what was really good, is that not only did I get bool to work almost perfectly, I also started work on the void C target type and finished that after my return from holiday last weekend, which lead to new optimization that I am putting in the 0.5.33 release that is coming soon, even before the void code generation is out.

The void C type cannot read values back, and unused values should not be used, so this gives errors for cases where that becomes obvious.

a or b

Consider this expression. The or expression, that one is going to producing a value, which is then released, but not used otherwise. New optimzation creates a conditional statement out of it, which takes a as the condition and if not true, then evaluates b but ignores it.

if not a:
    b

The void evaluation of b can then do further optimization for it.

Void code generation can therefore highlight missed opportunities for this kid of optimization, and found a couple of these. That is why I was going for it, and I feel it pays off. Code generation checking optimization here, is a really nice synergy between the two.

Plus I got all the tests to work with it, and solved the missing optimizations it found very easily. And instead of allocating an object now, not assigning is often creating more obvious code. And that too allowed me to find a couple of bugs by C compiler warnings.

Obviously I will want to run a compile all the world test before making it the default, which is why this will probably become part of 0.6.1 to be the default.

module_var

Previously variable codes were making a hard distinction for module variables and make them use their own helper codes. Now this is encapsulated in a normal C type class like nuitka_bool, or the one for PyObject * variables, and integrates smoothly, and even got better. A sign things are going smooth.

Goto Generators

Still not released. I delayed it after my holiday, and due to the heap generator change, after stabilizing the C types work, I want to first finish a tests/library/compile_python_module.py resume run, which will for a Anaconda3 compile all the code found in there.

Right now it’s still doing that, and even found a few bugs. The heap storage can still cause issues, as can changes to cloning nodes, which happens for try nodes and their finally blocks.

This should finish these days. I looked at performance numbers and found that develop is indeed only faster, and factory due to even more optimization will be yet faster, and often noteworthy.

Benchmarks

The Speedcenter of Nuitka is what I use right now, but it’s only showing the state of 3 branches and compared to CPython, not as much historical information. Also the organization of tests is poor. At least there is tags for what improved.

After release of Nuitka 0.6.0 I will show more numbers, and I will start to focus on making it easier to understand. Therefore no link right now, google if you are so keen. ;-)

Twitter

During the holiday sprint, and even after, I am going to Tweet a lot about what is going on for Nuitka. So follow me on twitter if you like, I will post important stuff as it happens there:

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Poll on Executable Names

So I put e.g. poll up on Twitter, which is now over. But it made me implement a new scheme, due to popular consensus

Hotfixes

Even more hotfixes. I even did 2 during my holiday, however packages built only later.

Threaded imports on 3.4 or higher of modules were not using the locking they should use. Multiprocessing on Windows with Python3 had even more problems, and the --include-package and --include-module were present, but not working.

That last one was actually very strange. I had added a new option group for them, but not added it to the parser. Result: Option works. Just does not show up in help output. Really?

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very like one you can help with.

Nuitka definitely needs more people to work on it.

Plans

Working down the release backlog. Things should be out. I am already working on what should become 0.6.1, but it’s not yet 0.5.33 released. Not a big deal, but 0.6.0 has 2 really important fixes for performance regressions that have happened in the past. One is for loops, making that faster is probably like the most important one. The other for constant indexing, probably also very important. Very much measurable in pystone at least.

In the mean time, I am preparing to get int working as a target C type, so e.g. comparisons of such values could be done in pure C, or relatively pure C.

Also, I noticed that e.g. in-place operations can be way more optimized and did stuff for 0.6.1 already in this domain. That is unrelated to C type work, but kind of follows a similar route maybe. How to compare mixed types we know of, or one type only. That kind of things needs ideas and experiments.

Having int supported should help getting some functions to C speeds, or at least much closer to it. That will make noticeable effects in many of the benchmarks. More C types will then follow one by one.

Donations

If you want to help, but cannot spend the time, please consider to donate to Nuitka, and go here:

Donate to Nuitka

Nuitka this week #5

2018-08-24T12:55:00+02:00

Goto Generators

Finished. Done. Finally.

Benchmarking was exciting. One program benchmark I had run in the past, was twice as fast than before, showing that the new implementation is indeed much faster, which is fantastic news.

Creating generator expressions and using them both got substantially faster and that is great.

It took me a fair amount of time to debug coroutines and asyncgen based on the new goto implementation. But the result is really good, and a fair amount of old bugs have been fixed. There always had been a segfault with asyncgen test that now has been eradicated.

One major observation is now, with only one C stack, debugging got a lot easier before, where context switches left much of the program state not reachable.

Benchmarks

Posted this one Twitter already:

Nuitka Speedcenter Builtin sum with generator

That one construct test has been a problem child, where Nuitka was slower than CPython 2.x, and very little faster than 3.x, and now with goto generators finally has become consistently faster.

I will explain what you see there in the next issue. The short version is that there is code, in which for one run, one line is used, and in another the other line is used, and then the “construct” is measure that way, by making the delta of the two. That construct performance is then compared between Python and Nuitka.

So if e.g. Nuitka is already better at looping, that won’t influence the number of making that sum call with a generator expression.

The alternative line uses the generator expression, to make sure the construction time is not counted. To measure that, there is another construct test, that just creates it.

Nuitka Speedcenter Generator Expression Creation

This one shows that stable Nuitka was already faster at creating them, but that the develop version got even faster again. As creating generator objects became more lightweight, that is also news.

There are constructs for many parts of Python, to shed a light on how Nuitka fares for that particular one.

Holiday

In my 2 weeks holiday, I will try and focus on the next big thing, C types, something also started in the past, and where recent changes as part of the heap storage, should make it really a lot easier to get it finished. In fact I don’t know right now, why my bool experimental work shouldn’t just prove to be workable.

I am not going to post a TWN issue next week, mostly because my home servers won’t be running, and the static site is rendered on one of them. Of course that would be movable, but I won’t bother.

I am going to post a lot on Twitter though.

Static Compilation

There is a GitHub issue where I describe how pyenv on macOS ought to be possible to use, and indeed, a brave soul has confirmed and even provided the concrete commands. All it takes now is somebody to fit this into the existing caching mechanism of Nuitka and to make sure the static library is properly patched to work with these commands.

Now is anyone of you going to create the code that will solve it for good?

Twitter

Follow me on twitter if you like, I will post important stuff as it happens there:

Follow @kayhayen

And lets not forget, having followers make me happy. So do re-tweets.

Hotfixes

And there have been yet again more hotfixes. Some are about coroutine and asyncgen corruptions for closes of frames. Multiprocessing plugin on Windows will work in all cases now.

Noteworthy was that the “0.5.32.6” was having a git merge problem on the cherry-pick that git didn’t tell me about, leading to crashes. That made it necessary to push an update right after. I was confused that I didn’t get a conflict, because there was one. But I am to blame for not checking the actual diff.

Bug Tracker

The next release will make GitHub the official tracker for Nuitka issues. I am working down the issues on the old tracker. The web site already pointed users there for a while, and I was set on this for some time, but yesterday I focused on taking action.

Basically what won me over is the easier templating of issues and pull requests that would have been possible with Roundup, but never happened. Also the OpenID integration that bugs.python.org has, never became available to me in a ready usable form.

Issue Backlog

Finishing goto “generators allowed” for around 10 issues to be closed alone, and I went over things, and checked out some stale issues, to see if they are dealt with, or pinging authors. I spent like half a day on this, bring down the issue count by a lot. Tedious work, but must be done too.

Also my inbox got a fair amount of cleanup, lots of issues pile up there, and from time to time, I do this, to get things straight. I raised issues for 2 things, that I won’t be doing immediately.

But actually as issues go, there really very little problematic stuff open right now, and nothing important really. I would almost call it issue clean.

Help Wanted

If you are interested, I am tagging issues help wanted and there is a bunch, and very like one you can help with.

Nuitka definitely needs more people to work on it.

Plans

The goto generator work could be released, but I want to make the compile all the world test before I do so. It is running right now, but I will not complete before I leave. Also I do not want to get regression reports in my holiday, and goto generators along with heap storage, mean there could be some.

I am going to work on C types now. There is a few closing down actions on what I observed doing goto generators. There are a few easy ways to get even slightly better performance, definitely smaller code out of generators. Not sure if I go there first, or for the C types work directly. I often like to get these kind of observations dealt with more immediately, but I don’t want to spend too much quality time on it.

Donations

As I have been asked this, yes, you can donate to Nuitka if you wish to further its development. Go here:

Donate to Nuitka

Nuitka this week #4

2018-08-17T14:55:00+02:00

Goto Generators

This continues TWN #3 where I explained what is all about.

Good news is, at the time Python2 generators were largely working with the new ways, in the mean time not only did all of the Python 2.7 test suite pass with goto generators, also did the Python 3.4 test suite, i.e. also the yield from is working with it.

The way it was done is to set m_yieldfrom in generators, and then to enter a state, where the code will only be resumed, when that sub-generator that currently it is yielding from, is finished. That makes it very much like normal yield. In fact, code generation is hardly different there.

Since the whole purpose is to get rid of make/get/setcontext, the next stop is coroutines. They have async for, async with and await but at the end of the day, the implementation comes down to yield from really with only a lot of sugar applied.

Right now, I am debugging “goto coroutines”. It’s hard to tell when it will be finished, and then asyncgen will be waiting still.

This is easily the largest change in a long time, esp. due to the heap storage changes that I already discussed. One this is finished, I expect to turn towards C types with relative ease.

Tox Plugin

Anthony Shaw took on Tox and Nuitka and created a plugin that allows using Nuitka. I am still wrapping my head around these things. It’s only a proof of concept yet. I will give it more coverage in the future.

Twitter

Follow me on twitter if you like, I will:

Follow @kayhayen

Hotfixes

So there have even more hotfixes. One addresses memory leaks found with the yield from while I was adding tests. Usually if I encounter an old issue that has a small fix, that is what I do, push out a hotfix using the git flow model. Also nested namespace packages for Python3, those are the ones without a __init__.py were not working after the original directory was removed, and that got fixed.

And right now, I have hotfixes for frames close method, which apparently was never updated to work properly for coroutines and asyncgen. That is going to be in the next hotfix.

Plans

So the heap storage seems pretty complete now, and goto generators are on the final stretch. As always, things feel right around the corner. But it’s unclear how much longer I will have to debug. I am pretty sure the bare work of doing asyncgen is going to be low. Debugging that too then, that is the hard part.

A new release seems justified, but I kind of do not want to make it without that major new code used. Because apparently during the debugging, I tend to find issues that need hotfixes, so I will wait for the goto generator work to finish.

Nuitka this week #3

2018-08-11T11:58:00+02:00

New Series Rationale

This is working out well so far. I think driving more attention at the things that are going on can only be good. Also to explain will always help. It also kind of motivates me a bit.

Twitter

Also as part of my communications offensive, I am using my Twitter account more regularly. I used to highlight important fixes, or occasionally releases of some importance there. I will continue to do only important stuff there, but with more regularity.

And I noticed in the past, even when I do not post, followers makes me happy. So here you go:

Follow @kayhayen

Goto Generators

This continues TWN #2 where I promised to speak more of it, and this is the main focus of my work on Nuitka right now.

Brief summary, context switches were how this was initially implemented. The main reason being that for C++ there never was going to be a way to save and restore state in the middle of an expression that involves constructors and destructors.

Fast forward some years, and C-ish entered the picture. No objects are used anymore, and Nuitka is purely C11 now, which has convenience of C++, but no objects. Instead goto is used a lot already. So every time an exception occurs, a goto is done, every time a branch is done, a loop exit or continue, you get it, another goto.

But so far, all Python level variables of a frame live on that C stack still, and the context switch is done with functions that swap stack. That is fast, but the imporant drawback is that it takes more memory. How deep of a stack will we need? And we can use really many, if you imagine a pool of 1000 coroutines, that quickly become impossible to deal with.

So, the new way of doing this basically goes like this:

def g():
    yield 1
    yield 2

This was some far becoming something along this lines:

PyObject *impl_g( NuitkaGenerator *generator )
{
     YIELD( const_int_1 );
     YIELD( const_int_2 );

     PyErr_SetException( StopIteration );
     return NULL;
}

The YIELD in there was basically doing the switching of the stacks and for the C code, it looked like a normal function call.

In the new approach, this is done:

PyObject *impl_g( NuitkaGenerator *generator )
{
     switch( generator->m_resume_point )
     {
          case 1: goto resume_1;
          case 2: goto resume_2;
     }

     generator->m_yielded = const_int_1;
     generator->resume_point = 1
     return NULL;
     resume_1:

     generator->m_yielded = const_int_2;
     generator->resume_point = 2
     return NULL;
     resume_2:

     PyErr_SetException( StopIteration );
     return NULL;
}

As you can see, the function has an initial dispatcher. Resume point 0 means we are starting at the top. Then every yield results in a function return with an updated resume point.

I experimented with this actually a long time ago, and experimental code was the result that remained in Nuitka. The problem left to solve was to store the variables that would normally live on the stack, in a heap storage. That is what I am currently working on.

This leads me to “heap storage”, which is what I am currently working on and will report on next week. Once that is there, goto generators can work, and will become the norm. Until then, I am refactoring a lot to get accesses to variable go through proper objects that know their storage locations and types.

Hotfixes

So there have been 2 more hotfixes. One was to make the enum and __new__ compatibility available that I talked about last week in TWN #2 <./nuitka-this-week-2.html#python3-enumerators> coupled with a new minor things.

And then another one, actually important, where Python3 __annotations__ by default was the empty dictionary, but then could be modified, corrupting the Nuitka internally used one severely.

Right now I have on factory another fix for nested namespace packages in Python3 and that might become another hotfix soon.

As you know, I am following the git flow model, where it’s easy to push out small fixes, and just those, on top of the last release. I tend to decide based on importance. However, I feel that with the important fixes in the hotfixes now, it’s probably time to make a full release, to be sure everybody gets those.

Plans

Finishing heap storage is my top priority right now and I hope to complete the refactorings necessary in the coming week. I will also talk about how it also enables C types work next week.

Until next week then!

Nuitka this week #2

2018-08-02T07:08:54+02:00

New Series Rationale

As discussed last week in TWN #1 this is a new series that I am using to highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

Python 3.7

I made the first release with official 3.7 support, huge milestone in terms of catching up. Generic classes posed a few puzzles, and need more refinements for error handling, but good code works now.

The class creation got a bit more complex, yet again, which will make it even hard to know the exact base classes to be used. But eventually we will manage to overcome this and statically optimize that.

MSI 3.7 files for Nuitka

Building the MSI files for Nuitka ran into a 3.7.0 regression of CPython failing to build them, that I reported and seems to be valid bug of theirs.

So they will be missing for some longer time. Actually I wasn’t so sure if they are all that useful, or working as expected for the runners, but with the -m nuitka mode of execution, that ought to be a non-issue. so it would be nice to keep them for those who use them for deployment internally.

Planned Mode

I have a change here. This is going to be a draft post until I publish it, so I might the link, or mention it on the list, but I do not think I will wait for feedback, where there is not going to be all that much.

So I am shooting this off the web site.

Goto Generators

This is an exciting field of work, that I have been busy with this week. I will briefly describe the issue at hand.

So generators in Python are more generally called coroutines in other places, and basically that is code shaking hands, executing resuming in one, handing back a piece of data back and forth.

In Python, the way of doing this is yield and more recently yield from as a convienant way for of doing it in a loop in Python3. I still recall the days when that was a statement. Then communication was one way only. Actually when I was still privately doing Nuitka based on then Python 2.5 and was then puzzled for Python 2.6, when I learned in Nuitka about it becoming an expression.

The way this is implemented in Python, is that execution of a frame is simply suspended, and another frame stack bytecode is activated. This switching is of course very fast potentially, the state is already fully preserved on the stack of the virtual machine, which is owned by the frame. For Nuitka, when it still was C++, it wasn’t going to be possible to interrupt execution without preserving the stack. So what I did was very similar, and I started to use makecontext/setcontext to implement what I call fibers.

Basically that is C level stack switching, but with a huge issue. Python does not grow stacks, but can need a lot of stack space below. Therefore 1MB or even 2MB per generator was allocated, to be able to make deep function calls if needed.

So using a lot of generators on 32 bits could easily hit a 2GB limit. And now with Python3.5 coroutines people use more and more of them, and hit memory issues.

So, goto generators, now that C is possible, are an entirely new solution. With it, Nuitka will use one stack only. Generator code will become re-entrant, store values between entries on the heap, and continue execution at goto destinations dispatched by a switch according to last exit of the generator.

So I am now making changes to cleanup the way variable declarations and accesses for the C variables are being made. More on that next week though. For now I am very exited about the many cleanups that stem from it. The code generation used to have many special bells and whistles, and they were generalized into one thing now, making for cleaner and easier to understand Nuitka code.

Python3 Enumerators

On interesting thing, is that an incompatibility related to __new__ will go away now.

The automatic staticmethod that we had to hack into it, because the Python core will do it for uncompiled functions only, had to be done while declaring the class. So it was visible and causing issues with at least the Python enum module, which wants to call your __new__ manually. Because why would it not?!

But turns out, for Python3 the staticmethod is not needed anymore. So this is now only done for Python2, where it is needed, and things work smoothly with this kind of code now too. This is currently in my factory testing and will probably become part of a hotfix if it turns out good.

Hotfixes

Immediately after the release, some rarely run test, where I compiled all the code on my machine, found 2 older bugs, obscure ones arguably, that I made into a hotfix, also because the test runner was having a regression with 3.7, which prevented some package builds. So that was 0.5.32.1 release.

And then I received a bug report about await where a self test of Nuitka fails and reports an optimization error. Very nice, the new exceptions that automatically dump involved nodes as XML made it immediately clear from the report, what is going on, even without having to reproduce anything. I bundled a 3.7 improvement for error cases in class creation with it. So that was the 0.5.32.2 release.

Plans

Finishing goto generators is my top priority, but I am also going over minor issues with the 3.7 test suite, fixing test cases there, and as with e.g. the enum issue, even known issues this now finds.

Until next week.

Nuitka this week #1

2018-07-27T07:08:54+02:00

New Series Rationale

I think I tend to prefer coding over communication too much. I think I need to make more transparent what I am doing. Also things, will be getting exciting continuously for a while now.

I used to status report posts, many years ago, every 3 months or so, and that was nice for me also to get an idea of what changed, but I stopped. What did not happen, was to successfully engage other people to contribute.

This time I am getting more intense. I will aim to do roughly weekly or bi-weekly reports, where I highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

Planned Mode

I will do it this fashion. I will write a post to the mailing list, right about wednesday every week or so. I need to pick a day. I am working from home that day, saving me commute time. I will invest that time into this.

The writing will not be too high quality at times. Bare with me there. Then I will check feedback from the list, if any. Hope is for it to point out the things where I am not correct, missing, or even engage right away.

Topics are going to be random, albeit repeating. I will try and make links to previous issues where applicable. Therefore also the TOC, which makes for link targets in the pages.

Locals Dict

When I am speaking of locals dict, I am talking of class scopes (and functions with exec statements). These started to use actual dictionary a while ago, which was a severe setback to optimization.

Right now, so for this week, after a first prototype was making the replacement of local dict assignment and references for Python2, and kind of worked through my buildbots, flawlessly, I immediately noticed that it would require some refactoring to not depend on the locals scopes to be only in one of the trace collections. Thinking of future inlining, maybe part of a locals scope was going to be in multiple functions, that ought to not be affected.

Therefore I made a global registry of locals scopes, and working on those, I checked its variables, if they can be forward propagated, and do this not per module, but after all the modules have been done. This is kind of a setback for the idea of module specific optimization (cacheable later on) vs. program optimization, but since that is not yet a thing, it can remain this way for now.

Once I did that, I was interested to see the effect, but to my horror, I noticed, that memory was not released for the locals dict nodes. It was way too involved with cyclic dependencies, which are bad. So that was problematic of course. Compilation to keep nodes in memory for both tracing the usage as a locals dict and temporary variables, wasn’t going to help scaling at all.

Solution is finalization

Nodes need Finalization

So replaced nodes reference a parent, and then the locals scope references variables, and trace collections referencing variables, which reference locals scopes, and accesses referencing traces, and so on. The garbage collector can handle some of this, but seems I was getting past that.

For a solution, I started to add a finalize method, which released the links for locals scopes, when they are fully propagated, on the next run.

Adding a finalize to all nodes, ought to make sure, memory is released soon, and might even find bugs, as nodes become unusable after they are supposedly unused. Obviously, there will currently be cases, where nodes becomes unused, but they are not finalized yet. Also, often this is more manual, because part of the node is to be released, but one child is re-used. That is messy.

Impact on Memory Usage

The result was a bit disappointing. Yes, memory usage of mercurial compilation went back again, but mostly to what it had been. Some classes are now having their locals dict forward propagated, but the effect is not always a single dictionary making yet. Right now, function definitions, are not forward at all propagated. This is a task I want to take on before next release though, but maybe not, there is other things too. But I am assuming that will make most class dictionaries created without using any variables at all anymore, which should make it really lean.

Type Hints Question

Then, asking about type hints, I got the usual question about Nuitka going to use it. And my stance is unchanged. They are just hints, not reliable. Need to behave the same if users do it wrong. Suggested to create decorated which make type hints enforced. But I expect nobody takes this on though. I need to make it a GitHub issue of Nuitka, although technically it is pure CPython work and ought to be done independently. Right now Nuitka is not yet there anyway yet, to take full advantage.

Python 3.7

Then, for Python 3.7, I have long gotten the 3.6 test suite to pass. I raised 2 bugs with CPython, one of which lead to update of a failing test. Nuitka had with large delay, caught of with what del __annotations__ was doing in a class. Only with the recent work for proper locals dict code generation, we could enforce a name to be local, and have proper code generation, that allows for it to be unset.

This was of course a bit of work. But the optimization behind was always kind of necessary to get right. But now, that I got this, think of my amazement when for 3.7 they reverted to the old behavior, where annotiatons then corrupt the module annotations

The other bug is a reference counting bug, where Nuitka tests were failing with CPython 3.7, and turns out, there is a bug in the dictionary implementation of 3.7, but it only corrupts counts reported, not actual objects, so it’s harmless, but means for 3.7.0 the reference count tests are disabled.

Working through the 3.7 suite, I am cherry picking commits, that e.g. allow the repr of compiled functions to contain <compiled_function ...> and the like. Nothing huge yet. There is now a subscript of type, and foremost the async syntax became way more liberal, so it is more complex for Nuitka to make out if it is a coroutine due to something happening inside a generator declared inside of it. Also cr_origin was added to coroutines, but that is mostly it.

Coroutine Compatibility

A bigger thing was that I debugged coroutines and their interaction with uncompiled and compiled coroutines awaiting one another, and turns out, there was a lot to improve.

The next release will be much better compatible with asyncio module and its futures, esp with exceptions to cancel tasks passed along. That required to clone a lot of CPython generator code, due to how ugly they mess with bytecode instruction pointers in yield from on an uncompiled coroutine, as they don’t work with send method unlike everything else has to.

PyLint Troubles

For PyLint, the 2.0.0 release found new things, but unfortunately for 2.0.1 there is a lot of regressions that I had to report. I fixed the versions of first PyLint, and now also Astroid, so Travis cannot suddenly start to fail due to a PyLint release finding new warnings.

Currently, if you make a PR on GitHub, a PyLint update will break it. And also the cron job on Travis that checks master.

As somebody pointed out, I am now using requires.io <https://requires.io/github/kayhayen/Nuitka/requirements/?branch=factory> to check for Nuitka dependencies. But since 1.9.2 is still needed for Python2, that kind of is bound to give alarms for now.

TODO solving

I have a habit of doing off tasks, when I am with my notebook in some place, and don’t know what to work on. So I have some 2 hours recently like this, and used it to look at TODO and resolve them.

I did a bunch of cleanups for static code helpers. There was one in my mind about calling a function with a single argument. That fast call required a local array with one element to put the arg into. That makes using code ugly.

Issues Encountered

So the enum module of Python3 hates compiled classes and their staticmethod around __new__. Since it manually unwraps __new__ and then calls it itself, it then finds that a staticmethod object cannot be called. It’s purpose is to sit in the class dictionary to give a descriptor that removes the self arg from the call.

I am contemplating submitting an upstream patch for CPython here. The hard coded check for PyFunction on the __new__ value is hard to emulate.

So I am putting the staticmethod into the dictionary passed already. But the undecorated function should be there for full compatibility.

If I were to make compiled function type that is both a staticmethod alike and a function, maybe I can work around it. But it’s ugly and a burden. But it would need no change. And maybe there is more core wanting to call __new__ manually

Plans

I intend to make a release, probably this weekend. It might not contain full 3.7 compatibility yet, although I am aiming at that.

Then I want to turn to “goto generators”, a scalability improvement of generators and coroutines that I will talk about next week then.

Until next week.

Nuitka Blog - Posts tagged NTW

Nuitka this week #16

Nuitka Onefile Gets More Flexible: --onefile-cache-mode and {PROGRAM_DIR}

Taking Control of Onefile Unpacking: --onefile-cache-mode

Relative Paths with {PROGRAM_DIR}

Nuitka Commercial and Open Source

Give it a Try!

Nuitka this week #15

Nuitka this week #14

All in with Nuitka

Why now?

What will change?

Nuitka Package Configuration

Streaming

Python 3.11

How you can help?

Becoming more active

Nuitka this week #13

Nuitka this week #12

Nuitka this week #11

Nuitka this week #10

Nuitka this week #9

Nuitka this week #8

Nuitka this week #7

Nuitka this week #6

Nuitka Onefile Gets More Flexible: `--onefile-cache-mode` and `{PROGRAM_DIR}`

Taking Control of Onefile Unpacking: `--onefile-cache-mode`

Relative Paths with `{PROGRAM_DIR}`