Nuitka on PyBench - Good and Bad

In case you wonder, [what Nuitka is](/pages/overview.html), look here. Over the 0.3.x release cycle, I have mostly looked at its performance with "pystone". I merely wanted to have a target to look at and enjoy the progress we have made there.

In the context of the Windows port then, Khalid Abu Bakr used the pybench on Windows and that got me interested. It's a nice collection of micro benchmarks, which is quite obviously aimed for looking CPython implementations only. In that it's quite good to check where Nuitka is good at, and where it can still take improvements for the milestone 2 stuff.

Enhancements to PyBench

  • The pybench refused to accept that Nuitka could use so little time on some tests, I needed to hack it to allow it.
  • Then it had "ZeroDivisionError" exceptions, because Nuitka can run fully predictable code not at all, thus with a time of 0ms, which gives interesting factors.
  • Also these are many results, we are going to care for regressions only, so there is an option now to output only tests with negative values.

The Interesting Parts

  • Nuitka currently has some fields where optimizations are already so effective as to render the whole benchmark pointless. Longterm, most of PyBench will not be looked at anymore, where the factor becomes "infinity", there is little point in looking at it. We will likely just use it as a test that optimizations didn't suddenly regress. Publishing the numbers will not be as interesting.
  • Then there are slow downs. These I take seriously, because of course I expect that Nuitka shall only be faster than CPython. Sometimes the implementation of Nuitka for some rarely used features is sub par though. I color coded these in red in the table below.
  • ComplexPythonFunctionCalls: These are twice as slow, which is an tribute to the fact, that the code in this domain is only as good as it needs to be. Of course function calls are very important, and this needs to be addressed.
  • TryRaiseExcept: This is much slower because of the cost of the raise statement, which is extremely high currently. For every raise, a frame object with a specific code object is created, so the traceback will point to the correct location. This is very inefficient, and wasteful. We need to be able to create code objects that can be used for all lines needed, and then we can re-use it and only have one frame object per function, which then can be re-used itself. There is already some work for that in [current git](/pages/download.html) (0.3.9 pre 2), but it's not yet complete at all.
  • WithRaiseExcept: Same problem as TryRaiseExcept, the exception raising is too expensive.
  • Note also that -90% is in fact much worse that +90%, the "diff" numbers from pybench make improvements look much better than regressions do. You can also checkout the comparison on the new [benchmark pages]( that I am just creating, they are based on codespeed, which I will blog upon separately.

Look at this table of results as produced by pybench:

Benchmark Results

**Test Name** **min CPython** **min Nuitka** **diff**
BuiltinFunctionCalls 76ms 54ms +41.0%
BuiltinMethodLookup 57ms 47ms +22.1%
CompareFloats 79ms 0ms +inf%
CompareFloatsIntegers 75ms 0ms +inf%
CompareIntegers 76ms 0ms +inf%
CompareInternedStrings 68ms 32ms +113.0%
CompareLongs 60ms 0ms +inf%
CompareStrings 86ms 62ms +38.2%
CompareUnicode 61ms 50ms +21.9%
ComplexPythonFunctionCalls 86ms 179ms -52.3%
ConcatStrings 98ms 99ms -0.6%
ConcatUnicode 127ms 124ms +2.3%
CreateInstances 76ms 52ms +46.8%
CreateNewInstances 58ms 47ms +22.1%
CreateStringsWithConcat 85ms 90ms -6.5%
CreateUnicodeWithConcat 74ms 68ms +9.5%
DictCreation 58ms 36ms +60.9%
DictWithFloatKeys 67ms 44ms +51.7%
DictWithIntegerKeys 64ms 30ms +113.8%
DictWithStringKeys 60ms 26ms +130.6%
ForLoops 47ms 15ms +216.2%
IfThenElse 67ms 16ms +322.5%
ListSlicing 69ms 70ms -0.9%
NestedForLoops 72ms 25ms +187.4%
NestedListComprehensions 87ms 42ms +105.9%
NormalClassAttribute 62ms 77ms -18.9%
NormalInstanceAttribute 56ms 24ms +129.7%
PythonFunctionCalls 72ms 34ms +116.1%
PythonMethodCalls 84ms 38ms +120.0%
Recursion 97ms 56ms +73.1%
SecondImport 61ms 47ms +31.6%
SecondPackageImport 66ms 29ms +125.4%
SecondSubmoduleImport 86ms 32ms +172.0%
SimpleComplexArithmetic 74ms 62ms +18.3%
SimpleDictManipulation 65ms 35ms +89.7%
SimpleFloatArithmetic 77ms 56ms +39.3%
SimpleIntFloatArithmetic 58ms 39ms +48.3%
SimpleIntegerArithmetic 59ms 37ms +57.7%
SimpleListComprehensions 75ms 33ms +128.7%
SimpleListManipulation 57ms 27ms +109.4%
SimpleLongArithmetic 68ms 57ms +19.9%
SmallLists 69ms 41ms +66.6%
SmallTuples 66ms 98ms -32.2%
SpecialClassAttribute 63ms 49ms +29.1%
SpecialInstanceAttribute 130ms 24ms +434.5%
StringMappings 67ms 62ms +8.5%
StringPredicates 69ms 59ms +16.6%
StringSlicing 73ms 47ms +54.8%
TryExcept 57ms 0ms +3821207.1%
TryFinally 65ms 26ms +153.4%
TryRaiseExcept 64ms 610ms -89.5%
TupleSlicing 76ms 67ms +12.7%
UnicodeMappings 88ms 91ms -2.9%
UnicodePredicates 64ms 59ms +8.8%
UnicodeProperties 69ms 63ms +8.8%
UnicodeSlicing 80ms 68ms +17.6%
WithFinally 84ms 26ms +221.2%
WithRaiseExcept 67ms 1178ms -94.3%


Comments powered by Disqus