Sander’s Blog

Boolean Array to (gmpy2) int

2022-07-15T05:00:00+00:00

This time another colleague was trying to work with gmpy2, a library for fast arithmetic operations on very large numbers. The specific problem was that the numbers had to be created from a numpy array with 1,000,000 booleans or more first, and the method my colleague had come up with wasn’t fast enough.

To beat: num = num.bit_set(i)

As a first attempt, they created a gmpy2.mpz object and manually set each bit with the bit_set() method.

import gmpy2

def mpz_by_bitset(bool_array):
    num = gmpy2.mpz(0)
    for i, bit in enumerate(bool_array[::-1]):
        if bit:
            num = num.bit_set(i)
    return num

Before considering performance on large arrays, let’s test it with a sample of 128 (=2^7) booleans:

import numpy as np
bool_array = np.random.randint(0,2,2**7, dtype=bool)
%timeit mpz_by_bitset(bool_array)

10.8 µs ± 4.13 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

This function does what we want, but raises two red flags for me:

We’re processing every bit individually
A new copy of the object is made every time we change a bit

Modern processors work on 64 bits at a time, or possibly even more with vector instructions, so only working on a single bit at a time means we’re using only about 1-2% of our processor’s power.

First Suggestion: making a binary string

In a first attempt to do better, we make use of the fact that Python’s integers can be infinitely large, and can be created from a binary string. We first manually create the binary string with ''.join(), create a Python int and then pass that to gmpy2:

def mpz_by_bin_string(bool_array):
    bin_string = '0b' + ''.join(map(str, bool_array.astype(int)))    
    return gmpy2.mpz(int(bin_string, base=2))
    
%timeit mpz_by_bin_string(bool_array)

80.2 µs ± 85.4 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

While this may seem more Pythonic, we’re now working with bits as a string instead of as numbers. This isn’t great for performance, and it shows: for our 128-bit example it’s about 8x slower.

Second Suggestion: numpy.packbits

With our first suggestion we are in the right direction though: a number in a computer is after all just a string of bits in memory. The only thing we want to do is take this string of bits we have, our boolean array, and let gmpy2 read it as a number it recognizes. So rather than doing this via a string as an intermediate step, we should just do this numerically. As gmpy2 has a method from_binary, this should be possible.

The problem is that a boolean in Python/numpy is stored as an integer, taking up at least one byte of space (‘00000000’ or ‘00000001’). Simply sticking those behind eachother will give us a number with a lot of zeros in the middle that don’t belong there: we only care about the last bit.

That problem can be solved with numpy.packbits, which packs an array of binary values into an array of type uint8 = 1 byte per value. That gets rid of all the zeros we don’t want. Now we can use numpy’s tobytes() to get the array directly as bytes.

Sadly, that solution fails in practise. While gmpy2.from_binary() accepts a bytestring as input, it fails when given the result of our bitpacked array as bytes. There seems to be some special formatting needed that isn’t documented. We need to use a Python integer as an intermediate for it to work:

def mpz_by_packbits(bool_array):
    bytestring = np.packbits(bool_array).tobytes()
    python_int = int.from_bytes(bytestring, byteorder='big')
    return gmpy2.mpz(python_int)

%timeit mpz_by_packbits(bool_array)

1.54 µs ± 10.2 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

Now that’s more like it! Even with the added step of using int.from_bytes instead of directly using gmpy2.from_binary, we’re still over 6x faster. But does that hold up for larger array sizes?

Automated Comparison with Perfplot

Perfplot is a cool little package that automates the creation of timing comparison plots like those I manually made in earlier posts. Using it we can quickly compare the different implementations and see how they scale. As test input size, I will use only multiples of complete bytes, just to prevent any possible issues with numpy.packbits.

import perfplot

out = perfplot.bench(
    setup=lambda n: np.random.randint(0,2,2**n),
    kernels=[mpz_by_bitset, mpz_by_bin_string, mpz_by_packbits],
    n_range=[n for n in range(3, 36, 4)],
    labels=['mpz.bit_set', 'int(bin_string)', 'np.packbits'],
    xlabel='2^n booleans',
    equality_check=False,
    max_time=20,
)

out.show(logy=True, logx=False)

The most obvious thing to notice is that the np.packbits version is clearly the fastest and scales best with larger sizes. It only takes about 10 seconds for an array of 2^30 booleans. That’s almost 1 GigaByte of memory for just the final bits!

The int(bin_string) version scales about the same as np.packbits, but is a lot slower overall. I can imagine the translation back and forth from integers to strings and back into an integer taking up a lot of overhead.

The bit_set version doesn’t actually perform too bad up to ~2^17 booleans, or about 16 KiloBytes. After that it scales a lot more poorly and does worse than even the int(bin_string) implementation. Perhaps the local copies that keep getting made are overflowing the processor’s cache at that point? Either way, it exceeds a runtime of 100 seconds when processing 2^23 booleans (=one MegaByte).

Conclusion

Iterating over the individual bits and using bit_set is indeed slower than the implementation using np.packbits, and scales surprisingly poorly for more than 2^17 booleans.

Also: Perfplot is cool!

Case Study 1: Row-wise equality checking

2021-11-14T05:00:00+00:00

Introduction

It shouldn’t come as a surprise that I like to optimize (Python) code. So when a colleague (who is more experienced in R than Python/Numpy) mentioned in a presentation that a certain calculation was rather slow, I was intrigued. Would I be able to help speed it up? In this blog post, I’ll take you along in the process I went through to optimize the code.

After confirming the code was indeed written in Python and asking for access, I took a look at the function. Below is a modified version showcasing what was relevant to the optimization.

def calculate(data, no_tests=100):  # no_tests is limited due to function speed
    n, d = data.shape               # typically n = 1_000, 10_000 or 100_000, d = 9

    for i in range(no_tests):
        for j in range(d):
            # prepare indices
            a_ind = np.concatenate([np.arange(0,j), np.arange(j+1,d)])
            # prepare auxiliary informationn of every individual to find peers
            aux = np.apply_along_axis(np.array2string,1, data[:,a_ind])
            # save data of selected
            selected = data[aux==aux[i],j]

            ... # perform some final processing of `selected`

Step 1: What does it do?

At first glance, it’s a double for-loop which creates some indices, uses them to transforms some of the data into strings as auxiliary data, and performs a selection based on equality in that auxiliary data. The first thing that stands out to me here is the np.array2string function call, as such a transformation is rather expensive, so I wonder if that is really needed. If not, that would probably be the place to start.

But let’s not get ahead of ourselves. Let’s consider what this function does step-by-step. Let’s set up a toy example with some sample data, pick some i,j in the middle of the loops, flatten the function and progress through a single run of the inner-most loop:

import numpy as np                                    # intermediate values:
n, d = 3, 4
i, j = 1, 2
data = np.arange(n*d).reshape(n, d) / 100             # [[0.  , 0.01, 0.02, 0.03],
                                                      #  [0.04, 0.05, 0.06, 0.07],
                                                      #  [0.08, 0.09, 0.1 , 0.11]]

range_1, range_2 = np.arange(0, j), np.arange(j+1,d)  # [0, 1], [3]
a_ind = np.concatenate([range_1, range_2])            # [0, 1, 3]

tmp = data[:,a_ind]                                   # [[0.  , 0.01, 0.03],
                                                      #  [0.04, 0.05, 0.07],
                                                      #  [0.08, 0.09, 0.11]]

aux = np.apply_along_axis(np.array2string, 1, tmp)    # ['[0.   0.01 0.03]',
                                                      #  '[0.04 0.05 0.07]',
                                                      #  '[0.08 0.09 0.11]']
comparisons = aux==aux[i]                             # [False, True, False]

selected = data[comparisons, j]                       # [0.1]

Looking at the flattened code and intermediate values from this sample run, we can see the process performed by the function much more clearly: In short, we exclude column j from the data and check whether any rows are equal to the selected row i. The original data in column j for each of those rows is then used for further calculations.

Step 2: What to Optimize

Now we know what the function does, we can try to optimize it. There are two candidate statements to optimize: creating the a_ind indices and determining which rows are equal using aux. Let’s test them both:

j, d = 3, 9
%timeit np.concatenate([range(0,j), range(j+1,d)])

7.18 µs ± 310 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

large_data = np.random.randn(1_000, 9)  # test speed using larger dataset
%timeit aux = np.apply_along_axis(np.array2string, 1, large_data); aux == aux[0]

130 ms ± 3.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

The equality check using np.array2string takes a factor of 20,000 more time than creating the indices. So as expected, that’s going to be the place to start. When comparing the speed, we’ll keep using the 1,000 x 9 large_data array we created here. This is large enough to give a good idea of the speed, but small enough to make testing fast.

Step 3: Optimizing within loops

The np.apply_along_axis(np.array2string, 1, large_data) statement is not just the slowest statement we found, but it’s extra worthwhile to optimize since it’s within the double nested loop. Any small improvement to a single run in that statement will add up quickly when run so many times in the loops. This means that, for now, we can just focus on optimizing that single statement as-is, without looking at the loops too.

Replacing np.array2string

First step is replacing row-equality check to no longer use strings. Instead of using np.array2string, we can check for element-wise equality between rows with np.allclose. This returns a single True or False when comparing two rows, just like with the strings previously. This results in a 4x speedup:

comparisons = np.apply_along_axis(np.allclose, 1, tmp, tmp[i])
# [False, True, False]
%timeit np.apply_along_axis(np.allclose, 1, large_data, large_data[0])

32.8 ms ± 1.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
-> 3.9x speedup

np.allclose vs np.equal

If we don’t need the tolerance-checking by np.allclose, we can even try just using np.equal instead. However, this returns the element-wise results, not a single True or False for each row:

comparisons = np.apply_along_axis(np.equal, 1, tmp, tmp[i])
# [[False, False, False],
#  [ True,  True,  True],
#  [False, False, False]]

To end up with the correct result, each row has to be summarized into True if the whole row is True, otherwise the row should be summarized to False. For this, we can use the np.all function, which works like an array-wide boolean AND operation.

# for example:
np.all([
    [ True,  True,  True],
    [ True,  True, False],
    [False, False, False],
], axis=1)
# [ True, False, False]

comparisons = np.all(np.apply_along_axis(np.equal, 1, tmp, tmp[i]), axis=1)
# [False, True, False]
%timeit np.all(np.apply_along_axis(np.equal, 1, large_data, large_data[0]), axis=1)

2.03 ms ± 59.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
-> 64x speedup

Perhaps surprisingly, this is another 16x faster! Even when adding the extra np.all call, it turns out that np.equal is much faster than np.allclose.

Removing np.apply_along_axis

Because of numpy’s broadcasting rules, we don’t actually need np.apply_along_axis to perform the element-wise comparison per row of data using np.equal. We can simply compare the entire array with the relevant row.

tmp == tmp[i]  # same as np.equal
# [[False, False, False],
#  [ True,  True,  True],
#  [False, False, False]]
np.all(tmp == tmp[i], axis=1)
# [False, True, False]

%timeit np.all(large_data == large_data[0], axis=1)

36.8 µs ± 2.06 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
-> 3,500x speedup

That’s a 3,500x speedup in total! Turns out that numpy by itself is pretty fast at performing these kinds of comparisons. With that, we’ve both simplified the code and made it a lot faster.

You may also notice that this version only takes ~5x as long as the index creation code we also timed earlier. It’s not worth it to spend time on this though, as it doesn’t scale with the amount of data rows, and we can speed it up using a different technique in the next step anyway.

Step 4: Factoring code out of loops

We’ve now made the code within the loops as fast as possible. The next step is to see if there is any work we’re unnecessarily doing again in any of the loops and well… making sure we don’t.

Inner loop

Remember that the inner loop was iterating over the 9 columns of our sample data? Each time we create the necessary indices and compare the data with the current row.

for i in range(no_tests):  # Outer loop
    for j in range(d):     # Inner loop
        a_ind = np.concatenate([range(0,j), range(j+1,d)])
        comparisons = np.all(data[:,a_ind] == data[i,a_ind], axis=1)

The indices depend on the column j, so within this loop we’re not doing any extra work there. But to calculate comparisons, only the columns of the a_ind indices are used. As a_ind only excludes a single column per iteration, each column is compared d-1 times. We can check this by printing a_ind for each iteration:

for j in range(d):
    a_ind = np.concatenate([range(0,j), range(j+1,d)])
    print(a_ind)
# [1 2 3]
# [0 2 3]
# [0 1 3]
# [0 1 2]

%%timeit
for i in range(10):
    for j in range(9):
        a_ind = np.concatenate([np.arange(0,j), np.arange(j+1,9)])
        comparisons = np.all(large_data[:,a_ind] == large_data[i,a_ind], axis=1)

3.51 ms ± 109 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The element-wise comparisons in data[:,a_ind] == data[i,a_ind] won’t change between iterations, so we’re actually making the same comparisons d-1 = 8 times! That makes this a good candidate for reintroducing an auxiliary variable, but this time outside of the inner loop:

%%timeit
for i in range(10):
    aux = large_data == large_data[i]
    for j in range(9):
        a_ind = np.concatenate([np.arange(0,j), np.arange(j+1,9)])
        comparisons = np.all(aux[a_ind], axis=1)

1.24 ms ± 65.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
-> 2.7x speedup

We’re saving 7/8th of the comparisons compared to previously, reducing the amount of work within the inner loop. But out of the potential 8x speedup, we ‘only’ get 2.7x. Why? I can only speculate… ¯\_(ツ)_/¯

Outer loop

We’ve taken the element-wise data comparison out of the inner loop because those results do not change depending on the index j of the inner loop. Using the same reasoning, we can extract the a_ind creation out of the outer loop. Yes, we need to determine a_ind once for each j, but they don’t change depending on i. That does mean we need to have a separate for j in range(d) loop outside of the outer loop to make the list of indices, but then we can keep reusing them in the inner loop.

You might wonder why we even bother with this if each run only takes 7 µs. As the size of large_data increases to many thousands or even millions, those microseconds per iteration can still add up. Besides, I think it makes the code cleaner in a way: it clearly separates what needs to be done within the inner and outer loop from what does not.

%%timeit
a_indices = np.array([
    np.concatenate([np.arange(0,j), np.arange(j+1,d)])
    for j in range(d)
])
for i in range(10):
    aux = large_data == large_data[i]
    for j, a_ind in enumerate(a_indices):
        comparisons = np.all(aux[a_ind], axis=1)

910 µs ± 24.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
-> 3.8x speedup

Nothing major, but an improvement nonetheless.

Step 5: Parallelize and benchmark

As a final step, we can see that the code within the outer loop is so-called “embarassingly parallel”. This means each outer loop iteration is independent of the others. In this case, that’s because an iteration only depends on i. Succesfully parallelizing code can give an additional speedup of up to the number of cores used. This can range anywhere from 4x on your local laptop to 256x on servers with the latest AMD EPYC cpus¹

However, that doesn’t mean it’s always worth it to actually write a parallel implementation. Parallelism in Python comes with some overhead, both in code complexity and in runtime. If these downsides don’t outweigh the speedup, it’s probably best to stick to a serial implementation.

In this case, we consider the parallel implementation below because the code we’re optimizing is simple to begin with and will be run on large data arrays with up to a million rows. This implementation is built using Python’s built-in multiprocessing.Pool and functools.partial.

from functools import partial
from multiprocessing import Pool, cpu_count

def _parallel_helper(i, data, a_indices):
    """Helper function to perform single run for parallelization"""
    n, d = data.shape
    aux = data == data[i]
    results = np.empty(d)
    for j, a_ind in enumerate(a_indices):
        comparisons = np.all(aux[a_ind], axis=1)
        results[j] = np.sum(comparisons)  # 'sum' as stand-in for final processing

def calc_parallel(data, no_tests, n_procs=cpu_count()):
    """Parallel implementation"""
    n, d = data.shape
    a_indices = np.array([
        np.concatenate([np.arange(0,j), np.arange(j+1,d)])
        for j in range(d)
    ])
    parallel_func = partial(_parallel_helper, data=data, a_indices=a_indices)
    with Pool(n_procs) as p:
        result = p.map(parallel_func, range(no_tests))
    return np.array(result)

The runtime overhead of starting a Pool is at least a second. This makes it unfair to compare to the previous optimized implementation for the large_data of size (1_000, 9), but as the data consists of more rows and more rows are compared, the parallel implementation should do better at some point.

To show this as clearly as possible, you can see the results of a small benchmarking experiment I’ve run in the graph below. It compares the parallel implementation with the implementation after each optimization step described in this blog. Each implementation is run with increasingly more rows of data, with no_tests set to n, until it takes more than 100 seconds to complete. The entire benchmarking code can be seen by clicking the spoiler tag below.

Benchmarking code

from functools import partial
from multiprocessing import Pool, cpu_count
from time import time
import numpy as np
import matplotlib.pyplot as plt

def calc_orig(data):
    """Original implementation"""
    n, d = data.shape
    results = np.empty((n,d))
    for i in range(n):
        for j in range(d):
            a_ind = np.concatenate([np.arange(0,j), np.arange(j+1,d)])
            aux = np.apply_along_axis(np.array2string,1, data[:,a_ind])
            peers_data = data[aux==aux[i],j]
            results[i,j] = np.sum(peers_data)
    return results

def calc_inner_loop_optimized(data):
    """Optimized code within inner loop"""
    n, d = data.shape
    results = np.empty((n,d))
    for i in range(n):
        for j in range(d):
            a_ind = np.concatenate([np.arange(0,j), np.arange(j+1,d)])
            comparisons = np.all(data[:,a_ind] == data[i,a_ind], axis=1)
            results[i,j] = np.sum(comparisons)
    return results

def calc_optimized(data):
    """Optimized code, including extractions from loop"""
    n, d = data.shape
    a_indices = np.array([
        np.concatenate([np.arange(0,j), np.arange(j+1,d)])
        for j in range(d)
    ])
    results = np.empty((n,d))
    for i in range(n):
        aux = data == data[i]
        for j, a_ind in enumerate(a_indices):
            comparisons = np.all(aux[a_ind], axis=1)
            results[i,j] = np.sum(comparisons)
    return results

def _parallel_helper(i, data, a_indices):
    """Helper function to perform single run for parallelization"""
    n, d = data.shape
    aux = data == data[i]
    results = np.empty(d)
    for j, a_ind in enumerate(a_indices):
        comparisons = np.all(aux[a_ind], axis=1)
        results[j] = np.sum(comparisons)

def calc_parallel(data, n_procs=cpu_count()):
    """Parallel implementation"""
    n, d = data.shape
    a_indices = np.array([
        np.concatenate([np.arange(0,j), np.arange(j+1,d)])
        for j in range(d)
    ])
    parallel_func = partial(_parallel_helper, data=data, a_indices=a_indices)
    with Pool(n_procs) as p:
        result = p.map(parallel_func, range(n))
    return np.array(result)

if __name__ == '__main__':
    magnitudes = range(1, 6)
    numbers = [1, 2, 5]
    N = [n * 10**m for m in magnitudes for n in numbers]
    functions = {
        'original': calc_orig,
        'inner loop optimized': calc_inner_loop_optimized,
        'fully optimized': calc_optimized,
        'parallel (4)': partial(calc_parallel, n_procs=4),
        'parallel (8)': partial(calc_parallel, n_procs=8),
        'parallel (12)': partial(calc_parallel, n_procs=12),
    }
    durations = {name: np.full(len(N), np.nan) for name in functions.keys()}
    skip = {name: False for name in functions.keys()}

    for idx, n in enumerate(N):
        print(n)
        data = np.random.randn(n, 9)
        for name, func in functions.items():
            if skip[name]:
                continue
            start = time()
            func(data)
            end = time()
            durations[name][idx] = end-start
            print(f'    {name}: {durations[name][idx]}')
            if durations[name][idx] > 100:
                skip[name] = True

    fig, ax = plt.subplots()
    ax.set_title('Speed comparison of implementations')
    ax.set_xscale('log')
    ax.set_xlabel('$n$')
    ax.set_yscale('log')
    ax.set_ylabel('time (s)')
    for name, duration in durations.items():
        ax.plot(N, duration, marker='+', ms=12, label=name)
    ax.legend(loc='lower right')
    ax.grid(which='major', color='black', alpha=.5, linestyle='-')
    ax.grid(which='minor', color='black', alpha=.25, linestyle=':')
    fig.savefig('case-study-1-benchmark.png')  # or .pdf or .svg

The log-log plot shows nicely that with each optimization step, we’re able to process more rows of data in the same amount of time. The parallel implementations are a slight exception to that rule when the number of rows n is less than 10,000. In those cases the constant runtime overhead of about one second means they’re actually slower than the fully optimized serial implementation.

For the parallel implementation, I’ve run it with a pool of 4, 8 and 12 processes on my 12-thread CPU. Similarly to inner loop optimization in step 4, we don’t see the speedup we would expect, but at least using more processes (up to n_procs=multiprocessing.cpu_count()) does reduce the runtime. Again, I can only speculate why we don’t get the speedup we might expect ¯\_(ツ)_/¯

Conclusion

And so we’ve gone from an R-inspired string-based implementation past a clean and minimal pure-Python one all the way to a fully optimized and parallelized implementation that can scale over many cores. I’ve had a lot of fun figuring out all these optimizations, and I hope you enjoyed coming along on this optimization journey with me.

Finally, I would like to thank my colleague for being open to having me tinker on their code.

For single CPUs/machines. More is possible on (super)computer clusters using e.g. Dask or Ray. ↩

Filtering elements from a list of tuples

2019-10-16T05:00:00+00:00

Recently I was writing some code to fill a 50 by 100 by 50 3D matrix with some calculated values, but only at the indices that passed a certain check: the first index has to be smaller than the second. The naive way to do this would be a nested for-loop with a continue if the check was not passed.

for a in range(x):
    for b in range(y):
        for c in range(z):
            if a <= b:
                continue

            matrix[a, b, c] = ...

This solution however, is a) long and b) less re-usable. Instead, I wanted to create the list of index-tuples beforehand, preferably using itertools.product to make the code cleaner. Simply put:

from itertools import product

indices = []
for a, b, c in product(range(x), range(y), range(z)):
    if a > b:
        indices.append((a, b, c))

which we can easily rewrite as a one-liner in Python. For this we have two options: using the built-in filter function with a lambda, or by using a list-comprehension.

So, which is faster? To test, we’ll just leave out the third index we’re not using for the check:

%timeit list(filter(lambda x: x[0] > x[1], product(range(10), range(10))))

15.3 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit [(a, b) for (a, b) in product(range(10), range(10)) if a>b]

8.76 µs ± 2.32 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In this small 10x10 example, the list-comprehension is almost twice as fast. What happens if we increase the ranges to the sizes we are actually using?

%timeit list(filter(lambda x: x[0] > x[1], product(range(50), range(100))))

691 µs ± 63.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit [(a, b) for (a, b) in product(range(50), range(100)) if a > b]

247 µs ± 21.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The list-comprehension is now more than twice as fast as the filter(lambda x: ...) combination! This does make some sense as list-comprehensions are considered more ‘pythonic’ for a while now, and have received a bit more love in terms of performance improvements.

This comparison still leaves room for improvement in two aspects:

We’re not considering the 3rd index yet
We’re comparing two across methods of accessing the values for a and b: by index and by tuple unpacking

As the filter(function, iterable) built-in only ever passes a single argument at a time to its function, we can’t use it to compare these two accessing methods. Because we already know it’s a lot slower too, we can just ignore it from now on.

So let’s add that third index and compare the two value accessing methods in a list-comprehension:

%timeit [xyz for xyz in product(range(10), range(10), range(50)) if xyz[0] > xyz[1]]

481 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit [(x, y, z) for x, y, z in product(range(10), range(10), range(50)) if x > y]

343 µs ± 21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The tuple unpacking seems to be about a quarter faster than the indexing method. Does this difference hold up if we increase the ranges in the first two indices?

%timeit [xyz for xyz in product(range(50), range(100), range(50)) if xyz[0] > xyz[1]]

23.6 ms ± 320 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit [(x, y, z) for x, y, z in product(range(50), range(100), range(50)) if x > y]

17.1 ms ± 2.89 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

(Don’t pay too much attention to the large difference in std. dev. for the two comparisons, as one only did 10 loops while the other did 100.)

So interestingly, it seems like unpacking and re-packing a tuple is faster to access two of its values than just accessing those values by index without unpacking! Although this results in code that is slightly longer, I do think it’s more readable, so I’m happy this turns out to be the (slightly) faster way :)

For a deeper understanding of what’s going on here, we should probably look into the actual interpreted instructions using e.g. the disassembly module dis. To be continued…?

Multi-dimensional Indexing in Numpy

2019-10-05T10:00:00+00:00

(This post is based on a StackOverflow answer)

What is tuple-indexing in Numpy?

If you want to index an element in a multi-dimensional (i.e. nested) list in Python, you have to do this manually:

>>> nested_list = [
...     [1, 2, 3],
...     [4, 5, 6],
...     [7, 8, 9],
... ]
>>> print(nested_list[1][1])
5

This is already ugly for this 2-dimensional example, and for lists that have more levels of nesting it only gets worse.

Numpy is a great python package for working with multi-dimensional numerical data. You can effectively use it the same as if it is still the same data-structure in Python.

>>> import numpy as np
>>> nested_array = np.array([
...     [1, 2, 3],
...     [4, 5, 6],
...     [7, 8, 9],
... ])
>>> print(nested_array[1][1])
5

However, Numpy also natively supports tuple-indexing:

>>> print(nested_array[(1, 1)])
5
>>> print(nested_array[1, 1])
5

So… which is faster?

Simple experiments: 1D, 2D and now also in 3D!

Let’s start with the 1D case to get a baseline:

A = np.zeros(8)
%timeit A[4]

183 ns ± 3 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Indexing a single value in a single-dimensional array takes just under 200 nanoseconds on my laptop. Now let’s see how that time scales to two-dimensional arrays.

A = np.zeros((8,8))
%timeit A[4][4]

373 ns ± 6.12 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%timeit A[4, 4]

202 ns ± 2.42 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

As you can see, using numpy’s tuple-indexing is still around 200 nanoseconds, while using list-indexing takes about twice as long!

What is probably happening, is that each A[x] is translated to A.__getitem__(x). This means that when x is a tuple, it is passed down to numpy’s own functions to handle the multi-dimensional indexing. But when using list-indexing, this is executed as a nested call to the __getitem__ method. For the 2D case we just tested, this results in

(A.__getitem__(4)).__getitem__(4)

where each __getitem__ takes ~200 nanoseconds as we’ve seen before. So for a 3D array, we expect numpy’s tuple-indexing to be constant-time, while list-indexing scales linearly:

A = np.zeros((8, 8, 8))
%timeit A[4][4][4]

614 ns ± 17.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%timeit A[4, 4, 4]

236 ns ± 14.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Yep, that’s about what we expected: just over 200 nanoseconds for the tuple-indexing, and three times that for the list-indexing.

Extended experiments up to 10D

To make sure we’re getting the full picture, let’s repeat the previous experiments all the way up to 10-dimensional arrays. We’ll reduce the size in each dimension to 4 to make sure Python doesn’t return a MemoryError when allocating the array.

The following script times the same indexing statement we’ve tested previously, and passes along the locally created array A to the globals parameter as an alternative to defining it as setup.

By using min(repeat(...)) instead of timeit(...) we get the minimum time out of 5 runs. This means our results are less likely influenced by any other processes that happen to be running at the same time during our script.

These experiments make heavy use of the Python 3.6+ f-strings. For more information on them, see here or let me know if you want me to write a blog post about them

from timeit import repeat
import matplotlib.pyplot as plt

list_idxing = []
tupl_idxing = []
ndims = range(1, 11)
for ndim in ndims:
    A = np.zeros([4]*ndim)
    list_idxing.append(
        min(repeat(f"A{'[2]'*ndim}", globals={'A': A}))
    )
    tupl_idxing.append(
        min(repeat(f"A[{','.join(['2']*ndim)}]", globals={'A': A}))
    )

plt.plot(ndims, list_idxing, label='list indexing')
plt.plot(ndims, tupl_idxing, label='tuple indexing')
plt.xlabel("Number of dimensions")
plt.ylabel("Microseconds per loop")
plt.legend(loc=0)
plt.show()

The resulting graph does indeed show that the time of tuple-indexing remains roughly constant, while list-indexing keeps growing linearly.

We have created the arrays to be increasingly larger for each of the above timings. To rule out whether that has any effect, we can repeat the experiments with the largest array already allocated, and just index the relevant slices from them:

list_idxing = []
tupl_idxing = []
B = np.zeros([4]*10)
ndims = range(1, 11)
for ndim in ndims:
    list_idxing.append(
        min(repeat(f"B{'[2]'*ndim}", globals={'B': B}))
    )
    tupl_idxing.append(
        min(repeat(f"B[{','.join(['2']*ndim)}]", globals={'B': B}))
    )

plt.plot(ndims, list_idxing, label='list indexing')
plt.plot(ndims, tupl_idxing, label='tuple indexing')
plt.xlabel("Number of dimensions")
plt.ylabel("Microseconds per loop")
plt.legend(loc=0)
plt.show()

Comparing this graph to the previous one, it does seem like the time for tuple-indexing is consistently at ~500 nanoseconds now instead of slowly increasing from ~200. So there does seem to be some overhead involved that depends on the number of dimensions of the array you’re indexing.

Comparing the graphs for list-indexing, we can see a similar shift upwards in the amount of time required. The overall trend is still linearly increasing, and does not compare any more favourably against tuple-indexing.

Conclusion

Using tuple-indexing is significantly faster than nested indexing.

Python’s timeit: a How-To

2019-09-28T10:00:00+00:00

As the main topic of this blog is going to be timing experiments in Python, let’s start by having a look at the main tool I use to perform these experiments: Python’s built-in timeit module.

In this post I will give a short and minimal introduction of the timeit module, and go over the three main ways to use it to time small snippets of Python code.

Content

Timey Wimey Python: timeit
In a script: import timeit
Commandline: python -m timeit
IPython/Jupyter magics: %timeit
Summary
- Pros and Cons

Timey Wimey Python: timeit

What is Python’s timeit module?

This module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times.

– timeit documentation

In contrast to profiling the runtime of your entire program, the timeit module is better suited to time small snippets. We can divide these comparisons in three classes, as illustrated by the examples I will use throughout this blogpost.

Standalone one-liners
- ", ".join(str(n) for n in range(100))
- ", ".join([str(n) for n in range(100)])
- ", ".join(map(str, range(100)))

Standalone multi-liners

 x = []
 for i in range(1000):
     x.append(i)

```
 x = []
 for i in range(1000):
     x += [i]
```
(ignoring for the sake of example that the list-comprehension [i for i in range(1000)] is twice as fast)

Either of the above with some setup required
Setup: text = "sample string"; char = "g"
- char in text
- text.find(char)

In a script: import timeit

As timeit is a Python module, you can import it and write Python scripts for your tests. This method is the most self-documenting and repeatable way of writing your timing experiments, and makes it easy to store the results for further processing.

Before we can run any experiments, we need to import the relevant function:

from timeit import timeit

The function has the following specification:

time_spent = timeit(stmt='pass', number=1_000_000)

By running timeit() without any further arguments, we execute the default statement stmt='pass' a million times, showing the minimal overhead of Python’s pass statement.

>>> timeit()
0.0090144

Unless specified otherwise, timeit() will always run the default of one million iterations. This also means you have to pick a useful number yourself to make it finish in a reasonable amount of time that’s not too long or too short. The returned value is the number of seconds (as a float) it has taken to run the statement number times.

Standalone one-liners

Let’s start with comparing the simple one-liners. We give the statement we want to time as a string, and specify a custom number of iterations. Note that when your statement already deals with double-quoted strings, the whole string should be given single-quoted, or vice-versa.

>>> from timeit import timeit
>>> timeit('", ".join(str(n) for n in range(100))', number=10_000)
0.3343242
>>> timeit('", ".join([str(n) for n in range(100)])', number=10_000)
0.2085556
>>> timeit('", ".join(map(str, range(100)))', number=10_000)
0.1828172

From the given number and returned time in seconds, we can calculate that each of these lines took 33.4, 20.9 and 18.3 microseconds to execute respectively.

Standalone multi-liners

Sometimes the snippet you want to test will consist of multiple statements. Python allows you to put multiple statements on a single line with a semicolon x = 1; y = 2. Then you can simply run your test as explained above for the standalone one-liners. This does not work for snippets with indented code like if or for though.

To time these snippets that have to span multiple lines, you can simply give timeit() a multi-line string as argument. The beautiful and Pythonic way to do this is using Python’s multi-line strings. (The non-Pythonic way is to add \n characters in your regular strings.)

>>> from timeit import timeit
>>> timeit('''x = []
... for i in range(1000):
...     x.append(i)''', number=10_000)
0.7762808
>>> timeit("""x = []
... for i in range(1000):
...     x += [i]""", number=10_000)
0.7307259

Setup required

If there is some setup that only has to be run once, including it in a multi-line snippet means it’s executed at every iteration. Then you’d be measuring something you don’t want to measure! Instead, you can pass this setup statement as a separate argument to timeit():

time_spent = timeit(stmt='pass', setup='pass', number=1_000_000)

Setup snippets most commonly consist of non-indented code, so you can usually just use semicolons to join those statements in a regular string. Remember that the stmt argument comes before the setup argument if you don’t pass them as keyword-arguments, even though the setup would normally be first.

>>> from timeit import timeit
>>> timeit(stmt='char in text',
...        setup='text = "sample string"; char = "g"',
...        number=1_000_000)
0.0358061
>>> timeit('text.find(char)',
...        'text = "sample string"; char = "g"',
...        number=1_000_000)
0.1525827

Commandline: python -m timeit

Python’s timeit module can also be run as a commandline tool.

$ python -m timeit -n 10000 'pass'
10000 loops, best of 3: 0.0109 usec per loop

The usage is similar to the imported function as shown in the previous section, although the -n option is more optional than in the imported function. What I mean with that? If you don’t specify a number of times to run your snippet, it will try successive powers of 10 (10, 100, 1000, …) until the total time spent is at least 0.2 seconds. It also reports the actual time per iteration, instead of having to calculate that yourself. This is reported in nsec, usec, msec or sec, for nano-, micro-, mili- and whole seconds respectively.

$ python3 -m timeit 'pass'
100000000 loops, best of 3: 0.00856 usec per loop

Standalone one-liners

Testing one-liners with the commandline is as easy as replacing pass from the introduction with the code you want to test, and the timeit tool will automatically report the time in a nice human-readable format.

$ python -m timeit '", ".join(str(n) for n in range(100))'
10000 loops, best of 3: 23.3 usec per loop
$ python -m timeit '", ".join([str(n) for n in range(100)])'
10000 loops, best of 3: 20 usec per loop
$ python -m timeit '", ".join(map(str, range(100)))'
100000 loops, best of 3: 16.4 usec per loop

Standalone multi-liners

As before, the simplest way to test a snippet of multiple lines is to join the statements with a semicolon if no indentation is required. When indentation is required, the other option is to pass multiple strings as arguments to the command. Note that you still have to add the indentation properly yourself! This can get tricky to count if your indentation is more than a single level deep, but is usually not too hard.

$ python -m timeit 'x = []' 'for i in range(1000):' '    x.append(i)'
10000 loops, best of 3: 74.2 usec per loop

$ python -m timeit 'x = []' 'for i in range(1000):' '    x += [i]'
10000 loops, best of 3: 71.3 usec per loop

Setup required

To add some initial setup such as imports or variable declarations, we can use the -s option:

$ python3 -m timeit -s 'pass' 'pass'
100000000 loops, best of 3: 0.00856 usec per loop

The setup does come before the statement to test in this case, so that makes it a bit more intuitive to read.

$ python -m timeit -s 'text = "sample string"; char = "g"' 'char in text'
10000000 loops, best of 3: 0.0335 usec per loop
$ python -m timeit -s 'text = "sample string"; char = "g"' 'text.find(char)'
10000000 loops, best of 3: 0.147 usec per loop

IPython/Jupyter magics: %timeit

When working in the IPython interactive shell, or in a Jupyter notebook with the IPython kernel, you have access to the so-called ‘magic commands’. For timeit, there is the %timeit magic. It simply takes the code you type after it and runs the timeit command on it! Like magic!

One small caveat is that the outcome is reported as a mean +/- standard deviation of multiple repetitions, while the original Python documentation suggests always using the minimum.

Standalone one-liners

For simple one-liners, just type the code as you would normally, and type %timeit before it. That’s all!

%timeit ", ".join(str(n) for n in range(100))

24.1 µs ± 314 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit ", ".join([str(n) for n in range(100)])

20.4 µs ± 266 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit ", ".join(map(str, range(100)))

18.2 µs ± 777 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Standalone multi-liners

The %timeit magic doesn’t quite work for multi-line snippets though. Why? Because magics with a single % are line-magics. For cell-magics, you just have to add another % to make it %%timeit. Then it will time all the code in your cell. No further difficulties whatsoever!

%%timeit
x = []
for i in range(1000):
    x.append(i)

75.1 µs ± 39.8 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%%timeit
x = []
for i in range(1000):
    x += [i]

72.4 µs ± 2.33 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Setup required

Now you might be thinking “but if this %timeit just takes your code and times it, where do I put my setup command?” The answer: just run it in some previous cell! As IPython already takes care of passing your code on to the timeit module properly, it also automatically passes along all current global variables. So we can simply first run a cell with our setup:

text = "sample string"
char = "g"

And use the simple %timeit magic to time the code we are actually interested in, without specifying which setup is associated with it.

%timeit char in text

49.9 ns ± 1.09 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

%timeit text.find(char)

178 ns ± 24.6 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Summary

I hope to have shown in this article how you can easily use Python’s timeit module to measure execution times of your snippets, whether for fun or profit. Although each has it’s pros and cons (see below), I will personally recommend to use IPython’s (%)%timeit magics as they are the most intuitive to use: just write the code as you would normally with the %timeit magic in front.

Pros and Cons

from timeit import timeit
- Pros:
  - Time taken returned within Python process, so can easily be used in further processing
- Cons:
  - Must specify number of iterations manually
  - Must manually calculate amount of time spent per iteration
python -m timeit
- Pros:
  - Gives nice and clear output
- Cons:
  - Multi-line snippets are less intuitive
  - Commandline can be ‘scary’ to some
(%)%timeit
- Pros:
  - Easiest to use
  - Setup is dealt with automatically
- Cons:
  - Needs IPython and/or Jupyter installed
  - Gives mean +/- standard deviation as result, while Python’s documentation suggests using the minimum

blog.init()

2019-09-21T09:13:36+00:00

So, it looks like I’ve started a blog!

I’m going to try to post all my small timing experiments (in Python 3.6+ of course) here for now, but who knows what other ideas I may later come up with.