Thursday, May 30, 2013

Parallel xasc

Sorting a table can be divided into two parts: determining the new order for the rows, and applying that ordering to the columns. While the former can’t be parallelized in q, the latter can. I don’t have any hard numbers handy at the moment, but with large tables and under the right conditions, I’ve seen noticeable speedups.

Note, BTW, that you can’t (and shouldn’t) write to disk from inside a peach, so this is only applicable to an ordinary in-memory table sort, not the on-disk variety (`c xasc`:t).

q)pxasc :{(count keys y)!flip{y x}[ iasc(raze x)#0!y]peach flip 0!y}
q)pxdesc:{(count keys y)!flip{y x}[idesc(raze x)#0!y]peach flip 0!y}