Python objects are scattered all over the place [on the heap] … performance degradation is the price for Python’s simple memory model. … NumPy is optimized for making use of blocks of contiguous memory.
You can substantially reduce memory fragmentation and GC pressure with only the standard library array module and memoryview builtin type, if your data suits that pattern. This is particularly useful to implement zero-copy algorithms for IO processing; as soon as the buffer is in memory anywere you just take pointers to slices rather than creating new objects.
JIT implementations of Python (PyPy, Pyjion, etc) are also usually pretty good at reducing the perf impact of Python’s memory model, at least if your program is reasonably sensible about what and when it allocates.
With progn and :=, it’s possible to combine multiple statements into one, so effectively create a lambda with multiple statements.
Sounds like you’re partway to updating onelinerizer.com for Python 3!
Sounds like you’re partway to updating onelinerizer.com for Python 3!
For the avoidance of doubt, the “obvious way” to do this (for an acculturated Python programmer) is with a nested def, which makes the progn thing non-obvious and therefore unpythonic. I strongly hinted at the obvious approach here, but konstell latched onto using a lambda instead (probably because she didn’t realize that named functions could also be closures). I saw a teaching opportunity in this, so I rolled with it. I got to dispel the myth that lambdas can only have one line and also introduced assignment expressions. I was going to get around to the obvious way, but we ran out of time.
With progn and :=, it’s possible to combine multiple statements into one, so effectively create a lambda with multiple statements.
konstell is using the terminology is a little imprecisely here. In Python, an “expression” evaluates to an object, while a “statement” is an instruction that does not evaluate to an object (not even None). Most statement types can contain expressions, however expressions cannot contain statements (exec() doesn’t count).
One of the simplest types of statements in Python is the “expression statement”, which contains a single expression and discards its result. A progn() expression can discard the results of subexpressions in a similar way, making them act like expression statements, but they are not technically Python statements. We also found an expression substitute for an assignment statement. It’s ultimately possible to use expressions for everything you’d normally use statements for, but this is not the “obvious way” to do it.
See my Drython and Hissp projects for more on “onlinerizing” Python.
Nice! I always enjoy reading these logs :-)
Numpy also has the enormous advantage of implementing all the numeric operators in C (or Fortran, or occasionally assembly. (If you want hardware accelerators, interop is a promising work in progress)
You can substantially reduce memory fragmentation and GC pressure with only the standard library
array
module andmemoryview
builtin type, if your data suits that pattern. This is particularly useful to implement zero-copy algorithms for IO processing; as soon as the buffer is in memory anywere you just take pointers to slices rather than creating new objects.JIT implementations of Python (PyPy, Pyjion, etc) are also usually pretty good at reducing the perf impact of Python’s memory model, at least if your program is reasonably sensible about what and when it allocates.
Sounds like you’re partway to updating onelinerizer.com for Python 3!
For the avoidance of doubt, the “obvious way” to do this (for an acculturated Python programmer) is with a nested
def
, which makes theprogn
thing non-obvious and therefore unpythonic. I strongly hinted at the obvious approach here, but konstell latched onto using alambda
instead (probably because she didn’t realize that named functions could also be closures). I saw a teaching opportunity in this, so I rolled with it. I got to dispel the myth that lambdas can only have one line and also introduced assignment expressions. I was going to get around to the obvious way, but we ran out of time.konstell is using the terminology is a little imprecisely here. In Python, an “expression” evaluates to an object, while a “statement” is an instruction that does not evaluate to an object (not even
None
). Most statement types can contain expressions, however expressions cannot contain statements (exec()
doesn’t count).One of the simplest types of statements in Python is the “expression statement”, which contains a single expression and discards its result. A
progn()
expression can discard the results of subexpressions in a similar way, making them act like expression statements, but they are not technically Python statements. We also found an expression substitute for an assignment statement. It’s ultimately possible to use expressions for everything you’d normally use statements for, but this is not the “obvious way” to do it.See my Drython and Hissp projects for more on “onlinerizing” Python.