An Apprentice Experiment in Python Programming, Part 2

konstell and gilch
29 Jul 2021 7:39 UTC
30 points
18 comments10 min readLW link

[Epistemic status: me trying to recall what happened in a 4.5-hour pair-programming session, during which I was focused on solving problems. This post was reconstructed from scratch code files, written messages (the session was over voice call) and my python shell history.]

Previously: https://​​www.lesswrong.com/​​posts/​​kv3RG7Ax8sgn2eog7/​​an-apprentice-experiment-in-python-programming

Three days ago gilch and I had another session on Python Programming, where we continued talking about decorators.

Lambda decorator

So far, all of the decorators we’ve seen have been functions that take in another function as parameter, then used by the syntax @<decorator_name>. Turns out we can use a lambda function as a decorator too:

@lambda f: print("hi")
def add(a, b):
    return a + b

When this file gets run in Python 3.9 interactive mode, the output is

hi

and when I call the add function:

>>> add(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable

Gilch asked me to write out the desugared version of this code. I wrote add = print("hi"), which was not correct. Gilch gave me the correct desugared code:

def add(a, b):
    return a + b

add = (lambda f: print("hi"))(add)

Partial

We did a brief introduction to partial. We had code from last time:

def subtract(a, b, c):
    return a - b - c

Accessing doc from within python shell:

>>> from functools import partial
>>> help(partial)

Using partial to create a function with the first and second arguments passed in:

>>> partial(subtract, 1, 2)
functools.partial(<function subtract at 0x7f34649af3a0>, 1, 2)
>>> partial_sub = partial(subtract, 1, 2)

Gilch pointed out that the line above, partial_sub = partial(subtract, 1, 2), could’ve been written as partial_sub = _ because in python shell, _ returns the last value that’s not None.

Testing the partial function:

>>> partial_sub(3)
-4

1 − 2 − 3 equals −4, so we’re getting what we expected.

Gilch then pointed out that arguments can be passed into partial as keyword arguments:

>>> partial(subtract, b=0, c=2)
functools.partial(<function subtract at 0x7f34649af3a0>, b=0, c=2)
>>> _(10)
8

Finally, there’s a way to implement partial functions using lambda:

>>> lambda x: subtract(x, 0, 2)
<function <lambda> at 0x7f346474ae50>
>>> _(10)
8

Running Bash in Python

While running help(partial) in the python shell, the topic of using less came up. I asked gilch if it was possible to pipe outputs from python shell to bash tools, gilch answered that we could not do it directly, but just like how bash can run python, python can also run bash:

>>> import os
>>> os.system("bash")

Now we’re in a bash shell, run by python:

~/code_stuff/python_scratch $ ls | less
~/code_stuff/python_scratch $ exit
exit
0
>>>

At the last line, we’re back in python shell again.

Lambda functions

Gilch gave me a number of lambda expressions in increasing complexity, asked me to explain what they did, then verify:

  • (lambda x: x+x)(3) evaluates to 6, this one was straightforward

  • (lambda x: lambda y: x + y)(2)(3) I assumed that this statement expanded inside out (i.e.2 was passed in as y and 3 was passed in as x) so I concluded that this would evaluate to 5. I got the correct answer, but my mistake was exposed by the next example:

  • (lambda x: lambda y: x + y)('a')('b') Using the same logic, I concluded that the result would be 'ba', but the actual result was 'ab'. Why did I make this mistake? Gilch gave me the next question:

  • (lambda x: lambda y: x + y)('a') evaluates to a lambda. A lambda of x or y? The first part of the expression, (lambda x: lambda y: x + y), is a lambda function that takes in a parameter x, and returns a lambda function lambda y: x + y. This lambda function that takes in x is then called with argument 'a'. The return value is lambda y: 'a' + y. And then it made sense why the previous question evaluated to 'ab' instead of 'ba'.

I think what gilch was getting at here was the syntax of function calls other than the format of f(x). Gilch also at some point pointed out the difference between a callable and a function.

Preprint

Gilch gave me the next challenge: write a decorator preprint such that it can be used in the following way:

@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")

>>> bob()
hi
Bob
>>> alice()
hello
Alice

I struggled here. I attempted to write out some vague idea in my head:

def greet_by_name(salutation, name):
    print(salutation)
    print(name)

def bob(salutation):
    from functools import partial
    return partial(greet_by_name, name="Bob")

Gilch reminded me that the use case was already specified; functions alice and bob were not to be modified. They then suggested that I wrote out the desugared version of the code. I struggled and gave an incorrect answer again until gilch provided me with the correct desugaring:

@preprint("hi")
def bob():
    print("Bob")

is equivalent to bob = (preprint("hi"))(bob).

Understanding decorator de-sugaring

As you can see, at this point, I was struggling with desugaring. So gilch presented me with a formula:

@<decorator expression>
def <name>(<args>):
    <body>

is equivalent to

def <name>(<args>):
    <body>

<name> = (<decorator expression>)(<name>)

Preprint

My memory of what exactly happened here is spotty, perhaps because I was focused on finding out how to write the preprint decorator. Eventually I came up with something like this:

def preprint(greetings):
    from functools import partial
    def greet_by_name(greeting, foo):
        def wrapper():
            print(greeting)
            foo()
        return wrapper
    return partial(greet_by_name, greetings)

This worked, but you may be able to see from the way I passed in greetings into greet_by_name, I was confused about why there needed to be 3 layers of definitions. Gilch gave me an alternative solution, and explained the pattern:

def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

Gilch explained that, for decorators that take in parameters, they are like a factory that produces different decorators based on input. Thus the first layer of definition processes the input, the second layer of definition is the decorator itself, and the innermost wrapper function is what replaces the function that ends up being decorated. This

@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")

is equivalent to

greet_by_name_hi = preprint("hi")
greet_by_name_hello = preprint("hello")
@greet_by_name_hi
def bob():
   print("Bob")
@greet_by_name_hello
def alice():
    print("Alice")

At this point, I was not quite getting the 3-layer structure—I identified that there was a pattern of factory-decorator-function but I was mostly mimicking the pattern. This lack of understanding would be exposed in a later example.

wraps

In the previous example, we had the decorator preprint:

def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

Gilch instructed me to add a doc string to a function being decorated by this decorator. So we have:

@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")

What happens when we try to access the doc string of bob?

>>> help(bob)

Help on function wrapper in module __main__:


wrapper()
(END)

Gilch then introduced wraps. According to the documentation, it calls update_wrapper to reassign some attributes of the wrapped function to the wrapper function.

from functools import wraps
def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        @wraps(function)
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")

>>> help(bob)
Help on function bob in module __main__:

bob()
    prints 'bob'
(END)

map

Gilch asked me if I knew what map in python did. I said map took in a function and an array and return the result of the function applied to each element of the array. Gilch responded that it was mostly correct, with the caveat that map took in an iterable instead of an array.

Gilch gave me a test case for which I needed to implement the decorator:

@<something>
def double(x):
    return x+x
assert list(double) == ['aa','bb','cc','dd']

I came up with

@lambda f: map(f, 'abcd')
def double(x):
    return x+x

When we run assert list(double) == ['aa','bb','cc','dd'], python did not complain. However when we ran list(double) again, we saw an empty array:

>>> list(double)
[]
>>> double
<map object at 0x7f3ad2d17940>
>>> list(double)
[]

Gilch explained that double was not an array, it was an iterable. After we had iterated through double in the assert statement, it became empty.

Callable

Gilch illustrated the difference between a function and a callable:

def my_if(b, then, else_):
   return then if b else else_

in which then if b else else_ is a callable. Two use cases:

>>> my_if(1<2, print('yes'), print('no'))
yes
no

>>> my_if(1<2, lambda:print('yes'), lambda:print('no'))()
yes
>>> my_if(1>2, lambda:print('yes'), lambda:print('no'))()
no

In the first case, print("yes") and print("no") are not callables (they get run and evaluated to None) whereas the lambda:print('yes') and lambda:print('no') in the second case are callables.

Similarly, when we put the print statements in functions, the functions are callables:

def print_yes():
    print("yes")

def print_no():
    print("no")

>>> my_if(1<2, print_yes, print_no)
<function print_yes at 0x7fa36ffbb280>
>>> my_if(1<2, print_yes, print_no)()
yes

Branching, with 1 branch

Gilch illustrated how we could use decorator for branching:

def when(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator

>>> @when(1<2):
... def result():
...     print("Yes.")
...     return 42
... 
Yes.
>>> assert result == 42
>>> @when(1>2):
... def result():
...     print("Yes.")
...     return 42
... 
>>> assert result == None

Branching, with 2 branches

Gilch asked me how I would implement branching such that if a given condition is true, one branch gets evaluated; if condition is false, another branch gets evaluated. This would essentially mean we decorate two functions. I came up with this solution:

def if_(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator

def else_(condition):
    def decorator(function):
        if not condition:
            return function()
    return decorator

condition = 1<2

@if_(condition)
def result():
    print("Yes.")
    return 42

@else_(condition)
def print_no():
    print("no!")

$ python3.9 -i map.py
Yes.
>>> result
42

And with condition = 1>2:

$ python3.9 -i map.py
no!
>>> result
>>>

The problem of this solution was too much code repetition. Also, I used 2 decorators instead of 1. Gilch said one way to do this was to decorate a class instead of a function.

Decorating a Class

Gilch gave me the test case, and asked me to write the decorator:

@if_(1>2)
class Result:
    def then():
        print("Yes.")
        return 42
    def else_():
        print("No.")
        return -1

Eventually I came up with the decorator:

def if_(condition):
    def decorator(c):
        if condition:
            return c.then()
        else:
            return c.else_()
    return decorator

$ python3.9 -i decorating_a_class.py 
Yes.
>>> Result
42

with condition reversed (@if_(1<2)):

$ python3.9 -i decorating_a_class.py 
No.
>>> Result
-1

Decorating a Class, Example 2

Gilch gave me another test case for which I needed to write the decorator:

@sign(x)
class Result:
    def positive():
        print('pos')
    def zero():
        print('zero')
    def negative():
        print('neg')

I modified the solution from the previous question to answer this one:

def sign(x):
    def decorator(c):
        if x > 0:
            return c.positive()
        elif x == 0:
            return c.zero()
        else:
            return c.negative()
    return decorator

This was basically the same example as the previous one in the sense that I didn’t need to change the structure of the decorator, only the specific functions within the innermost layer.

Making Branching Statements without Using if

I asked gilch if there were ways to make if statement without using if. Gilch gave me some examples:

  • return [c.then,c.else][not condition]()

  • return {True:c.then,False:c.else_}[bool(condition)]()

A More Difficult Example

Gilch gave me another test case:

@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42

And when the first line is changed to @if_(1>2), assert result == -1 should pass.

I figured out how to make the first branch work but not the second. At this point, it was getting quite late in my timezone and by brain was slowing down after hours of coding, so I told gilch I’d work on it the next day.

When I picked this problem up the next day, all of the wobbly parts of my understanding came back to bite me. I was running into a bunch of error messages like TypeError: 'NoneType' object is not callable and TypeError: <function> missing 1 required positional argument: 'greetings' (or TypeError: <function> takes 1 positional argument but 2 were given), and since the error messages would point at the line where result was defined instead of the line within the decorator where the problem was, I was struggling a lot debugging my code. I poked around the internet a lot to look at other examples as an attempt to understand how to expand decorators in my head, and eventually came across this slide from a PyCon talk that made things a bit more clear to me.

I sent gilch my solution:

def if_(condition):
    def decorator(function):
        if condition:
            def wrapper(*x):
                return function()
            return wrapper
        else:
            def decorator2(function):
                return function()
            return decorator2
    return decorator

# original test cases below
@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42

$ python3.9 -i cases.py
yes
>>> result
42

With the condition flipped:

$ python3.9 -i cases.py
no
>>> result
-1

Gilch noticed my confusion: “The way you named things indicates a lack of understanding. Naming should have been more like this.”

def if_(condition):
    def decorator(function):
        if condition:
            def decorator2(f):
                return function()
            return decorator2
        else:
            def decorator2(f):
                return f()
            return decorator2
    return decorator

Alternatively, gilch provided two more solutions:

def if_(condition):
    def decorator(function):
        if condition:
            return lambda _: function()
        else:
            return lambda f: f()
    return decorator

def if_(condition):
    def decorator(then_func):
        def decorator2(else_func):
            return (then_func if condition else else_func)()
        return decorator2
    return decorator

“They’re both a decorator you apply to the second definition, regardless of which branch is taken. So calling the first one a wrapper is the wrong way to think of it,” explained gilch.

Observations

  1. The concept of decorator being syntactic sugar of calling the decorator on a function then reassigning the result to the variable that used to be the function somehow didn’t stick for me. When gilch asked me to write the desugared version of functions, I kept thinking about further expanding the functions.

  2. There are a few things that I was not used to thinking about: functions of functions, functions that use variables that are in scope but don’t get passed in as parameters, functions that return callables. All these made decorators—especially parameterized decorators—”hard to fit into my head at once,” using gilch’s remark.

  3. Gilch’s teaching was primarily puzzle-based, which is quite different from the lecture-based learning that I have done. This is pretty cool, because for most of my life, solving problems has been in the context of some evaluation. Stress sucked the fun out of puzzles for me when I had trouble decoupling my self worth and the outcomes of those evaluations. Now I’m solving puzzles for no other reason beyond learning, I’ve realized this is also an opportunity to unlearn.

konstell and gilch
29 Jul 2021 7:39 UTC
30 points
18 comments10 min readLW link