Functions making functions

Dynamic Languages

One of the hallmarks of dynamic languages is that they try to put as much as possible off to the last minute. If we do the following

>>> 5 + 4
9
>>> "this" + "that"
'thisthat'

The meaning of "+" is not decided until the arguments are evaluated. This is true even if it is used in a function that has been compiled. Only at runtime is the meaning of "+" decided.

>>> def addem(a,b): return a+b
>>> addem(5,6)
11
>>> addem("Hello","World")
'HelloWorld'

Nesting Functions

Actually, even the compilation of functions is, in fact, a "runtime" activity that mostly happens as we import a module. So it is not too surprising that we can nest function definitions Like the following. Source for sample1.py

# sample1.py
def outer () :
   print "Outer 1"
   def inner () :
       print "Inner 1"
   print "Outer 2", inner
   inner()
print "Import", outer

Let's play with this a bit.

>>> import sample1
Import <function outer at 0x9d19fb4>
>>> sample1.outer()
Outer 1
Outer 2 <function inner at 0x9d8f56c>
Inner 1

Look closely at the timing and order of the print statements. The compilation of inner happens only when outer is run. The name "inner" is a local within the function outer so it is discarded once outer has returned control to its caller. In fact the whole compiled inner function is tossed out for garbage collection.

Notice too, that when we "print" a function, it gives us the name originally assigned when it's defined along with its memory address in hexidecimal.

Let's consider a slightly more complex example. Source for sample2.py

# sample2.py
def outer () :
   print "Outer 1"
   x = 42
   def inner () :
       print "Inner 1", x
   return inner

And now let's play with it

>>> import sample2
>>> sample2.outer()
Outer 1
<function inner at 0x9d1e224>
>>> sample2.outer()()
Outer 1
Inner 1 42

Notice that inner could access the local variable x in outer. This scoping rule is usually necessary for inner functions to be useful.

This time, outer returned the function it created and we can then use it however we want. Most of the time we'll assign the function to a local variable so we keep it.

>>> f1 = sample2.outer()
Outer 1
>>> print f1
<function inner at 0x9d1e87c>
>>> f1()
Inner 1 42

The function inner is not discarded since it can be reached through "f1", and because of that, the variable x inside inner is also retained.

And, (this is an "Ah-ha" moment) x can ONLY be reached through the inner function referenced by "f1". It is enclosed in the function. This is called a closure.

But "x" is pretty boring unless it can be manipulated. Let's try a variation. Source for sample3.py

# sample3.py
def outer () :
   print "Outer 1"
   x = 42
   def inner () :
       print "Inner 1", x
       x += 1                # This added
   return inner
>>> import sample4
>>> f2 = sample3.outer()
Outer 1
>>> f2()
Inner 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sample3.py", line 6, in inner
    print "Inner 1", x
UnboundLocalError: local variable 'x' referenced before assignment
>>>

Oops! Since inner is changing the value of "x", the Python compiler assumes "x" is a local variable to inner. And it doesn't work to make "x" global because globals must be defined at the module level. So much for enclosing it.

But there is a cute way to do what we want. If "x" instead of referencing a number (or string), references a mutable like a list, a dictionary or an object, the mutable can be mutated (changed). Let's have it be an object that could provide a whole private namespace. Source for sample4.py

# sample4.py
def outer () :
   print "Outer 1"
   class Dummy : pass   # just to build an empty instance
   x = Dummy()
   x.value = 42         # Create an attribute with initial value
   def inner () :
       print "Inner 1", x.value
       x.value += 1     # Update the attribute
   return inner
>>> import sample4
>>> f3 = sample4.outer()
Outer 1
>>> f3
<function inner at 0x9a12d14>
>>> f3()
Inner 1 42
>>> f3()
Inner 1 43
>>> f3()
Inner 1 44

The class Dummy was created to simply provide an empty instance that we can add attributes to and manipulate them at will. These attributes could include functions as well as variables. The instance and everything it contains is completely tucked inside the closure.

Here's a variation on this code. You are certainly familiar with the ticket dispensers one finds in waiting rooms. You pull a number and then you can sit in line instead of standing. Let's make a function that produces dispensers that simply can't be reset. No cheating allowed. Source for makeDispenser.py

# makeDispenser.py
#
def makeDispenser (startNum) :
   class NameSpace : pass
   myStuff = NameSpace()
   myStuff.nextNum = startNum
   def dispenser () :
       thisNum = myStuff.nextNum
       myStuff.nextNum += 1
       return thisNum
   return dispenser
>>> import makeDispenser
>>> aDisp = makeDispenser.makeDispenser (101)
>>> bDisp = makeDispenser.makeDispenser (201)
>>> print "Get ticket from A", aDisp()
Get ticket from A 101
>>> print "Get ticket from B", bDisp()
Get ticket from B 201
>>> print "Get ticket from A", aDisp()
Get ticket from A 102
>>> print "Get ticket from B", bDisp()
Get ticket from B 202

Getting ready for decorators

Python decorators are a syntactic mechanism to make some change in the original definition of a function. Suppose we have a function that computes the tip for a restaurant bill.

>>> def tip(amount) : return amount*.15
>>> tip(10)
1.5

Now, let's assume we want to double tips. We might write a function called "bigTip" like the following.

>>> def double(x)   : return x+x
>>> def bigTip(amt) : return double(tip(amt))
>>> bigTip(10)
3.0

Now, if we want to force every call to the tip to generate a big tip we might think we could simply do the following.

>>> tip = bigTip
>>> print tip(10)
RuntimeError: maximum recursion depth exceeded

Well, that won't work. If you look carefully you will see the recursion loop that keeps tip calling itself. But we can reach back a bit and make double operate on the function definition level. We'll have it create a new function that calls the original function passing through the parameter and then returns the result*2.

>>> def double (func) :
...     def wrapper(n) :
...             return func(n) * 2
...     return wrapper
...
>>> def tip(amount) : return amount*.15
>>> bigTip = double(tip)
>>> bigTip(10)
3.0
>>> tip = bigTip
>>> tip(10)
3.0
>>> print tip
<function wrapper at 0x9301cdc>

And this works. Notice that the varibable "tip" now points to the generated function wrapper created inside double. We can no longer even access the orginal tip. It's been swallowed up in a closure.

But so far, double is limited to operating on functions that take a single parameter. If we want to generalize the function parameters we need to use the special Python syntax to roll up the arguments of a function into a tuple. The rolled up tuple is generally given the name "args" and special syntax is to precede a variable name with a "*". Here's how it works.

>>> def xx(*args) : print args
>>> xx(1,2,3)
(1, 2, 3)
>>> xx("this", "is", [5,4,3,2,1])
('this', 'is', [5, 4, 3, 2, 1])

The leading "*" forces the roll-up and is valid only in function parameters. There is also a "**" which rolls up keyword arguments into a dictionary. It is generally given the name "kwargs".

>>> def xx(*args, **kwargs) : print args, kwargs
...
>>> xx(1,2,3)
(1, 2, 3) {}
>>> xx(1,2, debug=True)
(1, 2) {'debug': True}

Used together, we can pass arguments intact through an intermediate function. Let's rewrite double above which could be used for any function.

>>> def double (func) :
...     def wrapper(*args, **kwargs) :
...             return func(*args, **kwargs) * 2
...     return wrapper
...
>>> def tip (amt) : return amt * .15
...
>>> tip(15)
2.25
>>> tip = double(tip)
>>> tip(15)
4.5

So here our single parameter is passed through as it should be.

Python decorators (finally)

Now, all of this is made much more intuitive using the decorator syntax.

>>> @double
... def tax(amt) : return amt*.09
>>> tax(100)
18.0

The order of events is this. First the function tax is defined, and then it is "decorated" by double. The variable "tax" is then set to whatever the decorator function (double) returns. In this case hiding the original in a closure and manipulating its computation. As we will see other things are possible too.

Notice what happens when we do the following. The secret peeks through.

>>> print tax
<function wrapper at 0x8528534>

There are two more items of interest here. One is that decorators may be stacked.

>>> @double
... @double
... def tip (amt) : return amt * .15
...
>>> tip(40)
24.0

A 60% tipper!! .15*2*2.

So, actually, a decorator call like "@double" expects the next line will either start a function definition or supply another (inner) decorator.

Actually, the "@" starting a line expects an expression that evaluates to a function. Now so far, that has just been a variable referencing a function that's already defined. But here is where we can take a little trip down the rabbit hole.

Consider the following decorator. Notice that it has an argument. Source for nTimes.py

# nTimes.py
#
def nTimes(n) :
   "Create a decorator function that muliplies another functions return value by n"
   print "In function ntimes (arg=%s) creating decorator" % n
   def decorator(target) :
       print "In decorator creating the wrapper"
       def wrapper(*args, **kwargs) :
           print "Calling decorated function"
           return target(*args, **kwargs) * n
       return wrapper
   return decorator

Look carefully and see that calling nTimes creates and returns the function decorator. Calling decorator creates and returns wrapper, the decorated version of the function definition that follows.

>>> from nTimes import nTimes
>>> nTimes(3)
In function ntimes (arg=3) creating decorator
<function decorator at 0x7f2e069066e0>
>>> @nTimes(3)
... def tip(amt) : return amt * .15
...
In function ntimes (arg=3) creating decorator
In decorator creating the wrapper
>>> tip(10)
Calling decorated function
4.5

Notice carefully the order of the print statements.

Decorators for Side Effects

So far, the decorators we've written have modified a function. But that is not their only use nor even the major one. Often the decorator is there just for the side effects.

Here's an example.

Suppose we have a little language processer, like some of the Python for Fun projects and we want to tie or register a token, say "+", to a function that processes the token. A common thing to do might be the following.

Define the function. Let's call it "add"

>>> def add(a,b) : return a+b

and then have a dictionary that will map the token "+" to our function

>>> dispatch = {'+': add,  ... }

The dictionary might hold more information, like the number of arguments to pass to add.

This is where decorators can be quite elegant. This is how the equivalent might look.

@register('+')
def add(a,b) : return a+b

Now, the function register, like nTimes above, must create a decorator to apply to the function add. Here's how that might look.

# register.py
dispatch = {}
def register(token) :
   def decorator(func) :
       dispatch[token] = func
       return func
   return decorator

Let's try it out.

>>> from register import *
>>> print dispatch
{}
>>> @register('+')
... def add(a,b) : return a+b
...
>>> print dispatch
{'+': <function add at 0x7fe45a2352a8>}
>>>

A Real Example

The bottle.py minimilist web application framework uses decorators in a interesting way that is quite close to the previous example above. For example, the following

@route("/greet")
def hello () :
       return "<html>Hello, whoever you are</html>"

@route("/gruess/<name>")
@route("/greet/<name>")
def greet(name) :
       return "<html>Hello, nice to meet you, %s</html>" % name

will essentially "register" the function hello to a url ending with "/greet". A URL ending in either "gruess/lukas" or "greet/lukas" will call the function greet with "lukas". Notice that all 3 "@route" invokations take an argument which means the function route actually creates a different decorator function each time.

When the application is running, the string "<name>" works like a variable and is replaced by the actual word in the url. When the url is routed an *args list is built from the pattern matching and passed to the function greet. Here is what you get when you run the above as part of a mini-server (the program will have all of another 4 lines or so of Python).

http://localhost:8080/greet                # Call from your browser
Hello, whoever you are                     # returned to your window
-
http://localhost:8080/gruess/mickey
Hello, nice to meet you, mickey

It is, of course, important that the variables in the url align with the parameters in the decorated function.

The Point Being ...

So, decorators are cute. Are they necessary? There was some resistance to adding this syntax to the Python language.

Without decorators, the bottle code might look more like this.

def hello () :
       return "<html>Hello, whoever you are</html>"

def greet(name) :
       return "<html>Hello, nice to meet you, %s</html>" % name

route("/greet",         hello)
route("/gruess/<name>", greet)
route("/greet/<name>",  greet)

Certainly not wildly different. And the code within bottle.py would be much less obscure.

Still, the idea is that you will not be mucking with bottle.py, just using it. And the folks at bottle have already done the debugging of that piece.

But, if I were writing an application I would probably find extending it with custom decorators overkill. Because anyone maintaining the code, including myself, would probably have to muck with the decorators as well as the other code. This would increase the complexity of the code overall. And the major objective of programming should be simplicity.

.