OK, for anybody who's interested here's briefly what's going on...
Python is essentially a lexically scoped language where for any given function there are effectively three classes of scope:
- Local scope (i.e. within the current function).
- Enclosing scope (i.e. within the enclosing function, if any).
- Global scope (i.e. defined at the module level).
This is ignoring a couple of warts such as classes, which have their own scope which is more or less unconnected from other scopes, and the builtin scope, where all the language builtins live.
So, while a function is executing if you refer to a variable then it's searched first in the local scope, then in each enclosing scope in turn and finally in the global scope. The first match is always used. The other wrinkle to note is that when a function is defined, it stores a reference to its enclosing scope to make sure they don't disappear if the enclosing function terminates - the technical term for this is a closure. It's what allows things like this to work:
return arg + add_amount
my_add_5_func = create_adder_function(5)
# This will print '22'
The key point to note here is that in Python (unlike some other lexically scoped languages) only a function introduces its own scope and not constructs such as
for loops. This means that when you create a closure, you're just getting a reference to the enclosing scope. In the code in my first post, each of the functions defined in the loop ends up getting a reference to the same enclosing scope - in that example, the global scope, but it would work the same way within a function. This is because each repetition around the loop does not create its own scope, it merely executes in the local scope of the current function (or the global scope in that particular case).
This means that when the
lambda expressions are executed and refer to the variable
i, they're all referring to the same
i in the current scope and hence they all get the same value. If the inner function had been defined in a wrapper function as in my adding example just above, everything would work fine. This is because the wrapper function would be called each time, hence creating a new closure and a new instance of the counter variable. But in a
for loop no new closure, hence all the same value.
Finally, the question as to why the final code snippet works. This is because the round-bracket version of the list comprehension declares a generator expression which is lazily evaluated. Therefore, when
functions is defined in that example nothing has actually been evaluated - the
xrange generator expression is ready to generate integers, but it hasn't done yet.
When the second line then evaluates all the functions, the generator expression starts iterating through the
xrange one item at a time. Importantly, the
lambda expressions are being created and evaluated at the same time and hence the variable
i has the value you'd expect each time. The problem comes when you create all the closures first, and then evaluate the functions.
I'm not sure how good a job I did of explaining that, if anybody's curious feel free to ask. You might also read the blog post I did about this earlier this week.
By the way, thinking about closures as references to scopes as opposed to "freeze-frames" of values makes some things easier to understand. For example, the following Python 3 code:
x = 1
x += 1
x *= 2
return (increment, double)
inc, doub = outer()
If you execute this (with
python3 of course) then you'll see that both inner functions are sharing and modifying the same closure - they don't each have their own copy.