Forums

Quite a tricky one this week - I'll be impressed if anybody can pick this one apart!

Dinsdale is trying to write a wrapper around a Python dictionary - so far, he's just written the constructor and destructor with some debug in the latter and saved it in a file dictwrapper.py:

# dictwrapper.py

class DictWrapper(object):

    def __init__(self, init_dict):
        self.value = init_dict.copy()

    def __del__(self):
        print "DEBUG: Destroyed with " + repr(self.value)

At this point he decides to test what he's written so far, so he fires up an interactive interpreter and tries to instantiate his class. Unfortunately he doesn't quite understand how to create a Python dictionary, and he tries to use a list of 2-tuples:

>>> import dictwrapper
>>> my_init_dict = [(1,2), (3,4)]
>>> instance = dictwrapper.DictWrapper(my_init_dict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "dictwrapper.py", line 5, in __init__
    self.value = init_dict.copy()
AttributeError: 'list' object has no attribute 'copy'

This exception isn't surprising because the list object doesn't have the required copy() method that __init__() is trying to call. However, Dinsdale is a bit confused, so he tries to check whether his instance actually got created:

>>> print instance
Exception AttributeError: "'DictWrapper' object has no attribute 'value'" in <bound method DictWrapper.__del__ of <dictwrapper.DictWrapper object at 0x7fa5267bbdd0>> ignored
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'instance' is not defined

Now he's really confused. Are you? Can you explain the first exception message which appeared after the print statement and why it appeared where it did?

No takers? (^_^)

Yes: I'm confused :-)

I will now have to fire up an interpreter to play. Damn you Cartroo...

Heh, sorry matey! (^_^)

My advice: examine the error message carefully, it explains where it's being thrown from (and it's not the same point as the following traceback!). Convince yourself why that exception occurs. Then tackle the trickiest question of why it's being called at the time it is... Hint: you'd expect it to be called earlier!

OK, time's up...

There are two things going on here - one's useful to remember for future reference, one's a bit more obscure.

The first thing to note is something about __del__(), which many people think of as Python's "destructor" as opposed to the __init__() "constructor". However, if you're used to a language like C++ then it may come as a surprise to learn that __del__() is called whenever the object is garbage collected, even if it wasn't fully constructed yet. This means that really __del__() is more like the opposite of __new__() than __init__(), and it also means you have to be quite careful about what you do in it - you can't really make any assumptions about the state of the object when it's called, and that makes it rather less useful for general cleanup operations than you might think (in general it's better practice to use the context manager protocol if possible).

So, what's happening above is that __init__() is called a non-existent method on the list that's passed in and throwing an AttributeError as a result. This is fine. Normally, since the base class __new__() had already returned, we'd then expect __del__() to also be called, but it wasn't - that's odd. Here's the second trick... Because we didn't catch the AttributeError exception, the traceback associated with it still exists to supply sys.last_traceback and one of the stack frames associated with this still contains a reference to the created (but not fully initialised) DictWrapper.

You can see this in the transcript below where I first wrote some code to trawl through all the objects in the system (using gc.get_objects()) and find out what refers to any extant DictWrapper instances (with gc.get_referrers()), and then I show the same stack frame can be reached from sys.last_traceback:

>>> import dictwrapper
>>> import gc
>>> import sys
>>> import types
>>> 
>>> x = dictwrapper.DictWrapper(())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "dictwrapper.py", line 5, in __init__
    self.value = init_dict.copy()
AttributeError: 'tuple' object has no attribute 'copy'
>>> 
>>> for obj in gc.get_objects():
...   if isinstance(obj, dictwrapper.DictWrapper):
...     for i in gc.get_referrers(obj):
...       if isinstance(i, types.FrameType):
...         print repr(i)
... 
<frame object at 0xdd99f0>
>>> sys.last_traceback.tb_next.tb_frame
<frame object at 0xdd99f0>

So, while this stack frame still exists, the instance doesn't get garbage collected and __del__() isn't called. If we'd have constructed the DictWrapper instance inside a try..except block which caught the AttributeError we would have seen the __del__() method called immediately.

The final piece to the puzzle comes when executing print instance. Since __init__() threw an exception before completing, this exception propogated to the outermost scope and prevented the local instance variable from being set. This means when print instance is executed, another AttributeError is thrown, this time for instance in the local scope. During this process, sys.last_traceback is replaced by the current traceback object, since only the most recent exception's information is tracked. This means that the old stack frame is garbage collected and since this is the only remaining reference to the DictWrapper instance then this is collected too, causing its __del__() method to finally be called.

Since __del__() refers to self.value which was never actually set (since __init__() threw an exception first) then yet another AttributeError is thrown. Because this is in a __del__() method, however, Python ignores exceptions thrown and instead just prints an error message to stderr, which we can see displayed above just prior to the backtrace for the AttributeError for instance (which is totally unrelated).

There you go, clear as mud. Again, if anybody wants to clarify anything I'm more than happy to explain further.

Told you it was quite tricky!