Context Managers – Understanding the Python with keyword

Python With

The Python with statement is a very useful. This has been there since Python 2.5, and it’s been an ever-present feature used by almost every Python application now!

What’s so useful about this statement, so much that everyone uses it?

The most useful thing (the only thing in fact!) it does is that it opens and frees up resources.

Basically, it handles opening and closing any resources that you may need to use for a specific part of the program, and automatically closes it afterwards.

Let’s look at this statement in a bit more detail, using some examples now.


Why do we Need Context Managers?

Consider the scenario where you do file handling. In other languages, like C, we must manually open and close the file like this:

# Open the file
file_obj = open('input.txt', 'r')

# File operations come here
...

# Manually close the file
file_obj.close()

The with statement now abstracts this automatically for you, so that don’t need to close the file manually every-time!

The with statement has a context (block), under which it acts on. This of this as the scope of the statement.

When the program comes out of this context, with automatically closes your file!

Due to this, with is often referred to as a Context Manager.

So the same file handling procedures can be used like this, along with the with statement:

with open('input.txt', 'r') as file_obj:
    ...

Notice how this is very intuitive. The Python with statement will always close the file at the end, even if the program terminated abnormally even within the context/block.

This safety feature makes it the accepted (and recommended) choice for all Python programmers!


Using the Python with statement

Now, while there are a lot of classes which have implemented a facility for using with, we’re interested in looking at how it works, so that we can write one ourselves!

  • First, the with statement stores an object reference in a context object.

A context object is an object that contains extra information about its state, such as the module/scope, etc. This is useful since we can save or restore the state of this object.

So, there is some meaning to keeping a reference to the object!

Now, let’s move on. Once the context object is created, it calls the __enter__ dunder method on the object.

  • The __enter__ statement is the one that actually does the work of opening resources for the object, such as a file/socket. Normally, we can implement it to save the context object state, if needed.

Now, remember the as keyword? This actually returns the context object. Since we need the returned object by open(), we use the as keyword to get the context object.

Using as is optional, especially if you have a reference to the original context object somewhere else.

After this, we enter the nested block of statements.

Once the nested block has ended, OR, in case there is an Exception within this, the program always executes the __exit__ method on the context object!

This is the safety first feature we talked about earlier. So no matter what happens, we will always use __exit__ to free up resources and exit the context.

Finally, if feasible, __exit__ can be implemented so as to restore the context object state, so that it goes back to whatever state it belonged to.

Okay, that was quite a long explanation. To make it clearer, let’s look through an example of creating our own context manager for a Class.


Creating our own Context Managers for our Class

Consider the below class, for which we will have our own context manager for file handling.

class MyFileHandler():
    def __init__(self, filename, mode):
        self.filename = filename
        self.mode = mode
        # Originally, context object is None
        self.context_object = None


    # The context manager executes this first
    # Save the object state
    def __enter__(self):
        print("Entered the context!")
        self.context_object = open(self.filename, self.mode)
        return self.context_object


    # The context manager finally executes this before exiting
    # Information about any Exceptions encountered will go to
    # the arguments (type, value, traceback)
    def __exit__(self, type, value, traceback):
        print("Exiting the context....")
        print(f"Type: {type}, Value: {value}, Traceback: {traceback}")
        # Close the file
        self.context_manager.close()
        # Finally, restore the context object to it's old state (None)
        self.context_object = None

# We're simply reading the file using our context manager
with MyFileHandler('input.txt', 'r') as file_handle:
    for line in file_handle:
        print(line)

Observe the class methods closely. We have the __init__ method for our handler, which sets the initial state of the Context objects and relevant variables.

Now, the __enter__ dunder method saves the object state and opens the file. Now, we are inside the block.

After the block executes, the context manager executes __exit__ finally, where the original state of the context object is restored, and the file is closed.

Okay, now let’s check our Output now. This should work!

Output

Entered the context!
Hello from AskPython

This is the second line

This is the last line!
Exiting the context....
Type: None, Value: None, Traceback: None

Alright, seems that we got no errors! We’ve just implemented our own context managers for our Custom Class.

Now, there is another approach of creating a Context Manager, which uses generators.

However, this is a bit hacky and is generally not recommended, unless you know exactly what you’re doing since you must handle exceptions yourself.

But, for the purpose of completeness, you can look at using this approach here. I’d recommend you to read this once you’re familiar with the Class-based approach.


Conclusion

In this article, we learned about using Context Managers in Python, using the with statement.

References

  • A wonderful article by preshing on Python context managers