Generators in Python [With Easy Examples]

Generators in Python are powerful tools to make custom iterators that can be used wherever there is a need to generate a sequence of objects.

Pre-requisites

We need to know two important concepts before continuing on with generators in Python.

1. Iterables

An iterable is an object that returns an iterator if iter() is called on it. In other words, objects that are a sequence of other objects are typically iterables. Consider the example:

numbers = list([1, 2, 3, 4])
for element in numbers:
    print(element)

Here, number is a sequence of integers. If iter() is called on it, it will return a “list_iterator“, which is the reason why it can be used directly in a for-loop. In fact, list, dictionary, set, tuple, are all iterable classes.

Now that we’ve got an iterator on our hands, what do we do with it?

2. Iterators

Iterators are objects that are either returned by iter() (as we saw above), or they can be made by the programmer using a generator which we will be learning in this tutorial. They have three key properties:

1. They return an object when next() is called on it.
2. If there is no object to return, it will raise a StopIteration error.
3. Iteration only happens once. If you got an iterator for a list that contains five numbers, and you called next() four times, then you can only call next one more time, and after that, the iterator is of no use. That is, to re-iterate over the same list, a new and fresh iterator will be required.

Consider this example:

What are generators in Python?

Generators in python are functions that create an iterator.

The generator follows the same syntax as a function, but instead of writing return, we write yield whenever it needs to return something.

Creating a generator function

Let’s say we need to generate the first 10 perfect squares starting from 1.
This is the syntax:

def perfect_square():
    num = 1
    while(num <= 10):
        yield (num * num)
        num += 1

Let’s go through the code line by line:

• def perfect_square(): A normal start of a function block.
• num = 1: The only memory we need to generate any number of perfect squares.
• while(num <= 10): We only need to generate 10 perfect squares.
• yield(num * num): The most important and noticeable distinction from a normal function in Python. This is similar to a return statement in that it returns the generated perfect square. Notice that I’m saying generated because all the perfect squares this function returns are generated and not retrieved from memory.
• num += 1: Incrementing so that it yields the next perfect square.

Looking at the behavior of this generator. Simply calling it like a function will return a generator object.

This object is what we have to use. Calling next() on this will yield the first value, calling next() again will yield the second value and so on till the tenth value.

After that, calling next() will attempt to yield another value, but because the function is over, it will raise a StopIteration error.

While we could check for the exception at the end of the loop, a for-loop already does that for us. Recall that a for-loop accepts iterables like ranges, lists, tuples, etc. Similarly, the for-loop also accepts a generator.

for square in perfect_squares():
    print(square)

The above code will print the exact same thing we did before, try it out yourself!

Note that like an iterator, a generator object is not recyclable, so after finishing with squares (the generator object we used), we will need to get another object simply by doing squares = perfect_squares() again.

Also, note that a generator function and a generator object are different, the generator function (or simply generator) is used to return a generator object which yields all the required values.

Generator expressions

A simpler way to make a simple generator is using a generator expression.

Remember list comprehensions – To create a list with the first 10 perfect squares we can run the following:

squares_list = [num * num for num in range(1,11)]

Replacing “[” and “]” with “(” and “)” will instead create a generator that generates these values.

squares_list = (num * num for num in range(1,11))

Note that while the list is stored in memory and can be accessed anytime, the generator can only be used once.

Why do we need a generator?

Let’s see the size difference between the two. Importing the module sys and doing sys.getsizeof() will give us the size of the two objects.

We get:

• squares_list: 184 B
• squares_generator: 112 B

This isn’t a big difference. But what if we need 100 numbers, then the size becomes:

• squares_list: 904 B
• squares_generator: 112 B

For 10000 numbers:

• squares_list: 87616 B or 85.5 KB
• squares_generator: 112 B

It’s clear that if you need a huge sequence, like the first million Fibonacci numbers or the values of a function to print its graph, and you only need it once or twice, generators can save a lot of time (in coding) and space (in memory).

References

Python Wiki – Generators