In this article, we are going to look at an important functional pattern used in Python using the functools module. In programming, we often make use of higher-order functions. They are the functions that accept another function as a parameter and act upon it or return yet another function. Usually referred to as decorators in Python, they can be extremely useful as they allow the extension of an existing function, without any modification to the original function source code. They supercharge our functions and extend them to have extra functionalities.
The functools module is for using higher-order functions that are in-built in Python. Any function that is a callable object in Python, can be considered a function for using the functools module.
List of functools functions covered
partial()partialmethod()reduce()wraps()lru_cache()cache()cached_property()total_ordering()singledispatch()
Explanation and Usage
Let’s now get started with using the functools module further and understand each function practically.
1. partial()
partial is a function that takes another function as an argument. It takes a set of arguments both positional and keywords and those inputs are locked in arguments to the function.
Then partial returns what’s called a partial object that behaves like the original function with those arguments already defined.
It is used to transform a multi-argument function into a single argument function. They are much more readable, simple, easily typed, and provide efficient code completion.
For example, we will try to find out the square of numbers ranging from 0-10, first with the conventional function, and then get the same result using partial() from functools. This will help us understand its usage.
- Using the conventional function
def squared(num):
return pow(num, 2)
print(list(map(squared, range(0, 10))))
# Output: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
- Using
partial()from the functools
from functools import partial
print(list(map(partial(pow, exp=2), range(0, 10))))
# Output: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
2. partialmethod()
partialmethod function is similar to the partial function and used as a partial for class methods. It is designed to be a method definition rather than being directly callable. In simple terms, it’s the method that’s incorporated into our custom-defined class. It can be used to create convenience methods that pre-define some set values for that very method. Let’s see an example using this method.
from functools import partialmethod
class Character:
def __init__(self):
self.has_magic = False
@property
def magic(self):
return self.has_magic
def set_magic(self, magic):
self.has_magic = bool(magic)
set_has_magic = partialmethod(set_magic, True)
# Instantiating
witcher = Character()
# Check for Magical Powers
print(witcher.magic) # False
# Providing Magical Powers to our Witcher
witcher.set_has_magic()
print(witcher.magic) # True
3. reduce()
This method is used very often to output some kind of accumulated value, calculated by some pre-defined function. It takes a function as the first argument and an iterable as the second argument. It also has an initializer, which defaults to 0 if any particular value to the initializer is not specified in the function, and an iterator along to go over each item of the provided iterable.
Additional Reads: The reduce() function in Python
- Using
reduce()
from functools import reduce
# acc - accumulated/initial value (default_initial_value = 0)
# eachItem - update value from the iterable
add_numbers = reduce(lambda acc, eachItem: acc + eachItem, [10, 20, 30])
print(add_numbers) # 60
4. wraps()
wraps take in a function that it is wrapping and act as a function decorator used when defining a wrapper function. It updates the wrapper function’s attributes to match the wrapped function. If this update is not passed to the wrapper function using wraps(), the metadata for the wrapper function would be returned instead of the original function’s metadata or attributes, which are supposed to be the actual metadata for the entire function. To avoid these types of buggy programs, wraps() are very much useful.
There is another related function that is the update_wrapper() function, which is the same as the wraps(). The wraps() function is syntactic sugar over the update_wrapper() function and also a convenience function to invoke update_wrapper().
Let’s see an example for the above to get a better understanding of the function.
- Defining a decorator function without invoking wraps and trying to access the metadata
def my_decorator(func):
def wrapper(*args, **kwargs):
"""
Wrapper Docstring
"""
return func(*args, **kwargs)
return wrapper
@my_decorator
def original_func():
"""
Original Function Doctstring
"""
return "Something"
print(original_func())
print(original_func.__name__)
print(original_func.__doc__)
# Output
'''
Something
wrapper
Wrapper Docstring
'''
- Defining the same decorator function invoking
wraps()and trying to access the metadata
from functools import wraps
def my_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
"""
Wrapper Docstring
"""
return func(*args, **kwargs)
return wrapper
@my_decorator
def original_func():
"""
Original Function Doctstring
"""
return "Something"
print(original_func())
print(original_func.__name__)
print(original_func.__doc__)
# Output
"""
Something
original_func
Original Function Doctstring
"""
5. lru_cache(maxsize=128, typed=False)
It is a decorator that wraps around a function with a memoizing callable, which simply means that it makes the functional calls efficient while performing expensive I/O functional operations when the same arguments are used for that very particular expensive function. It essentially uses the cached result of the most recent maxsize calls. LRU in lru_cache() is an abbreviation for Least Recently Used.
It has a default maxsize of 128, which sets the number of last recent calls to be cached or saved to use later for the same operation at a later point. The typed=False caches the different types of functional argument values together, which in simple terms means that if we write type=True, the cached values for an integer 10 and a float 10.0 will be cached separately.
lru_cache function also has three other functions incorporated to perform some additional operations namely
- cache_parameters() – It returns a new dict showing the values for maxsize and typed. It is just for informational purposes and value mutation does not affect it.
- cache_info() – It is used to measure how effective the cache is to retune the maxsize if required. It outputs a named tuple displaying hits, misses, maxsize, and currsize of the wrapped function.
- cache_clear() – For clearing up the previously cached results.
There are also some points regarding the lru_cache() function to be kept in mind while using it.
- Since the underlying storage for results while using lru_cache() is a dictionary, the *args and **kwargs must be hashable.
- Since caching only works if a function returns the same result no matter how many times it’s being executed, it should be a pure function producing no side effects after execution.
For understanding, we will see the classic example of printing out the Fibonacci series. We know that for this calculation, the iteration is done using the same values again and again to output Fibonacci numbers when found. Well, caching values for those repeated numbers seems to be a good use case for lru_cache(). Let’s see the code for it.
Code:
from functools import lru_cache
@lru_cache(maxsize=32)
def fibonacci(n):
if n < 2:
return n
print(f"Running fibonacci for {n}")
return fibonacci(n - 1) + fibonacci(n - 2)
print([fibonacci(n) for n in range(15)])
print(fibonacci.cache_parameters())
print(fibonacci.cache_info())
Output:
Running fibonacci for 2
Running fibonacci for 3
Running fibonacci for 4
Running fibonacci for 5
Running fibonacci for 6
Running fibonacci for 7
Running fibonacci for 8
Running fibonacci for 9
Running fibonacci for 10
Running fibonacci for 11
Running fibonacci for 12
Running fibonacci for 13
Running fibonacci for 14
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]
{'maxsize': 32, 'typed': False}
CacheInfo(hits=26, misses=15, maxsize=32, currsize=15)
Explanation:
We can see that, for each of the different values in the range of 15, the print statement has provided output just once. Although the iterations have been done a whole lot more, by using lru_cache(), we are just able to see the unique results being printed and the repeated ones are cached. Next to the results is the information for the above function after being executed.
6. cache()
This function is a smaller and lightweight form of lru_cache() itself. But it has no fixed boundary for the number of cached values. Thus, we don’t need to specify the maxsize for it. It’s same as using lru_cache(maxsize=None).
Since no values that are being cached are forgotten by using this function, it makes cache() much faster than lru_cache() with a size limit. It’s a new addition to Python 3.9
One thing to keep in mind while using this function is that while implementing it with a function large variety of inputs, we can end up with very large cache size. Thus it should be used with caution and a previous thought through.
7. cached_property()
cached_property() is similar to the property() in Python allowing us to turn class attributes into properties or managed attributes as a built-in function. It additionally provides caching functionality and was introduced in Python 3.8
The value for the computation is calculated once and then saved as a normal attribute for the life of that instance. cache_property() also allows writing without a setter being defined, unlike the property() function.
Cached_property only runs:
- If the attribute is not already present
- If it has to perform lookups
Generally, if the attribute is already present in the function, it performs reads and writes like a normal attribute. For clearing the cached values, the attribute needs to be deleted, which indeed allows the cached_property() function to be invoked again.
- Without using catched_property(), see the output
class Calculator:
def __init__(self, *args):
self.args = args
@property
def addition(self):
print("Getting added result")
return sum(self.args)
@property
def average(self):
print("Getting average")
return (self.addition) / len(self.args)
my_instance = Calculator(10, 20, 30, 40, 50)
print(my_instance.addition)
print(my_instance.average)
"""
Output
Getting added result
150
Getting average
Getting added result
30.0
"""
- With catched_property(), see the output
from functools import cached_property
class Calculator:
def __init__(self, *args):
self.args = args
@cached_property
def addition(self):
print("Getting added result")
return sum(self.args)
@property
def average(self):
print("Getting average")
return (self.addition) / len(self.args)
my_instance = Calculator(10, 20, 30, 40, 50)
print(my_instance.addition)
print(my_instance.average)
"""
Output
Getting added result
150
Getting average
30.0
"""
Explanation:
We can see that the Getting added result has been printed twice while calculating the average since, for the first function, its result has not been cached. But when we used the @cached_property decorator, the result for the addition has been cached and thus used straight up from the memory to get the average.
8. total_ordering()
This higher-order function from functools, when used as a class decorator given that our class contains one or more rich comparison ordering methods, provides the rest of them without being explicitly defined in our class.
What it means is that while we use the comparison dunder/magic methods __gt__(), __lt__(), __ge__(), __le__() in our class, if we use define __eq__() and just one of the four other methods, the rest others will be automatically defined by the total_ordering() coming from the functools module.
One important thing to note is that it does slow down the speed of code execution and it creates a more complicated stack trace for the comparison methods that are not explicitly defined in our class.
Let’s see a simple example of it.
Code:
@functools.total_ordering
class Student:
def __init__(self, name, marks):
self.name = name
self.marks = marks
def __eq__(self, other):
return self.marks == other.marks
def __gt__(self, other):
return self.marks > other.marks
student_one = Student("John", 50)
student_two = Student("Peter", 70)
print(student_one == student_two)
print(student_one > student_two)
print(student_one >= student_two)
print(student_one < student_two)
print(student_one <= student_two)
"""
Output:
False
False
False
True
True
"""
Explanation:
In the above code, we have imported total_ordering() with a different syntax. It is the same as importing from the functools used in all the previous examples.
The class that we have created contains only two comparison methods. But by using the total_ordering() class decorator with it, we are enabling our class instances to derive the rest of the comparison methods by themselves.
9. singledispatch()
When we define a function, it implements the same operation for different input types of arguments. But what if we want a function to behave differently whenever the input type of the argument is different?
We send a list or a string or some other type, we want different outputs depending upon the data that we are sending. How can we achieve this?
The functools modules have the singledispatch() decorator which helps us to write such functions. The implementation is based on the type the argument being passed.
The register() attribute of the generic function along with the singledispatch() method is used to decorate the overloaded implementations. If the implementations are annotated with types like a statically typed language, the decorator infers the type of passed args automatically, else the type itself is an argument to the decorator.
Similarly, it can be implemented as a class method by using singledispatchmethod() to achieve the same results.
Let’s see an example to understand it better.
from functools import singledispatch
@singledispatch
def default_function(args):
return f"Default function arguments: {args}"
@default_function.register
def _(args: int) -> int:
return f"Passed arg is an integer: {args}"
@default_function.register
def _(args: str) -> str:
return f"Passed arg is a string: {args}"
@default_function.register
def _(args: dict) -> dict:
return f"Passed arg is a dict: {args}"
print(default_function(55))
print(default_function("hello there"))
print(default_function({"name": "John", "age": 30}))
print(default_function([1, 3, 4, 5, 6]))
print(default_function(("apple", "orange")))
"""
Output:
Passed arg is an integer: 55
Passed arg is a string: hello there
Passed arg is a dict: {'name': 'John', 'age': 30}
Default function arguments: [1, 3, 4, 5, 6]
Default function arguments: ('apple', 'orange')
"""
Explanation:
In the above example, we can see that the functional implementation is done based on the type of argument passed. The types that are not defined are executed by the default function, unlike the others.
Conclusion
In this article, we went through most of the functions provided by the functools module in Python. These higher-order functions provide some great ways to optimize our code which results in clean, efficient, easy to maintain, and reader-friendly programs.