Mastering Parallel Execution in Python: A Comprehensive Guide

By /
Parallel functions

Ever wondered how complex simulations finish so quickly or how massive datasets are processed in no time? This article demystifies the concept of parallel programming in Python. We’ll use simple functions to understand the core concepts. With parallelism, we can speed up simulations and handle data-intensive tasks efficiently. Let’s dive into the fascinating world of parallel programming!

Exploring Multithreading in Python

The first method for achieving parallel programming is Multithreading. It allows multiple threads of execution to run concurrently within a single process, enabling simultaneous execution of different tasks. Python’s threading module facilitates this by managing and creating threads at a higher level.

Multithreading Example: A Practical Approach

To understand the threading module, let’s run two simple parallel functions that print 1s and 0s simultaneously. Let’s dive in:

import threading
from time import sleep
 
def print_num(num):
    # function to print number 
    for i in range(5):
        sleep(1)
        print(num)

Explanation:

  • import threading: This line imports the threading module in Python, which provides functionalities to create and manage threads.
  • from time import sleep: This line imports the sleep function from the time module. The sleep function is used to introduce delays in the program execution.
  • print_num(num) is a function that takes a single argument num and prints its value five times with a 1-second delay between each print.

After this, we will write our main program and use threads to execute the function.

if __name__ =="__main__":
    # creating thread
    t1 = threading.Thread(target=print_num, args=(1,))
    t2 = threading.Thread(target=print_num, args=(0,))
    
    # starting thread 1
    t1.start()
    # starting thread 2
    t2.start()
    print("Threads are running")
    # join the threads with the main function
    t1.join()
    t2.join()
    print("Program finished")

Explanation:

  • if name == “main“: This line checks if the code is being executed as the main program.
  • t1 = threading.Thread(target=print_num, args=(1,)): Creates a Thread object called t1. The target argument specifies the function to be executed in the new thread and the args argument provides the arguments to be passed to the function(we are passing 1 in the first case and 0 in the second).

Now we have our 2 threads t1 and t2, we will start the execution by t1.start() and t2.start(). t1.join() and t2.join() waits for the t1 and t2 threads respectively to complete their execution before proceeding to the next commands.

Output:

Threads are running
1
0
1
0
1
0
1
0
1
0
Program finished

The program creates two threads that concurrently print the numbers 1 and 0. We can see in the output that our threads were running in parallel which resulted in a pattern of 1s and 0s. For detailed information on multithreading in python visit Multithreading in python

Diving into Multiprocessing in Python

Moving on to our second method, we have multiprocessing, which takes parallel programming in Python to a whole new level. While multithreading allows concurrent execution within a single process, multiprocessing enables us to distribute tasks across multiple processes, harnessing the full power of modern CPUs. Python’s multiprocessing module provides a convenient way to create and manage processes.

Multiprocessing Example: A Hands-on Guide

We can understand the multiprocessing module by using it to execute the print_num function. Both functions will run simultaneously, showcasing the power of multiprocessing. Let’s get started:

import multiprocessing
from time import sleep
 
def print_num(num):
    # function to print number 
    for i in range(5):
        sleep(1)
        print(num)

Explanation:

  • import multiprocessing: This line imports the multiprocessing module, which provides support for parallel execution of tasks.
  • from time import sleep: imports the sleep function from the time module.
  • print_num(num) is a function that takes a single argument num and prints its value five times with a 1-second delay between each print.
if __name__ =="__main__":
    # creating thread
    p1 = multiprocessing.Process(target=print_num, args=(1,))
    p2 = multiprocessing.Process(target=print_num, args=(0,))
    
    # starting process 1
    p1.start()
    # starting process 2
    p2.start()
    print("Processes are running")
    # join the process with the main function
    p1.join()
    p2.join()
    print("Program finished")

Explanation:

  • if name ==”main“: This line checks if the code is being executed as the main program.
  • p1 = multiprocessing.Process(target=print_num, args=(1,)) : This line creates a Process object called p1. The target argument specifies the function to be executed in the new process, and the args argument provides the arguments to be passed to the function. In this case, print_num is the target function, and 1 is the argument passed to it. The same is the case for p2 with 0 as the argument.
  • p1.start() and p2.start() starts the execution of the p1 and p2 processes.
  • p1.join() and p2.join() wait for their respective processes to complete their execution before proceeding to the next line.

Output:

Processes are running
1
0
1
0
1
0
1
0
1
0
Program finished

The program creates two processes that concurrently print the numbers 1 and 0. We can observe in the output that our processes were running in parallel, resulting in a pattern of 1s and 0s. This showcases the power of multiprocessing in achieving parallel execution. For more knowledge about multiprocessing visit Multiprocessing in python

Unveiling the Concurrent.futures Module in Python

Now, let’s explore our third method for achieving parallel programming in Python: concurrent.futures. This high-level module provides a powerful and intuitive interface for executing tasks concurrently, abstracting away the complexities of low-level thread or process management.

Concurrent.futures Example:

You might have seen the pattern in the usage of the modules till now, similar is the flow of using concurrent.futures module. Open your IDE and let’s begin!

import concurrent.futures

def function1():
    for i in range(5):
        print(f'Function 1: {i}')
    return 'Function 1 done'

def function2():
    for i in range(5):
        print(f'Function 2: {i}')
    return 'Function 2 done'

Explanation:

  • import concurrent.futures: This line imports the concurrent.futures module, which provides a high-level interface for asynchronously executing(parallel) functions.
  • function1() and function2() simply print their function name followed by numbers in the range 0-4 using a for loop.
# Create a ThreadPoolExecutor
executor = concurrent.futures.ThreadPoolExecutor()

# Submit functions to the executor
future1 = executor.submit(function1)
future2 = executor.submit(function2)

# Retrieve the results
result1 = future1.result()
result2 = future2.result()

# Print the results
print(result1)
print(result2)

# Shutdown the executor
executor.shutdown()

Explanation:

  • executor = concurrent.futures.ThreadPoolExecutor(): This line creates a ThreadPoolExecutor object called executor, which is responsible for managing and executing the submitted functions concurrently using threads.
  • future1 = executor.submit(function1): This line submits function1 to the executor for execution and returns a Future object called future1, which represents the result of the function execution. The same is done for submitting function2.
  • result1 = future1.result(): This line gets the result of function1 by calling the result method on the future1 object. This line will block until the function execution is complete and the result is available. This is followed by result2 that retrieves the result of function2.
  • executor.shutdown(): Shuts down the executor, releasing its resources and stopping the execution of any pending tasks.

Output:

Function 1: 0
Function 1: 1
Function 2: 0
Function 1: 2
Function 2: 1
Function 1: 3
Function 2: 2
Function 1: 4
Function 2: 3
Function 2: 4
Function 1 done
Function 2 done

Similar to what we saw in our previous 2 methods, we got the functions running in parallel as we can see in the output. The code executes two functions concurrently using a thread pool executor. The functions print numbers from 0 to 4 with a prefix indicating the function name. Finally, the results of each function execution are printed.

Conclusion: Master Parallel Execution in Python

We’ve explored the multithreading, multiprocessing, and concurrent.futures modules in Python, learning how to execute tasks in parallel, enhance performance, and manage concurrent tasks effectively. These modules are versatile, speeding up CPU-bound or I/O-bound tasks and facilitating asynchronous operations. They offer flexibility, unlocking the potential of parallelism and concurrency in our Python programs. How will you leverage these techniques in your next Python project?

References