Python Advanced - Concurrency with Multiprocessing

Utilizing multiprocessing for concurrency in Python

Concurrency is essential for improving the performance of applications by allowing multiple tasks to run simultaneously. In Python, the multiprocessing module provides a powerful way to achieve concurrency by creating multiple processes. This tutorial explores how to utilize multiprocessing for concurrency in Python.

Creating Processes

You can create a new process using the Process class from the multiprocessing module. Each process runs a target function independently:

import multiprocessing

def worker():
    print("Worker process")

if __name__ == "__main__":
    process = multiprocessing.Process(target=worker)
    process.start()
    process.join()
            

In this example, a new process is created to run the worker function. The start() method starts the process, and the join() method waits for the process to complete.

Using Process Pool

The Pool class from the multiprocessing module provides a convenient way to create a pool of worker processes and distribute tasks among them:

import multiprocessing

def worker(num):
    return num * num

if __name__ == "__main__":
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(worker, range(10))
        print(results)
            

In this example, a pool of 4 worker processes is created, and the map method is used to distribute the tasks (squaring numbers) among the processes. The results are collected and printed.

Inter-Process Communication

The multiprocessing module provides several ways for processes to communicate and share data, including pipes and queues:

import multiprocessing

def worker(pipe):
    pipe.send("Hello from worker")
    pipe.close()

if __name__ == "__main__":
    parent_conn, child_conn = multiprocessing.Pipe()
    process = multiprocessing.Process(target=worker, args=(child_conn,))
    process.start()
    print(parent_conn.recv())
    process.join()
            

In this example, a pipe is used for communication between the parent process and the worker process. The worker process sends a message through the pipe, which the parent process receives and prints.

Shared Memory

The multiprocessing module allows processes to share data using shared memory objects like Value and Array:

import multiprocessing

def worker(shared_value, shared_array):
    shared_value.value += 1
    for i in range(len(shared_array)):
        shared_array[i] += 1

if __name__ == "__main__":
    shared_value = multiprocessing.Value('i', 0)
    shared_array = multiprocessing.Array('i', [0, 1, 2, 3, 4])
    
    process = multiprocessing.Process(target=worker, args=(shared_value, shared_array))
    process.start()
    process.join()
    
    print(shared_value.value)  # Output: 1
    print(shared_array[:])    # Output: [1, 2, 3, 4, 5]
            

In this example, a shared value and a shared array are used to share data between the parent process and the worker process. The worker process modifies the shared data, and the changes are reflected in the parent process.

Process Synchronization

The multiprocessing module provides synchronization primitives like locks, events, and semaphores to manage access to shared resources:

import multiprocessing

def worker(lock, shared_counter):
    with lock:
        for _ in range(100):
            shared_counter.value += 1

if __name__ == "__main__":
    lock = multiprocessing.Lock()
    shared_counter = multiprocessing.Value('i', 0)
    
    processes = [multiprocessing.Process(target=worker, args=(lock, shared_counter)) for _ in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    
    print(shared_counter.value)  # Output: 400
            

In this example, a lock is used to synchronize access to the shared counter. Each worker process increments the counter 100 times, and the final value is printed in the parent process.

Summary

In this tutorial, you learned about utilizing multiprocessing for concurrency in Python. The multiprocessing module provides tools to create and manage multiple processes, enabling concurrent execution. You explored creating processes, using process pools, inter-process communication, shared memory, and process synchronization. Understanding and effectively using multiprocessing can significantly improve the performance of your Python applications by leveraging multiple CPU cores for parallel execution.