Python Advanced - Concurrency with Threading

Working with threads for concurrency in Python

Concurrency allows multiple tasks to run simultaneously, improving the efficiency and performance of your programs. Python provides the threading module to work with threads for concurrency. This tutorial explores how to use threading in Python to achieve concurrency.

Creating and Starting Threads

Threads can be created by instantiating the Thread class and passing a target function to be executed:

import threading

# Example of creating and starting threads
def print_numbers():
    for i in range(5):
        print(f"Number: {i}")

def print_letters():
    for letter in 'ABCDE':
        print(f"Letter: {letter}")

# Creating threads
thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_letters)

# Starting threads
thread1.start()
thread2.start()

# Waiting for threads to finish
thread1.join()
thread2.join()

print("Threads have finished execution.")
            

In this example, two threads are created to run the print_numbers and print_letters functions concurrently. The start method begins the execution of the threads, and the join method ensures that the main program waits for the threads to complete.

Thread Synchronization

Thread synchronization is necessary to avoid race conditions when multiple threads access shared resources. The threading module provides synchronization primitives like locks, semaphores, and condition variables:

import threading

# Example of thread synchronization using a lock
lock = threading.Lock()
counter = 0

def increment_counter():
    global counter
    for _ in range(1000):
        with lock:
            counter += 1

# Creating threads
threads = [threading.Thread(target=increment_counter) for _ in range(10)]

# Starting threads
for thread in threads:
    thread.start()

# Waiting for threads to finish
for thread in threads:
    thread.join()

print(f"Final counter value: {counter}")
            

In this example, a lock is used to synchronize access to the shared variable counter, ensuring that only one thread can increment the counter at a time.

Using threading.Event

The threading.Event class provides a way to communicate between threads and coordinate their actions:

import threading
import time

# Example of using threading.Event
event = threading.Event()

def wait_for_event():
    print("Waiting for event to be set...")
    event.wait()
    print("Event is set!")

def set_event():
    time.sleep(2)
    print("Setting event...")
    event.set()

# Creating threads
wait_thread = threading.Thread(target=wait_for_event)
set_thread = threading.Thread(target=set_event)

# Starting threads
wait_thread.start()
set_thread.start()

# Waiting for threads to finish
wait_thread.join()
set_thread.join()

print("Event handling complete.")
            

In this example, one thread waits for an event to be set, while another thread sets the event after a delay. The Event object is used to synchronize the actions of the threads.

Thread Communication with Queue

The queue.Queue class provides a thread-safe way to exchange data between threads:

import threading
import queue

# Example of thread communication using a queue
def producer(q):
    for i in range(5):
        q.put(i)
        print(f"Produced {i}")

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(f"Consumed {item}")
        q.task_done()

# Creating a queue
q = queue.Queue()

# Creating threads
producer_thread = threading.Thread(target=producer, args=(q,))
consumer_thread = threading.Thread(target=consumer, args=(q,))

# Starting threads
producer_thread.start()
consumer_thread.start()

# Waiting for the producer thread to finish
producer_thread.join()

# Adding a sentinel value to stop the consumer thread
q.put(None)
consumer_thread.join()

print("Queue handling complete.")
            

In this example, the producer thread puts items into the queue, and the consumer thread retrieves and processes them. A sentinel value (None) is used to signal the consumer thread to stop.

Threading with ThreadPoolExecutor

The concurrent.futures.ThreadPoolExecutor class provides a high-level interface for asynchronously executing callables using a pool of threads:

from concurrent.futures import ThreadPoolExecutor

# Example of using ThreadPoolExecutor
def task(n):
    print(f"Processing {n}")
    return n * n

# Creating a ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(task, i) for i in range(5)]

# Retrieving results
results = [future.result() for future in futures]
print(f"Results: {results}")
            

In this example, a thread pool is created with a maximum of three threads. Tasks are submitted to the pool, and their results are retrieved after execution.

Summary

In this tutorial, you learned about working with threads for concurrency in Python, including creating and starting threads, thread synchronization, using threading.Event, thread communication with Queue, and threading with ThreadPoolExecutor. Understanding these advanced threading techniques helps you write efficient and concurrent Python programs that can handle multiple tasks simultaneously.