Parallel Processing and GPUs
Parallel Processing and GPUs
As individual processor speeds plateau due to physical limits (heat, power, transistor size), parallelism has become the primary route to greater performance. A-Level requires understanding of different parallel architectures and their applications.
Why Parallel Processing?
Moore's Law (transistor count doubles every ~2 years) continues, but clock speed has stalled since ~2005 due to the power wall — higher frequencies generate unsustainable heat. The solution: use more cores rather than faster cores.
Amdahl's Law limits the speedup from parallelism:
Speedup = 1 / (s + (1−s)/n)
where s = fraction of the program that is serial (cannot be parallelised), n = number of processors.
Even with infinite processors, the serial portion limits the maximum speedup to 1/s. If 10% of a program is serial, maximum speedup = 10×.
Types of Parallel Processing
Flynn's Taxonomy
| Type | Meaning | Example |
|---|---|---|
| SISD | Single Instruction, Single Data | Traditional single-core CPU |
| SIMD | Single Instruction, Multiple Data | GPU, vector processors |
| MISD | Multiple Instruction, Single Data | Rare; fault-tolerant systems |
| MIMD | Multiple Instruction, Multiple Data | Multi-core CPUs, clusters |
Multi-Core Processors
A multi-core processor has 2 or more independent cores on a single chip, sharing cache and memory access.
Advantages:
- True parallelism: each core executes a different thread simultaneously
- Better energy efficiency than a single fast core
- Standard in all modern PCs, phones, and servers (2, 4, 8+ cores common)
Challenges:
- Software must be multi-threaded to benefit (not all programs are)
- Synchronisation between threads adds complexity (race conditions, deadlocks)
- Shared memory creates contention (multiple cores competing for memory access)
- Diminishing returns — doubling cores rarely doubles performance (Amdahl's Law)
GPU Architecture
A GPU (Graphics Processing Unit) is a massively parallel processor designed for tasks where the same operation is applied to many data points simultaneously (SIMD).
| Feature | CPU | GPU |
|---|---|---|
| Cores | Few (4–16), complex | Many (hundreds to thousands), simple |
| Design | Optimised for latency (fast single thread) | Optimised for throughput (many threads) |
| Best for | Sequential, branching logic | Parallel, data-parallel tasks |
| Clock speed | Higher (~4-5 GHz) | Lower (~1-2 GHz) |
| Cache | Large per core | Small per core |
GPU cores are simple: They lack the branch prediction, out-of-order execution, and large caches of CPU cores. But there are thousands of them, and they work together.
GPGPU (General-Purpose GPU Computing)
Originally designed for graphics rendering, GPUs are now used for general-purpose computation (GPGPU) in fields requiring massive parallelism:
| Application | Why GPU excels |
|---|---|
| Machine learning / AI | Training neural networks: matrix multiplication on millions of weights |
| Cryptocurrency mining | Hashing: same operation on many inputs |
| Scientific simulation | Physics, weather, molecular dynamics: same equations applied at every grid point |
| Image/video processing | Filters applied to every pixel independently |
| Data analytics | Same query/transform applied to millions of records |
Frameworks: CUDA (NVIDIA), OpenCL (cross-platform), TensorFlow/PyTorch (ML).
Distributed Computing
Multiple separate computers connected by a network, working on parts of a problem.
Types:
- Cluster computing: Dedicated machines in one location (e.g., university HPC cluster)
- Grid computing: Geographically distributed resources (e.g., SETI@home, Folding@home)
- Cloud computing: On-demand resources from providers (AWS, Azure, Google Cloud)
Advantages: Scalable (add more nodes), fault-tolerant (one node fails, others continue), cost-effective (use commodity hardware).
Challenges: Network latency, data distribution, synchronisation, programming complexity.
Virtual Machines and Containers
Virtualisation allows multiple operating systems to run on a single physical machine by abstracting the hardware.
Virtual Machine: A complete OS running inside a hypervisor. Full isolation but heavyweight (each VM has its own OS kernel).
Container: A lightweight package sharing the host OS kernel (e.g., Docker). Fast to start, lower overhead, but less isolation than a full VM.
Exam Tips
- Amdahl's Law calculation is a common exam question — practise with different values of s and n
- Know Flynn's taxonomy — especially SIMD (GPU) and MIMD (multi-core CPU)
- GPU vs CPU: the key distinction is few complex cores vs many simple cores — give specific application examples
- When discussing parallel processing benefits, always mention the limitations: not all tasks are parallelisable, synchronisation overhead, Amdahl's Law
- Multi-threading questions may ask about race conditions (two threads writing to the same variable) and deadlock (two threads each waiting for the other's resource)
- Cloud computing questions often ask about scalability and the difference between IaaS, PaaS, and SaaS