Chapter 1: A Brief Introduction to Roofline Analysis

Scaling Book Exercises – Chapter 1

Chapter: 1 – All About Rooflines
Source: handwritten Onyx Boox A4 PDF
Note: Transcribed from handwritten solutions; book markdown used only for exercise statements and notation.

Exercise 1 – int8 matmul

Exercise statement

Say we want to do the matmul $𝑋\left[𝐵,𝐷\right]{\cdot }_{𝐷}𝑌\left[𝐷,𝐹\right]\to 𝑍\left[𝐵,𝐹\right]$ in int8 precision (1 byte per parameter) instead of bfloat16 (2 bytes per parameter) since TPUs/GPUs can do matmuls faster in lower precision.

1. How many bytes need to be loaded from memory? How many need to be written back to memory?
2. How many total OPs are performed?
3. What is the arithmetic intensity?
4. What is a roofline estimate for ${𝑇}_{\text{math}}$ and ${𝑇}_{\text{comms}}$? What are reasonable upper and lower bounds for the runtime of the whole operation?

Assume our HBM bandwidth is $8.2\cdot {10}^{11}\text{bytes/s}$ and our int8 peak OPs/s is $3.94\cdot {10}^{14}$ (about 2x bfloat16).

Solution

Scan pages: 1-2

Assume the operation is $𝑋\left[𝐵,𝐷\right]{\cdot }_{𝐷}𝑌\left[𝐷,𝐹\right]\to 𝑍\left[𝐵,𝐹\right]$. Then, bytes in int8:

There are $𝐵𝐹$ dot products. A dot product has $𝐷$ multiplications and computes

hence $𝐷-1$ sums. Total OPs is then:

Arithmetic Intensity:

Roofline estimates:

Bounds of the whole operation:

Exercise 2 – int8 weights and bfloat16 activations

Exercise statement

In practice we often do different weight vs. activation quantization, so we might store our weights in very low precision but keep activations (and compute) in a higher precision. Say we want to quantize our weights in int8 but keep activations (and compute) in bfloat16. At what batch size do we become compute bound? Assume $1.97\cdot {10}^{14}$ bfloat16 FLOPs/s.

Hint: this means specifically bf16[B, D] * int8[D, F] -> bf16[B, F] where $𝐵$ is the “batch size”.

Solution

Scan pages: 3-4

Let the accelerator FLOPs/s for bf16 be:

Assume the operation is:

Assuming no conversion needed to cast int8 to bf16, we get:

Since:

we can approximate:

Hence:

“Per second I do more starts of the algorithm since less bytes to be transferred, so then per-algo FLOPs can be smaller.”

Exercise 3 – roofline plot for two matrix sizes

Exercise statement

Taking the setup from Question 2, make a roofline plot of peak FLOPs/s vs. $𝐵$ for $𝐹=𝐷=4096$ and $𝐹=𝐷=1024$. Use the exact number of bytes loaded, not an approximation.

Solution

Scan pages: 5-8

Let:

• Accelerator bfloat16 = $1.97\cdot {10}^{14}$.
• Accelerator HBM = $8.2\cdot {10}^{11}$.

Bandwidth limited performance can be defined as $\text{AI}\cdot \text{BW}$. Therefore the operation from Question 2 has:

Assuming $𝐷=𝐹$ the AI becomes:

Let $𝐷=2^{𝑘}$:

Note that $𝑘=12$ for $𝐷=4096$ and $𝑘=10$ for $𝐷=1024$. Taking logarithm we get:

For $𝐷=4096$:

For $𝐷=1024$:

Exercise 4 – different matrix for each batch element

Exercise statement

What if we wanted to perform $\text{int8}\left[𝐵,𝐷\right]{\cdot }_{𝐷}\text{int8}\left[𝐵,𝐷,𝐹\right]\to \text{int8}\left[𝐵,𝐹\right]$ where we imagine having a different matrix for each batch element. What is the arithmetic intensity of this operation?

Solution

Scan page: 9

Assume the operation is:

Hence:

Therefore:

Exercise 5 – memory roofline for H100 SXM

Exercise statement

Using the spec sheet provided by NVIDIA for the H100 SXM, calculate the batch size at which a bfloat16 matrix multiplication will become compute-bound. Note that the Tensor Core FLOPs numbers are twice the true value since they are only achievable with structured sparsity.

Solution

Scan pages: 9-10

Let Accelerator FLOPs = $\frac{1979\cdot {10}^{12}}{2}\approx 989.5\cdot {10}^{12}\approx 9.89\cdot {10}^{14}.$ To be computed bound we need:

Assume $𝐷\gg 𝐵$ and $𝐹\gg 𝐵$. Hence: