Understanding Bandwidth and Latency

Ars discusses issues of bandwidth and latency inside hardware architechture.

Latencies in real-world systems

Now let's take a look at a graph that scales out to much longer burst lengths than the 20-beat maximum in the previous graph.

Graph 2: Bandwidth vs. Burst Length

One thing you should know about this graph is that even though its curve may appear to be steeper than that of the previous graph, this is only because the previous graph shows fewer burst lengths (only up to 20). The graph shown above is actually much slower to ramp up to the peak bandwidth than the previous graph, and this is because I've used a 9-cycle memory read latency instead of the 3-cycle latency in the former graph. The 9-cycle latency number more accurately reflects the kind of latency seen by a real-world system. Let's look at why this is.

In systems where memory is connected directly to the CPU (whether main memory or a backside L2 or L3 cache), data takes only one bus clock cycle to travel between the memory and CPU. As I've said before, in the simplified diagrams above the data only appears to take multiple clock cycles to traverse the bus. This was just my way of depicting the fact that the CPU has to wait multiple bus cycles for a request to be filled, so as far as the CPU is concerned the bus appears to be multiple cycles in length. In reality, we could break down the hypothetical 3-cycle latency of our simplified system as follows:

Cycle # Event
1 The CPU's request for data travels across the bus to the DRAM.
2 The DRAM fills the CPU's request.
3 The requested data travels back across the bus to the CPU.


Of course, as should be clear from the preceding section on DRAM latencies, at bus speeds of 100MHz or better real DRAM takes more than one cycle to fill a request. So a more realistic description of read latency in a real system might look as follows:

Cycle # Event
2-5 The DRAM fills the CPU's request.
6 The requested data travels back across the bus to the CPU.

As I've pointed out earlier, the preceding diagrams showing the memory and CPU directly connected by a single bus have been simplified in order to make the basic concepts clear. In the kinds of PC systems that we usually cover at Ars, the CPU and main memory are not directly connected. (This may change in the future, though.) Instead, the north bridge of the core logic chipset sits in between the CPU and RAM, playing traffic cop by routing information in between the CPU, RAM, and the other parts of the system.

Figure 5

A set-up like the one depicted above adds two cycles of latency to a DRAM read: one cycle for the request to travel across the frontside bus to the northbridge, and one cycle for the northbridge to route the request from the FSB interface to the memory bus interface. As is illustrated in the diagram, this means that the request spends a total of 3 cycles just traveling to the DRAM, and then the data has to spend another 3 cycles traveling back to the CPU. The following chart breaks down the latencies for a system like the one pictured above:

Cycle # Event
1 The CPU's request for data travels across the FSB to the DRAM.
2 The chipset transfers the request from the FSB to the memory bus. 
5-8 The DRAM fills the CPU's request.
9 The chipset transfers the request from the memory bus to the FSB
10 The requested data travels back across the FSB to the CPU.


So in more complex systems the DRAM read latency increases simply because of the way the system is laid out. Furthermore, when other devices are contending via the northbridge for access to the DRAM, it's possible that the CPU will have to wait even longer to have a given request serviced if the DRAM is busy doing other things, thus adding more dead bus cycles to the read latency. The following chart illustrates the kind of detrimental effects that increases in read latency have on sustained system bandwidth.

Graph 3: bandwidth vs. burst length at different latencies

The different color curves in the above graph show how the bandwidth of systems with different read latencies ramps up in relation to burst length. The higher the read latency, the longer the burst length has to be before a given amount of sustained bus bandwidth can be attained. So systems with higher read latencies ramp up more slowly than those with lower read latencies. It should be obvious, then, that DRAM read latencies are a huge obstacle to achieving efficient bandwidth usage.

There are other obstacles as well. As I noted previously, the average burst length is not unlimited in a real-word system. In real systems there are multiple components vying for the use of the system memory, including I/O components, multiple CPUs, etc. For instance, if there are two CPUs trying to stream a large amount of data from memory (e.g. both CPUs are running data-intensive Altivec code) then the DRAM must alternate between servicing two sets of requests, which means alternating data tenures. And if each processor is reading data from a different row, then there will be dead cycles in between each processor's data tenure as the DRAM switches rows. Again, these dead bus cycles mean that bandwidth is being wasted. Even worse is the fact that one dead bus cycle can translate into multiple dead CPU cycles if the CPU can't find anything to busy itself with while waiting on the requested data.

Expand full story

You must to comment.