system
Given the following post, respond as if you're commenting on Reddit:
human
How powerful are GPUs really? Should I expect them to always outperform well written vectorized code?

In my computer architecture class my professor said that even well written and optimized SIMD code is usually going to be outperformed by just doing it in cuda instead. How true is this?
gpt
It's just obviously true, because they're not even in the ballpark.

A GPU can have up to 1000s of cores.

So he's right, even a naive CUDA implementation would beat all the work you put in to optimize the same workload with SIMD instructions.

At some point, the big bottleneck would definitely be memory bandwidth, GPU cores aren't pulling all their data from main memory, the GPU has it's own RAM optimized to work with those parallel cores, whereas for the CPU you are eventually going to bottleneck on how fast data can be pulled from main RAM as you optimize for how fast you process it.