The problem is that your camera is usually controlled by the Right Thumbstick.
Toggling the build mode on consoles, for instance, is done by pressing O/B/A on an Xbox/Sony/Switch controller. The problem is that your camera is usually controlled by the Right Thumbstick. Trying to quickly move your thumb from the stick to toggle build mode and back again to start placing structures can be a pain.