system
Given the following post, respond as if you're commenting on Reddit:
human
[40k] Why is technology stagnant?

What are the reasons for the ongoing stagnation? You'd think that the perpetual state of war would really drive innovation to secure the survival of the Empire. Why doesn't the empire has entire world researching for a superweapon to end it all?
gpt
Theres a bunch of reasons. Ill try to do them in chronological order but like everything i do thats probably wishful thinking.

**TL;DR** The Imperium has lost knowledge, they dont have the ability to read old code, they dont have the technical know-how to innovate without old code, they are superstitious and fear enemies from millenia past, they focus on the past instead of the future and the imperium is bloody gigantic so any changes take thousands of years to roll out. 

Ok so you have the Dark Age of Terra where the Iron Men, AIs created by the humans to do literally everything rebelled and tore apart the human civilization. This civilization considered the Rhino tank to be akin to a jeep and the Baneblade a "Light Tank", so when they say the Iron Men tore it apart its an indication that these Iron Men were really bloody powerful. Humanity prevailed but at significant cost and everything was lost, all that was left was hoarded and maintained but there were no resources for the majority of mankind stranded across the galaxy to innovate. 

Examples of this are from the Knights who pilot behemoths smaller than Titans but still devastating. These Knights are piloted by humans who have an implant in their minds to allow them control, these machines are vital for keeping many worlds safe as there are monsters that even they have trouble dealing with (each one is around 4-5 dreadnoughts tall). They have no room for innovation, indeed with no supplies coming in over the millenia many fell into disrepair and it was only with the Great Crusade were many found and patching up for continued service. 

Following the Iron Men's war, all research into AI has been forbidden upon pain of very very very agonizing death as orchestrated by the Mechanicus who consider it a very dangerous task indeed. The most powerful war machines of the Mechanicus (outside of their Knights and Titans) are golems which use punch-cards to fulfil their tasks, all AI has been stripped from other devices and the closest things that come to "intelligent" would be servitors which have their minds wiped and used to process information as machines. 

So we have a humanity that no longer has the ability to produce the amount of resources to innovate as all supply lines have been cut except for within star systems (and sometimes not even then), so stagnation is the only real way forward. Consider the Terminator armour used by the Holy Adeptus Astartes, it was designed to perform maintenance within fusion reactors but now has been repurposed to wage war. The Power Armour was for outside maintenance upon ships not the standard kit of an elite fighting army. Everything has been repurposed and new discoveries are simply rediscovering old things i.e. the Lightning fighter of the Imperial Navy which spent 300 years in testing after the discovery of its STC to ensure it was not corrupted. 

Then came the Horus Heresy were a full half of the Mechanicus rebelled and Mars was laid to waste by the Legions of Titans battling across its surface. Knowledge was lost forever as scrap-code (viruses) was inserted into data-cubes and the information inside rendered scrap for all eternity. In the aftermath there was barely enough there to maintain the Imperium itself and all attempts at innovation were halted as resources were poured into maintenance to ensure the xeno-hordes would not prevail. Over time this necessity became religion and now despite having the ability and the resources their religion prohibits any attempts at innovation and death awaits those foolish enough to challenge the Magos' words. 

Part of this fear is due to the Iron Men (despite them being millenia before) as their AI still exists within much of the technology of the Imperium to this day. The "pacification" of the Machine Spirit in many cases is literally pacifying the residual Iron Man spark to ensure it does not catastrophically destroy everything. This spark is not present in all machines but often times it is better to be safe and do it for all than to have your battleship's warp drive implode and drag the entire segmentum's fleet into the howling abyss of the Warp. 

The Mechanicus does not know which part of the code is essential for function and which is the Iron Men or if they are separate at all due to the fact that their language originally termed Binary has become mutated over time and descended into slang and shorthand common to all languages (think Latin and English). Their ability to read the very information necessary has been lost and so they rely on tradition and rituals to operate many of the more ancient machinery of the Imperium. At the same time this lack of ability to read operating language means that in innovating they will combine various pieces or use existing pieces to create new software which in turn might mean mixing Iron Men AI and creating an unstoppable machine that will cause untold devastation. 

Normal people cannot do anything technology related (outside of operating and basic maintenance) as the Mechanicus holds a monopoly and will enforce it with extreme prejudice. No-one is willing to push them as they hold the monopoly on technical knowledge. 

So in the end the Mechanicus lacking the ability to read information, mired in the past and tradition and built upon superstition does not innovate at all. 

In regards to technology elsewhere the Tau are innovating at an extreme rate. Not held back by the dogma of the Imperium they already have armour on their battlesuits equal if not superior to Imperium Power Armour and their weapons can melt entire armies. They merely lack the manpower to wage a sustained war and they are definitely trying to remedy that. 

I hope that helped and for further reading i recommend the new Mechanicus Codex as well as the Ciaphus Cain novelizations, the Fire Warrior novel, Baneblade, the Space Wolf saga and other pieces of fluff for peripheral knowledge.