There is No Big Brother or Small Brother: Knowledge Infusion in Language Models for Link Prediction and Question Answering