PURPOSE: This phase I/II trial is studying the side effects and best dose of erlotinib when given together with intensity-modulated radiation therapy and pemetrexed and to see how well they work in treating patients with recurrent or second primary head and neck cancer.