PURPOSE: This randomized phase II trial is studying giving erlotinib hydrochloride together with panitumumab to see how well it works with or without irinotecan hydrochloride as second-line therapy in treating patients with metastatic colorectal cancer.