RATIONALE: Erlotinib hydrochloride may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Monoclonal antibodies, such as panitumumab, can block tumor growth in different ways. Some block the ability of tumor cells to grow and spread. Others find tumor cells and help kill them or carry tumor-killing substances to them. Panitumumab may also stop the growth of colorectal cancer by blocking blood flow to the tumor. Drugs used in chemotherapy, such as irinotecan hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. It is not yet known whether erlotinib hydrochloride given together with panitumumab is more effective with or without irinotecan in treating patients with metastatic colorectal cancer.