This is a phase II study of the combination of panitumumab with irinotecan in malignant glioma patients. The primary objective of the study is to determine the activity of the combination of panitumumab with irinotecan as measured by 6-month progression-free survival. Secondary objectives include the following- to determine the safety of panitumumab in combination with irinotecan in patients with malignant glioma; to determine the effect of panitumumab in combination with irinotecan on corticosteroid dose for each patient; to explore any relationship between epidermal growth factor receptor (EGF-R) mutational analysis and efficacy or toxicity; and, to determine the response rate and overall survival of recurrent glioblastoma (GBM) patients treated with panitumumab in combination with irinotecan.