The primary purpose of this trial is to define the maximum tolerated and/or recommended phase II dose of the combination of panitumumab, oxaliplatin and capecitabine in patients undergoing radiation therapy for carcinoma of the thoracic esophagus or gastroesophageal junction. An additional primary objective is to describe the frequency and nature of grade III/IV and grade I/II toxicities associated with this regimen. Secondary objectives include describing 1-year disease-free survival and overall survival rates as well as to estimate clinical and pathologic complete response rates associated with this regimen.