This study is a pilot study open to patients with clinical stage T2N1 or T3N0-1 resectable rectal cancer. The investigational tool being evaluated is high-dose endorectal brachytherapy (HDRBT) which is an FDA approved method to administer endoluminal radiation for low rectal cancer. This study assumes that the rate of pathological response to HDRBT will be similar to historical controls (conventional neoadjuvant chemoradiation). All patients will receive postoperative standard 5-FU based chemotherapy (at the discretion of the medical oncologist). The clinical response will be assessed with endoscopic/ultrasound, pelvic MRI and FDG-PET/CT. In addition, tissue and serum will be collected to evaluate for biological predictors of response to therapy. The endpoints of this study include adverse events (gastrointestinal toxicity), quality of life as measured by the QLQ-C30, and tumor regression/response.