This is an open label phase II neoadjuvant clinical trial of Palbociclib in combination with endocrine therapy for hormone receptor positive early-stage breast cancer. The planned sample size is 180 participants. The study includes a "window treatment" phase followed by a treatment phase. In the window phase, participants will be treated with a two-week course of tamoxifen (Arm A) or letrozole (Arm B). In the treatment phase participants will be randomized to receive endocrine therapy in combination with palbociclib (Arm C) or endocrine therapy alone (Arm D) for a total duration of 24 weeks. Premenopausal patients with either invasive lobular or ductal carcinoma will be eligible to enroll directly into the treatment phase of the study. The study has two co-primary objectives: 1) To evaluate the difference in anti-proliferative activity of letrozole versus tamoxifen measured by changes in Ki67 from baseline to research biopsy (day 15) within cohorts of hormone receptor positive breast cancer for patients with invasive lobular and ductal carcinoma. 2) To evaluate the pathologic complete response (pCR) of endocrine therapy plus palbociclib and of endocrine therapy alone in breast cancer patients diagnosed with hormone receptor positive invasive breast cancer.