This phase II clinical trial studies how well fulvestrant and palbociclib works in treating older patients with breast cancer that responds to hormone treatment (hormone responsive) that cannot be removed by surgery. Estrogen can cause the growth of estrogen-receptor-positive breast cancer cells. Hormone therapy using fulvestrant may fight estrogen-receptor-positive breast cancer by blocking the use of estrogen by the tumor cells. Palbociclib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving fulvestrant together with palbociclib may be an effective treatment for hormone responsive breast cancer.