The goal of this trial is to assess the safety and efficacy of a multimodality approach combining hormones and immunotherapy in prostate cancer populations that are considered incurable and standardly treated with hormones alone, and represent clinical states prior to development of castration-resistant disease. There are 2 cohorts. The first will use ipilimumab and degarelix prior to and following radical prostatectomy in men with newly diagnosed, oligometastatic, castration-sensitive disease. The second cohort will include men who have already received definitive local therapy with radical prostatectomy but have since experienced biochemical and/or metastatic recurrence.