Patients who experience symptomatic or confirmed PD (or have best response of SD at 12 months) on nivolumab monotherapy will be eligible for consideration for Part B. Part B involves the addition of ipilimumab for up to 4 doses while maintaining nivolumab therapy. Dose of ipilimumab will be 1 mg/kg every 3 weeks together with nivolumab changed to 3 mg/kg every 3 weeks for up to 4 doses. Nivolumab will revert to 360 mg every 3 weeks after the completion of treatment with ipilimumab (beginning week 13-19 of Part B) for up to 48 weeks. Patients will be followed with serial imaging assessments weeks 12, 18 and 24 and then every 12 weeks after the initiation of ipilimumab. The tumor measurements at the time of ipilimumab institution will be the new baseline. If unequivocal symptomatic or confirmed new PD (as defined above) develops, treatment will be discontinued.