The phase 1b portion of the trial is a radiation dose escalation study to determine the maximum tolerated dose (MTD) of SBRT when given to patients previously and subsequently exposed to MK-3475. Because the class of PD-1 inhibitory antibodies conveys a risk of pneumonitis, there will be two parallel dose escalation arms- Arm A will include SBRT targets in the lung parenchyma, and Arm B will be limited to targets outside the lung parenchyma. Each arm will be separately escalated, and two MTDs will be determined. The starting dose will be 3000 cGy in 5 fractions; there will be one dose escalation cohort (3000 cGy in 3 fractions), and if necessary one dose de-escalation cohort (1000 cGy in a single fraction). If there is dose-limiting toxicity at the lowest cohort, that arm will be closed and SBRT to that site will be discontinued.