This phase I pilot trial studies the side effects of stereotactic radiosurgery and nivolumab in treating patients with newly diagnosed melanoma that has spread to the brain or spine. Stereotactic radiosurgery is a specialized radiation therapy that delivers a single, high dose of radiation directly to the tumor to more precisely target the cancer. Monoclonal antibodies, such as nivolumab may interfere with the ability of tumor cells to grow and spread. Giving stereotactic radiosurgery together with nivolumab may be a better treatment for melanoma.