This phase 2 trial studies how well pembrolizumab with or without vismodegib works in treating patients with skin basal cell cancer that has spread to other places in the body or cannot be removed by surgery. Monoclonal antibodies, such as pembrolizumab, blocks tumor growth in different ways by targeting certain cells. Vismodegib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. It is not yet known whether pembrolizumab is more effective alone or together with vismodegib in treating skin basal cell cancer.