This phase I/II trial studies the side effects and best dose of cetuximab when given together with pembrolizumab in treating patients with colorectal cancer that has spread from the primary site (place where it started) to other places in the body (metastatic) or that cannot be removed by surgery. Monoclonal antibodies, such as cetixumab and pembrolizumab, may block tumor growth in different ways by targeting certain cells.