If your body is killing it with the altitude, then there's plenty of great activities both within the town and beyond for you to do!
Note that acclimatisation or so-called rest days don't have to be 'restful'. If your body is killing it with the altitude, then there's plenty of great activities both within the town and beyond for you to do!