A stepped attenuator can be a big improvement compared to a cheap pot. Use the Java applet to calculate the resitor values. Simply fill in the desired total impedance, maximum level, minimum level, number of steps.
remember : With a 2 * 12 switch and using one position for off gives 2*11 steps.
With your mouse you can select the calculated values, with the key stroke Ctrl-C you can copy the values to the copy/paste buffer.
Don't try to find the exact values, just pick the closest value from the E24 range.