Recording and editing software can be free (GarageBand or Audacity).
If you already have a PC or tablet, the only equipment you'll need is a USB microphone. The Samson Q2U is a good choice, and only costs $60. Recording and editing software can be free (GarageBand or Audacity). And podcast hosting, like Transistor, is affordable on a monthly basis. It's possible to start a podcast for under $100.