If you’re going to offer a podcast that features music created by someone else, you might be responsible for paying royalties for the right to webcast that music. It doesn't seem that this has been completely worked out yet - although licensing companies which track royalty payments are fervently trying to figure out a workable plan. In the mean time, you would be advised to use "podcast safe" music.
Podcast safe music is designated by the creators as available for use in podcasts either for free or for a small fee. blogtalkradio.com has a list of sources you can check.
Aside from music, if your podcast consists primarily of voice - either your voice or someone else's voice who has agreed to be on your podcast - then you have little worries about copyrights and licensing fees. You own your voice - and the original content you create and speak. If someone agrees to be your guest, they have granted you a license to use their voice and distribute the content they speak within your podcast.
Remember: if you create a podcast - and especially if you incorporate original material which you created - it's a good idea that you signify that the material is copyrighted. During your wrap-up at the end, drop in that your show is "Copyright 2010 by Your Name or Company". That's a personal copyright and the law entitles you to it. It will also serve as a warning to someone who might be tempted to lift or steal something you created. Protect your intellectual property.