Thanks for writing. There's no good reason to dig out the soil and replace it. I'm not sure why your plants didn't do as well the second year, but I'd guess the soil is not the issue. If you wish to improve your soil, the best way to do so is to add organic matter, such as compost, every year. Organic matter feeds the soil as it breaks down.
As to why your plants didn't do so well, it could be any number of factors from weather to pests and diseases. Without more information, I'm afraid I can't help with that.
---Justin, Senior Garden Editor, BHG.com