It sounds like your peonies are under siege from the most common and aggravating peony disease, Botrytis paeoniae, a.k.a. Botrytis blight. Once the pathogen is present in your garden, you'll likely see it attack your peonies every year, with more severe attacks during wetter years. To lessen Botrytis's impact, remove and discard all of the peony plant debris in the fall, then add a 2-3-inch layer of mulch. If you're troubled enough by it, you can spray fungicide. But you'd have to start a spray regimen when the shoots first emerge from the soil in the spring and repeat every two weeks through July. The fungicide is used as a preventative, because it won't clear up Botrytis-damaged plants later in the season.