Transplanting your perennials from containers into the ground for winter is a terrific idea. September through mid-October will be fine for your area. Mulching them will both insulate and protect the crowns and help prevent heaving if the ground freezes and thaws repeatedly. Finally, I'm a little suspicious that your purple fountain grass won't be winter hardy in zone 4. You can sure give it a shot in the ground, but I'm skeptical about its durability.