Place the four sprigs rosemary on a cutting board; lightly crush with the back of a spoon. Set aside.
In a Dutch oven, bring the water to boiling; remove from heat. Add tea bags and the lightly crushed rosemary sprigs. Cover and let steep for 4 minutes. Using a slotted spoon or strainer, remove tea bags and rosemary; carefully press tea bags to get out as much liquid as possible. Cool slightly. Transfer to pitchers. Cover and chill for 4 to 24 hours.
Serve over ice. If desired, serve with additional rosemary sprigs.