This sounds like geranium budworm. The pest also attacks petunia and nicotiana, and is sometimes called tobacco budworm. The worms often tunnel into the flower buds, which is how they got their common name. On a small planting, handpicking them and destroying them is the most effective means of control. The worms are usually most active at dusk, so that's the best time to catch them.
If you decide to spray for them, spinosad or one of the synthetic pyrethroids (bifenthrin, cyfluthrin, permenthrin or related compounds) are most effective.