"You're right not to use a pesticide, because that will probably kill the pollinators your need. If only the petals are munched on the edges, there shouldn't be any problem. If the entire flower is missing, it's possible that a squirrel or other small animal is eating them. If the flowers are just shriveled and laying on the ground, they they are falling off naturally, as they usually do.
The male flowers have long skinny stems, and the female flowers have short, fat stems. Male flowers that fall off are just done doing their job. Female flowers that fall off have not been pollinated. If you are having unusually hot weather, that can also cause the plant to drop its flowers. "