Thanks for writing. The safest solution is a natural insecticide called Bt; it only affects caterpillars. Look for it at your local garden center. Keep in mind, though, that the caterpillars are the young form of passionflower butterflies --- so you may have fewer butterflies in your yard if you eradicate the caterpillars.
---Justin, Senior Garden Editor, BHG.com