The most organic treatment would be to hand-pick the worms and drop them in a bucket of soapy water. If that's not a good solution for you, look for an insecticidal soap or insecticide containing neem oil at your local garden center. These products are often considered organic. Note: Insecticidal soap has to be sprayed directly on the worms to stop them.
You can also go with a more hard-core chemical solution, such as the insecticide called Sevin. Look for this and other products at your local garden center. Be sure to follow the directions on the packaging -- if you misapply these products, they won't work very well