I suspect that what you're describing is actually a lichen. Is it relatively hard and crusty in appearance? If so, you need not do anything about it. Lichens grow on the surface of many objects, including tree and shrub branches and rocks. They don't harm the plants that they grow on.