If your plums fall off the tree when they are marble-size in diameter, they either are not getting pollinated or are damaged by an insect such as the plum curculio. Do you have more than one plum tree or are there other plum trees nearby that bloom at the same time as your tree? If not, you may need to plant another plum tree of a different variety to get cross-pollination.
Also, check the fallen fruits for scarring. Plum curculio adults make a crescent-shape slit in the developing fruit where they lay their eggs. For more on how to identify and control plum curculio, consult this bulletin from Texas A&M University: http://insects.tamu.edu/extension/publications/pcurculio.html