Ive never heard of ants harming tomato plants in the garden. But they may be a sign of other problems that you can diagnose. Ants will feed on tomato fruits that are cracked or otherwise damaged. Cracking of the tomato fruits can occur with erratic watering or too much nitrogen fertilizer after the fruit is set. Ants will get into the cracks and feed on the fruit then, but they are not really the cause of the damage. That goes for other damage from diseases and small animals too. If youre seeing ants on the plant itself, the ants are probably there farming aphids. They are attracted by stuff the aphids produce and are feeding on that. Check for aphids on the underside of leaves and knock them off with a good, strong blast of water. Get rid of the aphids and the ants will probably leave the plants alone. That is unless the ants across the fence are fire ants. Then theyre probably a bigger threat to you than to the plants!