It sounds like you have unmelted sugar crystals that “seeded” your fudge.
When sugar, butter, and chocolate are melted, they are unstable. They want to return to their crystalline forms—and if one crystal is present, it will replicate itself until you have a big chain reaction.
Try again. The remedy is to wash down the sides of your pot with hot water at the boiling stage—just a pastry brush dipped in hot water will do—to return any errant crystals into the boiling liquid, where they will dissolve.