In a 4- to 5-quart Dutch oven cook potatoes in enough boiling, salted water to cover for 25 minutes or until tender. Drain and return to Dutch oven.
Add 1/2 cup of the milk and butter. Use a potato masher or electric mixer to combine mixture until smooth. Add orange peel, orange juice,crystallized ginger if using, brown sugar, salt, ground ginger, and pepper; mash until well combined. Stir in enough remaining milk to make desired consistency.