Preheat oven to 350 degrees F. Place cherries or cranberries in a medium bowl; pour enough boiling water over to cover. Let stand for 5 minutes; drain.
Meanwhile, place 1/4 cup butter in a 9x1 1/2-inch round cake pan. Place cake pan in the oven until butter melts. Stir brown sugar and orange juice into melted butter. Arrange cherries or cranberries and walnuts evenly in pan. Set aside. Wipe out the medium bowl.
In the same bowl stir together flour, granulated sugar, baking powder, salt, and ginger. Add milk, 1/4 cup softened butter, the egg, and vanilla; beat with a wooden spoon until combined. Beat vigorously with spoon for 1 minute. (Batter may still be lumpy.) Spread batter in prepared pan.
Bake in the preheated oven for 30 to 35 minutes or until a toothpick inserted near center comes out clean. Cool in pan on wire rack for 5 minutes. Loosen side of cake; invert onto serving plate. Cool for 30 minutes; serve warm. If desired, top with whipped cream.