Line a 3 1/2- or 4-quart slow cooker with a disposable slow cooker liner. Add peaches, berries, and ginger to the prepared cooker; toss gently to coat.
In a medium bowl stir together all-purpose flour, white whole wheat flour, baking soda, baking powder, salt, and nutmeg; set aside.
In a large mixing bowl beat butter with an electric mixer on medium to high speed for 30 seconds. Add brown sugar; beat until combined. Beat in egg and vanilla. Beat in half of the flour mixture; beat in yogurt. Beat in the remaining flour mixture until combined. Spread batter over fruit mixture.
Place a clean kitchen towel over top of cooker; cover with cooker lid. Cook on high-heat setting for 2 1/4 to 2 1/2 hours or until a wooden toothpick inserted in the center of cake comes out clean. To ensure even cooking, carefully rotate crockery liner halfway through cooking, if possible.
Turn off cooker. If possible, remove crockery liner from cooker. Cool for 5 minutes. Remove lid and towel.
To unmold cake, place a large plate over the crockery liner or slow cooker. Using hot pads, carefully invert crockery liner or cooker onto plate (do not lift by the edges of the disposable liner). Remove disposable liner. Serve cake warm or at room temperature. If desired, serve with frozen yogurt.