In a large bowl combine 1 1/2 cups of the flour and the yeast. In a saucepan heat and stir milk, butter, sugar, and salt just until warm (120 degrees F to 130 degrees F) and butter almost melts; add to flour mixture along with the eggs and 1/4 cup of the peanut butter. Beat with a mixer on low 30 seconds, scraping bowl as needed. Beat on high 3 minutes. Stir in as much of the remaining flour as you can.
Turn dough out onto a lightly floured surface. Knead in enough of the remaining flour to make a moderately soft dough that is smooth and elastic (3 to 5 minutes total). Shape dough into a ball. Place dough in a lightly greased bowl, turning once to grease surface of the dough. Cover; let rise in a warm place until double in size (about 1 hour).
Punch down dough. Turn out onto a lightly floured surface. Cover; let rest 10 minutes. Meanwhile, lightly grease a 13x9-inch baking pan. For filling, in a small bowl stir together the remaining 1/2 cup peanut butter and the 1/4 cup jam or jelly.
Roll dough into an 18x12-inch rectangle. Spread filling over the dough, leaving a 1 inch unfilled along one of the long sides. Roll up rectangle, starting from the filled long side; pinch dough to seal seam. Slice into 12 pieces. Arrange rolls, cut sides up, in the prepared pan. Cover; let rise in a warm place until nearly double in size (about 30 minutes).
Preheat oven to 375 degrees F. Bake about 30 minutes or until golden. Cool 5 minutes. Carefully invert rolls onto a wire rack; cool slightly. Invert again onto a serving platter. Spread with Peanut Butter Icing. If desired, drizzle additional melted jam or jelly over the frosting.
Peanut Butter Icing
In a medium bowl combine powdered sugar and peanut butter. Stir in enough milk to make a spreading consistency.