Here's some advice for planning and hosting an easygoing caroling party:
Start early. Send a "save the date" card before Thanksgiving, then send invitations shortly after.
Pick a less-than-prime time. During the busy holiday season, people are more likely to make it to your party if you have it on a weeknight or a Sunday afternoon or evening.
Keep it (fairly) short and sweet -- three hours is a nice time frame. People can just stop by for a caroling open house atmosphere.
Pick a theme and a color scheme -- then keep your eye out for elements that fit both. As you prepare for the party, you might find a vase here, a scrap of ribbon there.
Vary the guest list. Try to put people together who don't know each other. A mix of young and old lends to the diversity of the list.
Serve simple food. Serve finger foods for quick and easy munching between songs. If you don't have time to bake, buy specialty cookies at the bakery or serve some of the goodies you always get as gifts.
If singers are shy, serve a little holiday cheer. Something festive and bubbly will get people in the mood and make them more apt to sing loudly.