Frost Damaged Lilies

Should I prune or remove damaged leaves and blooms from my Asiatic lilies? Some of the plants still look great but some of the leaves are brown as well as some of the blooms. Should I always remove blooms after they start to die or leave them and let them fall off?