Mullein is a common pasture and roadside weed, but there is also a domestic form grown in perennial borders. Probably the one you have is a wild plant. It really depends on if you like it or not. It will spread, but not aggressively. It's up to you if you like it or not.