The problem may be the variety of raspberry you're growing. For example, a spring-bearing variety planted in your climate might be suffering from frozen fruit buds. Although raspberries are hardy in Zones 4-9, they de-acclimate very quickly during warm winter weather. A cold snap later in winter can cause dieback or damage to fruit buds. My best suggestion is to build a raised bed in full sun, with lots of organic matter in the soil, and plant a new raspberry variety that's suitable for your climate. Consider growing an everbearing raspberry such as 'Heritage'. That way, even if the summer crop is frozen off, you'll get a crop in fall.