Wearing a bra 24/7 has actually been noted to increase chances of breast cancer and fibrocystic disease?. It's actually healthy to let your breasts move around a little and be free during part of the day you're awake, too... it really helps with lymphatic circulation, and can keep your breasts healthy.
Originally, bras weren't necessary to people, even with very large mammaries. Then eventually, everyone was told they had to wear one to keep from sagging or whatever. Now, most breasts need a bra, but it's like any other body part. If you put your arm in a sling for 6 months, it would be weak and feel like it needed a sling. Bras don't keep breasts from "sagging" and if your breasts are in a bra all of the time, they may even sag more in the end.
I'm not going to tell you to ditch a bra completely (unless you want to... that's all a personal choice) but I'd definitely recommend not wearing one at night, and even at other times when you feel comfortable taking it off.