Thanks for writing. Orange trees produce flowers with both male and female reproductive organs --- so you don't have a male tree. I can tell you that. But why it's not growing more, I can't be sure. Did you grow this plant from seed? Seed-grown varieties don't usually grow into the exact same variety as their parents, so it could be that's just what your particular plant does. Or it could be stunted from physical damage or growing conditions it does not like (if the soil is too heavy, if it doesn't get enough nutrition, etc).
For more help, I'd contact the folks at your county cooperative extension service. They should be able to help more if you can answer questions about the tree.
---Justin, Senior Garden Editor, BHG.com