American Apparel Flex Fleece Sizing?

I'm 5'9 and a UK size 8/ US size 4 and I get them in medium, they're the way you said you want it, baggy but not too baggy, but because you're quite a bit shorter, maybe you should get a small?