Dennis Martin wrote a great column about this very subject back in Fighting Arts, before it became defunct. You may wish to ask him if he still has a copy of the piece floating round.
So, on with your question. Obviously size counts for a lot...I am a firm believer in the adage "The bigger they are, the harder they hit." but, before you get discouraged, it's not a guarantee of success just as being small is no guarantee of failure.
When I first began on the doors I was only sixteen and, while tall, very skinny. (there's a photo of me doing a flying side kick over on the old GT forum someone may post here - I still can't figure out how damnit). Two of my instructors were small i.e. Mal Anderson and Tony Quinn were about your size and weight as was my older brother Anthony and they all successfully beat up/off groups of people. In Melbourne there were other small balltearers such as Trevor Barnes and Noel Hattwell was about 6'2 and 13 stone and punched and kicked like a sneaky mule.
So, while it's an asset (during the fight, not necessarily when you're standing in court trying to look innocent

) it's not the be all and end all of fighting.
Nick