Do I have to be as buff as a rhino to be Special Forces?

I'm tall but actually pretty skinny, 6'1 and like almost 140 lbs. I'm in good shape though, I have muscle but I'm very 'toned' not 'built' ... the recruiter told me that the big buff special forces guy is a movie thing, and they really look for the most fit, because stamina plays a bigger role... is that true, or are they just trying to get me to join faster or so?