A fast metabolism is not the only reason behind it. The answer to this question is quite complex as several factors are at play. Genetics, nutrition, and even behavioural factors are involved which help slim people to maintain their body weight. Each individual is different and the way they can maintain their weight depends on the fact how much the aforementioned factors interfere in their daily lives.
Also, most people who appear to eat a lot in front of you actually eat as much as you do. Just because they have sweets in front of you every day does not mean they eat more. It could be possible that they compensate for it later by having only two meals in a day. That means their daily calorie intake remains the same as yours.