While refined carbohydrates like sugar and white flour — which have been stripped of fiber and other nutrients — are definitely not healthy and are associated with weight gain, the opposite is true for unrefined carbohydrates like oats, bananas and sweet potatoes — with their fiber and nutrients left intact — which are healthy for a variety of reasons and are consistently associated with decreased body fat.
This would explain why people who eat plant-based diets, despite getting more of their calories from carbohydrates than those on animal-based diets, have lower average levels of body fat. More on that, over at Getting and Staying Lean.