We have been playing with 4 stacks of huts regardless of the number of players. The rules actually indicate that the stacks of huts (7 per stack) in play are based on the number of players. This means that during the game earlier, we ought to have played with 3 stacks of huts instead of 4.
I like the rule - it adds to the other rules such as limiting the number of players procuring resources each round in enhancing the game's scalability. The designers have done a good job of ensuring the game doesn't lose any luster with less than the full complement of players.