The first sentence could be expanded to
高いなんてもんじゃないよ。むちゃくちゃ高いんだよ。
It's not (just) huge. It's humongous.
and the first part would be written as
高いというものじゃないよ。
As for the second sentence, separating the sentence as
人が多かったの 何の って
the 何の is used to repeat the structure of the first part, but could be replaced with anything, indicating that the speaker can't even say there were many people, because "many" isn't enough to describe the situation. The って is used to strengthen the speaker's feeling about what he said, as he is quoting himself in a way, cf.
絶対危険だって
To compare the two phrases, let's take the sentence
このホテルは汚かった。
Turning this into
- 汚かったなんてもんじゃない。
- 汚かったのなんのって。
the first means that the hotel was extremely dirty and the word "dirty" would not be strong enough to express just how dirty it was, whilst the second means that it was dirty, ugly, etc. and no single word would be sufficient to express the various ways in which it was unpleasant.
A related phrase is
楽しい ったらありゃしない
which comes from
楽しい と言ったら、有りはしない
and also means, very roughly とても, in this case "it was as great as it could possibly have been".
P.S. This use of 何{なん} is the same as seen in
電話なり何なり
(thanks, Tim).
If 何 is already used, one uses かん instead, as in
何だかんだ
何でもかんでもない