Irregularity of あ-series in demonstratives

Question

Why is the あ-series in demonstratives irregular like the following:

ここ　　そこ　　あ そこ　　　　　　どこ
　　　　　　　(expected あこ)
こう　　そう　　あ あ　　　　　　　どう
　　　　　　　(expected あう)

Is it related to the fact that こ, そ, ど all end with the vowel "o", and あ ends with the vowel "a"?

Answer 1

あそこ (expected あこ)

There was an ako. From the 1775 text 物類称呼 (Iwanami Bunko ISBN4-00-302691-8 p. 146):

あそこ　こゝといふを 西國にて・あんなけ こんなけと云 肥前にて・そこねい　こゝねいと云　尾州にて・あそこなて　こゝなてと云　京にて・あこと云

However, there are ample usages of asoko in much earlier works from the 13th century and on, so this is likely an abbreviation of asoko > ako.

Historically, in Old Japanese there was the ko/so/ka system. The ka form are the ancestor of modern a-forms, with the initial /k/ dropping out leaving only /a/. However, in the Old Japanese corpus, the ka forms are extremely rare and of those few that exist, they are all attributed to an eastern dialect. As such, it is thought that the distal (遠称) series had not been fully developed until yet early middle Japanese. It was a two-way system of speaker vs. non-speaker rather than priximal/mesial/distal. So that is one reason to not expect regularity in the [k]a-series. Further, the early signs of the [k]a- series are found in eastern dialects, so yo can expect variation when it merges with the western dialects.

Also, you seem to expect *あう based on こう, そう, and どう. However, I must take issue with こう and そう. The etymology of こう is kaku > kau > kɔː > koː, with a once medial -k-. For そう, it is sau > sɔː > soː. These two do not share a common etymology, so you cannot draw any conclusions from this alone for a ああ vs. あう. Besides, if it were あう, this would have regularly developed into au > ɔː > oː (おう).

As explained above, I think the question is based on a false premise so it cannot be answered satisfactorily.

Answer 2

The historical answer to this still appears to be somewhat of a mystery. However, there is speculation on why the あ (of the こそあど) appears to be irregular.

Nomichi Sumire gives a hint in this answer as to why あ is different. 「ここ・そこ・あそこ・どこ」 were all written in Kanji in the past, like this: 「此処・其処・彼処・何処」 When written out in Kanji, all of the words appear to be the same length of characters. But this doesn't really answer your question.

The interesting thing about 「彼処{あそこ}」 is that it can also be read as 彼処{あこ}. So, at least one point in time, あこ (itself) was used.

There is other speculation about why it's currently 「あそこ」 instead of 「あこ」 in an essay (labeled as fictional... so take it with a grain of salt,) on this page: The Mystery of ASOKO.

Namely, the focus point of this essay is in this paragraph:

場所を尋ねたのに対し、「ここ」「そこ」と、 近称・中称で断定するのは良いが、 遠称で答えては正確な場所を特定できず、 わからないのと同じである。 だから「あこ」は存在しない。 その代わり、遠称を断定するために、 「こそあど」の遠称である「あ」を中称の「そこ」に加えて、 「あそこ」と言う表現を用いたらしいのである。 これなら明確に場所を特定しながらも、 「そこ」より遠くを示すことを伝えられるからだ。

Basically, (and earlier in the article,) the writer is suggesting that 「ここ」and「そこ」are used when the locations being discussed are known locations (near or kind of near places.) But when the place is somewhere far away... (when the location is not completely known,) perhaps 「あそこ」is used because it's adding the 「あ」of the distal pronoun (遠称) and adding it to the「そこ」of the mesioproximal pronoun (中称) as a way of saying something to the effect of: "it's a place somewhere far over （あ＋そこ）there".

Again, it's hard to know for sure when that idea seems to be just speculation by the writer. But it gives a better idea of why 「あそこ」came in to common usage. It could very well also have to do with the vowel idea of yours; with languages like Japanese and Pacific languages (like Hawaiian, for example,) keeping the consonant-vowel combination tends to be important.

For the second part of the example in your question: the「こう・そう・ああ・どう」, try thinking about it like this: each of those expected ones 「こう・そう・どう」 have a long "o" vowel sound. It's similar to when the 「ー」 sign is used for a long vowel sign. When you have a long vowel sound for 「あ」, what do you use? 「あー」, right? Which is a long「あ」sound.