This is a great question. I searched the Iwanami mathematical dictionary 『岩波数学辞典』 and Sasahara's 当て字 dictionary, 『当て字・当て読み 漢字表現辞典』, and did not find a definitive answer. Here's what I did find, though:
The word 函数 was invented in China, not Japan. The characters were chosen for phonetic value as well as meaning. It's possible that "box that numbers go into" was the intended interpretation, but we can't know for sure. What we can know is that it was the closest the translators involved could get to the "fun-" sound. [Edit: If I had bothered to read the Wikipedia article, I would know that it said that 函 was chosen as the /han/ sound for its meaning of "enclose, contain", and is apparently used in the sense that that the independent variable (x) is included within the dependent one (y). The "black box" interpretation is specifically rejected.]
The spelling 関数 was created in Japan once the postwar kanji and vocab lists (tōyō kanji, 学術用語集) were created without 函 on them. 関数 is ateji upon ateji. The two forms would have coexisted for a while before 関数 became dominant. As Tsuyoshi's answer says, this means that we can basically ignore the question of 関 vs 函; official kanji lists aren't able to cause sudden and discrete changes in everyone's usage, so it's no big deal if we see some inconsistency and overlap.
What follows is pure speculation.
- If "functor" was coined in the 1940s, then 関手 was probably coined in Japan, because by that period Western science was not absorbed via China.
- The English word is the root of "function" plus the Latinate suffix -or which means "(male) person who performs a task." Actor, director, operator, etc.
- So maybe whoever coined the word 関手 took the first half of 関数 and added the 手 of 運転手, 歌手, etc., by specific analogy with -or.
Basically I am in agreement with your theory and offering a reason why a suffix normally associated with people would have been chosen: because that's how the English breaks down, too. It may have been the very fact that 手 is no longer productive that made it the choice for calquing -or: as Tsuyoshi says, using a productive suffix like 者 might have carried too strong an implication of "(actual) person."
