Do you know how ~かねない and ~かねる work? かねる is a verb that literally means something like "to find it impossible/difficult" or "rather not". With this in mind, the phrase in question can be parsed like this:
- でんぐり返し: "roll (on a floor)"
- も: "even" (∼すら)
- うち: "to do (でんぐり返し)" (this 打つ is a verb collocated with でんぐり返し; see this)
- かね: continuative form of かねる
- まじく: "in a way that (something) seems unlikely/impossible" (continuative/adverbial form of まじ, which is an auxiliary for negative inference)
Put together, でんぐり返しもうちかねまじく literally means "in a way that even avoiding doing a roll seemed impossible". The ordinary way to say this in modern Japanese would be ひっくり返らずにはいられない(だろうという)ほど驚かされた (or more simply ひっくり返りそうなくらい驚かされた). So he was so surprised at the voice that he almost tumbled.