In hindsight, I do think the right approach is not to use calculators till secondary school (about 14~15 years of age). Having a feel for numbers and how they should behave under addition/multiplication/exponentiation and ability to mentally do two digit calculations as you said is critical, and should be an “automatic” capability