Many lattice based cryptosystems are constructed using linear algebra, which means they utilize little more then multiplication and addition operations (sometimes combined in fancy ways).
Assuming that you find an algorithm that is simple enough to perform mentally or with a simple calculator, the next practical problem will probably end up being the size of the numbers involved.
You cannot obtain any meaningful security if your numbers are not large enough. For example, if your brain or calculator can only deal with numbers that are less then 64-bits, then it does not matter if you have a provably secure algorithm - any real adversary can simply guess your key material.
Conventional wisdom says that you need at least 80-bit numbers to stave off basic brute force attacks. Conventional wisdom also says that normal people can't do math on 80+ bit numbers, but your adding machine/calculator might be able to deal with it. Of course, it's possible that your "adding machine" that can operate on big numbers is actually just a small computer in disguise, which means you'll effectively end up back at regular encryption software running on a computer...