When one generates a public and private key with say a total of 4096 bits, why does the program output what I assume to be base 64 instead of base 10 or binary?
It depends on the application if base 64 is being used to represent keys. Many applications that implement/use cryptography have been originally designed in a time where ASCII based communication was commonplace. If you would directly use BER / DER - a binary encodoing of parameters - then you had a high chance of losing data. For instance, you would not be able to send keys over mail or using text based FTP. Even today this can make sense, e.g. when integrating signatures in XML. Base 64 is then the safe default.
This is why many applications use a special format such as PEM which stands for privacy enhanced mail. It can of course be used for other textual interfaces as well, but it was originally designed for mail. Another term is "ASCII armor", which indicates the intent: protect binary encodings from being mangled by textual interfaces.
A decimal interface would not work; keys don't just consist of a single or multiple numbers. BER/DER are binary encoding engines for ASN.1, which is able to represent almost any data strucure including strings, dates, object identifiers and combinations of those. Furthermore, as already indicated, decimals would be very inefficient as well.
So often keys, certificates and signatures are first binary encoded and then "protected" using base 64.
Base-64 is simply a way to represent binary data using the ASCII character set.
It's used because a single base64 digit represents a whole number of bits (6 bits), where each decimal digit represents ~3.3 bits, which can make conversion a little tricky.
Being able to represent more bits per digit also means its more space efficient than decimal. It takes 683 base64 digits to represent a 4096-bit value, where it would take 1234 decimal digits.