100 MiB is nothing nowadays. Still, it is likely that most time is spend because of I/O operations, rather than encryption / decryption. Once the file is buffered in RAM, the operations will proceed much more quickly. Worse than that, during encryption most of the time is spend during compression of the input - not encrypting it. Try -z 0 so compression of ...
Symmetric cryptography could well work here, to be honest.
The main advantage of public cryptography in this setting is that you can encrypt by just using a public key. So you'd get the following operations:
create public / private key pair;
encrypt private key with password.
create random session key;
encrypt session key using public ...