I would like to know how openssl verifies a digital signature? I know that it uses this command to verify a signature:
openssl dgst -sha256 -verify pkypem -signature signbin msgbin > result
What I want to know is, what openssl does exactly with the public key, the signature and the message before verification.
I'm also interested in the signature creation process.
It depends on the type of key, and (thus) signature.
If it is an RSA key, by default OpenSSL uses the original PKCS1 'block type 1' signature scheme, now retronymed RSASSA-PKCS1-v1_5 and currently defined in PKCS1v2.2. OpenSSL commandline also supports the RSASSA-PSS scheme (commonly just PSS) defined in the preceding section of PKCS1v2.2, with the dgst -sigopt option (online copy of man page) whose relevant values are actually documented in the pkeyutl man page.
If it is a DSA key, OpenSSL implements the algorithm as specified by the standard, FIPS186 currently at revision -4 and available at https://csrc.nist.gov/publications/fips (edited: usually, but download currently hangs presumably due to the government shutdown even though the webpage does display which initially fooled me); a shorter, less bureaucratic and more reliably available description is in Wikipedia. One minor point not stated in FIPS186, or Wikipedia, is that the signature value, which is mathematically a pair of integers, is concretely represented using ASN.1 (as a SEQUENCE of two INTEGERs). (In contrast PKCS1 for RSA defines the representation of the signature, which is mathematically an integer in $[2,n-1]$, using the I2OSP and OS2IP primitives, as bigendian (unsigned) binary with size equal to that required to represent the modulus.)
If it is an EC key, OpenSSL implements ECDSA as specified by X9.62, which costs money, but is also available in document SEC1 at http://www.secg.org (which ironically can't easily be accessed by https because it has a mismatched certificate!), and again in Wikipedia. Again the signature is mathematically a pair of integers, represented in ASN.1.
All of the above sources cover signing, aka signature generation or creation, as well as verifying for a given algorithm.
If it is a GOST key, OpenSSL may or may not work depending on version and build or packaging; this support has fluctuated. And if it works I'm not familiar with what it does, except that it is somehow like ECDSA.
Note that the 'raw' format used by openssl dgst -sign/verify, and openssl pkeyutl -sign/verify which skips the (data) hashing step (and for RSASSA-PKCS1v1_5, optionally the ASN.1 encode/decode step), is not used by most other software. Most actual signatures (for things like emails and code files and XML or JSON data, and certificates and CRLs and OCSPs and timestamps, as well as the signatures used in protocols like SSL/TLS and SSH) are represented in a form that adds metadata about the signature algorithm and a way to find the publickey for verification, and often to validate that key, and sometimes contain or are attached to the data signed as well.
