Adding to other answers, I note that both schemes are related to (but clearly different from) those standardized in ISO/IEC 14888-3:2016 (non-functional preview):
- The BSI's EC-Schnorr original specification was similar to ISO/IEC 14888-3's EC-SDSA-opt, standing for Elliptic Curve Schnorr Digital Signature Algorithm optimized version, except that EC-Schnorr outputs $H(M\mathbin\|Q_x)$ but EC-SDSA-opt outputs $H(Q_x\mathbin\|M)$. The BSI wanted to simplify Smart Card implementations: with $M$ first, one can start the hashing of $M$ independently of the signature process.
ISO/IEC 14888-3 also defines EC-SDSA which outputs $H(Q_x\mathbin\|Q_y\mathbin\|M)$.
The BSI's current specification now matches the definition of EC-SDSA in ISO/IEC 14888-3.
- The question's libsecp256k1 (possibly matching this source) is similar to ISO/IEC 14888-3's EC-FSDSA (where the F stands for full), except that libsecp256k1 outputs (and later hashes) $Q_x$ where EC-FSDSA uses $Q_x\mathbin\|Q_y$. It follows that EC-FSDSA has a significantly larger signature than libsecp256k1 does (for $b$-bit security it is $6b$-bit, rather than $4b$-bit).
There are more variants related to libsecp256k1 in blockchain contexts. Update: like BIP-340 Schnorr, which seems to have taken over libsecp256k1 as in the question.
Summary of the schemes discussed:
$$
\begin{array}{l|rllr}
\text{scheme}&\text{public}&\,\,\,\text{ first}&\,\,\text{ second}&\text{sign. }\,\\
&\text{key }\,\,&\text{component}&\text{component}&\text{size }\,\,\,\\
\hline
\text{[Sc91]}&-d\,G&H(Q,M)&k+d\;h&b+2b\\
\text{EC-SDSA}&-d\,G&H(Q_x\mathbin\|Q_y\mathbin\|M)&k+d\;h&2b+2b\\
\text{EC-SDSA-opt}&-d\,G&H(Q_x\mathbin\|M)&k+d\;h&2b+2b\\
\text{EC-FSDSA}&-d\,G&Q_x\mathbin\|Q_y&k+d\;H(Q_x\mathbin\|Q_y\mathbin\|M)&4b+2b\\
\text{EC-Schnorr old}&d\,G&H(M\mathbin\|Q_x)&k-d\;h&2b+2b\\
\text{libsecp256k1}&d\,G&Q_x&k-d\;H(Q_x\mathbin\|M)&2b+2b\\
\operatorname{BIP-340}&(d\,G)_x&Q_x&\hat k+\hat d\;H(Q_x\mathbin\|\text{Pub}\mathbin\|M)&2b+2b\\
\end{array}
$$
- $M$ is the message to be signed.
- $G$ is the group generator, of order $n$.
- $d$ is the private key.
- $k$ is the secret drawn by the signer (randomly or pseudo-randomly) at each signature.
- $Q=k\,G=\underbrace{G+\ldots+G}_{k\text{ terms}}$.
- $Q_x$ (resp. $Q_y$) is the $x$ (resp. $y$) component of $Q$ as a bytestring.
- in BIP-340
- $\hat d$ is $d$ or $n-d$, such that $(\hat d\,G)_y$ is an even integer in $[0,p)$, where $p$ is the field order. Similarly, $\hat k$ is $k$ or $n-k$, such that $(\hat k\,G)_y$ is even. The BIP-340 specification uses $d',d,k',k$ where we use $d,\hat d,k,\hat k$.
- $\text{Pub}$ is the public key, a 32-byte bytestring. Including it in the hashed data simplifies (perhaps strengthens) security reduction in a multi-users configuration.
- $b$ is the intended security in bits; the group size $n$, $Q_x$ and $Q_y$ are about $2b$-bit.
- $H$ is the hash function, and $h$ is the result of the hash, often also the signature's first component. For $\text{[Sc91]}$, $h$ is $b$-bit wide, and the input of $H$ combines an unspecified fraction of $Q$ with $M$. In other schemes, $h$ is about $2b$-bit, or wider and reduced modulo the group order.
- An implicit $\bmod n$ is omitted for the signature's second component.
- Conversion between integer and bistring is omitted (though essential for interoperability).
$\text{[Sc91]}$ is Claus-Peter Schnorr, Efficient Signature Generation by Smart Cards, in Journal of cryptology, 1991; refer to this for an exposition and more bibliography.
In $\text{EC-Schnorr}$, the signature's first component is an ASN.1 integer rather than a bytestring.
$\text{EC-Schnorr}$ (resp. $\text{EC-SDSA}$, $\text{EC-SDSA-opt}$, $\text{EC-FSDSA}$ ) have registered Object IDentifier 0.4.0.127.0.7.1.1.4.3 (resp. 1.0.14888.3.0.11, 1.0.14888.3.0.13, 1.0.14888.3.0.12).
These 7 schemes are incompatible; Andrew S. Tanenbaum's "The nice thing about standards is that you have so many to choose from" is an understatement!
ETSI's Electronic Signatures and Infrastructures (ESI); Cryptographic Suites TS 119 312 V1.4.2 (2022-02) states:
For interoperability reasons only one version (EC-SDSA-opt) from the EC-XDSA Schnorr variants defined in ISO/IEC 14888-3 is selected by the present document. EC-SDSA in the optimized version has the small advantage of minimal data transfer for smart cards.
The stated advantage of EC-SDSA-opt against EC-FSDSA comes from the smaller signature size, and against EC-SDSA from the smaller return from the Smart Card in preparation of external computation of the hash: $Q_x$ for EC-SDSA-opt, versus $Q_x\mathbin\|Q_y$ for EC-SDSA and EC-FSDSA.
Hashing $M$ first is a specificity of EC-Schnorr. In constrained environments like Smart Cards, it allows offloading the bulk of the hashing of $M$ externally (also, on a multicore CPU, that allows hashing $M$ in a separate thread). Ed25519ph does the same. As long as the hash is secure, it should not make any difference in security.
I thank Ernst G. Giessmann for insights on the rationale of the various versions.