Adding to other answers, I note that both schemes are related to (but clearly different from) those standardized in ISO/IEC 14888-3:2016 (non-functional preview):
- The BSI's EC-Schnorr original specification was similar to ISO/IEC 14888-3's EC-SDSA-opt, standing for Elliptic Curve Schnorr Digital Signature Algorithm optimized version, except that EC-Schnorr outputs $H(M\mathbin\|Q_x)$ but EC-SDSA-opt outputs $H(Q_x\mathbin\|M)$.
ISO/IEC 14888-3 also defines EC-SDSA which outputs $H(Q_x\mathbin\|Q_y\mathbin\|M)$.
The BSI's current specification now matches the definition of EC-SDSA in ISO/IEC 14888-3.
- The question's libsecp256k1 (possibly matching this source) is similar to ISO/IEC 14888-3's EC-FSDSA (where the F stands for full), except that libsecp256k1 outputs (and later hashes) $Q_x$ where EC-FSDSA uses $Q_x\mathbin\|Q_y$. It follows that EC-FSDSA has a significantly larger signature than libsecp256k1 does (for $b$-bit security it is $6b$-bit, rather than $4b$-bit).
There are more variants related to libsecp256k1. A comment links to BIP-Schnorr.
Summary of the schemes discussed:
$$
\begin{array}{l|rllr}
\text{scheme}&\text{public}&\,\,\,\text{ first}&\,\,\text{ second}&\text{sign. }\,\\
&\text{key }\,\,&\text{component}&\text{component}&\text{size }\,\,\,\\
\hline
\text{[Sc91]}&-d\,G&H(Q,M)&k+d\;h&b+2b\\
\text{EC-SDSA}&-d\,G&H(Q_x\mathbin\|Q_y\mathbin\|M)&k+d\;h&2b+2b\\
\text{EC-SDSA-opt}&-d\,G&H(Q_x\mathbin\|M)&k+d\;h&2b+2b\\
\text{EC-FSDSA}&-d\,G&Q_x\mathbin\|Q_y&k+d\;H(Q_x\mathbin\|Q_y\mathbin\|M)&4b+2b\\
\text{EC-Schnorr old}&d\,G&H(M\mathbin\|Q_x)&k-d\;h&2b+2b\\
\text{libsecp256k1}&d\,G&Q_x&k-d\;H(Q_x\mathbin\|M)&2b+2b\\
\text{BIP-Schnorr}&d\,G&Q_x&k+d\;H(Q_x\mathbin\|\text{Pub}\mathbin\|M)&2b+2b\\
\end{array}
$$
- $M$ is the message to be signed.
- $G$ is the group generator.
- $k$ is the secret drawn by the signer (randomly or pseudo-randomly) at each signature.
- $Q=k\,G=\underbrace{G+\ldots+G}_{k\text{ terms}}$.
- $Q_x$ (resp. $Q_y$) is the $x$ (resp. $y$) component of $Q$ as a bytestring.
- $b$ is the intended security in bits; the group size $n$, $Q_x$ and $Q_y$ are about $2b$-bit.
- $H$ is the hash function, and $h$ is the result of the hash, often also the signature's first component. For $\text{[Sc91]}$, $h$ is $b$-bit wide, and the input of $H$ combines an unspecified fraction of $Q$ with $M$. In other schemes, $h$ is $2b$-bit wide.
- $\text{Pub}$ is the public key. Including it in the hashed data simplifies (perhaps strengthens) security reduction in a multi-users configuration.
- An implicit $\bmod n$ is omitted for the signature's second component.
- Conversion between integer and bistring is omitted (though essential for interoperability).
$\text{[Sc91]}$ is Claus-Peter Schnorr, Efficient Signature Generation by Smart Cards, in Journal of cryptology, 1991; refer to this for an exposition and more bibliography.
In $\text{EC-Schnorr}$, the signature's first component is an ASN.1 integer rather than a bytestring.
$\text{EC-Schnorr}$ (resp. $\text{EC-SDSA}$, $\text{EC-SDSA-opt}$, $\text{EC-FSDSA}$ ) have registered Object IDentifier 0.4.0.127.0.7.1.1.4.3 (resp. 1.0.14888.3.0.11, 1.0.14888.3.0.13, 1.0.14888.3.0.12).
These 7 schemes are incompatible; Andrew S. Tanenbaum's "The nice thing about standards is that you have so many to choose from" is an understatement!
ETSI's Electronic Signatures and Infrastructures (ESI); Cryptographic Suites TS 119 312 V1.2.1 (2017-05) states:
For interoperability reasons only one version (EC-SDSA-opt) from the EC-XDSA Schnorr variants defined in ISO/IEC 14888-3 is selected by the present document. EC-SDSA in the optimized version has the small advantage
of minimal data transfer for smart cards.
but that rationale works only against EC-FSDSA as far as I can tell. If the Smart Card itself computes or verifies the signature, as it should, what's transferred is the signature, which is $4b$-bit in both EC-SDSA-opt and EC-SDSA.
Hashing $M$ first is a specificity of EC-Schnorr; my guess is that it was designed for constrained environments like Smart Cards, where it allows offloading the bulk of the hashing of $M$ externally (also, on a multicore CPU, that allows hashing $M$ in a separate thread). As long as the hash is secure, it should not make any difference in security.