The question appears to be: we have algorithms that determine where $a$ is a quadratic residue modulo a prime $p$; what happens if we run the same algorithm, but instead of giving it a prime $p$, we give it a composite number $N$?
Well, there actually two common algorithms used to determine whether a value is a quadratic residue modulo a prime, and they differ in what they do if you feed them a composite:
The first test is to compute $a^{(p-1)/2} \bmod p$; the result is 1 if $a$ is a quadratic residue, -1 if $a$ is a quadratic nonresidue (and 0 if $a=0 \bmod p$)
What happens if you feed it a composite number; that is $a^{(N-1)/2} \bmod N$? Well, it is far more likely that the result will be something other than 0, 1 or -1; that would prove that $N$ is not a prime.
This is a slightly tighter version of the Fermat primality test; it is indeed useful as a quick check. However, there are composite numbers that will fool it with high probability; if we are searching for a prime number, and it passes the Fermat test, we generally want to go on with other tests as well.
The other test I referred to is computing the Legendre symbol; this is a way to compute a value that is equal to $a^{(p-1)/2} \bmod p$; however it uses a different algorithm (and is often faster, depending on what your computing resources are).
However, if you feed in a composite number $N$ into the Legendre symbol logic, it no longer is the same value as $a^{(N-1)/2} \bmod N$; instead, it will always remain either 0, 1 or -1 (and so is not useful for determining whether $N$ is composite); instead, you've just computed what's known as the Jacobi symbol.
If the Jacobi symbol is 0, then $a$ and $N$ are not relatively prime. If the Jacobi symbol is -1, then $a$ is not a quadratic residue. However, if it is 1, it might be (and it might not be; determining any more information appears to be a hard problem)