No, it is no longer secure. Kyber is an instantiation of a cryptosystem called "LPR Encryption".
Its public key is an LWE encryption of zero $(A, As + e_1)$.
Encryption proceeds by computing
- $u = r^tA + e_2$ (so $u^t = A^t r + e_2^t$), an LWE encryption of zero under the matrix $A^t$ (which is clearly correlated with $A$), and then
- $v = r^t (As + e_1) + (q/2)m + e_3$, an LWE encryption of zero using the random pad $(As + e_1)$.
To argue security, you first
Argue under the LWE assumption that $As + e_1$ is pseudorandom, i.e. you can replace it with a uniformly random string.
Argue that $v$ is now of the form $\langle r,pk\rangle + (q/2)m+e_3$, i.e. is an LWE encryption of $m$. You further need to argue that releasing $(rA^t + e_2)$ does not hurt security, but ignore this for now (it isn't needed to show the insecurity of your proposal).
Under your modification, this second step would no longer work, as $\langle r, pk\rangle + (q/2)m$ is not an LWE encryption (and is instead a "noiseless LWE encryption").
Noiseless LWE encryptions are known to be susceptible to simple attacks, say using Gaussian elimination.
So one could attack your cryptosystem by viewing ($pk, r^tpk + (q/2)m)$ as a noiseless LWE encryption of $m$, and running standard attacks in this setting.
The way I reached the above attack was simple --- look at the proof, see what step no longer works, and see if you can attack it.
I would highly encourage you to think through things in this way, rather than simply "It uses LWE so it's secure", as reasoning such as that is nearly never sound in cryptography.