Is there any algorithm capable of encrypt securely (symmetric) N numeric digits in a numeric message of M digits (N < M)? I am especially interested in a specific case in which N = 10 and M = N + 3. I tried the most famous algorithms, like DES and AES, but they did not get success. Does anyone know a way to encrypt messages with limitations like this?
One way of doing this is using a Format Preserving Encryption method.
This is a symmetric encryption method that is designed to convert (potentially) small messages into encrypted messages of exactly the same size. For example, it may take strings of 13 decimal digits, and encrypt them into strings of 13 decimal digits.
One such method would be the FF3 method from NIST SP 800-38G; if you set radix=10, then it can encrypt a string of 13 digits, giving you a string of 13 digits (which is exactly what you want).
Here is how you would use it in your example: you would take your $N=10$ digit number, and extend it to $M=13$ digits by appending 3 0's. Then, you'd hand your string of 13 digits (and the key) to the FPE algorithm to encrypt, and it'll give you another string of 13 digits; that's your ciphertext. Decryption is the the opposite; FPE decryption of the 13 digits, and then strip off the last three 0's (and, if they're not 0, then the ciphertext was invalid)
There are existing techniques for encrypting messages that can't be expressed in an even number of radix digits; however since your messages can, there's no reason to involve those.
All three methods in 800-38G can handle radix=10; I suggest FF3 because the pseudocode listed for FF1 in the NIST draft is wrong (their decryption code isn't an exact inverse of their encryption code; yes, NIST knows this), and FF2 has known (academic) weaknesses.