If you use the raw RSA operation ($M^d \bmod n$ or $M^e \bmod n$), then no, it is unsafe to use the same key, because an attacker could trick the private key holder into signing a message $M$ (i.e. generating $M^d$) which is actually an encrypted message ($M = P^e$), thus allowing the attacker to recover the original plaintext ($(P^e)^d = P$). (The dual attack leads to a forgery.) However, the raw RSA operation has a whole range of other flaws that make it unsafe as an encryption or signature scheme.
Encryption and signature schemes based on RSA use padding modes. The standard padding modes include discriminants so that an encryption payload does not look like a signature payload. Therefore, it is safe, cryptographically speaking, to use the same RSA key pair for signature and encryption, provided that the key pair is used safely for signature and used safely for encryption.
However this is a bad idea for a different reason: key management. Signature keys and encryption keys have different requirements in terms of backups, access control, repudiation, etc. The fallback for a signature key in case of a catastrophic event is to destroy it to avoid future forgeries, so a signature key does not need to be backed up extensively. Conversely, the fallback for an encryption key is to keep it around to decrypt existing documents, so it needs to be backed up reliably. In case of a leak of a signature key, pre-existing documents are not affected; it is only newly produced signatures that must be repudiated. Conversely, in case of a leak of an encryption key, the confidentiality of all pre-existing documents is at risk, whereas new documents simply need to be encrypted with different keys. The constraints push in different directions, so the keys need to be managed differently, they need to be different.