Assuming your deck is completely random and that it is not retrivable, how hard would it be for a casual person to decrypt a message encrypted with the Solitaire cipher?
Of course, it strongly depends on what exactly you mean when writing “casual person”. The cipher algorithm works similar to a shift cipher. As long as the attacker does not know the original order of the cards (of one or more decks of cards), then the cipher should be close to being unbreakable for a “casual person” – assuming that that casual person is someone we could call “an average Joe” and not a ”maths, statistics, or cryptanalytics fan”.
As for cryptanalytic security:
It should be noted that there is a chance the algorithm may repeat numbers since there is some bias (as pointed out by people like Paul Crowley in his Problems with bruce schneier's "solitaire"). While the probability that several numbers will be repeated consistently is rather low, you might want to take a look at his (and related) analysis nevertheless. After all, “having bias” is what broke RC4s neck too. (Yet, RC4 was used in a completely different context where several KBs of data were encrypted with a single key… which added to its bias becoming a weaknes.)
Also, it should not be ignored that if you encrypt with the same deck order and the same keyphrase twice (or more), then both messages can be solved by dragging probable cribs and expanding the solution… without ever attacking the Solitaire key-generation algorithm itself! This practically means Solitaire has the same weaknesses as an OTP system: key-distribution and user-discipline.
On the other hand, we should remember the Solitaire cipher is nothing but a pen-and-paper cipher so we can surely not expect the same security that cryptographic algorithms like AES provide. If you want to hide some personal information from your neighbour, Solitair will do a perfect job. But if you want to protect your country’s top secret information, I would definitely advise to use something else.