Looking at encrypted pictures is not a good test. I mean, if you can visually see non-random looking parts in an encrypted picture or text, then this implies that the encryption system is abysmally bad. But seeing nothing from a visual inspection does not prove that the system is secure; it just suggests that it will not be easily broken by a chimpanzee.
The same can be said, to some degree, about statistical tests like the Diehard suite. Successfully passing the tests is no proof of security, only proof that you did not make a mistake so obvious that even a brainless generic statistical test would detect it.
To know whether a given encryption system is secure, cryptographers use the following recipe:
- They publish the system: complete algorithm description and sample implementation with test vectors (the code is not the description; the description should make no assumption on the involved programming language, and yet be sufficient to reimplement it perfectly in any language).
- They lure other cryptographers into gnawing at it and try to break it. The "luring" is about making the algorithm interesting, either through a some kind of mathematical novelty in the structure (e.g. a reduction proof with regards to a well-known hard problem) or impressive performance (see below).
- After enough time and work, if the cryptographers found nothing bad to say about the algorithm, we can begin to say that it may be worthwhile to consider in future protocol designs.
Let me stress that "enough time" means "at least five years and one hundred cryptographers". Yes, it is very expensive unless you can convince cryptographers to work for free. NIST did it quite well back in 1998, for the AES competition: they just promised glory (but not a single dollar !) to the winner, and they all came. But the real source of AES strength is not the three years of competition, it is the ten years of subsequent widespread usage and thorough continuous investigation. We have come up with a few interesting attack ways, not on the algorithm itself but on its implementations (side channel attacks); the existence of some results on AES shows that research on AES has kept on, and found nothing fatal. That's the good thing: whatever weakness AES may have is certainly not obvious and we can begin to build trust on that.
Note that there are guaranteed ways to ensure that a given algorithm will not be studied, and thus may never be declared "safe enough":
- do not publish the algorithm;
- publish the algorithm only as code (especially in an ill-suited language such as any dialect of Basic);
- patent the algorithm;
- make it considerably slower than the AES.
If you do any of the following, you totally forfeit your chances of seeing your algorithm being independently investigated, and therefore it will never be considered "secure".
Performance is quite easier to evaluate than security. For a symmetric encryption system, the following characteristics are absolute requirements for the system to be considered a non-joke, performance-wise:
- The input data shall be any arbitrary sequence of bytes (no restriction like "printable characters only").
- For an input message of size $n$ bytes, the encrypted message should have length at most $n+r$ bytes where $r$ is a small overhead, no more than twenty bytes or so, independently of $n$. Proportional size increase, even by a small amount (such as encrypting 1 GB of data into 1.01 GB), is a total no-go.
- The encryption process shall use only small amounts of RAM and code; apart from the data itself, the code and RAM for state and constants shall fit in the level 1 caches of the CPU. On a modern, "big" CPU like the one in a desktop PC, there is typically 32 kB of level 1 cache for code, and about as much for data. Some smaller architectures have smaller caches (e.g. 8 kB for code and 8 kB for data on a MiPS-based home routeur). The encryption shall be amenable to streaming, i.e. encrypting gigabytes by small successive chunks, while keeping only a few hundreds of bytes of state in RAM.
- The processing time to encrypt $n$ bytes shall be, asymptotically, proportional to $n$ (time to encrypt 2 gigabytes shall be twice the time to encrypt 1 gigabyte).
Given these characteristics, you can define encryption performance with the following formula:
$$ T = k*n + i + f$$
- $T$ is the encryption time;
- $n$ is the input data size, in bytes;
- $i$ is the processing time needed to begin the processing (initialization which can be done before seeing the first byte of data; the "key schedule" fits there);
- $f$ is the processing time needed for finalization (after all data bytes have been seen, the time needed to close up things appropriately and output the last encrypted byte);
- $k$ is the asymptotic encryption speed.
If you express all these values in "clock cycles", then $k$ is the encryption speed in "clocks per byte". The smaller the better. To measure $k$, have your code repeatedly encrypt an 8 kB data buffer so that the total processing time takes at least a few second (see this answer for a more detailed treatment).
With OpenSSL on a basic desktop machine, I get AES speed close to 17 cycles per byte (the processor is an Intel i7; it can do much better, below 2 cycles per byte, using the special AES opcodes, but OpenSSL does not use them). Some stream ciphers achieve much better performance (e.g. less than 4 cycles per byte with Sosemanuk). That's your competition.
Bottom-line: If your system cannot go below 20 cycles per byte on a PC, don't even bother with publishing it: it will not be "interesting enough". Harsh but true: symmetric encryption has gone quite far and is considered to be a "mostly solved" problem. It takes quite a lot of effort to design something which has the slightest chance of being an improvement over already known algorithms.