According to Wikipedia, this homebrew cipher was originally used in PGP, before Phil Zimmermann replaced it with IDEA. Supposedly, insecurities in the algorithm were pointed out to him, leading to this change. While I have read the PGP 1.0 source code and saw the implementation of the cipher, that didn't help me actually understand it. I can't seem to find any complete description of the algorithm or, particularly, its weaknesses other than what is provided on Wikipedia:
The chosen key schedule produces a total of 8 permutation tables, each a permutation of the numbers 0 to 255. Each round consists of 4 operations: XORing the block with one of the permutation tables, shredding or permuting individual bits throughout the block, an unkeyed diffusion called raking, and a substitution step using the permutation tables as S-boxes. The shredding step can either permute all 8 bit-planes independently, or in groups of 4, depending on control bit 3. The permutation tables stay the same through all rounds of encryption on a given block, but if control bit 5 is set, then the tables are regenerated after each block.
I could also find a published quote from the book which Wikipedia used as a citation:
After Biham and Zimmermann go their food and sat down, Zimmermann took out a few pages of computer listings. Within minutes, Birham was finding fundamental flaws in Bass-O-Matic. Some of the flaws were subtle-weaknesses that made the algorithm susceptible to differential cryptanalysis, which was Birham’s speciality. Others were more embarrassing, like a conceptual error in Zimmermann’s algorithm that prevented the last bit of each byte from being properly encrypted. After ten minutes of Birham’s onslaught, Zimmermann realized that Bass-O-Matic was a lost cause.
How did BassOmatic work, and what were the weaknesses that lead to its replacement?