Base64 encoding regroups three consecutive 8-bit values (0-255) into four 6-bit values (0-63). Each value is used to index of a small ASCII array which is typically sequential, such as,
var base64Map = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=';
Base64 decoding is the reverse process. Each character's position is an indexof a matching ASCII array which results in a 6-bit value. Four of these are regrouped into three 8-bit values.
In your case, I suspect the base64Map was shuffled, rather than a separate shuffle operation of the encoded plaintext. If this is the case, then you need to match the order of the encoding array. If truly random, this would become a monumental effort. Perhaps try a Caesar Shift of the array.