Can you use SHAKE256_XOF with XOR to make a stream cipher?

Question

SHAKE256_XOF is an extendable output function based on SHA3-512. SHAKE256_XOF can be used to create an infinite length bit string with 256-bits of security. For example:

from Crypto.Hash.SHAKE256 import SHAKE256_XOF

shake = SHAKE256_XOF()
shake.update(b'key' + b'nonce')

for _ in range(10):
    print(shake.read(16).hex())

18068a140f4c0f1f85f17da13b049155
1fc6d25b11833480dae5fba50b8b73db
bf4284c0d9a32b9b374ae82d5c0514c6
...

SHAKE256_XOF is a pseudorandom function so XOR-ing its output with a plaintext should produce a ciphertext. My understanding is that this would be an asymmetric stream cipher. I am having a hard time finding research on this other than it being possible and Bruce Schneier saying "I know of no cryptanalytic analysis of particular one-way hash functions as block ciphers; wait for such analysis before you trust any of them." Finally, this cipher can be made authenticated by SHA3-512 as part of the seed. This leads to the final algorithm:

from Crypto.Hash.SHAKE256 import SHAKE256_XOF
from Crypto.Hash import SHA3_512

xor = lambda lhs, rhs: bytes(a ^ b for a, b in zip(lhs, rhs))

plaintext = b'plaintext'
key = b'key'
nonce = b'nonce'
digest = SHA3_512.new().update(plaintext).digest()

e_shake = SHAKE256_XOF().update(key + nonce + digest)
cipher_text = xor(e_shake.read(len(plaintext)), plaintext)

d_shake = SHAKE256_XOF().update(key + nonce + digest)
d_plaintext = xor(d_shake.read(len(cipher_text)), cipher_text)

print(d_plaintext)

1. Is there any literature on using SHAKE256_XOF as a cipher?
2. How does this method compare to AES256 in terms of security?
3. What are there weaknesses or disadvantages of this cipher? Obviously, the use of SHA3-512 to make it authenticated means it can't be efficiently encrypted in one pass, and decrypting requires calculating the SHA3-512 digest to authenticate (I am learning how authenticated encryption works so suggestions and feedback are welcomed).
4. Does it provide any possible advantage over AES256?
5. AES256 has 128-bit resistance against theoretical quantum attacks. Would this cipher maintain the min(key_length/2, 256) bits of resistance to theoretical quantum attacks that SHAKE256_XOF has.

Answer 1

In section 3.2 "Modes of use of sponge functions" of Cryptographic Sponge Functions (CSF for short for the purpose of this answer), the authors listed several "modes of operations" for different purposes, with stream cipher applications being one of them. Here's the table:

Functionality	Expression	Input	Output
$n$-bit hash function	$h = H(M)$	$M$	$\lfloor Z \rfloor _n$
$n$-bit randomized hash function	$h = H_R(M)$	$R|M$	$\lfloor Z \rfloor _n$
$n$-bit hash function instance differentiation	$h = H_D(M)$	$D|M$	$\lfloor Z \rfloor _n$
Slow $n$-bit one-way function	$h = H_{\text{slow}}(M)$	$M|0^N$	$\lfloor Z \rfloor _n$
$n$-bit MAC function	$T = MAC(K,[IV,]M)$	$K|IV|M$	$\lfloor Z \rfloor _n$
Random-access stream cipher ($n$-bit block)	$z_i = F(K,IV,i)$	$K|IV|i$	$\lfloor Z \rfloor _n$
Stream cipher	$Z=F(K,IV)$	$K|IV$	as is
Deterministic random bit generator (DRBG)	$z=DRBG(seed)$	seed	as is
Mask generating and key derivation function	$mask=F(seed,l)$	seed	$\lfloor Z \rfloor _l$

Is there any literature on using SHAKE256_XOF as a cipher?

Well, the paper has you covered.

How does this method compare to AES256 in terms of security?

This method has equal strength in terms of brutal force. a SHAKE stream cipher will be able to have longer periods than AES-256 either counter mode (CTR) or cipher feedback mode (CFB).

What are there weaknesses or disadvantages of this cipher? Obviously, the use of SHA3-512 to make it authenticated means it can't be efficiently encrypted in one pass and decrypting requires calculating the SHA3-512 digest to authenticate (I am learning how authenticated encryption works so suggestions and feedback are welcomed).

Actually, if you look at "duplex" modes (which the CSF paper also covers), AEADs are actually easier to implement in one pass.

Does it provide any possible advantage over AES256?

I've mentioned 2 (period length, 1-pass). It also has better parameter scalability than fixed-width block-cipher, so I'm counting this as the 3rd.

AES256 has 128-bit resistance against theoretical quantum attacks. Would this cipher maintain the min(key_length/2, 256) bits of resistance to theoretical quantum attacks that SHAKE256_XOF has.

Probably, but I'm not too good at maths.