My understanding is that the memory bandwidth of CPUs and GPUs is roughly one order of magnitude difference4, unlike cores which a GPU has many of and a CPU a handful. That is why PBKDF2-HMAC-SHA1 parallelizes very well (which needs 164 bytes of memory1, fitting inside the registers of a GPU's streaming processors) whereas bcrypt, scrypt, or Argon22 don't necessarily—not without a relatively large amount of independently accessible RAM per core.
The purpose of the parallelization parameter of Argon2 seems to be to fill the memory faster, i.e. to make it faster when running on a CPU as compared to cracking hardware6. The most common cracking hardware continues to be GPUs, and they have gigabytes of RAM, just that this big chunk is shared between all cores and therefore you lose the big advantage you had with non-memory-hard algorithms.
I've been able to partially confirm this experimentally: the speedup for PBKDF2, i.e. the number of hashes you can do per second on a CPU vs. on a GPU, is much greater than for bcrypt, scrypt, or Argon2. However, the comparative speedup when setting a higher L (lanes) value in Argon2 actually gets worse for the CPU. Setting L=2 might halve the time compared to L=1 when all else is equal, but on a GPU it might quarter the time instead, giving your attacker an advantage.3
The following two graphs show the number of hashes per second for the GPU and CPU with different parameters.
The CPU speed increases until roughly the number of cores available. The 4 MiB (second) graph is a bit less accurate because it runs very fast.
The next graph shows the speedup factor achieved with different parameters.
The speedup you get on CPU from using more lanes allows you to use more memory without taking more time (thus not annoying users more but getting a should-be-stronger hash). This should protecting better against a potential ASIC5, but it seems to weaken the protection from GPUs which typical attackers would actually use in practice.
Why is that? Should the GPU not be running into the same memory bandwidth issue as your CPU is, at least as soon as the CPU saturates the available memory bandwidth (and I assume the GPU always does due to the plentiful cores)? (Of course, this assumes the memory requirements are beyond the caches local to the streaming processor or streaming multiprocessor.)
The results do not seem to match the theory and intended goals as I understood them. Must the implementations I used be flawed, do I understand the theory wrong, the intended effect or the use of the lanes, is this hardware an outlier, or something else?
The raw data is available in this content-containing link.
Implementations and hardware used:
To run on the RX 5700 GPU: Ondrej Mosnáček's argon2-gpu-bench
$ ./argon2-gpu-bench -m opencl -M $((16*1024)) -T 32 -L 4 -t id | grep -a Mean\ com | grep -a per // outputs a value with a unit like "3.91383 us"
To run on the Ryzen 7 3700X CPU: the PHP function
password_hashwhich uses libsodium under the hood if I am not mistaken.
$ php -r '$t=microtime(true); password_hash("test", PASSWORD_ARGON2ID, ["memory_cost"=>1024*4, "time_cost"=>32, "threads"=>4]); print(microtime(true) - $t) . "\n";' // outputs the time taken in seconds
I also tried doing multiple calls (the way
argon2-gpu-benchdoes by default) but I think the function call overhead defeats any benefits because the results did not seem to be any different or significantly more stable.
It might be worth noting that the highest amount of memory tested with is 256 MiB and not multiple gigabytes as the CFRG recommends because with as little as 512 MiB:
$ ./argon2-gpu-bench -m opencl -M $((512*1024)) terminate called after throwing an instance of ‘cl::Error’ what(): clCreateBuffer Aborted
(And scrypt is no better, hashcat's implementation errored out at 16 MiB until last week and now 64 MiB is the maximum. The GPU reports having >4000 MB allocateable on both Windows and Linux. PHP has no issues computing Argon2 with e.g. 13 GiB.)
1 Ondrej Mosnáček "Key derivation functions and their GPU implementations" https://is.muni.cz/th/409879/fi_b/
2 Whenever I say just Argon2, I assume it goes for all variants, but Argon2id is the one I am specifically looking into.
5 The Argon2 paper writes: "We aim to maximize the cost of password cracking on ASICs. There can be different approaches to measure this cost, but we turn to one of the most popular – the time-area product [4, 16]. [...] The 50-nm DRAM implementation  takes 550 mm² per GByte; The Blake2b implementation in the 65-nm process should take about 0.1 mm²". I conclude from this that using more memory (M) increases the manufacturing cost of an ASIC more than more passes over said memory (T) do while the time it takes can be kept unchanged.
5 "The parallelism was probably one reason why Argon2 won the Password Hashing Competition. The use of processor cores allows for greater memory hardness (security) without increasing the execution time accordingly." and "Parallelism is actually used to fill memory more quickly. The more hardware threads you use, the faster the memory can be filled and operated on."