Sponge functions were only finalised in conference in 2007, whilst the initial NIST document came out in 2006. The DRBG behaviour of block ciphers and hashes has been known a lot earlier than 2006. So the repeatedly squeezed sponge technique was not available at the time of NIST's original publication, and even had it had been, it would not have been analysed/tested.
Economic might means that many countries follow the States' lead on technology and it's certification. So in order to use your piece of 'stuff', it may have to have NIST /FIPS certification. Whilst FIPS and NIST complement each other, especially in US financial and (non military) governmental circles, NIST makes FIPSes.
So in summary, it's more of a legal requirement, not a mathematical one. If the contract says "NIST-SP-800-90A", then "NIST-SP-800-90A" it. XOFs and KDFs are not usable at all in that context.
But you're right, and Sponge-based pseudo-random number generators is a good example design. However strong KDFs might not be great. Anything like Argon2 with low/trivial settings would negate their very purpose bringing more appropriate constructs into play.
Notes:
In addition to your points #1 and #2, there is #3, Resistance against state recovery (§4.2 of the linked document). It is as it sounds.
RadioGatún (2006) preceded the formal sponge, but seems to be been overlooked even though it was in a NIST competition.
Perhaps(?) NIST will revise again and incorporate XOFs in the future following SHA-3 2014 Workshop
- Many good folks will have worked on NIST-SP-800-90A. It is interesting then that page iii specifically has this acknowledgement, mentioning two people by name - "The National Institute of Standards and Technology (NIST) gratefully acknowledges and
appreciates contributions by Mike Boyle and Mary Baish from NSA for assistance in the
development of this Recommendation. NIST also thanks the many contributions by the public
and private sectors".
- See Federal Information Security
Modernization Act of 2014 § 3553 "Authority and functions of the Director and the Secretary" for opinion regarding #4.
Update
methods. That's something that is commonly missing from XOF's and KDF's (let alone stream ciphers). I think this detail is often overlooked. $\endgroup$ – Maarten Bodewes♦ Apr 11 '19 at 18:34