… SHA3 (Bouncycastle) constrains me …
Bouncycastle offers the NIST approved, fixed, and standardized output lengths of the keccak sponge function.
See, when talking about SHA-3, you're talking about those NIST standards. This was built upon the proposed and accepted keccak sponge function.
NIST standardized SHA-3 to those lengths (which libs like Bouncycastle cover) to match up with the SHA-2 output standard perfectly, so that SHA-3 can be used as an immediate drop-in replacement in case SHA-2 (let's just say) "goes down" one day.
Yet… just because NIST created/fixed those standard output lengths it doesn't mean that the underlying sponge function couldn't be used to create other output lengths.
In the end, the only thing you are missing is the fact that many programming libraries (including Bouncycastle) focus on providing functionality that adheres to standards, not custom functionality.
… is there a specific way to generate "arbitrary length" using bouncy castle implementation …
Well yes, if you look at the BC lightweight API you’ll discover the option to use SHAKE128 and SHAKE256 – which does exactly that. (hint:
If you think the SHAKE128/256 functions don’t fit your needs (they actually should), that's where programming comes in. If you want something special (read: non-standard), you'll have to grab the core sponge functionality and create your own corresponding functionality you think you need. Libraries like Bouncycastle will hardly be of much use here.
But it's not as if there aren't projects and sourcecode pieces out there on the web that already offer what you are looking for. Simply use your favorite search engine to find them for whatever programming language you deem suitable and fitting your needs.
… standardized version of Keccak is different from the original one that was submitted to NIST …
Yes, you are also correct related to SHA-3 and Keccak being different. They are both sponge functions… but SHA-3 was built upon Keccak, with a small tweak introduced by NIST. This small difference within the sponge functionalities of SHA-3 and Keccak makes them produce different outputs for the same output length.
Practically, both are considered cryptographically secure. So, when wanting arbitrary length of output, you'll have to decide what sponge function you want to use. Looking at Gimli, I'ld like to note that there are other sponge functions out there which are suitable for cryptographic purposes too. Depending on what exactly you need or expect, you have diverse choices.