Your question is kind of broad, but I think it's still more or less answerable. Let me just address your sub-questions one by one:
I am looking for the implementation of IPsec security algorithms for both authentication and Encryption. Is there any open source implementation available for the same?
Yes, plenty. If you specified which platform and/or language you're using, we could point you to some suitable ones, but just off the top of my head, the OpenSSL library is fairly extensive and portable.
And if someone supposed to implement these algorithms, then what background knowledge is required for someone new in the cryptography world?
It depends a lot on just which kinds of algorithms you want to (re)implement. Generally, high-level "derived" algorithms like HMAC or basic cipher modes like CBC, which simply build on top of existing crypto primitives, are relatively easy and safe to implement, whereas low-level "primitive" algorithms like block ciphers or hash functions are difficult to reimplement efficiently and securely.
In any case, the standard advice with crypto is that, if you don't have to implement something yourself, don't. Unless you're only doing it as a learning exercise, that is, and never intend to actually use your code for anything security-critical.
In any case, if you do end up implementing some crypto code yourself (especially, God forbid, if it's low-level), the very next thing you should do is read up on possible attacks against your code, and see if you can break it. If not, show it to someone who knows more about crypto than you, and see if they can break it. Repeat until your code gets broken, or you run out of experts to consult. :-)
Please recommend some links and documents that can help in understanding these algorithms for the implementation.
What you really need, to begin with, is a good introduction to cryptography and data security. I'm not really the best person to give specific recommendations here, but some books I've seen recommended by other include the Introduction to Modern Cryptography by Katz & Lindell, the Handbook of Applied Cryptography by Menezes, Oorschot & Vanstone, or, if those seem too intimidating in their scope, perhaps Cryptography: A Very Short Introduction by Piper & Murphy.
For learning about specific crypto algorithms, Wikipedia is often a very good starting point, and will typically have links to further material. If the algorithm is standardized, reading the relevant RFC / NIST SP / other standard is also often helpful. Many algorithms also have their own semi-official web pages maintained by their developers; for example, if you wanted to learn about HMAC, you could do a lot worse than starting at Mihir Bellare's HMAC page. And, of course, there's plenty of useful information right here on Crypto.SE, if you just search for it.
Also suggest some open source implementations of following algorithms: HMAC-MD5, HMAC-SHA1, DES-CBC, Triple-DES-CBC and AES, and the open source projects that have used it.
Those are all very common algorithms, and any half-decent crypto library (such as the OpenSSL library mentioned above) should support them.
Apart of the one I have listed above which are others algorithms that can be used for IPsec?
I'm not really familiar with IPsec myself, and the Wikipedia page just refers to RFC 4835, which seems to be the authoritative reference. It's a pretty short and readable document, so I suggest you just read through it. In particular, sections 3.1.1 and 3.2 list the encryption and authentication algorithms that IPsec implementations MUST, SHOULD and/or MAY support:
Requirement Encryption Algorithm (notes)
MUST NULL [RFC2410] (1)
MUST AES-CBC with 128-bit keys [RFC3602]
MUST- TripleDES-CBC [RFC2451]
SHOULD AES-CTR [RFC3686]
SHOULD NOT DES-CBC [RFC2405] (2)
Requirement Authentication Algorithm (notes)
MUST HMAC-SHA1-96 [RFC2404] (3)
SHOULD+ AES-XCBC-MAC-96 [RFC3566]
MAY NULL (1)
MAY HMAC-MD5-96 [RFC2403] (4)
What are the differences between authentication algorithms (HMAC-MD5 and HMAC-SHA1) and similarly what are differences between encryption algorithms (DES, 3DES, AES)?
MD5 is an old hash function that is considered broken for many purposes, although there are so far no known effective attacks on it when used specifically in the HMAC construction. SHA-1 is also considered somewhat weak these days, although not nearly as weak as MD5, and the NIST now recommends phasing it out (although HMAC is, again, considered an exception).
The hash functions currently recommended by NIST (and mostly by industry consensus) are the SHA-2 family of hashes, which are considered more secure than SHA-1. There's also an upcoming SHA-3 standard, which has not yet been finalized, but should be due any day now; it's not designated as a replacement to SHA-2, which is still considered secure, but as an alternative to it.
In any case, given the choice between MD5 and SHA-1, you should definitely go with SHA-1. But it would be better yet if SHA-2 were an option.
DES is an old block cipher from the 1970s, and should no longer be used; its main weakness is simply its short key length, which makes it vulnerable to brute force attacks on modern computers.
3DES, a.k.a. Triple DES, is an encryption scheme built on top of the DES cipher, which effectively doubles its key length, but also triples the time needed to encrypt and decrypt messages. It's still considered secure, although it's not as efficient as more modern ciphers.
AES is the modern replacement for DES, and the block cipher currently recommended by NIST. It has several improvements over DES, including significantly longer key length, twice as long block size (which matters for some application involving large amounts of data) and, frequently, higher speed. In general, if you have no specific reason to use something else, AES is the cipher you should be using these days: it's fast, it's secure, it's widespread and, above all, it's standard. To slightly misquote an old catchphrase, nobody ever got fired for using AES.