Fellas thank you so much for the insights. I did few tests the past couple of days and here's what I found:
On my old PC (Acer i5, 4GB RAM), I found that when I use Windows Audio (Exclusive Mode), I can go low with buffer size with absolutely no problems. Disabling the network adapter almost always make the DPC test (which EvilDragon suggested) better but I found that the results from thesycon.de and the clicking sounds are not related.
Same on my new PC (Dell Inspiron i7-7500U), the results from the DPC were not related to the clicking but what's strange is that even with Windows Audio (exclusive mode), I'd still hear the clicks.
Long story short, tried all different combinations on both PCs using various audio device type (windows audio, ASIO4ALL, exclusive audio) and the only common thing was that clicking was heard on both machines only when the "Device" setting was set to the M-Audio 2x2 Asio and this concluded that the problem is almost definitely coming from the compatibility of the M-Audio with the Dell Inspiron. A bit of research on that and I saw that there are a few folks complaining from latency and clicking problems from the M-Audio 2x2 so it must be that.
Still not sure how to solve this, if it's incompatibility then either the Dell or the M-Audio will have to be replaced. The topics in the M-Audio forums are left unresolved.
@Evildragon: I also found this program for DPC testing, it additionally tells which adapter is most likely causing the problem.
http://www.resplendence.com/latencymon
@Stamkorg: Wonderful guide, learned a lot from it even if it doesn't seem to solve my case (but good to have the PC in super shape anyway).
@Mossy: Did you mean to turn off both CPU overload detection and multicore rendering?