With the advancements in Artificial Intelligence (AI), AI techniques are being utilized in various real-world applications. While explainability of AI models is often not necessary when there are no significant consequences for incorrect outcomes, it remains crucial in safety-critical areas like Healthcare and Autonomous Vehicles. This has led to a growing demand for interpretability in Deep Learning models. The number of publications on Explainable AI/Interpretable ML has been increasing in recent years, as evident from major AI conferences such as ICML.