01698nas a2200169 4500008004100000245007400041210006900115260001300184520108200197100002101279700002401300700001701324700003001341700002601371700002001397856011101417 2026 eng d00aNeurosymbolic Hidden Neuron Analysis in Convolutional Neural Networks0 aNeurosymbolic Hidden Neuron Analysis in Convolutional Neural Net bElsevier3 a

This tutorial introduces a step-by-step, deductive pipeline for making the inner workings of neural networks more transparent by assigning human-understandable concepts to hidden neuron activations. The approach automatically maps neuron behavior to symbolic concepts drawn from structured knowledge sources and attaches an error margin to each label, providing a measure of confidence in its precision. While demonstrated in detail on the ADE20k scene dataset---including single-concept neurons, multiple neurons contributing to the same concept, and multi-concept neurons---the method is also applied to the SUN2012 dataset and adapted for a text classification task, highlighting its generalizability across modalities. The chapter is designed to be practical and educational, focusing on a replicable methodology that readers can adapt to varied applications. Through worked examples, visualizations, and evaluation strategies, the tutorial offers a clear, reusable framework for concept-based neuron analysis in both vision and language models.

1 aDalal, Abhilekha1 aSarma, Moumita, Sen1 aDas, Avishek1 aAkkamahadevi, Samatha, E.1 aVasserman, Eugene, Y.1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/neurosymbolic-hidden-neuron-analysis-convolutional-neural-networks01891nas a2200193 4500008004100000245005900041210005900100260002200159300001400181520124700195653003001442653002501472653001901497653002701516100001701543700002101560700002001581856009601601 2025 eng d00aHidden Neuron Activation Analysis on Labeled Text Data0 aHidden Neuron Activation Analysis on Labeled Text Data aUSAbACMc12/2025 a206 - 2103 a

Understanding the internal mechanisms of deep neural networks remains a central challenge in the field of Explainable Artificial Intelligence (XAI). With the rapid advancement of neural architectures in natural language processing (NLP), analyzing the role of hidden neurons in capturing and processing linguistic features has become increasingly important. This study investigates Hidden Neuron Activation Analysis on labeled text data to reveal how individual neurons contribute to a model’s decision-making process. We propose a model-agnostic explainability framework for text classifiers that identifies concepts activating specific neurons involved in classification. An LSTM-based network is trained on the AG News topic classification dataset, comprising four distinct classes, and the final Dense layer with 64 neurons was analyzed. In addition, statistical analyses like the Mann-Whitney U Test is conducted to assess the robustness and reliability of the system. The statistical analysis shows that, concepts plays important role in the decision making process of neural network. Our findings enhance interpretability in NLP models and offer a foundation for optimizing neural architectures in text classification tasks.

10aConcept-based Explanation10aDense Layer Analysis10aExplainable AI10aHidden Neuron Analysis1 aDas, Avishek1 aDalal, Abhilekha1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/hidden-neuron-activation-analysis-labeled-text-data01724nas a2200181 4500008004100000245006500041210006500106520107500171653002101246653001801267653004001285653002001325653001701345100003401362700002101396700002001417856010501437 2024 eng d00aAutomating CNN Neuron Interpretation using Concept Induction0 aAutomating CNN Neuron Interpretation using Concept Induction3 a

This paper presents an automation pipeline for interpreting hidden neuron activations in Convolutional Neural Networks (CNNs), a crucial objective of Explainable AI (XAI). Previously, our research group addressed this objective by employing concept induction and semantic reasoning using a concept hierarchy derived from the Wikipedia knowledge graph. However, the process was executed manually, taking several days to complete. In this study, we have fully automated the workflow, achieving consistent results while significantly reducing the execution time. The automation pipeline streamlines model training, data preparation, concept induction, image retrieval, classification, and statistical validation, thereby completely eliminating the manual intervention. This automation enables us to efficiently interpret and validate CNN neuron activations by modifying parameters, such as incorporating a broader range of training images and classes and examining additional concept induction results across various neuron layers using different analytical tools.

10aAutomation in AI10aDeep Learning10aExplainable Artificial Intelligence10aKnowledge Graph10aSemantic Web1 aAkkamahadevi, Samatha, Ereshi1 aDalal, Abhilekha1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/automating-cnn-neuron-interpretation-using-concept-induction01220nas a2200169 4500008004100000245006200041210006100103260001400164520066400178653000800842653002200850653001900872100002100891700002000912700002000932856009800952 2024 eng d00aError-margin Analysis for Hidden Neuron Activation Labels0 aErrormargin Analysis for Hidden Neuron Activation Labels bSpringer 3 a

Understanding how high-level concepts are represented with- in artificial neural networks is a fundamental challenge in the field of arti- ficial intelligence. While existing literature in explainable AI emphasizes the importance of labeling neurons with concepts to understand their functioning, they mostly focus on identifying what stimulus activates a neuron in most cases; this corresponds to the notion of recall in informa- tion retrieval. We argue that this is only the first-part of a two-part job; it is imperative to also investigate neuron responses to other stimuli, i.e., their precision. We call this the neuron label’s error margin.

10aCNN10aConcept Induction10aExplainable AI1 aDalal, Abhilekha1 aRayan, Rushrukh1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/error-margin-analysis-hidden-neuron-activation-labels02149nas a2200109 4500008004100000245010600041210006900147490000800216520166800224100002101892856012601913 2024 eng d00aUnderstanding Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning0 aUnderstanding Hidden Neuron Activations Using Structured Backgro0 vPhD3 a

A central challenge in Explainable AI (XAI) is accurately interpreting hidden neuron activations in deep neural networks (DNNs). Accurate interpretations help demystify the black-box nature of deep learning models by explaining what the system internally detects as relevant in the input. While some existing methods show that hidden neuron activations can be human-interpretable, systematic and automated approaches leveraging background knowledge remain underexplored. This thesis introduces a novel model-agnostic post-hoc XAI method that integrates a Wikipedia-derived concept hierarchy of approximately 2 million classes as background knowledge and employs OWL-reasoning-based Concept Induction to generate explanations. Our approach automatically assigns meaningful class expressions to neurons in the dense layers of Convolutional Neural Networks, outperforming prior methods both quantitatively and qualitatively.

In addition, we argue that understanding neuron behavior requires not only identifying what activates a neuron (recall) but also examining its precision—how it responds to other stimuli, which we define as the neuron's error margin, enhancing the granularity of neuron interpretation.

To visualize these findings, we present ConceptLens, an innovative tool that visualizes neuron activations and error margins. ConceptLens offers insights into the confidence levels of neuron activations and enables an intuitive understanding of neuron behavior through visual bar charts. Together, these contributions offer a holistic approach to interpreting DNNs, advancing the explainability and transparency of AI models.

1 aDalal, Abhilekha uhttps://daselab.cs.ksu.edu/publications/understanding-hidden-neuron-activations-using-structured-background-knowledge-and01357nas a2200217 4500008004100000245009200041210006900133260001400202520061900216653000800835653002200843653001900865653000800884100002100892700002000913700001800933700002600951700002400977700002001001856011801021 2024 eng d00aOn the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis0 aValue of Labeled Data and Symbolic Methods for Hidden Neuron Act bSpringer 3 a

We introduce a novel model-agnostic post-hoc Explainable AI method that provides meaningful interpretations for hidden neuron activations in a Convolutional Neural Network. Our approach uses a Wikipedia-derived concept hierarchy with approx. 2 million classes as background knowledge, and deductive reasoning based Concept Induc- tion for explanation generation. Additionally, we explore and compare the capabilities of off-the-shelf pre-trained multimodal-based explainable methods. Our evaluation shows that our neurosymbolic method holds a competitive edge in both quantitative and qualitative aspects.

10aCNN10aConcept Induction10aExplainable AI10aLLM1 aDalal, Abhilekha1 aRayan, Rushrukh1 aBarua, Adrita1 aVasserman, Eugene, Y.1 aSarker, Kamruzzaman1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/value-labeled-data-and-symbolic-methods-hidden-neuron-activation-analysis00758nas a2200217 4500008004100000245007100041210006900112100002000181700001900201700002000220700001900240700001900259700002000278700002500298700002400323700002100347700002100368700001900389700002200408856011000430 2023 eng d00aBridging RDF and Property Graphs: Linking KnowWhereGraph and SPOKE0 aBridging RDF and Property Graphs Linking KnowWhereGraph and SPOK1 aStephen, Shirly1 aShimizu, Cogan1 aHitzler, Pascal1 aSoman, Karthik1 aRose, Peter, W1 aMorris, John, H1 aBaranzini, Sergio, E1 aJanowicz, Krzysztof1 aChristou, Antrea1 aDalal, Abhilekha1 aCurrier, Kitty1 aSchildhauer, Mark uhttps://daselab.cs.ksu.edu/publications/bridging-rdf-and-property-graphs-linking-knowwheregraph-and-spoke01913nas a2200133 4500008004100000245007900041210006900120520138500189100002101574700002701595700001801622700002001640856011901660 2023 eng d00aExplaining Deep Learning Hidden Neuron Activations using Concept Induction0 aExplaining Deep Learning Hidden Neuron Activations using Concept3 a

One of the current key challenges in Explainable AI is in correctly interpreting activations of hidden neurons. It seems evident that accurate interpretations thereof would provide insights into the question what a deep learning system has internally detected as relevant on the input, thus lifting some of the black box character of deep learning systems.

The state of the art on this front indicates that hidden node activations appear to be interpretable in a way that makes sense to humans, at least in some cases. Yet, systematic automated methods that would be able to first hypothesize an interpretation of hidden neuron activations, and then verify it, are mostly missing. 

In this paper, we provide such a method and demonstrate that it provides meaningful interpretations. It is based on using large-scale background knowledge -- a class hierarchy of approx. 2 million classes curated from the Wikipedia Concept Hierarchy -- together with a symbolic reasoning approach called concept induction based on description logics that was originally developed for applications in the Semantic Web field. 

Our results show that we can automatically attach meaningful labels from the background knowledge to individual neurons in the dense layer of a Convolutional Neural Network through a hypothesis and verification process.

1 aDalal, Abhilekha1 aSarker, Md Kamruzzaman1 aBarua, Adrita1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/explaining-deep-learning-hidden-neuron-activations-using-concept-induction00817nas a2200289 4500008004100000245003200041210002800073100001900101700002000120700001900140700002000159700001300179700002400192700002200216700003400238700002100272700001800293700001400311700001800325700001800343700001900361700002500380700001600405700001600421700002200437856006800459 2023 eng d00aThe KnowWhereGraph Ontology0 aKnowWhereGraph Ontology1 aShimizu, Cogan1 aStephen, Shirly1 aCurrier, Kitty1 aHitzler, Pascal1 aZhu, Rui1 aJanowicz, Krzysztof1 aSchildhauer, Mark1 aMahdavinejad, Mohammad, Saeid1 aDalal, Abhilekha1 aBarua, Adrita1 aCai, Ling1 aMai, Gengchen1 aWang, Zhangyu1 aTian, Yuanyuan1 aNorouzi, Sanaz, Saki1 aLiu, Zilong1 aShi, Meilin1 aFisher, Colby, K. uhttps://daselab.cs.ksu.edu/publications/knowwheregraph-ontology00914nas a2200313 4500008004100000245004400041210003900085100001900124700002000143700001300163700001900176700002200195700002000217700002000237700002400257700002200281700003400303700002100337700001800358700002100376700002500397700001600422700001600438700001400454700001800468700001800486700001900504856007700523 2023 eng d00aThe KnowWhereGraph Ontology: A Showcase0 aKnowWhereGraph Ontology A Showcase1 aShimizu, Cogan1 aStephen, Shirly1 aZhu, Rui1 aCurrier, Kitty1 aSchildhauer, Mark1 aRehberger, Dean1 aHitzler, Pascal1 aJanowicz, Krzysztof1 aFisher, Colby, K.1 aMahdavinejad, Mohammad, Saeid1 aChristou, Antrea1 aBarua, Adrita1 aDalal, Abhilekha1 aNorouzi, Sanaz, Saki1 aLiu, Zilong1 aShi, Meilin1 aCai, Ling1 aMai, Gengchen1 aWang, Zhangyu1 aTian, Yuanyuan uhttps://daselab.cs.ksu.edu/publications/knowwheregraph-ontology-showcase00835nas a2200253 4500008004100000245006100041210005900102100001900161700002000180700002100200700001900221700003400240700002500274700002100299700001800320700002200338700002500360700001900385700002400404700002000428700002200448700002000470856009100490 2023 eng d00aKnowWhereGraph-Lite: A Perspective of the KnowWhereGraph0 aKnowWhereGraphLite A Perspective of the KnowWhereGraph1 aShimizu, Cogan1 aStephen, Shirly1 aChristou, Antrea1 aCurrier, Kitty1 aMahdavinejad, Mohammad, Saeid1 aNorouzi, Sanaz, Saki1 aDalal, Abhilekha1 aBarua, Adrita1 aFisher, Colby, K.1 aD’Onofrio, Anthony1 aThelen, Thomas1 aJanowicz, Krzysztof1 aRehberger, Dean1 aSchildhauer, Mark1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/knowwheregraph-lite-perspective-knowwheregraph00639nas a2200157 4500008004100000245010100041210006900142100002000211700002400231700001900255700002100274700002000295700001800315700002500333856012300358 2023 eng d00aOpenness and Transparency in Academic Publishing: A Decade of Data from the Semantic Web Journal0 aOpenness and Transparency in Academic Publishing A Decade of Dat1 aHitzler, Pascal1 aJanowicz, Krzysztof1 aShimizu, Cogan1 aDalal, Abhilekha1 aEberhart, Aaron1 aEells, Andrew1 aNorouzi, Sanaz, Saki uhttps://daselab.cs.ksu.edu/publications/openness-and-transparency-academic-publishing-decade-data-semantic-web-journal01563nas a2200121 4500008004100000245008100041210006900122520107200191100002101263700001901284700002001303856011801323 2021 eng d00aBridging Upper Ontology and Modular Ontology Modeling: A Tool and Evaluation0 aBridging Upper Ontology and Modular Ontology Modeling A Tool and3 a

Ontologies are increasingly used as schema for knowledge graphs in many application areas. As such, there are a variety of different approaches for their development. In this paper, we describe and evaluate UAO (for Upper Ontology Alignment Tool), which is an extension to CoModIDE, a graphical prote'ge' plugin for modular ontology modeling. UAO enables ontology engineers to combine modular ontology modeling with a more traditional ontology modeling approach based on upper ontologies. We posit -- and our evaluation supports this claim -- that the tool does indeed makes it easier to combine both approaches. Thus, UAO enables a best-of-both-worlds approach. The evaluation consists of a user study, and the results show that performing typical manual alignment modeling tasks is relatively easier with UAO than doing it with porte'ge' alone, in terms of the time required to complete the task and improving the correctness of the output. Additionally, our test subjects provided significantly higher ratings on the System Utilization Scale for UOA.

1 aDalal, Abhilekha1 aShimizu, Cogan1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/bridging-upper-ontology-and-modular-ontology-modeling-tool-and-evaluation01038nas a2200157 4500008004100000245008900041210006900130260001200199300001200211490000900223520046500232100002100697700001900718700002000737856012300757 2020 eng d00aModular Ontology Modeling Meets Upper Ontologies: The Upper Ontology Alignment Tool0 aModular Ontology Modeling Meets Upper Ontologies The Upper Ontol c10/2020 a119-1240 v27213 a

We provide an extension to the Prote'ge'-based modular ontology engineering tool CoModIDE, in order to make it possible for ontology engineers to adhere to traditional ontology modeling processes based on upper or foundational ontologies. As a bridge between the more recently proposed modular ontology modeling approach and more classical ones based on foundational ontologies, it enables a best-of-both-worlds approach for ontology engineering.

1 aDalal, Abhilekha1 aShimizu, Cogan1 aHitzler, Pascal uhttps://daselab.cs.ksu.edu/publications/modular-ontology-modeling-meets-upper-ontologies-upper-ontology-alignment-tool02942nas a2200133 4500008004100000245008900041210006900130260004800199300000700247490001200254520239600266100002102662856012502683 2020 eng d00aModular Ontology Modeling Meets Upper Ontologies: The Upper Ontology Alignment Tool0 aModular Ontology Modeling Meets Upper Ontologies The Upper Ontol aManhattanbKansas State Universityc11/2020 a350 vMasters3 a

Ontology modeling has become a primary approach to schema generation for data integration and knowledge graphs in many application areas. The quest for efficient approaches to model useful and re-useable ontologies has led to different ontology creation proposals over the years. The project focuses on two major approaches, modeling using a top-level ontology, and the other is modular ontology modeling.

The traditional approach is based on top-level ontology, and the strategy is to utilize ontology that is comprehensive enough to cover a broad spectrum of domains through their universal terminologies. In this way, all domain ontologies share a common top-level formal ontology in which their respective root nodes can be defined, and hence consistency is assured across the knowledge graph. Nevertheless, the most recent approach is quite different and is a refinement of the eXtreme Ontology Design methodology based on the ontology design patterns. Whole ontology is viewed as a collection of interconnected modules, and modules are developed around the classified fundamental notions according to experts' terminology or the use-case. Having developed modules in a fashion of divide and conquer, these modules are shareable and reusable among some other ontology if needed, and consequently, the ontology being FAIR is justified (findable, accessible, interoperable, and reusable).

Although, it has been argued that there are advantages to either paradigm, it is possible to have a combination of both approaches mentioned earlier, depending upon the use-case or the preferences of the ontology engineers. We provide an extension to the Protégé - based modular ontology engineering tool CoModIDE, in order to make it possible for ontology engineers to follow traditional, ad-hoc ontology modeling approach, alongside more modern paradigms such as modular ontology engineering. The project focuses on domain-level ontology developers or organizations dealing with ontology development, which may get help through the plugin in minimizing the tooling gap to unite paradigms and develop robust, flexible ontologies suitable to their needs. As a bridge between the more recently proposed modular ontology modeling approach and more classical ones based on foundational ontologies, it enables a best-of-both-worlds approach for ontology engineering. 

1 aDalal, Abhilekha uhttps://daselab.cs.ksu.edu/publications/modular-ontology-modeling-meets-upper-ontologies-upper-ontology-alignment-tool-0