Redefining Interpretability: From Filter-CAVs to Pattern-CAVs

Figure 1: CAVs are used to represent concepts, such as biases (e.g., a timestamp in fire classification tasks), in the latent space of DNNs.

15 July 2026

Frederik Pahde, Post-Doctoral Researcher in Explainable AI at Fraunhofer HHI

Leila Arras, PhD Candidate in Explainable AI at Fraunhofer HHI

Figure 1: CAVs are used to represent concepts, such as biases (e.g., a timestamp in fire classification tasks), in the latent space of DNNs.

Introduction

Since the rise of global eXplainable AI (XAI) in the quest to make Machine Learning (ML) more transparent, Concept Activation Vectors (CAVs) have become highly popular in interpretability research and are used for a plethora of high-stake decision-making tasks. They form the backbone of methods like TCAV [1], which conducts concept sensitivity testing, and mechanistic interpretability [2,3], which decodes a model's knowledge base. Recently, CAVs have also been applied in bias mitigation frameworks, such as ClArC [4,5], where they are used to represent a model’s perception of a bias to decrease its sensitivity towards the considered bias in a subsequent step.

However, our research published at ICLR 2025 reveals a critical flaw in how CAVs are typically calculated and introduces a more precise alternative: Pattern-CAVs.

Problem: Distractor Directions

Standard CAVs are usually derived from linear classifiers (like SVMs). These so-called Filter-CAVs are trained to find a boundary that separates the activations of two sample sets: one with and one without the concept to be represented. However, the directional precision of these resulting vectors is often underexplored.

Extraction filters (i.e., classifier weights) are known to be susceptible to distractor components [6], raising concerns about their precision. While linear classifiers optimize the separability of two classes for discriminative tasks, they may struggle to accurately identify the signal direction encoding the concept which is relevant for the explanatory perspective. This can be attributed to the significant influence of distractors (i.e., non-signal) directions contained in the data, which are picked up by extraction filters of linear models to optimize class-separability.

Solution: Pattern-CAV

To address this issue, we introduce Pattern-CAVs in our ICLR 2025 paper [7]. Unlike traditional methods that utilize a "backward" (discriminative) model to predict concept labels from activations, Pattern-CAVs employ a "forward" (generative) model.

More precisely, instead of searching for a decision boundary, we aim to find a Pattern-CAV that minimizes a reconstruction loss to express the observed activations in terms of the concept labels themselves:

In this equation, represents latent activations, the concept labels, a bias term and depicts the Frobenius norm. By focusing on reconstructing the signal rather than just separating classes, the Pattern-CAV isolates the true concept direction and ignores the distractors.

2D Toy Experiment

To illustrate the vulnerability of extraction filters to noise in CAVs, we conducted a 2D toy experiment with simulated activations composed of two components. Specifically, for concept labels , we generated activations , composed of a signal part , representing the concept direction, which is aligned with the x-axis and a distractor term D with random noise oriented parallel to a rotating vector. The resulting CAVs for different rotations (distractor direction depicted in grey) are shown in Fig. 2.

We observe that the Pattern-CAV (depicted in green) accurately identifies the concept direction across all rotations, while the Filter-CAV (depicted in magenta) tends to diverge, increasing the angle to the distractor direction to minimize the variance of data points along the weight vector. This experiment highlights that Filter-CAVs can deviate due to distractor components, while Pattern-CAVs consistently point toward the true concept direction.

Figure 2: A rotated distractor direction (grey) causes filter-CAVs (magenta) to diverge from the true concept direction (x-axis), while pattern-CAVs (green) are correctly oriented parallel to the x-axis. — Figure 1: A rotated distractor direction (grey) causes filter-CAVs (magenta) to diverge from the true concept direction (x-axis), while pattern-CAVs (green) are correctly oriented parallel to the x-axis.

Impact & Conclusion

Our findings indicate that while widely used filters can predict the presence of concepts, they are less effective at accurately modeling the direction of the concept signal due to the influence of distractor directions in the data. This limitation is particularly concerning for safety-critical applications of CAVs, such as TCAV for concept sensitivity testing and ClArC for model correction, that rely on precise concept directions.

To overcome this challenge, we introduced Pattern-CAVs, which focus solely on the concept signal and disregard distractor influences. Our paper provides both theoretical justifications and empirical evidence showing that Pattern-CAVs offer more accurate estimations of true concept directions than traditional filter-based CAVs.

The ability to precisely estimate concept directions within deep neural networks represents a significant step toward reliably understanding and steering the reasoning of ML models. This advancement is essential for developing and deploying trustworthy AI systems in safety-critical tasks, including natural disaster management and remote sensing of wildfires and floods.

References

Been Kim et al. "Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)", Proceedings of the 35th International Conference on Machine Learning (ICML). PMLR 80:2668-2677, 2018.
Neel Nanda et al. "Emergent Linear Representations in World Models of Self-Supervised Sequence Models", Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2023.
Zihao Wang et al. "Concept Algebra for (Score-Based) Text-Controlled Generative Models", Advances in Neural Information Processing Systems 36: 35331-35349, 2023.
Christopher Anders et al. "Finding and removing Clever Hans: Using explanation methods to debug and improve deep models", Information Fusion 77: 261-295, 2022.
Maximilian Dreyer et al. "From Hope to Safety: Unlearning Biases of Deep Models via Gradient Penalization in Latent Space", Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, No. 19, 2024.
Stefan Haufe et al. "On the interpretation of weight vectors of linear models in multivariate neuroimaging", Neuroimage 87: 96-110, 2014.
Frederik Pahde et al. "Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence", The Thirteenth International Conference on Learning Representations (ICLR), 2025.

Photos by Pexels: