RESEARCH

The main question is not only whether failure is linearly decodable from OpenVLA activations, but whether the learned warning signal survives a change in rollout initialization.

The evaluation therefore separates complete episodes into training and testing groups and measures performance on previously unseen rollout initializations.

The same framework can then be used to compare:

• clean → clean generalization;
• visual shift → visual shift generalization;
• clean → visual shift transfer;
• visual shift → clean transfer.

Longer term, the goal is to test whether similar activation-based warning signals appear across different VLA architectures.

1. Initial prototype. Modify CIFAR-100 to hold out classes, fine-tune a ResNet-50 on the remaining classes, and implement Integrated Gradients with Captum. This established the classifier and explanation pipeline before training the OOD detector.
2. Iteration with Karla Tran. Move to ImageNet with 10 dog classes as ID, 10 visually similar animal classes as near-OOD, and 10 dissimilar non-animal classes as far-OOD. Develop the pipeline for saving attribution maps and prediction metadata, with separate near, far, and combined detector experiments.
3. Iteration with Caio Miyake. Compare attribution methods and class selections, evaluate one combined detector on multiple OOD groups, and expand from map-based ResNets to MLPs using attribution statistics, classifier uncertainty, and perturbation-based features.

The original proposal also explored using a vision-language model to describe unusual evidence and an LLM to assess it against a system specification. The implemented experiments described here focus on attribution-based ID / OOD detection; they do not establish automated perception-contract verification.

I explored several ways to build that detector: training a ResNet directly on attribution maps, extracting numerical features from those maps and training an MLP, and adding uncertainty and perturbation-based features. I also changed the ID / OOD class selections and compared separate near-OOD and far-OOD detectors with a detector trained on both.

1. Phase 1 — Setup & baseline classifier. Define the distribution split, then train and evaluate the image classifier on ID classes.
2. Phase 2 — Generate attribution maps. Produce and save explanations for ID, near-OOD, and far-OOD images.
3. Phase 3 — Train the ID / OOD detector. Build separate train, validation, and test sets; compare detectors using the maps and derived features.

ID

Classes represented during the original image classifier's training.

Near-OOD

Unseen classes that look similar to the ID classes.

Far-OOD

Unseen classes that are more visually distinct from the ID classes.

The ImageNet setup selected 10 ID classes, 10 near-OOD classes, and 10 far-OOD classes. I constructed train, validation, and test subsets for several detector setups:

• Separate detectors: train and evaluate individual models for ID vs. near-OOD, ID vs. far-OOD, and ID vs. both OOD groups.
• One combined detector: train on both OOD groups, then evaluate the same model on near-OOD, far-OOD, and the combined set.
• Additional unseen classes: build new near-OOD and far-OOD sets containing classes and images the OOD detector was not trained on.

OOD is defined relative to the original image classifier. An OOD detector can still see examples from those classes during its own training, which is why the additional unseen-class evaluation is a separate check.

Image classification and map-based detection

ResNet-18 and ResNet-50 experiments, including a ResNet-50 image classifier and a separate ResNet-18 trained directly on attribution maps to predict ID / OOD.

Attribution methods

Integrated Gradients and Guided Backpropagation, with generated maps for ID, near-OOD, and far-OOD images. The pipeline produced tens of thousands of attribution maps from ImageNet-1K class subsets.

Feature-based detection

Multilayer perceptrons (MLPs) trained on numerical feature vectors extracted from attribution maps, with successive experiments expanding what each vector describes.

Nearest-neighbor baselines

Additional experiments on saved attribution maps compare a supervised k-nearest-neighbor classifier with an ID-only distance detector. The ID-only version measures distance to familiar maps and selects its threshold from ID validation examples.

image
  |
  v
classifier / prediction
  |
  v
attribution method
  |
  v
attribution map
  |
  +--> ResNet-18 detector
  |
  +--> extracted features
          |
          v
         MLP

both detector routes
  |
  v
ID / OOD

Alongside learning directly from the maps, I trained MLP detectors on progressively expanded feature sets. Each version tested a different way of representing the attribution pattern and the original classifier's response.

Big thanks to Will English for suggesting this part and even doing a writeup for it.

[read Will's writeup — PDF]

24 features — attribution statistics

Per-channel statistics such as minimum, maximum, mean, standard deviation, and fraction of positive attributions, plus global summaries including norms, spatial entropy, and concentration in the most strongly attributed pixels.

55 features — spatial and distributional structure

Expand the baseline with skewness, kurtosis, spatial descriptors, discrete cosine transform features, gradients, sparsity, and dynamic range.

58 features — classifier uncertainty

Add target-class confidence, softmax entropy, and target margin to the attribution-derived features.

61 features — perturbation-based signals

Add GARI, GARD, and their combined GAROD score: features examining the model's response to changes in background pixels and pixels identified as important.

583 features — descriptions across classes

Repeat the 58-feature extraction for all 10 classifier classes, then append the three GAROD-based features.

Additional implemented feature variants

A 65-feature extractor compares attribution maps for the two highest-scoring classes using similarity, distance, overlap, and energy ratios. A separate 70-feature extractor adds localization and compactness measurements to the 58-feature representation, including attribution center of mass and the extent of the strongest pixels.

The MLP experiments used batch normalization, GELU activations, dropout, AdamW, learning-rate scheduling, and early stopping. The map-based detector also used class-balanced sampling and normalization based on ID attribution statistics.

The saved 583-feature experiment trained separate MLPs for near-OOD, far-OOD, and both groups, using a 70% / 20% / 10% train, validation, and test split. Their recorded test results were:

ID vs. near-OOD

82.74% accuracy; AUROC 0.9091; AUPRC 0.9089.

ID vs. far-OOD

90.29% accuracy; AUROC 0.9652; AUPRC 0.9623.

ID vs. both OOD groups

84.18% accuracy; AUROC 0.9230; AUPRC 0.9582.

In the later nearest-neighbor experiment, the ID-only distance baseline recorded AUROC 0.6643 on near-OOD and 0.8631 on far-OOD. That experiment used resized attribution-energy maps and cosine distance, so it is a separate representation and setup rather than a controlled comparison with the MLP.

The recurring finding is that visually similar unseen classes are harder to separate from familiar classes. The results support attribution patterns as a useful detection signal in these experiments; they do not show that a confident prediction is safe or that every unseen class can be detected.

These are exploratory results on selected ImageNet subsets from individual saved runs. The 583-feature notebook computes feature normalization from the full ID pool, including its held-out portion. The scores therefore are not a fully independent preprocessing evaluation. The combined test set also contains two OOD groups for one ID group, so its accuracy and AUPRC should be read with that different class balance in mind. Evaluation on classes unseen by the detector is a separate experiment.