|
|
--- |
|
|
tags: |
|
|
- pattern-classification |
|
|
- multi-label-classification |
|
|
datasets: |
|
|
- maximuspowers/muat-mean-std |
|
|
--- |
|
|
|
|
|
# Pattern Classifier |
|
|
|
|
|
This model was trained to classify which patterns a subject model was trained on, based on neuron activation signatures. |
|
|
|
|
|
## Dataset |
|
|
|
|
|
- **Training Dataset**: [maximuspowers/muat-mean-std](https://huggingface.co/datasets/maximuspowers/muat-mean-std) |
|
|
- **Input Mode**: signature |
|
|
- **Number of Patterns**: 14 |
|
|
|
|
|
## Patterns |
|
|
|
|
|
The model predicts which of the following 14 patterns the subject model was trained to classify as positive: |
|
|
|
|
|
1. `palindrome` |
|
|
2. `sorted_ascending` |
|
|
3. `sorted_descending` |
|
|
4. `alternating` |
|
|
5. `contains_abc` |
|
|
6. `starts_with` |
|
|
7. `ends_with` |
|
|
8. `no_repeats` |
|
|
9. `has_majority` |
|
|
10. `increasing_pairs` |
|
|
11. `decreasing_pairs` |
|
|
12. `vowel_consonant` |
|
|
13. `first_last_match` |
|
|
14. `mountain_pattern` |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
- **Signature Encoder**: [512, 256, 256, 128] |
|
|
- **Activation**: relu |
|
|
- **Dropout**: 0.2 |
|
|
- **Batch Normalization**: True |
|
|
|
|
|
## Training Configuration |
|
|
|
|
|
- **Optimizer**: adam |
|
|
- **Learning Rate**: 0.001 |
|
|
- **Batch Size**: 16 |
|
|
- **Loss Function**: BCE with Logits (with pos_weight for training, unweighted for validation) |
|
|
|
|
|
## Test Set Performance |
|
|
|
|
|
- **F1 Macro**: 0.2832 |
|
|
- **F1 Micro**: 0.2498 |
|
|
- **Hamming Accuracy**: 0.7306 |
|
|
- **Exact Match Accuracy**: 0.0140 |
|
|
- **BCE Loss**: 0.4844 |
|
|
|
|
|
### Per-Pattern Performance (Test Set) |
|
|
|
|
|
| Pattern | Precision | Recall | F1 Score | |
|
|
|---------|-----------|--------|----------| |
|
|
| palindrome | 11.1% | 89.8% | 19.8% | |
|
|
| sorted_ascending | 59.7% | 56.6% | 58.1% | |
|
|
| sorted_descending | 15.8% | 66.2% | 25.5% | |
|
|
| alternating | 19.8% | 72.4% | 31.1% | |
|
|
| contains_abc | 30.8% | 57.6% | 40.1% | |
|
|
| starts_with | 9.1% | 59.4% | 15.8% | |
|
|
| ends_with | 10.3% | 73.8% | 18.1% | |
|
|
| no_repeats | 17.8% | 32.1% | 22.9% | |
|
|
| has_majority | 33.3% | 60.5% | 43.0% | |
|
|
| increasing_pairs | 23.3% | 35.3% | 28.1% | |
|
|
| decreasing_pairs | 19.4% | 60.9% | 29.5% | |
|
|
| vowel_consonant | 9.8% | 76.9% | 17.4% | |
|
|
| first_last_match | 15.3% | 96.6% | 26.5% | |
|
|
| mountain_pattern | 15.3% | 31.7% | 20.6% | |
|
|
|
