noanabeshima/tiny_model_cached_acts

Cached SAE/transcoder acts stored in CSR format. Not especially optimized for others' use/fleshed out. It's mostly just a space for me to place and download activations.

If you want to use them, do

import torch
from huggingface_hub import hf_hub_download

def load_feat_acts(fname, only_active_docs=False):
  local_path = hf_hub_download(repo_id="noanabeshima/tiny_model_cached_acts", filename=fname)
  csr_kwargs = torch.load(local_path)

  # The matrices might be stored in space-efficient formats that're incompatible with torch's sparse csr tensor.
  # Convert them back before constructing the matrix.
  csr_kwargs['crow_indices'] = csr_kwargs['crow_indices'].int()
  csr_kwargs['col_indices'] = csr_kwargs['col_indices'].int()
  values = csr_kwargs['values'].float()
  csr_kwargs['values'] = values/values.max()
  
  feat_acts = torch.sparse_csr_tensor(**csr_kwargs)
  return feat_acts

feat_acts = load_feat_acts(f"mlp_map_test/M2_S-2_R1_P0/{300}.pt").to_dense()

The activations are from 13% of the train split of https://huggingface.co/datasets/noanabeshima/TinyModelTokIds. I gather all the activations per feature and then take the smallest activations prefix so that each feature has at least 3K documents on which it activates and 12K token-activations.