A transformer-based classifier that predicts cell type from single-cell RNA-seq by representing each gene with its ESM-2 protein-language-model embedding (1280→512-dim) gated by expression. Trained on millions of cells from CellXGene Census across 61 leaf cell types, with hierarchical loss that respects the Cell Ontology DAG.

Highlights:

• • Expression-Gated Attention: Each token is the projected protein embedding scaled by L1-normalized log-expression — so the model attends to function, not just abundance.
• • Multi-CLS Readout: 8 CLS tokens through a 4-layer, 8-head transformer with flash attention in bf16; pooled into a 512-dim cell embedding.
• • Ontology-Aware Loss: MarginalizationLoss (leaf CrossEntropy + parent BCE) enforces consistent predictions up the Cell Ontology tree.
• • Production Pipeline: BioMart → UniProt → ESM-2 batched inference, with CellXGene streaming via tiledbsoma.

A deep neural classifier that predicts cell type directly from raw gene counts (~23K input features) and respects the Cell Ontology hierarchy. Trained on millions of cells from CellXGene Census with a custom loss that enforces parent-child consistency across 100+ cell types.

Highlights:

• • WideNN Architecture: 23K → 2048 → 1024 → 256 → ontology leaves, with BatchNorm + ReLU; ~50M parameters tuned for sparse single-cell input.
• • MarginalizationLoss: Combines leaf-level CrossEntropy with BCE over ancestor nodes so predictions stay consistent with the Cell Ontology DAG.
• • Streaming Data Pipeline: tiledbsoma-ml + torchdata to train directly off CellXGene Census without materializing the full matrix.
• • Foundation for Scipher: The hierarchical classification framework that Scipher's transformer plugs into.

A novel approach to single-cell analysis that aggregates 19,294 ProteinBERT embeddings into compact 512-dimensional cell representations, achieving 99.5% accuracy and 0.999 AUC on cancer classification with minimal training.

Key innovations:

• • Attention-Based Aggregation: Learns which genes matter regardless of expression level—capturing tumor suppressors that stay important when lowly expressed while ignoring noisy housekeeping genes.
• • Function-First Embedding: Maps cells to protein function space rather than raw expression, enabling biologically meaningful representations.
• • Lightweight Architecture: Only ~198K parameters trained for 7 epochs for attention pooling.
• • Interpretable Results: Attention weights reveal which genes drive classification decisions.