Supported MIL Models & Architectures
ColabBio provides a comprehensive suite of Multiple Instance Learning (MIL) architectures in WSI_MIL/src/models/, optimized for gigapixel Whole Slide Images (WSI) and multi-modal clinical endpoints (genomics, survival, grading).
graph TD
WSI["Gigapixel WSI Slide"] --> ENCODER["Foundation Feature Encoder<br/>(ResNet50 / UNI / Virchow / CTransPath)"]
ENCODER --> BAG["Instance Patch Embeddings Bag<br/>(N x 2048 / N x 768)"]
BAG --> M1["<b>abMIL</b><br/>Gated Attention Pooling<br/>(Ilse et al.)"]
BAG --> M2["<b>MultiTask-MIL</b><br/>Shared abMIL Backbone<br/>+ Parallel Heads"]
BAG --> M3["<b>Cox-MIL</b><br/>Attention Survival<br/>+ Breslow NLL Loss"]
BAG --> M4["<b>TransMIL</b><br/>Transformer Self-Attention<br/>+ PPEG 2D Encoding"]
BAG --> M5["<b>CLAM</b><br/>Clustering-constrained<br/>Instance Attention"]
M1 --> O1["Gleason / ISUP Classification<br/>+ Attention Heatmaps"]
M2 --> O2["Simultaneous Grade +<br/>HRD/BRCA + BCR Survival Risk"]
M3 --> O3["Continuous Risk Score θ<br/>(Time-to-Event / BCR / OS)"]
M4 --> O4["Complex Spatial / Subtyping<br/>(LUAD vs LUSC, Stroma-Tumor)"]
M5 --> O5["Low-data Regime Regularization<br/>+ Patch Pseudo-clustering"]
Model Comparison Matrix
Section titled “Model Comparison Matrix”| Model | Architecture | Outputs / Tasks | Primary Clinical Use Case | Compute Footprint |
|---|---|---|---|---|
| abMIL | Gated Attention Pooling | Single/Multiclass logits + patch attention weights | Standard histopathology grading (Gleason/ISUP), interpretable attention heatmaps | Low (O(N) linear in patches, ultra-fast) |
| MultiTask-MIL | Shared abMIL Backbone + Parallel Prediction Heads | Grade logits + Binary Biomarker logit + Continuous Hazard Risk | Unified multi-task profiling in 1 forward pass (e.g., TCGA-PRAD Gleason + HRD + BCR) | Low / Medium (shares slide pooling across all tasks) |
| Cox-MIL | Attention Pooling + Cox Proportional Hazards Head | Continuous log-hazard ratio (θ) | Prognostic stratification, Biochemical Recurrence (BCR), Overall Survival (OS) | Low (requires mini-batches with observed events) |
| TransMIL | Multi-Head Self-Attention + PPEG + [CLS] Token |
Slide classification / regression with inter-patch dependencies | Difficult morphological subtyping where long-range tissue interactions matter | Medium / High (quadratic/approximate self-attention) |
| CLAM | Gated Attention + Instance-level Pseudo-Clustering | Slide logits + top-k / bottom-k instance losses | Low-sample training cohorts, weakly supervised focus localization | Medium (calculates patch-level instance losses) |
1. abMIL (Attention-based Deep Multiple Instance Learning)
Section titled “1. abMIL (Attention-based Deep Multiple Instance Learning)”- Source Code:
WSI_MIL/src/models/AttentionModel.py - Paper Reference: Ilse et al., Attention-based Deep Multiple Instance Learning, ICML 2018.
Description
Section titled “Description”abMIL projects patch embeddings into a hidden space and applies gated attention to compute a slide-level representation:
- Gated attention mechanism combining
tanhandsigmoidnon-linearities to calculate instance importance. - Softmax normalization across all tiles in the biopsy slide.
- Linear classification head operating on the attention-weighted slide embedding.
Clinical Use Cases
Section titled “Clinical Use Cases”- Cancer Detection & Subtyping: Primary tumor detection in biopsy cores.
- Grading & Scoring: Gleason score and ISUP Grade Group (1-5) prediction in prostatectomy and biopsy specimens.
- Microscope Heatmaps: Attention weights directly drive the interactive heatmap overlays in the ColabBio Pathology Viewer (
cb_path_viewer).
2. MultiTask-MIL (Multi-Task abMIL)
Section titled “2. MultiTask-MIL (Multi-Task abMIL)”- Source Code:
WSI_MIL/src/models/MultiTaskMIL.py - Design Pattern: Multi-Task Clinical Profiling for Precision Oncology.
Description
Section titled “Description”Built directly on top of GatedAttentionModel, MultiTask-MIL uses a single shared tissue representation to feed three specialized task heads simultaneously:
- ISUP Head: 5-class linear classifier for histological grading (CrossEntropyLoss).
- HRD Head: Binary logit predicting Homologous Recombination Deficiency or DNA repair gene mutations (BRCA1/2, ATM, CDK12) (BCEWithLogitsLoss).
- BCR Head: Scalar continuous log-hazard score for Biochemical Recurrence survival analysis (Cox Proportional Hazards).
# Combined Multi-Task Losstotal_loss = w_isup * loss_ce + w_hrd * loss_bce + w_bcr * loss_coxClinical Use Cases
Section titled “Clinical Use Cases”- One-Stop Precision Oncology Workflows: Delivers histological grade, targeted therapy eligibility (PARP inhibitors like Olaparib), and relapse risk from a standard H&E slide in a single model forward pass.
- Triton GPU Efficiency: Minimizes inference latency and memory by eliminating redundant feature pooling across separate models.
3. Cox-MIL (Survival & Time-to-Event Analysis)
Section titled “3. Cox-MIL (Survival & Time-to-Event Analysis)”- Source Code:
WSI_MIL/src/models/CoxMIL.py - Paper Reference: Mobadersany et al., Predicting cancer outcomes from histology and genomics using convolutional networks, PNAS 2018.
Description
Section titled “Description”Cox-MIL predicts a patient-level log-hazard ratio using attention pooling, optimized against the Cox Proportional Hazards Negative Partial Log-Likelihood with Breslow handling of ties:
- Calculates risk scores across patients in the mini-batch.
- Numerically stabilized with
torch.logcumsumexpover the dynamic risk sets.
Clinical Use Cases
Section titled “Clinical Use Cases”- Time-to-Recurrence Prediction: Biochemical Recurrence (BCR) in prostate cancer or disease-free survival (DFS) in colorectal cancer.
- Clinical Trial Patient Stratification: Identifying high-risk patient subpopulations eligible for adjuvant chemotherapy or novel immunotherapy arms.
4. TransMIL (Correlated MIL with Transformer)
Section titled “4. TransMIL (Correlated MIL with Transformer)”- Source Code:
WSI_MIL/src/models/TransMIL.py - Paper Reference: Shao et al., TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification, NeurIPS 2021.
Description
Section titled “Description”Standard MIL treats patches as conditionally independent. TransMIL introduces inter-instance self-attention:
- Pyramid Position Encoding Generator (PPEG): Uses 7x7, 5x5, and 3x3 depthwise convolutions to capture multi-scale spatial morphology.
[CLS]Token Aggregation: Similar to ViT, a class token aggregates global slide semantics across multi-head self-attention layers.
Clinical Use Cases
Section titled “Clinical Use Cases”- Complex Subtyping: Discriminating between histologically similar subtypes (e.g. Lung Adenocarcinoma vs. Squamous Cell Carcinoma).
- Tumor Microenvironment (TME): Analyzing the spatial relationships between tumor nests, stroma, and Tumor-Infiltrating Lymphocytes (TILs).
5. CLAM (Clustering-constrained Attention MIL)
Section titled “5. CLAM (Clustering-constrained Attention MIL)”- Source Code:
WSI_MIL/src/models/MIL_model.py - Paper Reference: Lu et al., Data-efficient and weakly supervised computational pathology on whole-slide images, Nature Biomedical Engineering 2021.
Description
Section titled “Description”CLAM introduces an auxiliary instance-clustering loss during training:
- Softmax attention scores rank the $k$ most confident positive patches (Top-k) and $k$ negative patches (Bottom-k).
- Auxiliary linear instance classifiers are trained on these subsets to reinforce discriminative feature representations.
Clinical Use Cases
Section titled “Clinical Use Cases”- Small-Scale Cohorts: Regularizes training when the number of annotated patient slides is limited (N < 200).
- Weakly Supervised Localization: Provides high-confidence identification of small tumor foci without manual annotations.
Declarative Study Specification (study.yaml)
Section titled “Declarative Study Specification (study.yaml)”You can select and compare any of these architectures in your study pipelines:
version: "1.0"study: id: "tcga-prad-mil-benchmark" title: "Multi-Model Architecture Comparison on TCGA-PRAD"
models: - name: "unified_multitask_abmil" architecture: "MultiTaskMIL" input_dim: 2048 tasks: - id: "isup_grade" type: "multiclass" classes: 5 - id: "hrd_status" type: "binary_classification" - id: "bcr_risk" type: "cox_survival"
- name: "correlated_transmil" architecture: "TransMIL" input_dim: 2048 hidden_dim: 512 depth: 2 num_heads: 8
- name: "cox_bcr_predictor" architecture: "CoxMIL" input_dim: 2048