Skip to content

Supported MIL Models & Architectures

ColabBio provides a comprehensive suite of Multiple Instance Learning (MIL) architectures in WSI_MIL/src/models/, optimized for gigapixel Whole Slide Images (WSI) and multi-modal clinical endpoints (genomics, survival, grading).

graph TD
    WSI["Gigapixel WSI Slide"] --> ENCODER["Foundation Feature Encoder<br/>(ResNet50 / UNI / Virchow / CTransPath)"]
    ENCODER --> BAG["Instance Patch Embeddings Bag<br/>(N x 2048 / N x 768)"]

    BAG --> M1["<b>abMIL</b><br/>Gated Attention Pooling<br/>(Ilse et al.)"]
    BAG --> M2["<b>MultiTask-MIL</b><br/>Shared abMIL Backbone<br/>+ Parallel Heads"]
    BAG --> M3["<b>Cox-MIL</b><br/>Attention Survival<br/>+ Breslow NLL Loss"]
    BAG --> M4["<b>TransMIL</b><br/>Transformer Self-Attention<br/>+ PPEG 2D Encoding"]
    BAG --> M5["<b>CLAM</b><br/>Clustering-constrained<br/>Instance Attention"]

    M1 --> O1["Gleason / ISUP Classification<br/>+ Attention Heatmaps"]
    M2 --> O2["Simultaneous Grade +<br/>HRD/BRCA + BCR Survival Risk"]
    M3 --> O3["Continuous Risk Score &theta;<br/>(Time-to-Event / BCR / OS)"]
    M4 --> O4["Complex Spatial / Subtyping<br/>(LUAD vs LUSC, Stroma-Tumor)"]
    M5 --> O5["Low-data Regime Regularization<br/>+ Patch Pseudo-clustering"]

Model Architecture Outputs / Tasks Primary Clinical Use Case Compute Footprint
abMIL Gated Attention Pooling Single/Multiclass logits + patch attention weights Standard histopathology grading (Gleason/ISUP), interpretable attention heatmaps Low (O(N) linear in patches, ultra-fast)
MultiTask-MIL Shared abMIL Backbone + Parallel Prediction Heads Grade logits + Binary Biomarker logit + Continuous Hazard Risk Unified multi-task profiling in 1 forward pass (e.g., TCGA-PRAD Gleason + HRD + BCR) Low / Medium (shares slide pooling across all tasks)
Cox-MIL Attention Pooling + Cox Proportional Hazards Head Continuous log-hazard ratio (θ) Prognostic stratification, Biochemical Recurrence (BCR), Overall Survival (OS) Low (requires mini-batches with observed events)
TransMIL Multi-Head Self-Attention + PPEG + [CLS] Token Slide classification / regression with inter-patch dependencies Difficult morphological subtyping where long-range tissue interactions matter Medium / High (quadratic/approximate self-attention)
CLAM Gated Attention + Instance-level Pseudo-Clustering Slide logits + top-k / bottom-k instance losses Low-sample training cohorts, weakly supervised focus localization Medium (calculates patch-level instance losses)

1. abMIL (Attention-based Deep Multiple Instance Learning)

Section titled “1. abMIL (Attention-based Deep Multiple Instance Learning)”
  • Source Code: WSI_MIL/src/models/AttentionModel.py
  • Paper Reference: Ilse et al., Attention-based Deep Multiple Instance Learning, ICML 2018.

abMIL projects patch embeddings into a hidden space and applies gated attention to compute a slide-level representation:

  • Gated attention mechanism combining tanh and sigmoid non-linearities to calculate instance importance.
  • Softmax normalization across all tiles in the biopsy slide.
  • Linear classification head operating on the attention-weighted slide embedding.
  • Cancer Detection & Subtyping: Primary tumor detection in biopsy cores.
  • Grading & Scoring: Gleason score and ISUP Grade Group (1-5) prediction in prostatectomy and biopsy specimens.
  • Microscope Heatmaps: Attention weights directly drive the interactive heatmap overlays in the ColabBio Pathology Viewer (cb_path_viewer).

  • Source Code: WSI_MIL/src/models/MultiTaskMIL.py
  • Design Pattern: Multi-Task Clinical Profiling for Precision Oncology.

Built directly on top of GatedAttentionModel, MultiTask-MIL uses a single shared tissue representation to feed three specialized task heads simultaneously:

  1. ISUP Head: 5-class linear classifier for histological grading (CrossEntropyLoss).
  2. HRD Head: Binary logit predicting Homologous Recombination Deficiency or DNA repair gene mutations (BRCA1/2, ATM, CDK12) (BCEWithLogitsLoss).
  3. BCR Head: Scalar continuous log-hazard score for Biochemical Recurrence survival analysis (Cox Proportional Hazards).
# Combined Multi-Task Loss
total_loss = w_isup * loss_ce + w_hrd * loss_bce + w_bcr * loss_cox
  • One-Stop Precision Oncology Workflows: Delivers histological grade, targeted therapy eligibility (PARP inhibitors like Olaparib), and relapse risk from a standard H&E slide in a single model forward pass.
  • Triton GPU Efficiency: Minimizes inference latency and memory by eliminating redundant feature pooling across separate models.

3. Cox-MIL (Survival & Time-to-Event Analysis)

Section titled “3. Cox-MIL (Survival & Time-to-Event Analysis)”
  • Source Code: WSI_MIL/src/models/CoxMIL.py
  • Paper Reference: Mobadersany et al., Predicting cancer outcomes from histology and genomics using convolutional networks, PNAS 2018.

Cox-MIL predicts a patient-level log-hazard ratio using attention pooling, optimized against the Cox Proportional Hazards Negative Partial Log-Likelihood with Breslow handling of ties:

  • Calculates risk scores across patients in the mini-batch.
  • Numerically stabilized with torch.logcumsumexp over the dynamic risk sets.
  • Time-to-Recurrence Prediction: Biochemical Recurrence (BCR) in prostate cancer or disease-free survival (DFS) in colorectal cancer.
  • Clinical Trial Patient Stratification: Identifying high-risk patient subpopulations eligible for adjuvant chemotherapy or novel immunotherapy arms.

4. TransMIL (Correlated MIL with Transformer)

Section titled “4. TransMIL (Correlated MIL with Transformer)”
  • Source Code: WSI_MIL/src/models/TransMIL.py
  • Paper Reference: Shao et al., TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification, NeurIPS 2021.

Standard MIL treats patches as conditionally independent. TransMIL introduces inter-instance self-attention:

  1. Pyramid Position Encoding Generator (PPEG): Uses 7x7, 5x5, and 3x3 depthwise convolutions to capture multi-scale spatial morphology.
  2. [CLS] Token Aggregation: Similar to ViT, a class token aggregates global slide semantics across multi-head self-attention layers.
  • Complex Subtyping: Discriminating between histologically similar subtypes (e.g. Lung Adenocarcinoma vs. Squamous Cell Carcinoma).
  • Tumor Microenvironment (TME): Analyzing the spatial relationships between tumor nests, stroma, and Tumor-Infiltrating Lymphocytes (TILs).

5. CLAM (Clustering-constrained Attention MIL)

Section titled “5. CLAM (Clustering-constrained Attention MIL)”
  • Source Code: WSI_MIL/src/models/MIL_model.py
  • Paper Reference: Lu et al., Data-efficient and weakly supervised computational pathology on whole-slide images, Nature Biomedical Engineering 2021.

CLAM introduces an auxiliary instance-clustering loss during training:

  • Softmax attention scores rank the $k$ most confident positive patches (Top-k) and $k$ negative patches (Bottom-k).
  • Auxiliary linear instance classifiers are trained on these subsets to reinforce discriminative feature representations.
  • Small-Scale Cohorts: Regularizes training when the number of annotated patient slides is limited (N < 200).
  • Weakly Supervised Localization: Provides high-confidence identification of small tumor foci without manual annotations.

Declarative Study Specification (study.yaml)

Section titled “Declarative Study Specification (study.yaml)”

You can select and compare any of these architectures in your study pipelines:

version: "1.0"
study:
id: "tcga-prad-mil-benchmark"
title: "Multi-Model Architecture Comparison on TCGA-PRAD"
models:
- name: "unified_multitask_abmil"
architecture: "MultiTaskMIL"
input_dim: 2048
tasks:
- id: "isup_grade"
type: "multiclass"
classes: 5
- id: "hrd_status"
type: "binary_classification"
- id: "bcr_risk"
type: "cox_survival"
- name: "correlated_transmil"
architecture: "TransMIL"
input_dim: 2048
hidden_dim: 512
depth: 2
num_heads: 8
- name: "cox_bcr_predictor"
architecture: "CoxMIL"
input_dim: 2048