MedVFS Dataset Generation & Access Strategies

MedVFS (Medical Virtual File System) provides dual operating modes to accommodate both restricted compute environments (without root/kernel privileges) and high-performance streaming environments.

graph TD
    VFS_YAML["vfs.yaml Specification"] --> STRATEGY{"Environment Strategy"}
    
    STRATEGY -->|Restricted K8s Pods / No FUSE| STATIC["Static Materialization (symlink / download)"]
    STRATEGY -->|Root /dev/fuse Allowed / Edge VM| DYNAMIC["Dynamic FUSE Daemon (medvfs serve)"]
    
    STATIC --> PVC["Output PVC: /mnt/data/study-dataset"]
    STATIC --> MANIFEST["Reproducible manifest.json"]
    
    DYNAMIC --> POSIX_MOUNT["POSIX Mount: /data/vfs"]
    DYNAMIC --> HTTP_API["HTTP API (:8090) /stats, /events"]
    
    PVC --> ML["ML Training Pod (PyTorch / ABMIL)"]
    POSIX_MOUNT --> ML

---

1. Static Generation Mode (symlink / download)

In production Kubernetes clusters where pods cannot mount /dev/fuse due to security constraints, MedVFS runs as an init container or preprocessing job to materialize the virtual file tree.

Execution via CLI

medvfs setup \
  --config /etc/colabbio/vfs.yaml \
  --strategy symlink \
  --output /mnt/data/study-dataset

Generated File Hierarchy

MedVFS extracts records from OMOP, FHIR, and PACS, organizing them into a deterministic folder structure:

/mnt/data/study-dataset/
├── manifest.json
├── cohorts/
│   └── diabetic_nephropathy_cohort.csv
├── patients/
│   ├── PT_00101/
│   │   ├── clinical_records.json
│   │   ├── slides/
│   │   │   └── biopsy_cortex_20x.svs -> /mnt/storage/raw_wsi/9941.svs
│   │   └── radiology/
│   │       └── ct_contrast_axial.dcm -> /mnt/storage/pacs/ct_01.dcm
│   └── PT_00102/
│       └── ...

Study Manifest (manifest.json)

The generated manifest acts as an immutable audit snapshot for AI training:

{
  "study_id": "STUDY-ABMIL-2026",
  "generated_at": "2026-08-16T19:40:00Z",
  "total_patients": 420,
  "sources": [
    { "type": "omop", "rows": 420, "target": "cohorts/patients.csv" },
    { "type": "dicom", "series_count": 840, "target": "radiology/" },
    { "type": "omero", "image_count": 420, "target": "slides/" }
  ]
}

---

2. Dynamic FUSE Daemon Mode (medvfs serve)

When deployed on dedicated Edge VMs or nodes with /dev/fuse enabled, MedVFS operates as a resident background daemon providing zero-copy streaming with intelligent caching.

medvfs serve --config /etc/colabbio/vfs.yaml

Multi-Tier Caching Architecture

Cache Tier	Mechanism	Purpose
L1: Sequential Prefetch	Read-ahead blocks N+1 .. N+3	Accelerates sequential CSV/EHR record reading.
L2: Format-Aware Prefetch	Header parsing for SVS/TIFF/DICOM	Pre-loads image pyramid directory headers and metadata tags.
L3: Manifest Warmup	Background cache fill based on study cohort	Warms high-priority slide tiles prior to epoch training.

MedVFS Daemon HTTP API (:8090)

• GET /manifest: Returns the live virtual file system tree structure.
• GET /stats: Returns cache hit ratio, network bandwidth saved, and active POSIX file handles.
• GET /events: Server-Sent Events (SSE) stream announcing newly discovered DICOM/WSI files.