vfs.yaml Specification

1. Introduction

MedVFS (Medical Virtual Filesystem) is the data orchestration engine of the ColabBio platform. Its function is to act as an abstraction layer between heterogeneous hospital data systems (clinical databases, medical image servers, genomic repositories) and ML model training and validation containers.

The configuration for each execution is described in a single declarative file, vfs.yaml, which specifies:

1. What data is needed (sources, queries, filters).
2. How to access those sources (endpoints, authentication, secrets).
3. Where to materialize the results (K8s PVC, local directory).
4. How aggressive the extraction should be (parallelism, limits).

Design Philosophy

• Declarative: All configuration lives in vfs.yaml, zero code.
• Data Agnostic: A single format for OMOP, FHIR, DICOM, WSI, and Genomics.
• Data doesn’t move: For image/genomic sources, manifests with paths are generated, not copies.
• Total Reproducibility: Each execution produces an immutable snapshot + a signable manifest.json.
• Sovereignty: Data never leaves the hospital perimeter; the model goes to the data.
• Secure by Default: Secrets are never in plain YAML; always referenced (k8s://, env://).

2. Root Structure

version: "1"             # required — schema version
metadata:                # required — identifies the execution
  name: string
  namespace: string      # optional — target K8s namespace
  run_id: string         # optional — used as suffix in PVC/sandbox names

storage:                 # required — where to materialize outputs
  type: emptydir | hostpath | pvc
  host_path: string      # if type=hostpath
  pvc_name: string       # if type=pvc
  subpath: string        # optional subdirectory within the volume
  output_dir: string     # direct path override (dev/test)

setup:                   # required — execution control
  parallelism: int       # max concurrent adapters (default: 4)
  fail_fast: bool        # abort all on first error (default: false)

sources:                 # required — list of data sources
  - ...                  # (see below)

serve:                   # optional — serve mode configuration (FUSE daemon)
  fuse:
    mount_point: string  # FUSE mount path (e.g. /data/vfs)
    cache_dir: string    # block cache directory
    cache_max_gb: int    # max cache size in GB (default: 50)
    block_size_mb: int   # block size in MB (default: 4)
    prefetch:            # prefetch engine config
      enabled: bool
      strategy: string   # sequential | format_aware | manifest
      read_ahead_blocks: int  # L1 read ahead blocks
      concurrent_fetchers: int # parallel workers
      warmup_percent: int     # L3 warmup percentage
  http:
    port: int            # HTTP API port (default: 8090)
  live_monitoring: bool  # enable streaming adapters (fsnotify)

mount:                   # required — ML container mount config
  path: string           # base path where external container will see data
  readonly: bool         # true in external execution sandbox

3. Adapters Overview

Adapter (type)	Protocol	Source	Output
omop	SQL (pgx)	OMOP CDM PostgreSQL	CSV with query rows
fhir	FHIR R4 REST	HAPI FHIR / Any FHIR server	CSV with flattened fields
omero	OMERO.web API	OMERO Server (WSI)	CSV metadata + PNG thumbnails
orthanc	DICOMweb	Orthanc / PACS	CSV studies + JSON series manifests
genomics	NFS / GCS / S3	Genomic Filesystem	CSV manifest with validated paths
dicom	Local Scanner	Local DICOM Receiver	CSV + FUSE Symlink Tree