ML Systems
Efficient LLM/VLM Compression Research
Researching low-rank and activation-aware methods for reducing language and vision-language model memory and inference cost while preserving model quality.
- Python
- PyTorch
- CUDA
- Slurm
- Hugging Face
- Llama
- LLaVA
Problem
Large language and vision-language models are expensive to serve: memory footprint and inference latency scale with parameter count, which limits where and how cheaply they can be deployed.
Constraints
- Compression must preserve task quality within an acceptable tolerance, not just reduce size
- Evaluation has to hold across both language-only and vision-language tasks
- Methods should be reproducible on shared, resource-constrained compute (Slurm-scheduled GPUs)
Approach
Applies low-rank decomposition to weight matrices and activation-aware calibration to decide which layers tolerate aggressive rank reduction, then re-evaluates the compressed model rather than assuming compression is quality-neutral.
System design
Compression and evaluation pipeline
Each candidate model is compressed, then run through the same evaluation harness used for the uncompressed baseline so quality, memory, and speed are measured on equal footing.
- Baseline model (Llama / LLaVA checkpoints)
- Activation profiling to identify compressible layers
- Low-rank decomposition + calibration
- Compressed model checkpoint
- Evaluation harness (language + vision-language benchmarks)
- Memory / speed / quality comparison report
Tradeoffs and limitations
Compression ratio and task quality trade off directly, and the right operating point depends on the deployment target — the work focuses on characterizing that curve rather than optimizing a single fixed ratio.
Results
Full benchmark numbers are still being finalized and verified against the source materials before publishing — see the TODO in the project data file. This case study will be updated with specific figures once confirmed.
What I learned
Reproducible evaluation infrastructure matters as much as the compression method itself — most of the engineering effort went into making sure comparisons between baseline and compressed models were apples-to-apples.
Next steps
Extend evaluation to additional vision-language tasks and publish a reproducible benchmarking harness alongside the compression code.