Research Systems
This page summarizes my public research systems and artifacts. The systems are organized as a portfolio rather than as isolated projects: together they study how KV-cache can be placed, offloaded, scheduled, predicted, and moved across heterogeneous memory and GPU interconnects.
Portfolio map
| System | Role | Current scope |
|---|---|---|
| OrchKvCache | Tiered KV-cache substrate | HBM / DRAM / SSD object model, migration, eviction, and hotness-aware prefetch |
| HALO | Semantic-safe offloading | Full-attention-preserving KV tiering with algebraic identity guarantees |
| SEER | SLO-aware scheduling | Probabilistic sizing and schedulable KV-cache eviction for real-time decoding |
| XQP | Saliency audit | Measured audit of KV-saliency prediction from LLM attention dynamics |
| PeerKV | Cross-GPU KV handoff | NVLink handoff direction, contention, and cross-GPU KV movement behavior |
Systems
OrchKvCache
OrchKvCache studies KV-cache as a managed storage object. It explores a heterogeneous memory hierarchy with GPU HBM (fast GPU memory), host DRAM (larger warm memory), and SSD (large cold storage). The system tracks KV blocks, classifies them as hot or cold, migrates them across tiers, and schedules prefetches based on attention-derived hotness.
HALO
HALO focuses on semantic safety. Instead of discarding cold KV blocks, it moves them out of GPU memory and streams them back during decoding. Its tiering path preserves full-attention semantics through algebraic reconstruction rather than treating eviction as the only way to fit long contexts.
SEER
SEER treats KV-cache management as a soft real-time scheduling problem. It asks whether a KV policy can satisfy serving objectives such as P99 TPOT below a target deadline. The policy combines learned block usefulness, tier-specific IO cost, and a probabilistic sizing bound.
XQP
XQP is a measurement-driven audit of KV-block saliency prediction. It asks which cheap signals predict future important blocks, including within-layer attention magnitude, cross-layer hotness, query-key affinity, and recency. The goal is not to build the largest predictor, but to identify what the measured attention dynamics actually support.
PeerKV
PeerKV studies cross-GPU KV-cache handoff behavior on modern GPU interconnects. It focuses on whether handoff direction is a real scheduling knob, how NVLink contention shapes KV movement, and when cross-GPU KV placement helps or hurts long-context inference.
Design principle
The common theme is that long-context LLM inference should manage KV-cache explicitly, with clear distinctions between:
- Exact placement: data is moved but full-attention semantics are preserved.
- Approximate compression: data is kept but represented with lower precision.
- Lossy eviction: data may be dropped or ignored, creating quality risk.
- Latency-aware scheduling: placement decisions are constrained by SLOs.
- Interconnect-aware movement: cross-GPU KV handoff is shaped by directionality, contention, and GPU topology.