Machine Learning Engineer · LLM Inference & Systems
I am a Machine Learning Engineer at Zoom in Seattle, where I work on LLM inference systems — kernel-level optimization, speculative decoding, deployment auto-tuning for vLLM, and the infrastructure behind Zoom's AI services across ASR, translation, and reasoning.
Previously, I worked on ML inference at AWS Neuron (Amazon's Trainium accelerator team) and on distributed systems at Amazon. I received my M.S. in Computational Data Science from Carnegie Mellon University, and bachelor's degrees from the University of Michigan and Shanghai Jiao Tong University.
I contribute to MLC-AI (WebLLM, MLC-LLM), and previously to CMU Catalyst Lab and Intel Analytics-Zoo. My research interests span LLM inference systems, speculative decoding, ML compilation on custom accelerators, and federated learning.
A complete list is available on Google Scholar. ✱ denotes equal contribution.
Nature 649, pp. 1139–1146 · 2026 Nature Benchmark
Also known as Humanity's Last Exam. A 3,000-question benchmark spanning expert-level academic domains designed to be adversarial to frontier LLMs at the limit of human knowledge.
CoRR (arXiv:2412.15803) · MLC-AI · 2024 MLSys
A high-performance LLM inference engine that runs entirely in the browser, leveraging WebGPU for hardware acceleration. Brings privacy-preserving, serverless LLM inference to web applications without any backend infrastructure.
IEEE Transactions on Mobile Computing · 2024 Journal
Extended journal version of our FL-AAAI work. A contextual-bandit-based client selection method for federated learning over heterogeneous edge devices, with formal regret analysis and improvements over Oort in convergence speed and accuracy.
NeurIPS 2024 Workshop on Fine-Tuning in Modern ML: Principles & Scalability Workshop
A surgical layer-transplantation procedure converting attention layers in pretrained Transformers into Mamba-style state-space layers — trading compute for sub-quadratic inference while preserving most of the original model's quality with only 0.9 B tokens of fine-tuning and ~3× lower inference memory.
Int'l Workshop on Trustable, Verifiable & Auditable Federated Learning, AAAI 2022 Oral
A novel neural combinatorial contextual bandit (NCCB) intelligently selects clients in each federated round while preserving privacy, surpassing the state-of-the-art Oort in both convergence speed and final accuracy.
Contributed to the TVM-based MLC-LLM / WebLLM projects (>10k stars). Compiler-level optimizations for LLaMA, Gemma, Phi, Qwen; paged KV cache for memory-efficient inference; WebGPU in-browser execution with IndexedDB client-side caching for privacy-preserving on-device LLMs.
Resource-efficient framework constructing hybrid Transformer–SSM architectures via layer-importance-guided transplantation. ~3× lower inference memory with only 0.9 B fine-tuning tokens, preserving long-context performance.
An RL-based LLM router using contextualized query vectors and PPO to select among different quantized models, balancing quality vs. latency vs. cost.
UMTRI Multidisciplinary Development Program. Light-weight CNN (90.2% accuracy), Java labeling software, Bayesian CNN for weight uncertainty, and an end-to-end eye-gaze collector with PyGaze + React.
Lightweight collaborative super-resolution for on-device video inference (TensorFlow). 33.8 PSNR on 2K gaming datasets — 7.8% over BasicVSR; optimized Android inference pipelines for an 11.2% mobile deployment efficiency gain.
Constructed user-interaction graphs from GitHub Archive event streams with Spark; trained an unsupervised GraphSAGE model to embed contributor behavior and uncover correlations between linguistic / emoji patterns and group productivity.
Reinforcement-learning controller for building emissions; real-time dashboard backed by a streaming database.
React + Flask interface for voice-driven email; 7.8% WER speech-to-text with smart-reply via Dialogflow / Rasa.
In-browser LLM inference via WebGPU. Compiler-driven model deployment to web, mobile, and edge (10k+ stars).
Systems-for-AI research group at Carnegie Mellon working on efficient model serving and training.
Unified analytics + AI platform for Apache Spark and Ray. Contributed during 2021 Intel internship.
Public repos across ML systems, web infrastructure, and research codebases.
vLLMSGLangTensorRT-LLMCuTe DSLCutileTritonFlashAttentionTVM / MLC-LLM
PyTorchTensorFlowMLIRcuDNNMPIOpenMPNCCL
C++PythonCUDARustGoJavaTypeScriptScalaJavaScriptSQLBash
AWSKubernetesDockerTerraformSparkStep FunctionsSmithyAzureGCP
PostgreSQLMongoDBRedisKafkaCassandraBigQueryRedshiftInfluxDBTimescaleDB
M.S. Computational Data Science · System Track · School of Computer Science GPA 4.0 / 4.0 TA · Advanced Cloud Computing
B.S.E. Computer Science · Minor in Mathematics GPA 3.978 / 4.00 Dean's List · University Honors
B.S.E. Electrical & Computer Engineering · UM-SJTU Dual Degree GPA 3.89 / 4.00 Fan Xuji Scholarship · Undergrad Excellence
Winter Exchange Program · Kyoto, Japan
Happy to chat about ML infrastructure, LLM inference, research collaborations, or open-source contributions. Email is the fastest way:
hangrui@umich.edu