Inference Metrics Framework: TTFT TPS and Memory Profiling
Parent: Local Model Performance Evaluation: Ollama, MLX, and Hardware Architectures · Published reference · snapshot 2026-09-30
↓ Facts as markdownall context files
Source-anchored research for Inference Metrics Framework: TTFT TPS and Memory Profiling. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts and statements
- Time to First Token (TTFT) is compute-bound, determined by prompt batch parallel FLOPs on GPU compute cores, whereas auto-regressive token generation (TPS) is memory bandwidth-bound. (high confidence) [source]
- Measuring peak VRAM consumption requires tracking resident set size (RSS), active Metal wired memory allocations, and discrete GPU allocated VRAM during maximum prompt length ingestion. (high confidence) [source]
- Reporting mean tokens per second without measuring inter-token latency variance masks stutter caused by periodic garbage collection, dynamic KV cache reallocation, and memory compaction. (high confidence) [source]
- Automated evaluation pipelines must record thermal throttling states and host bus bandwidth saturation alongside inference latency to ensure benchmark reproducibility. (high confidence) [source]
Children
- No children recorded.