Optimizing Ornith 9B Model on a Single GPU: Costs, Latency, and Performance

We conducted a fine-tuning experiment on [Ornith-1.0-9B], an open-source 9 billion parameter model designed for coding tasks, using a 61,000-example dataset focused on reasoning and summarization. This was achieved on a single H200 GPU Droplet. The objective was to modify the model's output to present a structured reasoning summary, comprising a title, subtitle, plain-language summary, and a current-task field, instead of the model's full chain of thought.

Key Insights

The primary aim was to adjust how the model's reasoning is displayed, without altering its core task performance. This fine-tuning does not claim to enhance accuracy or agentic behavior. Instead, it focuses on the cost of training, latency, throughput, and any potential impact on inference performance. Notably, there was no negative effect on inference performance.

Motivation

Developers creating user-facing products often face a dilemma: either conceal the model's reasoning, which complicates debugging and trust-building, or present it in raw form, which can be overwhelming and sometimes inappropriate. A model that outputs a neat, structured summary of its reasoning without needing additional summarization processes provides an effective solution. The key question was whether this adjustment incurs any serving-time costs.

Training Setup

Ornith-1.0-9B, the smallest in its family of models, was fine-tuned using LLaMA-Factory on a single H200 GPU. This dense model, based on Qwen 3.5, operates efficiently in bf16 on an 80GB GPU. It's trained using reinforcement learning to build its own problem-solving framework.

The training dataset, composed of 61,000 examples, is tailored for producing structured reasoning summaries. Each example includes raw reasoning traces and their summarized versions, along with structured fields.

The following hyperparameters were used:

• Learning rate: 5e-5
• LR scheduler: Cosine
• Epochs: 3.0
• Batch size: 2 (effective 16 with gradient accumulation)
• Gradient accumulation: 8
• Max gradient norm: 1.0
• Cutoff length: 10,200
• Compute type: bf16

Training Results and Costs

• Epochs: 3.0
• Tokens processed: 75,092,496
• Total FLOPs: 3.790 × 10¹⁸
• Final train loss: 0.550
• Runtime: Approximately 24.7 hours
• Cost: $84.88 for the full dataset at $3.44/hour

Serving Setup

The fine-tuned model was deployed on the same H200 Droplet using a model serving platform, running behind an OpenAI-compatible endpoint. Latency and throughput were measured using a benchmarking tool that fired concurrent requests and recorded time-to-first-token (TTFT) and tokens-per-second (TPS).

During testing, two observations were noted: a higher TTFT on the first run at each concurrency level due to CUDA graph capture processes and one complete request failure out of 100 total.

Inference Results

• Fine-tuned Model Performance:

  • Concurrency 1: Avg TTFT 0.021s, Avg TPS 187.63 tok/s
  • Concurrency 5: Avg TTFT 0.036s, Avg TPS 178.94 tok/s
  • Concurrency 20: Avg TTFT 0.047s, Avg TPS 159.05 tok/s

• Comparison with Base Model:

  • Differences were negligible, indicating no additional serving overhead from fine-tuning.

Outcome and Use Cases

The fine-tune did not affect serving performance but changed the output's structure. Instead of a raw, unstructured reasoning chain, the fine-tuned model delivers concise, structured JSON objects that are easier to integrate into user interfaces.

This approach is suitable for applications where reasoning transparency is desired without revealing the entire thought process. However, it should not be considered as evidence of improved task accuracy.

Conclusion

This experiment demonstrates that fine-tuning for output format does not compromise performance, offering a practical solution for structured reasoning summaries in AI models.