Quality
Quality belongs to a model on your task, with your prompts, against your definition of correct. Running your own examples is Chapter 2. Trusting the result, with slices and gates, is Chapter 9.
estimate_footprint(candidate, serving, weight_bits=4)
Right-Sized AI Small Language Models in Production
Adapt, Compress, and Deploy SLMs from Server to Edge Device.
Capability is a function of training and engineering, not size.
Right-sizing is starting from the production constraint, not from the model catalog: identify the smallest credible model that could plausibly solve the task, then use training, compression, and deployment engineering to close whatever gap remains.
Kindle in other Amazon stores: UK · DE · FR · ES · IT · NL · JP · AU · BR · CA · MX
All the code is open: the companion repository on GitHub, one tag per chapter.
6Parts
15Chapters
5Appendices
417Passing tests
0GPUs to run them
The problem
Your model works in the demo and fails in production.
It misses the latency target, forces a GPU tier nobody budgeted for, or cannot run inside the customer environment that actually matters. The problem is rarely the model family. It is that the model was chosen on the wrong axis.
Right-Sized AI is a practitioner’s guide to the other approach: start from the constraint, pick the smallest model that can plausibly do the job, and use training and systems engineering to close the rest of the gap. The size of a model should be the conclusion of an argument, never its premise.
Where the numbers come from
No figure in this book is presented as measured by the author. Every quantitative table declares its provenance in its caption: computed, cited, or illustrative.
The book gives you arithmetic that predicts a number and tooling that produces one. It does not give you a table of results measured on someone else’s machine, because those do not transfer.
The Chapter 1 estimator, in your browser
Will it fit? Ask before you download.
State the bounds a deployment must clear, add the models you are considering, and see which ones the arithmetic rejects before you download a single weight. Arithmetic rules candidates out cheaply, and it never rules one in. A candidate that passes has earned the right to be measured on your task, and that is all it has earned.
1. Your envelope
Size for the worst case you will accept, not the case you expect.
2. Candidates
Enter a Hugging Face model id to read its config.json, or fill
in the shape by hand. Parameter counts come from the Hub’s safetensors metadata when
it has them; check them against the model card, because a “7B” is rarely
7.0.
3. Verdict
Memory is judged at the high end of the range, because an overestimate wastes some memory and an underestimate takes down the service.
What this page cannot tell you
The operating envelope has four terms, and arithmetic reaches two of them.
Cost
On rented hardware cost comes from throughput and utilization, both of which need measuring. On owned hardware it was fixed at procurement. That is Chapter 13.
Peak, not average
Peak memory comes during prefill of your longest prompt while other requests hold their caches. Decide what happens when a request exceeds the limit: rejection, truncation or queueing.
The arithmetic is a port of rightsize/sizing.py from the
book’s companion repository, checked against it value for value. Every figure here is
computed, not measured. The overhead band of 0.5 to 2.0 GiB is the observed range for
server runtimes.
Named concepts
Thirty-nine named concepts.
Each chapter names the concepts it introduces. Six of them, quoted from their definitions.
Ch 1
Operating envelope
The set of bounds a candidate must clear at the same time: task quality, latency, memory, and cost. A candidate that clears three and misses one has failed.
Ch 3
Embedding tax
The fraction of a model’s parameters held in its vocabulary table, not in its transformer stack.
Ch 6
Small-model penalty
The observed inverse relationship between model size and quantization damage: at the same bit width, smaller models lose more.
Ch 11
Concurrency ceiling
The number of requests a device can hold simultaneously: the memory left after weights and runtime overhead, divided by the KV cache one request claims at your context length.
Ch 12
Unattended constraint
The requirement that a deployed device recover from every failure without a human: power cut, corrupt download, out-of-memory kill, a model file that fails to load.
Ch 14
Inert technique
One whose premise does not hold in your deployment, as distinct from one that holds and does not help much.
Who this book is for
You are shipping a language model under a constraint you do not control.
That includes applied ML and AI engineers, platform and MLOps engineers, embedded and edge engineers putting models on hardware, and technical leads who have to defend a model choice with numbers.
You do not need to have fine-tuned, quantized, or served a model before. Those are what the book teaches. You do not need a GPU cluster either. Every technique is demonstrated on hardware an individual engineer can reach, and the book is explicit about which results require more.
What to expect
You will see opinions, not neutral surveys. Where the evidence supports a position, the book takes it and says why. Where the honest answer is that a larger model wins, the book says that too. Chapter 15 exists for exactly that reason.
Contents
Six parts, in the order a real project moves through.
Part I
The Right-Sizing Decision
- 1 The Case for Small
- 2 Your First Right-Sized Model
- 3 Where Small-Model Capability Comes From
Part II
Adapting the Model
- 4 Fine-Tuning and Adaptation
- 5 Distillation and Pruning
Part III
Making It Fit
- 6 Quantization I: What Precision Actually Costs
- 7 Quantization II: Formats, Runtimes, and Artifacts
Part IV
Making It Reliable
- 8 Structured Output, Tool Calling, and Constrained Decoding
- 9 Evaluation: Harnesses, Slices, and Gates
- 10 Composition: Routing, Cascades, and Speculative Decoding
Part V
Shipping It
- 11 Serving on Servers and Workstations
- 12 Edge and Embedded Deployment
- 13 Fleet Operations and On-Device Lifecycle
Part VI
Putting It Together
- 14 End-to-End Case Studies
- 15 Limits and the Road Ahead
A · Model Survey
B · Environment Setup and Hardware Reference
C · Book Scope and Stack Contract
D · Glossary
E · Memory and Throughput Math
The companion repository
One project, one tag per chapter. No GPU needed to test it.
The repository builds one project, Rightsize: a constraint-aware
measurement and deployment toolkit that grows across the book. It measures a model
against an envelope, then trains, compresses, serves, and ships it. At the release
tag, code-v1.0, the suite stands at 417 passing tests.
MIT License.
git clone https://github.com/ranjankumar-gh/right-sized-ai-codebase cd right-sized-ai-codebase uv sync uv run pytest tests/ -v
The test suite runs without a GPU and without downloading model weights. Anything that needs real hardware or a real model is behind an environment-variable guard and skips cleanly.
The author
Ranjan Kumar
Ranjan Kumar builds and writes about production AI systems. He holds an M.Tech in Artificial Intelligence from IIT Jodhpur and has 20 years of experience in the field. He writes at ranjankumar.in on production machine learning, model efficiency, and the engineering tradeoffs that shape deployed AI.
His other books:
Errata and feedback
The field this book describes moves quickly, and parts of it will drift. Confirmed
corrections and post-publication changes are tracked in ERRATA.md at the
root of the companion repository.
If you find an error, a stale runtime claim, or a production counterexample that materially changes the book’s guidance, the fastest route is an issue on the companion repository.