You do not need to design a model to use one, but a few terms help you choose models sensibly.
Parameters
Parameters are the numbers a model learns during training. More parameters usually mean more capability, and also more memory, time and cost to run.
Ways models are made smaller and faster
- Quantization stores the numbers with less precision (for example 4 or 8 bits instead of 16) so the model needs less memory. Quality drops slightly.
- Distillation trains a small "student" model to copy the behaviour of a large "teacher" model.
- Mixture of Experts (MoE) splits the model into specialist parts and uses only a few of them for each token, so a very large model can run faster than its size suggests.
Inference
Running a trained model is called inference. Two numbers matter: latency (how long until you get the first token or the full reply) and throughput (how many requests or tokens per second the system can handle). Long inputs, long outputs and big models all slow things down.
GPUs and accelerators
Models run on GPUs or similar accelerators because they do many multiplications in parallel. Memory on the GPU often decides which model can run at all, which is why quantization is popular for local models.