Search, Filter, Sort and Paginate with Mongoose, and Why Indexes Matter

A list of ten jobs is easy. A list of ten thousand cannot be sent in one answer, and a person cannot read it anyway. Real list endpoints let the client say which items (filter and search), in what order (sort) and how many at a time (pagination). This lesson gives GET /api/jobs all four, and shows the database feature that keeps it fast: indexes.

The query parameters

The contract is GET /api/jobs?q=bank&status=applied&sort=company&page=2&limit=10. Everything is optional, and everything arrives as text in req.query. The controller below turns those texts into a MongoDB filter and options.

// Lesson 18: search, filter, sort and pagination.
import Job, { STATUSES } from '../models/Job.js';
import { HttpError } from '../utils/HttpError.js';
import { pickJobFields } from '../middleware/validate.js';

const mustFind = (job) => {
  if (!job) throw new HttpError(404, 'Job not found');
  return job;
};

// Make user text safe to use inside a regular expression ("C++" must not break the search).
const escapeRegex = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');

const SORTS = {
  newest: { createdAt: -1 },
  oldest: { createdAt: 1 },
  company: { company: 1, createdAt: -1 },
};

// GET /api/jobs?q=bank&status=applied&sort=company&page=2&limit=10
export const listJobs = async (req, res) => {
  const page = Math.max(1, parseInt(req.query.page, 10) || 1);
  const limit = Math.min(50, Math.max(1, parseInt(req.query.limit, 10) || 10)); // never let a client ask for 1 million rows

  const filter = {};
  if (STATUSES.includes(req.query.status)) filter.status = req.query.status;
  const q = String(req.query.q || '').trim().slice(0, 80);
  if (q) {
    const re = new RegExp(escapeRegex(q), 'i');
    filter.$or = [{ company: re }, { role: re }, { location: re }];
  }

  const [items, total] = await Promise.all([
    Job.find(filter).sort(SORTS[req.query.sort] || SORTS.newest).skip((page - 1) * limit).limit(limit),
    Job.countDocuments(filter),
  ]);
  res.json({ items, page, limit, total, pages: Math.max(1, Math.ceil(total / limit)) });
};

export const getJob = async (req, res) => res.json(mustFind(await Job.findById(req.params.id)));

export const createJob = async (req, res) => {
  const job = await Job.create(pickJobFields(req.body));
  res.status(201).json(job);
};

export const updateJob = async (req, res) => {
  const job = await Job.findByIdAndUpdate(req.params.id, pickJobFields(req.body), { new: true, runValidators: true });
  res.json(mustFind(job));
};

export const deleteJob = async (req, res) => {
  mustFind(await Job.findByIdAndDelete(req.params.id));
  res.status(204).end();
};

Reading the controller

• Page and limit are parsed with parseInt and clamped: the page is at least 1, the limit between 1 and 50. Clamping matters: without it a client could ask for limit=1000000 and make your server fetch the whole collection into memory.
• Status is only used if it is one of the known values. Anything else is ignored rather than passed to the database.
• Search builds a case-insensitive regular expression over company, role and location with $or. The text is passed through escapeRegex first. Without it a visitor typing C++ or ( would break the pattern, and a crafted pattern could make the database do a lot of work (a *ReDoS* style attack). Escaping turns every special character into a literal one.
• Sort uses a lookup table, SORTS. The client sends a name such as company, and only names in the table work. Never pass a client-supplied value directly as a sort key.
• The answer includes items, page, limit, total and pages, so the client can draw page buttons without a second request.

Notice Promise.all([find..., countDocuments(...)]). The page of items and the total are independent questions, so they run at the same time, using the idea from lesson 4. skip((page - 1) * limit).limit(limit) is the classic way to paginate. It is simple and fine for thousands of documents. For very large collections, where skipping many documents becomes slow, teams switch to “cursor” pagination based on the last id seen.

Indexes

import mongoose from 'mongoose';

export const STATUSES = ['saved', 'applied', 'interview', 'offer', 'rejected'];

const jobSchema = new mongoose.Schema(
  {
    company: { type: String, required: [true, 'company is required'], trim: true, maxlength: 120 },
    role: { type: String, required: [true, 'role is required'], trim: true, maxlength: 120 },
    location: { type: String, trim: true, maxlength: 120, default: '' },
    status: { type: String, enum: { values: STATUSES, message: 'status must be one of: ' + STATUSES.join(', ') }, default: 'saved' },
    salaryNote: { type: String, trim: true, maxlength: 120, default: '' },
    notes: { type: String, trim: true, maxlength: 2000, default: '' },
    appliedOn: { type: Date },
  },
  { timestamps: true },
);

// Indexes make the common questions fast: "newest first" and "only this status, newest first".
jobSchema.index({ createdAt: -1 });
jobSchema.index({ status: 1, createdAt: -1 });

jobSchema.set('toJSON', {
  virtuals: true,
  versionKey: false,
  transform: (doc, ret) => { delete ret._id; return ret; },
});

export default mongoose.model('Job', jobSchema);

Without an index, MongoDB answers “newest first” by looking at every document and sorting them (a *collection scan*). With an index on createdAt, it already has them in order and just reads the first page. An index is a separate, sorted structure that costs a little space and a little time on every write, but makes the matching reads dramatically faster. The second index, on status then createdAt, serves the very common question “only this status, newest first”.

A useful rule: build indexes for the queries your app actually runs, and put the field you *filter by equality* first and the field you *sort by* last. In lesson 22 both indexes will start with user, because every query will begin “this person's jobs”. You can ask MongoDB how it ran a query with .explain('executionStats') and look for IXSCAN (index used) versus COLLSCAN (everything read).

Fake data to try it on

// Adds sample jobs so you have something to search.   Run:  npm run seed
import 'dotenv/config';
import mongoose from 'mongoose';
import { connectDB } from '../config/db.js';
import Job from '../models/Job.js';

const roles = ['Teller', 'Dispatcher', 'Line Supervisor', 'Electrician', 'Accounts Clerk', 'Delivery Driver', 'Warehouse Associate', 'Customer Support'];
const companies = ['Northwind Bank', 'Acme Logistics', 'Greenfield Foods', 'Sunrise Electricals', 'Lakeside Hospital', 'Metro Retail'];
const statuses = ['saved', 'applied', 'interview', 'offer', 'rejected'];

await connectDB();
const docs = Array.from({ length: 30 }, (_, i) => ({
  company: companies[i % companies.length],
  role: roles[i % roles.length],
  location: ['Pune', 'Lagos', 'Toronto', 'Leeds', 'Nairobi'][i % 5],
  status: statuses[i % statuses.length],
}));
await Job.insertMany(docs);
console.log(`Added ${docs.length} sample jobs`);
await mongoose.disconnect();

Your task

1. Run npm run seed to add 30 jobs, then try GET /api/jobs?limit=5&page=3.
2. Search for bank, then filter with status=interview, then combine them.
3. Try limit=9999 and page=-4. What does the server do?
4. Search for c++ and for (. Confirm there is no error. Then temporarily remove escapeRegex and try again.

How to check it worked

• total and pages change with the filters.
• Out-of-range values are clamped, not an error or a crash.
• Special characters in the search box work as plain text.

Summary

• Filter, search, sort and paginate on the server, and always clamp and whitelist what the client sends.
• Escape user text before putting it in a regular expression.
• Index the fields you filter and sort on; check with explain.