Pragmatic Scalability: What I Do Before Adding Servers or Splitting into Microservices
When a system starts feeling slow, flaky, or kind of unstable, two reflexes show up almost immediately: add more servers or split into microservices. Both can work, and both can save products. But both can also become expensive distractions when nobody has actually found the bottleneck. My approach is boring on purpose: measure first, then fix the cheapest constraint with the highest ROI.
The painful truth about scaling
A lot of people think scaling means handling more requests. That is only half the story. Real scaling means growing without turning the system into a stress machine, while keeping:
Low latency (especially p95/p99, not average)
Stability under traffic spikes and failures
Predictable cost (not we scaled and the bill doubled overnight)
Architecture maintainable (so every change does not feel like surgery)
My pragmatic order (the stuff I try before big moves)
Cache hot endpoints (carefully, not blindly)
Move heavy work to queues (stop expensive work inside requests)
Fix the database reality (queries and indexes)
Add minimum viable observability (so we stop guessing)
1) Cache hot endpoints (the right way, not the YOLO cache)
Hot endpoints are the routes everyone hits constantly: dashboards, summaries, profile data, and high-frequency listings. Caching these endpoints is often the best ROI in performance engineering, but it can also ship wrong data if done lazily.
Why caching gives fast ROI
Reduces database pressure
Cuts response latency
Absorbs traffic spikes without multiplying infrastructure
Practical cache key model (examples)
user:123:dashboard:v1 (short TTL like 60-300s)
tenant:45:stats:v2 (multi-tenant summaries)
tenant:45:orders:status=paid:page=2:sort=created_at_desc:v1
Where caching usually fails
One key fits all designs that ignore filters, pagination, and sorting
Global cache when data must be user or tenant scoped
No expiration and no invalidation strategy
Caching because we heard it is good instead of because we can explain it
If I cannot explain key format, TTL, and invalidation in one minute, I do not ship caching just to try.
2) Move heavy work to queues (stop doing expensive things inside requests)
A lot of products feel slow because the API is trying to do too much right now. The request-response path should be fast, predictable, and boring.
What goes to queue immediately
Email and SMS sending
PDF and report generation
Image processing
Slow external integrations
Webhooks with retries
Anything that risks timeouts under peak load
How product UX improves
The API shifts from I will do everything now to accepted and processing, with clearer status tracking. Users feel fewer timeouts and less randomness.
Classic danger: duplicate side effects
Jobs can retry, and retries can duplicate effects like billing, emails, or updates. Idempotency and queue backlog monitoring are not optional.
3) Optimize queries and indexes (DB is guilty until proven innocent)
In many systems, the app layer is not the bottleneck. The database is. Adding app servers often adds more concurrent pressure on the database.
Clear signs the DB is the constraint
Endpoint stays slow even after scaling app instances
High DB CPU or screaming disk IO
Slow query logs repeat the same offenders
N+1 query patterns
Performance degrades as tables grow
Lock waits, deadlocks, or connection pool saturation
What I do in practice (not fancy, just effective)
Identify highest-cost queries via APM or slow logs
Fix N+1 with proper eager loading or joins
Avoid SELECT * when not needed
Create indexes aligned with WHERE, JOIN, and ORDER BY
Validate plans with EXPLAIN and re-check after changes
Common traps
Too many indexes hurting writes and storage
Poor composite index order
Optimizing the wrong query because nobody measured
Tuning without baseline metrics (expensive guesswork)
Optimization without measurement is expensive guesswork.
4) Metrics and alerts (because guessing does not scale)
Without measurement, there is no control. Without alerts, you find out about incidents from customers.
The metrics I care about
p95/p99 latency per endpoint
Error rate (4xx/5xx)
Throughput (req/s)
DB query time and connection usage
Queue size and queue wait time
CPU and memory saturation
Cost anomaly signals (FinOps)
The alerts that actually help
p95/p99 above threshold for sustained windows
5xx spikes
Queue backlog growing
DB connections near limit
Cost anomaly alerts
I avoid CPU-only alerting. CPU is often a symptom, not the cause. A system can look fine on CPU while the DB is locked and everything times out.
Practical example: Microservices will fix it until metrics said otherwise
I have seen teams propose microservices because the system felt slow and unstable. The story was: it is too big, we need to split.
After measuring first, we found:
A dashboard endpoint recalculated expensive stats on every request
Missing indexes on the most-used filters
Report generation happening in the request path
No visibility into p95/p99 or queue health
DB connections spiking under traffic bursts
The pragmatic plan before any rewrite
Cache hot dashboard endpoints with tenant-scoped keys and short TTL
Move report generation to a queue (async with status tracking)
Add the right indexes and fix N+1 patterns
Add p95/p99, queue backlog, and DB connection alerts
Performance improved, stability improved, and cost became predictable without premature operational complexity.
And microservices? When do they become a loss?
I like microservices when context demands them. They become expensive when used too early, especially when:
Domain boundaries are still changing and likely to be split wrong
The team lacks mature CI/CD and observability
Services over-communicate (distributed monolith)
Operational overhead explodes (deployments, tracing, incidents, on-call fatigue)
In many cases, a well-designed modular monolith gives better ROI: clear boundaries, simpler deployment, and gradual evolution without distributed coordination overhead.
Conclusion
Before scaling infrastructure or splitting into services, I look for quick ROI in fundamentals.
I prioritize:
Cache where it clearly helps
Queues for heavy async work
Database optimization
Minimum viable observability
These steps solve a huge chunk of scaling pain and make the next architecture decision clearer, because now you are acting on evidence, not vibes.
Pragmatic next step
If you are dealing with rising cloud costs, slow APIs, unstable traffic spikes, or uncertainty between modular monolith and microservices, the first win is usually the same: measure, find the real bottleneck, and fix the highest-ROI constraint first.
Rising costs
Slow API
Spikes causing instability
Architecture uncertainty
I can deliver
Evidence-based bottleneck map
Quick wins in 1-2 weeks
Phased evolution plan in 1-2 months