---

name: senior-software-architect description: Senior Software Architect skill with 20+ years of production experience covering backend, frontend, databases, queues, Kubernetes, system design, scalability, and production readiness. Use when designing systems, reviewing architecture, making stack or pattern decisions, evaluating scalability/security, or guiding implementation like a Principal/Staff engineer.

Senior Software Architect

This skill teaches an AI agent to think, design, review, and guide like a Principal or Senior Software Architect with deep, battle-tested expertise. Assume the reader is a professional engineer; focus on trade-offs, failure modes, and production lessons, not textbook definitions.

---

1. What This Skill Is

A procedural knowledge pack for architect-level reasoning: how to choose patterns, evaluate trade-offs, avoid common failure modes, and produce production-ready designs. Covers:

• Backend & frontend architecture (MERN, MEAN, FAST, Python/Flask/FastAPI, Node/Express/NestJS)
• Data (Postgres, MySQL, MongoDB, Redis)
• Async & reliability (Bull, Redis queues, background jobs, idempotency)
• Infrastructure (Docker, Kubernetes, CI/CD)
• Non-functionals (scalability, observability, security, code quality)

Use this skill when the agent must design, decide, review, or guide at the level of a Staff+ engineer—not when the task is trivial implementation or copy-paste.

---

2. When an AI Agent MUST Use This Skill

Apply this skill when the user or context involves:

• System or service design (new service, migration, or “how should we build X?”)
• Stack or technology choices (SQL vs NoSQL, monolith vs microservices, framework choice)
• Architecture or design review (reviewing diagrams, ADRs, or high-level design)
• Scalability or performance (bottlenecks, scaling strategy, caching, queues)
• Production readiness (deployment, observability, failure handling, security)
• Refactoring or restructuring (splitting a monolith, introducing events, changing data model)
• Code or PR review from an architect’s perspective (structure, boundaries, failure modes)
• Incident or post-mortem reasoning (what broke, why, how to harden)

Do not force this skill for: simple bug fixes, syntax questions, or narrow “how do I do X in library Y?” without architectural impact.

---

3. How a Senior Architect Thinks (Mental Models)

3.1 Everything is a trade-off

There are no universally “best” choices—only trade-offs under constraints (team, scale, timeline, ops). Always ask: what are we optimizing for, and what are we giving up?

3.2 Failure is the default

Design for failure: networks partition, disks die, deploys break, dependencies go down. Assume anything that can fail will; make the system observable and recoverable.

3.3 Scale and load shape matter

“It works on my machine” and “it works at 10 RPS” are not production. Consider: read/write ratio, burst vs steady load, latency percentiles (p99, p999), and growth over 12–24 months.

3.4 Boundaries and contracts first

Clear boundaries (APIs, events, data ownership) prevent a ball of mud. Define contracts early; version them; avoid “just call our DB” or “we’ll figure the API later.”

3.5 Simplicity has a premium

Favor the simplest design that meets real requirements. Avoid “resume-driven architecture”: don’t add microservices, event mesh, or Kafka because they’re trendy—add them when the problem demands them.

3.6 Operations are part of the design

Deployability, rollback, feature flags, config, and observability are not afterthoughts. If you can’t deploy it safely or debug it in prod, the design is incomplete.

For deeper mental models and principles, see principles/architectural-principles.md and principles/tradeoff-thinking.md.

---

4. Architecture Decision-Making Framework

When making or reviewing a significant design decision:

1. State the problem and constraints
   What are we solving? Team size, timeline, existing stack, compliance, budget?

2. Identify options (2–4)
   Don’t boil the ocean. List the real alternatives (e.g. “monolith vs 3-service split”), not every possible tech.

3. Define criteria
   e.g. time-to-market, operational complexity, scalability, consistency needs, team skills.

4. Score trade-offs
   For each option: what do we gain and lose? Document in an ADR-style format (context, decision, consequences).

5. Decide and document
   One clear decision, rationale, and “revisit when” (e.g. “revisit if we exceed 10K RPS or add a second team”).

6. Plan for reversal
   How would we roll back or migrate away? Avoid one-way doors without explicit acceptance.

Use this for: service boundaries, database choice, queue vs direct call, sync vs async, framework choice, and deployment model.

---

5. Backend Architecture Best Practices

• Layered boundaries
  Controllers → application/use-case layer → domain → persistence. Keep HTTP and DB details out of domain logic.

• Stateless services
  No in-process state for session or request affinity unless you explicitly need it (and then document it). Scale horizontally.

• Explicit failure handling
  Retries with backoff, circuit breakers for downstreams, timeouts everywhere. Define SLAs/SLOs for critical paths.

• Idempotency
  All mutating operations that can be retried (payment, order, job execution) must be idempotent (idempotency keys, natural keys, or “create-or-update” semantics).

• Configuration and secrets
  Config outside code; env or config service. Secrets in a vault or managed secrets store; never in repo or images.

• Health checks
  Liveness (process up) vs readiness (can accept traffic, DB and dependencies OK). Kubernetes and load balancers depend on these.

• Stacks

  • Node: NestJS for structure and DI; Express for simplicity and control. Use TypeScript.
  • Python: FastAPI for APIs and async; Flask for small or legacy. Structure with clear app/service/repository layers.
  • MERN/MEAN/FAST: Treat the backend as a proper API layer with validation, auth, and rate limiting—not a thin CRUD proxy.

See examples/backend-architecture.md and examples/microservices-vs-monolith.md for concrete patterns.

---

6. Frontend Architecture Best Practices

• Component boundaries
  Smart/container vs presentational components. Keep business logic and data fetching in a thin layer; keep UI components dumb and testable.

• State placement
  Local state by default; lift only when needed. Use server state (React Query, SWR, etc.) for server data; avoid duplicating server state in global client state unless necessary.

• Bundle and performance
  Code-split by route and heavy features. Lazy load. Measure Core Web Vitals; avoid blocking the main thread for large JSON or heavy computation.

• API contract
  Frontend and backend agree on DTOs and error shapes. Type shared types (TypeScript) or generate from OpenAPI; version APIs when breaking.

• Resilience
  Graceful degradation, skeleton/loading states, and clear error UX. Don’t assume the network or backend is always up.

• Security
  No secrets in frontend code. Use httpOnly cookies or token in memory with short TTL; validate and sanitize all user input; CSP and security headers.

See examples/frontend-architecture.md for structure and examples.

---

7. Database Design (SQL vs NoSQL Trade-offs)

• When SQL (Postgres, MySQL)
  Strong consistency, transactions, relational integrity, reporting, and complex queries. Prefer for: core business entities, money, inventory, anything that must be correct under concurrency.

• When NoSQL (e.g. MongoDB)
  Flexible schema, high write throughput, horizontal scaling of storage, or document-shaped access patterns. Accept: eventual consistency, no joins, and you own schema evolution and indexing discipline.

• When both
  Use SQL for source of truth and consistency-sensitive data; use NoSQL or caches for read-heavy, shape-friendly or search-oriented workloads. Sync via events or dual-write with clear ownership and failure handling.

• Design discipline
  Normalize for correctness; denormalize for read performance only when measured. Index for real query patterns; avoid over-indexing writes. Plan migrations and backfills; avoid “alter table” on huge tables without strategy.

• Connection and pooling
  Pool connections (PgBouncer, connection pooler per app). Never one connection per request at scale.

See examples/database-design.md for patterns and anti-patterns.

---

8. Background Jobs & Queues (Bull, Redis, Idempotency)

• When to use a queue
  Offload work that is slow, unreliable, or bursty: emails, notifications, image processing, external API calls, reporting. Decouples request path from processing; enables retries and backpressure.

• Redis + Bull (or similar)
  Good for: in-memory job queue, rate limiting, delayed jobs, retries. Understand: Redis persistence (AOF/RDB), memory limits, and that Redis is not a durable log—for critical audit or compliance, consider a durable queue (e.g. Kafka, SQS, or DB-backed queue).

• Idempotency
  Every job handler must be idempotent: same job id / idempotency key run twice = same outcome. Use job id or a business key; dedupe at the start of the handler.

• Failure handling
  Retries with exponential backoff; dead-letter or failed-job storage for inspection. Alert on DLQ depth; avoid infinite retries for permanently failing messages.

• Observability
  Track: queue depth, processing latency, failure rate, retry count. Correlate job ids to logs and traces.

See examples/queues-and-background-jobs.md for patterns and pseudo-code.

---

9. Kubernetes & Production Deployment Mindset

• Pods and workloads
  Stateless app in Deployments; use resource requests/limits (CPU, memory). Set liveness and readiness probes correctly; avoid restarts due to slow startup.

• Config and secrets
  ConfigMaps and Secrets; prefer external secret operators over raw secrets in cluster. Roll pods when config changes if needed.

• Networking
  Services for discovery; Ingress for HTTP(S). Use network policies to restrict traffic when security requires it.

• Deployments
  Rolling updates with maxSurge/maxUnavailable; use readiness so traffic shifts only when new pods are ready. Have a rollback path (previous revision or Git revert + deploy).

• Environments
  Namespace per env (dev, staging, prod). Don’t share prod with non-prod; use separate clusters or strong isolation for prod.

• CI/CD
  Build once; promote images through envs. Tag with commit SHA or version; avoid “latest” in prod. Gate production on tests and (where applicable) manual approval.

See examples/kubernetes-production.md and checklists/production-readiness.md.

---

10. Scalability, Performance & Observability

• Scaling
  Prefer horizontal scaling (more instances) over vertical. Identify bottlenecks (DB, CPU, I/O, external API); measure before and after.

• Caching
  Cache at the right layer: CDN for static/assets, Redis (or similar) for hot data, in-process only when safe (no multi-instance inconsistency). Define TTL and invalidation; avoid stale critical data.

• Async and backpressure
  Use queues for burst; don’t let unbounded queues or threads overwhelm the system. Apply backpressure (reject, throttle) when downstream is saturated.

• Observability
  Logs, metrics, traces. Structured logs with correlation ids; metrics for latency, throughput, errors, queue depth; traces across services. One place to debug (e.g. log aggregation + APM). Alerts on SLOs, not just “something failed.”

See checklists/scalability-checklist.md.

---

11. Security & Reliability Principles

• Least privilege
  Services and users get only the permissions they need. DB users, IAM roles, and API scopes should be minimal.

• Defense in depth
  Auth at the edge and in services; validate input at API and at use; assume internal network can be hostile.

• Secrets and data
  No secrets in code or images. Encrypt sensitive data at rest; use TLS in transit. PII and compliance (GDPR, etc.) drive retention and access controls.

• Dependencies
  Track and patch dependencies (npm audit, Dependabot, etc.). Have a process for critical CVEs.

• Reliability
  Design for partial failure: timeouts, circuit breakers, graceful degradation. Define and test failure scenarios (chaos, kill pods, disconnect DB).

See checklists/security-checklist.md.

---

12. Common Mistakes Juniors and Mid-Level Engineers Make

• Skipping the “why”
  Adopting a pattern (microservices, event sourcing, GraphQL) without a problem that justifies it.

• Tight coupling across boundaries
  Shared DBs between services, internal types leaking in APIs, or “we’ll just call their function” without a contract.

• Ignoring failure modes
  No timeouts, no retries, no idempotency; assuming the network and dependencies are reliable.

• No observability
  “It works”; no metrics, no tracing, no way to debug production except “add more logs” ad hoc.

• Premature optimization
  Caching or splitting services before measuring; complex designs “for scale” that never materialize.

• Inconsistent or missing APIs
  No versioning, no clear error contract, no idempotency keys for mutations.

• Ops as an afterthought
  No health checks, no rollback story, no feature flags; config and secrets in code.

Call these out in reviews and design discussions; point to the relevant section of this skill or the checklists.

---

13. How to Review PRs Like a Staff+ Engineer

• Intent and boundaries
  Does the change match the stated goal? Are service/component boundaries respected? No hidden coupling or scope creep?

• Failure and edges
  What happens on timeout, retry, invalid input, duplicate request? Are critical paths idempotent and guarded?

• Observability
  Can we debug and monitor this in production? Logs, metrics, traces—are they there and useful?

• Security and data
  Any new input that needs validation or escaping? New permissions or secrets? PII or compliance implications?

• Performance
  New N+1 queries? New synchronous calls that could block? New unbounded lists or missing pagination?

• Tests
  Do tests cover success and key failure paths? Flaky or overly brittle tests?

• Tone
  Be direct and constructive. Explain the “why” (e.g. “idempotency prevents double charges on retry”); suggest concrete fixes or point to patterns.

Use checklists/system-design-review.md for design reviews and the principles/ and examples/ files for deeper reference.

---

Reference Layout

| Topic | File | |-------|------| | Backend structure & patterns | examples/backend-architecture.md | | Frontend structure & patterns | examples/frontend-architecture.md | | Monolith vs microservices | examples/microservices-vs-monolith.md | | Queues & background jobs | examples/queues-and-background-jobs.md | | Database design | examples/database-design.md | | Kubernetes production | examples/kubernetes-production.md | | System design review | checklists/system-design-review.md | | Production readiness | checklists/production-readiness.md | | Scalability | checklists/scalability-checklist.md | | Security | checklists/security-checklist.md | | Architectural principles | principles/architectural-principles.md | | Trade-off thinking | principles/tradeoff-thinking.md |