Most CI/CD advice is written by people running platform teams of thirty. If you are two to eight engineers shipping a product, you need a pipeline that a single person can hold in their head, debug at 11pm, and change without a planning meeting. The version below is roughly what we set up on every project we take through to production, and it fits in one workflow file plus a deploy script.
The single number that matters most: total feedback on a pull request should land under eight minutes. Past that, people open a second PR while waiting, stop reading the output, and start merging on vibes.
The stages, and what each one is allowed to block
Order matters because the cheap checks should fail first.
- Lint and typecheck: under 90 seconds. Blocks the merge.
eslint,tsc --noEmit,ruff,mypy. If this creeps past two minutes, split it or cache harder. - Unit tests: 2 to 3 minutes. Blocks. No network, no database, no sleep calls.
- Build: produces the artifact you will actually deploy. Blocks. Build once, reuse downstream, never rebuild at deploy time.
- Integration tests: against a real Postgres service container, not a mock or SQLite. 3 to 4 minutes. Blocks. This is where migration mistakes and query bugs surface.
- Deploy to preview: per-PR environment. Does not block the merge on infra failures, but it pages whoever owns the pipeline.
- Smoke test: 5 to 10 requests against the preview URL: health, login, one authenticated read, one write, one Stripe webhook replay. Blocks promotion, not the PR.
- Promote: on merge to main, the same artifact moves to production behind a health gate.
Everything else (visual diffs, load tests, dependency audits) runs nightly and reports rather than blocks. A flaky check that blocks merges is worse than no check, because the team learns to re-run until green.
The workflow file
Three things do most of the heavy lifting here: a concurrency group so pushing three times in a row does not run three full pipelines, a dependency cache, and a Postgres service container.
name: ci
on:
pull_request:
push:
branches: [main]
concurrency:
group: ci-${{ github.workflow }}-${{ github.head_ref || github.sha }}
cancel-in-progress: true
jobs:
integration:
runs-on: ubuntu-latest
timeout-minutes: 12
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app_test
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s --health-timeout 5s --health-retries 10
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/app_test
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- run: pip install -r requirements-dev.txt
- run: alembic upgrade head
- run: pytest -q tests/integration --maxfail=1
Keep the pipeline definition in the repo, next to the code it builds. Configuration living in a web console is configuration nobody can review, and it will drift.
Preview environments without the cloud bill
A per-PR URL is the highest-value thing you can add for a client project. Reviewers stop asking for screenshots and start clicking.
The trap is databases. One managed Postgres instance per PR gets expensive fast and takes 90 seconds to provision. Use one Postgres instance and a schema per PR instead:
-- provisioned at preview deploy time
CREATE SCHEMA IF NOT EXISTS pr_1482;
-- app connects with: ?options=-csearch_path%3Dpr_1482
-- teardown on merge/close:
DROP SCHEMA pr_1482 CASCADE;
Seed it from a fixture set, not a production dump. We keep a seed_demo.py that creates 3 orgs, 12 users across the role matrix, and a handful of paid and unpaid subscriptions so Stripe flows are exercisable. It runs in about four seconds.
Then actually tear it down. Wire a job to the pull_request: [closed] event that drops the schema and destroys the preview service. Skip this and you will find 60 idle environments in month three, all billing.
Migrations that cannot take the site down
Migrations run as a separate pipeline step, before the new app version rolls out, and must stay backward compatible with the currently running code. That constraint is what makes a rollback safe.
Use expand/contract, always:
- Add the new nullable column. Deploy.
- Backfill in batches, not one statement.
- Dual-write to old and new columns. Deploy.
- Switch reads to the new column. Deploy.
- Drop the old column, weeks later, once nothing references it.
Never rename a column in one deploy. A rename is a drop plus an add wearing a costume, and it breaks every process still running the old code during the rollout window.
Guard every migration with timeouts so a lock cannot stall a busy table:
SET lock_timeout = '3s';
SET statement_timeout = '30s';
ALTER TABLE orders ADD COLUMN fulfillment_status text;
CREATE INDEX CONCURRENTLY idx_orders_fulfillment
ON orders (fulfillment_status);
Without lock_timeout, an ALTER TABLE waiting behind a long read queues every subsequent query on that table. We have watched a 40 millisecond migration hold an exclusive lock for four minutes because it got stuck behind an analytics query, and the whole API timed out. Three seconds and a retry is the fix.
Have a tested down path, or an explicit decision that the fix is forward-only. Untested downgrade() functions are decorative.
Secrets
No secrets in the repo, ever, including .env.example files that quietly grew real values. Use OIDC federation from CI to the cloud provider so the pipeline exchanges a short-lived token instead of holding a long-lived access key. Scope secrets per environment: the preview deploy job should not be able to read the production database password. Rotate quarterly at minimum, and immediately when someone leaves.
If a key leaks, the order is rotate, then investigate. Revoke the credential, deploy the new one, confirm the service is healthy, and only then read logs to work out the blast radius. Teams that investigate first spend 40 minutes building a timeline while the key is still live.
Rollback in under two minutes
Four rules make this boring:
- Immutable artifacts, tagged by commit SHA.
app:9f2c1ab, neverapp:latest. - Deploy is a pointer flip. You are changing which already-built image the platform serves, not building anything.
- Health-gated rollout. New instances take traffic only after passing a real readiness check; a failing rollout reverts on its own.
- A rollback must never require a database rollback. If reverting the app needs a schema change too, the migration was not backward compatible. Go back to expand/contract.
Feature flags separate deploy from release. Ship the code dark, turn it on for internal accounts, then a percentage, then everyone. Turning off a flag takes seconds and needs no pipeline run.
You probably should not run Kubernetes yet
Kubernetes is excellent and it is not free. The bill is paid in attention: control plane upgrades every few months, ingress controller behavior, cert rotation, node pool sizing, resource requests nobody tuned, and one person carrying a pager for the cluster instead of the product. On a team of five that is 15 to 20 percent of an engineer, permanently.
Use a managed platform, a container service like ECS Fargate or Cloud Run, or two VMs behind a load balancer with systemd and a deploy script. All of them do rolling deploys, health checks, and rollbacks.
Honest signals you have outgrown that: roughly a dozen or more services with real inter-service traffic, multiple teams needing independent deploy cadence, genuine multi-region requirements, or a compute bill high enough that better bin-packing pays a salary. Wanting it on the resume is not a signal.
How we run this
On NorthStackHub projects the pipeline goes in during the first week, before there is much to deploy. Required status checks on main, a preview URL on every PR, migrations as their own gated step, artifacts tagged by SHA. By the time a client sees a staging build, we have rehearsed a rollback at least once on purpose. When we hand a project over or move into a maintenance retainer, the workflow file and the deploy script are the documentation, and a developer who has never seen the repo can read both in ten minutes.