HandyManny Documentation

Appearance

Rollback Procedures ​

This SOP covers rollback procedures for reverting failed deployments and recovering from production incidents.

When to Rollback ​

Rollback immediately if:

  • ❌ Application won't start after deployment
  • ❌ Critical functionality broken
  • ❌ Database migration failed
  • ❌ Security vulnerability introduced
  • ❌ Performance degradation >50%
  • ❌ User-facing errors affecting >10% of traffic

Investigate before rollback if:

  • ⚠️ Minor UI issues (can be hotfixed)
  • ⚠️ Non-critical feature not working
  • ⚠️ Configuration issue (might be fixable)
  • ⚠️ Performance degradation <20%

Quick Rollback (Git Revert) ​

Fastest method - use when:

  • No database migration involved
  • Simple code change
  • Container won't start
bash
# 1. SSH to VPS
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud

# 2. Navigate to project
cd /root/[project-name]

# 3. Check git log to find last working commit
git log --oneline -10

# 4. Revert to previous commit
git reset --hard [commit-hash]

# Example:
# git reset --hard a1b2c3d

# 5. Rebuild containers
docker compose down
docker compose up -d --build

# 6. Verify rollback
docker ps
docker logs -f [container-name]

Database Rollback (Prisma) ​

For Prisma migrations that need reverting:

Method 1: Rollback Migration ​

bash
# 1. SSH to VPS and enter container
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
docker exec -it [app-container] sh

# 2. Check migration status
npx prisma migrate status

# 3. Revert to specific migration
npx prisma migrate resolve --rolled-back [migration-name]

# 4. Reapply previous migration
npx prisma migrate deploy

# 5. Verify schema
npx prisma db pull

Method 2: Restore Database Backup ​

bash
# 1. Stop application containers
docker stop [app-container]

# 2. Drop current database and restore backup
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
docker exec [db-container] dropdb -U [user] [dbname]
docker exec -i [db-container] psql -U [user] -d postgres -c "CREATE DATABASE [dbname];"
docker exec -i [db-container] psql -U [user] -d [dbname] < /root/backups/[backup-file].sql

# 3. Restart application
docker start [app-container]

Database Rollback (SQLAlchemy/Alembic) ​

For Alembic migrations:

bash
# 1. SSH to VPS
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud

# 2. Enter application container
docker exec -it [container] sh

# 3. Check migration history
alembic history

# 4. Downgrade to previous revision
alembic downgrade -1  # go back one migration
# OR
alembic downgrade [revision-id]  # go to specific revision

# 5. Verify schema
alembic current

Docker Image Rollback ​

If new image is broken but previous image still exists:

bash
# 1. List available images
docker images | grep [project-name]

# 2. Find previous image tag
# Example output:
# expediente    latest    abc123   2 hours ago
# expediente    <none>    def456   1 day ago

# 3. Tag previous image as latest
docker tag def456 [project-name]:latest

# 4. Restart container with previous image
docker compose down
docker compose up -d

# 5. Verify
docker ps

Full Environment Rollback ​

Complete rollback including code, database, and environment:

1. Prepare Rollback ​

bash
# Identify target state
git log --oneline -20
git show [commit-hash]  # verify this is the working version

2. Stop Services ​

bash
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
cd /root/[project]

# Stop containers
docker compose down

3. Rollback Code ​

bash
# Revert to working commit
git reset --hard [commit-hash]

# Verify correct version
git log -1

4. Rollback Database (if needed) ​

Option A: Restore from backup

bash
# Restore PostgreSQL backup
docker exec -i [db-container] psql -U [user] -d postgres -c "DROP DATABASE [dbname];"
docker exec -i [db-container] psql -U [user] -d postgres -c "CREATE DATABASE [dbname];"
docker exec -i [db-container] psql -U [user] -d [dbname] < /root/backups/[backup-file].sql

Option B: Downgrade migration

bash
# For Prisma
docker exec [container] npx prisma migrate resolve --rolled-back [migration]

# For Alembic
docker exec [container] alembic downgrade -1

5. Restore Environment Variables (if changed) ​

bash
# Restore .env from backup or git
git show [commit-hash]:.env > .env

# OR manually edit
nano .env

6. Restart Services ​

bash
# Rebuild and start
docker compose up -d --build

# Monitor startup
docker compose logs -f

7. Verify Rollback ​

bash
# Check containers
docker ps

# Check logs
docker logs [container-name]

# Test application
curl -I https://[domain]

# Visual verification in browser

Partial Rollback (Hotfix) ​

When only specific files need reverting:

bash
# 1. SSH to VPS
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
cd /root/[project]

# 2. Checkout specific files from previous commit
git checkout [commit-hash] -- path/to/file.ts
git checkout [commit-hash] -- path/to/another/file.tsx

# 3. Verify changes
git diff

# 4. Rebuild
docker compose down
docker compose up -d --build

# 5. If successful, commit the partial revert
git add -A
git commit -m "hotfix: revert specific files from failed deployment"

Emergency Rollback Checklist ​

When rolling back in an emergency:

  • [ ] Notify team - Alert team members of rollback
  • [ ] Document issue - Screenshot errors, save logs
  • [ ] Stop deployment - Cancel any in-progress deploys
  • [ ] Verify backup available - Check database backup exists
  • [ ] Execute rollback - Follow procedure above
  • [ ] Test thoroughly - Verify rollback successful
  • [ ] Monitor metrics - Watch error rates, performance
  • [ ] Post-mortem - Document what went wrong and why
  • [ ] Fix forward - Address issue before next deploy

Backup Verification ​

Before any risky deployment, verify backups exist:

bash
# List database backups
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud 'ls -lh /root/backups/ | grep [dbname]'

# Verify backup is recent (within 24 hours)
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud 'find /root/backups/ -name "[dbname]-*.sql" -mtime -1'

# Test backup restore (to temporary database)
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
docker exec -i [db-container] psql -U [user] -d postgres -c "CREATE DATABASE test_restore;"
docker exec -i [db-container] psql -U [user] -d test_restore < /root/backups/[latest-backup].sql
docker exec [db-container] psql -U [user] -d test_restore -c "\dt"  # list tables
docker exec [db-container] psql -U [user] -d postgres -c "DROP DATABASE test_restore;"

Creating Backups Before Deployment ​

Always backup before risky deployments:

PostgreSQL Backup ​

bash
# Manual backup
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud

# Create timestamped backup
docker exec [db-container] pg_dump -U [user] [dbname] > /root/backups/[dbname]-$(date +%Y%m%d-%H%M%S).sql

# Verify backup created
ls -lh /root/backups/

# Keep only last 10 backups
cd /root/backups && ls -t [dbname]-*.sql | tail -n +11 | xargs rm -f

Application Files Backup ​

bash
# Backup entire application directory
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
tar -czf /root/backups/[project]-$(date +%Y%m%d-%H%M%S).tar.gz /root/[project]/

# Verify backup
ls -lh /root/backups/*.tar.gz

Docker Image Backup ​

bash
# Save current Docker image before rebuilding
docker tag [image]:latest [image]:backup-$(date +%Y%m%d-%H%M%S)

# Verify tagged
docker images | grep [image]

Rollback Testing ​

Test rollback procedures regularly:

bash
# 1. In staging environment, make a change
git commit -m "test change for rollback test"
docker compose down && docker compose up -d --build

# 2. Verify new version running
curl https://staging.[domain]

# 3. Practice rollback
git reset --hard HEAD~1
docker compose down && docker compose up -d --build

# 4. Verify rollback successful
curl https://staging.[domain]

Common Rollback Scenarios ​

Scenario 1: Deployment Broke Login ​

Problem: Users can't login after deployment

Rollback:

bash
# Quick git revert
ssh -i ~/.ssh/id_hostinger root@srv1139900.hstgr.cloud
cd /root/[project]
git reset --hard HEAD~1
docker compose down && docker compose up -d --build

Verify:

bash
# Test login endpoint
curl -X POST https://[domain]/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"test"}'

Scenario 2: Database Migration Corrupted Data ​

Problem: Migration ran but data is now corrupted

Rollback:

bash
# 1. Stop app
docker stop [app-container]

# 2. Restore database from backup
docker exec [db-container] dropdb -U [user] [dbname]
docker exec [db-container] createdb -U [user] [dbname]
docker exec -i [db-container] psql -U [user] -d [dbname] < /root/backups/[latest-backup].sql

# 3. Revert code
cd /root/[project]
git reset --hard [pre-migration-commit]

# 4. Restart app
docker start [app-container]

Scenario 3: Environment Variables Misconfigured ​

Problem: Forgot to update .env, app connecting to wrong services

Rollback:

bash
# 1. Restore .env from previous commit
git show HEAD~1:.env > .env

# 2. Restart containers (MUST restart for env changes)
docker compose down
docker compose up -d

# 3. Verify env vars
docker exec [container] env | grep -E "(DATABASE|API|SECRET)"

Scenario 4: Docker Build Failed ​

Problem: New Dockerfile breaks build

Rollback:

bash
# Revert Dockerfile
git checkout HEAD~1 -- Dockerfile
git checkout HEAD~1 -- docker-compose.yml

# Rebuild
docker compose down
docker compose up -d --build

Post-Rollback Actions ​

After successful rollback:

  1. Verify application health:

    • Test critical user flows
    • Check error monitoring (no new errors)
    • Monitor performance metrics

  2. Investigate root cause:

    • Review deployment logs
    • Identify what went wrong
    • Document in post-mortem

  3. Fix the issue:

    • Create bugfix branch
    • Fix the problem locally
    • Test thoroughly
    • Deploy fix when ready

  4. Update documentation:

    • Document what failed
    • Update deployment checklist
    • Add new test case

  5. Notify stakeholders:

    • Update team on status
    • Communicate fix timeline
    • Document lessons learned

Preventing Rollbacks ​

Best practices to avoid needing rollbacks:

  1. ✅ Test in staging first - Always deploy to staging before production
  2. ✅ Backup before deploy - Create database backup before risky changes
  3. ✅ Deploy during low traffic - Minimize user impact if rollback needed
  4. ✅ Use feature flags - Disable new features without code rollback
  5. ✅ Monitor closely - Watch logs/metrics immediately after deploy
  6. ✅ Incremental changes - Small, frequent deploys vs. large batches
  7. ✅ Database migrations - Test migrations thoroughly, use transactions
  8. ✅ Automated tests - Run test suite before deploying
  9. ✅ Rollback plan - Know rollback procedure before deploying
  10. ✅ Team availability - Don't deploy if team unavailable to respond

Last Updated: 2026-02-28 Related SOPs: Deployment SOP, Security Guide

HandyManny Documentation System

Copyright © 2026