What you will achieve

You will roll back a problematic release - first by reverting commits to undo the change in the working history, then by re-tagging an earlier version for re-deployment. By the end, you will be able to handle a "we need to undo v2.4.0 right now" incident calmly.

Two scenarios

• Roll back deployment - the production environment runs an older version while you fix forward.
• Roll back code - the commits themselves are reverted in main.

Scenario 1: deploy an older tag

Fastest path to safety. Identify the last good tag:

git tag --sort=-v:refname | head
# v2.4.0
# v2.3.5
# v2.3.4

Re-deploy v2.3.5 via your normal pipeline:

# Trigger the same pipeline that built v2.3.5
gh workflow run deploy.yml -f tag=v2.3.5

Verify in production. Now you have time to investigate.

Scenario 2: revert the commits

To undo the changes that caused the regression in main:

git checkout main
git pull
git revert <bad-sha>
git push

revert creates a new commit that undoes the targeted commit. History is preserved.

Reverting a merge commit

If the bad release came from merging a feature, revert the merge:

git log --merges -10
# find the merge sha
git revert -m 1 <merge-sha>
git push

-m 1 tells revert to undo by reapplying main's pre-merge state.

Reverting a range

git revert <older-sha>..<newer-sha>

This reverts each commit in the range as separate commits. Add -n to combine into one revert commit you stage and edit.

Tagging the rollback

Annotate the rollback for traceability:

git tag -a v2.4.1 -m "Roll back v2.4.0 due to checkout regression"
git push origin v2.4.1

Now v2.4.1 is the new "current" release, with the regression undone.

The forward-fix anti-pattern

Tempting but often wrong: "we will just fix forward". This works for small bugs but fails when:

• The fix takes hours and customers are impacted now.
• The bug is data-corrupting; every minute makes it worse.
• The fix needs careful testing.

Roll back first, fix forward at leisure.

Database migrations and rollback

Code rollback is easy; database rollback is hard. Best practices:

• Migrations should be additive (add columns, don't drop).
• Avoid breaking changes in migrations attached to risky releases.
• Have a tested down-migration path.
• Use feature flags to gate code paths that depend on schema changes.

Coordinating with the team

• Announce the rollback in the same channel as releases.
• Document the SHA being rolled back and why.
• Open a tracking issue for the forward fix.
• Schedule a post-mortem.

Avoiding rollbacks

• Canary releases - ship to 1% of traffic first.
• Feature flags - kill switches for new code paths.
• Pre-release validation - staging tests, smoke tests, automated rollback triggers.
• Smaller releases - five small releases per day fail less catastrophically than one weekly mega-release.

Re-applying after fix

If you reverted the commits and later want to bring them back fixed, revert the revert:

git revert <revert-sha>
# then add the fix on top
git commit -am "Fix the issue that caused the original revert"

Or rebase the original feature branch and re-merge.

The result

Rollbacks are routine, not catastrophic. The drill is: deploy older tag, revert offending commits, retag, communicate, post-mortem. Practise the drill on a staging environment before you need it in production.