Post-Mortem: A Disastrous Git Merge and the Resulting Workflow

Incident Summary

Date: 2025-03-14 Duration of impact: ~6 hours Commits lost: 48 across 3 contributors Recovery time: 12 engineering-hours Root cause: Unprotected main branch combined with git push --force

This write-up exists because the team agreed to document what happened, and because the details kept rattling around in my head and needed to go somewhere other than 3 AM anxiety spirals.

The Sequence of Events

Friday. Always a Friday.

One of our junior developers was stuck on a merge conflict inside a long-running refactoring branch. The kind of conflict where Git marks up half the file and the diff markers make nothing look right no matter how you read them. He'd been working at it for a while. Tried resolving manually. Got confused. Tried again. Eventually, through a sequence of steps I still don't fully understand, he ran git push origin main --force.

I've asked him about the exact command chain a couple of times. The answer is always some variation of "I was trying things from Stack Overflow." Which — yeah. Pasting commands you don't fully understand into a terminal at 4 PM on a Friday. I won't pretend I've never done that.

Any other week, branch protection on main would have stopped it cold. GitHub would have rejected the push with an error, end of story. But we'd been in the middle of migrating the repo from our old GitHub org to a new one earlier that week. During the migration, branch protection got turned off. Somebody was supposed to re-enable it afterward. Nobody did. No ticket for it. No checklist item. The task fell through the cracks in a way that felt obvious in hindsight and invisible at the time.

So the remote accepted the force push. Forty-eight commits from three contributors. Two days of integrated, reviewed, tested work. Gone from main. Replaced with whatever was on this junior dev's local branch, which was weeks behind the current state.

He didn't realize what had happened. Pinged me on Slack about twenty minutes later asking why his PR was showing weird diffs. That's when I looked at the commit log and felt my stomach drop.

Recovery

Next few hours were not great.

What saved us: git push --force doesn't destroy commits on the server immediately. Git's garbage collection hasn't run yet. The commit objects are still there, floating in the object store. They're just orphaned. No branch points to them anymore. If you can find the right hash, you can get everything back. If.

We SSH'd into the CI server, which had fetched main recently, and used git reflog to find the commit hash that main was pointing to before the force push landed.

git reflog show origin/main

Found the hash. Created a recovery branch and hard-reset to it:

git checkout -b recovery-main
git reset --hard abc123f

Then the tedious part — comparing the recovered branch against the now-corrupted main, verifying nothing was missing, cherry-picking the handful of commits that had landed between the last CI fetch and the force push.

git log --oneline recovery-main..main
git cherry-pick d4e5f6a
git cherry-pick 7b8c9d0

Worked. Got everything back. But it took the rest of the afternoon and part of the evening, and for about two hours nobody was sure we'd recover all of it. One developer whose commits were overwritten had already gone home. Had to call him and ask if he still had any local branches with commits that existed only on his machine.

He did. One commit that lived solely on his laptop.

That moment — conference room at 7 PM, waiting for someone to confirm over the phone that the branch was still there locally — was when the stress really registered. Not the initial discovery of the force push. That was more like shock. The weight showed up later, in the quiet parts, when you're just sitting there waiting and hoping.

Immediate Response: Branch Protection

Obvious first move. Protection went back on main within the hour. Beyond that — we wrote a post-migration checklist and stuck it in the team runbook. Any time repo settings get touched, org transfers happen, anything administrative — someone verifies protection rules as the last step. Two people sign off.

Feels like overkill until you remember why the checklist exists.

We also configured a Slack alert that fires whenever branch protection is disabled on any repo in the org. Took about fifteen minutes through GitHub webhooks. Should have existed from day one.

{
  "events": ["branch_protection_rule"],
  "config": {
    "url": "https://hooks.slack.com/services/xxx/yyy/zzz",
    "content_type": "json"
  }
}

Rethinking How Branches Work

Before the incident, branching was informal. People made branches. Sometimes well-named, sometimes called fix-stuff or anurag-temp. No convention. Developers rebased against main when they felt like it, or merged main into their branch, or occasionally worked directly off main for small fixes.

None of that was wrong in isolation. Worked fine when the team was three people sitting next to each other. By the time the incident happened, the team had grown to seven and the informal approach was showing cracks.

Afterward, naming conventions. I go deeper into the different branching models — GitFlow, GitHub Flow, trunk-based — in my branching strategies post. For our team, the immediate change was simpler. Every branch starts with a prefix:

git checkout main
git pull origin main
git checkout -b feat/payment-gateway-retry

feat/ for features. fix/ for bug fixes. chore/ for dependency updates and CI changes. hotfix/ for production emergencies.

Was skeptical at first. Felt bureaucratic. But scanning a list of 30 active branches on GitHub and knowing at a glance which ones are features versus fixes versus maintenance — that turned out to be useful enough to justify the overhead. CI pipeline runs different checks depending on the prefix too, which was a nice bonus.

The bigger change: how we keep branches up to date with main. Settled on local rebasing:

git fetch origin
git rebase origin/main

Not git merge origin/main into the feature branch. Merge approach works technically but creates merge bubbles in the history that make git log and git blame harder to read down the road. With rebase, the feature branch looks like it was started from the current tip of main, even if it's been in progress for a week.

The tradeoff: rebasing rewrites local history. Pushing to the remote after a rebase requires a force push. And given what had just happened, the words "force push" made people physically flinch.

The Force-With-Lease Compromise

This is where the most time was spent arguing.

One camp wanted force pushing banned entirely. Disable it across the org. Branch has diverged? Delete it, re-create it. Clean and safe.

The other camp — which included me — thought that was too restrictive. Rebasing is a useful workflow. The problem wasn't force pushing in general. The problem was force pushing to main, to a branch that other people depend on, without checking whether you're about to overwrite someone else's work.

Compromise: --force-with-lease. It tells Git to check whether the remote branch has been updated since your last fetch. If someone else pushed commits after your last fetch, the push gets rejected.

git push origin feat/payment-gateway-retry --force-with-lease

Not foolproof. If you do a git fetch right before pushing, --force-with-lease will let you overwrite whatever was fetched, because from Git's perspective you're "up to date." But it catches the most common accidental scenario — rebase locally, push without realizing a teammate pushed to the same branch an hour ago.

Set up a Git alias to reduce the typing:

git config --global alias.pushfl "push --force-with-lease"

git pushfl origin feat/whatever does the right thing. Some of the team uses the alias. Others type out the full flag. Doesn't matter, as long as bare --force isn't in anyone's muscle memory anymore.

And --force on main or develop? Blocked at the GitHub level. Can't do it even if you try. Non-negotiable.

Cleaning Up Before Merge

Smaller change, but one I care about more than I probably should.

Before the incident, main history was a mess. Commits reading WIP, fix, fix again, okay actually fix, linting, forgot to save. Opening git log read like someone's internal monologue during a debugging session.

Now everyone cleans up commits before opening a PR. Interactive rebase:

git rebase -i HEAD~5

Opens the editor with the last 5 commits. Squash them together. Reword. Reorder. The goal: one or two commits per feature that describe what actually changed.

A good commit message, for us:

feat: add retry logic to payment gateway

Previously, failed payment attempts returned a 500 to the user
immediately. This adds exponential backoff with 3 retries before
giving up. Timeout per attempt: 5s.

Closes #247

Not every PR lands with a message this clean. Some are still mediocre. Decided that's acceptable. The point isn't perfection — it's that main should be readable six months from now when someone needs to understand why the payment retry logic works the way it does.

Tooling That Survived

After the initial wave of policy changes, we added some automation. Some stuck. Some didn't.

What stuck: pre-commit hooks through Husky. If you're using TypeScript, these hooks pair well with patterns like discriminated unions and exhaustiveness checking that prevent whole categories of bugs at compile time. Every git commit fires a hook that runs ESLint and Prettier on staged files.

{
  "lint-staged": {
    "*.{js,ts,tsx}": ["eslint --fix", "prettier --write"],
    "*.{css,scss}": ["prettier --write"]
  }
}

Linting fails, commit is blocked. Annoying for about two weeks while everyone adjusted their editor configs. Now nobody notices. It just runs.

What also stuck: a prepare-commit-msg hook that prepends the branch prefix to the commit message automatically. Branch is feat/payment-retry, the commit message gets feat: added to the front. Small automation, but it means even lazy commit messages end up categorized.

What didn't stick: commitlint enforcing the conventional commits spec down to the character. Team hated it. Perfectly reasonable commit messages got rejected because of a capital letter after the colon or a subject line that was 73 characters instead of 72. Kept it for two months and then quietly removed it. Spirit of the rule was right — write clear messages. Enforcement was too rigid for the benefit it provided.

Still go back and forth on whether removing commitlint was the right call. Some messages have drifted back toward vague territory. But nobody's fighting the tooling anymore, which is worth something in its own right.

Advice for Teams Starting Fresh

Don't wait for the incident.

That's the blunt version. Branch protection could have existed from day one. Naming conventions too. --force-with-lease as a standard practice too. None of this is advanced Git knowledge. It's in the documentation. Every "Git best practices" article covers it.

We didn't do it because things were working fine. Small team. Everyone knew what everyone else was doing. Informal approach felt adequate. And it was — right up until it wasn't.

Second piece: be careful about over-correcting after something goes wrong. The temptation is to lock everything down, add friction to every step, make it impossible for bad things to happen. But too much friction and people find workarounds. They push to personal forks and PR from there. They stop making frequent small commits because the hooks are slow. They ignore conventions because enforcement is so rigid it's actively annoying.

We tried to find the middle. Protect the things that matter — main, production branches. Automate what's easy — linting, formatting. Set conventions for the rest — branch naming, commit messages — but don't die on every hill.

Handling Merge Conflicts Without Losing Your Mind

Merge conflicts are inevitable with multiple contributors, but there's a difference between a painful conflict and a manageable one. The single biggest factor: how long a branch lives before merging. A branch that's open for two days will have small, localized conflicts. A branch open for two weeks accumulates conflicts that span multiple files and multiple logical changes — untangling those takes real effort and carries real risk of introducing bugs.

We started tracking branch age and nudging developers when a branch passed the five-day mark. Not a hard rule. Just a Slack reminder: "hey, this branch is getting long in the tooth, consider rebasing or breaking it into smaller PRs." It cut the average conflict resolution time roughly in half, and more importantly reduced the number of conflicts that required calling someone else over to explain what their code was supposed to do.

Ten Months Later

Somewhere around 2,000 PRs merged since the incident. No force pushes have escaped a feature branch. No work lost.

But I'm not going to pretend everything is settled. We still debate the rebase-only policy. Developers who join from teams that used merge workflows find it confusing. Rebasing rewrites history, which means the local and remote branches diverge, which requires --force-with-lease, which requires understanding what that flag does and why bare --force is banned. That's a lot of context for someone who just wants to push their code.

A couple of people have asked whether we should just allow merge commits from main into feature branches and skip the whole rebase dance. Don't have a great answer. History is cleaner with rebase. But "cleaner history" is an aesthetic preference as much as a practical one, and I'm not sure it justifies the learning curve and the anxiety around force pushing.

Long-lived feature branches remain unsolved too. When a branch stays open for two weeks, rebasing against main every couple of days becomes a chore. Conflicts accumulate. Sometimes you resolve the same conflict three times in a week because main keeps moving. Strategies exist for this — feature flags, trunk-based development, smaller PRs — but we haven't committed to any of them. Muddle through case by case.

And some of the policies we put in place were emotional reactions to a bad day. I recognize that. How much process is the right amount of process doesn't have a formula, and the answer shifts as the team changes. Someone leaves, someone joins, codebase grows, deployment pipeline evolves. The rules from ten months ago might not be the right rules a year from now.

What we do have, and what I think matters most, is that the team actually discusses how we work with Git. Instead of everyone doing whatever they learned five years ago and hoping it doesn't collide with anyone else's approach. Whether our specific rules are optimal — can't say for certain. That the rules exist, and people understand why — that feels like it matters.

The junior dev who ran the force push is still on the team. One of the more careful committers now. Runs git status and git log before every push. Probably more cautious than strictly necessary. But I get it.

I still check that branch protection is enabled more often than is strictly rational. Tools like tmux and fzf make it faster to stay on top of these checks — having your git log and status visible at a glance helps catch problems before they escalate.

---

Further Resources

• Git Official Documentation — The complete reference for Git commands, internals, and workflows straight from the source.
• Atlassian Git Tutorials — Well-structured guides covering branching strategies, rebasing, merging, and collaborative workflows with visual diagrams.
• Pro Git Book (free online) — The definitive book on Git by Scott Chacon and Ben Straub, covering everything from basics to internals.