> For the complete documentation index, see [llms.txt](https://ultimatewebsolutions.gitbook.io/qa-knowledge-base/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ultimatewebsolutions.gitbook.io/qa-knowledge-base/critical-bug-in-production.md).

# Critical bug in production

First, I would assess the severity and business impact. If it's a critical production issue, the priority is to restore service or reduce the impact, for example through a rollback, feature flag, or emergency hotfix. <mark style="color:$danger;">**I would avoid making an untested direct change in production**</mark>.&#x20;

**The fix should be developed and tested through the appropriate environments as quickly as possible, then deployed to production** following the emergency change process.&#x20;

**Afterward, I would make sure the fix is also incorporated into the lower environments and codebase to prevent regression.**

### Instead of "Let's just change the code directly in production.":

* **Immediately assess the impact.**
* **Mitigate** if possible — rollback, disable feature, feature flag, etc.
* Prepare a **hotfix**.
* Test the hotfix as much as the situation allows.
* Deploy it to production using the emergency/change-management process.
* **Monitor production closely.**
* Apply the same fix to the lower environments/code branch so the problem doesn't reappear in the next release.
* Perform regression testing.

### A good production-bug process

Production bug
\
↓
\
Assess severity / business impact
\
↓
\
Can we mitigate immediately?
\
↓
\
Yes → Mitigate / rollback / feature flag
\
↓
\
Investigate root cause
\
↓
\
Develop fix
\
↓
\
DEV → QA → STAGING
\
↓
\
Regression / verification
\
↓
\
Production deployment
\
↓
\
Monitor