Safe System Changes

TL;DR

Making changes to computer systems can cause problems. This guide shows you how to reduce risks by testing thoroughly and having a way to quickly undo things if they go wrong.

1. Plan Your Change

1. What are you changing? Write down exactly what you’re going to do. Be specific!
2. Why are you changing it? Knowing the goal helps you test if the change worked correctly.
3. Who will be affected? Identify all users, systems, or services that might be impacted.
4. When will you make the change? Choose a quiet time (e.g., overnight) to minimise disruption.
5. How will you communicate? Tell people about the planned change and potential downtime beforehand.

2. Backups are Your Friend

Before *any* change, create a backup of everything that could be affected. This is your safety net.

1. Full backups: Copy all the data and system files.
2. Database dumps: If you’re changing a database, make a copy of its contents.

   mysqldump -u username -p database_name > backup.sql

3. Configuration file copies: Save copies of important settings files.
4. Test your backups! Make sure you can restore them before you start the change. This is crucial.

3. Test, Test, Test

Don’t make changes directly to the live system. Use a test environment that mirrors production as closely as possible.

1. Create a test environment: This should be separate from your live systems.
2. Apply the change to the test environment: Follow your plan exactly.
3. Run tests: Check everything that might be affected by the change.
   • Functional testing: Does it do what you expect?
   • Regression testing: Did anything else break? Run existing tests to confirm core functionality still works.
   • Performance testing: Is it as fast (or faster) as before?
4. Fix any problems: If tests fail, investigate and correct the issue in the test environment. Repeat steps 2 & 3 until everything works correctly.

4. Controlled Rollout

Even with thorough testing, things can still go wrong. A controlled rollout limits the impact of problems.

1. Canary release: Deploy the change to a small group of users first. Monitor closely for issues.
2. Phased deployment: Gradually roll out the change to more and more users over time.
3. Feature flags: Enable the new feature only for specific users or groups, allowing you to quickly turn it off if needed.

5. Have a Rollback Plan

If something goes wrong in production, you need to be able to quickly revert to the previous state.

1. Document the rollback steps: Write down exactly how to undo the change.
2. Automate if possible: Scripts can speed up the process and reduce errors. For example, restoring a database from backup.

   mysql -u username -p database_name < backup.sql

3. Practice the rollback plan: Test it in your test environment to ensure it works.
4. Monitor closely after deployment: Watch for errors or unexpected behaviour. Be ready to roll back if necessary.

6. Post-Change Review

After the change is complete, learn from the experience.

1. What went well? Identify things you did right and repeat them in future changes.
2. What could be improved? Find areas for improvement in your planning, testing, or deployment process.
3. Update documentation: Reflect any changes to procedures or configurations.