Data & Security

Customers expect their data to remain safe. They assume that the app is secure. You have to assume that it is not secure. Any line of code could potentially cause a hole that, even though it’s unlikely to be exploited, could if it were bring down your business. Review each line of code with this in mind.

Review all of the relevant security info for web sites.

There are tons and tons of random bots that just go around trying to attack every single website. This makes up a ton of web traffic. Assume that your site will be hit by these bots constantly.

Testing

Assume that nothing will work. Assume that what works in development will not work in production. Do everything you can to make things more likely to work, but never assume that they will.

Backups

Assume that you will arbitrarily lose all your data at least once a month. Have backups, and have backups for your backups. Have scripts that run backups. Have scripts ON ANOTHER SERVER that check to make sure that there are backups. Have notifications that let you know if either server is down. This sucks, but telling customers that you have lost their data feels so horrible that the extra time is worth it.

Logging

Always have full logs of everything that happens. Learn to read everything in var/log. In the cloud, logging can be a bit of a pain. It’s still worth it. If you don’t have logs, you will have issues b/c there will be problems and you won’t know what happened. This leads to fail.

Paranoia

Assume that your code is broken and there is something always wrong that you don’t know about. Constantly make sure that there is logging and backups for everything. Manually check that backups are working at least once a week. Assume that your servers will randomly fail. Have alerts setup so that if something starts to fail, you will know before your customers see errors.

Gut feelings

If you don’t understand why something happened, or why something is going wrong, don’t just fix the symptom. Figure out what the underlying problem is, even if it takes 20 hours of working in a row.

If you don’t do this, then the underlying problem will come back 3 months later to bite you in the form of 3 months of incorrect data, 3 months of data lost, or 3 months of a security hole being wide open. Figure out the root cause. Make sure you understand why things are going wrong.

Communication

Tell co-workers/your boss what’s going on as early on as possible. It’s better to chat someone and say “The site is having issues with x. I don’t know what it is yet. I am working on it now” then to wait for them to find the issue. As you go, communicate to explain what your status is. Once you have found the issue, tell whoever is in charge. As you try various fixes, tell whoever is in charge. Once it’s fixed, tell whoever is in charge. Once it’s fixed, ask why five times to prevent it from happening again.

Honesty

Everyone fucks up. The github founder accidentally dropped his production database. I’ve accidentally basically dropped the production database (see my post on Mongoid and delete_all). If he will fuck up, so will you. Don’t lie and make it seem like the problem wasn’t your fault. Don’t hide data that will prevent people from realizing that it’s your fault. Don’t wait 3 days to tell anyone because you’re afraid of the consequences. Unless you knowingly did something outright malicious, you’re not going to lose your job, you’re just going to feel shitty. You’ll feel less shitty if you’re honest about it.

Determination

If something is failing, don’t say “Well, this is hard, so we can’t do it.” Just fucking figure it out. If it’s hard, then figure out a way to make it easy. If it’s still hard, then just spend 20 hours learning what you need to in order to make it easy. Make sure that you communicate this with whoever is in charge, so that if, at some point, it really is too hard, they can make a decision.

Your machine

Lock your screen when you’re not around, even in the office. Lock your door in your apartment.

Realize that your machine, if it’s lost, probably has way to much secret info on it. Try to have as little secret info as possible. Assume that anyone who steals your machine will do the worst possible things that they can with what’s on your computer.