How Minecraft raised a generation of engineers

I was 14 the first time a server I ran fell over in front of an audience, on a school night. The console kept printing "Can't keep up! Is the server overloaded?" and about 30 people were frozen in place. For a few years now I've noticed that the good infrastructure and security people I meet, the ones born after 1995, almost all ran servers as teenagers. Mostly Minecraft servers, and I don't think that's a coincidence. This is some of what running one taught me.

• The machine I rented arrived as an IP address and a root password, with no interface. I learnt Linux because typing commands was the only way in. Closing the terminal killed the server, which took me an embarrassingly long time to work out, so I started using screen and later tmux. DNS came next, then an SRV record, because Minecraft ran on port 25565 and nobody remembers a port number. Firewalls came after I saw connection attempts on ports I'd never opened.

• I lost a world once, months of things people had built, and only learnt about backups after that. I remember how it felt to tell them. It wasn't a technical feeling.

• A server tries to run 20 ticks a second. That was my first service level objective, years before I knew the term. Most of us started by giving the heap more memory, then all of the system's memory, until the machine started swapping. Then the server froze for 3 seconds at a time, on a rhythm, and I had to learn about garbage collection.

• Like half the scene, I copy-pasted a long string of JVM flags from a forum without understanding a single one. Eventually I understood there's no setting that's correct for everyone. Then I started profiling, because guessing stopped working. A timings report shows where each tick went, and the lag everyone complained about turned out to be one plugin doing something expensive on an event that fired constantly.

• I was on the receiving end of DDoS attacks at 15. The first time, players dropped all at once and I couldn't reach the machine. Then the host emailed to say they'd null-routed the IP address to protect everyone else on the rack. The person behind an attack is usually someone you banned or a competing server owner. That was when I understood an outage could be on purpose, and that whoever did it would adapt.

• A proxy in front hides your machine's address, until you find out your origin leaked months ago through an old DNS record you forgot about. Layer 4 and layer 7 attacks also don't get solved the same way. Years later I worked at a company that sells the mitigation. I won't claim that was a plan.

• The Mirai botnet, the one that knocked a chunk of the internet offline in 2016, came out of this same scene. Paras Jha and Josiah White ran ProTraf Solutions, a DDoS mitigation firm. Part of the point of the botnet was knocking over rivals in the Minecraft server hosting market. They and Dalton Norman pleaded guilty in December 2017.

• Nobody in that scene chose Java, it was just what the plugin API was written in. You started because something annoyed you and no plugin did it. Documentation barely existed, so I read other people's code, which is still the fastest way I know to learn anything. I picked up SQL when I realised player data had to survive a restart. I learnt my first language under duress, and I think that's a good way to learn one.

• Plugins were sold for real money to teenagers running servers, and I was on both sides of that. I cracked plugins I couldn't afford, and I also sold plugins and tried to stop people cracking them. Some sellers used licence servers keyed to your IP address, which you got around by pointing the licence domain at 127.0.0.1 in your hosts file.

• Other sellers signed their jars. A signed jar keeps its signature files in a directory called META-INF, and the JVM won't load a class that doesn't match its recorded digest. You just delete the signature files, and it loads as an unsigned jar. A check that runs on hardware you don't control can raise the cost, but it can't win. I now run a company built entirely on that idea.

• Moderation took the most out of me. Griefing was the easy part, and it's why a lot of us learnt what an audit log is. People said nasty things in chat at 2 in the morning, and occasionally something appeared that frightened a 15-year-old, with no adult anywhere in the chain. Once you recruit staff, you find out that some of the people you give ban permissions to will use them to ban their friends' enemies.

• There was money too: donations, ranks, a store and chargebacks from a parent who saw a PayPal charge and had no idea what it was. We were doing trust and safety, fraud and customer support as teenagers, mostly badly.

• In June 2014 Mojang enforced its end user licence agreement, and servers had until the 1st of August to stop selling anything that affected gameplay. Then in September a DMCA notice landed on CraftBukkit. Wesley Wolfe, who had contributed something like 23,000 lines to it under the GPL, took the position that the licence terms hadn't been honoured. Nearly every server was built on CraftBukkit, and it was gone within a weekend. I was 17, and I realised everything I'd built sat on somebody else's platform.

While I believe this pattern is real, I want to be careful with it. Getting in needed a computer at home, a decent connection, unstructured hours and parents who tolerated a teenager on a server at midnight. Those spaces were often hostile too, and pushed out plenty of people who'd have been brilliant at this. Some of the best engineers I've worked with never touched any of it. Every cohort had its own version anyway: BBSes, IRC bots, phpBB forums, and today Roblox, FiveM and Discord bots.

When I want to know whether someone can operate a system, I ask what they ran before anyone was paying them to run it, and then what broke.

For 4 or 5 years, whenever something broke, nobody else was going to fix it.