One particular challenge that comes to mind was when we built out our own Matrix solution earlier this year. Even though the Matrix protocol was released 9 years ago, it’s a technology that’s still somewhat in its infancy, with a lot of questions around scale and application. We had a three person team with a deadline of six weeks to complete a stable, scalable backend to turn over to our Matrix developers.
Under “Job Title” on our company intranet, you wrote, “Literally web master.” I love that! But what exactly did you mean by it?
It’s really taught me how to apply fundamental reasoning in any situation. As I mentioned, I look at problems almost every day that I’ve never seen before, and have no idea where to start sometimes. My go-to method is always stepping back through first principles and working from there.
This has translated into my life in the same way. For example: I don’t really know anything about cars, but when my car breaks now, instead of it being an immobile piece of metal until someone that knows something can look at it, I step through the fundamentals of what that car needs to work. I’m often surprised at how simple things really are “under the hood,” if you just take a look with an open mind.
This is a good question. In general we all sort of have the mandate I mentioned previously, in that we’re all responsible for the availability, performance, reliability, and security of all Automattic’s services. However, some of us definitely have specialties or areas of expertise. For me in particular, I’m more of a generalist but help out from time to time with some of the project management around our data center operations, and I also help out our Opers focused on Tumblr Systems because I spent a couple of years focused on Tumblr Systems full time. —Rudy Faile
This is a good question. In general we all sort of have the mandate I mentioned previously, in that we’re all responsible for the availability, performance, reliability, and security of all Automattic’s services. However, some of us definitely have specialties or areas of expertise. For me in particular, I’m more of a generalist but help out from time to time with some of the project management around our data center operations, and I also help out our Opers focused on Tumblr Systems because I spent a couple of years focused on Tumblr Systems full time.
From the day you’re hired on, you’re handed an incredible level of trust and responsibility. I think Automattic is the apex of what remote work and a horizontal organizational structure should look like. On our “org chat” page there is a message that’s been posted there since I’ve been working here by our CEO Matt Mullenweg. It really encapsulates what I mean:
How would you describe the culture of Automattic and the people you get to work with?
“This shows an organizational chart that looks much like a traditional company. We’ve found keeping lines clear to be good for accountability, responsibility, and navigation. However the “hierarchy” does not imply worth or value—a key philosophy of Automattic is that good ideas can come from anywhere, and often do. Everyone in the company can influence any other part of the company; in fact if you believe in something, you are expected to.”—Matt
How does Sysops work at Automattic? What’s different about our approach?
What part of your role do you enjoy the most?
What new things is the Systems team working on, and how might they affect customer experience and the bottom line in the future?
One really nice thing about working in a distributed company is that we have great coverage across time zones. We all still have to respond to alerts, and everyone jumps on to help out, but no one has to work nights or anything like that (unless they want to 🙂). I think for the most part it’s a pretty simple hierarchy of importance, in that critical alerts take precedence over everything, and responding to other folks in the company is also pretty important to us. Everything else can slide to the right if it needs to.
This one’s easy! Everyone at Automattic in general is a lifelong learner. The first sentence of our creed that I talked about earlier is: “I will never stop learning.” In terms of learning specific to my role, there is really no choice, as technology progresses, developers want to use the latest and greatest tools for development, and end-users want to enjoy the benefits of them. We always have to keep a sharp lookout for emerging trends.
About what percentage of your time is spent interacting directly with our company’s customers?
It’s crucially important that we return accurate information or set things up properly for the requestor. These are the KPIs for systemsrequests, but we also try and get back to the requestor as quickly as possible—either with an acknowledgement that we’re working on it, a request for more information, or confirmation letting them know their request has been fulfilled, including the details.
However, all Automatticians, from the CEO on down, are required to interface directly with our customers during something we call a “support rotation,” which is a week-long period that happens every year. During this rotation you work in direct support, interfacing with Automattic’s customers directly—answering emails, communicating with them via chat, and everything else.
For those who may not know, what exactly is Sysops, and how does it contribute to a company’s stability, growth, and well being?
“Sysops” is short for “systems operators” (or systems/operations). The term is most often used in the context of networks, databases, or system administration. In general it refers to folks who manage and maintain computer systems, particularly servers.
Walk us through a typical work day.
How were you onboarded? Describe what you learned and how you learned it.
After that, we look into things like long-term projects and improvements to the system. For example, looking into recurring alerts and figuring out what can be improved there, or writing code for something else. While doing all of this, we are monitoring all of our alerting systems and Slack, responding to anyone who has a question, request, or report.
We went through multiple iterations, and there were times when I thought we not only would miss the deadline but perhaps not even be able to deliver the end product at all, given the constraints we had. However, in the Automattic Creed, we have a line that says: “I am in a marathon, not a sprint, and no matter how far away the goal is, the only way to get there is by putting one foot in front of another every day. Given time, there is no problem that’s insurmountable.” We kept this in mind as we continued to push forward day after day. We ended up meeting the deadline and delivering on time.
Then, I flew out to the data center to actually execute the project. Everything from racking the servers to configuring them to getting them in a state where they would be serving traffic to our customers. A lot of this I had never done before, but I was expected to look up everything I didn’t know to try and figure it out, asking questions if I got stuck.
I like solving problems, helping people, and automating things. The best part of the role comes when you can do all three of those things at once. For example, if someone comes to you with a problem, and you solve it by automating it and reducing their manual toil, you’ve just become their best friend 🙂. I like that a lot.
Hah. I think maybe literally everything 🙂. Considering the amount of servers we have and the number of customers we serve, we’re an extremely small team. I think, at most companies, you have structured divisions under infrastructure that specialize in each facet of systems administration. For example, folks who only work in the data centers, ops folks who only respond to alerts, infrastructure developers, database administrators, network specialists, so on-.
ABOUT AUTOMATTIC
This is kind of an interesting paradigm because when it comes to systems, in general our “customers” tend to be the other folks in the company. Developers come to us for help in implementing scalable solutions or deploying backend services, HR for access control, and everyone else for anything and everything in-between.
