log.andvari.net - Writing

Disappointing People Early

2026-02-02T12:00:00+00:00

One of my favourite phrases to remind people of as I'm fumbling about the business of producing reliable systems is "We Should Disappoint People Early". I've gotten enough funny looks about this that I figure I should explain myself.

The 'polite' assumption most folks make is that we should aim to never disappoint people at all. However, the wisdom is in the timing. If disappointment is coming (and it usually is, somewhere), you want it to arrive early enough that everyone can adjust, rather than late enough to cause real damage.

This applies everywhere: SLOs, product roadmaps, support response times, vendor relationships, and most critically, to the implicit promises we make when we don't say anything at all.

The Implicit Promise Problem

My erstwhile colleague Niall Murphy wrote an excellent piece on implicit SLOs and their dangers that crystallises something I've seen over and over. The short version: your users are already forming expectations about your service's reliability, whether you tell them what to expect or not.

If your service has been running at eleventy nines for the past year, your customers have noticed. They've built systems that depend on it. They've made architectural decisions predicated on "this thing is always up." They've stopped building fallbacks. You've made an implicit promise, and you probably didn't mean to.

The problem comes when you need to break that promise. Maybe there's cost-cutting. Maybe there's a re-architecture. Maybe you're just doing some planned maintenance that you've been putting off. Suddenly, you're at three nines instead of eleventy, and your customers are furious -- not because three nines is unreasonable, but because you violated an expectation they'd built up over time.

As Niall puts it: even if you smoothly manage the transition, if you fail 100x more often than you previously did, people are going to notice.

Why Stating an SLO is Better Than Not Stating One

The SRE book makes this point clearly: the business must establish what the availability target is for the system. Not the SRE team, not the platform engineers -- the business, informed by what users will actually tolerate and what makes commercial sense.

Ben Treynor's observation that "100% is the wrong reliability target for basically everything" is foundational here. You're not running a pacemaker. You get to decide what's reasonable, and more importantly, you get to tell people.

The alternative -- not stating an SLO -- is worse in every way. You've still made a promise; you just don't know what it is. Your customers have inferred one from your historical performance, and it's probably higher than you'd have chosen. You now have all the obligations of a promise without any of the negotiation that should have gone into making it.

You have to make trade-offs. You can't make trade-offs against a number you haven't decided on. When you decide on that number, you then need to staple it to people's faces and let them emote about it early.

The Same Pattern in Roadmaps

Product roadmaps suffer from exactly the same dynamic. A product (or platform!) team should not make a promise until they've conducted enough discovery work to understand what's truly required, and the only people who can make such a promise are the team responsible for delivering it. The corollary is stark -- if you're making roadmap commitments before you understand what delivery entails, you're setting up a future betrayal.

This is the implicit promise problem wearing different clothes. If you show a customer a roadmap with specific features and dates, you've made a promise. If the business context changes -- and it will -- you're now in the position of either delivering something suboptimal because you committed to it, or "breaking your promise" by adapting to reality.

The fix isn't to hide your roadmap. The fix is to disappoint people early by being explicit about the nature of roadmaps: they're current best guesses, subject to change, and the further out you look, the fuzzier they get. Folks who've learned this lesson put "subject to change" disclaimers on everything, use confidence percentages, and replace hard dates with buckets like "Now," "Next," and "Later." Obviously it's possible to hedge too much, but a certain amount of expectation-setting is healthy.

When roadmap planning is treated as a rigid forecast, it creates pressure and distrust. When it's treated as a dynamic, communication-first process, it builds trust and momentum -- even when timelines shift. The early disappointment of "this might change" is infinitely preferable to the late disappointment of "you promised."

Just like with outages, good partners and customers will understand that software and the means by which we make software are terrible, complicated, unpredictable beasts. A good customer and partner should be more interested in your response to an outage than they are on beating you over the head with it. Similarly, a good customer partner will appreciate you keeping them up to date on delivery and being honest about outcomes, rather than doing deadline gymnastics.

Vendor Relationships and the Shared Responsibility Trap

There's a particularly insidious version of this problem that plays out when companies move from on-premises infrastructure to PaaS or SaaS, in either Security or for provision of a dependency. The cloud shared responsibility model is supposed to be clear: the provider is responsible for security of the cloud, while the customer is responsible for security in the cloud. In practice, it's a mess of unstated expectations.

Gartner famously predicted that through 2025, 99% of cloud security failures will be the customer's fault. This sounds like finger-pointing, but it reflects a deeper truth: when companies adopt cloud services, they often expect the vendor to take over responsibilities that were never part of the deal. As a customer, you're intentionally abdicating duty of care about the details, in return for paying money for an outcome. It can be difficult to let go of the parts you've farmed out to your vendor.

When vendors aren't explicit about what they won't do, customers fill in the gaps with optimism. So, when something breaks, the disappointment isn't just about the outage -- it's about betrayed expectations that were never actually set. The NSA and CISA published guidance in March 2024 specifically because so many organisations were accelerating their cloud journeys "without proper planning and an appreciation for shared responsibilities."

This creates an ugly dynamic. If you're a vendor and you're not telling customers what failures to expect, they're going to want the full list of all things that can go wrong. Their nervousness translates into demands for breadth-first total coverage of all eventualities. But there are always failures you won't be able to enumerate specifically -- the unknown unknowns, the novel combinations, the things that have never happened before.

The answer is to disappoint early. Be explicit about what's in scope and what isn't. Document the failure modes you do know about. Make clear that you can't possibly predict everything, and explain what happens when the unexpected occurs, and why that makes you a safe pair of hands. The discomfort of that conversation now is vastly preferable to a customer discovering during an incident that their assumptions about your service were wrong. It also precludes a lot of demands from those same customers that are driven by vibes and nervousness at not feeling they know enough about potential failure modes and outcomes.

Support Response Times: The Hidden SLO

Every support team has an SLO, whether they've stated one or not. If you typically respond to tickets within 4 hours, your customers have noticed. They've calibrated their workflows around it. The first time you take 24 hours, they'll be furious -- not because 24 hours is unreasonable in isolation, but because you violated the expectation you accidentally created. This applies also to customers who only open the occasional support request - the feigned surprise that they don't get a 15-minutes response when the SLA is an hour (or more) will be familiar to anyone who's been on a customer support rotation. Customers who open tickets all the time will be more familiar with the expected performance because they've been told more recently.

This is where "disappoint early" becomes almost literal. If you know you can't offer priority support between midnight and 5 AM, say so upfront. If critical issues get faster response than routine questions, publish your priority matrix. If your support capacity means occasional delays during high-volume periods, tell people before they need help, not after they've been waiting.

Internal Customers Too

This isn't just about external customers. The same dynamic plays out inside your organisation, often more acutely.

If you're a platform team that offers services internally, your internal customers (the product teams) are forming expectations about your services just as external customers would. If you've been accidentally excellent at something -- say, keeping your deployment pipeline at sub-minute latencies -- you've now made an implicit promise. The moment that slips to five minutes because you've added some necessary safety checks, you'll hear about it.

The conversation is much easier if you've stated up front: "We target 95th percentile deploy times under 5 minutes." Now there's a conversation to be had. Maybe 5 minutes isn't good enough for some use cases. Maybe it's plenty. But at least everyone knows what they're working with.

Similarly with stakeholders and leadership. If you're asked "when will this be done?" and you say "soon" or "we're working on it," you've allowed them to fill in their own number. That number will be wrong, and it will be too optimistic. The disappointment arrives later, bigger, and with compound interest.

Saying "this will take four weeks" might disappoint someone today. But that disappointment is manageable. They can plan around it. They can reprioritise. They can have the argument about whether four weeks is acceptable now, when there's still time to do something about it.

The Discomfort is the Point

Stating an SLO feels uncomfortable precisely because it's a commitment. The same is true for publishing a support response time matrix, or putting "subject to change" on a roadmap, or listing the failure modes your SaaS platform won't protect against. Each of these forces you to confront the reality that you can't be all things to all people. It makes explicit something that was previously implicit and negotiable.

That discomfort is valuable. It's the discomfort of honesty. It's far preferable to the discomfort of a customer discovering during an outage that your service wasn't as reliable as they'd assumed, or finding out during an incident that their cloud vendor doesn't cover what they thought it did, or learning that the feature they were counting on got deprioritised.

The core of "disappoint early" is this: small disappointments now prevent large disappointments later. An SLO that seems modest compared to your historical performance might disappoint a customer today. A roadmap that says "this might change" feels less confident than one with firm dates. A vendor contract that explicitly lists what's out of scope seems less comprehensive than one that doesn't mention limitations at all.

But in each case, the explicit version creates space for honest conversations about what they actually need, what you can actually deliver, and what happens when those don't align. The implicit version creates the illusion of agreement that shatters on first contact with reality.

So.

If you're running a service without a stated SLO, you've already made a promise -- you just don't know what it is. Your customers have inferred one, and it's probably more optimistic than you'd like.

If you're sharing a roadmap without caveats about uncertainty, you've created expectations you may not be able to meet. You've also put yourself in a position where you've no way to climb down if the situation changes.

If you're a vendor who hasn't explicitly documented what failures customers should expect, they're filling in the blanks with assumptions that will turn into accusations when something goes wrong.

If you're working with stakeholders without explicit expectations about timelines, support responsiveness, or capabilities, you've allowed them to assume. Those assumptions will bite you.

The fix is the same in every case: have the uncomfortable conversation now. State what you can actually commit to. Document what's out of scope. Put confidence levels on your forecasts. Publish your priority matrix. Be boringly, repetitively explicit about constraints and limitations.

It's often a good idea to overcompensate when communicating constraints or expectations that may be 'disappointing.' The customer who understands your limitations upfront is a customer who can plan around them. The customer who discovers them during a crisis loses trust, a thing that is hard-won and sometimes impossible to repair.

It's paradoxical, but the path to being a more reliable partner -- to your customers, your stakeholders, your colleagues -- runs directly through being willing to disappoint them sooner.

Chasing Boring at Just the Right Speed

2026-01-22T12:00:00+00:00

Asking the Right Questions

A while back when I was looking for a fulltime gig (and when I was contracting, of course), I had the opportunity to do a bunch of interviews (including the kind of informal interviews you do to get a new contracting client). One of my boilerplate questions when asked "Any questions for me?" has always been "What does success look like for the person in this role?". I like to ask it of everyone, even folks who'd be reporting to me or folks far away on the org chart. It gives you a good insight into how stitched together the team is on the important bits, I find.

In the majority of cases, I tended to get at least one answer that boils down to "Fewer Outages, Lower MTTR".

"Fewer Outages", to me, is a weird one. I tend to respond to a "There are too many outages" complaint by asking "Well, how many is too many?". Just like 100% is the wrong SLO target for basically everything, "None" isn't the right target for number of outages. You take a reasonable and data-informed swag and aim for that. In the vast majority of cases, you're not running a nuclear power plant or a circulatory system, so you get to really think about what number is right.

Similarly, for MTTR (Mean Time To Recovery, or 'how long in seconds it takes you to mitigate the effect of an outage') the last while has seen a "race to the bottom". There are entire companies whose premise for what they'll do for you (and charge you a handsome fee, to boot) is lower your MTTR. They'll generally do it using an AI chatbot (and strand your systems knowledge and capability inside that same bot), but that's neither here nor there - the main gist is "MTTR is your problem", and it's a tempting one to latch onto. Easy to measure, easy to be seen to be grumpy about.

Doing it the Fastestest

The problem is that MTTR, as a primary focus, is a red herring. It's measuring the wrong thing; or at least, it's measuring something that's downstream of what you actually want.

If you're having the same kinds of outages over and over, getting really fast at recovering from them is like getting really good at bailing water out of a leaky boat. Sure, you're staying afloat, but water coming in is a bad sign, and you built the boat in the first place so should know what's up.

The real purpose of good incident practice isn't to get fast at recovery. It's to feed back what you learn into your software development lifecycle. Every incident is information. Every outage is telling you something about where your systems, processes, or assumptions are wrong. The goal isn't to recover quickly (though you should); it's to ensure you don't have to recover from the same thing twice.

If you wanted to be a little more philosophical about it (in terms your boss would probably hate), you could say that Outages (like all feedback) are a Gift. You pick yourself up, dust yourself off, and see what there is to learn from the (sometimes pretty shitty) experience.

I waxed slightly lyrical recently about Cloudflare's writeup on the outage they had in November 2025. The part that was valuable to me was not a power fantasy about how quickly everyone scrambled and dropped everything and mitigated the outage - in fact, I only checked just now how long the outage lasted (3 hours). I'm not concerned about those 3 hours. Less would be nice, but the real outcome is much more valuable. I'm not even a current Cloudflare customer, but the fact that I know what caused the outage, and I know things have changed means more than a much shorter outage, followed by "Sorry customers, it's very technical, won't happen again (probably)".

Doing it The Bestest

When I talk to teams about the point of good incident management (primarily retrospectives), I generally say that the point of doing them is twofold. That is:

Avoiding Repeats

If you're seeing repeat incidents -- the same service falling over the same way, the same capacity issues, the same configuration mistakes -- then your incident response process isn't doing its job, no matter how good your MTTR looks. You're optimising for the wrong end of the pipeline.

When you focus too hard on MTTR, you create incentives that work against learning. Teams get good at quick fixes and workarounds. They get good at restarting services and clearing queues. They don't necessarily get good at asking "why did this happen, and what systemic change prevents it from happening again?"

When I worked freelance, I ran into at least one shop that used a managed service partner. Their first instinct was to kick things, seemingly at random. Restart the service. Reboot the machine. Reset the network device.

Their MTRR numbers were amazing.

Generally when a particular service fell over, it was back within the hour. However, it was falling over every couple of weeks. When I probed for more information on contributing factors, I got crickets. No followups, no logs, no learning. Clearly there's some slider between "mitigate" and "understand" that makes them somewhat mutually exclusive at either end of the slider.

Raising All Boats

Post-incident reviews (or postmortems, or whatever you're calling them this week) should be producing action items that feed into your roadmap. Not just "add better monitoring" (though sure, maybe), but real changes: fix the race condition, add the circuit breaker, change the deployment process. Do a priority swap so we fix something we've identified as real, instead of adding additional whizz-bang that may be more speculative in terms of how happy it makes customers. For bonus points, involve your customer support friends to make sure what you're inserting into the SDLC is actually going to move the needle for customers.

The measure of a mature incident response process isn't how fast you recover. It's how rarely you see the same incident twice, and how much of the learning from your outages sticks, in the form of enduring shifts to how you write code and design systems. It's how often your outages are genuinely novel -- new and interesting failures that teach you something new about your systems, and real action on then finding yourself a better class of first-world problems.

So

So yes, resolve outages quickly. Absolutely. But don't let MTTR become the thing you're optimising for. The goal is to build systems and processes where you're constantly learning and improving, not systems where you're just really efficient at fighting the same fires over and over.

Your incidents should be novel, and retro outcomes should be real. If they're not, your incident process is failing you, regardless of what your status page says.

Critical Thinking and DEI

2025-02-16T00:00:00+00:00

Part of my day-job as head of Engineering at Google Ireland was attending talks and events/Q&A for local interest groups like Engineers Ireland. I was reminded today of a panel I was on around diversity in engineering in Ireland, and my allegedly 'fascinating' comments at it. That's not the part I remember most vividly, though.

After the presentation of the report, there was a Q&A session, with some good but not unusual questions. At one point, someone up the back raised a hand, and asked:

"What is Critical Thinking?"

The panel kind of went a bit quiet, but the cogs and wheels were turning in my head. It was exactly the right question at the right time, and a lot of stuff in that very moment fell into place in my head. Probably one of the best Q&A questions I'd ever seen, let alone had the privilege to answer.

Critical thinking is the ability to inspect a problem from different perspectives, and most crucially, to be able to reject hypotheses without bias. You need to be able to take your bright idea, and discard it in place of a better one, without attachment. To do it, you need people who have different frames of reference, who have had different 'crucible moments' and who have different contexts culturally that allow us to respectfully disagree and offer a broad set of views.

Homogeneous teams simply can't do it. We like to imagine we can transcend our personal experiences, biases and ego, but it simply isn't the case. That, to me, is what came to form the core of my thoughts on Diversity, Equity and Inclusion (or at least, the Diversity part). It forms a key part of my philosophy on building good teams. You simply can't build a good team from folks who look, think, and have experienced the same. Homogeneous teams are weak teams.

Another example: much earlier in my career, I was part of a small group in a leadership training tasked with coming up with a simple means of telling each other the number on a price tag without speaking. One of the team asked "How do we say a comma?". Myself and another Irish person in our group didn't see the point; just spell out the digits, why put in commas?

Turns out in big parts of continental Europe, price tags have a comma between the euro amount and the cents, whereas in Ireland there's a period. This may seem like a facile example, but these are the sort of small assumptions that get made every day that lead to accessibility issues, bias and poor quality outcomes in products you build.

There are a lot of things happening right now that make this a hot button issue; I care deeply about the social justice aspects of the naked white supremacy, homophobia and xenophobia that informs the current US administration's policy shifts. My friends are being hurt.

However, another aspect of pushing back against this wrong-headed, unjust and deeply stupid shift is what also reminded me of something I got to say to James Damore in the brief time he was at Google after his stupid-ass memo (this got me put on an external list of 'SJWs', a point of particular pride for me).

I told him, "You're not just wrong, you're incorrect."

As well as the plain prejudice, racism and homophobia that informs the current policy shifts, it's also worth pointing out that they simply weaken capability. We're being asked to build weaker teams, with narrower mindsets and much less interesting approaches to work.

While we're resisting and addressing these changes as leadership, remember that pointing out real, tangible risks to business and capability are part of the arsenal.

Believable AMAs for Genuine Leaders

2025-01-13T00:00:00+00:00

Communication Scales with Scale

Communicating with your team scales up as your team grows - it's an experience not unlike the individual engineer to senior engineer to staff engineer path. As well as scaling up in terms of impact, the very method of how you do it changes.

You very quickly grow out of being able to do a full matrix of 1:1 conversations, likely before you even go beyond single digit numbers of people. Even with a smaller team, this method can get pretty exhausting for all involved quickly. It's also pretty lossy - rather than actively tracking what you've communicated to each person, it's not unlikely that you'll assume you've said something about team direction, strategy, etc to everyone, when you might not have. It's possible to track this more exactly, but it's probably best not to. This is one instance where after you open a new spreadsheet, it's probably time to have a quiet word with yourself and see if you need to change the method.

Next in the toolbox is usually the ubiquitous stand-up or team meeting - a much better way to make sure you've let the whole team know about something, and ideally to give them the chance to clarify anything. This is probably the most common way to 'filter' down either communications from the broader organisation, or directions just for this team.

Once you scale beyond a team, things get tricky. You want to filter a message to all of these folks, but also give them the opportunity to ask questions and fill in the blanks they might have, that often you haven't thought of. Google's TGIF meetings used to do this; I attended a number of these in person when I'd visit HQ, and seeing Larry and Sergey being surprised by questions was pretty routine. This isn't a bad thing -- you're not going to think of everything, and often you'll miss things that will be brought up by thoughtful questions. The answers here were usually pretty frank; this changed later on when TGIF answers got pretty routinely leaked, often in real-time.

When does it become time to do something similar for your own team? In my case, it was when I started managing teams in various different offices - I had teams first in Dublin and California, and later NYC, Sydney and Seattle. While I did do my fair share of aeroplane time, it quickly became unworkable to do local "town hall meetings", even virtually. I still did them when I visited offices, but I wanted these to be in addition to folks getting all the information they needed day-to-day.

Tip: Perception is Reality

If you put a message out to a group of people, and most of them think you meant a certain thing, then that's what you meant.

This is a common pitfall from leaders of all levels - I've seen everyone from line-level managers to VP+ everywhere make the mistake of communicating carelessly, and not following up when it becomes clear that folks misunderstood what they meant. There's a hard problem here for most leaders. You're a professional communicator, you're good at this, so it's very tempting to think something like "Well, if folks misunderstood then that's not my problem".

What just happened is that you said words and people parsed those words into meaning, and your words failed to get the meaning across. In fact, they got another meaning across that you didn't mean to, making that the reality for those people. If a senior person shows up and says something, and most of the team comes away thinking "Dave will fire us if we make mistakes", then that's what you said. You've got work to do, or learning to do, or ideally both. Don't let hubris make you skip this part.

Communicating a Thing Starts with Communicating It

How you actually make decisions is beyond the scope of this article - let's assume for a moment you've done a good job at having all the right folks do all the right things to make a decision, charter a project, or otherwise determine a path forward.

Perhaps you've gotten a few leads together for a day or two and determined your roadmap for a year, or decided to make some team changes, or something of that ilk. You've spent hours internalising all the factors involved, and ideally used good judgement and data to come to a decision. Why would you expect folks on your team outside the room to immediately understand why a decision is made, just because you announced it? That's assuming you announce it at all.

The primary mistakes I've seen people make when it comes to communicating direction and change is:

Not communicating the decision, direction or change at all.

This is extremely common. It is a shockingly common assumption by rooms full of smart people that knowledge and context hard-won in that room are somehow magically transplanted to everyone affected by the outcome. I try to dedicate some "How are we communicating this?" time at the end of leadership get-togethers for this exact reason. There is real, difficult work in communicating things to folks; step one is acknowledging this and planning for it.
Communicating a decision or change, without providing any means of asking for details, or otherwise questioning it.

Every organisation differs in how decisions are done. On one extreme, there's the autocratic "One person makes all decisions" model. On the other end is the apocryphal "Absolute Consensus on all decisions, always" model. Chances are you're somewhere in the middle. Someone on the team isn't going to be happy with any given direction or decision. They're going to have some opinions and (usually loaded) questions. You want to put yourself in a position where these questions get answered, ideally in public. Otherwise people will answer these questions themselves. You're probably not going to like those answers. There's really no downside to inviting questions and comment on directions and decisions, yet I've seen many otherwise smart folks think they can quietly implement a decision and maybe nobody will mind or notice.
Not owning the decision

It's always tempting to defer to some external factor (deadlines, budget, leadership) when saying why a decision was made. This is a tricky balance, because while most good leaders don't like deferring to "Because I/we say so" as the reason a direction is being taken, that's ultimately what's happening. Yes, there are always contributing factors, but you've taken those into account to make a high-quality decision, which you should stand over as your own. If external factors seem to force your hand in all directions you take, then you're not really making decisions at all. Again, perception is reality. People are smart and will spot this right away.

Tip: Timing and Heads-Up(s)

As a conscientious leader or manager, you are likely highly practised at the art of delayed gratification. Most people aren't. If you're announcing something, 'fast follow' doesn't mean in a week or two when you can schedule a Q&A/AMA session. Let your ability to schedule this kind of session inform when you announce. Don't announce things on Friday and do the session on Monday; people will spend the weekend inventing answers to their own questions that will be way more compelling than any boring sensible answers you might eventually give.

If there are folks who need to know about something in advance, because they're key stakeholders, or simply because people will ask them about it first and they don't like surprises, do give them a heads-up. If you're worried about leaks, then that's a good instinct. Do this as late as is practical. Things will leak, accept this as a fact of life and plan accordingly.

Making it Real: The AMA

This section sets out tips on AMA ("Ask Me Anything") sessions. These can be held in relation to a specific announced thing, or be semi-regular. I've often made a point of doing these regularly (and per-timezone, for folks at an offset time from me).

Tip: A note on 'Corp Speak'

The jig is up, corp-speakers. Everyone knows you're doing it. You haven't cracked the ability to make people believe you through clever use of words.

One thing that increased numbers of virtual town-halls have given us is the ability to leave without tripping over chairs or being noticed, and the ability to watch attendee numbers in real-time. I've personally watched viewer numbers drop off a cliff on these kinds of calls when a particularly 'polished' answer comes out. This is a shame, as it drives disengagement from the message coming from leadership, and even if the rest of the content is credible and sincere, this can derail people's engagement with the decisions or directions being communicated. At best, folks will just think less of the corp-speaker.

One clue that this is not the True Way is that most CEOs are not corp-speakers when they're at home. In a company-private setting they are generally quite open and communicate as openly as they can. Generally when they defer a question to a corp-speaker, it's because they need protection from Saying the Wrong Thing, or smell a Trap. More on Traps later.

Setting Expectations: The '3 Answers' Model

I've seen several variants of this model of AMA, so I can't take any credit for it. However, it has worked for me for many years across teams and orgs.

The basic model (which you should absolutely tell attendees in advance) is that attendees can ask anything. The answer, however, can take one of 3 forms.

I answer the question to the best of my ability.
I tell you that I don't know.
I tell you why I'm not telling you, to the best of my ability.

First, you'll notice there's no "I will bloviate for a few minutes and hope you look bewildered and stop asking" option. The intention is that the answers are credible and concise, and engage with the question in good faith.

Option #1 is straightforward, and ideally is the option you take most of the time. Answer the question. Be prepared for follow-ups. If you've done your homework, this is the easy option. If someone's pointing out something you haven't thought of, say you haven't thought of that, and offer to follow up with them. Congratulations! You've successfully found one of the most tangible outcomes of holding these sessions: a high-value connection with an engaged team member.

Option #2 is just as straightforward on paper. It's not an answer many leaders like to give, however, as the pageantry innate to most corporate hierarchies generally assumes the senior person in the room to be all-knowing. What you're actually doing here is letting people know that you've really heard the question and you're not going to bullshit them by guessing. The ideal answer usually takes the form of "I don't know, but let me find out and get back to you", or "I don't know, let's talk after and we can find out".

Option #3 is the tricky one. The premise of giving this kind of answer is that it's more credible to say you're not answering than to use smoke and mirrors to try to steer people away from the subject. Some common reasons not to answer a question is that it's information private to an individual, it's information that's privileged (i.e. "I could answer but it would make you all Insiders and you probably don't want that"), or that it's information where answering would be worse for team cohesion than assuaging some folks' curiosity. I've been asked questions in AMAs related to individual performance of folks on the team, which are straightforward "that's not my information to give" answers. I've also had 'escalations' of decisions made at the team level, where it's appropriate to say something like "I'm not here to overrule decisions you don't like, someone else owns this decision and it's been devolved to them entirely".

Tip: A Note on Traps

You're likely going to run into questions that are designed to make you say something that'll either get you in "trouble", or make you take a firm position on something you don't need to. Always ask clarifying questions if you feel things are a bit vague.

I once joined a team where in my first regular AMA, I was asked "What is your opinion on monorepos?". This may seem like a benign question on its face, but the fact that someone chose to ask that means that chances are there's a monorepo holy war happening somewhere on the team and I'm being asked to weigh in. My answer was essentially "I will not referee your nerd fight", and this was Option #3 above.

On a more serious note, I've also been asked if a new female exec on my team is "able" to take on the teams she was hired to take on. Again, I could very easily (and emphatically) answer such a question, but it was more valuable to take Option #3 and say something along the lines of "I'm not answering that because I know what you're doing, knock it off."

Tip: Anonymous Questions

I used to take anonymous questions, but don't do that any more. You get more questions that folks would otherwise not ask, but the quality can vary massively (the above question about a female exec was anonymous, in case it wasn't obvious). What worked instead was to nominate some folks in the organisation as proxies for folks who weren't comfortable asking questions themselves, for whatever reason. These proxies weren't necessarily very senior folks, just people on the team who were considered safer options. The reasons for wanting to proxy a question don't really matter; it can be tempting for a leader to say "No anonymous or proxied questions because this is a safe space", but that's not strictly true. Also, perception is reality, as we've established.

Answers Aren't just Answers

A side-effect of some of these answers you might give (especially Options #2 and #3) is to let people know your style, and your mind on things. The quality and timbre of questions tends to adjust over time. This means that AMA questions are 'messier' than curated FAQs or simple top-down announcements. This can be a little intimidating, but is an investment in your credibility as a leader, and in your team understanding your mind on things. Whether we like it or not, teams tend to quietly imitate the style of their leadership. Over time, a culture of openness and of consequence-free questioning of the status quo more than makes up for any initial (or ongoing!) awkward moments. Like any investment, the key is to stick with it.

Plus Ça Change Management: 20 Years of SRE

2024-07-19T10:54:00+01:00

On July 19th 2004, I spent my first day at Google. I showed up at the Datacenter in Dublin, since that's apparently where SREs and production folks were going to sit. There wasn't a corp network connection, because this was Google in 2004. A couple of days later, I moved my stuff to Barrow Street and commandeered a desk. A few weeks after that, someone from facilities asked "Hey, isn't your team based in the DC?". "Nope", I said. They shrugged, and updated some spreadsheets, and then SRE was based in Barrow Street. Again, this was Google in 2004. We were making it up as we went along, in the best possible way.

I could claim I was doing SRE-adjacent things before this, as I suspect could many people, but I'm going with that day as when I became an SRE. The function was still figuring itself out; in many ways it still is. My work with Busy Teams as a freelancer isn't "SRE work" on paper, but in practice, it very much is. Resilience, defense in depth, common sense, backup plans for backup plans. Across industry, SRE/Reliability/Devops/ProdEng/Whatevsies is many things to many people. So, in the spirit of the core of the function, I've been thinking a little about uncomfortable gaps in capability. What have we not figured out yet?

Here's my short list of boiling hot takes:

Most companies haven't figured out how much they steady-state care about reliability. I expand on this in "6 Reasons you Don't need an SRE Team".
Assuming we've figured out that we care, we still can't agree how hard a problem this is. SRE (and traditional ops) begat DevOps, which at its core has the premise that busy developers could side-gig the production bits, and you can just wave your hand and say "You build it, you run it" and that'll cover you. This isn't true today, and was even less true when 'DevOps' started being a thing. If I squint my eyes, I can see a course correction happening with "Production Engineering", which has at its premise a lot more sensible of an acknowledgment that this whole area is hard (as in, requires smartness, innovation, and Real Engineering(tm)) as opposed to difficult (as in, it's boring and I don't want to do it). These are glacial shifts; it's taken more than 20 years for us to go in this circle back to acknowledgment that this is a real and specialised set of problems.
We actually run less and less of our own infra, and practices need updating. For a brief period before AI was suddenly what all startups are about, there were (and still are!) great startups producing big parts of your tool-chain as SaaS/PaaS. This is great for not having to build in in-house expertise in that particular area, but can leave you dead in the water if you don't have a good strategy around vendor management. I spoke about this a bit at SRECon EMEA 2023, and I do feel like a lot of our practices involve declaring an outsourced part of our tool-chain to be an opaque cuboid, and then not having defense in depth for when it goes away (temporarily or permanently). Many of the SRE practices as set out in various books/articles kind if assume you own your whole stack. This is becoming mostly untrue, and we are currently a Frog of Moderately Troubling Temperature here.

Anyway. It's Friday, it's 22 degrees outside in Dublin, and it's time to go enjoy the next 20. No shortage of things to do.

Production, SRE and the Architecture of the Built Environment

2024-02-15T16:23:00+00:00

Recycling old talk proposals for fun and...that's it!

Here's an amalgamation of the proposal text and some notes from a talk I proposed for SRECon EMEA last year -- I'll be talking about this to the Dublin SRE meetup in a couple of weeks, so here it is while the dust is dusted off.

Production, SRE and the Architecture of the Built Environment

6 Reasons You Don't Need an SRE Team

2023-06-21T05:40:00+01:00

The last several years have seen a huge upsurge in the popularity of the DevOps/SRE/Production Engineering model, with companies large and small adopting some of the practices and mindsets. One of the principal lessons many of these organisations (hopefully) learned was that it's close to impossible to adopt the SRE model as described by Google in the the first SRE book. It's a good approach to take on board the parts of the book that work for you, and to actively triage your time, energy and effort.

However, one premise doesn't see a lot of investigation or introspection -- that is, whether you should even have an SRE team at all. The existence of SRE (or "DevOps", or "Production Engineering", or "Platform Trust", or any of the other taxonomic manoeuvres I've seen) is still treated somewhat as a given.

This can be for a number of reasons -- there's a pre-existing "Operations" team, there's a strong leadership advocate, or the skills exist in the organisation and it's considered good to consolidate them. It can also often be for less good reasons; pure organisational momentum, a desire to silo-ise toil, and for reactionary reasons.

The arguments for having an SRE team can often seem obvious: Everyone likes having reliable systems; it's hard to argue against that. Everyone also likes someone else doing the hard bits. So, let's cover the counterpoint: 6 Reasons you shouldn't have an SRE team.

1. You're not Google

Even though Google's approaches to parts of the problem space are battle-tested and can help you with your approach, you're still not Google. You're not even Google in 2004.

As someone who was at Google in 2004, I will let you in on the secret sauce that made a large SRE team at Google a good idea.

Intractable Problems

Two factors drove the near-intractability of the situation Google found itself in around this time: Scale and Innovation. These sound like lovely soundbites, but the truth of the matter was that nobody was doing anything even remotely like what Google were doing at this time. Nobody had the same requirements.

In my last gig before Google, we were using nagios and mon. None of these sharded in any way that would work for us; they were designed for monitoring well-understood things about a few hundred hosts. At the time, Google had a couple of hundred thousand hosts, and were adding thousands more per month. Nothing in either the OSS world or that you could buy was even going to come close. There was no Prometheus, no Docker, no Terraform, and these wouldn't exist in any meaningful way for another 10+ years.

We had to build, deploy and somehow keep together things at a scale and complexity (related to the tools available) that we had never before seen in our lifetimes and likely will never see again. This meant that simply hiring buildings full of operators wasn't going to work at all. This was plainly obvious -- even back of envelope calculations showed a scaling vector that meant we'd run out of humans in pretty short order.

Unlimited Money

I'm not a finance person, but it's difficult to express just how much Google was able to invest around the software ecosystem. Everything from the hardware and cooling to the monitoring software and job scheduler was built in-house. This was mostly because of the "Intractable Problems" thing, but also because we could afford to. In many cases, we'd be fools not to; the industry in certain areas wasn't moving fast enough, or even in the right direction for us. Compartmentalising the big-picture reliability problems into a group that we could build or hire the best SREs on the planet into was a no-brainer.

To briefly get off the Google fanboy train and go back to the crux of the point: You're not Google. I don't mean that your product isn't a great product, or that you don't have real problems. I'm saying that the criteria you use to evaluate whether you need a full-blown SRE presence should be specific to you. In your checklist of things to potentially adopt from Google’s model, include an item about having a separate team at all.

(Also, you're not Google in 2004. If I want a reliable sharded database with global consistency today, I can buy it and have it today, using money. This was not even close to true in 2004)

2. You don't care that much about reliability.

It's very easy to say you care about reliability, and very difficult to figure out and assert how much you actually care. Even if you have a good set of SLIs and SLOs, that's not the whole picture of 'how much' you care. Is reliability so important to you that you need a whole entire team just to care about it? What about testing? Product Research? Customer feedback loops? Why did SRE get to the top of the list?

It's easy to assert that near-perfect reliability will drive user retention. It's very difficult to be so sure of your product's use case that you can put a number on how much unreliability you can tolerate. Saying "We're not a stock exchange, our users will tolerate 5 minutes of downtime every so often" is a strong thing to be able to assert; stronger and more rigorous than saying you need eleventy nines for your applications at all times.

This quote from Steven Levy's book on how earlier Google worked epitomises what was heard loud and clear throughout engineering. I once had Eric Schmidt visit my team's weekly meeting (I was on GMail SRE at the time), and the money quote was his response to a question about prioritising reliability vs. product features -- "I will choose reliability, every time". We framed that quote in our team area. Even to the jaded sysadmins and hackers we often saw ourselves as at the time, it was a clear message that reliability was a product feature, all the way from co-founder to CEO to us.

Is your CEO saying these things? Even behind closed doors? What about when you're under pressure to ship features?

The extent to which Google really, genuinely cared about reliability and made the company-level investment in it can often be forgotten - think about if you're there with them.

3. You're not sure what the team should do or own.

If you’re not able to very succinctly explain and have everyone understand what a team like SRE is there for, then the ambiguity and ability to re-legislate who does what is often quietly seen as an advantage.

Some places have this right; many places don't. If a team is chartered and has a remit, that remit should be set in stone and the requisite effort put in place to maintain that knowledge. In my talk at SRECon Europe 2022 I go into a bit of detail about stakeholders. It's been my personal experience that Stakeholders (i.e. product owners, dev partners, company leadership) either misunderstand or make it their business to misunderstand what an SRE group should be doing..

Even at Google (that wonderful flawless bastion of enlightened thought on the matter), I routinely ran into folks at VP level and above who didn't really know where their team's remit ended and SRE began, and really weren't inclined to learn. Free labour is free labour.

If you're not 100% sure what SRE (or any team, for that matter!) exists for, what it will do, and why it's a separate team, then you should strongly consider whether it being a separate team is a good idea.

4. You're doing it to avoid internalising inconvenient truths

Understanding reliability is everyone's job

If you, or engineering leads who work in your org don't think so, then hiring a separate team to care about it isn't going to help.

There is a continuing meme within software that I personally don't understand -- the idea that software can be produced to spec and then disappears into the ether. It existed to an extent in the area of software being shipped on Floppies/CDs/DVDs/tarballs, where the turnaround on bug reports, releases, patches, etc. was measured in development cycles. However, it has somehow survived into the era of there being exactly one running installation of your software that you care about.

I can understand how a team might shard the effort of day-to-day running of their software; I have less time for the idea that the entire ecosystem of care can be put on another team.

Low-value work

Reliability engineering is widely viewed as low-value work. Even if leadership is bought in, the meme is extremely prevalent at all levels of product engineering.

The usual response from many folks in Ops and SRE roles would be to say that it's not -- it's crucial to system reliability and customer trust, and so forth.

The good news is that you can both be right!

Further to my article on oncall being a waste of time, I'm here to make a humble request that people say this quiet part out loud. If it's the honest assessment of an engineer or leader that work is of low value, that actually forms the basis of a very interesting conversation to be had about priority and how this toil gets looked after. Some of the most stressed and least valuable SRE engagements I've seen were ones where people were talking past each other, with nobody acknowledging what everyone knew; that nobody wants to do this work, and it should be eliminated. The convenient presence of an SRE team means the conversation can often go around in circles. Speaking of which...

5. Your SRE team might be a red herring

Further to the above post about low-value work; another anti-pattern which means you should be very sure about your SRE team's remit is that they are a convenient outlet for lack of planning or responsibility on the part of other groups.

Even in a "you build it, you run it" shop, it's possible to indefinitely delay and defer platform reliability or modernisation work, on the grounds that an SRE team should either be doing this work wholesale, or assisting. It's a very easy position to take, that has the side-effect of deflecting responsibility from a product or service owner onto an SRE group. I've seen cases where the SRE group hasn't even been asked to do the work, this deflecting responsibility onto...nobody.

Would this happen if you were able to be 100% sure that everyone at your company completely understands SREs remit and has no issues with it? No!

Would it happen if you didn't have an SRE team at all? Also No!

6. You got a big fright

We've established that it's difficult to put your finger on how much you care about reliability. Even if you have done the work and really have a good handle on this; you may be one giant outage away from throwing that all away on optics.

As a thought experiment: go and corner your nearest C[TE]O and ask them how much they care about reliability. It's probably a lot, right?

Now, go ask them again the day after a multiple-hour outage. Now it's definitely a lot!

SRE as a brand is a double-edged sword -- it can be seen as a panacea for reliability problems, when often it's a more intrinsic cultural investment. I have personally seen SRE groups appear almost overnight, or have money/headcount thrown at them because we had some big outages and this is an audacious hail-mary pass of a move that really shows you care about reliability. If you're not careful, you could be telling your customers you care about reliability, while telling your product developers and leadership that they don't have to.

Conclusion

The model of large SRE teams covering many services in a vague and nebulous way that's open to repeated re-interpretation is mostly a side-effect of (a) cargo-culting the building of these large groups, or (b) retrofitting SRE/DevOps onto existing groups without the company-wide reliability focus required (or the fortitude to decide you didn't need such a large group to do SRE).

Most of the reasons Google built a large SRE presence were related to being better-equipped to throw money at all its problems than most companies in the last 100 years, and due to the utter absence of big parts of the software and hardware infrastructure needed to build reliable services.

The last 20 years have seen enormous advances in the SaaS and infrastructure software space. What was absent is rapidly becoming present; what was esoteric is rapidly becoming generic. To fully realise the SRE model is to reduce complexity, to make rational choices, to buy because you don't need to build.

The next stage in removing our production training wheels as an industry is to tear down the fence between SRE and Product Engineering, and make rational investments in reliability as a mindset, based on specific needs.

What Would It Take?

2023-05-12T15:48:00+01:00

One of the most constructive things I've found when facing a difficult work situation is to externalise and write things down -- I do quite well with talking things through with folks, and coaching and mentoring is something I get a lot of energy out of. When it's just me and my thoughts, the interlocutor is writing, and I often have to push myself to get that done.

When talking to others in this situation, I'd often frame it in simple "What would it take?" terms. In the case of whether to stick around in a job or stick with a project or idea, this would be "What would it take for you to quit today?", and the vital accompanying question of "What would it take to resolve all this, and to re-commit indefinitely?". In the case of a job, you're either going to quit, or you're not. Spending a lot of time agonising in between does pretty much everyone a disservice, yourself more than anyone. I think it's healthiest to always be going in the direction of one or the other, if you do feel there's work to be done there.

What I hadn't been realising is that this was a kind of abridged version of the spookily-named CIA Model used in formal coaching. Essentially, you're doing a bit of compartmentalising of issues in your own head into "Stuff I can fix", "Stuff I can affect", and "Stuff I just have to put up with". There's a lot to unpack there -- in some cases, there may be bright lines around ethics, capability and sometimes just bad timing that mean you have to self-select out of the situation. You may also decide that something you think you can fix is something you likely shouldn't, or that you need to think about if you should tolerate it long-term.

Conversely, taking a full checkpoint on what matters to you, and what the real situation is will lead you to clarity on next steps. It'd be a shame to not investigate options for how to move stuff from "I tolerate this" to "I'm okay with it", or better.

Most people take a little prodding to get there, and I'm no different. If there's no capture of what the sources of stress are and why they matter, we end up having to rely on the Amygdala, the "animal brain". It is very hard to reason logically when we think we're going to be eaten by a tiger. This is why we invented writing.

So, if you're at a point where you think you're phoning it in, or you're struggling to get the basics done; think about the theory first.

What about this situation do you directly Control?
What do you not control, but have some Influence over the outcome?
What parts are you going to have to Accept and deal with?

If you're able to successfully explore what the situation on the ground is, the slightly more practical parts that I've worked through with others and for myself are usually the more challenging practicalities:

What is the issue I'm dealing with?

Do I feel underappreciated and having my incentives change would fix it?
Do I feel like a co-worker is making my life difficult?
Is my employer doing a kind of business you don't like?
Am I not seeing enough customer conversions or sales?
All of these are valid. They all take up mental energy.

What would it take to resolve the issue and have me re-commit to the idea/project/job/etc. indefinitely? What's missing?

It costs mental energy to tolerate a situation, rather than having it be a fact of life that you're truly reconciled with.
Moving from tolerance to acceptance takes change; and it's almost never just rationalising the situation away in your head. Real change involves doing something you can't take back for free.

What would it take to give up and change direction completely? What's stopping me?

This is a challenging rhetorical question, but the actual answer is something you should try to write down. What would need to change in order for you to say "thus far, and no further" and do something more drastic than putitng up with the situation? What's your 'trapdoor'?
In each of these cases, you should consider making these things known, if you're able to be definite about them. This especially applies to business outcomes, where you might not be the only stakeholder.

This has worked for me in a long career at one employer, and for knowing when I'm able to make changes without compromising on core values, but also to provide an outlet for the Sunday Night Terrors when the Amygdala takes over. The right answer isn't always the most satisfying one, but it's easier once you put (figurative) pen to paper.

Oncall: An Equal-Opportunity Waste of Time

2022-11-04T21:13:00+00:00

I spent a number of days at SRECon 2022 in Amsterdam, and gave a talk that I'd had rattling around in my head for a wee while - roughtly based on a paper I left at Google when I left.

Laura was good enough to suggest it as a ;login: article, so here it is: Oncall: An Equal-Opportunity Waste of Time.

Working Deliberately with OKRs

2021-06-26T23:40:00+01:00

After being asked by a work colleague to say words about OKRs and if they're a good thing to adopt, I realised I had more words than fit in an email or short doc -- so, here's a slightly longer doc on my view of OKRs, tempered by 15 years or so of using them fairly successfully, and seeing them used...less successfully. They're one of those "Google does them so let's do them too" type things that seem like a no-brainer, but can often mask the right answer for where your practice is at.

Anyway, say hello to Working Deliberately with OKRs (with thanks to @ahidalgosre for casting a critical eye over it).

Bad Machinery Paper

2021-05-19T20:34:00+01:00

Thanks to my accomodating erstwhile colleagues, I'm able to publish the original paper (with minor redactions so it makes sense for an external audience). A version of this paper later became Chapter 29 of the SRE Book.

To add a little bit of extra colour, the document came out of some lessons learned on Storage SRE (specifically Bigtable/Colossus), that I ran directly back in the early 2010s -- The date I have on this document is 2016, but I know it was around long before that.

Storage SRE back then was something of an anomaly -- there were a number of teams that managed storage services (the above, but also some various other services, some of which where purely internal and never saw the light of day). To amalgamate these services meant that we potentially won a bunch of OMG synergy because of the base assumptions made in design that were unique to services that kept data. Google started out as a search company with a throwaway, re-scrape-able corpus (built on GFS, which wasn't designed to persist data), so it was natural for us to have a couple of goes at stateful services, and see what stuck.

So, while you might imagine that lumping services of a similar shape together is a good thing to reduce duplication of effort (and you'd be mostly right), other factors include service maturity, generalised attitude toward production, and even come down to the vagaries of the dev/SRE relationship, or of engineering leadership as a whole.

So around then, we had a Bigtable and Colossus service that generally worked remarkably well. We had challenges in observability, and in developing the tech around load-sharing of users on a shared compute/storage pool. This later became known as running a multi-single-tenant architecture, although at the time we saw it as fairly chaotic - not all the tooling had been done, and some of the assumptions and prior art we have the benefit of today wasn't a thing, yet.

While others looked at this as a purely technical problem, myself and a couple of key leads saw this as a problem of focus. Google in its earlier days had faced additional problems of scale like this -- where esentially we could keep digging, or we could put in place a set of attitudes and approaches that enabled us getting out of the hole we were in as far as interrupts are. Some of these problems fell into the "We will need 5000 people to edit config files" or "We will need more hard drives than humanity is making" variety, which really do need 'creative solutions').

One of the anti-patterns we observed at the time was a tendency among folks on both Dev and SRE teams to concentrate on speedy detection and resolution of issues (The 'Rocks with Eyes" method, as succinctly described at the time). Being able to know when things are busted is important -- but the followup is key. Oncall and interrupts are fundamentally a waste of time and should be minimised, so signing up for extra seems not a great approach.

I had a separate paper related to this one called something like 'email alerts are from the past' -- and my real-life experience of that was coaching some key team members through some of the above approaches. We had folks who felt that they needed to keep getting email alerts, as they had trained themselves to spot patterns, and believed they needed to keep doing this or we'd have more outages. You might step back from this a wee bit and recognise that this was a set of people who were inadvertently being profoundly disrespectful of their own time. It's not because they were bad people of course, it's because they were busy people, and firefighting had found its way into their muscle memory.

In the end (in this example), myself and senior TL on the team ended up unilaterally turning off email alerts. She sent me the code review, I approved it, done. It was clear that we weren't going to reach consensus on whether or not it was a good idea; and while I'm not a fan of this approach in general, it became clear we weren't going to get there (see also the 'consensus is nice' section). In this case, nothing happened. A few folks were briefly sad, but ended up with more time on the clock and more spoons.

That, to my mind, is the outcome of a successful strategy around interrupt management. The interrupts need to be addressed, and in a timely way -- but you as the team responsible hold the ability to do it in a way that doesn't treat people as machines. In addition, interupts are not a closed system - the interrupts you're getting a month from now need to be different from the ones you're getting today -- by finding a better set of first world problems.

But moreso, the way you give yourself the ability to step back and really address the root causes in a noisy system is to respect your time, reduce context switches, and give people both the well-clock time and the spoons to be able to do so.

Everybody's Free (To Use Their Best Judgement)

2016-09-26T11:22:00+01:00

This is something I wrote for new Googlers, drinking from the information hose. It's not formally part of the onboarding process, but enough people email me about it that I suspect it's passed around a lot informally.

(With Apologies to Mary Schmich)

Everybody's Free (To Use Their Best Judgment)

Conferences for Introverts

2016-08-20T17:29:00+01:00

This is a paper I wrote for consumption of people doing conferences and trainings at Google. I figured it was more generally applicable, so here it is: Conferences for Introverts