log.andvari.net

Disappointing People Early

2026-02-02T12:00:00+00:00

One of my favourite phrases to remind people of as I'm fumbling about the business of producing reliable systems is "We Should Disappoint People Early". I've gotten enough funny looks about this that I figure I should explain myself.

The 'polite' assumption most folks make is that we should aim to never disappoint people at all. However, the wisdom is in the timing. If disappointment is coming (and it usually is, somewhere), you want it to arrive early enough that everyone can adjust, rather than late enough to cause real damage.

This applies everywhere: SLOs, product roadmaps, support response times, vendor relationships, and most critically, to the implicit promises we make when we don't say anything at all.

The Implicit Promise Problem

My erstwhile colleague Niall Murphy wrote an excellent piece on implicit SLOs and their dangers that crystallises something I've seen over and over. The short version: your users are already forming expectations about your service's reliability, whether you tell them what to expect or not.

If your service has been running at eleventy nines for the past year, your customers have noticed. They've built systems that depend on it. They've made architectural decisions predicated on "this thing is always up." They've stopped building fallbacks. You've made an implicit promise, and you probably didn't mean to.

The problem comes when you need to break that promise. Maybe there's cost-cutting. Maybe there's a re-architecture. Maybe you're just doing some planned maintenance that you've been putting off. Suddenly, you're at three nines instead of eleventy, and your customers are furious -- not because three nines is unreasonable, but because you violated an expectation they'd built up over time.

As Niall puts it: even if you smoothly manage the transition, if you fail 100x more often than you previously did, people are going to notice.

Why Stating an SLO is Better Than Not Stating One

The SRE book makes this point clearly: the business must establish what the availability target is for the system. Not the SRE team, not the platform engineers -- the business, informed by what users will actually tolerate and what makes commercial sense.

Ben Treynor's observation that "100% is the wrong reliability target for basically everything" is foundational here. You're not running a pacemaker. You get to decide what's reasonable, and more importantly, you get to tell people.

The alternative -- not stating an SLO -- is worse in every way. You've still made a promise; you just don't know what it is. Your customers have inferred one from your historical performance, and it's probably higher than you'd have chosen. You now have all the obligations of a promise without any of the negotiation that should have gone into making it.

You have to make trade-offs. You can't make trade-offs against a number you haven't decided on. When you decide on that number, you then need to staple it to people's faces and let them emote about it early.

The Same Pattern in Roadmaps

Product roadmaps suffer from exactly the same dynamic. A product (or platform!) team should not make a promise until they've conducted enough discovery work to understand what's truly required, and the only people who can make such a promise are the team responsible for delivering it. The corollary is stark -- if you're making roadmap commitments before you understand what delivery entails, you're setting up a future betrayal.

This is the implicit promise problem wearing different clothes. If you show a customer a roadmap with specific features and dates, you've made a promise. If the business context changes -- and it will -- you're now in the position of either delivering something suboptimal because you committed to it, or "breaking your promise" by adapting to reality.

The fix isn't to hide your roadmap. The fix is to disappoint people early by being explicit about the nature of roadmaps: they're current best guesses, subject to change, and the further out you look, the fuzzier they get. Folks who've learned this lesson put "subject to change" disclaimers on everything, use confidence percentages, and replace hard dates with buckets like "Now," "Next," and "Later." Obviously it's possible to hedge too much, but a certain amount of expectation-setting is healthy.

When roadmap planning is treated as a rigid forecast, it creates pressure and distrust. When it's treated as a dynamic, communication-first process, it builds trust and momentum -- even when timelines shift. The early disappointment of "this might change" is infinitely preferable to the late disappointment of "you promised."

Just like with outages, good partners and customers will understand that software and the means by which we make software are terrible, complicated, unpredictable beasts. A good customer and partner should be more interested in your response to an outage than they are on beating you over the head with it. Similarly, a good customer partner will appreciate you keeping them up to date on delivery and being honest about outcomes, rather than doing deadline gymnastics.

Vendor Relationships and the Shared Responsibility Trap

There's a particularly insidious version of this problem that plays out when companies move from on-premises infrastructure to PaaS or SaaS, in either Security or for provision of a dependency. The cloud shared responsibility model is supposed to be clear: the provider is responsible for security of the cloud, while the customer is responsible for security in the cloud. In practice, it's a mess of unstated expectations.

Gartner famously predicted that through 2025, 99% of cloud security failures will be the customer's fault. This sounds like finger-pointing, but it reflects a deeper truth: when companies adopt cloud services, they often expect the vendor to take over responsibilities that were never part of the deal. As a customer, you're intentionally abdicating duty of care about the details, in return for paying money for an outcome. It can be difficult to let go of the parts you've farmed out to your vendor.

When vendors aren't explicit about what they won't do, customers fill in the gaps with optimism. So, when something breaks, the disappointment isn't just about the outage -- it's about betrayed expectations that were never actually set. The NSA and CISA published guidance in March 2024 specifically because so many organisations were accelerating their cloud journeys "without proper planning and an appreciation for shared responsibilities."

This creates an ugly dynamic. If you're a vendor and you're not telling customers what failures to expect, they're going to want the full list of all things that can go wrong. Their nervousness translates into demands for breadth-first total coverage of all eventualities. But there are always failures you won't be able to enumerate specifically -- the unknown unknowns, the novel combinations, the things that have never happened before.

The answer is to disappoint early. Be explicit about what's in scope and what isn't. Document the failure modes you do know about. Make clear that you can't possibly predict everything, and explain what happens when the unexpected occurs, and why that makes you a safe pair of hands. The discomfort of that conversation now is vastly preferable to a customer discovering during an incident that their assumptions about your service were wrong. It also precludes a lot of demands from those same customers that are driven by vibes and nervousness at not feeling they know enough about potential failure modes and outcomes.

Support Response Times: The Hidden SLO

Every support team has an SLO, whether they've stated one or not. If you typically respond to tickets within 4 hours, your customers have noticed. They've calibrated their workflows around it. The first time you take 24 hours, they'll be furious -- not because 24 hours is unreasonable in isolation, but because you violated the expectation you accidentally created. This applies also to customers who only open the occasional support request - the feigned surprise that they don't get a 15-minutes response when the SLA is an hour (or more) will be familiar to anyone who's been on a customer support rotation. Customers who open tickets all the time will be more familiar with the expected performance because they've been told more recently.

This is where "disappoint early" becomes almost literal. If you know you can't offer priority support between midnight and 5 AM, say so upfront. If critical issues get faster response than routine questions, publish your priority matrix. If your support capacity means occasional delays during high-volume periods, tell people before they need help, not after they've been waiting.

Internal Customers Too

This isn't just about external customers. The same dynamic plays out inside your organisation, often more acutely.

If you're a platform team that offers services internally, your internal customers (the product teams) are forming expectations about your services just as external customers would. If you've been accidentally excellent at something -- say, keeping your deployment pipeline at sub-minute latencies -- you've now made an implicit promise. The moment that slips to five minutes because you've added some necessary safety checks, you'll hear about it.

The conversation is much easier if you've stated up front: "We target 95th percentile deploy times under 5 minutes." Now there's a conversation to be had. Maybe 5 minutes isn't good enough for some use cases. Maybe it's plenty. But at least everyone knows what they're working with.

Similarly with stakeholders and leadership. If you're asked "when will this be done?" and you say "soon" or "we're working on it," you've allowed them to fill in their own number. That number will be wrong, and it will be too optimistic. The disappointment arrives later, bigger, and with compound interest.

Saying "this will take four weeks" might disappoint someone today. But that disappointment is manageable. They can plan around it. They can reprioritise. They can have the argument about whether four weeks is acceptable now, when there's still time to do something about it.

The Discomfort is the Point

Stating an SLO feels uncomfortable precisely because it's a commitment. The same is true for publishing a support response time matrix, or putting "subject to change" on a roadmap, or listing the failure modes your SaaS platform won't protect against. Each of these forces you to confront the reality that you can't be all things to all people. It makes explicit something that was previously implicit and negotiable.

That discomfort is valuable. It's the discomfort of honesty. It's far preferable to the discomfort of a customer discovering during an outage that your service wasn't as reliable as they'd assumed, or finding out during an incident that their cloud vendor doesn't cover what they thought it did, or learning that the feature they were counting on got deprioritised.

The core of "disappoint early" is this: small disappointments now prevent large disappointments later. An SLO that seems modest compared to your historical performance might disappoint a customer today. A roadmap that says "this might change" feels less confident than one with firm dates. A vendor contract that explicitly lists what's out of scope seems less comprehensive than one that doesn't mention limitations at all.

But in each case, the explicit version creates space for honest conversations about what they actually need, what you can actually deliver, and what happens when those don't align. The implicit version creates the illusion of agreement that shatters on first contact with reality.

So.

If you're running a service without a stated SLO, you've already made a promise -- you just don't know what it is. Your customers have inferred one, and it's probably more optimistic than you'd like.

If you're sharing a roadmap without caveats about uncertainty, you've created expectations you may not be able to meet. You've also put yourself in a position where you've no way to climb down if the situation changes.

If you're a vendor who hasn't explicitly documented what failures customers should expect, they're filling in the blanks with assumptions that will turn into accusations when something goes wrong.

If you're working with stakeholders without explicit expectations about timelines, support responsiveness, or capabilities, you've allowed them to assume. Those assumptions will bite you.

The fix is the same in every case: have the uncomfortable conversation now. State what you can actually commit to. Document what's out of scope. Put confidence levels on your forecasts. Publish your priority matrix. Be boringly, repetitively explicit about constraints and limitations.

It's often a good idea to overcompensate when communicating constraints or expectations that may be 'disappointing.' The customer who understands your limitations upfront is a customer who can plan around them. The customer who discovers them during a crisis loses trust, a thing that is hard-won and sometimes impossible to repair.

It's paradoxical, but the path to being a more reliable partner -- to your customers, your stakeholders, your colleagues -- runs directly through being willing to disappoint them sooner.

Chasing Boring at Just the Right Speed

2026-01-22T12:00:00+00:00

Asking the Right Questions

A while back when I was looking for a fulltime gig (and when I was contracting, of course), I had the opportunity to do a bunch of interviews (including the kind of informal interviews you do to get a new contracting client). One of my boilerplate questions when asked "Any questions for me?" has always been "What does success look like for the person in this role?". I like to ask it of everyone, even folks who'd be reporting to me or folks far away on the org chart. It gives you a good insight into how stitched together the team is on the important bits, I find.

In the majority of cases, I tended to get at least one answer that boils down to "Fewer Outages, Lower MTTR".

"Fewer Outages", to me, is a weird one. I tend to respond to a "There are too many outages" complaint by asking "Well, how many is too many?". Just like 100% is the wrong SLO target for basically everything, "None" isn't the right target for number of outages. You take a reasonable and data-informed swag and aim for that. In the vast majority of cases, you're not running a nuclear power plant or a circulatory system, so you get to really think about what number is right.

Similarly, for MTTR (Mean Time To Recovery, or 'how long in seconds it takes you to mitigate the effect of an outage') the last while has seen a "race to the bottom". There are entire companies whose premise for what they'll do for you (and charge you a handsome fee, to boot) is lower your MTTR. They'll generally do it using an AI chatbot (and strand your systems knowledge and capability inside that same bot), but that's neither here nor there - the main gist is "MTTR is your problem", and it's a tempting one to latch onto. Easy to measure, easy to be seen to be grumpy about.

Doing it the Fastestest

The problem is that MTTR, as a primary focus, is a red herring. It's measuring the wrong thing; or at least, it's measuring something that's downstream of what you actually want.

If you're having the same kinds of outages over and over, getting really fast at recovering from them is like getting really good at bailing water out of a leaky boat. Sure, you're staying afloat, but water coming in is a bad sign, and you built the boat in the first place so should know what's up.

The real purpose of good incident practice isn't to get fast at recovery. It's to feed back what you learn into your software development lifecycle. Every incident is information. Every outage is telling you something about where your systems, processes, or assumptions are wrong. The goal isn't to recover quickly (though you should); it's to ensure you don't have to recover from the same thing twice.

If you wanted to be a little more philosophical about it (in terms your boss would probably hate), you could say that Outages (like all feedback) are a Gift. You pick yourself up, dust yourself off, and see what there is to learn from the (sometimes pretty shitty) experience.

I waxed slightly lyrical recently about Cloudflare's writeup on the outage they had in November 2025. The part that was valuable to me was not a power fantasy about how quickly everyone scrambled and dropped everything and mitigated the outage - in fact, I only checked just now how long the outage lasted (3 hours). I'm not concerned about those 3 hours. Less would be nice, but the real outcome is much more valuable. I'm not even a current Cloudflare customer, but the fact that I know what caused the outage, and I know things have changed means more than a much shorter outage, followed by "Sorry customers, it's very technical, won't happen again (probably)".

Doing it The Bestest

When I talk to teams about the point of good incident management (primarily retrospectives), I generally say that the point of doing them is twofold. That is:

Avoiding Repeats

If you're seeing repeat incidents -- the same service falling over the same way, the same capacity issues, the same configuration mistakes -- then your incident response process isn't doing its job, no matter how good your MTTR looks. You're optimising for the wrong end of the pipeline.

When you focus too hard on MTTR, you create incentives that work against learning. Teams get good at quick fixes and workarounds. They get good at restarting services and clearing queues. They don't necessarily get good at asking "why did this happen, and what systemic change prevents it from happening again?"

When I worked freelance, I ran into at least one shop that used a managed service partner. Their first instinct was to kick things, seemingly at random. Restart the service. Reboot the machine. Reset the network device.

Their MTRR numbers were amazing.

Generally when a particular service fell over, it was back within the hour. However, it was falling over every couple of weeks. When I probed for more information on contributing factors, I got crickets. No followups, no logs, no learning. Clearly there's some slider between "mitigate" and "understand" that makes them somewhat mutually exclusive at either end of the slider.

Raising All Boats

Post-incident reviews (or postmortems, or whatever you're calling them this week) should be producing action items that feed into your roadmap. Not just "add better monitoring" (though sure, maybe), but real changes: fix the race condition, add the circuit breaker, change the deployment process. Do a priority swap so we fix something we've identified as real, instead of adding additional whizz-bang that may be more speculative in terms of how happy it makes customers. For bonus points, involve your customer support friends to make sure what you're inserting into the SDLC is actually going to move the needle for customers.

The measure of a mature incident response process isn't how fast you recover. It's how rarely you see the same incident twice, and how much of the learning from your outages sticks, in the form of enduring shifts to how you write code and design systems. It's how often your outages are genuinely novel -- new and interesting failures that teach you something new about your systems, and real action on then finding yourself a better class of first-world problems.

So

So yes, resolve outages quickly. Absolutely. But don't let MTTR become the thing you're optimising for. The goal is to build systems and processes where you're constantly learning and improving, not systems where you're just really efficient at fighting the same fires over and over.

Your incidents should be novel, and retro outcomes should be real. If they're not, your incident process is failing you, regardless of what your status page says.

Adrenaline In Moderation, Please

2025-03-03T17:34:00+00:00

One of the more impactful bits of "DEI work" we (ohai Sara Smollett) did at the G was helping to re-write our job descriptions for SRE (I later re-wrote the internal job ladder for SRE, but that's probably not as directly related).

Circa 2010 or so, our JDs needed a bit of a refresh, and they were pretty typical of the time - they selected for a kind of exceptionalism that is kind of gross to many folks. Words like "Rockstar" and "Adrenalin Junkie" and "Code Jockey" and the like. Definitely a pre-cursor to the only slightly more benign "10x engineer", so we haven't gotten away from that sort of thinking, sadly.

Friends, certain phrases in your job descriptions make certain kinds of people self-select out. They select for certain backgrounds, and people with certain attitudes, and this is to the exclusion of people who are just as good and better at their jobs that balk at filling a magical role you're using weird hyperbole to describe. You're ideally trying to encourage qualified people to apply, by describing the real requirements and real outcomes of the job. I've specifically and repeatedly learned this from female engineers I've worked with, but it also applies culturally (there used to be an informal training given internally on "How To Always Say You Are Awesome Even If It Is Sometimes Lies (for Europeans)").

The same goes for inflated experience requirements. You might put 10+ years of Kubernetes experience on a JD today and hope it weeds out time-wasters, but all it does is weed out every single person who's not prepared to lie about their experience. This especially applies to people for whom being accurate in their speech and writing is important, such as our neurodivergent friends (or people otherwise formed that way).

Instead, use language that describes why the role is exciting or interesting. Be realistic in how you describe it. Be forthright in letting folks know the kind of candidates you want to talk to, and remember that your words matter when it comes to what's probably the first piece of prose your future co-workers see that's considered representative of who you are.

You don't even strictly need to call it DEI work, if that unfortunately matters somehow. Go take a look today and see what you can do.

Critical Thinking and DEI

2025-02-16T00:00:00+00:00

Part of my day-job as head of Engineering at Google Ireland was attending talks and events/Q&A for local interest groups like Engineers Ireland. I was reminded today of a panel I was on around diversity in engineering in Ireland, and my allegedly 'fascinating' comments at it. That's not the part I remember most vividly, though.

After the presentation of the report, there was a Q&A session, with some good but not unusual questions. At one point, someone up the back raised a hand, and asked:

"What is Critical Thinking?"

The panel kind of went a bit quiet, but the cogs and wheels were turning in my head. It was exactly the right question at the right time, and a lot of stuff in that very moment fell into place in my head. Probably one of the best Q&A questions I'd ever seen, let alone had the privilege to answer.

Critical thinking is the ability to inspect a problem from different perspectives, and most crucially, to be able to reject hypotheses without bias. You need to be able to take your bright idea, and discard it in place of a better one, without attachment. To do it, you need people who have different frames of reference, who have had different 'crucible moments' and who have different contexts culturally that allow us to respectfully disagree and offer a broad set of views.

Homogeneous teams simply can't do it. We like to imagine we can transcend our personal experiences, biases and ego, but it simply isn't the case. That, to me, is what came to form the core of my thoughts on Diversity, Equity and Inclusion (or at least, the Diversity part). It forms a key part of my philosophy on building good teams. You simply can't build a good team from folks who look, think, and have experienced the same. Homogeneous teams are weak teams.

Another example: much earlier in my career, I was part of a small group in a leadership training tasked with coming up with a simple means of telling each other the number on a price tag without speaking. One of the team asked "How do we say a comma?". Myself and another Irish person in our group didn't see the point; just spell out the digits, why put in commas?

Turns out in big parts of continental Europe, price tags have a comma between the euro amount and the cents, whereas in Ireland there's a period. This may seem like a facile example, but these are the sort of small assumptions that get made every day that lead to accessibility issues, bias and poor quality outcomes in products you build.

There are a lot of things happening right now that make this a hot button issue; I care deeply about the social justice aspects of the naked white supremacy, homophobia and xenophobia that informs the current US administration's policy shifts. My friends are being hurt.

However, another aspect of pushing back against this wrong-headed, unjust and deeply stupid shift is what also reminded me of something I got to say to James Damore in the brief time he was at Google after his stupid-ass memo (this got me put on an external list of 'SJWs', a point of particular pride for me).

I told him, "You're not just wrong, you're incorrect."

As well as the plain prejudice, racism and homophobia that informs the current policy shifts, it's also worth pointing out that they simply weaken capability. We're being asked to build weaker teams, with narrower mindsets and much less interesting approaches to work.

While we're resisting and addressing these changes as leadership, remember that pointing out real, tangible risks to business and capability are part of the arsenal.

Believable AMAs for Genuine Leaders

2025-01-13T00:00:00+00:00

Communication Scales with Scale

Communicating with your team scales up as your team grows - it's an experience not unlike the individual engineer to senior engineer to staff engineer path. As well as scaling up in terms of impact, the very method of how you do it changes.

You very quickly grow out of being able to do a full matrix of 1:1 conversations, likely before you even go beyond single digit numbers of people. Even with a smaller team, this method can get pretty exhausting for all involved quickly. It's also pretty lossy - rather than actively tracking what you've communicated to each person, it's not unlikely that you'll assume you've said something about team direction, strategy, etc to everyone, when you might not have. It's possible to track this more exactly, but it's probably best not to. This is one instance where after you open a new spreadsheet, it's probably time to have a quiet word with yourself and see if you need to change the method.

Next in the toolbox is usually the ubiquitous stand-up or team meeting - a much better way to make sure you've let the whole team know about something, and ideally to give them the chance to clarify anything. This is probably the most common way to 'filter' down either communications from the broader organisation, or directions just for this team.

Once you scale beyond a team, things get tricky. You want to filter a message to all of these folks, but also give them the opportunity to ask questions and fill in the blanks they might have, that often you haven't thought of. Google's TGIF meetings used to do this; I attended a number of these in person when I'd visit HQ, and seeing Larry and Sergey being surprised by questions was pretty routine. This isn't a bad thing -- you're not going to think of everything, and often you'll miss things that will be brought up by thoughtful questions. The answers here were usually pretty frank; this changed later on when TGIF answers got pretty routinely leaked, often in real-time.

When does it become time to do something similar for your own team? In my case, it was when I started managing teams in various different offices - I had teams first in Dublin and California, and later NYC, Sydney and Seattle. While I did do my fair share of aeroplane time, it quickly became unworkable to do local "town hall meetings", even virtually. I still did them when I visited offices, but I wanted these to be in addition to folks getting all the information they needed day-to-day.

Tip: Perception is Reality

If you put a message out to a group of people, and most of them think you meant a certain thing, then that's what you meant.

This is a common pitfall from leaders of all levels - I've seen everyone from line-level managers to VP+ everywhere make the mistake of communicating carelessly, and not following up when it becomes clear that folks misunderstood what they meant. There's a hard problem here for most leaders. You're a professional communicator, you're good at this, so it's very tempting to think something like "Well, if folks misunderstood then that's not my problem".

What just happened is that you said words and people parsed those words into meaning, and your words failed to get the meaning across. In fact, they got another meaning across that you didn't mean to, making that the reality for those people. If a senior person shows up and says something, and most of the team comes away thinking "Dave will fire us if we make mistakes", then that's what you said. You've got work to do, or learning to do, or ideally both. Don't let hubris make you skip this part.

Communicating a Thing Starts with Communicating It

How you actually make decisions is beyond the scope of this article - let's assume for a moment you've done a good job at having all the right folks do all the right things to make a decision, charter a project, or otherwise determine a path forward.

Perhaps you've gotten a few leads together for a day or two and determined your roadmap for a year, or decided to make some team changes, or something of that ilk. You've spent hours internalising all the factors involved, and ideally used good judgement and data to come to a decision. Why would you expect folks on your team outside the room to immediately understand why a decision is made, just because you announced it? That's assuming you announce it at all.

The primary mistakes I've seen people make when it comes to communicating direction and change is:

Not communicating the decision, direction or change at all.

This is extremely common. It is a shockingly common assumption by rooms full of smart people that knowledge and context hard-won in that room are somehow magically transplanted to everyone affected by the outcome. I try to dedicate some "How are we communicating this?" time at the end of leadership get-togethers for this exact reason. There is real, difficult work in communicating things to folks; step one is acknowledging this and planning for it.
Communicating a decision or change, without providing any means of asking for details, or otherwise questioning it.

Every organisation differs in how decisions are done. On one extreme, there's the autocratic "One person makes all decisions" model. On the other end is the apocryphal "Absolute Consensus on all decisions, always" model. Chances are you're somewhere in the middle. Someone on the team isn't going to be happy with any given direction or decision. They're going to have some opinions and (usually loaded) questions. You want to put yourself in a position where these questions get answered, ideally in public. Otherwise people will answer these questions themselves. You're probably not going to like those answers. There's really no downside to inviting questions and comment on directions and decisions, yet I've seen many otherwise smart folks think they can quietly implement a decision and maybe nobody will mind or notice.
Not owning the decision

It's always tempting to defer to some external factor (deadlines, budget, leadership) when saying why a decision was made. This is a tricky balance, because while most good leaders don't like deferring to "Because I/we say so" as the reason a direction is being taken, that's ultimately what's happening. Yes, there are always contributing factors, but you've taken those into account to make a high-quality decision, which you should stand over as your own. If external factors seem to force your hand in all directions you take, then you're not really making decisions at all. Again, perception is reality. People are smart and will spot this right away.

Tip: Timing and Heads-Up(s)

As a conscientious leader or manager, you are likely highly practised at the art of delayed gratification. Most people aren't. If you're announcing something, 'fast follow' doesn't mean in a week or two when you can schedule a Q&A/AMA session. Let your ability to schedule this kind of session inform when you announce. Don't announce things on Friday and do the session on Monday; people will spend the weekend inventing answers to their own questions that will be way more compelling than any boring sensible answers you might eventually give.

If there are folks who need to know about something in advance, because they're key stakeholders, or simply because people will ask them about it first and they don't like surprises, do give them a heads-up. If you're worried about leaks, then that's a good instinct. Do this as late as is practical. Things will leak, accept this as a fact of life and plan accordingly.

Making it Real: The AMA

This section sets out tips on AMA ("Ask Me Anything") sessions. These can be held in relation to a specific announced thing, or be semi-regular. I've often made a point of doing these regularly (and per-timezone, for folks at an offset time from me).

Tip: A note on 'Corp Speak'

The jig is up, corp-speakers. Everyone knows you're doing it. You haven't cracked the ability to make people believe you through clever use of words.

One thing that increased numbers of virtual town-halls have given us is the ability to leave without tripping over chairs or being noticed, and the ability to watch attendee numbers in real-time. I've personally watched viewer numbers drop off a cliff on these kinds of calls when a particularly 'polished' answer comes out. This is a shame, as it drives disengagement from the message coming from leadership, and even if the rest of the content is credible and sincere, this can derail people's engagement with the decisions or directions being communicated. At best, folks will just think less of the corp-speaker.

One clue that this is not the True Way is that most CEOs are not corp-speakers when they're at home. In a company-private setting they are generally quite open and communicate as openly as they can. Generally when they defer a question to a corp-speaker, it's because they need protection from Saying the Wrong Thing, or smell a Trap. More on Traps later.

Setting Expectations: The '3 Answers' Model

I've seen several variants of this model of AMA, so I can't take any credit for it. However, it has worked for me for many years across teams and orgs.

The basic model (which you should absolutely tell attendees in advance) is that attendees can ask anything. The answer, however, can take one of 3 forms.

I answer the question to the best of my ability.
I tell you that I don't know.
I tell you why I'm not telling you, to the best of my ability.

First, you'll notice there's no "I will bloviate for a few minutes and hope you look bewildered and stop asking" option. The intention is that the answers are credible and concise, and engage with the question in good faith.

Option #1 is straightforward, and ideally is the option you take most of the time. Answer the question. Be prepared for follow-ups. If you've done your homework, this is the easy option. If someone's pointing out something you haven't thought of, say you haven't thought of that, and offer to follow up with them. Congratulations! You've successfully found one of the most tangible outcomes of holding these sessions: a high-value connection with an engaged team member.

Option #2 is just as straightforward on paper. It's not an answer many leaders like to give, however, as the pageantry innate to most corporate hierarchies generally assumes the senior person in the room to be all-knowing. What you're actually doing here is letting people know that you've really heard the question and you're not going to bullshit them by guessing. The ideal answer usually takes the form of "I don't know, but let me find out and get back to you", or "I don't know, let's talk after and we can find out".

Option #3 is the tricky one. The premise of giving this kind of answer is that it's more credible to say you're not answering than to use smoke and mirrors to try to steer people away from the subject. Some common reasons not to answer a question is that it's information private to an individual, it's information that's privileged (i.e. "I could answer but it would make you all Insiders and you probably don't want that"), or that it's information where answering would be worse for team cohesion than assuaging some folks' curiosity. I've been asked questions in AMAs related to individual performance of folks on the team, which are straightforward "that's not my information to give" answers. I've also had 'escalations' of decisions made at the team level, where it's appropriate to say something like "I'm not here to overrule decisions you don't like, someone else owns this decision and it's been devolved to them entirely".

Tip: A Note on Traps

You're likely going to run into questions that are designed to make you say something that'll either get you in "trouble", or make you take a firm position on something you don't need to. Always ask clarifying questions if you feel things are a bit vague.

I once joined a team where in my first regular AMA, I was asked "What is your opinion on monorepos?". This may seem like a benign question on its face, but the fact that someone chose to ask that means that chances are there's a monorepo holy war happening somewhere on the team and I'm being asked to weigh in. My answer was essentially "I will not referee your nerd fight", and this was Option #3 above.

On a more serious note, I've also been asked if a new female exec on my team is "able" to take on the teams she was hired to take on. Again, I could very easily (and emphatically) answer such a question, but it was more valuable to take Option #3 and say something along the lines of "I'm not answering that because I know what you're doing, knock it off."

Tip: Anonymous Questions

I used to take anonymous questions, but don't do that any more. You get more questions that folks would otherwise not ask, but the quality can vary massively (the above question about a female exec was anonymous, in case it wasn't obvious). What worked instead was to nominate some folks in the organisation as proxies for folks who weren't comfortable asking questions themselves, for whatever reason. These proxies weren't necessarily very senior folks, just people on the team who were considered safer options. The reasons for wanting to proxy a question don't really matter; it can be tempting for a leader to say "No anonymous or proxied questions because this is a safe space", but that's not strictly true. Also, perception is reality, as we've established.

Answers Aren't just Answers

A side-effect of some of these answers you might give (especially Options #2 and #3) is to let people know your style, and your mind on things. The quality and timbre of questions tends to adjust over time. This means that AMA questions are 'messier' than curated FAQs or simple top-down announcements. This can be a little intimidating, but is an investment in your credibility as a leader, and in your team understanding your mind on things. Whether we like it or not, teams tend to quietly imitate the style of their leadership. Over time, a culture of openness and of consequence-free questioning of the status quo more than makes up for any initial (or ongoing!) awkward moments. Like any investment, the key is to stick with it.

October 14 2024

2024-10-14T15:09:00+01:00

Bit of a shuffle in what musical projects I'm working on - two groups I'm with have fizzled (at least, my involvement with them has) and one other is coming up.

One of the bits I've signed up for is playing samples on stage through front-of-house, so I've invested in a Boss RC-5 loop station.

These have seen a resurgence recently, solo artists have been using them pretty extensively to build up tracks (Ed Sheeran put out his own). The basic gist is that you can record a few bars of music through them, then have them repeat and play over them.

One of the other things they do is play back samples - modern ones like the RC-5 have storage you can drop .wav files of a certain type on there. In the case of the RC-5, it specifically wants certain formats -- for folks beating their heads against this, the cheat sheet is:

sox existingfile.wav -e floating-point -b 32 -r 44.1k looperfile.wav

Another bothersome thing about the RC-5 at least is the amount of fiddling to get loops/samples playing the way you want - setting tempo, repeat, the sample name, and so on. Of course, my first instinct is that I'd like to be able to backup and restore everything. Thankfully, you can do this -- if you plug the pedal into USB and go to Setup->Storage, it'll appear as a USB device. The storage is built-in, so swapping SD cards isn't a thing. You can then use rsync to back up and restore.

For setting up your setting and completely avoiding inputting text using a scroll wheel, you'll need to edit the XML file (what year is this?) at //ROLAND/DATA on the device. A lot of the fields are self-explanatory, the main bother being the Loop name -- this is done as an XML element of the ASCII codes of each letter in the name, like so:

```

"My Loop"

77 121 32 76 111 111 112 32 32 32 32 32 ```

This is a bit bothersome, so I took the opportunity to learn some more golang -- check out https://github.com/gerrowadat/looperutil for a wee utility for editing XML files and pretty-printing your setup.

Golang's XML parser is one of those patterns I'm not sure if I love or hate - make some structs that look like the DOM of the XML file, and "magic" happens. Works for my purposes, though.

If you're nervous, then looperutil xmlname "My Loop" will spit out the above, and you can paste it into your editor like a farmer.

Overall, this solidifies Roland/Boss as being still probably the kings of pedals that don't fuck around -- not only have they always been built like tanks, the ways to interact with them (be it stepping on them, or XML) still seems solid since I bought my first OC-2 in...1995 or so?

Now, it only remains to actually get out and play some gigs. As the serially defunct blog author said, "more on that Later"...

Plus Ça Change Management: 20 Years of SRE

2024-07-19T10:54:00+01:00

On July 19th 2004, I spent my first day at Google. I showed up at the Datacenter in Dublin, since that's apparently where SREs and production folks were going to sit. There wasn't a corp network connection, because this was Google in 2004. A couple of days later, I moved my stuff to Barrow Street and commandeered a desk. A few weeks after that, someone from facilities asked "Hey, isn't your team based in the DC?". "Nope", I said. They shrugged, and updated some spreadsheets, and then SRE was based in Barrow Street. Again, this was Google in 2004. We were making it up as we went along, in the best possible way.

I could claim I was doing SRE-adjacent things before this, as I suspect could many people, but I'm going with that day as when I became an SRE. The function was still figuring itself out; in many ways it still is. My work with Busy Teams as a freelancer isn't "SRE work" on paper, but in practice, it very much is. Resilience, defense in depth, common sense, backup plans for backup plans. Across industry, SRE/Reliability/Devops/ProdEng/Whatevsies is many things to many people. So, in the spirit of the core of the function, I've been thinking a little about uncomfortable gaps in capability. What have we not figured out yet?

Here's my short list of boiling hot takes:

Most companies haven't figured out how much they steady-state care about reliability. I expand on this in "6 Reasons you Don't need an SRE Team".
Assuming we've figured out that we care, we still can't agree how hard a problem this is. SRE (and traditional ops) begat DevOps, which at its core has the premise that busy developers could side-gig the production bits, and you can just wave your hand and say "You build it, you run it" and that'll cover you. This isn't true today, and was even less true when 'DevOps' started being a thing. If I squint my eyes, I can see a course correction happening with "Production Engineering", which has at its premise a lot more sensible of an acknowledgment that this whole area is hard (as in, requires smartness, innovation, and Real Engineering(tm)) as opposed to difficult (as in, it's boring and I don't want to do it). These are glacial shifts; it's taken more than 20 years for us to go in this circle back to acknowledgment that this is a real and specialised set of problems.
We actually run less and less of our own infra, and practices need updating. For a brief period before AI was suddenly what all startups are about, there were (and still are!) great startups producing big parts of your tool-chain as SaaS/PaaS. This is great for not having to build in in-house expertise in that particular area, but can leave you dead in the water if you don't have a good strategy around vendor management. I spoke about this a bit at SRECon EMEA 2023, and I do feel like a lot of our practices involve declaring an outsourced part of our tool-chain to be an opaque cuboid, and then not having defense in depth for when it goes away (temporarily or permanently). Many of the SRE practices as set out in various books/articles kind if assume you own your whole stack. This is becoming mostly untrue, and we are currently a Frog of Moderately Troubling Temperature here.

Anyway. It's Friday, it's 22 degrees outside in Dublin, and it's time to go enjoy the next 20. No shortage of things to do.

July 16 2024

2024-07-16T17:18:00+01:00

Going to be playing a bit of catch-up, since it's been a few months.

Work is still workin' -- I've got some coaching going on, and am still working mainly with one team on consulting. It's been fun times, and the gig is flexible enough that I've gone essentially part time for July and August. More time for...activities!

Since last time, I headed over to Morocco for a few weeks on the bike (shipped the bike to Malaga, and rode via the ferry around the more rural parts, including the earthquake-stricken parts).

It's definitely an eye-opener to see parts of the world that are still so rapidly developing -- someone who's done the trip a few times said that every time they come back, there's less gravel and more tarmac. This is obviously a good thing for most people, but it brings home that there are still parts of the world you should go see before they disappear. It was always on my list to go to Kyiv and Visit Pripyat; guess that's on hold.

Of course, the minute I got off the bike to do any other activities, I managed to nearly murder myself -- I flipped a dune buggy off a sand dune in the Sahara, which landed on my foot, breaking it (not too badly, but enough to be swollen and sore for a bit). No sign f an emergency department out there, so that had to wait a few days 'til it was home time.

That'll learn me to do things. It's July now (about 3 months later) so all back to close to 100%.

Plans for rest-of-year are to maybe update here more? And also keep myself busy with working with teams on fun projects.

April 01 2024

2024-04-01T11:54:00+01:00

A few months further into my "not working full-time and seeing how that goes" journey.

I've picked up a contract for the consulting end of my 'job' since January, that'll likely last til the summer or thereabouts -- it'd been fun doing the good and useful parts of the job I used to do, and being able to leave it alone when I leave work. Because I'm not 'full-time', I can take the odd day when I feel like it (not getting paid, of course), and the freedom of that is pretty good. I'd imagine I landed on my feet a bit with my first contract -- the sponsor is pretty chilled out about how much I work as long as the results we've talked about are happening. As with a lot of this stuff, I'm playing it by ear. Maybe it is normal?

I have a fairly...puritanical attitude to work, which it's hard to shake off. I occasionally get flashes of anxiety that I'm not in a big impressive tenured job that I can keep forever (Like I used to be, right? :-)). So I'm keeping an eye on the market as well, and if something really interesting comes up, I'll take a look. However, being able to do this for a while has been pretty good, especially since I actually picked up some work that's fun times. It's in IT, which isn't a world I've lived in at a technical level for a long, long time. However, the basics are similar -- busy teams, technical debt, trust, fear of change, and smart people keeping the lights on at the core that really do need a push in certain directions. Like with coaching, I get to do the fun parts.

Sometimes I wonder if I'd be better suited to being an IC -- if I've so successfully insured myself to the hassle-y parts of management that my skills there grew and tech stuff atrophied - I've been told I tend to underestimate my technical skills, so it's a thought in the back of my mind that perhaps at some future point I'll do IC again. Not urgent as a thought, but a thought nonetheless.

I've also found myself having to push myself to take time off -- even when public holidays come up (the team I'm working with aren't based in Ireland), I've had to remember and normalise not working. I'm heading to Morocco on the motorbike in a few weeks, so that'll cover me for time off for a while, right? :-)

Overall, I think I'm still learning things about myself that lay immobile for the longest time when I was working full-time - what I'm into, what motivates me, what my fears are, and a sense of normalcy about that being okay and being settled with it without absolutely everything being 'looked after'. A kind of corporate deprogramming, if you will.

Production, SRE and the Architecture of the Built Environment

2024-02-15T16:23:00+00:00

Recycling old talk proposals for fun and...that's it!

Here's an amalgamation of the proposal text and some notes from a talk I proposed for SRECon EMEA last year -- I'll be talking about this to the Dublin SRE meetup in a couple of weeks, so here it is while the dust is dusted off.

Production, SRE and the Architecture of the Built Environment

November 21 2023

2023-11-21T14:51:00+00:00

The last few weeks were a bit of a hive of activity. I went back to work, started a small business, and went from no-busy-at-all levels of work-related concentration to quite-busy-and-not-very-stressful-yet. I'm going to be doing more than just coaching, but that's for another time.

Earlier this year, I asked J to marry me, and she said alright I guess. We're both very low-key about this sort of thing, so had a wee ceremony as mandated by the state (8 minutes!) and caught a few days away, being fed to an astonishing degree.

J is mostly not online, so that's about all the detail y'alls get -- this is a very nice thing. She has been family for a few years at least by now, so it's tempting to describe this as a formality or delve too deep into this Not Being a Big Deal, No Really. But, it is. I like this lady a lot and now I get to like her forever. Pretty class.

WOMC: Nomad Variables as Secrets

2023-11-03T00:00:00+00:00

Standard disclaimer: Maybe don't do this in production? I hear Hashicorp also does a secret store :-)

Nomad 1.4.0 added support for 'variables', which are essentially key/value blobs protected by nomad's ACL system. As someone who had steadfastly avoided either running vault, or doing secrets properly, I knew an opportunity to do secrets improperly when I saw it, so here we are.

If you want a 'working' example, my nomad-homelab repo has the example for my SSL certs, where I populate nomad with letsencrypt SSL certs and give whatever jobs need them access.

Getting data into nomad Variables

Assuming you have access keys and auth set up, working with variables is pretty easy.

Each variable is referred to by a path, which doesn't mean much aside from tidiness, aside from in one way: If you create a variable named nomad/jobs/myjob, tasks in the myjob job have access to the variable.

Each variable is actually a set of key/value pairs, so if I wanted to create a variable for myjob and put a 'authtoken' key in there, I'd do something like:

nomad var put nomad/jobs/myjob authtoken=mySeCrEtAuThToKeN

You can then happily get the variable with nomad var get and so on. See file_to_nomad_var.sh for an example of how to load file contents into a nomad variable key (although it looks like nomad has since grown functionality for using - as the value on the command line to accept key contents via stdin).

Access Controls

Variables have the same kind of access controls that other nomad entities have -- via an acl policy. The short version of how to get this done is that you should write (and then either keep or throw away) an acl policy file. This generally looks like the following:

```

mypolicyfile.hcl

namespace "default" { variables { path "my/secret/stuff" { capabilities = ["read", "list"] } } } ```

You can then insert a policy for each job to give access to the my/secret/stuff variable with:

nomad acl policy apply -namespace default -job myjob myjob-secretstuff-policy - < mypolicyfile.hcl

The policy can be named anything, so known yourself out with the tidiness. If you'd like to see this being used in anger, see grant_job_access_to_cert.sh from my SSL example.

Accessing Variables from nomad Jobs

Variables are accessible from nomad jobs at invocation time -- that is, when the alloc is being built. So, the best way to access and use a variable is to write out its contents in a template within the alloc, so even if nomad itself goes away, the container still has everything it needs.

In the .hcl file for your job, you can dump a particular variable you have access to to a file in the secrets directory by using template directives within a task stanza:

template { data = "{{ with nomadVar \"my/secret/stuff\" }}{{ .authkey }}{{ end }}" destination = "secrets/authkey.txt" change_mode = "signal" change_signal = "SIGHUP" perms = 700 }

Some things to note here:

my/secret/stuff is the variable name, authkey is the key within that variable.
You can place the output file outside of secrets/ if you like, I guess.
You can of course have multiple template {} stanzas with differet variables and outputs/behaviour.
change_mode and change_signal determine what happens if the variable gets updated. In this case, the task gets HUP'd if someone updates the nomad variable. This can be handy for stuff like SSL certs.

Why not production?

Any number of reasons I can think of:

People with admin access to nomad have access to all variables. You probably don't want that in production.
I'm pretty sure nomad shunts all this content around via raft and I'm not sure how secure the data is at rest (or in transit). I'm gonna guess less so than vault :-)
There's a proper way to do this that isn't chintzy as frig and that the people who come after you are more likely to understand.

Other Thoughts

Nomad Variables are kind of like K8S ConfigMap, so I could probably abuse them to store actual configuration data pretty trivially -- for jobs that just have a single config file and everything else can get blown away with the container, this is probably fine. Very few of the apps I run at home are like that, though.

Offsiting DNS for Fun and Profit

2023-10-23T10:46:00+01:00

Further to my last post where I went away for a few weeks, that away-time featured a power cut at home. Things....did not come back cleanly, and required intervention to come back -- not least of which was DNS. Because of course.

My setup had been running DNS locally for the local zone, including delegating to consul for where nomad jobs are. So, I looked into getting rid of this -- being able to use any resolver to resolve things at home would be nice, and being able to just hand out an external resolver to most clients will also be good. I have redundant internet connections at home, so I'm going to explicitly make it so not being able to reach external resolvers is the falure mode, rather than recovering from a power cut :-)

Thankfully, the major cloud providers will...provide. Google Cloud DNS costs pennies a month and you can talk to it over an API, so I started fiddling around with that.

I'd been kind of looking around for yet another excuse to learn go, so I've made clouddns-sync to do the heavy lifting. It's still pretty janky and needs tests, but it works, for now.

The probably more useful parts for other people is the clouddns-sync getzonefile and clouddns-sync putzonefile fnctionality - I used this for getting my existing zone into Cloud DNS. It probably doesn't produce or consume all kinds of zone file fragments, but in my defense, it's DNS. Show me software that does.

The more me-specific part is updating records for nomad jobs -- I can do this one-off with clouddns-sync nomad_sync -- which does a one-off poll of where jobs and allocs are in a local nomad cluster, and then updates a zone in cloud DNS. I'll likely cron it for now and then see if I can do something smarter with running it in nomad itself and useing channels and proper montoring and all the good stuff you're meant to do.

All that's then left is delegating zones to gcloud's resolvers, and blowing away my local resolvers once I'm hapy with that. There's probably a paranoid concern about leaking where things are running on my local (i.e. rfc1918) network, but gcloud doesn't allow full zone transfers, and I'd imagine that's lower on my list of worries once someone's popped my home network.

October 20 2023

2023-10-20T17:56:00+01:00

A lot has happened over the last few months, so rather than a big catch-all post, I'm going to do a few posts over the next few weeks.

The last week or so have been my first 'in the office', so to speak, since March. As is patently obvious to anyone who's been within earshot, I'm going to try freelancing for a while. This has involved me getting boring life and company admin stuff set up. But, I digress...

One of the major things I've been working on with myself is the concept of asking permission. I grew up in a very permission-based setting. That's not unusual of course, but I'm used to having to ask if I can do things -- as I went out into the Big Bad World, it has often been a very internalised obstacle for me. I feel like doing a thing might be nice, and there's no real obstacle aside from me missing having a voice sometimes telling me "No". That voice can sometimes feel unkind and capricious, and it's only recently I've recognised it as my own. In the meantime, I'd gotten very good at self-denial.

So, to work on this a bit I planned a longer trip for myself, on the bike. This would involve a wee bit of structure (the first 10 days or so were an organised trip with a group) and then another 3 weeks or so of just doing what I want. What it was supposed to turn into was about 5000km of going where I like.

It mostly got there. It was a pretty nice jaunt around some mountains in Italy, Slovenia, Austria and etc. (The photo is me! It's from Passo Gardenia in Italy, in the Dolomites). The latter part was interesting, since a lot of the structure fell out (I was supposed to go to a wedding in Sardinia, but ended up getting covid instead). I spent the last few weeks getting to a point where I was just chilling out, making my way randomly. Get up, figure out a cardinal direction, book a place to stay, and go.

While I'd like to say that this was a relevatory and life-changing experience, I'm only going to say about 80% of that :-) It was pretty amazing, but I further discovered that it's okay to not want to be on a mad adventure sometimes. I was definitely done by the end of the trip -- which is another good thing to know about myself. But, It was an interesting exercise in knowing what I'm like, rather than forcing myself to be like something.

I have a tendency, when I get into things, to really get into things. I'm still trying to figure out how much of it is for my personal happiness and joy in experiencing things, and how much is performative.

Like many other things that live inside my brain, it's a work in progress :-)

6 Reasons You Don't Need an SRE Team

2023-06-21T05:40:00+01:00

The last several years have seen a huge upsurge in the popularity of the DevOps/SRE/Production Engineering model, with companies large and small adopting some of the practices and mindsets. One of the principal lessons many of these organisations (hopefully) learned was that it's close to impossible to adopt the SRE model as described by Google in the the first SRE book. It's a good approach to take on board the parts of the book that work for you, and to actively triage your time, energy and effort.

However, one premise doesn't see a lot of investigation or introspection -- that is, whether you should even have an SRE team at all. The existence of SRE (or "DevOps", or "Production Engineering", or "Platform Trust", or any of the other taxonomic manoeuvres I've seen) is still treated somewhat as a given.

This can be for a number of reasons -- there's a pre-existing "Operations" team, there's a strong leadership advocate, or the skills exist in the organisation and it's considered good to consolidate them. It can also often be for less good reasons; pure organisational momentum, a desire to silo-ise toil, and for reactionary reasons.

The arguments for having an SRE team can often seem obvious: Everyone likes having reliable systems; it's hard to argue against that. Everyone also likes someone else doing the hard bits. So, let's cover the counterpoint: 6 Reasons you shouldn't have an SRE team.

1. You're not Google

Even though Google's approaches to parts of the problem space are battle-tested and can help you with your approach, you're still not Google. You're not even Google in 2004.

As someone who was at Google in 2004, I will let you in on the secret sauce that made a large SRE team at Google a good idea.

Intractable Problems

Two factors drove the near-intractability of the situation Google found itself in around this time: Scale and Innovation. These sound like lovely soundbites, but the truth of the matter was that nobody was doing anything even remotely like what Google were doing at this time. Nobody had the same requirements.

In my last gig before Google, we were using nagios and mon. None of these sharded in any way that would work for us; they were designed for monitoring well-understood things about a few hundred hosts. At the time, Google had a couple of hundred thousand hosts, and were adding thousands more per month. Nothing in either the OSS world or that you could buy was even going to come close. There was no Prometheus, no Docker, no Terraform, and these wouldn't exist in any meaningful way for another 10+ years.

We had to build, deploy and somehow keep together things at a scale and complexity (related to the tools available) that we had never before seen in our lifetimes and likely will never see again. This meant that simply hiring buildings full of operators wasn't going to work at all. This was plainly obvious -- even back of envelope calculations showed a scaling vector that meant we'd run out of humans in pretty short order.

Unlimited Money

I'm not a finance person, but it's difficult to express just how much Google was able to invest around the software ecosystem. Everything from the hardware and cooling to the monitoring software and job scheduler was built in-house. This was mostly because of the "Intractable Problems" thing, but also because we could afford to. In many cases, we'd be fools not to; the industry in certain areas wasn't moving fast enough, or even in the right direction for us. Compartmentalising the big-picture reliability problems into a group that we could build or hire the best SREs on the planet into was a no-brainer.

To briefly get off the Google fanboy train and go back to the crux of the point: You're not Google. I don't mean that your product isn't a great product, or that you don't have real problems. I'm saying that the criteria you use to evaluate whether you need a full-blown SRE presence should be specific to you. In your checklist of things to potentially adopt from Google’s model, include an item about having a separate team at all.

(Also, you're not Google in 2004. If I want a reliable sharded database with global consistency today, I can buy it and have it today, using money. This was not even close to true in 2004)

2. You don't care that much about reliability.

It's very easy to say you care about reliability, and very difficult to figure out and assert how much you actually care. Even if you have a good set of SLIs and SLOs, that's not the whole picture of 'how much' you care. Is reliability so important to you that you need a whole entire team just to care about it? What about testing? Product Research? Customer feedback loops? Why did SRE get to the top of the list?

It's easy to assert that near-perfect reliability will drive user retention. It's very difficult to be so sure of your product's use case that you can put a number on how much unreliability you can tolerate. Saying "We're not a stock exchange, our users will tolerate 5 minutes of downtime every so often" is a strong thing to be able to assert; stronger and more rigorous than saying you need eleventy nines for your applications at all times.

This quote from Steven Levy's book on how earlier Google worked epitomises what was heard loud and clear throughout engineering. I once had Eric Schmidt visit my team's weekly meeting (I was on GMail SRE at the time), and the money quote was his response to a question about prioritising reliability vs. product features -- "I will choose reliability, every time". We framed that quote in our team area. Even to the jaded sysadmins and hackers we often saw ourselves as at the time, it was a clear message that reliability was a product feature, all the way from co-founder to CEO to us.

Is your CEO saying these things? Even behind closed doors? What about when you're under pressure to ship features?

The extent to which Google really, genuinely cared about reliability and made the company-level investment in it can often be forgotten - think about if you're there with them.

3. You're not sure what the team should do or own.

If you’re not able to very succinctly explain and have everyone understand what a team like SRE is there for, then the ambiguity and ability to re-legislate who does what is often quietly seen as an advantage.

Some places have this right; many places don't. If a team is chartered and has a remit, that remit should be set in stone and the requisite effort put in place to maintain that knowledge. In my talk at SRECon Europe 2022 I go into a bit of detail about stakeholders. It's been my personal experience that Stakeholders (i.e. product owners, dev partners, company leadership) either misunderstand or make it their business to misunderstand what an SRE group should be doing..

Even at Google (that wonderful flawless bastion of enlightened thought on the matter), I routinely ran into folks at VP level and above who didn't really know where their team's remit ended and SRE began, and really weren't inclined to learn. Free labour is free labour.

If you're not 100% sure what SRE (or any team, for that matter!) exists for, what it will do, and why it's a separate team, then you should strongly consider whether it being a separate team is a good idea.

4. You're doing it to avoid internalising inconvenient truths

Understanding reliability is everyone's job

If you, or engineering leads who work in your org don't think so, then hiring a separate team to care about it isn't going to help.

There is a continuing meme within software that I personally don't understand -- the idea that software can be produced to spec and then disappears into the ether. It existed to an extent in the area of software being shipped on Floppies/CDs/DVDs/tarballs, where the turnaround on bug reports, releases, patches, etc. was measured in development cycles. However, it has somehow survived into the era of there being exactly one running installation of your software that you care about.

I can understand how a team might shard the effort of day-to-day running of their software; I have less time for the idea that the entire ecosystem of care can be put on another team.

Low-value work

Reliability engineering is widely viewed as low-value work. Even if leadership is bought in, the meme is extremely prevalent at all levels of product engineering.

The usual response from many folks in Ops and SRE roles would be to say that it's not -- it's crucial to system reliability and customer trust, and so forth.

The good news is that you can both be right!

Further to my article on oncall being a waste of time, I'm here to make a humble request that people say this quiet part out loud. If it's the honest assessment of an engineer or leader that work is of low value, that actually forms the basis of a very interesting conversation to be had about priority and how this toil gets looked after. Some of the most stressed and least valuable SRE engagements I've seen were ones where people were talking past each other, with nobody acknowledging what everyone knew; that nobody wants to do this work, and it should be eliminated. The convenient presence of an SRE team means the conversation can often go around in circles. Speaking of which...

5. Your SRE team might be a red herring

Further to the above post about low-value work; another anti-pattern which means you should be very sure about your SRE team's remit is that they are a convenient outlet for lack of planning or responsibility on the part of other groups.

Even in a "you build it, you run it" shop, it's possible to indefinitely delay and defer platform reliability or modernisation work, on the grounds that an SRE team should either be doing this work wholesale, or assisting. It's a very easy position to take, that has the side-effect of deflecting responsibility from a product or service owner onto an SRE group. I've seen cases where the SRE group hasn't even been asked to do the work, this deflecting responsibility onto...nobody.

Would this happen if you were able to be 100% sure that everyone at your company completely understands SREs remit and has no issues with it? No!

Would it happen if you didn't have an SRE team at all? Also No!

6. You got a big fright

We've established that it's difficult to put your finger on how much you care about reliability. Even if you have done the work and really have a good handle on this; you may be one giant outage away from throwing that all away on optics.

As a thought experiment: go and corner your nearest C[TE]O and ask them how much they care about reliability. It's probably a lot, right?

Now, go ask them again the day after a multiple-hour outage. Now it's definitely a lot!

SRE as a brand is a double-edged sword -- it can be seen as a panacea for reliability problems, when often it's a more intrinsic cultural investment. I have personally seen SRE groups appear almost overnight, or have money/headcount thrown at them because we had some big outages and this is an audacious hail-mary pass of a move that really shows you care about reliability. If you're not careful, you could be telling your customers you care about reliability, while telling your product developers and leadership that they don't have to.

Conclusion

The model of large SRE teams covering many services in a vague and nebulous way that's open to repeated re-interpretation is mostly a side-effect of (a) cargo-culting the building of these large groups, or (b) retrofitting SRE/DevOps onto existing groups without the company-wide reliability focus required (or the fortitude to decide you didn't need such a large group to do SRE).

Most of the reasons Google built a large SRE presence were related to being better-equipped to throw money at all its problems than most companies in the last 100 years, and due to the utter absence of big parts of the software and hardware infrastructure needed to build reliable services.

The last 20 years have seen enormous advances in the SaaS and infrastructure software space. What was absent is rapidly becoming present; what was esoteric is rapidly becoming generic. To fully realise the SRE model is to reduce complexity, to make rational choices, to buy because you don't need to build.

The next stage in removing our production training wheels as an industry is to tear down the fence between SRE and Product Engineering, and make rational investments in reliability as a mindset, based on specific needs.

June 19 2023

2023-06-19T23:53:00+01:00

My last few weeks have ended up with me having about 30% of my regular day-job meeting/stuff-I-gotta-do load; due to a perfect storm of school stuff and other adventures (I'm currently collecting driving license categories! Trailer learner permit coming and sitting some theory tests for Truck license).

Other adventures recently was dusting off the campervan for this year - it turned into more adventure than we strictly wanted when the alternator in the van decided to die half-way to where we were going. This territorial Robin quickly reminded us we were only on his turf temporarily.

Next week I get to probably make a poor attempt at lacquering a wooden floor, and burying some cables in the garden that i definitely won't dig up again. Living The Dream. Still Alive. May think about work come....October or thereabouts?

June 08 2023

2023-06-08T23:12:00+01:00

I've mostly ported my homelab across, and took the opportunity to add actual monitoring and actual resiliency. I'll do a post on the janky setup.

I've also (thanks to some kind mastodaoine) gotten set up with restic to some remote storage. I'm glad software like this is still being made. Very little messing about and a well-implemented version of the thing we thought we could do ourselves with a shell script. You know who we are.

In other news, I'm also roughly half-way through a diploma course in Coaching and Mentoring. I'm pretty sure I'll end up using it in any case, but it's one of those things where I also want to try out if it turn out to be a source of energy to me. it certainly was when I was working in a larger enterprise, and it'll be interesting to see if that was all due to being invested in outcomes, as opposed to happy with the method and with helping individuals.

(Sorry for the 'writing prompts' post calling out how I haven't been posting here. I'm trying various methods to gather my thoughts as I sound out how my brain works these days. But that's another post).

What Would It Take?

2023-05-12T15:48:00+01:00

One of the most constructive things I've found when facing a difficult work situation is to externalise and write things down -- I do quite well with talking things through with folks, and coaching and mentoring is something I get a lot of energy out of. When it's just me and my thoughts, the interlocutor is writing, and I often have to push myself to get that done.

When talking to others in this situation, I'd often frame it in simple "What would it take?" terms. In the case of whether to stick around in a job or stick with a project or idea, this would be "What would it take for you to quit today?", and the vital accompanying question of "What would it take to resolve all this, and to re-commit indefinitely?". In the case of a job, you're either going to quit, or you're not. Spending a lot of time agonising in between does pretty much everyone a disservice, yourself more than anyone. I think it's healthiest to always be going in the direction of one or the other, if you do feel there's work to be done there.

What I hadn't been realising is that this was a kind of abridged version of the spookily-named CIA Model used in formal coaching. Essentially, you're doing a bit of compartmentalising of issues in your own head into "Stuff I can fix", "Stuff I can affect", and "Stuff I just have to put up with". There's a lot to unpack there -- in some cases, there may be bright lines around ethics, capability and sometimes just bad timing that mean you have to self-select out of the situation. You may also decide that something you think you can fix is something you likely shouldn't, or that you need to think about if you should tolerate it long-term.

Conversely, taking a full checkpoint on what matters to you, and what the real situation is will lead you to clarity on next steps. It'd be a shame to not investigate options for how to move stuff from "I tolerate this" to "I'm okay with it", or better.

Most people take a little prodding to get there, and I'm no different. If there's no capture of what the sources of stress are and why they matter, we end up having to rely on the Amygdala, the "animal brain". It is very hard to reason logically when we think we're going to be eaten by a tiger. This is why we invented writing.

So, if you're at a point where you think you're phoning it in, or you're struggling to get the basics done; think about the theory first.

What about this situation do you directly Control?
What do you not control, but have some Influence over the outcome?
What parts are you going to have to Accept and deal with?

If you're able to successfully explore what the situation on the ground is, the slightly more practical parts that I've worked through with others and for myself are usually the more challenging practicalities:

What is the issue I'm dealing with?

Do I feel underappreciated and having my incentives change would fix it?
Do I feel like a co-worker is making my life difficult?
Is my employer doing a kind of business you don't like?
Am I not seeing enough customer conversions or sales?
All of these are valid. They all take up mental energy.

What would it take to resolve the issue and have me re-commit to the idea/project/job/etc. indefinitely? What's missing?

It costs mental energy to tolerate a situation, rather than having it be a fact of life that you're truly reconciled with.
Moving from tolerance to acceptance takes change; and it's almost never just rationalising the situation away in your head. Real change involves doing something you can't take back for free.

What would it take to give up and change direction completely? What's stopping me?

This is a challenging rhetorical question, but the actual answer is something you should try to write down. What would need to change in order for you to say "thus far, and no further" and do something more drastic than putitng up with the situation? What's your 'trapdoor'?
In each of these cases, you should consider making these things known, if you're able to be definite about them. This especially applies to business outcomes, where you might not be the only stakeholder.

This has worked for me in a long career at one employer, and for knowing when I'm able to make changes without compromising on core values, but also to provide an outlet for the Sunday Night Terrors when the Amygdala takes over. The right answer isn't always the most satisfying one, but it's easier once you put (figurative) pen to paper.

May 09 2023

2023-05-09T23:30:00+01:00

In among posts about fiddling with electronics and occasional life updates, here's a life update!

My partner (and now fiancee, I guess?) isn't very present on the internet, and escews social media in fairly impressive ways. But, we've been together for close to 15 years, so we decided to make it official.

I say 'we' -- we still did the proposal and...dare I say formal part of things, but it very much felt like we were taking a further step on a closeness that has stood to us and will endure, I think. We're both very private, and very contrary when it comes to going through motions, so I suspect this'll be a paperwork thing

There's a calm in knowing that no matter where your head takes you, you get to come home to (or, indeed, stay home with) your favourite person.

Even if my head is tired and uncooperative and in 6 different directions (more on that later!), this is Real and Special and Lovely.

Another 50 years or so, then we'll see.

April 25 2023

2023-04-25T01:00:00+01:00

Further adventures in pretending homelab is real work!

This week, I've actually been hacking on getting restic running semi-regularly. It turns out (or at least, I've found by experimentation) that the way nomad does scheduled/periodic jobs means that the periodic jobs don't have access to variables the core job does. I may be doing something wrong, but I also wanted to write come code, so there's a hacky threaded restic job runner over here that I still have to tidy up and document a bit.

My main thing I'm working on is replacing consul. I don't have any need for its more advanced nonsense, and all I was using it for was finding jobs. It has a DNS port it listens on that you can point named at (and I do, here). It can be a bit flakey, for reasons I'm not sure about (and that are proably my fault).

So, I've decided to implement just that fuctionality myself, and rip consul out of the homelab. I'm not quite done yet, but I think nothing is talking to consul at this point. This change and some prometheus changes probably did most of it.

The code is at nomad-dns-exporter and, while terrifying, is still solid enough to run my lab on. It's also got some very basic prometheus metrics to scrape (I cannot prometheus my way out of a wet paper bag at this point, but we'll get there).

If you want to follow along at home, try:

virtualenv .venv source .venv/bin/activate pip3 install -r requirements.txt python3 nomad-dns-exporter.py --nomad_server=ip.of.any.nomad.server

Then test with dig myjob.job.nomad @localhost -p5333 - I shoould probably do smarter things with SRV records or something, but that's it for now. Next up is to grit my teeth and turn down consul to see what happens.

April 17 2023

2023-04-17T07:00:00+01:00

I've been gradually rolling out more homelab updates, including whats turned into a bit of a rebuild and de-flakiness effort. Nothing like a bit of sunlight to realise that things need a proper looking at :-)

It's also been pretty good for my brain, I think -- having the benefit of really looking into and learning how things work properly, rather than hacking on it until it bends to my often fickle and capricious will.

Anyway, this week's make-go-proper effort was starting to use a docker registry for localy cached images. I had been going about it arseways by having a local registry that I downloaded and pushed images to. Turns out A More Properer Way is to configure all your docker hosts to use a registry that's configured as a cacheing proxy. This also means if my home internet is dead, I'm able to move tasks around without it blocking on the new nomad hosts fetching the remote docker image.

The tl;dr, is to set up a registry with proxy.remoteurl set to a remote proxy. here's how i do it at home with nomad -- note that you can't push images to this registry, so I run a separate proxy-only registry. Then add the following to /etc/docker/daemon.json on your docker hosts:

{ "registry-mirrors": ["https://my-local-mirror.local:5001"] }

Probably overkill, but that's probably the point?

#worksonmycluster

2023-04-07T00:14:00+01:00

Now that I've a bit more time on my hands (more on that later) the predictable swing back toward being able to do actual tech fiddling has started (I Hope).

I've been running nomad from Hashicorp as the compute coordinator for my home setup for a couple of years now, and it's been keeping things going in a fairly unobtrusive way, so I'm going to publish a few more of my 'orrible hacks to get it to do as I say at all costs. I find it a lot more amenable to doing things without having to gut its innards or be a YAML savant -- all while being able to actually understand what's going on with my tiny wee brain.

the initial results are and are going to be at https://github.com/gerrowadat/nomad-homelab, and maybe that means I'll update stuff here? Maybe. Anyway, come and see how I do SSL certificate distribution probably wrongly.

November 06 2022

2022-11-06T21:32:00+00:00

It's November! That most wonderful time of the year when your brain and being need a little more maintenance.

Part of my maintenance schedule is taking the worsening weather as a prompt to feel less guilty about not going outside and doing outside things -- i.e. getting to coding and home sysadmin projects before Advent of Code kicks off.

Having ditched Kubernetes at home for various reasons, I've been running Nomad instead for homelab stuff. It is a lot more straightforward for my purposes, and reminds me a lot more of Borg from my time in the chocolate mines.

I'd also been running Consul for service discovery (mostly via DNS interface), and was planning on doing proper secrets management with Vault -- However, Nomad 1.4 added a few nice features, including something that's not as shiny as Vault, but works for my own purposes. So, I was looking to slim back to just Nomad and get rid of the rest of the ecosystem entirely.

After racking my brains for a cool and down-with-the-kids project name, enter nomad-dns-exporter, which is now happily pointing nginx and other various scripts and such at nomad services via DNS, no consul required (unless I turn down consul and discover it is, of course :-))

It's a homelab, so I get to replace some proper software with duct tape, and learned a few things along the way -- so, homelab mission accomplished!

Oncall: An Equal-Opportunity Waste of Time

2022-11-04T21:13:00+00:00

I spent a number of days at SRECon 2022 in Amsterdam, and gave a talk that I'd had rattling around in my head for a wee while - roughtly based on a paper I left at Google when I left.

Laura was good enough to suggest it as a ;login: article, so here it is: Oncall: An Equal-Opportunity Waste of Time.

2021 Retrospective

2022-01-01T04:10:00+00:00

2021 was a year. I've seen the receipts.

In summing it up, I guess I'll preface it by saying that this, more than any previous year, was one where the shared experience of being the connected human was a more closely bonded one than ever. The shared experience definitely had a timbre, a scent, a feeling to it, and in most individual examples it was pretty below average. It's rare to talk to someone who said they've had an amazing year and meant it.

It can almost feel like saying that things happened and that they might been pretty good is a source of guilt; as a catholic-raised Irish person, I can tell you a thing or two about that. So, with that in mind, here's how things went, and a few aspirations, thoughts (definitely not resolutions) for 2022.

One thing I should mention, since it kind of encompasses all of this -- I've been seeing a psychotherapist and been back on SSRIs since late 2020. It's not a big thing, it's helped a lot, and I think it's important to mention. It's not a sign of weakness to look after mental health, and to be seen to do so, even if you're meant to be an omnipotent entity (and yes, even if that requirement exists only in your own head).

Work

Probably warrants its own entry? Or a few entries. I changed jobs in June, from being at Google for almost 17 years, to a much smaller place that's in a very interesting growth phase. One of the interesting parts about this was that the new place is fully remote, in a way that doesn't disenfranchise remote workers (i.e. everyone is remote). This has been something I've wanted to do basically forever. It was pretty clear from what was happening at G that they were not taking remote work seriously, so that was one of the contributing factors. Coming into 2021, I didn't know what I wanted to do, but I knew offices as we know them are from the past. I was kinda done being '8 hours from the mothership', kinda done getting on a plane every 6 weeks ro so to remain relevant and defeat information compartmentalisation, and also kinda done with a lot of things that are inevitable when your employee count goes into 6 figures.

The new place has been great! Due to the remote work piece being built into how they do business, I feel a lot more connected, ironically. I've yet to meet any of my co-workers in person, although it's alleged this will happen in June 2022. Sounds about right :-)

As part of this, I've also been spending some time at a co-working space. I'd be lying if I said I don't miss the office ambience sometimes, so it's good to have the option. Home is still home. Speaking of which...

Home

Home is still as it is. Keeping healthy and sane is a bit of an uphill struggle, as I suspect it is for many. We did manage to make it out when restrictions permitted (we even made it to Finland!). Overall, I'd call this ticking over. There's a degree of 'normalcy' that many people are waiting for the other foot to drop on that I suspect we'll just have to declare someday.

Sadly, the Small Cat left us at easter time, from lung cancer. She showed up at what at the time was my doorstep, 12 years ago, right as myself and J started seeing each other. She was my cat, but also our cat.

One of the things we said we'd do if we ever lost Small Cat (she....didn't like other cats, and barely tolerated our other one) was to start fostering kittens. So, we took in a few litters, and eventually as things go, we met Tortellini, our 'failed foster':

We weren't planning on another tortie, but they tend to find you.

Hobbies, Etc.

I did manage to do a little bit of 'getting on my motorbike and going far away' this year - managed a trip to Scotland with some biker mates, and got just the right amount of rained on.

I've also been studying, practicing and getting ready for the RoSPA test, which is one of the tougher civilian motorbike certifications. I'm hoping to do some blood bike work in 2022, all going well (the test is a prerequisite). It's been a great way to keep up skills and progress.

I also started doing gigs again -- I kinda wound down the involvment with one group I played with, and started playing bass for a Ukelele band in Dublin. Gigs have been pretty quiet since things went south around december-time, but there should be a lot going on there in 2022 all going well.

I also did something I'd been meaning to do for a long while, and joined a shooting club. I've been meaning to break into it for a while now, and it's one of those "just bite the bullet" things (the upfront costs are singificant, and the licensing rules are weird). It's been gratifying to see actual improvement in my shooting over the last few months. Again, more to come on that.

All in all, given the circomstances, it's been a pretty good year. I've had to take calculated risks to do so, and I've most likely been very lucky not to get sick or to lose anyone close to me. It's part of a longer journey, and I'm happier with the amount of control I have over things (macropolicies keeping me at home notwithstanding).

I'd like to get a bit more relaxed about using the advantages of fully remote work -- I think it'd be a shame to have this kind of position and then work from my home office every single day. But, baby steps :-)

I mentioned it on my magical exploding twitter feed, but I feel like there's very much an onld-meaning meme going around that Everthing is Awful and we must not buck this narrative. On balance, things are pretty awful. But, I think we also have a responsibility to celebrate the okay things. The world is giving us pleny of excuses to be morose; let's not help it too much.

To 2022. Because it beats the alternatives!

A Short History

2021-07-22T14:30:00+01:00

I'm allowed one cheat blog post, as a treat.

The twitter thread I posted a while back seems to have gotten some attention. It's reproduced on threadreader but that too shall pass (and I auto-delete my tweets after 90 days), so here's a reproduction.

The SRE book is a history book, not a manual.

SREs, at their fundament, are organisational glue. They are people who are curious and understand how the world works. They turn their hand to things. When things don't go SRE's way, they pick up the pieces and try to fit things back together again.

If you need something well outside the remit of what you'd like your average engineer to do, find the nearest concentration of SREs. Want some business sense? Sure. Need someone to capacity plan? OK. Want some esoteric knowledge about weird systems side-effects? SREs will go learn it, for seemingly no good reason other than because it Must be Known.

If you get a load of these kinds of people together, they can move mountains. If you give them a seemingly hopeless situation, they will pull you out. If you give SREs an innocuous-looking situation, they will tell you which lava flows will liquify, and which will stay solid. SREs will point out the oncoming train, and patiently explain how to turn the train we're on.

This is their common skill: The ability to use the silver bullets of curiosity, pragmatism and bias toward action to grow systems as they need to.

What happens when you get extremely good SREs and expose them to an essentially unlimited pile of money, infrastructure and opportunity? You get a story. The story of the SRE book is the one of what worked for Google SRE early on, in that context. It's a series of parables. There are teachings there, not instructions. It's no surprise that many of the authors of the SRE book were based in Dublin, or Irish -- if you learn by stories, you teach by stories :-)

And so when I say to people that they should take what they read in the SRE book with a grain of salt, I mean that there is truth and wisdom in there, that need skill and wisdom to tease out. The skill you bring as the reader is in deciphering what are the fundaments, the core of wisdom behind the practices described. The ideas you can take away, and reasonably expect to make useful in your own practice.

The circumstances that gave rise to the collation of the SRE book may never be reproduced in our lifetimes. I'll always also maintain that it should be taken as a set of examples of what SREs did when faced with seemingly insurmountable challenges, and lived to tell the tale.

And, if you're reading the book and the thought process and conclusions based on the problems described speak to you, a pragmatist, a realist, a curious tinkerer, then you're already an SRE. Leave the book aside, and take what you've learned with you on your journey.

Working Deliberately with OKRs

2021-06-26T23:40:00+01:00

After being asked by a work colleague to say words about OKRs and if they're a good thing to adopt, I realised I had more words than fit in an email or short doc -- so, here's a slightly longer doc on my view of OKRs, tempered by 15 years or so of using them fairly successfully, and seeing them used...less successfully. They're one of those "Google does them so let's do them too" type things that seem like a no-brainer, but can often mask the right answer for where your practice is at.

Anyway, say hello to Working Deliberately with OKRs (with thanks to @ahidalgosre for casting a critical eye over it).

June 13 2021

2021-06-13T16:46:00+01:00

This past week was my first week at elastic.co, where I am now a Director of Engineering - taking on the SRE group and making Elastic Cloud go (Elastic Cloud is essentially the SaaS version of Elastic Search - i.e. you can of course run it yourself, or you can pay us, because tl;dr money is cheaper than opportunity/time and also we'll likely do a better job).

Although I'm 3 days in, it's popped up a bunch of really interesting oportunities and problems -- some I'm /very/ familar with as someone who tells nerds what to do, some are completely not. This is actually exactly what I was looking for. A set of areas where I can both bring the expertise I have, and also be stretched (with a side of being closer to the newer parts of the industry, and being somewhere where remote work is taken seriously).

Of course, this has the side-effect of me being absolutely wrecked at the end of each day, and having to keep tabs on my time and not overdo it after 5 weeks off (the above is a good forcing function: Marble tha cat came and sat on me on my first evening, and it was No More Work time).

There's a mountain of research out there on how to get started at a new enterprise. I tend to consume this stuff and run it through a filter of there being really no subsititute for a bit of cop on. Aside from being only partially successful in general, 'management by numbers' is one of those things that people spot right away, if they're in any way smart (and hopefully they are!).

I'd recently been recommended The First 90 Days, which like most management books have a bunch of decent general-purpose pieces of advice, but are for the most part not to be taken literally. In this case, the book kinda assumes you can sit on your hole for a few weeks to plan, and avoid commitment for your first wee while.

In reality, given how business works these days, there are always going to be things that don't wait. In my case, it was probably the first-day war room for the fastly outage -- yes, I could have ignored it because it's my first day; however, there is a huge amount to learn from observing your new practice's workflows and dynamics, unfettered by them knowing who you are, and how you are :-)

So, as I've said to folks internally, the 90 days plan is of course still a thing -- however, in the meantime I'm practicing what I'm calling "Listening and having standards". That is, the active listening and onboarding you'd expect from any new lead, but with a side of asking awkward questions, and setting a standard for rigour that folks will come to know. Part of being an active global lead is having people know your mind, without needing to hear from you directly -- I don't think it's ever too early to set out your stall, even if it's asking the kind of questions to which the answer either matters to you, or where finding the answer is a useful exercise in and of itself.

June 06 2021

2021-06-06T00:06:00+01:00

The last few days (among other things) saw a little work on the previous project, as well as taking advantage of having new hardware to mess about with.

My desktop OS journey has a 12-year (or so) mac gap in it -- which is to say that I used to use a window manager and go through all the tweaks and business associated (the kool kids at DCU all used Window Maker, but when I started at Google, they handed me a macbook (the early, pre-magsafe one, circa 2004). I was pretty happy that I could have a desktop that just works, and that I can replace at very short notice. Much of the work I did at Google involved using shells and a web browser, so macs were just fine. I then happily used macbooks until the touch bar, which closed me down overnight. Ugh, horrible.

I then spent 6 months or so trying to work from a pixelbook on Chrome OS. Google's deep integration with chrome and BeyondCorp meant this was a fine machine to use, until it started to do random things, and you can't really do anything to fix it (aside from powerwashing and hoping for the best).

So, for my last year or so at Google, I was back on gLinux, which was fairly standard gnome desktop. I didn't fiddle with it, it just worked, happy out.

Anyway, $NEWJOB sent me a ThinkPad T14 -- this seems to be an AMD equivalent-or-so of the T490 I have as a personal laptop. So, I took to installing Arch Linux, and trying out a few window managers.

Arch, to me, seems to be just a little beyond the amount of thinking I'd like to do about my linux distro. I must say the documentation was /excellent/. I learned new things about the modern linux userspace (systemd is still newfangled to me, and I've only really set up wifi by clicking buttons.). I gave it a shot for a few days, before giving up on sound. So, doable, but more time than I'm willing to put in when good old familiar Ubuntu is waiting there. So, good experiment, but ultimately I have shit to do that isn't dealing with linux on a machine that needs to work and for which there are better options for my expertise/desire-for-hassle ratio.

As for window managers, I had toyed a bit with xmonad a few years back - I know no haskell at all, so it was mainly trying a tiling window manager. This time around, i3wm was a natural one to try. I have to say it's one of those 'exactly what it says on the tin' style things. The only tweaks I've done so far is making i3status say stuff that is useful (and talk to the thinkpad battery properly), and multi-profile/workspace groups stuff.

The latter was done with i3-workspace-groups - it's a hacky enough way of doing it, but works great. The way i3 is setup to be modular and take instructions from various sources is super smart. I want to be able to have separate sets of workspaces for work, home, etc. and this seems to do it. Config is here. Maybe this is old hat for long-term linux desktop users, but I'm personally glad to be back to getting things just-so, rather than using a 'supported distro', so to speak.

The proof of its efficacy, of course, will come on Tuesday. More about that soon :-)

May 24 2021

2021-05-24T22:14:00+01:00

Showed up to a few low-key conferences last week (Multitasking between SLOconf and o11yfest for a few days).

It felt great to be catching up with some folks I hadn't in a while -- not having the GOOG hat on was a refreshing change. It's like the manager/director/VP/whatever hat in any job you're in - you like to imagine you can show up one day and sit backwards in a chair and say you're not a manager/director/whatever for a second, but that's not how it works. So let's say that showing up to a conference as a regular punter and as a Director at a FAANG are very different experiences.

Another good sign is signing up for a few more projects at home. While the month over xmas was home-assistant, then messing with Kubernetes before throwing it out in favour of nomad, this break has been a lot more about really re-charging.

So this week, the bits arrived for one of the few projects I'm allowing myself (aside from home sysadmin). Since I'm a fan of stupidly simple interfaces, I wanted to do a 'traffic light' style interface for presence, that I can plug into Slack, or even other things (Slack is about to become a large part of my day-to-day workflow, as part of a global org, so presence is important). I know I could run Keybow OS on it and mess about with LUA, but where's the fun in that?

The RPi Zero and keybow are dinky! I dunno, I still get a kick out of really tiny things that have a full userspace on them. I might see if I can pick up a small handful of Cherry MX Blue switches, since I dont' mind those three wee buttons being clicky (I run browns in my regular keyboard).

The code I've messed about with so far is here ('anseo' means 'here' or 'present' in Irish). It also gave me a chance to actually learn more about asyncio in Python, which is a feature that appeared since the last time I wrote python code in anger, and looks much nicer than messing around with threads (so far...).

Bad Machinery Paper

2021-05-19T20:34:00+01:00

Thanks to my accomodating erstwhile colleagues, I'm able to publish the original paper (with minor redactions so it makes sense for an external audience). A version of this paper later became Chapter 29 of the SRE Book.

To add a little bit of extra colour, the document came out of some lessons learned on Storage SRE (specifically Bigtable/Colossus), that I ran directly back in the early 2010s -- The date I have on this document is 2016, but I know it was around long before that.

Storage SRE back then was something of an anomaly -- there were a number of teams that managed storage services (the above, but also some various other services, some of which where purely internal and never saw the light of day). To amalgamate these services meant that we potentially won a bunch of OMG synergy because of the base assumptions made in design that were unique to services that kept data. Google started out as a search company with a throwaway, re-scrape-able corpus (built on GFS, which wasn't designed to persist data), so it was natural for us to have a couple of goes at stateful services, and see what stuck.

So, while you might imagine that lumping services of a similar shape together is a good thing to reduce duplication of effort (and you'd be mostly right), other factors include service maturity, generalised attitude toward production, and even come down to the vagaries of the dev/SRE relationship, or of engineering leadership as a whole.

So around then, we had a Bigtable and Colossus service that generally worked remarkably well. We had challenges in observability, and in developing the tech around load-sharing of users on a shared compute/storage pool. This later became known as running a multi-single-tenant architecture, although at the time we saw it as fairly chaotic - not all the tooling had been done, and some of the assumptions and prior art we have the benefit of today wasn't a thing, yet.

While others looked at this as a purely technical problem, myself and a couple of key leads saw this as a problem of focus. Google in its earlier days had faced additional problems of scale like this -- where esentially we could keep digging, or we could put in place a set of attitudes and approaches that enabled us getting out of the hole we were in as far as interrupts are. Some of these problems fell into the "We will need 5000 people to edit config files" or "We will need more hard drives than humanity is making" variety, which really do need 'creative solutions').

One of the anti-patterns we observed at the time was a tendency among folks on both Dev and SRE teams to concentrate on speedy detection and resolution of issues (The 'Rocks with Eyes" method, as succinctly described at the time). Being able to know when things are busted is important -- but the followup is key. Oncall and interrupts are fundamentally a waste of time and should be minimised, so signing up for extra seems not a great approach.

I had a separate paper related to this one called something like 'email alerts are from the past' -- and my real-life experience of that was coaching some key team members through some of the above approaches. We had folks who felt that they needed to keep getting email alerts, as they had trained themselves to spot patterns, and believed they needed to keep doing this or we'd have more outages. You might step back from this a wee bit and recognise that this was a set of people who were inadvertently being profoundly disrespectful of their own time. It's not because they were bad people of course, it's because they were busy people, and firefighting had found its way into their muscle memory.

In the end (in this example), myself and senior TL on the team ended up unilaterally turning off email alerts. She sent me the code review, I approved it, done. It was clear that we weren't going to reach consensus on whether or not it was a good idea; and while I'm not a fan of this approach in general, it became clear we weren't going to get there (see also the 'consensus is nice' section). In this case, nothing happened. A few folks were briefly sad, but ended up with more time on the clock and more spoons.

That, to my mind, is the outcome of a successful strategy around interrupt management. The interrupts need to be addressed, and in a timely way -- but you as the team responsible hold the ability to do it in a way that doesn't treat people as machines. In addition, interupts are not a closed system - the interrupts you're getting a month from now need to be different from the ones you're getting today -- by finding a better set of first world problems.

But moreso, the way you give yourself the ability to step back and really address the root causes in a noisy system is to respect your time, reduce context switches, and give people both the well-clock time and the spoons to be able to do so.

May 11 2021

2021-05-11T17:06:00+01:00

Let's try every week or so? No excuses about no time :-)

The last week or two have been decompressing, switching off. I found/find myself unable to do most of the technical/outside/etc. stuff that would usually count as "Doing stuff I want", in favour of a sort of haze of...meh? It's not even "Oh, woe is me", it's more of a "Nope" to doing basically anything. I take it as the brain trying to heal itself :-)

So, that's what that's like :-) I can already feel things lifting, though -- so I think once there's a little more structure I'll be good to go. I do realise it's up to me to provide it, so that's this week's job.

I have been able to get out a bunch, though. Yesterday was the first day when inter-county travel was allowed, so I made it out to the parents, for the first time since xmas. There is a significant backlog of being fed to be done.

Next week involves putting a bit of structure (i.e. fixing sleep cycle), starting small on some exercise, and maybe making a list of project work to do (I suspect learning about prometheus is bubbling to the top of the backlog). I also need to fix my horrible hacky way of using local disk in Nomad allocs (squint your eyes and behold). Plus other stuff that's now popping into my head that means I should probably go and make that list now...

May 05 2021

2021-05-05T00:50:00+01:00

So, last week was my last week at Google. I'm writing this from the weird void of disbelief where I've not quite settled into the 6 or so weeks of time off I have before the new thing (which I'm trying not to talk about, because I do want to take this time really off).

It seems customary, almost, for long-time Googlers (and especially SREs, for some reason) to go into the reasons why they left. A sort of "Why daveoc is leaving" doc/rant/diatribe, which generally gets people riled up.

In realiy, I could probably invent such a doc. I'm unhappy with many things G is doing -- but, that's not the primary reason (or in the top three) reasons I'm leaving.

In short, if I had to narrow things down, it'd come down to:

I've been at G too long -- I mean in general principles. 17 years is a very long time to do anything. You're not the same as everyone else. You have baggage, either your own or other people's.
I want to see if I can do the things I enjoyed at G elsewhere, and be successful. I mean, I am pretty sure I can, but there's knowing and then there's knowing.
I've always imagined that at some point I'll work remotely properly. 10-15 years ago, I imagined both the technology and work practices would evolve to make it workable. I hoped that this would happen during my working life, and it looks kind of like it has. It's definitely possible to find a company taking remote work seriously. G have been pretty open about not doing that. The office is still king for them, and that's fine. But, I want to try it seriously.
I also feel like I should be more public about what I do - I've had a few external tech talks, but not a huge number -- I've had a number of papers I published internally at Google that people have taken lessons from (one even made it into the book, and some others can be found in sanitised form in my Writing section). I of course had the ability to do so at G, but for various reasons I hadn't gotten into the swing of it.
I happened to find a role that ticked all the above boxes and more.

So, that's it. I continue the journey with bridges unburned, but hands firmly wiped.

Of course, to coincide with being a jobless hippie for a little while, my new bike showed up and got collected today -- just in time for a lovely spell of a few days of showers. Ireland's lockdown is easing to inter-county travel from next Monday, so I suspect this machine will be getting some exercise in the coming weeks. That's about as much as I've figured out about what I'm doing. Which, to me, sounds pretty fine right now.

May 1 2021

2021-05-01T12:15:00+01:00

One of my excuses for not updating this more often is the fiddly nature of making sure I have a working Pelican install. I mean, the real reason is laziness, but that's a handy excuse.

"Woudn't it be great", my insomnia-brain said to itself, "If I could just check a markdown file into git and everything just happened?".

So, a bit of fiddling later, a combination of GitHub Pages and a GitHub Action does just that. So, no more excuses, right?

The repo will give you an idea of how to go about it if you feel like emulating. It still ticks my box of "I have the content in a form that can be used to publish elsewhere if I want", so nerd brain is also happy.

Hopefully longer post incoming, I guess I have some news, right?

January 03 2021

2021-01-03T01:23:00+00:00

I'm now most of the way through the ~month or so I took off work. Around the start of it, I made a few plans, but I also resolved to unapologetically spend much of it just relaxing, and feeling non-guilty about that. It's something I've been working on, and I'm fairly pleased with the results. Sometimes the things you do are for you. It's one of the things I know and then steadfastly refuse my own counsel on.

One of the outcomes was solving a few of the outstanding home and home automation things I've been loking at doing for ages. The above is a Shelly 1 connected to the pinout of my garage door opener. I didn't have much say in what kind of opener and ended up with the kind with an Internet of Shit widget that just doesn't work properly, and a Manufacturer that don't take too kindly to hobbyists by not publishing any schematics. Thankfully, I happened to know that shorting two of the terminals on the back worked the same as a remote control press, and here we are.

One of the outcomes (unplanned, but here we are) of this break was that i've started writing code agin in earnest. I hope it lasts when I go back to work; I partially credit Advent of Code and some nice folks over on the Irish Tech community Slack for that -- but i've picked up on some other stuff as well. (btw, my attempts up until I ran ot of time are on github, including practicing on previous years).

(I guess if this is the last entry for another year or two, you'll know how going back to work went).

December 31 2020

2020-12-31T15:40:00+00:00

My last attempt at doing the Home Automation/Domotics/Whatyacallit a few years back mostly involved buying expensive widgets, determining they are shite, or marketed at US power and outlets, and then discovering that setup and upgrades generally involved one ore more of downloading a random .exe from random websites, soldering, or tracking the ongoing battle between hardware manufacturers and hobbyists.

Thankfully, things have gotten a lot better in the last few years. One of my personal projects in the time off over xmas this year was finally fiddling with Home Assistant, which got done on the houses's brand new Kubernetes cluster (but, more on that later).

It's been, quite honestly, a breath of fresh air. I might be a little unhappy with HA devs racing to the bottom in terms of what market to cater for (but that's purely a selfish concern), but it's...refreshing to see an ecosystem of support build around something.

It also helps that manufacturers like Shelly are openly suporting offline use and flashing their kit to something like Tasmota. One of the key concerns last time was just how flakey it all was - and subject to the whims of manufacturers.

I'm still working on the trickier parts of integrating the random assortment of Internet of Shit devices at home, but this is a very promising start.

December 09 2020

2020-12-09T01:28:00+00:00

So I've set both a really good example (take your vacation!) and a really bad example (Don't leave it all til the last minute!) for work folks, and therefore today was my last day at work in 2020. It's been a year.

One of the things I'm especially looking forward to this time round is taking on projects and doing things in an active way -- a lot of the time, my primary complaint when I take some time off is that I get to the end and feel like I've wasted it. I often feel like I can't get things done because there's some crucial thing I need to box all my time off for. A few months back, someone casually remarked to me that that sounded like burnout. So, taking this time was one of the things I wanted to do.

The other was actually going and doing something about some of the ways my brain worked. That means drugs and therapy. I'd done some advocacy work (where I could) at work around mental health, and even while I was doing that, I kinda realised I felt a bit hypocritical, because I was basically untreated (aside from a few abortive attempts to go to a therapist that amounted to being handed a burned CD of whale noises and being told to google CBT).

So, long story short, I've been back on SSRIs (I was on them for a brief time about 10 years ago), and I'm seeing a much more engaged and useful therapy person. It's been pretty great, actually. I feel a lot more like myself again - like a version that isn't really struggling as much, without the mind racing constantly.

One of the side-effects of this, of course, is that I've started to write code and do some wee projects again. I hadn't felt up to this in a long time. Probably years. It's a good feeling. What's helping right now is Advent of Code, but I've also got a few things going at home. Hence the picture of the haul of RPi stuff -- which I do intend to write some more technical posts on, as I progress. A month off is long enough to form some good habits; maybe logging more stuff is one of them? No Promises :-)

On my list is:

RPi cluster to replace an aging NUC machine, probably running MicroK8s
Some Shelly lightswitches (they run over WiFi, and can be flashed with Tasmota)
Probably other Zigbee stuff, assuming it arrives.
Fiddling with HA
Also fiddling with Mycroft, possibly as a replacement for Google Assistant (Nothing against it as such, but I do want more stuff to work when we've no internet at home).

I note at this point that I do tend to go big or go home on hobbyist stuff. A month is a long time, let's see where we get.

But first, some decompression. And to put up the tree.

November 22 2020

2020-11-22T19:08:00+00:00

Short a quick trip to Zurich in January 2020, the trip I was on was the last time I travelled outside Ireland, until now. I also don't plan on going anywhere until mid-2021 or thereabouts.

Much as all that's been written about how this year has been, one of the largest lessons I take away from it is that it called me and many folks in probably a similar position on some assumptions.

What would happen if you didn't have to go to work every day? Either because you were a remote worker, or just because. You'd have more 'time in the day', fewer excuses about getting exercise and eating healthy, and so on. Notwithstanding that COVID has been a terrible experience for pretty much everyone, after a certain point you have to get to something like normalcy, and see if it's what you expected.

In my case, I put on a load of weight, didn't really leave the house for a few months, and went back on SSRIs. I found a good therapist and I've mostly got the extra few kilos back off again, with more to come. It took a while but I think I am adjusting.

I had always had in my mind that I'd take some time off before I retire. By 'some', I mean maybe a year or two. It always seemed strange to me that people would work until they're in their late 60s, after they've lost the ability, freedom and inclination to do the things they can't do because of work.

I had an imaginary expectation that if I suddeny didn't have to go to work any more, that fulfilment and relaxation would fill the gaps -- in reality, I really hadn't thought about it all that much. If I stopped working tomorrow, I would enjoy the situation for...maybe a month? Two months? I'd then have to do something. My platonic ideal had been having nothing to do, and now I know that just makes me anxious :-)

This past 9 months has been a microcosm of that. My last almost 20 years has been me being on the road every month or two, and that's all gone now. Do I miss it? Honestly, no. I enjoyed some parts of travel, there were good experiences; but I'm really not crazy to get back on the road again. I could take or leave business travel, honestly. I often said before that the amount of business travel I do kinds ruins holidays for me - the minute I go near the airport I'm in work mode. I'm trying to manage the situation, and getting stressed about everything going smoothly, and waiting for the trip to be over so I can go home. So in some ways I am kinda looking forward to travel for travel's sake again. But, I can wait.

I could write more words about the future of work than will fit here -- I think this year has been an accelerant to that progress, but we are heading toward a brick wall where we have to deal with the social nature of humans and how we can't substitute that with technology.

Anyway. Same time next year?

November 09 2019

2019-11-09T14:14:00+00:00

It's been over two years, and while I'm still patiently waiting for the demise of Medium, it only took re-running pelican-quickstart to make my makefiles work again. Plus we haven't deprecated serving static content just yet, so there's that.

Most of the selling points for blog platforms these days is some sort of federation thing, or the promise that more people will read your nonsense, or other stuff that's nothing to do with producing an actual log. It's something LiveJournal got right, in a way.

Anyway.

At the moment, I'm on a train from work on Seattle, to San Jose, which is also near work. In a toss-up between getting a 2-hour flight and sitting on my ass in the bay area for two days, I choose....22-hour train journey. It's being a pretty good opportunity to switch off, when these are few and far between these days, more than I'd care to admit. The very fact that I'm on the (offline editing) blog thing will give you an idea of the extent of the problem I have with disconnecting :-)

I last took the Coast Starlight maybe 8 or 9 years ago, that time from Oakland to Los Angeles. I remember it as a fascinating drive-by of some of the smaller parts of American life -- as someone who's really never visited the USA for pleasure purposes, it's easy to forget it's where millions of people live in small towns, and rolling past and getting a look at that brings me a sense of ease that even with an open text editor (the modern equivalent of a notebook), I can't properly try to express with words. It's a good feeling and I'll take it.

This trip to the Pacific Northwest, short as it was, was also a good reminder that it's a place I enjoy being at. Maybe it's the weather? Or the closeness to nature in the planning of the city. Anyway, it puts me in a mind to having a go at coming here for nonwork reasons. That doesn't happen a lot.

Ayway, that's yisser lot. Like always, I promise myself I'll update this more, if my brain is in the space to do so. And, most likely like always, it never bubbles to the top. Seeya when I see you :-)

An Update on Life

2017-02-06T11:15:00+00:00

The last few months have been a bit hectic, let's say that.

Work

In work, a few things happened. I got promoted to 'Director', which is the top end of the management progression, bottom of executive. What either of these mean, I'll probably never know. About a year or so ago, I expressed a bit of existential angst about how my job had become more about pottering about among the various pies I had a finger in, making generally well-meaning statements and giving advice. I felt like I wasn't getting a lot of work done, frankly. My boss at the time said "This is what a Director does", and that was a bit terrifying. I guess now I know what it feels like (all over again) to go from full-time engineering getting-things done to management, yet done in some other stratum. Your tolerance for delayed gratification has to go up another notch, as the stuff I'm up to now has a 3-5 year timeframe, as opposed to yearly or so, as opposed to quarterly.

It's been fun times thus far. Trying to find the balance between getting stuff done, and being hands-off. I honestly think getting this balance right is one of the primary things that stands in a middle manager's favour - the balance between getting stuck in and not letting the people under you grow, and feeling like you're not really doing anything. It's also a very situational thing. Every time I've gotten promoted in the last 5 years or so, I've thought "Right, that's me I guess. Topped out.". What this means in my head is that I've honestly no idea or concrete advice for other folks who want to go down the same path. Every Director I've ever spoken to about it had a different story. They're not going to tell you how to do it, they can only tell you how they did it.

The other thing that happened is that I guess I'm now head of SRE for Dublin. Again, this role means different things for different people - I'm not terribly figurehead-y in some ways, and I'm lucky to have lots of great folks around me to make sure I get mercilessly taken down a notch if I'm doing not enough or too much of things. I'm also happy to be back into the 'existential terror' mode of operation. I believe everyone should be reasonably sure they can do about 80% of their job. The other 20% are what keeps it interesting. This only happened a month ago, so I'm still figuring it out.

Home

Lots of stuff here as well! I am adulting very much, after a few years of not so much. I am generally bad at adulting so this is good, I guess?

Firstly, and I guess most significantly - myself and my partner of 8 years now live together properly and for realsies. There are various and sundry private reasons why this wasn't the case, and the thing that happened to make this happen is also private, but this is a big deal for me - sharing my life with someone else in so complete a way is a new thing for me (yes, despite being nearly 40, I've never actually lived with someone I was going out with). This has done wonders for my general well-being, in many, many small ways.

Semi-related - we're selling the house I bought almost 3 years ago, and moving into Dublin city proper. This was something I was thinking about for a long while, and the above thing also precipitated it. There are a couple of reasons why this is happening now, which I guess are relevant to most commuter-town folks.

For reference, I currently live about 40km outside Dublin, and I work in about as far east (i.e. the opposite end) in Dublin as you can. I commute by car or motorbike mostly, and it takes about an hour, at best. I've been doing this for 10 years. I had figured this was worth it, as I like having privacy, space, and so forth. Also, the way the property market is, you get about twice as much bang for your buck compared to living in Dublin. Even if there were houses and gardens the size of our current place in Dublin, you're talking most of a million quid, whereas we're selling up for not even close to half that, and doing well out of it. Things are amazingly Dublin-centric.

I guess the reasons break down to:

Commute finally killing me.

It's getting worse rather than better. Economy is recovering, lots of people have cars and need to use them. My commute took 45-50 minutes on a good day 2-3 years back. right before christmas, I had a few days where it took over 2 hours. Fuck everything about that noise. I think having the motorbike took the sting out for a bit, but I realised back in the middle of last year that I was literally driving into Dublin every day, including weekends. Why don't I live there? Good question, obvious answer.

Health (mental and phyiscal).

I live along a main road. There's nowhere to walk if you don't like being run over by a truck. You have to drive everywhere. What this meant in practical terms is that I got little to no exercise, unless I made a point of it, drove to a place, exercised (walking, cycling, etc.), and then loaded up and went home. Back when I lived in Dublin before and moved out to the country, I distinctly remember putting on about 10kg just by missing out on the walk to the bus stop or down the shops. I spent a good chunk of the last 2 years getting literally no exercise. Walk 10 feet to the car, into and out of work, then that's it. I've put in about another 10kg in the lat 2 years, with really no sign of it slowing down. There was really only one way that was going. I'm not a gym person, and I find it very difficult to rationalise or will myself to get exercise that doesn't have a 'point'. I'm the least physically fit and heaviest I've ever been right now. I'm getting to an age where that's actually pretty dangerous. I'd rather wake up now than in a few years when I have my first coronary.

Secondly, it's just secluded. I don't have an active social life. I went to a pub a few weeks back on a whim, and realised I haven't done that in probably a year or more. All the folks I'd go to the pub with are either work peope, or live in Dublin. I already have pretty severe social anxiety, that isn't getting helped by living in a hermitage. So, even if it makes me a bit uncomfortable, something has to change.

Services.

Services for peope who live outside cities and large towns are getting worse, rather than better. Ireland has always been a city-centric (and specifically Dublin-centric) country, and this shows no signs of going in the other direction. Broadband is pathetic (I get about 50Mb on an LTE connection, with a 20GB per day download limit). In the last few years, getting tradespeople to come out to a place has gotten harder as the economy picks up. Case in point, my electric gates broke down a few weeks back. The original installer said "I won't be able to come out to you for at least 3 weeks", and replied "dunno" when I asked what else I could do. Rural Ireland is kind of fucked. By that I mean living outside towns. I could move down the road a mile and probably do a lot better with services - the appetite to go the last mile is just not there.

Because I can.

I've been pretty lucky in that I have a job that pays well, and I really don't have any outlays or expensive habits (aside from Airsoft kit, which granted isn't that expensive in the grander scheme of things, if you compare it to golf or something).

So, I have the ability to live in Dublin, that I perhaps less so had 10 years ago, when I first moved out. So, it's time to embrace the privilege a little bit.

Anyway, all these reasons got written down sometime last year, and we started looking, as well as putting our place on the market. It sold reasonably quickly, and right now we're going to rent for a bit and take our time. I'm going to have to get used to some stuff I used to take for granted (being able to walk! to a shop! public transport! taxis that don't cost 80 quid!), and I'm looking forward to it.

Of course, all of the above manifests itself as my being a ball of abject stress for probably the next 6-12 months. But, it'll all be worth it, right?

Everybody's Free (To Use Their Best Judgement)

2016-09-26T11:22:00+01:00

This is something I wrote for new Googlers, drinking from the information hose. It's not formally part of the onboarding process, but enough people email me about it that I suspect it's passed around a lot informally.

(With Apologies to Mary Schmich)

Everybody's Free (To Use Their Best Judgment)

Conferences for Introverts

2016-08-20T17:29:00+01:00

This is a paper I wrote for consumption of people doing conferences and trainings at Google. I figured it was more generally applicable, so here it is: Conferences for Introverts

June 12 2016

2016-06-12T12:04:00+01:00

I was away last week with work to an offsite at the Club Med, which as far as I can gather is one of those soviet-era collective holiday camps for people who weren't in a soviet. I bet the soviets never had nespresso on demand.

Bizarrely, the weather in Ireland was actually better than in the south of France. No complaints though :-)

This week, the weather at home is fighting with itself -- sunshine and high (for Ireland) temperatures, with "showers" that range in intensity from light drizzle to attempted murder. Will still try to bike it to work as I can this week. Actually the first full week in many weeks, so trying to git 'er done moreso than I have been.

June 05 2016

2016-06-05T19:44:00+01:00

The weather turns and turns. We're well into the season where things spring out, and make you cut them back, or just be happy they're there. Our garden has rhubarb. Enough for a tiny put of jam, or perhaps a Mr. Kipling's style rhubarb tartlet or two.

It also is the season for camping, weekend activities that involve outside, and being on the motorbike a lot more. I was away for much of last week, looking forward to more of the outside, assuming the last week or so wasn't a trick.

May 25 2016

2016-05-25T13:05:00+01:00

Further to the Reddit AMA I was involved in a few years back (I am /u/sre_pointyhair) - "The Guy" showed up at my desk last week.

Alas, I was away, and he had to make do with doing the unspeakable things we do to keep servers running from my desk for a while.

Thankfully, nothing exploded.

Imperfect Machines

2016-05-19T23:33:00+01:00

I finally got my grubby mitts on an actual dead-tree copy of the SRE book. I've wanted this to exist for many years, and it's finally out there. I wrote a chapter on working with teams with high interrupt loads where the team consists of humans. It's a paraphrase of a paper I wrote a few years back that's used internally across a few teams.

Much as we all need to get grunge work done, often the 'fair' way to divide it is unfair to everyone individually. People work best where they get to do one thing well for as long an uninterrupted session of time as possible. It's a concept that I get pushed back on quite a bit, as it's dangerously close to claiming human can't multitask, and people love to claim they can. But, it's a pretty simple concept that's worked well where applied. Polarise time, sanity may well follow.

April 30 2016

2016-04-30T19:14:00+01:00

Back from a few days in Prague and beyond, at Border War. This was a very decent one - and I ended up not being as wrecked as I thought (due to the previously mentioned unfitness). Did about 45km of hiking over the 3 days of the event, which is a lot for me.

This (as usual) has sparked plans for more trips. Work has been busy (in a good way) for a while, so plans for non-work are spinning up, which I guess is the real sign of spring (I've got things I could be doing for about half the weekends until July already).

More of this sort of thing, plz.

The Technical Contrarian

2016-04-16T22:17:00+01:00

So, case in point (since we love examples).

A couple of years ago, when I first started fiddling with Pelican, one of the first things in the docs was 'okay, set up virtualenv and do such a thing'. My initial reaction, as an old-skool sysadmin weenie who hasn't had to actually run a linux install that I care about in many years, was to go 'pfft', and just get python going on the machine I wanted to use pelican on (i.e. my macbook). This, of course, resulted in a world of hurt, as the first thing you do is pick between pip and easy_install, and your first step being to pick a package manager is never a good sign (I've used python for years, so I'm kinda familiar with its vagaries, but y'know. All the same.).

But, step one for me was to run into virtualenv, declare it to be newfangled, and do what I know. I don't feel like I'm a person who rejects things out of hand for being newfangled all on their own. However, that was my initial instinct.

So, when I decided I'd do a bit more writing a few weeks back, I threw ludditism to the wind, and said "OK, let's use virtualenv'. I was up and running in less than 5 minutes. Now I have a (kind of, I assume) idempotent virtual environment for python sitting in dropbox, that I assume will work on other OSX installs. Or maybe elsewhere. Have yet to test.

But the point is, it was quick, painless, and I probably could have done the same thing a few years ago and saved a bunch of hassle. But I didn't -- and I'm interested in why.

It may, of course, go back to one of the truisms I like to come out with, which is People enjoy things they remember. I remember how to install stuff from source, how ./configure && make && make install was the old curl blah | bash, and even though the logical brain knows that the thing that's had thousands of hours of work, and tens of thousands of hours of nerd scrutiny applied to it, and is what the software maintainer recommends will probably work, and may surprise you, you're still gonna go "Install packages? Yeah, I've done that. I'll take it from here, documentation". It's like following the first half of the IKEA instructions and then reckoning you can wing it from there.

Nothing like new ways of doing things to make you feel a bit left behind, eh?

But, it's hard to see what's actually a good way of doing things, and what's generally a bad idea. This varies by your risk aversion, and how bothered you are, and how much you like/trust/care for the community behind this stuff. You used to be able to get a good feel for what's a decent idea as far as sysadmin goes. Now it seems more like you have to be careful about picking your poison.

Other case in point! -- Docker Hub.

I've been playing a bit with Docker for a while, and I've been happy enough with how it functions so far. I haven't in any way stressed it as far as functionality goes (mainly I've been using it at home for Plex, Sonarr, SabNZBd, for, uh, reasons). However, one of the first things you get exposed to is docker hub -- which is basically a giant Makefile repository for docker images.

It gives me the fear.

I don't know why it gives me the fear in a more substantial way than, say, running any software, but I'm still kinda finding it hard in my brain to distinguish it from the sudo | bash school of package management, in terms of trusting your machine to a random thing from internet strangers.

But, we run software written by strangers all the time, right? There's no 'score' assigned to any of this software like docker images from docker hub, right? I've spent long enough reading occasional stories about how some prominent OSS people are deranged maniacs to know that the scrutiny leveled at, say, the sabnzbd or sonarr software itself is far less than the 111 stars and 100k+ pulls that the highest-rated plex Dockerfile has. Hell, Plex is closed source so who knows what it's doing?

You might trust the crowd. That's fine, too. It might result in just as simple a life as only trusting yourself, all things considered.

I guess it comes back to one of the key interests I have, that I've yet to tease out into an 'area of study', really -- the effect of human instinct, emotion and judgement on how we go about interacting with software, especially software we rely on to stay working. People mistake 'correct' for 'safe' or 'familiar' or 'comfortable' all the time, and there's very little acknowledgement of the basic emotions at play. People imagine it as a very dry, scientific discipline -- but some of the most strident arguments I've encountered have been over two perfectly fine methods, software elements, processes, or whatever you run into when dealing with these ridiculous computer contraptions we deal with.

Maybe you think vi is better than emacs because you have muscle memory, or because your girlfriend tought you to use it and that's a happy memory, or you personally have it set up just-so, and it's a classic nerd-brain trope to imagine everyone is just like you, because things are simpler that way?

Or maybe vi's better than emacs because it just is and I'm tired of this argument, so shut your damn hell mouth.

Border War Prep 2016

2016-04-16T21:54:00+01:00

Some of the gear that's being shipped to Czech Republic (or Czechia if you don't please) for Border War next week. I and 20-odd other tubby nerd militia follow it all next week. This will be my third time in as many years, and probably the unfittest I've been in about that time. Let's see if I manage to hold my own or expire (It's 3 days of camping and about 30-40km of hiking with a load of gear on you).

Thankfully, the weather looks to be clearing up and staying cleared up, which means a bunch of motorbike commuting (and gadding about). This is a good thing for the brains.

April 11 2016

2016-04-11T00:42:00+01:00

I almost signed up for a medium account. A brief moment of madness before I realised that the likes of medium accounts are also transient. Reminded, again, by my recent de-facto abandonment of G+ as a thing to publish things on. I am not a crackpot, our outlets for writing are transient and we're in danger of not having a narrative at all if (some of us) don't self-curate :-)

In since-last-year news, things are progressing, for certain values of progress. As far as the Internet at large is concerned, I now have about twice as many people who have the good|bad|indifferent fortune of being in my organisation at work (I know, "my organisation", right?). Things have gotten really busy since about the start of this year, but in a good way.

It is my solemn belief that the average person should know how to do about 90% of their job. Thereabouts.

I also wrote a chapter for the SRE Book. Not on anything technical of course. On manager-nerd stuff. A dabbler in code and people. It was a paper I'd had sitting internally for a while, that is now seeing the light of day, in edited form. I am glad this book exists, as I have honestly believed that what we're doing is capital-I Important.

I also get on. I get on okay. How are you?

February 17 2015

2015-02-17T20:59:00+00:00

The weather's turning. It's one of those nice times of year when it's cold and mostly dry, but without a winter in front of us. Things are growing at home and we're seeing what pops up on our first spring in the new place.

So, on Valentine's day, I reminded myself of how much I love myself, and got a new motorbike. It's the little things. Love you lots babe xoxo.

I'm currently in London, doing a short trip for work. Have managed only short trips so far (to Zurich, and now here) -- I've a few longer trips coming up, but they should be mostly uneventful. I don't have the appetite for long trips any more, to anywhere. There's too much at home.

As the weather gets nicer, I also mean to get fitter (i.e. fitter than not fit at all). The house buying moved us further out from work, and introduced stress that means I've been off the self-powered bike for a while. I've a pretty big Airsoft Event coming up in April, and not dropping dead from running for a few hundred metres would be just great. I'm funny with exercise in that I can't stick to it unless it has a point. So, bike commuting was great. Gym, not so much.

Of course, I say this after inhaling a Pretty Nice Burger in a local place here. Whoops-a-daisy.

January 11 2015

2015-01-11T15:30:00+00:00

Domestic weekend! The first week back at work was one of those stealth first-weeks-back, where the first part is relief that nothing exploded in my 3 weeks away, and the rest was EVERYTHING HAPPENING. But it's the weekend now, and Amazon has a fancy Karcher window vacuum on sale. A window vacuum, if you don't mind. It actually worked, which restored my faith in branding (I wouldn't have bothered with one and would have declared them bullshit if Karcher hadn't been making one).

Somoene mentioned Medium to me last week, as a means to 'tweet a thing with more words than a tweet'. Over the years, I've used a bunch of things to keep an accessible log. I used to use LiveJournal until some russian mobsters bought it. I used to keep up with people on some NNTP groups, or via email. I don't really do email with people any more, which is a shame in a sort of compresed-time way. Remember when we used to write letters? I don't.

So, I'm pretty sure that some HTML files in a directory aren't going to get deprecated or fall out of favour any time soon. I can carry a means to reproduce them anywhere with me, I can rejigger them to some other format if the world moves on, I can use some other form of syndication, because I have the originals in markdown. You could argue that it won't fall out of favour because it isn't in favour, and you'd be right :-)

I sometimes think about how temporary a lot of social media is. People from the older school of personal journaling or websites came into it with an expectation that it'd be a record of their travels through life. It really hasn't become that, and I think that's fine - it is what it defines itself to be, which is a moving set of goalposts. Will Facebook be around in 40+ years, when I hope to be still here? I don't know, and neither do you. Will it offer a means to search for the times I talked about a particular thing at any point in the past? It doesn't even do that now!

And, just in the interests of fairness, neither do G+ or twitter in any meaninfgul way.

But, all this is a huge distraction from how I need to clean my kitchen.

January 01 2015

2015-01-01T02:05:00+00:00

So, every October or so, I ask HR at work for a report of people on my team (I have somehow tricked The Company, or they have tricked me, into being responsible for about 40 people) who haven't taken enough vacation, since we can only carry over 5 days. Due to accounting messing about for my leave earlier this year, It turns out I had 17 days left, and could only carry over 5. "Whoops", I thought.

So, I took 3 weeks off. My leave in the summer was good, and overdue, but stressful and punctuated by having to call people, stand over people as they do jobs, worry about things going well (Ever had to hope a plumber has a steady hand drilling a hole in a 500 euro ceramic sink, as you race to a kitchen suppliers to get parts they forgot to send out and you needed today? That was a fun tuesday morning). A lot of it felt a lot more like work than I would have liked. So, this time round, I made a small list of "Things I will feel like a failure as a person for not getting done". I then promptly did about half of them, then declared "Fuck it, it's Christmas" and sat around enjoying the outcome of the previous time spent working. It has been glorious.

2014 has been one of the more eventful years in the last few. Of course, I moved house, for the first time in almost 10 years. The above picture is my workbench for doing electronics messing and airsoft, which only got unpacked and set up this week. This will give you an idea of my approach to moving. I still have a roomful of boxes.

I also did more of my "Doing things is better than not doing things" stuff. For me, as an on-again off-again GAD sufferer, this has brought about an amazing sense of well-being. The alternative to figuring out how to be alive like this is the slow death of coasting through life without ever challenging yourself. I'm doing it in small way. I'm not about to take up lion-taming, but I'm doing my bit. I did the mandatory training and started riding a motorcycle, which was a big unnecessary thing I thought would help, and it has. It's easy (and especially easy for me) to rationalise away any thoughts of doing things that aren't strictly necessary - it's a defence mechanism against failure. I'm trying my best to do things when my brain goes "Bleh". I'm also trying my best to fulfil a few of the "Wouldn't it be cool if..." ideas I often have and then squirrel away for 'later'. There's no later.

Hopes for 2015? I'm trying hard to have specific ones. It's not something I think about a lot, and the 'five year plans are for losers' attitude that got me through my 20s has worn thin a little bit. So, I'm still thinking about it. I hope I can do more stuff that I've only thought of doing but then threw away the idea before. I hope I can figure out a reproducible way of making myself happy. I hope I can be a better partner, a better manager, and a more open person without stressing myself out or being uncomfortable all the time.

It starts with doing things. Because that is, after all, better than not doing things.

December 07 2014

2014-12-07T13:48:00+00:00

The last of this year's travel is ongoing. I haven't had a weekend in NYC in a while, and it's managed not to snow this time. A decent day yesterday walking around the park and times square in the rain, and visiting MoMA for a bit.

This week is more synergy - a meeting of managers from all over to talk about The Company's storage, the teams that run it, and various other bits. It's the stuff I'm interested enough in that it doesn't feel like work. Good times.

Here until thursday or so, the heading home to a bunch of time off for xmas and new years. I've alwys taken the two weeks surrounding xmas and new year's off, for my entire working life. Of course, this means that the week before the week before xmas always ends up being an exercise in clock-watching (yes, even if I do like my job. It's not quite at the level of 'xmas every day' just yet).

So, this year, I took that week off as well. I'm sure nothing bad can come of this.

November 12 2014

2014-11-12T22:15:00+00:00

Home again, at least until next month.

You'd almost think they missed me.

November 4 2014

2014-11-04T22:02:00+00:00

Excellent korean BBQ is a lot less cheaty than sausage and stodge.

As hoped, got a bunch of stuff done on the plane, and managed to not have an awful flight. I can tell my sinuses aren't going to like it later, but for the moment I can take solace in meat.

Mountain View is as it usually is, except it feels a lot more like it's reached its tipping point as far as straining the local infrastructure goes. People don't want to live near the office (because let's face it, Mountain View itself is a bit of a shithole), so they end up commuting >1h from San Francisco (which is also a shithole) every day. At some point, being in the centre of the universe gets outweighed by a declining quality of life. Hopefully something creative happens, since I don't think it's a problem unique to The Company.

One more day in the office, then hitting the red-eye to NYC tomorrow night. I expect that to be a less pleasant flight, but at least I'll then be in NYC.

November 1 2014

2014-11-01T21:43:00+00:00

Off to Mountain View in the morning for 3 days, then onwards to New York. I haven't been to MTV since January, or NYC since June, so I guess I'm due in both.

This is the aeroplane survival kit. I don't sleep on planes, and Aer Lingus have transatlantic WiFi now, so this is indispensble. That's the work phone (Nexus 4), work tablet (Nexus 10), and the newly-acquired Moto 360 smartwatch, which the jury is still out on (pretty shiny so far though).

Three other pieces of indisensible kit are:

Anker ASTRO E5 battery pack for charging hungry gadgets on the road.
Anker 5-way USB charging thingo (2.4A on each of 5 ports for maximum charging goodness)
ASUS WL-330NUL USB ethernet and mobile hotspot, for sharing stupid one-mac-address wifi with all your gadgets (seriously, everyone should have one of these).

Hopefully I'll get some stuff done on the plane - I've yet to properly emulate the distraction-deprivation environment that planes give you.

Seeya on the other side, internet.

October 21 2014

2014-10-21T21:36:00+01:00

At a work conference in Munich, which I've never been to before. From what I have seen of restaurants here, it is a Paleo disaster, so I've been pretty cheat-y so far.

As middle-european cities go, it's pretty nice -- ageing but efficient train service, pants-shitting taxi rides from the airport at 220kph, and good food and company.

I get to spend essentially no free time here, but maybe I'll come back sometime. For now, I get to go back home for a week, then do a short-in-duration but long-in-distance jaunt. The joy.

October 18 2014

2014-10-18T14:30:00+01:00

More inactivity! Mainly it's been ridiculously busy in work, mostly around performance reviews and a number of things sneaking up on me. They're sneaky, those things.

Doing a little more travel before the end of the year -- Munich, London, Mountain View and New York (twice). Yay. I've been very travel-light this year, so I guess I gots to pay my dues.

In other news, I own a motorbike now! It's a 2004 Honda NT650V Deauville. It'll do for now - I meanly needed a bigger bike after I discovered that the 250cc bike I was on was hardly pulling me, and I found it way, way easier to ride the school bike (an SV650S). Of course, I'd pick a day with gale-force winds to pick it up.

I've also booked my test, on advice of the instructor - I did an insurance assessment and was given a grade equivalent to 'has a full license and is a daily rider'. Which had the nice side-effect of 30% off my insurance. Let's hope I can keep that up when I don't have a dude following me and roaring at me over a radio.

October 05 2014

2014-10-05T11:51:00+01:00

Trying to keep this thing up by doing worthwhile things. That was the idea, right?

I've been trying Paleo for the last few weeks, with a drop of about 1.1kg per week so far. Seems normal enough for exercise free diet change (from previous experience), and it's a bit easier on the palate than just caloric reduction or other stuff - we'll see how long it lasts. I've noticed a bit more energy certainly, and less feeling bloated and horrible. The first couple of weeks off work really took it out of me, and the brain can't really perform if the body isn't keeping up. It's too far out to be cycling to work any more, so diet will have to come first.

Motorbike stuff is still in the pant-shitting terror mode for a lot of it, which I hear is quite normal and healthy. I know I see it in the car as well, but inconsiderate arseholes are legion on Irish roads - people should have to spend some time on a bike or motorbike to understand this. On the other hand, we've had some great weather the last few weeks, and I'm enjoying the hell out of the long straight bits. Back in this week for some more refresher lessons.

Have also solidified travel for the next while - going to Munich, London, Mountain View and NYC before the end of the year; in short trips for the most part. Also, got a ticket for Border War next year - I'm so there I'm not even here.

September 16th 2014

2014-09-16T14:15:00+01:00

Away in Zurich for a few days with work. Partially to meet a bunch of people, but also to get into the practice of being away from home in small doses. I've a few more trips coming up (London, Munich, etc) that'll also help.

I've been dealing with a bit of low-level anxiety about leaving the house empty - it's a good bit more secluded than the old place, which means I don't like leaving it alone. For this and a few future trips, I have someone going in there pretty much every day (or living there), but I just have to deal with it -- Alarm systems, CCTV, home automation only go so far. If stuff happens, it happens. Bizarrely, in my head it's the cats I'd be more worried about. Stuff is just stuff.

I was in Zurich in June, but didn't get to see any of it -- so I'm getting around this time by walking. I really like old european cities, where people actually use the space. It's one of the things I don't like about Dublin -- people don't look like they're using the space and kind of scurry around between Things They Gotta Do. Hard to explain. What I have learned is that I need a comfortable pair of walking shoes, and to become less of a fat sack of shit (I stopped getting any regular exercise right when I bought the new house and stress took over. Need to reverse that trend.)

I got to do some gaming over the weekend at Heckcon, which was nice - I've been trying to do a little more of this, since it was one of the things I consciously enjoyed that I just petered out with doing. I'm getting set up for Gaelcon in a month or so, and may even run a game. There's also a set of people that I only really run into at cons, and it'll be good to get kept up.

September 12th 2014

2014-09-12T21:11:00+01:00

Second week back, and seeing what my job ends up looking like for the next while. Think 'busy'. I'm trying to nail down travel until the end of the year, it being increasingly impossible. I'll be in Zurich for a while next week, and after that is anyone's guess.

Other delightful happenings: Someone working on another building in the campus our building is in managed to reverse a van into the motorbike. The only real damage was to the airblade windscreen, which got broken pretty much clean in half (see picture). After removing the remains, I discovered that all it was really doing was redirecting wind into my face, and riding the bike got a bit nicer to do. So, thanks clumsy van driver I guess.

September 7th 2014

2014-09-07T21:11:00+01:00

First week back at work - done!

Things have been more physically tiring than otherwise. I've been trying to nail down the next couple of months travel wise, and managed to make it into work on the bike two days out of five, which was good. That's been helped along by some good weather, and unhelped by some impatient drivers (that get caught on my helmet cam and youtubed).

This weekend saw some Airsoft, and some moving of larger items from the old house. It's not quite the insuroutable chore I've assumed it would be any more, a couple more trips will do it, I think.

August 31st 2014

2014-08-31T21:30:00+01:00

Last day of leave, and a day of normalcy and domesticity. Mainly hung around and enjoyed the last of the days since the end of June, when I last went near work. Tomorrow will see some hectic times, when I figure out what's happened while I've been away. If I've been doing it right, very little! The best work disappears, as they say.

These few months off have given me a bit of time to think about larger things -- I've gotten a lot of stuff done, and have thought about a lot more stuff. My brain has not been terribly healthy these last few years, and part of what I hoped to figure out was what I might end up doing (or not doing) to make it be a little healtier. I had a couple of bad signs, like mood swings that weren't linked to any external stimuli, that mean I may end up seeing a professional again at some point. However, on other fronts things are a lot better.

For one, I've finally acknowledged that I'm actually not made happy by being lazy. The last proper frame of reference I had with enough time to introspect on it was college and soon after, where I could do stuff like spend a week playing Baldur's Gate, or similar goofing off. I often stressed myself out in recent years because I don't seem to get the time to do that sort of thing any more. However, given the opportunity to do so, I found myself not really able for it - the thought of not doing something productive, or not doing something at least worthwhile outweighed the stress for not being able to goof off (if that makes sense). So tl;dr: The marathon laziness sessions I get stressed about not doing are actually bad for me when I get to do them. I don't like them. The reason in my head, at least, seems to be that the part of my brain that wants things is a bit disconnected from the part that enjoys experiences and feeds back. It also probably means that I did some growing as a person without really noticing. So, I got to rationalise away a source of stress :-)

For another: this leave almost acted as a catalyst for realising what I'm going to do, ever, and what I'm not going to do. For a long time, I've had in the back of my brain "Someday I'll do X", where X is a thing that's completely possible for me to do now. Someday I'll go travel in a particular place. Someday I'll learn a particular skill. It'll happen in the empty space that I can retionalise away as "When I have time off work". I just had time off work, and only got one or two of these things done (the motorcycle thing, and some other social things I may go into later). Now that I'm back at work, I realise: there is no later. Unless you're prepared to make a long list and wait for retirement (and hope you make it in one piece and are able-bodied enough to tick them off) you're just not going to do these things. You sweep them away forward into a fold in time, which usually means 'never'. So, if I find myself going "I'd like to do X sometime when I've time", the time is now. Or soon. Make plans.

I've been extraordinarily lucky in my work life - I have a job I love doing, that pays well and is flexible about lifestyle. I have a Significant Other that is the world to me, and that I want to share my life with if I can. I have most of my life left to make of it what I can. The last thing I need is self-denial and some-other-time'ism holding me back. I need to keep that in my head as I get back into a routine, so it doesn't become too routine.

Of course, if I could find my work ID, this would all be a lot more straightforward.

August 23rd 2014

2014-08-23T22:24:00+01:00

Not a lot got done today, which is as it should be. Went for a wee motorcycle ride, which turned into a spin to, and around the phoenix park. Getting more confident and enjoying it more now that there are fewer moments of pant-shitting terror. More of this this week, I think.

August 22nd 2014

2014-08-22T19:06:00+01:00

I think this is the last of the work on the new house. Today, the painting of the exterior finished, the plumbing was finished inside, and the garage/shed got alarmed (we quickly calmed it down). Just in time for a week left in my time off work!

All that's left is to clear out the old house of stuff I care about, and point the "Make ready for new tenants" ray-gun at it, care of the local letting agent. Hopefully I'll get that done early next week, just in time for a few days off with nothing to do before going back to work.

I think, all in all, that despite the house move taking up a big chunk of my 2 months off, I'm still an awful lot happier with how things have turned out than if I'd kept trying to do it while working full-time. I can't imagine having done all this properly with only evenings and weekends.

I'll do a proper write-up of thoughts on this at some point, this margin cannot contain, etc.

August 20th 2014

2014-08-20T01:20:00+01:00

Sleep cycle being fucked up. That's the holiday sensation I know so well!

Last weekend, I was at a large (well, the largest of its kind in Ireland), Airsoft game, which meant a lot of driving and walking around, and not a lot of sleep. My feeble hu-man body is still getting over it, via some seriously lazy weekdays.

Tomorrow should see the beginnings of the hideous sickly-yellow colour the previous owners painted the outside of the house disappearing, to be replaced by a hopefully less sickly light green. I am not pretending to be even remotely competent at painting, so we're getting someone in to do it.

Me, I've been taking the last few weeks of my leave to spend time with favourite lady, enjoy the new place, and get to some nerding -- I've been messing about with LightwaveRF and getting it working nicely with my existing z-wave setup (and learning LUA and writing more golang in the process). More on that later.

August 10th 2014

2014-08-10T18:21:00+01:00

A week between updates, but it's been a reasonably busy one. Still moving a bunch of stuff from the old house, still trying to get out on the (motor)bike every day or so (in between impressive rainshowers).

Friday was a small get-together with some folks, as a bit of a "Yep, I definitely live here now" wake-up call. The smaller touches are also happening, like home automation and the like. I think tis is possibly the first day I've been able to sit in the office and get some solid nerding done. I'm currently experimenting with using LightwaveRF stuff for light switches, since their switch gear is a bit cheaper and more predictable tan z-wave (which I'm using for everything else).

This week, I have to sort out some smaller bits (like the CCTV company that have left me waiting a solid 3 weeks to finish an install. Bad show, MAC Alarms). Also, some more moving (I still havent'e mptied the old house. The last big job is getting that rented out).

Back to the nerd-mobile.

August 3rd 2014

2014-08-03T11:40:00+01:00

Updates getting more infrequent, so I should make it my business to do more interesting things, right?

Thursday and friday were the IBT course, which I passed, making me apparently eligible to ride a motorycle of any size without restriction, as long as I wear a newbie hi-vis. That seems kind of terrifying.

Yesterday, I picked up the rest of the bike gear/PPE, so all I'm missing now is a bike. And for this rain to feck off so it's safe to ride it.

This week looks like it's moving the rest of the stuff from the old house, and finding a rental agency that wants to turn the old house from a mess into paying tenants. Possibly interspersed with bike shopping, but there's no rush on that.

July 30th 2014

2014-07-30T16:20:00+01:00

Thge last few days haven't felt like particularly urgent ones, which is good -- I think I'm winding down the urgency level overall, and the time off work is kicking in. I had originally intended to take a month off, and was convinced to take 2 by a co-worker. In retrospect, only taking a month would have been a mistake. I think I'm only really getting into things now, 4 weeks in. Having to go back to work next monday would be bad.

The computers/gear/network gubbins are mostly set up in the new place, which makes things more tolerable entertainment-wise. The software side of it will come together - I use some dodgy python scripts to do a lot of the automation stuff, and will have to do stuff like turn downloads off while I'm at home (Internet connection going from 200Mb->6Mb), and various other bits.

The next few days are a bit busier, my bike training is tomorrow, and Favourite Lady gets home as well. Hopefully updates will pick up.

July 27th 2014

2014-07-27T23:12:00+01:00

26 and 27 have been days off. I said before about needing to take 'days off' from the moving and the Stuff I Gotta Do - these days turned out that way.

Friday night, I got one of the annoying headaches I get if I get too tired or stressed, so that, if nothing else, was the sign that it was time for a break. So, yesterday and today was mostly lounging around, being amused by the cats noticing things. There's a lot more wildlife outside the windows of this house (more kinds of birds, rabbits and hares, and a local cat they haven't seen yet). It's been a tiring experience when you've got lots of things to notice.

Ping has stopped hiding, at least, which is encouraging. She's more of a creature of habit, and didn't explore the house at all the first day. She's currently trotting around smelling things, so that's good.

Tomorrow, I get to do some delicious physical labour, moving things from the old house, and moving lumber and various bits from the back garden to the site of the skip I really must get for it one of these days.

This week is probably the first week with nothign house-related scheduled -- the only thing scheduled is my IBT course on thurs/fri, and some nerd stuff on thursday evening. That'll do.

July 25th 2014

2014-07-25T13:19:00+01:00

Yesterday was Moving Day for small cats everywhere. As predicted, Stranger didn't care and just wanted her head scratched, and Ping stared bloody murder and found some hiding places.

Overnight was uneventful, aside from the CCTV making noises at 0330 and complaining about SMART errors. Guess it doesn't like being on its side. Whoops.

Today, I am generally being around so the cats can have some semblance of normality, and I can catch up on lounging. Some goodies arrived from Vesternet so I'm going to take a ook at that later.

For now, I'm reading The Finder Library and enjoying the sun while it lasts. The cats are making various sorties into the rest of the house and being hilarious.

July 23rd 2014

2014-07-23T09:38:00+01:00

Yesterday, some dudes in a JCB showed up outside the gate to fill in a giant pothole that trucks and buses had been disappearing down for the last few months. It is my understanding that I should figure out who caused this to happen and then vote for them.

The very last of the work on the house drags oooooon. Today, an electrician does a number of small jobs, which sorts that out (and turns the tiny pointless room containing the entrance to the attic into a decent comms room). Tomorrow (hopefully) a plumber gets to drain and rebalance my radiators in order to fix a tiny drip from one of the radiator valves. The CCTV also gets finished off either today or tomorrow.

Still, I am an awful lot more moved-in than I was. The cats are used to me disappearing for a few days at a time, so them moving later in the week is fine, I think.

July 20th 2014

2014-07-21T12:46:00+01:00

I have moved, the cats have not. This means they get really needy when I'm in the old house. See above for an example: I sat down and Ping immediately sat on my arm.

House business: More furniture has arrived, the blinds and CCTV are being installed tomorrow (if people show up). Then the Cats may move, since all that's left is some electrical work.

Other business: Went to see "Monty Python Live (almost)" in VUE in Liffey Valley. Some nice touches added to the older bits, and they knew which bits to go Just Too Far on (and it was either abundantly clear that Cleese didn't want to be there, or he was putting it on really convincingly).

July 17th 2014

2014-07-17T23:17:00+01:00

If you ever want to get funny looks in an electronics shop, break out a measuring tape and start measuring microwaves. I got an incredulous "They're all standard sizes" today in Power City, while standing in front of a load of microwaves that were clearly Very Different Sizes. Our old microwave (which I actually saw on sale for an eye-watering amount) is too big for the fancy flippy arrangement in the new kitchen, so I got one that physically fits, and performs the one function of a microwave I've ever used ("Nuke on full power for N minutes"). The old microwave was also a browner, grill, and worryingly had a button marked 'Chaos Defrost'. So it gets flogged/donated/something.

Also today was the BER assessor looking at the house post-renovations. His only on-the-spot recommandations were something like "Eh, there are still one or two bulbs that aren't CFLs, and that's about it". So that's good. This assesment means free gubmint money to cover some small part of the heating/insulation work.

I was planning on tomorrow being a day off, but the solar installers never showed today, and some furniture is arriving tomorrow, as I found out today (2 weeks early!). Maybe I'll gt the weekend off?

July 16th 2014

2014-07-17T09:35:00+01:00

I own a sealed copy of Katamari Damacy for the PS2. You find these things out when you're moving and doing a cull of crap. Why do I own this? I don't even own a PS2 any more.

This was probably most of the day. That, and being asleep. I've found that between one thing and another, I'm still having to get up in the morning. This is, as they say, not what I signed up for. Going to have to sort that one out.

The evening saw some shooting of people, and then to bed for a delightful 5 hour nap, in time for getting up for the last of the builders (turning on the solar system, and the BER assessment).

July 15th 2014

2014-07-15T21:48:00+01:00

Today turned out a good bit more productive than expected. Headed up to Newry for some shopping (Well, for a particular item), then got back and did some painting -- Blinds are arriving on Friday hopefully, so the reveals around the windows needed to be painted before then. Good old Magnolia. The safe bet.

This evening is my first night spent sleeping in the new house. I have to say it feels a bit strange not to be surrounded by other people in houses attached to mine. Getting used to the noises of a new house, the random settling and banging noises throughout, the outside noises seeping in. Compared to the house in Clane, it's deathly quiet.

Tomorrow, I'm actually not sure what's on the cards. I've been asked to do some driving instruction, and will be going shooting people in the evening.

I also need to book some training. Check out what I bought in Newry:

July 14th 2014

2014-07-14T21:48:00+01:00

Eating an actual meal in what was essentially a building site a week or so ago is a strange experience.

In any case, today was waaaaay less productive than I thought it would be -- the last vestiges of Builder disapeared (the skip full of old stuff), and I got a new freezer. Aside from that I played a bunch of Hearthstone, which I'm not sure I like yet, and moved a few small bits from house to house. Tomorrow will be probably a writeoff too, going to do some shopping...

July 13th 2014

2014-07-13T21:48:00+01:00

As old-house starts looking less like a house, new-house starts looking like a way better house than old-house.

Today I bought new glasses, discovered that my prescription hasn't changed in 3 years (which I think is good), and then unpacked some nice chairs and a table we ordered a while back.

Then, back to old-house, and coming home to this:

I suspect tomorrow may be a mostly-day-off. I need to get back on the bike again, and this weekend has been hectic. Weekends and weekdays haven't blurred into a meaningless amalgam just yet, so I need to get on that!

July 12th 2014

2014-07-12T20:39:00+01:00

Busy day yesterday (and less busy today) so catching up today. The large furniture parts are moved, and some other stuff was delivered (this is a table and chairs). Lead times on furniture are up to a month or so, so this'll keep happening over the next while. I was able to get a little bit done, but these are mostly still boxes in a corridor.

Later, headed into Swords to a birthday thing for an Airsoft-buddy. I'm not usually a pub person, because I usually have a work or driving excuse, but I figured that this is the closest to a complete absence of an excuse I'll have, so I actually showed up. The plan was to be dirty stopouts, then up in the morning for some shoot shoots. That worked out delightfully, followed by some helping J and Oli move, followed by crashing on their spare bed for a while.

Home now, and having to face the terrifying prospect of an evening without television (it's still here, but I'm not arsed plugging it in). Tomorrow, there are some new-house bits to do, and some more J/Oli moving (their old place is close to Croker, so we have to wait for the bogger concentration to die down in the evening).

July 10th 2014

2014-07-10T23:57:00+01:00

2nd Van Day, and the old house is even barer. This will probably be my last sleep there. The cats get to stay here a while longer, until I have some painting and DIY and other non-cat-friendly stuff done.

The new house has what we assume is a colony of wild rabbits nearby, we usually scatter one or two whenever we walk around the back. They came in handy today when we left out some cuttings from a tree in the front, which they duly disposed of by nibbling and running around nervously.

Tomorrow I get solar panels turned on, and get to make a lot of phone calls (one of which will be about my being largely unable to make mobile phone calls in the new house. So Rural.).

July 9th 2014

2014-07-09T23:32:00+01:00

Starting to move. First, large furniture items, so rented another van from another place, and moved bed, TV stuff, various other bits. A slight oversight means I don't have a functional bed anywhere. Thankfully, I have a nice couch. It feels a lot more real and the house in Clane feels a lot more bare.

Later, headed up to Bellurgan for a site walkaround and meeting about a large Airsoft game, The Rionegro Insurgency, I'm helping with running next month. Five weeks to go, and surely more about this as the time gets nearer.

Sleep time. I have the Van tomorrow again, so will get the last of the large stuff moved, and can then move smaller bits at my leisure (this is the theory).

July 8th 2014

2014-07-08T21:46:00+01:00

A simple day in one way. Helped J move cats and possessions, rented a van and negotiated extremely tiny streets with same. Only managed to knock the wing mirror off once.

Afternoon-food in Woodstock, which never disappoints. Om nom.

Heading out in the morning to pick up the van for my own move, of at least large pieces of furniture that don't fit in the jeep. Oh joy.

July 7th 2014

2014-07-07T22:40:00+01:00

Another late morning. I am entitled. The only useful thing that happened today was painting one of the bedrooms. Tomorrow I get to drive a giant van around, helping J move to a new place. I then go get a different giant van for a few days to move my larger items. It's giant van week! So excite.

July 5th 2014

2014-07-06T14:36:00+01:00

No photo today. If there was, it'd be a soiled lab coat with some scary-looking weaponry on it. Went to an afternoon/evening/overnight Airsoft game, with a Walking Dead theme -- I got to stick on a lab coat and pretend to do unspeakable things to walkers in a loft. This went on until 3am or so. Good times.

I'm only out of bed now, which gives me a low-level anxiety about wasting the day still. I've been trying to intentionally take some days off from "Things I have to do" for "Things I'd like to do", and this may end up being one. As of tomorrow, I have No Excuse Whatsoever to not move house, so today may well be not doing anything.

July 4th 2014

2014-07-04T12:02:00+01:00

After a slight hiccup on the home stretch (a plumber having a go at drilling a hole in a ceramic sink, and succeeding against all odds), the house buildings works are done, finished, kaputt.

Now it's time for minor DIY plus moving, no more excuses. Also furniture shopping. Joy.

July 1st 2014

2014-07-01T22:47:00+01:00

Skipped a day, yesterday being mostly lost to what I'm assuming were sinus headaches, but which could have been stress or tiredness or a combination of both. Last time I bought a house, I ended up on a treadmill in a local clinic having some heart palpitations checked out. So things are going pretty well, considering.

Today was Kitchen day, the installers arriving first thing (as well as the Alarm guy, so now my alarm will send me a frantic SMS message if it goes off). After only slight mishaps (the hole in the ceiling for the extractor fan was measured based off old plans, so was a bit off. Whoops) I now have some of a kitchen. The main bits (sink, hob, oven) will go in hopefully tomorrow.

I also had new washer and drier delivered, and got to hear an entertaining rant from the delivery guy (apparently 'Robertstown' is some vague amalgam of a couple of townlands that can mean any where from just outside Naas to Enfield, so about 30 square km). A happy throwback to living in town is that I'm able to give directions to my house via well-known local pubs.

I also primed/sealed some plaster reveals in the rooms I didn't have profesionally painted, and hung some curtain rails. Aren't I a clever primate. I also now own a spirit level you could beat someone to death with (which, in fairness, is a critical criterion for any hand tool).

Tomorrow, since the actual tradespeople don't arrive until the evening, I'm going to take a day off house stuff. I actually haven't had a proper lie-in since my leave started, so I'm going to try that, then see what happens.

June 29th 2014

2014-06-29T20:13:00+01:00

My capacity to think of a witty title for every day is going to run dry eventully, so let's try this for a title convention -- protip: posts with the date as a title probably either have limited re-sale value or were boring in the first place :-)

Today I went and played some Airsoft, then met J in IKEA and picked up a few small bits (no furniture yet, or at all from IKEA if I get my way), then headed home via seeing the new floor. This was the last unfinished surface in the house, the walls and ceilings having been finished yesterday post-windows.

Since it was a big floor (spanning two rooms), we decided to go fairly swish, and weren't disappointed.

Must Do Things

2014-06-28T19:00:00+01:00

I'm guessing it's going to take at least a week to figure out that I don't have a schedule. Sadly, house renovations mean I actually do have a schedule of people showing up for the next week or so.

Today, I failed to sell an Oven, and succeeded in buying a new Airsoft gun. So there's something.

I also lit the first fire in the new house, coincidentally because I needed to dry the place out for a floor that's being laid tomorrow.

Last Day O' School

2014-06-27T22:37:00+01:00

Finished up in work today, not in again until the 1st September. Wandered around doing some house stuff (measuring windows for blinds, seeing if stuff needs doing tomorrow).

Next wek, the interior painting gets finished, along with the new kitchen arriving tuesday, and the last of the floors going down. We're then well into 'no excuse' territory :-)

Googlers who are following this: I should warn you -- it's not going to involve much highbrow insight until I sort out this block and tackle. Right now, this is "My Project", if you've read the excellent Nerd Handbook article.

I'm also going to be doing something essentially every day, as much as I can. I'm considering adding Disqus so people can comment, but in the meantime, email me.

For now, though, I'm tired. It's been a logn day of work. Time for a new start tomorrow.

Ow

2014-06-24T21:26:00+01:00

Finally (I am a male Irish person, after all. We're allergic to health professionals) went to the Physio today for a niggling problem I've had with one of my calves for the last while. What I thought was a bit of stiffness due to lack of exercise and a need to HTFU is actually Achilles Tendinitis, which is a lot more scientific of a diagnosis. Icepacks and stretches for a while. Just in time for wanting to use my calves a bunch, yay.

Also, figured out the last big renovation job on the house (a floor in two of the rooms). A good bit more than I was expecting to spend, but should be commensurately nicer. All things going well, we could be done by end of next week. Could be. All things going well. Yep.

Don't Forget: You're Here Forever

2014-06-23T19:28:00+01:00

Some zealous schwag-merchant deep in the innards of The Company was a month early with this -- probably giving me some time to mount it over the mantel of my plush leather and mahogony office before the big day.

Ironically, I'll not be in work for the big day -- I'm taking all of July and August off, so generally chill out, catch up on being lazy, and have more than 2 weeks off for the first time in my working life. I figure after 15 years or so I'm probably overdue.

Generally, my list of things to do for the two months amount to:

Move into new house.
Rent out old house.
Ride my bike a bunch.
Maybe get a new hat.

This seems like a solid set of objectives. I have vague plans to write more code, play more music, replay some games I've been meaning to. Anything after that seems like overachieving.

So, this week saw lots of mail show up at the office, some of which is being squirreled away for hacking on later. Today, the Spark board arrived, which will be fun, if I decide what to do with it at some point.

The idea of being able to talk over the network to stuff that's not supposed to is fascinating (which is why I've also picked up some z-wave toys for the new house, which I'll get to). This will be fun to mess with.

Ta-ta

2014-06-21T17:13:00+01:00

Having nothing to write about but "I made soup!" is something of a dampener on things (I didn't make soup).

Part of the point of this log is to capture some of that upcoming leave of absence I'm taking from work - I'm taking all of July and August off, and this is my last week in work before turning into a pumpkin.

The general idea is that if I've got nothing to write about every day or two, then I should probably be doing more stuff. I've put off a lot of things over the years, I should at least clear some things (I haven't made a list intentionally - that kind of invites disappointment, I think).

So, today was mostly sleeping and dealing with Jetlag. Tomorrow, hopefully less so.

Excuse the Dust

2014-06-19T17:06:00+01:00

Still fiddling with Pelican -- it's got a reasonably nice codebase, but the devs don't appear to be all that responsive. I've had to patch a few of their outstanding pull requests already to get things working just-so, this one most notably. Also some fiddling with setting up a lightbox for images, and importing old stuff from Blogger.

Today in NYC wasn't as oppressively hot as yesterday (and probably about the same as at home. I think I might have missed the Irish summer in my 3-day trip away). At least they're less apologetic about air conditioning here.

One more day here and I get to go see if my new house is still standing.

Tap tap tap

2014-06-18T21:39:00+01:00

So here's another attempt at keeping a log of some kind. the last one died because I stopped having (what I saw as) interesting things on it, blogger's editor bugged the fuck out of me, and general forgetting.

Part of my problem with having a web log is in the min-maxing of content -- you either detail every last thing and risk (hell, more than risk) being boring, or you hold out until you have some earth-shattering insight, and every post has to have one of those.

This race to the middle has been the death of previous attempts, I think. This time around I'm going to try to do something, anything on a regular basis. The aspiration is daily, or several times a week. We'll see how often it is. It's going to be boring and not-boring in about equal measure, I hope.

I'm also going to see if I can hack a bit on Pelican to see how it can munge some of the metadata I provide. I've been doing 'snippets' at work using markdown for a while, and have found it useful for grepping. The source files for this should hopefully be useful in the same way.

Using GMail as a relay with Postfix

2011-07-22T17:25:00+01:00

I fiddled a bit with this, so I thought I'd do a no-nonsense guide to doing this. There are other guides, but a lot of them are incomplete and will leave you with queueing issues or annoyances.

The science bit:

:::python
smtp_use_tls = yes
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options=noanonymous
smtp_sasl_tls_security_options=noanonymous
# Send all mail through smtp.gmail.com
relayhost = smtp.gmail.com
# Stop getting a catpcha, hopefully.
smtp_destination_rate_delay = 60
relay_destination_rate_delay = 60

Then, edit /etc/postfix/sasl_passwd:

:::python
smtp.gmail.com myemail@gmail.com:MyPaSsWoRd

It doesn't have to be an @gmail account, it can be a GAFYD account, once you've logged into it once to activate it.

Then just generate the map file for the sasl_passwd file and reload. As root:

:::python
postmap sasl_passwd
postfix reload

And you're done. Monitor your queue a few times over the first few days to make sure stupid stuff hasn't happened (the delay between mails might get tweaked), but in general this has worked for me.

Have fun.

Alive and Thinkin'

2011-05-26T13:55:00+01:00

I am alive. Apologies to folks who were following the electronics stuff -- the short version is that it worked, although I didn't get to use it 'in anger' as such. The other thing I learned is that when traveling through UK customs on the week of a royal wedding, having a small plastic box with wires coming out of it and a 9v battery strapped to it with tape is not a clever thing to have in your car (thankfully the customs official had a sense of humour).

Since then I've been doing a bunch of for-money work, and only took time out this week to relax. I occasionally take a week off to hang around at home and get 'life administrivia' done. Think of it as a concrete block on the see-saw of work-life balance.

I've been playing around a bit with generalised messaging for domotics updates. I'm thinking of a centralised controller that can accept events from various signals, such as the Current Cost meter, a wifi device coming in range, or a small IC hanging off the doorbell circuit, for example. A lot of these devices don't have particularly fully-functional network stacks, so I'll have to bite the bullet and get the handful of ZigBee radios I got recently working. Doorbell will be the fun one, I think. My house already has a (private) twitter account, and DMs me for various stuff (unknown hosts grabbing a DHCP lease on the network, etc.). Adding more stuff to this is the idea.

I'm also interested in the idea of cloud applications with local storage or local encryption keys -- so all your data is stored encrypted on the server side, and only you have the keys, so the content is decrypted at render-time. Stuff like 1password does this with secure stuff like password databases, but I think web browsing machines are zippy enough to do this for userdata. I have to see if it's been done already (it probably has), but I want to do some fiddling with it.

So yes, same old. Too much science to do, never enough time.

My Eyes!

2011-04-15T23:19:00+01:00

I've not had a lot of time to go at electronics in the past while (mainly a combination of work things and family things kicking my ass), but I got to some of it tonight -- The clever parts of the design and build are basically done, so what's left is producing the final piece.

Aside from the motion sensor parts, there's also the bright light, to mess with people's night vision. For this, I got 4 super-bright LEDs, and did monstrous things with solder to produce a 5x2 header, that was fully connected along the long side. This gave me the 4 LEDs connected in parallell. This is pretty bright with the 5v applied from the analog output of the Arduino:

The final result will be built on an Arduno Fio board, which only has a 3.3v signal (It's powered off a 3.7v LiPo battery), so I'll need to amplify that somehow. I'm thinking I might try to power the board and LEDs off a regular 9V instead, or have the LiPo powering the board and the 9V powering the buzzer and leds, gated on the signal line of the board with some transistors. I may be of course talking out my ass. I'm going to pick up some various transistors over the weekend and experimenting. My guess is the gate and emitter of a transistor need a common ground, which won't work with using separate power sources. We'll see.

Of course, deadlines, help. I'm supposed to be using this thing next saturday. So, full steam ahead.

Finding Day

2011-04-03T10:15:00+01:00

One year ago today, a very small ball of fur decided to mew outside our front door in particular. We let her in and looked after her for a bit, and eventually decided she could stay.

We're not sure exactly how old she is, we're guessing se was about 6-7 months when she came to us, but I guess her finding day is as good a day as any to be happy she's around.

Happy finding day, Ping.

I Seeeeee Youuuuu

2011-03-25T22:30:00+00:00

So, first steps. I picked up a few PIR Motion sensors. These are a specific kind, there are a few other kinds, but they should have datasheets. You might recognise the bulbous white shape of them from those motion-sensing light switches. They're about a fiver each and fun to play with.

The comments section on Sparkfun had a very good tutorial on basic use, but also explains the basic principles of its operation. It has 3 pins -- DC, Ground and Alarm.

Science Bit

The alarm pin is what's called an Open Collector, which means it's either connected to ground, or to 'nothing'. To better understand this, let's assume you don't know what a transistor is (I know I sure didn't a few weeks ago).

Speaking simplistically, a transistor is a component where you can vary one voltage based on a much smaller voltage. In this way you can amplify a voltage pretty easily.

Here's a diagram!

Usually transistors are tiny and embedded or are tiny and on a breadboard, but let's assume for a moment that it's a box with 3 wires coming out of it. They're called collector, emitter and base. Emitter is always connected to ground, and provides the reference voltage, or definition of 'zero'. Base is a small voltage in relation to ground, and the voltage applied to the base controls how much voltage is allowed flow between the emitter and the collector. The difference between the voltage on the base, and the voltage allowed through the collector is called the 'gain'.

However! The voltage on the collector doesn't come from nowhere. It has to be supplied, and the voltage 'allowed' to flow is defined by the voltage on the base. So, if you apply 3.3v to the base, and 9v to the collector (the voltage marked 'V~cc~' on the diagram there), you get a certain voltage that's actually 'allowed' through the transistor, and can be used in other applications by connecting them before the resistor (At 'V~out~' on the diagram).

The zig-zaggy thing there on the diagram is a resistor. We put it across the larger V~cc~ voltage to lower it a bit, since transistors are usually only rated to take a certain voltage. The function of V~cc~ and the resistor is to 'pull up' the voltage across the emitter and collector. This is why the resistor there is often called a 'pull-up resistor'.

Transistors are handy for switches when you want to control a large voltage using a small voltage. They're also useful if you want to take a tiny voltage you get from a signal somewhere and amplify it.

Anyway, why this is relevant is that an 'Open Collector' is the collector from a transistor that's available as a signal. However, instead of usual 'binary' signals that might produce a 'True' voltage or a 'False' lack of voltage, it's either connected to ground, or not. Whether or not it is connected to ground depends on the base voltage. My guess is that the PIR sensor circuitry provides the 'base' voltage and lets us use any V~cc~ voltage we want, in order to 'pull up' the output.

So, if you go look at the circuit diagram and build instructions in the tutorial mentioned above, it works just so. The one modification I made here is that as well as pulling up the voltage, instead of connecting back to the LED on the Arduino board, I connected up to a big old Piezo Buzzer I got in Maplin.

The finished build looked like this:

So, the end result is that when you wave your hand in front of the PIR, the buzzer makes an annoying noise. Success!

Next up would be making things a bit smaller -- There are smaller, cheaper Arduino boards you can get, and I'll do a build soon that's a little easier to get into a project box and lug around with you.