WOMC: Nomad Variables as Secrets
Fri 03 November 2023
Standard disclaimer: Maybe don't do this in production? I hear Hashicorp also does a secret store :-)
Nomad 1.4.0 added support for 'variables', which are essentially key/value blobs protected by nomad's ACL system. As someone who had steadfastly avoided either running vault, or doing secrets properly, I knew an opportunity to do secrets improperly when I saw it, so here we are.
If you want a 'working' example, my nomad-homelab repo has the example for my SSL certs, where I populate nomad with letsencrypt SSL certs and give whatever jobs need them access.
Getting data into nomad Variables
Assuming you have access keys and auth set up, working with variables is pretty easy.
Each variable is referred to by a path, which doesn't mean much aside from tidiness, aside from in one way: If you create
a variable named nomad/jobs/myjob, tasks in the myjob job have access to the variable.
Each variable is actually a set of key/value pairs, so if I wanted to create a variable for myjob and put a 'authtoken' key in there, I'd do something like:
nomad var put nomad/jobs/myjob authtoken=mySeCrEtAuThToKeN
You can then happily get the variable with nomad var get and so on. See file_to_nomad_var.sh for an example of how to load file contents into a nomad variable key (although it looks like nomad has since grown functionality for using - as the value on the command line to accept key contents via stdin).
Access Controls
Variables have the same kind of access controls that other nomad entities have -- via an acl policy. The short version of how to get this done is that you should write (and then either keep or throw away) an acl policy file. This generally looks like the following:
# mypolicyfile.hcl
namespace "default" {
variables {
path "my/secret/stuff" {
capabilities = ["read", "list"]
}
}
}
You can then insert a policy for each job to give access to the my/secret/stuff variable with:
nomad acl policy apply -namespace default -job myjob myjob-secretstuff-policy - < mypolicyfile.hcl
The policy can be named anything, so known yourself out with the tidiness. If you'd like to see this being used in anger, see grant_job_access_to_cert.sh from my SSL example.
Accessing Variables from nomad Jobs
Variables are accessible from nomad jobs at invocation time -- that is, when the alloc is being built. So, the best way to access and use a variable is to write out its contents in a template within the alloc, so even if nomad itself goes away, the container still has everything it needs.
In the .hcl file for your job, you can dump a particular variable you have access to to a file in the secrets directory by using template directives within a task stanza:
template {
data = "{{ with nomadVar \"my/secret/stuff\" }}{{ .authkey }}{{ end }}"
destination = "secrets/authkey.txt"
change_mode = "signal"
change_signal = "SIGHUP"
perms = 700
}
Some things to note here:
my/secret/stuffis the variable name,authkeyis the key within that variable.- You can place the output file outside of
secrets/if you like, I guess. - You can of course have multiple
template {}stanzas with differet variables and outputs/behaviour. change_modeandchange_signaldetermine what happens if the variable gets updated. In this case, the task gets HUP'd if someone updates the nomad variable. This can be handy for stuff like SSL certs.
Why not production?
Any number of reasons I can think of:
- People with admin access to nomad have access to all variables. You probably don't want that in production.
- I'm pretty sure nomad shunts all this content around via raft and I'm not sure how secure the data is at rest (or in transit). I'm gonna guess less so than vault :-)
- There's a proper way to do this that isn't chintzy as frig and that the people who come after you are more likely to understand.
Other Thoughts
Nomad Variables are kind of like K8S ConfigMap, so I could probably abuse them to store actual configuration data pretty trivially -- for jobs that just have a single config file and everything else can get blown away with the container, this is probably fine. Very few of the apps I run at home are like that, though.
Category: Tech Tagged: log #worksonmycluster tech