Ansible: 10 thing I immediately put to use in production

So in June 2024 I followed an excellent Warp9 masterclass on Ansible.
It was so inspring and effective, I could immediately put it to good use.

1. Create an inventory to all the servers for my clients.

First, I created a central inventory grouping all the servers I have access to for all my clients.

Using this inventory, I can quickly see what servers and VM's I am 'registered' with, what is required to access them, and what my role is in being able to access those servers.

2. Deploy and rotate public ssh keys to the server inventory.

Until now, managing ssh access to all those servers has been a pain.
I like to rotate my SSH keys frequently, but if I forgot to update one server, it might well be that I'd lose access to that server. Leading to an 'archive' of ssh keys, just in case, which defeats the purpose of rotating ssh keys.

Using an ansible playbook, a role, and the inventory I created earlier, I am now able to add SSH keys to all servers at once. The role I created also allows to revoke SSH public keys on those servers.

Besides managing my own access, I am now able to centrally manage the SSH keys of other developers too, and in my vars I can specify when they need to be revoked or rotated.

First, I created a central inventory grouping all the servers I have access to for all my clients.

Using this inventory, I can quickly see what servers and VM's I am 'registered' with, what is required to access them, and what my role is in being able to access those servers.

2. Deploy and rotate public ssh keys to the server inventory.

Until now, managing ssh access to all those servers has been a pain.
I like to rotate my SSH keys frequently, but if I forgot to update one server, it might well be that I'd lose access to that server. Leading to an 'archive' of ssh keys, just in case, which defeats the purpose of rotating ssh keys.

Using an ansible playbook, a role, and the inventory I created earlier, I am now able to add SSH keys to all servers at once. The role I created also allows to revoke SSH public keys on those servers.

Besides managing my own access, I am now able to centrally manage the SSH keys of other developers too, and in my vars I can specify when they need to be revoked or rotated.

3. Perform OS updates and patches on all my nodes at once

In my lab, keeping all the nodes running linux up to date has been a pain. With an ansible role, I can easily apt update all the nodes in my two homelab locations at once, with an option to reboot or not if required.

4. Compliance checks

Use ansible to centrally detect compliance on node configuration, such as wether `sudo` is allowed without a password, or wether certain directories have the right permissions.

5. Install, setup and manage ZFS

I use OpenZFS for reliable storage, with encryption. I also incrementally sync pools to a remote location for efficient backup and recovery strategies. I rotate my ssh keys frequently.

Using Ansible, I can centrally manage all those processes, and I can also use it to set up datasets with a consistent set of options (such as compression, deduplication, encryption) from a central node.

6. Run recurring jobs with Morpheus

Morpheus is a handy little tool I run in docker on a control node. It is basically an UI to run playbooks. It can fetch playbooks from a git repository, and provides output and task status.

It can also automate these tasks (such as backups, compliance checks) based on a cron schedule. Simple and nice!

7. Install docker runtime on nodes

Most of the times, my nodes (in the home lab as well as ec2 instances) just run docker or some container execution environment. And then I target these with a CD pipeline.

So I can easily set up a node for this purpose, combined with my 'ssh keys' role, without having to ssh into the node myself.

8. Create deployment targets for my CD pipeline

Some projects in my cd pipeline require some custom actions in the deploy phase when they target a node, such as executing migration scripts, or updating docker images in a docker swarm configuration file (or an .env file). With ansible, I can create the boilerplate for a new project on a node.

9. Install sets of SysOps and developer tools

Install sets of developer/sysops friendly tools on nodes that will be accesses frequently in the terminal for development, proof of concepts or manual monitoring. So along my favorites are installs of zsh, nano, htop, iotop, net-tools, git, jq, dnsutils and many more.

10. Set up telemetry on nodes

Another one that is frequently or even always required on the nodes is some libraries or services to make them observable. With ansible, it is efficient to select nodes and make them observable by, for example, AWS CloudWatch agent.

11. Install kubernetes control plane and register/deregister worker nodes

The home lab contains a k8s control plane and some worker nodes, managed by microk8s.
Using ansible I can install the required libraries on new worker nodes, register the nodes as workers at the control plane, and remove them.

12. Install and update TLS Certificates

Updating TLS (ssl-) certificates has been such a manual job... every node of different clients has them installed in different locations, and require different operations depending on the server software used. It is more easy to deploy a new certificate, and backup the old one, using ansible, and then run a test to see if the certificate is installed correctly. And also to check if the configuration is still valid before attempting any actions.

13. Manage NFS access

For my NFS share, add users/hosts and set the NFS options. Centrally and managed as code.

14. Connect with HashiCorp Vault

And then finally, still a todo, install HashiCorp vault in production mode to integrate with Ansible and provide passwords and other confidential information to ansible playbooks. It is much better than Ansible's vault and more or less industry standard, and it is supposed to work great.

Use ansible to centrally detect compliance on node configuration, such as wether `sudo` is allowed without a password, or wether certain directories have the right permissions.

I use OpenZFS for reliable storage, with encryption. I also incrementally sync pools to a remote location for efficient backup and recovery strategies. I rotate my ssh keys frequently.

Using Ansible, I can centrally manage all those processes, and I can also use it to set up datasets with a consistent set of options (such as compression, deduplication, encryption) from a central node.

SemaphoreUI is a handy little tool I run in docker on a control node. It is basically an UI to run playbooks. It can fetch playbooks from a git repository, and provides output and task status.

It can also automate these tasks (such as backups, compliance checks) based on a cron schedule. Simple and nice!

Most of the times, my nodes (in the home lab as well as ec2 instances) just run docker or some container execution environment. And then I target these with a CD pipeline.

So I can easily set up a node for this purpose, combined with my 'ssh keys' role, without having to ssh into the node myself.

Some projects in my cd pipeline require some custom actions in the deploy phase when they target a node, such as executing migration scripts, or updating docker images in a docker swarm configuration file (or an .env file). With ansible, I can create the boilerplate for a new project on a node.

Install sets of developer/sysops friendly tools on nodes that will be accesses frequently in the terminal for development, proof of concepts or manual monitoring. So along my favorites are installs of zsh, nano, htop, iotop, net-tools, git, jq, dnsutils and many more.

Another one that is frequently or even always required on the nodes is some libraries or services to make them observable. With ansible, it is efficient to select nodes and make them observable by, for example, AWS CloudWatch agent.

The home lab contains a k8s control plane and some worker nodes, managed by microk8s.
Using ansible I can install the required libraries on new worker nodes, register the nodes as workers at the control plane, and remove them.

Updating TLS (ssl-) certificates has been such a manual job... every node of different clients has them installed in different locations, and require different operations depending on the server software used. It is more easy to deploy a new certificate, and backup the old one, using ansible, and then run a test to see if the certificate is installed correctly. And also to check if the configuration is still valid before attempting any actions.

For my NFS share, add users/hosts and set the NFS options. Centrally and managed as code.

And then finally, still a todo, install HashiCorp Vault in production mode to integrate with Ansible and provide passwords and other confidential information to ansible playbooks. It is much better than Ansible's vault and more or less industry standard, and it is supposed to work great.

• Ansible
• Configuration Management
• DevOps
• Linux