Upgraded Adventure

Patroni for postgres clusters on prem

Sun, 19 Apr 2026 00:00:00 +0000

A common pattern in deploying applications is ensuring they have a reliable persistence layer. This often takes the form of a database. If an application, or rather the service owner which has to deliver that application, can assume that it can connect to a database via configuration parameters, this reduces the scope of the application deployment.

The problem then gets shifted to the platform owner to ensure that there is indeed a database server for the application ready when it’s deployed. In a scenario where there are several applications deployed into the platform, the platform owner now has a set of design choices to make:

Do we have a single database server serving all applications in a given scope, or does each application get its own databse server?
Do we deploy database servers in the same scope as the application, or outside of the application scope?
Does database management lie with the service owner or with the platform owner?

The correct choices will probably depend from platform to platform. From my point of view, the important thing is to have options available to be able to even pose the questions in the first place.

Managed databases on prem

The platforms I own now are all custom-built, on-premise. There are no platform functions other than those I build into them, which reveals many of the ways a platform engineer needs to make design choices. Coming from cloud-native environments where databases are available as a service, I had to go back to the drawing board.

First, I had to recognise that the only storage I have available is the block storage attached to virtual machines which constitute the Nomad agents. In some cases, these attached disks could not even be requested at agent launch time, but had to be added when the VM itself was provisioned - either you had an attached disk or you didn’t and you couldn’t be sure across the fleet whether was the case. However, one has to start with some basic assumptions. I assumed that I could request certain agents which would have attached block storage, and that block storage would be reliable: it would be snapshotted, backed up, etc.

I wanted a way to bring up a reliable database service, as a Nomad job. It should be able to have servers spread across several nodes to protect against single node failures, and provide the ability to do rolling updates across nodes, taking them out of the cluster temporarily, then re-joining.

All of this is possible of course, and probably well-trodden paths for a database administrator, but I am not the DBA! I wanted something that DBAs have refined over the course of their collective years of experience into something which would allow me to perform their magic without necessarily understanding the minutae.

Some research lead me to Patroni.

Patroni fits into my platform

Since these are intended as lab notes, I will skip the various alternatives which were discarded. Patroni is a management layer for Postgres. These are my words; they call it a “template for HA Postgres”. It’s been around for more than 10 years at the time of writing, and is used by a few big names e.g. EnterpriseDB – let’s just say that’s a good signal in my book! However Patroni had one key design consideration which fits into my idea of a platform - distributed configuration. It uses a DCS (distributed configuration store) to managed postgres deployments externally. The DCS is the source of truth for the database. This in turn is kept in a kv store which is replicated across a Raft cluster. This KV store is used not only to write configuration data, but also agree on the leader or primary, since it is the one with a lock on the configuration. If the primary fails, it loses the lock on the configuration, and an election is triggered, resulting in one of the replicas being promoted to the new primary (leader).

It should be noted that what sealed the deal for me was the built-in ability to use Consul as a DCS.

A little bit of engineering

Patroni is a python library; it is the application which manages Postgres. I consider it to be a kind of postgres harness – but you still need the postgres binary.

In order to deploy it in my platforms, it would run as a Nomad job, where the task driver used would inevitably be one of the container runtimes, Docker or Podman.

Images

I first needed to have containers that actually had patroni in them. I considered using pre-start tasks in the default postgres containers to install patroni, but opted instead of building my own containers with patroni pre-installed. I wrote a Packer template to do this, with essentially only two changes:

Add the dependencies to the image, according to the Patroni installation documentation
Change the entrypoint of the container from its defalt to /patroni/bin/patroni.

The Packer template has two builders for the most common architectures in my platforms - AMD64 and ARM64.

Deployment

We have all of the ingredients necessary to deploy as many clusters as we like:

Consul cluster for DCS
Patroni-ready postgres images
Nomad orchestrator

The final step would be to write the Nomad job and deploy. I took the case where we would need a default database server as a fundamental dependency for platform services themslves. This would serve several databases to platform service rather than platform customers, which would probably merit having their own deployments within their namespace, scope, etc.

The first nice thing is that we can simply declare a capacity using the Nomad group count paramter; this opens up possibilities of scaling later.

The group also has its own network, where we declare the Patroni API port as well as the postgres port of course:

network {
      # Expose the patroni API. This listens on 8008 on the container
      port "patroni_api" {
        to = 8008
      }
      # A dynamic port is reserved for postgres
      port "postgres" {}

    }

Each instance of the task in the group should get its own service registered in Consul. I made the mistake in initial attempts to declare the service at the level of the group rather than the task, which ended up giving routing errors. This allowed each instance of Patroni in the group to have a unique service ID, and was necessary to get the DCS leader election to work.

We declare the patroni service with its healthcheck and the postgres service in the group.task.service block:

service {
  name     = "patroni-api-${var.tenant}"
  port     = "patroni_api"
  provider = "consul"

  check {
    name     = "alive"
    type     = "tcp"
    interval = "10s"
    timeout  = "2s"
  }
  check {
    name     = "healthy"
    type     = "http"
    interval = "10s"
    timeout  = "2s"
    path     = "/healthy"
  }
}

service {
  name     = "postgres-${var.tenant}"
  port     = "postgres"
  provider = "consul"

  check {
    name     = "alive"
    type     = "tcp"
    interval = "10s"
    timeout  = "2s"
  }
}

Here ${var.tenant} is a Nomad variable to distinguish who this postgres cluster instance belongs to.

The next tricky part was writing the patroni.yml configuration template. Each patroni instance needed to be unique within a given scope in order to perform leadership elections. For example, in a 3 node cluster (1 primary, 2 replicas), they all need a patroni.yml file which differs by the name of the instance, but has the same namespace and scope, as well as allows each instance to communicate with each other.

I decided to configure them with the scope as the name of the tenant, a namespace selected by hand (operations) and using the Nomad allocation runtime variables as identifiers:

scope: ${var.tenant}
namespace: operations
name: {{ env "NOMAD_ALLOC_NAME" }}-{{ env "NOMAD_ALLOC_ID" }}

That would end up being templated as

scope: default
namespace: operations
name: patroni-postgres.patroni[0]-8ae3416a-2a73-a3e0-ea10-9d50596a442d

for the first ([0]) instance of the group.

We configure the Patroni REST API to be served at the task’s specific URL:

restapi:
  connect_address: {{ env "NOMAD_ADDR_patroni_api" }}
  listen: {{ env "NOMAD_ADDR_patroni_api" }}

Then we come to the tricky part: the postgres configuration. This was tricky because I know almost nothing about configuring postgres servers. There were however only two configurations I needed to actually think about. The first was the users and their authentication when bootstrapping and managing the clusters. Patroni defines 3 users which it uses to manage postgres behind the scenes: superuser, rewind and replication. These should all have different credentials.

I had stored them already in the platform Vault as secure KV pairs, so that was used in the template with a Vault lookup: . Again the Nomad runtime variables gave me the port which postgres was served on in the deployment:

postgresql:
  authentication:
{{- with secret "hashiatho.me-v2/data_plane" }}
    replication:
      password: "{{ .Data.data.postgres_replication_password }}"
      username: {{ .Data.data.postgres_replication_user }}
      sslmode: allow
    rewind:
      password: "{{ .Data.data.postgres_rewind_password}}"
      username: {{ .Data.data.postgres_rewind_user }}
      sslmode: allow
    superuser:
      password: "{{ .Data.data.postgres_root_password }}"
      username: "{{ .Data.data.postgres_root_user }}"
      sslmode: allow
  bin_dir: ''
  connect_address: {{ env "NOMAD_ADDR_postgres" }}
  data_dir: /var/lib/postgresql/17/data
  listen: {{ env "NOMAD_ADDR_postgres" }}
  parameters:
    password_encryption: md5
  pg_hba:
    - host all all 0.0.0.0/0 md5
    - host replication {{ .Data.data.postgres_replication_user }} 127.0.0.1/32 md5
    - host replication {{ .Data.data.postgres_replication_user }} 127.0.0.1/32 md5
    - host replication {{ .Data.data.postgres_replication_user }} 192.168.1.0/24 md5
{{- end }}

To be honest the pg_hba key was the result of trial and error, as well as a bit of good old searching on errors, but essentially, I needed to add the local network CIDRs which the replication user would communicate on.

Operations

So, now we can deploy postgres clusters with reliable replication, using Consul to manage leadership election. We can scale them with a single parameter (count) and manage all of the configuration centrally from the Consul KV.

There are many interesting aspects to look into next, especially from an operations point of view:

Can I update recovery and replication configuration just by updating entries in Consul?
Can I perform blue-green deployments and ensure that all data is replicated during the deployment? Will this obviate the need for persistent disks in certain cases?
How can I use the Vault database secrets provider to provision one-time credentials for database users?

I have also already had a few failures of database nodes in prod on one of our platforms, where essentially replicas were not rebuilt after a downtime due to divergence in history. Whatever that means - I have no idea really.

There is also the need to actually back up databases and define point-in-time snapshots so that we can reproduce deployments elsewhere, e.g. during CI/CD or testing procedures.

I now feel like I understand Patroni well enough that I can use it to solve these needs, perhaps adding a few other tools like Barman. The data plane of the platform is taking fine shape!

Managing security and risk in a complex system

Mon, 14 Jul 2025 00:00:00 +0000

Problem Statement
Setting the stage
Seeing Risk
Owning Risk
Where have I seen this before
- Providing value to customers closes the feedback loop
Putting Dependency Track to use
- Tracking dependencies, measuring risk, ensuring compliance
- Services in the Security Plane of the platform
Conclusion
Footnotes and References

Problem Statement

In this post, we’ll tackle a common problem faced by those responsible for shipping software for a client: ensuring that software supply chain is kept secure and compliant with requirements of the contract.

Setting the stage

Let’s say you’re building a set of software services for a client, which takes the form of a product composed of multiple services, managed on behalf of the client. The goal of this set of services is to provide access to a defined set of cloud services, data sets and other APIs, all of which are also part of your product. Now, let’s say that the client intends to release this product for use by third parties and in order to do that, they need to state with certainty that the product is secure and compliant with all relevant regulations and standards.

It is safe to assume that each of these services is implemented directly using one or another programming language, which inevitably depend on a set of libraries or frameworks. These direct dependencies may themselves also depend on other libraries, packages or modules. In reality then, the system is composed not just of the specific implementation of services, but also all of this graph of dependencies.

Given this understanding of the composition of the system, of known, assumed and unknown dependencies, let us consider what it would take to keep the system safe. This would, at a minimum, require the ability to observe the full composition of the system, as well as the presence of known vulnerabilities in the dependencies.

Seeing Risk

Let’s assume we can actually see all of the dependencies: how would we achieve that? It would involve some tool which inspected the system and extracted the graph of all of the dependencies. This is only feasible when the system declares what it depends on. This varies from case to case:

In the case of an application, this will come in the form of a language-specific manifest file, such as a requirements.txt for Python, a go.mod for Go, or a package.json for Node.js.
In the case of a service, this will come in the form of a service-specific file, such as a Dockerfile a Docker Compose file for Docker, a manifest file or Helm chart for Kubernetes, or the providers section of a Terraform configuration.
In the case of a virtual or bare-metal machine, this will come from interrogating the filesystem for installed packages or modules.

Scanning these various endpoints, whether they be package managers, deployment manifests, or actual deployed environments, provides us with a list of discovered versions and their versions. This is the first step in being able to see the risk of the system: the simple ability to generate an inventory. The key capability however is being able to take appropriate action based on whether a given component is known to contain a given vulnerability. Vulnerabilities do not inherently carry risk, however, since they must be exploitable in order to pose a threat, and whether they are exploitable or not depends on several factors, including the configuration and deployment of that component in the actual system.

Nonetheless, the ability to map CVEs¹ to components allows us to locate points of investigation, and perhaps even keep an audit trail of the system’s posture.

Owning Risk

The system is not a flat, amorphous blob of components. There is a structure to the system which comes from the way in which humans have decided to organise their work around building, maintaining and deploying it. In some cases, there will be a single system owner, perhaps an enterprise architect, who has the final responsibility for the system’s security posture. In others, each service in the system will have self-contained ownership, and agree to a common set of policies in order to guarantee a consistent security posture in the system even though they take action on known vulnerabilities autonomously. The point is that it in order to ensure that the system as a whole is secure, it is necessary to know who should take action on which component. Ownership is thus the ability to map vulnerabilities to components, via services to actual people who take responsibility for action.

Where have I seen this before

Way back in before 2020 some time, I forget exactly when, I was working in the SANREN group at the Council for Scientific and Industrial Research (CSIR) in Pretoria, South Africa. SANREN stands for “South African National Research and Education Network”, and aside from provisioning high-capacity fibre connections throughout South African research institutions and universities, it also had the mandate to deliver services to these customers. One of these services was the South African National Grid, which I was responsible for, which coordinated the distributed computing infrastructure offered by the universities and laboratories connected to the SANREN network. However, SANREN was very active in providing end-user services such as the identity federation, on-demand file transfer between individuals, etc. Most of these were simply instantiations of a product commonly-used by peers in the research networking community. These surely provided some value to the communities served by the infrastructure, but they were too generic to be considered invaluable. The real wins were where we could give actionable advice to customers.

Providing value to customers closes the feedback loop

I will permit myself a brief digression here to consider how infrastructures are perceived and how they become sustainable.

The question can be posed as such:

How can invisible infrastructures keep proving their value to customers?

An infrastructure works well when it becomes invisible; nobody sits and thinks about how the network provides value to them, they just use the things which the network enables. This invisibility is something which often breaks the feedback loop between customer and provider, since when things are working well, the value is invisible, and only when the infrastructure fails to do its job does it become visible. By its very nature, it is only perceived negatively.

A second consideration is whether there is a direct feedback loop between the end users of the services offered by the infrastructure and those who actually pay for it (the customer). In the absence of such a loop, the sustainability of the infrastructure, depending on its continued viability, requires arduous measurement and reporting of the customer satisfaction and usage metrics, which inevitably drives up the cost and reduces the efficiency. Instead, what if the infrastructure provider itself provided services directly to the customer? In SANREN’s case, these are typically the Information Technology departments of the institutes served by the network². These IT organisations are comprised of IT professionals, with their budget directly allocated to IT services for the institute. Part of their mission is usually to protect the network and IT resources associated with them, either within their institute or across the network in the case of research and scientific collaborations.

Assessing and owning risks was one of those activities which simply could not be performed by the respective IT departments, given their limited resources and vast set of users. If SANREN could provide them with a service to assess security risks and provide specific actionable recommendations assigned to specific people this would have gone a long way towards helping them perform their overall mission. This would have been a service provided directly to the customer, providing a strong feedback loop between the invisible infrastructure and those who ultimately depend on it.

It was my buddy Schalk Peach way back around 2017-2018 who helped me understand how powerful this feedback loop between vulnerability, component and person would be. He was a postgrad student at the CSIR SANREN team during my last few years there, and we had almost daily conversations about the workflows and entity models that would be involved in providing a service like this. At the time, we were focussed on perimeter security and directing customer network administrators to specific mitigation actions based on the unintelligible vulnerability reports from Nessus³.

Every now again I think about him and how far ahead of the curve he was.

Putting Dependency Track to use

Now I’m part of the team delivering a managed service to researchers on behalf of the European Commission, and we are facing the same challenges as a decade ago. A customer like the EC does not mess around when it comes to compliance and quality, and the system we were building is delivered by a great number of independent contractors, so I knew that assuring all interested parties that the system was secure and up to date would require specific tooling. Indeed at the outset my mind flew right back to Schalk and his magical vulnerability-remedy system… if only we had something like that to observe and track our dependencies!

Well, it turns out that in the decade or so since I last bothered myself with platform security, the OWASP foundation has brought a great many projects to maturity, including one called “Dependency Track”.

Tracking dependencies, measuring risk, ensuring compliance

Dependency Track’s strength comes from its ability to keep a detailed database of Software Bill of Materials (SBOM), as well as a local database of all known vulnerabilities. Vulnerabilities are almost always published with an associated component, so all we needed to do was declare the components of our system, and their associated SBOMs. Dependency Track also has an overlay system which allows one to declare that certain projects are owned by certain teams, recreating that all-important feedback loop we mentioned repeatedly before.

This functionality allowed security teams at a glance to know which vulnerabilities were present where, and indeed what the relevant service owner was doing about.

We ended up bolting on a custom notification handler to Dependency Track to allow us to accurately communicate the discovery of vulnerabilities both with the client and the security team.

Services in the Security Plane of the platform

In the larger context of providing a platform providing support functions for the delivery of a wide set of services, Dependency Track falls cleanly into the “Security Plane”. The final goal is not measurement and visibility, but actually taking remedial action, we should envision the connection of task and issue tracking services with the dependency tracker. However, dependencies of software components are not the only source of risk and security vulnerability in the system. Misconfiguration and insufficient resources can also contribute to security risks, or risks of denial of service. These are detected with other tools, such as penetration and load testing tools. These findings too are associated with specific components, owned by specific people in the organisation.

This could be done with DefectDojo, another tool in the OWASP stable.

These of course only address the system design and architecture. There are of course also the operational aspects, the day-to-day events happening in the system, and the inevitable attacks it will face if connected to the internet. To this end a security information and event management (SIEM) system is required, where alerts could also be passed to the overall view of the security posture, potentially correlated with known vulnerabilities and outdated components.

Conclusion

We have gone to great lengths to find a set of tools which would allow teams delivering software to declare their composition and dependencies to a central service, and in turn obtain visibility and actionable advice on how to respond to known vulnerabilities. This could be deployed in the context of platform engineering within the “security plane”, offering a platform-level internal service to internal customers wishing to deploy in the platform. All of the tools discussed here are open-source and deployable on-prem. Indeed I have deployed them in a Nomad cluster built for the purposes of providing the platform-level functions to our client, as well as a similar deployment in EGI internally. We gain insight into component-level risk, the ability to map that risk to actual people, and the ability to provide them with specific concrete advice on how to mitigate the risk. With the addition of a SIEM, we would be able to have real-time observability into security events, and thus ensure not only the architectural security of the system, but also its operational security.

Total visibility and collaboration between platform engineers, SRE and service owners is the goal, and we are within its reach.

Footnotes and References

Common Vulnerabilities and Exposures (CVE) is a dictionary of publicly known information security vulnerabilities and exposures. CVE is maintained by the MITRE Corporation. ↩
The Association of South African University Directors of IT (ASAUDIT) was in my understanding the interface between the infrastructure provider and the institutes – it has apparently transformed into Higher Education IT South Africa ↩
I think it was Nessus, but it was a while ago. I do remember that it was open source at the time, whatever the tool was. ↩

A terraform module for Vault on DigitalOcean

Mon, 16 Jun 2025 00:00:00 +0000

Vault in a Managed Platform
- Solution Context
One Goal – clean architectural interfaces
Design Decisions
Using the module.
Conclusion

Vault in a Managed Platform

This is a short writeup of the Terraform module I wrote for a Vault deployment. I had to solve a few technical challenges and take a few decisions during the development process, so I wanted to write them down for posterity.

TL;DR You can find the module in the terraform registry

Solution Context

There are plenty of ways to deploy Vault, but this one has a few considerations which you may find yourself identifying with. The overall context is that this service (Vault) is a component of an internal developer platform which supports the delivery of workloads to downstream users. I am writing as the platform engineer who is responsible for delivering the platform as a whole.

I followed the Hashicorp Well-Architected Framework for Vault, applying this to the Digital Ocean platform.

Another important design decision here was that I consider Vault to the be the first component of a platform, with zero dependencies. This means that, at the time of deployment, we have no knowledge of any other services which may enhance the operations, such as service discovery, or observability. These will indeed come later, as part of the platform’s iterative deployment, but it is important to explicitly exclude them from our scope since we want to make a clean Terraform module.

There are a few questionable decisions which I have made during the design of this module, specifically the choice of Tailscale as an overlay network, and the use of self-signed certificates for TLS. I will justify these and perhaps discuss how changes to the module may make them optional.

One Goal – clean architectural interfaces

I wanted to see if I could reduce the goal of this module to a single concern, i.e. it should only deploy Vault. Nothing else should be contained within it – if anything else is needed, it should be expressed as input variables. Likewise, the module should be amenable to anything that depends on it upstream – it should produce outputs which are consumable by other modules which need to build on it.

Finally, the module should deploy Vault with zero opinions about what Vault should do. There should be no internal configuration for secret stores, authentication mechanisms, or other configuration. This will vary greatly from team to team and use case to use case, so should be left up to the actual users of the service to define. The overall picture then is of a production-ready, workload and userbase agnostic Vault deployment which can be customised by whoever requests it. This will make the module better for use in a self-service environment, such as the internal developer platform we are considering in our context.

Design Decisions

Design decisions are the constraints and opinions imposed on the module that express what it is designed to do. Our design decisions are informed by the the single goal we have set above, and constrain the user experience of the service such that it becomes consistent with the rest of the services in the platform.

These decisions are meant to reduce the scope of the module in order to make it maintainable, and help operators decide when it is appropriate to use it.

The design decisions are listed below:

Design decisions will be documented using the ADR format.
The resource provider will be DigitalOcean.
Vault will be served by configured processes on virtual machines.
Vault will be configured using userdata at provision time.
Vault data will be persisted to attached storage.
The virtual machines will be configured with private networking in a virtual private network.
Networking over the public interface will be disabled via firewall rules which permit access only to specific services, and not the Vault service.
An overlay network will be provisioned and Vault will communicate only via this overlay network.
mTLS will be configured between Vault instances.
Certificates will be generated with a private CA, as part of the Terraform state.
The final state will be a sealed Vault cluster with no configured secret stores, authentication mechanisms or policies.

Discussion

Why do we take these specific decisions? Are they objectively better than alternatives? The answer is no, they are not objectively better, indeed there can be no objectivity here, because we are building something with a specific intent in mind. This is a module for use by teams who depend on a platform which is built by a small team of platform engineers. There are constraints all over this scenario, so what we design must take those constraints into account, and respond to the subjectivity of the situation. We could have decided to deploy Vault via a helm chart on a DOKS cluster, or decided to expose it only via a load balancer. We could have decided to add a backup operator which backed up raft state to an object store, or we could have decided to use a managed database as Vault backend. All of these would have introduced additional complexity and dependencies, while keeping the actual service (Vault) constant.

Design dependencies

Although we declare zero dependency on other services, we do indeed need some bedrock on which to deploy our Vault. These are our design dependencies. If you find yourself disagreeing with the need for these, then this module is not for you. I find it important however to explicitly state these dependencies so that these divergences in expectations can be immediately identified.

Rather than designing a module that is generic and satisfies any deployment scenario, I wanted to develop separate modules for specific scenarios. It is perhaps true that there will be some redundancy and repetition across these modules, but this is done consciously in order to maintain clean interfaces.

The dependencies of this module are:

A virtual private network (VPC) which we can use to isolate our Vault deployment from other platform services, or managed workloads.
A Digital Ocean project, so that we can add resources to it and manage them.
A tailscale organisation which we join droplets to.
A pre-existing Vault cluster which contains the secrets necessary to initialise the providers of the dependencies above.

The first two are useful for organising and billing resources and are provided by an upstream module.

The third is a design choice to ensure that we have secure overlay to provide access to the instances of the cluster, and we can expose the service only via that interface.

Control plane vs security plane

The last dependency one might seem like a unsatisfiable requirement – how can I deploy Vault if I need Vault to deploy Vault!? The impression of redundancy may change if one remembers that we are deploying an instance of Vault for a specific team or community, so it is one of potentially many instances, provided as a managed service to that team or community. It is therefore part of a security plane, not part of the control plane.

Another way to put it might be that the users of the provisioned Vault instance are the members of the team or community which are served by the platform, whilst the users of the Vault cluster we depend on are the members of the platform engineering team which build the platform itself.

Using the module.

Let’s close out this long navel gaze by giving a demonstration of how to use the module. The module contains a few examples of how to use it, we will use the “simple” example here, which makes no assumptions about the pre-existing resources, and deploys into a commonly-used region (AMS3).

We first declare the provider configurations: We will need a DigitalOcean and Tailscale API tokens in order to create the resources, and those tokens are stored in Vault. We look them up with a few data calls, then use them to configure the providers:

# Vault configured with environment variables.
# The user only needs to know where the central operations Vault instance is, and have a token with the relevant permissions issued to them.

provider "vault" {}

# We have declared a variable to hold the value of the KV mount path.
# The path is also given to us by the operations team, who are responsible for
# separation of concerns.
data "vault_kv_secret_v2" "do" {
  mount = var.do_kv_mount_path
  name  = "tokens"
}

# Similarly, the tailscale tokens are stored in the same mount,
# using a different key.
data "vault_kv_secret_v2" "tailscale" {
  mount = var.do_kv_mount_path
  name  = "tailscale"
}

# Now we can configure the DigitalOcean provider
provider "digitalocean" {
  token = data.vault_kv_secret_v2.do.data["terraform"]
}

# And the Tailscale provider
provider "tailscale" {
  api_key = data.vault_kv_secret_v2.tailscale.data.api_key
}

Vault needs to be deployed into a VPC, but that is not of concern to the Vault module, so it needs to created with its own module. In this case, it’s being kept in the same state as the Vault module itself, so it will be deleted when Vault itself is deleted. This may be the case in a short-lived, entirely standalone project for example. In other cases, an existing VPC may be desired, provisioned by a different team.

Having created the VPC and project resources, we can then deploy Vault into it:

# First, we make the VPC and project, using a handy existing module in our registry
module "vpc" {
  source          = "brucellino/vpc/digitalocean"
  version         = "2.0.0"
  # The project and VPC have been declared above as variables,
  # but we omit the details here for brevity.
  project         = var.project
  vpc_name        = var.vpc_name
  vpc_region      = "ams3"
  vpc_description = "Vault VPC"
}

# Now we make our Vault service
module "vault_cluster" {
  create_instances         = true
  instances                = 3
  depends_on               = [module.vpc]
  source                   = "../../"
  # These use the same variable as the VPC module,
  # we could have used the outputs from the module instead.
  vpc_name                 = var.vpc_name
  project_name             = var.project.name
  # We have declared a local variable previously,
  # which contains the IP we want to allow SSH access from.
  ssh_inbound_source_cidrs = [local.addr]
  region_from_data         = false
  region                   = "ams3"
  # This is the Digital Ocean token which allows Vault to lookup
  # other instances in the platform to peer with.
  auto_join_token          = data.vault_kv_secret_v2.do.data["vault_auto_join"]
}

Conclusion

Hey presto, you now have a self-service module for creating Vault clusters on Digital Ocean. Using this module requires knowledge of the location of a central Vault instance where platform secrets are kept, and a token from that Vault that allows access only to the secrets you need to provision resources in Digital Ocean and Tailscale.

Once you’ve created the cluster, you have have full control over it, and can subsequently terraform it with the another module, fit for your specific needs. Maybe you want to connect it to your Identity Provider, maybe you want to use it to issue Nomad tokens for jobs, maybe you want to connect your managed databases in your application workloads to it… the choice is yours.

And that’s the whole point at the end of the day!

Self service can be hard to do right. With this simple module, we have taken explicit design decisions, paring down the scope of what one can do with these powerful providers, and putting just enough control in the hands of the user that they can create their resources themselves in a predictable and consistent manner.

A Platform for Improving Quality of Infrastructure Engineering in Open Science (I)

Sun, 01 Jun 2025 00:00:00 +0000

This is the writeup of a presentation given to EGI 2025 Conference in 2025.

EGI has been managing federated infrastructure and delivering services for more than 20 years now. During this time, it has participated in countless infrastructure projects and brought even more services to user communities across Europe and beyond. One key to the long-term stability and maturity of these services is the FitSM standard, which provides requirements and guidelines for managing services in federated environments.

However, EGI has inherited infrastructure and resources from federation partners over the years and continued maintaining this legacy platform without explicitly evolving the methods and practices involved in actually building them.

Whilst FitSM is appropriate and useful for governing and managing services at a mature stage of development, it doesn’t provide the opinions, guidelines, or patterns for quickly ramping up and iterating the development of infrastructure components themselves—or services deployed within them.

This has been addressed in environments where fast iteration towards high-quality services creates competitive advantage. One need only refer to DevOps practices for a good frame of reference. A shorthand introduction to this article might be:

The key for making change fast is DevOps, whilst the key for managing mature services is FitSM

Lack of constraints leads to lack of function

The collaboration, observation, and fast feedback patterns of DevOps allow us to reach a position where system governance becomes feasible more quickly. At that point, we can start extracting value from the FitSM standard with its clear requirements.

In our context, software systems are built and delivered by federations—independent teams under separate domains. There are so many tools available in every phase of the software development lifecycle that independent teams almost certainly evolve different toolkits. This creates a situation where the freedom to select tools for local DevOps optimisation ends up frustrating efforts to globally optimise FitSM governance.

Since each team delivers with different toolkits, implementing machine-readable procedures becomes prohibitively complex. The only tool truly shared by all teams is human language. FitSM requirements are then implemented by adopting this lowest common denominator—human-readable processes, procedures, and policies¹.

Furthermore, procedures tend to be intentionally vague, given the opacity across domain boundaries. Teams in one area lack visibility into another’s workings and must describe procedures in terms of generic interfaces rather than specific actions. Whilst this has benefits—abstract interfaces allow tooling to change without altering procedures—there are better ways to achieve this whilst providing superior system understanding.

Patterns impose constraints

This complexity problem has been encountered repeatedly in enterprises large and small. It’s a consistent hurdle that becomes inevitable once a certain scale is reached. The boundary between distinct domains is analogous to the traditional boundary between development and operations concerns.

Just as we realised that better dev-ops collaboration could lead to superior overall outcomes, there were plenty of ways to do it “wrong”. The frustration from lacklustre DevOps adoption returns led some in the industry to consider what patterns or antipatterns were present. This approach led to “Team Topologies”²—a guide to identifying which interaction model suits a given environment.

Emerging from this research was the concept of a “platform”—an abstract set of functions which, when organised together, allow software delivery teams to perform tasks with less cognitive load, fewer dependencies, and faster feedback. A few iterations later, we arrive at a semi-codification of this pattern: platform engineering.

Platform engineering is the practice of building platforms³ designed for delivery. Another way to think of this: a platform is “the product that helps us deliver services to our customers”. This links our fast-flow, fast-feedback, tool-integration DevOps world with our quality-oriented, requirements-first, process-aligned service management FitSM world.

The platform engineering pattern helps various domains adopt consistent patterns, even without overall consensus on specific tooling. It organises tools according to their function and position in the software delivery lifecycle, identifying certain “planes”:

Developer Control Plane: Contains all necessary tooling for developers to build applications. This includes version control for applications and the platform infrastructure’s source code. In mature cases, it includes an internal service catalogue and API gateway that developers use to identify reusable components and avoid duplication.
Integration and Delivery Plane: Work from the Developer Control Plane flows here. This includes continuous integration (CI) pipelines, artifactories, and continuous deployment (CD) pipelines. A crucial component is the platform orchestrator, which deploys new artefacts into deployment environments based on platform definitions, configuration constraints, and policies.
Resource Plane: Contains the actual operating environment for services offered to customers. This is the traditional “Ops” part of DevOps, bound to reliability, security, and cost objectives. It includes all environments: testing, staging, and production.
Observability Plane: Monitors all platform components and provides the bedrock for feedback. It collects metrics from platform components and services deployed in the Resource Plane, alerting based on defined service level objectives. It also aggregates logs and traces across the platform.
Security Plane: Responsible for declaring, enforcing, and supporting security and safety across the platform and its workloads.

We’re still hiding tooling complexities behind the “plane” abstraction, but at least we now have a common model across different domains.

Overcoming boundaries

The platform engineering pattern tames DevOps complexity, but we’re still left with organisational boundaries inherent in federations. Have we improved the situation by adopting platforms, or created more work since every team must now build one?

Organisational boundaries won’t simply disappear. Even though our platform describes interaction patterns, we still need to implement those interactions. Code commits must flow through CI, workload definitions need auditing, securing, and delivery to various resource planes across boundaries (i.e., heterogeneous toolkits).

The question becomes:

How can we build a platform whilst respecting organisational boundaries?

Given that platform components, or entire planes, may be provided by different organisations:

How can I consume platform component services or expose my component’s services across opaque organisational boundaries?

This may be analogous to the “Bezos API Mandate”⁴. To paraphrase⁵:

All teams will expose their data and functionality through service interfaces
Teams must communicate through these interfaces
No other form of interprocess communication is allowed
Technology choice doesn’t matter
All service interfaces must be designed to be externalisable

Wouldn’t it be brilliant if all platform functionality needed to deliver services to users were exposed as APIs?

Workflows in platforms vs tools

Most toolkit tools do expose APIs for external consumption. The problem is that we’ve deliberately avoided referencing explicit tooling in our platform pattern. When platforms are eventually created using actual tools, and the only way to use those tools in delivery workflows is via their specific APIs, we’re back to the complexity we tried to address.

We’d have to integrate with numerous APIs and account for countless specific configurations to use the platform. This clearly isn’t an improvement!

Event-driven architectures

Tightly-integrated end-to-end workflows for change management, release, deployment, and monitoring—i.e., all service management system requirements—are thus doomed. They’re either too narrow in scope (single org or team), too costly to implement (too many different tools and APIs), or too expensive to maintain (tools change, requiring ongoing API integration work).

But what if we didn’t need end-to-end workflows? An alternative would be taking an event-driven system view. This allows writing procedures as combinations of specific events, actors, and actions:

--- title: Event-Driven Architecture --- flowchart LR TriggerEvent(Trigger Event) --> Actor Actor --> Action[[Action]] Action --> ResultEvent(Result Event)

In this scenario, actors subscribe to events, and know what to do when a given event occurs – they take the relevant action. These actions can be codified accordingle, whilst the SMS policy defines actors. What remains is deciding the triggering event and determining how actors are notified.

This is where the CDEvents specification comes in. As we’ll see in the next post, common event specification in software systems will allow us to build tool-agnostic platforms for delivering software systems to customers across organisational boundaries.

This will be discussed in Part II, where we dive into the CDEvents specification and show how it can solve our complexity and integration problems.

Footnotes and References

Perhaps one day large language models (LLMs) will make human language a valid interface for implementing machine-readable workflows, but we’re most definitely not there yet. ↩
Skelton, M., & Pais, M. (2019). Team topologies: organizing business and technology teams for fast flow. It Revolution. ↩
The term “platform” appears repeatedly in this article. To be clear, we’re referring to the same platform that Evan Bottcher discusses in What I talk about when I talk about platforms ↩
The only attribution I could find was Steve Yegge’s second-hand, accidental post. It has been almost lost to time thanks to Google+’s shutdown (irony of ironies). ↩
This isn’t faithfully reproduced—I’ve edited and omitted several parts. See the original for greater context. ↩

Attracting and retaining new users

Wed, 01 May 2024 00:00:00 +0000

This is a discussion article for work done in the context of EGI user support.

So, you want to run a training event

There are some events which you need to run repeatedly, where the workflow is more or less constant, even though the specific details of each instance may change. One of the north stars of engineering in our line of work is to eliminate toil¹: work that is

manual
repetitive
automatable
tactical
no enduring value

In this case, we’ll take a look at preparation for training events from a perspective of eliminating toil, and see what the effects might be on participant experience, as well as overhead in running the event.

Treating Training Environments as Products

When a user comes to a training event, whether we like it or not, whether we realise it or not, we are selling them an experience, before they actually use the service. If that experience is a good one, and if they perceive that the service they are using actually provides utility, then they will come back and do whatever is necessary

A lot of ink has been spilled regarding developer experience (DX), and how to improve it, culiminating perhaps in the practice of platform engineering. One of the main lessons that has been learned is that in order to be successful, the platform needs to be designed as a product, making it appealing and engaging for the end users, and creating “golden paths” for them.

With this as backdrop, let’s see if we can apply these ideas to a training event where we are exposing potential users to a service for the first time. A traditional training event might start at “zero” and walk the user through all of the initial steps necessary to obtain access to the platform, eventually getting to the point where they have access to the actual service they want to use.

Let’s take a look at what might be considered a “good” user journey – one which attempts to demonstrate the value (utility) of the service to the user as soon as possible:

--- title: Demonstrating service utility during training config: journey: actorColors: ["#59c9a5", "#465775"] --- journey section Awareness Discovers service via EGI page: 5: User Registers to event: 6: User section Access Has pre-prepared credentials: 4: Operator Pre-registered in group / VO: 4: Operator Accesses service: 5: User

Here, we split the journey into two sections – Awareness and Access, showing the user emotion from positive (high) to negative (low) as they complete their journey to accessing the service². The goal is to get the user in front of the service as fast as possible, with as little distraction as possible. In order to achieve this, however, we have created an artificial scenario where the user is assuming an identity which we have already created and approved in the group. We are hiding complexity from the user in order to use the little time we have in contact with them to greatest benefit.

Our goal is to deliver value to them, their goal is to conduct their research.

We need to spend every moment of the time they have dedicated to spend with us towards convincing them that our service can help them reach their goal.

As you can see though, this requires some intervention on the part of the operator in that they need to provision the users and the training environment which the attendees of the tutorial will use. As it turns out there is already a good recipe for creating the training environment, but what about provisioning the users?

The invisible barrier

This has until now been a source of toil – or rather the toil has been rejected and implicitly pushed onto the users. We know that there is actually a huge barrier for them to accessing our services, the Authentication and Authorisation of the user. This barrier is there by design in order to place access controls and service level quotas on the services we provide. The main problem is that it creates a huge mental separation between us and the attendees since we have already been granted access to the service, while they have not. We don’t feel the disillusionment and pain of having to overcome that barrier, while they do – and often it is the first emotion they have when trying our services. Let’s take a closer look at that Access section, as it stands for the average first-time user.

journey section Access Requests service: 8: User Selects IdP: 6: User Logs in IdP: 5: User Info Release IdP: 4: User Info Release SP: 3: User Access Denied: 1: User

Since the trainer doesn’t know who the users are, or which IdP they will use to log in, they can’t pre-approve actual real people, so the user journey that starts at the service login page is a dead-end.

In fact, the user needs to first be enrolled in a group or Virtual Organisation (VO), and thus authorised to access the service. What is more, they need to be authorised to access that service in a specific context – i.e. as a member of a specific group or VO.

Taking stock of this means understanding that there is actually a whole other journey for the user to take – entering a specific group.

Golden paths for first time users

Understanding that we now actually have two journeys for the user to complete, we can keep their goals clearer, rather than sending them on a winding path with dead ends:

First, we create a paved road³ for using the service. Outcome: They want to use the service.
Then, we show them the paved road for accessing the service. Outcome: they understand the trust and community aspects of the service.

Separating these two experiences can be done in different sessions of a tutorial, and will help the attendees remain focussed on one goal at a time.

Eliminating Toil

So much for the product development and retention side of things – we previously mentioned that pre-populating the training users would be a source of toil, even if it made the user experience much better. What if we could eliminate that toil by encoding the process?

The user pre-registration workflow looks a bit like this:

--- title: Preparation for User Training config: useMaxWidth: true theme: base themeVariables: primaryColor: "#00ff00" flowchart: fontSize: 32px --- sequenceDiagram autoNumber actor A as IdP Admin participant i as IDP V->>A: We need users A->>i: Create usernames and passwords A->>O: We have users actor O as Operator participant r as RPA O->>r: Invoke Process activate r participant c as Check-In r->>c: Sign Up c->>c: Create New User r->>c: Petition VO deactivate r actor V as VO Manager c->>V: Notify Request V->>c: Approve Request c->>c: Add to VO V->>U: Here are your credentials for the training U->>s: Request service s->>c: Authentication c->>i: Authenticate\nto IdP i->>c: Return\nAttributes c->>s: Authorize s->>U: Grant\nAccess actor U as User participant s as Service

Here⁴ we can see that User only really needs to interact with the Service, and they experience the login procedure as it should be, thanks to the pre-registration of the identities in Check-In.

We introduce a new actor here called “RPA”, which stands for Robotic Process Automation. This is actually some code which executes the process of first signup on behalf of the users – this is the “toily” bit which we would like to eliminate from human actors (the Operator in this case).

The process has been implemented using Robot Framework, by impersonating the user itself, and driving a browser to complete the tasks the user would have done. This is broken down into tasks shown in the sequence diagram above:

Signup: Sign up a new user to Check-In
Join VO: Petition to join the VO
Clean Up: Collect created EPUIDs and send them to Check-In admin for deletion.

The first two are mentioned in the sequence diagram above, while the last is a process which happens once the training event is completed. In principle, we could have a multi-actor RPA, which means that we could invoke the roles of VO Manager or IDP Admin as well, but for now, we are only focussing on the part in the middle: registering users and requesting to join the VO.

Let’s take a closer look at the code.

*** Settings ***
Library             Browser
Library             DataDriver    users.csv
Resource            login_resources.robot

Suite Teardown      Close the Browser
Task Setup          Open the Browser
Task Template       Signup

*** Tasks ***
# Sign with user    Default    UserData
Signup    Default    UserData

Here we can see the single task referred to above. It is parametrised to use an input file of users, containing their usernames and passwords.

We had to write the task keyword Signup ourselves – it’s contained in the login_resources.robot file you see declared as a Resource in the settings part:

*** Keywords ***
Signup
    [Arguments]    ${username}    ${password}
    # Click on EGI SSO where the users have been created
    Click    selector=${EGI_SSO_SELECTOR}

    Wait For Navigation    url=https://sso.egi.eu/egissoidp/profile/SAML2/Redirect/SSO?execution=e1s2
    # fill in the login form
    Provide Credentials    username=${username}    password=${password}

    # Accept info release
    Click    css=.grid-item > button:nth-child(1)
    Wait For Load State    domcontentloaded    timeout=30s

    # Follow the signup flow and accept info release and Terms and Conditions
    Click    css=div.grid-item:nth-child(1) > button:nth-child(1)
    Wait For Navigation    url=https://aai.egi.eu/registry/co_petitions/start/coef:2

    Click    selector=a[href='/registry/co_petitions/start/coef:2/done:core']
    Click    css=.checkbutton
    Click    css=div.ui-dialog-buttonpane:nth-child(11) > div:nth-child(1) > button:nth-child(1)
    Check Checkbox    selector=id=CoTermsAndConditions1

    # Submit the request and wait for the server to log us out
    Click    css=div.submit:nth-child(1) > input:nth-child(1)
    Wait For Navigation    https://aai.egi.eu/registry/pages/public/loggedout

Provide Credentials
    [Arguments]    ${username}    ${password}
    Fill Text    id=username    txt=${username}
    Fill Text    id=password    txt=${password}

The various keywords you see there such as Click, Wait For Navigation are all in the Robot Framwork Browser library which is a python wrapper around Playwright. It drives an actual browser to complete the workflow.

Once the user has created a unique ID in Check-In, we can complete the registration by petitioning to join a VO.

*** Tasks ***
Join VO    Default    UserData

*** Keywords ***
Join VO
    [Arguments]    ${username}    ${password}
    Click    selector=${EGI_SSO_SELECTOR}
    Fill Text    id=username    txt=${USERNAME}
    Fill Text    id=password    txt=${PASSWORD}
    # Click Login button
    Click    css=.grid-item > button:nth-child(1)
    Click    css=div.grid-item:nth-child(1) > button:nth-child(1)
    # This does not redirect me to the VO enrollment page
    Go To    ${VO_SIGNUP_URL}
    # We still have the cookie, so select the favourite
    Click    css=#favouritesubmit
    # Review AUP
    Click    selector=.checkbutton
    Click    css=div.ui-dialog-buttonpane:nth-child(11) > div:nth-child(1) > button:nth-child(1)
    Check Checkbox    css=#CoTermsAndConditions114
    Click    css=div.submit:nth-child(1) > input:nth-child(1)

    Wait For Navigation    https://aai.egi.eu/registry/
    # Pending acknowledgement notification

The cleanup task is similar, but logs into Check-In and populates a file with EPUIDs to be sent to the Check-In admin for deletion.

Discussion

Have we really eliminated toil

We now have two simple tasks that have been codified: a repeatable procedure which can be reliably performed by a computer! We have removed the work from the Operator, eliminating part of the toil in this procedure. We have written these tasks to perform actors as a single role so far – that of the prospective user

The creation of the users in the IDP – this is perhaps already encoded, but does not notify the next actor in the sequence (Operator)
Approval of the VO registration request – this can be done by a VO admin, either in bulk via the API, or as part of the registration process, but requires a change in actor. Currently the notifications are handled via Check-In’s builtin notification systems, so there is a connection between these two procedures (request and approval of VO membership).
The user need to be notified about their credentials. This is probably done in person by the training instructor. Since this involves the transfer of sensitive data, it’s probably a good idea to transfer this information in person.
Users still need to be removed from Check-In by the Check-In admin.

Have we improved the user’s experience

With this pre-registration in place, let’s take a look at how the attendee experiences the training. Let’s remember that we have just a few hours to convince them that they should use our services, and that their decision will be based on their experience of the service itself – does it actually provide them with value? Taking a closer look at that hypothetical “good” experience we mentioned above, from the user’s point of view it looks like this:

--- title: User config: journey: actorColors: ["#59c9a5", "#465775"] --- journey section Awareness Discovers service via EGI page: 5: User Registers to event: 6: User section Access Provides credentials: 5: Trainer Logs into Check-In: 3: User Releases Information: 3: User Authorised to use Service: 9: User

The user initially feels a bit of a dip in their enthusiasm – when their curiosity is focussed on the service itself, every click we put between them and the service is a disappointment. However, here we design a “peak-end” experience – the final result of the user’s journey through login is entirely positive as they are granted immediate access to the service.

Natural extension: RPA for training

So, we have removed some toil and improved the user experience for these crucial first-touch events – we can probably consider this an unconditional win! But improvement is itself a journey, not a destination, so let’s consider what would be the natural extension of this improvement, for next time.

The goals here are the same: Create a satisfying experience for new users in a training environment, while eliminating toil from the people involved.

Let’s consider that we have a lot of time between when the user registers for the event, and when they actually participate to the event. We could use this time to enable an email drip campaign to help the approach the biggest obstacle of all, which is also only experienced once at the beginning: signup to Check-In.

This in fact consists of two different tasks -

Create an entity/account in Check-In
Request to join a VO

Each of these tasks should end on a positive note and should be completed “quickly”, giving the impression that they are making progress, and achieving results.

Now, we don’t yet have an email service which can be programmed via API with transactional emails, but creating something like that isn’t hard. We could imagine a series of hooks starting from the first interaction of the new user – the registration for the event in Indico – with a persistence layer to keep track of progression, to implement such an end-to-end workflow for nudging users onto the service before the training event. This would result in the same experience as what we have just shown for pre-registration, but without the downside of having to overcome the registration process after the training.

This is a very common approach for any cloud service or SaaS, since it’s a well-known heuristic that users are easily distracted and discouraged by longer onboarding paths.

Another aspect which we need to better perform is some actual user research while new users are onboarded. How do they really feel when accessing our services for the first time? What are they really thinking as they go through the motions of creating accounts and requesting access to service? It would be good to have some better feedback based on consistent user research during the event, rather than responding to surveys after the fact.

Finally, we can indeed define golden paths for new users, and these can be extended from first touch to production usage, with the adoption of these RPA tools and a little bit of automation.

At the end of the day, this is all about keeping the customer engaged, making sure they achieve small, continuous wins, and ensuring that we demonstrate utility first.

Footnotes and References

See The Google SRE book for more. ↩
I pulled these emotions graded from 0-10 straight out of the air, I have no data to back this up. It would be really interesting to measure user happiness along their journey as part of the typical training activity. ↩
The “Paved Road” or “Golden Path” are used interchangeably here, referring to an easy, intuitive, straightforward path from start to goal. Read about some of the differences on the Octopus blog ↩
I keep getting kicked in the teeth for my optimistic view of mermaidjs. Man, creating this diagram was a pain, and the box and destroy features just straight up didn’t work. Oh well. Next time. ↩

Consul for external services

Sun, 24 Mar 2024 00:00:00 +0000

Table of Contents

Service registration vs service discovery
Registering services in a modern service catalogue
Using Consul as Service Catalogue
- Problem statement: Declared vs Actual states
- Terraforming federated services into Consul
Results
- I promised you UX improvements
- Discussion

Service registration vs service discovery

This is a short experiment in using Consul as a service discovery tool for services in federated infrastructures.

Traditionally, services were not “discovered” but rather registered in a configuration database, which then acted as a source of truth for clients wishing to find out information regarding the available resources in the federation. This configuration database (CMDB) remains an authoritative source of truth, since service owners manually register their inventory therein and there is a high level of manual verification, however it is not easily queryable.

One of the main infrastructure services is the information system, which is implemented with the Berkeley Database Information System (BDII) with a specific LDAP schema. The BDII provides a hierarchical way to propagate service registration from so-called “sites” up to a centralised service catalogue. This model was adopted for more than a decade and supported a truly massive computing effort in the pursuit of new knowledge.

My memory may be failing me after almost 7 years away from actually working in this environment, but I remember actually using the BDII being extremely laborious. There was a lot to actually remember, which ended up written in wiki after wiki, generating a great amount of toil and thus cost, and communities inevitably ended up needing to centralise and codify knowledge about the state of the infrastructure in single instances.

Registering services in a modern service catalogue

A common design pattern in the 2020s is to use a service mesh to register services and define their connection permissions. This is conceptually similar to the BDII approach of creating and registering services from the ground up, but instead of using LDAP with LDIF updates, we use DNS.

This brings a radically different approach to actually using the infrastructure, i.e in user experience (UX). Relying on DNS means we no longer need to remember arcane ldapsearch commands with undecipherable filters, we just need to call a DNS name. The service catalogue also takes care of organising and routing traffic to the desired endpoints so that applications do not need to be aware of changes in the infrastructure.

Using Consul as Service Catalogue

Services deployed in a Kubernetes cluster for example get these benefits for free as part of the environment they are running in, but we do not have that luxury in the fedearated infrastructure world. While some of these services may end up deployed in Kubernetes within a specific local context (site), it is inconceivable that the entire federation would be. The services can still be registered as external services which services within a Kubernetes cluster can discover.

However, one does not need to adopt the full apparatus of Kubernetes to benefit from this. Consul from Hashicorp is well-adapted to bridging legacy and cloud-native infrastructure, since it is designed to be multi-platform. I wanted to investigate how it could be used together with existing infrastructure in order to improve the UX of communities which have adopted modern tooling but nevertheless need to use existing federated resources.

Before we get to the big picture stuff though, let’s get down in the weeds to see how this could be done effectively.

Problem statement: Declared vs Actual states

The source of truth hasn’t changed, and we don’t want to break the Service Management standard we’re used to: FitSM. Rather, we want to use existing structures to improve UX.

Service discovery can be considered part of the Configuration Management process (CONFM), which has the following specific requirements¹ according to the standard:

PR11.1 The scope of configuration management shall be defined together with the types of configuration items (CIs) and relationships to be considered.
PR11.2 The level of detail of configuration information shall be sufficient to support effective control over CIs.
PR11.3 Information on CIs and their relationships with other CIs shall be maintained in a configuration management database (CMDB).
PR11.4 CIs shall be controlled and changes to CIs tracked in the CMDB.
PR11.5 The information stored in the CMDB shall be verified at planned intervals

The configuration management database is commonly understood to be the GOCDB, the operations database which implements the functionality required to satisfy the requirements above. It is a source of information, which is is supposed to describe the actual state of the world. Unfortunately, it is not authoritative in that sense, since it is merely a database, and the items in it have no controls associated. There are no connectivity checks, health checks, performance, etc - it’s all manually added.

Now, it is true that those checks are delegated to a different service (the monitoring service ARGO), but again there is no way to interrogate the actual state of the infrastructure in an operational sense. If I my application or community to use the currently “good” services, I need to somehow hook into the monitoring system, find which is the currently good set of services via some arcane query, and keep doing that all the time.

Terraforming federated services into Consul

What if we could create an environment where the healthy services were simply discoverable by the thing we all know and use every day: DNS. I’ll show now how Terraform can be used to query the GOCDB, register services in a Consul catalogue and then use the Consul DNS interface to use and discover these services.

UX improvement:

This exercise will demonstrate how make NGI or ROC-level BDII endpoints available via DNS.

Before: User needs to query GOCDB to find a relevant endpoint, remember a default one, or hardcode it in the environment :anger:
After: User can look up the currently healthy endpoint using a DNS name :heart_eyes:

We’ll do the following things to:

Create a query script to provide external data to Terraform
Create a Consul external node to assign the declared services to
Register the declared services along with simple health checks in the external node as external services
Deploy an external service monitor to discover and perform monitoring checks on these external services

We will do all this in Terraform, so first things first, we initialize our providers²:

terraform {
  required_version = "~> 1.7.0"
  required_providers {
    consul = {
      source  = "hashicorp/consul"
      version = "~> 2.20"
    }
    external = {
      source  = "hashicorp/external"
      version = "~> 2.3"
    }
  }
}

External Terraform Data source

Next, we’ll need to use the GOCDB as a data source in Terraform. In order to do this, we use the external data source from Hashicorp. This is implemented as an arbitrary program by the user (me), which returns a JSON that Terraform can parse. I wrote it in Python:

#!/bin/env python3
import requests
import xmltodict
import json

def get_top_bdii():
    data = requests.get(
        "https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=Top-BDII"
    )
    xpars = xmltodict.parse(data.text, strip_whitespace=True)
    j = json.dumps(xpars["results"], allow_nan=False)

    r = {"output": str(j)}
    print(json.dumps(r))

if __name__ == "__main__":
    get_top_bdii()

This queries the GOCDB to get the Top-BDII service endpoints. The HTTP call is unauthenticated and returns an XML response which we parse into JSON. The JSON is added to an output dict which is returned to Terraform via stdin as required.

External Consul Node

We now create single external node (EGI) to keep things simple, where we can register all of the declared services³. We declare it as “external” using the node metadata, so that the monitor can discover it and monitor services on it:

resource "consul_node" "egi" {
  name    = "EGI"
  address = "https://egi.eu"
  meta = {
    "external-node" = "true"
  }
}

Consul Service

Now the meaty bit - we need to parse the result of the GOCDB lookup to create the services, and register them on the external node.

In order to do this, I first declare a local variable service_endpoints which filters out the result of the GOCDB data:

locals {
  service_endpoints = { for v in jsondecode(data.external.bdiis.result.output).SERVICE_ENDPOINT : v["@PRIMARY_KEY"] => {
    key                = v["@PRIMARY_KEY"]
    configuration_item = v.GOCDB_PORTAL_URL
    hostname           = v.HOSTNAME
    sitename           = v.SITENAME
    in_production      = v.IN_PRODUCTION
    scopes             = v.SCOPES
    monitored          = v.NODE_MONITORED
    notifications      = v.NOTIFICATIONS
    country            = v.COUNTRY_NAME
    roc                = v.ROC_NAME
    scopes             = [v.SCOPES.SCOPE]
    }
  }
}

Now we have a nice variable of type map which we can loop over the keys of in order to register the service declared there in our Consul catalog.

Since there is a 1-to-many mapping between sites and service instances, we need to register the service and check with unique names. We will use a combination of service name and primary key as defined in GOCDB for this.

In order to avoid repeating ourselves, we will use the for_each keyword for the consul_service resource and loop over the local.service_endpoints keys.

for_each   = { for i in local.service_endpoints : "${i.sitename}-${i.key}" => i }
  node       = consul_node.egi.name
  address    = each.value.hostname
  port       = 2170
  name       = "top-bdii"
  service_id = "top-bdii_${each.key}"
  tags = concat(
    [each.value.sitename],
    [each.value.roc],
    flatten([each.value.scopes])
  )
  check {
    tls_skip_verify                   = true
    check_id                          = "service:${each.value.sitename}-${each.value.key}"
    name                              = "${each.value.sitename} top bdii check"
    interval                          = "1m0s"
    timeout                           = "20s"
    tcp                               = "${each.value.hostname}:2170"
    notes                             = "${each.value.sitename} TCP check"
    deregister_critical_service_after = "720h0m0s"
  }
  meta = {
    primary_key   = each.value.key
    ci            = each.value.configuration_item
    hostname      = each.value.hostname
    sitename      = each.value.sitename
    in_production = each.value.in_production
    monitored     = each.value.monitored
    notifications = each.value.notifications
    country       = each.value.country
    roc           = each.value.roc
  }
}

We’ve added a series of key-value metadata to reproduce the kind of information that one would find in the GOCDB. As we’ll see, many of the instances registered there are not alive, so their service checks will immediately fail and the service will soon be deregistered. We only register one health check for now, which is a TCP check on the LDAP server. It would be far better and more accurate to add an actual LDAP search script check⁴, but we’ll get into that later in the discussion below.

External Service Monitor

Finally, we need to run an external service monitor.

I am obviously doing this with something you might not have: a beautiful Nomad cluster. The external service monitor can be run next to any Consul agent, so in principle you could run it as a systemd unit on one of your Consul agents. I’m using Nomad so that I can easily manage the deployment and lifecycle. The final resource is therefore a nomad_job:

resource "nomad_job" "consul_esm" {
  jobspec = templatefile("${path.module}/consul-esm.jobspec.hcl", {
    consul_esm_version = var.consul_esm_version,
    # spread over available nodes
    count = 5
  })
  rerun_if_dead = true
}

with associated Job Specification.

Results

Now, let’s deploy this monstruosity and discuss the result. The Terraform plan looks sane, with a bunch of

# module.example.consul_service.top-bdii["AEGIS01-IPB-SCL-1183G0"] will be created
  + resource "consul_service" "top-bdii" {
      + address    = "bdii.ipb.ac.rs"
      + datacenter = (known after apply)
      + id         = (known after apply)
      + meta       = {
          + "ci"            = "https://goc.egi.eu/portal/index.php?Page_Type=Service&id=1183"
          + "country"       = "Serbia"
          + "hostname"      = "bdii.ipb.ac.rs"
          + "in_production" = "Y"
          + "monitored"     = "Y"
          + "notifications" = "N"
          + "primary_key"   = "1183G0"
          + "roc"           = "NGI_AEGIS"
          + "sitename"      = "AEGIS01-IPB-SCL"
        }
      + name       = "top-bdii"
      + node       = "EGI"
      + port       = 2170
      + service_id = "top-bdii_AEGIS01-IPB-SCL-1183G0"
      + tags       = [
          + "AEGIS01-IPB-SCL",
          + "NGI_AEGIS",
          + "EGI",
        ]

      + check {
          + check_id                          = "service:AEGIS01-IPB-SCL-1183G0"
          + deregister_critical_service_after = "720h0m0s"
          + interval                          = "1m0s"
          + method                            = "GET"
          + name                              = "AEGIS01-IPB-SCL top bdii check"
          + notes                             = "AEGIS01-IPB-SCL TCP check"
          + status                            = (known after apply)
          + tcp                               = "bdii.ipb.ac.rs:2170"
          + timeout                           = "20s"
          + tls_skip_verify                   = true
        }
    }

The apply operation added 83 resources in just under 18s. Below is a screencast of the events in Consul while the services are registered, and eventually become healthy.

As you can see, they are tagged as registered by Terraform in the EGI external node. The Nomad job running the external monitor takes a few seconds to come up and perform the monitoring checks which eventually makes healthy service instances go green.

So, in a few seconds, we have both registered the external services, and are monitoring them with a basic tcp liveness check. Let’s see if this makes any difference to a user.

I promised you UX improvements

Now, imagine I’m member of the alice VO and I want to find a top-bdii:

host alice.top-bdii.service.consul
alice.top-bdii.service.consul is an alias for topbdii.grif.fr.
topbdii.grif.fr is an alias for lpnhe-topbdii.in2p3.fr.
lpnhe-topbdii.in2p3.fr is an alias for lpnhe-gs9013.in2p3.fr.
lpnhe-gs9013.in2p3.fr has address 134.158.159.13
lpnhe-gs9013.in2p3.fr has IPv6 address 2001:660:3036:197:134:158:159:13

Consul creates DNS entries for all services and their tags in its catalogue, and only returns healthy instances. Since it’s DNS, there’s automatically round-robin so we don’t risk hitting a given instance too hard.

Since we’ve tagged services with their site name as well as NGI and ROC names, we can also find local, national or regional instances:

host ngi_france.top-bdii.service.consul
ngi_france.top-bdii.service.consul is an alias for lapp-bdii01.in2p3.fr.
lapp-bdii01.in2p3.fr has address 134.158.84.162
lapp-bdii01.in2p3.fr has IPv6 address 2001:660:5310:420:7::1

Discussion

Of course, I’m hiding a few details from you here, dear reader⁵, but bear with me – we are looking at this from the user’s point of view. If a community decided that they wanted to use a modern stack, but use some of EGI’s federated services, they could use this approach to discover infrastructure services. This could greatly improve the performance of workflow engines for example which need to keep an up-to-date list of healthy compute endpoints. The last time I touched this problem, it was done using either a hardcoded list of endpoints or an unweidly and unreliable GIIS lookup. Being able to find things just by using DNS seems to me a much better approach.

I’m also looking only at Top-BDII services here. I decided to start there because it was easy to write a health check for it and I’m pretty much guaranteed that these will be open to the world. It seems a bit redundant to put service discovery systems (top-bdiis) into another service discovery system (Consul). We could replace the GOCDB query and find UIs in the same way though, which might be a bit more useful eventually.

Another point is the combination of service discovery and service availability checks. I mentioned above that the GOCDB only declares the desired state of the world, but Consul adds to that by including a current state check. There is no historical and statistical information in this, so it’s by no means a replacement for something like ARGO. However, if you have a piece of infrastructure which needs to query a service topology in order to configure itself, this is a better way to go.

In conclusion, the whole concept of external services is really useful here. I can easily envisage a scenario where a community comes up with a set of applications which gets deployed into its platform, but needs to augment them with infrastgructure or compute and data services from the federation. This fun little experiment shows just how easy it is to terraform the federation into your environment.

I’m not proposing anything radical here, but I am intrigued by the idea of including the entire federation into a set of peered Consul datacentres, replacing the entire bdii infrastructure with a combination of Consul’s service mesh and key-value store. I have a sneaking suspicion it would be quite handy in creating the controls which are required to satisfy the FitSM CONFM process requirements. Consul’s documentation says it should be able to scale effectively… but I’m more interested in entirely eliminating the need for site information services and using it as a distributed source of truth for configuration items.

For now it’s just a thought experiment, but I really want to scratch that itch.– I look forward to extending the approach to see what else we can do :star:

Taken from the FitsSM standard, section on Configuration Management Process ↩
Note that we will be using environment variables to configure the Consul provider (address and token). The backend is not declared here, but in the actual instantiation, the backend is also Consul. ↩
I thought about binding these declared external services to actual exsting nodes, and then requesting that the monitor run on those nodes, but there is currently no operator loop between terraforming the services and the node state. Nodes could therefore potentially fail, taking the services registered on them with them, so I decided on the “fake” external node. ↩
This kind of script check is exactly what the Nagios-based ARGO monitor does. ↩
First of all, I’ve got my environment set up to be able to use the Consul DNS by having an agent running locally. ↩

Deploying Krateo with Terraform

Sun, 11 Jun 2023 00:00:00 +0000

This post will describe my experience in getting up and running with Krateo in a toy environment on Digital Ocean.

I will pay particular attention to the paper cuts encountered while getting this up.

Problem Statement

Let’s set a few criteria for ourselves, to see if this little experience was successful:

Zero-touch declaration: I shouldn’t have to do anything but write a declaration of what I want. No scripts, no manual intervention, no checking things somewhere.
Zero-magic: The declaration contains everything I need to know. I shouldn’t have to invoke a deus ex-machina at some point, assuming that there is something else I already know.
Minimise work: I should have to write the smallest possible declaration, the smallest possible set of moving parts

The solution will consist of a Kubernetes cluster with Krateo installed on it, exposed by a load balancer, with a DNS name associated with it. You will need:

A Kubernetes cluster
A DNS zone which you control

Approach

I’m using Digital Ocean (DOKS) to create the Kubernetes cluster, and Cloudflare to manage my DNS zone. This is an almost zero-cost way to set up the basic infrastructure required for the problem¹, but acounts on these platforms are a prerequisite to using them.

My desire was to be able to implement solutions to the problem in different ways and evaluate them. These solutions are in an accompanying repository: brucellino/jubilant-umbrella

Terraform implementation

The first implementation (and the only one so far) was done with Hashicorp Terraform. This choice satisfies the three criteria state above, by containing a single definition of all of the infrastructure, with zero manual intervention and no undeclared steps.

The core of the implementation are two resources:

The first is the actual K8s cluster required to install Krateo to, while the second satisfies the requirement that the endpoint be passed to the ingress controller in order to expose the services.

There are several other tidbits which I found necessary to add, whether for conciseness, elegance, or one of the three requirements stated above.

Vault provider to configure cloud providers

I have tokens for Cloudflare and Digital Ocean stored in my Hashicorp Vault, which are consumed as Terraform data sources in order to pass them to provisioner {} blocks for the respective cloud providers. While this is not strictly required, it’s a default engineering practice I always employ when building infrastructure with Terraform and adds a bit of safety to the process. Making the provider secrets declarative instead of hiding them in environment variables or special files which cannot be committed to the repository makes a clear separation of concerns between the infrastructure’s code and data.

Somebody else can thus more easily re-use this terraform module, just by passing the relevant Vault parameters to their data lookup:

data "vault_kv_secret_v2" "do" {
  mount = var.do_vault_mount
  name  = var.do_vault_secret
}

provider "digitalocean" {
  token = data.vault_kv_secret_v2.do.data["terraform"]
}

This has the downside of having to include another provider (Vault), but that’s a satisfactory tradeoff for the safety and re-usability that we gain, in my opinion.

Installing Krateo

Krateo installation is an imperative task; the Krateo CLI has to be executed against the cluster, and cannot simply be declared into existence. This fact breaks the first requirement (zero-touch declaration) at first glance, unless we can find some way around it. The default approach would be to

Terraform Digital Ocean to declare the K8s cluster into existence
execute krateo init --kubeconfig ${output from first step}
Terraform Cloudflare to declare the DNS record into existence

This kind of imperative, step-by-step approach is prone to being unreliable, and is one of the main reasons that declarative formats are strongly suggested. Yes, we can codify these steps into a pipeline, and that’s a great start if there’s no alternative, but any pipeline can fail unpredictably. Besides the reliability of the pipeline, we’re adding extra work by forcing steps to be taken, in a specific order. We have to keep several things in our head at the same time, which are not explicitly and irrevocably linked between them. In the software development world, this is the kind of thing that would compile fine, and then generate runtime errors, forcing the developer (operator in our case), to break a state of flow with an interruption, go back and debug.

Long story short, I really wanted to make the deployment as declarative as possible, so I chose to add a null_resource resource linked to the creation of the kubernetes cluster, in order to represent the imperative Krateo installation.

Results and Discussion

The final results of this experiment may be summarised as such:

The criteria set in the problem statement were respected, save for one
Krateo installation was done successfully, but
I couldn’t get it to expose the app endpoint properly

These results are discussed in a bit more detail below, but all-in all, I’d give myself a 70% satisfaction rating.

Interactivity and Concerns

The first paper cut I encountered was having to deal with the Krateo CLI interactivity. While I could use the attributes from the digitalocean_kubernetes_cluster to write the kubeconfig file using a local_file resource, and thus pass it to krateo init --kubeconfig, but the CLI expects human input in order to configure the app endpoint. I found this somewhat inelegant, based on the criteria I’ve set myself, and I would have preferred to declare the domain name in some way. I ended up having to pass it to the CLI via the command line in the local_exec provisioner used in the null_resource representing the Krateo installation:

echo '${var.cf_zone}' | ./krateo init --kubeconfig ${local_file.k8sconfig.filename}"

Here we can see that we pass the cloudflare zone represented by the variable cf_zone to the krateo init execution via a shell pipe – old school.

The second thing I needed to take care of was to implement a way to cleanly remove all of the resources that were created by Krateo during installation. In this small experiment, the only such resource was a loadbalancer created to expose the app service. This is not in the Terraform state, but instead in the Krateo state. Krateo isn’t in the Terraform state either, only the null_resource representing it – so if we do a terraform destroy, the resources that Terraform knows about will be destroyed, but not those Krateo made.

Luckily, Krateo implements a cleanup target for its CLI, so we can invoke that at destroy time by adding a relevant Terraform provisioner, which should run. Putting it all together:

resource "null_resource" "k_install" {
  triggers = {
    kube_config = local_file.k8sconfig.filename
  }
  provisioner "local-exec" {
    when        = create
    command     = "curl -fSL ${local.krateo_release_url} | tar xz krateo >krateo"
    interpreter = ["/bin/bash", "-c"]
  }

  provisioner "local-exec" {
    when        = create
    command     = "echo '${var.cf_zone}' | ./krateo init --kubeconfig ${local_file.k8sconfig.filename}"
    interpreter = ["/bin/bash", "-c"]
  }

  provisioner "local-exec" {
    when        = destroy
    command     = "./krateo uninstall --kubeconfig kubeconfig-krateo-control-plane"
    interpreter = ["/bin/bash", "-c"]
  }
}

Change the endpoint

What about if I wanted to change the endpoint? Krateo expects, as mentioned above, an input parameter to allow it to tell the ingress controller how to expose its services. This is a hardcoded to app.<domain> where <domain> is the top-level domain that you are deploying Krateo to. This is almost certainly something that can be changed by applying a different configuration, but it would have been nice to have this configurable via the same init function.

The good news was that running init again with a different TLD resulted in the desired configuration being applied.

Unpredictable load balancer name

During installation, Krateo creates an ingress controller which manages a Digital Ocean loadbalancer. The public IP of that load balancer is required in order to add the A record to the DNS in order to interact with the Krateo App, but since this load balancer is managed by Krateo, it is not known to the Terraform state.

I first tried to add an external loadbalancer, which I wanted to tell Krateo about, but that didn’t work out of the box - Krateo ignored it and added it’s own. The data block that should discover this loadbalancer does depend on the Krateo installation null_resource, but there is a delay between when Krateo exits and when the loadbalancer becomes available. However, the real deal breaker is the inability to declare the name of the loadbalancer a-priori, which necessarily introduces esoteric knowledge – magic information that I just need to know and can’t derive.

I ended up breaking requirements 2 and 3 described in the beginning of this post, by

having to add esoteric knowledge (the name of the Krateo-managed loadbalancer)
having to run terraform twice (ugh, gross) in order to pick up the load balancer

# LB created by krateo.
data "digitalocean_loadbalancer" "krateo" {
  depends_on = [null_resource.k_install]
  # id         = "d72d4916-9023-4616-b292-33032dda4799" # <- obtained from the console
  name = "a6434671d1dde4647804e9cd6261d5d6" # <- obtained from the console.
}

resource "cloudflare_record" "k" {
  zone_id = data.cloudflare_zone.k.id
  type    = "A"
  proxied = true
  name    = join(".", ["app", var.cf_zone])
  value   = data.digitalocean_loadbalancer.krateo.ip
}

The name attribute, in my ignorance of how to use Krateo effectively, cannot be known in advance, and is computed by Krateo. I could probably add a null resource to run a doctl in order to look up the load balancer and pass its attributes to the cloudflare_record resource, or a kubectl to do something similar, but I didn’t want to add extra tooling at this point and indeed, I wanted to force the issue by surfacing this “problem”.

This is, in my opinion, a documentation problem more than a design problem, since I can definitely imagine ways to get around this, but they all make me throw up in my mouth a bit.

SSL and ingress errors

The showstopper for me was the inability to actually access the app.<domain> URL due to SSL and ingress errors. Behind the scenes, I could see that all Krateo components had been installed, and everything was reporting healthy. However, I was unable to access the UI, since the URL gave HTTP 522 errors (timeouts). I didn’t spend too much time investigating, but my suspicion was that

A firewall rule was blocking the connection between the LB and the services in the cluster - either at the infrastructure (Digital Ocean) level, or at the API gateway² level
Somewhere a selector was improperly configured – perhaps an authentication service was missing which the API gateway was sending requests to, resulting in the 522

Whatever the true reason, I’m confident that this could be resolved by adding a few kubernetes_xyz resources using the Terraform Kubernetes provider.

Summary

The goal of this little exercise was to get my hands dirty with Krateo, while staying true to some of the engineering principles I hold dear. I was about 70% successful at this. I have no doubt that it’s possible – easy even – to deploy Krateo in this way, with a bit more understanding of how it is supposed to work. I have a suspicion that a bit more detail in the documentation could have helped, or perhaps a tutorial showing how to modify the vanilla installation with a few kubectls after krateo init. I don’t expect the Krateo CLI to do everything after all!

I should also remind the reader that deploying Krateo is very likely a one-time event. This is a service which will act as the control plane for your entire infrastructure after all, so I don’t expect folks to be doing krateo init once a week in the end-use environment. However, for people like me who will eventually end up doing it for clients, the process isn’t 100% yet.

Who governs the governor

There is however a deeper question which has bugged me throughout this exercise:

Am I doing it wrong?

Krateo is for governance, it’s supposed to contain the control plane for everything. But it needs to emerge from the void, something I’ve written about before. The demos I’ve seen before start with “Create a Kubernetes cluster”³, and I can imagine that when used in an enterprise environment, it will be a bit more like “Install Krateo into an existing cluster”. But who creates those resources? It can’t be Krateo because it doesn’t exist in that environment yet. Does Krateo become self-aware after installation and resolve the bootstrap paradox by then managing itself?

I can’t shake the ghost of Godel whispering in my ear:

“the system is necessarily incomplete”.

If something extra is always required to invoke a governor, if this is indeed an emergent property, what is the most elegant way of expressing this?

I do not have an answer to this yet. Do you?

To give an idea, the cost for the cluster and associated resources was 10 euro cents for 3 hours of use. ↩
The API gateway used by Krateo is Kong ↩
As we’ve seen, this is also not sufficient – you need a few other resources in order to properly deploy Krateo. While the DNS domain is mentioned, there are indeed some other requirements which are not declared explicitly. ↩

We are all platform engineers now

Sat, 10 Jun 2023 00:00:00 +0000

We are all platform engineers now

Platform Engineering is clearly the hotness amongst the cool kids, and has been so since at least early 2022. As with all the good ideas that eventually end up mangled by the IT industry, Platform Ops has both been around for a longer time than I bet most folks think, and also doesn’t actually express anything about its nature.

Terms like these are useful for driving the marketing department, naming products and generally starting useful conversations. It’s an opener, but doesn’t actually get you all the way to actually solving a problem.

One thing I am convinced by is that these terms arise because there really are problems worth solving, and perhaps since the birth the term “Agile”, these are ever more business problems rather technological problems. With nigh infinite compute capacity, the questions then became “how the heck do we get things done now?”.

In this vein, the exhortation to collaborate in DevOps movement, the codification of good practices for reliability in SRE and the realisation in DevSecOps that collaboration with the security and safety side of business gave greater benefits if included early, finds a continuation in this pattern of “Platform”.

It’s hard to tell from within my little bubble, but the term “Platform Engineering” seems to have an irresistible attraction in 2023. There will certainly be “haters” – contrarian folks who find their intelligence or experience insulted by the very idea that the knowledge and skill contained within them can be packaged into a product (ugh).

“Where’s the workmanship in that?”

they will scowl and who can blame them? Our market is swimming in terrible products which only exist to make their vendors a dime.

“I can do that in a weekend!”

they will counter, and who can blame them? Most of the products we’re seeing come out are just compositions of other things.

“There’s no way a product, no matter how customisable and extensible, will be able to meet all use cases”

is the line they will draw in the sand, and who can blame them?

Well, yes, these are all good points. I’ve been involved in building things we called “platforms” for over 10 years and I can say for sure that all the things people are talking about in this wave, we’ve been trying to build in one way or another for that whole time.

We didn’t ever call them “internal developer platforms” when we were building science gateways back in 2013, we definitely didn’t treat the worldwide compute grid as a product back in 2003, but these were definitely platforms. You want something else? Build it your own damn self.

So, of course people did – frameworks under toolkits under apps, all in such a delirious state of entropy that nobody could accurately predict what would be around. Remember GIFEE¹? Yeah, I almost forgot about that too, but luckily I was around in 2016 and I’ve seen enough failed experiments know why spidey sense is tingling now that we’re making a hotness.

What’s changed

The OG platforms (including early AWS) were born to serve the entire damn planet and were by necessity inflexible. You get these functions, that’s it. Everybody gets the same. Self-service? yes, you can consume these very specific things, and we’ll bill you for them. We had effort after effort to build a better piece of infrastructure, to expose more specific functions, to bring in new capabilities. I’ll bet that a it’s a commonly-held opinion that all of this converged in the creation of Kubernetes: the codification of all conceivable functions in a data center, in a network, and between applications.

This didn’t solve any damn problems – it created more of them!

What’s changed since the first iterations of platform is that we are now coming to terms with Spidey Rules: with great power comes great responsibility.

Now, I’ve been around, but I haven’t been to every business out there, I don’t know the story of your struggles, I’m not going to pretend I do. But I’m willing to bet that many organisations over a certain age have found themselves mired in transition due to the inability to make decisions responsibly across the lifecycle of the application. Each individual group or function chose “the right tools for the job”, perhaps doing their best to optimise locally²

What exactly are we talking about

I find it better to my taste to be explicit when talking about a subject, so as to help myself understand the boundaries of my own knowledge, and better hone my own opinions. For this reason, I tend to think about – and hence reason about – Platform Engineering and PlatformOps differently.

Did you just google that? Did you have to go to page 2 to feel the same irony I’m feeling? In 2016 had a hashtag and drive IBM to adopt the cloud model, if you believe the things you read on page 2. ↩
This is the generous point of view. A more realistic one is that choices were driven by vendors and personal connections at every level. ↩

Back to serving others

Wed, 04 Jan 2023 00:00:00 +0000

Working through war and plague
Mission and purpose
A detour
Lessons
- Who am I not
- Who am I
Back to service

Working through war and plague

It is not the stroke of midnight, but the breaking of the first dawn of the year which prompts me to reflect on that moment of equilibrium between the year past, and that which is to come.

Allow me, dear reader, to pause for a moment of self-indulgent personal reflection. Nay, come with me, if you will, on a stroll past ourselves as we try to digest how the past few years have moulded us. This is a story about me, but I’m not so special that you are unlikely to find fragments of yourself in my story.

For the past few years, I have described myself as “2020 fugee” – a refugee from the upheaval, stagnation and unrest that was brought upon us by that year of reckoning. I’ve had my share of feeling adrift, moving between one place and the next over the course of my life, but in that carefree past it was more often than not a flight of desire, not of obligation. The stasis of the pandemic, having to accept that things must stand still for a time, that we must all stay where we are, that events must simply be cancelled, erased, that milestones must be deleted and that everything will be done later when it would be safe again, this stasis must now come to an end. It is time to start moving again.

The irony in many cases, including mine, is that I came out of that period completely exhausted, mentally and socially. The mere idea of having to interact, to earn social capital, to be present for others that were not my immediate family, became for a period too much to bear.

I sometimes think about what happened to us during that vaguely defined time of pandemic, and whether we can ever recover from it as a society. The pessimist in me says that some cracks were laid bare and that once we see them, we can never unsee them.

However, perhaps we can go about fixing them.

Mission and purpose

For as long as I can remember, I have been working for common goals, finding my own purpose through a larger mission. First, this was the study of the universe through a series of postgraduate degrees in physics culminating in a Ph.D. and several postdoc positions. Scientific research has for decades been a team sport, and even more so in the physics domain. Personally I found myself perennially in collaborations large and small, working with others towards what we were all convinced was a greater good. During the latter period of my scientific career, this peaked as I worked in what was then a giant collaboration of more than three thousand researchers and engineers. The sense of community I felt then has been something that I will never forget and have sought ever since. There was a mission, and I felt that my work had a purpose through achieving that mission.

My decision to leave that environment was taken half-heartedly. I would have happily stayed in physics, had I not realised two difficult truths:

I was not the world’s greatest physicist, indeed I was quite mediocre and the competition was fierce
Physics as a career would not give me the stability and growth path I wanted.

However, one cannot so easily change one’s essence as one can one’s job. My next career was still in service, more explicitly so, as I returned to South Africa to help build public infrastructure for computing. Looking back on those ten years from 2008 to 2018, I find it hard to judge myself as having had any success. As a harsh critic, I would say that all of the work I did was just playing at the various roles of project manager, community manager, trainer, etc. The end result was that there was nothing really to judge, because the goal posts kept moving, so I can’t even call the outcome of those years a failure! Perhaps this is too harsh a critique, but I will let others be the judge of that.

One theme that did remain constant, I think, was that of working for the benefit of others. Yes, I had my personal agendas which I was following, whether I knew it or not, but this does not detract from the fact that the projects, infrastructure and initiatives that I was trying to move forward were always for the direct benefit of others. This gave me that warm and fuzzy feeling of somehow “doing good”.

A detour

I joined EGI enthusiastically in 2018 to continue this mission, and out of the blue came the biggest opportunity for detour in my life so far. I made the decision to leave the world of research and public service to enter the private sector, because I felt that the time had come to put myself to the test. I had had my share of projects in which I was but a part, which didn’t depend explicitly on me, but which I contributed to and I had always felt the frustration that my peers and collaborators were not “doing it right”. A whiff of arrogance had started to surround me, I suspect, and I found myself correcting folks, pointing out “the bigger picture”, arguing based on grand principles and generally pretending like I knew it all.

What if I didn’t?

But what if I did?

The opportunity to work in a tight, elite team presented itself as I was called to join UEFA’s new DevOps team. I still can’t believe that I was presented with that opportunity and the absolutely amazing environment I was presented with. I found the best colleagues – truly wonderful people, great at their job all at the top of their game – as well as a fertile environment in which to prove myself. I found myself the least prepared, least knowledgeable, least technically capable, least sure of all of my colleagues and that feeling of being uncomfortable and challenged every day was thrilling.

My time at UEFA gave me the chance to really see what I was capable of and held a mirror up to myself, not just the environment I was working in.

Lessons

Needless to say, I learned a lot during that time; lot of technology, yes, but also a lot about how things really work in the real world. Or rather, how things fail in the real world, all of the ways that they work on paper but not in practice. The most valuable experience I gained during that time was not the subtleties of the cloud, performance tuning or monitoring tricks – it was what it takes to be successful, how to think and execute quickly how to solve problems permanently instead of fix issues. I learned a few things about myself which I hope to bring into 2023 and beyond.

Who am I not

First of all, I learned a few things about who I am not:

Not the smartest person in the room: just as when I was a physicist, I am never going to be the smartest person in a team. I have a meandering education and experience and I’m working with folks who have spent their lives working in this specific environment and are really good at it. This means that I have to remain humble and listen to others when they talk because they probably know things that I don’t.
Not the hardest worker in the room: I have a family, I have responsibilities outside of the office, I cannot afford to be dedicated to the work more than 8 or 9 hours a day (even that is a stretch in a normal week). I am older than most of my colleagues, my energy levels are not what they used to be, and I choose to dedicate the best part of that energy to my personal life. I also see this as a job, not a calling. I am here to get work done by being professional, not by being passionate. That is ok, but it means that I need to have habits and set clear expectations both for my colleagues as well as myself and my family.
Not a rock star: I don’t crave recognition. I don’t need people to call out my name, invite me to guest blogposts or podcasts, conference appearances. I don’t need an audience. I do not want to be the face of anything, because I know that it’s all been done before. I don’t want to be bamboozled by hype… get real and show me the data.
Nothing new to say: I don’t have original opinions in this environment. Perhaps I never did! but I’m certain that whatever I have to say has already been said before by those more eloquent and experienced than me. I have seen a bit, done a bit and learned a bit, but after all, we’re talking about computers here. It’s not quantum mechanics. My role is not to come up with edgy hot takes, it is to learn and improve those around me. There is more than one way to lead; “Linkedinfluencer” is not my style.

Who am I

So, am I just the opposite of the things I am not? To an extent, I am defining myself in opposition to things which I have learned that I am not, but there are also things which I am, not just things that I am because I am not their opposite.

I am:

Conscious and appreciative of counter-narratives: I get sceptical when I hear “just-so” stories, broad generalisations and aphorisms at work. “This technology will revolutionise…”, “we have do do things this way because…”, “the data shows…”. These are all narratives, stories that people tell to support an existing point of view. These are really useful, they help us to get on the same page, to get the point across – heck, I do this all the the time myself! – but they are just stories. They are not the truth. The truth is there is always a counter-narrative and if we’re not aware of it, we risk talking ourselves into a corner.
Capable of living in other peoples’ skins: I’ve been around, I’ve seen and worked in places that my colleagues barely know the names of. I know that people think and work in different ways, and that while these may not be original, they are indeed diverse. I became aware that the world is a racist, classist place when I first stepped out of high school, and almost all my experience since then has been a clash between principles of equity and the messed up way the world seems to be organised. Working in Africa and working in Europe are wildly different experiences, with people having wildly different biases, points of view and priorities. Some of these people may have violently opposed world views to yours, but they are still people. Whatever the official line, you don’t have “workers” at work, you have people and those people are frikkin weird… and I am frikkin here for it…
Multifaced experience: I am trained as a scientist and at heart I still reason about the world as a scientist. However, I speak four languages, I have lived in several cities, on several continents. I’ve been poor, I’ve written code, managed projects, I’ve been responsible for liaising with high school principals and foreign ambassadors, I’ve had folks die on me and seen my babies born. I’ve had good managers, I’ve had no managers and I’ve had terrible managers. I’ve been told “whatever it costs, just get it done”, and I’ve been told “there’s no budget, just get it done”. I haven’t really found a way to write this on a CV, but it’s really the biggest thing I’ve got going for me, from an employer’s point of view. Throw me in the deep end fam, I’ve seen it before, I’m good to go.
I care about quality: Quality makes the difference. I am not, by nature, a person who cares about the nitty-gritty details. My scientific training (and perhaps the reason I was so drawn to it), taught me to look for patterns, for general laws, to see the big picture. But in the real world of people, this doesn’t make the difference, it’s quality that makes the difference. Being prepared, anticipating pain, building things that are free of defects, designing for elegant function, rather than appearance.

Back to service

Coming back full circle, my purpose has always been found through others, it’s time to accept and embrace that again. I have spent the last 2 years more or less focussed on myself at work, to the detriment of the community out there I had found myself a part of.

I am taking a turn away from myself and back towards serving others, this time as engineering manager at the company I work at. In my mind, this is an opportunity to “do it right” – to put my own personal convictions and ideas to the test in a challenging way – but also a way to rediscover purpose through empowering others and improving their lives. I want to bring a rigorous approach to designing processes in our company such that they actually make our lives better first: less miscommunication, less toil and unrewarding work, fewer meetings, more context when it’s needed, fewer surprises, putting the right tools and information in the right hands, at the right time. I want to design for elegance, for happiness, for efficiency.

It’s time to channel my inner Deming!

My own experience has been that a good manager can mean the difference between happiness and despair, between fulfillment and frustration, the reason people stay and the reason people leave. Good managers come in different forms. Some of them are memorable for your personal relationship with them, others are able to change the system so that it’s better by design. I don’t know what effect I will eventually have in this new role, but I intend to bring those things that I am, and those things that I am not, to this new role.

I don’t want to be the superhero that everyone sends their troubles to and that then magically makes those troubles disappear – I am not that guy! I want to make things better for others by making my own life a pit of stress… I want to solve problems by removing them from the system.

I want to see people shine. I want to shrug off the lethargy and I want to feel us all picking up that spring in our stride again, to have fun at work, to build things that last.

If this sounds like a nice place to work, hook me up. If the place you work already sounds like this, good for you and your team, and hey, let’s compare notes.

Have a good 2023 y’all,

The Dude has spoken.

Issuing host certificates from Vault with Ansible

Sun, 23 Oct 2022 00:00:00 +0000

Overview
The Vault CA
Issueing certificates
Deciding on whether to issue a certificate
Declaratively determining issue_cert
Further enhancement
Footnotes and References

Overview

I recently released a new Ansible role which is responsible for providing a base layer of configuration machines in a cluster.

This layer is responsible for preparing the machine to host other services, but without depending explicitly on them and as such, it is a somewhat abstract component. It can be applied to any machine by a controller which has access to the In particular, this release deals with the issuing of a X.509 certificate to a machine in order to allow it to communicate securely with infrastructure services such as the service mesh or orchestration services¹.

The Vault CA

The certificate chain we are provisioning is managed by Vault². We have configured our Vault instance with an Intermediate CA, following the Vault “Build your own CA” tutorial. I had previously encoded this as a Terraform module, more as an exercise than anything else.

The real state is currently defined in github.com/brucellino/vaultatho.me. Most importantly, the Vault PKI secret backend role resource is defined there:

# H@H Intermediate CA role

resource "vault_pki_secret_backend_role" "hah_int_role" {
  backend = vault_mount.hah_pki_int.path
  name    = "hah_int_role"
  key_usage = [
    "DigitalSignature",
    "KeyEncipherment",
    "KeyAgreement"
  ]
  allowed_domains = [
    "*.service.consul",
    "*.node.consul",
    "*.node.dc1.consul",
    "*.hashiatho.me",
    "*.station"
  ]

  allow_bare_domains = true
  allow_subdomains   = true
  allow_glob_domains = true
  allow_ip_sans      = true
}

Calls to this endpoint with a valid token would result in the issueing of a certificate to the caller.

Issueing certificates

Before describing exactly how we manage to deliver the certifiate to the host, let’s consider some details. We have a situation where a controller is applying a configuration to an entity in our inventory. The controller has access to a secret which allows it to authenticate to Vault, particularly the token allows calls to issue new certificates, i.e. the /pki/issue/:name endpoint.

Our controller is actually a machine running an Ansible playbook. I had the choice of one of two Ansible modules:

In the first case, we would have to call the API endpoint directly and pass several parameters in the header to request the certificate, whilst in the second, we could call the wrapper module with a similar set of parameters.

In this case, I opted for the latter, with something like this:

- name: Issue certificate to host
  when: (issue_cert | bool)
  block:
    - name: Issue cert from Vault
      community.hashi_vault.vault_pki_generate_certificate: # noqa syntax-check
        role_name: hah_int_role
        common_name: "{{ ansible_fqdn }}.node.consul"
        engine_mount_point: "pki_hah_int"
        url: "{{ lookup('env', 'VAULT_ADDR') }}"
        token: "{{ lookup('env', 'VAULT_TOKEN') }}"
        alt_names:
          - "{{ ansible_hostname }}.node.consul"
          - "{{ ansible_hostname }}.hashiatho.me"
      register: cert_data

Deciding on whether to issue a certificate

The astute reader will not the the task is conditional: when: (issue_cert | bool). This is a check on a boolean fact which we use to determine whether we should issue a certificate (true) or not (false).

How do we know when to set this value to true? Surely we should issue a new certificate when a new certificate is required. Decomposing this statement, a new certificate is required when:

There is no private key present
The public certificate is invalid
1. issued to incorrect host or hostname changed
2. corrupted file
3. expired

In principle, if the private key is present, but the certificate is not valid for some reason, we could look up the public key and CA data in Vault, as long as we knew the serial number of the certificate, but simply revoking the cert and issueing a new one is far simpler.

Declaratively determining `issue_cert`

There is a lot of logic in the process of determining whether to issue the certificate anew. The first few runs of the playbook which tests this role resulted in hundreds of new certificates, one for each run, since the Vault PKI endpoint isn’t a stateful service.

On the other hand I could have implemented the logic in a big script which did all the computing… but this didn’t feel like the right thing to do and would nevertheless result in a lot of extra work.

In the end, I settled on an approach using a few invokations of the stat module and setting facts on the fly:

First, we check all of the certificate files:

- name: Stat cert files
  ansible.builtin.stat:
    path: "/etc/tls/hashi@home/.pem"
  register: stat
  loop:
    - certificate
    - private_key
    - issuing_ca

We get back a large dictionary which we can query to see when the stat on files shows that they are not present:

- name: Set issue_cert
  delegate_to: localhost
  ansible.builtin.set_fact:
    issue_cert: "{{false in (stat.results | community.general.json_query('[*].stat.exists')) }}"

If the certificate files are all present, we can enter the decision branch where we check decide to issue the certificate based on the validity of the existing certificate. We perform a x509_certificate_info and get back the cert info. If it’s valid, issue_cert is set to false. If not, not only is issue_cert set to true, but we also remove the corrupt or invalid files:

- name: Check Certs
  when: false not in (stat.results | community.general.json_query('[*].stat.exists'))
  block:
    # If this fails -- either if the cert is not present or if it is not a valid cert
    # Then the rescue is invoked
    - name: Get cert facts
      community.crypto.x509_certificate_info:
        path: /etc/tls/hashi@home/certificate.pem
      register: cert_info
    - name: Set expired fact
      ansible.builtin.set_fact:
        issue_cert: cert_info.expired
  rescue:
    - name: Remove Corrupt Cert
        ansible.builtin.file:
          path: "/etc/tls/hashi@home/{{ item }}.pem"
          state: absent
      loop:
        - certificate
        - private_key
        - issuing_ca
    - name: Set issue_cert fact
        ansible.builtin.set_fact:
          issue_cert: true

After all of that, we finally have a good value for issue_cert, which is used, as shown above, to determine whether or not to issue a new certificate and deliver it to the host:

- name: Issue certificate to host
  when: (issue_cert | bool)
  block:
    - name: Issue cert from Vault
        community.hashi_vault.vault_pki_generate_certificate: # noqa syntax-check
          role_name: hah_int_role
          common_name: "{{ ansible_fqdn }}.node.consul"
          engine_mount_point: "pki_hah_int"
          url: "{{ lookup('env', 'VAULT_ADDR') }}"
          token: "{{ lookup('env', 'VAULT_TOKEN') }}"
          alt_names:
            - "{{ ansible_hostname }}.node.consul"
            - "{{ ansible_hostname }}.hashiatho.me"
        register: cert_data

    - name: Deliver certs
      ansible.builtin.copy:
        dest: "/etc/tls/hashi@home/{{ item }}.pem"
        content: "{{ cert_data.data.data[item] }}"
        mode: 0644
        owner: root
        group: root
      loop:
        - certificate
        - issuing_ca
        - private_key

Further enhancement

This approach can safely and reliably issue an x.509 credential chain to a host from the Vault CA via the controller. It takes care of checking whether a certificate needs to be issued before actually calling the CA for a new cert, but there are a few aspects can be further improved in later versions.

First of all, we can call the PKI endpoint to tidy the certificate store when a certificate is determined to be invalid. Even more correctly, we should probably revoke a certificate if we are issueing new one, so that we can update the CRLs. This can probably be quite easily implemented as an Ansible handler, but the cert serial number is required in order to call the revokation function. This seems like a good case to use the Consul KV store (or indeed the Vault KV store) to keep host names (keys) and cert serial numbers (values) easily retrievable, rather than having to do a filter on a big list of serial numbers returned by /pki/certs.

For the full role, see @brucellino/ansible-role-base-platform-pi, and open an issue if you would like to discuss!

Footnotes and References

In my case, these are Consul and Nomad. ↩
See the Vault blog ↩

Upgraded Adventure

Patroni for postgres clusters on prem

Managed databases on prem

Patroni fits into my platform

A little bit of engineering

Images

Deployment

Operations

Managing security and risk in a complex system

Problem Statement

Setting the stage

Seeing Risk

Owning Risk

Where have I seen this before

Providing value to customers closes the feedback loop

Putting Dependency Track to use

Tracking dependencies, measuring risk, ensuring compliance

Services in the Security Plane of the platform

Conclusion

Footnotes and References

A terraform module for Vault on DigitalOcean

Vault in a Managed Platform

Solution Context

One Goal – clean architectural interfaces

Design Decisions

Discussion

Design dependencies

Control plane vs security plane

Using the module.

Conclusion

A Platform for Improving Quality of Infrastructure Engineering in Open Science (I)

Lack of constraints leads to lack of function

Patterns impose constraints

Overcoming boundaries

Workflows in platforms vs tools

Event-driven architectures

Footnotes and References

Attracting and retaining new users

So, you want to run a training event

Treating Training Environments as Products

The invisible barrier

Golden paths for first time users

Eliminating Toil

Making the signup robot

Discussion

Have we really eliminated toil

Have we improved the user’s experience

Natural extension: RPA for training

Footnotes and References

Consul for external services

Service registration vs service discovery

Registering services in a modern service catalogue

Using Consul as Service Catalogue

Problem statement: Declared vs Actual states

Terraforming federated services into Consul

External Terraform Data source

External Consul Node

Consul Service

External Service Monitor

Results

I promised you UX improvements

Discussion

Deploying Krateo with Terraform

Problem Statement

Approach

Terraform implementation

Vault provider to configure cloud providers

Installing Krateo

Results and Discussion

Interactivity and Concerns

Change the endpoint

Unpredictable load balancer name

SSL and ingress errors

Summary

Who governs the governor

We are all platform engineers now

We are all platform engineers now

What’s changed

What exactly are we talking about

Back to serving others

Declaratively determining `issue_cert`