Stuck on Elasticsearch 7.10 in AWS, and the route to 9.x

If you run Elasticsearch on AWS's managed service and you're on 7.10, you're not behind on maintenance. You're at the end of the road, and the road ended in 2021.

In January 2021 Elastic announced that from 7.11 onwards, Elasticsearch and Kibana would no longer ship under Apache 2.0, but under a dual SSPL / Elastic License v2 model. That made 7.10.2 the last Apache-licensed release. In April 2021 AWS forked exactly that release into OpenSearch, and in September 2021 Amazon Elasticsearch Service was renamed Amazon OpenSearch Service.

The consequence is easy to miss until it's your problem: the managed service never offered Elasticsearch beyond 7.10, and it never will. Its upgrade path leads to OpenSearch, which has been a separate product for five years now and has diverged accordingly. If what you want is Elasticsearch proper — 8.x, 9.x, with Elastic's feature set and its ecosystem compatibility — the managed service is a dead end by construction, not by neglect.

To be fair to AWS: 7.10 is still under Standard Support on Amazon OpenSearch Service, with no end date announced, so it does keep receiving bug fixes and security fixes at no extra charge. AWS still lists it as a valid target if you're coming from an older 6.x or 7.x version. This is not a security cliff.

It's the surroundings that tell you where this goes. What you're running is a fork snapshot that AWS maintains, not a version upstream is still actively developing. AWS has published end-of-Standard-Support dates and a paid Extended Support tier for the versions on either side of 7.10: legacy versions up to 6.7, and 7.1 through 7.8, left Standard Support in November 2025 and carry an extra flat fee per normalized instance hour after that. For 7.10 itself, no end date is announced. That is not the same as never.

Getting to 9.x is four hops, not one

That was the position a client of mine was in, and 9.x was where they needed to end up. You can't jump there. Every major upgrade requires you to be on the last minor of the major you're leaving, which makes the route 7.10 → 7.17 → 8.19 → 9.x. On top of that, a cluster upgrade doesn't change the version an index was created with, so indices created under 7.10 need reindexing before the 9.x step or they arrive on the other side as read-only archive.

That's a lot of one-way doors to walk through on a cluster that serves traffic. What the situation really called for was two versions running at the same time, so indices could be moved across, verified, and cut over one at a time instead of all at once. A managed domain doesn't offer that shape. So: self-managed on EC2.

What that costs you, stated plainly

A managed domain handles patching, backups, monitoring and node replacement. Self-managed means you build all of that yourself, and you build it before you need it rather than after. Everything below is that bill coming due. It's worth it when you need a capability the managed service structurally cannot give you, and it's a bad trade when you just want a search index and a quiet life.

The shape of it

Three Terraform modules, joined by an SSM parameter holding an AMI ID.

An image pipeline on EC2 Image Builder that bakes Ubuntu with Elasticsearch, Docker and the metrics exporters, and publishes the AMI ID it produced.
A node module: one standalone EBS volume, an Auto Scaling Group of exactly one, a Route 53 record, and a tightly scoped IAM role.
A shared S3 bucket for snapshots, one prefix per node.

The node module is keyed by variant name, so v7, v8 and v9 are three instances of the same module with their own versions and their own volumes. That coexistence is the entire point of the exercise.

Keeping the image pipeline as a sibling module rather than nesting it inside the node module matters more than it looks. The two change at completely different rates — the image rarely, the fleet constantly — and nesting them means every five-second config change queues behind a half-hour image build. Building and rolling out are separate concerns, and the SSM parameter between them is the only coupling.

How to build it

Keep identity out of the image. Bake a volume ID into the AMI and every change costs you a 20 to 40 minute rebuild, and that same image can never serve a second node or a restore test. The image knows nothing: not which volume belongs to it, not which DNS name, not which Elasticsearch version. The boot script supplies all of that.
Data on a standalone EBS volume, ASG at min = max = desired = 1. The ASG throws the node away and brings up a new one; the volume stays put. The cost: an EBS volume lives in one Availability Zone, so the node is pinned to a single subnet, and an AZ outage is an outage until you restore a snapshot elsewhere. That's inherent to one-volume-one-node. It's a trade, not a defect, but write it down somewhere.
Mount by filesystem label, never by device path. A volume you attach as /dev/sdf shows up on Nitro as /dev/nvme1n1, and that numbering isn't stable across reboots. The label lives in the superblock, so it survives snapshot, restore, and attach in a different slot. Use XFS: an EBS restore is a block-level clone, and with btrfs you end up with two volumes carrying the same FSID, which breaks exactly the restore-and-reattach workflow the volume exists for. One consequence of running variants side by side is that every volume carries the same label. That's safe as long as each instance may only attach its own volume, which the IAM policy enforces. It is not safe to attach two of them to one machine, which is tempting during a migration. Move data with reindex-from-remote or through S3 snapshots, never by cross-attaching volumes.
Attach in three separate waits, each with its own timeout. Wait for the volume to report available, call attach, wait for in-use, wait for the device node to appear. Three independent asynchronous transitions. Collapse them into one retry loop and you've built something that works until the day it doesn't.
Make it impossible for the container to start without data. This one is worth walking through, because two guards that look sufficient both turn out to be advisory.

The mount unit carries nofail, deliberately, so that a missing volume never wedges the boot — you want to be able to log in and fix it. That means a failed attach leaves the mount point present and empty rather than absent. The service unit carries RequiresMountsFor=/elasticsearch, so systemd will not start Elasticsearch without the mount. That looks like it closes the gap, and it doesn't: with restart: unless-stopped, dockerd starts the container by itself when the daemon comes up. The systemd unit never runs, so its condition never applies. Docker then does one more helpful thing and creates the bind source as an empty directory because it's missing.

Chain those together and a node whose volume failed to attach comes up with Elasticsearch initialising a brand new, empty index on the root disk, reporting green, and accepting writes into a directory that disappears at the next node replacement. Nothing in the health checks is lying. There is a healthy Elasticsearch. It just isn't yours.

The fix is to bind /elasticsearch/data, a subdirectory the boot script creates only after the mount has succeeded, instead of the mount point itself, and to set create_host_path: false, which on the Compose version in use here stops Docker from inventing it — worth confirming on your own version rather than assuming, since not every Compose release has honored that flag. Since that subdirectory can only exist on the mounted volume, the interlock moves from advisory at the systemd layer to enforced at the container layer, and restart: unless-stopped can stay where it's genuinely useful for crash recovery. One related detail that bites: mounting a filesystem replaces the mount point's ownership, so any chown you did at bake time is discarded. It has to happen in the boot script, after the mount, against the subdirectory.

This came out of a review, not out of a failing build. A green build only proves the happy path executes.
Let the node update its own DNS record from the boot script. A Lambda on an ASG lifecycle hook gives you a guarantee that DNS is correct before traffic arrives. With one node and no load balancer in front, that guarantee buys very little and gives you three times as much that can break. Revisit it if you ever go multi-node.
Two restore paths, two different recovery points. EBS snapshots capture the whole volume and put the data directory back exactly as it was, on-disk format included. You need them, because Elasticsearch refuses to start against a data directory a newer version has touched — not just a newer major, a newer patch release. Rolling back to your previous image therefore looks like a clean Terraform rollback and hands you a node that won't boot; the volume snapshot is the real rollback. Next to that, Elasticsearch snapshots to S3: incremental, per index, taken far more often, so your recovery point is finer and you can restore a single index instead of the world. Enforce retention through the Elasticsearch API, not with a lifecycle rule on the bucket. That rule deletes segment blobs Elasticsearch still references in its own repository metadata, and takes older snapshots down with it.
Pin the image ID, and make node replacement opt-in. Don't let the node track the latest successful build, or a deploy that has nothing to do with Elasticsearch will roll your only Elasticsearch node on a day nobody touched Elasticsearch. The same applies to the ASG's instance refresh: once that block is present, any launch template change triggers a replacement, and launch templates hash on bytes. A feature that's conditional in behaviour is not conditional in the template. I found that out by restarting two nodes that didn't use the feature I'd just added.

Docker now, probably Podman next

Elasticsearch runs as a digest-pinned container under Docker. The container defines its own healthcheck, which is what makes the systemd unit honest: systemctl start elasticsearch only returns once Elasticsearch actually answers on 9200, rather than once a process has been spawned.

The security posture took some care. The Elastic image defaults to uid 1000, and on an Ubuntu host uid 1000 is the ubuntu account — interactive shell, passwordless sudo, home directory. Container uids are host uids, so a container escape at the default lands you on a real account. The node instead gets a dedicated elasticsearch system account at uid/gid 9200 with no home and no login shell, the container is pinned to user: "9200:0" with no-new-privileges: true, and the config tree is root-owned. The gid stays 0 because Elastic's image ships group-readable and owned elasticsearch:root, which is the same mechanism OpenShift's random-uid model relies on.

What's left is the Docker socket, which is root-equivalent: anyone in the docker group effectively has root on the host. The image build asserts that the docker group has no members, so that a future usermod -aG docker fails the build instead of quietly shipping a privilege escalation. That assertion exists to defend against the runtime's architecture, which is a decent sign the runtime is worth revisiting.

So the next image is likely Podman, for two structural reasons rather than fashion. First, there is no daemon: containers are started by systemd units, which puts RequiresMountsFor= back in charge and makes the interlock above declarative again instead of something enforced by a carefully chosen bind path. Second, there is no root-equivalent socket, and therefore no group whose emptiness needs asserting at build time.

What's holding it up is real. Rootless Podman runs the container inside a user namespace, so container uid 9200 maps to a subordinate uid on the host. Both the data volume's ownership scheme and the restore path — which already has to repair ownership when a snapshot arrives owned by a different uid — would need rework, and that is precisely the code path where a mistake means Elasticsearch cannot write to its own index. So it's a planned migration with a throwaway test node in front of it, not a swap.

What the image tunes underneath

The AMI sets the kernel parameters Elasticsearch needs on the host rather than in the container, because they aren't namespaced per container and setting them alongside the container would do nothing. vm.max_map_count is raised well above the distribution default, since Lucene memory-maps aggressively and Elasticsearch fails its own bootstrap check if it's left too low. Swappiness goes right down so the heap is never paged out, and the file descriptor and memlock limits are raised so memory locking works at all.

Memory allocation is a decision, not a default. The heap is 50% of node RAM, capped just under 32 GB: past that the JVM drops compressed object pointers, so a larger heap holds fewer objects and you pay for RAM to get less out of it. The other half stays with the operating system for the page cache, which is what actually makes Lucene fast. The heap value is written in exactly one place, which sounds obvious and wasn't — an earlier version set it both as an environment variable and in jvm.options.d. Both applied identically, so nothing misbehaved, and changing either one alone would have silently done nothing at all.

Standing up a clone next to the live node

This is the part that pays for the rest of it. Because the node is a Terraform module keyed by variant, and because the data exists in two independently restorable forms, putting a second node next to the running one is a map entry and an apply rather than a project.

There are two ways to fill it. Point the new variant at an EBS snapshot of the live volume and it comes up with a block-level copy of the data directory exactly as it was, on-disk format included, which is what you want when the versions differ. Or give it a blank volume and tell it to seed itself from the S3 snapshot repository on first boot. That's slower, but it works across major versions, which the EBS route does not.

Each variant publishes its own DNS record, so the clone is addressable the moment it's up. Point a test harness at it, run the reindex, compare results against the node still serving. When you're satisfied, the cutover is a DNS change, and at a 60 second TTL that's about a minute.

One mechanism, two uses. For recovery: if the live node is lost, including losing the entire Availability Zone its volume is pinned to, you restore the snapshot into a different subnet and bring a node up on it. The AZ pinning in step 2 is a genuine constraint, and this is what makes it survivable instead of fatal. For upgrades and configuration changes: exactly the same procedure, on purpose. Stand the new version up beside the old one, move the data, verify, switch. Roll back by switching back, because the old node is still sitting there untouched.

That overlap is the real win, and it isn't really about speed. Most disaster recovery procedures rot quietly, because nobody runs them until the day they're needed, which is the worst imaginable day to find out the runbook is wrong. Here the recovery path and the upgrade path are the same path, so it gets walked on an ordinary Tuesday, deliberately, with the old node still running next to it. The first-boot restore was itself proven on a throwaway node built for the purpose rather than on anything that mattered: blank volume, seeded from the live node's repository, then deliberately refreshed to confirm it didn't restore itself a second time.

To be clear about what this is: manual blue/green. No traffic shifting, no automated health gate, no automatic rollback. For a single search node behind a stable DNS name that's the right amount of machinery, but it's a procedure someone follows, not a button someone presses.

Whether you actually need to reindex

Worth being precise about this, because it decides whether an upgrade is an afternoon or a project. The rule that matters: Elasticsearch has full read and write support for indices created in the immediately previous major version. It's the index's creation version that counts, not whichever cluster happens to be serving it today. So booting an 8.x node on data carried over from 7.17, or restoring a 7.17 snapshot straight into an 8.x cluster, needs no reindex at all — on either the EBS or the S3 path. Elastic's own snapshot compatibility table shows a plain, unconditional pass for a 7.x-created index restored into any 8.x version.

The trap is indices that were created back in 6.x and have simply been carried forward, unreindexed, inside a 7.x cluster ever since. Those still fail on 8.x, silently, until you look: GET /my-index/_settings?filter_path=**.version.created tells you the truth regardless of what version the cluster reports. Run that, or the Upgrade Assistant, before assuming a 7.x cluster is clean.

If something does need reindexing, and the versions live on separate nodes the way they do here, the mechanism is reindex-from-remote: whitelist the source node's host in the target node's reindex.remote.whitelist, then call POST _reindex on the target with source.remote.host pointing at it. That's the same HTTP-based migration this design already relies on for moving data between variants, so it costs nothing extra to reach for.

None of this applies to the step this project hasn't taken yet. Jumping two majors at once — a 7.x-created index straight into 9.x — is a different case entirely and does need either reindexing or the archive/searchable-snapshot route. One major at a time, which is the whole reason this design runs versions side by side instead of leaping, stays free.

Where it landed

Staging runs a 7.17 node, seeded from a snapshot of the real data, with the 8.x and 9.x variants defined and ready to switch on. The managed domain is still there and untouched: migrating off it is a separate, deliberate step, and bundling it into this change would have meant two risky things happening at once.

Node replacement now completes in minutes, and restoring from a snapshot is a bounded, documented procedure rather than a support ticket. A version change on a managed domain is a different order of magnitude entirely.

What's next: standby nodes, and then real clusters

Everything above is deliberately a cluster of one. That was the right starting point, because a single node makes every failure mode easy to reason about and there was no point solving distributed problems before the single-node ones were solid. It is not the destination.

The next step is a standby: a second node, in a different Availability Zone, kept current from the snapshot repository. That alone removes the sharpest edge in the current design, since the data volume lives in one AZ and an AZ outage is currently an outage until a snapshot comes up somewhere else. A standby turns that into a DNS change.

After that, an actual cluster with replicas, which is a bigger change than adding a node. A handful of decisions in this design were made because there is exactly one node, and every one of them reverses:

Discovery is set to single-node, so there are no peers, so the transport layer never needs to advertise a reachable address to anyone. Real peers bring back both the transport port and the networking decisions that came with it.
Replicas never allocate on one node, so the cluster sits permanently at yellow and the restore automation treats yellow as success. With replicas, green becomes meaningful again and that assumption has to go.
Every volume currently carries the same filesystem label, which is safe only because each instance may attach exactly one volume. More nodes per version means that label has to become node-aware.
The DNS choice in step 6 was made on the grounds that there is no load balancer and nothing routes on health. Put a load balancer in front and the ordering guarantee that was not worth paying for suddenly is.

The snapshot bucket was already built with this in mind, which is why it's a separate module rather than something the node owns: backup data should outlive the compute that produced it, and a multi-node cluster has many writers, none of which should control the bucket's lifecycle. Podman fits in the same window, since moving the runtime is easier to justify while the node topology is changing anyway.

How much of this was AI-assisted

Worth being open about: a large part of the design documents, the Terraform and the runbooks here were written with AI assistance, using Claude. Not as a novelty. It changed what was economically possible on a project this size.

The obvious effect is pace. Getting from a first design note to a working staging node, with the plans and runbooks written up alongside it rather than promised for later, went considerably faster than it would have done writing every line by hand.

The less obvious effect matters more, and it's the one I'd point at. Verification I would never have been able to justify the hours for became routine. The boot script's volume-state function was extracted exactly as bash renders it through the Terraform template, then driven through failure, terminal-state, success and flaky-then-success cases under a real set -euo pipefail — and run against both the fixed and the pre-fix version, so the difference was demonstrated rather than assumed. The format and restore helpers were pulled out exactly as bash sees them through a YAML block scalar and a quoted heredoc, then pushed down every refusal path: missing variables, an unresolvable device, the marker short-circuit exiting before any network call, a newer failed snapshot correctly skipped in favour of an older successful one. For a single search node, that is not a test suite anyone would sign off on building by hand. Here it cost minutes.

The boundary was deliberate. The agent had no access to live AWS at all: terraform apply and the AWS CLI were blocked at the tooling level before any of the Elasticsearch work began. Every apply was run by a human, read by a human, and handed back. The agent wrote code, plans and runbooks, and stopped at the edge where something could actually break.

It is also not a substitute for review, which is the part I'd want anyone reading this to take away. The four defects a green build could never have caught were all found by reading the code, not by generating it.

Looking back, the interesting part of this project wasn't building the happy path. That came together quickly, and it was never where the risk sat. It was one habit, applied to every check I wrote: take the failure that check exists to catch, cause it deliberately, and confirm the check actually fails. That turned up three assertions which could not fail at all — each one negated in a way that makes bash skip it entirely, so all three reported “validation passed” regardless of what they were given. Four defects surfaced this way in total, and a build that went green would never have found any of them.

I write these up to show the reasoning behind the infrastructure work I do, including the decisions that had to be revisited along the way.

Datum: Augustus, 2026

This page in English