Moving a training pipeline off a hyperscaler without stopping it

Every migration plan assumes a quiet period. Machine learning teams do not get one — there is always a model due, and "we are migrating this month" is not an answer anyone accepts.

So the plan has to be sequenced so that work continues throughout, and so that any phase can be abandoned without leaving you stranded. This is the same shape as a general workload exit, but the ordering is different, because in ML the data is heavier than the compute and the compute is easier to move than anyone expects.

Phase 0: find out what you are actually using

Before anything moves, separate the estate into three buckets:

• Plain compute. Instances running your containers. These move essentially for free.
• Storage. Datasets, checkpoints, artefacts. Heavy, slow, and the thing you are billed to remove.
• Managed services. Hosted feature stores, managed training orchestrators, proprietary model registries, vendor-specific pipeline DSLs. These are the work.

The third bucket determines whether this is a two-week job or a two-quarter one. A team running plain containers on GPU instances with data in object storage can move in a fortnight. A team whose pipeline is expressed in a vendor's orchestration format is rewriting that part regardless — decide early whether to port it to something portable, or accept staying.

One more inventory item, easily missed: CUDA and driver versions. Your container almost certainly assumes a driver stack. Check the target provides it before you promise a date.

Phase 1: move the data, and only the data

Data is the long pole, so start it immediately and let it run in the background while everything else continues where it is.

Sequence:

1. Copy the canonical datasets to the target's object storage. One-time egress cost — budget it explicitly, because it will be the largest single line of the migration and there is no way to avoid it.
2. Dual-write new data from the point of copy onward, so the two stay in sync without a second bulk transfer later.
3. Do not move checkpoints in bulk. Move the ones backing production models. The rest are history; leave them, let the lifecycle rule expire them, and pay the storage rather than the egress.
4. Verify by checksum, not by object count. A truncated transfer that produced the right number of objects is the classic way to discover a problem eight weeks later.

Nothing has changed for the team at this point. Runs continue on the old platform against the old copy.

Phase 2: reproduce exactly one run

Pick a real, recent, non-trivial run — not a toy — and reproduce it end to end on the new platform. One run. Resist the urge to move the whole fleet.

The bar is not bit-for-bit identical output. Different driver versions, different kernel selection and non-deterministic reductions make that unrealistic and chasing it wastes weeks. The bar is: final metrics within run-to-run noise, and wall-clock time within a sensible margin.

This phase is where you find the real problems, and they are always the same ones: a base image that assumed a vendor-specific init, credentials injected by a metadata service that does not exist here, a dataloader path hard-coded to a bucket URL scheme, a library pinned to a driver version. All fixable, all invisible until you try.

Time-box it. If one run cannot be reproduced in two weeks, the blocker is structural and you need to know that now rather than after committing the team.

Phase 3: shadow, and measure the thing that matters

Run both platforms in parallel for a few weeks. New experiments go to the new platform; anything in flight finishes where it started.

What to compare — and it is not the hourly rate:

• Cost per completed run, including failures and idle time.
• Time from submit to first step, which is queueing plus provisioning plus staging, and which the team feels far more than any per-hour figure.
• Interruption rate, if you are using preemptible capacity anywhere.
• Storage and egress for a representative month.

Shadow periods have a habit of becoming permanent, so set an end date at the start and hold it. Two platforms is the most expensive state in the whole exercise: you pay for both, and the team maintains two of everything.

Phase 4: switch the default

The cutover is a policy change rather than an event. New experiments default to the new platform; in-flight runs finish where they are; a documented escape hatch exists for a week.

Two things to do on the same day: point the model registry and CI at the new artefact store, and change who has permission to launch on the old platform. Migrations do not fail at the switch, they fail because launching on the old thing stayed possible and convenient.

Phase 5: decommission, properly

This is the phase teams skip, and it is where the savings actually live. Compute stops billing when you stop it. Storage does not.

Checklist:

• Delete or expire old checkpoints and intermediate datasets — usually the single largest remaining line.
• Remove snapshots, machine images and container registry contents.
• Delete idle endpoints, notebooks, schedulers and the always-on head node.
• Cancel committed-use or reserved capacity where the terms allow, and diarise the ones that do not.
• Keep exactly one archived copy of anything with a regulatory retention period, and write down where it is.

Give it a named owner and a date. A migration that is 95% done bills like one that is 40% done.

The traps, collected

In roughly the order they bite: driver and CUDA version drift; credentials from a metadata service that does not exist on the target; bucket URL schemes hard-coded in dataloaders; IAM policies nobody can fully enumerate; container registry authentication in CI; a managed feature store with no export; and per-run egress from a dataset that is still on the old platform, which quietly makes the shadow period cost more than either platform alone.

None of these are hard. All of them take a day each, and the plan should assume five of them.

The honest timeline

Plain containers, data in object storage, no managed orchestration: two to four weeks, most of it waiting for bytes to copy. Add a vendor pipeline DSL or a managed feature store and it is a quarter, most of it rewriting. Anyone quoting you three weeks has not looked at your IAM policies yet.

Antyxsoft GPU instances are launching soon in Greece, Spain and the USA, with no egress charge on the data you move in or out. Join the waitlist and tell us what you plan to run.