Key Takeaways
- Deploying an ML model to a fleet of physically separate industrial plants is not the same problem as deploying to a fleet of cloud regions: the deployable unit has to be the model plus its site-specific configuration, versioned together, not the model alone.
- A validation gate which only looks at aggregate model quality against a generic hold out set will pass models that are inaccurate for a certain site’s sensor calibration or operating range; validation has to be site aware in order to protect a live process.
- Staged, canary-style rollout, a pilot site first, observed, then the remaining fleet, gives a multi-site deployment the same safety property canary releases give a web service, applied to a setting where the “user” is a physical process rather than a browser session.
- It is to say updating a site with a specific version of what is known to be a good model, not just putting back old code which may or may not work, also that which only works if we have perfect deployment traceability back to an exact registry entry.
- Consolidation of a disorganized, manual, site by site release process into a single automated pipeline reports that practitioner which built it saw deploy time to go down greatly and made model reuse across sites a reality; as with any single team’s self reported before and after results that is a directional result not an independently verified one.
Executive Summary
Deploying a machine-learning model through one centrally hosted service is mostly a solved engineering task. The usual process is familiar: build and test the model, package it in a container, release it, monitor a dashboard, and roll it back when something seems wrong. Deploying the same class of model to a fleet of physically separate industrial plants is a different problem wearing the same name. The issue at hand is that which the model function across many sites is a difficult task. Each site has its own sensor calibration, history of process data, and different expectations around change control. What we see is that a model which performs well and safely at one site does not by default do the same at a different site. This article we put forth a CI/CD pipeline which takes machine learning based process quality optimization models out of manual plant by plant deployment to an automated and repeatable process across many plants. We cover model versioning along with site specific config, the pre deployment automated validation which is put in place before a model goes live, the deployment which is done in stages across sites, and the roll back we have put in for when a model performs below par in production.
Introduction
Most public MLOps writing; and much of the tooling built around it; quietly assumes a particular kind of deployment target: a cloud region, a container orchestration cluster, or an API endpoint that scales horizontally over a fast, reliable network. For a recommendation service or fraud-scoring API, that assumption generally works. It holds much less well for a model that optimizes quality and emissions-related metrics on a live industrial process, running against a distinct physical production line at each of several plants.
Three differences matter immediately. First out that the deployment target is a single entity which it is not, in fact we have many separate physical targets each with their own set of sensors, their own history of process data, and also often their own on site computing which is separate from a shared cloud environment. Second, network conditions between a central platform and a plant floor do not reliably resemble a cloud region’s internal network: bandwidth can be limited and connectivity intermittent, so a deployment mechanism that assumes it can always reach every site on demand will eventually be wrong about that. Third, and most consequentially, a bad deployment does not just return an error to a caller; it can push a model that mis-optimizes a live physical process, and heavy-industrial operations already run under change-control expectations, inherited from process-safety and operational-technology practice, that a typical cloud release pipeline was never designed to satisfy.
Moving out of the manual, one time implementation of machine learning models in individual plants which was the past and into an automatic, version controlled, phased approach. With that setup, the same underlying model could be validated and safely reused across sites. The work did not begin with a decision to build a new MLOps platform. Also we began with models which were already in production at one site and a deployment strategy that hit a wall at a few sites.
Background
Here we look at models which look at process quality and emissions in a live industrial setting. We trained and in some cases retrained these models using plant level data. As it is the case that some production plants have the same base process but different variations of it, we had a practical chance to use the models at multiple sites as well as a theoretical one. Also with the effort of fine tuning or retraining the models on each site’s data we were able to make the same modeling approach work for more than one plant if we could do so safely and in the framework of standardized enterprise to plant integration models for better coordination of activities across many manufacturing sites which is also report.
Before this pipeline existed, getting a model from “trained and validated by a data scientist” into operation on a live process at a particular plant was a manual procedure that was rarely documented. The model artifact had to be exported, handed to whoever had systems access at that plant, adjusted for site-specific parameters by hand, and deployed manually. Coordination took place through email or in person; no system tracked it. Models did get deployed, so the process worked in that limited sense. It did not scale, though. Each new site required starting over, deployments could differ from one another, and there was no dependable, searchable record showing which model version and configuration were running at which plant at a particular point in time.
Problem Analysis
Three technical challenges shaped the pipeline’s design:
- A model is not portable by itself; a model plus its site configuration is. The trained model artifact contains the learned parameters, but a process-optimization model needs site-specific inputs as well: sensor tag mappings, calibration offsets, and the operating ranges treated as normal for a particular line. Those inputs differ across plants, even when the underlying process is nominally identical. The problem with treating “the model” as the deployment unit and leaving site configuration until later is that it made the manual process error-prone. A model paired with the wrong site configuration is not merely somewhat inaccurate; it is reasoning correctly about the wrong physical reality.
- Validating a model against held-out data is necessary but not sufficient. In a plant historical hold out set a model may do well but it may also perform badly on a different plant’s sensor calibration or normal operating range. We see that validation which looks at total model quality against a generic hold out set will accept models the hold out set did not label as defective.
- A bad deployment to a live process is a different class of incident than a bad deployment to a stateless service. Rolling back a web service reports to be putting in use the prior version of it; also the world outside of that service is not impacted. Rolling back a model that has been actively influencing an industrial process for hours means reasoning about what state the process itself was left in, not only which software version is currently running; an asymmetry that has to be designed for explicitly, not assumed away as an edge case.
These three problems are what the pipeline described below was built to address, and they are also the three places a naive “just add CI/CD” approach tends to quietly fail; echoing the broader, well-documented finding that most of the sustained engineering effort in a production ML system lies outside the model itself, in exactly this kind of surrounding infrastructure.
Architecture and Design
Source in version control, build and validation, a versioned model registry, staged per site deployment, and live process monitoring which includes a roll back to the registry.
Two simpler designs were considered and set aside, which is what actually justifies the one built here. Pushing an approved model to every site simultaneously, skipping the staged pilot step, is the fastest path from “approved” to “everywhere” and was the closest approximation of what the old manual process did once one site’s deployment was copied to the rest; it was rejected because it collapses “validated in general” and “safe for this specific site” into one irreversible action, so a site-specific problem the gate missed becomes a fleet-wide problem the moment it surfaces. A second option, running inference as a shared, centrally hosted service every site calls over the network instead of running the model locally, would have simplified versioning to a single always-current endpoint, but it makes every site’s live process dependent on network reachability to that service; precisely the assumption the Introduction argues does not hold reliably here. Per-site deployment of a validated, versioned artifact, staged through a pilot first, kept both properties: a site can keep running its last-known-good model if connectivity drops, and a bad model is caught before it reaches more than one site.
Model training code, validation logic, and each site’s configuration are included in the same version controlled repository as opposed to existing as separate tracked entities. At a time when a config change (a new calibration offset, an updated operating range) causes the model to do something different at a site it should go through the same review, history and rollback processes as a code change, not a separate and less controlled path the same declarative, versioned, and auditable approach which is at the base of the config as code practice.
A build stage packages a candidate model, its runtime dependencies, and its serving code into a versioned container image, the same packaging discipline used for any production software artifact. Packaging as a container instead of shipping a bare model file matters less for portability in the abstract and more for a concrete operational property: the exact image that passes validation is the image that runs at each site, closing the “it worked when the data scientist ran it locally” gap the old manual process depended on trusting instead of verifying.
The automated validation gate comes between build and promotion. Most of the design effort went into this stage because it must establish enough trust for a candidate model to interact with a live process. Before promotion, each candidate faces several checks: its data schema and input ranges are compared with the expected sensor inputs for that site; its performance is measured against the currently deployed model using held-out data and, when available, site-specific historical data; and its outputs are tested against known-safe operating ranges. An aggregate accuracy number alone is not enough.That last point echoes long-standing ML engineering guidance to catch a problem before a model is exported for serving rather than after it has already reached production.
A versioned model registry is the boundary between “validated” and “deployed.” Models promoted through the gate are tagged, and every deployment references a specific registry entry rather than a mutable “latest” pointer, following the same principle that a released version is treated as immutable. Rollback is when a site’s deployment reference is pointed at a certain past registry entry which we are not doing a full previous state reconstruction out of memory.
Deployment is done in stages not at the very large scale. Once a new model is approved, we introduce it at one pilot site, or at a small group of sites, and monitor it closely for a set period. The remaining sites do not receive the model until the pilot reports no issues. This is the canary release method, commonly used for production releases in software development and now included in progressive delivery tools. Here, we apply it to a live physical process rather than a web service. Site-specific settings are added during deployment as an overlay to the already validated model. The validated container image is therefore reused across sites instead of being rebuilt for each one. That distinction made scalable model reuse practical: the model and pipeline are shared, while the configuration overlay adapts the validated model to each plant’s particular process.
Figure 1: CI/CD pipeline for multi-site industrial ML deployment, from source control through build/validate, the versioned model registry, staged per-site deployment, and live-process monitoring with a rollback path back to the registry.Implementation
Each stage of the pipeline is an automated step triggered by a change to version control, structured like a conventional software CI/CD pipeline, adapted for the fact that the artifact under change control is sometimes code, sometimes a retrained model, and sometimes a site’s configuration.
A representative (simplified, illustrative) pipeline definition looks like this:
stages:
- build
- validate
- register
- deploy-pilot
- deploy-fleet
build:
script:
- docker build -t registry/process-model:$CI_COMMIT_SHA .
- docker push registry/process-model:$CI_COMMIT_SHA
validate:
script:
- python validate_candidate.py \
--model registry/process-model:$CI_COMMIT_SHA \
--site-configs configs/ \
--baseline-tag approved
artifacts:
reports: [validation_report.json]
register:
script:
- python register_model.py \
--model registry/process-model:$CI_COMMIT_SHA \
--report validation_report.json \
--tag candidate
rules:
- if: '$VALIDATION_STATUS == "pass"'
deploy-pilot:
script:
- python deploy.py --site pilot-plant --model $APPROVED_TAG
when: manual
deploy-fleet:
script:
- python deploy.py --sites all --exclude pilot-plant --model $APPROVED_TAG
when: manual
needs: [deploy-pilot]
A few decisions here are load-bearing, not incidental. The validation step is a required gate, not an advisory one: register only runs, and only tags a build candidate for promotion, when validate reports a pass, so a failed validation cannot be bypassed by a later stage running anyway. Pilot and fleet wide deployment are separate and distinct stages which are not combined into a single “deploy everywhere” step, also deploy fleet includes the pilot stage as a dependency, thus the stage rollout is enforced by the pipeline’s structure which in turn does not leave it up to the engineer to put in that order.
The validation script itself is where the site-awareness described above actually gets enforced:
def validate_candidate(model, site_configs, baseline_tag):
results = {}
for site, config in site_configs.items():
preds = model.predict(load_holdout(site), config=config)
baseline_preds = load_baseline(baseline_tag).predict(
load_holdout(site), config=config
)
results[site] = {
"schema_ok": check_schema(preds, config.expected_schema),
"within_safe_bounds": check_bounds(preds, config.safe_ranges),
"beats_baseline": score(preds) >= score(baseline_preds) - TOLERANCE,
}
passed = all(
r["schema_ok"] and r["within_safe_bounds"] and r["beats_baseline"]
for r in results.values()
)
return passed, results
The loop over site_configs is the part that distinguishes this from a conventional single-target validation step: the same candidate model is evaluated once per site it is expected to serve, against that site’s own holdout data and safe operating ranges, and a candidate only passes overall if it passes for every site it is being validated against. A model that is a genuine improvement at one site but violates a safety bound at another does not get a passing result by averaging the two; both have to hold.
Figure 2: Validation gate decision flow, tracing a candidate model from the validation battery through either a canary deployment to a pilot site or a blocked promotion that never touches a live process.Performance and Scalability
The most direct performance effect of moving from a manual, per-site process to an automated pipeline was on deployment time, the elapsed time from “a validated model exists” to “it is running against a live process at a site.” The author of this pipeline reports we saw a large reduction in deployment time with the introduction of automation which we did not see in the past with manual processes, and that what had been difficult to do before in terms of reusing a validated model at other sites which would have required repeating manual steps each time was made practical. As with any stand alone report of a before and after which is self reported the size of that change is a function of how manual and broken down the former process was; a team that has a more disciplined manual process to begin with will see a smaller relative benefit from the same level of automation.
Scalability issues in this case present a different picture than what we are used to in high volume data pipelines. We see that the validation gate’s cost grows nearly linearly as we scale out the number of sites a candidate is validated against which in turn requires each site to run its own hold out evaluation; as a fleet grows from a few plants to several dozen that step must remain parallel across sites which if not the gate run time becomes the bottleneck. The staged-rollout step scales differently: pilot observation windows are inherently sequential per model version, but once a pilot period concludes, fleet-wide promotion can run in parallel, bounded mainly by how many sites can safely be touched concurrently without exceeding any one site’s own capacity to absorb and monitor a change.
Security Considerations
A pipeline that can push a new model to a live industrial process is, by construction, part of the plant’s attack surface, not just the data platform’s. Access to the registry’s promotion path and to the deployment stage is restricted and audited separately from general repository access, since reviewing code and authorizing a change that reaches a live process are different privileges. Container images are built from a controlled base and pulled from a private registry, and image provenance (what commit, what pipeline run, produced this exact image) is retained, since supply-chain integrity for a model artifact is a documented, field-specific concern distinct from conventional application-security risk. Secrets that grant access to site systems are at deploy time pulled from a managed secrets store instead of embedded in pipeline config or carried over from the old manual process’s ad hoc credential sharing.
In that which is industrial we see an element that pure cloud native MLOps security does not at present include. Plant floor systems typically are in an operational technology network which is purposely separated from general IT, with a risk which is more of an availability and physical safety issue as opposed to confidentiality; the deployment strategy has to take that into account which is to say respect that segmentation instead of assuming a direct unobstructed path to all sites’ control systems, also we have formal frameworks for industrial control system cyber security which exist because that boundary we see different failure modes then what is present in a typical enterprise network boundary. On the model-specific side, treating a deployed model as tamper-evident; verifying the artifact actually running at a site matches what the registry says was approved; is one of the newer risk categories ML-specific security guidance has begun to formalize explicitly.
Operational Excellence
Published operational-excellence guidance for ML workloads converges on a common point: model quality is not a one-time property established at training time, but something that has to be watched continuously against live data because input data itself keeps evolving after deployment. In this pipeline, three categories of signal matter most once a model is live at a site: is its live output behaving the way validation predicted, is the deployment mechanism itself healthy (did the last promotion complete, is a site’s running version what the registry says it should be), and is the underlying process data still arriving in the shape the model expects. The first is what most directly protects against a class of failure the validation gate cannot fully prevent: a model can pass every check against historical holdout data and still drift out of alignment with a live process over time, since the relationship between sensor inputs and correct process behavior is not guaranteed to stay static; a well-documented phenomenon in deployed ML systems generally. Monitoring compares live behavior against the expectations validation established, not just against a fixed threshold, so a slow divergence is visible before it becomes a large one.
Rollback is treated as a first-class operational action, not an emergency improvisation. Because every deployment references a specific, versioned registry entry, rolling a site back means re-pointing its deployment reference at the prior last-known-good entry and re-running the same deployment step against it; functionally similar to reverting a container orchestrator to a prior rollout revision; instead of inventing a separate manual recovery procedure under pressure. A model rollback is logged and reviewable the same way a forward promotion is, consistent with the change-management expectations a plant already runs under for anything touching a live process.
Trade-offs and Limitations
None of this is for free, and the issues at hand must be put out there in the open. We have a validation gate which has each candidate go up against every site it targets which in turn creates delay between “a model looks good in training” and “it is live somewhere. That delay is intentional; it is the point of the design. Still we have teams which are under pressure to ship a fix fast which may see that as friction. Staged rollout plays the same role. By design, it is slower than deploying everywhere at once, while a genuinely urgent fix is better handled through an expedited rollback path than pushed through the same multi-day pilot-then-fleet sequence intended for routine releases.
Containerized, registry-backed deployment also assumes a baseline of on-site compute and connectivity that not every plant necessarily has. A site with meaningfully constrained hardware or an unreliable connection to the central platform needs either a lighter-weight deployment mechanism suited to intermittent connectivity, or an explicit exception process; forcing it through the same pipeline built for better-connected sites treats every site as interchangeable, which is itself a source of the kind of site-specific failure the rest of this design tries to guard against.
Finally, this pipeline concentrates real authority, the ability to change what a live industrial process does, into fewer automated decision points than the old manual process had. That is the point; fewer, better-tested decision points beat many inconsistent manual ones; but it also means a defect in the validation logic itself has a wider blast radius than one engineer’s manual judgment call would have had, so that logic is held to the same review and testing discipline as the models it gates, not treated as trusted infrastructure exempt from scrutiny.
Production Lessons Learned
The most instructive lesson from operating this pipeline did not come from the pipeline mechanics themselves; it came from a gap in what the validation gate was actually checking. Early in the process, site-aware validation checked a candidate’s predictions against each site’s historical holdout data. The per-site configuration overlay; including calibration offsets and operating-range definitions; was applied only at deployment, after validation, so validation did not exercise it. A candidate which passed all validation tests across each site’s hold out data with the platform’s reference configuration was put into production and we see at only one site that the live results differ from what we saw in validation which we trace back to that site’s calibration which had shifted input distribution in a way that the reference configuration validation didn’t see. Nothing was broken in the sense of a crash or an error; the gate had simply been asked, and correctly answered, a slightly different question than the one that actually mattered at deployment time.
Validation is now of a type that each candidate is put through each site’s actual config which we aren’t using a shared reference config out of which the version of the validation script we put forward in the Implementation section comes. The lesson generalized past this one incident: any input that can differ by site has to be part of what gets validated per site, not applied as a deployment-time detail after the part of the pipeline meant to catch problems. A validation gate is only as good as how real world relevant what it puts the candidate through is and that is something which has to be proactively maintained as the system’s config space grows.
A second, smaller lesson concerned the pilot-site observation window. A window size for how quickly a stateless service’s error rate reveals a problem was initially too short for a process-optimization model, whose effect on a quality or emissions metric can take longer to become statistically distinguishable from normal process variation than a latency spike takes to show up on a dashboard. The window was lengthened, and the monitoring comparison was changed from a fixed threshold to a comparison against the range of variation validation had already characterized for that site, so a canary period is judged against what “normal” actually looks like for that process rather than a generic bar borrowed from a different kind of system.
Best Practices
Validate every candidate against every site’s own configuration and holdout data, not a shared reference configuration; a passing result against the wrong configuration is a false negative for the exact failure mode that matters most, as the incident above shows.
- Keep the pilot site the same across successive model versions rather than rotating it. A pilot site which is the baseline range for what is normal process variation only becomes a trusted benchmark for the next canary once we have that data from several past rollouts.
- we should put the full validation report with the registry entry it produced, for both pass and fail results, not just a pass or fail flag; also for a rejected candidate we should note down the specific failure reason in such a way that months down the road when someone looks at why a similar model was turned down we can still answer that.
- Give for emergency rollbacks their own expedited track which leaves out the multi-day staged promotion sequence instead of trying to rush through routine promotion pipelines; a fast track which at the same time only includes pre approved registry entries maintains speed and safety in separate domains.
- Treat the validation logic itself as a reviewed, tested artifact with its own change history, since a silent gap in what it checks, not a bad individual model, is the failure mode most likely to reach a live process undetected.
- Size of monitored thresholds and canary windows based on what has been measured for that particular site and process of variation validation, not a generic default from unrelated stateless service tooling.
Future Outlook
The most likely near-term change for a pipeline like this is to narrow the distance between what the validation gate checks and what happens in a live process. Instead of just at deployment which is after we have historical holdout data we will have live performance constantly compared to what was established in the validation phase. If the two begin to diverge, it could prompt a proactive review or recommend rolling back; rather than relying on someone to spot the problem. Also we see a trend towards a standardized deployment at the site level. As we mature in our use of lightweight Kubernetes compatible solutions for edge and intermittent environments [18] this type of pipeline will less rely on site specific scripts and more so on a portable target which is the same across the board no matter the connection reliability. That in turn will address the site to site variability which was the original issue we were trying to solve.
Conclusion
Treating CI/CD for machine-learning models deployed across physically distributed industrial plants as a simple variation on conventional cloud CI/CD is where naive automation usually fails. A live industrial process will not tolerate the same mistakes as a stateless web service. The difference came from applying three decisions consistently, not from choosing any one tool: a model could not move without the site configuration it required; validation results could not be generalized to sites where the model had not actually been tested; and a rollback had to be a precise, versioned pointer to a model already known to be safe. None of this eliminated risk. It made risk visible, tested, and recoverable before a bad model ever reached a live process; a narrower, and more achievable, goal than eliminating risk altogether.
Leave a comment