Container orchestration for machine learning means running training and inference as scheduled cluster workloads rather than as jobs someone starts by hand. OpenShift is Red Hat’s commercial distribution of Kubernetes, and the question worth answering is narrow: what does that platform layer actually have to provide before a model behaves like production software?
Quick Overview
This article stays at the platform layer. It is about what a Kubernetes or OpenShift cluster must be configured to do — how it schedules accelerators, isolates tenants, mounts data, serves an endpoint and reports what it is doing — so that machine learning stops being an exception to the way everything else is operated.
Three boundaries are worth stating before anything else, because they decide whether this text is the one you want.
The model lifecycle — experiment tracking, promotion criteria, the split of ownership between data science and engineering, the discipline of getting a notebook to behave like a service — sits above the platform and is a separate subject with a separate answer. It is treated in MLOps: taking machine learning models to production, and repeating it here would produce two half-articles instead of one useful one.
The general case of container orchestration — what Kubernetes and Docker do for ordinary cloud applications, how deployments and services relate, why the pair took over application delivery — is also covered elsewhere and is not restated. What follows assumes you know that layer and asks only what changes when the workload is a training run holding a GPU overnight, or an inference endpoint whose latency budget is measured in milliseconds.
The choice of vendor is the third boundary. This is not a buying guide. OpenShift is used here as a concrete, widely deployed instance of the platform layer, and where a mechanism exists identically in upstream Kubernetes, that is said plainly rather than presented as a product advantage.
What is left after those three subtractions is the part nobody writes down: the configuration work that stands between a cluster that runs web services well and a cluster that runs machine learning without surprising anyone.
What OpenShift Adds on Top of Kubernetes
OpenShift is a Kubernetes distribution with an opinionated set of components assembled and supported as one product. For machine learning teams the practical additions are an integrated image registry, a configuration management layer, autoscaling wired in by default, and a monitoring stack that is present without anyone having to choose one.
The mechanism that carries most of the ML-specific behaviour is the operator: a controller that extends the Kubernetes API with custom resources and reconciles them the way the built-in controllers reconcile Deployments. OpenShift AI, previously named OpenShift Data Science, ships as an operator and brings prepared environments for common frameworks — TensorFlow, PyTorch and the notebook images around them — as declarative objects rather than as installation instructions.
That is the genuine value: the environment becomes something you declare and version rather than something you build. It is also where the cost sits. The additions carry licence fees and a materially higher resource floor than a plain Kubernetes cluster, and the operators themselves consume capacity that is then unavailable to the workloads they exist to support. Teams that skip that arithmetic discover it during the first capacity crunch.
Where OpenShift and Plain Kubernetes Actually Diverge
Most published comparisons list features present in both and attribute them to one. The differences that survive scrutiny are narrower than the marketing suggests.
Built-in delivery tooling. OpenShift ships continuous integration and delivery components — Tekton-based pipelines, Argo CD-based GitOps — as supported parts of the platform. Upstream Kubernetes leaves the choice open, which means more setup and more freedom to use whatever the organisation already runs.
Access control. Role-based access control exists in both. What OpenShift adds is the integration work: identity providers wired in, and Security Context Constraints as an additional admission layer governing what a pod is permitted to do. For teams handling regulated training data, the integration is the product.
Projects instead of namespaces. OpenShift wraps namespaces in a “project” abstraction with its own lifecycle and default policies. Whether that simplifies tenancy or adds a layer depends on how many teams share the cluster.
Accelerator support. GPU scheduling works in both, through operators that expose devices to the scheduler as extended resources. OpenShift ships a supported path to it; upstream Kubernetes reaches the same place with the vendor’s own operator.
A catalogue of prepared components. OperatorHub makes installing an ML platform component a declarative act. Helm charts do a comparable job on any cluster, without the catalogue.
| Aspect | OpenShift | Upstream Kubernetes | What it means for ML |
|---|---|---|---|
| Licensing | Commercial subscription | Open source | Higher entry cost, support included |
| Delivery tooling | Shipped and supported | Assembled per organisation | Faster start, less freedom of choice |
| Identity and access | Integrated with enterprise directories | RBAC present, integration is yours | Matters when training data is regulated |
| Resource floor | Higher | Lower | Platform overhead competes with workloads |
| Support model | Vendor support contract | Community and internal expertise | Decisive for regulated production |
There are also alternatives that remove the platform question entirely rather than answering it. Managed services — SageMaker, Vertex AI, Azure Machine Learning — hand over cluster operation in exchange for portability and cost control. Kubeflow on plain Kubernetes keeps the control and drops the subscription, at the price of assembling and maintaining the pieces yourself. None of these options is wrong; they differ in who is on call when the scheduler misbehaves.
Preparing a Cluster to Take ML Work
A cluster that runs web services competently is not yet a cluster that runs machine learning. The preparation is a sequence, and skipping a step usually shows up as an unexplained queue rather than as an error.
Install the platform operators. OpenShift AI, Open Data Hub or their upstream equivalents establish the custom resources everything else refers to. Each operator adds reconciliation load and consumes capacity, so the install list should be short and deliberate.
Expose the accelerators. GPUs reach the scheduler through a device plugin and a driver operator. The failure mode here is version alignment: driver, operator and platform release form a compatibility triangle, and the newest of anything is frequently the one that does not fit. Planning for a lag between a hardware generation shipping and the cluster being able to schedule it is realism, not pessimism.
Choose storage per access pattern, not per convenience. Training reads large files sequentially; feature stores and metadata want low-latency random access; artefacts want durability and cheap capacity. A single storage class chosen for the whole platform will be wrong for at least one of those, and the symptom will be presented to you as “the GPUs are idle”.
Size the network for the data path. Distributed training moves gradients between nodes and moves datasets from storage to compute. In clusters where nobody measured the data path, the bottleneck is almost always there and almost never where the first investigation looks.
Set requests and limits before the first workload, not after. Initial estimates are always wrong; the point is to have a number to correct rather than an unbounded pod to evict.
Wire delivery and the registry. Pipelines need somewhere to push images and somewhere to record what they built. The integrated registry is convenient and has capacity characteristics worth checking before it holds every model image the organisation has ever produced.
Doing this well is architecture work rather than administration, and it is the part of the job that transfers between platforms — which is why it is the core of a course like Docker, Kubernetes and OpenShift for Developers rather than a vendor-specific checklist.
The Platform Components That Matter for Machine Learning
OpenShift ships a great many components. A short list of them changes how ML workloads behave; the rest are ordinary platform furniture.
OpenShift AI provides prepared data science environments — notebook images, framework builds, the wiring between them. It removes environment assembly, not infrastructure ownership; the cluster is still yours to run.
Pipelines, built on Tekton, expresses a workflow as Kubernetes objects. Every step is a pod, which means training steps inherit the scheduling, quota and isolation behaviour of everything else in the cluster instead of living in a parallel universe.
Serverless, built on Knative, gives request-driven scaling including scale to zero. For an endpoint whose traffic is bursty and mostly absent, this is the difference between paying for capacity and paying for use — at the price of cold-start latency on the first request after idle.
Service Mesh, built on Istio, gives traffic control and transport encryption between components. It earns its complexity when an ML system is several services — preprocessing, prediction, explanation — rather than one.
GitOps, built on Argo CD, makes the cluster’s desired state a reviewed artefact in version control. For models this matters more than for ordinary services, because “which version was serving when that decision was made” is a question somebody will eventually ask under audit.
Data Foundation provides storage as a cluster service. Rook, Ceph, Longhorn and cloud-native volumes solve the same problem with different operational profiles.
The monitoring stack, built on Prometheus and Grafana, is present by default rather than assembled.
Every one of these exists as an open-source project runnable on any conformant cluster. What the distribution sells is assembly, joint versioning and a single support contract. The general mechanics of how the container platform underneath them schedules and heals workloads are covered in Kubernetes and Docker: Automating Cloud Application Management, which describes the same substrate for ordinary applications; the sections below assume it and address only what machine learning adds.
Running Kubeflow Pipelines on OpenShift
Kubeflow is the open-source toolkit for expressing machine learning workflows as Kubernetes-native objects. Running it on OpenShift is a common combination and a genuinely fiddly one.
The install path is the first decision. The catalogue offers an operator, which is the low-effort route; more complex deployments frequently end up installing components directly, because the operator’s opinions and the cluster’s constraints do not always agree. Neither is wrong, but choosing the convenient route and then fighting it for a quarter is a recognisable pattern.
Authentication is where most of the integration effort actually goes. Kubeflow carries its own notion of user identity, the platform carries another, and making one defer to the other so that a single sign-on grants coherent permissions in both is configuration work that no tutorial completes for you. Getting it wrong is not a cosmetic problem: it is how a notebook ends up able to read a dataset its owner was never granted.
Storage class selection decides pipeline responsiveness. Metadata and artefacts are accessed constantly and in small pieces, and putting them on capacity-optimised storage produces a pipeline that spends its time waiting rather than computing.
Accelerator access needs profiles that describe what a user may claim. Where a single physical GPU is shared, partitioning technologies such as Multi-Instance GPU give hard isolation between tenants rather than the polite fiction of everyone agreeing to be reasonable.
The integration between Kubeflow’s own pipeline runner and the platform’s delivery tooling is the last mile and the most common place to stall, usually on version compatibility between components that release on independent schedules.
A pair of failure modes deserve naming, because both look like platform faults and neither is. The Kubeflow interface degrades as pipeline history accumulates, which is a query and caching problem rather than a cluster problem. And the metadata components will consume whatever memory they are permitted to, so leaving their limits unset is how a control-plane component evicts a training job.
Lighter alternatives exist and are underused. MLflow handles tracking and registry duties with far less machinery. Tekton with purpose-built ML tasks stays inside the platform’s own primitives. Argo Workflows does orchestration alone and does it well. The right answer depends on how much of Kubeflow the team will genuinely use, and the honest answer for many teams is: less than they installed.
Scheduling Compute: Requests, Limits and Autoscaling
Training is the most resource-hungry thing most organisations run, and the scheduler can only be as good as the declarations it is given.
Requests and limits are the interface to the scheduler. A request is what the pod is guaranteed and what placement decisions are made against; a limit is where it gets throttled or killed. Workloads that declare neither are scheduled on hope and evicted on arithmetic.
Horizontal autoscaling adds and removes replicas in response to load, which is the right shape for inference. Vertical autoscaling adjusts the resources of individual pods based on observed use, which is the right shape for long-running training jobs whose appetite nobody estimated correctly. The Kubernetes documentation page Autoscaling Workloads sets out both mechanisms and the boundary between them, and the boundary is worth respecting: applying them to the same workload simultaneously produces two controllers arguing.
Placement rules put work on the right hardware. Node selectors, affinity rules and taints with tolerations are how GPU nodes stop hosting web front-ends that wandered onto them. The reference behaviour is documented under Assigning Pods to Nodes, and getting it wrong is expensive in a specific way: the accelerator sits idle while something that does not need it holds the node.
Quotas bound what a tenant can consume. The Resource Quotas mechanism caps aggregate consumption per namespace, which is the only reliable way to stop one team’s hyperparameter sweep from starving everyone else’s inference.
Priority classes decide who wins under contention. Pod Priority and Preemption defines how a higher-priority pod displaces a lower-priority one. Production inference should outrank exploratory training, and if that ordering is not encoded, it will be discovered during an incident.
None of this is set once. Utilisation metrics are the input to a periodic correction, and clusters where nobody performs that correction drift into a state where everything is over-requested, the scheduler refuses work, and the nodes look empty. Reasoning about this behaviour under real contention is exactly what Advanced Docker and Kubernetes exists to teach.
Serving a Model as an API
Training produces an artefact. Serving turns it into something the rest of the business can call, and there are several defensible ways to do it.
Purpose-built inference servers — TensorFlow Serving, TorchServe, ONNX Runtime — take a model in a known format and expose it efficiently, with batching and hardware acceleration handled for you. They are deployed as ordinary Deployments with accelerator access, and they constrain you to the formats they understand.
A web framework wrapping the model — Flask, FastAPI, Django — gives complete control of the request path, which matters when preprocessing is idiosyncratic or when the endpoint has to speak an existing internal protocol. The cost is that you now maintain a service, including the parts the inference servers would have maintained for you.
A serving controller such as KServe makes the endpoint a declarative resource, with versioned rollout, canary traffic splitting and autoscaling described rather than scripted. It is the strongest option for teams already invested in the Kubeflow ecosystem and the heaviest for teams who are not.
A serverless endpoint on Knative scales with demand and down to nothing. The Knative Serving documentation describes the request-driven model directly; the trade-off is the cold start, which is invisible for a batch consumer and unacceptable for an interactive one.
A decomposed pipeline splits preprocessing, prediction and explanation into separate services connected through the mesh. It scales each part independently and costs you a distributed system to debug.
| Approach | Strength | Weakness | Fits best |
|---|---|---|---|
| Inference server | Optimised, format-aware | Constrained to supported formats | Standard framework outputs |
| Web framework | Total control of the request path | You maintain everything | Unusual preprocessing, legacy integration |
| Serving controller | Declarative versions and rollout | Heavy dependency footprint | Established Kubeflow platforms |
| Serverless | Pays only for traffic | Cold-start latency | Bursty or intermittent demand |
| Decomposed services | Independent scaling | Distributed-system complexity | Multi-model systems |
Whichever is chosen, the same four things decide whether it survives contact with production traffic: declared resources that match observed use, a scaling policy that reacts before the queue does, persistence for model artefacts and configuration, and monitoring that reports prediction quality alongside latency. An endpoint that is fast and wrong is not a working endpoint.
Open Data Hub as an Assembled Platform
Open Data Hub is the upstream project that assembles storage, notebooks, pipelines and serving components into something resembling a platform. It is best understood as a curated assembly rather than as a product, and the distinction predicts most of the experience of using it.
The assembly gives a coherent entry point: notebook environments from prepared images, so that a data scientist’s first day does not begin with dependency archaeology; managed access to training data; pipeline orchestration inherited from Kubeflow; and serving integration through Seldon Core or KServe. Standardising notebook images alone removes a category of “works on my machine” that consumes a surprising amount of senior time.
The costs are equally consistent across deployments. Navigation between components is not uniform, because they are separate projects with separate interfaces wearing a shared badge. Version compatibility between those components is a recurring maintenance obligation rather than a one-off install task. And the serving path that behaves well for moderate models needs deliberate attention once model artefacts become large enough that pulling an image is itself a scheduling event.
| Platform | Advantage over Open Data Hub | Cost relative to it | Typical fit |
|---|---|---|---|
| Managed cloud ML service | No infrastructure to operate | Higher spend, vendor coupling | Teams needing speed over control |
| Cloud vendor’s ML suite | Deep integration with that vendor’s stack | Portability | Organisations already committed |
| Kubeflow alone | Smaller resource footprint | More assembly work | Teams fluent in Kubernetes |
| Spark-centric platforms | Strong large-dataset processing | Cost, less flexibility | Data-engineering-heavy workloads |
Open Data Hub shortens the distance to a working platform, not to one that fits a specific organisation. Treating installation as completion is the most reliable way to be disappointed by it.
Continuous Delivery for Containerised Models
Delivery for machine learning is ordinary delivery plus two things that ordinary software does not have: the artefact is derived from data, and it degrades without anyone changing it.
Version everything the result depends on. Code, pipeline definitions, configuration and experiment metadata belong in version control. Model binaries and datasets belong in tools built for them — a registry, or a data versioning layer — because a Git repository holding gigabyte artefacts stops being a Git repository in any useful sense.
Test at the levels that can actually fail. Unit tests for preprocessing and postprocessing logic. Integration tests for the pipeline end to end. Load tests for the serving endpoint. Quality tests for the model against a held-out set. Drift tests for the input distribution. That is a genuine hierarchy: the first three are software testing, and the last two are the ones without which a green pipeline means nothing.
Automate the path from training to serving. A change in code or data triggers training; the resulting model is registered; an image is built and tested; deployment follows a strategy that limits exposure — blue-green, or canary with a small traffic share first.
Keep a human gate where the consequence is external. Promotion to production, automatic rollback on degradation, and an audit trail from training run to serving version are governance requirements, not ceremony. When a decision affecting a customer is challenged, the ability to answer “this model, trained on this data, served from this date” is the answer.
Close the loop. Production performance feeds monitoring; monitoring detects drift; drift triggers retraining. Without the loop, continuous delivery for a model delivers the first version continuously and every subsequent problem manually.
Monitoring the Runtime, Not Only the Model
An ML platform has two monitoring problems that are usually conflated: is the infrastructure healthy, and is the model still right? Both need answering, and they need different instruments.
The platform stack gives infrastructure metrics — processor, memory, network and storage per pod and per node — with alerting and enough retention to see a trend rather than a moment. That is the baseline every workload gets.
Application-level monitoring is where ML diverges. The metrics that matter are inference latency, throughput, queue depth and prediction quality, and they only exist if the serving code is instrumented to emit them. Nothing in the platform knows what a good prediction looks like.
Accelerator monitoring is its own discipline: utilisation, memory occupancy and thermal state per device, plus the transfer between host and device that is frequently the real bottleneck in a job that looks compute-bound.
Distributed tracing earns its keep once the system is more than one service, because latency in a decomposed inference path is a property of the path rather than of any component in it. Centralised, structured logging correlated with those traces is what turns an incident from an archaeology exercise into a query.
The last layer is dashboards built for the people who own the models rather than for the people who own the cluster: pipeline status, drift indicators, resource consumption attributed per project. Metrics nobody assembled into a view are metrics nobody reads.
Versioning Models and the Images That Carry Them
A model in production has two identities: the artefact and the container image that serves it. Version them separately and consistently, or spend incident time establishing which was which.
For models, a registry records the artefact together with training parameters, evaluation metrics and the dataset version it came from. A naming scheme that encodes framework, task and lineage beats a timestamp nobody can decode. Lifecycle tags — candidate, production, baseline — let automation refer to a role rather than to a specific version.
For images, semantic versioning with immutable tags is the discipline that makes rollback a decision rather than a rebuild. A moving tag pointing at whatever was built most recently guarantees that at some point nobody will know what is running. Vulnerability scanning on push closes the gap between a base image being fine when it was chosen and being fine today.
For rollout, blue-green keeps two versions live and switches traffic; canary shifts traffic gradually; A/B splits it deliberately to compare outcomes; shadow deployment sends real traffic to a new version whose answers nobody acts on. Each answers a different question, and choosing between them is a question about risk appetite rather than about tooling.
For dependencies, the record that matters is which model version was trained on which data version, which serving API contract it satisfies, and which libraries were baked into the image. That triple is what makes a result reproducible.
For retention, an explicit policy beats infinite accumulation. Keeping every artefact ever produced is a storage bill; keeping the ones that were ever served, along with what produced them, is an audit position.
Where the Infrastructure Bill Actually Comes From
Machine learning spend is dominated by accelerators that are idle and storage that is warm when it should be cold. Almost everything else is rounding.
Right-size before optimising. Vertical autoscaling recommendations against observed use turn guessed requests into measured ones, and the correction is usually downward by more than anyone expects. Quality-of-service classes then decide who is evicted first when the arithmetic gets tight.
Place work deliberately. Affinity and anti-affinity rules control spread; taints reserve expensive nodes for the workloads that justify them; a descheduler corrects placement that was sensible when it was made and is no longer.
Use interruptible capacity for interruptible work. Training that checkpoints can run on preemptible instances at a fraction of the price. Training that does not checkpoint cannot, and adding checkpointing is usually the cheaper project.
Share accelerators. Partitioning and time-slicing let several small jobs occupy one device that a single job would have held while using a fraction of it. Queueing rather than allocating is what keeps the device busy.
Turn things off. Development and test environments that hibernate outside working hours, clusters that scale down at night, and time-to-live policies on temporary resources remove a class of spend that nobody defends once it is visible.
Tier the storage. Hot, warm and cold tiers with automatic migration, columnar formats for datasets that are read far more often than written, and compression on anything archival. Storage grows quietly, which is why it is usually the second-largest surprise.
Attribute the cost. Resource tags that map spend to a team or project convert an infrastructure bill into a set of decisions someone owns. Alerts on thresholds make the conversation happen before the invoice does.
Securing Data, Models and the Cluster Around Them
Security for an ML platform covers three assets that fail differently: the training data, the model, and the infrastructure serving both.
Data. Encryption at rest through the storage layer, encryption in transit through automatic transport security, and access governed by role-based rules plus the admission constraints that decide what a pod may do at all. Training data is frequently the most sensitive material an organisation holds, and it is routinely the least protected because it lives in the analytics estate rather than in the production one.
Models. A trained model is intellectual property and an attack surface at once. Restricting who can retrieve it, signing artefacts so that tampering is detectable, validating inputs to blunt adversarial probing, and monitoring usage patterns for extraction attempts are the controls that apply. A public endpoint with no rate limiting is a model being given away one query at a time.
Containers. Scan images for known vulnerabilities, keep base images minimal so there is less to scan, run without privileges unless something genuinely requires them, and enforce signature policies at admission. The last one is what makes the previous three binding rather than advisory.
The network. The Network Policies mechanism defines which pods may talk to which, and the default of unrestricted internal traffic is the one setting most worth changing on a cluster that hosts multiple tenants. A mesh adds encryption and finer control between services; segmentation isolates the components whose compromise would be worst.
Identity and audit. Integration with the organisation’s directory, stronger authentication for privileged access, granular authorisation, automatic rotation of certificates and secrets, and a log of every operation on data and models. Compliance regimes ask for provenance, and provenance is a logging decision made long before anyone asks.
Privacy. Where the data is personal, the platform can support anonymisation and pseudonymisation before training, techniques that bound what a model reveals about individual records, and federated approaches that keep raw data where it was collected. These are design choices with accuracy costs, and pretending otherwise is how they get abandoned mid-project.
Dividing the Work Between ML and Platform Teams
Most ML platform failures are not technical. They are two groups optimising for different things with no agreement about where one stops.
The workable division puts the platform team in charge of the substrate — cluster, scheduling, storage classes, delivery machinery, monitoring infrastructure — and the ML team in charge of what runs on it, including the quality of what it produces. The interface between them is a set of declarative resources rather than a queue of requests, which is the whole point of expressing the platform as an API.
Shared infrastructure only works with shared artefacts: common repositories, one delivery system, one monitoring stack, and reusable images so that every project does not rediscover how to build a container with a framework in it. Where those are shared, “it works in the pipeline” means the same thing to both groups.
The GitOps posture is what makes the boundary reviewable. Infrastructure and configuration expressed as code in a repository means changes go through review, and disagreements happen in a pull request instead of in an incident. The same is true of the model side once serving definitions live beside everything else.
The organisational shapes that work are unremarkable: cross-functional teams with both capabilities, rotation between them so each understands the other’s constraints, and incentives that reward the platform team for what teams ship rather than for cluster uptime alone. A self-service internal platform is the mature form of this, and it is a product with users rather than an infrastructure project with tickets.
Debugging Performance in a Containerised ML System
Performance problems in containerised ML have an unhelpful property: the symptom is almost never located where the cause is. A systematic order saves more time than any individual tool.
Start with system metrics. Processor, memory, storage and network per pod and per node, correlated with the load that triggered the problem. This eliminates the largest category of causes quickly and identifies which resource is actually scarce.
Profile the workload itself. Framework profilers show where time goes inside training or inference, and the answer is frequently data loading rather than computation. Accelerator utilisation alongside host-to-device transfer separates a job that is compute-bound from one that is starved.
Trace the request path. In a decomposed system, latency is distributed across hops, and only tracing shows which hop owns it.
Read the logs as a correlated series. Centralised, structured logs aligned with metrics and traces turn a guess into a search. Logs collected but not correlated are an archive, not an instrument.
Check the network explicitly. Latency and throughput between components, network policy rules that silently drop traffic, name resolution, and load-balancing behaviour. Policy misconfiguration presents as intermittent application failure and is diagnosed as anything but.
Then look at the ML-specific causes. Input drift, batch size interacting badly with memory, and inefficient data loading pipelines account for most problems that survive the earlier steps.
In production, isolate rather than experiment. Shadow deployments test a change against real traffic without exposing anyone to it, feature flags allow selective disablement, and circuit breakers stop a failing component from taking its callers with it.
Handling Training Data at Cluster Scale
Data management determines how fast the platform can work, and it is the part most often designed after everything else has been built around it.
Match storage architecture to data shape. Structured, unstructured and streaming data have different access patterns; object storage, distributed file systems and platform-native volumes solve different problems. Tiering across hot, warm and cold is what keeps the cost of retaining everything from growing without limit.
Record metadata as a first-class asset. Lineage — which dataset produced which features, which features fed which model — is what makes a result explainable months later. Cataloguing and tagging make datasets findable, and findability is the difference between reuse and reproduction.
Version data as deliberately as code. A data versioning tool, a record of what changed between versions, and deterministic processing pipelines together make reproducibility achievable rather than aspirational.
Process where the data is. Distributed processing frameworks, caching for repeated operations, materialised views for common aggregations, and columnar formats for read-heavy access. The recurring lesson is that moving data is more expensive than processing it.
Validate continuously. Automated schema and distribution checks, deduplication, and drift monitoring catch the failures that silently degrade a model without breaking anything.
Distribute with locality in mind. Scheduling compute near the data, sharding for parallel reads, and minimising the transfer volume are the levers that make distributed training scale at all.
Govern access and enable reuse together. Role-based access to datasets, automatic anonymisation of personal fields, and usage logging satisfy compliance. Internal catalogues, standard access interfaces and documented datasets are what stop three teams building the same feature pipeline independently.
Capacity Planning for Demand That Refuses to Be Steady
ML demand is heterogeneous and spiky in a way that ordinary services are not. Planning capacity for it means planning for several different workloads that happen to share a cluster.
Classify the workloads first. Training, inference, extraction and interactive exploration have different resource profiles and different timing patterns — batch against real-time, periodic against unpredictable. Sizing a cluster for their sum rather than for their peak concurrency is how money is wasted; sizing for their peak concurrency without knowing it is how queues form.
Build node pools around those classes. Separate pools for accelerator work, memory-heavy work and general compute, with instance types chosen per pool. Burst capacity into a second environment is worth designing before it is needed.
Autoscale at every level that can move. Cluster autoscaling adjusts node count; horizontal autoscaling adjusts replicas; vertical autoscaling adjusts individual pod resources. They operate at different timescales and each covers a gap the others cannot.
Encode the priorities. Priority classes, topology spread constraints and quotas together determine what happens under contention. Without them, contention is resolved arbitrarily, and arbitrary resolution during a production incident is indistinguishable from an outage.
Plan proactively where the pattern is known. Historical usage supports predictive scaling for recurring peaks, and periodic right-sizing of existing deployments recovers capacity that accumulated as slack.
Use the specialised hardware fully. Sharing and time-slicing accelerators, and matching job size to device partition, is what raises utilisation on the most expensive resource in the cluster.
Design explicitly for the idle periods. Scale-to-zero serving, job queues for batch work, and interruptible capacity for anything that tolerates restart are what make an uneven demand curve affordable rather than merely survivable.
What Deployments Report — and Where the Reports Stop Being Usable
Published accounts of ML platform adoption follow a recognisable pattern, and the pattern is more useful than any individual figure in it.
The reported benefits cluster in the same places regardless of sector: a shorter path from a finished model to a served endpoint, because deployment stops being bespoke; better behaviour under load, because scaling is a platform property rather than a project decision; and stronger audit positions, because the platform records what it did. Organisations under regulatory supervision consistently report the last of these as the benefit that justified the work, which is not the benefit the technology is usually sold on.
The reported difficulties cluster equally consistently. Integration with existing systems takes longer than planned, because the existing systems were not designed to be integrated with. Performance at genuine data scale requires optimisation work that the pilot did not reveal. Licensing is a real line in the budget rather than a rounding error. And the platform requires people who understand it, which is a hiring and training commitment that outlasts the project.
A word of caution about the genre. Vendor case studies and conference talks report improvements without a baseline, a measurement method or a control, and the numbers they carry cannot be verified by a reader. This article therefore reports the shape of what deployments describe and declines to repeat the figures attached to them. A percentage without a method behind it is not evidence, and reproducing one here would only launder it.
The pattern that does survive scrutiny is a hybrid posture: experimentation on whatever is cheapest and most flexible, production on the platform that carries the compliance and support obligations, and some workloads deliberately left on plain Kubernetes because they do not need what the distribution adds. That posture is a decision about where each cost belongs, and it is available to any organisation willing to run more than one thing.
Operational Failure Modes and How Teams Contain Them
The problems that recur in production ML are a short list, and each has a containment strategy that is well understood and frequently not implemented.
Models degrade without anyone touching them. Input distributions move, and the model that was accurate at launch quietly stops being. Containment is drift monitoring on input features, alerting on prediction-quality metrics rather than on infrastructure health alone, and gradual rollout so that a new version proves itself on a slice of traffic before it owns all of it.
Dependencies drift apart. Library versions that agreed in development disagree in production. Containment is self-contained images with pinned dependencies, a small set of maintained base images rather than one per team, and automated scanning that flags an image whose dependencies have become a liability.
Load arrives in the wrong shape. Containment is scaling on metrics that lead demand rather than lag it — queue depth or upstream business signals rather than processor utilisation alone — and load testing that exercises the serving path at the volume it will actually see.
Failures are hard to reproduce. Containment is centralised logging with distributed tracing, shadow deployment for testing against live traffic without consequence, and an incident record that turns each outage into something the next person can search.
Costs grow silently. Containment is periodic right-sizing against observed use, interruptible capacity for deferrable work, and automatic shutdown of environments outside working hours.
None of these is exotic. They recur because each is cheap to implement before it is needed and expensive to retrofit afterwards, and the moment of realisation is reliably the moment the team has no capacity to act. Teams that treat container images, scheduling and delivery as engineering disciplines in their own right — the ground covered by Advanced Docker Techniques — tend to arrive at these controls before the incident that would have taught them.
Where Container-Based ML Is Heading
The direction of travel is reasonably clear, and most of it is consolidation rather than novelty.
Platform practices are becoming standard rather than distinctive. The automation, reproducibility and auditability that were differentiators are becoming baseline expectations, increasingly merged with security practice rather than sitting beside it, and increasingly delivered as an internal platform rather than as a per-team assembly.
Inference is moving toward the data. Running models at the edge, on constrained hardware near where data is produced, with training centralised and serving distributed, is a hybrid architecture that container tooling is steadily getting better at expressing.
Accelerators are diversifying. Beyond GPUs sit purpose-built inference chips and reconfigurable hardware, and the platform’s job is to abstract access to them so that a workload declares what it needs rather than which device it wants.
Training is decentralising where data cannot move. Federated approaches keep data in place and move the model instead, which is a containerisation problem as much as a machine learning one, and a privacy posture that regulation is likely to reward.
Automation is moving up the stack. Automated architecture and hyperparameter search turns experimentation into a scheduling problem, which is precisely what an orchestrator is for.
The infrastructure is starting to learn. Predictive scaling and anomaly detection applied to the cluster itself, rather than to what it runs, is a modest but real trend.
Models are being treated as code. Versioned, tested and deployed on the same terms as applications, with tooling converging on that assumption.
Access is broadening. Low-code interfaces for building and deploying models widen participation, and they raise the governance question of who is accountable for what those users deploy.
Transparency is becoming a platform feature. Support for explainability, bias testing and automatically generated model documentation is moving from research tooling into platform tooling, driven less by preference than by regulation.
Tracking these is not the same as adopting them. The organisations that benefit are the ones that experiment deliberately, on a platform that makes experiments cheap to run and cheap to abandon.
When OpenShift Is the Right Substrate — and When It Is Not
OpenShift is a solid substrate for ML orchestration. It is not automatically the right one, and the argument that it is usually skips the comparison.
It fits organisations that need vendor support, have compliance obligations that make an audit trail load-bearing, run infrastructure they must keep control of, and have or intend to build the platform skills to operate it. Under those conditions the subscription buys something specific and the resource overhead is a fair price.
It fits poorly where the workload is small enough that a managed service is cheaper than an operating team, where the organisation has no platform capability and no plan to acquire one, or where the flexibility of assembling upstream components matters more than having them assembled.
The decision rests on things absent from every feature list: the infrastructure and skills already present, the security and compliance obligations that actually apply, the scale and budget of the operation, and the technology direction the organisation intends to hold.
Whatever the platform, the constant is the working relationship between the people who build models and the people who run the cluster, together with the discipline of measuring what production is doing rather than assuming it. Platforms change more often than that requirement does.
Frequently Asked Questions
Is OpenShift better than plain Kubernetes for machine learning?
It is better assembled, not more capable. OpenShift ships supported delivery tooling, integrated identity and a monitoring stack, which shortens the path to a working platform and matters most under compliance obligations. Upstream Kubernetes with the same open-source components reaches a comparable place with more setup and lower cost.
What does a cluster need before it can run ML workloads?
An accelerator path exposed to the scheduler, storage classes matched to the access patterns of training and serving, requests and limits declared on every workload, quotas per tenant, and a monitoring stack that reports accelerator use. Missing any one of them produces the same symptom: expensive hardware sitting idle while jobs queue.
How does Kubeflow relate to OpenShift?
Kubeflow is the orchestration layer for ML workflows; OpenShift is the platform it runs on. Kubeflow contributes pipeline definitions, experiment tracking and serving components, while the platform contributes scheduling, isolation, access control and delivery machinery. The integration work concentrates on making one identity system authoritative for both.
Should we use a managed cloud service instead of running our own cluster?
If the team has no platform capability and the workload does not carry data-residency or control requirements, a managed service removes work that is not the point of the project. If the organisation must control where data lives and what the infrastructure costs, running the cluster is the price of that control.