Skip to content
MLOps

What Is MLOps? One Model Taken From Notebook to Production

Ihsan Ullah · 17 min read
What Is MLOps? Labelled training data flows into a model, then through five MLOps stages: track experiments, quality gate, register @champion, serve with an API, monitor for drift

Quick answer: MLOps (machine learning operations) is the set of practices and tools that take a machine learning model out of a notebook and keep it working in production. It covers tracking experiments, testing models before release, versioning and registering them, deploying them behind an API, and monitoring them for data drift so you know when to retrain. Where DevOps manages code, MLOps also has to manage data and models, which change in ways ordinary software does not.

Ask “what is MLOps?” and most guides answer with the lifecycle stages and a long table of tools. This one takes a single model through those stages in Python. We will use the wine classifier from our supervised vs unsupervised learning guide: track two candidate models with MLflow, block any model below a quality bar, register the winner, serve it with FastAPI, and catch a data drift problem before the labels arrive. Every output shown below comes from running the code.

What is MLOps?

MLOps is to machine learning what DevOps is to software: a way of working that makes releasing and running models repeatable instead of a one-off manual effort. Google Cloud describes it as practices for managing the full machine learning lifecycle, from development to deployment and monitoring.

The need comes from a simple fact. A model is not finished when it scores well in a notebook. Once it is live, the world it predicts keeps changing, new data arrives, someone retrains it, and a different version ends up in production. Without MLOps, nobody can answer basic questions such as “which model is running right now?”, “what data was it trained on?” or “is it still accurate?”

The problem was described well in Google’s 2015 paper Hidden Technical Debt in Machine Learning Systems. Its best-known figure shows that in real-world ML systems, the model code is only a small part; most of the work is the surrounding data collection, configuration, serving and monitoring. MLOps is the discipline for that surrounding work.

MLOps vs DevOps

MLOps borrows a lot from DevOps (version control, automated testing, CI/CD, monitoring) but adds problems that only appear when the behavior of a system is learned from data.

DevOpsMLOps
What changesCodeCode, data and models
What can break silentlyLess often: bad code usually throws errorsOften: a model keeps returning answers while its accuracy falls
TestingUnit and integration tests on codeCode tests plus data validation and model quality checks
VersioningSource codeSource code, datasets, model files and their metrics
Release cycleBuild and deploy the appRetrain, evaluate, register and deploy the model, often on a schedule
MonitoringUptime, errors, latencyAll of those, plus input drift and prediction quality

Google’s architecture guide adds a third practice to CI and CD: continuous training (CT), meaning models are retrained and redeployed automatically when new data or a drop in quality calls for it.

The MLOps lifecycle

A model in production goes round a loop rather than a straight line:

  1. Prepare data: collect, clean and validate the training data.
  2. Train and track: run experiments and record the code, parameters, data and metrics of each one.
  3. Validate: test the best candidate against a quality bar before anyone can ship it.
  4. Register: store the approved model with a version number, so you always know which model is which.
  5. Deploy: serve the model, usually behind an API or inside a batch job.
  6. Monitor: watch the inputs and predictions for drift and errors, and feed problems back to step 1.
The MLOps lifecycle in six steps: prepare data, train and track, validate with a quality gate, register a version as champion, deploy behind an API, and monitor for drift before retraining
The MLOps lifecycle is a loop: monitoring feeds back into retraining.

The worked example below follows steps 2 to 6. Step 1 is the subject of data engineering; our ETL examples show what that work looks like in practice.

One model from notebook to production, step by step

You need Python 3.11 or later and three packages (the standard extra adds the FastAPI command-line tool, the Uvicorn server and HTTPX for testing):

pip install scikit-learn mlflow "fastapi[standard]"

The example uses the wine dataset built into scikit-learn (178 wines, 13 chemical measurements, 3 grape varieties), so nothing needs downloading. It was run with scikit-learn 1.9.1, MLflow 3.18.0 and FastAPI 0.162.0.

Step 1: Train candidates and track every run

In a notebook, results live in cell outputs and disappear. An experiment tracker records each run (its parameters, metrics and the trained model file) so you can compare runs and reproduce any of them later. MLflow is one of the most widely used open-source options.

# step1_train_track.py
import mlflow
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

mlflow.set_tracking_uri("sqlite:///mlflow.db")   # every run is saved in this local file
mlflow.set_experiment("wine-classifier")

X, y = load_wine(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

candidates = {
    "logreg": make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000)),
    "forest": RandomForestClassifier(n_estimators=200, max_depth=5, random_state=42),
}

for name, model in candidates.items():
    with mlflow.start_run(run_name=name):
        cv_acc = cross_val_score(model, X_train, y_train, cv=5).mean()
        model.fit(X_train, y_train)
        test_acc = accuracy_score(y_test, model.predict(X_test))

        mlflow.log_param("model_type", name)
        mlflow.log_metrics({"cv_accuracy": cv_acc, "test_accuracy": test_acc})
        # MLflow saves scikit-learn models in the safer "skops" format by default.
        # Tree models need their tree type marked as trusted (we built it ourselves).
        mlflow.sklearn.log_model(
            model, name="model", input_example=X_train.head(3),
            skops_trusted_types=["sklearn.tree._tree.Tree"] if name == "forest" else None)
        print(f"{name:7s} cv_accuracy={cv_acc:.3f}  test_accuracy={test_acc:.3f}")
logreg  cv_accuracy=0.985  test_accuracy=1.000
forest  cv_accuracy=0.970  test_accuracy=1.000

Both models score 100% on the 45-wine test set, so the test set alone cannot separate them. Cross-validated accuracy on the training data can: 98.5% for logistic regression against 97.0% for the random forest. Run mlflow ui --backend-store-uri sqlite:///mlflow.db and open the address it prints to compare the two runs side by side.

One current detail is worth knowing. Recent MLflow versions save scikit-learn models in the skops format by default (outside Databricks) because, unlike Python’s pickle, it is designed so that loading a model file does not run hidden code, as long as you only trust types you have checked. The catch is that skops refuses model types it has not been told to trust. When we first ran this script, logging the random forest failed with an “untrusted types” error naming sklearn.tree._tree.Tree. Because we built that model ourselves, listing that one type in skops_trusted_types is safe. Only trust types for model files you created or fully trust.

Step 2: Gate on quality, then register the winner

Before a model can be deployed, it should pass an automatic check. A quality gate is just code that refuses to continue when a model falls below the bar you set. Models that pass go into the model registry, which gives each approved model a version number. An alias such as champion points at whichever version should be live.

# step2_gate_register.py
import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()

# Pick the best run by cross-validated accuracy
exp = client.get_experiment_by_name("wine-classifier")
best = client.search_runs([exp.experiment_id], order_by=["metrics.cv_accuracy DESC"], max_results=1)[0]
name, acc = best.data.params["model_type"], best.data.metrics["cv_accuracy"]
print(f"Best run: {name}  cv_accuracy={acc:.3f}")

# Quality gate: refuse to ship anything below the bar
MIN_ACCURACY = 0.95
if acc < MIN_ACCURACY:
    raise SystemExit(f"Blocked: {acc:.3f} is below {MIN_ACCURACY}")

# Register it and point the "champion" alias at this version
version = mlflow.register_model(f"runs:/{best.info.run_id}/model", "wine-classifier")
client.set_registered_model_alias("wine-classifier", "champion", version.version)
print(f"Registered wine-classifier version {version.version} as @champion")
Best run: logreg  cv_accuracy=0.985
Successfully registered model 'wine-classifier'.
Created version '1' of model 'wine-classifier'.
Registered wine-classifier version 1 as @champion

Two ideas here carry over to every real MLOps setup:

  • The gate is automatic. In a team, this script runs in CI (for example a GitHub Actions job) every time someone changes the training code. Nobody can skip it on a busy Friday.
  • Deployments point at an alias, not a file. To release a new model, you move @champion to the new version. To roll back, you move it back. The serving code never changes; a running API picks up the new version the next time it restarts or reloads the model.

Step 3: Serve the model behind an API

Other applications usually get predictions over HTTP. FastAPI turns a Python function into a web API, and checks that each request contains the right fields with the right types before your code runs.

# serve.py
import mlflow
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel

mlflow.set_tracking_uri("sqlite:///mlflow.db")
model = mlflow.pyfunc.load_model("models:/wine-classifier@champion")  # whatever version is champion

class Wine(BaseModel):
    alcohol: float
    malic_acid: float
    ash: float
    alcalinity_of_ash: float
    magnesium: float
    total_phenols: float
    flavanoids: float
    nonflavanoid_phenols: float
    proanthocyanins: float
    color_intensity: float
    hue: float
    od280_od315_of_diluted_wines: float   # the dataset's name for this column has a slash
    proline: float

app = FastAPI(title="Wine classifier")

@app.post("/predict")
def predict(wine: Wine):
    row = wine.model_dump()
    row["od280/od315_of_diluted_wines"] = row.pop("od280_od315_of_diluted_wines")
    pred = model.predict(pd.DataFrame([row]))
    return {"variety": int(pred[0])}

Start it with fastapi dev serve.py and FastAPI also gives you interactive documentation at /docs. To test it without starting a server, use FastAPI’s test client:

# step3_test_api.py
from fastapi.testclient import TestClient
from sklearn.datasets import load_wine
from serve import app

X, y = load_wine(return_X_y=True, as_frame=True)
sample = X.iloc[0].to_dict()
sample["od280_od315_of_diluted_wines"] = sample.pop("od280/od315_of_diluted_wines")

client = TestClient(app)
response = client.post("/predict", json=sample)
print(response.status_code, response.json(), "| true variety:", int(y.iloc[0]))
200 {'variety': 0} | true variety: 0

The awkward column name is a realistic detail. Real datasets often have names that are not valid in code or JSON, and the API layer is where you translate between the two. A test like this one belongs in CI next to the quality gate.

Step 4: Monitor for data drift

This is the step most beginner projects skip, and the one that matters most once a model is live. Data drift means the data arriving in production no longer looks like the training data. The model will still return confident answers; they will just be worse.

To show this, we simulate a new harvest: the same wines, but with 1.5 percentage points more alcohol and 40% stronger colour. A two-sample Kolmogorov-Smirnov (KS) test compares each feature’s distribution in the new batch with the training data, and flags features where the difference is too large to be chance.

# step4_drift.py
import pandas as pd
from scipy.stats import ks_2samp
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
import mlflow

X, y = load_wine(return_X_y=True, as_frame=True)
X_train, X_new, y_train, y_new = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

# Simulate a new harvest: 1.5 points more alcohol and 40% stronger colour
X_drift = X_new.copy()
X_drift["alcohol"] += 1.5
X_drift["color_intensity"] *= 1.4

def drift_report(reference, current, alpha=0.01):
    """Two-sample Kolmogorov-Smirnov test on every feature."""
    rows = []
    for col in reference.columns:
        stat, p = ks_2samp(reference[col], current[col])
        rows.append((col, round(stat, 3), p, p < alpha))
    return pd.DataFrame(rows, columns=["feature", "ks_stat", "p_value", "drift"])

for label, batch in [("normal batch", X_new), ("new-harvest batch", X_drift)]:
    report = drift_report(X_train, batch)
    flagged = report.loc[report.drift, "feature"].tolist()
    print(f"{label}: {len(flagged)} drifted feature(s) {flagged}")

# Labels usually arrive later. When they do, check accuracy too.
mlflow.set_tracking_uri("sqlite:///mlflow.db")
model = mlflow.pyfunc.load_model("models:/wine-classifier@champion")
for label, batch in [("normal batch", X_new), ("new-harvest batch", X_drift)]:
    acc = (model.predict(batch) == y_new.values).mean()
    print(f"{label}: accuracy={acc:.3f}")
normal batch: 0 drifted feature(s) []
new-harvest batch: 2 drifted feature(s) ['alcohol', 'color_intensity']
normal batch: accuracy=1.000
new-harvest batch: accuracy=0.800
Drift check results on the wine model: normal batch 0 drifted features and 100% accuracy; new-harvest batch 2 drifted features (alcohol, colour intensity) and 80% accuracy; the check needs no labels
Results from the code in this article: the drift check flags the problem before labels arrive.

The drift check flagged exactly the two features we changed, and nothing on the normal batch. Accuracy on the drifted batch fell from 100% to 80%: one in five wines now gets the wrong variety.

The order matters. In real systems, the true answers (labels) often arrive days or weeks after the prediction, if at all. The drift check needs no labels, so it raises the alarm as soon as the strange batch arrives, long before anyone could measure the accuracy drop. That early warning is what lets a team investigate, collect new labels and retrain before the damage spreads.

Two cautions about this simple check. First, a KS test on large batches will flag tiny, harmless differences, so teams usually also look at how big the shift is (the KS statistic) rather than the p-value alone. Second, testing 13 features at once raises the chance of a false alarm, which is why we used a stricter threshold of 0.01 instead of the usual 0.05. A common, even stricter rule is to divide the threshold by the number of features tested (0.05 / 13 is about 0.004). Open-source monitoring tools such as Evidently package these checks with reports, dashboards and alerts.

What would make this production-grade

The four scripts show the core ideas, but they run by hand on one machine. A production setup automates the same steps:

  • Version the data as well as the code, with a tool such as DVC or a data lake table format, so every model can be traced to its exact training data.
  • Run training as a pipeline on a schedule or trigger, using an orchestrator such as Airflow, Prefect or Kubeflow Pipelines, instead of running scripts by hand.
  • Run the quality gate and API tests in CI on every change, and deploy only from CI.
  • Package the API in a container (Docker) so it runs the same way on a laptop, a server or Kubernetes.
  • Schedule the drift check, log its results, and alert someone (or trigger retraining) when it fires.
  • Use a shared MLflow server instead of a local SQLite file, so the whole team sees the same runs and registry.

MLOps maturity levels

Google’s widely cited guide, MLOps: Continuous delivery and automation pipelines in machine learning, describes three levels. They are a useful way to see where a team is and what to automate next.

LevelWhat it looks likeTypical sign
0: Manual processData scientists train models by hand in notebooks and hand a model file to engineers to deployReleases are rare and nobody monitors the model in production
1: ML pipeline automationTraining runs as an automated pipeline with data and model validation, so models are retrained on new data (continuous training)The pipeline, not just the model, is what gets deployed
2: CI/CD pipeline automationChanges to the pipeline code itself are built, tested and deployed automaticallyThe team can try new features and model ideas quickly and safely

Our example sits between level 0 and level 1: experiments are tracked, a quality gate and registry are in place, and drift is checked, but nothing runs automatically yet. For a small team, that is a sensible place to start.

MLOps tools by stage

There are hundreds of MLOps tools. These are well-known open-source options for each stage; managed cloud platforms (Amazon SageMaker, Google Vertex AI, Azure Machine Learning) bundle many of them into one service.

StageWhat it doesOpen-source examples
Experiment trackingRecords parameters, metrics and model files for every runMLflow (Weights & Biases is a popular commercial alternative)
Data versioningTracks which data each model was trained onDVC, lakeFS
Pipeline orchestrationRuns training and data steps on a schedule or triggerAirflow, Prefect, Kubeflow Pipelines
Model registryStores approved model versions and which one is liveMLflow Model Registry
ServingExposes models as an API or batch jobFastAPI, BentoML, KServe
MonitoringDetects drift and quality problems in productionEvidently, Prometheus + Grafana
Feature storeShares consistent features between training and servingFeast

Start with the first two rows and a simple API. Add the rest when the pain they solve actually appears. If you work in data engineering, many of these tools will look familiar; our guide to data engineering skills and Python libraries for data engineering cover the shared foundation.

Common MLOps mistakes

  • Shipping the notebook. If a model can only be rebuilt by re-running a notebook in the right order, it cannot be reproduced. Move training into scripts and track every run.
  • Trusting one test score. Both our models scored 100% on the test set. Cross-validation and a written quality bar are what actually separate good from lucky.
  • Deploying files instead of versions. Copying model.pkl to a server makes rollback guesswork. Deploy from a registry using versions or aliases.
  • Monitoring only uptime. An API can be 100% available while its predictions get steadily worse. Monitor the inputs and, when labels arrive, the accuracy.
  • Loading model files you did not create. Pickle-based model files can run code when loaded. Prefer safer formats such as skops, and only trust types you have checked.
  • Over-tooling on day one. A Kubernetes cluster, a feature store and five dashboards will not help a team that cannot yet answer “which model is live?”. Fix the basics first.

Frequently asked questions

What is MLOps in simple words?

MLOps is the set of practices that get machine learning models into production and keep them working there. It covers tracking experiments, testing and versioning models, deploying them, and monitoring them so you know when they need retraining.

What is the difference between MLOps and DevOps?

DevOps automates building, testing and releasing software code. MLOps applies the same ideas to machine learning, but also has to manage data and trained models, test model quality, and watch for data drift, because a model can get worse without any code changing.

What is data drift?

Data drift happens when the data a model receives in production stops looking like the data it was trained on, for example when customer behaviour or a measurement process changes. The model keeps making predictions, but they become less accurate. Statistical tests such as the Kolmogorov-Smirnov test can detect it without needing labels.

Which tools should I learn first for MLOps?

Start with Python, Git and Docker, then an experiment tracker such as MLflow and a web framework such as FastAPI for serving. After that, learn one orchestrator (Airflow or Prefect) and one monitoring approach. Cloud platforms like SageMaker or Vertex AI are easier to pick up once these basics make sense.

Is MLOps a good career?

MLOps skills are in demand because many companies struggle to move models beyond the prototype stage. The role sits between data science, data engineering and DevOps, so people from any of those backgrounds can move into it by adding the missing pieces.

Key points

  • MLOps is what turns a model in a notebook into a system you can reproduce, release safely and keep accurate.
  • It differs from DevOps because data and models change too, and models can fail silently.
  • In our worked example, MLflow tracked two models, a quality gate picked logistic regression (98.5% cross-validated accuracy), the registry made it @champion, and FastAPI served it.
  • A simple drift check flagged the two shifted features on a new batch before labels were available; accuracy on that batch later turned out to be 80% instead of 100%.
  • Start small: track experiments, gate on quality, register versions and monitor inputs. Automate the rest as your team grows.

Sources

Quiz: test what you learned

Welcome to your Quiz: What Is MLOps? One Model Taken From Notebook to Production

Ihsan Ullah

Written & tested by Ihsan Ullah

Data analyst with 3 years of experience and a Bachelor's degree in Computer Science. I write clear, practical guides on data, SQL, Python and AI.

More about me →

Leave a comment

Your email address will not be published. Required fields are marked *