Send data to MPM

Start logging model predictions, as well as new models, simply by sending Comet your model's input features and predictions. There is no additional configuration required.

There are two main integration paths when it comes to MPM:

• Sending data from the application making the predictions
• Using log forwarding

If the inference service is written in Python and predictions are not tracked already, we recommend sending events straight from the application using our MPM Python SDK.

If you are using a managed inference service like Sagemaker or Seldon, you can simply enable Data Capture or Request / Response logging and forward these events to MPM using our Rest API.

Integration with Experiment Management¶

Using MPM in conjunction with Experiment Management and Comet's Model Registry allows you to track models from development all the way to production ! In order to do this, you simply need to ensure that the model name used when sending MPM events matches up with the model name in the Model Registry.

MPM can also be used standalone, in which case models are automatically created in the Comet Model Registry whenever you send the first event for a new model.

MPM Python SDK¶

The MPM Python SDK has been developed with production inference services in mind and includes a number of optimizations to make sure the logging overhead is kept at a minimum.

The MPM SDK can be installed using:

pip install comet-mpm

export COMET_URL=https://<path_to_your_deployment>/

Sending prediction events¶

Once installed, logging events to MPM takes just three lines of code:

from comet_mpm import CometMPM

api_key = "<Your API key>"
workspace_name = "<Your workspace name>"

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key
)

MPM.log_event(
    prediction_id="1",
    input_features={'age': 29, 'country': 'UK'},
    output_features={'value': True, 'probability': 0.5},
    labels={'value': False}
)

You can check that individual events are correctly ingested in Comet by navigating to the debugger tab in MPM. If the event has been ingested correctly, it will appear in the predictions tab, if there was an error ingesting the data it will appear in the Ingestion Errors section with an error message explaining the issue. It is worth noting that it can take up to 5 minutes for the events to be present in the debugger tab.

If you are using FastAPI, you will also need to implement a shutdown event as detailed here

Sending labels¶

You can also send labels using the SDK:

from comet_mpm import CometMPM

api_key = "<Your API key>"
workspace_name = "<Your workspace name>"

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key
)


MPM.log_label(
    prediction_id="1",
    labels={"value": False}
)

Sending batches of events¶

To send a batch of events to MPM, you can use the log_dataframe method to log a pandas DataFrame:

from comet_mpm import CometMPM
import pandas as pd

api_key = "<Your API key>"
workspace_name = "<Your workspace name>"

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key,
)

df = pd.DataFrame({
    'age': [29, 30, 31],
    'prediction_id': [100, 200, 300],
    'country': ['UK', 'US', 'UK'],
    'label': [True, False, True],
    'predicted_value': [True, False, True]
})

MPM.log_dataframe(
    dataframe=df,
    prediction_id_column='prediction_id',
    feature_columns=['age', 'country'],
    output_features_columns=['predicted_value'],
    labels_columns=['label']
)

Sending training distributions¶

If you would like to compute feature drift based on a training distribution, you will need to upload these to the MPM platform. You can achieve this by using the upload_dataset_csv method in the MPM SDK:

from comet_mpm import CometMPM

api_key = "<Your API key>"
workspace_name = "<Your workspace name>"

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key,
)

MPM.upload_dataset_csv(
    file_path='path/to/your/training/distribution.csv',
    dataset_type='training-events', # This is an important field as it marks this CSV as training data
    dataset_name='training-distribution',
)

Managing errors¶

The core philosophy that guides the MPM SDK is that we never want to crash your production application just because of a MPM logging error. As a result, we don't raise exceptions unless you explicitly ask us to.

Raising errors during initialization:

When initializing the MPM SDK, you can set the raise_on_error_during_init parameter to True to raise an exception if there is an issue with the initialization of the MPM SDK:

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key,
    raise_on_error_during_init=True
)

Raising errors during logging:

The MPM Python SDK sends data using a background thread for performance reasons and as a result cannot raise exceptions when you call .log_event in case of an issue with data logging.

In order to track errors that occur during the submission of events to MPM, you can use the CometMPM.has_logging_errors method and CometMPM.get_logging_errors method:

from comet_mpm import CometMPM
import pandas as pd

api_key = "<Your API key>"
workspace_name = "<Your workspace name>"

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name='demo-mpm-model',
    model_version='1.2.0',
    api_key=api_key,
)

MPM.log_event(
    prediction_id="1",
    input_features={'age': 29, 'country': 'UK'},
    output_features={'value': True, 'probability': 0.5},
    labels={'value': False}
)

MPM.end()

if MPM.has_logging_error():
    errors = MPM.get_logging_errors()
    print(errors)

For each error, we keep track of: * message: The error message * logger_name: The logger name * timestamp: The timestamp * data_affected: Description of what data was affected (optional), typically the batch of events that were not logged to MPM * traceback: Traceback information (optional)

Troubleshooting¶

Most MPM logging problems are silent by design: the SDK will not crash your inference service, so a misconfiguration usually shows up as missing data rather than as an error. Start by calling CometMPM.get_logging_errors() after MPM.end(), as described above, and then work through the cases below.

Events were logged but do not appear in the UI

Ingestion is asynchronous. Events are accepted by the API first and moved into the query layer by a separate process, so there is a short delay between a successful log_event call and the data being visible in charts or returned by the SDK's query methods. If events are still missing several minutes later, carry on through the cases below.

Event is out of date

Comet rejects events whose timestamp is older than the installation's event age limit. This is the most common cause of a backfill that reports success but lands only partial data: recent events are accepted and older ones are dropped, per event.

The limit is 90 days on Comet Cloud, and 90 days is also the default for self-hosted installations. Self-hosted deployments can change it with the MPM_EVENT_AGE_LIMIT_DAYS environment variable, so if you are unsure what your installation uses, check that value before assuming the age limit is not the cause.

If you need to load history older than the limit, raise it before sending rather than after: events rejected at ingestion time are not retried.

Event timestamp can not be in future

The timestamp parameter of log_event and log_label is seconds since the epoch in UTC, and the timestamp_column of log_dataframe is the same. Passing milliseconds puts every event far in the future, and Comet rejects future-dated events. If you have milliseconds, divide by 1000:

MPM.log_event(
    prediction_id="1",
    input_features={"age": 29, "country": "UK"},
    output_features={"value": True, "probability": 0.5},
    timestamp=timestamp_in_milliseconds / 1000,
)

A malformed or negative value is rejected as Invalid timestamp.

Some events are missing after logging a large batch

The SDK sends events from a background thread, and MPM.end() waits for that queue to drain before returning. If the queue is still draining when the wait ends, the remaining events are discarded and a warning is printed reporting how many were lost.

When logging a large backfill, pass a timeout that suits the volume rather than relying on the default:

MPM.end(timeout=1800)

The same value can be set with the COMET_MPM_JOIN_TIMEOUT environment variable. Note that MPM.end() is what flushes the queue, so a script that exits without calling it has no opportunity to finish sending.

Nothing is logged at all

Check the messages printed when CometMPM is constructed. If Comet reports that Model Production Monitoring is not installed in your installation, MPM is not enabled for that deployment and no amount of retrying will help — contact your Comet administrator. Initialize with raise_on_error_during_init=True to turn that message into an exception:

MPM = CometMPM(
    workspace_name=workspace_name,
    model_name="demo-mpm-model",
    model_version="1.2.0",
    api_key=api_key,
    raise_on_error_during_init=True,
)

A transient failure at startup — a restart, a proxy, a slow network — is reported as a warning and does not stop logging; events continue to be buffered and sent.

Comet URL conflict detected between config and API Key

Comet API keys can carry the URL of the installation they belong to. If you also set COMET_URL (or COMET_URL_OVERRIDE) and the two disagree, the SDK logs this warning and uses the configured URL, not the one in the key. That is usually what you want when pointing a key at a test deployment — but if data is arriving somewhere unexpected, this warning tells you which destination won.

MPM Rest API¶

To track model performance, MPM needs to have access to a model's input features, output features and labels. In addition to the Python API, this data can be sent via a Rest API.

The MPM Rest API includes three methods that can be used to upload MPM events:

• https://www.comet.com/mpm/v2/events/batch: Used to upload batches of events that contain input and output features
• https://www.comet.com/mpm/v2/labels: Used to upload labels

Sending input and output features to MPM¶

The JSON payload for the https://www.comet.com/mpm/v2/events/batch POST endpoints should contain the following attributes:

Here is an example payload:

{
    "data": [{
        "workspaceName": "...",
        "modelName": "Credit Scoring",
        "modelVersion": "1.0.0",
        "timestamp": 1615922560000,
        "predictionId": "000001",
        "features": {
            "feature_1":0.34,
            "feature_2": "dog",
        },
        "predictions": {
            "value": "true",
            "probability": 0.86
        },
        "labels": {
            "value": "true"
        }
    }]
}

To test, you can use:

workspaceName=<workspace_name>
api_key=<api_key>

current_timestamp=$(date -v-1H +%s000)
payload='{"data": [{"features": {"categorical_feature_0": "value_1", "numerical_feature_0": 0.5841119597210334}, "modelName": "test-model", "modelVersion": "1.0.0", "predictions": {"predicted_value": "true"}, "predictionId": "e347539b-a1df-432e-aa4a-3fe93805d3be", "timestamp": '$current_timestamp', "workspaceName": "'$workspaceName'"}]}'

curl -s -d "$payload" \
    -H "Content-Type: application/json"\
    -H "Authorization: $api_key"\
    -X POST https://www.comet.com/mpm/v2/events/batch

Sending labels to MPM¶

Ground truth labels can be sent to Comet and are used to compute accuracy related metrics (Accuracy, Precision, Recall, F1-score, etc). You can send the labels either with the input features and predictions or at any time after that, we will take care of updating the relevant metrics.

If you wish to send labels to Comet independently from the prediction, you can use the https://www.comet.com/mpm/v2/labels/batch POST endpoint with a JSON payload containing the following attributes:

Learn more¶

• About Comet's MPM