✨ Get task agreement matrix | API Reference

{
  "matrix": [
    [
      1,
      0.75,
      0.5
    ],
    [
      0.75,
      1,
      0.6
    ],
    [
      0.5,
      0.6,
      1
    ]
  ],
  "users": [
    {
      "avatar": null,
      "email": "annotator1@example.com",
      "first_name": "John",
      "id": 1,
      "initials": "JD",
      "last_name": "Doe",
      "username": "annotator1"
    },
    {
      "avatar": null,
      "email": "annotator2@example.com",
      "first_name": "Jane",
      "id": 2,
      "initials": "JS",
      "last_name": "Smith",
      "username": "annotator2"
    },
    {
      "id": "model:v1",
      "username": "Model v1"
    }
  ]
}

This endpoint is not available in Label Studio Community Edition. Learn more about Label Studio Enterprise

Returns a pairwise agreement matrix between selected participants for a single task, averaged across all active dimensions or a single specified dimension.

Authentication

AuthorizationToken

The token (or API key) must be passed as a request header. You can find your user token on the User Account page in Label Studio. Example:

curl https://label-studio-host/api/projects -H "Authorization: Token [your-token]"

Path parameters

project_pkintegerRequired

Project ID

task_pkintegerRequired

Task ID

Query parameters

dimensionintegerOptional

Dimension ID to compute agreement for. If not provided, averages across all active dimensions.

selectionobjectRequired

JSON object specifying which participants to include in the agreement matrix

Response

This endpoint returns an object.

matrixlist of lists of doubles

Pairwise agreement scores matrix where matrix[i][j] is the agreement between users[i] and users[j]

userslist of maps from strings to any

Expanded participant details (annotators and/or model versions)