Braintrust Data API v1.0.0

API specification for the backend data server. The API is hosted globally at https://api.braintrustdata.com or in your own environment. The v1 API is currently in preview mode and unstable until March 1, 2024. We may make backwards incompatible changes before then, as we learn from our users.

Base URLs:

https://api.braintrustdata.com

License: Apache 2.0

Authentication

HTTP Authentication, scheme: bearer
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key] to your HTTP request. You can create an API key in the Braintrust organization settings page.

Projects

Create project

Code samples

const inputBody = JSON.stringify({
  name: "string",
  org_name: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/project

Create a new project. If there is an existing project with the same name as the one specified in the request, will return the existing project unmodified

Body parameter

{
  "name": "string",
  "org_name": "string"
}

Parameters

Name	In	Type	Required	Description
body	body	CreateProject	true	Any desired information about the new project object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new project object	Project
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Create or replace project

Code samples

const inputBody = JSON.stringify({
  name: "string",
  org_name: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project", {
  method: "PUT",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PUT /v1/project

Create or replace a new project. If there is an existing project with the same name as the one specified in the request, will replace the existing project with the provided fields

Body parameter

{
  "name": "string",
  "org_name": "string"
}

Parameters

Name	In	Type	Required	Description
body	body	CreateProject	true	Any desired information about the new project object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new project object	Project
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

List projects

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/project

List out all projects. The projects are sorted by creation date, with the most recently-created projects coming first

Parameters

Name	In	Type	Required	Description
limit	query	AppLimitParam	false	Limit the number of objects to return
starting_after	query	StartingAfter	false	Pagination cursor id.
ending_before	query	EndingBefore	false	Pagination cursor id.
project_name	query	ProjectName	false	Name of the project to search for
org_name	query	OrgName	false	Filter search results to within a particular organization

Detailed descriptions

starting_after: Pagination cursor id.

For example, if the final item in the last page you fetched had an id of foo, pass starting_after=foo to fetch the next page. Note: you may only pass one of starting_after and ending_before

ending_before: Pagination cursor id.

For example, if the initial item in the last page you fetched had an id of foo, pass ending_before=foo to fetch the previous page. Note: you may only pass one of starting_after and ending_before

Example responses

200 Response

{
  "objects": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
      "name": "string",
      "created": "2019-08-24T14:15:22Z",
      "deleted_at": "2019-08-24T14:15:22Z",
      "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns a list of project objects	Inline
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

Response Schema

Status Code 200

Name	Type	Required	Restrictions	Description
» objects	[Project]	true	none	A list of project objects
»» id	string(uuid)	true	none	Unique identifier for the project
»» org_id	string(uuid)	true	none	Unique id for the organization that the project belongs under
»» name	string	true	none	Name of the project
»» created	string,null(date-time)	false	none	Date of project creation
»» deleted_at	string,null(date-time)	false	none	Date of project deletion, or null if the project is still active
»» user_id	string,null(uuid)	false	none	Identifies the user who created the project

🔒

This endpoint requires authentication.

Get project

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project/{project_id}", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/project/{project_id}

Get a project object by its id

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the project object	Project
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Partially update project

Code samples

const inputBody = JSON.stringify({
  name: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project/{project_id}", {
  method: "PATCH",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PATCH /v1/project/{project_id}

Partially update a project object. Specify the fields to update in the payload. Any object-type fields will be deep-merged with existing content. Currently we do not support removing fields or setting them to null. As a workaround, you may retrieve the full object with GET /project/{id} and then replace it with PUT /project.

Body parameter

{
  "name": "string"
}

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id
body	body	PatchProject	false	Fields to update

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the experiment object	Project
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Delete project

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project/{project_id}", {
  method: "DELETE",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

DELETE /v1/project/{project_id}

Delete a project object by its id

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the deleted project object	Project
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Insert project logs events

Code samples

const inputBody = JSON.stringify({
  events: [
    {
      input: null,
      output: null,
      expected: null,
      scores: {
        property1: 1,
        property2: 1,
      },
      metadata: {
        property1: null,
        property2: null,
      },
      metrics: {
        start: 0,
        end: 0,
        property1: null,
        property2: null,
      },
      context: {
        caller_functionname: "string",
        caller_filename: "string",
        caller_lineno: 0,
        property1: null,
        property2: null,
      },
      span_attributes: {
        name: "string",
        type: "llm",
        property1: null,
        property2: null,
      },
      id: "string",
      _object_delete: true,
      _is_merge: false,
      _parent_id: "string",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project_logs/{project_id}/insert", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/project_logs/{project_id}/insert

Insert a set of events into the project logs

Body parameter

{
  "events": [
    {
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      },
      "id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id
body	body	InsertProjectLogsEventRequest	true	An array of project logs events to insert

Example responses

200 Response

{
  "row_ids": ["string"]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the inserted row ids	InsertEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch project logs (POST form)

Code samples

const inputBody = JSON.stringify({
  limit: 0,
  max_xact_id: 0,
  max_root_span_id: "string",
  filters: [
    {
      type: "path_lookup",
      path: ["string"],
      value: null,
    },
  ],
  version: 0,
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project_logs/{project_id}/fetch", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/project_logs/{project_id}/fetch

Fetch the events in a project logs. Equivalent to the GET form of the same path, but with the parameters in the request body rather than in the URL query

Body parameter

{
  "limit": 0,
  "max_xact_id": 0,
  "max_root_span_id": "string",
  "filters": [
    {
      "type": "path_lookup",
      "path": ["string"],
      "value": null
    }
  ],
  "version": 0
}

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id
body	body	FetchEventsRequest	false	Filters for the fetch query

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "log_id": "g",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchProjectLogsEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch project logs (GET form)

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project_logs/{project_id}/fetch", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/project_logs/{project_id}/fetch

Fetch the events in a project logs. Equivalent to the POST form of the same path, but with the parameters in the URL query rather than in the request body

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id
limit	query	FetchLimitParam	false	limit the number of traces fetched
max_xact_id	query	MaxXactId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
max_root_span_id	query	MaxRootSpanId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
version	query	Version	false	Retrieve a snapshot of events from a past time

Detailed descriptions

limit: limit the number of traces fetched

Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest _xact_id. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier _xact_id. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by id) from your combined result set.

The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

max_xact_id: Together, max_xact_id and max_root_span_id form a pagination cursor

Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id). See the documentation of limit for an overview of paginating fetch queries.

max_root_span_id: Together, max_xact_id and max_root_span_id form a pagination cursor

version: Retrieve a snapshot of events from a past time

The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "log_id": "g",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchProjectLogsEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Feedback for project logs events

Code samples

const inputBody = JSON.stringify({
  feedback: [
    {
      id: "string",
      scores: {
        property1: 1,
        property2: 1,
      },
      expected: null,
      comment: "string",
      metadata: {
        property1: null,
        property2: null,
      },
      source: "app",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "text/plain",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/project_logs/{project_id}/feedback", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/project_logs/{project_id}/feedback

Log feedback for a set of project logs events

Body parameter

{
  "feedback": [
    {
      "id": "string",
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "expected": null,
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
project_id	path	ProjectId	true	Project id
body	body	FeedbackProjectLogsEventRequest	true	An array of feedback objects

Example responses

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	No return value	None
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Experiments

Create experiment

Code samples

const inputBody = JSON.stringify({
  project_id: "405d8375-3514-403b-8c43-83ae74cfe0e9",
  name: "string",
  description: "string",
  repo_info: {
    commit: "string",
    branch: "string",
    tag: "string",
    dirty: true,
    author_name: "string",
    author_email: "string",
    commit_message: "string",
    commit_time: "string",
    git_diff: "string",
  },
  base_exp_id: "4838cee2-a665-4545-aa9f-483678c01a6b",
  dataset_id: "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  dataset_version: "string",
  public: true,
  metadata: {
    property1: null,
    property2: null,
  },
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/experiment

Create a new experiment. If there is an existing experiment in the project with the same name as the one specified in the request, will create a new experiment from name, suffixed with a unique identifier

Body parameter

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Parameters

Name	In	Type	Required	Description
body	body	CreateExperiment	true	Any desired information about the new experiment object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new experiment object	Experiment
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Create or replace experiment

Code samples

const inputBody = JSON.stringify({
  project_id: "405d8375-3514-403b-8c43-83ae74cfe0e9",
  name: "string",
  description: "string",
  repo_info: {
    commit: "string",
    branch: "string",
    tag: "string",
    dirty: true,
    author_name: "string",
    author_email: "string",
    commit_message: "string",
    commit_time: "string",
    git_diff: "string",
  },
  base_exp_id: "4838cee2-a665-4545-aa9f-483678c01a6b",
  dataset_id: "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  dataset_version: "string",
  public: true,
  metadata: {
    property1: null,
    property2: null,
  },
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment", {
  method: "PUT",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PUT /v1/experiment

Create or replace a new experiment. If there is an existing experiment in the project with the same name as the one specified in the request, will replace the existing experiment with the provided fields

Body parameter

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Parameters

Name	In	Type	Required	Description
body	body	CreateExperiment	true	Any desired information about the new experiment object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new experiment object	Experiment
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

List experiments

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/experiment

List out all experiments. The experiments are sorted by creation date, with the most recently-created experiments coming first

Parameters

Name	In	Type	Required	Description
limit	query	AppLimitParam	false	Limit the number of objects to return
starting_after	query	StartingAfter	false	Pagination cursor id.
ending_before	query	EndingBefore	false	Pagination cursor id.
experiment_name	query	ExperimentName	false	Name of the experiment to search for
project_name	query	ProjectName	false	Name of the project to search for
org_name	query	OrgName	false	Filter search results to within a particular organization

Detailed descriptions

starting_after: Pagination cursor id.

For example, if the final item in the last page you fetched had an id of foo, pass starting_after=foo to fetch the next page. Note: you may only pass one of starting_after and ending_before

ending_before: Pagination cursor id.

Example responses

200 Response

{
  "objects": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "name": "string",
      "description": "string",
      "created": "2019-08-24T14:15:22Z",
      "repo_info": {
        "commit": "string",
        "branch": "string",
        "tag": "string",
        "dirty": true,
        "author_name": "string",
        "author_email": "string",
        "commit_message": "string",
        "commit_time": "string",
        "git_diff": "string"
      },
      "commit": "string",
      "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
      "deleted_at": "2019-08-24T14:15:22Z",
      "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
      "dataset_version": "string",
      "public": true,
      "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
      "metadata": {
        "property1": null,
        "property2": null
      }
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns a list of experiment objects	Inline
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

Response Schema

Status Code 200

Name	Type	Required	Restrictions	Description
» objects	[Experiment]	true	none	A list of experiment objects
»» id	string(uuid)	true	none	Unique identifier for the experiment
»» project_id	string(uuid)	true	none	Unique identifier for the project that the experiment belongs under
»» name	string	true	none	Name of the experiment. Within a project, experiment names are unique
»» description	string,null	false	none	Textual description of the experiment
»» created	string,null(date-time)	false	none	Date of experiment creation
»» repo_info	RepoInfo	false	none	Metadata about the state of the repo when the experiment was created
»»» commit	string,null	false	none	SHA of most recent commit
»»» branch	string,null	false	none	Name of the branch the most recent commit belongs to
»»» tag	string,null	false	none	Name of the tag on the most recent commit
»»» dirty	boolean,null	false	none	Whether or not the repo had uncommitted changes when snapshotted
»»» author_name	string,null	false	none	Name of the author of the most recent commit
»»» author_email	string,null	false	none	Email of the author of the most recent commit
»»» commit_message	string,null	false	none	Most recent commit message
»»» commit_time	string,null	false	none	Time of the most recent commit
»»» git_diff	string,null	false	none	If the repo was dirty when run, this includes the diff between the current state of the repo and the most recent commit.
»» commit	string,null	false	none	Commit, taken directly from `repo_info.commit`
»» base_exp_id	string,null(uuid)	false	none	Id of default base experiment to compare against when viewing this experiment
»» deleted_at	string,null(date-time)	false	none	Date of experiment deletion, or null if the experiment is still active
»» dataset_id	string,null(uuid)	false	none	Identifier of the linked dataset, or null if the experiment is not linked to a dataset
»» dataset_version	string,null	false	none	Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
»» public	boolean	true	none	Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
»» user_id	string,null(uuid)	false	none	Identifies the user who created the experiment
»» metadata	object,null	false	none	User-controlled metadata about the experiment
»»» additionalProperties	any	false	none	none

🔒

This endpoint requires authentication.

Get experiment

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/experiment/{experiment_id}

Get an experiment object by its id

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the experiment object	Experiment
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Partially update experiment

Code samples

const inputBody = JSON.stringify({
  name: "string",
  description: "string",
  repo_info: {
    commit: "string",
    branch: "string",
    tag: "string",
    dirty: true,
    author_name: "string",
    author_email: "string",
    commit_message: "string",
    commit_time: "string",
    git_diff: "string",
  },
  base_exp_id: "4838cee2-a665-4545-aa9f-483678c01a6b",
  dataset_id: "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  dataset_version: "string",
  public: true,
  metadata: {
    property1: null,
    property2: null,
  },
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}", {
  method: "PATCH",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PATCH /v1/experiment/{experiment_id}

Partially update an experiment object. Specify the fields to update in the payload. Any object-type fields will be deep-merged with existing content. Currently we do not support removing fields or setting them to null. As a workaround, you may retrieve the full object with GET /experiment/{id} and then replace it with PUT /experiment.

Body parameter

{
  "name": "string",
  "description": "string",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
body	body	PatchExperiment	false	Fields to update

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the experiment object	Experiment
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Delete experiment

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}", {
  method: "DELETE",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

DELETE /v1/experiment/{experiment_id}

Delete an experiment object by its id

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the deleted experiment object	Experiment
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Insert experiment events

Code samples

const inputBody = JSON.stringify({
  events: [
    {
      input: null,
      output: null,
      expected: null,
      scores: {
        property1: 1,
        property2: 1,
      },
      metadata: {
        property1: null,
        property2: null,
      },
      metrics: {
        start: 0,
        end: 0,
        property1: null,
        property2: null,
      },
      context: {
        caller_functionname: "string",
        caller_filename: "string",
        caller_lineno: 0,
        property1: null,
        property2: null,
      },
      span_attributes: {
        name: "string",
        type: "llm",
        property1: null,
        property2: null,
      },
      id: "string",
      dataset_record_id: "string",
      _object_delete: true,
      _is_merge: false,
      _parent_id: "string",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}/insert", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/experiment/{experiment_id}/insert

Insert a set of events into the experiment

Body parameter

{
  "events": [
    {
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      },
      "id": "string",
      "dataset_record_id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
body	body	InsertExperimentEventRequest	true	An array of experiment events to insert

Example responses

200 Response

{
  "row_ids": ["string"]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the inserted row ids	InsertEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch experiment (POST form)

Code samples

const inputBody = JSON.stringify({
  limit: 0,
  max_xact_id: 0,
  max_root_span_id: "string",
  filters: [
    {
      type: "path_lookup",
      path: ["string"],
      value: null,
    },
  ],
  version: 0,
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}/fetch", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/experiment/{experiment_id}/fetch

Fetch the events in an experiment. Equivalent to the GET form of the same path, but with the parameters in the request body rather than in the URL query

Body parameter

{
  "limit": 0,
  "max_xact_id": 0,
  "max_root_span_id": "string",
  "filters": [
    {
      "type": "path_lookup",
      "path": ["string"],
      "value": null
    }
  ],
  "version": 0
}

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
body	body	FetchEventsRequest	false	Filters for the fetch query

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "dataset_record_id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchExperimentEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch experiment (GET form)

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}/fetch", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/experiment/{experiment_id}/fetch

Fetch the events in an experiment. Equivalent to the POST form of the same path, but with the parameters in the URL query rather than in the request body

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
limit	query	FetchLimitParam	false	limit the number of traces fetched
max_xact_id	query	MaxXactId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
max_root_span_id	query	MaxRootSpanId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
version	query	Version	false	Retrieve a snapshot of events from a past time

Detailed descriptions

limit: limit the number of traces fetched

The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

max_xact_id: Together, max_xact_id and max_root_span_id form a pagination cursor

max_root_span_id: Together, max_xact_id and max_root_span_id form a pagination cursor

version: Retrieve a snapshot of events from a past time

The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "dataset_record_id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchExperimentEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Feedback for experiment events

Code samples

const inputBody = JSON.stringify({
  feedback: [
    {
      id: "string",
      scores: {
        property1: 1,
        property2: 1,
      },
      expected: null,
      comment: "string",
      metadata: {
        property1: null,
        property2: null,
      },
      source: "app",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "text/plain",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/experiment/{experiment_id}/feedback", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/experiment/{experiment_id}/feedback

Log feedback for a set of experiment events

Body parameter

{
  "feedback": [
    {
      "id": "string",
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "expected": null,
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
body	body	FeedbackExperimentEventRequest	true	An array of feedback objects

Example responses

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	No return value	None
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Summarize experiment

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch(
  "https://api.braintrustdata.com/v1/experiment/{experiment_id}/summarize",
  {
    method: "GET",
 
    headers: headers,
  },
)
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/experiment/{experiment_id}/summarize

Summarize experiment

Parameters

Name	In	Type	Required	Description
experiment_id	path	ExperimentId	true	Experiment id
summarize_scores	query	SummarizeScores	false	Whether to summarize the scores and metrics. If false (or omitted), only the metadata will be returned.
comparison_experiment_id	query	ComparisonExperimentId	false	The experiment to compare against, if summarizing scores and metrics. If omitted, will fall back to the `base_exp_id` stored in the experiment metadata, and then to the most recent experiment run in the same project. Must pass `summarize_scores=true` for this id to be used

Example responses

200 Response

{
  "project_name": "string",
  "experiment_name": "string",
  "project_url": "http://example.com",
  "experiment_url": "http://example.com",
  "comparison_experiment_name": "string",
  "scores": {
    "property1": {
      "name": "string",
      "score": 1,
      "diff": -1,
      "improvements": 0,
      "regressions": 0
    },
    "property2": {
      "name": "string",
      "score": 1,
      "diff": -1,
      "improvements": 0,
      "regressions": 0
    }
  },
  "metrics": {
    "property1": {
      "name": "string",
      "metric": 0,
      "unit": "string",
      "diff": 0,
      "improvements": 0,
      "regressions": 0
    },
    "property2": {
      "name": "string",
      "metric": 0,
      "unit": "string",
      "diff": 0,
      "improvements": 0,
      "regressions": 0
    }
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	experiment summary	SummarizeExperimentResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Datasets

Create dataset

Code samples

const inputBody = JSON.stringify({
  project_id: "405d8375-3514-403b-8c43-83ae74cfe0e9",
  name: "string",
  description: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/dataset

Create a new dataset. If there is an existing dataset in the project with the same name as the one specified in the request, will return the existing dataset unmodified

Body parameter

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string"
}

Parameters

Name	In	Type	Required	Description
body	body	CreateDataset	true	Any desired information about the new dataset object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new dataset object	Dataset
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Create or replace dataset

Code samples

const inputBody = JSON.stringify({
  project_id: "405d8375-3514-403b-8c43-83ae74cfe0e9",
  name: "string",
  description: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset", {
  method: "PUT",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PUT /v1/dataset

Create or replace a new dataset. If there is an existing dataset in the project with the same name as the one specified in the request, will replace the existing dataset with the provided fields

Body parameter

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string"
}

Parameters

Name	In	Type	Required	Description
body	body	CreateDataset	true	Any desired information about the new dataset object

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the new dataset object	Dataset
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

List datasets

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/dataset

List out all datasets. The datasets are sorted by creation date, with the most recently-created datasets coming first

Parameters

Name	In	Type	Required	Description
limit	query	AppLimitParam	false	Limit the number of objects to return
starting_after	query	StartingAfter	false	Pagination cursor id.
ending_before	query	EndingBefore	false	Pagination cursor id.
dataset_name	query	DatasetName	false	Name of the dataset to search for
project_name	query	ProjectName	false	Name of the project to search for
org_name	query	OrgName	false	Filter search results to within a particular organization

Detailed descriptions

starting_after: Pagination cursor id.

For example, if the final item in the last page you fetched had an id of foo, pass starting_after=foo to fetch the next page. Note: you may only pass one of starting_after and ending_before

ending_before: Pagination cursor id.

Example responses

200 Response

{
  "objects": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "name": "string",
      "description": "string",
      "created": "2019-08-24T14:15:22Z",
      "deleted_at": "2019-08-24T14:15:22Z",
      "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns a list of dataset objects	Inline
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

Response Schema

Status Code 200

Name	Type	Required	Restrictions	Description
» objects	[Dataset]	true	none	A list of dataset objects
»» id	string(uuid)	true	none	Unique identifier for the dataset
»» project_id	string,null(uuid)	false	none	Unique identifier for the project that the dataset belongs under
»» name	string	true	none	Name of the dataset. Within a project, dataset names are unique
»» description	string,null	false	none	Textual description of the dataset
»» created	string,null(date-time)	false	none	Date of dataset creation
»» deleted_at	string,null(date-time)	false	none	Date of dataset deletion, or null if the dataset is still active
»» user_id	string,null(uuid)	false	none	Identifies the user who created the dataset

🔒

This endpoint requires authentication.

Get dataset

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/dataset/{dataset_id}

Get a dataset object by its id

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the dataset object	Dataset
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Partially update dataset

Code samples

const inputBody = JSON.stringify({
  name: "string",
  description: "string",
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}", {
  method: "PATCH",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

PATCH /v1/dataset/{dataset_id}

Partially update a dataset object. Specify the fields to update in the payload. Any object-type fields will be deep-merged with existing content. Currently we do not support removing fields or setting them to null. As a workaround, you may retrieve the full object with GET /dataset/{id} and then replace it with PUT /dataset.

Body parameter

{
  "name": "string",
  "description": "string"
}

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
body	body	PatchDataset	true	Fields to update

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the experiment object	Dataset
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Delete dataset

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}", {
  method: "DELETE",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

DELETE /v1/dataset/{dataset_id}

Delete a dataset object by its id

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id

Example responses

200 Response

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the deleted dataset object	Dataset
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Insert dataset events

Code samples

const inputBody = JSON.stringify({
  events: [
    {
      input: null,
      expected: null,
      metadata: {
        property1: null,
        property2: null,
      },
      id: "string",
      _object_delete: true,
      _is_merge: false,
      _parent_id: "string",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}/insert", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/dataset/{dataset_id}/insert

Insert a set of events into the dataset

Body parameter

{
  "events": [
    {
      "input": null,
      "expected": null,
      "metadata": {
        "property1": null,
        "property2": null
      },
      "id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
body	body	InsertDatasetEventRequest	true	An array of dataset events to insert

Example responses

200 Response

{
  "row_ids": ["string"]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the inserted row ids	InsertEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch dataset (POST form)

Code samples

const inputBody = JSON.stringify({
  limit: 0,
  max_xact_id: 0,
  max_root_span_id: "string",
  filters: [
    {
      type: "path_lookup",
      path: ["string"],
      value: null,
    },
  ],
  version: 0,
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}/fetch", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/dataset/{dataset_id}/fetch

Fetch the events in a dataset. Equivalent to the GET form of the same path, but with the parameters in the request body rather than in the URL query

Body parameter

{
  "limit": 0,
  "max_xact_id": 0,
  "max_root_span_id": "string",
  "filters": [
    {
      "type": "path_lookup",
      "path": ["string"],
      "value": null
    }
  ],
  "version": 0
}

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
body	body	FetchEventsRequest	false	Filters for the fetch query

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
      "input": null,
      "expected": null,
      "metadata": {
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "root_span_id": "string"
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchDatasetEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Fetch dataset (GET form)

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}/fetch", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/dataset/{dataset_id}/fetch

Fetch the events in a dataset. Equivalent to the POST form of the same path, but with the parameters in the URL query rather than in the request body

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
limit	query	FetchLimitParam	false	limit the number of traces fetched
max_xact_id	query	MaxXactId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
max_root_span_id	query	MaxRootSpanId	false	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor
version	query	Version	false	Retrieve a snapshot of events from a past time

Detailed descriptions

limit: limit the number of traces fetched

The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

max_xact_id: Together, max_xact_id and max_root_span_id form a pagination cursor

max_root_span_id: Together, max_xact_id and max_root_span_id form a pagination cursor

version: Retrieve a snapshot of events from a past time

The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Example responses

200 Response

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
      "input": null,
      "expected": null,
      "metadata": {
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "root_span_id": "string"
    }
  ]
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the fetched rows	FetchDatasetEventsResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Feedback for dataset events

Code samples

const inputBody = JSON.stringify({
  feedback: [
    {
      id: "string",
      comment: "string",
      metadata: {
        property1: null,
        property2: null,
      },
      source: "app",
    },
  ],
});
const headers = {
  "Content-Type": "application/json",
  Accept: "text/plain",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}/feedback", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/dataset/{dataset_id}/feedback

Log feedback for a set of dataset events

Body parameter

{
  "feedback": [
    {
      "id": "string",
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
body	body	FeedbackDatasetEventRequest	true	An array of feedback objects

Example responses

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	No return value	None
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Summarize dataset

Code samples

const headers = {
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/dataset/{dataset_id}/summarize", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1/dataset/{dataset_id}/summarize

Summarize dataset

Parameters

Name	In	Type	Required	Description
dataset_id	path	DatasetId	true	Dataset id
summarize_data	query	SummarizeData	false	Whether to summarize the data. If false (or omitted), only the metadata will be returned.

Example responses

200 Response

{
  "project_name": "string",
  "dataset_name": "string",
  "project_url": "http://example.com",
  "dataset_url": "http://example.com",
  "data_summary": {
    "total_records": 0
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	dataset summary	SummarizeDatasetResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Other

Hello world endpoint

Code samples

const headers = {
  Accept: "text/plain",
};
 
fetch("https://api.braintrustdata.com/v1", {
  method: "GET",
 
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

GET /v1

Default endpoint. Simply replies with 'Hello, World!'. Authorization is not required

Example responses

200 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Hello world string	string
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🌐

This operation does not require authentication

CrossObject

Cross-object insert

Code samples

const inputBody = JSON.stringify({
  experiment: {
    property1: {
      events: [
        {
          input: null,
          output: null,
          expected: null,
          scores: {
            property1: 1,
            property2: 1,
          },
          metadata: {
            property1: null,
            property2: null,
          },
          metrics: {
            start: 0,
            end: 0,
            property1: null,
            property2: null,
          },
          context: {
            caller_functionname: "string",
            caller_filename: "string",
            caller_lineno: 0,
            property1: null,
            property2: null,
          },
          span_attributes: {
            name: "string",
            type: "llm",
            property1: null,
            property2: null,
          },
          id: "string",
          dataset_record_id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          scores: {
            property1: 1,
            property2: 1,
          },
          expected: null,
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
    property2: {
      events: [
        {
          input: null,
          output: null,
          expected: null,
          scores: {
            property1: 1,
            property2: 1,
          },
          metadata: {
            property1: null,
            property2: null,
          },
          metrics: {
            start: 0,
            end: 0,
            property1: null,
            property2: null,
          },
          context: {
            caller_functionname: "string",
            caller_filename: "string",
            caller_lineno: 0,
            property1: null,
            property2: null,
          },
          span_attributes: {
            name: "string",
            type: "llm",
            property1: null,
            property2: null,
          },
          id: "string",
          dataset_record_id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          scores: {
            property1: 1,
            property2: 1,
          },
          expected: null,
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
  },
  dataset: {
    property1: {
      events: [
        {
          input: null,
          expected: null,
          metadata: {
            property1: null,
            property2: null,
          },
          id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
    property2: {
      events: [
        {
          input: null,
          expected: null,
          metadata: {
            property1: null,
            property2: null,
          },
          id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
  },
  project_logs: {
    property1: {
      events: [
        {
          input: null,
          output: null,
          expected: null,
          scores: {
            property1: 1,
            property2: 1,
          },
          metadata: {
            property1: null,
            property2: null,
          },
          metrics: {
            start: 0,
            end: 0,
            property1: null,
            property2: null,
          },
          context: {
            caller_functionname: "string",
            caller_filename: "string",
            caller_lineno: 0,
            property1: null,
            property2: null,
          },
          span_attributes: {
            name: "string",
            type: "llm",
            property1: null,
            property2: null,
          },
          id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          scores: {
            property1: 1,
            property2: 1,
          },
          expected: null,
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
    property2: {
      events: [
        {
          input: null,
          output: null,
          expected: null,
          scores: {
            property1: 1,
            property2: 1,
          },
          metadata: {
            property1: null,
            property2: null,
          },
          metrics: {
            start: 0,
            end: 0,
            property1: null,
            property2: null,
          },
          context: {
            caller_functionname: "string",
            caller_filename: "string",
            caller_lineno: 0,
            property1: null,
            property2: null,
          },
          span_attributes: {
            name: "string",
            type: "llm",
            property1: null,
            property2: null,
          },
          id: "string",
          _object_delete: true,
          _is_merge: false,
          _parent_id: "string",
        },
      ],
      feedback: [
        {
          id: "string",
          scores: {
            property1: 1,
            property2: 1,
          },
          expected: null,
          comment: "string",
          metadata: {
            property1: null,
            property2: null,
          },
          source: "app",
        },
      ],
    },
  },
});
const headers = {
  "Content-Type": "application/json",
  Accept: "application/json",
  Authorization: "Bearer {access-token}",
};
 
fetch("https://api.braintrustdata.com/v1/insert", {
  method: "POST",
  body: inputBody,
  headers: headers,
})
  .then(function (res) {
    return res.json();
  })
  .then(function (body) {
    console.log(body);
  });

POST /v1/insert

Insert events and feedback across object types

Body parameter

{
  "experiment": {
    "property1": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "dataset_record_id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "dataset_record_id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  },
  "dataset": {
    "property1": {
      "events": [
        {
          "input": null,
          "expected": null,
          "metadata": {
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "expected": null,
          "metadata": {
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  },
  "project_logs": {
    "property1": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  }
}

Parameters

Name	In	Type	Required	Description
body	body	CrossObjectInsertRequest	false	A mapping from event object type -> object id -> events to insert

Example responses

200 Response

{
  "experiment": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  },
  "dataset": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  },
  "project_logs": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  }
}

400 Response

"string"

Responses

Status	Meaning	Description	Schema
200	OK	Returns the inserted row ids for the events on each individual object	CrossObjectInsertResponse
400	Bad Request	The request was unacceptable, often due to missing a required parameter	string
401	Unauthorized	No valid API key provided	string
403	Forbidden	The API key doesn’t have permissions to perform the request	string
500	Internal Server Error	Something went wrong on Braintrust's end. (These are rare.)	string

🔒

This endpoint requires authentication.

Schemas

ExperimentId

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

Experiment id

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	Experiment id

DatasetId

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

Dataset id

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	Dataset id

ProjectId

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

Project id

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	Project id

ExperimentName

"string"

Name of the experiment to search for

Properties

Name	Type	Required	Restrictions	Description
anonymous	string	false	none	Name of the experiment to search for

DatasetName

"string"

Name of the dataset to search for

Properties

Name	Type	Required	Restrictions	Description
anonymous	string	false	none	Name of the dataset to search for

ProjectName

"string"

Name of the project to search for

Properties

Name	Type	Required	Restrictions	Description
anonymous	string	false	none	Name of the project to search for

OrgName

"string"

Filter search results to within a particular organization

Properties

Name	Type	Required	Restrictions	Description
anonymous	string	false	none	Filter search results to within a particular organization

AppLimitParam

Limit the number of objects to return

Properties

Name	Type	Required	Restrictions	Description
anonymous	integer	false	none	Limit the number of objects to return

FetchLimitParam

limit the number of traces fetched

The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

Properties

Name	Type	Required	Restrictions	Description
anonymous	integer	false	none	limit the number of traces fetched Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest `_xact_id`. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier `_xact_id`. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by `id`) from your combined result set. The `limit` parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

StartingAfter

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

Pagination cursor id.

For example, if the final item in the last page you fetched had an id of foo, pass starting_after=foo to fetch the next page. Note: you may only pass one of starting_after and ending_before

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	Pagination cursor id. For example, if the final item in the last page you fetched had an id of `foo`, pass `starting_after=foo` to fetch the next page. Note: you may only pass one of `starting_after` and `ending_before`

EndingBefore

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

Pagination cursor id.

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	Pagination cursor id. For example, if the initial item in the last page you fetched had an id of `foo`, pass `ending_before=foo` to fetch the previous page. Note: you may only pass one of `starting_after` and `ending_before`

MaxXactId

Together, max_xact_id and max_root_span_id form a pagination cursor

Properties

Name	Type	Required	Restrictions	Description
anonymous	integer(int64)	false	none	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple `(_xact_id, root_span_id)`. See the documentation of `limit` for an overview of paginating fetch queries.

MaxRootSpanId

"string"

Together, max_xact_id and max_root_span_id form a pagination cursor

Properties

Name	Type	Required	Restrictions	Description
anonymous	string	false	none	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple `(_xact_id, root_span_id)`. See the documentation of `limit` for an overview of paginating fetch queries.

Version

Retrieve a snapshot of events from a past time

The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Properties

Name	Type	Required	Restrictions	Description
anonymous	integer(int64)	false	none	Retrieve a snapshot of events from a past time The version id is essentially a filter on the latest event transaction id. You can use the `max_xact_id` returned by a past fetch as the version to reproduce that exact fetch.

SummarizeScores

true

Whether to summarize the scores and metrics. If false (or omitted), only the metadata will be returned.

Properties

Name	Type	Required	Restrictions	Description
anonymous	boolean	false	none	Whether to summarize the scores and metrics. If false (or omitted), only the metadata will be returned.

ComparisonExperimentId

"497f6eca-6276-4993-bfeb-53cbbbba6f08"

The experiment to compare against, if summarizing scores and metrics. If omitted, will fall back to the base_exp_id stored in the experiment metadata, and then to the most recent experiment run in the same project. Must pass summarize_scores=true for this id to be used

Properties

Name	Type	Required	Restrictions	Description
anonymous	string(uuid)	false	none	The experiment to compare against, if summarizing scores and metrics. If omitted, will fall back to the `base_exp_id` stored in the experiment metadata, and then to the most recent experiment run in the same project. Must pass `summarize_scores=true` for this id to be used

SummarizeData

true

Whether to summarize the data. If false (or omitted), only the metadata will be returned.

Properties

Name	Type	Required	Restrictions	Description
anonymous	boolean	false	none	Whether to summarize the data. If false (or omitted), only the metadata will be returned.

Project

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "name": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

Properties

Name	Type	Required	Restrictions	Description
id	string(uuid)	true	none	Unique identifier for the project
org_id	string(uuid)	true	none	Unique id for the organization that the project belongs under
name	string	true	none	Name of the project
created	string,null(date-time)	false	none	Date of project creation
deleted_at	string,null(date-time)	false	none	Date of project deletion, or null if the project is still active
user_id	string,null(uuid)	false	none	Identifies the user who created the project

CreateProject

{
  "name": "string",
  "org_name": "string"
}

Properties

Name	Type	Required	Restrictions	Description
name	string	true	none	Name of the project
org_name	string,null	false	none	For nearly all users, this parameter should be unnecessary. But in the rare case that your API key belongs to multiple organizations, you may specify the name of the organization the project belongs in.

PatchProject

{
  "name": "string"
}

Properties

Name	Type	Required	Restrictions	Description
name	string,null	false	none	Name of the project

InsertEventsResponse

{
  "row_ids": ["string"]
}

Properties

Name	Type	Required	Restrictions	Description
row_ids	[string]	true	none	The ids of all rows that were inserted, aligning one-to-one with the rows provided as input

InsertProjectLogsEventReplace

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The arguments that uniquely define a user input(an arbitrary, JSON serializable object).
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare logs.
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the project logs event. Use "start" and "end" to track the time span over which the project logs event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event finished
context	object,null	false	none	Context is additional information about the code that produced the project logs event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the project logs event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the project logs event
» caller_filename	string,null	false	none	Name of the file in code where the project logs event was created
» caller_lineno	integer,null	false	none	Line of code where the project logs event was created
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only
id	string,null	false	none	A unique identifier for the project logs event. If you don't provide one, BrainTrust will generate one for you
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the project logs event deleted. Deleted events will not show up in subsequent fetches for this project logs
_is_merge	boolean,null	false	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_parent_id	string,null	false	none	Use the `_parent_id` field to create this row as a subspan of an existing row. It cannot be specified alongside `_is_merge=true`. Tracking hierarchical relationships are important for tracing (see the guide for full details). For example, say we have logged a row `{"id": "abc", "input": "foo", "output": "bar", "expected": "boo", "scores": {"correctness": 0.33}}`. We can create a sub-span of the parent row by logging `{"_parent_id": "abc", "id": "llm_call", "input": {"prompt": "What comes after foo?"}, "output": "bar", "metrics": {"tokens": 1}}`. In the webapp, only the root span row `"abc"` will show up in the summary view. You can view the full trace hierarchy (in this case, the `"llm_call"` row) by clicking on the "abc" row.

Enumerated Values

Property	Value
type	llm
type	score
type	function
type	eval
type	task
type	tool
_is_merge	false

InsertProjectLogsEventMerge

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": true,
  "_merge_paths": [["string"]]
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The arguments that uniquely define a user input(an arbitrary, JSON serializable object).
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare logs.
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the project logs event. Use "start" and "end" to track the time span over which the project logs event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event finished
context	object,null	false	none	Context is additional information about the code that produced the project logs event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the project logs event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the project logs event
» caller_filename	string,null	false	none	Name of the file in code where the project logs event was created
» caller_lineno	integer,null	false	none	Line of code where the project logs event was created
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only
id	string,null	false	none	A unique identifier for the project logs event. If you don't provide one, BrainTrust will generate one for you
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the project logs event deleted. Deleted events will not show up in subsequent fetches for this project logs
_is_merge	boolean	true	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_merge_paths	array,null	false	none	The `_merge_paths` field allows controlling the depth of the merge. It can only be specified alongside `_is_merge=true`. `_merge_paths` is a list of paths, where each path is a list of field names. The deep merge will not descend below any of the specified merge paths. For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": {"b": 10}, "c": {"d": 20}}, "output": {"a": 20}}`. If we merge a new row as `{"_is_merge": true, "_merge_paths": [["input", "a"], ["output"]], "input": {"a": {"q": 30}, "c": {"e": 30}, "bar": "baz"}, "output": {"d": 40}}`, the new row will be `{"id": "foo": "input": {"a": {"q": 30}, "c": {"d": 20, "e": 30}, "bar": "baz"}, "output": {"d": 40}}`. In this case, due to the merge paths, we have replaced `input.a` and `output`, but have still deep-merged `input` and `input.c`.

Enumerated Values

Property	Value
type	llm
type	score
type	function
type	eval
type	task
type	tool
_is_merge	true

InsertProjectLogsEvent

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

A project logs event

Properties

anyOf

Name	Type	Required	Restrictions	Description
anonymous	InsertProjectLogsEventReplace	false	none	none

Name	Type	Required	Restrictions	Description
anonymous	InsertProjectLogsEventMerge	false	none	none

InsertProjectLogsEventRequest

{
  "events": [
    {
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      },
      "id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[InsertProjectLogsEvent]	true	none	A list of project logs events to insert

ProjectLogsEvent

{
  "id": "string",
  "_xact_id": 0,
  "created": "2019-08-24T14:15:22Z",
  "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "log_id": "g",
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_id": "string",
  "span_parents": ["string"],
  "root_span_id": "string",
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  }
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	A unique identifier for the project logs event. If you don't provide one, BrainTrust will generate one for you
_xact_id	integer(int64)	true	none	The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)
created	string,null(date-time)	false	none	The timestamp the project logs event was created
org_id	string(uuid)	true	none	Unique id for the organization that the project belongs under
project_id	string(uuid)	true	none	Unique identifier for the project
log_id	string	true	none	A literal 'g' which identifies the log as a project log
input	any	false	none	The arguments that uniquely define a user input(an arbitrary, JSON serializable object).
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare logs.
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the project logs event. Use "start" and "end" to track the time span over which the project logs event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the project logs event finished
context	object,null	false	none	Context is additional information about the code that produced the project logs event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the project logs event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the project logs event
» caller_filename	string,null	false	none	Name of the file in code where the project logs event was created
» caller_lineno	integer,null	false	none	Line of code where the project logs event was created
span_id	string	true	none	A unique identifier used to link different project logs events together as part of a full trace. See the tracing guide for full details on tracing
span_parents	array,null	false	none	An array of the parent `span_ids` of this project logs event. This should be empty for the root span of a trace, and should most often contain just one parent element for subspans
root_span_id	string	true	none	The `span_id` of the root of the trace this project logs event belongs to
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only

Enumerated Values

Property	Value
log_id	g
type	llm
type	score
type	function
type	eval
type	task
type	tool

FetchProjectLogsEventsResponse

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "org_id": "a40f5d1f-d889-42e9-94ea-b9b33585fc6b",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "log_id": "g",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[ProjectLogsEvent]	true	none	A list of fetched events

PathLookupFilter

{
  "type": "path_lookup",
  "path": ["string"],
  "value": null
}

A path-lookup filter describes an equality comparison against a specific sub-field in the event row. For instance, if you wish to filter on the value of c in {"input": {"a": {"b": {"c": "hello"}}}}, pass path=["input", "a", "b", "c"] and value="hello"

Properties

Name	Type	Required	Restrictions	Description
type	string	true	none	Denotes the type of filter as a path-lookup filter
path	[string]	true	none	List of fields describing the path to the value to be checked against. For instance, if you wish to filter on the value of `c` in `{"input": {"a": {"b": {"c": "hello"}}}}`, pass `path=["input", "a", "b", "c"]`
value	any	false	none	The value to compare equality-wise against the event value at the specified `path`. The value must be a "primitive", that is, any JSON-serializable object except for objects and arrays. For instance, if you wish to filter on the value of "input.a.b.c" in the object `{"input": {"a": {"b": {"c": "hello"}}}}`, pass `value="hello"`

Enumerated Values

Property	Value
type	path_lookup

FetchEventsFilters

[
  {
    "type": "path_lookup",
    "path": ["string"],
    "value": null
  }
]

A list of filters on the events to fetch. Currently, only path-lookup type filters are supported, but we may add more in the future

Properties

Name	Type	Required	Restrictions	Description
anonymous	array,null	false	none	A list of filters on the events to fetch. Currently, only path-lookup type filters are supported, but we may add more in the future

FetchEventsRequest

{
  "limit": 0,
  "max_xact_id": 0,
  "max_root_span_id": "string",
  "filters": [
    {
      "type": "path_lookup",
      "path": ["string"],
      "value": null
    }
  ],
  "version": 0
}

Properties

Name	Type	Required	Restrictions	Description
limit	integer,null	false	none	limit the number of traces fetched Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest `_xact_id`. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier `_xact_id`. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by `id`) from your combined result set. The `limit` parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.
max_xact_id	integer,null(int64)	false	none	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple `(_xact_id, root_span_id)`. See the documentation of `limit` for an overview of paginating fetch queries.
max_root_span_id	string,null	false	none	Together, `max_xact_id` and `max_root_span_id` form a pagination cursor Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple `(_xact_id, root_span_id)`. See the documentation of `limit` for an overview of paginating fetch queries.
filters	FetchEventsFilters	false	none	A list of filters on the events to fetch. Currently, only path-lookup type filters are supported, but we may add more in the future
version	integer,null(int64)	false	none	Retrieve a snapshot of events from a past time The version id is essentially a filter on the latest event transaction id. You can use the `max_xact_id` returned by a past fetch as the version to reproduce that exact fetch.

FeedbackProjectLogsItem

{
  "id": "string",
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "expected": null,
  "comment": "string",
  "metadata": {
    "property1": null,
    "property2": null
  },
  "source": "app"
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	The id of the project logs event to log feedback for. This is the row `id` returned by `POST /v1/project_logs/{project_id}/insert`
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the project logs event
» additionalProperties	number,null	false	none	none
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not
comment	string,null	false	none	An optional comment string to log about the project logs event
metadata	object,null	false	none	A dictionary with additional data about the feedback. If you have a `user_id`, you can log it here and access it in the Braintrust UI.
» additionalProperties	any	false	none	none
source	string,null	false	none	The source of the feedback. Must be one of "external" (default), "app", or "api"

Enumerated Values

Property	Value
source	app
source	api
source	external

FeedbackProjectLogsEventRequest

{
  "feedback": [
    {
      "id": "string",
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "expected": null,
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
feedback	[FeedbackProjectLogsItem]	true	none	A list of project logs feedback items

RepoInfo

{
  "commit": "string",
  "branch": "string",
  "tag": "string",
  "dirty": true,
  "author_name": "string",
  "author_email": "string",
  "commit_message": "string",
  "commit_time": "string",
  "git_diff": "string"
}

Metadata about the state of the repo when the experiment was created

Properties

Name	Type	Required	Restrictions	Description
anonymous	object,null	false	none	Metadata about the state of the repo when the experiment was created
commit	string,null	false	none	SHA of most recent commit
branch	string,null	false	none	Name of the branch the most recent commit belongs to
tag	string,null	false	none	Name of the tag on the most recent commit
dirty	boolean,null	false	none	Whether or not the repo had uncommitted changes when snapshotted
author_name	string,null	false	none	Name of the author of the most recent commit
author_email	string,null	false	none	Email of the author of the most recent commit
commit_message	string,null	false	none	Most recent commit message
commit_time	string,null	false	none	Time of the most recent commit
git_diff	string,null	false	none	If the repo was dirty when run, this includes the diff between the current state of the repo and the most recent commit.

Experiment

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "commit": "string",
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "deleted_at": "2019-08-24T14:15:22Z",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Properties

Name	Type	Required	Restrictions	Description
id	string(uuid)	true	none	Unique identifier for the experiment
project_id	string(uuid)	true	none	Unique identifier for the project that the experiment belongs under
name	string	true	none	Name of the experiment. Within a project, experiment names are unique
description	string,null	false	none	Textual description of the experiment
created	string,null(date-time)	false	none	Date of experiment creation
repo_info	RepoInfo	false	none	Metadata about the state of the repo when the experiment was created
commit	string,null	false	none	Commit, taken directly from `repo_info.commit`
base_exp_id	string,null(uuid)	false	none	Id of default base experiment to compare against when viewing this experiment
deleted_at	string,null(date-time)	false	none	Date of experiment deletion, or null if the experiment is still active
dataset_id	string,null(uuid)	false	none	Identifier of the linked dataset, or null if the experiment is not linked to a dataset
dataset_version	string,null	false	none	Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
public	boolean	true	none	Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
user_id	string,null(uuid)	false	none	Identifies the user who created the experiment
metadata	object,null	false	none	User-controlled metadata about the experiment
» additionalProperties	any	false	none	none

CreateExperiment

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Properties

Name	Type	Required	Restrictions	Description
project_id	string(uuid)	true	none	Unique identifier for the project that the experiment belongs under
name	string,null	false	none	Name of the experiment. Within a project, experiment names are unique
description	string,null	false	none	Textual description of the experiment
repo_info	RepoInfo	false	none	Metadata about the state of the repo when the experiment was created
base_exp_id	string,null(uuid)	false	none	Id of default base experiment to compare against when viewing this experiment
dataset_id	string,null(uuid)	false	none	Identifier of the linked dataset, or null if the experiment is not linked to a dataset
dataset_version	string,null	false	none	Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
public	boolean,null	false	none	Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
metadata	object,null	false	none	User-controlled metadata about the experiment
» additionalProperties	any	false	none	none

PatchExperiment

{
  "name": "string",
  "description": "string",
  "repo_info": {
    "commit": "string",
    "branch": "string",
    "tag": "string",
    "dirty": true,
    "author_name": "string",
    "author_email": "string",
    "commit_message": "string",
    "commit_time": "string",
    "git_diff": "string"
  },
  "base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "dataset_version": "string",
  "public": true,
  "metadata": {
    "property1": null,
    "property2": null
  }
}

Properties

Name	Type	Required	Restrictions	Description
name	string,null	false	none	Name of the experiment. Within a project, experiment names are unique
description	string,null	false	none	Textual description of the experiment
repo_info	RepoInfo	false	none	Metadata about the state of the repo when the experiment was created
base_exp_id	string,null(uuid)	false	none	Id of default base experiment to compare against when viewing this experiment
dataset_id	string,null(uuid)	false	none	Identifier of the linked dataset, or null if the experiment is not linked to a dataset
dataset_version	string,null	false	none	Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
public	boolean,null	false	none	Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
metadata	object,null	false	none	User-controlled metadata about the experiment
» additionalProperties	any	false	none	none

InsertExperimentEventReplace

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "dataset_record_id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The arguments that uniquely define a test case (an arbitrary, JSON serializable object). Later on, Braintrust will use the `input` to know whether two test cases are the same between experiments, so they should not contain experiment-specific state. A simple rule of thumb is that if you run the same experiment twice, the `input` should be identical
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate your experiments while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare experiments
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the experiment event. Use "start" and "end" to track the time span over which the experiment event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event finished
context	object,null	false	none	Context is additional information about the code that produced the experiment event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the experiment event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the experiment event
» caller_filename	string,null	false	none	Name of the file in code where the experiment event was created
» caller_lineno	integer,null	false	none	Line of code where the experiment event was created
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only
id	string,null	false	none	A unique identifier for the experiment event. If you don't provide one, BrainTrust will generate one for you
dataset_record_id	string,null	false	none	If the experiment is associated to a dataset, this is the event-level dataset id this experiment event is tied to
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the experiment event deleted. Deleted events will not show up in subsequent fetches for this experiment
_is_merge	boolean,null	false	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_parent_id	string,null	false	none	Use the `_parent_id` field to create this row as a subspan of an existing row. It cannot be specified alongside `_is_merge=true`. Tracking hierarchical relationships are important for tracing (see the guide for full details). For example, say we have logged a row `{"id": "abc", "input": "foo", "output": "bar", "expected": "boo", "scores": {"correctness": 0.33}}`. We can create a sub-span of the parent row by logging `{"_parent_id": "abc", "id": "llm_call", "input": {"prompt": "What comes after foo?"}, "output": "bar", "metrics": {"tokens": 1}}`. In the webapp, only the root span row `"abc"` will show up in the summary view. You can view the full trace hierarchy (in this case, the `"llm_call"` row) by clicking on the "abc" row.

Enumerated Values

Property	Value
type	llm
type	score
type	function
type	eval
type	task
type	tool
_is_merge	false

InsertExperimentEventMerge

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "dataset_record_id": "string",
  "_object_delete": true,
  "_is_merge": true,
  "_merge_paths": [["string"]]
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The arguments that uniquely define a test case (an arbitrary, JSON serializable object). Later on, Braintrust will use the `input` to know whether two test cases are the same between experiments, so they should not contain experiment-specific state. A simple rule of thumb is that if you run the same experiment twice, the `input` should be identical
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate your experiments while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare experiments
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the experiment event. Use "start" and "end" to track the time span over which the experiment event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event finished
context	object,null	false	none	Context is additional information about the code that produced the experiment event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the experiment event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the experiment event
» caller_filename	string,null	false	none	Name of the file in code where the experiment event was created
» caller_lineno	integer,null	false	none	Line of code where the experiment event was created
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only
id	string,null	false	none	A unique identifier for the experiment event. If you don't provide one, BrainTrust will generate one for you
dataset_record_id	string,null	false	none	If the experiment is associated to a dataset, this is the event-level dataset id this experiment event is tied to
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the experiment event deleted. Deleted events will not show up in subsequent fetches for this experiment
_is_merge	boolean	true	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_merge_paths	array,null	false	none	The `_merge_paths` field allows controlling the depth of the merge. It can only be specified alongside `_is_merge=true`. `_merge_paths` is a list of paths, where each path is a list of field names. The deep merge will not descend below any of the specified merge paths. For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": {"b": 10}, "c": {"d": 20}}, "output": {"a": 20}}`. If we merge a new row as `{"_is_merge": true, "_merge_paths": [["input", "a"], ["output"]], "input": {"a": {"q": 30}, "c": {"e": 30}, "bar": "baz"}, "output": {"d": 40}}`, the new row will be `{"id": "foo": "input": {"a": {"q": 30}, "c": {"d": 20, "e": 30}, "bar": "baz"}, "output": {"d": 40}}`. In this case, due to the merge paths, we have replaced `input.a` and `output`, but have still deep-merged `input` and `input.c`.

Enumerated Values

Property	Value
type	llm
type	score
type	function
type	eval
type	task
type	tool
_is_merge	true

InsertExperimentEvent

{
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  },
  "id": "string",
  "dataset_record_id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

An experiment event

Properties

anyOf

Name	Type	Required	Restrictions	Description
anonymous	InsertExperimentEventReplace	false	none	none

Name	Type	Required	Restrictions	Description
anonymous	InsertExperimentEventMerge	false	none	none

InsertExperimentEventRequest

{
  "events": [
    {
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      },
      "id": "string",
      "dataset_record_id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[InsertExperimentEvent]	true	none	A list of experiment events to insert

ExperimentEvent

{
  "id": "string",
  "dataset_record_id": "string",
  "_xact_id": 0,
  "created": "2019-08-24T14:15:22Z",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
  "input": null,
  "output": null,
  "expected": null,
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "metadata": {
    "property1": null,
    "property2": null
  },
  "metrics": {
    "start": 0,
    "end": 0,
    "property1": null,
    "property2": null
  },
  "context": {
    "caller_functionname": "string",
    "caller_filename": "string",
    "caller_lineno": 0,
    "property1": null,
    "property2": null
  },
  "span_id": "string",
  "span_parents": ["string"],
  "root_span_id": "string",
  "span_attributes": {
    "name": "string",
    "type": "llm",
    "property1": null,
    "property2": null
  }
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	A unique identifier for the experiment event. If you don't provide one, BrainTrust will generate one for you
dataset_record_id	string,null	false	none	If the experiment is associated to a dataset, this is the event-level dataset id this experiment event is tied to
_xact_id	integer(int64)	true	none	The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the experiment (see the `version` parameter)
created	string,null(date-time)	false	none	The timestamp the experiment event was created
project_id	string(uuid)	true	none	Unique identifier for the project that the experiment belongs under
experiment_id	string(uuid)	true	none	Unique identifier for the experiment
input	any	false	none	The arguments that uniquely define a test case (an arbitrary, JSON serializable object). Later on, Braintrust will use the `input` to know whether two test cases are the same between experiments, so they should not contain experiment-specific state. A simple rule of thumb is that if you run the same experiment twice, the `input` should be identical
output	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate your experiments while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare experiments
» additionalProperties	number,null	false	none	none
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
metrics	object,null	false	none	Metrics are numerical measurements tracking the execution of the code that produced the experiment event. Use "start" and "end" to track the time span over which the experiment event was produced
» additionalProperties	any	false	none	none
» start	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event started
» end	number,null	false	none	A unix timestamp recording when the section of code which produced the experiment event finished
context	object,null	false	none	Context is additional information about the code that produced the experiment event. It is essentially the textual counterpart to `metrics`. Use the `caller_*` attributes to track the location in code which produced the experiment event
» additionalProperties	any	false	none	none
» caller_functionname	string,null	false	none	The function in code which created the experiment event
» caller_filename	string,null	false	none	Name of the file in code where the experiment event was created
» caller_lineno	integer,null	false	none	Line of code where the experiment event was created
span_id	string	true	none	A unique identifier used to link different experiment events together as part of a full trace. See the tracing guide for full details on tracing
span_parents	array,null	false	none	An array of the parent `span_ids` of this experiment event. This should be empty for the root span of a trace, and should most often contain just one parent element for subspans
root_span_id	string	true	none	The `span_id` of the root of the trace this experiment event belongs to
span_attributes	object,null	false	none	Human-identifying attributes of the span, such as name, type, etc.
» additionalProperties	any	false	none	none
» name	string,null	false	none	Name of the span, for display purposes only
» type	string,null	false	none	Type of the span, for display purposes only

Enumerated Values

Property	Value
type	llm
type	score
type	function
type	eval
type	task
type	tool

FetchExperimentEventsResponse

{
  "events": [
    {
      "id": "string",
      "dataset_record_id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
      "input": null,
      "output": null,
      "expected": null,
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "metadata": {
        "property1": null,
        "property2": null
      },
      "metrics": {
        "start": 0,
        "end": 0,
        "property1": null,
        "property2": null
      },
      "context": {
        "caller_functionname": "string",
        "caller_filename": "string",
        "caller_lineno": 0,
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "span_parents": ["string"],
      "root_span_id": "string",
      "span_attributes": {
        "name": "string",
        "type": "llm",
        "property1": null,
        "property2": null
      }
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[ExperimentEvent]	true	none	A list of fetched events

FeedbackExperimentItem

{
  "id": "string",
  "scores": {
    "property1": 1,
    "property2": 1
  },
  "expected": null,
  "comment": "string",
  "metadata": {
    "property1": null,
    "property2": null
  },
  "source": "app"
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	The id of the experiment event to log feedback for. This is the row `id` returned by `POST /v1/experiment/{experiment_id}/insert`
scores	object,null	false	none	A dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the experiment event
» additionalProperties	number,null	false	none	none
expected	any	false	none	The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not
comment	string,null	false	none	An optional comment string to log about the experiment event
metadata	object,null	false	none	A dictionary with additional data about the feedback. If you have a `user_id`, you can log it here and access it in the Braintrust UI.
» additionalProperties	any	false	none	none
source	string,null	false	none	The source of the feedback. Must be one of "external" (default), "app", or "api"

Enumerated Values

Property	Value
source	app
source	api
source	external

FeedbackExperimentEventRequest

{
  "feedback": [
    {
      "id": "string",
      "scores": {
        "property1": 1,
        "property2": 1
      },
      "expected": null,
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
feedback	[FeedbackExperimentItem]	true	none	A list of experiment feedback items

ScoreSummary

{
  "name": "string",
  "score": 1,
  "diff": -1,
  "improvements": 0,
  "regressions": 0
}

Summary of a score's performance

Properties

Name	Type	Required	Restrictions	Description
name	string	true	none	Name of the score
score	number	true	none	Average score across all examples
diff	number	true	none	Difference in score between the current and comparison experiment
improvements	integer	true	none	Number of improvements in the score
regressions	integer	true	none	Number of regressions in the score

MetricSummary

{
  "name": "string",
  "metric": 0,
  "unit": "string",
  "diff": 0,
  "improvements": 0,
  "regressions": 0
}

Summary of a metric's performance

Properties

Name	Type	Required	Restrictions	Description
name	string	true	none	Name of the metric
metric	number	true	none	Average metric across all examples
unit	string	true	none	Unit label for the metric
diff	number	true	none	Difference in metric between the current and comparison experiment
improvements	integer	true	none	Number of improvements in the metric
regressions	integer	true	none	Number of regressions in the metric

SummarizeExperimentResponse

{
  "project_name": "string",
  "experiment_name": "string",
  "project_url": "http://example.com",
  "experiment_url": "http://example.com",
  "comparison_experiment_name": "string",
  "scores": {
    "property1": {
      "name": "string",
      "score": 1,
      "diff": -1,
      "improvements": 0,
      "regressions": 0
    },
    "property2": {
      "name": "string",
      "score": 1,
      "diff": -1,
      "improvements": 0,
      "regressions": 0
    }
  },
  "metrics": {
    "property1": {
      "name": "string",
      "metric": 0,
      "unit": "string",
      "diff": 0,
      "improvements": 0,
      "regressions": 0
    },
    "property2": {
      "name": "string",
      "metric": 0,
      "unit": "string",
      "diff": 0,
      "improvements": 0,
      "regressions": 0
    }
  }
}

Summary of an experiment

Properties

Name	Type	Required	Restrictions	Description
project_name	string	true	none	Name of the project that the experiment belongs to
experiment_name	string	true	none	Name of the experiment
project_url	string(uri)	true	none	URL to the project's page in the Braintrust app
experiment_url	string(uri)	true	none	URL to the experiment's page in the Braintrust app
comparison_experiment_name	string,null	false	none	The experiment which scores are baselined against
scores	object,null	false	none	Summary of the experiment's scores
» additionalProperties	ScoreSummary	false	none	Summary of a score's performance
metrics	object,null	false	none	Summary of the experiment's metrics
» additionalProperties	MetricSummary	false	none	Summary of a metric's performance

Dataset

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string",
  "created": "2019-08-24T14:15:22Z",
  "deleted_at": "2019-08-24T14:15:22Z",
  "user_id": "a169451c-8525-4352-b8ca-070dd449a1a5"
}

Properties

Name	Type	Required	Restrictions	Description
id	string(uuid)	true	none	Unique identifier for the dataset
project_id	string,null(uuid)	false	none	Unique identifier for the project that the dataset belongs under
name	string	true	none	Name of the dataset. Within a project, dataset names are unique
description	string,null	false	none	Textual description of the dataset
created	string,null(date-time)	false	none	Date of dataset creation
deleted_at	string,null(date-time)	false	none	Date of dataset deletion, or null if the dataset is still active
user_id	string,null(uuid)	false	none	Identifies the user who created the dataset

CreateDataset

{
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "name": "string",
  "description": "string"
}

Properties

Name	Type	Required	Restrictions	Description
project_id	string,null(uuid)	false	none	Unique identifier for the project that the dataset belongs under
name	string	true	none	Name of the dataset. Within a project, dataset names are unique
description	string,null	false	none	Textual description of the dataset

PatchDataset

{
  "name": "string",
  "description": "string"
}

Properties

Name	Type	Required	Restrictions	Description
name	string	true	none	Name of the dataset. Within a project, dataset names are unique
description	string,null	false	none	Textual description of the dataset

InsertDatasetEventReplace

{
  "input": null,
  "expected": null,
  "metadata": {
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The argument that uniquely define an input case (an arbitrary, JSON serializable object)
expected	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object)
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
id	string,null	false	none	A unique identifier for the dataset event. If you don't provide one, BrainTrust will generate one for you
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the dataset event deleted. Deleted events will not show up in subsequent fetches for this dataset
_is_merge	boolean,null	false	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_parent_id	string,null	false	none	Use the `_parent_id` field to create this row as a subspan of an existing row. It cannot be specified alongside `_is_merge=true`. Tracking hierarchical relationships are important for tracing (see the guide for full details). For example, say we have logged a row `{"id": "abc", "input": "foo", "output": "bar", "expected": "boo", "scores": {"correctness": 0.33}}`. We can create a sub-span of the parent row by logging `{"_parent_id": "abc", "id": "llm_call", "input": {"prompt": "What comes after foo?"}, "output": "bar", "metrics": {"tokens": 1}}`. In the webapp, only the root span row `"abc"` will show up in the summary view. You can view the full trace hierarchy (in this case, the `"llm_call"` row) by clicking on the "abc" row.

Enumerated Values

Property	Value
_is_merge	false

InsertDatasetEventMerge

{
  "input": null,
  "expected": null,
  "metadata": {
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": true,
  "_merge_paths": [["string"]]
}

Properties

Name	Type	Required	Restrictions	Description
input	any	false	none	The argument that uniquely define an input case (an arbitrary, JSON serializable object)
expected	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object)
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
id	string,null	false	none	A unique identifier for the dataset event. If you don't provide one, BrainTrust will generate one for you
_object_delete	boolean,null	false	none	Pass `_object_delete=true` to mark the dataset event deleted. Deleted events will not show up in subsequent fetches for this dataset
_is_merge	boolean	true	none	The `_is_merge` field controls how the row is merged with any existing row with the same id in the DB. By default (or when set to `false`), the existing row is completely replaced by the new row. When set to `true`, the new row is deep-merged into the existing row For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": 5, "b": 10}}`. If we merge a new row as `{"_is_merge": true, "id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"a": 5, "b": 11, "c": 20}}`. If we replace the new row as `{"id": "foo", "input": {"b": 11, "c": 20}}`, the new row will be `{"id": "foo", "input": {"b": 11, "c": 20}}`
_merge_paths	array,null	false	none	The `_merge_paths` field allows controlling the depth of the merge. It can only be specified alongside `_is_merge=true`. `_merge_paths` is a list of paths, where each path is a list of field names. The deep merge will not descend below any of the specified merge paths. For example, say there is an existing row in the DB `{"id": "foo", "input": {"a": {"b": 10}, "c": {"d": 20}}, "output": {"a": 20}}`. If we merge a new row as `{"_is_merge": true, "_merge_paths": [["input", "a"], ["output"]], "input": {"a": {"q": 30}, "c": {"e": 30}, "bar": "baz"}, "output": {"d": 40}}`, the new row will be `{"id": "foo": "input": {"a": {"q": 30}, "c": {"d": 20, "e": 30}, "bar": "baz"}, "output": {"d": 40}}`. In this case, due to the merge paths, we have replaced `input.a` and `output`, but have still deep-merged `input` and `input.c`.

Enumerated Values

Property	Value
_is_merge	true

InsertDatasetEvent

{
  "input": null,
  "expected": null,
  "metadata": {
    "property1": null,
    "property2": null
  },
  "id": "string",
  "_object_delete": true,
  "_is_merge": false,
  "_parent_id": "string"
}

A dataset event

Properties

anyOf

Name	Type	Required	Restrictions	Description
anonymous	InsertDatasetEventReplace	false	none	none

Name	Type	Required	Restrictions	Description
anonymous	InsertDatasetEventMerge	false	none	none

InsertDatasetEventRequest

{
  "events": [
    {
      "input": null,
      "expected": null,
      "metadata": {
        "property1": null,
        "property2": null
      },
      "id": "string",
      "_object_delete": true,
      "_is_merge": false,
      "_parent_id": "string"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[InsertDatasetEvent]	true	none	A list of dataset events to insert

DatasetEvent

{
  "id": "string",
  "_xact_id": 0,
  "created": "2019-08-24T14:15:22Z",
  "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
  "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
  "input": null,
  "expected": null,
  "metadata": {
    "property1": null,
    "property2": null
  },
  "span_id": "string",
  "root_span_id": "string"
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	A unique identifier for the dataset event. If you don't provide one, BrainTrust will generate one for you
_xact_id	integer(int64)	true	none	The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the dataset (see the `version` parameter)
created	string,null(date-time)	false	none	The timestamp the dataset event was created
project_id	string,null(uuid)	false	none	Unique identifier for the project that the dataset belongs under
dataset_id	string(uuid)	true	none	Unique identifier for the dataset
input	any	false	none	The argument that uniquely define an input case (an arbitrary, JSON serializable object)
expected	any	false	none	The output of your application, including post-processing (an arbitrary, JSON serializable object)
metadata	object,null	false	none	A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the `prompt`, example's `id`, or anything else that would be useful to slice/dice later. The values in `metadata` can be any JSON-serializable type, but its keys must be strings
» additionalProperties	any	false	none	none
span_id	string	true	none	A unique identifier used to link different dataset events together as part of a full trace. See the tracing guide for full details on tracing
root_span_id	string	true	none	The `span_id` of the root of the trace this dataset event belongs to

FetchDatasetEventsResponse

{
  "events": [
    {
      "id": "string",
      "_xact_id": 0,
      "created": "2019-08-24T14:15:22Z",
      "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
      "dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
      "input": null,
      "expected": null,
      "metadata": {
        "property1": null,
        "property2": null
      },
      "span_id": "string",
      "root_span_id": "string"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
events	[DatasetEvent]	true	none	A list of fetched events

FeedbackDatasetItem

{
  "id": "string",
  "comment": "string",
  "metadata": {
    "property1": null,
    "property2": null
  },
  "source": "app"
}

Properties

Name	Type	Required	Restrictions	Description
id	string	true	none	The id of the dataset event to log feedback for. This is the row `id` returned by `POST /v1/dataset/{dataset_id}/insert`
comment	string,null	false	none	An optional comment string to log about the dataset event
metadata	object,null	false	none	A dictionary with additional data about the feedback. If you have a `user_id`, you can log it here and access it in the Braintrust UI.
» additionalProperties	any	false	none	none
source	string,null	false	none	The source of the feedback. Must be one of "external" (default), "app", or "api"

Enumerated Values

Property	Value
source	app
source	api
source	external

FeedbackDatasetEventRequest

{
  "feedback": [
    {
      "id": "string",
      "comment": "string",
      "metadata": {
        "property1": null,
        "property2": null
      },
      "source": "app"
    }
  ]
}

Properties

Name	Type	Required	Restrictions	Description
feedback	[FeedbackDatasetItem]	true	none	A list of dataset feedback items

DataSummary

{
  "total_records": 0
}

Summary of a dataset's data

Properties

Name	Type	Required	Restrictions	Description
anonymous	object,null	false	none	Summary of a dataset's data
total_records	integer	true	none	Total number of records in the dataset

SummarizeDatasetResponse

{
  "project_name": "string",
  "dataset_name": "string",
  "project_url": "http://example.com",
  "dataset_url": "http://example.com",
  "data_summary": {
    "total_records": 0
  }
}

Summary of a dataset

Properties

Name	Type	Required	Restrictions	Description
project_name	string	true	none	Name of the project that the dataset belongs to
dataset_name	string	true	none	Name of the dataset
project_url	string(uri)	true	none	URL to the project's page in the Braintrust app
dataset_url	string(uri)	true	none	URL to the dataset's page in the Braintrust app
data_summary	DataSummary	false	none	Summary of a dataset's data

CrossObjectInsertResponse

{
  "experiment": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  },
  "dataset": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  },
  "project_logs": {
    "property1": {
      "row_ids": ["string"]
    },
    "property2": {
      "row_ids": ["string"]
    }
  }
}

Properties

Name	Type	Required	Restrictions	Description
experiment	object,null	false	none	A mapping from experiment id to row ids for inserted `events`
» additionalProperties	InsertEventsResponse	false	none	none
dataset	object,null	false	none	A mapping from dataset id to row ids for inserted `events`
» additionalProperties	InsertEventsResponse	false	none	none
project_logs	object,null	false	none	A mapping from project id to row ids for inserted `events`
» additionalProperties	InsertEventsResponse	false	none	none

CrossObjectInsertRequest

{
  "experiment": {
    "property1": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "dataset_record_id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "dataset_record_id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  },
  "dataset": {
    "property1": {
      "events": [
        {
          "input": null,
          "expected": null,
          "metadata": {
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "expected": null,
          "metadata": {
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  },
  "project_logs": {
    "property1": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    },
    "property2": {
      "events": [
        {
          "input": null,
          "output": null,
          "expected": null,
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "metadata": {
            "property1": null,
            "property2": null
          },
          "metrics": {
            "start": 0,
            "end": 0,
            "property1": null,
            "property2": null
          },
          "context": {
            "caller_functionname": "string",
            "caller_filename": "string",
            "caller_lineno": 0,
            "property1": null,
            "property2": null
          },
          "span_attributes": {
            "name": "string",
            "type": "llm",
            "property1": null,
            "property2": null
          },
          "id": "string",
          "_object_delete": true,
          "_is_merge": false,
          "_parent_id": "string"
        }
      ],
      "feedback": [
        {
          "id": "string",
          "scores": {
            "property1": 1,
            "property2": 1
          },
          "expected": null,
          "comment": "string",
          "metadata": {
            "property1": null,
            "property2": null
          },
          "source": "app"
        }
      ]
    }
  }
}

Properties

Name	Type	Required	Restrictions	Description
experiment	object,null	false	none	A mapping from experiment id to a set of log events and feedback items to insert
» additionalProperties	object	false	none	none
»» events	array,null	false	none	A list of experiment events to insert
»» feedback	array,null	false	none	A list of experiment feedback items
dataset	object,null	false	none	A mapping from dataset id to a set of log events and feedback items to insert
» additionalProperties	object	false	none	none
»» events	array,null	false	none	A list of dataset events to insert
»» feedback	array,null	false	none	A list of dataset feedback items
project_logs	object,null	false	none	A mapping from project id to a set of log events and feedback items to insert
» additionalProperties	object	false	none	none
»» events	array,null	false	none	A list of project logs events to insert
»» feedback	array,null	false	none	A list of project logs feedback items

Span Walkthrough