Please read the LangSmith architectural overview and guide on connecting to external ClickHouse before proceeding with this guide.
Architecture Overview
The architecture of using LangSmith-managed ClickHouse with your self-hosted LangSmith instance is similar to using a fully self-hosted ClickHouse instance, with a few key differences:- You will need to set up a private network connection between your LangSmith instance and the LangSmith-managed ClickHouse instance. This is to ensure that your data is secure and that you can connect to the ClickHouse instance from your self-hosted LangSmith instance.
- With this option, sensitive information (inputs and outputs) of your traces will be stored in cloud object storage (S3 or GCS) within your cloud instead of ClickHouse to ensure that sensitive information doesn’t leave your VPC. For more details on where particular data fields are stored, refer to Data storage.
- The LangSmith team will monitor your ClickHouse instance and ensure that it is running smoothly. This allows us to track metrics like run-ingestion delay and query performance.
Requirements
- You must use a supported blob storage option. Read the blob storage guide for more information.
- To use private endpoints, ensure that your VPC is in a ClickHouse Cloud supported region. Otherwise, you will need to use a public endpoint we will secure with firewall rules. Your VPC will need to have a NAT gateway to allow us to allowlist your traffic.
- You must have a VPC that can connect to the LangSmith-managed ClickHouse service. You will need to work with our team to set up the necessary networking.
- You must have a LangSmith self-hosted instance running. You can use our managed ClickHouse service with both Kubernetes and Docker installations.
Data storage
ClickHouse stores runs and feedback data, specifically:- All feedback data fields.
- Some run data fields.
inputs, outputs, errors, manifests, extras, and events of a run, since these fields may contain LLM prompts and completions. With LangSmith-managed ClickHouse, these sensitive fields are stored in cloud object storage (S3 or GCS) within your cloud, while the rest of the run data is stored in ClickHouse, ensuring sensitive information never leaves your VPC.
Stored feedback data fields
Because all feedback data is stored in ClickHouse, do not send sensitive information in feedback (scores and annotations/comments) or in any other run fields that are mentioned in Stored run data fields.
| Field Name | Type | Description |
|---|---|---|
| id | UUID | Unique identifier for the record itself |
| created_at | datetime | Timestamp when the record was created |
| modified_at | datetime | Timestamp when the record was last modified |
| session_id | UUID | Unique identifier for the experiment or tracing project the run was a part of |
| run_id | UUID | Unique identifier for a specific run within a session |
| key | string | A key describing the criteria of the feedback, eg “correctness” |
| score | number | Numerical score associated with the feedback key |
| value | string | Reserved for storing a value associated with the score. Useful for categorical feedback. |
| comment | string | Any comment or annotation associated with the record. This can be a justification for the score given. |
| correction | object | Reserved for storing correction details, if any |
| feedback_source | object | Object containing information about the feedback source |
| feedback_source.type | string | The type of source where the feedback originated, eg “api”, “app”, “evaluator” |
| feedback_source.metadata | object | Reserved for additional metadata, currently |
| feedback_source.user_id | UUID | Unique identifier for the user providing feedback |
Stored run data fields
Run data fields are split between the managed ClickHouse database and your cloud object storage (e.g., S3 or GCS).For run fields stored in object storage, only a reference or pointer is kept in ClickHouse. For example,
inputs and outputs content are offloaded to S3/GCS, with the ClickHouse record storing corresponding S3 URLs in the inputs_s3_urls and outputs_s3_urls fields.| Field | Storage Location |
|---|---|
id | ClickHouse |
name | ClickHouse |
inputs | Object Storage |
run_type | ClickHouse |
start_time | ClickHouse |
end_time | ClickHouse |
extra | Object Storage |
error | Object Storage |
outputs | Object Storage |
events | Object Storage |
tags | ClickHouse |
trace_id | ClickHouse |
dotted_order | ClickHouse |
status | ClickHouse |
child_run_ids | ClickHouse |
direct_child_run_ids | ClickHouse |
parent_run_ids | ClickHouse |
feedback_stats | ClickHouse |
reference_example_id | ClickHouse |
total_tokens | ClickHouse |
prompt_tokens | ClickHouse |
completion_tokens | ClickHouse |
total_cost | ClickHouse |
prompt_cost | ClickHouse |
completion_cost | ClickHouse |
first_token_time | ClickHouse |
session_id | ClickHouse |
in_dataset | ClickHouse |
parent_run_id | ClickHouse |
execution_order (deprecated) | ClickHouse |
serialized | ClickHouse |
manifest_id (deprecated) | ClickHouse |
manifest_s3_id | ClickHouse |
inputs_s3_urls | ClickHouse |
outputs_s3_urls | ClickHouse |
price_model_id | ClickHouse |
app_path | ClickHouse |
last_queued_at | ClickHouse |
share_token | ClickHouse |