Integrate LangSmith
What is LangSmith
LangSmith is a platform for building production-grade LLM applications. It is used for developing, collaborating, testing, deploying, and monitoring LLM applications.
For more details, please refer to LangSmith.
How to Configure LangSmith
1. Register/Login to LangSmith
2. Create a Project
Create a project in LangSmith. After logging in, click New Project on the homepage to create your own project. The project will be used to associate with applications in Dify for data monitoring.
Once created, you can view all created projects in the Projects section.
3. Create Project Credentials
Find the project settings Settings in the left sidebar.
Click Create API Key to create project credentials.
Select Personal Access Token for subsequent API authentication.
Copy and save the created API key.
4. Integrating LangSmith with Dify
Configure LangSmith in the Dify application. Open the application you need to monitor, open Monitoring in the side menu, and select Tracing app performance on the page.
After clicking configure, paste the API Key and project name created in LangSmith into the configuration and save.
The configured project name needs to match the project set in LangSmith. If the project names do not match, LangSmith will automatically create a new project during data synchronization.
Once successfully saved, you can view the monitoring status on the current page.
Viewing Monitoring Data in LangSmith
Once configured, the debug or production data from applications within Dify can be monitored in LangSmith.
When you switch to LangSmith, you can view detailed operation logs of Dify applications in the dashboard.
Detailed LLM operation logs through LangSmith will help you optimize the performance of your Dify application.
Monitoring Data List
Workflow/Chatflow Trace Information
Used to track workflows and chatflows
| Workflow | LangSmith Chain |
|---|---|
workflow_app_log_id/workflow_run_id | id |
user_session_id | - placed in metadata |
workflow_{id} | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
Model token consumption | usage_metadata |
metadata | extra |
error | error |
[workflow] | tags |
"conversation_id/none for workflow" | conversation_id in metadata |
conversion_id | parent_run_id |
Workflow Trace Info
workflow_id - Unique identifier of the workflow
conversation_id - Conversation ID
workflow_run_id - ID of the current run
tenant_id - Tenant ID
elapsed_time - Time taken for the current run
status - Run status
version - Workflow version
total_tokens - Total tokens used in the current run
file_list - List of processed files
triggered_from - Source that triggered the current run
workflow_run_inputs - Input data for the current run
workflow_run_outputs - Output data for the current run
error - Errors encountered during the current run
query - Query used during the run
workflow_app_log_id - Workflow application log ID
message_id - Associated message ID
start_time - Start time of the run
end_time - End time of the run
workflow node executions - Information about workflow node executions
Metadata
workflow_id - Unique identifier of the workflow
conversation_id - Conversation ID
workflow_run_id - ID of the current run
tenant_id - Tenant ID
elapsed_time - Time taken for the current run
status - Run status
version - Workflow version
total_tokens - Total tokens used in the current run
file_list - List of processed files
triggered_from - Source that triggered the current run
Message Trace Information
Used to track LLM-related conversations
| Chat | LangSmith LLM |
|---|---|
message_id | id |
user_session_id | - placed in metadata |
“message_{id}" | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
Model token consumption | usage_metadata |
metadata | extra |
error | error |
["message", conversation_mode] | tags |
conversation_id | conversation_id in metadata |
conversion_id | parent_run_id |
Message Trace Info
message_id - Message ID
message_data - Message data
user_session_id - User session ID
conversation_model - Conversation mode
message_tokens - Number of tokens in the message
answer_tokens - Number of tokens in the answer
total_tokens - Total number of tokens in the message and answer
error - Error information
inputs - Input data
outputs - Output data
file_list - List of processed files
start_time - Start time
end_time - End time
message_file_data - File data associated with the message
conversation_mode - Conversation mode
Metadata
conversation_id - Conversation ID
ls_provider - Model provider
ls_model_name - Model ID
status - Message status
from_end_user_id - ID of the sending user
from_account_id - ID of the sending account
agent_based - Whether the message is agent-based
workflow_run_id - Workflow run ID
from_source - Message source
Moderation Trace Information
Used to track conversation moderation
| Moderation | LangSmith Tool |
|---|---|
user_id | - placed in metadata |
“moderation" | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
metadata | extra |
[moderation] | tags |
message_id | parent_run_id |
Moderation Trace Info
message_id - Message ID
user_id: User ID
workflow_app_log_id - Workflow application log ID
inputs - Moderation input data
message_data - Message data
flagged - Whether the content is flagged for attention
action - Specific actions taken
preset_response - Preset response
start_time - Moderation start time
end_time - Moderation end time
Metadata
message_id - Message ID
action - Specific actions taken
preset_response - Preset response
Suggested Question Trace Information
Used to track suggested questions
| Suggested Question | LangSmith LLM |
|---|---|
user_id | - placed in metadata |
suggested_question | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
metadata | extra |
[suggested_question] | tags |
message_id | parent_run_id |
Message Trace Info
message_id - Message ID
message_data - Message data
inputs - Input content
outputs - Output content
start_time - Start time
end_time - End time
total_tokens - Number of tokens
status - Message status
error - Error information
from_account_id - ID of the sending account
agent_based - Whether the message is agent-based
from_source - Message source
model_provider - Model provider
model_id - Model ID
suggested_question - Suggested question
level - Status level
status_message - Status message
Metadata
message_id - Message ID
ls_provider - Model provider
ls_model_name - Model ID
status - Message status
from_end_user_id - ID of the sending user
from_account_id - ID of the sending account
workflow_run_id - Workflow run ID
from_source - Message source
Dataset Retrieval Trace Information
Used to track knowledge base retrieval
| Dataset Retrieval | LangSmith Retriever |
|---|---|
user_id | - placed in metadata |
dataset_retrieval | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
metadata | extra |
[dataset_retrieval] | tags |
message_id | parent_run_id |
Dataset Retrieval Trace Info
message_id - Message ID
inputs - Input content
documents - Document data
start_time - Start time
end_time - End time
message_data - Message data
Metadata
message_id - Message ID
ls_provider - Model provider
ls_model_name - Model ID
status - Message status
from_end_user_id - ID of the sending user
from_account_id - ID of the sending account
agent_based - Whether the message is agent-based
workflow_run_id - Workflow run ID
from_source - Message source
Tool Trace Information
Used to track tool invocation
| Tool | LangSmith Tool |
|---|---|
user_id | - placed in metadata |
tool_name | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
metadata | extra |
["tool", tool_name] | tags |
message_id | parent_run_id |
Tool Trace Info
message_id - Message ID
tool_name - Tool name
start_time - Start time
end_time - End time
tool_inputs - Tool inputs
tool_outputs - Tool outputs
message_data - Message data
error - Error information, if any
inputs - Inputs for the message
outputs - Outputs of the message
tool_config - Tool configuration
time_cost - Time cost
tool_parameters - Tool parameters
file_url - URL of the associated file
Metadata
message_id - Message ID
tool_name - Tool name
tool_inputs - Tool inputs
tool_outputs - Tool outputs
tool_config - Tool configuration
time_cost - Time cost
error - Error information, if any
tool_parameters - Tool parameters
message_file_id - Message file ID
created_by_role - Role of the creator
created_user_id - User ID of the creator
Generate Name Trace Information
Used to track conversation title generation
| Generate Name | LangSmith Tool |
|---|---|
user_id | - placed in metadata |
generate_name | name |
start_time | start_time |
end_time | end_time |
inputs | inputs |
outputs | outputs |
metadata | extra |
[generate_name] | tags |
Generate Name Trace Info
conversation_id - Conversation ID
inputs - Input data
outputs - Generated conversation name
start_time - Start time
end_time - End time
tenant_id - Tenant ID
Metadata
conversation_id - Conversation ID
tenant_id - Tenant ID
Last updated
