Integrate LangSmith

What is LangSmith

LangSmith is a platform for building production-grade LLM applications. It is used for developing, collaborating, testing, deploying, and monitoring LLM applications.

For more details, please refer to LangSmith.

How to Configure LangSmith

2. Create a Project

Create a project in LangSmith. After logging in, click New Project on the homepage to create your own project. The project will be used to associate with applications in Dify for data monitoring.

Once created, you can view all created projects in the Projects section.

3. Create Project Credentials

Find the project settings Settings in the left sidebar.

Click Create API Key to create project credentials.

Select Personal Access Token for subsequent API authentication.

Copy and save the created API key.

4. Integrating LangSmith with Dify

Configure LangSmith in the Dify application. Open the application you need to monitor, open Monitoring in the side menu, and select Tracing app performance on the page.

After clicking configure, paste the API Key and project name created in LangSmith into the configuration and save.

The configured project name needs to match the project set in LangSmith. If the project names do not match, LangSmith will automatically create a new project during data synchronization.

Once successfully saved, you can view the monitoring status on the current page.

Viewing Monitoring Data in LangSmith

Once configured, the debug or production data from applications within Dify can be monitored in LangSmith.

When you switch to LangSmith, you can view detailed operation logs of Dify applications in the dashboard.

Detailed LLM operation logs through LangSmith will help you optimize the performance of your Dify application.

Monitoring Data List

Workflow/Chatflow Trace Information

Used to track workflows and chatflows

Workflow Trace Info

workflow_id - Unique identifier of the workflow
conversation_id - Conversation ID
workflow_run_id - ID of the current run
tenant_id - Tenant ID
elapsed_time - Time taken for the current run
status - Run status
version - Workflow version
total_tokens - Total tokens used in the current run
file_list - List of processed files
triggered_from - Source that triggered the current run
workflow_run_inputs - Input data for the current run
workflow_run_outputs - Output data for the current run
error - Errors encountered during the current run
query - Query used during the run
workflow_app_log_id - Workflow application log ID
message_id - Associated message ID
start_time - Start time of the run
end_time - End time of the run
workflow node executions - Information about workflow node executions
Metadata
- workflow_id - Unique identifier of the workflow
- conversation_id - Conversation ID
- workflow_run_id - ID of the current run
- tenant_id - Tenant ID
- elapsed_time - Time taken for the current run
- status - Run status
- version - Workflow version
- total_tokens - Total tokens used in the current run
- file_list - List of processed files
- triggered_from - Source that triggered the current run

Message Trace Information

Used to track LLM-related conversations

Message Trace Info

message_id - Message ID
message_data - Message data
user_session_id - User session ID
conversation_model - Conversation mode
message_tokens - Number of tokens in the message
answer_tokens - Number of tokens in the answer
total_tokens - Total number of tokens in the message and answer
error - Error information
inputs - Input data
outputs - Output data
file_list - List of processed files
start_time - Start time
end_time - End time
message_file_data - File data associated with the message
conversation_mode - Conversation mode
Metadata
- conversation_id - Conversation ID
- ls_provider - Model provider
- ls_model_name - Model ID
- status - Message status
- from_end_user_id - ID of the sending user
- from_account_id - ID of the sending account
- agent_based - Whether the message is agent-based
- workflow_run_id - Workflow run ID
- from_source - Message source

Moderation Trace Information

Used to track conversation moderation

Moderation Trace Info

message_id - Message ID
user_id: User ID
workflow_app_log_id - Workflow application log ID
inputs - Moderation input data
message_data - Message data
flagged - Whether the content is flagged for attention
action - Specific actions taken
preset_response - Preset response
start_time - Moderation start time
end_time - Moderation end time
Metadata
- message_id - Message ID
- action - Specific actions taken
- preset_response - Preset response

Dataset Retrieval Trace Information

Used to track knowledge base retrieval

Dataset Retrieval Trace Info

message_id - Message ID
inputs - Input content
documents - Document data
start_time - Start time
end_time - End time
message_data - Message data
Metadata
- message_id - Message ID
- ls_provider - Model provider
- ls_model_name - Model ID
- status - Message status
- from_end_user_id - ID of the sending user
- from_account_id - ID of the sending account
- agent_based - Whether the message is agent-based
- workflow_run_id - Workflow run ID
- from_source - Message source

Tool Trace Information

Used to track tool invocation

Tool Trace Info

message_id - Message ID
tool_name - Tool name
start_time - Start time
end_time - End time
tool_inputs - Tool inputs
tool_outputs - Tool outputs
message_data - Message data
error - Error information, if any
inputs - Inputs for the message
outputs - Outputs of the message
tool_config - Tool configuration
time_cost - Time cost
tool_parameters - Tool parameters
file_url - URL of the associated file
Metadata
- message_id - Message ID
- tool_name - Tool name
- tool_inputs - Tool inputs
- tool_outputs - Tool outputs
- tool_config - Tool configuration
- time_cost - Time cost
- error - Error information, if any
- tool_parameters - Tool parameters
- message_file_id - Message file ID
- created_by_role - Role of the creator
- created_user_id - User ID of the creator