Integrate LangSmith

What is LangSmith

LangSmith is a platform for building production-grade LLM applications. It is used for developing, collaborating, testing, deploying, and monitoring LLM applications.

For more details, please refer to LangSmith.


How to Configure LangSmith

1. Register/Login to LangSmith

2. Create a Project

Create a project in LangSmith. After logging in, click New Project on the homepage to create your own project. The project will be used to associate with applications in Dify for data monitoring.

Once created, you can view all created projects in the Projects section.

3. Create Project Credentials

Find the project settings Settings in the left sidebar.

Click Create API Key to create project credentials.

Select Personal Access Token for subsequent API authentication.

Copy and save the created API key.

4. Integrating LangSmith with Dify

Configure LangSmith in the Dify application. Open the application you need to monitor, open Monitoring in the side menu, and select Configure on the page.

After clicking configure, paste the API Key and project name created in LangSmith into the configuration and save.

The configured project name needs to match the project set in LangSmith. If the project names do not match, LangSmith will automatically create a new project during data synchronization.

Once successfully saved, you can view the monitoring status on the current page.

Viewing Monitoring Data in LangSmith

Once configured, the debug or production data from applications within Dify can be monitored in LangSmith.

When you switch to LangSmith, you can view detailed operation logs of Dify applications in the dashboard.

Detailed LLM operation logs through LangSmith will help you optimize the performance of your Dify application.

Monitoring Data List

Workflow/Chatflow Trace Information

Used to track workflows and chatflows

WorkflowLangSmith Chain

workflow_app_log_id/workflow_run_id

id

user_session_id

- placed in metadata

workflow_{id}

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

Model token consumption

usage_metadata

metadata

extra

error

error

[workflow]

tags

"conversation_id/none for workflow"

conversation_id in metadata

conversion_id

parent_run_id

Workflow Trace Info

  • workflow_id - Unique identifier of the workflow

  • conversation_id - Conversation ID

  • workflow_run_id - ID of the current run

  • tenant_id - Tenant ID

  • elapsed_time - Time taken for the current run

  • status - Run status

  • version - Workflow version

  • total_tokens - Total tokens used in the current run

  • file_list - List of processed files

  • triggered_from - Source that triggered the current run

  • workflow_run_inputs - Input data for the current run

  • workflow_run_outputs - Output data for the current run

  • error - Errors encountered during the current run

  • query - Query used during the run

  • workflow_app_log_id - Workflow application log ID

  • message_id - Associated message ID

  • start_time - Start time of the run

  • end_time - End time of the run

  • workflow node executions - Information about workflow node executions

  • Metadata

    • workflow_id - Unique identifier of the workflow

    • conversation_id - Conversation ID

    • workflow_run_id - ID of the current run

    • tenant_id - Tenant ID

    • elapsed_time - Time taken for the current run

    • status - Run status

    • version - Workflow version

    • total_tokens - Total tokens used in the current run

    • file_list - List of processed files

    • triggered_from - Source that triggered the current run

Message Trace Information

Used to track LLM-related conversations

ChatLangSmith LLM

message_id

id

user_session_id

- placed in metadata

“message_{id}"

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

Model token consumption

usage_metadata

metadata

extra

error

error

["message", conversation_mode]

tags

conversation_id

conversation_id in metadata

conversion_id

parent_run_id

Message Trace Info

  • message_id - Message ID

  • message_data - Message data

  • user_session_id - User session ID

  • conversation_model - Conversation mode

  • message_tokens - Number of tokens in the message

  • answer_tokens - Number of tokens in the answer

  • total_tokens - Total number of tokens in the message and answer

  • error - Error information

  • inputs - Input data

  • outputs - Output data

  • file_list - List of processed files

  • start_time - Start time

  • end_time - End time

  • message_file_data - File data associated with the message

  • conversation_mode - Conversation mode

  • Metadata

    • conversation_id - Conversation ID

    • ls_provider - Model provider

    • ls_model_name - Model ID

    • status - Message status

    • from_end_user_id - ID of the sending user

    • from_account_id - ID of the sending account

    • agent_based - Whether the message is agent-based

    • workflow_run_id - Workflow run ID

    • from_source - Message source

Moderation Trace Information

Used to track conversation moderation

ModerationLangSmith Tool

user_id

- placed in metadata

“moderation"

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

metadata

extra

[moderation]

tags

message_id

parent_run_id

Moderation Trace Info

  • message_id - Message ID

  • user_id: User ID

  • workflow_app_log_id - Workflow application log ID

  • inputs - Moderation input data

  • message_data - Message data

  • flagged - Whether the content is flagged for attention

  • action - Specific actions taken

  • preset_response - Preset response

  • start_time - Moderation start time

  • end_time - Moderation end time

  • Metadata

    • message_id - Message ID

    • action - Specific actions taken

    • preset_response - Preset response

Suggested Question Trace Information

Used to track suggested questions

Suggested QuestionLangSmith LLM

user_id

- placed in metadata

suggested_question

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

metadata

extra

[suggested_question]

tags

message_id

parent_run_id

Message Trace Info

  • message_id - Message ID

  • message_data - Message data

  • inputs - Input content

  • outputs - Output content

  • start_time - Start time

  • end_time - End time

  • total_tokens - Number of tokens

  • status - Message status

  • error - Error information

  • from_account_id - ID of the sending account

  • agent_based - Whether the message is agent-based

  • from_source - Message source

  • model_provider - Model provider

  • model_id - Model ID

  • suggested_question - Suggested question

  • level - Status level

  • status_message - Status message

  • Metadata

    • message_id - Message ID

    • ls_provider - Model provider

    • ls_model_name - Model ID

    • status - Message status

    • from_end_user_id - ID of the sending user

    • from_account_id - ID of the sending account

    • workflow_run_id - Workflow run ID

    • from_source - Message source

Dataset Retrieval Trace Information

Used to track knowledge base retrieval

Dataset RetrievalLangSmith Retriever

user_id

- placed in metadata

dataset_retrieval

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

metadata

extra

[dataset_retrieval]

tags

message_id

parent_run_id

Dataset Retrieval Trace Info

  • message_id - Message ID

  • inputs - Input content

  • documents - Document data

  • start_time - Start time

  • end_time - End time

  • message_data - Message data

  • Metadata

    • message_id - Message ID

    • ls_provider - Model provider

    • ls_model_name - Model ID

    • status - Message status

    • from_end_user_id - ID of the sending user

    • from_account_id - ID of the sending account

    • agent_based - Whether the message is agent-based

    • workflow_run_id - Workflow run ID

    • from_source - Message source

Tool Trace Information

Used to track tool invocation

ToolLangSmith Tool

user_id

- placed in metadata

tool_name

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

metadata

extra

["tool", tool_name]

tags

message_id

parent_run_id

Tool Trace Info

  • message_id - Message ID

  • tool_name - Tool name

  • start_time - Start time

  • end_time - End time

  • tool_inputs - Tool inputs

  • tool_outputs - Tool outputs

  • message_data - Message data

  • error - Error information, if any

  • inputs - Inputs for the message

  • outputs - Outputs of the message

  • tool_config - Tool configuration

  • time_cost - Time cost

  • tool_parameters - Tool parameters

  • file_url - URL of the associated file

  • Metadata

    • message_id - Message ID

    • tool_name - Tool name

    • tool_inputs - Tool inputs

    • tool_outputs - Tool outputs

    • tool_config - Tool configuration

    • time_cost - Time cost

    • error - Error information, if any

    • tool_parameters - Tool parameters

    • message_file_id - Message file ID

    • created_by_role - Role of the creator

    • created_user_id - User ID of the creator

Generate Name Trace Information

Used to track conversation title generation

Generate NameLangSmith Tool

user_id

- placed in metadata

generate_name

name

start_time

start_time

end_time

end_time

inputs

inputs

outputs

outputs

metadata

extra

[generate_name]

tags

Generate Name Trace Info

  • conversation_id - Conversation ID

  • inputs - Input data

  • outputs - Generated conversation name

  • start_time - Start time

  • end_time - End time

  • tenant_id - Tenant ID

  • Metadata

    • conversation_id - Conversation ID

    • tenant_id - Tenant ID

Last updated