Model Provider Plugin

Prerequisites

Dify CLI
Basic Python programming skills and understanding of object-oriented programming
Familiarity with the API documentation of the model provider you want to integrate

Step 1: Create and Configure a New Plugin Project

Initialize the Project

dify plugin init

Choose Model Plugin Template

Select the LLM type plugin template from the available options. This template provides a complete code structure for model integration.

Configure Plugin Permissions

For a model provider plugin, configure the following essential permissions:

Models - Base permission for model operations
LLM - Permission for large language model functionality
Storage - Permission for file operations (if needed)

Directory Structure Overview

After initialization, your plugin project will have a directory structure similar to this (assuming a provider named my_provider supporting LLM and Embedding):

models/my_provider/
├── models                # Model implementation and configuration directory
│   ├── llm               # LLM type
│   │   ├── _position.yaml  (Optional, controls sorting)
│   │   ├── model1.yaml     # Configuration for specific model
│   │   └── llm.py          # LLM implementation logic
│   └── text_embedding    # Embedding type
│       ├── _position.yaml
│       ├── embedding-model.yaml
│       └── text_embedding.py
├── provider              # Provider-level code directory
│   └── my_provider.py    # Provider credential validation
└── manifest.yaml         # Plugin manifest file

Step 2: Understand Model Configuration Methods

Dify supports two model configuration methods that determine how users will interact with your provider’s models:

Predefined Models (`predefined-model`)

These are models that only require unified provider credentials to use. Once a user configures their API key or other authentication details for the provider, they can immediately access all predefined models. Example: The OpenAI provider offers predefined models like gpt-3.5-turbo-0125 and gpt-4o-2024-05-13. A user only needs to configure their OpenAI API key once to access all these models.

Custom Models (`customizable-model`)

These require additional configuration for each specific model instance. This approach is useful when models need individual parameters beyond the provider-level credentials. Example: Xinference supports both LLM and Text Embedding, but each model has a unique model_uid. Users must configure this model_uid separately for each model they want to use. These configuration methods can coexist within a single provider. For instance, a provider might offer some predefined models while also allowing users to add custom models with specific configurations.

Step 3: Create Model Provider Files

Creating a new model provider involves two main components:

Provider Configuration YAML File - Defines the provider’s basic information, supported model types, and credential requirements
Provider Class Implementation - Implements authentication validation and other provider-level functionality

3.1 Create Model Provider Configuration File

The provider configuration is defined in a YAML file that declares the provider’s basic information, supported model types, configuration methods, and credential rules. This file will be placed in the root directory of your plugin project. Here’s an annotated example of the anthropic.yaml configuration file:

# Basic provider identification
provider: anthropic                # Provider ID (must be unique)
label:
  en_US: Anthropic                 # Display name in UI
description:
  en_US: Anthropic's powerful models, such as Claude 3.
  zh_Hans: Anthropic 的强大模型，例如 Claude 3。
icon_small:
  en_US: icon_s_en.svg            # Small icon for provider (displayed in selection UI)
icon_large:
  en_US: icon_l_en.svg            # Large icon (displayed in detail views)
background: "#F0F0EB"             # Background color for provider in UI

# Help information for users
help:
  title:
    en_US: Get your API Key from Anthropic
    zh_Hans: 从 Anthropic 获取 API Key
  url:
    en_US: https://console.anthropic.com/account/keys

# Supported model types and configuration approach
supported_model_types:
  - llm                           # This provider offers LLM models
configurate_methods:
  - predefined-model              # Uses predefined models approach

# Provider-level credential form definition
provider_credential_schema:
  credential_form_schemas:
    - variable: anthropic_api_key  # Variable name for API key
      label:
        en_US: API Key
      type: secret-input           # Secure input for sensitive data
      required: true
      placeholder:
        zh_Hans: 在此输入你的 API Key
        en_US: Enter your API Key
    - variable: anthropic_api_url
      label:
        en_US: API URL
      type: text-input             # Regular text input
      required: false
      placeholder:
        zh_Hans: 在此输入你的 API URL
        en_US: Enter your API URL

# Model configuration
models:
  llm:                            # Configuration for LLM type models
    predefined:
      - "models/llm/*.yaml"       # Pattern to locate model configuration files
    position: "models/llm/_position.yaml"  # File defining display order

# Implementation file locations
extra:
  python:
    provider_source: provider/anthropic.py  # Provider class implementation
    model_sources:
      - "models/llm/llm.py"                 # Model implementation file

Custom Model Configuration

If your provider supports custom models, you need to add a model_credential_schema section to define what additional fields users need to configure for each individual model. This is typical for providers that support fine-tuned models or require model-specific parameters. Here’s an example from the OpenAI provider:

model_credential_schema:
  model: # Fine-tuned model name field
    label:
      en_US: Model Name
      zh_Hans: 模型名称
    placeholder:
      en_US: Enter your model name
      zh_Hans: 输入模型名称
  credential_form_schemas:
  - variable: openai_api_key
    label:
      en_US: API Key
    type: secret-input
    required: true
    placeholder:
      zh_Hans: 在此输入你的 API Key
      en_US: Enter your API Key
  - variable: openai_organization
    label:
        zh_Hans: 组织 ID
        en_US: Organization
    type: text-input
    required: false
    placeholder:
      zh_Hans: 在此输入你的组织 ID
      en_US: Enter your Organization ID
  # Additional fields as needed...

For complete model provider YAML specifications, please refer to the Model Schema documentation.

3.2 Write Model Provider Code

Next, create a Python file for your provider class implementation. This file should be placed in the /provider directory with a name matching your provider (e.g., anthropic.py). The provider class must inherit from ModelProvider and implement at least the validate_provider_credentials method:

import logging
from dify_plugin.entities.model import ModelType
from dify_plugin.errors.model import CredentialsValidateFailedError
from dify_plugin import ModelProvider

logger = logging.getLogger(__name__)


class AnthropicProvider(ModelProvider):
    def validate_provider_credentials(self, credentials: dict) -> None:
        """
        Validate provider credentials by testing them against the API.
        
        This method should attempt to make a simple API call to verify
        that the credentials are valid.
        
        :param credentials: Provider credentials as defined in the YAML schema
        :raises CredentialsValidateFailedError: If validation fails
        """
        try:
            # Get an instance of the LLM model type and use it to validate credentials
            model_instance = self.get_model_instance(ModelType.LLM)
            model_instance.validate_credentials(
                model="claude-3-opus-20240229", 
                credentials=credentials
            )
        except CredentialsValidateFailedError as ex:
            # Pass through credential validation errors
            raise ex
        except Exception as ex:
            # Log and re-raise other exceptions
            logger.exception(f"{self.get_provider_schema().provider} credentials validate failed")
            raise ex

The validate_provider_credentials method is crucial as it’s called whenever a user tries to save their provider credentials in Dify. It should:

Attempt to validate the credentials by making a simple API call
Return silently if validation succeeds
Raise CredentialsValidateFailedError with a helpful message if validation fails

For Custom Model Providers

For providers that exclusively use custom models (where each model requires its own configuration), you can implement a simpler provider class. For example, with Xinference:

from dify_plugin import ModelProvider

class XinferenceProvider(ModelProvider):
    def validate_provider_credentials(self, credentials: dict) -> None:
        """
        For custom-only model providers, validation happens at the model level.
        This method exists to satisfy the abstract base class requirement.
        """
        pass

Step 4: Implement Model-Specific Code

After setting up your provider, you need to implement the model-specific code that will handle API calls for each model type you support. This involves:

Creating model configuration YAML files for each specific model
Implementing the model type classes that handle API communication

For detailed instructions on these steps, please refer to:

Model Design Rules - Standards for integrating predefined models
Model Schema - Standards for model configuration files

4.1 Define Model Configuration (YAML)

For each specific model, create a YAML file in the appropriate model type directory (e.g., models/llm/) to define its properties, parameters, and features. Example (claude-3-5-sonnet-20240620.yaml):

model: claude-3-5-sonnet-20240620   # API identifier for the model
label:
  en_US: claude-3-5-sonnet-20240620 # Display name in UI
model_type: llm                     # Must match directory type
features:                           # Special capabilities
  - agent-thought
  - vision
  - tool-call
  - stream-tool-call
  - document
model_properties:                   # Inherent model properties
  mode: chat                        # "chat" or "completion"
  context_size: 200000              # Maximum context window
parameter_rules:                    # User-adjustable parameters
  - name: temperature
    use_template: temperature       # Reference predefined template
  - name: top_p
    use_template: top_p
  - name: max_tokens
    use_template: max_tokens
    required: true
    default: 8192
    min: 1
    max: 8192
pricing:                           # Optional pricing information
  input: '3.00'
  output: '15.00'
  unit: '0.000001'                 # Per million tokens
  currency: USD

4.2 Implement Model Calling Code (Python)

Create a Python file for each model type you’re supporting (e.g., llm.py in the models/llm/ directory). This class will handle API communication, parameter transformation, and result formatting. Here’s an example implementation structure for an LLM:

import logging
from typing import Union, Generator, Optional, List
from dify_plugin.provider_kits.llm import LargeLanguageModel # Base class
from dify_plugin.provider_kits.llm import LLMResult, LLMResultChunk, LLMUsage # Result classes
from dify_plugin.provider_kits.llm import PromptMessage, PromptMessageTool # Message classes
from dify_plugin.errors.provider_error import InvokeError, InvokeAuthorizationError # Error classes

logger = logging.getLogger(__name__)

class MyProviderLargeLanguageModel(LargeLanguageModel):
    def _invoke(self, model: str, credentials: dict, prompt_messages: List[PromptMessage],
                model_parameters: dict, tools: Optional[List[PromptMessageTool]] = None,
                stop: Optional[List[str]] = None, stream: bool = True,
                user: Optional[str] = None) -> Union[LLMResult, Generator[LLMResultChunk, None, None]]:
        """
        Core method for invoking the model API.
        
        Parameters:
            model: The model identifier to call
            credentials: Authentication credentials
            prompt_messages: List of messages to send
            model_parameters: Parameters like temperature, max_tokens
            tools: Optional tool definitions for function calling
            stop: Optional list of stop sequences
            stream: Whether to stream responses (True) or return complete response (False)
            user: Optional user identifier for API tracking
            
        Returns:
            If stream=True: Generator yielding LLMResultChunk objects
            If stream=False: Complete LLMResult object
        """
        # Prepare API request parameters
        api_params = self._prepare_api_params(
            credentials, model_parameters, prompt_messages, tools, stop
        )
        
        try:
            # Call appropriate helper method based on streaming preference
            if stream:
                return self._invoke_stream(model, api_params, user)
            else:
                return self._invoke_sync(model, api_params, user)
        except Exception as e:
            # Handle and map errors
            self._handle_api_error(e)
    
    def _invoke_stream(self, model: str, api_params: dict, user: Optional[str]) -> Generator[LLMResultChunk, None, None]:
        """Helper method for streaming API calls"""
        # Implementation details for streaming calls
        pass
        
    def _invoke_sync(self, model: str, api_params: dict, user: Optional[str]) -> LLMResult:
        """Helper method for synchronous API calls"""
        # Implementation details for synchronous calls
        pass
        
    def validate_credentials(self, model: str, credentials: dict) -> None:
        """
        Validate that the credentials work for this specific model.
        Called when a user tries to add or modify credentials.
        """
        # Implementation for credential validation
        pass
        
    def get_num_tokens(self, model: str, credentials: dict, 
                       prompt_messages: List[PromptMessage],
                       tools: Optional[List[PromptMessageTool]] = None) -> int:
        """
        Estimate the number of tokens for given input.
        Optional but recommended for accurate cost estimation.
        """
        # Implementation for token counting
        pass
        
    @property
    def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:
        """
        Define mapping from vendor-specific exceptions to Dify standard exceptions.
        This helps standardize error handling across different providers.
        """
        return {
            InvokeAuthorizationError: [
                # List vendor-specific auth errors here
            ],
            # Other error mappings
        }

The most important method to implement is _invoke, which handles the core API communication. This method should:

Transform Dify’s standardized inputs into the format required by the provider’s API
Make the API call with proper error handling
Transform the API response into Dify’s standardized output format
Handle both streaming and non-streaming modes

Step 5: Debug and Test Your Plugin

Dify provides a remote debugging capability that allows you to test your plugin during development:

In your Dify instance, go to “Plugin Management” and click “Debug Plugin” to get your debug key and server address
Configure your local environment with these values in a .env file:

INSTALL_METHOD=remote
REMOTE_INSTALL_HOST=<your-dify-domain-or-ip>
REMOTE_INSTALL_PORT=5003
REMOTE_INSTALL_KEY=****-****-****-****-****

Run your plugin locally with python -m main and test it in Dify

Step 6: Package and Publish

When your plugin is ready:

Package it using the scaffolding tool:

dify plugin package models/<provider_name>

Test the packaged plugin locally before submitting
Submit a pull request to the Dify official plugins repository

For more details on the publishing process, see the Publishing Overview.

Reference Resources

Quick Integration of a New Model - How to add new models to existing providers
Basic Concepts of Plugin Development - Return to the plugin development getting started guide
Model Schema - Learn detailed model configuration specifications
General Specifications - Learn about plugin manifest file configuration
Dify Plugin SDK Reference - Look up base classes, data structures, and error types

Edit this page | Report an issue

Getting Started

Features & Specs

Development Guides & Walkthroughs

Publishing

Model Provider Plugin

Prerequisites

Step 1: Create and Configure a New Plugin Project

Initialize the Project

Choose Model Plugin Template

Configure Plugin Permissions

Directory Structure Overview

Step 2: Understand Model Configuration Methods

Predefined Models (`predefined-model`)

Custom Models (`customizable-model`)

Step 3: Create Model Provider Files

3.1 Create Model Provider Configuration File

Custom Model Configuration

3.2 Write Model Provider Code

For Custom Model Providers

Step 4: Implement Model-Specific Code

4.1 Define Model Configuration (YAML)

4.2 Implement Model Calling Code (Python)

Step 5: Debug and Test Your Plugin

Step 6: Package and Publish

Reference Resources

Getting Started

Features & Specs

Development Guides & Walkthroughs

Publishing

Documentation Index

​Prerequisites

​Step 1: Create and Configure a New Plugin Project

​Initialize the Project

​Choose Model Plugin Template

​Configure Plugin Permissions

​Directory Structure Overview

​Step 2: Understand Model Configuration Methods

​Predefined Models (predefined-model)

​Custom Models (customizable-model)

​Step 3: Create Model Provider Files

​3.1 Create Model Provider Configuration File

​Custom Model Configuration

​3.2 Write Model Provider Code

​For Custom Model Providers

​Step 4: Implement Model-Specific Code

​4.1 Define Model Configuration (YAML)

​4.2 Implement Model Calling Code (Python)

​Step 5: Debug and Test Your Plugin

​Step 6: Package and Publish

​Reference Resources

Prerequisites

Step 1: Create and Configure a New Plugin Project

Initialize the Project

Choose Model Plugin Template

Configure Plugin Permissions

Directory Structure Overview

Step 2: Understand Model Configuration Methods

Predefined Models (`predefined-model`)

Custom Models (`customizable-model`)

Step 3: Create Model Provider Files

3.1 Create Model Provider Configuration File

Custom Model Configuration

3.2 Write Model Provider Code

For Custom Model Providers

Step 4: Implement Model-Specific Code

4.1 Define Model Configuration (YAML)

4.2 Implement Model Calling Code (Python)

Step 5: Debug and Test Your Plugin

Step 6: Package and Publish

Reference Resources