“Models” have been fully integrated into a “Plugin” ecosystem. For detailed development instructions on model plugins, please refer to Plugin Development. The following content has been archived.
After completing the supplier integration, the next step is to integrate the models under the supplier.
First, we need to determine the type of model to be integrated and create the corresponding model type module
in the directory of the respective supplier.
The currently supported model types are as follows:
llm
Text Generation Modeltext_embedding
Text Embedding Modelrerank
Rerank Modelspeech2text
Speech to Texttts
Text to Speechmoderation
ModerationTaking Anthropic
as an example, Anthropic
only supports LLM, so we create a module
named llm
in model_providers.anthropic
.
For predefined models, we first need to create a YAML file named after the model under the llm
module
, such as: claude-2.1.yaml
.
It is recommended to prepare all model configurations before starting the implementation of the model code.
Similarly, you can refer to the YAML configuration information in the directories of other suppliers under the model_providers
directory. The complete YAML rules can be found in: Schema.
Next, create a Python file with the same name llm.py
under the llm
module
to write the implementation code.
Create an Anthropic LLM class in llm.py
, which we will name AnthropicLargeLanguageModel
(name can be arbitrary), inheriting from the __base.large_language_model.LargeLanguageModel
base class, and implement the following methods:
LLM Invocation
Implement the core method for LLM invocation, supporting both streaming and synchronous responses.
When implementing, note to use two functions to return data, one for handling synchronous responses and one for streaming responses. Since Python recognizes functions containing the yield
keyword as generator functions, returning a fixed data type of Generator
, synchronous and streaming responses need to be implemented separately, like this (note the example below uses simplified parameters, actual implementation should follow the parameter list above):
Precompute Input Tokens
If the model does not provide a precompute tokens interface, return 0 directly.
Model Credentials Validation
Similar to supplier credentials validation, this validates the credentials for a single model.
Invocation Error Mapping Table
When a model invocation error occurs, it needs to be mapped to the InvokeError
type specified by Runtime, facilitating Dify to handle different errors differently.
Runtime Errors:
InvokeConnectionError
Invocation connection errorInvokeServerUnavailableError
Invocation service unavailableInvokeRateLimitError
Invocation rate limit reachedInvokeAuthorizationError
Invocation authorization failedInvokeBadRequestError
Invocation parameter errorFor interface method descriptions, see: Interfaces, and for specific implementation, refer to: llm.py.
provider
(string) Supplier identifier, e.g., openai
label
(object) Supplier display name, i18n, can be set in en_US
English and zh_Hans
Chinese
zh_Hans
(string) [optional] Chinese label name, if zh_Hans
is not set, it will default to en_US
.en_US
(string) English label namedescription
(object) [optional] Supplier description, i18n
zh_Hans
(string) [optional] Chinese descriptionen_US
(string) English descriptionicon_small
(string) [optional] Supplier small icon, stored in the _assets
directory under the respective supplier implementation directory, follows the same language strategy as label
zh_Hans
(string) [optional] Chinese iconen_US
(string) English iconicon_large
(string) [optional] Supplier large icon, stored in the _assets
directory under the respective supplier implementation directory, follows the same language strategy as label
zh_Hans
(string) [optional] Chinese iconen_US
(string) English iconbackground
(string) [optional] Background color value, e.g., #FFFFFF, if empty, the default color value will be displayed on the front end.help
(object) [optional] Help information
title
(object) Help title, i18n
zh_Hans
(string) [optional] Chinese titleen_US
(string) English titleurl
(object) Help link, i18n
zh_Hans
(string) [optional] Chinese linken_US
(string) English linksupported_model_types
(array[ModelType]) Supported model typesconfigurate_methods
(array[ConfigurateMethod]) Configuration methodsprovider_credential_schema
(ProviderCredentialSchema) Supplier credential schemamodel_credential_schema
(ModelCredentialSchema) Model credential schemaEdit this page | Report an issue