Integrate Local Models Deployed by Ollama
Ollama is a local inference framework client that allows one-click deployment of LLMs such as Llama 2, Mistral, Llava, etc. Dify supports integrating LLM and Text Embedding capabilities of large language models deployed with Ollama.
Quick Integration
Download and Launch Ollama
Download Ollama
Visit https://ollama.ai/download to download the Ollama client for your system.
Run Ollama and Chat with Llava
After successful launch, Ollama starts an API service on local port 11434, which can be accessed at
http://localhost:11434
.For other models, visit Ollama Models for more details.
Integrate Ollama in Dify
In
Settings > Model Providers > Ollama
, fill in:Model Name:
llava
Base URL:
http://<your-ollama-endpoint-domain>:11434
Enter the base URL where the Ollama service is accessible.
If Dify is deployed using docker, consider using the local network IP address, e.g.,
http://192.168.1.100:11434
or the docker host machine IP address, e.g.,http://172.17.0.1:11434
.For local source code deployment, use
http://localhost:11434
.Model Type:
Chat
Model Context Length:
4096
The maximum context length of the model. If unsure, use the default value of 4096.
Maximum Token Limit:
4096
The maximum number of tokens returned by the model. If there are no specific requirements for the model, this can be consistent with the model context length.
Support for Vision:
Yes
Check this option if the model supports image understanding (multimodal), like
llava
.
Click "Save" to use the model in the application after verifying that there are no errors.
The integration method for Embedding models is similar to LLM, just change the model type to Text Embedding.
Use Ollama Models
Enter
Prompt Eng.
page of the App that needs to be configured, select thellava
model under the Ollama provider, and use it after configuring the model parameters.
FAQ
⚠️ If you are using docker to deploy Dify and Ollama, you may encounter the following error:
This error occurs because the Ollama service is not accessible from the docker container. localhost
usually refers to the container itself, not the host machine or other containers. To resolve this issue, you need to expose the Ollama service to the network.
Setting environment variables on Mac
If Ollama is run as a macOS application, environment variables should be set using launchctl
:
For each environment variable, call
launchctl setenv
.Restart Ollama application.
If the above steps are ineffective, you can use the following method:
The issue lies within Docker itself, and to access the Docker host. you should connect to
host.docker.internal
. Therefore, replacinglocalhost
withhost.docker.internal
in the service will make it work effectively.
Setting environment variables on Linux
If Ollama is run as a systemd service, environment variables should be set using systemctl
:
Edit the systemd service by calling
systemctl edit ollama.service
. This will open an editor.For each environment variable, add a line
Environment
under section[Service]
:Save and exit.
Reload
systemd
and restart Ollama:
Setting environment variables on Windows
On windows, Ollama inherits your user and system environment variables.
First Quit Ollama by clicking on it in the task bar
Edit system environment variables from the control panel
Edit or create New variable(s) for your user account for
OLLAMA_HOST
,OLLAMA_MODELS
, etc.Click OK/Apply to save
Run
ollama
from a new terminal window
How can I expose Ollama on my network?
Ollama binds 127.0.0.1 port 11434 by default. Change the bind address with the OLLAMA_HOST
environment variable.
More Information
For more information on Ollama, please refer to:
Last updated