25 lines
478 B
Markdown
25 lines
478 B
Markdown
# rkllm server
|
|
|
|
Hosts a simple flask-based chat interface to a rkllm-model at localhost:8080.
|
|
|
|
|
|
## Install
|
|
|
|
On a r3588 system:
|
|
|
|
```sh
|
|
git clone "<repo>/rkllm_server"
|
|
cd rkllm_server
|
|
python3 -m venv venv
|
|
source venv/bin/activate
|
|
pip install -r requirements.txt
|
|
deactivate
|
|
```
|
|
|
|
you can now start the server with:
|
|
|
|
```sh
|
|
bash ./start_server.sh "/path/to/model.rkllm"
|
|
```
|
|
|
|
The first time on each boot it will ask for a sudo password to fix the npu speed (see [fix_freq_rk3588.sh]).
|