lucretia/embeddingsearch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated README to reflect the recent addition of the Indexer

Browse Source
This commit is contained in:
EzFeDezy
2025-05-27 22:30:52 +02:00
parent 7096f6591f
commit 371f9c7411
3 changed files with 144 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
docs/Indexer.md
View File

@@ -0,0 +1,83 @@
# Overview
## Installing the dependencies
## Ubuntu 24.04
1. Install the .NET SDK: `sudo apt update && sudo apt install dotnet-sdk-8.0 -y`
2. Install the python SDK: `sudo apt install python3 python3.12 python3.12-dev`
## Windows
Download the [.NET SDK](https://dotnet.microsoft.com/en-us/download) or follow these steps to use WSL:
1. Install Ubuntu in WSL (`wsl --install` and `wsl --install -d Ubuntu`)
2. Enter your WSL environment `wsl.exe` and configure it
3. Update via `sudo apt update && sudo apt upgrade -y && sudo snap refresh`
4. Continue here: [Ubuntu 24.04](#Ubuntu-24.04)
# Configuration
The configuration is located in `src/Indexer` and conforms to the [ASP.NET configuration design pattern](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-9.0), i.e. `src/Indexer/appsettings.json` is the base configuration, and `/src/Indexer/appsettings.Development.json` overrides it.
If you plan to use multiple environments, create any `appsettings.{YourEnvironment}.json` (e.g. `Development`, `Staging`, `Prod`) and set the environment variable `DOTNET_ENVIRONMENT` accordingly on the target machine.
## Setup
If you just installed the server and want to configure it:
1. Open `src/server/appsettings.Development.json`
2. If your search server is not on the same machine as the indexer, update "BaseUri" to reflect the URL to the server.
3. If your search server requires API keys, (i.e. it's operating outside of the "Development" environment) set `"ApiKey": "<your key here>"` beneath `"BaseUri"` in the `"Embeddingsearch"` section.
4. Create your own indexing script(s) in `src/Indexer/Scripts/` and configure their use as
## Structure
```json
"EmbeddingsearchIndexer": {
"Worker":
[ // This is a list; you can have as many "workers" as you want
{
"Name": "example",
"Searchdomains": [
"example"
],
"Script": "Scripts/example.py",
"Calls": [ // This is also a list. You can have as many different calls as you need.
{
"Type": "interval", // See: Call types
"Interval": 60000
}
]
}
]
}
```
## Call types
- `interval`
- What does it do: The script gets called periodically based on the specified `Interval` parameter.
- Parameters:
- Interval (in milliseconds)
- `schedule` (WIP)
- What does it do: The script gets called based on the provided schedule
- Parameters: (WIP)
- `fileupdate` (WIP)
- What does it do: The script gets called whenever a file is updated in the specified subdirectory
- Parameters: (WIP)
# Scripting
## Python
To ease scripting, tools.py contains all definitions of the .NET objects passed to the script. This includes attributes and methods.
These are not yet defined in a way that makes them 100% interactible with the Dotnet CLR, meaning some methods that require anything more than strings or other simple data types to be passed are not yet supported. (WIP)
### Required elements
Here is an overview of required elements by example:
```python
from tools import * # Import all tools that are provided for ease of scripting
def init(toolset: Toolset): # defining an init() function with 1 parameter is required.
pass # Your code would go here.
# DO NOT put a main loop here! Why?
# This function prevents the application from initializing and maintains exclusive control over the GIL
def update(toolset: Toolset): # defining an update() function with 1 parameter is required.
pass # Your code would go here.
```
### Using the toolset passed by the .NET CLR
The use of the toolset is laid out in good example by `src/Indexer/Scripts/example.py`.
Currently, `Toolset`, as provided by the IndexerService to the Python script, contains 3 elements:
1. (only for `update`, not `init`) `callbackInfos` - an object that provides all information regarding the callback. (e.g. what file was updated)
2. `client` - a .NET object that has the functions as described in `src/Indexer/Scripts/tools.py`. It's the client that - according to the configuration - communicates with the search server and executes the API calls.
3. `filePath` - the path to the script, as specified in the configuration
## Golang
TODO
## Javascript
TODO

41
docs/Server.md
View File

@@ -1,4 +1,4 @@
# Server
# Overview
The server by default
- runs on port 5146
- Uses Swagger UI in development mode (`/swagger/index.html`)
@@ -6,7 +6,7 @@ The server by default
# Installing the dependencies
## Ubuntu 24.04
1. `sudo apt update && sudo apt install dotnet-sdk-8.0 -y`
1. Install the .NET SDK: `sudo apt update && sudo apt install dotnet-sdk-8.0 -y`
## Windows
Download the [.NET SDK](https://dotnet.microsoft.com/en-us/download) or follow these steps to use WSL:
1. Install Ubuntu in WSL (`wsl --install` and `wsl --install -d Ubuntu`)
@@ -19,8 +19,39 @@ Download the [.NET SDK](https://dotnet.microsoft.com/en-us/download) or follow t
- Linux/WSL: `sudo apt install mysql-server`
- Windows: [MySQL Community Server](https://dev.mysql.com/downloads/mysql/)
2. connect to it: `sudo mysql -u root` (Or from outside of WSL: `mysql -u root`)
3. Create the database
3. Create the database:
`CREATE DATABASE embeddingsearch; use embeddingsearch;`
4. Create the user
4. Create the user (replace "somepassword! with a secure password):
`CREATE USER 'embeddingsearch'@'%' identified by "somepassword!"; GRANT ALL ON embeddingsearch.* TO embeddingsearch; FLUSH PRIVILEGES;`
5. Create the tables using the CLI tool: `dotnet build` and `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --database --setup`
5. Create the tables using the CLI tool: `cd src/cli; dotnet build` and `bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --database --setup` (replace the variables with the actual values)
# Configuration
## Environments
The configuration is located in `src/server/` and conforms to the [ASP.NET configuration design pattern](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-9.0), i.e. `src/server/appsettings.json` is the base configuration, and `/src/server/appsettings.Development.json` overrides it.
If you plan to use multiple environments, create any `appsettings.{YourEnvironment}.json` (e.g. `Development`, `Staging`, `Prod`) and set the environment variable `DOTNET_ENVIRONMENT` accordingly on the target machine.
## Setup
If you just installed the server and want to configure it:
1. Open `src/server/appsettings.Development.json`
2. Change the password in the "SQL" section (`pwd=<your password goes here>;`)
3. If your Ollama instance does not run locally, update "OllamaURL" to point at your Ollama instance.
4. If you plan on using the server in production:
1. Set the environment variable `DOTNET_ENVIRONMENT` to something that is not "Development". (e.g. "Prod")
2. Rename the `appsettings.Development.json` - replace "Development" with whatever you chose. (e.g. "Prod")
3. Set API keys in the "ApiKeys" section (generate keys using the `uuid` command on Linux)
# API
## Accessing the api
Once started, the server's API can be comfortably be viewed and manipulated via swagger.
By default it is accessible under: `http://localhost:5146/swagger/index.html`
To make an API request from within swagger:
1. Open one of the actions ("GET" / "POST")
2. Click the "Try it out" button. The input fields (if there are any for your action) should now be editable.
3. Fill in the necessary information
4. Click "Execute"
## Restricting access
API keys do **not** get checked in Development environment!
Set up a non-development environment as described in [Configuration>Setup](#setup) to enable API key authentication.