# Overview
The indexer by default
- runs on port 5210
- Uses Swagger UI in development mode (endpoint: `/swagger/index.html`)
- Ignores API keys when in development mode
- Uses Elmah error logging (endpoint: `/elmah`, local files: `~/logs`)
- Uses serilog logging (local files: `~/logs`)
- Uses HealthChecks (endpoint: `/healthz`)
## Docker installation
(On Linux you might need root privileges, thus use `sudo` where necessary)
1. [Configure the indexer](docs/Indexer.md#configuration)
2. [Set up your indexing script(s)](docs/Indexer.md#scripting)
3. Navigate to the `src` directory
4. Build the docker container: `docker build -t embeddingsearch-indexer -f Indexer/Dockerfile .`
5. Run the docker container: `docker run --net=host -t embeddingsearch-indexer` (the `-t` is optional, but you get more meaningful output. Or use `-d` to run it in the background)
## Installing the dependencies
## Ubuntu 24.04
1. Install the .NET SDK: `sudo apt update && sudo apt install dotnet-sdk-10.0 -y`
2. Install the python SDK: `sudo apt install python3 python3.13 python3.13-dev`
    - Note: Python 3.14 is not supported yet
## Windows
Download and install the [.NET SDK](https://dotnet.microsoft.com/en-us/download) or follow these steps to use WSL:
1. Install Ubuntu in WSL (`wsl --install` and `wsl --install -d Ubuntu`)
2. Enter your WSL environment `wsl.exe` and configure it
3. Update via `sudo apt update && sudo apt upgrade -y && sudo snap refresh`
4. Continue here: [Ubuntu 24.04](#Ubuntu-24.04)
# Configuration
The configuration is located in `src/Indexer` and conforms to the [ASP.NET configuration design pattern](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-9.0), i.e. `src/Indexer/appsettings.json` is the base configuration, and `/src/Indexer/appsettings.Development.json` overrides it.

If you plan to use multiple environments, create any `appsettings.{YourEnvironment}.json` (e.g. `Development`, `Staging`, `Prod`) and set the environment variable `DOTNET_ENVIRONMENT` accordingly on the target machine.
## Setup
If you just installed the indexer and want to configure it:
1. Open `src/Indexer/appsettings.Development.json`
2. If your search server is not on the same machine as the indexer, update "BaseUri" to reflect the URL to the server.
3. If you configured API keys for the search server, set `"ApiKey": "<your key here>"` beneath `"BaseUri"` in the `"Server"` section.
4. Create your own indexing script(s) in `src/Indexer/Scripts/` and configure them as shown below
## Structure
```json
  "Indexer": {
    "Workers":
    [ // This is a list; you can have as many "workers" as you want
      {
        "Name": "example",
        "Script": "Scripts/example.py",
        "Calls": [ // This is also a list. A worker may have multiple different calls.
          {
            "Type": "interval", // See: Call types
            "Interval": 60000 // Parameter(s) as specified for the call type
          }
        ]
      },
      {
        "Name": "secondWorker",
        /* ... */
      }
    ],
    "ApiKeys": ["YourApiKeysHereForTheIndexer"], // API Keys for if you want to protect the indexer's API
    "Server": {
      "BaseUri": "http://localhost:5000", // URL to the embeddingsearch server
      "ApiKey": "ServerApiKeyHere" // API Key set in the server
    }
  }
```
## Call types
- `runonce`
  - What does it do: The script gets called once at startup. Use this if you need a main loop.
  - (Remember the call runs in `update()` like the others!)
  - Parameters: None
- `interval`
    - What does it do: The script gets called periodically based on the specified `Interval` parameter.
    - Parameters:
        - Interval (in milliseconds)
- `schedule`
    - What does it do: The script gets called based on the provided schedule
    - Parameters:
      - Schedule ([Quartz Cron expression](https://www.quartz-scheduler.net/documentation/quartz-3.x/tutorial/crontrigger.html), e.g. "0/5 * * * * ?")
- `fileupdate`
    - What does it do: The script gets called whenever a file is updated in the specified subdirectory
    - Parameters:
      - Path (e.g. "Scripts/example_content")
# Scripting
Scripts should be put in `src/Indexer/Scripts/`. If you look there, by default you will find some example scripts that can be taken as reference when building your own.

For configuration of the scripts see: [Structure](#structure)

The next few sections explain some core concepts/patterns. If you want to skip to explicit code examples, look here:
- [Python](#python)
- [Roslyn](#c-roslyn)
## General
Scripts need to define the following functions:
- `init()`
  - Is run at startup. Put all initialization code here.
  - Do not put a main loop here! Might cause other workers not to initialize and other unintended behavior!
- `update()`
  - Is called by the calls as specified in [Call types](#call-types)
  - A main loop might work best here using the `runonce` call
## probMethods
Probmethods are used to join the multiple similarity values from multiple models and multiple datapoints into one single result.

They need to be specified when constructing a datapoint or an entity (see: [src/Indexer/Scripts/example.py](/src/Indexer/Scripts/example.py) in method `index_files`)

Currently the following probMethods are implemented:
- "Mean"
- "HarmonicMean"
- "QuadraticMean"
- "GeometricMean"
- "ExtremeValuesEmphasisWeightedAverage" or "EVEWavg"
- "HighValueEmphasisWeightedAverage" or "HVEWAvg"
- "LowValueEmphasisWeightedAverage" or "LVEWAvg"
- "DictionaryWeightedAverage"

### Mean
Averages the values by accumulating the sums and dividing by the number of entries.

$\frac{1}{n} \sum_{i=1}^{n} x_i$

### HarmonicMean
Calculates the harmonic mean, but also avoids division by 0 issues
$$
\text{HarmonicMean}(L) = \begin{cases}0,
& \text{if } n_{nz} = 0 \\\left( \frac{n_{nz}}{\sum\limits_{x_i \in L,\ x_i \neq 0} \frac{1}{x_i}} \right) \cdot \left( \frac{n_{nz}}{n_T} \right), & \text{otherwise}
\end{cases}
$$
with
- $n_{nz}$ being the number of non-zero elements
- $n_T$ being the total number of elements
### QuadraticMean
Calculates the quadratic mean.
$$
\text{QuadraticMean}(L) = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} x_i^2 }
$$
### GeometricMean
Calculates the geometric mean.
$$
\text{GeometricMean}(L) = \begin{cases}0, & \text{if } n = 0\\\left(\prod\limits_{i=1}^{n} x_i \right)^{\frac{1}{n}}, & \text{otherwise}\end{cases}
$$

### ExtremeValuesEmphasisWeightedAverage
aka. EVEWavg

Calculates a weighted average where values near 0 or 1 are weighted much more heavily.

A single `1` makes the whole function return 1, as it has "infinite" weight.
Similarly any `0` causes the function to return 0.

(If both a `0` and a `1` are present, the function returns 1)
$$
\text{EVEWA}(L) = \begin{cases}1, & \text{if } \exists, x_i = 1 \\0, & \text{if } \exists, x_i = 0 \\\frac{ \sum\limits_{i=1}^{n} \frac{x_i}{x_i(1 - x_i)} }{ \sum\limits_{i=1}^{n} \frac{1}{x_i(1 - x_i)} }, & \text{otherwise}\end{cases}
$$

### HighValueEmphasisWeightedAverage
aka. HVEWAvg

Calculates a weighted average where values near 1 are weighted much more heavily. Lower values are weighted less.

A single `1` makes the whole function return 1, as it has "infinite" weight.
A `0` has zero weight.
$$
\text{HVEWA}(L) = \begin{cases}1, & \text{if } \exists, x_i = 1 \\\frac{ \sum\limits_{i=1}^{n} \frac{x_i}{1 - x_i} }{ \sum\limits_{i=1}^{n} \frac{1}{1 - x_i} }, & \text{otherwise}\end{cases}
$$
### LowValueEmphasisWeightedAverage
aka. LVEWAvg

Calculates a weighted average where values near 0 are weighted much more heavily. Higher values are weighted less.

A single `0` makes the whole function return 0, as it has "infinite" weight.
A `1` has zero weight.
$$
\text{LVEWA}(L) = \begin{cases}1, & \text{if } \exists, x_i = 1 \\\frac{ n}{ \sum\limits_{i=1}^{n} \frac{1}{x_i} }, & \text{otherwise}\end{cases}
$$
### DictionaryWeightedAverage
Calculates a weighted average as specified by the user.

$$
\text{DWA}(L, D) = \frac{ \sum\limits_{i=1}^{n} w_i x_i }{ \sum\limits_{i=1}^{n} w_i }
$$

Where:
- $L = \{(k_1, x_1), (k_2, x_2), \dots, (k_n, x_n)\}$ is the list of key–value pairs
  - $x_i$ is the float value associated with key $k_i$
- $D : k_i \mapsto w_i$ is a dictionary mapping keys $k_i$ to weights $w_i \in \mathbb{R}$
$


e.g.:
```
probmethod_datapoint = "DictionaryWeightedAverage:{\"ollama:bge-m3\": 4, \"ollama:mxbai-embed-large\": 1}"
probmethod_entity = "DictionaryWeightedAverage:{\"title\": 2, \"filename\": 0.1, \"text\": 0.25}"
```

## Python
To ease scripting, tools.py contains all definitions of the .NET objects passed to the script. This includes attributes and methods.

These are not yet defined in a way that makes them 100% interactible with the Dotnet CLR, meaning some methods that require anything more than strings or other simple data types to be passed are not yet supported. (WIP)
### Supported file extensions
- .py
### Code elements
Here is an overview of code elements by example:
```python
from tools import * # Import all tools that are provided for ease of scripting

def init(toolset: Toolset): # defining an init() function with 1 parameter is required.
    pass # Your code would go here.
    # Don't put a main loop here! Why?
    # This function prevents the application from initializing and maintains exclusive control over the GIL

def update(toolset: Toolset): # defining an update() function with 1 parameter is required.
    pass # Your code - including possibly a main loop - would go here.
```
### Using the toolset passed by the .NET CLR
The use of the toolset is laid out in good example by `src/Indexer/Scripts/example.py`.

Currently, `Toolset`, as provided by the IndexerService to the Python script, contains 3 elements:
1. (only for `update`, not `init`) `callbackInfos` - an object that provides all information regarding the callback. (e.g. what file was updated)
2. `client` - a .NET object that has the functions as described in `src/Indexer/Scripts/tools.py`. It's the client that - according to the configuration - communicates with the search server and executes the API calls.
3. `filePath` - the path to the script, as specified in the configuration
## C# (Roslyn)
### Supported file extensions
- .csx
### Code elements
**important hint:** As shown in the last two lines of the example code, simply declaring the class is **not** enough. One must also return an object of said class!
```csharp
// #load directives are disregarded at compile time. Its use is currently for syntax highlighting only
#load "../../Client/Client.cs"
#load "../Models/Script.cs"
#load "../Models/Interfaces.cs"
#load "../Models/WorkerResults.cs"
#load "../../Shared/Models/SearchdomainResults.cs"
#load "../../Shared/Models/JSONModels.cs"
#load "../../Shared/Models/EntityResults.cs"

using Shared.Models;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Logging;

// Required: a class that extends Indexer.Models.IScript
public class ExampleScript : Indexer.Models.IScript
{
    public Indexer.Models.ScriptToolSet ToolSet;
    public Client.Client client;

    // Optional: constructor
    public ExampleScript()
    {
        //System.Console.WriteLine("DEBUG@example.cs - Constructor"); // logger not passed here yet
    }

    // Required: Init method as required to extend IScript
    public int Init(Indexer.Models.ScriptToolSet toolSet)
    {
        ToolSet = toolSet;
        ToolSet.Logger.LogInformation("DEBUG@example.csx - Init");
        return 0; // Required: int error value return
    }

    // Required: Updaet method as required to extend IScript
    public int Update(Indexer.Models.ICallbackInfos callbackInfos)
    {
        ToolSet.Logger.LogInformation("DEBUG@example.csx - Update");
        EntityQueryResults test = ToolSet.Client.EntityQueryAsync(defaultSearchdomain, "DNA").Result;
        var firstResult = test.Results.ToArray()[0];
        ToolSet.Logger.LogInformation(firstResult.Name);
        ToolSet.Logger.LogInformation(firstResult.Value.ToString());
        return 0; // Required: int error value return
    }

    // Required: int error value return
    public int Stop()
    {
        ToolSet.Logger.LogInformation("DEBUG@example.csx - Stop");
        return 0; // Required: int error value return
    }
}

// Required: return an instance of your IScript-extending class
return new ExampleScript();
```
## Lua
TODO
## Javascript
TODO