Removed remaining CLI mentions
This commit is contained in:
12
README.md
12
README.md
@@ -6,7 +6,6 @@ Embeddingsearch is a DotNet C# library that uses Embedding Similarity Search (si
|
|||||||
This repository comes with
|
This repository comes with
|
||||||
- a server (accessible via API calls & swagger)
|
- a server (accessible via API calls & swagger)
|
||||||
- a clientside library
|
- a clientside library
|
||||||
- a CLI module (deprecated)
|
|
||||||
- a scripting based indexer service that supports
|
- a scripting based indexer service that supports
|
||||||
- Python
|
- Python
|
||||||
- Golang (WIP)
|
- Golang (WIP)
|
||||||
@@ -32,20 +31,12 @@ This repository comes with
|
|||||||
3. [Configure the indexer](docs/Indexer.md#configuration)
|
3. [Configure the indexer](docs/Indexer.md#configuration)
|
||||||
4. [Set up your indexing script(s)](docs/Indexer.md#scripting)
|
4. [Set up your indexing script(s)](docs/Indexer.md#scripting)
|
||||||
5. Run with `dotnet build && dotnet run` (Or `/usr/bin/dotnet build && /usr/bin/dotnet run`)
|
5. Run with `dotnet build && dotnet run` (Or `/usr/bin/dotnet build && /usr/bin/dotnet run`)
|
||||||
## CLI
|
|
||||||
Before anything follow these steps:
|
|
||||||
1. Enter the project's `src` directory (used as the working directory in all examples)
|
|
||||||
2. Build the project: `dotnet build`
|
|
||||||
All user-defined parameters are denoted using the `$` symbol. I.e. `$mysql_ip` means: replace this with your MySQL IP address or set it as a local variable in your terminal session.
|
|
||||||
|
|
||||||
All commands, parameters and examples are documented here: [docs/CLI.md](docs/CLI.md)
|
|
||||||
# Known issues
|
# Known issues
|
||||||
| Issue | Solution |
|
| Issue | Solution |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Failed to load /usr/lib/dotnet/host/fxr/8.0.15/libhostfxr.so, error: /snap/core20/current/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/dotnet/host/fxr/8.0.15/libhostfxr.so) | You likely installed dotnet via snap instead of apt. Try running the CLI using `/usr/bin/dotnet` instead of `dotnet`. |
|
|
||||||
| Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Invalid attempt to access a field before calling Read() | The searchdomain you entered does not exist |
|
| Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Invalid attempt to access a field before calling Read() | The searchdomain you entered does not exist |
|
||||||
| Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'localhost' for user 'embeddingsearch' using method 'caching_sha2_password' failed with message: Access denied for user 'embeddingsearch'@'localhost' (using password: YES) | TBD |
|
| Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'localhost' for user 'embeddingsearch' using method 'caching_sha2_password' failed with message: Access denied for user 'embeddingsearch'@'localhost' (using password: YES) | TBD |
|
||||||
| System.DllNotFoundException: Could not load libpython3.12.so with flags RTLD_NOW \| RTLD_GLOBAL: libpython3.12.so: cannot open shared object file: No such file or directory | Install python3.12-dev via apt |
|
| System.DllNotFoundException: Could not load libpython3.12.so with flags RTLD_NOW \| RTLD_GLOBAL: libpython3.12.so: cannot open shared object file: No such file or directory | Install python3.12-dev via apt. Also: try running the indexer using `/usr/bin/dotnet` instead of `dotnet` (make sure dotnet is installed via apt) |
|
||||||
# To-do
|
# To-do
|
||||||
- (High priority) Add default indexer
|
- (High priority) Add default indexer
|
||||||
- Library
|
- Library
|
||||||
@@ -77,7 +68,6 @@ All commands, parameters and examples are documented here: [docs/CLI.md](docs/CL
|
|||||||
- Level 2: `{"Name": "...", "Value": 0.53, "Datapoints": [{"Name": "title", "Value": 0.65, "Embeddings": [{"Model": "bge-m3", "Value": 0.87}, {...}]}, {...}]}`
|
- Level 2: `{"Name": "...", "Value": 0.53, "Datapoints": [{"Name": "title", "Value": 0.65, "Embeddings": [{"Model": "bge-m3", "Value": 0.87}, {...}]}, {...}]}`
|
||||||
- Add "Click-Through" result evaluation (For each entity: store a list of queries that led to the entity being chosen by the user. Then at query-time choose the best-fitting entry and maybe use it as another datapoint? Or use a separate weight function?)
|
- Add "Click-Through" result evaluation (For each entity: store a list of queries that led to the entity being chosen by the user. Then at query-time choose the best-fitting entry and maybe use it as another datapoint? Or use a separate weight function?)
|
||||||
- Reranker/Crossencoder/RAG (or anything else beyond initial retrieval) support
|
- Reranker/Crossencoder/RAG (or anything else beyond initial retrieval) support
|
||||||
- Remove the CLI
|
|
||||||
- Improve error messaging for when retrieving a searchdomain fails.
|
- Improve error messaging for when retrieving a searchdomain fails.
|
||||||
- Remove the `id` collumns from the database tables where the table is actually identified (and should be unique by) the name, which should become the new primary key.
|
- Remove the `id` collumns from the database tables where the table is actually identified (and should be unique by) the name, which should become the new primary key.
|
||||||
- Improve performance & latency (Create ready-to-go processes where each contain an n'th share of the entity cache, ready to perform a query. Prepare it after creating the entity cache.)
|
- Improve performance & latency (Create ready-to-go processes where each contain an n'th share of the entity cache, ready to perform a query. Prepare it after creating the entity cache.)
|
||||||
|
|||||||
86
docs/CLI.md
86
docs/CLI.md
@@ -1,86 +0,0 @@
|
|||||||
# CLI
|
|
||||||
## Database
|
|
||||||
### Create or check
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --database --create [--setup]`
|
|
||||||
|
|
||||||
Without the `--setup` parameter a "dry-run" is performed. I.e. no actions are taken. Only the database is checked for read access and that all tables exist.
|
|
||||||
|
|
||||||
## Searchdomain
|
|
||||||
### Create a searchdomain
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --create -s $searchdomain_name`
|
|
||||||
|
|
||||||
Creates the searchdomain as specified under `$searchdomain_name`
|
|
||||||
|
|
||||||
### List searchdomains
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --list`
|
|
||||||
|
|
||||||
List all searchdomains
|
|
||||||
|
|
||||||
### Update searchdomain
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --update -s $searchdomain_name [-n $searchdomain_newname] [-S $searchdomain_newsettings]`
|
|
||||||
|
|
||||||
Set a new name and/or update the settings for the searchdomain.
|
|
||||||
|
|
||||||
### Delete searchdomain
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --delete -s $searchdomain_name`
|
|
||||||
|
|
||||||
Deletes a searchdomain and its corresponding entites.
|
|
||||||
|
|
||||||
## Entity
|
|
||||||
### Create / Index entity
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e $entities_as_JSON`
|
|
||||||
Creates the entities using the json string as specified under $entities_as_JSON
|
|
||||||
|
|
||||||
Example:
|
|
||||||
- Linux: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{"name": "myfile.txt", "probmethod": "weighted_average", "searchdomain": "mysearchdomain", "attributes": {"mimetype": "text-plain"}, "datapoints": [{"name": "text", "text": "this is the full text", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}, {"name": "filepath", "text": "/home/myuser/myfile.txt", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}]}]'`
|
|
||||||
- Powershell: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{\"name\": \"myfile.txt\", \"probmethod\": \"weighted_average\", \"searchdomain\": \"mysearchdomain\", \"attributes\": {\"mimetype\": \"text-plain\"}, \"datapoints\": [{\"name\": \"text\", \"text\": \"this is the full text\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}, {\"name\": \"filepath\", \"text\": \"\/home\/myuser\/myfile.txt\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}]}]'`
|
|
||||||
|
|
||||||
Only the json:
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"name": "myfile.txt",
|
|
||||||
"probmethod": "weighted_average",
|
|
||||||
"searchdomain": "mysearchdomain",
|
|
||||||
"attributes": {
|
|
||||||
"mimetype": "text-plain"
|
|
||||||
},
|
|
||||||
"datapoints": [
|
|
||||||
{
|
|
||||||
"name": "text",
|
|
||||||
"text": "this is the full text",
|
|
||||||
"probmethod_embedding": "weighted_average",
|
|
||||||
"model": [
|
|
||||||
"bge-m3",
|
|
||||||
"nomic-embed-text",
|
|
||||||
"paraphrase-multilingual"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "filepath",
|
|
||||||
"text": "/home/myuser/myfile.txt",
|
|
||||||
"probmethod_embedding": "weighted_average",
|
|
||||||
"model": [
|
|
||||||
"bge-m3",
|
|
||||||
"nomic-embed-text",
|
|
||||||
"paraphrase-multilingual"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
### Evaluate query (i.e. "search"; that what you're here for)
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --evaluate -o $ollama_URL -s $searchdomain_name -q $query_string [-n $max_results]`
|
|
||||||
|
|
||||||
Executes a search using the specified query string and outputs the results.
|
|
||||||
|
|
||||||
### List entities
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --list -s $searchdomain_name`
|
|
||||||
|
|
||||||
Lists all entities in that domain (together with its attributes and datapoints and probmethod)
|
|
||||||
|
|
||||||
### Delete entity
|
|
||||||
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --remove -s $searchdomain_name -n $entity_name`
|
|
||||||
|
|
||||||
Deletes the entity specified by `$entity_name`.
|
|
||||||
Reference in New Issue
Block a user