From 60bfcff539dd8f405d0ad2f066953463b910f3ff Mon Sep 17 00:00:00 2001 From: LD-Reborn Date: Sun, 8 Jun 2025 14:06:30 +0200 Subject: [PATCH] Removed remaining CLI mentions --- README.md | 12 +------- docs/CLI.md | 86 ----------------------------------------------------- 2 files changed, 1 insertion(+), 97 deletions(-) delete mode 100644 docs/CLI.md diff --git a/README.md b/README.md index 55cba9b..9867638 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,6 @@ Embeddingsearch is a DotNet C# library that uses Embedding Similarity Search (si This repository comes with - a server (accessible via API calls & swagger) - a clientside library -- a CLI module (deprecated) - a scripting based indexer service that supports - Python - Golang (WIP) @@ -32,20 +31,12 @@ This repository comes with 3. [Configure the indexer](docs/Indexer.md#configuration) 4. [Set up your indexing script(s)](docs/Indexer.md#scripting) 5. Run with `dotnet build && dotnet run` (Or `/usr/bin/dotnet build && /usr/bin/dotnet run`) -## CLI -Before anything follow these steps: -1. Enter the project's `src` directory (used as the working directory in all examples) -2. Build the project: `dotnet build` -All user-defined parameters are denoted using the `$` symbol. I.e. `$mysql_ip` means: replace this with your MySQL IP address or set it as a local variable in your terminal session. - -All commands, parameters and examples are documented here: [docs/CLI.md](docs/CLI.md) # Known issues | Issue | Solution | | --- | --- | -| Failed to load /usr/lib/dotnet/host/fxr/8.0.15/libhostfxr.so, error: /snap/core20/current/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/dotnet/host/fxr/8.0.15/libhostfxr.so) | You likely installed dotnet via snap instead of apt. Try running the CLI using `/usr/bin/dotnet` instead of `dotnet`. | | Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Invalid attempt to access a field before calling Read() | The searchdomain you entered does not exist | | Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'localhost' for user 'embeddingsearch' using method 'caching_sha2_password' failed with message: Access denied for user 'embeddingsearch'@'localhost' (using password: YES) | TBD | -| System.DllNotFoundException: Could not load libpython3.12.so with flags RTLD_NOW \| RTLD_GLOBAL: libpython3.12.so: cannot open shared object file: No such file or directory | Install python3.12-dev via apt | +| System.DllNotFoundException: Could not load libpython3.12.so with flags RTLD_NOW \| RTLD_GLOBAL: libpython3.12.so: cannot open shared object file: No such file or directory | Install python3.12-dev via apt. Also: try running the indexer using `/usr/bin/dotnet` instead of `dotnet` (make sure dotnet is installed via apt) | # To-do - (High priority) Add default indexer - Library @@ -77,7 +68,6 @@ All commands, parameters and examples are documented here: [docs/CLI.md](docs/CL - Level 2: `{"Name": "...", "Value": 0.53, "Datapoints": [{"Name": "title", "Value": 0.65, "Embeddings": [{"Model": "bge-m3", "Value": 0.87}, {...}]}, {...}]}` - Add "Click-Through" result evaluation (For each entity: store a list of queries that led to the entity being chosen by the user. Then at query-time choose the best-fitting entry and maybe use it as another datapoint? Or use a separate weight function?) - Reranker/Crossencoder/RAG (or anything else beyond initial retrieval) support -- Remove the CLI - Improve error messaging for when retrieving a searchdomain fails. - Remove the `id` collumns from the database tables where the table is actually identified (and should be unique by) the name, which should become the new primary key. - Improve performance & latency (Create ready-to-go processes where each contain an n'th share of the entity cache, ready to perform a query. Prepare it after creating the entity cache.) diff --git a/docs/CLI.md b/docs/CLI.md deleted file mode 100644 index f683a30..0000000 --- a/docs/CLI.md +++ /dev/null @@ -1,86 +0,0 @@ -# CLI -## Database -### Create or check -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --database --create [--setup]` - -Without the `--setup` parameter a "dry-run" is performed. I.e. no actions are taken. Only the database is checked for read access and that all tables exist. - -## Searchdomain -### Create a searchdomain -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --create -s $searchdomain_name` - -Creates the searchdomain as specified under `$searchdomain_name` - -### List searchdomains -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --list` - -List all searchdomains - -### Update searchdomain -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --update -s $searchdomain_name [-n $searchdomain_newname] [-S $searchdomain_newsettings]` - -Set a new name and/or update the settings for the searchdomain. - -### Delete searchdomain -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --searchdomain --delete -s $searchdomain_name` - -Deletes a searchdomain and its corresponding entites. - -## Entity -### Create / Index entity -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e $entities_as_JSON` -Creates the entities using the json string as specified under $entities_as_JSON - -Example: -- Linux: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{"name": "myfile.txt", "probmethod": "weighted_average", "searchdomain": "mysearchdomain", "attributes": {"mimetype": "text-plain"}, "datapoints": [{"name": "text", "text": "this is the full text", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}, {"name": "filepath", "text": "/home/myuser/myfile.txt", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}]}]'` -- Powershell: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{\"name\": \"myfile.txt\", \"probmethod\": \"weighted_average\", \"searchdomain\": \"mysearchdomain\", \"attributes\": {\"mimetype\": \"text-plain\"}, \"datapoints\": [{\"name\": \"text\", \"text\": \"this is the full text\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}, {\"name\": \"filepath\", \"text\": \"\/home\/myuser\/myfile.txt\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}]}]'` - -Only the json: -```json -[ - { - "name": "myfile.txt", - "probmethod": "weighted_average", - "searchdomain": "mysearchdomain", - "attributes": { - "mimetype": "text-plain" - }, - "datapoints": [ - { - "name": "text", - "text": "this is the full text", - "probmethod_embedding": "weighted_average", - "model": [ - "bge-m3", - "nomic-embed-text", - "paraphrase-multilingual" - ] - }, - { - "name": "filepath", - "text": "/home/myuser/myfile.txt", - "probmethod_embedding": "weighted_average", - "model": [ - "bge-m3", - "nomic-embed-text", - "paraphrase-multilingual" - ] - } - ] - } -] -``` -### Evaluate query (i.e. "search"; that what you're here for) -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --evaluate -o $ollama_URL -s $searchdomain_name -q $query_string [-n $max_results]` - -Executes a search using the specified query string and outputs the results. - -### List entities -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --list -s $searchdomain_name` - -Lists all entities in that domain (together with its attributes and datapoints and probmethod) - -### Delete entity -`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --remove -s $searchdomain_name -n $entity_name` - -Deletes the entity specified by `$entity_name`. \ No newline at end of file