lucretia/embeddingsearch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fixed malformed JSON, added To-do

Browse Source
This commit is contained in:
EzFeDezy
2025-04-23 23:44:23 +02:00
parent 6b3e363e35
commit 6e0d82ad3c
1 changed files with 37 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
README.md
View File

@@ -71,45 +71,47 @@ Deletes a searchdomain and its corresponding entites.
## Entity ## Entity
### Create / Index entity ### Create / Index entity
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e $entity_as_JSON` `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e $entities_as_JSON`
Creates the entity using the json string as specified under $entity_as_JSON Creates the entities using the json string as specified under $entities_as_JSON
Example: Example:
- Linux: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '{"name": "myfile.txt", "probmethod": "weighted_average", "searchdomain": "mysearchdomain", "attributes": {"mimetype": "text-plain"}, "datapoints": [{"name": "text", "text": "this is the full text", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}, {"name": "filepath", "text": "/home/myuser/myfile.txt", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}]}'` - Linux: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{"name": "myfile.txt", "probmethod": "weighted_average", "searchdomain": "mysearchdomain", "attributes": {"mimetype": "text-plain"}, "datapoints": [{"name": "text", "text": "this is the full text", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}, {"name": "filepath", "text": "/home/myuser/myfile.txt", "probmethod_embedding": "weighted_average", "model": ["bge-m3", "nomic-embed-text", "paraphrase-multilingual"]}]}]'`
- Powershell: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '{\"name\": \"myfile.txt\", \"probmethod\": \"weighted_average\", \"searchdomain\": \"mysearchdomain\", \"attributes\": {\"mimetype\": \"text-plain\"}, \"datapoints\": [{\"name\": \"text\", \"text\": \"this is the full text\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}, {\"name\": \"filepath\", \"text\": \"\/home\/myuser\/myfile.txt\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}]}'` - Powershell: `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --index -o $ollama_URL -s $searchdomain_name -e '[{\"name\": \"myfile.txt\", \"probmethod\": \"weighted_average\", \"searchdomain\": \"mysearchdomain\", \"attributes\": {\"mimetype\": \"text-plain\"}, \"datapoints\": [{\"name\": \"text\", \"text\": \"this is the full text\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}, {\"name\": \"filepath\", \"text\": \"\/home\/myuser\/myfile.txt\", \"probmethod_embedding\": \"weighted_average\", \"model\": [\"bge-m3\", \"nomic-embed-text\", \"paraphrase-multilingual\"]}]}]'`
Only the json: Only the json:
```json ```json
{ [
"name": "myfile.txt", {
"probmethod": "weighted_average", "name": "myfile.txt",
"searchdomain": "mysearchdomain", "probmethod": "weighted_average",
"attributes": { "searchdomain": "mysearchdomain",
"mimetype": "text-plain" "attributes": {
}, "mimetype": "text-plain"
"datapoints": [
{
"name": "text",
"text": "this is the full text",
"probmethod_embedding": "weighted_average",
"model": [
"bge-m3",
"nomic-embed-text",
"paraphrase-multilingual"
]
}, },
{ "datapoints": [
"name": "filepath", {
"text": "/home/myuser/myfile.txt", "name": "text",
"probmethod_embedding": "weighted_average", "text": "this is the full text",
"model": [ "probmethod_embedding": "weighted_average",
"bge-m3", "model": [
"nomic-embed-text", "bge-m3",
"paraphrase-multilingual" "nomic-embed-text",
] "paraphrase-multilingual"
} ]
] },
} {
"name": "filepath",
"text": "/home/myuser/myfile.txt",
"probmethod_embedding": "weighted_average",
"model": [
"bge-m3",
"nomic-embed-text",
"paraphrase-multilingual"
]
}
]
}
]
``` ```
### Evaluate query (i.e. "search"; that what you're here for) ### Evaluate query (i.e. "search"; that what you're here for)
`src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --evaluate -o $ollama_URL -s $searchdomain_name -q $query_string [-n $max_results]` `src/cli/bin/Debug/net8.0/cli -h $mysql_ip -p $mysql_port -U $mysql_username -P $mysql_password --entity --evaluate -o $ollama_URL -s $searchdomain_name -q $query_string [-n $max_results]`
@@ -134,7 +136,8 @@ Deletes the entity specified by `$entity_name`.
| Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'localhost' for user 'embeddingsearch' using method 'caching_sha2_password' failed with message: Access denied for user 'embeddingsearch'@'localhost' (using password: YES) | TBD | | Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'localhost' for user 'embeddingsearch' using method 'caching_sha2_password' failed with message: Access denied for user 'embeddingsearch'@'localhost' (using password: YES) | TBD |
# To-do # To-do
- Implement environment variable use - Add "Click-Through" result evaluation (For each entity: store a list of queries that led to the entity being chosen by the user. Then at query-time choose the best-fitting entry and maybe use it as another datapoint? Or use a separate weight function?)
- Implement environment variable use in CLI
- fix the `--help` functionality - fix the `--help` functionality
- Rename `cli` to something unique but still short, e.g. `escli`? - Rename `cli` to something unique but still short, e.g. `escli`?
- Improve error messaging for when retrieving a searchdomain fails. - Improve error messaging for when retrieving a searchdomain fails.