Use JinaAI models for embeddings (#14252)

* add generic onnx model class and use jina ai clip models for all embeddings

* fix merge confligt

* add generic onnx model class and use jina ai clip models for all embeddings

* fix merge confligt

* preferred providers

* fix paths

* disable download progress bar

* remove logging of path

* drop and recreate tables on reindex

* use cache paths

* fix model name

* use trust remote code per transformers docs

* ensure tokenizer and feature extractor are correctly loaded

* revert

* manually download and cache feature extractor config

* remove unneeded

* remove old clip and minilm code

* docs update
This commit is contained in:
Josh Hawkins
2024-10-09 15:31:54 -06:00
committed by GitHub
parent dbeaf43b8f
commit d4925622f9
7 changed files with 277 additions and 331 deletions
+1 -1
View File
@@ -73,7 +73,7 @@ class EmbeddingsContext:
def __init__(self, db: SqliteVecQueueDatabase):
self.embeddings = Embeddings(db)
self.thumb_stats = ZScoreNormalization()
self.desc_stats = ZScoreNormalization(scale_factor=3, bias=-2.5)
self.desc_stats = ZScoreNormalization()
# load stats from disk
try: