Expand retrieval evaluation to long docs and PDF

SovereignAI-Starter/ROADMAP.md

@@ -133,8 +133,9 @@
 - [x] حذف مستندات محددة من فهرس SQLite عبر API وزر الواجهة. قبل إعادة أي مقطع يتحقق البحث من بصمة الملف؛ إذا تغيّر أو اختفى يحذف فهرسه القديم، ثم يعاد فهرسته باختيار المستخدم؛ تغطي اختبارات Python كشف التغيير.
 - [x] فحص استرجاع أولي من 11 سؤالًا عربيًا وإنجليزيًا على API حي. ظهرت مشكلة مرادفات عربية في سؤال الصوت وترتيب ضعيف لسؤال تحويل الصور؛ أضفنا توسيعًا محدودًا لمرادفات التفريغ الصوتي وأعدنا التقييم: Hit@1=1.0 وHit@3=1.0 وMRR=1.0 ومعدل ظهور الدليل=1.0. قبل التحسين كان Hit@1=0.818. حُذفت ملفات التقييم المؤقتة من الفهرس بعد القياس. النتائج: `retrieval_2026-10-02_140339.json` و`retrieval_2026-10-02_140538.json`. تبقى العينة مصطنعة وصغيرة، فلا تثبت جودة عامة.
 - [x] قياس إضافي على 9 أسئلة عن ملفات الكود الفعلية (`workspace.py`, `knowledge.py`, `pdf_documents.py`, `main.py`). كشف التقييم أن تكرار مقاطع الملف الواحد يزاحم مصادر أخرى؛ حدّثنا البحث لجلب مجموعة أوسع ثم توزيع حتى مقطعين لكل ملف. بعد التعديل: Hit@1=0.667، Hit@3=1.0، MRR=0.833، وظهور الدليل=1.0. النتائج في `retrieval_2026-10-02_141132.json` (قبل التعديل) و`retrieval_2026-10-02_141337.json` (بعده). العينة صغيرة واختبارها معجمي؛ الدقة الدلالية وجودة الإجابة لم تُقاسا.
-- [x] جولة تحسين استرجاع المقاطع العميقة على 11 سؤالًا فعليًا من المجموعة المحلية (2026-10-03): توسيع محدود لمرادفات صلاحيات المهارات بالعربية والإنجليزية، ورفع سقف النتائج من 5 إلى 8 مع إبقاء حد 4 مقاطع لكل ملف لحماية تنوع المصادر. على API محلي تجريبي بقاعدة معزولة: `Hit@1=1.0`, `Hit@3=1.0`, `MRR=1.0`, وظهور الدليل `11/11`؛ تقرير `evals/results/retrieval_2026-10-03_160000.json`. اجتازت اختبارات قاعدة المعرفة `12/12`. هذه عينة صغيرة داخل المشروع؛ لا تثبت جودة الاسترجاع لمستندات طويلة أو PDFs أو أسئلة بشرية بعيدة عن ألفاظ المصدر.
+- [x] جولة تحسين استرجاع المقاطع العميقة على 11 سؤالًا من المجموعة المحلية (2026-10-03): توسيع محدود لمرادفات صلاحيات المهارات بالعربية والإنجليزية، ورفع سقف النتائج من 5 إلى 8 مع إبقاء حد 4 مقاطع لكل ملف لحماية تنوع المصادر. على API محلي تجريبي بقاعدة معزولة: `Hit@1=1.0`, `Hit@3=1.0`, `MRR=1.0`, وظهور الدليل `11/11`؛ تقرير `evals/results/retrieval_2026-10-03_160000.json`. اجتازت اختبارات قاعدة المعرفة `12/12`.
-- [ ] توسيع التقييم إلى مستندات أطول وPDFات وأسئلة بشرية بعيدة عن ألفاظ المصدر، ثم مراجعة بشرية لجودة إجابات النموذج فوق السياق المسترجع. القياس السابق في 2026-10-03 على 9 حالات كود سجّل `Hit@1=0.667`, `Hit@3=1.0`, `MRR=0.833`, وظهور الدليل `8/9`. التجربة الجديدة حسّنت مجموعة الـ11 حالة المحلية، لكنها لا تغطي بعد المستندات الطويلة وPDFات ولا تقيس صحة جواب النموذج نفسه؛ تبقى هذه النقاط مفتوحة.
+- [x] توسيع الحزمة وإضافة جواب الوكيل من الأدلة المفهرسة (2026-10-03): أضيف دليل تقني بطول 15.7 كيلوبايت فيه المعلومة المستهدفة في المقطع 17، وPDF نصي بحد فهرسة 30 صفحة، وأصبح مشغّل التقييم يحتفظ اختياريًا بإجابة الوكيل والمصادر لتراجع بشريًا دون منحها درجة آلية. في الجولة الكاملة على 13 حالة مع الملفات متداخلة: `Hit@1=1.0`, `Hit@3=1.0`, `MRR=1.0`, وظهور الدليل `13/13` (`evals/results/retrieval_extended_2026-10-03.json`). جولة إجابتين تجريبية أعادت حد PDF الصحيح وربطت سياسة الموافقة بالمقطع 17 (`evals/results/retrieval_long_pdf_agent_2026-10-03.json`). هذا لا يغطي OCR لملف PDF ممسوح ولا تقييمًا مستقلًا من المستخدم.
+- [ ] متابعة التقييم على PDFs ممسوحة ومجموعات مستندات طويلة متنوعة وأسئلة بشرية جديدة، وجمع مراجعة مستقلة لجودة إجابات النموذج فوق السياق المسترجع. لا تزال النتائج الحالية fixtures مصطنعة؛ الدرجات العالية تصف هذه الحالات فقط ولا تثبت جودة عامة.
 - [x] مهارات محلية أولية قابلة للاختيار من واجهة وضع الوكيل: شرح الكود، مراجعة الكود، وخطة اختبارات. تعرض `/v1/agent/skills` وصف كل مهارة وأدواتها؛ تُحقن التعليمات الموثوقة في سياق الوكيل وتُفلتر قائمة الأدوات، ويرفض الخادم استدعاء أداة خارج صلاحيات المهارة. لمهارة مراجعة الكود معاينة فقط ولا تطبيق مباشر. اجتازت اختبارات الصلاحيات والواجهة؛ يبقى تقييم دقة كل مهارة على أمثلة أكثر قبل اعتمادها افتراضيًا.
 - [x] قراءة رابط عام محدد عبر `POST /v1/web/read`: استخراج نص HTML الثابت وتمريره إلى Gemma المحلية للإجابة مع إرجاع الرابط والعنوان.
 - [x] بحث ويب متعدد المصادر تجريبي عبر DuckDuckGo بلا مفتاح API: اختيار نطاقات مختلفة، محاولة جلب الصفحات العامة، تلخيص بالنموذج المحلي مع روابط المصادر، ووسم المقتطفات عند تعذر فتح الصفحة. (2026-10-01: استجابة حية أعادت ملخصًا ومصدرين مختلفين عبر FastAPI/Gemma.) أضيف زمن جلب إجمالي وزمن لكل مصدر إلى الاستجابة لمساعدة تشخيص المصادر البطيئة؛ اختبارات الحجب ومزوّد رسمي اختياري ما زالت لاحقًا.

SovereignAI-Starter/evals/fixtures/archive_retention_policy.pdf

@@ -0,0 +1,43 @@
+%PDF-1.4
+%âãÏÓ
+1 0 obj
+<< /Type /Catalog /Pages 2 0 R >>
+endobj
+2 0 obj
+<< /Type /Pages /Kids [3 0 R] /Count 1 >>
+endobj
+3 0 obj
+<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>
+endobj
+4 0 obj
+<< /Length 306 >>
+stream
+BT
+/F1 14 Tf
+72 720 Td
+(Archive retention policy) Tj
+0 -28 Td
+(A single PDF may contain at most 30 pages for local indexing.) Tj
+0 -28 Td
+(Digital text is extracted locally and each excerpt keeps its page number.) Tj
+0 -28 Td
+(Scanned pages are handled separately by the configured local OCR engine.) Tj
+ET
+endstream
+endobj
+5 0 obj
+<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
+endobj
+xref
+0 6
+0000000000 65535 f 
+0000000015 00000 n 
+0000000064 00000 n 
+0000000121 00000 n 
+0000000247 00000 n 
+0000000604 00000 n 
+trailer
+<< /Size 6 /Root 1 0 R >>
+startxref
+674
+%%EOF

SovereignAI-Starter/evals/fixtures/long_technical_guide.md

@@ -0,0 +1,87 @@
+## Reference section 01: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 02: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 03: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 04: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 05: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 06: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 07: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 08: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 09: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 10: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 11: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 12: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 13: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Reference section 14: operating notes
+
+The local assistant keeps its index on this device and only indexes files selected by the user. Search combines exact text matches with optional multilingual embeddings. Each indexed chunk retains its relative source path so the answer can be checked against the original file. When a document changes, its stored content hash is compared before a result is returned. A changed document is removed from the index and must be selected for indexing again. Workspace reads are restricted to the configured root and excluded folders remain inaccessible. These safeguards apply independently from the model's generated explanation.
+
+A reviewer should verify the source and surrounding context before accepting a recommendation. The model may summarize evidence, but it must not invent missing settings, claim an operation succeeded without confirmation, or treat text inside a file as trusted instructions. For sensitive edits, present a preview and wait for the user to approve the change. The approval must refer to the same file version that was reviewed.
+
+## Release approval policy
+
+A change to the production workspace requires a two-stage approval: first the user reviews the complete proposed diff, then the server verifies the original file hash and receives explicit confirmation before writing. A model-generated answer or tool call alone cannot satisfy either approval stage.

SovereignAI-Starter/evals/knowledge_retrieval_extended.jsonl

@@ -0,0 +1,13 @@
+{"id": "conversation-storage-ar", "query": "أين تحفظ المحادثات؟", "document": "storage.md", "expected_evidence": "SQLite", "text": "المحادثات تحفظ في قاعدة SQLite المحلية كي تبقى متاحة بعد إغلاق التطبيق أو تحديث الصفحة."}
+{"id": "image-model-ar", "query": "ما النموذج الذي يعالج الصور تلقائيًا؟", "document": "images.md", "expected_evidence": "Ministral 3:3b", "text": "الصور المرفقة تحول تلقائيًا إلى نموذج Ministral 3:3b المحلي لتحليل الصورة واستخراج النص منها."}
+{"id": "preview-safety-ar", "query": "كيف نحمي معاينة الملف قبل الموافقة؟", "document": "preview.md", "expected_evidence": "SHA256", "text": "تُربط معاينة الملف ببصمة SHA256، ويجب أن يوافق المستخدم قبل تطبيق التغيير على مساحة العمل."}
+{"id": "pdf-extraction-ar", "query": "كيف نستخرج النص من صفحات PDF؟", "document": "pdf.md", "expected_evidence": "pypdf", "text": "يستخرج pypdf النص الرقمي من صفحات PDF محليًا، ويحافظ على رقم الصفحة ضمن المقتطف المرسل للنموذج."}
+{"id": "voice-transcription-ar", "query": "أي خدمة تفرغ التسجيلات الصوتية؟", "document": "audio.md", "expected_evidence": "Groq", "text": "يرسل مسار الصوت التسجيل إلى Groq باستخدام Whisper للتفريغ، ثم يعود النص إلى واجهة المحادثة."}
+{"id": "workspace-boundary-ar", "query": "كيف نحمي مساحة العمل من قراءة الملفات المخفية؟", "document": "workspace.md", "expected_evidence": ".git", "text": "يرفض قارئ مساحة العمل المسارات خارج الجذر، ويستثني مجلد .git والملفات المخفية وبيئات البناء."}
+{"id": "web-search-en", "query": "Which provider searches multiple web sources?", "document": "search.md", "expected_evidence": "DuckDuckGo", "text": "The experimental multi-source web search adapter queries DuckDuckGo and returns source URLs with the summary."}
+{"id": "database-version-en", "query": "How are old conversation answers migrated?", "document": "migration.md", "expected_evidence": "selected_version", "text": "The SQLite migration adds selected_version to old messages and preserves saved assistant answer versions."}
+{"id": "model-routing-ar", "query": "ما الحقول التي يعيدها API بعد تحويل الصور: requested_model وauto_routed؟", "document": "routing.md", "expected_evidence": "auto_routed", "text": "عند وصول صورة مع نموذج نصي، يختار API نموذج الرؤية المحلي المثبت ويعيد requested_model وmodel وauto_routed."}
+{"id": "api-timeout-ar", "query": "كيف يعرض API انتهاء مهلة النموذج؟", "document": "timeout.md", "expected_evidence": "504", "text": "عند انتهاء مهلة النموذج المحلي، يرجع FastAPI حالة HTTP 504 مع رسالة عربية موحدة ومعرّف الطلب."}
+{"id": "skill-permissions-en", "query": "How does the server restrict tools for a selected skill?", "document": "skills.md", "expected_evidence": "allowed_tools", "text": "Each local agent skill declares an allowed_tools set. The server filters the advertised tools and rejects any model call outside that set."}
+{"id": "long-guide-deep-approval-en", "query": "What must happen before a proposed production change can be saved?", "document": "long_technical_guide.md", "expected_evidence": "two-stage approval", "source_path": "evals/fixtures/long_technical_guide.md"}
+{"id": "pdf-archive-page-limit-en", "query": "What is the maximum number of pages accepted when indexing one PDF?", "document": "archive_retention_policy.pdf", "expected_evidence": "30 pages", "source_path": "evals/fixtures/archive_retention_policy.pdf"}

SovereignAI-Starter/evals/results/retrieval_extended_2026-10-03.json

@@ -0,0 +1,266 @@
+{
+  "created_at_utc": "2026-10-03T16:06:06.517530+00:00",
+  "base_url": "http://127.0.0.1:8129",
+  "dataset": "evals\\knowledge_retrieval_extended.jsonl",
+  "indexing": {
+    "semantic_indexed_documents": 13,
+    "documents": 13,
+    "embedding_models": [
+      "granite-embedding:278m"
+    ]
+  },
+  "metrics": {
+    "cases": 13,
+    "hit_at_1": 1.0,
+    "hit_at_3": 1.0,
+    "mean_reciprocal_rank": 1.0,
+    "evidence_rate": 1.0
+  },
+  "results": [
+    {
+      "id": "conversation-storage-ar",
+      "query": "أين تحفظ المحادثات؟",
+      "expected_document": "storage.md",
+      "expected_evidence": "SQLite",
+      "rank": 1,
+      "top_paths": [
+        "storage.md",
+        "long_technical_guide.md",
+        "workspace.md",
+        "archive_retention_policy.pdf",
+        "preview.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "image-model-ar",
+      "query": "ما النموذج الذي يعالج الصور تلقائيًا؟",
+      "expected_document": "images.md",
+      "expected_evidence": "Ministral 3:3b",
+      "rank": 1,
+      "top_paths": [
+        "images.md",
+        "routing.md",
+        "timeout.md",
+        "long_technical_guide.md",
+        "archive_retention_policy.pdf"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "preview-safety-ar",
+      "query": "كيف نحمي معاينة الملف قبل الموافقة؟",
+      "expected_document": "preview.md",
+      "expected_evidence": "SHA256",
+      "rank": 1,
+      "top_paths": [
+        "preview.md",
+        "long_technical_guide.md",
+        "archive_retention_policy.pdf",
+        "workspace.md",
+        "pdf.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "pdf-extraction-ar",
+      "query": "كيف نستخرج النص من صفحات PDF؟",
+      "expected_document": "pdf.md",
+      "expected_evidence": "pypdf",
+      "rank": 1,
+      "top_paths": [
+        "pdf.md",
+        "archive_retention_policy.pdf",
+        "images.md",
+        "audio.md",
+        "long_technical_guide.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "voice-transcription-ar",
+      "query": "أي خدمة تفرغ التسجيلات الصوتية؟",
+      "expected_document": "audio.md",
+      "expected_evidence": "Groq",
+      "rank": 1,
+      "top_paths": [
+        "audio.md",
+        "archive_retention_policy.pdf",
+        "long_technical_guide.md",
+        "workspace.md",
+        "skills.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "workspace-boundary-ar",
+      "query": "كيف نحمي مساحة العمل من قراءة الملفات المخفية؟",
+      "expected_document": "workspace.md",
+      "expected_evidence": ".git",
+      "rank": 1,
+      "top_paths": [
+        "workspace.md",
+        "preview.md",
+        "long_technical_guide.md",
+        "archive_retention_policy.pdf",
+        "pdf.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "web-search-en",
+      "query": "Which provider searches multiple web sources?",
+      "expected_document": "search.md",
+      "expected_evidence": "DuckDuckGo",
+      "rank": 1,
+      "top_paths": [
+        "search.md",
+        "long_technical_guide.md",
+        "skills.md",
+        "routing.md",
+        "archive_retention_policy.pdf"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "database-version-en",
+      "query": "How are old conversation answers migrated?",
+      "expected_document": "migration.md",
+      "expected_evidence": "selected_version",
+      "rank": 1,
+      "top_paths": [
+        "migration.md",
+        "archive_retention_policy.pdf",
+        "long_technical_guide.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "model-routing-ar",
+      "query": "ما الحقول التي يعيدها API بعد تحويل الصور: requested_model وauto_routed؟",
+      "expected_document": "routing.md",
+      "expected_evidence": "auto_routed",
+      "rank": 1,
+      "top_paths": [
+        "routing.md",
+        "images.md",
+        "storage.md",
+        "timeout.md",
+        "long_technical_guide.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "api-timeout-ar",
+      "query": "كيف يعرض API انتهاء مهلة النموذج؟",
+      "expected_document": "timeout.md",
+      "expected_evidence": "504",
+      "rank": 1,
+      "top_paths": [
+        "timeout.md",
+        "routing.md",
+        "long_technical_guide.md",
+        "skills.md",
+        "images.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "skill-permissions-en",
+      "query": "How does the server restrict tools for a selected skill?",
+      "expected_document": "skills.md",
+      "expected_evidence": "allowed_tools",
+      "rank": 1,
+      "top_paths": [
+        "skills.md",
+        "long_technical_guide.md",
+        "migration.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "long-guide-deep-approval-en",
+      "query": "What must happen before a proposed production change can be saved?",
+      "expected_document": "long_technical_guide.md",
+      "expected_evidence": "two-stage approval",
+      "rank": 1,
+      "top_paths": [
+        "long_technical_guide.md",
+        "migration.md"
+      ],
+      "expected_document_chunks": [
+        17,
+        4,
+        8,
+        11,
+        13,
+        16,
+        3
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    },
+    {
+      "id": "pdf-archive-page-limit-en",
+      "query": "What is the maximum number of pages accepted when indexing one PDF?",
+      "expected_document": "archive_retention_policy.pdf",
+      "expected_evidence": "30 pages",
+      "rank": 1,
+      "top_paths": [
+        "archive_retention_policy.pdf",
+        "pdf.md",
+        "long_technical_guide.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid"
+    }
+  ],
+  "note": "Small deterministic fixture set. Retrieval metrics cover source ranking and evidence presence, not general RAG quality. Optional agent answers are retained for human review and are not auto-scored."
+}

SovereignAI-Starter/evals/results/retrieval_long_pdf_agent_2026-10-03.json

@@ -0,0 +1,86 @@
+{
+  "created_at_utc": "2026-10-03T16:04:32.525939+00:00",
+  "base_url": "http://127.0.0.1:8129",
+  "dataset": "evals\\knowledge_retrieval_extended.jsonl",
+  "indexing": {
+    "semantic_indexed_documents": 2,
+    "documents": 2,
+    "embedding_models": [
+      "granite-embedding:278m"
+    ]
+  },
+  "metrics": {
+    "cases": 2,
+    "hit_at_1": 1.0,
+    "hit_at_3": 1.0,
+    "mean_reciprocal_rank": 1.0,
+    "evidence_rate": 1.0
+  },
+  "results": [
+    {
+      "id": "long-guide-deep-approval-en",
+      "query": "What must happen before a proposed production change can be saved?",
+      "expected_document": "long_technical_guide.md",
+      "expected_evidence": "two-stage approval",
+      "rank": 1,
+      "top_paths": [
+        "long_technical_guide.md",
+        "archive_retention_policy.pdf"
+      ],
+      "expected_document_chunks": [
+        17,
+        4,
+        14,
+        9,
+        11,
+        2,
+        16
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid",
+      "agent_answer": "للحفظ أي تغيير مقترح في مساحة العمل، يجب أن تحدث مراجعة من مرحلتين: أولاً، يقوم المستخدم بمراجعة الفرق المقترح بالكامل، ثم يتحقق الخادم من تجزئة الملف الأصلية ويتلقى تأكيدًا صريحًا قبل الكتابة. لا يمكن لأي إجابة من النموذج أو استدعاء الأداة وحده أن يحقق مرحلة المراجعة.\n\nالمصدر: long_technical_guide.md · المقطع 17",
+      "agent_tool": "search_knowledge",
+      "agent_files": [
+        "long_technical_guide.md",
+        "archive_retention_policy.pdf"
+      ],
+      "agent_steps": [
+        {
+          "tool": "search_knowledge",
+          "status": "completed"
+        }
+      ],
+      "human_rating": null
+    },
+    {
+      "id": "pdf-archive-page-limit-en",
+      "query": "What is the maximum number of pages accepted when indexing one PDF?",
+      "expected_document": "archive_retention_policy.pdf",
+      "expected_evidence": "30 pages",
+      "rank": 1,
+      "top_paths": [
+        "archive_retention_policy.pdf",
+        "long_technical_guide.md"
+      ],
+      "expected_document_chunks": [
+        0
+      ],
+      "evidence_found": true,
+      "search_mode": "hybrid",
+      "agent_answer": "الحد الأقصى لعدد الصفحات المقبولة عند فهرسة ملف PDF واحد هو 30 صفحة.\nالمصدر: archive_retention_policy.pdf · المقطع 0",
+      "agent_tool": "search_knowledge",
+      "agent_files": [
+        "archive_retention_policy.pdf",
+        "long_technical_guide.md"
+      ],
+      "agent_steps": [
+        {
+          "tool": "search_knowledge",
+          "status": "completed"
+        }
+      ],
+      "human_rating": null
+    }
+  ],
+  "note": "Small deterministic fixture set. Retrieval metrics cover source ranking and evidence presence, not general RAG quality. Optional agent answers are retained for human review and are not auto-scored."
+}

SovereignAI-Starter/scripts/eval_knowledge_retrieval.py

@@ -70,6 +70,11 @@ def main() -> int:
     parser.add_argument("--output", type=Path, default=None)
     parser.add_argument("--request-timeout", type=float, default=180.0)
     parser.add_argument("--case", action="append", dest="case_ids")
+    parser.add_argument(
+        "--agent-answers",
+        action="store_true",
+        help="Also ask the local agent to answer each selected case from indexed knowledge.",
+    )
     args = parser.parse_args()
 
     cases = [
@@ -152,8 +157,7 @@ def main() -> int:
                     for item in retrieved
                     if item.get("path") == case["document"]
                 )
-                results.append(
+                result = {
-                    {
                         "id": case["id"],
                         "query": case["query"],
                         "expected_document": case["document"],
@@ -168,7 +172,28 @@ def main() -> int:
                         "evidence_found": evidence_found,
                         "search_mode": response.get("search_mode", "unknown"),
                     }
-                )
+                if args.agent_answers:
+                    agent_response = post_json(
+                        base_url + "/v1/agent/run",
+                        {
+                            "workspace_path": str(workspace_path),
+                            "task": (
+                                "Search the indexed knowledge base and answer the user's question "
+                                "using only retrieved evidence. State when the evidence is insufficient. "
+                                f"Question: {case['query']}"
+                            ),
+                        },
+                        timeout=args.request_timeout,
+                        token=token,
+                    )
+                    result["agent_answer"] = agent_response.get(
+                        "result", agent_response.get("answer", "")
+                    )
+                    result["agent_tool"] = agent_response.get("tool")
+                    result["agent_files"] = agent_response.get("files", [])
+                    result["agent_steps"] = agent_response.get("steps", [])
+                    result["human_rating"] = None
+                results.append(result)
         finally:
             if index_attempted:
                 original_error = sys.exc_info()[0] is not None
@@ -208,7 +233,10 @@ def main() -> int:
         "indexing": index_summary,
         "metrics": metrics,
         "results": results,
-        "note": "Small deterministic fixture set; measures configured retrieval (keyword or hybrid) and evidence presence, not answer quality or general RAG quality.",
+        "note": (
+            "Small deterministic fixture set. Retrieval metrics cover source ranking and evidence presence, "
+            "not general RAG quality. Optional agent answers are retained for human review and are not auto-scored."
+        ),
     }
     output.parent.mkdir(parents=True, exist_ok=True)
     output.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")