build: add preliminary dependency component inventory

2026-10-04 04:50:16 +03:00
parent f6a2e8c3d3
commit 36a8db510c
6 changed files with 5028 additions and 0 deletions
@@ -25,6 +25,7 @@

 ## ما لم يُراجع بعد قبل التوزيع

+- جرد مكونات التطوير متاح الآن في [`sbom/component-inventory.json`](sbom/component-inventory.json)، مع طريقة التوليد والحدود في [`sbom/README.md`](sbom/README.md). هذا لا يغلق مراجعة التراخيص: 101 ملف رخصة Flutter تحتاج تصنيفًا يدويًا، كما أن الجرد ليس خاصًا بإصدار إنتاج نهائي.
 - مطابقة مصدر كل artifact محلي وبصمته وملف الرخصة المطابق له، ثم إعداد حزمة إشعارات الرخصة للنماذج التي ستُوزع فعلًا.
 - مراجعة شاملة لاعتماديات Python وFlutter وWindows، وإنتاج SBOM وإشعاراتها.
 - رخصة PDFium وإشعارها، ورخصة كل وزن OCR محلي مستخدم فعليًا، بما في ذلك ملفات EasyOCR العربية والإنجليزية.
@@ -169,6 +169,7 @@
 - [x] إنشاء مجموعة أولية من خمسة أسئلة عربية/برمجية مع معايير مراجعة بشرية وفحوص شكل/قيمة محدودة، ومشغّل يسجل الأجوبة ووقت كل طلب واسم أداة الوكيل في JSON. تجربة واحدة على Gemma 4 E2B وQwen 2.5 1.5B وحُفظت النتائج في `evals/results/` (2026-10-02).
 - تكرار التقييم مع إجابات مرجعية ومراجعين/درجات بشرية، وقياس استهلاك الذاكرة واعتماد مقارنة قابلة للتكرار. (2026-10-03: أُعيد التشغيل بمعايير أوسع: Gemma وQwen أكملتا 5/5 طلبات بلا أخطاء؛ Gemma مرّرت 9/9 فحوص شكلية، وQwen 8/9، ومتوسط الزمن 14.77 مقابل 6.25 ثانية. إجابة Qwen عن SQLite كانت غير دقيقة رغم اجتياز فحص وجود ثلاث نقاط، ما يثبت أن هذه الفحوص لا تقيس صحة المعنى. التقريرين النهائيين `evals/results/gemma4_e2b_2026-10-03_183058.json` و`evals/results/qwen2.5_1.5b-instruct-q4_K_M_2026-10-03_183140.json`. التقييمات السابقة المؤرخة `175827` و`180307` تستخدم فحوصًا أضيق. أُضيف `scripts/capture_ollama_runtime.ps1`: أظهر `/api/ps` حجم Gemma المحمّل 6,733,158,152 بايت وQwen 1,169,980,128 بايت، و`size_vram=0` لكليهما. اللقطتان `evals/results/ollama_runtime_2026-10-03_180846.json` و`...180934.json` لحظيتان وليستا peak؛ لا تربطان كل PID بالنموذج ولم تتوفر قراءة إجمالي/متاح RAM بسبب رفض CIM. لم تُسجل درجات بشرية بعد؛ تبقى مراجعة بشرية، قياس ذروة RAM/VRAM موثوق ومتكرر، وجولات إضافية للمقارنة.) الجولة السابقة كانت Gemma 8/10 بمتوسط 16.32 ثانية وQwen 5/10 بمتوسط 8.93 ثانية. أُصلحت منذها مشكلة الحساب المباشر مقابل الأداة وصيغة Qwen الرياضية؛ مشغّل التقييم يستخدم جلسة loopback مؤقتة ويلغيها.
 - [x] جرد أولي لترخيص وسوم Ollama المحلية في `LICENSE_REVIEW_2026-10.md` (2026-10-04): فُحصت `ollama show --license` للأوزان الخمسة المثبتة وبطاقات المصادر الرسمية. Gemma 4 E2B وMinistral 3:3b وQwen2.5 وGranite Embedding أظهرت Apache 2.0؛ `gemma3:1b` تستخدم شروط Gemma الإضافية. وثّق الجرد أيضًا تدفق Groq الصوتي الخارجي واحتفاظ البيانات المحتمل.
+- [x] إنشاء جرد مكونات آلي أولي `sbom/component-inventory.json` باستخدام `scripts/generate_component_inventory.py`: لقطة 2026-10-04 تضم 50 توزيع Python من `.venv` و104 حزم Flutter من `pubspec.lock`، مع بصمات 104 ملفات رخصة وبصمة أرشيف `NOTICES.Z` المضمّن في Windows Debug. اجتازت اختبارات محلّل الجرد 4/4 وفحوص JSON والمرجع الفريد. تُرك التصنيف اليدوي مطلوبًا لـ101 ملف رخصة عثرت عليها الأداة دون تصنيف تلقائي. الملف ليس جرد إصدار إنتاج محصورًا ولا مراجعة قانونية ولم يُتحقق من مخطط CycloneDX الرسمي؛ لا يتضمن علاقات الاعتماد أو كامل مكونات Windows الأصلية وPDFium وأوزان OCR/النماذج.
 - [ ] اعتماد النماذج والاعتماديات لنسخة تجارية محددة: مطابقة مصدر وبصمة كل وزن سيُشحن، حزمة SBOM وإشعارات Python/Flutter/Windows/PDFium/OCR، معالجة قيود Gemma 3 أو استبعادها، وشروط Groq وإفصاح الصوت. لا يُعد الجرد الأولي أعلاه موافقة قانونية أو جاهزية إصدار.
 - تحسين أولي عبر prompts وRAG والأدوات؛ هذه غالبًا تعالج نقص المعرفة أو القدرة على الفعل دون تغيير أوزان النموذج.
 - عند توفر GPU مناسب: تجربة LoRA/QLoRA على بيانات مرخصة ومنقحة، ومقارنة النتائج بالمجموعة المرجعية قبل اعتماد adapter.
@@ -0,0 +1,24 @@
+# Preliminary component inventory
+
+`component-inventory.json` is a point-in-time CycloneDX 1.5 inventory assembled from the active Python virtual environment, `flutter_app/pubspec.lock`, and (when supplied) the bundled Flutter `NOTICES.Z` file. It is an engineering aid, not a complete or legally approved commercial SBOM.
+
+The generator performs local JSON and uniqueness checks; this snapshot has not yet been validated against the official [CycloneDX 1.5 JSON schema](https://github.com/CycloneDX/specification/blob/master/schema/bom-1.5.schema.json). The `pub` PURL type is listed by the [Package-URL type registry](https://github.com/package-url/purl-spec/blob/main/purl-types-index.json).
+
+## Snapshot captured on 2026-10-04
+
+- 50 Python distributions from `.venv`: 8 direct runtime requirements, 1 optional OCR requirement, and 41 installed transitive/development packages. Python versions are the versions installed in this local environment; `requirements.txt` still uses ranges and is not a fully pinned production lock.
+- 104 Flutter pub packages from the resolved lock: 15 direct main, 3 direct development, and 86 transitive/SDK packages.
+- 104 package license files have SHA-256 evidence. For 101 components the inventory can find a license file but deliberately does not classify its legal terms automatically. The remaining declarations come from distribution/package metadata or the Flutter SDK license notice.
+- The Windows Debug artifact bundled `NOTICES.Z` (SHA-256 `eb0096c70ca8a2b23d1a806f1fddb5ce379730712347b267b7c7de4599b8ba70`; 1,836,646 bytes after decompression). The application build already carries this Flutter notice archive.
+
+The inventory does not include dependency edges, a release-only Python environment, all native Windows/C++ components, a complete PDFium notice review, OCR model-weight files, or model weights. It does not decide whether any license permits a specific commercial distribution. Continue the manual source, hash, notice, and terms review in `LICENSE_REVIEW_2026-10.md` for the exact release artifacts.
+
+## Regenerate
+
+From the repository root, after installing Python requirements in `.venv` and resolving Flutter packages:
+
+```powershell
+& .\.venv\Scripts\python.exe .\scripts\generate_component_inventory.py
+```
+
+To hash the notices archive from a Windows build, add `--flutter-notices <path-to-NOTICES.Z>`. The script reads installed Python distribution metadata, the Flutter lock and package cache, and the passed notice archive; it does not install packages or inspect model weights.
@@ -0,0 +1,328 @@
+"""Generate a preliminary CycloneDX inventory from the local Python and Flutter locks."""
+
+from __future__ import annotations
+
+import argparse
+import gzip
+import hashlib
+import importlib.metadata
+import json
+import re
+from datetime import UTC, datetime
+from pathlib import Path
+from urllib.parse import unquote, urlparse
+from uuid import uuid4
+
+
+def digest(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def canonical_name(value: str) -> str:
+    return re.sub(r"[-_.]+", "-", value).lower()
+
+
+def requirement_names(path: Path) -> set[str]:
+    names: set[str] = set()
+    for raw_line in path.read_text(encoding="utf-8").splitlines():
+        line = raw_line.split("#", 1)[0].strip()
+        match = re.match(r"([A-Za-z0-9_.-]+)", line)
+        if match:
+            names.add(canonical_name(match.group(1)))
+    return names
+
+
+def python_components(root: Path) -> list[dict[str, object]]:
+    direct = requirement_names(root / "requirements.txt")
+    optional = requirement_names(root / "requirements-ocr.txt")
+    components: list[dict[str, object]] = []
+    seen: set[str] = set()
+    for distribution in importlib.metadata.distributions():
+        name = distribution.metadata.get("Name")
+        version = distribution.version
+        if not name or not version:
+            continue
+        key = canonical_name(name)
+        if key in seen:
+            continue
+        seen.add(key)
+        raw_license = (
+            distribution.metadata.get("License-Expression")
+            or distribution.metadata.get("License")
+            or ""
+        ).strip()
+        if not raw_license:
+            raw_license = next(
+                (
+                    value.removeprefix("License :: ")
+                    for value in distribution.metadata.get_all("Classifier", [])
+                    if value.startswith("License :: ")
+                ),
+                "",
+            )
+        scope = (
+            "optional-ocr"
+            if key in optional
+            else "direct-runtime"
+            if key in direct
+            else "installed-transitive-or-development"
+        )
+        components.append(
+            component(
+                ecosystem="pypi",
+                name=name,
+                version=version,
+                scope=scope,
+                raw_license=raw_license,
+                license_source="installed Python distribution metadata",
+            )
+        )
+    return sorted(components, key=lambda item: str(item["name"]).lower())
+
+
+def component(
+    *,
+    ecosystem: str,
+    name: str,
+    version: str,
+    scope: str,
+    raw_license: str,
+    license_source: str,
+    license_file: Path | None = None,
+) -> dict[str, object]:
+    normalized = canonical_name(name) if ecosystem == "pypi" else name.lower()
+    result: dict[str, object] = {
+        "type": "library",
+        "bom-ref": f"pkg:{ecosystem}/{normalized}@{version}",
+        "name": name,
+        "version": version,
+        "purl": f"pkg:{ecosystem}/{normalized}@{version}",
+        "properties": [
+            {"name": "inventory.scope", "value": scope},
+            {"name": "inventory.license_source", "value": license_source},
+        ],
+    }
+    if raw_license:
+        result["licenses"] = [{"license": {"name": raw_license}}]
+    else:
+        result["properties"].append(
+            {"name": "inventory.license_status", "value": "not-present-in-metadata"}
+        )
+    if license_file is not None:
+        result["properties"].extend(
+            [
+                {
+                    "name": "inventory.license_file",
+                    "value": license_file.name,
+                },
+                {
+                    "name": "inventory.license_file_sha256",
+                    "value": digest(license_file),
+                },
+            ]
+        )
+    return result
+
+
+def parse_lockfile(path: Path) -> list[dict[str, str]]:
+    packages: list[dict[str, str]] = []
+    current: dict[str, str] | None = None
+    package_header = re.compile(r"^  ([A-Za-z0-9_]+):$")
+    for line in path.read_text(encoding="utf-8").splitlines():
+        match = package_header.match(line)
+        if match:
+            if current is not None and current.get("version"):
+                packages.append(current)
+            current = {"name": match.group(1)}
+            continue
+        if current is None:
+            continue
+        for field in ("dependency", "source", "version"):
+            field_match = re.match(rf"^    {field}:\s*(.*?)\s*$", line)
+            if field_match:
+                current[field] = field_match.group(1).strip('"\'')
+    if current is not None and current.get("version"):
+        packages.append(current)
+    return packages
+
+
+def package_roots(config_path: Path) -> dict[str, Path]:
+    config = json.loads(config_path.read_text(encoding="utf-8"))
+    roots: dict[str, Path] = {}
+    for package in config.get("packages", []):
+        uri = urlparse(package.get("rootUri", ""))
+        if uri.scheme == "file":
+            raw_path = unquote(uri.path)
+            if re.match(r"^/[A-Za-z]:/", raw_path):
+                raw_path = raw_path[1:]
+            roots[package["name"]] = Path(raw_path)
+    return roots
+
+
+def flutter_license_file(package_root: Path) -> Path | None:
+    try:
+        files = {item.name.lower(): item for item in package_root.iterdir() if item.is_file()}
+    except OSError:
+        return None
+    for name in ("license", "license.txt", "license.md", "copying", "copying.txt"):
+        if name in files:
+            return files[name]
+    return None
+
+
+def flutter_components(root: Path) -> list[dict[str, object]]:
+    app_root = root / "flutter_app"
+    packages = parse_lockfile(app_root / "pubspec.lock")
+    roots = package_roots(app_root / ".dart_tool" / "package_config.json")
+    components: list[dict[str, object]] = []
+    for package in packages:
+        name = package["name"]
+        version = package["version"]
+        scope = package.get("dependency", "transitive")
+        package_root = roots.get(name)
+        license_file = flutter_license_file(package_root) if package_root else None
+        declared = ""
+        license_source = "resolved pubspec.lock; license declaration unavailable"
+        if package.get("source") == "sdk" and package_root is not None:
+            sdk_license = package_root.parent.parent / "LICENSE"
+            if sdk_license.is_file():
+                license_file = sdk_license
+                declared = "Flutter SDK root license notice (BSD-style)"
+                license_source = "Flutter SDK root LICENSE inherited by SDK package"
+        if package_root:
+            pubspec = package_root / "pubspec.yaml"
+            if pubspec.exists() and not declared:
+                match = re.search(
+                    r"(?m)^license:\s*(.*?)\s*$",
+                    pubspec.read_text(encoding="utf-8", errors="replace"),
+                )
+                if match:
+                    declared = match.group(1).strip('"\'')
+                    license_source = "package pubspec.yaml declaration"
+        if not declared and license_file:
+            declared = f"See included {license_file.name} file; not automatically classified"
+            license_source = "package license file (hash recorded; human classification required)"
+        components.append(
+            component(
+                ecosystem="pub",
+                name=name,
+                version=version,
+                scope=scope,
+                raw_license=declared,
+                license_source=license_source,
+                license_file=license_file,
+            )
+        )
+    return sorted(components, key=lambda item: str(item["name"]).lower())
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--project-root",
+        type=Path,
+        default=Path(__file__).resolve().parents[1],
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=None,
+        help="Output JSON path (default: sbom/component-inventory.json)",
+    )
+    parser.add_argument(
+        "--flutter-notices",
+        type=Path,
+        default=None,
+        help="Optional Flutter build's bundled NOTICES.Z archive for evidence/hash",
+    )
+    args = parser.parse_args()
+    root = args.project_root.resolve()
+    output = args.output or root / "sbom" / "component-inventory.json"
+    components = python_components(root) + flutter_components(root)
+    metadata: dict[str, object] = {
+        "timestamp": datetime.now(UTC).isoformat(),
+        "tools": [{"name": "generate_component_inventory.py"}],
+        "component": {
+            "type": "application",
+            "name": "SovereignAI-Starter",
+            "version": "1.0.0",
+        },
+        "properties": [
+            {
+                "name": "inventory.note",
+                "value": "Preliminary local environment inventory; not a legal approval or a complete commercial SBOM.",
+            },
+            {
+                "name": "inventory.python_requirements_sha256",
+                "value": digest(root / "requirements.txt"),
+            },
+            {
+                "name": "inventory.ocr_requirements_sha256",
+                "value": digest(root / "requirements-ocr.txt"),
+            },
+            {
+                "name": "inventory.flutter_lock_sha256",
+                "value": digest(root / "flutter_app" / "pubspec.lock"),
+            },
+        ],
+    }
+    if args.flutter_notices:
+        notice_bytes = args.flutter_notices.read_bytes()
+        metadata["properties"].extend(
+            [
+                {
+                    "name": "inventory.flutter_notices_sha256",
+                    "value": hashlib.sha256(notice_bytes).hexdigest(),
+                },
+                {
+                    "name": "inventory.flutter_notices_uncompressed_bytes",
+                    "value": str(len(gzip.decompress(notice_bytes))),
+                },
+            ]
+        )
+    document = {
+        "bomFormat": "CycloneDX",
+        "specVersion": "1.5",
+        "serialNumber": f"urn:uuid:{uuid4()}",
+        "version": 1,
+        "metadata": metadata,
+        "components": components,
+    }
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(
+        json.dumps(document, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+    python_count = sum(item["purl"].startswith("pkg:pypi/") for item in components)
+    pub_count = sum(item["purl"].startswith("pkg:pub/") for item in components)
+    missing = sum("licenses" not in item for item in components)
+    unclassified = sum(
+        any(
+            prop["name"] == "inventory.license_source"
+            and "human classification required" in prop["value"]
+            for prop in item["properties"]
+        )
+        for item in components
+    )
+    hashed_licenses = sum(
+        any(prop["name"] == "inventory.license_file_sha256" for prop in item["properties"])
+        for item in components
+    )
+    print(
+        json.dumps(
+            {
+                "output": str(output),
+                "python_components": python_count,
+                "flutter_components": pub_count,
+                "components_missing_license_evidence": missing,
+                "license_files_hashed": hashed_licenses,
+                "license_files_requiring_manual_classification": unclassified,
+                "complete_commercial_audit": False,
+            },
+            ensure_ascii=False,
+        )
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,99 @@
+from __future__ import annotations
+
+import json
+import shutil
+import unittest
+from pathlib import Path
+from uuid import uuid4
+
+from scripts.generate_component_inventory import (
+    canonical_name,
+    component,
+    parse_lockfile,
+    requirement_names,
+)
+
+
+class ComponentInventoryTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.fixture_root = (
+            Path(__file__).resolve().parent
+            / f".component-inventory-test-{uuid4().hex}"
+        )
+        self.fixture_root.mkdir()
+
+    def tearDown(self) -> None:
+        shutil.rmtree(self.fixture_root, ignore_errors=True)
+
+    def test_python_package_names_are_canonicalized_for_purls(self) -> None:
+        self.assertEqual(canonical_name("python_multipart"), "python-multipart")
+        self.assertEqual(canonical_name("Pillow"), "pillow")
+
+    def test_requirements_parser_ignores_comments_and_extras(self) -> None:
+        requirements = self.fixture_root / "requirements.txt"
+        requirements.write_text(
+            "fastapi>=0.1\nuvicorn[standard]>=0.2 # comment\n# skip\n",
+            encoding="utf-8",
+        )
+        self.assertEqual(requirement_names(requirements), {"fastapi", "uvicorn"})
+
+    def test_pub_lock_parser_keeps_scope_source_and_version(self) -> None:
+        lockfile = self.fixture_root / "pubspec.lock"
+        lockfile.write_text(
+            """# Generated by pub
+packages:
+  demo_package:
+    dependency: direct main
+    source: hosted
+    version: \"1.2.3\"
+  flutter:
+    dependency: direct main
+    source: sdk
+    version: \"0.0.0\"
+sdks:
+  dart: \"^3.0.0\"
+""",
+            encoding="utf-8",
+        )
+        self.assertEqual(
+            parse_lockfile(lockfile),
+            [
+                {
+                    "name": "demo_package",
+                    "dependency": "direct main",
+                    "source": "hosted",
+                    "version": "1.2.3",
+                },
+                {
+                    "name": "flutter",
+                    "dependency": "direct main",
+                    "source": "sdk",
+                    "version": "0.0.0",
+                },
+            ],
+        )
+
+    def test_component_records_license_file_hash_as_evidence(self) -> None:
+        license_file = self.fixture_root / "LICENSE"
+        license_file.write_text("sample notice\n", encoding="utf-8")
+        result = component(
+            ecosystem="pub",
+            name="demo_package",
+            version="1.2.3",
+            scope="direct main",
+            raw_license="See included LICENSE",
+            license_source="test fixture",
+            license_file=license_file,
+        )
+        self.assertEqual(result["purl"], "pkg:pub/demo_package@1.2.3")
+        self.assertTrue(
+            any(
+                property_item["name"] == "inventory.license_file_sha256"
+                for property_item in result["properties"]
+            )
+        )
+        self.assertEqual(json.loads(json.dumps(result)), result)
+
+
+if __name__ == "__main__":
+    unittest.main()