48c839d927
This is the snapshot the production landing site (nibiru-framework.com) is deployed from. Brings together the recent splash + docs migration to the v4 "Cosmos" design system, the new in-framework AI module, and the framework groundwork that backs the framework-reference extraction. What lands: - docs/: Astro + Starlight site with the v4 dark cosmic palette, GalaxyHero canvas constellation, Mission Control chat (wired to /api/oracle → api.neuronetz.ai via providers.mjs Ollama), 5-panel MMVC stage (Model · AI · Module · Controller · View), translated EN/DE/JA/ES/FR content, PWA + sitemap + llms.txt + Umami analytics. - docs/design-system/: canonical mockup bundle (source/index-v2.html for splash, source/docs-system.html + preview/ for docs, SPEC.md, tokens). - docs/scripts/extraction/framework-reference-v2.md: deep framework reference (~1.6k lines, file:line citations, every public factory and idiom — basis for the LoRA training corpus. - application/module/ai/: AI module with chat / embed / RAG / agent plugins, plus pdoQuery / httpGet / fileRead tools and Modelfile + smoke-test in training/. - application/module/users/: user / ACL / form-factory traits used as the reference plugin pattern for the framework docs. - application/settings/config/database/: schema + seed migrations including the AI module tables (200–203). - Form factory + autogenerator changes the framework-reference-v2 covers. Production secrets stay out: docs/.env, settings.production.ini and ai.production.ini are all gitignored (.example files are in tree). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
119 lines
2.5 KiB
Plaintext
119 lines
2.5 KiB
Plaintext
# =============================================================================
|
|
# robots.txt for nibiru-framework.com
|
|
#
|
|
# Policy: open. We want every search engine, every AI training crawler,
|
|
# every retrieval/RAG agent to be able to read these docs. The whole point
|
|
# of publishing this site is so that humans AND models can learn Nibiru.
|
|
#
|
|
# Wildcard rule below allows everything; AI-specific bots are listed
|
|
# explicitly so their operators can verify they are welcome here.
|
|
# =============================================================================
|
|
|
|
# ── Search engines ──────────────────────────────────────────────────────────
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
User-agent: Bingbot
|
|
Allow: /
|
|
|
|
User-agent: DuckDuckBot
|
|
Allow: /
|
|
|
|
User-agent: Yandexbot
|
|
Allow: /
|
|
|
|
User-agent: Baiduspider
|
|
Allow: /
|
|
|
|
# ── AI training / search crawlers — explicitly welcomed ─────────────────────
|
|
# OpenAI
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
|
|
User-agent: OAI-SearchBot
|
|
Allow: /
|
|
|
|
# Anthropic
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
|
|
User-agent: Claude-Web
|
|
Allow: /
|
|
|
|
User-agent: anthropic-ai
|
|
Allow: /
|
|
|
|
# Google AI training
|
|
User-agent: Google-Extended
|
|
Allow: /
|
|
|
|
# Apple AI training
|
|
User-agent: Applebot-Extended
|
|
Allow: /
|
|
|
|
User-agent: Applebot
|
|
Allow: /
|
|
|
|
# Meta
|
|
User-agent: meta-externalagent
|
|
Allow: /
|
|
|
|
User-agent: FacebookBot
|
|
Allow: /
|
|
|
|
# Perplexity
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
User-agent: Perplexity-User
|
|
Allow: /
|
|
|
|
# Other AI / LLM crawlers
|
|
User-agent: YouBot
|
|
Allow: /
|
|
|
|
User-agent: Bytespider
|
|
Allow: /
|
|
|
|
User-agent: Amazonbot
|
|
Allow: /
|
|
|
|
User-agent: Diffbot
|
|
Allow: /
|
|
|
|
User-agent: cohere-ai
|
|
Allow: /
|
|
|
|
User-agent: cohere-training-data-crawler
|
|
Allow: /
|
|
|
|
User-agent: Mistral-AI-User
|
|
Allow: /
|
|
|
|
User-agent: omgili
|
|
Allow: /
|
|
|
|
User-agent: omgilibot
|
|
Allow: /
|
|
|
|
# Common Crawl — the dataset most LLMs train on
|
|
User-agent: CCBot
|
|
Allow: /
|
|
|
|
# Internet Archive
|
|
User-agent: ia_archiver
|
|
Allow: /
|
|
|
|
# ── Default policy: allow everything ───────────────────────────────────────
|
|
User-agent: *
|
|
Allow: /
|
|
|
|
# Don't index or crawl the SSR API endpoint — it's not content.
|
|
Disallow: /api/
|
|
|
|
# ── Sitemaps ───────────────────────────────────────────────────────────────
|
|
Sitemap: https://nibiru-framework.com/sitemap-index.xml
|