# SEO Farm Architecture

## 1. Общая схема

```text
React/Vite UI
  |
  v
Local Backend
  |
  +-- Project Storage
  |     |
  |     +-- Postgres DB (source of truth)
  |     +-- Object Storage
  |     |     +-- MinIO in local/dev
  |     |     +-- S3 in hosted mode later
  |     +-- Project Source Adapters
  |     |     +-- local_folder (MVP)
  |     |     +-- later: git_repo
  |     |     +-- later: zip_upload
  |     |     +-- later: site_crawl
  |     +-- project_sources
  |     +-- scan_versions
  |     +-- page/section/media logical records
  |     +-- page_versions / section_versions / media_versions
  |     +-- originalText snapshots
  |     +-- workingText drafts
  |     +-- run history
  |     +-- changesets
  |     +-- backups
  |     +-- exports
  |     +-- reports/raw dumps/previews/frames
  |
  +-- Task Runner
        |
        +-- Model Provider
        |     |
        |     +-- MVP: Codex Exec Provider
        |     |     |
        |     |     +-- codex exec
        |     |
        |     +-- Later: OpenAI API Provider
        |     +-- Later: Other API Provider
        |     +-- Later: Local LLM Provider
        |
        +-- SEO Tools
        |     |
        |     +-- HTML/Text Extractor
        |     +-- Structure Checker
        |     +-- Keyword Checker
        |     +-- Over-SEO Checker
        |     +-- Page Map Checker
        |
        +-- Wordstat Layer
        |     |
        |     +-- MCP-KV Wordstat
        |
        +-- Text Quality Layer
        |     |
        |     +-- ru-text
        |
        +-- Media SEO Layer
              |
              +-- HTML media parser
              +-- optional: image-size
              +-- optional: Sharp
              +-- optional: FFmpeg
              +-- Model Provider for image/video description
```

## 1.1. Статус инструментов

```text
MVP, точно используем:
  - React
  - Vite
  - Node.js backend
  - codex exec
  - ModelProvider abstraction
  - MCP-KV Wordstat
  - ru-text
  - Cheerio
  - local SEO checks
  - Postgres
  - MinIO or other S3-compatible object storage
  - structured JSON model outputs
  - YAML/JSON/Markdown exports for debug/import-export only

Опционально для Media SEO, подключать когда дойдём до медиа:
  - image-size
  - Sharp
  - FFmpeg

Не используем в MVP:
  - автопостинг
  - Яндекс Метрика
  - robots/sitemap editor
  - hosted/cloud mode
  - Codex skills as dependency
```

Future enhancement:

```text
Codex skills can be added later to improve repeatable model workflows:
  - seo-seed-extractor
  - seo-keyword-classifier
  - seo-copy-editor-ru
  - media-seo-alt-writer

Skills are not required for MVP. First stabilize tools and reports.
```

## 1.2. Project Source Adapters

```text
SourceAdapter
  |
  +-- local_folder
  |     +-- MVP source type
  |     +-- uses targetProjectPath
  |
  +-- git_repo
  |     +-- future hosted/local source
  |
  +-- zip_upload
  |     +-- future hosted/local source
  |
  +-- site_crawl
        +-- future audit-only source
```

For MVP we start with `local_folder`, but the application should keep the source type in the database from day one so local and hosted modes use the same upper pipeline.

## 2. Основной поток данных

```text
1. User creates/selects SEO Farm project
   |
   v
2. User sets project source for the tested site/codebase
   |
   +-- MVP: local_folder -> targetProjectPath
   |
   v
3. Backend creates scan_version, scans source, and builds projectIndex
   |
   v
4. Baseline Audit runs on discovered pages/assets/media
   |
   v
5. User selects page or page group for SEO work
   |
   v
6. Backend creates originalText snapshots and workingText drafts
   |
   v
7. Model Provider extracts meanings and seed queries
   |
   v
8. User reviews/edits seed queries
   |
   v
9. Wordstat Layer collects keyword data
   |
   v
10. keywordService cleans/classifies keyword results
   |
   v
11. User reviews keyword decisions
   |
   v
12. Keyword Map Builder proposes section mapping and limits
    |
    v
13. User approves/edits Keyword Map
    |
    v
14. Optimization Plan is generated and approved
    |
    v
15. Model Provider proposes text edits into workingText
    |
    v
16. Media SEO proposes alt/caption/filename edits
    |
    v
17. SEO Tools + ru-text validate text and media edits
    |
    v
18. UI shows report and diff
    |
    v
19. User applies changes to target project
    |
    v
20. User can export archive/reports
```

MVP works with existing code/text only:

```text
existing HTML / existing draft / existing site
  -> analyze
  -> collect keywords
  -> edit
  -> final validate
```

Creating text from scratch is a future mode, not part of MVP.

## 3. UI Modules

```text
app/
  Projects
    -> create/select local project
    -> set target project path
    -> scan project

  Project Dashboard
    -> project status
    -> discovered pages
    -> baseline audit status
    -> recent runs
    -> working drafts
    -> changesets
    -> exports

  Baseline Audit
    -> project/page structure
    -> title/description/h1/h2
    -> discovered text blocks
    -> media issues
    -> technical SEO issues

  Pages / Source
    -> discovered pages
    -> selected page
    -> extracted sections
    -> extracted headings
    -> original text
    -> working text

  Seeds
    -> seed queries from source text
    -> manual seed editing
    -> select seeds for Wordstat

  Wordstat
    -> run MCP-KV Wordstat
    -> show frequency/dynamics/regions
    -> raw keyword results
    -> high/mid/low groups

  Keyword Cleaning
    -> normalize/deduplicate Wordstat queries
    -> mark obvious trash
    -> classify ambiguous queries through Codex
    -> approve/edit statuses: use/support/article/risky/trash

  Keyword Map
    -> suggested keyword-to-section map
    -> approve/edit suggestions
    -> primary/secondary keywords
    -> soft forms
    -> exact-match limits

  Optimization Plan
    -> sections to edit
    -> sections to leave unchanged
    -> keywords to add/reduce
    -> media tasks
    -> length/tone risks

  Rewrite
    -> select section
    -> edit working text
    -> generate variants through Model Provider
    -> run SEO checks
    -> run ru-text
    -> show original vs working diff
    -> apply only after user approval

  Media SEO
    -> list images/videos/posters
    -> current alt/title/aria/captions
    -> preview
    -> suggested alt/descriptions
    -> media SEO warnings

  Final Validation
    -> title/description/h1/h2
    -> competing meanings
    -> keyword usage
    -> over-SEO warnings
    -> text length/design limits
    -> ru-text status
    -> media SEO status

  Apply / Export
    -> affected files
    -> changeset
    -> manual approval
    -> apply to target project
    -> zip target project
    -> export reports

  History / Reports
    -> scans
    -> audits
    -> Wordstat runs
    -> validation reports
    -> media SEO reports
    -> keyword decisions
    -> rewrites
    -> changesets
    -> exports
```

## 4. Backend Modules

```text
server/
  routes/
    projects
    scan
    pages
    sources
    seeds
    wordstat
    keywords
    keyword-cleaning
    keyword-map
    optimization-plan
    audit
    rewrite
    media
    validate
    apply
    export
    history
    reports

  providers/model/
    ModelProvider.ts
    CodexExecProvider.ts
    OpenAiApiProvider.ts      # future
    CustomHttpProvider.ts     # future

  providers/storage/
    StorageProvider.ts
    MinioStorageProvider.ts

  services/
    db
    projectSourceService
    storageProvider
    taskRunner
    projectStorage
    projectScanService
    reconcileService
    pageWorkspaceService
    wordstatService
    keywordService
    keywordMapService
    optimizationPlanService
    seoAuditService
    ruTextService
    mediaAuditService
    validationService
    changesetService
    applyService
    exportService
    reportService
```

## 5. Tool Responsibilities

| Layer | Tool | Responsibility |
|---|---|---|
| UI | React | Interface, state, workflows |
| UI | Vite | Dev server and build |
| Backend | Node.js | Local server and task orchestration |
| Backend | Express/Fastify | HTTP API for UI |
| Storage | Postgres | Source of truth for projects, sources, versions, approvals, runs, validations, changesets |
| Storage | MinIO/S3 | Raw dumps, previews, reports, exports, backups, scan snapshots |
| Model | `codex exec` | MVP model execution |
| Model | ModelProvider | Switch between Codex/API/local models |
| Wordstat | MCP-KV Wordstat | Raw keyword data from Yandex Wordstat |
| Text parsing | Cheerio | Parse HTML and extract title, meta, headings, text blocks, media tags |
| SEO checks | Local scripts | Title, description, h1/h2, keywords, spam, length |
| Text quality | ru-text | Editorial SEO correction, tone, readability, typographic checks |
| Media parsing | Cheerio | Find img/picture/video/source/poster |
| Images | image-size, optional | Read image dimensions for media report |
| Images | Sharp, optional | Generate previews/thumbnails if browser preview is not enough |
| Video | FFmpeg, optional | Extract representative frames from videos |
| Files | YAML/JSON/Markdown | Export, debug, import/export artifacts only; not the main storage layer |

## 6. Model Provider Abstraction

```text
ModelProvider
  |
  +-- CodexExecProvider
  |     +-- calls: codex exec
  |     +-- use case: local MVP
  |
  +-- OpenAiApiProvider
  |     +-- calls: OpenAI API
  |     +-- use case: remote app / hosted version
  |
  +-- CustomHttpProvider
  |     +-- calls: any compatible API
  |     +-- use case: future integrations
  |
  +-- LocalLlmProvider
        +-- calls: local model runtime
        +-- use case: offline/private mode
```

## 7. Wordstat Flow

```text
Seed queries
  |
  v
wordstatService
  |
  v
MCP-KV Wordstat
  |
  +-- top requests
  +-- related requests
  +-- dynamics
  +-- regions
  |
  v
raw keyword dump
  |
  v
keywordService
  |
  +-- clean by local rules
  +-- classify
  +-- frequency type: high/mid/low
  +-- status: use/support/article/risky/trash
```

## 7.1. Keyword Cleaning Logic

Чистка мусора — это не stopword/удаление частых слов. Это смысловая фильтрация Wordstat-ключей на релевантность проекту. Делает её `keywordService`: локальные правила проекта плюс Codex-классификация спорных запросов.

```text
Raw Wordstat keywords
  |
  v
keywordService
  |
  +-- normalize text
  +-- remove duplicates
  +-- apply negative intents
  +-- apply project blacklist
  +-- group close phrases
  +-- mark obvious trash
  +-- ask Model Provider to classify ambiguous phrases
  |
  v
User review
```

Базовые причины выкинуть запрос:

```text
- вакансии
- обучение / курсы
- скачать
- бесплатно
- реферат / диплом
- бытовой смысл вместо B2B
- чужая отрасль
- слишком широкий запрос без связи с продуктом
- запрос конфликтует с primary_intent страницы
```

Результат хранится как статус:

```text
use / support / article / risky / trash
```

## 8. Text SEO Flow

```text
Source text / HTML
  |
  v
extractText
  |
  +-- sections
  +-- headings
  +-- visible text
  |
  v
Model Provider
  |
  +-- meanings
  +-- seed queries
  |
  v
SEO audit
  |
  +-- title/description/h1/h2
  +-- competing meanings
  +-- keyword usage
  +-- over-SEO
  +-- length/design limits
  |
  v
Rewrite variants
  |
  v
ru-text + SEO checks
  |
  v
User approval
```

## 8.1. Over-SEO Checker

Проверка перетошноты/переблева ключами описана отдельно в `CHECK_OVERSEO_SPEC.md`.

Text.ru используется только как референс по смыслу показателей. Их API не подключаем.

`seo-analyzer` и похожие HTML-аудиторы не заменяют этот слой: они полезны для title/h1/alt/структуры, но не решают именно перетошноту текста.

```text
checkOverseo
  |
  +-- exact keyword count
  +-- soft_forms count from keywords.yml
  +-- keyword density by section
  +-- classic nausea: sqrt(max word/phrase frequency)
  +-- academic nausea: top word/phrase share in text
  +-- repeated exact phrases
  +-- repeated bigrams/trigrams
  +-- proximity: same keyword too close
  +-- title/description/h1 overuse
  +-- alt keyword stuffing
  +-- section-level warnings
```

Источник правил:

```text
keywords.yml
  - phrase
  - soft_forms
  - section
  - exact_limit
  - max_section_count
  - status
  - notes

page-map.yml
  - primary_intent
  - section meanings
  - forbidden_intents
  - max_lines_hint
```

В MVP не нужен сложный морфологический движок. Для русского текста сначала используем:

```text
- точные ключи;
- вручную/моделью заданные soft_forms;
- нормализацию регистра и пробелов;
- подсчёт повторов по секциям.
```

Если этого станет мало, позже можно добавить отдельный морфологический модуль, но он не входит в MVP.

## 9. Media SEO Flow

```text
Source HTML/CSS
  |
  v
extractMedia
  |
  +-- img
  +-- picture/source/srcset
  +-- video/source/poster
  +-- CSS backgrounds
  |
  v
mediaAuditService
  |
  +-- current alt/title/aria
  +-- filename
  +-- captions
  +-- section context
  +-- dimensions
  +-- file size
  |
  +-- optional image-size
  +-- optional Sharp previews
  +-- optional FFmpeg video frames
  |
  v
Model Provider
  |
  +-- describe image/video frame
  +-- suggest alt/caption/description
  |
  v
SEO checks
  |
  +-- missing alt
  +-- weak alt
  +-- duplicate alt
  +-- decorative media
  +-- weak/non-descriptive filename
  +-- keyword stuffing in alt
  +-- missing video context
  |
  v
User approval
```

## 10. Storage Layout

```text
Postgres
  - projects
  - project_sources
  - scan_versions
  - pages
  - page_versions
  - sections
  - section_versions
  - media_assets
  - media_versions
  - runs
  - seeds
  - wordstat_jobs
  - wordstat_results
  - keyword_decisions
  - keyword_map_items
  - optimization_plans
  - drafts
  - draft_items
  - validations
  - validation_issues
  - changesets
  - changeset_items
  - exports

Object Storage
  - projects/project-id/scans/scan-version-id/project-index.json
  - projects/project-id/raw/wordstat/*.json
  - projects/project-id/snapshots/original/*.json
  - projects/project-id/snapshots/working/*.json
  - projects/project-id/reports/*.md
  - projects/project-id/previews/*
  - projects/project-id/video-frames/*
  - projects/project-id/backups/changeset-id/*
  - projects/project-id/exports/*
```

`project.json` can exist as export/debug metadata, but the source of truth lives in Postgres:

```json
{
  "id": "nodedc",
  "name": "Node.DC",
  "source": {
    "type": "local_folder",
    "targetProjectPath": "C:/Users/const/Desktop/landing-correction"
  },
  "scan": {
    "include": ["**/*.html", "**/*.md", "css/**/*.css", "images/**/*", "video/**/*"],
    "exclude": ["node_modules/**", "dist/**", ".git/**"]
  }
}
```

Postgres stores project state and history:

```text
projects
project_sources
scan_versions
pages
page_versions
sections
section_versions
media_assets
media_versions
runs
seeds
wordstat_jobs
wordstat_results
keyword_decisions
keyword_map_items
optimization_plans
drafts
draft_items
validations
validation_issues
changesets
changeset_items
exports
```

Rules:

```text
Postgres = source of truth
Object Storage = heavy/snapshot/report artifacts
YAML/JSON/Markdown = export/debug/import-export only
```

## 10.1. Repeated Scan / Reconcile Model

Every scan creates a new `scan_version`.

Important rule:

```text
DOM selector != entity identity
```

Stable identities:

```text
page_id
section_id
media_id
```

Versioned snapshots:

```text
page_version
section_version
media_version
```

When a new scan is created, backend runs reconcile:

```text
matched
changed
new
orphaned
conflicted
```

Matching should use more than one signal:

```text
- source path/url
- heading/context
- normalized text hash
- selector fingerprint
- neighboring structure
```

Approvals, keyword decisions, drafts, and media decisions must link to stable logical IDs, not only to the current selector.

Text editing model:

```text
target project file
  |
  v
extract text
  |
  +-- originalText snapshot
  |
  +-- workingText draft
        |
        +-- user edits
        +-- Codex edits
        +-- SEO checks
        +-- ru-text checks
        |
        v
      diff
        |
        v
      user approval
        |
        v
      changeset
        |
        +-- dry-run
        +-- backup affected files to object storage
        +-- apply through backend
        +-- verify references/renames
        +-- rollback on failure
        |
        v
      apply to target project
```

MVP stores one `workingText` draft per source/section. Full draft versioning and rollback history are future enhancements.

## 11. External Links

Related local docs:

- `SEO_FARM_BRIEF.md`
- `USER_FLOW.md`
- `CHECK_OVERSEO_SPEC.md`
- `IMPLEMENTATION_GUARDRAILS.md`
- `UI_UX_SPEC.md`
- `TASK_MANAGER.md`

- Codex exec: https://developers.openai.com/codex/noninteractive
- Codex with ChatGPT plan: https://help.openai.com/en/articles/11369540-using-codex-with-your-chatgpt-plan
- React: https://react.dev/
- Vite: https://vite.dev/guide/
- MCP-KV Wordstat: https://mcp-kv.ru/docs/wordstat-mcp-setup
- MCP-KV SEO tools, optional/reference: https://mcp-kv.ru/mcp-server-seo
- ru-text: https://github.com/talkstream/ru-text
- Cheerio: https://cheerio.js.org/docs/api/
- image-size: https://github.com/image-size/image-size
- Sharp: https://www.npmjs.com/package/sharp
- FFmpeg: https://ffmpeg.org/ffmpeg-doc.html
- Text.ru SEO analysis reference: https://text.ru/seo
- Text.ru API/check description, reference for analysis model only: https://text.ru/api-check
