kaiyuan/chatgpt-on-wechat
1
0
Fork 0
mirror of https://github.com/zhayujie/chatgpt-on-wechat.git synced 2026-07-20 21:57:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(skill): multi-provider image generation with auto-fallback

Browse Source 
- Add Gemini, Seedream (Volcengine Ark), Qwen (DashScope), MiniMax
  providers to image-generation skill with universal sequential
  fallback: OpenAI → Gemini → Seedream → Qwen → MiniMax → LinkAI
- Each provider filters unsupported size tiers to valid values
  (e.g. Seedream 1K→2K, Qwen 3K→2K, Gemini 3K→2K)
- Pinned model only tries its native provider; auto-routing uses
  each provider's default model
- Support skill-namespaced config (config.skill.image-generation.model
  → SKILL_IMAGE_GENERATION_MODEL env var)
- Add image lightbox (click-to-enlarge) in web console
- Add  docs for built-in skills (skill-creator, knowledge-wiki,
  image-generation) under docs/skills/
This commit is contained in:
zhayujie
2026-04-23 12:39:39 +08:00
parent 81e8bb62ae
commit 68ce2e5232
16 changed files with 2189 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

158
docs/en/skills/image-generation.mdx Normal file
View File

@@ -0,0 +1,158 @@
---
title: image-generation - Image Generation
description: Text-to-image / image-to-image / multi-image fusion with automatic multi-provider routing and fallback
---
A general-purpose image generation and editing skill supporting six providers: OpenAI, Gemini, Seedream (Volcengine Ark), Qwen (DashScope), MiniMax, and LinkAI. No need to choose a model manually — the script automatically selects a configured provider based on a fixed priority order.
## Model Selection
`image-generation` uses a "fixed priority + automatic fallback" strategy — just configure your keys and it works:
1. **Priority order**: `OpenAI → Gemini → Seedream → Qwen → MiniMax → LinkAI`
2. **Unconfigured providers are skipped**: only providers with an API key participate
3. **Automatic fallback on failure**: on errors like 401, model not enabled, or network issues, the next provider is tried
4. **Specified model goes first**: if a specific model name is provided, its provider is promoted to the front
### Supported Models
| Provider | Models / Aliases | Notes |
| --- | --- | --- |
| OpenAI | `gpt-image-2`, `gpt-image-1` | General-purpose, high quality, supports `quality` parameter |
| Gemini Nano Banana | `nano-banana-2`, `nano-banana-pro`, `nano-banana` | Corresponds to `gemini-3.1-flash`, `gemini-3-pro`, `gemini-2.5-flash` image variants |
| Seedream (Volcengine Ark) | `seedream-5.0-lite`, `seedream-4.5` | Native 2K–4K, up to 14 reference images for fusion |
| Qwen (DashScope) | `qwen-image-2.0`, `qwen-image-2.0-pro` | Strong with Chinese text rendering and text-image layouts |
| MiniMax | `image-01` | Fast and simple image generation |
| LinkAI | Any model | Universal proxy, used as fallback |
<Note>
By default, the Agent does not pick a model — it uses automatic routing. If you want a specific model, just say so in the conversation, e.g. "use seedream to draw a cat" or "generate a poster with gpt-image-2". You can also pin a default model via the "Custom Configuration" section below.
</Note>
## Custom Configuration
### API Key Setup
You need **at least one** provider key. Configuring multiple providers enables automatic fallback. There are three ways to set up keys:
#### Option 1: Automatic Reuse of Existing Keys
If you have already configured model keys in the web console or `config.json` (e.g. `openai_api_key`, `gemini_api_key`, etc.), these keys are **automatically synced** to the corresponding environment variables at startup. In other words, if your chat model works, image generation can use the same key with zero extra configuration.
#### Option 2: Configure in config.json
Add the key fields directly to `config.json`:
```json
{
"openai_api_key": "sk-xxx",
"openai_api_base": "https://api.openai.com/v1",
"gemini_api_key": "AIza-xxx",
"ark_api_key": "xxx",
"dashscope_api_key": "sk-xxx",
"minimax_api_key": "xxx",
"linkai_api_key": "xxx"
}
```
A restart is required after changes. Each key also has a corresponding `*_api_base` field for custom endpoints.
#### Option 3: Configure via Conversation
Send an API key in the chat and the Agent will save it to `~/cow/.env` using the `env_config` tool — **no restart needed**. For example:
```
Set OPENAI_API_KEY to sk-xxx
```
Or:
```
Configure ARK_API_KEY as xxx
```
### API Key Reference
| Environment Variable | config.json Field | Provider | Default Base URL |
| --- | --- | --- | --- |
| `OPENAI_API_KEY` | `openai_api_key` | OpenAI | `https://api.openai.com/v1` |
| `GEMINI_API_KEY` | `gemini_api_key` | Gemini | `https://generativelanguage.googleapis.com` |
| `ARK_API_KEY` | `ark_api_key` | Volcengine Ark (Seedream) | `https://ark.cn-beijing.volces.com/api/v3` |
| `DASHSCOPE_API_KEY` | `dashscope_api_key` | Alibaba DashScope (Qwen) | `https://dashscope.aliyuncs.com` |
| `MINIMAX_API_KEY` | `minimax_api_key` | MiniMax | `https://api.minimaxi.com` |
| `LINKAI_API_KEY` | `linkai_api_key` | LinkAI | `https://api.link-ai.tech` |
### Pinning a Default Model
To force all image generation through a specific provider's model, add this to `config.json`:
```json
"skill": {
"image-generation": {
"model": "seedream-5.0-lite"
}
}
```
At startup, this is automatically converted to the environment variable `SKILL_IMAGE_GENERATION_MODEL`, and the script will always use this model's provider for generation.
## Enabling and Disabling
`image-generation` is a built-in skill that **automatically adjusts its status based on API keys**:
- **Key configured**: the skill is active — the Agent will invoke it when asked to draw
- **Key not configured**: the skill still appears in context (marked as "needs configuration") — the Agent will guide the user to set up a key rather than failing silently
To control it manually:
```text
/skill disable image-generation # Disable (won't be invoked even if keys are present)
/skill enable image-generation # Re-enable
```
In the terminal: `cow skill disable image-generation` / `cow skill enable image-generation`.
## Parameters
| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | string | Yes | — | Image description |
| `image_url` | string / list | No | null | Input image(s) for editing — local path or URL. Pass multiple for multi-image fusion |
| `quality` | string | No | auto | `low` / `medium` / `high` — only some providers support this |
| `size` | string | No | auto | `512` / `1K` / `2K` / `3K` / `4K`, or pixel value like `1024x1024` |
| `aspect_ratio` | string | No | null | `1:1` / `3:2` / `2:3` / `16:9` / `9:16` / `21:9`; Gemini also supports `1:4` / `4:1` / `1:8` / `8:1` |
<Warning>
**Higher quality and larger size cost more and take longer.**
- For everyday conversations and quick previews, use the defaults (`auto`) or `quality=low` + `size=1K` — roughly 20 seconds
- For posters or when the user explicitly asks for high resolution, use `quality=high` + `size=2K/4K` — may take 1–5 minutes depending on the model
</Warning>
## Output
On success:
```json
{
"model": "doubao-seedream-5-0-260128",
"images": [
{"url": "/path/to/output.png"}
]
}
```
On failure: `{ "error": "..." }`. After an error, **do not retry directly** — it is almost always a configuration issue (wrong key, incorrect API base, model not enabled). Have the user fix the configuration first.
## Common Use Cases
- **Text-to-image**: generate illustrations, posters, icons, avatars, storyboards, etc. from a description
- **Image-to-image**: change styles, swap elements, add decorations or text on an existing image
- **Multi-image fusion**: combine multiple reference images into one (outfit swaps, character group photos, etc.)
<Note>
- Bash timeout should be set to 600 seconds. Each provider has a 300-second HTTP timeout, but the script may try multiple providers sequentially
- Input images are automatically compressed to ≤ 4 MB with the longest edge ≤ 4096 px
- Gemini / Seedream / Qwen / MiniMax do not support the `quality` parameter — passing it has no effect
- Seedream defaults to 2K; `seedream-5.0-lite` supports up to 3K; `seedream-4.5` supports up to 4K
</Note>

112
docs/en/skills/knowledge-wiki.mdx Normal file
View File

@@ -0,0 +1,112 @@
---
title: knowledge-wiki - Knowledge Base
description: Maintain a local structured knowledge base with automatic archiving, categorisation, and cross-referencing
---
Organises notes, insights, and reference materials from your conversations into a structured local knowledge base, automatically maintaining an index and cross-references between pages.
`knowledge-wiki` maintains a `knowledge/` directory in your workspace — essentially the Agent's "second brain". The skill is marked `always: true`, so it is **always loaded** and requires no external dependencies.
## When It Triggers
- You share an article, document, or URL that you want to keep for future reference
- A conversation produces conclusions worth retaining long-term
- You want to look up something you accumulated earlier
## Directory Structure
```
knowledge/
├── index.md # Global index (must be maintained)
├── log.md # Operation log (append-only)
└── <category>/ # Category subdirectories (grouped by content)
└── <slug>.md # Knowledge page (lowercase-hyphenated filename)
```
## Three Core Operations
### 1. Ingest
When you share some material, the Agent will:
1. Read and understand the original content, extracting key information
2. Decide which category it belongs to — check `index.md` first; create a new category if none fits
3. Generate a knowledge page at `knowledge/<category>/<slug>.md`
4. Update the index `index.md` and the log `log.md`
### 2. Synthesise
When a conversation produces new conclusions or insights:
1. Create a new knowledge page under an appropriate category
2. Add cross-links to and from related existing pages
3. Update the index and log
### 3. Query
When you ask about previously accumulated knowledge:
1. Search `index.md` for potentially relevant pages
2. Open specific pages with the `read` tool
3. Supplement with `memory_search` if needed
4. Include links to knowledge pages in the answer so you can click through to the source
## Page Format
```markdown
# Page Title
> Source: <source URL or brief description>
Body content. Link between pages using relative paths:
[Related Page](../category/related-page.md)
## Key Points
- ...
## Related Pages
- [Page A](../category/page-a.md) — why it's related
```
<Note>
- `> Source:` records where this knowledge came from. Always include it when there is a clear source
- Cross-references are important: when creating or updating a page, remember to add back-links in the related pages too
- **Only link to pages that already exist.** If a concept deserves its own page, create it first, then add the link
</Note>
## Index Format
`knowledge/index.md` uses a flat list grouped by category, one knowledge page per line:
```markdown
# Knowledge Index
## Category A
- [Page Title](category-a/page-slug.md) — one-line summary
## Category B
- [Page Title](category-b/page-slug.md) — one-line summary
```
No tables, no emojis. Category names and organisation can be adjusted freely.
## Log Format
`knowledge/log.md` is append-only — newest entries go at the bottom:
```markdown
## [YYYY-MM-DD] ingest | Page Title
## [YYYY-MM-DD] synthesize | Page Title
```
## Writing Guidelines
- **Filenames**: lowercase with hyphens, e.g. `machine-learning.md`
- **One topic per page** — link related content across pages
- **Update, don't duplicate** — if a page already exists, update it rather than creating a new one
- **Always update the index** `knowledge/index.md` after any change
- **Distill, don't copy** — capture the key points, not the entire source
- **Use full paths when referencing knowledge pages in conversations**, e.g. `[Title](knowledge/<category>/<slug>.md)`. Use relative paths only for inter-page links
- **Include links when answering questions based on knowledge pages** so users can dig deeper

180
docs/en/skills/skill-creator.mdx Normal file
View File

@@ -0,0 +1,180 @@
---
title: skill-creator - Skill Creator
description: Create, install, and update skills — standardises SKILL.md format and directory structure
---
`skill-creator` is a "meta-skill" that helps the Agent create, install, and update other skills, ensuring every skill follows a consistent `SKILL.md` format and directory layout.
## When It Triggers
- The user wants to install a skill from a URL or remote repository
- The user wants to create a brand-new skill from scratch
- An existing skill needs upgrading or restructuring
## What Is a Skill?
A skill is a reusable instruction set plus optional scripts and assets. It injects domain expertise into the Agent so it can handle specific tasks like a specialist.
A skill typically contains:
1. **Specialised workflow** — step-by-step instructions for a category of tasks
2. **Tool usage** — how to call a particular API or process a particular file format
3. **Domain knowledge** — team conventions, business rules, data schemas, etc.
4. **Attached resources** — scripts, reference docs, templates, etc.
<Note>
**Core principle: less is more.** Only write what the Agent wouldn't figure out on its own. For every line you add, ask yourself: is it worth the tokens?
</Note>
## Directory Structure
```
skill-name/
├── SKILL.md # Required: skill definition
│ ├── YAML frontmatter (name / description are mandatory)
│ └── Markdown body (instructions + examples)
└── Optional resources
├── scripts/ # Executable scripts (Python / Bash, etc.)
├── references/ # Large reference docs the Agent reads on demand
└── assets/ # Templates, icons, etc. used directly in output
```
## SKILL.md Specification
Frontmatter fields in the SKILL.md header:
| Field | Description |
| --- | --- |
| `name` | Skill name — lowercase with hyphens, must match the directory name |
| `description` | **The most important field.** Clearly state what the skill does and when to use it. The Agent reads this to decide whether to invoke it. All trigger-related descriptions go here, not in the body |
| `metadata.cowagent.requires.bins` | System CLI tools that must be installed |
| `metadata.cowagent.requires.env` | Required environment variables (all must be present) |
| `metadata.cowagent.requires.anyEnv` | Multiple API keys — at least one must be set |
| `metadata.cowagent.requires.anyBins` | Multiple tools — at least one must be installed |
| `metadata.cowagent.always` | Set to `true` to always load, skipping dependency checks |
| `metadata.cowagent.emoji` | Display emoji (optional) |
| `metadata.cowagent.os` | OS restriction, e.g. `["darwin", "linux"]` |
<Note>
The `category` field does not need to be set manually — the system automatically sets it to `skill`.
</Note>
Two ways to declare API key dependencies:
```yaml
metadata:
cowagent:
requires:
env: ["MYAPI_KEY"] # Must be present
```
```yaml
metadata:
cowagent:
requires:
anyEnv: ["OPENAI_API_KEY", "LINKAI_API_KEY"] # At least one
```
**Skills are auto-enabled/disabled based on dependencies**: they activate when all required environment variables are present and deactivate when any are missing — no need for manual `/skill enable`.
## Resource Directories
| Directory | What goes here | What does NOT go here |
| --- | --- | --- |
| `scripts/` | Code that needs to run repeatedly, or scripts that produce deterministic results | Demo-only code snippets |
| `references/` | Documents **over 500 lines** that genuinely won't fit in SKILL.md (e.g. a full DB schema) | General API docs, tutorials, examples |
| `assets/` | Files that appear in the final output (templates, icons, boilerplate, etc.) | Explanatory documentation |
<Warning>
**In principle, everything goes in `SKILL.md`** — only split into resource directories when it truly won't fit.
Do not add `README.md`, `CHANGELOG.md`, or `INSTALLATION_GUIDE.md` to a skill — put everything in `SKILL.md`. Resource directories should only contain scripts that actually run or assets that are actually used.
</Warning>
## Installing External Skills
After installation, the skill lands in `<workspace>/skills/<name>/`.
| Source | How to install |
| --- | --- |
| URL (single file) | curl / web_fetch |
| URL (zip archive) | Download and extract |
| Local SKILL.md | Read directly |
| Local zip archive | Extract |
Installation steps:
1. Locate the `SKILL.md` (may be at the root or in a subdirectory of the archive)
2. Read the `name` from the frontmatter
3. Copy the **entire skill directory** (including `SKILL.md`, `scripts/`, `assets/`, etc.) to `<workspace>/skills/<name>/`
4. If the archive contains an `INSTALL.md` or similar setup script, run it — but the final result must still reside under `<workspace>/skills/<name>/`
## Creating a Skill from Scratch
Recommended order:
1. **Clarify requirements** — ask the user for a few concrete use cases (don't ask too many at once)
2. **Plan the structure** — does this skill need scripts? Reference docs? Template assets?
3. **Scaffold** — use the init script:
```bash
scripts/init_skill.py <skill-name> --path <workspace>/skills [--resources scripts,references,assets] [--examples]
```
4. **Fill in content** — write SKILL.md, add scripts and resources. Always test scripts after writing them
5. **Validate** (optional):
```bash
scripts/quick_validate.py <workspace>/skills/<skill-name>
```
6. **Iterate** — keep improving based on real-world usage feedback
## Naming Conventions
- Use only lowercase letters, digits, and hyphens. Normalise user-given names, e.g. `Plan Mode` → `plan-mode`
- Maximum 64 characters
- Keep it short, start with a verb, make it self-explanatory
- Use tool names as prefixes when appropriate, e.g. `gh-address-comments`, `linear-address-issue`
- The directory name and the `name` field must match exactly
## Three-Level Loading
Skills are not loaded into context all at once — they use a three-level progressive loading mechanism:
1. **Metadata** (`name` + `description`) — always in context (~100 words). The Agent uses this to decide whether to invoke the skill
2. **SKILL.md body** — loaded only when the skill is activated; keep it under 500 lines
3. **Resource files** — read on demand by the Agent
For skills with multiple variants (e.g. multi-cloud deployment), organise like this:
```
cloud-deploy/
├── SKILL.md # Main workflow and provider selection logic
└── references/
├── aws.md
├── gcp.md
└── azure.md
```
When the user picks AWS, the Agent only reads `aws.md` — no need to load all three providers.
## Common Design Patterns
**Step-by-step**: numbered steps with corresponding scripts.
```markdown
1. Analyse form structure (run analyze_form.py)
2. Generate field mappings (edit fields.json)
3. Auto-fill the form (run fill_form.py)
```
**Branching**: different flows based on user intent.
```markdown
1. Determine operation type:
**Creating new content?** → follow the "Create" workflow
**Editing existing content?** → follow the "Edit" workflow
```
**Template-based**: when output format has strict requirements, include a template in SKILL.md for the Agent to follow.