fix: client resource usage bug

支持ZhipuAI GLM系列模型和画图代码

.github/workflows/deploy-image-arm.yml

@@ -19,6 +19,7 @@ env:
 
 jobs:
   build-and-push-image:
+    if: github.repository == 'zhayujie/chatgpt-on-wechat'
     runs-on: ubuntu-latest
     permissions:
       contents: read

.github/workflows/deploy-image.yml

@@ -19,6 +19,7 @@ env:
 
 jobs:
   build-and-push-image:
+    if: github.repository == 'zhayujie/chatgpt-on-wechat'
     runs-on: ubuntu-latest
     permissions:
       contents: read

.gitignore

@@ -29,4 +29,5 @@ plugins/banwords/lib/__pycache__
 !plugins/hello
 !plugins/role
 !plugins/keyword
-!plugins/linkai
+!plugins/linkai
+client_config.json

README.md

@@ -1,37 +1,54 @@
 # 简介
 
-> ChatGPT近期以强大的对话和信息整合能力风靡全网，可以写代码、改论文、讲故事，几乎无所不能，这让人不禁有个大胆的想法，能否用他的对话模型把我们的微信打造成一个智能机器人，可以在与好友对话中给出意想不到的回应，而且再也不用担心女朋友影响我们 ~~打游戏~~ 工作了。
+> 本项目是基于大模型的智能对话机器人，支持微信、企业微信、公众号、飞书、钉钉接入，可选择GPT3.5/GPT4.0/Claude/文心一言/讯飞星火/通义千问/Gemini/LinkAI/ZhipuAI，能处理文本、语音和图片，通过插件访问操作系统和互联网等外部资源，支持基于自有知识库定制企业AI应用。
 
 最新版本支持的功能如下：
 
-- [x] **多端部署：** 有多种部署方式可选择且功能完备，目前已支持个人微信，微信公众号和企业微信应用等部署方式
-- [x] **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4, claude, 文心一言, 讯飞星火
-- [x] **语音识别：** 可识别语音消息，通过文字或语音回复，支持 azure, baidu, google, openai等多种语音模型
-- [x] **图片生成：** 支持图片生成 和 图生图（如照片修复），可选择 Dell-E, stable diffusion, replicate, midjourney模型
-- [x] **丰富插件：** 支持个性化插件扩展，已实现多角色切换、文字冒险、敏感词过滤、聊天记录总结等插件
-- [X] **Tool工具：** 与操作系统和互联网交互，支持最新信息搜索、数学计算、天气和资讯查询、网页总结，基于 [chatgpt-tool-hub](https://github.com/goldfishh/chatgpt-tool-hub) 实现
-- [x] **知识库：** 通过上传知识库文件自定义专属机器人，可作为数字分身、领域知识库、智能客服使用，基于 [LinkAI](https://chat.link-ai.tech/console) 实现
-
-> 欢迎接入更多应用，参考 [Terminal代码](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/terminal/terminal_channel.py)实现接收和发送消息逻辑即可接入。 同时欢迎增加新的插件，参考 [插件说明文档](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins)。
+- [x] **多端部署：** 有多种部署方式可选择且功能完备，目前已支持个人微信、微信公众号和、企业微信、飞书、钉钉等部署方式
+- [x] **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4, claude, Gemini, 文心一言, 讯飞星火, 通义千问，ChatGLM
+- [x] **语音能力：** 可识别语音消息，通过文字或语音回复，支持 azure, baidu, google, openai(whisper/tts) 等多种语音模型
+- [x] **图像能力：** 支持图片生成、图片识别、图生图（如照片修复），可选择 Dall-E-3, stable diffusion, replicate, midjourney, CogView-3, vision模型
+- [x] **丰富插件：** 支持个性化插件扩展，已实现多角色切换、文字冒险、敏感词过滤、聊天记录总结、文档总结和对话、联网搜索等插件
+- [x] **知识库：** 通过上传知识库文件自定义专属机器人，可作为数字分身、智能客服、私域助手使用，基于 [LinkAI](https://link-ai.tech) 实现
 
 # 演示
 
-https://user-images.githubusercontent.com/26161723/233777277-e3b9928e-b88f-43e2-b0e0-3cbc923bc799.mp4
+https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db-8706-40ce9f3f1b1e
 
 Demo made by [Visionn](https://www.wangpc.cc/)
 
-# 交流群
+# 商业支持
 
-添加小助手微信进群，请备注 "wechat"：
+> 我们还提供企业级的 **AI应用平台**，包含知识库、Agent插件、应用管理等能力，支持多平台聚合的应用接入、客户端管理、对话管理，以及提供
+SaaS服务、私有化部署、稳定托管接入 等多种模式。
+>
+> 目前已在私域运营、智能客服、企业效率助手等场景积累了丰富的 AI 解决方案， 在电商、文教、健康、新消费等各行业沉淀了 AI 落地的最佳实践，致力于打造助力中小企业拥抱 AI 的一站式平台。
+
+企业服务和商用咨询可联系产品顾问：
+
+<img width="240" src="https://img-1317903499.cos.ap-guangzhou.myqcloud.com/docs/product-manager-qrcode.jpg">
+
+# 开源社区
+
+添加小助手微信加入开源项目交流群：
 
 <img width="240" src="./docs/images/contact.jpg">
 
 # 更新日志
->**2023.09.01：** 增加 [企微个人号](#1385) 通道，[claude](1388)，讯飞星火模型
+
+>**2023.11.11：** [1.5.3版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.3) 和 [1.5.4版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.4)，新增Google Gemini、通义千问模型
+
+>**2023.11.10：** [1.5.2版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.2)，新增飞书通道、图像识别对话、黑名单配置
+
+>**2023.11.10：** [1.5.0版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.0)，新增 `gpt-4-turbo`, `dall-e-3`, `tts` 模型接入，完善图像理解&生成、语音识别&生成的多模态能力
+
+>**2023.10.16：** 支持通过意图识别使用LinkAI联网搜索、数学计算、网页访问等插件，参考[插件文档](https://docs.link-ai.tech/platform/plugins)
+
+>**2023.09.26：** 插件增加 文件/文章链接 一键总结和对话的功能，使用参考：[插件说明](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins/linkai#3%E6%96%87%E6%A1%A3%E6%80%BB%E7%BB%93%E5%AF%B9%E8%AF%9D%E5%8A%9F%E8%83%BD)
 
 >**2023.08.08：** 接入百度文心一言模型，通过 [插件](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins/linkai) 支持 Midjourney 绘图
 
->**2023.06.12：** 接入 [LinkAI](https://chat.link-ai.tech/console) 平台，可在线创建个人知识库，并接入微信、公众号及企业微信中，打造专属客服机器人。使用参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
+>**2023.06.12：** 接入 [LinkAI](https://link-ai.tech/console) 平台，可在线创建领域知识库，并接入微信、公众号及企业微信中，打造专属客服机器人。使用参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
 
 >**2023.04.26：** 支持企业微信应用号部署，兼容插件，并支持语音图片交互，私人助理理想选择，[使用文档](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/wechatcom/README.md)。(contributed by [@lanvent](https://github.com/lanvent) in [#944](https://github.com/zhayujie/chatgpt-on-wechat/pull/944))
 
@@ -43,19 +60,21 @@ Demo made by [Visionn](https://www.wangpc.cc/)
 
 >**2023.03.09：** 基于 `whisper API`(后续已接入更多的语音`API`服务) 实现对微信语音消息的解析和回复，添加配置项 `"speech_recognition":true` 即可启用，使用参考 [#415](https://github.com/zhayujie/chatgpt-on-wechat/issues/415)。(contributed by [wanggang1987](https://github.com/wanggang1987) in [#385](https://github.com/zhayujie/chatgpt-on-wechat/pull/385))
 
->**2023.03.02：** 接入[ChatGPT API](https://platform.openai.com/docs/guides/chat) (gpt-3.5-turbo)，默认使用该模型进行对话，需升级openai依赖 (`pip3 install --upgrade openai`)。网络问题参考 [#351](https://github.com/zhayujie/chatgpt-on-wechat/issues/351)
-
 >**2023.02.09：** 扫码登录存在账号限制风险，请谨慎使用，参考[#58](https://github.com/AutumnWhj/ChatGPT-wechat-bot/issues/158)
 
 # 快速开始
 
+快速开始文档：[项目搭建文档](https://docs.link-ai.tech/cow/quick-start)
+
 ## 准备
 
-### 1. OpenAI账号注册
+### 1. 账号注册
 
-前往 [OpenAI注册页面](https://beta.openai.com/signup) 创建账号，参考这篇 [教程](https://www.pythonthree.com/register-openai-chatgpt/) 可以通过虚拟手机号来接收验证码。创建完账号则前往 [API管理页面](https://beta.openai.com/account/api-keys) 创建一个 API Key 并保存下来，后面需要在项目中配置这个key。
+项目默认使用OpenAI接口，需前往 [OpenAI注册页面](https://beta.openai.com/signup) 创建账号，创建完账号则前往 [API管理页面](https://beta.openai.com/account/api-keys) 创建一个 API Key 并保存下来，后面需要在项目中配置这个key。接口需要海外网络访问及绑定信用卡支付。
 
-> 项目中默认使用的对话模型是 gpt3.5 turbo，计费方式是约每 500 汉字 (包含请求和回复) 消耗 $0.002，图片生成是每张消耗 $0.016。
+> 默认对话模型是 openai 的 gpt-3.5-turbo，计费方式是约每 1000tokens (约750个英文单词 或 500汉字，包含请求和回复) 消耗 $0.002，图片生成是Dell E模型，每张消耗 $0.016。
+
+项目同时也支持使用 LinkAI 接口，无需代理，可使用 文心、讯飞、GPT-3、GPT-4 等模型，支持 定制化知识库、联网搜索、MJ绘图、文档总结和对话等能力。修改配置即可一键切换，参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
 
 ### 2.运行环境
 
@@ -71,6 +90,8 @@ git clone https://github.com/zhayujie/chatgpt-on-wechat
 cd chatgpt-on-wechat/
 ```
 
+注: 如遇到网络问题可选择国内镜像 https://gitee.com/zhayujie/chatgpt-on-wechat
+
 **(2) 安装核心依赖 (必选)：**
 > 能够使用`itchat`创建机器人，并具有文字交流功能所需的最小依赖集合。
 ```bash
@@ -82,23 +103,7 @@ pip3 install -r requirements.txt
 ```bash
 pip3 install -r requirements-optional.txt
 ```
-> 如果某项依赖安装失败请注释掉对应的行再继续。
-
-其中`tiktoken`要求`python`版本在3.8以上，它用于精确计算会话使用的tokens数量，强烈建议安装。
-
-
-使用`google`或`baidu`语音识别需安装`ffmpeg`，
-
-默认的`openai`语音识别不需要安装`ffmpeg`。
-
-参考[#415](https://github.com/zhayujie/chatgpt-on-wechat/issues/415)
-
-使用`azure`语音功能需安装依赖，并参考[文档](https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstarts/setup-platform?pivots=programming-language-python&tabs=linux%2Cubuntu%2Cdotnet%2Cjre%2Cmaven%2Cnodejs%2Cmac%2Cpypi)的环境要求。
-:
-
-```bash
-pip3 install azure-cognitiveservices-speech
-```
+> 如果某项依赖安装失败可注释掉对应的行再继续
 
 ## 配置
 
@@ -173,7 +178,7 @@ pip3 install azure-cognitiveservices-speech
 **5.LinkAI配置 (可选)**
 
 + `use_linkai`: 是否使用LinkAI接口，开启后可国内访问，使用知识库和 `Midjourney` 绘画, 参考 [文档](https://link-ai.tech/platform/link-app/wechat)
-+ `linkai_api_key`: LinkAI Api Key，可在 [控制台](https://chat.link-ai.tech/console/interface) 创建
++ `linkai_api_key`: LinkAI Api Key，可在 [控制台](https://link-ai.tech/console/interface) 创建
 + `linkai_app_code`: LinkAI 应用code，选填
 
 **本说明文档可能会未及时更新，当前所有可选的配置项均在该[`config.py`](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/config.py)中列出。**
@@ -185,17 +190,16 @@ pip3 install azure-cognitiveservices-speech
 如果是开发机 **本地运行**，直接在项目根目录下执行：
 
 ```bash
-python3 app.py
+python3 app.py                                    # windows环境下该命令通常为 python app.py
 ```
-终端输出二维码后，使用微信进行扫码，当输出 "Start auto replying" 时表示自动回复程序已经成功运行了（注意：用于登录的微信需要在支付处已完成实名认证）。扫码登录后你的账号就成为机器人了，可以在微信手机端通过配置的关键词触发自动回复 (任意好友发送消息给你，或是自己发消息给好友)，参考[#142](https://github.com/zhayujie/chatgpt-on-wechat/issues/142)。
 
+终端输出二维码后，使用微信进行扫码，当输出 "Start auto replying" 时表示自动回复程序已经成功运行了（注意：用于登录的微信需要在支付处已完成实名认证）。扫码登录后你的账号就成为机器人了，可以在微信手机端通过配置的关键词触发自动回复 (任意好友发送消息给你，或是自己发消息给好友)，参考[#142](https://github.com/zhayujie/chatgpt-on-wechat/issues/142)。
 
 ### 2.服务器部署
 
 使用nohup命令在后台运行程序：
 
 ```bash
-touch nohup.out                                   # 首次运行需要新建日志文件  
 nohup python3 app.py & tail -f nohup.out          # 在后台运行程序并通过日志输出二维码
 ```
 扫码登录后程序即可运行于服务器后台，此时可通过 `ctrl+c` 关闭日志，不会影响后台程序的运行。使用 `ps -ef | grep app.py | grep -v grep` 命令可查看运行于后台的进程，如果想要重新启动程序可以先 `kill` 掉对应的进程。日志关闭后如果想要再次打开只需输入 `tail -f nohup.out`。此外，`scripts` 目录下有一键运行、关闭程序的脚本供使用。
@@ -266,12 +270,12 @@ volumes:
 
 FAQs： <https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs>
 
-或直接在线咨询 [项目小助手](https://chat.link-ai.tech/app/Kv2fXJcH)  (beta版本，语料完善中，回复仅供参考)
+或直接在线咨询 [项目小助手](https://link-ai.tech/app/Kv2fXJcH)  (beta版本，语料完善中，回复仅供参考)
+
+## 开发
+
+欢迎接入更多应用，参考 [Terminal代码](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/terminal/terminal_channel.py) 实现接收和发送消息逻辑即可接入。 同时欢迎增加新的插件，参考 [插件说明文档](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins)。
 
 ## 联系
 
-欢迎提交PR、Issues，以及Star支持一下。程序运行遇到问题可以查看 [常见问题列表](https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs) ，其次前往 [Issues](https://github.com/zhayujie/chatgpt-on-wechat/issues) 中搜索。
-
-如果你想了解更多项目细节，与开发者们交流更多关于AI技术的实践，欢迎加入星球:
-
-<a href="https://public.zsxq.com/groups/88885848842852.html"><img width="360" src="./docs/images/planet.jpg"></a>
+欢迎提交PR、Issues，以及Star支持一下。程序运行遇到问题可以查看 [常见问题列表](https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs) ，其次前往 [Issues](https://github.com/zhayujie/chatgpt-on-wechat/issues) 中搜索。个人开发者可加入开源交流群参与更多讨论，企业用户可联系[产品顾问](https://img-1317903499.cos.ap-guangzhou.myqcloud.com/docs/product-manager-qrcode.jpg)咨询。

app.py

@@ -3,11 +3,13 @@
 import os
 import signal
 import sys
+import time
 
 from channel import channel_factory
-from common.log import logger
-from config import conf, load_config
+from common import const
+from config import load_config
 from plugins import *
+import threading
 
 
 def sigterm_handler_wrap(_signo):
@@ -23,6 +25,21 @@ def sigterm_handler_wrap(_signo):
     signal.signal(_signo, func)
 
 
+def start_channel(channel_name: str):
+    channel = channel_factory.create_channel(channel_name)
+    if channel_name in ["wx", "wxy", "terminal", "wechatmp", "wechatmp_service", "wechatcom_app", "wework",
+                        const.FEISHU, const.DINGTALK]:
+        PluginManager().load_plugins()
+
+    if conf().get("use_linkai"):
+        try:
+            from common import linkai_client
+            threading.Thread(target=linkai_client.start, args=(channel,)).start()
+        except Exception as e:
+            pass
+    channel.startup()
+
+
 def run():
     try:
         # load config
@@ -40,14 +57,11 @@ def run():
 
         if channel_name == "wxy":
             os.environ["WECHATY_LOG"] = "warn"
-            # os.environ['WECHATY_PUPPET_SERVICE_ENDPOINT'] = '127.0.0.1:9001'
 
-        channel = channel_factory.create_channel(channel_name)
-        if channel_name in ["wx", "wxy", "terminal", "wechatmp", "wechatmp_service", "wechatcom_app", "wework"]:
-            PluginManager().load_plugins()
+        start_channel(channel_name)
 
-        # startup channel
-        channel.startup()
+        while True:
+            time.sleep(1)
     except Exception as e:
         logger.error("App startup failed!")
         logger.exception(e)

bot/ali/ali_qwen_bot.py

@@ -0,0 +1,214 @@
+# encoding:utf-8
+
+import json
+import time
+from typing import List, Tuple
+
+import openai
+import openai.error
+import broadscope_bailian
+from broadscope_bailian import ChatQaMessage
+
+from bot.bot import Bot
+from bot.ali.ali_qwen_session import AliQwenSession
+from bot.session_manager import SessionManager
+from bridge.context import ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from common import const
+from config import conf, load_config
+
+class AliQwenBot(Bot):
+    def __init__(self):
+        super().__init__()
+        self.api_key_expired_time = self.set_api_key()
+        self.sessions = SessionManager(AliQwenSession, model=conf().get("model", const.QWEN))
+
+    def api_key_client(self):
+        return broadscope_bailian.AccessTokenClient(access_key_id=self.access_key_id(), access_key_secret=self.access_key_secret())
+
+    def access_key_id(self):
+        return conf().get("qwen_access_key_id")
+
+    def access_key_secret(self):
+        return conf().get("qwen_access_key_secret")
+
+    def agent_key(self):
+        return conf().get("qwen_agent_key")
+
+    def app_id(self):
+        return conf().get("qwen_app_id")
+
+    def node_id(self):
+        return conf().get("qwen_node_id", "")
+
+    def temperature(self):
+        return conf().get("temperature", 0.2 )
+
+    def top_p(self):
+        return conf().get("top_p", 1)
+
+    def reply(self, query, context=None):
+        # acquire reply content
+        if context.type == ContextType.TEXT:
+            logger.info("[QWEN] query={}".format(query))
+
+            session_id = context["session_id"]
+            reply = None
+            clear_memory_commands = conf().get("clear_memory_commands", ["#清除记忆"])
+            if query in clear_memory_commands:
+                self.sessions.clear_session(session_id)
+                reply = Reply(ReplyType.INFO, "记忆已清除")
+            elif query == "#清除所有":
+                self.sessions.clear_all_session()
+                reply = Reply(ReplyType.INFO, "所有人记忆已清除")
+            elif query == "#更新配置":
+                load_config()
+                reply = Reply(ReplyType.INFO, "配置已更新")
+            if reply:
+                return reply
+            session = self.sessions.session_query(query, session_id)
+            logger.debug("[QWEN] session query={}".format(session.messages))
+
+            reply_content = self.reply_text(session)
+            logger.debug(
+                "[QWEN] new_query={}, session_id={}, reply_cont={}, completion_tokens={}".format(
+                    session.messages,
+                    session_id,
+                    reply_content["content"],
+                    reply_content["completion_tokens"],
+                )
+            )
+            if reply_content["completion_tokens"] == 0 and len(reply_content["content"]) > 0:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+            elif reply_content["completion_tokens"] > 0:
+                self.sessions.session_reply(reply_content["content"], session_id, reply_content["total_tokens"])
+                reply = Reply(ReplyType.TEXT, reply_content["content"])
+            else:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+                logger.debug("[QWEN] reply {} used 0 tokens.".format(reply_content))
+            return reply
+
+        else:
+            reply = Reply(ReplyType.ERROR, "Bot不支持处理{}类型的消息".format(context.type))
+            return reply
+
+    def reply_text(self, session: AliQwenSession, retry_count=0) -> dict:
+        """
+        call bailian's ChatCompletion to get the answer
+        :param session: a conversation session
+        :param retry_count: retry count
+        :return: {}
+        """
+        try:
+            prompt, history = self.convert_messages_format(session.messages)
+            self.update_api_key_if_expired()
+            # NOTE 阿里百炼的call()函数未提供temperature参数，考虑到temperature和top_p参数作用相同，取两者较小的值作为top_p参数传入，详情见文档 https://help.aliyun.com/document_detail/2587502.htm
+            response = broadscope_bailian.Completions().call(app_id=self.app_id(), prompt=prompt, history=history, top_p=min(self.temperature(), self.top_p()))
+            completion_content = self.get_completion_content(response, self.node_id())
+            completion_tokens, total_tokens = self.calc_tokens(session.messages, completion_content)
+            return {
+                "total_tokens": total_tokens,
+                "completion_tokens": completion_tokens,
+                "content": completion_content,
+            }
+        except Exception as e:
+            need_retry = retry_count < 2
+            result = {"completion_tokens": 0, "content": "我现在有点累了，等会再来吧"}
+            if isinstance(e, openai.error.RateLimitError):
+                logger.warn("[QWEN] RateLimitError: {}".format(e))
+                result["content"] = "提问太快啦，请休息一下再问我吧"
+                if need_retry:
+                    time.sleep(20)
+            elif isinstance(e, openai.error.Timeout):
+                logger.warn("[QWEN] Timeout: {}".format(e))
+                result["content"] = "我没有收到你的消息"
+                if need_retry:
+                    time.sleep(5)
+            elif isinstance(e, openai.error.APIError):
+                logger.warn("[QWEN] Bad Gateway: {}".format(e))
+                result["content"] = "请再问我一次"
+                if need_retry:
+                    time.sleep(10)
+            elif isinstance(e, openai.error.APIConnectionError):
+                logger.warn("[QWEN] APIConnectionError: {}".format(e))
+                need_retry = False
+                result["content"] = "我连接不到你的网络"
+            else:
+                logger.exception("[QWEN] Exception: {}".format(e))
+                need_retry = False
+                self.sessions.clear_session(session.session_id)
+
+            if need_retry:
+                logger.warn("[QWEN] 第{}次重试".format(retry_count + 1))
+                return self.reply_text(session, retry_count + 1)
+            else:
+                return result
+
+    def set_api_key(self):
+        api_key, expired_time = self.api_key_client().create_token(agent_key=self.agent_key())
+        broadscope_bailian.api_key = api_key
+        return expired_time
+
+    def update_api_key_if_expired(self):
+        if time.time() > self.api_key_expired_time:
+            self.api_key_expired_time = self.set_api_key()
+
+    def convert_messages_format(self, messages) -> Tuple[str, List[ChatQaMessage]]:
+        history = []
+        user_content = ''
+        assistant_content = ''
+        system_content = ''
+        for message in messages:
+            role = message.get('role')
+            if role == 'user':
+                user_content += message.get('content')
+            elif role == 'assistant':
+                assistant_content = message.get('content')
+                history.append(ChatQaMessage(user_content, assistant_content))
+                user_content = ''
+                assistant_content = ''
+            elif role =='system':
+                system_content += message.get('content')
+        if user_content == '':
+            raise Exception('no user message')
+        if system_content != '':
+            # NOTE 模拟系统消息，测试发现人格描述以"你需要扮演ChatGPT"开头能够起作用，而以"你是ChatGPT"开头模型会直接否认
+            system_qa = ChatQaMessage(system_content, '好的，我会严格按照你的设定回答问题')
+            history.insert(0, system_qa)
+        logger.debug("[QWEN] converted qa messages: {}".format([item.to_dict() for item in history]))
+        logger.debug("[QWEN] user content as prompt: {}".format(user_content))
+        return user_content, history
+
+    def get_completion_content(self, response, node_id):
+        if not response['Success']:
+            return f"[ERROR]\n{response['Code']}:{response['Message']}"
+        text = response['Data']['Text']
+        if node_id == '':
+            return text
+        # TODO: 当使用流程编排创建大模型应用时，响应结构如下，最终结果在['finalResult'][node_id]['response']['text']中，暂时先这么写
+        # {
+        #     'Success': True,
+        #     'Code': None,
+        #     'Message': None,
+        #     'Data': {
+        #         'ResponseId': '9822f38dbacf4c9b8daf5ca03a2daf15',
+        #         'SessionId': 'session_id',
+        #         'Text': '{"finalResult":{"LLM_T7islK":{"params":{"modelId":"qwen-plus-v1","prompt":"${systemVars.query}${bizVars.Text}"},"response":{"text":"作为一个AI语言模型，我没有年龄，因为我没有生日。\n我只是一个程序，没有生命和身体。"}}}}',
+        #         'Thoughts': [],
+        #         'Debug': {},
+        #         'DocReferences': []
+        #     },
+        #     'RequestId': '8e11d31551ce4c3f83f49e6e0dd998b0',
+        #     'Failed': None
+        # }
+        text_dict = json.loads(text)
+        completion_content =  text_dict['finalResult'][node_id]['response']['text']
+        return completion_content
+
+    def calc_tokens(self, messages, completion_content):
+        completion_tokens = len(completion_content)
+        prompt_tokens = 0
+        for message in messages:
+            prompt_tokens += len(message["content"])
+        return completion_tokens, prompt_tokens + completion_tokens

bot/ali/ali_qwen_session.py

@@ -0,0 +1,62 @@
+from bot.session_manager import Session
+from common.log import logger
+
+"""
+    e.g.
+    [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Who won the world series in 2020?"},
+        {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
+        {"role": "user", "content": "Where was it played?"}
+    ]
+"""
+
+class AliQwenSession(Session):
+    def __init__(self, session_id, system_prompt=None, model="qianwen"):
+        super().__init__(session_id, system_prompt)
+        self.model = model
+        self.reset()
+
+    def discard_exceeding(self, max_tokens, cur_tokens=None):
+        precise = True
+        try:
+            cur_tokens = self.calc_tokens()
+        except Exception as e:
+            precise = False
+            if cur_tokens is None:
+                raise e
+            logger.debug("Exception when counting tokens precisely for query: {}".format(e))
+        while cur_tokens > max_tokens:
+            if len(self.messages) > 2:
+                self.messages.pop(1)
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "assistant":
+                self.messages.pop(1)
+                if precise:
+                    cur_tokens = self.calc_tokens()
+                else:
+                    cur_tokens = cur_tokens - max_tokens
+                break
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "user":
+                logger.warn("user message exceed max_tokens. total_tokens={}".format(cur_tokens))
+                break
+            else:
+                logger.debug("max_tokens={}, total_tokens={}, len(messages)={}".format(max_tokens, cur_tokens, len(self.messages)))
+                break
+            if precise:
+                cur_tokens = self.calc_tokens()
+            else:
+                cur_tokens = cur_tokens - max_tokens
+        return cur_tokens
+
+    def calc_tokens(self):
+        return num_tokens_from_messages(self.messages, self.model)
+
+def num_tokens_from_messages(messages, model):
+    """Returns the number of tokens used by a list of messages."""
+    # 官方token计算规则："对于中文文本来说，1个token通常对应一个汉字；对于英文文本来说，1个token通常对应3至4个字母或1个单词"
+    # 详情请产看文档：https://help.aliyun.com/document_detail/2586397.html
+    # 目前根据字符串长度粗略估计token数，不影响正常使用
+    tokens = 0
+    for msg in messages:
+        tokens += len(msg["content"])
+    return tokens

bot/baidu/baidu_wenxin.py

@@ -16,7 +16,10 @@ class BaiduWenxinBot(Bot):
 
     def __init__(self):
         super().__init__()
-        self.sessions = SessionManager(BaiduWenxinSession, model=conf().get("baidu_wenxin_model") or "eb-instant")
+        wenxin_model = conf().get("baidu_wenxin_model") or "eb-instant"
+        if conf().get("model") and conf().get("model") == "wenxin-4":
+            wenxin_model = "completions_pro"
+        self.sessions = SessionManager(BaiduWenxinSession, model=wenxin_model)
 
     def reply(self, query, context=None):
         # acquire reply content

bot/bot_factory.py

@@ -43,4 +43,18 @@ def create_bot(bot_type):
     elif bot_type == const.CLAUDEAI:
         from bot.claude.claude_ai_bot import ClaudeAIBot
         return ClaudeAIBot()
+
+    elif bot_type == const.QWEN:
+        from bot.ali.ali_qwen_bot import AliQwenBot
+        return AliQwenBot()
+
+    elif bot_type == const.GEMINI:
+        from bot.gemini.google_gemini_bot import GoogleGeminiBot
+        return GoogleGeminiBot()
+
+    elif bot_type == const.ZHIPU_AI:
+        from bot.zhipuai.zhipuai_bot import ZHIPUAIBot
+        return ZHIPUAIBot()
+
+
     raise RuntimeError

bot/chatgpt/chat_gpt_bot.py

@@ -148,8 +148,9 @@ class ChatGPTBot(Bot, OpenAIImage):
                     time.sleep(10)
             elif isinstance(e, openai.error.APIConnectionError):
                 logger.warn("[CHATGPT] APIConnectionError: {}".format(e))
-                need_retry = False
                 result["content"] = "我连接不到你的网络"
+                if need_retry:
+                    time.sleep(5)
             else:
                 logger.exception("[CHATGPT] Exception: {}".format(e))
                 need_retry = False

bot/chatgpt/chat_gpt_session.py

@@ -1,5 +1,6 @@
 from bot.session_manager import Session
 from common.log import logger
+from common import const
 
 """
     e.g.  [
@@ -56,15 +57,16 @@ class ChatGPTSession(Session):
 def num_tokens_from_messages(messages, model):
     """Returns the number of tokens used by a list of messages."""
 
-    if model in ["wenxin", "xunfei"]:
+    if model in ["wenxin", "xunfei", const.GEMINI]:
         return num_tokens_by_character(messages)
 
     import tiktoken
 
-    if model in ["gpt-3.5-turbo-0301", "gpt-35-turbo"]:
+    if model in ["gpt-3.5-turbo-0301", "gpt-35-turbo", "gpt-3.5-turbo-1106"]:
         return num_tokens_from_messages(messages, model="gpt-3.5-turbo")
     elif model in ["gpt-4-0314", "gpt-4-0613", "gpt-4-32k", "gpt-4-32k-0613", "gpt-3.5-turbo-0613",
-                   "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-35-turbo-16k"]:
+                   "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-35-turbo-16k", "gpt-4-turbo-preview",
+                   "gpt-4-1106-preview", const.GPT4_TURBO_PREVIEW, const.GPT4_VISION_PREVIEW]:
         return num_tokens_from_messages(messages, model="gpt-4")
 
     try:

bot/gemini/google_gemini_bot.py

@@ -0,0 +1,75 @@
+"""
+Google gemini bot
+
+@author zhayujie
+@Date 2023/12/15
+"""
+# encoding:utf-8
+
+from bot.bot import Bot
+import google.generativeai as genai
+from bot.session_manager import SessionManager
+from bridge.context import ContextType, Context
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf
+from bot.baidu.baidu_wenxin_session import BaiduWenxinSession
+
+
+# OpenAI对话模型API (可用)
+class GoogleGeminiBot(Bot):
+
+    def __init__(self):
+        super().__init__()
+        self.api_key = conf().get("gemini_api_key")
+        # 复用文心的token计算方式
+        self.sessions = SessionManager(BaiduWenxinSession, model=conf().get("model") or "gpt-3.5-turbo")
+
+    def reply(self, query, context: Context = None) -> Reply:
+        try:
+            if context.type != ContextType.TEXT:
+                logger.warn(f"[Gemini] Unsupported message type, type={context.type}")
+                return Reply(ReplyType.TEXT, None)
+            logger.info(f"[Gemini] query={query}")
+            session_id = context["session_id"]
+            session = self.sessions.session_query(query, session_id)
+            gemini_messages = self._convert_to_gemini_messages(self._filter_messages(session.messages))
+            genai.configure(api_key=self.api_key)
+            model = genai.GenerativeModel('gemini-pro')
+            response = model.generate_content(gemini_messages)
+            reply_text = response.text
+            self.sessions.session_reply(reply_text, session_id)
+            logger.info(f"[Gemini] reply={reply_text}")
+            return Reply(ReplyType.TEXT, reply_text)
+        except Exception as e:
+            logger.error("[Gemini] fetch reply error, may contain unsafe content")
+            logger.error(e)
+
+    def _convert_to_gemini_messages(self, messages: list):
+        res = []
+        for msg in messages:
+            if msg.get("role") == "user":
+                role = "user"
+            elif msg.get("role") == "assistant":
+                role = "model"
+            else:
+                continue
+            res.append({
+                "role": role,
+                "parts": [{"text": msg.get("content")}]
+            })
+        return res
+
+    def _filter_messages(self, messages: list):
+        res = []
+        turn = "user"
+        for i in range(len(messages) - 1, -1, -1):
+            message = messages[i]
+            if message.get("role") != turn:
+                continue
+            res.insert(0, message)
+            if turn == "user":
+                turn = "assistant"
+            elif turn == "assistant":
+                turn = "user"
+        return res

bot/linkai/link_ai_bot.py

@@ -1,34 +1,39 @@
 # access LinkAI knowledge base platform
 # docs: https://link-ai.tech/platform/link-app/wechat
 
+import re
 import time
-
 import requests
-
+import config
 from bot.bot import Bot
 from bot.chatgpt.chat_gpt_session import ChatGPTSession
-from bot.openai.open_ai_image import OpenAIImage
 from bot.session_manager import SessionManager
 from bridge.context import Context, ContextType
 from bridge.reply import Reply, ReplyType
 from common.log import logger
-from config import conf
+from config import conf, pconf
+import threading
+from common import memory, utils
+import base64
+import os
 
-
-class LinkAIBot(Bot, OpenAIImage):
+class LinkAIBot(Bot):
     # authentication failed
     AUTH_FAILED_CODE = 401
     NO_QUOTA_CODE = 406
 
     def __init__(self):
         super().__init__()
-        self.sessions = SessionManager(ChatGPTSession, model=conf().get("model") or "gpt-3.5-turbo")
+        self.sessions = LinkAISessionManager(LinkAISession, model=conf().get("model") or "gpt-3.5-turbo")
         self.args = {}
 
     def reply(self, query, context: Context = None) -> Reply:
         if context.type == ContextType.TEXT:
             return self._chat(query, context)
         elif context.type == ContextType.IMAGE_CREATE:
+            if not conf().get("text_to_image"):
+                logger.warn("[LinkAI] text_to_image is not enabled, ignore the IMAGE_CREATE request")
+                return Reply(ReplyType.TEXT, "")
             ok, res = self.create_img(query, 0)
             if ok:
                 reply = Reply(ReplyType.IMAGE_URL, res)
@@ -47,10 +52,10 @@ class LinkAIBot(Bot, OpenAIImage):
         :param retry_count: 当前递归重试次数
         :return: 回复
         """
-        if retry_count >= 2:
+        if retry_count > 2:
             # exit from retry 2 times
             logger.warn("[LINKAI] failed after maximum number of retry times")
-            return Reply(ReplyType.ERROR, "请再问我一次吧")
+            return Reply(ReplyType.TEXT, "请再问我一次吧")
 
         try:
             # load config
@@ -58,28 +63,61 @@ class LinkAIBot(Bot, OpenAIImage):
                 logger.info(f"[LINKAI] won't set appcode because a plugin ({context['generate_breaked_by']}) affected the context")
                 app_code = None
             else:
-                app_code = context.kwargs.get("app_code") or conf().get("linkai_app_code")
+                plugin_app_code = self._find_group_mapping_code(context)
+                app_code = context.kwargs.get("app_code") or plugin_app_code or conf().get("linkai_app_code")
             linkai_api_key = conf().get("linkai_api_key")
 
             session_id = context["session_id"]
+            session_message = self.sessions.session_msg_query(query, session_id)
+            logger.debug(f"[LinkAI] session={session_message}, session_id={session_id}")
 
-            session = self.sessions.session_query(query, session_id)
-            model = conf().get("model") or "gpt-3.5-turbo"
+            # image process
+            img_cache = memory.USER_IMAGE_CACHE.get(session_id)
+            if img_cache:
+                messages = self._process_image_msg(app_code=app_code, session_id=session_id, query=query, img_cache=img_cache)
+                if messages:
+                    session_message = messages
+
+            model = conf().get("model")
             # remove system message
-            if session.messages[0].get("role") == "system":
+            if session_message[0].get("role") == "system":
                 if app_code or model == "wenxin":
-                    session.messages.pop(0)
-
+                    session_message.pop(0)
             body = {
                 "app_code": app_code,
-                "messages": session.messages,
+                "messages": session_message,
                 "model": model,     # 对话模型的名称, 支持 gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-4, wenxin, xunfei
                 "temperature": conf().get("temperature"),
                 "top_p": conf().get("top_p", 1),
                 "frequency_penalty": conf().get("frequency_penalty", 0.0),  # [-2,2]之间，该值越大则更倾向于产生不同的内容
                 "presence_penalty": conf().get("presence_penalty", 0.0),  # [-2,2]之间，该值越大则更倾向于产生不同的内容
+                "session_id": session_id,
+                "channel_type": conf().get("channel_type", "wx")
             }
-            logger.info(f"[LINKAI] query={query}, app_code={app_code}, mode={body.get('model')}")
+            try:
+                from linkai import LinkAIClient
+                client_id = LinkAIClient.fetch_client_id()
+                if client_id:
+                    body["client_id"] = client_id
+                    # start: client info deliver
+                    if context.kwargs.get("msg"):
+                        body["session_id"] = context.kwargs.get("msg").from_user_id
+                        if context.kwargs.get("msg").is_group:
+                            body["is_group"] = True
+                            body["group_name"] = context.kwargs.get("msg").from_user_nickname
+                            body["sender_name"] = context.kwargs.get("msg").actual_user_nickname
+                        else:
+                            if body.get("channel_type") in ["wechatcom_app"]:
+                                body["sender_name"] = context.kwargs.get("msg").from_user_id
+                            else:
+                                body["sender_name"] = context.kwargs.get("msg").from_user_nickname
+
+            except Exception as e:
+                pass
+            file_id = context.kwargs.get("file_id")
+            if file_id:
+                body["file_id"] = file_id
+            logger.info(f"[LINKAI] query={query}, app_code={app_code}, model={body.get('model')}, file_id={file_id}")
             headers = {"Authorization": "Bearer " + linkai_api_key}
 
             # do http request
@@ -92,7 +130,22 @@ class LinkAIBot(Bot, OpenAIImage):
                 reply_content = response["choices"][0]["message"]["content"]
                 total_tokens = response["usage"]["total_tokens"]
                 logger.info(f"[LINKAI] reply={reply_content}, total_tokens={total_tokens}")
-                self.sessions.session_reply(reply_content, session_id, total_tokens)
+                self.sessions.session_reply(reply_content, session_id, total_tokens, query=query)
+    
+                agent_suffix = self._fetch_agent_suffix(response)
+                if agent_suffix:
+                    reply_content += agent_suffix
+                if not agent_suffix:
+                    knowledge_suffix = self._fetch_knowledge_search_suffix(response)
+                    if knowledge_suffix:
+                        reply_content += knowledge_suffix
+                # image process
+                if response["choices"][0].get("img_urls"):
+                    thread = threading.Thread(target=self._send_image, args=(context.get("channel"), context, response["choices"][0].get("img_urls")))
+                    thread.start()
+                    if response["choices"][0].get("text_content"):
+                        reply_content = response["choices"][0].get("text_content")
+                reply_content = self._process_url(reply_content)
                 return Reply(ReplyType.TEXT, reply_content)
 
             else:
@@ -107,7 +160,7 @@ class LinkAIBot(Bot, OpenAIImage):
                     logger.warn(f"[LINKAI] do retry, times={retry_count}")
                     return self._chat(query, context, retry_count + 1)
 
-                return Reply(ReplyType.ERROR, "提问太快啦，请休息一下再问我吧")
+                return Reply(ReplyType.TEXT, "提问太快啦，请休息一下再问我吧")
 
         except Exception as e:
             logger.exception(e)
@@ -116,6 +169,66 @@ class LinkAIBot(Bot, OpenAIImage):
             logger.warn(f"[LINKAI] do retry, times={retry_count}")
             return self._chat(query, context, retry_count + 1)
 
+    def _process_image_msg(self, app_code: str, session_id: str, query:str, img_cache: dict):
+        try:
+            enable_image_input = False
+            app_info = self._fetch_app_info(app_code)
+            if not app_info:
+                logger.debug(f"[LinkAI] not found app, can't process images, app_code={app_code}")
+                return None
+            plugins = app_info.get("data").get("plugins")
+            for plugin in plugins:
+                if plugin.get("input_type") and "IMAGE" in plugin.get("input_type"):
+                    enable_image_input = True
+            if not enable_image_input:
+                return
+            msg = img_cache.get("msg")
+            path = img_cache.get("path")
+            msg.prepare()
+            logger.info(f"[LinkAI] query with images, path={path}")
+            messages = self._build_vision_msg(query, path)
+            memory.USER_IMAGE_CACHE[session_id] = None
+            return messages
+        except Exception as e:
+            logger.exception(e)
+
+    def _find_group_mapping_code(self, context):
+        try:
+            if context.kwargs.get("isgroup"):
+                group_name = context.kwargs.get("msg").from_user_nickname
+                if config.plugin_config and config.plugin_config.get("linkai"):
+                    linkai_config = config.plugin_config.get("linkai")
+                    group_mapping = linkai_config.get("group_app_map")
+                    if group_mapping and group_name:
+                        return group_mapping.get(group_name)
+        except Exception as e:
+            logger.exception(e)
+            return None
+
+    def _build_vision_msg(self, query: str, path: str):
+        try:
+            suffix = utils.get_path_suffix(path)
+            with open(path, "rb") as file:
+                base64_str = base64.b64encode(file.read()).decode('utf-8')
+                messages = [{
+                    "role": "user",
+                    "content": [
+                        {
+                            "type": "text",
+                            "text": query
+                        },
+                        {
+                            "type": "image_url",
+                            "image_url": {
+                                "url": f"data:image/{suffix};base64,{base64_str}"
+                            }
+                        }
+                    ]
+                }]
+                return messages
+        except Exception as e:
+            logger.exception(e)
+
     def reply_text(self, session: ChatGPTSession, app_code="", retry_count=0) -> dict:
         if retry_count >= 2:
             # exit from retry 2 times
@@ -138,7 +251,7 @@ class LinkAIBot(Bot, OpenAIImage):
             }
             if self.args.get("max_tokens"):
                 body["max_tokens"] = self.args.get("max_tokens")
-            headers = {"Authorization": "Bearer " +  conf().get("linkai_api_key")}
+            headers = {"Authorization": "Bearer " + conf().get("linkai_api_key")}
 
             # do http request
             base_url = conf().get("linkai_api_base", "https://api.link-ai.chat")
@@ -180,3 +293,175 @@ class LinkAIBot(Bot, OpenAIImage):
             time.sleep(2)
             logger.warn(f"[LINKAI] do retry, times={retry_count}")
             return self.reply_text(session, app_code, retry_count + 1)
+
+    def _fetch_app_info(self, app_code: str):
+        headers = {"Authorization": "Bearer " + conf().get("linkai_api_key")}
+        # do http request
+        base_url = conf().get("linkai_api_base", "https://api.link-ai.chat")
+        params = {"app_code": app_code}
+        res = requests.get(url=base_url + "/v1/app/info", params=params, headers=headers, timeout=(5, 10))
+        if res.status_code == 200:
+            return res.json()
+        else:
+            logger.warning(f"[LinkAI] find app info exception, res={res}")
+
+    def create_img(self, query, retry_count=0, api_key=None):
+        try:
+            logger.info("[LinkImage] image_query={}".format(query))
+            headers = {
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {conf().get('linkai_api_key')}"
+            }
+            data = {
+                "prompt": query,
+                "n": 1,
+                "model": conf().get("text_to_image") or "dall-e-2",
+                "response_format": "url",
+                "img_proxy": conf().get("image_proxy")
+            }
+            url = conf().get("linkai_api_base", "https://api.link-ai.chat") + "/v1/images/generations"
+            res = requests.post(url, headers=headers, json=data, timeout=(5, 90))
+            t2 = time.time()
+            image_url = res.json()["data"][0]["url"]
+            logger.info("[OPEN_AI] image_url={}".format(image_url))
+            return True, image_url
+
+        except Exception as e:
+            logger.error(format(e))
+            return False, "画图出现问题，请休息一下再问我吧"
+
+
+    def _fetch_knowledge_search_suffix(self, response) -> str:
+        try:
+            if response.get("knowledge_base"):
+                search_hit = response.get("knowledge_base").get("search_hit")
+                first_similarity = response.get("knowledge_base").get("first_similarity")
+                logger.info(f"[LINKAI] knowledge base, search_hit={search_hit}, first_similarity={first_similarity}")
+                plugin_config = pconf("linkai")
+                if plugin_config and plugin_config.get("knowledge_base") and plugin_config.get("knowledge_base").get("search_miss_text_enabled"):
+                    search_miss_similarity = plugin_config.get("knowledge_base").get("search_miss_similarity")
+                    search_miss_text = plugin_config.get("knowledge_base").get("search_miss_suffix")
+                    if not search_hit:
+                        return search_miss_text
+                    if search_miss_similarity and float(search_miss_similarity) > first_similarity:
+                        return search_miss_text
+        except Exception as e:
+            logger.exception(e)
+
+
+    def _fetch_agent_suffix(self, response):
+        try:
+            plugin_list = []
+            logger.debug(f"[LinkAgent] res={response}")
+            if response.get("agent") and response.get("agent").get("chain") and response.get("agent").get("need_show_plugin"):
+                chain = response.get("agent").get("chain")
+                suffix = "\n\n- - - - - - - - - - - -"
+                i = 0
+                for turn in chain:
+                    plugin_name = turn.get('plugin_name')
+                    suffix += "\n"
+                    need_show_thought = response.get("agent").get("need_show_thought")
+                    if turn.get("thought") and plugin_name and need_show_thought:
+                        suffix += f"{turn.get('thought')}\n"
+                    if plugin_name:
+                        plugin_list.append(turn.get('plugin_name'))
+                        if turn.get('plugin_icon'):
+                            suffix += f"{turn.get('plugin_icon')} "
+                        suffix += f"{turn.get('plugin_name')}"
+                        if turn.get('plugin_input'):
+                            suffix += f"：{turn.get('plugin_input')}"
+                    if i < len(chain) - 1:
+                        suffix += "\n"
+                    i += 1
+                logger.info(f"[LinkAgent] use plugins: {plugin_list}")
+                return suffix
+        except Exception as e:
+            logger.exception(e)
+
+    def _process_url(self, text):
+        try:
+            url_pattern = re.compile(r'\[(.*?)\]\((http[s]?://.*?)\)')
+            def replace_markdown_url(match):
+                return f"{match.group(2)}"
+            return url_pattern.sub(replace_markdown_url, text)
+        except Exception as e:
+            logger.error(e)
+
+    def _send_image(self, channel, context, image_urls):
+        if not image_urls:
+            return
+        max_send_num = conf().get("max_media_send_count")
+        send_interval = conf().get("media_send_interval")
+        try:
+            i = 0
+            for url in image_urls:
+                if max_send_num and i >= max_send_num:
+                    continue
+                i += 1
+                if url.endswith(".mp4"):
+                    reply_type = ReplyType.VIDEO_URL
+                elif url.endswith(".pdf") or url.endswith(".doc") or url.endswith(".docx") or url.endswith(".csv"):
+                    reply_type = ReplyType.FILE
+                    url = _download_file(url)
+                    if not url:
+                        continue
+                else:
+                    reply_type = ReplyType.IMAGE_URL
+                reply = Reply(reply_type, url)
+                channel.send(reply, context)
+                if send_interval:
+                    time.sleep(send_interval)
+        except Exception as e:
+            logger.error(e)
+
+
+def _download_file(url: str):
+    try:
+        file_path = "tmp"
+        if not os.path.exists(file_path):
+            os.makedirs(file_path)
+        file_name = url.split("/")[-1]  # 获取文件名
+        file_path = os.path.join(file_path, file_name)
+        response = requests.get(url)
+        with open(file_path, "wb") as f:
+            f.write(response.content)
+        return file_path
+    except Exception as e:
+        logger.warn(e)
+
+
+class LinkAISessionManager(SessionManager):
+    def session_msg_query(self, query, session_id):
+        session = self.build_session(session_id)
+        messages = session.messages + [{"role": "user", "content": query}]
+        return messages
+
+    def session_reply(self, reply, session_id, total_tokens=None, query=None):
+        session = self.build_session(session_id)
+        if query:
+            session.add_query(query)
+        session.add_reply(reply)
+        try:
+            max_tokens = conf().get("conversation_max_tokens", 2500)
+            tokens_cnt = session.discard_exceeding(max_tokens, total_tokens)
+            logger.debug(f"[LinkAI] chat history, before tokens={total_tokens}, now tokens={tokens_cnt}")
+        except Exception as e:
+            logger.warning("Exception when counting tokens precisely for session: {}".format(str(e)))
+        return session
+
+
+class LinkAISession(ChatGPTSession):
+    def calc_tokens(self):
+        if not self.messages:
+            return 0
+        return len(str(self.messages))
+
+    def discard_exceeding(self, max_tokens, cur_tokens=None):
+        cur_tokens = self.calc_tokens()
+        if cur_tokens > max_tokens:
+            for i in range(0, len(self.messages)):
+                if i > 0 and self.messages[i].get("role") == "assistant" and self.messages[i - 1].get("role") == "user":
+                    self.messages.pop(i)
+                    self.messages.pop(i - 1)
+                    return self.calc_tokens()
+        return cur_tokens

bot/openai/open_ai_image.py

@@ -15,7 +15,7 @@ class OpenAIImage(object):
         if conf().get("rate_limit_dalle"):
             self.tb4dalle = TokenBucket(conf().get("rate_limit_dalle", 50))
 
-    def create_img(self, query, retry_count=0, api_key=None):
+    def create_img(self, query, retry_count=0, api_key=None, api_base=None):
         try:
             if conf().get("rate_limit_dalle") and not self.tb4dalle.get_token():
                 return False, "请求太快了，请休息一下再问我吧"
@@ -24,7 +24,8 @@ class OpenAIImage(object):
                 api_key=api_key,
                 prompt=query,  # 图片描述
                 n=1,  # 每次生成图片的数量
-                size=conf().get("image_create_size", "256x256"),  # 图片大小,可选有 256x256, 512x512, 1024x1024
+                model=conf().get("text_to_image") or "dall-e-2",
+                # size=conf().get("image_create_size", "256x256"),  # 图片大小,可选有 256x256, 512x512, 1024x1024
             )
             image_url = response["data"][0]["url"]
             logger.info("[OPEN_AI] image_url={}".format(image_url))
@@ -36,7 +37,7 @@ class OpenAIImage(object):
                 logger.warn("[OPEN_AI] ImgCreate RateLimit exceed, 第{}次重试".format(retry_count + 1))
                 return self.create_img(query, retry_count + 1)
             else:
-                return False, "提问太快啦，请休息一下再问我吧"
+                return False, "画图出现问题，请休息一下再问我吧"
         except Exception as e:
             logger.exception(e)
-            return False, str(e)
+            return False, "画图出现问题，请休息一下再问我吧"

bot/session_manager.py

@@ -69,7 +69,7 @@ class SessionManager(object):
             total_tokens = session.discard_exceeding(max_tokens, None)
             logger.debug("prompt tokens used={}".format(total_tokens))
         except Exception as e:
-            logger.debug("Exception when counting tokens precisely for prompt: {}".format(str(e)))
+            logger.warning("Exception when counting tokens precisely for prompt: {}".format(str(e)))
         return session
 
     def session_reply(self, reply, session_id, total_tokens=None):
@@ -80,7 +80,7 @@ class SessionManager(object):
             tokens_cnt = session.discard_exceeding(max_tokens, total_tokens)
             logger.debug("raw total_tokens={}, savesession tokens={}".format(total_tokens, tokens_cnt))
         except Exception as e:
-            logger.debug("Exception when counting tokens precisely for session: {}".format(str(e)))
+            logger.warning("Exception when counting tokens precisely for session: {}".format(str(e)))
         return session
 
     def clear_session(self, session_id):

bot/xunfei/xunfei_spark_bot.py

@@ -40,10 +40,14 @@ class XunFeiBot(Bot):
         self.app_id = conf().get("xunfei_app_id")
         self.api_key = conf().get("xunfei_api_key")
         self.api_secret = conf().get("xunfei_api_secret")
-        # 默认使用v2.0版本，1.5版本可设置为 general
-        self.domain = "generalv2"
-        # 默认使用v2.0版本，1.5版本可设置为 "ws://spark-api.xf-yun.com/v1.1/chat"
-        self.spark_url = "ws://spark-api.xf-yun.com/v2.1/chat"
+        # 默认使用v2.0版本: "generalv2"
+        # v1.5版本为 "general"
+        # v3.0版本为: "generalv3"
+        self.domain = "generalv3"
+        # 默认使用v2.0版本: "ws://spark-api.xf-yun.com/v2.1/chat"
+        # v1.5版本为: "ws://spark-api.xf-yun.com/v1.1/chat"
+        # v3.0版本为: "ws://spark-api.xf-yun.com/v3.1/chat"
+        self.spark_url = "ws://spark-api.xf-yun.com/v3.1/chat"
         self.host = urlparse(self.spark_url).netloc
         self.path = urlparse(self.spark_url).path
         # 和wenxin使用相同的session机制
@@ -56,7 +60,8 @@ class XunFeiBot(Bot):
             request_id = self.gen_request_id(session_id)
             reply_map[request_id] = ""
             session = self.sessions.session_query(query, session_id)
-            threading.Thread(target=self.create_web_socket, args=(session.messages, request_id)).start()
+            threading.Thread(target=self.create_web_socket,
+                             args=(session.messages, request_id)).start()
             depth = 0
             time.sleep(0.1)
             t1 = time.time()
@@ -83,20 +88,27 @@ class XunFeiBot(Bot):
                     depth += 1
                     continue
             t2 = time.time()
-            logger.info(f"[XunFei-API] response={reply_map[request_id]}, time={t2 - t1}s, usage={usage}")
-            self.sessions.session_reply(reply_map[request_id], session_id, usage.get("total_tokens"))
+            logger.info(
+                f"[XunFei-API] response={reply_map[request_id]}, time={t2 - t1}s, usage={usage}"
+            )
+            self.sessions.session_reply(reply_map[request_id], session_id,
+                                        usage.get("total_tokens"))
             reply = Reply(ReplyType.TEXT, reply_map[request_id])
             del reply_map[request_id]
             return reply
         else:
-            reply = Reply(ReplyType.ERROR, "Bot不支持处理{}类型的消息".format(context.type))
+            reply = Reply(ReplyType.ERROR,
+                          "Bot不支持处理{}类型的消息".format(context.type))
             return reply
 
     def create_web_socket(self, prompt, session_id, temperature=0.5):
         logger.info(f"[XunFei] start connect, prompt={prompt}")
         websocket.enableTrace(False)
         wsUrl = self.create_url()
-        ws = websocket.WebSocketApp(wsUrl, on_message=on_message, on_error=on_error, on_close=on_close,
+        ws = websocket.WebSocketApp(wsUrl,
+                                    on_message=on_message,
+                                    on_error=on_error,
+                                    on_close=on_close,
                                     on_open=on_open)
         data_queue = queue.Queue(1000)
         queue_map[session_id] = data_queue
@@ -108,7 +120,8 @@ class XunFeiBot(Bot):
         ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
 
     def gen_request_id(self, session_id: str):
-        return session_id + "_" + str(int(time.time())) + "" + str(random.randint(0, 100))
+        return session_id + "_" + str(int(time.time())) + "" + str(
+            random.randint(0, 100))
 
     # 生成url
     def create_url(self):
@@ -122,22 +135,21 @@ class XunFeiBot(Bot):
         signature_origin += "GET " + self.path + " HTTP/1.1"
 
         # 进行hmac-sha256进行加密
-        signature_sha = hmac.new(self.api_secret.encode('utf-8'), signature_origin.encode('utf-8'),
+        signature_sha = hmac.new(self.api_secret.encode('utf-8'),
+                                 signature_origin.encode('utf-8'),
                                  digestmod=hashlib.sha256).digest()
 
-        signature_sha_base64 = base64.b64encode(signature_sha).decode(encoding='utf-8')
+        signature_sha_base64 = base64.b64encode(signature_sha).decode(
+            encoding='utf-8')
 
         authorization_origin = f'api_key="{self.api_key}", algorithm="hmac-sha256", headers="host date request-line", ' \
                                f'signature="{signature_sha_base64}"'
 
-        authorization = base64.b64encode(authorization_origin.encode('utf-8')).decode(encoding='utf-8')
+        authorization = base64.b64encode(
+            authorization_origin.encode('utf-8')).decode(encoding='utf-8')
 
         # 将请求的鉴权参数组合为字典
-        v = {
-            "authorization": authorization,
-            "date": date,
-            "host": self.host
-        }
+        v = {"authorization": authorization, "date": date, "host": self.host}
         # 拼接鉴权参数，生成url
         url = self.spark_url + '?' + urlencode(v)
         # 此处打印出建立连接时候的url,参考本demo的时候可取消上方打印的注释，比对相同参数时生成的url与自己代码生成的url是否一致
@@ -190,11 +202,15 @@ def on_close(ws, one, two):
 # 收到websocket连接建立的处理
 def on_open(ws):
     logger.info(f"[XunFei] Start websocket, session_id={ws.session_id}")
-    thread.start_new_thread(run, (ws,))
+    thread.start_new_thread(run, (ws, ))
 
 
 def run(ws, *args):
-    data = json.dumps(gen_params(appid=ws.appid, domain=ws.domain, question=ws.question, temperature=ws.temperature))
+    data = json.dumps(
+        gen_params(appid=ws.appid,
+                   domain=ws.domain,
+                   question=ws.question,
+                   temperature=ws.temperature))
     ws.send(data)
 
 
@@ -212,7 +228,8 @@ def on_message(ws, message):
         content = choices["text"][0]["content"]
         data_queue = queue_map.get(ws.session_id)
         if not data_queue:
-            logger.error(f"[XunFei] can't find data queue, session_id={ws.session_id}")
+            logger.error(
+                f"[XunFei] can't find data queue, session_id={ws.session_id}")
             return
         reply_item = ReplyItem(content)
         if status == 2:

bot/zhipuai/zhipu_ai_image.py

@@ -0,0 +1,29 @@
+from common.log import logger
+from config import conf
+
+
+# ZhipuAI提供的画图接口
+
+class ZhipuAIImage(object):
+    def __init__(self):
+        from zhipuai import ZhipuAI
+        self.client = ZhipuAI(api_key=conf().get("zhipu_ai_api_key"))
+
+    def create_img(self, query, retry_count=0, api_key=None, api_base=None):
+        try:
+            if conf().get("rate_limit_dalle"):
+                return False, "请求太快了，请休息一下再问我吧"
+            logger.info("[ZHIPU_AI] image_query={}".format(query))
+            response = self.client.images.generations(
+                prompt=query,
+                n=1,  # 每次生成图片的数量
+                model=conf().get("text_to_image") or "cogview-3",
+                size=conf().get("image_create_size", "1024x1024"),  # 图片大小,可选有 256x256, 512x512, 1024x1024
+                quality="standard",
+            )
+            image_url = response.data[0].url
+            logger.info("[ZHIPU_AI] image_url={}".format(image_url))
+            return True, image_url
+        except Exception as e:
+            logger.exception(e)
+            return False, "画图出现问题，请休息一下再问我吧"

bot/zhipuai/zhipu_ai_session.py

@@ -0,0 +1,51 @@
+from bot.session_manager import Session
+from common.log import logger
+
+
+class ZhipuAISession(Session):
+    def __init__(self, session_id, system_prompt=None, model="glm-4"):
+        super().__init__(session_id, system_prompt)
+        self.model = model
+        self.reset()
+
+    def discard_exceeding(self, max_tokens, cur_tokens=None):
+        precise = True
+        try:
+            cur_tokens = self.calc_tokens()
+        except Exception as e:
+            precise = False
+            if cur_tokens is None:
+                raise e
+            logger.debug("Exception when counting tokens precisely for query: {}".format(e))
+        while cur_tokens > max_tokens:
+            if len(self.messages) > 2:
+                self.messages.pop(1)
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "assistant":
+                self.messages.pop(1)
+                if precise:
+                    cur_tokens = self.calc_tokens()
+                else:
+                    cur_tokens = cur_tokens - max_tokens
+                break
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "user":
+                logger.warn("user message exceed max_tokens. total_tokens={}".format(cur_tokens))
+                break
+            else:
+                logger.debug("max_tokens={}, total_tokens={}, len(messages)={}".format(max_tokens, cur_tokens,
+                                                                                       len(self.messages)))
+                break
+            if precise:
+                cur_tokens = self.calc_tokens()
+            else:
+                cur_tokens = cur_tokens - max_tokens
+        return cur_tokens
+
+    def calc_tokens(self):
+        return num_tokens_from_messages(self.messages, self.model)
+
+
+def num_tokens_from_messages(messages, model):
+    tokens = 0
+    for msg in messages:
+        tokens += len(msg["content"])
+    return tokens

bot/zhipuai/zhipuai_bot.py

@@ -0,0 +1,149 @@
+# encoding:utf-8
+
+import time
+
+import openai
+import openai.error
+from bot.bot import Bot
+from bot.zhipuai.zhipu_ai_session import ZhipuAISession
+from bot.zhipuai.zhipu_ai_image import ZhipuAIImage
+from bot.session_manager import SessionManager
+from bridge.context import ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf, load_config
+from zhipuai import ZhipuAI
+
+
+# ZhipuAI对话模型API
+class ZHIPUAIBot(Bot, ZhipuAIImage):
+    def __init__(self):
+        super().__init__()
+        self.sessions = SessionManager(ZhipuAISession, model=conf().get("model") or "ZHIPU_AI")
+        self.args = {
+            "model": conf().get("model") or "glm-4",  # 对话模型的名称
+            "temperature": conf().get("temperature", 0.9),  # 值在(0,1)之间(智谱AI 的温度不能取 0 或者 1)
+            "top_p": conf().get("top_p", 0.7),  # 值在(0,1)之间(智谱AI 的 top_p 不能取 0 或者 1)
+        }
+        self.client = ZhipuAI(api_key=conf().get("zhipu_ai_api_key"))
+
+    def reply(self, query, context=None):
+        # acquire reply content
+        if context.type == ContextType.TEXT:
+            logger.info("[ZHIPU_AI] query={}".format(query))
+
+            session_id = context["session_id"]
+            reply = None
+            clear_memory_commands = conf().get("clear_memory_commands", ["#清除记忆"])
+            if query in clear_memory_commands:
+                self.sessions.clear_session(session_id)
+                reply = Reply(ReplyType.INFO, "记忆已清除")
+            elif query == "#清除所有":
+                self.sessions.clear_all_session()
+                reply = Reply(ReplyType.INFO, "所有人记忆已清除")
+            elif query == "#更新配置":
+                load_config()
+                reply = Reply(ReplyType.INFO, "配置已更新")
+            if reply:
+                return reply
+            session = self.sessions.session_query(query, session_id)
+            logger.debug("[ZHIPU_AI] session query={}".format(session.messages))
+
+            api_key = context.get("openai_api_key") or openai.api_key
+            model = context.get("gpt_model")
+            new_args = None
+            if model:
+                new_args = self.args.copy()
+                new_args["model"] = model
+            # if context.get('stream'):
+            #     # reply in stream
+            #     return self.reply_text_stream(query, new_query, session_id)
+
+            reply_content = self.reply_text(session, api_key, args=new_args)
+            logger.debug(
+                "[ZHIPU_AI] new_query={}, session_id={}, reply_cont={}, completion_tokens={}".format(
+                    session.messages,
+                    session_id,
+                    reply_content["content"],
+                    reply_content["completion_tokens"],
+                )
+            )
+            if reply_content["completion_tokens"] == 0 and len(reply_content["content"]) > 0:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+            elif reply_content["completion_tokens"] > 0:
+                self.sessions.session_reply(reply_content["content"], session_id, reply_content["total_tokens"])
+                reply = Reply(ReplyType.TEXT, reply_content["content"])
+            else:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+                logger.debug("[ZHIPU_AI] reply {} used 0 tokens.".format(reply_content))
+            return reply
+        elif context.type == ContextType.IMAGE_CREATE:
+            ok, retstring = self.create_img(query, 0)
+            reply = None
+            if ok:
+                reply = Reply(ReplyType.IMAGE_URL, retstring)
+            else:
+                reply = Reply(ReplyType.ERROR, retstring)
+            return reply
+
+        else:
+            reply = Reply(ReplyType.ERROR, "Bot不支持处理{}类型的消息".format(context.type))
+            return reply
+
+    def reply_text(self, session: ZhipuAISession, api_key=None, args=None, retry_count=0) -> dict:
+        """
+        call openai's ChatCompletion to get the answer
+        :param session: a conversation session
+        :param session_id: session id
+        :param retry_count: retry count
+        :return: {}
+        """
+        try:
+            # if conf().get("rate_limit_chatgpt") and not self.tb4chatgpt.get_token():
+            #     raise openai.error.RateLimitError("RateLimitError: rate limit exceeded")
+            # if api_key == None, the default openai.api_key will be used
+            if args is None:
+                args = self.args
+            # response = openai.ChatCompletion.create(api_key=api_key, messages=session.messages, **args)
+            response = self.client.chat.completions.create(messages=session.messages, **args)
+            # logger.debug("[ZHIPU_AI] response={}".format(response))
+            # logger.info("[ZHIPU_AI] reply={}, total_tokens={}".format(response.choices[0]['message']['content'], response["usage"]["total_tokens"]))
+
+            return {
+                "total_tokens": response.usage.total_tokens,
+                "completion_tokens": response.usage.completion_tokens,
+                "content": response.choices[0].message.content,
+            }
+        except Exception as e:
+            need_retry = retry_count < 2
+            result = {"completion_tokens": 0, "content": "我现在有点累了，等会再来吧"}
+            if isinstance(e, openai.error.RateLimitError):
+                logger.warn("[ZHIPU_AI] RateLimitError: {}".format(e))
+                result["content"] = "提问太快啦，请休息一下再问我吧"
+                if need_retry:
+                    time.sleep(20)
+            elif isinstance(e, openai.error.Timeout):
+                logger.warn("[ZHIPU_AI] Timeout: {}".format(e))
+                result["content"] = "我没有收到你的消息"
+                if need_retry:
+                    time.sleep(5)
+            elif isinstance(e, openai.error.APIError):
+                logger.warn("[ZHIPU_AI] Bad Gateway: {}".format(e))
+                result["content"] = "请再问我一次"
+                if need_retry:
+                    time.sleep(10)
+            elif isinstance(e, openai.error.APIConnectionError):
+                logger.warn("[ZHIPU_AI] APIConnectionError: {}".format(e))
+                result["content"] = "我连接不到你的网络"
+                if need_retry:
+                    time.sleep(5)
+            else:
+                logger.exception("[ZHIPU_AI] Exception: {}".format(e), e)
+                need_retry = False
+                self.sessions.clear_session(session.session_id)
+
+            if need_retry:
+                logger.warn("[ZHIPU_AI] 第{}次重试".format(retry_count + 1))
+                return self.reply_text(session, api_key, args, retry_count + 1)
+            else:
+                return result

bridge/bridge.py

@@ -18,20 +18,33 @@ class Bridge(object):
             "text_to_voice": conf().get("text_to_voice", "google"),
             "translate": conf().get("translate", "baidu"),
         }
-        model_type = conf().get("model")
+        model_type = conf().get("model") or const.GPT35
         if model_type in ["text-davinci-003"]:
             self.btype["chat"] = const.OPEN_AI
         if conf().get("use_azure_chatgpt", False):
             self.btype["chat"] = const.CHATGPTONAZURE
-        if model_type in ["wenxin"]:
+        if model_type in ["wenxin", "wenxin-4"]:
             self.btype["chat"] = const.BAIDU
         if model_type in ["xunfei"]:
             self.btype["chat"] = const.XUNFEI
+        if model_type in [const.QWEN]:
+            self.btype["chat"] = const.QWEN
+        if model_type in [const.GEMINI]:
+            self.btype["chat"] = const.GEMINI
+        if model_type in [const.ZHIPU_AI]:
+            self.btype["chat"] = const.ZHIPU_AI
+
         if conf().get("use_linkai") and conf().get("linkai_api_key"):
             self.btype["chat"] = const.LINKAI
+            if not conf().get("voice_to_text") or conf().get("voice_to_text") in ["openai"]:
+                self.btype["voice_to_text"] = const.LINKAI
+            if not conf().get("text_to_voice") or conf().get("text_to_voice") in ["openai", const.TTS_1, const.TTS_1_HD]:
+                self.btype["text_to_voice"] = const.LINKAI
+
         if model_type in ["claude"]:
             self.btype["chat"] = const.CLAUDEAI
         self.bots = {}
+        self.chat_bots = {}
 
     def get_bot(self, typename):
         if self.bots.get(typename) is None:
@@ -61,6 +74,11 @@ class Bridge(object):
     def fetch_translate(self, text, from_lang="", to_lang="en") -> Reply:
         return self.get_bot("translate").translate(text, from_lang, to_lang)
 
+    def find_chat_bot(self, bot_type: str):
+        if self.chat_bots.get(bot_type) is None:
+            self.chat_bots[bot_type] = create_bot(bot_type)
+        return self.chat_bots.get(bot_type)
+
     def reset_bot(self):
         """
         重置bot路由

bridge/context.py

@@ -9,11 +9,15 @@ class ContextType(Enum):
     IMAGE = 3  # 图片消息
     FILE = 4  # 文件信息
     VIDEO = 5  # 视频信息
+    SHARING = 6  # 分享信息
 
     IMAGE_CREATE = 10  # 创建图片命令
+    ACCEPT_FRIEND = 19 # 同意好友请求
     JOIN_GROUP = 20  # 加入群聊
     PATPAT = 21  # 拍了拍
     FUNCTION = 22  # 函数调用
+    EXIT_GROUP = 23 #退出
+
 
     def __str__(self):
         return self.name

channel/channel.py

@@ -8,6 +8,7 @@ from bridge.reply import *
 
 
 class Channel(object):
+    channel_type = ""
     NOT_SUPPORT_REPLYTYPE = [ReplyType.VOICE, ReplyType.IMAGE]
 
     def startup(self):

channel/channel_factory.py

@@ -1,40 +1,45 @@
 """
 channel factory
 """
+from common import const
+from .channel import Channel
 
 
-def create_channel(channel_type):
+def create_channel(channel_type) -> Channel:
     """
     create a channel instance
     :param channel_type: channel type code
     :return: channel instance
     """
+    ch = Channel()
     if channel_type == "wx":
         from channel.wechat.wechat_channel import WechatChannel
-
-        return WechatChannel()
+        ch = WechatChannel()
     elif channel_type == "wxy":
         from channel.wechat.wechaty_channel import WechatyChannel
-
-        return WechatyChannel()
+        ch = WechatyChannel()
     elif channel_type == "terminal":
         from channel.terminal.terminal_channel import TerminalChannel
-
-        return TerminalChannel()
+        ch = TerminalChannel()
     elif channel_type == "wechatmp":
         from channel.wechatmp.wechatmp_channel import WechatMPChannel
-
-        return WechatMPChannel(passive_reply=True)
+        ch = WechatMPChannel(passive_reply=True)
     elif channel_type == "wechatmp_service":
         from channel.wechatmp.wechatmp_channel import WechatMPChannel
-
-        return WechatMPChannel(passive_reply=False)
+        ch = WechatMPChannel(passive_reply=False)
     elif channel_type == "wechatcom_app":
         from channel.wechatcom.wechatcomapp_channel import WechatComAppChannel
-
-        return WechatComAppChannel()
+        ch = WechatComAppChannel()
     elif channel_type == "wework":
         from channel.wework.wework_channel import WeworkChannel
-
-        return WeworkChannel()
-    raise RuntimeError
+        ch = WeworkChannel()
+    elif channel_type == const.FEISHU:
+        from channel.feishu.feishu_channel import FeiShuChanel
+        ch = FeiShuChanel()
+    elif channel_type == const.DINGTALK:
+        from channel.dingtalk.dingtalk_channel import DingTalkChanel
+        ch = DingTalkChanel()
+    else:
+        raise RuntimeError
+    ch.channel_type = channel_type
+    return ch

channel/chat_channel.py

@@ -4,13 +4,13 @@ import threading
 import time
 from asyncio import CancelledError
 from concurrent.futures import Future, ThreadPoolExecutor
+from concurrent import futures
 
 from bridge.context import *
 from bridge.reply import *
 from channel.channel import Channel
 from common.dequeue import Dequeue
-from common.log import logger
-from config import conf
+from common import memory
 from plugins import *
 
 try:
@@ -18,6 +18,8 @@ try:
 except Exception as e:
     pass
 
+handler_pool = ThreadPoolExecutor(max_workers=8)  # 处理消息的线程池
+
 
 # 抽象类, 它包含了与消息通道无关的通用处理逻辑
 class ChatChannel(Channel):
@@ -26,7 +28,6 @@ class ChatChannel(Channel):
     futures = {}  # 记录每个session_id提交到线程池的future对象, 用于重置会话时把没执行的future取消掉，正在执行的不会被取消
     sessions = {}  # 用于控制并发，每个session_id同时只能有一个context在处理
     lock = threading.Lock()  # 用于控制对sessions的访问
-    handler_pool = ThreadPoolExecutor(max_workers=8)  # 处理消息的线程池
 
     def __init__(self):
         _thread = threading.Thread(target=self.consume)
@@ -91,9 +92,11 @@ class ChatChannel(Channel):
         # 消息内容匹配过程，并处理content
         if ctype == ContextType.TEXT:
             if first_in and "」\n- - - - - - -" in content:  # 初次匹配 过滤引用消息
+                logger.debug(content)
                 logger.debug("[WX]reference query skipped")
                 return None
 
+            nick_name_black_list = conf().get("nick_name_black_list", [])
             if context.get("isgroup", False):  # 群聊
                 # 校验关键字
                 match_prefix = check_prefix(content, conf().get("group_chat_prefix"))
@@ -105,6 +108,12 @@ class ChatChannel(Channel):
                         if match_prefix:
                             content = content.replace(match_prefix, "", 1).strip()
                     if context["msg"].is_at:
+                        nick_name = context["msg"].actual_user_nickname
+                        if nick_name and nick_name in nick_name_black_list:
+                            # 黑名单过滤
+                            logger.warning(f"[WX] Nickname {nick_name} in In BlackList, ignore")
+                            return None
+
                         logger.info("[WX]receive group at")
                         if not conf().get("group_at_off", False):
                             flag = True
@@ -124,6 +133,12 @@ class ChatChannel(Channel):
                         logger.info("[WX]receive group voice, but checkprefix didn't match")
                     return None
             else:  # 单聊
+                nick_name = context["msg"].from_user_nickname
+                if nick_name and nick_name in nick_name_black_list:
+                    # 黑名单过滤
+                    logger.warning(f"[WX] Nickname '{nick_name}' in In BlackList, ignore")
+                    return None
+
                 match_prefix = check_prefix(content, conf().get("single_chat_prefix", [""]))
                 if match_prefix is not None:  # 判断如果匹配到自定义前缀，则返回过滤掉前缀+空格后的内容
                     content = content.replace(match_prefix, "", 1).strip()
@@ -171,9 +186,8 @@ class ChatChannel(Channel):
         reply = e_context["reply"]
         if not e_context.is_pass():
             logger.debug("[WX] ready to handle context: type={}, content={}".format(context.type, context.content))
-            if e_context.is_break():
-                context["generate_breaked_by"] = e_context["breaked_by"]
             if context.type == ContextType.TEXT or context.type == ContextType.IMAGE_CREATE:  # 文字和图片消息
+                context["channel"] = e_context["channel"]
                 reply = super().build_reply_content(context.content, context)
             elif context.type == ContextType.VOICE:  # 语音消息
                 cmsg = context["msg"]
@@ -203,12 +217,16 @@ class ChatChannel(Channel):
                     else:
                         return
             elif context.type == ContextType.IMAGE:  # 图片消息，当前仅做下载保存到本地的逻辑
-                cmsg = context["msg"]
-                cmsg.prepare()
+                memory.USER_IMAGE_CACHE[context["session_id"]] = {
+                    "path": context.content,
+                    "msg": context.get("msg")
+                }
+            elif context.type == ContextType.SHARING:  # 分享信息，当前无默认逻辑
+                pass
             elif context.type == ContextType.FUNCTION or context.type == ContextType.FILE:  # 文件消息及函数调用等，当前无默认逻辑
                 pass
             else:
-                logger.error("[WX] unknown context type: {}".format(context.type))
+                logger.warning("[WX] unknown context type: {}".format(context.type))
                 return
         return reply
 
@@ -234,14 +252,15 @@ class ChatChannel(Channel):
                         reply = super().build_text_to_voice(reply.content)
                         return self._decorate_reply(context, reply)
                     if context.get("isgroup", False):
-                        reply_text = "@" + context["msg"].actual_user_nickname + "\n" + reply_text.strip()
+                        if not context.get("no_need_at", False):
+                            reply_text = "@" + context["msg"].actual_user_nickname + "\n" + reply_text.strip()
                         reply_text = conf().get("group_chat_reply_prefix", "") + reply_text + conf().get("group_chat_reply_suffix", "")
                     else:
                         reply_text = conf().get("single_chat_reply_prefix", "") + reply_text + conf().get("single_chat_reply_suffix", "")
                     reply.content = reply_text
                 elif reply.type == ReplyType.ERROR or reply.type == ReplyType.INFO:
                     reply.content = "[" + str(reply.type) + "]\n" + reply.content
-                elif reply.type == ReplyType.IMAGE_URL or reply.type == ReplyType.VOICE or reply.type == ReplyType.IMAGE:
+                elif reply.type == ReplyType.IMAGE_URL or reply.type == ReplyType.VOICE or reply.type == ReplyType.IMAGE or reply.type == ReplyType.FILE or reply.type == ReplyType.VIDEO or reply.type == ReplyType.VIDEO_URL:
                     pass
                 else:
                     logger.error("[WX] unknown reply type: {}".format(reply.type))
@@ -322,7 +341,7 @@ class ChatChannel(Channel):
                         if not context_queue.empty():
                             context = context_queue.get()
                             logger.debug("[WX] consume context: {}".format(context))
-                            future: Future = self.handler_pool.submit(self._handle, context)
+                            future: Future = handler_pool.submit(self._handle, context)
                             future.add_done_callback(self._thread_pool_callback(session_id, context=context))
                             if session_id not in self.futures:
                                 self.futures[session_id] = []

channel/dingtalk/dingtalk_channel.py

@@ -0,0 +1,100 @@
+"""
+钉钉通道接入
+
+@author huiwen
+@Date 2023/11/28
+"""
+
+# -*- coding=utf-8 -*-
+from channel.dingtalk.dingtalk_message import DingTalkMessage
+from bridge.context import Context
+from bridge.reply import Reply
+from common.log import logger
+from common.singleton import singleton
+from config import conf
+from common.expired_dict import ExpiredDict
+from bridge.context import ContextType
+from channel.chat_channel import ChatChannel
+import logging
+from dingtalk_stream import AckMessage
+import dingtalk_stream
+
+
+@singleton
+class DingTalkChanel(ChatChannel, dingtalk_stream.ChatbotHandler):
+    dingtalk_client_id = conf().get('dingtalk_client_id')
+    dingtalk_client_secret = conf().get('dingtalk_client_secret')
+
+    def setup_logger(self):
+        logger = logging.getLogger()
+        handler = logging.StreamHandler()
+        handler.setFormatter(
+            logging.Formatter('%(asctime)s %(name)-8s %(levelname)-8s %(message)s [%(filename)s:%(lineno)d]'))
+        logger.addHandler(handler)
+        logger.setLevel(logging.INFO)
+        return logger
+
+    def __init__(self):
+        super().__init__()
+        super(dingtalk_stream.ChatbotHandler, self).__init__()
+        self.logger = self.setup_logger()
+        # 历史消息id暂存，用于幂等控制
+        self.receivedMsgs = ExpiredDict(60 * 60 * 7.1)
+        logger.info("[dingtalk] client_id={}, client_secret={} ".format(
+            self.dingtalk_client_id, self.dingtalk_client_secret))
+        # 无需群校验和前缀
+        conf()["group_name_white_list"] = ["ALL_GROUP"]
+
+    def startup(self):
+        credential = dingtalk_stream.Credential(self.dingtalk_client_id, self.dingtalk_client_secret)
+        client = dingtalk_stream.DingTalkStreamClient(credential)
+        client.register_callback_handler(dingtalk_stream.chatbot.ChatbotMessage.TOPIC, self)
+        client.start_forever()
+
+    def handle_single(self, cmsg: DingTalkMessage):
+        # 处理单聊消息
+        if cmsg.ctype == ContextType.VOICE:
+            logger.debug("[dingtalk]receive voice msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.IMAGE:
+            logger.debug("[dingtalk]receive image msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.PATPAT:
+            logger.debug("[dingtalk]receive patpat msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.TEXT:
+            expression = cmsg.my_msg
+        cmsg.content = conf()["single_chat_prefix"][0] + cmsg.content
+        context = self._compose_context(cmsg.ctype, cmsg.content, isgroup=False, msg=cmsg)
+        if context:
+            self.produce(context)
+
+    def handle_group(self, cmsg: DingTalkMessage):
+        # 处理群聊消息
+        if cmsg.ctype == ContextType.VOICE:
+            logger.debug("[dingtalk]receive voice msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.IMAGE:
+            logger.debug("[dingtalk]receive image msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.PATPAT:
+            logger.debug("[dingtalk]receive patpat msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.TEXT:
+            expression = cmsg.my_msg
+        cmsg.content = conf()["group_chat_prefix"][0] + cmsg.content
+        context = self._compose_context(cmsg.ctype, cmsg.content, isgroup=True, msg=cmsg)
+        context['no_need_at'] = True
+        if context:
+            self.produce(context)
+
+    async def process(self, callback: dingtalk_stream.CallbackMessage):
+        try:
+            incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
+            dingtalk_msg = DingTalkMessage(incoming_message)
+            if incoming_message.conversation_type == '1':
+                self.handle_single(dingtalk_msg)
+            else:
+                self.handle_group(dingtalk_msg)
+            return AckMessage.STATUS_OK, 'OK'
+        except Exception as e:
+            logger.error(e)
+            return self.FAILED_MSG
+
+    def send(self, reply: Reply, context: Context):
+        incoming_message = context.kwargs['msg'].incoming_message
+        self.reply_text(reply.content, incoming_message)

channel/dingtalk/dingtalk_message.py

@@ -0,0 +1,44 @@
+from bridge.context import ContextType
+from channel.chat_message import ChatMessage
+import json
+import requests
+from common.log import logger
+from common.tmp_dir import TmpDir
+from common import utils
+from dingtalk_stream import ChatbotMessage
+
+class DingTalkMessage(ChatMessage):
+    def __init__(self, event: ChatbotMessage):
+        super().__init__(event)
+        
+        self.msg_id = event.message_id
+        msg_type = event.message_type
+        self.incoming_message =event
+        self.sender_staff_id = event.sender_staff_id
+        self.other_user_id = event.conversation_id
+        self.create_time = event.create_at
+        if event.conversation_type=="1":
+            self.is_group = False
+        else:
+            self.is_group = True
+        
+
+        if msg_type == "text":
+            self.ctype = ContextType.TEXT
+            
+            self.content = event.text.content.strip()
+        elif msg_type == "audio":
+            
+            # 钉钉支持直接识别语音，所以此处将直接提取文字，当文字处理
+            self.content = event.extensions['content']['recognition'].strip()
+            self.ctype = ContextType.TEXT
+        self.from_user_id = event.sender_id
+        self.to_user_id = event.chatbot_user_id
+        self.other_user_nickname = event.conversation_title
+        
+        user_id = event.sender_id
+        nickname =event.sender_nick
+
+        
+
+      

channel/feishu/feishu_channel.py

@@ -0,0 +1,254 @@
+"""
+飞书通道接入
+
+@author Saboteur7
+@Date 2023/11/19
+"""
+
+# -*- coding=utf-8 -*-
+import uuid
+
+import requests
+import web
+from channel.feishu.feishu_message import FeishuMessage
+from bridge.context import Context
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from common.singleton import singleton
+from config import conf
+from common.expired_dict import ExpiredDict
+from bridge.context import ContextType
+from channel.chat_channel import ChatChannel, check_prefix
+from common import utils
+import json
+import os
+
+URL_VERIFICATION = "url_verification"
+
+
+@singleton
+class FeiShuChanel(ChatChannel):
+    feishu_app_id = conf().get('feishu_app_id')
+    feishu_app_secret = conf().get('feishu_app_secret')
+    feishu_token = conf().get('feishu_token')
+
+    def __init__(self):
+        super().__init__()
+        # 历史消息id暂存，用于幂等控制
+        self.receivedMsgs = ExpiredDict(60 * 60 * 7.1)
+        logger.info("[FeiShu] app_id={}, app_secret={} verification_token={}".format(
+            self.feishu_app_id, self.feishu_app_secret, self.feishu_token))
+        # 无需群校验和前缀
+        conf()["group_name_white_list"] = ["ALL_GROUP"]
+        conf()["single_chat_prefix"] = []
+
+    def startup(self):
+        urls = (
+            '/', 'channel.feishu.feishu_channel.FeishuController'
+        )
+        app = web.application(urls, globals(), autoreload=False)
+        port = conf().get("feishu_port", 9891)
+        web.httpserver.runsimple(app.wsgifunc(), ("0.0.0.0", port))
+
+    def send(self, reply: Reply, context: Context):
+        msg = context.get("msg")
+        is_group = context["isgroup"]
+        if msg:
+            access_token = msg.access_token
+        else:
+            access_token = self.fetch_access_token()
+        headers = {
+            "Authorization": "Bearer " + access_token,
+            "Content-Type": "application/json",
+        }
+        msg_type = "text"
+        logger.info(f"[FeiShu] start send reply message, type={context.type}, content={reply.content}")
+        reply_content = reply.content
+        content_key = "text"
+        if reply.type == ReplyType.IMAGE_URL:
+            # 图片上传
+            reply_content = self._upload_image_url(reply.content, access_token)
+            if not reply_content:
+                logger.warning("[FeiShu] upload file failed")
+                return
+            msg_type = "image"
+            content_key = "image_key"
+        if is_group:
+            # 群聊中直接回复
+            url = f"https://open.feishu.cn/open-apis/im/v1/messages/{msg.msg_id}/reply"
+            data = {
+                "msg_type": msg_type,
+                "content": json.dumps({content_key: reply_content})
+            }
+            res = requests.post(url=url, headers=headers, json=data, timeout=(5, 10))
+        else:
+            url = "https://open.feishu.cn/open-apis/im/v1/messages"
+            params = {"receive_id_type": context.get("receive_id_type") or "open_id"}
+            data = {
+                "receive_id": context.get("receiver"),
+                "msg_type": msg_type,
+                "content": json.dumps({content_key: reply_content})
+            }
+            res = requests.post(url=url, headers=headers, params=params, json=data, timeout=(5, 10))
+        res = res.json()
+        if res.get("code") == 0:
+            logger.info(f"[FeiShu] send message success")
+        else:
+            logger.error(f"[FeiShu] send message failed, code={res.get('code')}, msg={res.get('msg')}")
+
+
+    def fetch_access_token(self) -> str:
+        url = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal/"
+        headers = {
+            "Content-Type": "application/json"
+        }
+        req_body = {
+            "app_id": self.feishu_app_id,
+            "app_secret": self.feishu_app_secret
+        }
+        data = bytes(json.dumps(req_body), encoding='utf8')
+        response = requests.post(url=url, data=data, headers=headers)
+        if response.status_code == 200:
+            res = response.json()
+            if res.get("code") != 0:
+                logger.error(f"[FeiShu] get tenant_access_token error, code={res.get('code')}, msg={res.get('msg')}")
+                return ""
+            else:
+                return res.get("tenant_access_token")
+        else:
+            logger.error(f"[FeiShu] fetch token error, res={response}")
+
+
+    def _upload_image_url(self, img_url, access_token):
+        logger.debug(f"[WX] start download image, img_url={img_url}")
+        response = requests.get(img_url)
+        suffix = utils.get_path_suffix(img_url)
+        temp_name = str(uuid.uuid4()) + "." + suffix
+        if response.status_code == 200:
+            # 将图片内容保存为临时文件
+            with open(temp_name, "wb") as file:
+                file.write(response.content)
+
+        # upload
+        upload_url = "https://open.feishu.cn/open-apis/im/v1/images"
+        data = {
+            'image_type': 'message'
+        }
+        headers = {
+            'Authorization': f'Bearer {access_token}',
+        }
+        with open(temp_name, "rb") as file:
+            upload_response = requests.post(upload_url, files={"image": file}, data=data, headers=headers)
+            logger.info(f"[FeiShu] upload file, res={upload_response.content}")
+            os.remove(temp_name)
+            return upload_response.json().get("data").get("image_key")
+
+
+
+class FeishuController:
+    # 类常量
+    FAILED_MSG = '{"success": false}'
+    SUCCESS_MSG = '{"success": true}'
+    MESSAGE_RECEIVE_TYPE = "im.message.receive_v1"
+
+    def GET(self):
+        return "Feishu service start success!"
+
+    def POST(self):
+        try:
+            channel = FeiShuChanel()
+
+            request = json.loads(web.data().decode("utf-8"))
+            logger.debug(f"[FeiShu] receive request: {request}")
+
+            # 1.事件订阅回调验证
+            if request.get("type") == URL_VERIFICATION:
+                varify_res = {"challenge": request.get("challenge")}
+                return json.dumps(varify_res)
+
+            # 2.消息接收处理
+            # token 校验
+            header = request.get("header")
+            if not header or header.get("token") != channel.feishu_token:
+                return self.FAILED_MSG
+
+            # 处理消息事件
+            event = request.get("event")
+            if header.get("event_type") == self.MESSAGE_RECEIVE_TYPE and event:
+                if not event.get("message") or not event.get("sender"):
+                    logger.warning(f"[FeiShu] invalid message, msg={request}")
+                    return self.FAILED_MSG
+                msg = event.get("message")
+
+                # 幂等判断
+                if channel.receivedMsgs.get(msg.get("message_id")):
+                    logger.warning(f"[FeiShu] repeat msg filtered, event_id={header.get('event_id')}")
+                    return self.SUCCESS_MSG
+                channel.receivedMsgs[msg.get("message_id")] = True
+
+                is_group = False
+                chat_type = msg.get("chat_type")
+                if chat_type == "group":
+                    if not msg.get("mentions") and msg.get("message_type") == "text":
+                        # 群聊中未@不响应
+                        return self.SUCCESS_MSG
+                    if msg.get("mentions")[0].get("name") != conf().get("feishu_bot_name") and msg.get("message_type") == "text":
+                        # 不是@机器人，不响应
+                        return self.SUCCESS_MSG
+                    # 群聊
+                    is_group = True
+                    receive_id_type = "chat_id"
+                elif chat_type == "p2p":
+                    receive_id_type = "open_id"
+                else:
+                    logger.warning("[FeiShu] message ignore")
+                    return self.SUCCESS_MSG
+                # 构造飞书消息对象
+                feishu_msg = FeishuMessage(event, is_group=is_group, access_token=channel.fetch_access_token())
+                if not feishu_msg:
+                    return self.SUCCESS_MSG
+
+                context = self._compose_context(
+                    feishu_msg.ctype,
+                    feishu_msg.content,
+                    isgroup=is_group,
+                    msg=feishu_msg,
+                    receive_id_type=receive_id_type,
+                    no_need_at=True
+                )
+                if context:
+                    channel.produce(context)
+                logger.info(f"[FeiShu] query={feishu_msg.content}, type={feishu_msg.ctype}")
+            return self.SUCCESS_MSG
+
+        except Exception as e:
+            logger.error(e)
+            return self.FAILED_MSG
+
+    def _compose_context(self, ctype: ContextType, content, **kwargs):
+        context = Context(ctype, content)
+        context.kwargs = kwargs
+        if "origin_ctype" not in context:
+            context["origin_ctype"] = ctype
+
+        cmsg = context["msg"]
+        context["session_id"] = cmsg.from_user_id
+        context["receiver"] = cmsg.other_user_id
+
+        if ctype == ContextType.TEXT:
+            # 1.文本请求
+            # 图片生成处理
+            img_match_prefix = check_prefix(content, conf().get("image_create_prefix"))
+            if img_match_prefix:
+                content = content.replace(img_match_prefix, "", 1)
+                context.type = ContextType.IMAGE_CREATE
+            else:
+                context.type = ContextType.TEXT
+            context.content = content.strip()
+
+        elif context.type == ContextType.VOICE:
+            # 2.语音请求
+            if "desire_rtype" not in context and conf().get("voice_reply_voice"):
+                context["desire_rtype"] = ReplyType.VOICE
+
+        return context

channel/feishu/feishu_message.py

@@ -0,0 +1,63 @@
+from bridge.context import ContextType
+from channel.chat_message import ChatMessage
+import json
+import requests
+from common.log import logger
+from common.tmp_dir import TmpDir
+from common import utils
+
+
+class FeishuMessage(ChatMessage):
+    def __init__(self, event: dict, is_group=False, access_token=None):
+        super().__init__(event)
+        msg = event.get("message")
+        sender = event.get("sender")
+        self.access_token = access_token
+        self.msg_id = msg.get("message_id")
+        self.create_time = msg.get("create_time")
+        self.is_group = is_group
+        msg_type = msg.get("message_type")
+
+        if msg_type == "text":
+            self.ctype = ContextType.TEXT
+            content = json.loads(msg.get('content'))
+            self.content = content.get("text").strip()
+        elif msg_type == "file":
+            self.ctype = ContextType.FILE
+            content = json.loads(msg.get("content"))
+            file_key = content.get("file_key")
+            file_name = content.get("file_name")
+
+            self.content = TmpDir().path() + file_key + "." + utils.get_path_suffix(file_name)
+
+            def _download_file():
+                # 如果响应状态码是200，则将响应内容写入本地文件
+                url = f"https://open.feishu.cn/open-apis/im/v1/messages/{self.msg_id}/resources/{file_key}"
+                headers = {
+                    "Authorization": "Bearer " + access_token,
+                }
+                params = {
+                    "type": "file"
+                }
+                response = requests.get(url=url, headers=headers, params=params)
+                if response.status_code == 200:
+                    with open(self.content, "wb") as f:
+                        f.write(response.content)
+                else:
+                    logger.info(f"[FeiShu] Failed to download file, key={file_key}, res={response.text}")
+            self._prepare_fn = _download_file
+        else:
+            raise NotImplementedError("Unsupported message type: Type:{} ".format(msg_type))
+
+        self.from_user_id = sender.get("sender_id").get("open_id")
+        self.to_user_id = event.get("app_id")
+        if is_group:
+            # 群聊
+            self.other_user_id = msg.get("chat_id")
+            self.actual_user_id = self.from_user_id
+            self.content = self.content.replace("@_user_1", "").strip()
+            self.actual_user_nickname = ""
+        else:
+            # 私聊
+            self.other_user_id = self.from_user_id
+            self.actual_user_id = self.from_user_id

channel/wechat/wechat_channel.py

@@ -15,6 +15,7 @@ import requests
 from bridge.context import *
 from bridge.reply import *
 from channel.chat_channel import ChatChannel
+from channel import chat_channel
 from channel.wechat.wechat_message import *
 from common.expired_dict import ExpiredDict
 from common.log import logger
@@ -25,7 +26,7 @@ from lib import itchat
 from lib.itchat.content import *
 
 
-@itchat.msg_register([TEXT, VOICE, PICTURE, NOTE])
+@itchat.msg_register([TEXT, VOICE, PICTURE, NOTE, ATTACHMENT, SHARING])
 def handler_single_msg(msg):
     try:
         cmsg = WechatMessage(msg, False)
@@ -36,7 +37,7 @@ def handler_single_msg(msg):
     return None
 
 
-@itchat.msg_register([TEXT, VOICE, PICTURE, NOTE], isGroupChat=True)
+@itchat.msg_register([TEXT, VOICE, PICTURE, NOTE, ATTACHMENT, SHARING], isGroupChat=True)
 def handler_group_msg(msg):
     try:
         cmsg = WechatMessage(msg, True)
@@ -95,7 +96,7 @@ def qrCallback(uuid, status, qrcode):
         print(qr_api4)
         print(qr_api2)
         print(qr_api1)
-
+        _send_qr_code([qr_api1, qr_api2, qr_api3, qr_api4])
         qr = qrcode.QRCode(border=1)
         qr.add_data(url)
         qr.make(fit=True)
@@ -109,23 +110,46 @@ class WechatChannel(ChatChannel):
     def __init__(self):
         super().__init__()
         self.receivedMsgs = ExpiredDict(60 * 60)
+        self.auto_login_times = 0
 
     def startup(self):
-        itchat.instance.receivingRetryCount = 600  # 修改断线超时时间
-        # login by scan QRCode
-        hotReload = conf().get("hot_reload", False)
-        status_path = os.path.join(get_appdata_dir(), "itchat.pkl")
-        itchat.auto_login(
-            enableCmdQR=2,
-            hotReload=hotReload,
-            statusStorageDir=status_path,
-            qrCallback=qrCallback,
-        )
-        self.user_id = itchat.instance.storageClass.userName
-        self.name = itchat.instance.storageClass.nickName
-        logger.info("Wechat login success, user_id: {}, nickname: {}".format(self.user_id, self.name))
-        # start message listener
-        itchat.run()
+        try:
+            itchat.instance.receivingRetryCount = 600  # 修改断线超时时间
+            # login by scan QRCode
+            hotReload = conf().get("hot_reload", False)
+            status_path = os.path.join(get_appdata_dir(), "itchat.pkl")
+            itchat.auto_login(
+                enableCmdQR=2,
+                hotReload=hotReload,
+                statusStorageDir=status_path,
+                qrCallback=qrCallback,
+                exitCallback=self.exitCallback,
+                loginCallback=self.loginCallback
+            )
+            self.user_id = itchat.instance.storageClass.userName
+            self.name = itchat.instance.storageClass.nickName
+            logger.info("Wechat login success, user_id: {}, nickname: {}".format(self.user_id, self.name))
+            # start message listener
+            itchat.run()
+        except Exception as e:
+            logger.error(e)
+
+    def exitCallback(self):
+        try:
+            from common.linkai_client import chat_client
+            if chat_client.client_id and conf().get("use_linkai"):
+                _send_logout()
+                time.sleep(2)
+                self.auto_login_times += 1
+                if self.auto_login_times < 100:
+                    chat_channel.handler_pool._shutdown = False
+                    self.startup()
+        except Exception as e:
+            pass
+
+    def loginCallback(self):
+        logger.debug("Login success")
+        _send_login_success()
 
     # handle_* 系列函数处理收到的消息后构造Context，然后传入produce函数中处理Context和发送回复
     # Context包含了消息的所有信息，包括以下属性
@@ -138,10 +162,12 @@ class WechatChannel(ChatChannel):
     #        msg: ChatMessage消息对象
     #        origin_ctype: 原始消息类型，语音转文字后，私聊时如果匹配前缀失败，会根据初始消息是否是语音来放宽触发规则
     #        desire_rtype: 希望回复类型，默认是文本回复，设置为ReplyType.VOICE是语音回复
-
     @time_checker
     @_check
     def handle_single(self, cmsg: ChatMessage):
+        # filter system message
+        if cmsg.other_user_id in ["weixin"]:
+            return
         if cmsg.ctype == ContextType.VOICE:
             if conf().get("speech_recognition") != True:
                 return
@@ -167,11 +193,13 @@ class WechatChannel(ChatChannel):
             logger.debug("[WX]receive voice for group msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.IMAGE:
             logger.debug("[WX]receive image for group msg: {}".format(cmsg.content))
-        elif cmsg.ctype in [ContextType.JOIN_GROUP, ContextType.PATPAT]:
+        elif cmsg.ctype in [ContextType.JOIN_GROUP, ContextType.PATPAT, ContextType.ACCEPT_FRIEND, ContextType.EXIT_GROUP]:
             logger.debug("[WX]receive note msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.TEXT:
             # logger.debug("[WX]receive group msg: {}, cmsg={}".format(json.dumps(cmsg._rawmsg, ensure_ascii=False), cmsg))
             pass
+        elif cmsg.ctype == ContextType.FILE:
+            logger.debug(f"[WX]receive attachment msg, file_name={cmsg.content}")
         else:
             logger.debug("[WX]receive group msg: {}".format(cmsg.content))
         context = self._compose_context(cmsg.ctype, cmsg.content, isgroup=True, msg=cmsg)
@@ -208,3 +236,48 @@ class WechatChannel(ChatChannel):
             image_storage.seek(0)
             itchat.send_image(image_storage, toUserName=receiver)
             logger.info("[WX] sendImage, receiver={}".format(receiver))
+        elif reply.type == ReplyType.FILE:  # 新增文件回复类型
+            file_storage = reply.content
+            itchat.send_file(file_storage, toUserName=receiver)
+            logger.info("[WX] sendFile, receiver={}".format(receiver))
+        elif reply.type == ReplyType.VIDEO:  # 新增视频回复类型
+            video_storage = reply.content
+            itchat.send_video(video_storage, toUserName=receiver)
+            logger.info("[WX] sendFile, receiver={}".format(receiver))
+        elif reply.type == ReplyType.VIDEO_URL:  # 新增视频URL回复类型
+            video_url = reply.content
+            logger.debug(f"[WX] start download video, video_url={video_url}")
+            video_res = requests.get(video_url, stream=True)
+            video_storage = io.BytesIO()
+            size = 0
+            for block in video_res.iter_content(1024):
+                size += len(block)
+                video_storage.write(block)
+            logger.info(f"[WX] download video success, size={size}, video_url={video_url}")
+            video_storage.seek(0)
+            itchat.send_video(video_storage, toUserName=receiver)
+            logger.info("[WX] sendVideo url={}, receiver={}".format(video_url, receiver))
+
+def _send_login_success():
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_login_success()
+    except Exception as e:
+        pass
+
+def _send_logout():
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_logout()
+    except Exception as e:
+        pass
+
+def _send_qr_code(qrcode_list: list):
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_qrcode(qrcode_list)
+    except Exception as e:
+        pass

channel/wechat/wechat_message.py

@@ -7,7 +7,6 @@ from common.tmp_dir import TmpDir
 from lib import itchat
 from lib.itchat.content import *
 
-
 class WechatMessage(ChatMessage):
     def __init__(self, itchat_msg, is_group=False):
         super().__init__(itchat_msg)
@@ -28,13 +27,24 @@ class WechatMessage(ChatMessage):
             self._prepare_fn = lambda: itchat_msg.download(self.content)
         elif itchat_msg["Type"] == NOTE and itchat_msg["MsgType"] == 10000:
             if is_group and ("加入群聊" in itchat_msg["Content"] or "加入了群聊" in itchat_msg["Content"]):
-                self.ctype = ContextType.JOIN_GROUP
-                self.content = itchat_msg["Content"]
                 # 这里只能得到nickname， actual_user_id还是机器人的id
                 if "加入了群聊" in itchat_msg["Content"]:
+                    self.ctype = ContextType.JOIN_GROUP
+                    self.content = itchat_msg["Content"]
                     self.actual_user_nickname = re.findall(r"\"(.*?)\"", itchat_msg["Content"])[-1]
                 elif "加入群聊" in itchat_msg["Content"]:
+                    self.ctype = ContextType.JOIN_GROUP
+                    self.content = itchat_msg["Content"]
                     self.actual_user_nickname = re.findall(r"\"(.*?)\"", itchat_msg["Content"])[0]
+
+            elif is_group and ("移出了群聊" in itchat_msg["Content"]):
+                self.ctype = ContextType.EXIT_GROUP
+                self.content = itchat_msg["Content"]
+                self.actual_user_nickname = re.findall(r"\"(.*?)\"", itchat_msg["Content"])[0]
+                    
+            elif "你已添加了" in itchat_msg["Content"]:  #通过好友请求
+                self.ctype = ContextType.ACCEPT_FRIEND
+                self.content = itchat_msg["Content"]
             elif "拍了拍我" in itchat_msg["Content"]:
                 self.ctype = ContextType.PATPAT
                 self.content = itchat_msg["Content"]
@@ -42,6 +52,14 @@ class WechatMessage(ChatMessage):
                     self.actual_user_nickname = re.findall(r"\"(.*?)\"", itchat_msg["Content"])[0]
             else:
                 raise NotImplementedError("Unsupported note message: " + itchat_msg["Content"])
+        elif itchat_msg["Type"] == ATTACHMENT:
+            self.ctype = ContextType.FILE
+            self.content = TmpDir().path() + itchat_msg["FileName"]  # content直接存临时目录路径
+            self._prepare_fn = lambda: itchat_msg.download(self.content)
+        elif itchat_msg["Type"] == SHARING:
+            self.ctype = ContextType.SHARING
+            self.content = itchat_msg.get("Url")
+
         else:
             raise NotImplementedError("Unsupported message type: Type:{} MsgType:{}".format(itchat_msg["Type"], itchat_msg["MsgType"]))
 
@@ -80,5 +98,5 @@ class WechatMessage(ChatMessage):
         if self.is_group:
             self.is_at = itchat_msg["IsAt"]
             self.actual_user_id = itchat_msg["ActualUserName"]
-            if self.ctype not in [ContextType.JOIN_GROUP, ContextType.PATPAT]:
+            if self.ctype not in [ContextType.JOIN_GROUP, ContextType.PATPAT, ContextType.EXIT_GROUP]:
                 self.actual_user_nickname = itchat_msg["ActualNickName"]

channel/wechatmp/passive_reply.py

@@ -49,7 +49,7 @@ class Query:
 
                 # New request
                 if (
-                    from_user not in channel.cache_dict
+                    channel.cache_dict.get(from_user) is None
                     and from_user not in channel.running
                     or content.startswith("#")
                     and message_id not in channel.request_cnt  # insert the godcmd
@@ -131,8 +131,10 @@ class Query:
 
                 # Only one request can access to the cached data
                 try:
-                    (reply_type, reply_content) = channel.cache_dict.pop(from_user)
-                except KeyError:
+                    (reply_type, reply_content) = channel.cache_dict[from_user].pop(0)
+                    if not channel.cache_dict[from_user]:  # If popping the message makes the list empty, delete the user entry from cache
+                        del channel.cache_dict[from_user]
+                except IndexError:
                     return "success"
 
                 if reply_type == "text":
@@ -146,7 +148,7 @@ class Query:
                             max_split=1,
                         )
                         reply_text = splits[0] + continue_text
-                        channel.cache_dict[from_user] = ("text", splits[1])
+                        channel.cache_dict[from_user].append(("text", splits[1]))
 
                     logger.info(
                         "[wechatmp] Request {} do send to {} {}: {}\n{}".format(

channel/wechatmp/wechatmp_channel.py

@@ -10,6 +10,7 @@ import requests
 import web
 from wechatpy.crypto import WeChatCrypto
 from wechatpy.exceptions import WeChatClientException
+from collections import defaultdict
 
 from bridge.context import *
 from bridge.reply import *
@@ -46,7 +47,7 @@ class WechatMPChannel(ChatChannel):
             self.crypto = WeChatCrypto(token, aes_key, appid)
         if self.passive_reply:
             # Cache the reply to the user's first message
-            self.cache_dict = dict()
+            self.cache_dict = defaultdict(list)
             # Record whether the current message is being processed
             self.running = set()
             # Count the request from wechat official server by message_id
@@ -82,24 +83,28 @@ class WechatMPChannel(ChatChannel):
             if reply.type == ReplyType.TEXT or reply.type == ReplyType.INFO or reply.type == ReplyType.ERROR:
                 reply_text = reply.content
                 logger.info("[wechatmp] text cached, receiver {}\n{}".format(receiver, reply_text))
-                self.cache_dict[receiver] = ("text", reply_text)
+                self.cache_dict[receiver].append(("text", reply_text))
             elif reply.type == ReplyType.VOICE:
-                try:
-                    voice_file_path = reply.content
-                    with open(voice_file_path, "rb") as f:
-                        # support: <2M, <60s, mp3/wma/wav/amr
-                        response = self.client.material.add("voice", f)
-                        logger.debug("[wechatmp] upload voice response: {}".format(response))
-                        # 根据文件大小估计一个微信自动审核的时间，审核结束前返回将会导致语音无法播放，这个估计有待验证
-                        f_size = os.fstat(f.fileno()).st_size
-                        time.sleep(1.0 + 2 * f_size / 1024 / 1024)
-                        # todo check media_id
-                except WeChatClientException as e:
-                    logger.error("[wechatmp] upload voice failed: {}".format(e))
-                    return
-                media_id = response["media_id"]
-                logger.info("[wechatmp] voice uploaded, receiver {}, media_id {}".format(receiver, media_id))
-                self.cache_dict[receiver] = ("voice", media_id)
+                voice_file_path = reply.content
+                duration, files = split_audio(voice_file_path, 60 * 1000)
+                if len(files) > 1:
+                    logger.info("[wechatmp] voice too long {}s > 60s , split into {} parts".format(duration / 1000.0, len(files)))
+
+                for path in files:
+                    # support: <2M, <60s, mp3/wma/wav/amr
+                    try:
+                        with open(path, "rb") as f:
+                            response = self.client.material.add("voice", f)
+                            logger.debug("[wechatmp] upload voice response: {}".format(response))
+                            f_size = os.fstat(f.fileno()).st_size
+                            time.sleep(1.0 + 2 * f_size / 1024 / 1024)
+                            # todo check media_id
+                    except WeChatClientException as e:
+                        logger.error("[wechatmp] upload voice failed: {}".format(e))
+                        return
+                    media_id = response["media_id"]
+                    logger.info("[wechatmp] voice uploaded, receiver {}, media_id {}".format(receiver, media_id))
+                    self.cache_dict[receiver].append(("voice", media_id))
 
             elif reply.type == ReplyType.IMAGE_URL:  # 从网络下载图片
                 img_url = reply.content
@@ -119,7 +124,7 @@ class WechatMPChannel(ChatChannel):
                     return
                 media_id = response["media_id"]
                 logger.info("[wechatmp] image uploaded, receiver {}, media_id {}".format(receiver, media_id))
-                self.cache_dict[receiver] = ("image", media_id)
+                self.cache_dict[receiver].append(("image", media_id))
             elif reply.type == ReplyType.IMAGE:  # 从文件读取图片
                 image_storage = reply.content
                 image_storage.seek(0)
@@ -134,7 +139,7 @@ class WechatMPChannel(ChatChannel):
                     return
                 media_id = response["media_id"]
                 logger.info("[wechatmp] image uploaded, receiver {}, media_id {}".format(receiver, media_id))
-                self.cache_dict[receiver] = ("image", media_id)
+                self.cache_dict[receiver].append(("image", media_id))
         else:
             if reply.type == ReplyType.TEXT or reply.type == ReplyType.INFO or reply.type == ReplyType.ERROR:
                 reply_text = reply.content

channel/wework/wework_channel.py

@@ -120,7 +120,7 @@ def _check(func):
 
 
 @wework.msg_register(
-    [ntwork.MT_RECV_TEXT_MSG, ntwork.MT_RECV_IMAGE_MSG, 11072, ntwork.MT_RECV_VOICE_MSG])
+    [ntwork.MT_RECV_TEXT_MSG, ntwork.MT_RECV_IMAGE_MSG, 11072, ntwork.MT_RECV_LINK_CARD_MSG,ntwork.MT_RECV_FILE_MSG, ntwork.MT_RECV_VOICE_MSG])
 def all_msg_handler(wework_instance: ntwork.WeWork, message):
     logger.debug(f"收到消息: {message}")
     if 'data' in message:

channel/wework/wework_message.py

@@ -8,6 +8,7 @@ import pilk
 from bridge.context import ContextType
 from channel.chat_message import ChatMessage
 from common.log import logger
+from ntwork.const import send_type
 
 
 def get_with_retry(get_func, max_retries=5, delay=5):
@@ -41,15 +42,40 @@ def cdn_download(wework, message, file_name):
     data = message["data"]
     aes_key = data["cdn"]["aes_key"]
     file_size = data["cdn"]["size"]
-    file_type = 2
-    file_id = data["cdn"]["file_id"]
 
     # 获取当前工作目录，然后与文件名拼接得到保存路径
     current_dir = os.getcwd()
     save_path = os.path.join(current_dir, "tmp", file_name)
 
-    result = wework.c2c_cdn_download(file_id, aes_key, file_size, file_type, save_path)
-    logger.debug(result)
+    # 下载保存图片到本地
+    if "url" in data["cdn"].keys() and "auth_key" in data["cdn"].keys():
+        url = data["cdn"]["url"]
+        auth_key = data["cdn"]["auth_key"]
+        # result = wework.wx_cdn_download(url, auth_key, aes_key, file_size, save_path)  # ntwork库本身接口有问题，缺失了aes_key这个参数
+        """
+        下载wx类型的cdn文件，以https开头
+        """
+        data = {
+            'url': url,
+            'auth_key': auth_key,
+            'aes_key': aes_key,
+            'size': file_size,
+            'save_path': save_path
+        }
+        result = wework._WeWork__send_sync(send_type.MT_WXCDN_DOWNLOAD_MSG, data)  # 直接用wx_cdn_download的接口内部实现来调用
+    elif "file_id" in data["cdn"].keys():
+        if message["type"] == 11042:
+            file_type = 2
+        elif message["type"] == 11045:
+            file_type = 5
+        file_id = data["cdn"]["file_id"]
+        result = wework.c2c_cdn_download(file_id, aes_key, file_size, file_type, save_path)
+    else:
+        logger.error(f"something is wrong, data: {data}")
+        return
+
+    # 输出下载结果
+    logger.debug(f"result: {result}")
 
 
 def c2c_download_and_convert(wework, message, file_name):
@@ -105,6 +131,18 @@ class WeworkMessage(ChatMessage):
                 self.ctype = ContextType.IMAGE
                 self.content = os.path.join(current_dir, "tmp", file_name)
                 self._prepare_fn = lambda: cdn_download(wework, wework_msg, file_name)
+            elif wework_msg["type"] == 11045:  # 文件消息
+                print("文件消息")
+                print(wework_msg)
+                file_name = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+                file_name = file_name + wework_msg['data']['cdn']['file_name']
+                current_dir = os.getcwd()
+                self.ctype = ContextType.FILE
+                self.content = os.path.join(current_dir, "tmp", file_name)
+                self._prepare_fn = lambda: cdn_download(wework, wework_msg, file_name)
+            elif wework_msg["type"] == 11047:  # 链接消息
+                self.ctype = ContextType.SHARING
+                self.content = wework_msg['data']['url']
             elif wework_msg["type"] == 11072:  # 新成员入群通知
                 self.ctype = ContextType.JOIN_GROUP
                 member_list = wework_msg['data']['member_list']
@@ -156,6 +194,7 @@ class WeworkMessage(ChatMessage):
                 if conversation_id:
                     room_info = get_room_info(wework=wework, conversation_id=conversation_id)
                     self.other_user_nickname = room_info.get('nickname', None) if room_info else None
+                    self.from_user_nickname = room_info.get('nickname', None) if room_info else None
                     at_list = data.get('at_list', [])
                     tmp_list = []
                     for at in at_list:

common/const.py

@@ -5,8 +5,24 @@ BAIDU = "baidu"
 XUNFEI = "xunfei"
 CHATGPTONAZURE = "chatGPTOnAzure"
 LINKAI = "linkai"
-
-VERSION = "1.3.0"
-
 CLAUDEAI = "claude"
-MODEL_LIST = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4", "wenxin", "xunfei","claude"]
+QWEN = "qwen"
+GEMINI = "gemini"
+ZHIPU_AI = "glm-4"
+
+
+# model
+GPT35 = "gpt-3.5-turbo"
+GPT4 = "gpt-4"
+GPT4_TURBO_PREVIEW = "gpt-4-0125-preview"
+GPT4_VISION_PREVIEW = "gpt-4-vision-preview"
+WHISPER_1 = "whisper-1"
+TTS_1 = "tts-1"
+TTS_1_HD = "tts-1-hd"
+
+MODEL_LIST = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4", "wenxin", "wenxin-4", "xunfei", "claude", "gpt-4-turbo",
+              "gpt-4-turbo-preview", "gpt-4-1106-preview", GPT4_TURBO_PREVIEW, QWEN, GEMINI, ZHIPU_AI]
+
+# channel
+FEISHU = "feishu"
+DINGTALK = "dingtalk"   

common/linkai_client.py

@@ -0,0 +1,55 @@
+from bridge.context import Context, ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from linkai import LinkAIClient, PushMsg
+from config import conf, pconf, plugin_config
+from plugins import PluginManager
+
+
+chat_client: LinkAIClient
+
+class ChatClient(LinkAIClient):
+    def __init__(self, api_key, host, channel):
+        super().__init__(api_key, host)
+        self.channel = channel
+        self.client_type = channel.channel_type
+
+    def on_message(self, push_msg: PushMsg):
+        session_id = push_msg.session_id
+        msg_content = push_msg.msg_content
+        logger.info(f"receive msg push, session_id={session_id}, msg_content={msg_content}")
+        context = Context()
+        context.type = ContextType.TEXT
+        context["receiver"] = session_id
+        context["isgroup"] = push_msg.is_group
+        self.channel.send(Reply(ReplyType.TEXT, content=msg_content), context)
+
+    def on_config(self, config: dict):
+        if not self.client_id:
+            return
+        logger.info(f"从控制台加载配置: {config}")
+        local_config = conf()
+        for key in local_config.keys():
+            if config.get(key) is not None:
+                local_config[key] = config.get(key)
+        if config.get("reply_voice_mode"):
+            if config.get("reply_voice_mode") == "voice_reply_voice":
+                local_config["voice_reply_voice"] = True
+            elif config.get("reply_voice_mode") == "always_reply_voice":
+                local_config["always_reply_voice"] = True
+        # if config.get("admin_password") and plugin_config["Godcmd"]:
+        #     plugin_config["Godcmd"]["password"] = config.get("admin_password")
+        #     PluginManager().instances["Godcmd"].reload()
+        # if config.get("group_app_map") and pconf("linkai"):
+        #     local_group_map = {}
+        #     for mapping in config.get("group_app_map"):
+        #         local_group_map[mapping.get("group_name")] = mapping.get("app_code")
+        #     pconf("linkai")["group_app_map"] = local_group_map
+        #     PluginManager().instances["linkai"].reload()
+
+
+def start(channel):
+    global chat_client
+    chat_client = ChatClient(api_key=conf().get("linkai_api_key"),
+                        host="link-ai.chat", channel=channel)
+    chat_client.start()

common/memory.py

@@ -0,0 +1,3 @@
+from common.expired_dict import ExpiredDict
+
+USER_IMAGE_CACHE = ExpiredDict(60 * 3)

common/utils.py

@@ -1,6 +1,6 @@
 import io
 import os
-
+from urllib.parse import urlparse
 from PIL import Image
 
 
@@ -49,3 +49,8 @@ def split_string_by_utf8_length(string, max_length, max_split=0):
         result.append(encoded[start:end].decode("utf-8"))
         start = end
     return result
+
+
+def get_path_suffix(path):
+    path = urlparse(path).path
+    return os.path.splitext(path)[-1].lstrip('.')

config-template.json

@@ -1,7 +1,10 @@
 {
-  "open_ai_api_key": "YOUR API KEY",
-  "model": "gpt-3.5-turbo",
   "channel_type": "wx",
+  "model": "",
+  "open_ai_api_key": "YOUR API KEY",
+  "text_to_image": "dall-e-2",
+  "voice_to_text": "openai",
+  "text_to_voice": "openai",
   "proxy": "",
   "hot_reload": false,
   "single_chat_prefix": [
@@ -16,23 +19,17 @@
     "ChatGPT测试群",
     "ChatGPT测试群2"
   ],
-  "group_chat_in_one_session": [
-    "ChatGPT测试群"
-  ],
   "image_create_prefix": [
-    "画",
-    "看",
-    "找"
+    "画"
   ],
-  "speech_recognition": false,
+  "speech_recognition": true,
   "group_speech_recognition": false,
   "voice_reply_voice": false,
-  "conversation_max_tokens": 1000,
+  "conversation_max_tokens": 2500,
   "expires_in_seconds": 3600,
   "character_desc": "你是ChatGPT, 一个由OpenAI训练的大型语言模型, 你旨在回答并解决人们的任何问题，并且可以使用多种语言与人交流。",
   "temperature": 0.7,
-  "top_p": 1,
-  "subscribe_msg": "感谢您的关注！\n这里是ChatGPT，可以自由对话。\n支持语音对话。\n支持图片输入。\n支持图片输出，画字开头的消息将按要求创作图片。\n支持tool、角色扮演和文字冒险等丰富的插件。\n输入{trigger_prefix}#help 查看详细指令。",
+  "subscribe_msg": "感谢您的关注！\n这里是AI智能助手，可以自由对话。\n支持语音对话。\n支持图片输入。\n支持图片输出，画字开头的消息将按要求创作图片。\n支持tool、角色扮演和文字冒险等丰富的插件。\n输入{trigger_prefix}#help 查看详细指令。",
   "use_linkai": false,
   "linkai_api_key": "",
   "linkai_app_code": ""

config.py

@@ -16,7 +16,7 @@ available_setting = {
     "open_ai_api_base": "https://api.openai.com/v1",
     "proxy": "",  # openai使用的代理
     # chatgpt模型， 当use_azure_chatgpt为true时，其名称为Azure上model deployment名称
-    "model": "gpt-3.5-turbo",  # 还支持 gpt-3.5-turbo-16k, gpt-4, wenxin, xunfei
+    "model": "gpt-3.5-turbo",  # 还支持 gpt-4, gpt-4-turbo, wenxin, xunfei, qwen
     "use_azure_chatgpt": False,  # 是否使用azure的chatgpt
     "azure_deployment_id": "",  # azure 模型部署名称
     "azure_api_version": "",  # azure api版本
@@ -32,10 +32,15 @@ available_setting = {
     "group_name_white_list": ["ChatGPT测试群", "ChatGPT测试群2"],  # 开启自动回复的群名称列表
     "group_name_keyword_white_list": [],  # 开启自动回复的群名称关键词列表
     "group_chat_in_one_session": ["ChatGPT测试群"],  # 支持会话上下文共享的群名称
+    "nick_name_black_list": [],  # 用户昵称黑名单
+    "group_welcome_msg": "",  # 配置新人进群固定欢迎语，不配置则使用随机风格欢迎 
     "trigger_by_self": False,  # 是否允许机器人触发
+    "text_to_image": "dall-e-2",  # 图片生成模型，可选 dall-e-2, dall-e-3
+    "image_proxy": True,  # 是否需要图片代理，国内访问LinkAI时需要
     "image_create_prefix": ["画", "看", "找"],  # 开启图片回复的前缀
     "concurrency_in_session": 1,  # 同一会话最多有多少条消息在处理中，大于1可能乱序
-    "image_create_size": "256x256",  # 图片大小,可选有 256x256, 512x512, 1024x1024
+    "image_create_size": "256x256",  # 图片大小,可选有 256x256, 512x512, 1024x1024 (dall-e-3默认为1024x1024)
+    "group_chat_exit_group": False, 
     # chatgpt会话参数
     "expires_in_seconds": 3600,  # 无操作会话的过期时间
     # 人格描述
@@ -49,7 +54,7 @@ available_setting = {
     "top_p": 1,
     "frequency_penalty": 0,
     "presence_penalty": 0,
-    "request_timeout": 60,  # chatgpt请求超时时间，openai接口默认设置为600，对于难问题一般需要较长时间
+    "request_timeout": 180,  # chatgpt请求超时时间，openai接口默认设置为600，对于难问题一般需要较长时间
     "timeout": 120,  # chatgpt重试超时时间，在这个时间内，将会自动重试
     # Baidu 文心一言参数
     "baidu_wenxin_model": "eb-instant",  # 默认使用ERNIE-Bot-turbo模型
@@ -62,15 +67,25 @@ available_setting = {
     # claude 配置
     "claude_api_cookie": "",
     "claude_uuid": "",
+    # 通义千问API, 获取方式查看文档 https://help.aliyun.com/document_detail/2587494.html
+    "qwen_access_key_id": "",
+    "qwen_access_key_secret": "",
+    "qwen_agent_key": "",
+    "qwen_app_id": "",
+    "qwen_node_id": "",  # 流程编排模型用到的id，如果没有用到qwen_node_id，请务必保持为空字符串
+    # Google Gemini Api Key
+    "gemini_api_key": "",
     # wework的通用配置
     "wework_smart": True,  # 配置wework是否使用已登录的企业微信，False为多开
     # 语音设置
-    "speech_recognition": False,  # 是否开启语音识别
+    "speech_recognition": True,  # 是否开启语音识别
     "group_speech_recognition": False,  # 是否开启群组语音识别
     "voice_reply_voice": False,  # 是否使用语音回复语音，需要设置对应语音合成引擎的api key
     "always_reply_voice": False,  # 是否一直使用语音回复
     "voice_to_text": "openai",  # 语音识别引擎，支持openai,baidu,google,azure
-    "text_to_voice": "baidu",  # 语音合成引擎，支持baidu,google,pytts(offline),azure,elevenlabs
+    "text_to_voice": "openai",  # 语音合成引擎，支持openai,baidu,google,pytts(offline),azure,elevenlabs
+    "text_to_voice_model": "tts-1",
+    "tts_voice_id": "alloy",
     # baidu 语音api配置， 使用百度语音识别和语音合成时需要
     "baidu_app_id": "",
     "baidu_api_key": "",
@@ -110,6 +125,18 @@ available_setting = {
     "wechatcomapp_secret": "",  # 企业微信app的secret
     "wechatcomapp_agent_id": "",  # 企业微信app的agent_id
     "wechatcomapp_aes_key": "",  # 企业微信app的aes_key
+
+    # 飞书配置
+    "feishu_port": 80,  # 飞书bot监听端口
+    "feishu_app_id": "",  # 飞书机器人应用APP Id
+    "feishu_app_secret": "",  # 飞书机器人APP secret
+    "feishu_token": "",  # 飞书 verification token
+    "feishu_bot_name": "",  # 飞书机器人的名字
+    
+    # 钉钉配置
+    "dingtalk_client_id": "",  # 钉钉机器人Client ID 
+    "dingtalk_client_secret": "",  # 钉钉机器人Client Secret 
+    
     # chatgpt指令自定义触发词
     "clear_memory_commands": ["#清除记忆"],  # 重置会话指令，必须以#开头
     # channel配置
@@ -121,7 +148,12 @@ available_setting = {
     "plugin_trigger_prefix": "$",  # 规范插件提供聊天相关指令的前缀，建议不要和管理员指令前缀"#"冲突
     # 是否使用全局插件配置
     "use_global_plugin_config": False,
-    # 知识库平台配置
+    "max_media_send_count": 3,     # 单次最大发送媒体资源的个数
+    "media_send_interval": 1,  # 发送图片的事件间隔，单位秒
+    # 智谱AI 平台配置
+    "zhipu_ai_api_key": "",
+    "zhipu_ai_api_base": "https://open.bigmodel.cn/api/paas/v4",
+    # LinkAI平台配置
     "use_linkai": False,
     "linkai_api_key": "",
     "linkai_app_code": "",

lib/itchat/LICENSE

@@ -0,0 +1,9 @@
+**The MIT License (MIT)**
+
+Copyright (c) 2017 LittleCoder ([littlecodersh@Github](https://github.com/littlecodersh))
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

plugins/config.json.template

@@ -33,6 +33,12 @@
             "max_tasks": 3,
             "max_tasks_per_user": 1,
             "use_image_create_prefix": true
+        },
+        "summary": {
+            "enabled": true,
+            "group_enabled": true,
+            "max_file_size": 5000,
+            "type": ["FILE", "SHARING"]
         }
     }
 }

plugins/godcmd/godcmd.py

@@ -4,8 +4,10 @@ import json
 import os
 import random
 import string
+import logging
 from typing import Tuple
 
+import bridge.bridge
 import plugins
 from bridge.bridge import Bridge
 from bridge.context import ContextType
@@ -134,9 +136,9 @@ ADMIN_COMMANDS = {
 
 # 定义帮助函数
 def get_help_text(isadmin, isgroup):
-    help_text = "通用指令：\n"
+    help_text = "通用指令\n"
     for cmd, info in COMMANDS.items():
-        if cmd == "auth":  # 不提示认证指令
+        if cmd in ["auth", "set_openai_api_key", "reset_openai_api_key", "set_gpt_model", "reset_gpt_model", "gpt_model"]:  # 不显示帮助指令
             continue
         if cmd == "id" and conf().get("channel_type", "wx") not in ["wxy", "wechatmp"]:
             continue
@@ -149,7 +151,7 @@ def get_help_text(isadmin, isgroup):
 
     # 插件指令
     plugins = PluginManager().list_plugins()
-    help_text += "\n目前可用插件有："
+    help_text += "\n可用插件"
     for plugin in plugins:
         if plugins[plugin].enabled and not plugins[plugin].hidden:
             namecn = plugins[plugin].namecn
@@ -201,6 +203,7 @@ class Godcmd(Plugin):
 
         self.password = gconf["password"]
         self.admin_users = gconf["admin_users"]  # 预存的管理员账号，这些账号不需要认证。itchat的用户名每次都会变，不可用
+        global_config["admin_users"] = self.admin_users
         self.isrunning = True  # 机器人是否运行中
 
         self.handlers[Event.ON_HANDLE_CONTEXT] = self.on_handle_context
@@ -263,14 +266,16 @@ class Godcmd(Plugin):
                     if not isadmin and not self.is_admin_in_group(e_context["context"]):
                         ok, result = False, "需要管理员权限执行"
                     elif len(args) == 0:
-                        ok, result = True, "当前模型为: " + str(conf().get("model"))
+                        model = conf().get("model") or const.GPT35
+                        ok, result = True, "当前模型为: " + str(model)
                     elif len(args) == 1:
                         if args[0] not in const.MODEL_LIST:
                             ok, result = False, "模型名称不存在"
                         else:
-                            conf()["model"] = args[0]
+                            conf()["model"] = self.model_mapping(args[0])
                             Bridge().reset_bot()
-                            ok, result = True, "模型设置为: " + str(conf().get("model"))
+                            model = conf().get("model") or const.GPT35
+                            ok, result = True, "模型设置为: " + str(model)
                 elif cmd == "id":
                     ok, result = True, user
                 elif cmd == "set_openai_api_key":
@@ -308,8 +313,10 @@ class Godcmd(Plugin):
                     except Exception as e:
                         ok, result = False, "你没有设置私有GPT模型"
                 elif cmd == "reset":
-                    if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI, const.BAIDU, const.XUNFEI]:
+                    if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI, const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI]:
                         bot.sessions.clear_session(session_id)
+                        if Bridge().chat_bots.get(bottype):
+                            Bridge().chat_bots.get(bottype).sessions.clear_session(session_id)
                         channel.cancel_session(session_id)
                         ok, result = True, "会话已重置"
                     else:
@@ -332,15 +339,19 @@ class Godcmd(Plugin):
                             ok, result = True, "配置已重载"
                         elif cmd == "resetall":
                             if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI,
-                                           const.BAIDU, const.XUNFEI]:
+                                           const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI]:
                                 channel.cancel_all_session()
                                 bot.sessions.clear_all_session()
                                 ok, result = True, "重置所有会话成功"
                             else:
                                 ok, result = False, "当前对话机器人不支持重置会话"
                         elif cmd == "debug":
-                            logger.setLevel("DEBUG")
-                            ok, result = True, "DEBUG模式已开启"
+                            if logger.getEffectiveLevel() == logging.DEBUG:  # 判断当前日志模式是否DEBUG
+                                logger.setLevel(logging.INFO)
+                                ok, result = True, "DEBUG模式已关闭"
+                            else:
+                                logger.setLevel(logging.DEBUG)
+                                ok, result = True, "DEBUG模式已开启"
                         elif cmd == "plist":
                             plugins = PluginManager().list_plugins()
                             ok = True
@@ -458,3 +469,17 @@ class Godcmd(Plugin):
         if context["isgroup"]:
             return context.kwargs.get("msg").actual_user_id in global_config["admin_users"]
         return False
+
+
+    def model_mapping(self, model) -> str:
+        if model == "gpt-4-turbo":
+            return const.GPT4_TURBO_PREVIEW
+        return model
+
+    def reload(self):
+        gconf = plugin_config[self.name]
+        if gconf:
+            if gconf.get("password"):
+                self.password = gconf["password"]
+            if gconf.get("admin_users"):
+                self.admin_users = gconf["admin_users"]

plugins/hello/hello.py

@@ -6,6 +6,7 @@ from bridge.reply import Reply, ReplyType
 from channel.chat_message import ChatMessage
 from common.log import logger
 from plugins import *
+from config import conf
 
 
 @plugins.register(
@@ -21,27 +22,49 @@ class Hello(Plugin):
         super().__init__()
         self.handlers[Event.ON_HANDLE_CONTEXT] = self.on_handle_context
         logger.info("[Hello] inited")
+        self.config = super().load_config()
 
     def on_handle_context(self, e_context: EventContext):
         if e_context["context"].type not in [
             ContextType.TEXT,
             ContextType.JOIN_GROUP,
             ContextType.PATPAT,
+            ContextType.EXIT_GROUP
         ]:
             return
-
         if e_context["context"].type == ContextType.JOIN_GROUP:
+            if "group_welcome_msg" in conf():
+                reply = Reply()
+                reply.type = ReplyType.TEXT
+                reply.content = conf().get("group_welcome_msg", "")
+                e_context["reply"] = reply
+                e_context.action = EventAction.BREAK_PASS  # 事件结束，并跳过处理context的默认逻辑
+                return
             e_context["context"].type = ContextType.TEXT
             msg: ChatMessage = e_context["context"]["msg"]
             e_context["context"].content = f'请你随机使用一种风格说一句问候语来欢迎新用户"{msg.actual_user_nickname}"加入群聊。'
             e_context.action = EventAction.BREAK  # 事件结束，进入默认处理逻辑
+            if not self.config or not self.config.get("use_character_desc"):
+                e_context["context"]["generate_breaked_by"] = EventAction.BREAK
             return
-
+        
+        if e_context["context"].type == ContextType.EXIT_GROUP:
+            if conf().get("group_chat_exit_group"):
+                e_context["context"].type = ContextType.TEXT
+                msg: ChatMessage = e_context["context"]["msg"]
+                e_context["context"].content = f'请你随机使用一种风格跟其他群用户说他违反规则"{msg.actual_user_nickname}"退出群聊。'
+                e_context.action = EventAction.BREAK  # 事件结束，进入默认处理逻辑
+                return
+            e_context.action = EventAction.BREAK
+            return
+            
         if e_context["context"].type == ContextType.PATPAT:
             e_context["context"].type = ContextType.TEXT
             msg: ChatMessage = e_context["context"]["msg"]
             e_context["context"].content = f"请你随机使用一种风格介绍你自己，并告诉用户输入#help可以查看帮助信息。"
             e_context.action = EventAction.BREAK  # 事件结束，进入默认处理逻辑
+            if not self.config or not self.config.get("use_character_desc"):
+                e_context["context"]["generate_breaked_by"] = EventAction.BREAK
             return
 
         content = e_context["context"].content

plugins/keyword/keyword.py

@@ -2,7 +2,7 @@
 
 import json
 import os
-
+import requests
 import plugins
 from bridge.context import ContextType
 from bridge.reply import Reply, ReplyType
@@ -51,15 +51,37 @@ class Keyword(Plugin):
         content = e_context["context"].content.strip()
         logger.debug("[keyword] on_handle_context. content: %s" % content)
         if content in self.keyword:
-            logger.debug(f"[keyword] 匹配到关键字【{content}】")
+            logger.info(f"[keyword] 匹配到关键字【{content}】")
             reply_text = self.keyword[content]
 
             # 判断匹配内容的类型
-            if (reply_text.startswith("http://") or reply_text.startswith("https://")) and any(reply_text.endswith(ext) for ext in [".jpg", ".jpeg", ".png", ".gif", ".webp"]):
-                # 如果是以 http:// 或 https:// 开头，且.jpg/.jpeg/.png/.gif结尾，则认为是图片 URL
+            if (reply_text.startswith("http://") or reply_text.startswith("https://")) and any(reply_text.endswith(ext) for ext in [".jpg", ".jpeg", ".png", ".gif", ".img"]):
+            # 如果是以 http:// 或 https:// 开头，且".jpg", ".jpeg", ".png", ".gif", ".img"结尾，则认为是图片 URL。
                 reply = Reply()
                 reply.type = ReplyType.IMAGE_URL
                 reply.content = reply_text
+                
+            elif (reply_text.startswith("http://") or reply_text.startswith("https://")) and any(reply_text.endswith(ext) for ext in [".pdf", ".doc", ".docx", ".xls", "xlsx",".zip", ".rar"]):
+            # 如果是以 http:// 或 https:// 开头，且".pdf", ".doc", ".docx", ".xls", "xlsx",".zip", ".rar"结尾，则下载文件到tmp目录并发送给用户
+                file_path = "tmp"
+                if not os.path.exists(file_path):
+                    os.makedirs(file_path)
+                file_name = reply_text.split("/")[-1]  # 获取文件名
+                file_path = os.path.join(file_path, file_name)
+                response = requests.get(reply_text)
+                with open(file_path, "wb") as f:
+                    f.write(response.content)
+                #channel/wechat/wechat_channel.py和channel/wechat_channel.py中缺少ReplyType.FILE类型。
+                reply = Reply()
+                reply.type = ReplyType.FILE
+                reply.content = file_path
+            
+            elif (reply_text.startswith("http://") or reply_text.startswith("https://")) and any(reply_text.endswith(ext) for ext in [".mp4"]):
+            # 如果是以 http:// 或 https:// 开头，且".mp4"结尾，则下载视频到tmp目录并发送给用户
+                reply = Reply()
+                reply.type = ReplyType.VIDEO_URL
+                reply.content = reply_text
+                
             else:
             # 否则认为是普通文本
                 reply = Reply()
@@ -68,7 +90,7 @@ class Keyword(Plugin):
             
             e_context["reply"] = reply
             e_context.action = EventAction.BREAK_PASS  # 事件结束，并跳过处理context的默认逻辑
-
+            
     def get_help_text(self, **kwargs):
         help_text = "关键词过滤"
         return help_text

plugins/linkai/README.md

@@ -1,16 +1,16 @@
 ## 插件说明
 
-基于 LinkAI 提供的知识库、Midjourney绘画等能力对机器人的功能进行增强。平台地址: https://chat.link-ai.tech/console
+基于 LinkAI 提供的知识库、Midjourney绘画、文档对话等能力对机器人的功能进行增强。平台地址: https://link-ai.tech/console
 
 ## 插件配置
 
-将 `plugins/linkai` 目录下的 `config.json.template` 配置模板复制为最终生效的 `config.json`。 (如果未配置则会默认使用`config.json.template`模板中配置，功能默认关闭，可通过指令进行开启)。
+将 `plugins/linkai` 目录下的 `config.json.template` 配置模板复制为最终生效的 `config.json`。 (如果未配置则会默认使用`config.json.template`模板中配置，但功能默认关闭，需要可通过指令进行开启)。
 
-以下是配置项说明：
+以下是插件配置项说明：
 
 ```bash
 {
-    "group_app_map": {            # 群聊 和 应用编码 的映射关系
+    "group_app_map": {               # 群聊 和 应用编码 的映射关系
         "测试群名称1": "default",      # 表示在名称为 "测试群名称1" 的群聊中将使用app_code 为 default 的应用
         "测试群名称2": "Kv2fXJcH"
     },
@@ -21,19 +21,31 @@
         "max_tasks": 3,           # 支持同时提交的总任务个数
         "max_tasks_per_user": 1,  # 支持单个用户同时提交的任务个数
         "use_image_create_prefix": true   # 是否使用全局的绘画触发词，如果开启将同时支持由`config.json`中的 image_create_prefix 配置触发
+    },
+    "summary": {
+        "enabled": true,              # 文档总结和对话功能开关
+        "group_enabled": true,        # 是否支持群聊开启
+        "max_file_size": 5000,        # 文件的大小限制，单位KB，默认为5M，超过该大小直接忽略
+        "type": ["FILE", "SHARING", "IMAGE"]  # 支持总结的类型，分别表示 文件、分享链接、图片，其中文件和链接默认打开，图片默认关闭
     }
 }
-
 ```
+
+根目录 `config.json` 中配置，`API_KEY` 在 [控制台](https://link-ai.tech/console/interface) 中创建并复制过来:
+
+```bash
+"linkai_api_key": "Link_xxxxxxxxx"
+```
+
 注意：
 
- - 配置项中 `group_app_map` 部分是用于映射群聊与LinkAI平台上的应用， `midjourney` 部分是 mj 画图的配置，可根据需要进行填写，未填写配置时默认不开启相应功能
+ - 配置项中 `group_app_map` 部分是用于映射群聊与LinkAI平台上的应用， `midjourney` 部分是 mj 画图的配置，`summary` 部分是文档总结及对话功能的配置。三部分的配置相互独立，可按需开启
  - 实际 `config.json` 配置中应保证json格式，不应携带 '#' 及后面的注释
  - 如果是`docker`部署，可通过映射 `plugins/config.json` 到容器中来完成插件配置，参考[文档](https://github.com/zhayujie/chatgpt-on-wechat#3-%E6%8F%92%E4%BB%B6%E4%BD%BF%E7%94%A8)
 
 ## 插件使用
 
-> 使用插件中的知识库管理功能需要首先开启`linkai`对话，依赖全局 `config.json` 中的 `use_linkai` 和 `linkai_api_key` 配置；而midjourney绘画功能则只需填写 `linkai_api_key` 配置，`use_linkai` 无论是否关闭均可使用。具体可参考 [详细文档](https://link-ai.tech/platform/link-app/wechat)。
+> 使用插件中的知识库管理功能需要首先开启`linkai`对话，依赖全局 `config.json` 中的 `use_linkai` 和 `linkai_api_key` 配置；而midjourney绘画 和 summary文档总结对话功能则只需填写 `linkai_api_key` 配置，`use_linkai` 无论是否关闭均可使用。具体可参考 [详细文档](https://link-ai.tech/platform/link-app/wechat)。
 
 完成配置后运行项目，会自动运行插件，输入 `#help linkai` 可查看插件功能。
 
@@ -73,7 +85,25 @@
 
 注意事项：
 1. 使用 `$mj open` 和 `$mj close` 指令可以快速打开和关闭绘图功能
-2. 海外环境部署请将 `img_proxy` 设置为 `False`
+2. 海外环境部署请将 `img_proxy` 设置为 `false`
 3. 开启 `use_image_create_prefix` 配置后可直接复用全局画图触发词，以"画"开头便可以生成图片。
 4. 提示词内容中包含敏感词或者参数格式错误可能导致绘画失败，生成失败不消耗积分
 5. 若未收到图片可能有两种可能，一种是收到了图片但微信发送失败，可以在后台日志查看有没有获取到图片url，一般原因是受到了wx限制，可以稍后重试或更换账号尝试；另一种情况是图片提示词存在疑似违规，mj不会直接提示错误但会在画图后删掉原图导致程序无法获取，这种情况不消耗积分。
+
+### 3.文档总结对话功能
+
+#### 配置
+
+该功能依赖 LinkAI的知识库及对话功能，需要在项目根目录的config.json中设置 `linkai_api_key`， 同时根据上述插件配置说明，在插件config.json添加 `summary` 部分的配置，设置 `enabled` 为 true。
+
+如果不想创建 `plugins/linkai/config.json` 配置，可以直接通过 `$linkai sum open` 指令开启该功能。
+
+#### 使用
+
+功能开启后，向机器人发送 **文件**、 **分享链接卡片**、**图片** 即可生成摘要，进一步可以与文件或链接的内容进行多轮对话。如果需要关闭某种类型的内容总结，设置 `summary`配置中的type字段即可。
+
+#### 限制
+
+ 1. 文件目前 支持 `txt`, `docx`, `pdf`, `md`, `csv`格式，文件大小由 `max_file_size` 限制，最大不超过15M，文件字数最多可支持百万字的文件。但不建议上传字数过多的文件，一是token消耗过大，二是摘要很难覆盖到全部内容，只能通过多轮对话来了解细节。
+ 2. 分享链接 目前仅支持 公众号文章，后续会支持更多文章类型及视频链接等
+ 3. 总结及对话的 费用与 LinkAI 3.5-4K 模型的计费方式相同，按文档内容的tokens进行计算

plugins/linkai/config.json.template

@@ -10,5 +10,11 @@
         "max_tasks": 3,
         "max_tasks_per_user": 1,
         "use_image_create_prefix": true
+    },
+    "summary": {
+        "enabled": true,
+        "group_enabled": true,
+        "max_file_size": 5000,
+        "type": ["FILE", "SHARING"]
     }
 }

plugins/linkai/linkai.py

@@ -1,17 +1,21 @@
 import plugins
 from bridge.context import ContextType
 from bridge.reply import Reply, ReplyType
-from config import global_config
 from plugins import *
 from .midjourney import MJBot
+from .summary import LinkSummary
 from bridge import bridge
-
+from common.expired_dict import ExpiredDict
+from common import const
+import os
+from .utils import Util
 
 @plugins.register(
     name="linkai",
     desc="A plugin that supports knowledge base and midjourney drawing.",
     version="0.1.0",
     author="https://link-ai.tech",
+    desire_priority=99
 )
 class LinkAI(Plugin):
     def __init__(self):
@@ -23,7 +27,11 @@ class LinkAI(Plugin):
             self.config = self._load_config_template()
         if self.config:
             self.mj_bot = MJBot(self.config.get("midjourney"))
-        logger.info("[LinkAI] inited")
+        self.sum_config = {}
+        if self.config:
+            self.sum_config = self.config.get("summary")
+        logger.info(f"[LinkAI] inited, config={self.config}")
+
 
     def on_handle_context(self, e_context: EventContext):
         """
@@ -34,10 +42,44 @@ class LinkAI(Plugin):
             return
 
         context = e_context['context']
-        if context.type not in [ContextType.TEXT, ContextType.IMAGE, ContextType.IMAGE_CREATE]:
+        if context.type not in [ContextType.TEXT, ContextType.IMAGE, ContextType.IMAGE_CREATE, ContextType.FILE, ContextType.SHARING]:
             # filter content no need solve
             return
 
+        if context.type in [ContextType.FILE, ContextType.IMAGE] and self._is_summary_open(context):
+            # 文件处理
+            context.get("msg").prepare()
+            file_path = context.content
+            if not LinkSummary().check_file(file_path, self.sum_config):
+                return
+            if context.type != ContextType.IMAGE:
+                _send_info(e_context, "正在为你加速生成摘要，请稍后")
+            res = LinkSummary().summary_file(file_path)
+            if not res:
+                if context.type != ContextType.IMAGE:
+                    _set_reply_text("因为神秘力量无法获取内容，请稍后再试吧", e_context, level=ReplyType.TEXT)
+                return
+            summary_text = res.get("summary")
+            if context.type != ContextType.IMAGE:
+                USER_FILE_MAP[_find_user_id(context) + "-sum_id"] = res.get("summary_id")
+                summary_text += "\n\n💬 发送 \"开启对话\" 可以开启与文件内容的对话"
+            _set_reply_text(summary_text, e_context, level=ReplyType.TEXT)
+            os.remove(file_path)
+            return
+
+        if (context.type == ContextType.SHARING and self._is_summary_open(context)) or \
+                (context.type == ContextType.TEXT and LinkSummary().check_url(context.content)):
+            if not LinkSummary().check_url(context.content):
+                return
+            _send_info(e_context, "正在为你加速生成摘要，请稍后")
+            res = LinkSummary().summary_url(context.content)
+            if not res:
+                _set_reply_text("因为神秘力量无法获取文章内容，请稍后再试吧~", e_context, level=ReplyType.TEXT)
+                return
+            _set_reply_text(res.get("summary") + "\n\n💬 发送 \"开启对话\" 可以开启与文章内容的对话", e_context, level=ReplyType.TEXT)
+            USER_FILE_MAP[_find_user_id(context) + "-sum_id"] = res.get("summary_id")
+            return
+
         mj_type = self.mj_bot.judge_mj_task_type(e_context)
         if mj_type:
             # MJ作图任务处理
@@ -49,10 +91,38 @@ class LinkAI(Plugin):
             self._process_admin_cmd(e_context)
             return
 
+        if context.type == ContextType.TEXT and context.content == "开启对话" and _find_sum_id(context):
+            # 文本对话
+            _send_info(e_context, "正在为你开启对话，请稍后")
+            res = LinkSummary().summary_chat(_find_sum_id(context))
+            if not res:
+                _set_reply_text("开启对话失败，请稍后再试吧", e_context)
+                return
+            USER_FILE_MAP[_find_user_id(context) + "-file_id"] = res.get("file_id")
+            _set_reply_text("💡你可以问我关于这篇文章的任何问题，例如：\n\n" + res.get("questions") + "\n\n发送 \"退出对话\" 可以关闭与文章的对话", e_context, level=ReplyType.TEXT)
+            return
+
+        if context.type == ContextType.TEXT and context.content == "退出对话" and _find_file_id(context):
+            del USER_FILE_MAP[_find_user_id(context) + "-file_id"]
+            bot = bridge.Bridge().find_chat_bot(const.LINKAI)
+            bot.sessions.clear_session(context["session_id"])
+            _set_reply_text("对话已退出", e_context, level=ReplyType.TEXT)
+            return
+
+        if context.type == ContextType.TEXT and _find_file_id(context):
+            bot = bridge.Bridge().find_chat_bot(const.LINKAI)
+            context.kwargs["file_id"] = _find_file_id(context)
+            reply = bot.reply(context.content, context)
+            e_context["reply"] = reply
+            e_context.action = EventAction.BREAK_PASS
+            return
+
+
         if self._is_chat_task(e_context):
             # 文本对话任务处理
             self._process_chat_task(e_context)
 
+
     # 插件管理功能
     def _process_admin_cmd(self, e_context: EventContext):
         context = e_context['context']
@@ -63,7 +133,7 @@ class LinkAI(Plugin):
 
         if len(cmd) == 2 and (cmd[1] == "open" or cmd[1] == "close"):
             # 知识库开关指令
-            if not _is_admin(e_context):
+            if not Util.is_admin(e_context):
                 _set_reply_text("需要管理员权限执行", e_context, level=ReplyType.ERROR)
                 return
             is_open = True
@@ -81,7 +151,7 @@ class LinkAI(Plugin):
             if not context.kwargs.get("isgroup"):
                 _set_reply_text("该指令需在群聊中使用", e_context, level=ReplyType.ERROR)
                 return
-            if not _is_admin(e_context):
+            if not Util.is_admin(e_context):
                 _set_reply_text("需要管理员权限执行", e_context, level=ReplyType.ERROR)
                 return
             app_code = cmd[2]
@@ -94,11 +164,39 @@ class LinkAI(Plugin):
             # 保存插件配置
             super().save_config(self.config)
             _set_reply_text(f"应用设置成功: {app_code}", e_context, level=ReplyType.INFO)
-        else:
-            _set_reply_text(f"指令错误，请输入{_get_trigger_prefix()}linkai help 获取帮助", e_context,
-                            level=ReplyType.INFO)
             return
 
+        if len(cmd) == 3 and cmd[1] == "sum" and (cmd[2] == "open" or cmd[2] == "close"):
+            # 知识库开关指令
+            if not Util.is_admin(e_context):
+                _set_reply_text("需要管理员权限执行", e_context, level=ReplyType.ERROR)
+                return
+            is_open = True
+            tips_text = "开启"
+            if cmd[2] == "close":
+                tips_text = "关闭"
+                is_open = False
+            if not self.sum_config:
+                _set_reply_text(f"插件未启用summary功能，请参考以下链添加插件配置\n\nhttps://github.com/zhayujie/chatgpt-on-wechat/blob/master/plugins/linkai/README.md", e_context, level=ReplyType.INFO)
+            else:
+                self.sum_config["enabled"] = is_open
+                _set_reply_text(f"文章总结功能{tips_text}", e_context, level=ReplyType.INFO)
+            return
+
+        _set_reply_text(f"指令错误，请输入{_get_trigger_prefix()}linkai help 获取帮助", e_context,
+                        level=ReplyType.INFO)
+        return
+
+    def _is_summary_open(self, context) -> bool:
+        if not self.sum_config or not self.sum_config.get("enabled"):
+            return False
+        if context.kwargs.get("isgroup") and not self.sum_config.get("group_enabled"):
+            return False
+        support_type = self.sum_config.get("type") or ["FILE", "SHARING"]
+        if context.type.name not in support_type:
+            return False
+        return True
+
     # LinkAI 对话任务处理
     def _is_chat_task(self, e_context: EventContext):
         context = e_context['context']
@@ -112,7 +210,7 @@ class LinkAI(Plugin):
         """
         context = e_context['context']
         # 群聊应用管理
-        group_name = context.kwargs.get("msg").from_user_nickname
+        group_name = context.get("msg").from_user_nickname
         app_code = self._fetch_group_app_code(group_name)
         if app_code:
             context.kwargs['app_code'] = app_code
@@ -130,7 +228,7 @@ class LinkAI(Plugin):
 
     def get_help_text(self, verbose=False, **kwargs):
         trigger_prefix = _get_trigger_prefix()
-        help_text = "用于集成 LinkAI 提供的知识库、Midjourney绘画等能力。\n\n"
+        help_text = "用于集成 LinkAI 提供的知识库、Midjourney绘画、文档总结、联网搜索等能力。\n\n"
         if not verbose:
             return help_text
         help_text += f'📖 知识库\n - 群聊中指定应用: {trigger_prefix}linkai app 应用编码\n'
@@ -140,6 +238,7 @@ class LinkAI(Plugin):
         help_text += f"🎨 绘画\n - 生成: {trigger_prefix}mj 描述词1, 描述词2.. \n - 放大: {trigger_prefix}mju 图片ID 图片序号\n - 变换: {trigger_prefix}mjv 图片ID 图片序号\n - 重置: {trigger_prefix}mjr 图片ID"
         help_text += f"\n\n例如：\n\"{trigger_prefix}mj a little cat, white --ar 9:16\"\n\"{trigger_prefix}mju 11055927171882 2\""
         help_text += f"\n\"{trigger_prefix}mjv 11055927171882 2\"\n\"{trigger_prefix}mjr 11055927171882\""
+        help_text += f"\n\n💡 文档总结和对话\n - 开启: {trigger_prefix}linkai sum open\n - 使用: 发送文件、公众号文章等可生成摘要，并与内容对话"
         return help_text
 
     def _load_config_template(self):
@@ -150,22 +249,23 @@ class LinkAI(Plugin):
                 with open(plugin_config_path, "r", encoding="utf-8") as f:
                     plugin_conf = json.load(f)
                     plugin_conf["midjourney"]["enabled"] = False
+                    plugin_conf["summary"]["enabled"] = False
                     return plugin_conf
         except Exception as e:
             logger.exception(e)
 
-# 静态方法
-def _is_admin(e_context: EventContext) -> bool:
-    """
-    判断消息是否由管理员用户发送
-    :param e_context: 消息上下文
-    :return: True: 是, False: 否
-    """
-    context = e_context["context"]
+
+def _send_info(e_context: EventContext, content: str):
+    reply = Reply(ReplyType.TEXT, content)
+    channel = e_context["channel"]
+    channel.send(reply, e_context["context"])
+
+
+def _find_user_id(context):
     if context["isgroup"]:
-        return context.kwargs.get("msg").actual_user_id in global_config["admin_users"]
+        return context.kwargs.get("msg").actual_user_id
     else:
-        return context["receiver"] in global_config["admin_users"]
+        return context["receiver"]
 
 
 def _set_reply_text(content: str, e_context: EventContext, level: ReplyType = ReplyType.ERROR):
@@ -173,6 +273,15 @@ def _set_reply_text(content: str, e_context: EventContext, level: ReplyType = Re
     e_context["reply"] = reply
     e_context.action = EventAction.BREAK_PASS
 
-
 def _get_trigger_prefix():
     return conf().get("plugin_trigger_prefix", "$")
+
+def _find_sum_id(context):
+    return USER_FILE_MAP.get(_find_user_id(context) + "-sum_id")
+
+def _find_file_id(context):
+    user_id = _find_user_id(context)
+    if user_id:
+        return USER_FILE_MAP.get(user_id + "-file_id")
+
+USER_FILE_MAP = ExpiredDict(conf().get("expires_in_seconds") or 60 * 30)

plugins/linkai/midjourney.py

@@ -5,10 +5,10 @@ import requests
 import threading
 import time
 from bridge.reply import Reply, ReplyType
-import aiohttp
 import asyncio
 from bridge.context import ContextType
 from plugins import EventContext, EventAction
+from .utils import Util
 
 INVALID_REQUEST = 410
 NOT_FOUND_ORIGIN_IMAGE = 461
@@ -49,7 +49,7 @@ task_name_mapping = {
 
 
 class MJTask:
-    def __init__(self, id, user_id: str, task_type: TaskType, raw_prompt=None, expires: int = 60 * 30,
+    def __init__(self, id, user_id: str, task_type: TaskType, raw_prompt=None, expires: int = 60 * 6,
                  status=Status.PENDING):
         self.id = id
         self.user_id = user_id
@@ -88,6 +88,8 @@ class MJBot:
         context = e_context['context']
         if context.type == ContextType.TEXT:
             cmd_list = context.content.split(maxsplit=1)
+            if not cmd_list:
+                return None
             if cmd_list[0].lower() == f"{trigger_prefix}mj":
                 return TaskType.GENERATE
             elif cmd_list[0].lower() == f"{trigger_prefix}mju":
@@ -114,6 +116,9 @@ class MJBot:
             return
 
         if len(cmd) == 2 and (cmd[1] == "open" or cmd[1] == "close"):
+            if not Util.is_admin(e_context):
+                Util.set_reply_text("需要管理员权限执行", e_context, level=ReplyType.ERROR)
+                return
             # midjourney 开关指令
             is_open = True
             tips_text = "开启"

plugins/linkai/summary.py

@@ -0,0 +1,94 @@
+import requests
+from config import conf
+from common.log import logger
+import os
+
+
+class LinkSummary:
+    def __init__(self):
+        pass
+
+    def summary_file(self, file_path: str):
+        file_body = {
+            "file": open(file_path, "rb"),
+            "name": file_path.split("/")[-1],
+        }
+        url = self.base_url() + "/v1/summary/file"
+        res = requests.post(url, headers=self.headers(), files=file_body, timeout=(5, 300))
+        return self._parse_summary_res(res)
+
+    def summary_url(self, url: str):
+        body = {
+            "url": url
+        }
+        res = requests.post(url=self.base_url() + "/v1/summary/url", headers=self.headers(), json=body, timeout=(5, 180))
+        return self._parse_summary_res(res)
+
+    def summary_chat(self, summary_id: str):
+        body = {
+            "summary_id": summary_id
+        }
+        res = requests.post(url=self.base_url() + "/v1/summary/chat", headers=self.headers(), json=body, timeout=(5, 180))
+        if res.status_code == 200:
+            res = res.json()
+            logger.debug(f"[LinkSum] chat open, res={res}")
+            if res.get("code") == 200:
+                data = res.get("data")
+                return {
+                    "questions": data.get("questions"),
+                    "file_id": data.get("file_id")
+                }
+        else:
+            res_json = res.json()
+            logger.error(f"[LinkSum] summary error, status_code={res.status_code}, msg={res_json.get('message')}")
+            return None
+
+    def _parse_summary_res(self, res):
+        if res.status_code == 200:
+            res = res.json()
+            logger.debug(f"[LinkSum] url summary, res={res}")
+            if res.get("code") == 200:
+                data = res.get("data")
+                return {
+                    "summary": data.get("summary"),
+                    "summary_id": data.get("summary_id")
+                }
+        else:
+            res_json = res.json()
+            logger.error(f"[LinkSum] summary error, status_code={res.status_code}, msg={res_json.get('message')}")
+            return None
+
+    def base_url(self):
+        return conf().get("linkai_api_base", "https://api.link-ai.chat")
+
+    def headers(self):
+        return {"Authorization": "Bearer " + conf().get("linkai_api_key")}
+
+    def check_file(self, file_path: str, sum_config: dict) -> bool:
+        file_size = os.path.getsize(file_path) // 1000
+
+        if (sum_config.get("max_file_size") and file_size > sum_config.get("max_file_size")) or file_size > 15000:
+            logger.warn(f"[LinkSum] file size exceeds limit, No processing, file_size={file_size}KB")
+            return False
+
+        suffix = file_path.split(".")[-1]
+        support_list = ["txt", "csv", "docx", "pdf", "md", "jpg", "jpeg", "png"]
+        if suffix not in support_list:
+            logger.warn(f"[LinkSum] unsupported file, suffix={suffix}, support_list={support_list}")
+            return False
+
+        return True
+
+    def check_url(self, url: str):
+        if not url:
+            return False
+        support_list = ["http://mp.weixin.qq.com", "https://mp.weixin.qq.com"]
+        black_support_list = ["https://mp.weixin.qq.com/mp/waerrpage"]
+        for black_url_prefix in black_support_list:
+            if url.strip().startswith(black_url_prefix):
+                logger.warn(f"[LinkSum] unsupported url, no need to process, url={url}")
+                return False
+        for support_url in support_list:
+            if url.strip().startswith(support_url):
+                return True
+        return False

plugins/linkai/utils.py

@@ -0,0 +1,28 @@
+from config import global_config
+from bridge.reply import Reply, ReplyType
+from plugins.event import EventContext, EventAction
+
+
+class Util:
+    @staticmethod
+    def is_admin(e_context: EventContext) -> bool:
+        """
+        判断消息是否由管理员用户发送
+        :param e_context: 消息上下文
+        :return: True: 是, False: 否
+        """
+        context = e_context["context"]
+        if context["isgroup"]:
+            actual_user_id = context.kwargs.get("msg").actual_user_id
+            for admin_user in global_config["admin_users"]:
+                if actual_user_id and actual_user_id in admin_user:
+                    return True
+            return False
+        else:
+            return context["receiver"] in global_config["admin_users"]
+
+    @staticmethod
+    def set_reply_text(content: str, e_context: EventContext, level: ReplyType = ReplyType.ERROR):
+        reply = Reply(level, content)
+        e_context["reply"] = reply
+        e_context.action = EventAction.BREAK_PASS

plugins/plugin.py

@@ -21,6 +21,9 @@ class Plugin:
             if os.path.exists(plugin_config_path):
                 with open(plugin_config_path, "r", encoding="utf-8") as f:
                     plugin_conf = json.load(f)
+
+                # 写入全局配置内存
+                plugin_config[self.name] = plugin_conf
         logger.debug(f"loading plugin config, plugin_name={self.name}, conf={plugin_conf}")
         return plugin_conf
 
@@ -43,3 +46,6 @@ class Plugin:
 
     def get_help_text(self, **kwargs):
         return "暂无帮助信息"
+
+    def reload(self):
+        pass

plugins/source.json

@@ -15,6 +15,10 @@
     "timetask": {
       "url": "https://github.com/haikerapples/timetask.git",
       "desc": "一款定时任务系统的插件"
+    },
+    "Apilot": {
+      "url": "https://github.com/6vision/Apilot.git",
+      "desc": "通过api直接查询早报、热榜、快递、天气等实用信息的插件"
     }
   }
 }

plugins/tool/README.md

@@ -3,11 +3,19 @@
 使用说明(默认trigger_prefix为$)：  
 ```text
 #help tool: 查看tool帮助信息，可查看已加载工具列表  
-$tool 命令: 根据给出的{命令}使用一些可用工具尽力为你得到结果。  
+$tool 工具名 命令: （pure模式）根据给出的{命令}使用指定 一个 可用工具尽力为你得到结果。
+$tool 命令: （多工具模式）根据给出的{命令}使用 一些 可用工具尽力为你得到结果。  
 $tool reset: 重置工具。  
 ```
 ### 本插件所有工具同步存放至专用仓库：[chatgpt-tool-hub](https://github.com/goldfishh/chatgpt-tool-hub)
 
+2024.01.16更新
+1. 新增工具pure模式，支持单个工具调用
+2. 新增消息转发工具：email, sms, wechat, 可以根据规则向其他平台发送消息
+3. 替换visual-dl（更名为visual）实现，目前识别图片链接效果较好。
+4. 修复了0.4版本大部分工具返回结果不可靠问题
+
+新版本工具名共19个，不一一列举，相应工具需要的环境参数见`tool.py`里的`_build_tool_kwargs`函数
 
 ## 使用说明
 使用该插件后将默认使用4个工具, 无需额外配置长期生效：
@@ -24,7 +32,7 @@ $tool reset: 重置工具。
 
 > 注1：url-get默认配置、browser需额外配置，browser依赖google-chrome，你需要提前安装好
 
-> 注2：当检测到长文本时会进入summary tool总结长文本，tokens可能会大量消耗！
+> 注2：（可通过`browser_use_summary`或 `url_get_use_summary`开关）当检测到长文本时会进入summary tool总结长文本，tokens可能会大量消耗！
 
 这是debian端安装google-chrome教程，其他系统请自行查找
 > https://www.linuxjournal.com/content/how-can-you-install-google-browser-debian
@@ -34,9 +42,10 @@ $tool reset: 重置工具。
 
 > terminal调优记录：https://github.com/zhayujie/chatgpt-on-wechat/issues/776#issue-1659347640
 
-### 4. meteo-weather
+### 4. meteo
 ###### 回答你有关天气的询问, 需要获取时间、地点上下文信息，本工具使用了[meteo open api](https://open-meteo.com/)
 注：该工具需要较高的对话技巧，不保证你问的任何问题均能得到满意的回复
+注2：当前版本可只使用这个工具，返回结果较可控。
 
 > meteo调优记录：https://github.com/zhayujie/chatgpt-on-wechat/issues/776#issuecomment-1500771334
 
@@ -65,18 +74,12 @@ $tool reset: 重置工具。
 #### 6.2. morning-news *
 ###### 每日60秒早报，每天凌晨一点更新，本工具使用了[alapi-每日60秒早报](https://alapi.cn/api/view/93)
 
-```text
-可配置参数：
-1. morning_news_use_llm: 是否使用LLM润色结果，默认false（可能会慢）
-```
-
 > 该tool每天返回内容相同
 
 #### 6.3. finance-news
 ###### 获取实时的金融财政新闻
 
-> 该工具需要解决browser tool 的google-chrome依赖安装
-
+> 该工具需要用到browser工具解决反爬问题
 
 
 ### 7. bing-search *
@@ -99,18 +102,33 @@ $tool reset: 重置工具。
 > 0.4.2更新，例子：帮我找一篇吴恩达写的论文
 
 ### 11. summary
-###### 总结工具，该工具必须输入一个本地文件的绝对路径
+###### 总结工具，该工具可以支持输入url
 
 > 该工具目前是和其他工具配合使用，暂未测试单独使用效果
 
-### 12. image2text
-###### 将图片转换成文字，底层调用imageCaption模型，该工具必须输入一个本地文件的绝对路径
+### 12. visual
+###### 将图片转换成文字，底层调用ali dashscope `qwen-vl-plus`模型
 
 ### 13. searxng-search *
 ###### 一个私有化的搜索引擎工具
 
 > 安装教程：https://docs.searxng.org/admin/installation.html
 
+### 14. email *
+###### 发送邮件
+
+### 15. sms *
+###### 发送短信
+
+### 16. stt *
+###### speak to text 语音识别
+
+### 17. tts *
+###### text to speak 文生语音
+
+### 18. wechat *
+###### 向好友、群组发送微信
+
 ---
 
 ###### 注1：带*工具需要获取api-key才能使用(在config.json内的kwargs添加项)，部分工具需要外网支持  
@@ -120,7 +138,7 @@ $tool reset: 重置工具。
 ###### 默认工具无需配置，其它工具需手动配置，以增加morning-news和bing-search两个工具为例：
 ```json
 {
-  "tools": ["bing-search", "news", "你想要添加的其他工具"],  // 填入你想用到的额外工具名，这里加入了工具"bing-search"和工具"news"(news工具会自动加载morning-news、finance-news等子工具)
+  "tools": ["bing-search", "morning-news", "你想要添加的其他工具"],  // 填入你想用到的额外工具名，这里加入了工具"bing-search"和工具"morning-news"
   "kwargs": {
       "debug": true, // 当你遇到问题求助时，需要配置
       "request_timeout": 120,  // openai接口超时时间
@@ -137,7 +155,6 @@ $tool reset: 重置工具。
   - `debug`: 输出chatgpt-tool-hub额外信息用于调试
   - `request_timeout`: 访问openai接口的超时时间，默认与wechat-on-chatgpt配置一致，可单独配置
   - `no_default`: 用于配置默认加载4个工具的行为，如果为true则仅使用tools列表工具，不加载默认工具
-  - `top_k_results`: 控制所有有关搜索的工具返回条目数，数字越高则参考信息越多，但无用信息可能干扰判断，该值一般为2
   - `model_name`: 用于控制tool插件底层使用的llm模型，目前暂未测试3.5以外的模型，一般保持默认
 
 ---

plugins/tool/config.json.template

@@ -3,10 +3,10 @@
     "python",
     "url-get",
     "terminal",
-    "meteo-weather"
+    "meteo"
   ],
   "kwargs": {
-    "top_k_results": 2,
+    "debug": false,
     "no_default": false,
     "model_name": "gpt-3.5-turbo"
   }

plugins/tool/tool.py

@@ -1,23 +1,20 @@
-import json
-import os
-
 from chatgpt_tool_hub.apps import AppFactory
 from chatgpt_tool_hub.apps.app import App
-from chatgpt_tool_hub.tools.all_tool_list import get_all_tool_names
+from chatgpt_tool_hub.tools.tool_register import main_tool_register
 
 import plugins
 from bridge.bridge import Bridge
 from bridge.context import ContextType
 from bridge.reply import Reply, ReplyType
 from common import const
-from config import conf
+from config import conf, get_appdata_dir
 from plugins import *
 
 
 @plugins.register(
     name="tool",
     desc="Arming your ChatGPT bot with various tools",
-    version="0.4",
+    version="0.5",
     author="goldfishh",
     desire_priority=0,
 )
@@ -36,10 +33,12 @@ class Tool(Plugin):
         if not verbose:
             return help_text
         help_text += "\n使用说明：\n"
-        help_text += f"{trigger_prefix}tool " + "命令: 根据给出的{命令}使用一些可用工具尽力为你得到结果。\n"
+        help_text += f"{trigger_prefix}tool " + "命令: 根据给出的{命令}模型来选择使用哪些工具尽力为你得到结果。\n"
+        help_text += f"{trigger_prefix}tool 工具名 " + "命令: 根据给出的{命令}使用指定工具尽力为你得到结果。\n"
         help_text += f"{trigger_prefix}tool reset: 重置工具。\n\n"
+
         help_text += f"已加载工具列表: \n"
-        for idx, tool in enumerate(self.app.get_tool_list()):
+        for idx, tool in enumerate(main_tool_register.get_registered_tool_names()):
             if idx != 0:
                 help_text += ", "
             help_text += f"{tool}"
@@ -91,17 +90,28 @@ class Tool(Plugin):
 
                     e_context.action = EventAction.BREAK
                     return
-
                 query = content_list[1].strip()
+                
+                use_one_tool = False
+                for tool_name in main_tool_register.get_registered_tool_names():
+                    if query.startswith(tool_name):
+                        use_one_tool = True
+                        query = query[len(tool_name):]
+                        break
 
                 # Don't modify bot name
                 all_sessions = Bridge().get_bot("chat").sessions
                 user_session = all_sessions.session_query(query, e_context["context"]["session_id"]).messages
 
-                # chatgpt-tool-hub will reply you with many tools
                 logger.debug("[tool]: just-go")
                 try:
-                    _reply = self.app.ask(query, user_session)
+                    if use_one_tool:
+                        _func, _ = main_tool_register.get_registered_tool()[tool_name]
+                        tool = _func(**self.app_kwargs)
+                        _reply = tool.run(query)
+                    else:
+                        # chatgpt-tool-hub will reply you with many tools
+                        _reply = self.app.ask(query, user_session)
                     e_context.action = EventAction.BREAK_PASS
                     all_sessions.session_reply(_reply, e_context["context"]["session_id"])
                 except Exception as e:
@@ -126,53 +136,111 @@ class Tool(Plugin):
         request_timeout = kwargs.get("request_timeout")
 
         return {
-            "debug": kwargs.get("debug", False),
-            "openai_api_key": conf().get("open_ai_api_key", ""),
-            "open_ai_api_base": conf().get("open_ai_api_base", "https://api.openai.com/v1"),
-            "deployment_id": conf().get("azure_deployment_id", ""),
-            "proxy": conf().get("proxy", ""),
+            # 全局配置相关
+            "log": True,  # tool 日志开关
+            "debug": kwargs.get("debug", False),  # 输出更多日志
+            "no_default": kwargs.get("no_default", False),  # 不要默认的工具，只加载自己导入的工具
+            "think_depth": kwargs.get("think_depth", 2),  # 一个问题最多使用多少次工具
+            "proxy": conf().get("proxy", ""),  # 科学上网
             "request_timeout": request_timeout if request_timeout else conf().get("request_timeout", 120),
+            "temperature": kwargs.get("temperature", 0),  # llm 温度，建议设置0
+            # LLM配置相关
+            "llm_api_key": conf().get("open_ai_api_key", ""),  # 如果llm api用key鉴权，传入这里
+            "llm_api_base_url": conf().get("open_ai_api_base", "https://api.openai.com/v1"),  # 支持openai接口的llm服务地址前缀
+            "deployment_id": conf().get("azure_deployment_id", ""),  # azure openai会用到
             # note: 目前tool暂未对其他模型测试，但这里仍对配置来源做了优先级区分，一般插件配置可覆盖全局配置
-            "model_name": tool_model_name if tool_model_name else conf().get("model", "gpt-3.5-turbo"),
-            "no_default": kwargs.get("no_default", False),
-            "top_k_results": kwargs.get("top_k_results", 3),
-            # for news tool
-            "news_api_key": kwargs.get("news_api_key", ""),
+            "model_name": tool_model_name if tool_model_name else conf().get("model", const.GPT35),
+            # 工具配置相关
+            # for arxiv tool
+            "arxiv_simple": kwargs.get("arxiv_simple", True),  # 返回内容更精简
+            "arxiv_top_k_results": kwargs.get("arxiv_top_k_results", 2),  # 只返回前k个搜索结果
+            "arxiv_sort_by": kwargs.get("arxiv_sort_by", "relevance"),  # 搜索排序方式 ["relevance","lastUpdatedDate","submittedDate"]
+            "arxiv_sort_order": kwargs.get("arxiv_sort_order", "descending"),  # 搜索排序方式 ["ascending", "descending"]
+            "arxiv_output_type": kwargs.get("arxiv_output_type", "text"),  # 搜索结果类型 ["text", "pdf", "all"]
             # for bing-search tool
             "bing_subscription_key": kwargs.get("bing_subscription_key", ""),
+            "bing_search_url": kwargs.get("bing_search_url", "https://api.bing.microsoft.com/v7.0/search"),  # 必应搜索的endpoint地址，无需修改
+            "bing_search_top_k_results": kwargs.get("bing_search_top_k_results", 2),  # 只返回前k个搜索结果
+            "bing_search_simple": kwargs.get("bing_search_simple", True),  # 返回内容更精简
+            "bing_search_output_type": kwargs.get("bing_search_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for email tool
+            "email_nickname_mapping": kwargs.get("email_nickname_mapping", "{}"),  # 关于人的代号对应的邮箱地址，可以不输入邮箱地址发送邮件。键为代号值为邮箱地址
+            "email_smtp_host": kwargs.get("email_smtp_host", ""),  # 例如 'smtp.qq.com'
+            "email_smtp_port": kwargs.get("email_smtp_port", ""),  # 例如 587
+            "email_sender": kwargs.get("email_sender", ""),  # 发送者的邮件地址
+            "email_authorization_code": kwargs.get("email_authorization_code", ""),  # 发送者验证秘钥（可能不是登录密码）
             # for google-search tool
             "google_api_key": kwargs.get("google_api_key", ""),
             "google_cse_id": kwargs.get("google_cse_id", ""),
+            "google_simple": kwargs.get("google_simple", True),   # 返回内容更精简
+            "google_output_type": kwargs.get("google_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for finance-news tool
+            "finance_news_filter": kwargs.get("finance_news_filter", False),  # 是否开启过滤
+            "finance_news_filter_list": kwargs.get("finance_news_filter_list", []),  # 过滤词列表
+            "finance_news_simple": kwargs.get("finance_news_simple", True),   # 返回内容更精简
+            "finance_news_repeat_news": kwargs.get("finance_news_repeat_news", False),  # 是否过滤不返回。该tool每次返回约50条新闻，可能有重复新闻
+            # for morning-news tool
+            "morning_news_api_key": kwargs.get("morning_news_api_key", ""),   # api-key
+            "morning_news_simple": kwargs.get("morning_news_simple", True),   # 返回内容更精简
+            "morning_news_output_type": kwargs.get("morning_news_output_type", "text"),  # 搜索结果类型 ["text", "image"]
+            # for news-api tool
+            "news_api_key": kwargs.get("news_api_key", ""),
             # for searxng-search tool
-            "searx_search_host": kwargs.get("searx_search_host", ""),
+            "searxng_search_host": kwargs.get("searxng_search_host", ""),
+            "searxng_search_top_k_results": kwargs.get("searxng_search_top_k_results", 2),  # 只返回前k个搜索结果
+            "searxng_search_output_type": kwargs.get("searxng_search_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for sms tool
+            "sms_nickname_mapping": kwargs.get("sms_nickname_mapping", "{}"),  # 关于人的代号对应的手机号，可以不输入手机号发送sms。键为代号值为手机号
+            "sms_username": kwargs.get("sms_username", ""),  # smsbao用户名
+            "sms_apikey": kwargs.get("sms_apikey", ""),  # smsbao
+            # for stt tool
+            "stt_api_key": kwargs.get("stt_api_key", ""),  # azure
+            "stt_api_region": kwargs.get("stt_api_region", ""),  # azure
+            "stt_recognition_language": kwargs.get("stt_recognition_language", "zh-CN"),  # 识别的语言类型 部分：en-US ja-JP ko-KR yue-CN zh-CN
+            # for tts tool
+            "tts_api_key": kwargs.get("tts_api_key", ""),  # azure
+            "tts_api_region": kwargs.get("tts_api_region", ""),  # azure
+            "tts_auto_detect": kwargs.get("tts_auto_detect", True),  # 是否自动检测语音的语言
+            "tts_speech_id": kwargs.get("tts_speech_id", "zh-CN-XiaozhenNeural"),  # 输出语音ID
+            # for summary tool
+            "summary_max_segment_length": kwargs.get("summary_max_segment_length", 2500),  # 每2500tokens分段，多段触发总结tool
+            # for terminal tool
+            "terminal_nsfc_filter": kwargs.get("terminal_nsfc_filter", True),  # 是否过滤llm输出的危险命令
+            "terminal_return_err_output": kwargs.get("terminal_return_err_output", True),  # 是否输出错误信息
+            "terminal_timeout": kwargs.get("terminal_timeout", 20),  # 允许命令最长执行时间
+            # for visual tool
+            "caption_api_key": kwargs.get("caption_api_key", ""),  # ali dashscope apikey
+            # for browser tool
+            "browser_use_summary": kwargs.get("browser_use_summary", True),  # 是否对返回结果使用tool功能
+            # for url-get tool
+            "url_get_use_summary": kwargs.get("url_get_use_summary", True),  # 是否对返回结果使用tool功能
+            # for wechat tool
+            "wechat_hot_reload": kwargs.get("wechat_hot_reload", True),  # 是否使用热重载的方式发送wechat
+            "wechat_cpt_path": kwargs.get("wechat_cpt_path", os.path.join(get_appdata_dir(), "itchat.pkl")),  # wechat 配置文件（`itchat.pkl`）
+            "wechat_send_group": kwargs.get("wechat_send_group", False),  # 是否向群组发送消息
+            "wechat_nickname_mapping": kwargs.get("wechat_nickname_mapping", "{}"),  # 关于人的代号映射关系。键为代号值为微信名（昵称、备注名均可）
+            # for wikipedia tool
+            "wikipedia_top_k_results": kwargs.get("wikipedia_top_k_results", 2),  # 只返回前k个搜索结果
             # for wolfram-alpha tool
             "wolfram_alpha_appid": kwargs.get("wolfram_alpha_appid", ""),
-            # for morning-news tool
-            "morning_news_api_key": kwargs.get("morning_news_api_key", ""),
-            # for visual_dl tool
-            "cuda_device": kwargs.get("cuda_device", "cpu"),
-            "think_depth": kwargs.get("think_depth", 3),
-            "arxiv_summary": kwargs.get("arxiv_summary", True),
-            "morning_news_use_llm": kwargs.get("morning_news_use_llm", False),
         }
 
     def _filter_tool_list(self, tool_list: list):
         valid_list = []
         for tool in tool_list:
-            if tool in get_all_tool_names():
+            if tool in main_tool_register.get_registered_tool_names():
                 valid_list.append(tool)
             else:
                 logger.warning("[tool] filter invalid tool: " + repr(tool))
         return valid_list
 
     def _reset_app(self) -> App:
-        tool_config = self._read_json()
-        app_kwargs = self._build_tool_kwargs(tool_config.get("kwargs", {}))
+        self.tool_config = self._read_json()
+        self.app_kwargs = self._build_tool_kwargs(self.tool_config.get("kwargs", {}))
 
         app = AppFactory()
-        app.init_env(**app_kwargs)
-
+        app.init_env(**self.app_kwargs)
         # filter not support tool
-        tool_list = self._filter_tool_list(tool_config.get("tools", []))
+        tool_list = self._filter_tool_list(self.tool_config.get("tools", []))
 
-        return app.create_app(tools_list=tool_list, **app_kwargs)
+        return app.create_app(tools_list=tool_list, **self.app_kwargs)

requirements-optional.txt

@@ -13,20 +13,30 @@ langid # language detect
 #install plugin
 dulwich
 
-# wechaty
-wechaty>=0.10.7
-wechaty_puppet>=0.4.23
-# pysilk_mod>=1.6.0 # needed by send voice only in wechaty
-
-# wechatmp wechatcom
+# wechatmp && wechatcom
 web.py
 wechatpy
 
 # chatgpt-tool-hub plugin
-chatgpt_tool_hub==0.4.6
+chatgpt_tool_hub==0.5.0
 
 # xunfei spark
 websocket-client==1.2.0
 
 # claude bot
 curl_cffi
+
+# tongyi qwen
+broadscope_bailian
+
+# google
+google-generativeai
+
+# linkai
+linkai>=0.0.3.5
+
+# dingtalk
+dingtalk_stream
+
+# zhipuai
+zhipuai>=2.0.1

requirements.txt

@@ -1,4 +1,4 @@
-openai>=0.27.8
+openai==0.27.8
 HTMLParser>=0.0.2
 PyQRCode>=1.2.1
 qrcode>=7.4.2

voice/ali/ali_api.py

@@ -0,0 +1,152 @@
+# coding=utf-8
+"""
+Author: chazzjimel
+Email: chazzjimel@gmail.com
+wechat：cheung-z-x
+
+Description:
+
+"""
+
+import json
+import time
+import requests
+import datetime
+import hashlib
+import hmac
+import base64
+import urllib.parse
+import uuid
+
+from common.log import logger
+from common.tmp_dir import TmpDir
+
+
+def text_to_speech_aliyun(url, text, appkey, token):
+    """
+    使用阿里云的文本转语音服务将文本转换为语音。
+
+    参数:
+    - url (str): 阿里云文本转语音服务的端点URL。
+    - text (str): 要转换为语音的文本。
+    - appkey (str): 您的阿里云appkey。
+    - token (str): 阿里云API的认证令牌。
+
+    返回值:
+    - str: 成功时输出音频文件的路径，否则为None。
+    """
+    headers = {
+        "Content-Type": "application/json",
+    }
+
+    data = {
+        "text": text,
+        "appkey": appkey,
+        "token": token,
+        "format": "wav"
+    }
+
+    response = requests.post(url, headers=headers, data=json.dumps(data))
+
+    if response.status_code == 200 and response.headers['Content-Type'] == 'audio/mpeg':
+        output_file = TmpDir().path() + "reply-" + str(int(time.time())) + "-" + str(hash(text) & 0x7FFFFFFF) + ".wav"
+
+        with open(output_file, 'wb') as file:
+            file.write(response.content)
+        logger.debug(f"音频文件保存成功，文件名：{output_file}")
+    else:
+        logger.debug("响应状态码: {}".format(response.status_code))
+        logger.debug("响应内容: {}".format(response.text))
+        output_file = None
+
+    return output_file
+
+
+class AliyunTokenGenerator:
+    """
+    用于生成阿里云服务认证令牌的类。
+
+    属性:
+    - access_key_id (str): 您的阿里云访问密钥ID。
+    - access_key_secret (str): 您的阿里云访问密钥秘密。
+    """
+
+    def __init__(self, access_key_id, access_key_secret):
+        self.access_key_id = access_key_id
+        self.access_key_secret = access_key_secret
+
+    def sign_request(self, parameters):
+        """
+        为阿里云服务签名请求。
+
+        参数:
+        - parameters (dict): 请求的参数字典。
+
+        返回值:
+        - str: 请求的签名签章。
+        """
+        # 将参数按照字典顺序排序
+        sorted_params = sorted(parameters.items())
+
+        # 构造待签名的查询字符串
+        canonicalized_query_string = ''
+        for (k, v) in sorted_params:
+            canonicalized_query_string += '&' + self.percent_encode(k) + '=' + self.percent_encode(v)
+
+        # 构造用于签名的字符串
+        string_to_sign = 'GET&%2F&' + self.percent_encode(canonicalized_query_string[1:])  # 使用GET方法
+
+        # 使用HMAC算法计算签名
+        h = hmac.new((self.access_key_secret + "&").encode('utf-8'), string_to_sign.encode('utf-8'), hashlib.sha1)
+        signature = base64.encodebytes(h.digest()).strip()
+
+        return signature
+
+    def percent_encode(self, encode_str):
+        """
+        对字符串进行百分比编码。
+
+        参数:
+        - encode_str (str): 要编码的字符串。
+
+        返回值:
+        - str: 编码后的字符串。
+        """
+        encode_str = str(encode_str)
+        res = urllib.parse.quote(encode_str, '')
+        res = res.replace('+', '%20')
+        res = res.replace('*', '%2A')
+        res = res.replace('%7E', '~')
+        return res
+
+    def get_token(self):
+        """
+        获取阿里云服务的令牌。
+
+        返回值:
+        - str: 获取到的令牌。
+        """
+        # 设置请求参数
+        params = {
+            'Format': 'JSON',
+            'Version': '2019-02-28',
+            'AccessKeyId': self.access_key_id,
+            'SignatureMethod': 'HMAC-SHA1',
+            'Timestamp': datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"),
+            'SignatureVersion': '1.0',
+            'SignatureNonce': str(uuid.uuid4()),  # 使用uuid生成唯一的随机数
+            'Action': 'CreateToken',
+            'RegionId': 'cn-shanghai'
+        }
+
+        # 计算签名
+        signature = self.sign_request(params)
+        params['Signature'] = signature
+
+        # 构造请求URL
+        url = 'http://nls-meta.cn-shanghai.aliyuncs.com/?' + urllib.parse.urlencode(params)
+
+        # 发送请求
+        response = requests.get(url)
+
+        return response.text

voice/ali/ali_voice.py

@@ -0,0 +1,81 @@
+# -*- coding: utf-8 -*-
+"""
+Author: chazzjimel
+Email: chazzjimel@gmail.com
+wechat：cheung-z-x
+
+Description:
+ali voice service
+
+"""
+import json
+import os
+import re
+import time
+
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from voice.voice import Voice
+from voice.ali.ali_api import AliyunTokenGenerator
+from voice.ali.ali_api import text_to_speech_aliyun
+from config import conf
+
+
+class AliVoice(Voice):
+    def __init__(self):
+        """
+        初始化AliVoice类，从配置文件加载必要的配置。
+        """
+        try:
+            curdir = os.path.dirname(__file__)
+            config_path = os.path.join(curdir, "config.json")
+            with open(config_path, "r") as fr:
+                config = json.load(fr)
+            self.token = None
+            self.token_expire_time = 0
+            # 默认复用阿里云千问的 access_key 和 access_secret
+            self.api_url = config.get("api_url")
+            self.app_key = config.get("app_key")
+            self.access_key_id = conf().get("qwen_access_key_id") or config.get("access_key_id")
+            self.access_key_secret = conf().get("qwen_access_key_secret") or config.get("access_key_secret")
+        except Exception as e:
+            logger.warn("AliVoice init failed: %s, ignore " % e)
+
+    def textToVoice(self, text):
+        """
+        将文本转换为语音文件。
+
+        :param text: 要转换的文本。
+        :return: 返回一个Reply对象，其中包含转换得到的语音文件或错误信息。
+        """
+        # 清除文本中的非中文、非英文和非基本字符
+        text = re.sub(r'[^\u4e00-\u9fa5\u3040-\u30FF\uAC00-\uD7AFa-zA-Z0-9'
+                      r'äöüÄÖÜáéíóúÁÉÍÓÚàèìòùÀÈÌÒÙâêîôûÂÊÎÔÛçÇñÑ，。！？,.]', '', text)
+        # 提取有效的token
+        token_id = self.get_valid_token()
+        fileName = text_to_speech_aliyun(self.api_url, text, self.app_key, token_id)
+        if fileName:
+            logger.info("[Ali] textToVoice text={} voice file name={}".format(text, fileName))
+            reply = Reply(ReplyType.VOICE, fileName)
+        else:
+            reply = Reply(ReplyType.ERROR, "抱歉，语音合成失败")
+        return reply
+
+    def get_valid_token(self):
+        """
+        获取有效的阿里云token。
+
+        :return: 返回有效的token字符串。
+        """
+        current_time = time.time()
+        if self.token is None or current_time >= self.token_expire_time:
+            get_token = AliyunTokenGenerator(self.access_key_id, self.access_key_secret)
+            token_str = get_token.get_token()
+            token_data = json.loads(token_str)
+            self.token = token_data["Token"]["Id"]
+            # 将过期时间减少一小段时间（例如5分钟），以避免在边界条件下的过期
+            self.token_expire_time = token_data["Token"]["ExpireTime"] - 300
+            logger.debug(f"新获取的阿里云token：{self.token}")
+        else:
+            logger.debug("使用缓存的token")
+        return self.token

voice/ali/config.json.template

@@ -0,0 +1,6 @@
+{
+    "api_url": "https://nls-gateway-cn-shanghai.aliyuncs.com/stream/v1/tts",
+    "app_key": "",
+    "access_key_id": "",
+    "access_key_secret": ""
+}

voice/factory.py

@@ -33,4 +33,13 @@ def create_voice(voice_type):
         from voice.elevent.elevent_voice import ElevenLabsVoice
 
         return ElevenLabsVoice()
+
+    elif voice_type == "linkai":
+        from voice.linkai.linkai_voice import LinkAIVoice
+
+        return LinkAIVoice()
+    elif voice_type == "ali":
+        from voice.ali.ali_voice import AliVoice
+
+        return AliVoice()
     raise RuntimeError

voice/linkai/linkai_voice.py

@@ -0,0 +1,83 @@
+"""
+google voice service
+"""
+import random
+import requests
+from voice import audio_convert
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf
+from voice.voice import Voice
+from common import const
+import os
+import datetime
+
+class LinkAIVoice(Voice):
+    def __init__(self):
+        pass
+
+    def voiceToText(self, voice_file):
+        logger.debug("[LinkVoice] voice file name={}".format(voice_file))
+        try:
+            url = conf().get("linkai_api_base", "https://api.link-ai.chat") + "/v1/audio/transcriptions"
+            headers = {"Authorization": "Bearer " + conf().get("linkai_api_key")}
+            model = None
+            if not conf().get("text_to_voice") or conf().get("voice_to_text") == "openai":
+                model = const.WHISPER_1
+            if voice_file.endswith(".amr"):
+                try:
+                    mp3_file = os.path.splitext(voice_file)[0] + ".mp3"
+                    audio_convert.any_to_mp3(voice_file, mp3_file)
+                    voice_file = mp3_file
+                except Exception as e:
+                    logger.warn(f"[LinkVoice] amr file transfer failed, directly send amr voice file: {format(e)}")
+            file = open(voice_file, "rb")
+            file_body = {
+                "file": file
+            }
+            data = {
+                "model": model
+            }
+            res = requests.post(url, files=file_body, headers=headers, data=data, timeout=(5, 60))
+            if res.status_code == 200:
+                text = res.json().get("text")
+            else:
+                res_json = res.json()
+                logger.error(f"[LinkVoice] voiceToText error, status_code={res.status_code}, msg={res_json.get('message')}")
+                return None
+            reply = Reply(ReplyType.TEXT, text)
+            logger.info(f"[LinkVoice] voiceToText success, text={text}, file name={voice_file}")
+        except Exception as e:
+            logger.error(e)
+            return None
+        return reply
+
+    def textToVoice(self, text):
+        try:
+            url = conf().get("linkai_api_base", "https://api.link-ai.chat") + "/v1/audio/speech"
+            headers = {"Authorization": "Bearer " + conf().get("linkai_api_key")}
+            model = const.TTS_1
+            if not conf().get("text_to_voice") or conf().get("text_to_voice") in ["openai", const.TTS_1, const.TTS_1_HD]:
+                model = conf().get("text_to_voice_model") or const.TTS_1
+            data = {
+                "model": model,
+                "input": text,
+                "voice": conf().get("tts_voice_id"),
+                "app_code": conf().get("linkai_app_code")
+            }
+            res = requests.post(url, headers=headers, json=data, timeout=(5, 120))
+            if res.status_code == 200:
+                tmp_file_name = "tmp/" + datetime.datetime.now().strftime('%Y%m%d%H%M%S') + str(random.randint(0, 1000)) + ".mp3"
+                with open(tmp_file_name, 'wb') as f:
+                    f.write(res.content)
+                reply = Reply(ReplyType.VOICE, tmp_file_name)
+                logger.info(f"[LinkVoice] textToVoice success, input={text}, model={model}, voice_id={data.get('voice')}")
+                return reply
+            else:
+                res_json = res.json()
+                logger.error(f"[LinkVoice] textToVoice error, status_code={res.status_code}, msg={res_json.get('message')}")
+                return None
+        except Exception as e:
+            logger.error(e)
+            # reply = Reply(ReplyType.ERROR, "遇到了一点小问题，请稍后再问我吧")
+            return None

voice/openai/openai_voice.py

@@ -9,7 +9,9 @@ from bridge.reply import Reply, ReplyType
 from common.log import logger
 from config import conf
 from voice.voice import Voice
-
+import requests
+from common import const
+import datetime, random
 
 class OpenaiVoice(Voice):
     def __init__(self):
@@ -24,6 +26,32 @@ class OpenaiVoice(Voice):
             reply = Reply(ReplyType.TEXT, text)
             logger.info("[Openai] voiceToText text={} voice file name={}".format(text, voice_file))
         except Exception as e:
-            reply = Reply(ReplyType.ERROR, str(e))
+            reply = Reply(ReplyType.ERROR, "我暂时还无法听清您的语音，请稍后再试吧~")
         finally:
             return reply
+
+
+    def textToVoice(self, text):
+        try:
+            api_base = conf().get("open_ai_api_base") or "https://api.openai.com/v1"
+            url = f'{api_base}/audio/speech'
+            headers = {
+                'Authorization': 'Bearer ' + conf().get("open_ai_api_key"),
+                'Content-Type': 'application/json'
+            }
+            data = {
+                'model': conf().get("text_to_voice_model") or const.TTS_1,
+                'input': text,
+                'voice': conf().get("tts_voice_id") or "alloy"
+            }
+            response = requests.post(url, headers=headers, json=data)
+            file_name = "tmp/" + datetime.datetime.now().strftime('%Y%m%d%H%M%S') + str(random.randint(0, 1000)) + ".mp3"
+            logger.debug(f"[OPENAI] text_to_Voice file_name={file_name}, input={text}")
+            with open(file_name, 'wb') as f:
+                f.write(response.content)
+            logger.info(f"[OPENAI] text_to_Voice success")
+            reply = Reply(ReplyType.VOICE, file_name)
+        except Exception as e:
+            logger.error(e)
+            reply = Reply(ReplyType.ERROR, "遇到了一点小问题，请稍后再问我吧")
+        return reply