feat:新加入定时任务功能

This commit is contained in:
ktianc 2023-07-03 23:49:23 +08:00
parent 7debee74e0
commit 9e59170a4b
56 changed files with 3217 additions and 198 deletions

View File

@ -19,13 +19,14 @@
Kinit 是一套全部开源的快速开发平台,毫无保留给个人及企业免费使用。
- 后端采用 Python 语言现代、快速(高性能) [FastAPI](https://fastapi.tiangolo.com/zh/) 异步框架 + [SQLAlchemy](https://www.sqlalchemy.org/) 异步操作 [MySQL](https://www.mysql.com/) 数据库。
- PC端采用 [vue-element-plus-admin](https://gitee.com/kailong110120130/vue-element-plus-admin) 、[Vue3](https://cn.vuejs.org/guide/introduction.html)、[Element Plus](https://element-plus.gitee.io/zh-CN/guide/design.html)、[TypeScript](https://www.tslang.cn/)等主流技术开发。
- 移动端采用 [uni-app](https://uniapp.dcloud.net.cn/component/)[Vue2](https://v2.cn.vuejs.org/v2/guide/)[uView 2](https://www.uviewui.com/components/intro.html)为主要技术开发
- 新加入 [Typer](https://typer.tiangolo.com/) 命令行应用,简单化数据初始化,数据表模型迁移。
- 后端采用 Python 语言现代、快速(高性能) [FastAPI](https://fastapi.tiangolo.com/zh/) 异步框架 + [SQLAlchemy](https://www.sqlalchemy.org/) 异步操作 [MySQL](https://www.mysql.com/) 数据库;
- PC端采用 [vue-element-plus-admin](https://gitee.com/kailong110120130/vue-element-plus-admin) 、[Vue3](https://cn.vuejs.org/guide/introduction.html)、[Element Plus](https://element-plus.gitee.io/zh-CN/guide/design.html)、[TypeScript](https://www.tslang.cn/)等主流技术开发;
- 移动端采用 [uni-app](https://uniapp.dcloud.net.cn/component/)[Vue2](https://v2.cn.vuejs.org/v2/guide/)[uView 2](https://www.uviewui.com/components/intro.html)为主要技术开发;
- 后端加入 [Typer](https://typer.tiangolo.com/) 命令行应用,简单化数据初始化,数据表模型迁移等操作;
- 新加入定时任务功能,采用 [APScheduler](https://github.com/agronholm/apscheduler) 定时任务框架 + [Redis](https://redis.io/) 消息队列 + [MongoDB](https://www.mongodb.com/) 持久存储;
- 权限认证使用[(哈希)密码和 JWT Bearer 令牌的 OAuth2](https://fastapi.tiangolo.com/zh/tutorial/security/oauth2-jwt/),支持多终端认证系统。
- 支持加载动态权限菜单,多方式轻松权限控制,按钮级别权限控制。
- 已加入常见的`Redis``MYSQL``MongoDB`数据库异步操作。
- 已加入常见的 [MySQL](https://www.mysql.com/) + [MongoDB](https://www.mongodb.com/) + [Redis](https://redis.io/) 数据库异步操作。
- 开箱即用的中后台解决方案,可以用来作为新项目的启动模版,也可用于学习参考。并且时刻关注着最新技术动向,尽可能的第一时间更新。
- 与 [vue-element-plus-admin](https://gitee.com/kailong110120130/vue-element-plus-admin) 前端框架时刻保持同步更新。
@ -119,6 +120,8 @@ github地址https://github.com/vvandk/kinit
- [x] 命令行操作:新加入 `Typer` 命令行应用,简单化数据初始化,数据表模型迁移。
- [x] 定时任务:在线操作(添加、修改、删除)任务调度包含查看任务执行结果日志。
## 移动端内置功能
- [x] 登录认证:支持用户使用手机号+密码方式登录,微信手机号一键登录方式。
@ -133,10 +136,9 @@ github地址https://github.com/vvandk/kinit
## TODO
- [ ] 考虑支持多机部署方案,如果接口使用多机,那么用户是否支持统一认证
- [ ] **自动化编排服务使用docker-compose部署项目**
- [ ] **定时任务:定时执行数据库备份**
- [ ] **可视化低代码表单接入低代码表单https://vform666.com/vform3.html?from=element_plus**
- [ ] 多租户方案
- [ ] 自动化编排服务使用docker-compose部署项目
- [ ] 可视化低代码表单:接入低代码表单,[vform3](https://vform666.com/vform3.html?from=element_plus)
## 前序准备
@ -145,7 +147,8 @@ github地址https://github.com/vvandk/kinit
- [Python3](https://www.python.org/downloads/windows/):熟悉 python3 基础语法
- [FastAPI](https://fastapi.tiangolo.com/zh/) - 熟悉后台接口 Web 框架.
- [Typer](https://typer.tiangolo.com/) - 熟悉命令行工具的使用
- [MySQL](https://www.mysql.com/) 和 [MongoDB](https://www.mongodb.com/) - 熟悉数据存储数据库
- [MySQL](https://www.mysql.com/) 和 [MongoDB](https://www.mongodb.com/) 和 [Redis](https://redis.io/) - 熟悉数据存储数据库
- [iP查询接口文档](https://user.ip138.com/ip/doc)IP查询第三方服务有1000次的免费次数
### PC端
@ -157,30 +160,23 @@ github地址https://github.com/vvandk/kinit
- [Vue-Router-Next](https://gitee.com/link?target=https%3A%2F%2Fnext.router.vuejs.org%2F) - 熟悉 vue-router 基本使用
- [Element-Plus](https://gitee.com/link?target=https%3A%2F%2Felement-plus.org%2F) - element-plus 基本使用
- [Mock.js](https://gitee.com/link?target=https%3A%2F%2Fgithub.com%2Fnuysoft%2FMock) - mockjs 基本语法
- [vue3-json-viewer](https://gitee.com/isfive/vue3-json-viewer)简单易用的json内容展示组件,适配vue3和vite。
- [SortableJS/vue.draggable.next](https://github.com/SortableJS/vue.draggable.next)Vue 组件 Vue.js 3.0 允许拖放和与视图模型数组同步。
- [高德地图API (amap.com)](https://lbs.amap.com/api/jsapi-v2/guide/webcli/map-vue1):地图 JSAPI 2.0 是高德开放平台免费提供的第四代 Web 地图渲染引擎。
### 移动端
- [uni-app](https://uniapp.dcloud.net.cn/component/) - 熟悉 uni-app 基本语法
- [Vue2](https://v2.cn.vuejs.org/v2/guide/) - 熟悉 Vue 基础语法
- [uView UI 2](https://www.uviewui.com/components/intro.html)uView UI 组件的基本使用
### 依赖包
#### PC端
- [vue3-json-viewer](https://gitee.com/isfive/vue3-json-viewer)简单易用的json内容展示组件,适配vue3和vite。
- [vue3-slide-verify](https://github.com/monoplasty/vue3-slide-verify):滑块验证码插件 vue3 + typescript。
- [SortableJS/vue.draggable.next](https://github.com/SortableJS/vue.draggable.next)Vue 组件 Vue.js 3.0 允许拖放和与视图模型数组同步。
- [高德地图API (amap.com)](https://lbs.amap.com/api/jsapi-v2/guide/webcli/map-vue1):地图 JSAPI 2.0 是高德开放平台免费提供的第四代 Web 地图渲染引擎, 以 WebGL 为主要绘图手段,本着“更轻、更快、更易用”的服务原则,广泛采用了各种前沿技术,交互体验、视觉体验大幅提升,同时提供了众多新增能力和特性。
#### 移动端
- [uni-read-pages](https://github.com/SilurianYang/uni-read-pages) :自动读取 `pages.json` 所有配置。
- [uni-simple-router](https://hhyang.cn/v2/start/quickstart.html) 在uni-app中使用vue-router的方式进行跳转路由路由拦截。
#### 后端
### 定时任务
- [iP查询接口文档](https://user.ip138.com/ip/doc)IP查询第三方服务有1000次的免费次数
- [Python3](https://www.python.org/downloads/windows/) -熟悉 python3 基础语法
- [APScheduler](https://github.com/agronholm/apscheduler) - 熟悉定时任务框架
- [MongoDB](https://www.mongodb.com/) 和 [Redis](https://redis.io/) - 熟悉数据存储数据库
## 安装和使用
@ -323,6 +319,12 @@ Redis (推荐使用最新稳定版)
# 微信小程序配置
wx_server_app_id
wx_server_app_secret
# 邮箱配置
email_access
email_password
email_server
email_port
```
6. 启动
@ -354,6 +356,51 @@ pnpm run dev
pnpm run build:pro
```
### 定时任务
1. 安装依赖
```
# 安装依赖库
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
# 第三方源:
1. 阿里源: https://mirrors.aliyun.com/pypi/simple/
```
2. 修改项目数据库配置信息
`application/config` 目录中
- development.py开发环境
- production.py生产环境
```python
"""
MongoDB 数据库配置
与接口是同一个数据库
"""
MONGO_DB_NAME = "数据库名称"
MONGO_DB_URL = f"mongodb://用户名:密码@地址:端口/?authSource={MONGO_DB_NAME}"
"""
Redis 数据库配置
与接口是同一个数据库
"""
REDIS_DB_URL = "redis://:密码@地址:端口/数据库名称"
```
3. 启动
```
python3 main.py
```
### 访问项目
- 访问地址http://localhost:5000 (默认为此地址,如有修改请按照配置文件)

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

View File

@ -0,0 +1,33 @@
import request from '@/config/axios'
export const getTaskListApi = (params: any): Promise<IResponse> => {
return request.get({ url: '/vadmin/system/tasks', params })
}
export const addTaskListApi = (data: any): Promise<IResponse> => {
return request.post({ url: '/vadmin/system/tasks', data })
}
export const delTaskListApi = (dataId: string): Promise<IResponse> => {
return request.delete({ url: `/vadmin/system/tasks?_id=${dataId}` })
}
export const putTaskListApi = (dataId: string, data: any): Promise<IResponse> => {
return request.put({ url: `/vadmin/system/tasks?_id=${dataId}`, data })
}
export const getTaskApi = (dataId: string): Promise<IResponse> => {
return request.get({ url: `/vadmin/system/task?_id=${dataId}` })
}
export const getTaskGroupOptionsApi = (): Promise<IResponse> => {
return request.get({ url: '/vadmin/system/task/group/options' })
}
export const getTaskRecordListApi = (params: any): Promise<IResponse> => {
return request.get({ url: '/vadmin/system/task/records', params })
}
export const runOnceTaskApi = (dataId: string): Promise<IResponse> => {
return request.post({ url: `/vadmin/system/task?_id=${dataId}` })
}

View File

@ -14,7 +14,7 @@ defineProps({
</script>
<template>
<ElCard :class="[prefixCls]" shadow="never">
<ElCard :class="[prefixCls]" shadow="never" class="!border-0">
<template v-if="title" #header>
<div class="flex items-center">
<span class="text-16px font-700">{{ title }}</span>

View File

@ -6,3 +6,9 @@
width: 100% !important;
}
// 解决element-plus 中的日期组件宽度无法设置100%的问题
.el-date-editor .el-input__wrapper {
width: 100% !important;
}

View File

@ -64,4 +64,6 @@
--app-footer-height: 50px;
--transition-time-02: 0.2s;
--app-content-bg-color-new: #ffffff;
}

View File

@ -8,8 +8,8 @@
export interface DictDetail {
label: string
value: string
disabled: boolean
is_default: boolean
disabled?: boolean
is_default?: boolean
}
export const selectDictLabel = (datas: DictDetail[], value: string) => {
if (!value) {

View File

@ -0,0 +1,20 @@
<script setup lang="ts">
import { PropType } from 'vue'
import { Descriptions } from '@/components/Descriptions'
import { DescriptionsSchema } from '@/types/descriptions'
defineProps({
currentRow: {
type: Object as PropType<Nullable<any>>,
default: () => null
},
detailSchema: {
type: Array as PropType<DescriptionsSchema[]>,
default: () => []
}
})
</script>
<template>
<Descriptions :schema="detailSchema" :data="currentRow || {}" />
</template>

View File

@ -0,0 +1,115 @@
import { FormSchema } from '@/types/form'
import { TableColumn } from '@/types/table'
import { reactive } from 'vue'
export const columns = reactive<TableColumn[]>([
{
field: 'job_id',
label: '任务编号',
show: true,
disabled: true,
width: '240px',
span: 24
},
{
field: 'name',
label: '任务名称',
show: true,
disabled: true,
span: 24
},
{
field: 'group',
label: '任务分组',
show: true,
span: 24
},
{
field: 'job_class',
label: '调用目标',
show: true,
span: 24
},
{
field: 'exec_strategy',
label: '执行策略',
show: true,
span: 24
},
{
field: 'expression',
label: '表达式',
show: true,
span: 24
},
{
field: 'start_time',
label: '开始执行时间',
show: false,
width: '200px',
span: 24
},
{
field: 'end_time',
label: '执行完成时间',
width: '200px',
show: true,
span: 24
},
{
field: 'process_time',
label: '耗时(秒)',
width: '110px',
show: true,
span: 24
},
{
field: 'retval',
label: '任务返回值',
show: true,
span: 24
},
{
field: 'exception',
label: '异常信息',
show: false,
span: 24
},
{
field: 'traceback',
label: '堆栈跟踪',
show: false,
width: '100px',
span: 24
},
{
field: 'action',
width: '100px',
label: '操作',
show: true,
disabled: true,
span: 24
}
])
export const searchSchema = reactive<FormSchema[]>([
{
field: 'job_id',
label: '任务编号',
component: 'Input',
componentProps: {
clearable: true,
style: {
width: '240px'
}
}
},
{
field: 'name',
label: '任务名称',
component: 'Input',
componentProps: {
clearable: true
}
}
])

View File

@ -0,0 +1,138 @@
<script lang="ts">
export default {
name: 'SystemRecordTask'
}
</script>
<script setup lang="ts">
import { ContentWrap } from '@/components/ContentWrap'
import { Table } from '@/components/Table'
import { getTaskRecordListApi } from '@/api/vadmin/system/task'
import { useTable } from '@/hooks/web/useTable'
import { columns, searchSchema } from './components/task.data'
import { ref, watch, nextTick } from 'vue'
import { ElButton, ElRow } from 'element-plus'
import { RightToolbar } from '@/components/RightToolbar'
import { Dialog } from '@/components/Dialog'
import Detail from './components/Detail.vue'
import { Search } from '@/components/Search'
import { useCache } from '@/hooks/web/useCache'
import { useRouter } from 'vue-router'
import { DictDetail, selectDictLabel } from '@/utils/dict'
import { useDictStore } from '@/store/modules/dict'
import { FormSetPropsType } from '@/types/form'
const { wsCache } = useCache()
const { currentRoute } = useRouter()
const job_id = currentRoute.value.query.job_id
const { register, elTableRef, tableObject, methods } = useTable({
getListApi: getTaskRecordListApi,
response: {
data: 'data',
count: 'count'
}
})
tableObject.params = { job_id: job_id }
const dialogVisible = ref(false)
const dialogTitle = ref('')
const execStrategyOptions = ref<DictDetail[]>([])
const searchSetSchemaList = ref([] as FormSetPropsType[])
if (typeof job_id === 'string') {
searchSetSchemaList.value.push({
field: 'job_id',
path: 'value',
value: job_id
})
}
const getOptions = async () => {
const dictStore = useDictStore()
const dictOptions = await dictStore.getDictObj(['vadmin_system_task_exec_strategy'])
execStrategyOptions.value = dictOptions.vadmin_system_task_exec_strategy
}
getOptions()
const view = (row: any) => {
dialogTitle.value = '操作记录'
tableObject.currentRow = row
dialogVisible.value = true
}
const { getList, setSearchParams } = methods
getList()
const tableSize = ref('default')
watch(tableSize, (val) => {
tableSize.value = val
})
const route = useRouter()
const cacheTableHeadersKey = route.currentRoute.value.fullPath
watch(
columns,
async (val) => {
wsCache.set(cacheTableHeadersKey, JSON.stringify(val))
await nextTick()
elTableRef.value?.doLayout()
},
{
deep: true
}
)
</script>
<template>
<ContentWrap>
<Search
:schema="searchSchema"
:setSchemaList="searchSetSchemaList"
@search="setSearchParams"
@reset="setSearchParams"
/>
<div class="mb-8px flex justify-between">
<ElRow />
<RightToolbar
@get-list="getList"
v-model:table-size="tableSize"
v-model:columns="columns"
:cache-table-headers-key="cacheTableHeadersKey"
/>
</div>
<Table
v-model:limit="tableObject.limit"
v-model:page="tableObject.page"
:columns="columns"
:data="tableObject.tableData"
:loading="tableObject.loading"
:selection="false"
:size="tableSize"
:border="true"
:pagination="{
total: tableObject.count
}"
@register="register"
>
<template #exec_strategy="{ row }">
{{ selectDictLabel(execStrategyOptions, row.exec_strategy) }}
</template>
<template #action="{ row }">
<ElButton type="primary" link size="small" @click="view(row)"> 详情 </ElButton>
</template>
</Table>
<Dialog v-model="dialogVisible" :title="dialogTitle" width="900px">
<Detail :detail-schema="columns" :current-row="tableObject.currentRow" />
</Dialog>
</ContentWrap>
</template>

View File

@ -0,0 +1,112 @@
<script setup lang="ts">
import { Form } from '@/components/Form'
import { useForm } from '@/hooks/web/useForm'
import { PropType, reactive, watch } from 'vue'
import { useValidator } from '@/hooks/web/useValidator'
import { schema } from './task.data'
import { DictDetail } from '@/utils/dict'
import { ElRadioGroup, ElRadio } from 'element-plus'
const { required } = useValidator()
const props = defineProps({
currentRow: {
type: Object as PropType<Nullable<any>>,
default: () => null
},
execStrategyOptions: {
type: Object as PropType<DictDetail[]>,
default: () => null
},
taskGroupOptions: {
type: Object as PropType<DictDetail[]>,
default: () => null
}
})
const rules = reactive({
name: [required()],
exec_strategy: [required()],
expression: [{ required: true, message: '请填写表达式', trigger: 'blur' }],
job_class: [required()]
})
const { register, methods, elFormRef } = useForm({
schema: schema
})
watch(
() => props.currentRow,
(currentRow) => {
if (!currentRow) return
const { setValues } = methods
setValues(currentRow)
},
{
deep: true,
immediate: true
}
)
watch(
() => props.execStrategyOptions,
(execStrategyOptions) => {
if (!execStrategyOptions) return
const { setSchema } = methods
setSchema([
{
field: 'exec_strategy',
path: 'componentProps.options',
value: execStrategyOptions
}
])
},
{
deep: true,
immediate: true
}
)
watch(
() => props.taskGroupOptions,
(taskGroupOptions) => {
if (!taskGroupOptions) return
const { setSchema } = methods
setSchema([
{
field: 'group',
path: 'componentProps.options',
value: taskGroupOptions
}
])
},
{
deep: true,
immediate: true
}
)
const handleChange = (form) => {
form['start_date'] = null
form['end_date'] = null
form['expression'] = null
elFormRef.value?.clearValidate('expression')
}
defineExpose({
elFormRef,
getFormData: methods.getFormData
})
</script>
<template>
<Form :rules="rules" @register="register">
<template #exec_strategy="form">
<ElRadioGroup v-model="form['exec_strategy']" @change="handleChange(form)">
<ElRadio v-for="(item, $index) in execStrategyOptions" :key="$index" :label="item.value">{{
item.label
}}</ElRadio>
</ElRadioGroup>
</template>
</Form>
</template>

View File

@ -0,0 +1,326 @@
import { FormSchema } from '@/types/form'
import { TableColumn } from '@/types/table'
import { reactive } from 'vue'
export const columns = reactive<TableColumn[]>([
{
field: '_id',
label: '任务编号',
show: true,
disabled: true,
width: '240px',
span: 24
},
{
field: 'name',
label: '任务名称',
width: '200px',
show: true,
disabled: true,
span: 24
},
{
field: 'group',
label: '任务分组',
show: true,
span: 24
},
{
field: 'job_class',
label: '调用目标',
show: true,
span: 24
},
{
field: 'exec_strategy',
label: '执行策略',
component: 'Radio',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
}
}
},
{
field: 'expression',
label: '表达式',
show: true,
span: 24
},
{
field: 'is_active',
label: '任务状态',
show: true,
width: '100px',
span: 24
},
{
field: 'last_run_datetime',
label: '最近一次执行时间',
show: true,
width: '200px',
span: 24
},
{
field: 'remark',
label: '任务备注',
show: true,
span: 24
},
{
field: 'create_datetime',
label: '创建时间',
show: true,
width: '200px',
span: 24
},
{
field: 'action',
label: '操作',
show: true,
disabled: false,
width: '240px',
span: 24
}
])
export const schema = reactive<FormSchema[]>([
{
field: 'name',
label: '任务名称',
component: 'Input',
colProps: {
span: 12
},
componentProps: {
style: {
width: '100%'
}
}
},
{
field: 'group',
label: '任务分组',
colProps: {
span: 12
},
component: 'Select',
componentProps: {
style: {
width: '100%'
},
allowCreate: true,
filterable: true,
defaultFirstOption: true,
placeholder: '请选择任务分组,支持直接输入添加'
}
},
{
field: 'job_class',
label: '调用目标',
component: 'Input',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
},
placeholder:
'调用示例test.main.Test("kinit", 1314, True);参数仅支持字符串,整数,浮点数,布尔类型。'
}
},
{
field: 'exec_strategy',
label: '执行策略',
colProps: {
span: 24
},
component: 'Radio',
componentProps: {
style: {
width: '100%'
}
},
value: 'interval'
},
{
field: 'expression',
label: '表达式',
component: 'Input',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
},
placeholder:
'interval 表达式,五位,分别为:秒 分 时 天 周例如10 * * * * 表示每隔 10 秒执行一次任务。'
},
ifshow: (values) => values.exec_strategy === 'interval'
},
{
field: 'expression',
label: '表达式',
component: 'Input',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
},
placeholder: 'cron 表达式,六位或七位,分别表示秒、分钟、小时、天、月、星期几、年'
},
ifshow: (values) => values.exec_strategy === 'cron'
},
{
field: 'expression',
label: '执行时间',
component: 'DatePicker',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
},
type: 'datetime',
format: 'YYYY-MM-DD HH:mm:ss',
valueFormat: 'YYYY-MM-DD HH:mm:ss'
},
ifshow: (values) => values.exec_strategy === 'date'
},
{
field: 'start_date',
label: '开始时间',
colProps: {
span: 12
},
component: 'DatePicker',
componentProps: {
style: {
width: '100%'
},
type: 'datetime',
format: 'YYYY-MM-DD HH:mm:ss',
valueFormat: 'YYYY-MM-DD HH:mm:ss'
},
ifshow: (values) => values.exec_strategy !== 'date'
},
{
field: 'end_date',
label: '结束时间',
colProps: {
span: 12
},
component: 'DatePicker',
componentProps: {
style: {
width: '100%'
},
type: 'datetime',
format: 'YYYY-MM-DD HH:mm:ss',
valueFormat: 'YYYY-MM-DD HH:mm:ss'
},
ifshow: (values) => values.exec_strategy !== 'date'
},
{
field: 'is_active',
label: '任务状态',
colProps: {
span: 8
},
component: 'Radio',
componentProps: {
style: {
width: '100%'
},
options: [
{
label: '正常',
value: true
},
{
label: '停用',
value: false
}
]
},
value: true
},
{
field: '',
label: '',
colProps: {
span: 16
},
component: 'Text',
value:
'创建或更新任务完成后,如果任务状态与设置的不符,请尝试刷新数据或查看调度日志,任务状态可能会有延迟(几秒)。'
},
{
field: 'remark',
label: '备注说明',
component: 'Input',
colProps: {
span: 24
},
componentProps: {
style: {
width: '100%'
},
maxlength: '1000',
showWordLimit: true,
type: 'textarea',
rows: '3'
}
},
{
field: 'active',
label: ' ',
colProps: {
span: 24
}
}
])
export const searchSchema = reactive<FormSchema[]>([
{
field: 'name',
label: '任务名称',
component: 'Input',
componentProps: {
clearable: true,
style: {
width: '214px'
}
}
},
{
field: '_id',
label: '任务编号',
component: 'Input',
componentProps: {
clearable: true,
style: {
width: '214px'
}
}
},
{
field: 'group',
label: '任务分组',
component: 'Select',
componentProps: {
style: {
width: '214px'
},
options: []
}
}
])

View File

@ -0,0 +1,289 @@
<script lang="ts">
export default {
name: 'HelpTask'
}
</script>
<script setup lang="ts">
import { ContentWrap } from '@/components/ContentWrap'
import { Table } from '@/components/Table'
import {
getTaskListApi,
addTaskListApi,
delTaskListApi,
putTaskListApi,
getTaskApi,
getTaskGroupOptionsApi,
runOnceTaskApi
} from '@/api/vadmin/system/task'
import { useTable } from '@/hooks/web/useTable'
import { columns, searchSchema } from './components/task.data'
import { ref, watch, nextTick, unref } from 'vue'
import { ElRow, ElCol, ElButton, ElSwitch, ElMessageBox, ElMessage } from 'element-plus'
import { RightToolbar } from '@/components/RightToolbar'
import { useDictStore } from '@/store/modules/dict'
import { selectDictLabel, DictDetail } from '@/utils/dict'
import { FormSetPropsType } from '@/types/form'
import { Search } from '@/components/Search'
import { useI18n } from '@/hooks/web/useI18n'
import { Dialog } from '@/components/Dialog'
import { useCache } from '@/hooks/web/useCache'
import { useRouter } from 'vue-router'
import Write from './components/Write.vue'
const { wsCache } = useCache()
const { t } = useI18n()
const { push } = useRouter()
const { register, elTableRef, tableObject, methods } = useTable({
getListApi: getTaskListApi,
delListApi: delTaskListApi,
response: {
data: 'data',
count: 'count'
}
})
const { getList, setSearchParams } = methods
const tableSize = ref('default')
watch(tableSize, (val) => {
tableSize.value = val
})
const route = useRouter()
const cacheTableHeadersKey = route.currentRoute.value.fullPath
watch(
columns,
async (val) => {
wsCache.set(cacheTableHeadersKey, JSON.stringify(val))
await nextTick()
elTableRef.value?.doLayout()
},
{
deep: true
}
)
const dialogVisible = ref(false)
const dialogTitle = ref('')
const loading = ref(false)
const actionType = ref('')
const execStrategyOptions = ref<DictDetail[]>([])
const taskGroupOptions = ref<DictDetail[]>([])
const searchSetSchemaList = ref([] as FormSetPropsType[])
const getOptions = async () => {
const dictStore = useDictStore()
const dictOptions = await dictStore.getDictObj(['vadmin_system_task_exec_strategy'])
execStrategyOptions.value = dictOptions.vadmin_system_task_exec_strategy
const res = await getTaskGroupOptionsApi()
taskGroupOptions.value = res.data.map((item) => {
return {
label: item.value,
value: item.value
}
})
searchSetSchemaList.value.push({
field: 'group',
path: 'componentProps.options',
value: taskGroupOptions.value
})
}
getOptions()
//
const toRecord = (row: any) => {
if (row) {
console.log(row)
push(`/system/record/task?job_id=${row._id}`)
} else {
push(`/system/record/task`)
}
}
//
const addAction = async () => {
dialogTitle.value = '添加定时任务'
tableObject.currentRow = null
dialogVisible.value = true
actionType.value = 'add'
}
//
const updateAction = async (row: any) => {
const res = await getTaskApi(row._id)
if (res) {
dialogTitle.value = '编辑定时任务'
tableObject.currentRow = res.data
dialogVisible.value = true
actionType.value = 'edit'
}
}
//
const delData = async (row: any) => {
const { delListApi } = methods
await delListApi(false, row._id)
}
//
const runOnceTask = async (row: any) => {
ElMessageBox.confirm('是否确认立即执行一次任务', t('common.delWarning'), {
confirmButtonText: t('common.delOk'),
cancelButtonText: t('common.delCancel'),
type: 'warning'
}).then(async () => {
const res = await runOnceTaskApi(row._id)
if (res) {
if (res.data > 0) {
ElMessage.success('任务成功被消费者接收!')
} else {
ElMessage.error('执行失败,未有消费者接收任务,请检查定时任务程序状态!')
}
}
})
}
const writeRef = ref<ComponentRef<typeof Write>>()
//
const save = async () => {
const write = unref(writeRef)
await write?.elFormRef?.validate(async (isValid) => {
if (isValid) {
loading.value = true
let data = await write?.getFormData()
try {
const res = ref({} as any)
if (data?.exec_strategy === 'date') {
data.start_date = null
data.end_date = null
}
if (actionType.value === 'add') {
res.value = await addTaskListApi(data)
if (res.value) {
dialogVisible.value = false
const result = res.value.data
if (result.is_active) {
if (result.subscribe_number > 0) {
ElMessage.success('创建成功,任务成功被消费者接收!')
} else {
ElMessage.info('创建成功,未有消费者接收任务,请检查定时任务程序状态!')
}
} else {
ElMessage.success('创建成功!')
}
getList()
}
} else if (actionType.value === 'edit') {
res.value = await putTaskListApi(data?._id, data)
if (res.value) {
dialogVisible.value = false
const result = res.value.data
if (result.is_active) {
if (result.subscribe_number > 0) {
ElMessage.success('更新成功,任务已重新被消费者接收!')
} else {
ElMessage.info('更新成功,未有消费者接收任务,请检查定时任务程序状态!')
}
} else {
ElMessage.success('更新成功!')
}
getList()
}
}
} finally {
loading.value = false
}
}
})
}
// cron
const generateCronExpression = () => {
ElMessage.info('下一个版本更新')
}
getList()
</script>
<template>
<ContentWrap>
<Search
:schema="searchSchema"
:setSchemaList="searchSetSchemaList"
@search="setSearchParams"
@reset="setSearchParams"
/>
<div class="mb-8px flex justify-between">
<ElRow>
<ElCol :span="1.5">
<ElButton type="primary" @click="addAction">添加定时任务</ElButton>
<ElButton type="primary" @click="toRecord(null)">调度日志</ElButton>
<ElButton type="primary" @click="generateCronExpression">生成 Cron 表达式</ElButton>
</ElCol>
</ElRow>
<RightToolbar
@get-list="getList"
v-model:table-size="tableSize"
v-model:columns="columns"
:cache-table-headers-key="cacheTableHeadersKey"
/>
</div>
<Table
v-model:limit="tableObject.limit"
v-model:page="tableObject.page"
:columns="columns"
:data="tableObject.tableData"
:loading="tableObject.loading"
:selection="false"
:size="tableSize"
:border="true"
:pagination="{
total: tableObject.count
}"
@register="register"
>
<template #is_active="{ row }">
<ElSwitch :value="row.is_active" size="small" disabled />
</template>
<template #exec_strategy="{ row }">
{{ selectDictLabel(execStrategyOptions, row.exec_strategy) }}
</template>
<template #action="{ row }">
<ElButton type="primary" link size="small" @click="updateAction(row)">
{{ t('exampleDemo.edit') }}
</ElButton>
<ElButton type="primary" link size="small" @click="toRecord(row)"> 调度日志 </ElButton>
<ElButton type="primary" link size="small" @click="runOnceTask(row)"> 执行一次 </ElButton>
<ElButton type="danger" link size="small" @click="delData(row)">
{{ t('exampleDemo.del') }}
</ElButton>
</template>
</Table>
<Dialog v-model="dialogVisible" :title="dialogTitle" width="800px">
<Write
ref="writeRef"
:current-row="tableObject.currentRow"
:exec-strategy-options="execStrategyOptions"
:task-group-options="taskGroupOptions"
/>
<template #footer>
<ElButton type="primary" :loading="loading" @click="save">
{{ t('exampleDemo.save') }}
</ElButton>
<ElButton @click="dialogVisible = false">{{ t('dialogDemo.close') }}</ElButton>
</template>
</Dialog>
</ContentWrap>
</template>

View File

@ -11,7 +11,7 @@ from fastapi.security import OAuth2PasswordBearer
"""
系统版本
"""
VERSION = "1.8.4"
VERSION = "1.9.0"
"""安全警告: 不要在生产中打开调试运行!"""
DEBUG = True
@ -55,7 +55,7 @@ ACCESS_TOKEN_EXPIRE_MINUTES = 1440
"""refresh_token 过期时间用于刷新token使用两天"""
REFRESH_TOKEN_EXPIRE_MINUTES = 1440 * 2
"""access_token 缓存时间用于刷新token使用30分钟"""
ACCESS_TOKEN_CACHE_MINUTES = 60 * 2
ACCESS_TOKEN_CACHE_MINUTES = 30
"""
挂载临时文件目录并添加路由访问此路由不会在接口文档中显示
@ -134,3 +134,9 @@ MIDDLEWARES = [
"core.middleware.register_demo_env_middleware" if DEMO else None,
"core.middleware.register_jwt_refresh_middleware"
]
"""
定时任务配置
"""
# 发布/订阅通道,与定时任务程序相互关联,请勿随意更改
SUBSCRIBE = 'kinit_queue'

View File

@ -8,7 +8,6 @@
from aioredis import Redis
from fastapi import APIRouter, Depends, Body, UploadFile, Request
from sqlalchemy.orm import joinedload
from core.database import redis_getter
from utils.response import SuccessResponse, ErrorResponse
from . import schemas, crud, models

View File

@ -7,13 +7,14 @@
# @desc : 数据库 增删改查操作
import random
from typing import List
# sqlalchemy 查询操作https://segmentfault.com/a/1190000016767008
# sqlalchemy 关联查询https://www.jianshu.com/p/dfad7c08c57a
# sqlalchemy 关联查询详细https://blog.csdn.net/u012324798/article/details/103940527
from motor.motor_asyncio import AsyncIOMotorDatabase
from sqlalchemy.ext.asyncio import AsyncSession
from . import models, schemas
from core.crud import DalBase
from core.mongo_manage import MongoManage
class LoginRecordDal(DalBase):
@ -72,3 +73,9 @@ class SMSSendRecordDal(DalBase):
def __init__(self, db: AsyncSession):
super(SMSSendRecordDal, self).__init__(db, models.VadminSMSSendRecord, schemas.SMSSendRecordSimpleOut)
class OperationRecordDal(MongoManage):
def __init__(self, db: AsyncIOMotorDatabase):
super(OperationRecordDal, self).__init__(db, "operation_record", schemas.OperationRecordSimpleOut)

View File

@ -1,3 +1,3 @@
from .login import LoginRecord, LoginRecordSimpleOut
from .sms import SMSSendRecord, SMSSendRecordSimpleOut
from .operation import OpertionRecord, OpertionRecordSimpleOut
from .operation import OperationRecord, OperationRecordSimpleOut

View File

@ -11,9 +11,10 @@
from typing import Optional, List
from pydantic import BaseModel
from core.data_types import DatetimeStr, ObjectIdStr
class OpertionRecord(BaseModel):
class OperationRecord(BaseModel):
telephone: Optional[str] = None
user_id: Optional[str] = None
user_name: Optional[str] = None
@ -29,10 +30,10 @@ class OpertionRecord(BaseModel):
tags: Optional[List[str]] = None
process_time: Optional[str] = None
params: Optional[str] = None
create_datetime: Optional[str] = None
class OpertionRecordSimpleOut(OpertionRecord):
class OperationRecordSimpleOut(OperationRecord):
create_datetime: DatetimeStr
class Config:
orm_mode = True

View File

@ -6,12 +6,14 @@
# @desc : 主要接口文件
from fastapi import APIRouter, Depends
from motor.motor_asyncio import AsyncIOMotorDatabase
from utils.response import SuccessResponse
from . import crud, schemas
from apps.vadmin.auth.utils.current import AllUserAuth
from apps.vadmin.auth.utils.validation.auth import Auth
from core.mongo import get_database, DatabaseManage
from .params import LoginParams, OperationParams, SMSParams
from core.database import mongo_getter
app = APIRouter()
@ -29,11 +31,11 @@ async def get_record_login(p: LoginParams = Depends(), auth: Auth = Depends(AllU
@app.get("/operations", summary="获取操作日志列表")
async def get_record_operation(
p: OperationParams = Depends(),
db: DatabaseManage = Depends(get_database),
db: AsyncIOMotorDatabase = Depends(mongo_getter),
auth: Auth = Depends(AllUserAuth())
):
count = await db.get_count("operation_record", **p.to_count())
datas = await db.get_datas("operation_record", v_schema=schemas.OpertionRecordSimpleOut, **p.dict())
count = await crud.OperationRecordDal(db).get_count(**p.to_count())
datas = await crud.OperationRecordDal(db).get_datas(**p.dict())
return SuccessResponse(datas, count=count)

View File

@ -11,16 +11,22 @@
# sqlalchemy 关联查询详细https://blog.csdn.net/u012324798/article/details/103940527
import json
import os
from typing import List, Union
from enum import Enum
from typing import List, Union, Any
from aioredis import Redis
from fastapi.encoders import jsonable_encoder
from motor.motor_asyncio import AsyncIOMotorDatabase
from pymongo.results import InsertOneResult, UpdateResult
from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import joinedload
from application.settings import STATIC_ROOT
from application.settings import STATIC_ROOT, SUBSCRIBE
from core.mongo_manage import MongoManage
from utils.file.file_manage import FileManage
from . import models, schemas
from core.crud import DalBase
from core.exception import CustomException
from utils import status
class DictTypeDal(DalBase):
@ -167,3 +173,325 @@ class SettingsTabDal(DalBase):
}
for tab in datas
}
class TaskDal(MongoManage):
class JobOperation(Enum):
add = "add_job"
def __init__(self, db: AsyncIOMotorDatabase):
super(TaskDal, self).__init__(db, "vadmin_system_task", schemas.TaskSimpleOut)
async def get_task(
self,
_id: str = None,
v_return_none: bool = False,
v_schema: Any = None,
**kwargs
) -> dict | None:
"""
获取单个数据默认使用 ID 查询否则使用关键词查询
包括临时字段 last_run_datetimeis_active
is_active: 只有在 scheduler_task_jobs 任务运行表中存在相同 _id 才表示任务添加成功任务状态才为 True
last_run_datetime: scheduler_task_record 中获取该任务最近一次执行完成的时间
:param _id: 数据 ID
:param v_return_none: 是否返回空 None否则抛出异常默认抛出异常
:param v_schema: 指定使用的序列化对象
"""
if _id:
kwargs["_id"] = ("ObjectId", _id)
params = self.filter_condition(**kwargs)
pipeline = [
{
'$addFields': {
'str_id': {'$toString': '$_id'}
}
},
{
'$lookup': {
'from': 'scheduler_task_jobs',
'localField': 'str_id',
'foreignField': '_id',
'as': 'matched_jobs'
}
},
{
'$lookup': {
'from': 'scheduler_task_record',
'localField': 'str_id',
'foreignField': 'job_id',
'as': 'matched_records'
}
},
{
'$addFields': {
'is_active': {
'$cond': {
'if': {'$ne': ['$matched_jobs', []]},
'then': True,
'else': False
}
},
'last_run_datetime': {
'$ifNull': [
{'$arrayElemAt': ['$matched_records.create_datetime', -1]},
None
]
}
}
},
{
'$project': {
'matched_records': 0,
'matched_jobs': 0
}
},
{
'$match': params
},
{
'$facet': {
'documents': [
{'$limit': 1},
]
}
}
]
# 执行聚合查询
cursor = self.collection.aggregate(pipeline)
result = await cursor.to_list(length=None)
data = result[0]['documents']
if not data and v_return_none:
return None
elif not data:
raise CustomException("未查找到对应数据", code=status.HTTP_404_NOT_FOUND)
data = data[0]
if data and v_schema:
return jsonable_encoder(v_schema(**data))
return data
async def get_tasks(
self,
page: int = 1,
limit: int = 10,
v_schema: Any = None,
v_order: str = None,
v_order_field: str = None,
**kwargs
):
"""
获取任务信息列表
添加了两个临时字段
is_active: 只有在 scheduler_task_jobs 任务运行表中存在相同 _id 才表示任务添加成功任务状态才为 True
last_run_datetime: scheduler_task_record 中获取该任务最近一次执行完成的时间
"""
v_order_field = v_order_field if v_order_field else 'create_datetime'
v_order = -1 if v_order in self.ORDER_FIELD else 1
params = self.filter_condition(**kwargs)
pipeline = [
{
'$addFields': {
'str_id': {'$toString': '$_id'}
}
},
{
'$lookup': {
'from': 'scheduler_task_jobs',
'localField': 'str_id',
'foreignField': '_id',
'as': 'matched_jobs'
}
},
{
'$lookup': {
'from': 'scheduler_task_record',
'localField': 'str_id',
'foreignField': 'job_id',
'as': 'matched_records'
}
},
{
'$addFields': {
'is_active': {
'$cond': {
'if': {'$ne': ['$matched_jobs', []]},
'then': True,
'else': False
}
},
'last_run_datetime': {
'$ifNull': [
{'$arrayElemAt': ['$matched_records.create_datetime', -1]},
None
]
}
}
},
{
'$project': {
'matched_records': 0,
'matched_jobs': 0
}
},
{
'$match': params
},
{
'$facet': {
'documents': [
{'$sort': {v_order_field: v_order}},
{'$limit': limit},
{'$skip': (page - 1) * limit}
],
'count': [{'$count': 'total'}]
}
}
]
# 执行聚合查询
cursor = self.collection.aggregate(pipeline)
result = await cursor.to_list(length=None)
datas = result[0]['documents']
count = result[0]['count'][0]['total'] if result[0]['count'] else 0
if count == 0:
return [], 0
elif v_schema:
datas = [jsonable_encoder(v_schema(**data)) for data in datas]
elif self.schema:
datas = [jsonable_encoder(self.schema(**data)) for data in datas]
return datas, count
async def add_task(self, rd: Redis, data: dict) -> int:
"""
添加任务到消息队列
使用消息无保留策略无保留是指当发送者向某个频道发送消息时如果没有订阅该频道的调用方就直接将该消息丢弃
:params rd: redis 对象
:params data: 行数据字典
:return: 接收到消息的订阅者数量
"""
exec_strategy = data.get("exec_strategy")
job_params = {
"name": data.get("_id"),
"job_class": data.get("job_class"),
"expression": data.get("expression")
}
if exec_strategy == "interval" or exec_strategy == "cron":
job_params["start_date"] = data.get("start_date")
job_params["end_date"] = data.get("end_date")
message = {
"operation": self.JobOperation.add.value,
"task": {
"exec_strategy": data.get("exec_strategy"),
"job_params": job_params
}
}
return await rd.publish(SUBSCRIBE, json.dumps(message).encode('utf-8'))
async def create_task(self, rd: Redis, data: schemas.Task) -> dict:
"""
创建任务
"""
data_dict = data.dict()
is_active = data_dict.pop('is_active')
insert_result = await super().create_data(data_dict)
obj = await self.get_task(insert_result.inserted_id, v_schema=schemas.TaskSimpleOut)
# 如果分组不存在则新增分组
group = await TaskGroupDal(self.db).get_data(value=data.group, v_return_none=True)
if not group:
await TaskGroupDal(self.db).create_data({"value": data.group})
result = {
"subscribe_number": 0,
"is_active": is_active
}
if is_active:
# 创建任务成功后, 如果任务状态为 True则向消息队列中发送任务
result['subscribe_number'] = await self.add_task(rd, obj)
return result
async def put_task(self, rd: Redis, _id: str, data: schemas.Task) -> dict:
"""
更新任务
"""
data_dict = data.dict()
is_active = data_dict.pop('is_active')
await super(TaskDal, self).put_data(_id, data)
obj: dict = await self.get_task(_id, v_schema=schemas.TaskSimpleOut)
# 如果分组不存在则新增分组
group = await TaskGroupDal(self.db).get_data(value=data.group, v_return_none=True)
if not group:
await TaskGroupDal(self.db).create_data({"value": data.group})
try:
# 删除正在运行中的 Job
await SchedulerTaskJobsDal(self.db).delete_data(_id)
except CustomException as e:
pass
result = {
"subscribe_number": 0,
"is_active": is_active
}
if is_active:
# 更新任务成功后, 如果任务状态为 True则向消息队列中发送任务
result['subscribe_number'] = await self.add_task(rd, obj)
return result
async def delete_task(self, _id: str) -> bool:
"""
删除任务
"""
result = await super(TaskDal, self).delete_data(_id)
try:
# 删除正在运行中的 Job
await SchedulerTaskJobsDal(self.db).delete_data(_id)
except CustomException as e:
pass
return result
async def run_once_task(self, rd: Redis, _id: str) -> int:
"""
执行一次任务
"""
obj: dict = await self.get_data(_id, v_schema=schemas.TaskSimpleOut)
message = {
"operation": self.JobOperation.add.value,
"task": {
"exec_strategy": "once",
"job_params": {
"name": obj.get("_id"),
"job_class": obj.get("job_class")
}
}
}
return await rd.publish(SUBSCRIBE, json.dumps(message).encode('utf-8'))
class TaskGroupDal(MongoManage):
def __init__(self, db: AsyncIOMotorDatabase):
super(TaskGroupDal, self).__init__(db, "vadmin_system_task_group")
class TaskRecordDal(MongoManage):
def __init__(self, db: AsyncIOMotorDatabase):
super(TaskRecordDal, self).__init__(db, "scheduler_task_record")
class SchedulerTaskJobsDal(MongoManage):
def __init__(self, db: AsyncIOMotorDatabase):
super(SchedulerTaskJobsDal, self).__init__(db, "scheduler_task_jobs", is_object_id=False)

View File

@ -1,2 +1,3 @@
from .dict_type import DictTypeParams
from .dict_detail import DictDetailParams
from .task import TaskParams

View File

@ -0,0 +1,30 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/25 14:50
# @File : task.py
# @IDE : PyCharm
# @desc : 简要说明
from fastapi import Depends
from core.dependencies import Paging, QueryParams
class TaskParams(QueryParams):
"""
列表分页
"""
def __init__(self, name: str = None, _id: str = None, group: str = None, params: Paging = Depends()):
super().__init__(params)
self.name = ("like", name)
self.group = group
self._id = ("ObjectId", _id)
class TaskRecordParams(QueryParams):
"""
列表分页
"""
def __init__(self, job_id: str = None, name: str = None, params: Paging = Depends()):
super().__init__(params)
self.job_id = ("like", job_id)
self.name = ("like", name)

View File

@ -1,3 +1,4 @@
from .dict import DictType, DictDetails, DictTypeSimpleOut, DictDetailsSimpleOut, DictTypeSelectOut
from .settings_tab import SettingsTab, SettingsTabSimpleOut
from .settings import Settings, SettingsSimpleOut
from .task import Task, TaskSimpleOut

View File

@ -0,0 +1,33 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/25 15:08
# @File : task.py
# @IDE : PyCharm
# @desc : 简要说明
from typing import Optional
from pydantic import BaseModel, Field
from core.data_types import DatetimeStr, ObjectIdStr
class Task(BaseModel):
name: str
group: Optional[str] = None
job_class: str
exec_strategy: str
expression: str
is_active: Optional[bool] = True # 临时字段,不在表中创建
remark: Optional[str] = None
start_date: Optional[DatetimeStr] = None
end_date: Optional[DatetimeStr] = None
class TaskSimpleOut(Task):
id: ObjectIdStr = Field(..., alias='_id')
create_datetime: DatetimeStr
update_datetime: DatetimeStr
last_run_datetime: Optional[DatetimeStr] = None # 临时字段,不在表中创建
class Config:
orm_mode = True

View File

@ -9,19 +9,22 @@
from typing import List
from aioredis import Redis
from fastapi import APIRouter, Depends, Body, UploadFile, Form
from motor.motor_asyncio import AsyncIOMotorDatabase
from sqlalchemy.ext.asyncio import AsyncSession
from application.settings import ALIYUN_OSS
from core.database import db_getter, redis_getter
from core.database import db_getter, redis_getter, mongo_getter
from utils.file.aliyun_oss import AliyunOSS, BucketConf
from utils.file.file_manage import FileManage
from utils.response import SuccessResponse, ErrorResponse
from utils.sms.code import CodeSMS
from utils.tools import generate_string
from . import schemas, crud
from core.dependencies import IdList
from apps.vadmin.auth.utils.current import AllUserAuth, FullAdminAuth, OpenAuth
from apps.vadmin.auth.utils.validation.auth import Auth
from .params import DictTypeParams, DictDetailParams
from .params import DictTypeParams, DictDetailParams, TaskParams
from apps.vadmin.auth import crud as vadminAuthCRUD
from .params.task import TaskRecordParams
app = APIRouter()
@ -150,7 +153,11 @@ async def get_settings_tabs_values(tab_id: int, auth: Auth = Depends(FullAdminAu
@app.put("/settings/tabs/values", summary="更新系统配置信息")
async def put_settings_tabs_values(datas: dict = Body(...), auth: Auth = Depends(FullAdminAuth()), rd: Redis = Depends(redis_getter)):
async def put_settings_tabs_values(
datas: dict = Body(...),
auth: Auth = Depends(FullAdminAuth()),
rd: Redis = Depends(redis_getter)
):
return SuccessResponse(await crud.SettingsDal(auth.db).update_datas(datas, rd))
@ -167,3 +174,87 @@ async def get_settings_privacy(auth: Auth = Depends(FullAdminAuth())):
@app.get("/settings/agreement", summary="获取用户协议")
async def get_settings_agreement(auth: Auth = Depends(FullAdminAuth())):
return SuccessResponse((await crud.SettingsDal(auth.db).get_data(config_key="web_agreement")).config_value)
###########################################################
# 定时任务管理
###########################################################
@app.get("/tasks", summary="获取定时任务列表")
async def get_tasks(
p: TaskParams = Depends(),
db: AsyncIOMotorDatabase = Depends(mongo_getter),
auth: Auth = Depends(AllUserAuth())
):
datas, count = await crud.TaskDal(db).get_tasks(**p.dict())
return SuccessResponse(datas, count=count)
@app.post("/tasks", summary="添加定时任务")
async def post_tasks(
data: schemas.Task,
db: AsyncIOMotorDatabase = Depends(mongo_getter),
rd: Redis = Depends(redis_getter),
auth: Auth = Depends(AllUserAuth())
):
return SuccessResponse(await crud.TaskDal(db).create_task(rd, data))
@app.put("/tasks", summary="更新定时任务")
async def put_tasks(
_id: str,
data: schemas.Task,
db: AsyncIOMotorDatabase = Depends(mongo_getter),
rd: Redis = Depends(redis_getter),
auth: Auth = Depends(AllUserAuth())
):
return SuccessResponse(await crud.TaskDal(db).put_task(rd, _id, data))
@app.delete("/tasks", summary="删除单个定时任务")
async def delete_task(
_id: str,
db: AsyncIOMotorDatabase = Depends(mongo_getter),
auth: Auth = Depends(AllUserAuth())
):
return SuccessResponse(await crud.TaskDal(db).delete_task(_id))
@app.get("/task", summary="获取定时任务详情")
async def get_task(
_id: str,
db: AsyncIOMotorDatabase = Depends(mongo_getter),
auth: Auth = Depends(AllUserAuth())
):
return SuccessResponse(await crud.TaskDal(db).get_task(_id, v_schema=schemas.TaskSimpleOut))
@app.post("/task", summary="执行一次定时任务")
async def run_once_task(
_id: str,
db: AsyncIOMotorDatabase = Depends(mongo_getter),
rd: Redis = Depends(redis_getter),
auth: Auth = Depends(AllUserAuth())
):
return SuccessResponse(await crud.TaskDal(db).run_once_task(rd, _id))
###########################################################
# 定时任务分组管理
###########################################################
@app.get("/task/group/options", summary="获取定时任务分组选择项列表")
async def get_task_group_options(db: AsyncIOMotorDatabase = Depends(mongo_getter), auth: Auth = Depends(AllUserAuth())):
return SuccessResponse(await crud.TaskGroupDal(db).get_datas(limit=0))
###########################################################
# 定时任务调度日志
###########################################################
@app.get("/task/records", summary="获取定时任务调度日志列表")
async def get_task_records(
p: TaskRecordParams = Depends(),
db: AsyncIOMotorDatabase = Depends(mongo_getter),
auth: Auth = Depends(AllUserAuth())
):
count = await crud.TaskRecordDal(db).get_count(**p.to_count())
datas = await crud.TaskRecordDal(db).get_datas(**p.dict())
return SuccessResponse(datas, count=count)

View File

@ -63,7 +63,7 @@ class DalBase:
:param v_or: 或逻辑查询
:param v_order: 排序默认正序 desc 是倒叙
:param v_order_field: 排序字段
:param v_return_none: 是否返回空 None抛出异常默认抛出异常
:param v_return_none: 是否返回空 None抛出异常默认抛出异常
:param v_schema: 指定使用的序列化对象
:param kwargs: 查询参数
"""

View File

@ -9,6 +9,9 @@
"""
自定义数据类型 - 官方文档https://pydantic-docs.helpmanual.io/usage/types/#custom-data-types
"""
import datetime
from bson import ObjectId
from .validator import *
@ -23,6 +26,9 @@ class DatetimeStr(str):
def validate(cls, v):
if isinstance(v, str):
return v
elif isinstance(v, dict):
# 转换为datetime对象
v = datetime.datetime.strptime(v.get("$date"), "%Y-%m-%dT%H:%M:%S.%fZ")
return v.strftime("%Y-%m-%d %H:%M:%S")
@ -59,3 +65,20 @@ class DateStr(str):
if isinstance(v, str):
return v
return v.strftime("%Y-%m-%d")
class ObjectIdStr(str):
@classmethod
def __get_validators__(cls):
yield cls.validate
@classmethod
def validate(cls, v):
if isinstance(v, str):
return v
elif isinstance(v, dict):
return v.get("$oid")
elif isinstance(v, ObjectId):
return str(v)
return v

View File

@ -14,9 +14,10 @@ from aioredis import Redis
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.ext.declarative import declared_attr, declarative_base
from sqlalchemy.orm import sessionmaker
from application.settings import SQLALCHEMY_DATABASE_URL, REDIS_DB_ENABLE
from application.settings import SQLALCHEMY_DATABASE_URL, REDIS_DB_ENABLE, MONGO_DB_ENABLE
from fastapi import Request
from core.exception import CustomException
from motor.motor_asyncio import AsyncIOMotorDatabase
def create_async_engine_session(database_url: str):
@ -93,10 +94,22 @@ async def db_getter():
def redis_getter(request: Request) -> Redis:
"""
获取关系数据库
获取 redis 数据库对象
数据库依赖项它将在单个请求中使用然后在请求完成后将其关闭
全局挂载使用一个数据库对象
"""
if not REDIS_DB_ENABLE:
raise CustomException("请先配置Redis数据库链接并启用", desc="请启用 application/settings.py: REDIS_DB_ENABLE")
return request.app.state.redis
def mongo_getter(request: Request) -> AsyncIOMotorDatabase:
"""
获取 mongo 数据库对象
全局挂载使用一个数据库对象
"""
if not MONGO_DB_ENABLE:
raise CustomException(msg="请先开启 MongoDB 数据库连接!", desc="请启用 application/settings.py: MONGO_DB_ENABLE")
return request.app.state.mongo

View File

@ -8,8 +8,8 @@
from fastapi import FastAPI
from motor.motor_asyncio import AsyncIOMotorClient
from application.settings import REDIS_DB_URL, MONGO_DB_URL, MONGO_DB_NAME, EVENTS
from core.mongo import db
from utils.cache import Cache
import aioredis
from contextlib import asynccontextmanager
@ -82,11 +82,13 @@ async def connect_mongo(app: FastAPI, status: bool):
:return:
"""
if status:
client: AsyncIOMotorClient = AsyncIOMotorClient(MONGO_DB_URL, maxPoolSize=10, minPoolSize=10)
app.state.mongo_client = client
app.state.mongo = client[MONGO_DB_NAME]
print("Connecting to Mongo")
await db.connect_to_database(path=MONGO_DB_URL, db_name=MONGO_DB_NAME)
else:
print("Mongo connection closed")
await db.close_database_connection()
app.state.mongo_client.close()

View File

@ -19,8 +19,9 @@ from fastapi.routing import APIRoute
from user_agents import parse
from application.settings import OPERATION_RECORD_METHOD, MONGO_DB_ENABLE, IGNORE_OPERATION_FUNCTION,\
DEMO_WHITE_LIST_PATH, DEMO
from core.mongo import get_database
from utils.response import ErrorResponse
from apps.vadmin.record.crud import OperationRecordDal
from core.database import mongo_getter
def write_request_log(request: Request, response: Response):
@ -112,8 +113,7 @@ def register_operation_record_middleware(app: FastAPI):
"create_datetime": datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"params": json.dumps(params)
}
db = await get_database()
await db.create_data("operation_record", document)
await OperationRecordDal(mongo_getter(request)).create_data(document)
return response

View File

@ -1,13 +0,0 @@
from .database_manage import DatabaseManage
from .mongo_manage import MongoManage
from application.settings import MONGO_DB_ENABLE
from core.exception import CustomException
from utils import status
db = MongoManage()
async def get_database() -> DatabaseManage:
if not MONGO_DB_ENABLE:
raise CustomException(msg="请先开启 MongoDB 数据库连接!", code=status.HTTP_ERROR)
return db

View File

@ -1,47 +0,0 @@
from abc import abstractmethod
from typing import Any
class DatabaseManage:
"""
This class is meant to be extended from
./mongo_manage.py which will be the actual connection to mongodb.
"""
@property
def client(self):
raise NotImplementedError
@property
def db(self):
raise NotImplementedError
# database connect and close connections
@abstractmethod
async def connect_to_database(self, path: str, db_name: str):
pass
@abstractmethod
async def close_database_connection(self):
pass
@abstractmethod
async def create_data(self, collection: str, data: dict):
pass
@abstractmethod
async def get_datas(
self,
collection: str,
page: int = 1,
limit: int = 10,
v_schema: Any = None,
v_order: str = None,
v_order_field: str = None,
**kwargs
):
pass
@abstractmethod
async def get_count(self, collection: str, **kwargs) -> int:
pass

View File

@ -1,82 +0,0 @@
import json
from typing import Any
from bson.json_util import dumps
from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorDatabase
from core.mongo import DatabaseManage
from pymongo.results import InsertOneResult
class MongoManage(DatabaseManage):
"""
This class extends from ./database_manage.py
which have the abstract methods to be re-used here.
博客https://www.cnblogs.com/aduner/p/13532504.html
mongodb 官网https://www.mongodb.com/docs/drivers/motor/
motor 文档https://motor.readthedocs.io/en/stable/
"""
client: AsyncIOMotorClient = None
db: AsyncIOMotorDatabase = None
async def connect_to_database(self, path: str, db_name: str):
self.client = AsyncIOMotorClient(path, maxPoolSize=10, minPoolSize=10)
self.db = self.client[db_name]
async def close_database_connection(self):
self.client.close()
async def create_data(self, collection: str, data: dict) -> InsertOneResult:
return await self.db[collection].insert_one(data)
async def get_datas(
self,
collection: str,
page: int = 1,
limit: int = 10,
v_schema: Any = None,
v_order: str = None,
v_order_field: str = None,
**kwargs
):
"""
使用 find() 要查询的一组文档 find() 没有I / O也不需要 await 表达式它只是创建一个 AsyncIOMotorCursor 实例
当您调用 to_list() 或为循环执行异步时 (async for) 查询实际上是在服务器上执行的
"""
params = self.filter_condition(**kwargs)
cursor = self.db[collection].find(params)
# 对查询应用排序(sort),跳过(skip)或限制(limit)
cursor.sort("create_datetime", -1).skip((page - 1) * limit).limit(limit)
datas = []
async for row in cursor:
del row['_id']
data = json.loads(dumps(row))
if v_schema:
data = v_schema.parse_obj(data).dict()
datas.append(data)
return datas
async def get_count(self, collection: str, **kwargs) -> int:
params = self.filter_condition(**kwargs)
return await self.db[collection].count_documents(params)
@classmethod
def filter_condition(cls, **kwargs):
"""
过滤条件
"""
params = {}
for k, v in kwargs.items():
if not v:
continue
elif isinstance(v, tuple):
if v[0] == "like" and v[1]:
params[k] = {'$regex': v[1]}
elif v[0] == "between" and len(v[1]) == 2:
params[k] = {'$gte': f"{v[1][0]} 00:00:00", '$lt': f"{v[1][1]} 23:59:59"}
else:
params[k] = v
return params

View File

@ -0,0 +1,165 @@
import datetime
import json
from typing import Any
from bson import ObjectId
from bson.errors import InvalidId
from bson.json_util import dumps
from fastapi.encoders import jsonable_encoder
from motor.motor_asyncio import AsyncIOMotorDatabase
from pymongo.results import InsertOneResult, UpdateResult
from core.exception import CustomException
from utils import status
class MongoManage:
"""
mongodb 数据库管理器
博客https://www.cnblogs.com/aduner/p/13532504.html
mongodb 官网https://www.mongodb.com/docs/drivers/motor/
motor 文档https://motor.readthedocs.io/en/stable/
"""
# 倒叙
ORDER_FIELD = ["desc", "descending"]
def __init__(self, db: AsyncIOMotorDatabase, collection: str, schema: Any = None, is_object_id: bool = True):
self.db = db
self.collection = db[collection]
self.schema = schema
self.is_object_id = is_object_id
async def get_data(
self,
_id: str = None,
v_return_none: bool = False,
v_schema: Any = None,
**kwargs
) -> dict | None:
"""
获取单个数据默认使用 ID 查询否则使用关键词查询
:param _id: 数据 ID
:param v_return_none: 是否返回空 None否则抛出异常默认抛出异常
:param v_schema: 指定使用的序列化对象
"""
if _id and self.is_object_id:
kwargs["_id"] = ObjectId(_id)
params = self.filter_condition(**kwargs)
data = await self.collection.find_one(params)
if not data and v_return_none:
return None
elif not data:
raise CustomException("查找失败,未查找到对应数据", code=status.HTTP_404_NOT_FOUND)
elif data and v_schema:
return jsonable_encoder(v_schema(**data))
return data
async def create_data(self, data: dict | Any) -> InsertOneResult:
"""
创建数据
"""
if not isinstance(data, dict):
data = jsonable_encoder(data)
data['create_datetime'] = datetime.datetime.now()
data['update_datetime'] = datetime.datetime.now()
result = await self.collection.insert_one(data)
# 判断插入是否成功
if result.acknowledged:
return result
else:
raise CustomException("创建新数据失败", code=status.HTTP_ERROR)
async def put_data(self, _id: str, data: dict | Any) -> UpdateResult:
"""
更新数据
"""
if not isinstance(data, dict):
data = jsonable_encoder(data)
new_data = {'$set': data}
result = await self.collection.update_one({'_id': ObjectId(_id) if self.is_object_id else _id}, new_data)
if result.matched_count > 0:
return result
else:
raise CustomException("更新失败,未查找到对应数据", code=status.HTTP_404_NOT_FOUND)
async def delete_data(self, _id: str):
"""
删除数据
"""
result = await self.collection.delete_one({'_id': ObjectId(_id) if self.is_object_id else _id})
if result.deleted_count > 0:
return True
else:
raise CustomException("删除失败,未查找到对应数据", code=status.HTTP_404_NOT_FOUND)
async def get_datas(
self,
page: int = 1,
limit: int = 10,
v_schema: Any = None,
v_order: str = None,
v_order_field: str = None,
v_return_objs: bool = False,
**kwargs
):
"""
使用 find() 要查询的一组文档 find() 没有I / O也不需要 await 表达式它只是创建一个 AsyncIOMotorCursor 实例
当您调用 to_list() 或为循环执行异步时 (async for) 查询实际上是在服务器上执行的
"""
params = self.filter_condition(**kwargs)
cursor = self.collection.find(params)
if v_order or v_order_field:
v_order_field = v_order_field if v_order_field else 'create_datetime'
v_order = -1 if v_order in self.ORDER_FIELD else 1
cursor.sort(v_order_field, v_order)
if limit != 0:
# 对查询应用排序(sort),跳过(skip)或限制(limit)
cursor.skip((page - 1) * limit).limit(limit)
datas = []
async for row in cursor:
data = json.loads(dumps(row))
datas.append(data)
if not datas or v_return_objs:
return datas
elif v_schema:
datas = [jsonable_encoder(v_schema(**data)) for data in datas]
elif self.schema:
datas = [jsonable_encoder(self.schema(**data)) for data in datas]
return datas
async def get_count(self, **kwargs) -> int:
"""
获取统计数据
"""
params = self.filter_condition(**kwargs)
return await self.collection.count_documents(params)
@classmethod
def filter_condition(cls, **kwargs):
"""
过滤条件
"""
params = {}
for k, v in kwargs.items():
if not v:
continue
elif isinstance(v, tuple):
if v[0] == "like" and v[1]:
params[k] = {'$regex': v[1]}
elif v[0] == "between" and len(v[1]) == 2:
params[k] = {'$gte': f"{v[1][0]} 00:00:00", '$lt': f"{v[1][1]} 23:59:59"}
elif v[0] == "ObjectId" and v[1]:
try:
params[k] = ObjectId(v[1])
except InvalidId:
raise CustomException("任务编号格式不正确!")
else:
params[k] = v
return params

View File

@ -104,4 +104,4 @@ if __name__ == '__main__':
# print(generate_invitation_code())
# print(int(datetime.datetime.now().timestamp()))
# print(datetime.datetime.today() + datetime.timedelta(days=7))
print(generate_string())
print(generate_string(15))

32
kinit-task/.gitignore vendored Normal file
View File

@ -0,0 +1,32 @@
# Editor directories and files
.idea/
.idea
.vscode
*.suo
*.ntvs*
*.njsproj
*.sln
logs/*
!logs/.gitkeep
temp/*
!temp/.gitkeep
!static/.gitkeep
!alembic/versions/.gitkeep
# dotenv
.env
# virtualenv
venv/
ENV/
# Spyder project settings
.spyderproject
# Rope project settings
.ropeproject
*.db
.DS_Store
__pycache__
!migrations/__init__.py
*.pyc

410
kinit-task/README.md Normal file
View File

@ -0,0 +1,410 @@
# kinit-task
**感谢若依http://demo.ruoyi.vip/**
定时任务功能:
- [x] 支持添加四种定时任务:
- [x] 添加 date 指定日期时间执行定时任务
- [x] 添加 Cron 表达式定时任务
- [x] 添加 Interval 时间间隔定时任务
- [x] 支持立即执行任务功能
- [x] 使用 redis 消息队列功能动态添加任务
- [x] 使用 mongodb 数据库存储持久化保存任务
- [x] 任务表达式使用类路径表示,支持添加初始化参数:支持字符串,布尔类型,长整型,浮点型,整型
- [x] 每次任务执行完成后,记录日志到 mongodb 数据中:开始/结束执行时间,耗时,任务返回值,异常信息
## 使用
1. 安装依赖
```
# 安装依赖库
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
# 第三方源:
1. 阿里源: https://mirrors.aliyun.com/pypi/simple/
```
2. 修改项目数据库配置信息
`application/config` 目录中
- development.py开发环境
- production.py生产环境
```python
"""
MongoDB 数据库配置
与接口是同一个数据库
"""
MONGO_DB_NAME = "数据库名称"
MONGO_DB_URL = f"mongodb://用户名:密码@地址:端口/?authSource={MONGO_DB_NAME}"
"""
Redis 数据库配置
与接口是同一个数据库
"""
REDIS_DB_URL = "redis://:密码@地址:端口/数据库名称"
```
3. 启动
```
python3 main.py
```
## APScheduler
官方文档https://apscheduler.readthedocs.io/en/master/userguide.html
Githubhttps://github.com/agronholm/apscheduler
PYPIhttps://pypi.org/project/APScheduler/
安装/更新
```
pip install -U APScheduler -i https://mirrors.aliyun.com/pypi/simple/
```
### 使用
```
# 添加任务
from apscheduler.schedulers.blocking import BlockingScheduler
def job():
print('Hello world!')
scheduler = BlockingScheduler()
scheduler.add_job(job, 'interval', minutes=1)
scheduler.start()
```
```
# 立即执行
from apscheduler.schedulers.background import BackgroundScheduler
def job():
print("Hello, world!")
scheduler = BackgroundScheduler()
# 立即执行任务
scheduler.add_job(job, next_run_time=datetime.now(), id='my_job')
scheduler.start()
```
```
# 判断是否存在
from apscheduler.schedulers.background import BackgroundScheduler
def my_job():
print('Hello, world!')
scheduler = BackgroundScheduler()
scheduler.add_job(my_job, 'interval', seconds=10, id='my_job')
scheduler.start()
# 检查任务是否存在
if scheduler.get_job('my_job'):
print('任务存在')
else:
print('任务不存在')
```
```
# 删除任务
from apscheduler.schedulers.background import BackgroundScheduler
def my_job():
print('Hello, world!')
scheduler = BackgroundScheduler()
scheduler.add_job(my_job, 'interval', seconds=10, id='my_job')
scheduler.start()
# 删除任务
scheduler.remove_job('my_job')
```
```
# 添加参数
from apscheduler.schedulers.background import BackgroundScheduler
def job(arg1, arg2):
print('This is a job with arguments: {}, {}'.format(arg1, arg2))
scheduler = BackgroundScheduler()
scheduler.add_job(job, 'interval', seconds=5, args=('hello', 'world'))
scheduler.start()
```
```
# 获取当前正在执行的任务列表
from apscheduler.schedulers.background import BackgroundScheduler
def job():
print('This is a job.')
scheduler = BackgroundScheduler()
scheduler.add_job(job, 'interval', seconds=5)
scheduler.start()
# 获取当前正在执行的任务列表
jobs = scheduler.get_jobs()
for job in jobs:
print(job)
```
### 添加定时任务 add_job 方法
APScheduler的`add_job`方法用于添加定时任务。除了使用Cron表达式来指定定时任务的调度规则之外`add_job`方法还支持其他几种方法来设置定时任务的执行时间。以下是`add_job`方法常用的几种调度方式:
- date指定一个具体的日期和时间来执行任务。
```python
scheduler.add_job(job_function, 'date', run_date='2023-06-30 12:00:00')
```
在上述示例中任务将在指定的日期和时间2023年6月30日12:00:00执行。
- interval指定一个时间间隔来执行任务。
```python
scheduler.add_job(job_function, 'interval', minutes=30)
```
在上述示例中任务将每隔30分钟执行一次。
- cron使用Cron表达式来指定任务的执行时间。
```python
scheduler.add_job(job_function, 'cron', hour=8, minute=0, day_of_week='0-4')
```
在上述示例中任务将在每个工作日的早上8点执行。
- timedelta指定一个时间间隔来执行任务但相对于当前时间的偏移量。
```python
from datetime import timedelta
scheduler.add_job(job_function, 'interval', seconds=10, start_date=datetime.now() + timedelta(seconds=5))
```
在上述示例中任务将在当前时间的5秒后开始执行然后每隔10秒执行一次。
这些方法提供了不同的方式来安排定时任务的执行时间。你可以根据具体需求选择适合的调度方式,并结合相关参数来设置定时任务的执行规则。无论使用哪种方法,都可以通过`add_job`方法将任务添加到调度器中,以便按照预定的时间规则执行任务。
### cron 触发器
`cron`触发器是`APScheduler`中常用的一种触发器类型用于基于cron表达式来触发任务。它提供了灵活且精确的任务调度规则可以在特定的日期和时间点上触发任务。
以下是关于`cron`触发器的详细解释:
1. **创建触发器:**要创建一个`cron`触发器,可以使用`CronTrigger`类并指定cron表达式作为参数。cron表达式是一种字符串格式用于指定任务触发的时间规则。它由多个字段组成每个字段表示时间的不同部分例如分钟、小时、日期等。示例代码如下
```python
from apscheduler.triggers.cron import CronTrigger
# 创建每天上午10点触发的cron触发器
trigger = CronTrigger(hour=10)
```
在上述示例中我们创建了一个每天上午10点触发的`cron`触发器。
2. **添加触发器到任务:**创建触发器后,可以将它与任务相关联,以定义任务的调度规则。可以使用`add_job()`方法的`trigger`参数将触发器添加到任务中。示例代码如下:
```python
from apscheduler.schedulers.blocking import BlockingScheduler
def job_function():
# 任务逻辑
scheduler = BlockingScheduler()
scheduler.add_job(job_function, trigger=CronTrigger(hour=10))
```
在上述示例中,我们将`cron`触发器添加到了名为`job_function`的任务中使得该任务在每天上午10点触发。
3. **cron表达式**cron表达式由多个字段组成用空格分隔。每个字段表示时间的不同部分具体如下
- `分钟`范围是0-59。
- `小时`范围是0-23。
- `日期`范围是1-31。
- `月份`范围是1-12。
- `星期几`范围是0-6其中0表示星期日1表示星期一以此类推。
通过在cron表达式中指定相应的字段值可以创建各种复杂的调度规则。例如`0 12 * * *`表示每天中午12点触发`0 8-18 * * MON-FRI`表示工作日每小时从早上8点到下午6点之间的整点触发。
```python
from apscheduler.triggers.cron import CronTrigger
# 创建每周一至周五上午10点触发的cron触发器
trigger = CronTrigger(hour=10, day_of_week='mon-fri')
```
在上述示例中我们创建了一个每周一至周五上午10点触发的`cron`触发
### interval 触发器
`interval`触发器是`APScheduler`中常用的一种触发器类型,用于在固定的时间间隔内重复触发任务。它基于时间间隔而不是具体的日期和时间来触发任务,适用于需要以固定间隔执行的周期性任务。
以下是关于`interval`触发器的详细解释:
1. **创建触发器:**要创建一个`interval`触发器,可以使用`IntervalTrigger`类并指定时间间隔参数。时间间隔可以以秒、分钟、小时或者天为单位进行设置。示例代码如下:
```python
from apscheduler.triggers.interval import IntervalTrigger
# 创建每5秒触发一次的interval触发器
trigger = IntervalTrigger(seconds=5)
```
在上述示例中我们创建了一个每5秒触发一次的`interval`触发器。
2. **添加触发器到任务:**创建触发器后,可以将它与任务相关联,以定义任务的调度规则。可以使用`add_job()`方法的`trigger`参数将触发器添加到任务中。示例代码如下:
```python
from apscheduler.schedulers.blocking import BlockingScheduler
def job_function():
# 任务逻辑
scheduler = BlockingScheduler()
scheduler.add_job(job_function, trigger=IntervalTrigger(seconds=5))
```
在上述示例中,我们将`interval`触发器添加到了名为`job_function`的任务中使得该任务每隔5秒触发一次。
3. **触发器选项:**`IntervalTrigger`类还提供了其他可选的参数,用于进一步定制触发器的行为,例如:
- `start_date`:指定触发器的开始日期和时间。
- `end_date`:指定触发器的结束日期和时间。
- `timezone`:指定触发器的时区。
这些选项可以通过在创建触发器时传递相应的参数来设置。
```python
from apscheduler.triggers.interval import IntervalTrigger
from datetime import datetime, timedelta
from pytz import timezone
# 创建每5分钟触发一次的interval触发器从指定日期开始结束日期为一周后
start_date = datetime(2023, 1, 1, 0, 0, 0)
end_date = start_date + timedelta(weeks=1)
tz = timezone('US/Eastern')
trigger = IntervalTrigger(minutes=5, start_date=start_date, end_date=end_date, timezone=tz)
```
在上述示例中我们创建了一个每5分钟触发一次的`interval`触发器,并指定了开始日期、结束日期和时区。
通过使用`interval`触发器,可以方便地设置任务以固定的时间间隔重复执行。可以根据实际需求设置触发器的参数,以满足不同的周期性任务调度要求。
### 事件监听器
事件监听器是`APScheduler`中的一种机制,用于监视和响应调度器中的事件。当特定事件发生时,监听器将执行预定义的操作,例如记录日志、发送通知或执行自定义逻辑。下面是关于`APScheduler`事件监听器的详细解释:
1. **事件类型:**`APScheduler`中有多个事件类型,每个事件类型对应着不同的调度器行为或状态变化。一些常见的事件类型包括:
- `EVENT_JOB_ADDED`:当添加新任务时触发。
- `EVENT_JOB_REMOVED`:当移除任务时触发。
- `EVENT_JOB_MODIFIED`:当修改任务时触发。
- `EVENT_JOB_EXECUTED`:当任务执行完成时触发。
- `EVENT_JOB_ERROR`:当任务执行出错时触发。
- `EVENT_SCHEDULER_STARTED`:当调度器启动时触发。
- `EVENT_SCHEDULER_SHUTDOWN`:当调度器关闭时触发。
### max_instances 最大实例
报错:
```
Execution of job "Test.main (trigger: interval[0:00:01], next run at: 2023-06-21 16:23:23 CST)" skipped: maximum number of running instances reached (1)
```
这个错误提示表示当前正在运行的任务实例数量已达到最大限制,因此调度器跳过了该任务的执行。该错误通常是由于设置了并发执行任务的限制导致的。
默认情况下,`APScheduler`允许同时运行的任务实例数量是1也就是说同一任务在执行时不会与自身的其他实例并发执行。这是为了防止任务并发执行引发的问题例如资源冲突或意外的行为。
如果你想要允许任务并发执行,可以通过调整调度器的配置来增加最大并发实例数量。在创建调度器时,可以传递`max_instances`参数来设置最大并发实例数量。示例如下:
```python
from apscheduler.schedulers.blocking import BlockingScheduler
def job_function():
# 任务逻辑
scheduler = BlockingScheduler()
scheduler.add_job(job_function, 'interval', seconds=1, max_instances=5) # 设置最大并发实例为5
scheduler.start()
```
在上述示例中,`max_instances`参数被设置为5表示同一任务可以最多同时运行5个实例。如果同时触发了多个任务实例调度器将会并发执行它们直到达到最大实例数量为止。
请注意,增加最大并发实例数量可能会增加系统负载和资源消耗。因此,在设置并发执行之前,请确保你的系统能够处理额外的负载,并根据实际需求进行适当的配置。
### 如何达到了最大实例,那么超过最大实例的其他任务会延后执行吗
当调度器的任务实例数量达到最大限制时,超过最大实例数量的其他任务将会被延后执行。`APScheduler`会自动管理任务的执行队列,并按照任务的调度规则和最大实例限制进行调度。
具体来说,当有新的任务触发时,调度器会检查当前正在运行的任务实例数量是否已经达到最大限制。如果是,则该任务的执行会被延后,直到有可用的执行实例。延后执行的任务会继续保持在调度器的执行队列中,等待下一个可用的执行时机。
值得注意的是,任务的延后执行是相对于其原定的调度时间而言的。也就是说,即使一个任务被延后执行,它仍然会尽可能地在其下一个调度时间点之前执行,以保持任务的调度准确性。
以下是一个示例,演示了当任务实例数量达到最大限制时,其他任务会被延后执行:
```python
from apscheduler.schedulers.blocking import BlockingScheduler
def job_function():
# 任务逻辑
scheduler = BlockingScheduler()
scheduler.add_job(job_function, 'interval', seconds=10, max_instances=2) # 设置最大并发实例为2
# 添加多个任务,超过最大实例数量
for i in range(5):
scheduler.add_job(job_function, 'interval', seconds=5)
scheduler.start()
```
在上述示例中我们设置了最大并发实例数量为2然后添加了5个间隔为5秒的任务。由于最大实例数量限制为2所以前两个任务会立即开始执行而后面的三个任务会被延后执行等待前面的任务完成并释放实例后才能执行。
总结起来,超过最大实例数量的任务会被放入调度器的执行队列中,并在合适的时机进行延后执行,以保证任务的调度准确性和最大实例限制的有效性。
## Redis
### 什么是Redis消息订阅与发布
Redis是一个开源的高性能内存数据库支持数据结构丰富、灵活的持久化、复制以及分片。而Redis消息订阅与发布就是Redis中的一种发布/订阅模型,调用方可以订阅某个频道并接收消息,也可以向某个频道发送消息,同时所有订阅该频道的调用方都可以收到这个消息。该模型使用了发布/订阅的方式来实现多个调用方之间的信息通信。
### Redis消息保留策略
Redis提供了两种消息保留策略分别是无保留和有保留。无保留是指当发送者向某个频道发送消息时如果没有订阅该频道的调用方就直接将该消息丢弃。而有保留则是指Redis能够将消息保存下来直到有订阅该频道的调用方出现时再将该消息发送给该调用方。在Redis中消息保留策略由两个参数来控制即PUBLISH命令的NX、EX和PX选项。NX选项表示只有当至少有一个订阅者收到消息时该消息才会被保留。EX和PX选项用于控制保留策略的时间EX表示以秒为单位的保留时间PX表示以毫秒为单位的保留时间。开发者可以根据实际需要配置相应的保留策略。

View File

@ -0,0 +1,7 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 13:39
# @File : __init__.py
# @IDE : PyCharm
# @desc : 简要说明

View File

@ -0,0 +1,23 @@
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2021/10/19 15:47
# @File : development.py
# @IDE : PyCharm
# @desc : 数据库生产配置文件
"""
MongoDB 数据库配置
与接口是同一个数据库
"""
MONGO_DB_NAME = "数据库名称"
MONGO_DB_URL = f"mongodb://用户名:密码@地址:端口/?authSource={MONGO_DB_NAME}"
"""
Redis 数据库配置
与接口是同一个数据库
"""
REDIS_DB_URL = "redis://:密码@地址:端口/数据库名称"

View File

@ -0,0 +1,23 @@
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2021/10/19 15:47
# @File : production.py
# @IDE : PyCharm
# @desc : 数据库开发配置文件
"""
MongoDB 数据库配置
与接口是同一个数据库
"""
MONGO_DB_NAME = "数据库名称"
MONGO_DB_URL = f"mongodb://用户名:密码@地址:端口/?authSource={MONGO_DB_NAME}"
"""
Redis 数据库配置
与接口是同一个数据库
"""
REDIS_DB_URL = "redis://:密码@地址:端口/数据库名称"

View File

@ -0,0 +1,51 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 13:39
# @File : settings.py
# @IDE : PyCharm
# @desc : 简要说明
import os
"""项目根目录"""
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEBUG = True
"""
引入数据库配置
"""
if DEBUG:
from application.config.development import *
else:
from application.config.production import *
"""
发布/订阅通道
与接口相互关联请勿随意更改
"""
SUBSCRIBE = 'kinit_queue'
"""
MongoDB 集合
与接口相互关联相互查询请勿随意更改
"""
# 用于存放任务调用日志
SCHEDULER_TASK_RECORD = "scheduler_task_record"
# 用于存放运行中的任务
SCHEDULER_TASK_JOBS = "scheduler_task_jobs"
# 用于存放任务信息
SCHEDULER_TASK = "vadmin_system_task"
"""
定时任务脚本目录
"""
TASKS_ROOT = "tasks"

View File

@ -0,0 +1,7 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:10
# @File : __init__.py
# @IDE : PyCharm
# @desc : 简要说明

View File

@ -0,0 +1,57 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 14:42
# @File : listener.py
# @IDE : PyCharm
# @desc : 简要说明
import datetime
import json
from apscheduler.events import JobExecutionEvent
from core.mongo import get_database
import pytz
from application.settings import SCHEDULER_TASK_RECORD, SCHEDULER_TASK
from core.logger import logger
def before_job_execution(event: JobExecutionEvent):
# print("在执行定时任务前执行的代码...")
shanghai_tz = pytz.timezone("Asia/Shanghai")
start_time: datetime.datetime = event.scheduled_run_time.astimezone(shanghai_tz)
end_time = datetime.datetime.now(shanghai_tz)
process_time = (end_time - start_time).total_seconds()
job_id = event.job_id
if "-temp-" in job_id:
job_id = job_id.split("-")[0]
# print("任务标识符:", event.job_id)
# print("任务开始执行时间:", start_time.strftime("%Y-%m-%d %H:%M:%S"))
# print("任务执行完成时间:", end_time.strftime("%Y-%m-%d %H:%M:%S"))
# print("任务耗时(秒):", process_time)
# print("任务返回值:", event.retval)
# print("异常信息:", event.exception)
# print("堆栈跟踪:", event.traceback)
result = {
"job_id": job_id,
"start_time": start_time.strftime("%Y-%m-%d %H:%M:%S"),
"end_time": end_time.strftime("%Y-%m-%d %H:%M:%S"),
"process_time": process_time,
"retval": json.dumps(event.retval),
"exception": json.dumps(event.exception),
"traceback": json.dumps(event.traceback)
}
db = get_database()
try:
task = db.get_data(SCHEDULER_TASK, job_id, is_object_id=True)
result["job_class"] = task.get("job_class", None)
result["name"] = task.get("name", None)
result["group"] = task.get("group", None)
result["exec_strategy"] = task.get("exec_strategy", None)
result["expression"] = task.get("expression", None)
except ValueError as e:
result["exception"] = str(e)
logger.error(f"任务编号:{event.job_id},报错:{e}")
db.create_data(SCHEDULER_TASK_RECORD, result)

30
kinit-task/core/logger.py Normal file
View File

@ -0,0 +1,30 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:10
# @File : logger.py
# @IDE : PyCharm
# @desc : 日志管理器
import os
import time
from loguru import logger
from application.settings import BASE_DIR
"""
# 日志简单配置
# 具体其他配置 可自行参考 https://github.com/Delgan/loguru
"""
# 移除控制台输出
logger.remove(handler_id=None)
log_path = os.path.join(BASE_DIR, 'logs')
if not os.path.exists(log_path):
os.mkdir(log_path)
log_path_info = os.path.join(log_path, f'info_{time.strftime("%Y-%m-%d")}.log')
log_path_error = os.path.join(log_path, f'error_{time.strftime("%Y-%m-%d")}.log')
info = logger.add(log_path_info, rotation="00:00", retention="3 days", enqueue=True, encoding="UTF-8", level="INFO")
error = logger.add(log_path_error, rotation="00:00", retention="3 days", enqueue=True, encoding="UTF-8", level="ERROR")

View File

@ -0,0 +1,7 @@
from .mongo_manage import MongoManage
db = MongoManage()
def get_database() -> MongoManage:
return db

View File

@ -0,0 +1,118 @@
import datetime
from typing import Any
from bson import ObjectId
from bson.errors import InvalidId
from pymongo import MongoClient
from pymongo.results import InsertOneResult, UpdateResult
from pymongo.mongo_client import MongoClient as MongoClientType
from pymongo.database import Database
class MongoManage:
"""
mongodb 数据库管理器
mongodb 官网https://www.mongodb.com/docs/drivers/pymongo/
"""
client: MongoClientType = None
db: Database = None
def connect_to_database(self, path: str, db_name: str) -> None:
"""
连接 mongodb 数据库
:param path: mongodb 链接地址
:param db_name: 数据库名称
:return:
"""
self.client = MongoClient(path)
self.db = self.client[db_name]
def close_database_connection(self) -> None:
"""
关闭 mongodb 数据库连接
:return:
"""
self.client.close()
def create_data(self, collection: str, data: dict) -> InsertOneResult:
"""
创建单个数据
:param collection: 集合
:param data: 数据
"""
data['create_datetime'] = datetime.datetime.now()
data['update_datetime'] = datetime.datetime.now()
result = self.db[collection].insert_one(data)
# 判断插入是否成功
if result.acknowledged:
return result
else:
raise ValueError("创建新数据失败")
def get_data(
self,
collection: str,
_id: str = None,
v_return_none: bool = False,
v_schema: Any = None,
is_object_id: bool = False,
**kwargs
) -> dict | None:
"""
获取单个数据默认使用 ID 查询否则使用关键词查询
:param collection: 集合
:param _id: 数据 ID
:param v_return_none: 是否返回空 None否则抛出异常默认抛出异常
:param is_object_id: 是否为 ObjectId
:param v_schema: 指定使用的序列化对象
"""
if _id and is_object_id:
kwargs["_id"] = ObjectId(_id)
params = self.filter_condition(**kwargs)
data = self.db[collection].find_one(params)
if not data and v_return_none:
return None
elif not data:
raise ValueError("查询单个数据失败,未找到匹配的数据")
elif data and v_schema:
return v_schema(**data).dict()
return data
def put_data(self, collection: str, _id: str, data: dict, is_object_id: bool = False) -> UpdateResult:
"""
更新数据
"""
new_data = {'$set': data}
result = self.db[collection].update_one({'_id': ObjectId(_id) if is_object_id else _id}, new_data)
if result.matched_count > 0:
return result
else:
raise ValueError("更新数据失败,未找到匹配的数据")
@classmethod
def filter_condition(cls, **kwargs) -> dict:
"""
过滤条件
"""
params = {}
for k, v in kwargs.items():
if not v:
continue
elif isinstance(v, tuple):
if v[0] == "like" and v[1]:
params[k] = {'$regex': v[1]}
elif v[0] == "between" and len(v[1]) == 2:
params[k] = {'$gte': f"{v[1][0]} 00:00:00", '$lt': f"{v[1][1]} 23:59:59"}
elif v[0] == "ObjectId" and v[1]:
try:
params[k] = ObjectId(v[1])
except InvalidId:
raise ValueError("任务编号格式不正确!")
else:
params[k] = v
return params

View File

@ -0,0 +1,7 @@
from .redis_manage import RedisManage
db = RedisManage()
def get_database() -> RedisManage:
return db

View File

@ -0,0 +1,37 @@
import redis
class RedisManage:
"""
redis 数据库管理器
"""
rd: redis.Redis = None
def connect_to_database(self, path: str) -> None:
"""
连接 redis 数据库
:param path: mongodb 链接地址
:return:
"""
self.rd = redis.from_url(path)
def close_database_connection(self) -> None:
"""
关闭 redis 连接
:return:
"""
self.rd.close()
def subscribe(self, channel: str):
"""
订阅
:param channel: 频道
:return:
"""
pubsub = self.rd.pubsub()
pubsub.subscribe(channel)
return pubsub

View File

@ -0,0 +1,347 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:10
# @File : scheduler.py
# @IDE : PyCharm
# @desc : 简要说明
import datetime
import importlib
from typing import List
import re
from apscheduler.jobstores.base import JobLookupError
from apscheduler.jobstores.mongodb import MongoDBJobStore
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from apscheduler.triggers.date import DateTrigger
from apscheduler.triggers.interval import IntervalTrigger
from apscheduler.job import Job
from .listener import before_job_execution
from apscheduler.events import EVENT_JOB_EXECUTED
from application.settings import MONGO_DB_NAME, SCHEDULER_TASK_JOBS, TASKS_ROOT
from core.mongo import get_database
class Scheduler:
TASK_DIR = TASKS_ROOT
COLLECTION = SCHEDULER_TASK_JOBS
def __init__(self):
self.scheduler = None
self.db = None
def start(self, listener: bool = True) -> None:
"""
创建调度器
:return:
"""
self.scheduler = BackgroundScheduler()
if listener:
# 注册事件监听器
self.scheduler.add_listener(before_job_execution, EVENT_JOB_EXECUTED)
self.scheduler.add_jobstore(self.__get_mongodb_job_store())
self.scheduler.start()
def __get_mongodb_job_store(self) -> MongoDBJobStore:
"""
获取 MongoDBJobStore
:return: MongoDBJobStore
"""
self.db = get_database()
return MongoDBJobStore(database=MONGO_DB_NAME, collection=self.COLLECTION, client=self.db.client)
def add_job(
self,
job_class: str,
trigger: CronTrigger | DateTrigger | IntervalTrigger,
name: str = None,
*args,
**kwargs
) -> None | Job:
"""
date触发器用于在指定的日期和时间触发一次任务它适用于需要在特定时间点执行一次的任务例如执行一次备份操作
:param job_class: 类路径
:param trigger: 触发条件
:param name: 任务名称
:return:
"""
job_class = self.__import_module(job_class)
if job_class:
return self.scheduler.add_job(job_class.main, trigger=trigger, args=args, kwargs=kwargs, id=name)
else:
raise ValueError(f"添加任务失败,未找到该模块下的方法:{job_class}")
def add_cron_job(
self,
job_class: str,
expression: str,
start_date: str = None,
end_date: str = None,
timezone: str = "Asia/Shanghai",
name: str = None,
args: tuple = (),
**kwargs
) -> None | Job:
"""
通过 cron 表达式添加定时任务
:param job_class: 类路径
:param expression: cron 表达式六位或七位分别表示秒分钟小时星期几
:param start_date: 触发器的开始日期时间可选参数默认为 None
:param end_date: 触发器的结束日期时间可选参数默认为 None
:param timezone: 时区表示触发器应用的时区可选参数默认为 None使用上海默认时区
:param name: 任务名称
:param args: 非关键字参数
:return:
"""
second, minute, hour, day, month, day_of_week, year = self.__parse_cron_expression(expression)
trigger = CronTrigger(
second=second,
minute=minute,
hour=hour,
day=day,
month=month,
day_of_week=day_of_week,
year=year,
start_date=start_date,
end_date=end_date,
timezone=timezone
)
return self.add_job(job_class, trigger, name, *args, **kwargs)
def add_date_job(self, job_class: str, expression: str, name: str = None, args: tuple = (), **kwargs) -> None | Job:
"""
date触发器用于在指定的日期和时间触发一次任务它适用于需要在特定时间点执行一次的任务例如执行一次备份操作
:param job_class: 类路径
:param expression: date
:param name: 任务名称
:param args: 非关键字参数
:return:
"""
trigger = DateTrigger(run_date=expression)
return self.add_job(job_class, trigger, name, *args, **kwargs)
def add_interval_job(
self,
job_class: str,
expression: str,
start_date: str | datetime.datetime = None,
end_date: str | datetime.datetime = None,
timezone: str = "Asia/Shanghai",
jitter: int = None,
name: str = None,
args: tuple = (),
**kwargs
) -> None | Job:
"""
date触发器用于在指定的日期和时间触发一次任务它适用于需要在特定时间点执行一次的任务例如执行一次备份操作
:param job_class: 类路径
:param expressioninterval 表达式分别为例如设置 10 * * * * 表示每隔 10 秒执行一次任务
:param end_date: 表示任务的结束时间可以设置为 datetime 对象或者字符串
例如设置 end_date='2023-06-23 10:00:00' 表示任务在 2023 6 23 10 点结束
:param start_date: 表示任务的起始时间可以设置为 datetime 对象或者字符串
例如设置 start_date='2023-06-22 10:00:00' 表示从 2023 6 22 10 点开始执行任务
:param timezone表示时区可以设置为字符串或 pytz.timezone 对象例如设置 timezone='Asia/Shanghai' 表示使用上海时区
:param jitter表示时间抖动可以设置为整数或浮点数例如设置 jitter=2 表示任务的执行时间会在原定时间上随机增加 0~2 秒的时间抖动
:param name: 任务名称
:param args: 非关键字参数
:return:
"""
second, minute, hour, day, week = self.__parse_interval_expression(expression)
trigger = IntervalTrigger(
weeks=week,
days=day,
hours=hour,
minutes=minute,
seconds=second,
start_date=start_date,
end_date=end_date,
timezone=timezone,
jitter=jitter
)
return self.add_job(job_class, trigger, name, *args, **kwargs)
def run_job(self, job_class: str, args: tuple = (), **kwargs) -> None:
"""
立即执行一次任务但不会执行监听器只适合只需要执行任务不需要记录的任务
:param job_class: 类路径
:param args: 类路径
:return: 类实例
"""
job_class = self.__import_module(job_class)
job_class.main(*args, **kwargs)
def remove_job(self, name: str) -> None:
"""
删除任务
:param name: 任务名称
:return:
"""
try:
self.scheduler.remove_job(name)
except JobLookupError as e:
raise ValueError(f"删除任务失败, 报错:{e}")
def get_job(self, name: str) -> Job:
"""
获取任务
:param name: 任务名称
:return:
"""
return self.scheduler.get_job(name)
def has_job(self, name: str) -> bool:
"""
判断任务是否存在
:param name: 任务名称
:return:
"""
if self.get_job(name):
return True
else:
return False
def get_jobs(self) -> List[Job]:
"""
获取所有任务
:return:
"""
return self.scheduler.get_jobs()
def get_job_names(self) -> List[str]:
"""
获取所有任务
:return:
"""
jobs = self.scheduler.get_jobs()
return [job.id for job in jobs]
def __import_module(self, expression: str):
"""
反射模块
:param expression: 类路径
:return: 类实例
"""
module, args = self.__parse_string_to_class(expression)
module_pag = self.TASK_DIR + '.' + module[0:module.rindex(".")]
module_class = module[module.rindex(".") + 1:]
try:
# 动态导入模块
pag = importlib.import_module(module_pag)
return getattr(pag, module_class)(*args)
except ModuleNotFoundError:
raise ValueError(f"未找到该模块:{module_pag}")
except AttributeError:
raise ValueError(f"未找到该模块下的方法:{module_class}")
except TypeError as e:
raise ValueError(f"参数传递错误:{args}, 详情:{e}")
@staticmethod
def __parse_cron_expression(expression: str) -> tuple:
"""
解析 cron 表达式
:param expression: cron 表达式支持六位或七位分别表示秒分钟小时星期几
:return: 解析后的秒分钟小时星期几年字段的元组
"""
fields = expression.strip().split()
if len(fields) not in (6, 7):
raise ValueError("无效的 Cron 表达式")
parsed_fields = [None if field in ('*', '?') else field for field in fields]
if len(fields) == 6:
parsed_fields.append(None)
return tuple(parsed_fields)
@staticmethod
def __parse_interval_expression(expression: str) -> tuple:
"""
解析 interval 表达式
:param expression: interval 表达式分别为例如设置 10 * * * * 表示每隔 10 秒执行一次任务
:return:
"""
# 将传入的 interval 表达式拆分为不同的字段
fields = expression.strip().split()
if len(fields) != 5:
raise ValueError("无效的 interval 表达式")
parsed_fields = [int(field) if field != '*' else 0 for field in fields]
return tuple(parsed_fields)
@classmethod
def __parse_string_to_class(cls, expression: str) -> tuple:
"""
使用正则表达式匹配类路径和参数
:param expression: 表达式
:return:
"""
pattern = r'([\w.]+)(?:\((.*)\))?'
match = re.match(pattern, expression)
if match:
class_path = match.group(1)
arguments = match.group(2)
if arguments:
arguments = cls.__parse_arguments(arguments)
else:
arguments = []
return class_path, arguments
return None, None
@staticmethod
def __parse_arguments(args_str) -> list:
"""
解析类路径参数字符串
:param args_str: 类参数字符串
:return:
"""
arguments = []
for arg in re.findall(r'"([^"]*)"|(\d+\.\d+)|(\d+)|([Tt]rue|[Ff]alse)', args_str):
if arg[0]:
# 字符串参数
arguments.append(arg[0])
elif arg[1]:
# 浮点数参数
arguments.append(float(arg[1]))
elif arg[2]:
# 整数参数
arguments.append(int(arg[2]))
elif arg[3]:
# 布尔参数
if arg[3].lower() == 'true':
arguments.append(True)
else:
arguments.append(False)
return arguments
def shutdown(self) -> None:
"""
关闭调度器
:return:
"""
self.scheduler.shutdown()

0
kinit-task/logs/.gitkeep Normal file
View File

164
kinit-task/main.py Normal file
View File

@ -0,0 +1,164 @@
import atexit
import datetime
import json
import random
from enum import Enum
from apscheduler.jobstores.base import ConflictingIdError
from core.scheduler import Scheduler
from core.mongo import get_database as get_mongo
from application.settings import MONGO_DB_NAME, MONGO_DB_URL, REDIS_DB_URL, SUBSCRIBE, SCHEDULER_TASK, \
SCHEDULER_TASK_RECORD
from core.redis import get_database as get_redis
from core.logger import logger
class ScheduledTask:
class JobExecStrategy(Enum):
interval = "interval"
date = "date"
cron = "cron"
once = "once"
def __init__(self):
self.mongo = None
self.scheduler = None
self.rd = None
def add_job(self, exec_strategy: str, job_params: dict) -> None:
"""
添加定时任务
:param exec_strategy: 执行策略
:param job_params: 执行参数
:return:
"""
name = job_params.get("name", None)
error_info = None
try:
if exec_strategy == self.JobExecStrategy.interval.value:
self.scheduler.add_interval_job(**job_params)
elif exec_strategy == self.JobExecStrategy.cron.value:
self.scheduler.add_cron_job(**job_params)
elif exec_strategy == self.JobExecStrategy.date.value:
self.scheduler.add_date_job(**job_params)
elif exec_strategy == self.JobExecStrategy.once.value:
# 这种方式会自动执行事件监听器,用于保存执行任务完成后的日志
job_params["name"] = f"{name}-temp-{random.randint(1000, 9999)}"
self.scheduler.add_date_job(**job_params, expression=datetime.datetime.now())
else:
raise ValueError("无效的触发器")
except ConflictingIdError as e:
# 任务编号已存在,重复添加报错
error_info = "任务编号已存在"
except ValueError as e:
error_info = e.__str__()
if error_info:
logger.error(f"任务编号:{name},报错:{error_info}")
self.error_record(name, error_info)
def error_record(self, name: str, error_info: str) -> None:
"""
添加任务失败记录并且将任务状态改为 False
:param name: 任务编号
:param error_info: 报错信息
:return:
"""
try:
self.mongo.put_data(SCHEDULER_TASK, name, {"is_active": False})
task = self.mongo.get_data(SCHEDULER_TASK, name)
# 执行你想要在任务执行前执行的代码
result = {
"job_id": name,
"job_class": task.get("job_class", None),
"name": task.get("name", None),
"group": task.get("group", None),
"exec_strategy": task.get("exec_strategy", None),
"expression": task.get("expression", None),
"start_time": datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"end_time": datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"process_time": 0,
"retval": "任务添加失败",
"exception": error_info,
"traceback": None
}
self.mongo.create_data(SCHEDULER_TASK_RECORD, result)
except ValueError as e:
logger.error(f"任务编号:{name}, 报错:{e}")
def run(self) -> None:
"""
启动监听订阅消息阻塞
:return:
"""
self.start_mongo()
self.start_scheduler()
self.start_redis()
pubsub = self.rd.subscribe(SUBSCRIBE)
logger.info("已成功启动程序,等待接收消息...")
print("已成功启动程序,等待接收消息...")
# 处理接收到的消息
for message in pubsub.listen():
if message['type'] == 'message':
data = json.loads(message['data'].decode('utf-8'))
operation = data.get("operation")
task = data.get("task")
content = f"接收到任务:任务操作方式({operation}),任务详情:{task}"
logger.info(content)
print(content)
getattr(self, operation)(**task)
def start_mongo(self) -> None:
"""
启动 mongo
:return:
"""
self.mongo = get_mongo()
self.mongo.connect_to_database(MONGO_DB_URL, MONGO_DB_NAME)
print("成功连接 MongoDB")
def start_scheduler(self) -> None:
"""
启动定时任务
:return:
"""
self.scheduler = Scheduler()
self.scheduler.start()
print("成功启动 Scheduler")
def start_redis(self) -> None:
"""
启动 redis
:return:
"""
self.rd = get_redis()
self.rd.connect_to_database(REDIS_DB_URL)
print("成功连接 Redis")
def close(self) -> None:
"""
# pycharm 执行停止,该函数无法正常被执行,怀疑是因为阻塞导致或 pycharm 的强制退出导致
# 报错导致得退出,会被执行
关闭程序
:return:
"""
self.mongo.close_database_connection()
self.scheduler.shutdown()
self.rd.close_database_connection()
if __name__ == '__main__':
main = ScheduledTask()
atexit.register(main.close)
main.run()

View File

@ -0,0 +1,11 @@
APScheduler==3.10.1
colorama==0.4.6
dnspython==2.3.0
loguru==0.7.0
pymongo==4.3.3
pytz==2023.3
six==1.16.0
tzdata==2023.3
tzlocal==5.0.1
win32-setctime==1.1.0
redis==4.5.5

View File

@ -0,0 +1,7 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:07
# @File : __init__.py
# @IDE : PyCharm
# @desc : 简要说明

View File

@ -0,0 +1,7 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:08
# @File : __init__.py
# @IDE : PyCharm
# @desc : 简要说明

View File

@ -0,0 +1,26 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# @version : 1.0
# @Create Time : 2023/6/21 10:08
# @File : mian.py
# @IDE : PyCharm
# @desc : 简要说明
import datetime
import time
class Test:
def __init__(self, name: str, age: int):
self.name = name
self.age = age
def main(self) -> str:
"""
主入口函数
:return:
"""
print('{}, 定时任务测试实例,参数为: {}, {}'.format(datetime.datetime.now(), self.name, self.age))
time.sleep(3)
return '任务执行完成'