Skip to content

Commit ecde367

Browse files
aiceflowermayinrainkinsanitiesv-kkhuang“v_kkhuang”
authored
[1.9.0 release] [PES][EC][publicservice][spark] feat: ResultSet field processing, engine switch, task diagnosis and more (#5356)
* fix: 调整执行阶段 * code optimization (#770) * code optimization * code optimization * add aisql type * code optimization * update python version * update ldap expire time to 20m * update conf * code optimization * update progress * update code * update progress * code optimization * code optimization * Code optimization (#771) * bug fix * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * print dynamic param * code review fix * code optimization * code optimization * code optimization * Security work order repair (#772) * bug fix * bug fix * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * code optimization * code optimization * update max executors * Add Spark 3 default (#774) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * bug fix (#775) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * revent spring-webflux (#776) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Security work order repair (#777) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * add spark conf dir (#778) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.12.0 code review fix (#779) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * velocity exclusion (#780) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir * bug fix * velocity exclusion --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * update thymeleaf version (#781) * bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置,预防spark 默认配置没有用户设置的参数,导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir * bug fix * velocity exclusion * update thymeleaf version --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * update version to 1.13.0 * Code optimization (#786) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * 优化 Hive 和 Spark 配置重置逻辑 (#785) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * HDFS file system request to add retry mechanism (#783) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Spark supports forced conversion: user configuration, department configuration (#782) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * [1.13.0]Python code adds exception command check (#784) * Python code adds exception command check * update entrance intercept * update entrance intercept --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * [1.13.0]add datasource rpc GetInfoPublishedByUserIpPortAction (#787) * add datasource rpc GetInfoPublishedByUserIpPortAction * add datasource rpc GetInfoPublishedByUserIpPortAction --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * [1.13.0]Spark supports forced conversion: user configuration, department conf (#788) * Spark supports forced conversion: user configuration, department configuration * spark3 update --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.13.0 webank temp (#789) * support retry by templateconf * update retry logic * code optimization * code optimization * code optimization * code format * support conn param execute task (#792) * [1.13.0]Data source supports automated publishing (#791) * Data source supports automated publishing * Data source supports automated publishing * Data source supports automated publishing * Data source supports automated publishing --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * fix compile error * code optimization * some code retry optimization * upd: 样式优化 * upd: 样式优化 (#794) * Dev 1.13.0 bug fix (#793) * code opmitization * code opmization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: Casion <casionone@gmail.com> * code optimization * [1.13.0]code opmitization (#795) * code opmitization * code opmization * code opmization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: Casion <casionone@gmail.com> * update jdbc * upd: 依赖升级 * upd: 高度计算逻辑修改 * code optimization * upd: 放大样式修改 * compile error fix * upd: height adjustment * code optimization * Dev 1.13.0 bug fix (#799) * code opmitization * code opmization * code opmization * bug fix * bug fix * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: Casion <casionone@gmail.com> * template conf optimization * classcastexception fix * code optimization & format * code optimization * params optimization * code optimization * fix: 逻辑兜底 * task error receiver sync * code optimization * Dev 1.13.0 bug fix (#800) * code opmitization * code opmization * code opmization * bug fix * bug fix * bug fix * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: Casion <casionone@gmail.com> * add log * code update * add switch * update code to status rpc * add log * bug fix * code review fix * code review fix (#802) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.13.0 code review fix (#803) * code review fix * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * code optimization * add conf retry switch * code uptate * update code add retry code * Dev 1.13.0 code review fix (#804) * code review fix * bug fix * revent flink dependency --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.13.0 code review fix (#805) * code review fix * bug fix * revent flink dependency * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Update version to 1.14.0 * close python safe check (#808) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * fix resource reset error (#807) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * 【1.14.0】Add Doctor functionality and support for switching between AI SQL dynamic engines (#806) * 添加 Doctor功能和 AI SQL动态引擎切换支持 * 添加doctoris服务请求超时配置并优化动态引擎类型获取 * ```添加调用Doctoris diagnose接口的日志记录 ``` * ```优化动态引擎切换功能和日志记录 - 修改Python安全检查开关的默认值为false --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Add BML client connection information retrieval function (#809) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * support change engine type for aisql (#810) Co-authored-by: aiceflower <kinsanities@sina.com> * fix todo & adjust interceptor (#811) Co-authored-by: aiceflower <kinsanities@sina.com> * Dev 1.14.0 ai sql support (#812) * 添加 Doctor功能和 AI SQL动态引擎切换支持 * 添加doctoris服务请求超时配置并优化动态引擎类型获取 * ```添加调用Doctoris diagnose接口的日志记录 ``` * ```优化动态引擎切换功能和日志记录 - 修改Python安全检查开关的默认值为false * ```优化动态引擎切换功能和日志记录 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Fix resource statistics logic to correctly handle empty resources (#813) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.14.0 bug fix (#814) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.14.0 bug fix (#815) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 更新了代码类型和运行类型关系配置,将Python 3 (py3) 添加到Python代码类型支持的列表中。 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * code review fix (#816) Co-authored-by: aiceflower <kinsanities@sina.com> * Dev 1.14.0 bug fix (#817) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 更新了代码类型和运行类型关系配置,将Python 3 (py3) 添加到Python代码类型支持的列表中。 * code review fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.14.0 bug fix (#818) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 更新了代码类型和运行类型关系配置,将Python 3 (py3) 添加到Python代码类型支持的列表中。 * code review fix * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.14.0 bug fix (#819) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 更新了代码类型和运行类型关系配置,将Python 3 (py3) 添加到Python代码类型支持的列表中。 * code review fix * Security work order repair * Optimize HTTP client connection pool settings and update dependency versions * Code optimization * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.14.0 bug fix (#820) * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 修复资源统计逻辑以正确处理空资源 * 更新了代码类型和运行类型关系配置,将Python 3 (py3) 添加到Python代码类型支持的列表中。 * code review fix * Security work order repair * Optimize HTTP client connection pool settings and update dependency versions * Code optimization * Code optimization * Code optimization * optimization log * code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: aiceflower <kinsanities@sina.com> * Update version to 1.15.0-wds * 在JobHistory的搜索和查询接口中添加了runType参数,允许根据运行类型筛选作业历史记录 (#823) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * 【1.15.0】Dev 1.15.0 sys user login (#822) * 禁止系统用户登录 * Code optimization * Code optimization * Code optimization * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * 【1.15.0】Linkis token rsa (#821) * linkis token rsa * linkis token rsa * Code optimization * Code optimization * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * spark3 dynamic params (#825) Co-authored-by: aiceflower <kinsanities@sina.com> * linkis keytab (#824) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * upd: 令牌管理新增token查看;全局历史优化 * fix: 初始化问题修复 * upd: 格式化 * update conf & code format (#829) Co-authored-by: aiceflower <kinsanities@sina.com> * Code optimization (#830) Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * i18n optimization * fix: 新增不禁用名字输入 * Dev 1.15.0 code review fix (#832) * Code optimization * Code optimization * Code optimization * Security order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * fix: has specific task id, ignore runtype * Dev 1.15.0 code review fix (#834) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.15.0 webank spark3 (#836) * update conf & code format * update spark3 conf --------- Co-authored-by: aiceflower <kinsanities@sina.com> * add log for reuse engine * add log * update log * update shuff * Dev 1.15.0 code review fix (#837) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * The openfile interface supports masking field information (#839) * Code optimization (#843) * update version to 1.8.0 * Upgrade `LINKIS_VERSION` to `1.8.0` to Fix Integration Test Failure in GitHub Actions (#5250) * build(ci): update Linkis version to 1.8.0 Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * build(ci): update Docker publish workflow to use the current repository Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> --------- Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * entrance and linkismanager performance optimization (#853) * update version to 1.14.2 * Slow SQL repair * add shell executor conf * Revert "add shell executor conf" This reverts commit 324e2ece0512edbcd94fa9cc0c4deebe63021011. * Revert "Slow SQL repair" This reverts commit b4310df5fd8619459741cc46846ff2462ecafdcb. * 【1.14.2】Emergency Version Repair (#840) * Slow SQL repair * add shell executor conf * push fix resource error code * Code optimization * fix compile error * code format&fix compile error * optimization code * Dev 1.14.2 fix slow (#841) * Slow SQL repair * add shell executor conf * push fix resource error code * Code optimization * Code optimization * optimization get resource slow query (#842) --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: v-kkhuang <62878639+v-kkhuang@users.noreply.github.com> Co-authored-by: aiceflower <kinsanities@sina.com> * add cache for ec reuse (#854) * update version to 1.14.2 * Slow SQL repair * add shell executor conf * Revert "add shell executor conf" This reverts commit 324e2ece0512edbcd94fa9cc0c4deebe63021011. * Revert "Slow SQL repair" This reverts commit b4310df5fd8619459741cc46846ff2462ecafdcb. * 【1.14.2】Emergency Version Repair (#840) * Slow SQL repair * add shell executor conf * push fix resource error code * Code optimization * fix compile error * code format&fix compile error * optimization code * Dev 1.14.2 fix slow (#841) * Slow SQL repair * add shell executor conf * push fix resource error code * Code optimization * Code optimization * optimization get resource slow query (#842) * Dev 1.14.3 datasource update (#844) * Code optimization * Code optimization * Code optimization * Code optimization * update version 1.14.3 * hive query update * Code optimization * Code optimization * Dev 1.14.3 datasource update (#846) * Code optimization * Code optimization * Code optimization * Code optimization * update version 1.14.3 * hive query update * Code optimization * Code optimization * Code optimization * Code optimization * update version to 1.14.0 back * update log (#847) Co-authored-by: aiceflower <kinsanities@sina.com> * add cache for reuse engine (#849) Co-authored-by: aiceflower <kinsanities@sina.com> * Dev 1.14.3 datasource update (#848) * Code optimization * Code optimization * Code optimization * Code optimization * update version 1.14.3 * hive query update * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * revent commit * Code optimization * Code optimization * Code optimization * Dev 1.14.3 datasource update (#850) * Code optimization * Code optimization * Code optimization * Code optimization * update version 1.14.3 * hive query update * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * revent commit * Code optimization * Code optimization * Code optimization * Code optimization * Dev 1.14.3 webank cache (#851) * add cache for reuse engine * add lock --------- Co-authored-by: aiceflower <kinsanities@sina.com> * code format & fix compile error --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: v-kkhuang <62878639+v-kkhuang@users.noreply.github.com> Co-authored-by: aiceflower <kinsanities@sina.com> * Dev 1.15.0 code review fix (#852) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. * bug fix * code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * update lock timeout * Dev 1.15.0 code review fix (#855) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. * bug fix * code optimization * code optimization * code optimization * code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Fix KIND image loading, script typo, and cache directory creation (#5251) * fix(ci): pass `USING_KIND` variable to `install-mysql.sh` Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * fix(ci): correct typo in script name Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * fix(ci): create `TAR_CACHE_ROOT` directory if not exists Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> --------- Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * Dev 1.15.0 code review fix (#856) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. * bug fix * code optimization * code optimization * code optimization * code optimization * code review fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Dev 1.15.0 code review fix (#857) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. * bug fix * code optimization * code optimization * code optimization * code optimization * code review fix * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * chore: prepare to release 1.8.0 (#5254) Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * Dev 1.15.0 code review fix (#858) * Code optimization * Code optimization * Code optimization * Security order repair * 修复诊断异常 * Code optimization * add shell executor conf * aisql support mcp * fix linkis client login error * Revert "add shell executor conf" This reverts commit af6c1dca6eda08d090585357de72f70b797d7eaa. * bug fix * code optimization * code optimization * code optimization * code optimization * code review fix * Code optimization * Code optimization * Code optimization --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * Update version to 1.16.0 * add ai prompt base rule * [1.16.0]sensetive sql check (#860) * engine log url support * Code optimization * [1.16.0]feat: 历史任务信息获取接口添加brief参数优化 (#859) * feat: 历史任务信息获取接口添加brief参数优化 - 添加brief参数支持,当brief=true时返回简要信息 - 复用现有getJobHistoryByIdAndNameNoCode方法 - 新增jobHistory2BriefTaskVO转换方法 - 提升接口性能,减少数据传输量 - 保持向下兼容 * Code optimization * Code optimization * feat: 从get接口读查openEngineLog用的参数 (#861) * [1.16.0] get support engine log (#862) * engine log url support * Code optimization * refactor(web): remove request /ecrHistoryList (#863) * feat: 从get接口读查openEngineLog用的参数 * refactor: remove request /ecrHistoryList * fix(web): 总是用engineLogPath * fix(web): 显示引擎日志的时候才显示UDF日志 (#864) * fix(web): 显示引擎日志的时候才显示UDF日志 * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization (#866) * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Dev 1.16.0 code review fix (#867) * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * fix(web): 兼容老接口 (#868) * Code optimization * Code optimization * Dev 1.16.0 code review fix (#869) * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * Code optimization * support azure (#5214) * support azure * remove file * add azure conf --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”> * chore(web): update axios (#870) * Code optimization * Safety Work Order Repair * Add OAuth2 authentication support (#5253) * feat(mg-gateway): add OAuth2 authentication support - Add OAuth2 authentication configuration to GatewayConfiguration - Implement OAuth2Authentication - Update `SecurityFilter` and `UserRestful` to process OAuth2 request Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * feat(mg-gateway): add OAuth configuration - Add OAuth-related properties to `linkis-mg-gateway.properties` - Include support for GitHub OAuth as an example Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * style: reformat code Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * feat(mg-gateway): add OAuth in frontend - Add OAuth login option to the login page - Implement OAuth callback route and component - Add translations for OAuth login text Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * docs: add OAuth authentication documentation --------- Signed-off-by: kazutoiris <78157415+kazutoiris@users.noreply.github.com> * fix azure compile error (#5264) * fix azure compile error * fix azure compile error * fix storage test error --------- Co-authored-by: aiceflower <kinsanities@sina.com> * remove default token (#5265) * fix compile error * fix token security * fix token security --------- Co-authored-by: aiceflower <kinsanities@sina.com> * fix token security (#5266) Co-authored-by: aiceflower <kinsanities@sina.com> * Merge upstream changes and apply additional improvements (#5279) * feat: merge from upstream --- * Optimize task planner only create job and codeLogical Task (peacewong) * Add Job ID information to the log of the entire link of the task (peacewong) * Fix log out cookie not to clear (peacewong) * Task progress exceeding 1 will give fake progress (peacewong) * Enable to custom the storage path for yarn logs in ECM. (davidhua) * Change the default value of blank-placeholder in Flink engine; (davidhua) * optimize errorcode (peacewong) * Clean the job resources after submitting of flink-jar job in flink engine. (davidhua) * Exclude the FLINK_SHIP_DIRECTORIES from server in cleaner of Flink Engine. (davidhua) * code format (peacewong) * engineconn-plugin-core - add config for MaxMetaspaceSize (alexkun) * remove jdk8 judge (peacewong) * optimize engine log (#98) (huangKai-2323) * update task cancel msg (#97) (huangKai-2323) * update log (#94) (huangKai-2323) * fix shell upload blocking (#95) (huangKai-2323) * 【1.1.10】Fix the garbled code problem of spark function (#93) (huangKai-2323) * set version 1.1.10-wedatasphere (casionone) * 【1.1.10】Repair of the parameter validation problem configured by the management console (#96) (huangKai-2323) * fix yarn resource add + not correct (#102) (Casion) * optimize scriptis code print (#103) (Casion) * support spark.external.default.jars (#101) (Casion) * spark task add resource info log (#104) (Casion) * 【1.1.10】Entry tenant label supports common configuration (#91) (huangKai-2323) * fix csv export possible missing escape characters (#105) (Casion) * Improve exception information and add path information (#4351) Co-authored-by: casionone <casionone@gmail.com> (binbincheng) * revert of shell upload blocking (casionone) * move dir check to HDFSCacheLogWriter.init (#100) (aiceflower) * Dev 1.3.2 update tenant (#106) (huangKai-2323) * Fix custom variable settings (#107) (Casion) * engine bml resource compatible prefix v format (#108) (Casion) * Revert "move dir check to HDFSCacheLogWriter.init (#100)" (#109) (peacewong) * Fix unlock (peacewong) * Fix unlock (#111) (peacewong) * fix NPE bug (#112) (Casion) * Fix unlock (peacewong) * update to retry exception (peacewong) * remove default value 0 (peacewong) * 1.1.10 webank bugfix (#116) (Casion) * code review optimize (peacewong) * optimize about submit empty code after code preprocessing (#113) (Casion) * Fix resultset (#115) (Casion) * Dev 1.3.2 fix config cache bug (#117) (huangKai-2323) * Revert "support spark.external.default.jars (#101)" (casionone) * Dev 1.3.2 update tenant (#114) (huangKai-2323) * scala support jar (#119) (Casion) * fix(common util):fix getStringToMap function bug about ECResourceInfoUtils (#121) (ahaoyao) * revert DoubleType NaN (casionone) * fix spark resource log NPE (casionone) * Revert "【1.1.10】Fix the garbled code problem of spark function (#93)" (casionone) * Code optimization (#125) (huangKai-2323) * bug fix and feature optimize (#124) (Yonghao Mei) * add 1.1.10 webank web (#127) (Yonghao Mei) * feat:Add single test to ByteTimeUtils and fix code that triggers null pointer exception (#126) (ahaoyao) * fix:Fix spelling errors in words (#129) (ahaoyao) * feat:add ldap-userNameFormat example value (#128) (ahaoyao) * Revert "Fix custom variable settings (#107)" This reverts commit dd10fbcc (casionone) * fix JobHistoryQueryServiceImpl cache bug (#133) (huangKai-2323) * update spark datatype (huangKai-2323) * Compatible with trino version 407. (HmhWz) * add dss gateway support parser. (#135) (mhHao) * Supports the conversion of positive and negative byte numbers to gb (ahaoyao) * Optimization of token exception scenario description (ahaoyao) * Basic Information Management Delete Interface Add Administrator Authentication (ahaoyao) * 1. Add the eclist interface input parameter: queueName 2. Add the eclist interface parameter: lastUnlockTimestamp (huangKai-2323) * adjust sqoop ec (casionone) * add datasource dependency (casionone) * add missing import (casionone) * fix spark garbled code bug (huangKai-2323) * fix spark garbled code bug (huangKai-2323) * add instanceName parameters (huangKai-2323) * Code Formatting (huangKai-2323) * adjust dss gateway support (casionone) * Code Formatting (huangKai-2323) * add doubleToStringTest (huangKai-2323) * fix Fixed session issue (peacewong) * update to ms (peacewong) * format code (huangKai-2323) * format code (huangKai-2323) * [1.1.11] Optimize UDF function logic (#131) (huangKai-2323) * update get configuration logic (#140) (aiceflower) * fix class no found (#144) (Casion) * update test (peacewong) * add log (peacewong) * Parse to specific instance for flow/entrance/kill in DSSGatewayParser. (HmhWz) * version num to 1.1.11-wedatasphere (casionone) * fix:select exclude route label is empty (#147) (Casion) * added askEngineHook (#143) (Casion) * 1.1.11 webank bdpclient (#142) (Casion) * Optimize jdbc driver (peacewong) * fix Fixed session issue (peacewong) * update to ms (peacewong) * add log (peacewong) * add license (peacewong) * Dev 1.1.11 ec list add unlock time (#148) (huangKai-2323) * update var name (peacewong) * Fix error Status call back (peacewong) * Dev 1.4.0 jdbc optimize (#149) (peacewong) * fix eclist api:filter after group by (#151) (Casion) * Optimization of token exception scenario description (#152) (ahaoyao) * ec list add unlock time---code_view (#153) (huangKai-2323) * optimize udf permissions (#150) (huangKai-2323) * Fix duplicate error codes (ahaoyao) * remove duplicate dependency of disruptor (casionone) * JDBC Optimize (peacewong) * Fix build error (peacewong) * Dev 1.1.11 optimize udf permissions (#154) (huangKai-2323) * Code Format (peacewong) * fix web bugs and optimization (#156) (Yonghao Mei) * Dev 1.1.11 webank fixoom (#157) (peacewong) * Fix issue with annotation task hang at the end of Scala code (peacewong) * add runScheduleIntercept (peacewong) * Fix storage error (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Add creator level job limit (peacewong) * remove CacheOutputExecuteResponse (peacewong) * Fix build error (peacewong) * The dss prod label interface forwarding problem was fixed. (HmhWz) * add scheduler conf (peacewong) * feat: do not kill ec when ecm restart part-1 (#4184) (#4452) * feat: do not kill ec when ecm restart(#4192) (CharlieYan) * fix: do not clean ec resource and service info when handler ecm close event(#4184) (#4510) (CharlieYan) * update to 1.1.13-wedatasphere (casionone) * add sql connection (peacewong) * add sql connection (#168) (Casion) * Add TicketID to Node (peacewong) * Fixed ECM not cleaning up when starting EC exception (peacewong) * move File to java dir (peacewong) * Because the information needs to be pushed to createEngine, there is no need to stop here, because create will stop (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Fix Integration Test (peacewong) * code optimize (peacewong) * Fix NPE bug #4621 (peacewong) * itsm json fix (huangKai-2323) * print real ip (ahaoyao) * Add exception logs when kill job interface (ahaoyao) * delete setStatus (ahaoyao) * properties values support the Chinese (ahaoyao) * update error code (huangKai-2323) * Optimization of incorrect 500 scenario return results (ahaoyao) * Modify comments (ahaoyao) * remove author info (ahaoyao) * Implement custom DefaultErrorAttributes (ahaoyao) * add linkis_ps_udf_user_load index (huangKai-2323) * sync 1.4.0 code (huangKai-2323) * fix spark memory count error (huangKai-2323) * Security Work Order Upgrade (ahaoyao) * udf all add description (huangKai-2323) * Code optimization (huangKai-2323) * Code optimization (huangKai-2323) * Code optimization (huangKai-2323) * add timo out task scan (peacewong) * remove node scorer service (peacewong) * Optimization of log issues, adding queue names (ahaoyao) * 【1.1.13】fix queueresources sql bug (#166) (v-kkhuang) * 【1.1.13】interface fileinfo add totalline (#165) (v-kkhuang) * 【1.1.13】remove tenant for entrance (#163) (v-kkhuang) * 【1.1.13】Adding specified parameter validation when modifying configuration files (#180) (v-kkhuang) * 【1.1.13】 Operation and maintenance optimization: Service version information reporting (#182) (v-kkhuang) * Service support merge ps-data-source-manager ps-metadataquery into ps-datasource (#185) (Casion) * remove node scorer service invoke (peacewong) * FIX conflict (peacewong) * Fix build issue (peacewong) * add mapper (casionone) * feature add config template (casionone) * remove try and warn (peacewong) * add test for TemplateConfigKey dao (casionone) * code optimize (casionone) * fix bug and code optimize (casionone) * code optimize (casionone) * queue resources and add parameter maxapps (huangKai-2323) * fix oom & shell error optimize (casionone) * 【1.1.13】queue resources and add parameter maxapps (#190) (v-kkhuang) * Adapt data source script fallback (ahaoyao) * 【1.1.13】fix licensed check error (#192) (v-kkhuang) * spotless:apply (casionone) * add sys funtion script (#187) (ahaoyao) * A single executor engine type should not create multiple executors #4713 (peacewong) * set default value to false (peacewong) * code optimization (huangKai-2323) * optimize shell error msg (casionone) * fix shell file path bug (huangKai-2323) * Support configuration template (#195) (Casion) * Fix scala dont exit issue (peacewong) * openEngineLog check with permission (casionone) * 【1.1.13】 Operation and maintenance optimization: Service version information reporting (#203) (v-kkhuang) * 【1.1.13】Adding specified parameter validation when modifying configuration files (#201) (v-kkhuang) * openEngineLog check with historyAdmin permission configuration template optimize (casionone) * revert sql result deal with \n (casionone) * rollback (ahaoyao) * code optimization (huangKai-2323) * code optimize after code review (#209) (v-kkhuang) * code optimize after code review (#206) (Casion) * fix NPE for YarnResourceRequester (#212) (v-kkhuang) * Optimization for linkis web and new features (#211) (Yonghao Mei) * fix bug get user special limit config setting (casionone) * upgrade commons-fileupload (#214) (Casion) * fixed 'job is killed by connection lost' (casionone) * code opmitization (huangKai-2323) * fix eclist heartbeat_msg (casionone) * optimize cli (peacewong) * add hive current feature (peacewong) * code format (peacewong) * Dev 1.1.13 webank web (#217) (Yonghao Mei) * [Bugfix] Task state changes should before engine state changes (#4775) (人生有如两个橘子) * add ex exit send task log (peacewong) * add appconn plugin (xlinliu) * Revert "add appconn plugin" (xlinliu) * set timeout (#4794) (人生有如两个橘子) * update null logical (peacewong) * Fix Build error (peacewong) * remove useless method (peacewong) * add ecm instance log (huangKai-2323) * When requesting EC, print linkismanager instance information (#219) (v-kkhuang) * Optimization of token exception description (#220) (ahaoyao) * EC start with app version parameter (#224) (v-kkhuang) * Spark & Hive Engine Log Ooptimization (#221) (v-kkhuang) * hive ec supports concurrency and when ec exit push task failed log (#223) (peacewong) * support cross cluster label task (#222) (lemonjuice) * upgrade to 1.1.14-wedatasphere (casionone) * 【1.1.14】New User Configuration and Configuration Management for Linkis Management Console (#225) (v-kkhuang) * fix update rule mapper (lemonjuicelove) * fix line break bug (casionone) * code format (casionone) * fix cache log write bug (#229) (Casion) * fix null value bug (peacewong) * fix resultset (ahaoyao) * fix Duplicate resource display (ahaoyao) * Fix npe (peacewong) * merge dev-1.1.13-webank-merge-1.12-webank-flink into dev-1.1.14-webank (#236) (Alexyang) * 1. linkis-engineplugin-hive - fix compile error (alexkun) * 1. linkis-engineplugin-hive - fix compile error (alexkun) * fix ClassCastException Ljava.lang.Object; cannot be cast to scala.runtime.Nothing (casionone) * 【1.1.14】fix user conf error (#237) (v-kkhuang) * 【1.1.14】update log (#238) (v-kkhuang) * Dev 1.1.14 webank jayceyang (#239) (lemonjuice) * support ec startup params with templateid (#234) (Casion) * Fix JDBC Fixed session issue (peacewong) * add template rpc response entity (casionone) * remove hooks (peacewong) * Fix queue not exit exception (peacewong) * fix Extra invisible characters (ahaoyao) * Optimize EC request linkismanager and print logs (#242) (v-kkhuang) * Fix configuration management and user configuration bugs (#241) (v-kkhuang) * fix conf template id apply api NPE (casionone) * switch on (peacewong) * 【1.1.14】code view fix (#247) (v-kkhuang) * Remove cross cluster (#246) (lemonjuice) * 【1.1.14】base conf data sql syc (#244) (v-kkhuang) * optimize token desc (ahaoyao) * Fix default null value issue (peacewong) * Fix default null value issue (peacewong) * add configKey and userConfig page; support download all result set wh… (#248) (Yonghao Mei) * add tryCatch to log and progress method (peacewong) * change LinkisErrorCodeHandler to public (#253) (Alexyang) * delete duplicate line (peacewong) * Fix thread throw exception stop (peacewong) * fix sql where label miss and (aiceflower) * Fix null instance issue (peacewong) * Fix NPE issue (peacewong) * update attempts default retry (peacewong) * update try and warn (peacewong) * Fix hive get progress npe (peacewong) * Fix build error (peacewong) * update default value (peacewong) * update null replace default false (peacewong) * service merge (huangKai-2323) * error code (huangKai-2323) * update error code dml (huangKai-2323) * update hive progress Get the completed stage for progress calculation (peacewong) * Fix fs not init bug (peacewong) * add status (huangKai-2323) * eureka instance list (#257) (v-kkhuang) * 【1.1.15】Configuration cache optimization (#255) (v-kkhuang) * update error code dml (#264) (v-kkhuang) * 【1.1.15】Spark engine add conf (#258) (v-kkhuang) * feat-4868: Delete the configurations of the master and slave nodes. (#254) (Casion) * Update vertion to 1.1.15-wedatasphere (Casion) * fix compile bug (#266) (Casion) * update bean name (#267) (aiceflower) * update lock owner field (#273) (aiceflower) * add txt file type (#270) (v-kkhuang) * 【1.1.15】fix spark conf error (#269) (v-kkhuang) * 【1.1.15】Fix Datatype Decimal Error (#265) (v-kkhuang) * 【1.1.15】Security Work Order Upgrade (#256) (v-kkhuang) * 【1.1.15】fix query config key error (#274) (v-kkhuang) * 【1.1.15】fix decimal test code (#275) (v-kkhuang) * Fix progress no push bug (peacewong) * 【1.1.15】job remove running msg (#276) (v-kkhuang) * Dev 1.1.15 webank acrossclusters (#278) (Casion) * update template conf support query with associated with the engine type (casionone) * 【1.1.15】fix spark conf bug (#279) (v-kkhuang) * 【1.1.15】filesystem rename add no admin check (#280) (v-kkhuang) * fix null exception (#282) (lemonjuice) * Dev 1.1.15 fix entrance oom (#281) (Casion) * add yarnCluster label (#284) (aiceflower) * 【1.1.15】code view fix (#286) (v-kkhuang) * [Linkis-1.1.15-Code Review] fix code review (#285) (lemonjuice) * 【1.1.15】spark code opmitization (#287) (v-kkhuang) * linkis console web optimization and add new features (#283) (Yonghao Mei) * Dev 1.1.15 webank hive (#289) (aiceflower) * 【1.1.15】bug fix (#288) (v-kkhuang) * Revert "【1.1.15】bug fix (#288)" (casionone) * bug fix ps server merege and configuration api (casionone) * optimization for console web (#290) (Yonghao Mei) * fix group name may contain with _ such as APP_A (casionone) * Fix progress get 0.1 fake progress (peacewong) * update log of match error code to error level (peacewong) * fix conf error (v_kkhuang) * switch jvm user to false (peacewong) * 【1.1.15】update dml (#292) (v-kkhuang) * update engine error msg (v_kkhuang) * update engine error msg (v_kkhuang) * update engine error msg (v_kkhuang) * update hive log4j conf (v_kkhuang) * code opmizition (v_kkhuang) * Update vertion to 1.1.16-wedatasphere (aiceflower) * update setOwner (aiceflower) * update error coode (v_kkhuang) * update error coode (v_kkhuang) * update fun name (aiceflower) * update func name (aiceflower) * update deprecated desc (aiceflower) * update error coode (v_kkhuang) * formate code (v_kkhuang) * update dml (v_kkhuang) * update msg (v_kkhuang) * add configration test (v_kkhuang) * update error msg (v_kkhuang) * Add the linkis task id to trino in the first line of tsql. (HmhWz) * code format spotless (casionone) * code support to set template conf name (casionone) * trino add query taskid info &code optimize (casionone) * update hdfs check on (aiceflower) * fix log permision (aiceflower) * update hive log4j conf (v_kkhuang) * add configration test (v_kkhuang) * code optimize and add some unit test (casionone) * add configration test (#307) (v-kkhuang) * 【1.1.16】fix regex bug (#306) (v-kkhuang) * update hive log4j conf (#308) (v-kkhuang) * linkis ecm stimate actual memory 100% (casionone) * fix result set sort bug (mayinrain) * fix bug (casionone) * fix log4j bug (v_nikeyang) * fix log4j bug (v_nikeyang) * Dev 1.1.15 webank streamis fink load yaml (#303) (yangwenzea) * fix log4j bug (v_nikeyang) * add clusterinfo and code optimize (casionone) * update error log mark red (aiceflower) * update regex (v_kkhuang) * remove tispark conf (v_kkhuang) * code opmitization (v_kkhuang) * update entrance capacity to 5000 (aiceflower) * update ecm protected to 10g by default (aiceflower) * 1. plugin-flink - fix lockservice in FlinkSpringConfiguration.scala 2. application-engineconn.yml - add param allow-bean-definition-overriding with default value true 3. code format (Alexyang) * 1. rename FlinkSpringConfiguration (Alexyang) * fix log4j bug (v_nikeyang) * fix log4j bug (v_nikeyang) * update period to 10 and fix ECM log info (aiceflower) * update value can empty (aiceflower) * fix gc log bug (v_nikeyang) * push iast fix code (v_kkhuang) * Fix linkis cli npe error (#317) (mhHao) * fix code review (v_nikeyang) * Update vertion to 1.1.17-wedatasphere (Casion) * Revert "Revert "add appconn plugin"" (xlinliu) * add appconn plugin (xlinliu) * update dss.version for linkis-engine-appconn (xlinliu) * Code optimization (v_kkhuang) * resolve The problem when application in yarn is always acceptable, the flink ec will be blocked in detach mode. (wushengyeyouya) * remove query mysql meta logic (aiceflower) * down csv ignore \n (#322) (Casion) * [1.1.17]ECM Support file download (#318) (v-kkhuang) * fix linkis_cg_lm_across_cluster_rule to linkis_ps_configuration_across_cluster_rule (#321) (lemonjuice) * [1.1.17]Linkis supports task prediction function (#319) (v-kkhuang) * add get sheets info api (#323) (aiceflower) * Dev 1.1.17 task taskprediction (#324) (v-kkhuang) * appconn code formatter (casionone) * fix some bugs and code optimizes (#325) (Casion) * fix cogfig bug (#327) (v-kkhuang) * Dev 1.1.17 task taskprediction (#326) (v-kkhuang) * fix sql check action error (casionone) * remove appconn plugin (xlinliu) * Dev 1.1.17 sheetbug (#332) (aiceflower) * [1.1.17]Add permission verification to the interface (#328) (v-kkhuang) * upgrade snakeyaml to 2.0 (#331) (aiceflower) * add cluster resource threshold (#330) (lemonjuice) * Safety work order repair (#329) (v-kkhuang) * fix bigxlsx parse error (#334) (aiceflower) * async resue engine (#335) (Casion) * wds.linkis.server.version set default value (#336) (aiceflower) * fix combile error (aiceflower) * [1.1.17] task taskprediction (#339) (v-kkhuang) * [1.1.17] add jobhistory admin (#340) (v-kkhuang) * [1.1.17]fix config bug (#338) (v-kkhuang) * fix cluster threshold logs (#337) (lemonjuice) * [1.1.17] fix hive result error (#341) (v-kkhuang) * Dev 1.1.17 task taskprediction (#343) (v-kkhuang) * Dev 1.1.17 code view fix (#344) (v-kkhuang) * parse xlsx code update (#345) (aiceflower) * add snakeyaml dep for flink (aiceflower) * modify parse excel order (aiceflower) * fix excel load into hive & log optimize (#347) (Casion) * 增加hbase引擎 (peacewong) * Modify the type of the key get from the connectParams (#349) (chenmutime) * Rollback code (#350) (v-kkhuang) * chore: add download log function; optimization for web (#348) (Yonghao Mei) * [1.1.17]application-eureka yarn to properties (#342) (v-kkhuang) * mdq revert (#352) (aiceflower) * fix scala killTask (casionone) * wds.linkis.entrance.max.capacity 5000 -> 1000 (#353) (v-kkhuang) * 回滚版本 (peacewong) * Update vertion to 1.1.18-wedatasphere (Casion) * 修改判断bug (peacewong) * Update vertion to 1.1.19-wedatasphere (Casion) * [wip][1.1.18]add monitor service (#333) (v-kkhuang) * 1. FlinkEngineConnFactory - add default ha_mode 'NONE' for manage ec (#357) (#360) (Casion) * code optimization (#359) (v-kkhuang) * Dev 1.1.18 webank acrosscluster (#358) (lemonjuice) * mss log dependency (#356) (v-kkhuang) * [wip][1.1.18]open file limit 30m (#355) (v-kkhuang) * add error code 01006 (#354) (v-kkhuang) * fix spark scala task kill (#361) (Casion) * check crossQueue length (#363) (lemonjuice) * Dev 1.1.18 webank fixbug (#366) (Casion) * 修复JDBC连接泄漏问题 (peacewong) * 优化判断逻辑 (peacewong) * 1. fix FlinkEnvConfiguration - remove dependency for HighAvailablilityOption in flink-core.jar (Alexyang) * bug fix (casionone) * fix NPE of api persistTable (casionone) * optimize spark scala job thread isInterrupted (casionone) * add across cluster queue feature in web (#371) (Yonghao Mei) * fix bug when killTask may kill main thread (casionone) * code optimize after code review (casionone) * [1.1.19]Optimize monitor code (#362) (v-kkhuang) * syn pr (#367) (v-kkhuang) * [wip][1.1.19]Optimization of Linkis Resource Prediction Interface (#364) (v-kkhuang) * update netty version 4.1.86.Final ->4.1.100.Final (#374) (v-kkhuang) * update sheetInfo result data map to array list (#375) (v-kkhuang) * [1.1.19]revent pom.xml (#376) (v-kkhuang) * optimize loop array queue cache (#378) (peacewong) * add last month variable (#379) (v-kkhuang) * fix 1.1.19 pom error (#381) (v-kkhuang) * fix if to nonEquals (peacewong) * [1.1.19] fix hive partition bug (#365) (v-kkhuang) * [1.1.19]File system openLog increases 30M limit (#370) (v-kkhuang) * [1.1.19] Tenant label adding department label management (#380) (v-kkhuang) * [1.1.19]Dev 1.1.19 add last variable (#382) (v-kkhuang) * The submit interface supports replacing the executeUser parameter (#383) (v-kkhuang) * [1.1.19] tenant add department (#385) (v-kkhuang) * 【1.1.19】 fix monitor error (#386) (v-kkhuang) * [1.1.19] fix monitor error (#388) (v-kkhuang) * optimize res path (#384) (peacewong) * [1.1.19]tenant support start/stop (#387) (v-kkhuang) * 1.add error code 01007 (#389) (v-kkhuang) * [1.1.19] mv hdfs file (#346) (v-kkhuang) * hive add application log (#368) (v-kkhuang) * [WIP] fix delete rule api (#369) (lemonjuice) * code revert (#391) (aiceflower) * fix department error (#392) (v-kkhuang) * Dev 1.1.18 mv hdfs file (#390) (v-kkhuang) * [1.1.19]fix tenant error (#393) (v-kkhuang) * update service instance (#396) (v-kkhuang) * Dev 1.1.19 add last variable (#395) (v-kkhuang) * [1.1.19] fix hdfs length error (#394) (v-kkhuang) * remove write invoke setPermission (peacewong) * fix cross rule list api (lemonjuicelove) * update case error (#397) (Casion) * Dev 1.1.19 tenant add department (#398) (v-kkhuang) * optimize ec classpath env, support task conf classpath files (peacewong) * fix cross rule query list api (lemonjuicelove) * Dev 1.1.19 hive add application log (#399) (v-kkhuang) * [1.1.19]update error code (#400) (v-kkhuang) * [1.1.19]fix chmod error (#401) (v-kkhuang) * [1.1.19]fix openFile error (#402) (v-kkhuang) * [1.1.19]fix chmod error (#403) (v-kkhuang) * [1.1.19]add archive shell (#404) (v-kkhuang) * Dev 1.1.18 fix hive partition bug (#406) (v-kkhuang) * fix python kill error (#407) (Casion) * add afterExecutorExecute (peacewong) * dev-1.1.19:across cluster code review (#408) (lemonjuice) * [1.1.19]code review fix (#409) (v-kkhuang) * fix monitor (aiceflower) * add qml conf (#411) (Casion) * update fs (aiceflower) * Update vertion to 1.1.20-wds (Casion) * remove node exist status deal (peacewong) * Fix bug oom process exit (peacewong) * code format (peacewong) * support job tags into yarn app info (peacewong) * fix slow sql (#412) (v-kkhuang) * opt code (#413) (Casion) * Feature: Add result set read column maximum limit (peacewong) * Control whether engine logs are only viewed by admin (“v_kkhuang”) * Revert "Control whether engine logs are only viewed by admin" (“v_kkhuang”) * chore: 1.1.19 web update (#410) (Yonghao Mei) * monitor fix (#415) (v-kkhuang) * update logger level warn -> error (#416) (v-kkhuang) * Optimize the download of Excel result set header color: red>black (#417) (v-kkhuang) * The number of rows in the result set cannot exceed 5000 rows (#419) (v-kkhuang) * User configuration function optimization (#418) (v-kkhuang) * Control whether engine logs are only viewed by admin (#420) (v-kkhuang) * [wip][1.1.20]ecm list add msg (#421) (v-kkhuang) * optimize concurrent ec execute logic (peacewong) * init list size reduce memory usage (peacewong) * update current ec max task num value,should < linkis.engineconn.concurrent.thread.num (peacewong) * update ec transition should to judge running task (peacewong) * [1.1.20]Jobhistory add taskids query (#414) (v-kkhuang) * get across cluster queue (#422) (aiceflower) * optimize concurrent ec execute logic (peacewong) * [wip][1.1.20]update bml select (#424) (v-kkhuang) * jdbc support params set engineversion (#425) (Casion) * Dev 1.1.20 webank acrosscluster (#426) (lemonjuice) * should throw exception (peacewong) * fix cross cluster resource bug (lemonjuicelove) * code optimize (peacewong) * update code (#427) (aiceflower) * code format (lemonjuicelove) * update isOnline (#428) (aiceflower) * Update code (Casion) * add trace log (peacewong) * fix cross cluster (lemonjuicelove) * use all path (peacewong) * add error code 13002 (#430) (v-kkhuang) * [1.1.20] update cloum length 128 ->256 (#431) (v-kkhuang) * add cross cluster notes (lemonjuicelove) * fix cross cluster resource bug (lemonjuicelove) * add across resource show switch (aiceflower) * fix entrance error (#433) (v-kkhuang) * [1.1.20]Add SDK data type return (#429) (v-kkhuang) * merge 1.1.19 webank to 1.1.20-webank (#434) (Casion) * Update acrossQueues (Casion) * remove maxnumber (Casion) * roll back (peacewong) * Revert "roll back" (peacewong) * roll back (peacewong) * remove spark tags (peacewong) * update show order (aiceflower) * Fix entrance oom add job clear method (peacewong) * code format (peacewong) * Fix entrance oom add job clear method (peacewong) * 1.Fix FS support do not close fs 2.do not to create configuration everytime (peacewong) * update version (peacewong) * Fix Test (peacewong) * add create times (peacewong) * add create times (peacewong) * Feature: Add result set read column maximum limit (peacewong) * Fix NPE (peacewong) * Fix col value length cause oom bug (peacewong) * Fix build issue (peacewong) * fix bug of spark ec not exit successfully (casionone) * add row max check (peacewong) * add row max check (peacewong) * Fix case issue (peacewong) * Fix add class path bug (peacewong) * Fix exception thrown (peacewong) * 统一结果集行列限制返回 (peacewong) * fix cache loger may missing logs (#435) (Casion) * [1.1.20] fix jobhistory bug (#441) (v-kkhuang) * [1.1.20]fix sdk data type (#440) (v-kkhuang) * fix get user-creator conf : *-* > hduser - * (#439) (v-kkhuang) * [1.1.20]fix can not update "" error (#438) (v-kkhuang) * add bml clear task sh (#437) (v-kkhuang) * Dev 1.1.20 webank acrosscluster (#442) (lemonjuice) * Fix ConcurrentEC max running number (peacewong) * need to check fileSystem == cache fileSystem (peacewong) * fix compile error (aiceflower) * 【1.1.20】buf fix (#443) (v-kkhuang) * add storage info method (peacewong) * add hook execute exception (peacewong) * 1. update task execute thread (#444) (peacewong) * Update version to 1.2.0-wds (Casion) * remove write to file log (aiceflower) * code review fix (#445) (v-kkhuang) * update param check (aiceflower) * [1.1.20]support dss crosscluster filiter (#446) (v-kkhuang) * [1.1.20]fix bml get new version error (#447) (v-kkhuang) * [1.1.20]revent row limit 5000 (#448) (v-kkhuang) * add hive ds annotation (aiceflower) * merge 1.1.20 to 1.2.0 (#449) (Casion) * update monitor logger print (#450) (v-kkhuang) * chore: optimize log component; adjust globalhistory search condition; bug fix (#451) (Yonghao Mei) * Update version to 1.3.0-wds (Casion) * update openfile api remove col length limit for data service (#453) (#455) (Casion) * 代码格式化 (peacewong) * 优化状态翻转逻辑 (peacewong) * 增加登录态缓存 (peacewong) * fix jdbc nullexpection error (#457) (v-kkhuang) * Dev 1.3.0 johnnwang (#456) (peacewong) * add taskId label (#460) (aiceflower) * add participle to the pipeline (#458) (v-kkhuang) * [1.3.0]add error code 13010, 43019 (#459) (v-kkhuang) * support appconn update healthy status (#461) (aiceflower) * [1.3.0]Task logs support Yarn link redirection (#462) (v-kkhuang) * add healthy log (#463) (aiceflower) * 为并发引擎增加execute once 自动退出 (peacewong) * Code optimization (#464) (v-kkhuang) * add shutdown for unhealthy node (#465) (aiceflower) * fix column_limit_display not set error (aiceflower) * code format (aiceflower) * add result col over size msg (aiceflower) * code format (aiceflower) * add enableLimit for client (aiceflower) * update error code 22001 (#466) (v-kkhuang) * update task log url (#467) (v-kkhuang) * Optimize log data yarn URL (#468) (v-kkhuang) * upd: ecm management page add batch option;Optimize large result set displaying;across cluster queue management input limit adjust (#469) (Yonghao Mei) * Update version to 1.4.0-wds (Casion) * [1.4.0]New department administrator added to the global history page (#473) (v-kkhuang) * JobHistoryClear adds configurable days (#472) (v-kkhuang) * Upload audit logs from Linkis to HDFS and delete them (#474) (v-kkhuang) * [1.4.0]Add ecminstance parameter filtering to the list interface (#475) (v-kkhuang) * engine reuse support resource judge (#471) (aiceflower) * remove once label to reuse ec (#470) (aiceflower) * Dev 1.3.0 webank hive fix (#477) (peacewong) * add partiton check interceptor (#478) (aiceflower) * add log (aiceflower) * filter ide creator (aiceflower) * filter ide creator (aiceflower) * code format (aiceflower) * update fs msg (#480) (v-kkhuang) * [1.4.0] department user view jobhistory log (#479) (v-kkhuang) * update reuse logic (aiceflower) * enable partition check by default (aiceflower) * remove redundent log (aiceflower) * [1.4.0]bug fix (#481) (v-kkhuang) * revert partition check (#483) (aiceflower) * Dev 1.4.0 webank appconn optimize (#482) (peacewong) * [1.4.0]bug fix (#484) (v-kkhuang) * [1.4.0]add error code 43042,43043 (#476) (v-kkhuang) * 增加db空闲异常的catch (peacewong) * fix write rs error when ThreadDath (#485) (aiceflower) * code review fix (aiceflower) * code review fix (#487) (v-kkhuang) * [1.4.0]slow sql fix (#486) (v-kkhuang) * add engineType check for ec reuse (aiceflower) * Revert "add engineType check for ec reuse" (aiceflower) * ec reuse add engine type judge (aiceflower) * add delimiter for args param (#490) (GLeXios) * [1.4.0]Monitor Code optimization (#489) (v-kkhuang) * 修复offline的异常 (peacewong) * Update version to 1.5.0-wds (Casion) * Security Work Order Repair (#491) (v-kkhuang) * 调整强退的逻辑和次数 (peacewong) * chore: 1.4.0 web (#493) (Yonghao Mei) * support jdbc ec (#503) (aiceflower) * add hive cluster task support (#496) (Casion) * [1.5.0]Obtain MD5 value based on hdfs file name (#497) (v-kkhuang) * [1.5.0]Add token verification interface (#499) (v-kkhuang) * add service version (#500) (v-kkhuang) * [1.5.0]Guidelines for adding resource parameters in task logs (#501) (v-kkhuang) * hive add mapreduce.job.reduce.slowstart.completedmaps conf (#494) (v-kkhuang) * monitor add dept task index to ims (#498) (v-kkhuang) * Optimize error code information (#506) (v-kkhuang) * [1.5.0]token add audit log (#505) (v-kkhuang) * hive cluseter support oncelabel (#507) (aiceflower) * 1.fix linkiscli npe bug. 2.support jobserver/BdpClient config (#508) (mhHao) * 1.5.0特性 (#509) (peacewong) * add once label (aiceflower) * Dev 1.5.0 logs add conf tips (#511) (v-kkhuang) * [1.5.0]Code optimization (#510) (v-kkhuang) * 增加随机延迟 (peacewong) * add once label for hive (aiceflower) * code format (#512) (aiceflower) * update add once label logic (aiceflower) * code optimization (aiceflower) * 修改引擎的task id为唯一的任务id (peacewong) * [1.5.0]Fix a bug where resources are negative (#513) (v-kkhuang) * update error msg (#514) (aiceflower) * update error message (aiceflower) * runtime label judge (aiceflower) * Deal with empty extraParams on starrocks (#515) (jefftlin) * optimization code (aiceflower) * add log (aiceflower) * add default for empty label (aiceflower) * Dev 1.5.0 bug fix (#516) (v-kkhuang) * 修改默认值为5 (peacewong) * add sso login (peacewong) * 修改方法 (peacewong) * 将sso-login开头的请求处理都单独 (peacewong) * remove log (aiceflower) * [1.5.0]code review fix (#518) (v-kkhuang) * fix hive log yarn url bug (#517) (v-kkhuang) * 增加上固定数 (peacewong) * Spring 升级 (peacewong) * fix build (peacewong) * fix build (peacewong) * fix build (peacewong) * Spring 升级 (peacewong) * fix build (peacewong) * Spring 升级支持 (peacewong) * 修复build 问题 (peacewong) * 移除knife4j (peacewong) * 移除knife4j 回退 (peacewong) * 增加依赖 (peacewong) * 排除不必要的依赖 (peacewong) * 代码优化 (peacewong) * 移除knife4j (peacewong) * 修复异常信息不打印问题 (peacewong) * 优化空指针问题 (peacewong) * hive 需要引入jersery client (peacewong) * update version to 1.6.0 (peacewong) * Add built-in variables: run_mon_now (#526) (v-kkhuang) * Modify the default expiration time of the token (#525) (v-kkhuang) * DBS interface supports returning libraries that can be written (#524) (v-kkhuang) * merge 1.5.0 to 1.6.0 (#528) (aiceflower) * ECM 支持更多退出码重试,以及移除不必要的上报 (peacewong) * 增加admin 相关信息 (peacewong) * fix file not exit error (#531) (v-kkhuang) * [1.6.0]Optimizing Linkis built-in parameters (#530) (v-kkhuang) * [1.6.0]Kill existing engine after updating configuration (#527) (v-kkhuang) * task submit to fix ec support (#529) (aiceflower) * [1.6.0]Optimize fs and use Hadoop to read (#520) (v-kkhuang) * Dev 1.6.0 kill engine by conf (#532) (v-kkhuang) * fix compile error (aiceflower) * Fix NPE (peacewong) * Security work order processing (#533) (v-kkhuang) * fix variable error (#534) (v-kkhuang) * shell 引擎设置job 信息到env id (peacewong) * 解决FS递归同名目录没有展示的问题 (peacewong) * 1.add isValid parms (#535) (v-kkhuang) * remove taskId label (#536) (aiceflower) * Fix python 进程不退出 (peacewong) * Dev 1.6.0 fix bug (#537) (v-kkhuang) * 增加日志打印 (peacewong) * chore: 1.6.0 (#538) (Yonghao Mei) * [1.6.0]spark add dependency for fix bug (#539) (v-kkhuang) * [1.6.0]code review fix (#540) (v-kkhuang) * bug fix (aiceflower) * code optimization (aiceflower) * [1.6.0] code review fix (#541) (v-kkhuang) * Fix the issue of multiple log prints (#542) (v-kkhuang) * Dev 1.6.0 fix bug (#543) (v-kkhuang) * 支持设置Feign超时参数 (peacewong) * Fix NPE (peacewong) * Once Client 优化支持单例 (peacewong) * 主动推送日志修改,支持缓存t (peacewong) * update version from 1.6.0 to 1.7.0 (aiceflower) * 日志优化 (peacewong) * 修复大小写问题 (peacewong) * EC结果集路径优化 (peacewong) * 移除无用代码 (peacewong) * Bug 修复 (peacewong) * 增加数据库相关访问的Retry (peacewong) * 修复异常设置错误问题 (peacewong) * merge 1.6.0 update to 1.7.0 (#545) (aiceflower) * update template match multline (aiceflower) * 调整pes的多个模块 (peacewong) * merge server module of basedata-manager、errorde-server、variable、scriptis-dev-server to be linkis-publicservice module (peacewong) * 修复编译问题 (peacewong) * 修复编译问题 (peacewong) * 解决编译问题 (peacewong) * Optimization of Jobhistory Task Information Query Interface (“v_kkhuang”) * revent Optimization of Jobhistory Task Information Query Interface (“v_kkhuang”) * add nebula ecp (#553) (aiceflower) * update monitor sh timeout conf && ec clear sh timeout conf (#552) (v-kkhuang) * [wip][1.7.0]jobhistory add fields such as whether to reuse engines (#548) (v-kkhuang) * Dev 1.7.0 jobhistory upadte get (#554) (v-kkhuang) * [wip][1.7.0]ECM resource reset and user resource reset (#547) (v-kkhuang) * support query specified column with get result set (#551) (aiceflower) * fix compile error (aiceflower) * fix comiple error (aiceflower) * code format (aiceflower) * update openFile (#555) (Casion) * update openFile api (#557) (Casion) * optimization code (#558) (Casion) * fix: fix token npe (#559) (taoran1250) * [1.7.0]Add download button to the history of the management console (#560) (v-kkhuang) * code optimization (#561) (Casion) * Resource reset code optimization (#564) (v-kkhuang) * fix: optimizing code (#562) (taoran1250) * Dev 1.7.0 webank fsfix (#563) (aiceflower) * [1.7.0]Dev 1.7.0 bug fix (#565) (v-kkhuang) * add code type and run type relation for nebula (aiceflower) * linkis-cli support parameter passing in http headers. (HmhWz) * 代码格式化 (peacewong) * 增加请求拦截耗时打印 (peacewong) * Dev 1.7.0 bug fix (#567) (v-kkhuang) * code optimization (aiceflower) * fix gateway start error (aiceflower) * Dev 1.7.0 bug fix (#569) (v-kkhuang) * 修改代码更加优雅 (peacewong) * Dev 1.7.0 bug fix (#570) (v-kkhuang) * fix IAST bug (#568) (taoran1250) * [1.7.0]code review fix (#571) (v-kkhuang) * Fix Security Work Order: Upgrade Spring Version:5.3.27 -> 5.3.34 (#573) (v-kkhuang) * 【1.7.0】SDK interface adds new parameters (#574) (v-kkhuang) * chore: add spring-boot-actuator dependency (#572) (taoran1250) * optimization & format code (aiceflower) * IAST BUG FIX (#575) (taoran1250) * 【1.7.0】fix jobhistory query with code (#576) (v-kkhuang) * [1.7.0]fix download slowly (#577) (v-kkhuang) * 结果集路径bug修复 (peacewong) * Dev 1.7.0 bug fix (#578) (v-kkhuang) * 代码格式化 (peacewong) * 修复ConcurrentHashMap 遍历Values的并发问题 (peacewong) * 增加用户部门信息查询的接口 (peacewong) * rpc white check bug fix (aiceflower) * 增加Spark opts (peacewong) * Dev 1.7.0 bug fix (#579) (v-kkhuang) * Dev 1.7.0 bug fix (#580) (v-kkhuang) * Dev 1.7.0 bug fix (#581) (v-kkhuang) * optimazation code (aiceflower) * code optimization (aiceflower) * null value bug fix (aiceflower) * update sdk action (#589) (v-kkhuang) * merge 1.7.1 to 1.8.0 (#592) (aiceflower) * YYYYMMddHH 变量时分秒修改为取当前值 (peacewong) * Fix jdbc 复杂类型的值转换bug (peacewong) * feat: add variables print in log && add nebula space config (#595) (taoran1250) * 1.8.0 requirements (#596) (aiceflower) * code review (#598) (aiceflower) * update python module download dir to engine tmp dir (aiceflower) * update version from 1.7.0 to 1.8.0 (aiceflower) * code format (aiceflower) * update python hook support zip file (aiceflower) * feat: add multiple result set support (#601) (finyuq) * update rsa key length (aiceflower) * feat: change ims alert to ecc_receiver (#600) (taoran1250) * code format (aiceflower) * feat: add python version to Log && mybatis-plus dependency update (#602) (taoran1250) * update python …
1 parent 5df9bc3 commit ecde367

458 files changed

Lines changed: 37294 additions & 4388 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/README.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Linkis AI 开发文档导航
2+
3+
> **版本信息**
4+
> - 文档版本: 1.0.0
5+
> - 最后更新: 2025-01-28
6+
> - 适用版本: Apache Linkis 1.17.0+
7+
8+
---
9+
10+
## 🚀 快速开始
11+
12+
### 新手必读(按顺序阅读)
13+
1. **[项目核心规约](./project-context.md)** - 包含技术栈、架构设计、开发规范和模板
14+
2. **[强制性开发规则](./rules.md)** - 必须无条件遵守的开发规则
15+
3. **[模块文档](#模块文档索引)** - 根据你要开发的功能选择对应模块
16+
17+
### 常见开发场景快速跳转
18+
- 🆕 新增 REST 接口 → [REST接口开发模板](#rest接口开发)
19+
- ⚙️ 添加配置项 → [配置管理规范](#配置管理)
20+
- 🗄️ 修改数据库 → [数据库变更规范](#数据库变更)
21+
- 🐛 异常处理 → [异常处理规范](#异常处理)
22+
- 📝 日志记录 → [日志规范](#日志规范)
23+
24+
---
25+
26+
## 📚 核心文档索引
27+
28+
### 🎯 开发规范文档
29+
| 文档 | 用途 | 何时查看 |
30+
|------|------|----------|
31+
| [project-context.md](./project-context.md) | 项目角色定位、技术栈、架构设计、开发模板 | 开始任何开发工作前必读 |
32+
| [rules.md](./rules.md) | 强制性开发规则、需求实现步骤 | 每次开发新需求时参考 |
33+
34+
### 🏗️ 模块文档索引
35+
36+
#### 微服务治理服务(基础设施层)
37+
| 服务 | 文档 | 主要功能 |
38+
|------|------|----------|
39+
| Gateway | [gateway.md](./modules/microservice-governance/gateway.md) | API网关、路由转发、安全认证 |
40+
| Eureka | [eureka.md](./modules/microservice-governance/eureka.md) | 服务注册与发现 |
41+
| 概览 | [README.md](./modules/microservice-governance/README.md) | 微服务治理服务概述 |
42+
43+
#### 计算治理服务(核心业务层)
44+
| 服务 | 文档 | 主要功能 |
45+
|------|------|----------|
46+
| Entrance | [entrance.md](./modules/computation-governance/entrance.md) | 任务提交入口、调度管理 |
47+
| JobHistory | [jobhistory.md](./modules/computation-governance/jobhistory.md) | 任务历史记录查询 |
48+
| Manager | [manager.md](./modules/computation-governance/manager.md) | 资源管理、应用管理 |
49+
| ECM | [ecm.md](./modules/computation-governance/ecm.md) | 引擎连接管理 |
50+
| 概览 | [README.md](./modules/computation-governance/README.md) | 计算治理服务概述 |
51+
52+
#### 公共增强服务(支撑服务层)
53+
| 服务 | 文档 | 主要功能 |
54+
|------|------|----------|
55+
| PublicService | [publicservice.md](./modules/public-enhancements/publicservice.md) | 公共服务、文件管理 |
56+
| Configuration | [configuration.md](./modules/public-enhancements/configuration.md) | 配置管理 |
57+
| BML | [bml.md](./modules/public-enhancements/bml.md) | 大数据物料库 |
58+
| DataSource | [datasource.md](./modules/public-enhancements/datasource.md) | 数据源管理 |
59+
| Context | [context.md](./modules/public-enhancements/context.md) | 上下文服务 |
60+
| Monitor | [monitor.md](./modules/public-enhancements/monitor.md) | 监控服务 |
61+
| 概览 | [README.md](./modules/public-enhancements/README.md) | 公共增强服务概述 |
62+
63+
---
64+
65+
## 🔍 按功能快速查找
66+
67+
### <a name="rest接口开发"></a>REST接口开发
68+
- **开发模板**: [project-context.md - REST接口层](./project-context.md#1-rest接口层)
69+
- **API规范**: [project-context.md - API设计规范](./project-context.md#api设计规范)
70+
- **参考示例**:
71+
- Entrance接口: [entrance.md - API Interfaces](./modules/computation-governance/entrance.md#api-interfaces)
72+
- Configuration接口: [configuration.md - API Interfaces](./modules/public-enhancements/configuration.md#api-interfaces)
73+
74+
### <a name="配置管理"></a>配置管理
75+
- **配置规范**: [project-context.md - 配置管理规范](./project-context.md#配置管理规范)
76+
- **配置示例库**: [project-context.md - 常用配置示例库](./project-context.md#常用配置示例库)
77+
- **配置模板**: [project-context.md - 配置类](./project-context.md#4-配置类)
78+
- **参考实现**: linkis-jobhistory/conf/JobhistoryConfiguration
79+
80+
### <a name="数据库变更"></a>数据库变更
81+
- **变更规则**: [rules.md - 数据库修改原则](./rules.md#数据库修改原则)
82+
- **DDL脚本位置**: `linkis-dist/package/db/linkis_ddl.sql`
83+
- **DML脚本位置**: `linkis-dist/package/db/linkis_dml.sql`
84+
- **表结构参考**: 各模块文档的 "Database Table Structures" 章节
85+
86+
### <a name="异常处理"></a>异常处理
87+
- **异常规范**: [project-context.md - 异常处理规范](./project-context.md#异常处理规范)
88+
- **统一异常**: `org.apache.linkis.common.exception.LinkisException`
89+
- **常见错误**: [project-context.md - 常见错误及避免方法](./project-context.md#常见错误及避免方法)
90+
91+
### <a name="日志规范"></a>日志规范
92+
- **日志规范**: [project-context.md - 日志规范](./project-context.md#日志规范)
93+
- **Logger定义**: 必须使用 `LoggerFactory.getLogger(ClassName.class)`
94+
- **日志级别**: ERROR/WARN/INFO/DEBUG 使用场景
95+
96+
---
97+
98+
## 🎨 开发模板快速复制
99+
100+
### 新增功能完整流程
101+
```
102+
1. 查看 rules.md - 需求实现步骤
103+
2. 创建需求文档和设计文档
104+
3. 使用 project-context.md 中的代码模板:
105+
- REST接口层模板
106+
- 服务层模板
107+
- 数据访问层模板
108+
- 配置类模板
109+
4. 添加功能开关(默认false)
110+
5. 记录数据库变更
111+
6. 编写测试和文档
112+
```
113+
114+
### REST接口模板快速链接
115+
👉 [project-context.md - 新功能开发模板](./project-context.md#新功能开发模板)
116+
117+
### 配置类模板快速链接
118+
👉 [project-context.md - 配置类](./project-context.md#4-配置类)
119+
120+
---
121+
122+
## ⚠️ 重要提醒
123+
124+
### 🚫 禁止操作(来自 rules.md)
125+
- **数据库结构**: 除非明确指定,严禁修改现有表结构
126+
- **第三方依赖**: 不允许引入新的第三方依赖库
127+
- **核心接口**: 不得修改现有公共接口的签名
128+
129+
### ✅ 必须遵守
130+
- **最小改动原则**: 所有功能实现必须遵循最小改动原则
131+
- **功能可配置**: 所有功能必须增加功能开关,默认关闭
132+
- **向后兼容**: 新增功能必须考虑向后兼容性
133+
134+
---
135+
136+
## 💡 开发技巧
137+
138+
### 编程语言选择
139+
- **Java**: REST API、Service层、Entity类、配置类
140+
- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象
141+
142+
### 字符编码
143+
统一使用 `StandardCharsets.UTF_8`,禁止使用字符串 `"UTF-8"`
144+
145+
### 统一返回体
146+
所有REST接口返回 `org.apache.linkis.server.Message`
147+
148+
---
149+
150+
## 📖 如何使用这些文档
151+
152+
### 场景1: 我要在 Entrance 服务中新增一个接口
153+
1. 阅读 [entrance.md](./modules/computation-governance/entrance.md) 了解服务结构
154+
2. 查看 [project-context.md - REST接口层模板](./project-context.md#1-rest接口层)
155+
3. 参考 entrance.md 中现有接口实现
156+
4. 遵循 [rules.md](./rules.md) 中的开发规则
157+
5. 添加功能开关配置
158+
159+
### 场景2: 我需要添加一个新的配置项
160+
1. 查看 [project-context.md - 配置管理规范](./project-context.md#配置管理规范)
161+
2. 参考 [project-context.md - 配置类模板](./project-context.md#4-配置类)
162+
3. 查看 `JobhistoryConfiguration` 实现示例
163+
4. 在当前模块的 conf 目录下的 Configuration 类中添加
164+
165+
### 场景3: 我需要修改数据库表
166+
1. 查看 [rules.md - 数据库修改原则](./rules.md#数据库修改原则)
167+
2. 确认是否能通过新增字段实现(优先选择)
168+
3. 将变更记录到 `linkis-dist/package/db/linkis_ddl.sql`
169+
4. 如有初始化数据,记录到 `linkis-dist/package/db/linkis_dml.sql`
170+
171+
---
172+
173+
## 🔄 文档更新记录
174+
175+
| 版本 | 日期 | 更新内容 | 更新人 |
176+
|------|------|----------|--------|
177+
| 1.0.0 | 2025-01-28 | 创建导航文档,优化文档结构 | AI |
178+
179+
---
180+
181+
## 📞 帮助与反馈
182+
183+
如果文档中有不清楚的地方,请:
184+
1. 先查看对应模块的详细文档
185+
2. 查看 project-context.md 中的开发模板和示例
186+
3. 参考现有代码实现
187+
188+
**记住**: 遵循规范比快速开发更重要!
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Computation Governance Services
2+
3+
The computation governance services handle the core computation task lifecycle management in Linkis.
4+
5+
## Service Modules
6+
7+
- [Entrance Service](./entrance.md) - Task submission and entrance point
8+
- [Manager Service](./manager.md) - Resource and application management
9+
- [ECM Service](./ecm.md) - Engine Connection Manager
10+
- [JobHistory Service](./jobhistory.md) - Task execution history tracking
11+
12+
## Overview
13+
14+
These services form the core of Linkis' computation governance capabilities, managing the complete lifecycle of computation tasks from submission to execution and monitoring.
15+
16+
## Common Features
17+
18+
### Task Lifecycle Management
19+
- Task submission and validation
20+
- Task scheduling and resource allocation
21+
- Task execution monitoring
22+
- Task result management
23+
- Task error handling and recovery
24+
25+
### Engine Management
26+
- Dynamic engine connection creation
27+
- Engine lifecycle management
28+
- Engine resource monitoring
29+
- Engine scaling capabilities
30+
31+
### Resource Governance
32+
- Multi-tenant resource isolation
33+
- Load balancing across engines
34+
- Resource usage tracking
35+
- Quota management
36+
37+
## API Interface Summary
38+
39+
### Entrance Service APIs
40+
- Task submission: `POST /api/entrance/submit`
41+
- Task status query: `GET /api/entrance/{id}/status`
42+
- Task progress: `GET /api/entrance/{id}/progress`
43+
- Task log retrieval: `GET /api/entrance/{id}/log`
44+
- Task cancellation: `GET /api/entrance/{id}/kill`
45+
46+
### Manager Service APIs
47+
- Engine instance management
48+
- Resource allocation and monitoring
49+
- Node status querying
50+
- Engine creation requests
51+
52+
### ECM Service APIs
53+
- Engine connection management
54+
- Engine lifecycle operations
55+
- Resource reporting
56+
- Engine metrics collection
57+
58+
### JobHistory Service APIs
59+
- Job history querying
60+
- Job detail retrieval
61+
- Job statistics reporting
62+
63+
## Database Schema Summary
64+
65+
### Job History Group Table
66+
```sql
67+
CREATE TABLE `linkis_ps_job_history_group_history` (
68+
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment',
69+
`job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId',
70+
`submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job',
71+
`execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job',
72+
`source` text DEFAULT NULL COMMENT 'job source',
73+
`labels` text DEFAULT NULL COMMENT 'job labels',
74+
`params` text DEFAULT NULL COMMENT 'job params',
75+
`progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress',
76+
`status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout',
77+
`log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log',
78+
`error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails',
79+
`error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails',
80+
`created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time',
81+
`updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time',
82+
`instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances',
83+
`metrics` text DEFAULT NULL COMMENT 'Job Metrics',
84+
`engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type',
85+
`execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path',
86+
`result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets',
87+
`observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job',
88+
PRIMARY KEY (`id`),
89+
KEY `idx_created_time` (`created_time`),
90+
KEY `idx_submit_user` (`submit_user`)
91+
);
92+
```
93+
94+
### Job History Detail Table
95+
```sql
96+
CREATE TABLE `linkis_ps_job_history_detail` (
97+
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment',
98+
`job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory',
99+
`result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets',
100+
`execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job',
101+
`result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array',
102+
`job_group_info` text DEFAULT NULL COMMENT 'Job group info/path',
103+
`created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time',
104+
`updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time',
105+
`status` varchar(32) DEFAULT NULL COMMENT 'status',
106+
`priority` int(4) DEFAULT 0 COMMENT 'order of subjob',
107+
PRIMARY KEY (`id`)
108+
);
109+
```
110+
111+
### Common Lock Table
112+
```sql
113+
CREATE TABLE `linkis_ps_common_lock` (
114+
`id` int(11) NOT NULL AUTO_INCREMENT,
115+
`lock_object` varchar(255) COLLATE utf8_bin DEFAULT NULL,
116+
`locker` VARCHAR(255) CHARSET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'locker',
117+
`time_out` longtext COLLATE utf8_bin,
118+
`update_time` datetime DEFAULT CURRENT_TIMESTAMP,
119+
`create_time` datetime DEFAULT CURRENT_TIMESTAMP,
120+
PRIMARY KEY (`id`),
121+
UNIQUE KEY `uniq_lock_object` (`lock_object`)
122+
);
123+
```
124+
125+
## RPC Methods Summary
126+
127+
### Entrance Service RPCs
128+
- `submitTask(TaskRequest request)`
129+
- `getTaskStatus(String taskId)`
130+
- `cancelTask(String taskId)`
131+
- `getTaskResult(String taskId)`
132+
133+
### Manager Service RPCs
134+
- `requestEngine(EngineRequest request)`
135+
- `releaseEngine(String engineId)`
136+
- `getEngineStatus(String engineId)`
137+
- `getNodeMetrics(String nodeId)`
138+
139+
### ECM Service RPCs
140+
- `createEngineConnection(EngineCreateRequest request)`
141+
- `terminateEngineConnection(String engineId)`
142+
- `reportEngineResourceUsage(String engineId, ResourceUsage usage)`
143+
- `getEngineMetrics(String engineId)`
144+
145+
### JobHistory Service RPCs
146+
- `saveJobHistory(JobHistory history)`
147+
- `queryJobHistory(JobHistoryQuery query)`
148+
- `getJobDetails(Long jobId)`
149+
- `updateJobStatus(Long jobId, String status)`
150+
151+
## Dependencies
152+
153+
- linkis-commons - Shared utilities
154+
- linkis-protocol - Communication protocols
155+
- linkis-rpc - Remote procedure calls
156+
- Various engine connection plugins
157+
- Spring Cloud ecosystem

0 commit comments

Comments
 (0)