0.3.1 release (#115)

* 1. add stream.log.enable in log-collector

* remove unused file.

* refactor JobLogRestfulApi, add lock for updating register info

* 告警信息查询支持分页

* upd: 告警表格接口与交互改为分页展示

* fix not alarm bug

* upd: 执行历史表格接口与交互改为分页展示

* 作业历史信息查询支持分页

* fix: 表格loading持续存在异常修复

* jobName空判断，长度最长支持2000

* Revert "upd: 执行历史表格接口与交互改为分页展示"

* add support for skipping uris in job-audit-log module

* fix comments in linkis.properties

* refactor StreamisHeartbeatHandler

* add ddl for table 'linkis_stream_register_info'

* update version from '0.3.6-wedatasphere-SNAPSHOT' to '0.3.7-wedatasphere-SNAPSHOT'

* fix bug in RegularUtil

* upd: 执行历史表格接口与交互改为分页展示

* upd: 支持现有执行历史任务metajson查看与下载，接口待联调

* add cost time for audit log.

* add metajson to StreamTask on job starting

* fix val bug

* fix jobconfig bug

* upd: 支持job级资源下载，接口待联调

* upd: 优化启动确认检查逻辑，增加批量启动、停止相关交互

* job materials download

* fix job materials download

* add product name

* fix path name problem

* fix job material download

* upd: 支持job级资源下载，接口联调

* add error codes solutions

* change 'contains' to 'equals' in AuditLogAspect

* Revert "change 'contains' to 'equals' in AuditLogAspect"

This reverts commit 00fba0a.

* change 'contains' to 'equals' in AuditLogAspect

* fix some problems in material download

* fix root path

* fix execute statement

* add costTimeMills in StreamAuditLogMapper.xml

* add field 'job_startup_config' in linkis_stream_task upgrade_dml.sql

* rename field 'job_startup_config' to 'job_start_config' in table linkis_stream_task

* add log in DefaultStreamTaskService

* upd: 审计日志信息增加接口耗时

* fix: metajson 弹窗文案修改

* fix error code and delete linkis error code

* fix error code statement

* fix error code solution in task table

* change root path

* fix execute problem

* upd: 执行历史任务metajson查看与下载接口联调

* fix root path

* fix problems in root path

* fix: metajson 弹窗文案修改

* errorCode solution optimization

* upd: 任务列表增加解决方案跳转链接

* product param optimization

* upd: 批量停止弹窗footer按钮与批量启动保持统一

* upd: 批量启动失败时不显示任务完成弹窗

* fix error code match

* fix String interception

* fix some problems

* fix error code default comments

* 修改配置名称

* fix: 修改baseurl，解决环境异常

* remove fields 'metainfo' and 'worksapcename' in josn of MetaJsonInfo

* fix error code default comments

* fix error code default comments

* fix metajson

* fix metajson

* add flink manager ec init extra params

* fix value

* fix AbstractLinkisJobStateFetcher

* set productName cache

* fix error msg

* upd: 知识库URL改为文件配置式

* fix: 任务列表：重启操作检查失败时关闭所有浮窗而不是仅关闭检查浮窗导致卡死

* fix: 任务列表：批量修改标签后清空选中列表

* fix high available inspect

* 1. remove unused resources.
2. change application.yml to application.properties in streamis-server

* exclude snakeyaml in streamis-server and appconn

* 1. remove unused resources.

* 1. remove unused yml resources.

* 1. exclude yml resources.

* upd: 任务列表：增加对上传的控制

* fix: 批量停止成功后隐藏批量停止弹窗

* update version from '0.3.7-wedatasphere-SNAPSHOT' to '0.3.8-wedatasphere-SNAPSHOT'

* add hook interface.

* add gitignore

* add token authentication for job upload

* add token authentication for project files upload

* add token authentication switch

* fix npe in map put

* upd: 添加水印

* upd: 水印出现页面调整

* add job template upload

* upd: streamis子页面获取用户相关base信息存入localstorage

* job template-change jobContent

* job template-upload jobContent

* upd: job配置与args参数是否支持编辑与上传逻辑一致由配置文件配置

* fix job template-upload

* fix job template-upload

* fix job template upload

* fix job template upload in projectServer

* add template in jobConf

* add template in jobContent

* fix template in jobContent

* add param separates

* fix job template in jobContent

* add interface for getting username

* add JobShutdownHook and DefaultEmptyJobShutdownHook

* change type of timeoutMills from 'int' to 'long'

* fix job template in job config

* fix npe as project does not have a template

* add jsonUtil

* add download interface for job template

* change task param

* upd: 接入用户名接口，水印颜色加深

* upd: job 任务详情增加项目模板查看与下载

* add job shutdown hook

* fix Object

* add configuration in jobContent and jobConfig

* upd: joblist增加是否弹出跳过hook异常弹窗相关逻辑

* upd: jobconfig、detail模板编辑权限调整，配置内容实用模板与自定义并集结果

* upd: job 任务详情模板调整

* chore: 更新依赖与配置文件

* fix jobTemplate download

* register default hook 'emptyJobShutdownHook'

* change registering default hook 'emptyJobShutdownHook'

* fix error code match

* upd: 调整joblist跳过hook异常逻辑

* fix: 解决editable获取的参数错误问题

* change wait radix on executing the hook

* fix: 上传接口content-type设置调整

* fix: joblist 停止poptip改为挂载到body解决被table遮断的问题

* fix: 执行弹窗关闭时同步关闭停止信息弹窗

* fix: 字段错误修复

* fix: hasHook接口字段类型变更为数组，同步更新判断TrueFalse逻辑

* refactor executeJobShutdownHook

* upd: jardetail 项目模板增加为null时的处理

* fix some problems

* fix some problems

* refactor executeJobShutdownHook

* fix job template problems

* fix jetty-server

* upd: jobconfig增加模板与job标记

* upd: jobdetail 模板下载改为前端下载

* fix job shutdown hook

* Revert "fix jetty-server"

This reverts commit 27d23d3.

* fix: jobconfig 模板字段问题修复

* fix job shutdown hook

* fix job template

* fix job template

* fix job shutdown hook

* remove cache pool in DefaultStreamTaskService

* fix: jobconfig 模版字段 问题修复

* upd: flinks参数添加重复相关处理逻辑，job不可重复，template可和job重复

* fix job template

* fix:jobconfig 保存阶段自定义参数判重逻辑去除

* fix error code match

* change the type of flink separate

* fix 0.3.8 code review

* 1. add cancel in StreamisJobShutdownHook
2. refactor msg in hook execution

* fix 0.3.8 code review

* fix 0.3.8 code review

* fix 0.3.8 code review

* fix 0.3.8 code review

* add flink yarn log switch and log

* 1. update version to '0.3.1-SNAPSHOT'
2. update linkis version to '1.5.0'

* 1. update version to '0.3.1'
2. update linkis version to '1.5.0'
3. refactor pom.xml

* fix dss version

* update docs

* update sql

---------

Co-authored-by: gelxiogong <[email protected]>
Co-authored-by: v_nikeyang <[email protected]>
Co-authored-by: bert_cai <[email protected]>

Author: 4 people

.gitignore

@@ -28,3 +28,8 @@ streamis-appconn/target
 /streamis-jobmanager/streamis-job-log/job-log-collector/xspark-streamis-log-collector/target/
 /streamis-jobmanager/streamis-job-log/streamis-job-log-common/target/
 /streamis-jobmanager/streamis-job-log/streamis-job-log-server/target/
+/streamis-jobmanager/streamis-job-errorcode/target/
+/streamis-jobmanager/streamis-job-audit-log/target/
+
+**/._*
+**/.flattened-pom.xml

README-ZH.md

@@ -52,7 +52,7 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com
 
 ## 依赖的生态组件
 
-| 依赖的应用工具    | 描述                                                         | Streamis 兼容版本                               |
+| 依赖的应用工具    | 描述                                                         | Streamis 兼容版本 |
 |--------------|---------------------------------------------------------------|---------------------------------------------|
 | [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | 数据应用开发管理集成框架。以工作流式的图形化拖拽开发体验，将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出应用等，数据应用开发全流程场景需求。 | >= DSS1.1.2（已发布）                         |
 | [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis，通过提供 REST/WebSocket/JDBC/SDK 等标准接口，上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | >= Linkis1.4.0（已发布） |

README.md

@@ -52,7 +52,7 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/
 
 ## Depended ecosystems
 
-| Depended Component | Description | Streamis compatibility       |
+| Depended Component | Description | Streamis compatibility |
 | -------------- | -------------------------------------------------------------- |------------------------------|
 | [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. | >= DSS1.1.2 (Released)    |
 | [Linkis](https://github.com/apache/linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | >= Linkis1.4.0 (Released) |
@@ -115,7 +115,7 @@ please refer to [Streamis Installation and Deployment Document](docs/en_US/0.3.0
 
        For any questions or suggestions, please kindly submit an [issue](https://github.com/WeBankFinTech/Streamis/issues).
 
-       You can scan the QR code below to join our WeChat group to get more immediate response.
+       You can scan the QR code below to join our WeChat and QQ group to get more immediate response.
 
 ![comminicate](images/zh_CN/readme/communication.png)
 

assembly/pom.xml

@@ -20,7 +20,7 @@
     <parent>
         <artifactId>streamis</artifactId>
         <groupId>com.webank.wedatasphere.streamis</groupId>
-        <version>0.3.0</version>
+        <version>0.3.1</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

conf/config.sh

@@ -40,6 +40,6 @@ GATEWAY_PORT=9001
 STREAMIS_SERVER_INSTALL_IP=127.0.0.1
 STREAMIS_SERVER_INSTALL_PORT=9400
 
-STREAMIS_VERSION=0.3.0
+STREAMIS_VERSION=0.3.1
 
 STREAMIS_FILE_NAME="STREAMIS-$STREAMIS_VERSION"

db/streamis_ddl.sql

@@ -222,6 +222,7 @@ CREATE TABLE `linkis_stream_task` (
   `submit_user` varchar(50) DEFAULT NULL,
   `linkis_job_id` varchar(200) DEFAULT NULL,
   `linkis_job_info` mediumtext,
+  `job_start_config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'streamis job startup config',
   PRIMARY KEY (`id`) USING BTREE
 ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT COMMENT='任务表';
 
@@ -252,9 +253,39 @@ CREATE TABLE `linkis_stream_project_privilege` (
 ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='项目权限表';
 
 
+DROP TABLE IF EXISTS `linkis_stream_error_code`;
+
+CREATE TABLE `linkis_stream_error_code` (
+  `id` bigint(20) NOT NULL AUTO_INCREMENT,
+  `error_code` varchar(50) COLLATE utf8_bin NOT NULL,
+  `error_desc` varchar(1024) COLLATE utf8_bin NOT NULL,
+  `error_regex` varchar(1024) COLLATE utf8_bin DEFAULT NULL,
+  `error_type` int(3) DEFAULT '0',
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
+
+DROP TABLE IF EXISTS `linkis_stream_audit_log`;
+
+CREATE TABLE `linkis_stream_audit_log` (
+  `id` bigint(20) NOT NULL AUTO_INCREMENT,
+  `user` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `proxy_user` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_name` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_desc` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_type` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `operate_time` datetime DEFAULT NULL,
+  `input_parameters` text COLLATE utf8_bin,
+  `output_parameters` text COLLATE utf8_bin,
+  `project_name` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `client_ip` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `cost_time_mills` bigint(20) NULL,
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
+
+
 ALTER TABLE `linkis_stream_project` ADD create_time datetime DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD last_update_by varchar(50) DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD last_update_time datetime DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD is_deleted tinyint unsigned DEFAULT 0;
-
+ALTER TABLE linkis_stream_project_files ADD source varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL;
 SET FOREIGN_KEY_CHECKS = 1;