[BUG] Importing a large amount of monitoring config occur poor experience and may trigger a long database transaction

[MasamiYui]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I imported 500 monitoring configuration files, it took 8 minutes.

Expected Behavior

During this time, I didn’t see any data increase on the monitoring center list page, which made me suspect if the process was stuck.

Steps To Reproduce

1.Enter 'monitors' from left menu
2.Select 'Import Monitor'
3.Import a large monitor config file
4.Waiting long time,no data increase

Environment

HertzBeat version(s):1.6.1

Debug logs

No response

Anything else?

I found that MonitorServiceImpl.java used @transactional at the class level,
@Transactional(rollbackFor = Exception.class) public class MonitorServiceImpl implements MonitorService

This occur a poor experience and was unnecessary.

If each configuration import was treated as a separate transaction, I would have seen the data being imported continuously, making me feel that the process was working.

The same issue might also at AlertDefineServiceImpl.java.

[yunfan24]

Hello, I have a question. If each configuration is imported as a separate transaction, how can we ensure the integrity of the imported data? In other words, if one configuration import fails, how should it be handled?

[MasamiYui]

Hello, I have a question. If each configuration is imported as a separate transaction, how can we ensure the integrity of the imported data? In other words, if one configuration import fails, how should it be handled?

I think failure can be ignored may be more reasonable ;or adding a progress bar or a place to check the progress will be much better for batch import.

[tomsun28]

The problem may cause by the detect, the will detect all monitors when import.

        try {
            detectMonitor(monitor, params, collector);
        } catch (Exception ignored) {
        }

+1, adding a progress bar or a place to check the progress will be much better for batch import.

[MasamiYui]

The problem may cause by the detect, the will detect all monitors when import.

        try {
            detectMonitor(monitor, params, collector);
        } catch (Exception ignored) {
        }

+1, adding a progress bar or a place to check the progress will be much better for batch import.

OK，If possible, assign this task to me to optimize.

[tomsun28]

hi welcome, what the idea to solve it, remove detect or use asynchronous detection? It seems that the real-time status of monitoring will change according to this detect

[zuobiao-zhou]

hi welcome, what the idea to solve it, remove detect or use asynchronous detection? It seems that the real-time status of monitoring will change according to this detect

Yes, the current process involves performing a detect before creation to obtain the accurate monitoring availability status.

[zuobiao-zhou]

I think it is essential to correctly display availability after creation; otherwise, it may lead to confusion among users, similar issues were frequently raised in the WeChat feedback group prior to the overhaul of this feature.

[tomsun28]

yes +1

[MasamiYui]

My idea is to maintain the synchronization detection；and add 「user task center」 on the page. When batch imported tasks are executing, the task send some message about process and result to notify 「user task center」 , then 「user task center」 update task status . This way, users can obtain task progress and final results even user already leave the import page

[tomsun28]

My idea is to maintain the synchronization detection；and add 「user task center」 on the page. When batch imported tasks are executing, the task send some message about process and result to notify 「user task center」 , then 「user task center」 update task status . This way, users can obtain task progress and final results even user already leave the import page

hi, if user operate in monitors center, but he turn in the「user task center」to see the result, I feel this is a bit of a jump. How about the where to operate, where to obtain? maybe we can add batch import monitors task history page, like the 「user task center」, it show the task status and import result and so on.

[MasamiYui]

My idea is to maintain the synchronization detection；and add 「user task center」 on the page. When batch imported tasks are executing, the task send some message about process and result to notify 「user task center」 , then 「user task center」 update task status . This way, users can obtain task progress and final results even user already leave the import page

hi, if user operate in monitors center, but he turn in the「user task center」to see the result, I feel this is a bit of a jump. How about the where to operate, where to obtain? maybe we can add batch import monitors task history page, like the 「user task center」, it show the task status and import result and so on.

Great!

[zqr10159]

I think it's just an import monitor, there's no need to make a separate task viewer interface, it seems like you can submit it and then the backend uses sse to push events at fixed intervals, the frontend pops up a progress bar notification in the top right corner, and the same user retains the notification no matter which page they switch to, and you can turn it off after it's over, what do you guys think?

[MasamiYui]

hi,I want to try implement it, this is my initial vision, but i hava some question, and please point out if any problems：
backend:
Emit msg to client and trigger progress update per 10%

public void importConfig(String fileName, InputStream is) {
    //...
    try {
        for (int i = 0; i < totalSize; i++) {
            monitorService.validate(formList.get(i), false);
            monitorService.addMonitor(formList.get(i).getMonitor(), formList.get(i).getParams(),
                    formList.get(i).getCollector(), monitorDto.getGrafanaDashboard());
            // Calculate current progress
            int currentPercent = (i + 1) * 100 / totalSize;
            int completed = i + 1; // Number of completed items
            // Publish event at defined progress thresholds or at the last item
            if (currentPercent >= nextPercent || i == totalSize - 1) {
                emitter.send(SseEmitter.event()
                            .id(String.valueOf(System.currentTimeMillis()))
                            .name("IMPORT_TASK_EVENT")
                            .data(new ImportTaskProcess(fileName, currentPercent, totalSize, completed)));//emit ImportTaskProcess Object
                nextPercent += progressStep;
            }
        }
        //...emit success
    } catch (Exception e) {
          //...emit fail
        ));
    }
}

Front:
Add importTask reponse at notify.component.ts，and then the frontend pops up a progress bar notification in the top right corner，my question is does frontend pops already stantard implement at the project, if not any suggest about style?

[zqr10159]

hi,I want to try implement it, this is my initial vision, but i hava some question, and please point out if any problems：
backend:
Emit msg to client and trigger progress update per 10%

public void importConfig(String fileName, InputStream is) {
    //...
    try {
        for (int i = 0; i < totalSize; i++) {
            monitorService.validate(formList.get(i), false);
            monitorService.addMonitor(formList.get(i).getMonitor(), formList.get(i).getParams(),
                    formList.get(i).getCollector(), monitorDto.getGrafanaDashboard());
            // Calculate current progress
            int currentPercent = (i + 1) * 100 / totalSize;
            int completed = i + 1; // Number of completed items
            // Publish event at defined progress thresholds or at the last item
            if (currentPercent >= nextPercent || i == totalSize - 1) {
                emitter.send(SseEmitter.event()
                            .id(String.valueOf(System.currentTimeMillis()))
                            .name("IMPORT_TASK_EVENT")
                            .data(new ImportTaskProcess(fileName, currentPercent, totalSize, completed)));//emit ImportTaskProcess Object
                nextPercent += progressStep;
            }
        }
        //...emit success
    } catch (Exception e) {
          //...emit fail
        ));
    }
}

Front:
Add importTask reponse at notify.component.ts，and then the frontend pops up a progress bar notification in the top right corner，my question is does frontend pops already stantard implement at the project, if not any suggest about style?

It hasn't been realized yet. You can realize your own style. If necessary, we will modify it again.