Merge branch 'RESTAPI-1203-Kibana-dashboard-for-F7t-K8s' into 'master'

Added JSON log for Kibana

See merge request firecrest/firecrest!321

.gitlab-ci.yml

@@ -96,7 +96,7 @@ deploy_dev:
         helm uninstall -n ${CI_NAMESPACE_DEV} "$app" || true
         helm install --wait --wait-for-jobs --timeout 240s -n ${CI_NAMESPACE_DEV} -f values-dev.yaml "$app" $app;
       done
-    # Cluster is deployed separatelly ALWAYS with tag = latest
+    # Cluster is deployed separately ALWAYS with tag = latest
     - |
       echo -e "global:\n  registry: ${CI_REGISTRY_PREFIX}\n  tag: latest\n  namespace: ${CI_NAMESPACE_DEV}\n" > values-cluster-dev.yaml;
 

CHANGELOG.md

@@ -14,22 +14,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added new "general" section to status/parameters describing `FIRECREST_VERSION` and `FIRECREST_BUILD` timestamp
 - Environment variable `F7T_HOME_ENABLED` to set `False` if `$HOME` is not mounted on systems executing FirecREST commands
 - Add support in the `GET /compute/jobs` endopint to poll for jobs of any user
+- Environment variable `F7T_LOG_KIBANA` to enable/disable the JSON format log reporting, detected and parsed by Kibana, and the related Helm chart's annotations.
 
 ### Changed
 
-- SLURM scheduler now uses `--export` option for passing environment variables to a job
-- Variable `F7T_REALM_RSA_PUBLIC_KEYS` changed to `F7T_AUTH_PUBLIC_KEYS`
-- Variable `F7T_REALM_RSA_TYPE_` changed to `F7T_AUTH_ALGORITHMS`
-- Added default values on helm charts
-- Upgrade `requests` library to version `2.32.0`
-
+- SLURM scheduler now uses `--export` option for passing environment variables to a job.
+- Variable `F7T_REALM_RSA_PUBLIC_KEYS` changed to `F7T_AUTH_PUBLIC_KEYS`.
+- Variable `F7T_REALM_RSA_TYPE_` changed to `F7T_AUTH_ALGORITHMS`.
+- Added default values on helm charts.
+- Upgrade `requests` library to version `2.32.0`.
+- System availability is tested using `whoami` instead of `ls -l` on a filesystem. Filesystem failures reported in `status/systems/<SYSTEM>` do not affect the availability of a system.
+- Filesystem check is executed with the command `ls -1f`, skipping listing and sorting of entries.
 
 ### Fixed
 
 - Fix parsing in `GET /utilities/ls` endpoint.
 - The job fields `job_data_out` and `job_file_err` from `GET /compute/jobs` will be empty for jobs that are still pending (so that there is no confusion with older output/error files).
-- Added retry on task creation workflow
-- Error message when `$HOME` is not mounted
+- Added retry on task creation workflow.
+- Error message when `$HOME` is not mounted.
 
 ## [1.16.0]
 
@@ -39,12 +41,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added endpoints `POST /utilities/compress`, `POST /utilities/extract`, `POST /storage/xfer-internal/compress` and `POST /storage/xfer-internal/extract` for file compression and extraction.
 - Added recurisive option to ls utilities command `&recursive=true`.
 - Added the endpoint `/compute/partitions` to retrieve information about partitions in the scheduling queue.
-- Added grep support for tail and head command. `&grep=pattern`
+- Added grep support for tail and head command. `&grep=pattern`.
 - Added `examples` directory for practical use cases of FirecREST.
 
 ### Changed
 
-- Environment variable names
+- Environment variable names:
   - Added: `F7T_CERTIFICATOR_HOST`, `F7T_COMPUTE_HOST`, `F7T_RESERVATION_HOST`, `F7T_STATUS_HOST`, `F7T_STORAGE_HOST`, `F7T_TASKS_HOST`, `F7T_UTILITIES_HOST`
   - Replaced
     - `F7T_SYSTEMS_PUBLIC` by `F7T_SYSTEMS_PUBLIC_NAME`
@@ -65,7 +67,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed error on pipeline when releasing production version.
 - Fixed response in task after timeout in one of the commands.
 - Handle `ChunkedEncodingError` error in task creation to avoid crashing and returning 500 Error.
-- Fixed compute task error message on system not available
+- Fixed compute task error message on system not available.
 
 ## [1.15.0]
 
@@ -88,10 +90,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - Fixed demo images dependency declarations preventing docker-compose to build successfully.
-- Fixed check when submitted an empty batch file on `POST /compute/jobs/upload`
-- Fixed error message when `GET /status/systems` encounters error in one filesystem
-- Fixed SSH connection error catching
-- Fixed secured "ssh-keygen" command execution
+- Fixed check when submitted an empty batch file on `POST /compute/jobs/upload`.
+- Fixed error message when `GET /status/systems` encounters error in one filesystem.
+- Fixed SSH connection error catching.
+- Fixed secured "ssh-keygen" command execution.
 
 ## [1.14.0]
 
@@ -103,15 +105,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add description for each parameter in `GET /status/parameters` response.
 - Add support for Object Storage Tenants in S3v4 object storage. The associated environment variable is `F7T_S3_TENANT` and it can be empty or be `null` or `none` when the tenant is not needed. Otherwise the tenant name has to be set.
 - The task that is returned from a successful `GET /jobs/acct` would returns the attribute `time`, which is `cputime` from slurm. The attribute will remain and `cputime` and `elapsed` will be also returned. Similarly, `time_left` is actually the time of termination of the jobs. `time_left` will remain for compatibility reasons, but `elapsed` attribute will also be returned.
-- Added `F7T_AUTH_ISSUER` to specify the JWT token issuer to be checked by Kong GW
-- Removed `F7T_AUTH_REALM` and `F7T_AUTH_URL` which are no longer needed
+- Added `F7T_AUTH_ISSUER` to specify the JWT token issuer to be checked by Kong GW.
+- Removed `F7T_AUTH_REALM` and `F7T_AUTH_URL` which are no longer needed.
 
 ### Changed
 
-- CI/CD pipeline is now adapted to create helm charts images and push to a repository when TDS or Prod are tagged
-  - Also secrets now can be managed from ExternalSecrets on K8s deployment
-  - Deployment on TDS triggers ArgoCD deployment
-- Demo and k8s deployments have the Swagger UI API specification at unauthenticated `/docs` endpoint
+- CI/CD pipeline is now adapted to create helm charts images and push to a repository when TDS or Prod are tagged:
+  - Also secrets now can be managed from ExternalSecrets on K8s deployment.
+  - Deployment on TDS triggers ArgoCD deployment.
+- Demo and k8s deployments have the Swagger UI API specification at unauthenticated `/docs` endpoint.
 
 ### Fixed
 
@@ -122,24 +124,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - Tasks microservice now provides filtering by a subset of tasks with a `tasks` parameter
-  - `GET /tasks?tasks=<taskid>,<taskid>,<taskid>`
+  - `GET /tasks?tasks=<taskid>,<taskid>,<taskid>`.
 
-- Tasks microservice now reports in task metadata the system for which the task was created
+- Tasks microservice now reports in task metadata the system for which the task was created.
 
-- For storage tasks, now the `source` and `target` path are part of the `data` field on the response for all statuses
+- For storage tasks, now the `source` and `target` path are part of the `data` field on the response for all statuses.
 
-- For certificator container sets up the environment variables `F7T_CA_KEY_PATH` and `F7T_PUB_USER_KEY_PATH` as absolute paths in the container for the CA Private key (`ca-key`) and the user key (`user-key.pub`), respectively. If not set, the default directory will be root (ie, `/ca-key`)
+- For certificator container sets up the environment variables `F7T_CA_KEY_PATH` and `F7T_PUB_USER_KEY_PATH` as absolute paths in the container for the CA Private key (`ca-key`) and the user key (`user-key.pub`), respectively. If not set, the default directory will be root (ie, `/ca-key`).
 
-- For the rest of microservices, the value to set is `F7T_PRIV_USER_KEY_PATH` (`user-key`)
+- For the rest of microservices, the value to set is `F7T_PRIV_USER_KEY_PATH` (`user-key`).
 
 - The `head` endpoint has a new argument: `skip_ending`. The output will be the whole file, without the last NUM bytes/lines of each file.
 
 - The `tail` endpoint has a new argument: `skip_beginning`. The output will start with byte/line NUM of each file.
 
 ### Fixed
 
-- Demo template UI client has been fixed in order to integrate latest changes
-- Fixed correct header when the result of an operation in the system is `Not a directory` to `X-Not-A-Directory`
+- Demo template UI client has been fixed in order to integrate latest changes.
+- Fixed correct header when the result of an operation in the system is `Not a directory` to `X-Not-A-Directory`.
 - Fixed the automatic change of the filename in uploaded files with empty spaces and other special characters.
 - Fixed the issue with parsing `ls` when encountering filenames with the `$` character and whitespace.
 
@@ -149,30 +151,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
-- Flask version upgraded to `2.3.2`
-- Cryptography version upgraded to `39.0.2`
-- Certificator now uses `f7t-base` as base image
+- Flask version upgraded to `2.3.2`.
+- Cryptography version upgraded to `39.0.2`.
+- Certificator now uses `f7t-base` as base image.
 
 ### Fixed
 
 ## [1.12.0]
 
 ### Added
 
-- More tests for `/storage/xfer-external/upload` endpoint
+- More tests for `/storage/xfer-external/upload` endpoint.
 ### Changed
 
-- For `/storage/xfer-external/upload` now is possible for the form parameter `targetPath` to be a file path or a directory path
+- For `/storage/xfer-external/upload` now is possible for the form parameter `targetPath` to be a file path or a directory path.
 
 ### Fixed
 
-- Upgrade to `redis == 4.5.4` in `tasks` API
+- Upgrade to `redis == 4.5.4` in `tasks` API.
 
 ## [1.11.1]
 
 ### Fixed
 
-- Automated process for extracting the version number and using it in K8s OpenAPI pod
+- Automated process for extracting the version number and using it in K8s OpenAPI pod.
 
 ## [1.11.0]
 
@@ -181,7 +183,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add github workflow for automatic releases.
 - Add new endpoints for `head` and `tail` commands.
 - Add endpoint for the `whoami` command.
-- Add `X-Size-Limit` to `/utilities/download` and `/utilities/view` endpoints API specification
+- Add `X-Size-Limit` to `/utilities/download` and `/utilities/view` endpoints API specification.
 
 ### Changed
 

deploy/demo/common/common.env

@@ -19,6 +19,10 @@ F7T_AUTH_ALGORITHMS=RS256
 F7T_AUTH_ROLE='firecrest-sa'
 # DEBUG FLAG
 F7T_DEBUG_MODE=True
+# KIBANA LOG
+F7T_LOG_KIBANA=True
+# LOG TYPE
+F7T_LOG_TYPE='file'
 #-------
 # microservices IPs
 F7T_CERTIFICATOR_IP=192.168.220.11

deploy/docker/base/requirements.txt

@@ -6,4 +6,5 @@ requests==2.32.0
 jaeger_client==4.5.0
 Flask-Opentracing==1.1.0
 gunicorn==22.0.0
-Flask-Caching==2.1.0
+Flask-Caching==2.1.0
+python-json-logger==2.0.7

deploy/k8s/compute/templates/deploy.compute.yaml

@@ -3,8 +3,8 @@ kind: Deployment
 metadata:
   name: deploy-compute
   namespace: {{ .Values.global.namespace }}
-  {{ if ne .Values.global.environment "dev" }}
   annotations:
+  {{ if ne .Values.global.environment "dev" }}
     configmap.reloader.stakater.com/reload: "compute-env-file,common-env-file"    
   {{ end }}
 spec:
@@ -14,6 +14,18 @@ spec:
       app: compute
   template:
     metadata:
+      annotations:
+      {{ if .Values.global.F7T_LOG_KIBANA | default false }}
+        co.elastic.logs/processors.decode_json_fields.fields.0: "message"
+        co.elastic.logs/processors.decode_json_fields.target: ""
+        co.elastic.logs/processors.decode_json_fields.overwrite_keys: "true"
+        co.elastic.logs/processors.decode_json_fields.add_error_key: "true"
+        co.elastic.logs/processors.decode_json_fields.when.regexp.message: '^\{'
+        co.elastic.logs/processors.add_fields.target: ""
+        co.elastic.logs/processors.add_fields.fields.data_stream.type: "logs"
+        co.elastic.logs/processors.add_fields.fields.data_stream.dataset: {{ .Values.global.namespace | replace "-" "." | quote }}
+        co.elastic.logs/processors.add_fields.fields.data_stream.namespace: "firecrest"
+      {{ end }}
       labels:
         app: compute
     spec:

deploy/k8s/config/templates/cm.common.yaml

@@ -25,6 +25,7 @@ data:
   F7T_JAEGER_AGENT: {{ .Values.F7T_JAEGER_AGENT | default "" | quote }}
   F7T_LOG_TYPE: {{ .Values.global.F7T_LOG_TYPE | default "file" | quote }}
   F7T_LOG_PATH: {{ .Values.global.F7T_LOG_PATH | default "/var/log" | quote }}
+  F7T_LOG_KIBANA: {{ .Values.global.F7T_LOG_KIBANA | default "False" | quote }}
   F7T_OBJECT_STORAGE: {{ .Values.F7T_OBJECT_STORAGE | default "s3v4" | quote }}
   F7T_AUTH_PUBLIC_KEYS: {{ include "list.listPubKeys" . | default "" | quote }}
   F7T_AUTH_ALGORITHMS: {{ include "list.listPubKeyTypes" . | default "RS256" | quote }}

deploy/k8s/status/templates/deploy.status.yaml

@@ -3,8 +3,8 @@ kind: Deployment
 metadata:
   name: deploy-status
   namespace: {{ .Values.global.namespace }}
-  {{ if ne .Values.global.environment "dev" }}
   annotations:
+  {{ if ne .Values.global.environment "dev" }}
     configmap.reloader.stakater.com/reload: "status-env-file,common-env-file"
   {{ end }}
 spec:
@@ -14,6 +14,18 @@ spec:
       app: status
   template:
     metadata:
+      annotations:
+      {{ if .Values.global.F7T_LOG_KIBANA | default false }}
+        co.elastic.logs/processors.decode_json_fields.fields.0: "message"
+        co.elastic.logs/processors.decode_json_fields.target: ""
+        co.elastic.logs/processors.decode_json_fields.overwrite_keys: "true"
+        co.elastic.logs/processors.decode_json_fields.add_error_key: "true"
+        co.elastic.logs/processors.decode_json_fields.when.regexp.message: '^\{'
+        co.elastic.logs/processors.add_fields.target: ""
+        co.elastic.logs/processors.add_fields.fields.data_stream.type: "logs"
+        co.elastic.logs/processors.add_fields.fields.data_stream.dataset: {{ .Values.global.namespace | replace "-" "." | quote }}
+        co.elastic.logs/processors.add_fields.fields.data_stream.namespace: "firecrest"
+      {{ end }}
       labels:
         app: status
         role: microservice

deploy/k8s/storage/templates/deploy.storage.yaml

@@ -3,8 +3,8 @@ kind: Deployment
 metadata:
   name: deploy-storage
   namespace: {{ .Values.global.namespace }}
-  {{ if ne .Values.global.environment "dev" }}
   annotations:
+  {{ if ne .Values.global.environment "dev" }}
     configmap.reloader.stakater.com/reload: "storage-env-file,common-env-file"
     secret.reloader.stakater.com/reload: "firecrest-common-secrets"
   {{ end }}
@@ -15,6 +15,18 @@ spec:
       app: storage
   template:
     metadata:
+      annotations:
+      {{ if .Values.global.F7T_LOG_KIBANA | default false }}
+        co.elastic.logs/processors.decode_json_fields.fields.0: "message"
+        co.elastic.logs/processors.decode_json_fields.target: ""
+        co.elastic.logs/processors.decode_json_fields.overwrite_keys: "true"
+        co.elastic.logs/processors.decode_json_fields.add_error_key: "true"
+        co.elastic.logs/processors.decode_json_fields.when.regexp.message: '^\{'
+        co.elastic.logs/processors.add_fields.target: ""
+        co.elastic.logs/processors.add_fields.fields.data_stream.type: "logs"
+        co.elastic.logs/processors.add_fields.fields.data_stream.dataset: {{ .Values.global.namespace | replace "-" "." | quote }}
+        co.elastic.logs/processors.add_fields.fields.data_stream.namespace: "firecrest"
+      {{ end }}
       labels:
         app: storage
     spec:

deploy/k8s/utilities/templates/deploy.utilities.yaml

@@ -3,8 +3,8 @@ kind: Deployment
 metadata:
   name: deploy-utilities
   namespace: {{ .Values.global.namespace }}
-  {{ if ne .Values.global.environment "dev" }}
   annotations:
+  {{ if ne .Values.global.environment "dev" }}
     configmap.reloader.stakater.com/reload: "utilities-env-file,common-env-file"    
   {{ end }}
 spec:
@@ -14,6 +14,18 @@ spec:
       app: utilities
   template:
     metadata:
+      annotations:
+      {{ if .Values.global.F7T_LOG_KIBANA | default false }}
+        co.elastic.logs/processors.decode_json_fields.fields.0: "message"
+        co.elastic.logs/processors.decode_json_fields.target: ""
+        co.elastic.logs/processors.decode_json_fields.overwrite_keys: "true"
+        co.elastic.logs/processors.decode_json_fields.add_error_key: "true"
+        co.elastic.logs/processors.decode_json_fields.when.regexp.message: '^\{'
+        co.elastic.logs/processors.add_fields.target: ""
+        co.elastic.logs/processors.add_fields.fields.data_stream.type: "logs"
+        co.elastic.logs/processors.add_fields.fields.data_stream.dataset: {{ .Values.global.namespace | replace "-" "." | quote }}
+        co.elastic.logs/processors.add_fields.fields.data_stream.namespace: "firecrest"
+      {{ end }}
       labels:
         app: utilities
       generateName: firecrest.utilities-1-

deploy/k8s/values-dev.yaml

@@ -75,6 +75,7 @@ global:
   include_reservations: true
   F7T_DEBUG_MODE: true
   F7T_LOG_TYPE: "stdout"
+  F7T_LOG_KIBANA: true
   F7T_GATEWAY_HOST: "svc-kong"
   F7T_GATEWAY_PORT: "8000"
   F7T_SWAGGER_HOST: "http://svc-openapi"

examples/UI-client-credentials/src/client.py

@@ -266,10 +266,6 @@ def list_jobs_with_f7t(system_name: str, job_ids: list[int]) -> dict:
 
         return {"jobs": [], "error": 1}
 
-
-
-
-
 
 def mkdir(jobName):
 
@@ -314,7 +310,6 @@ def list_jobs():
 
     if f7t_jobs["error"] == 0:
         return {"rows": f7t_jobs["jobs"]}
-
 
     return jsonify(response="Error listing jobs"), 400
 
@@ -353,7 +348,6 @@ def results():
 
 def write_sbatch(jobTemplate, jobName="f7t_test", ntasks=1, account=None, partition=None,
                 constraint=None,reservation=None,jobDir=None, problem_ini_file=None, problem_msh_file=None,step=1,lastJobId=None):
-
     try:
         # sbatch templates directory
         basePath = os.path.abspath(os.path.dirname(__file__))
@@ -378,7 +372,6 @@ def write_sbatch(jobTemplate, jobName="f7t_test", ntasks=1, account=None, partit
             app.logger.error(f"Error: {e}")
         return {"error": 1, "msg":"Couldn't create sbatch file"}
 
-
     try:
         # creating temp sbatch file in client to upload in SYSTEM_NAME
         td = tempfile.mkdtemp(prefix="job")
@@ -407,8 +400,6 @@ def write_sbatch(jobTemplate, jobName="f7t_test", ntasks=1, account=None, partit
             app.logger.error(e)
             app.logger.error(type(e))
         return {"error": 1, "msg":"Couldn't create sbatch file"}
-
-
     return {"error": 0, "path":sbatch_file_path}
 
 
@@ -419,9 +410,7 @@ def background_submit_postproc(jobTemplate,jobName,ntasks,partition,constraint,r
     global POST_JOB_ID
 
     try:
-
         # write sbatch file using the jobTemplate
-
         res = write_sbatch(jobTemplate=jobTemplate, jobName=f"{jobName}_post",ntasks=ntasks,account=USER_GROUP, partition=partition,
                 constraint=constraint, reservation=reservation, jobDir=targetPath, problem_ini_file=problem_ini_file, problem_msh_file=problem_msh_file)
     except Exception as e:
@@ -516,8 +505,6 @@ def background_submit_task(steps,jobTemplate,jobName,ntasks,partition,constraint
             if DEBUG:
                 app.logger.debug(f"Job submission data: {job}")
 
-
-
             lastJobId = job["jobid"]
             data = job
             status = 200