DLPX-89763 DLPX-86523 delphix-platform changes

[justsanjeev]

Problem

CIS is looking or a single home directory filesystem mounted at the /home location, currently we have the home dataset is mounted on /export/home

Due to that we see the below issues in the CIS Report

• (1.45) 7402 Status of the '/home' partition in the '/etc/fstab' file
• (1.46) 13248 Status of Mount Partition '/home' using mount command
• (1.47) 7403 Status of the 'nodev' mount option setting for the '/home partition' defined in the '/etc/ fstab' file
• (1.48) 14601 Status of the 'nodev' option for '/home' partition using 'mount' command

Solution

• Mounting the home dataset to /home.
• Post upgrade creating a softlink /export/home to /home so that the exiting tests and dependencies have minimal impact.

We have now implemented the delphix-legacy-link.service for the /export/home -> /home creation, we earlier tried to do the sae in the delphix-platform.services and tested to implement the change with ExecStart and ExecStartPost. It dod not work and then this change was implemented to achieve the same after the delphix-platform.services are up completely.

Testing Done

• Build : appliance-build-orchestrator-pre-push/10952 - ☑️ [Partially]

• Pre Checkin tests 172,051 - 🔴

  • Reason: Failed because of the reason that the build tried to navigate to the directory /export/home before the softlink creation.
  • we created a fix for this issue - https://github.com/delphix/qa-infra/pull/2270 for this successful run.
  • We Re-Built the Pre Checkin Tests - 172175 - ✅

• 2025.2.0.0 VM Upgrade Test:

  • Created a DE from 2025.2.0.0 branch - dlpx-qa-2025200-qar-171974-766f97e2.dlpxdc.co
  • Added the source and target environments on the above engine.
  • Created a dsource from an unstructured dataset created on source, and provisioned a vdb.

• Took the upgrade Image s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4295/upgrade-artifacts/internal-qa.upgrade.tar from above build.

• Upgrade the 2025.2.0.0 Engine (Deferred).

• After Upgrade, on above upgraded DE

  • /export/home -> /home symbolic link creation. - ✅

    sanjeev.rohilla@APAC-SANJEEV ~ % ssh [email protected]
    ([email protected]) Password: 
    delphix@ip-10-110-247-6:~$ ls -ltr /export/home
    lrwxrwxrwx 1 root root 5 Apr 23 08:22 /export/home -> /home
    delphix@ip-10-110-247-6:~$ 
    

  • From /var/log/syslog - The service execution is successful. -- ✅

    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Finished Delphix Appliance Platform Service.
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Create /export/home symlink to /home after platform setup...
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix Nginx reverse proxy...
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix default PostgreSQL database server...
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix Perfomance Statistics...
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Started Delphix Telegraf Metric Collection Agent.
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Docker Application Container Engine...
    Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Ensuring /export/home is a symlink to /home...
    Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: Waiting for /run/containerd/containerd.sock to listen
    Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: num passes: 0
    Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: /export/home is a mountpoint, unmounting...
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: export-home.mount: Succeeded.
    Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: /run/containerd/containerd.sock is listening
    Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Removing existing /export/home directory...
    Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Creating symlink: /export/home -> /home
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: delphix-legacy-link.service: Succeeded.
    Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Finished Create /export/home symlink to /home after platform setup.
    

• Full upgrade ✅

  Apr 23 08:35:04 ip-10-110-247-6 systemd[1]: delphix-legacy-link.service: Succeeded.
  Apr 23 08:35:04 ip-10-110-247-6 systemd[1]: Finished Create /export/home symlink to /home after platform setup.
  

• Screenshot From Engine

• CIS Test: Uploading Compliance_Report_CISReport21Apr2025_dephx3sr1_20250421.pdf… - ✅

We have also tested with the same logic of symbolic link creation in the delphix-platform.service and below is the setup details-

[justsanjeev]

We have made the latest changes here and added the below logic to delphix-platform.service to take care of the /export/home mount point and soft link /export/home -> /home

# Create /export/home symlink to /home if it doesn't already exist
ExecStartPost=/bin/sh -c '\
  echo "Ensuring /export/home is a symlink to /home..."; \
  if mountpoint -q /export/home; then \
    echo "/export/home is a mountpoint, unmounting..."; \
    umount /export/home || { echo "Failed to unmount /export/home"; exit 1; }; \
  fi; \
  if [ -e /export/home ] && [ ! -L /export/home ]; then \
    echo "Removing existing /export/home directory..."; \
    rm -rf /export/home; \
  fi; \
  if [ ! -d /export ]; then \
    mkdir /export; \
  fi; \
  if [ -d /export ] && [ ! -L /export/home ]; then \
    echo "Creating symlink: /export/home -> /home"; \
    ln -s /home /export/home; \
  fi;'

We this change we got a build : appliance-build-orchestrator-pre-push/10842.

• Here we have the successful upgrade tests.
• But to verify @sebroy 's comment, I again the same testing

-- Created an app data unstructured vdb on a target
-- Upgraded the engine
-- delphix-mgmt did not come up due to delphix-platform

Also my /var/log/syslog show

Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1218 [main]initcall_blacklisted =p "initcall %s blacklisted\012"
Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Ensuring /export/home is a symlink to /home...
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: /export/home is a mountpoint, unmounting...
Apr 15 12:04:11 ip-10-110-212-88 sh[156704]: umount: /export/home: target is busy.
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Failed to unmount /export/home
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Control process exited, code=exited, status=1/FAILURE
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Failed with result 'exit-code'.
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: Failed to start Delphix Appliance Platform Service.

Upon executing below

delphix@ip-10-110-212-88:~$ sudo systemctl enable delphix-platform.service
delphix@ip-10-110-212-88:~$ sudo systemctl restart delphix-mgmt

And my log shows that the services has started successfully

Apr 15 12:37:42 ip-10-110-212-88 dynamic-debug[165503]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Ensuring /export/home is a symlink to /home...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: /export/home is a mountpoint, unmounting...
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 systemd[164396]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Removing existing /export/home directory...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Creating symlink: /export/home -> /home
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Finished Delphix Appliance Platform Service.
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Starting Delphix management service...

Setup Present for the same setup: dlpx-qa-2025200-qar-170885-766f97e2.dlpxdc.co
Upgraded from: 2025.2.0.0
Upgrade Tar: s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4235/upgrade-artifacts/internal-qa.upgrade.tar
An Engine retained in the Error State: dlpx-qa-2025200-qar-170888-766f97e2.dlpxdc.co

[justsanjeev]

Now here, we have created an alternative PR where we have added the logic to cleanup the /export/home mount point and creating the soft link /export/home -> /home.

We have added another service delphix-legacy-link.service

This Build is Validated : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/10846/
ManualUpgrade with an Unstructured database mounted: ✅
Manual Upgraded Engine, with deffered and Full Upgrade : dlpx-qa-2025200-qar-170959-766f97e2.dlpxdc.co
Failed Pre Checkin Test Rebuild: pre_checkin

[justsanjeev]

Post changing my code as per above comments, I have Build appliance-build-orchestrator-pre-push/10897 running.