[irods/irods#7229] Add support for iRODS 5 #231

korydraughn · 2024-11-07T17:47:15Z

~~This is a WIP.~~

The change to the shared memory is required because iRODS 5 temporarily allocates more shared memory during a reload of the configuration.

Feel free to ask questions and provide feedback.

irods_testing_environment/irods_setup.py

irods_testing_environment/irods_config.py

irods_testing_environment/irods_setup.py

irods_testing_environment/odbc_setup.py

irods_testing_environment/irods_setup.py

korydraughn · 2024-11-13T02:11:11Z

I've confirmed the changes in this PR work.

There was one thing that caught my eye. When using -vvv, I noticed a strange message which seemed to imply the IrodsController replacement for irodsctl wasn't working. This was observed in the output during setup. I'm not entirely sure it was real. The results of the tests appear to be correct.

I believe it was during federation testing.

korydraughn · 2024-11-13T14:35:56Z

I've confirmed the changes in this PR work.

There was one thing that caught my eye. When using -vvv, I noticed a strange message which seemed to imply the IrodsController replacement for irodsctl wasn't working. This was observed in the output during setup. I'm not entirely sure it was real. The results of the tests appear to be correct.

I believe it was during federation testing.

The following is from a unit test run for iRODS 4.3.3 (using this PR) and is exactly the message I was referring to. The test passed BTW.

2024-11-13 08:39:01,619 - setting up iRODS catalog provider [ubuntu-2004-postgres-14_irods-catalog-provider_1]                                                                         
2024-11-13 08:39:01,619 - executing on [ubuntu-2004-postgres-14_irods-catalog-provider_1] [python3 -c 'from scripts.irods.controller import IrodsController; IrodsController().stop()']
2024-11-13 08:39:01,623 - http://localhost:None "POST /v1.46/containers/52d5b5581ff...2058e6c0aa/exec HTTP/1.1" 201 74                         
2024-11-13 08:39:01,627 - http://localhost:None "POST /v1.46/exec/84fb2a73d01...f71a192b7b58/start HTTP/1.1" 101 0                               
2024-11-13 08:39:01,629 - unable to find user irods: no matching entries in passwd file

Notice the last line from the logs above. It is produced by the following:

irods_testing_environment/irods_testing_environment/irods_setup.py

Lines 439 to 449 in 615a2e2

    
           try: 
        
               if stop_irods(container) != 0: 
        
                   logging.debug(f'[{container.name}] failed to stop iRODS server before setup') 
        
           except Exception as e: 
        
               error_msg = f'[{container.name}] failed to stop iRODS server before setup: {str(e)}' 
        
               if "unable to find user irods" in str(e): 
        
                   # If the user didn't exist at this point then the service probably wasn't started. 
        
                  logging.debug(error_msg) 
        
               else: 
        
                  logging.error(error_msg) 
        
                  raise e

I then did the same thing again, but using tip-of-main for the testing environment and got this output.

2024-11-13 09:29:58,974 - setting up iRODS catalog provider [ubuntu-2004-postgres-14_irods-catalog-provider_1]                                                
2024-11-13 09:29:58,974 - executing on [ubuntu-2004-postgres-14_irods-catalog-provider_1] [/var/lib/irods/irodsctl stop]                                      
2024-11-13 09:29:58,979 - http://localhost:None "POST /v1.46/containers/b74d1b63aef4bd...a8a9e77e3f5c026f0e/exec HTTP/1.1" 201 74
2024-11-13 09:29:58,982 - http://localhost:None "POST /v1.46/exec/a5728b962df32a38fdc...fc055d783447e6/start HTTP/1.1" 101 0      
2024-11-13 09:29:58,984 - unable to find user irods: no matching entries in passwd file

Notice the use of irodsctl and the last log message.

Based on the output of each run, I'm sure this is expected/intentional. Therefore, everything is working correctly.

trel · 2024-11-13T15:03:11Z

but i don't see in the log ... failed to stop iRODS server...

korydraughn · 2024-11-13T15:12:03Z

That log message appears after those lines.

trel · 2024-11-13T15:19:47Z

oh, okay - so we're failing to stop(), and that's 'okay'?

korydraughn · 2024-11-13T15:40:52Z

That code snippet is designed to cover multiple situations based on the state of the world. Based on what I saw for tip-of-main for this repo, I think we're okay.

alanking

Looks good overall. Just some suggested improvements.

irods_testing_environment/irods_config.py

irods_testing_environment/irods_setup.py

irods_testing_environment/odbc_setup.py

projects/almalinux-8/almalinux-8-mariadb-10.11/docker-compose.yml

irods_testing_environment/irods_config.py

trel · 2024-11-13T20:54:10Z

do we have a linter we can run? something that will clean this kind of thing up?

alanking · 2024-11-13T21:41:58Z

Created #233 for linter workflow. I also kinda addressed all the problems it found in this branch: https://github.com/alanking/irods_testing_environment/tree/ruffaction Not sure if we actually want those changes but there they are.

korydraughn · 2024-11-13T22:35:38Z

I say open a PR for that.

alanking · 2024-11-14T16:04:06Z

Okay, I opened the PR here: #234 Left it in draft in case we wanted to put this in first.

korydraughn · 2024-11-14T16:09:16Z

Yes, I'd like to get this PR merged first unless you feel it's easier to merge the one you just opened. The edits in this PR aren't difficult to apply on top of yours.

Thoughts?

alanking · 2024-11-14T16:27:19Z

No preference. I don't think the merge conflict will be too bad, so I don't mind dealing with it in the other PR. In other words, let's put this one in first.

trel · 2024-11-14T17:28:34Z

agreed - linter/formatter after please.

irods_testing_environment/irods_config.py

korydraughn · 2024-12-04T22:17:52Z

Attached issue numbers to commits.

Just one more comment to resolve before this is ready.

alanking

If there's no further reservations about the ODBC drivers, I say # it.

korydraughn · 2024-12-05T14:47:31Z

Still thinking about the ODBC driver issue. I'm going to back that change out so we can investigate it more.

The other changes are fine.

korydraughn · 2024-12-05T14:58:14Z

Don't merge this yet. Got the wrong issue number for irods/irods.

… config.

…ion of federation initialization. Changes to config files in iRODS 5 are not automatically loaded, therefore, the testing environment must instruct the iRODS 5 server to reload its config files before launching the tests.

…on for containers from 64mb to 100mb. The tests for iRODS 5 lean heavily on configuration reloading. This operation results in at least two agent factories appearing, increasing the amount of shared memory needed by the container. This commit makes it so that the tests don't terminate due to a server reload.

… irodsctl. irodsctl was removed in iRODS 5. This commit replaces the testing environment's use of irodsctl with the python code invoked via irodsctl.

… for setup_irods.py.

korydraughn · 2024-12-05T15:00:42Z

Removed the ODBC driver commit and fixed issue numbers.

Ready for merging.

korydraughn changed the title ~~Add support for iRODS 5~~ [irods/irods#7229] Add support for iRODS 5 Nov 7, 2024

trel reviewed Nov 7, 2024

View reviewed changes

irods_testing_environment/irods_setup.py Show resolved Hide resolved

korydraughn commented Nov 7, 2024

View reviewed changes

irods_testing_environment/irods_config.py Outdated Show resolved Hide resolved

korydraughn commented Nov 7, 2024

View reviewed changes

irods_testing_environment/irods_setup.py Show resolved Hide resolved

korydraughn commented Nov 7, 2024

View reviewed changes

irods_testing_environment/irods_setup.py Outdated Show resolved Hide resolved

korydraughn commented Nov 7, 2024

View reviewed changes

irods_testing_environment/odbc_setup.py Outdated Show resolved Hide resolved

korydraughn commented Nov 9, 2024

View reviewed changes

irods_testing_environment/irods_setup.py Outdated Show resolved Hide resolved

korydraughn force-pushed the irods5 branch 2 times, most recently from e2c0b4f to 44cdd5d Compare November 13, 2024 02:02

korydraughn marked this pull request as ready for review November 13, 2024 02:11

korydraughn mentioned this pull request Nov 13, 2024

iRODS 5: [#7229] Redesign startup/shutdown implementation of server (main) irods/irods#7988

Merged

9 tasks

alanking reviewed Nov 13, 2024

View reviewed changes

irods_testing_environment/irods_config.py Outdated Show resolved Hide resolved

korydraughn force-pushed the irods5 branch from eefdfb6 to ba285fd Compare December 4, 2024 21:21

alanking mentioned this pull request Dec 4, 2024

Add GitHub Actions for Ruff and Black, and make them green #234

Draft

alanking reviewed Dec 4, 2024

View reviewed changes

irods_testing_environment/irods_config.py Show resolved Hide resolved

korydraughn force-pushed the irods5 branch from ba285fd to 15e1def Compare December 4, 2024 22:16

alanking approved these changes Dec 5, 2024

View reviewed changes

korydraughn force-pushed the irods5 branch from 15e1def to e1077a7 Compare December 5, 2024 14:57

korydraughn added 5 commits December 5, 2024 09:58

[irods/irods#7229,irods#235] iRODS 5: Add irodsAgent entry to rsyslog…

35b429d

… config.

[irods/irods#7229,irods/irods#7996,irods#235] iRODS 5: Replace use of…

8b08b86

… irodsctl. irodsctl was removed in iRODS 5. This commit replaces the testing environment's use of irodsctl with the python code invoked via irodsctl.

[irods/irods#7229,irods#235] iRODS 5: Update generation of input file…

54e39f7

… for setup_irods.py.

korydraughn force-pushed the irods5 branch from e1077a7 to 54e39f7 Compare December 5, 2024 14:59

alanking merged commit 56e7c81 into irods:main Dec 5, 2024

korydraughn deleted the irods5 branch December 5, 2024 15:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[irods/irods#7229] Add support for iRODS 5 #231

[irods/irods#7229] Add support for iRODS 5 #231

korydraughn commented Nov 7, 2024 •

edited

Loading

korydraughn commented Nov 13, 2024

korydraughn commented Nov 13, 2024

trel commented Nov 13, 2024

korydraughn commented Nov 13, 2024

trel commented Nov 13, 2024

korydraughn commented Nov 13, 2024

alanking left a comment

trel commented Nov 13, 2024

alanking commented Nov 13, 2024

korydraughn commented Nov 13, 2024

alanking commented Nov 14, 2024

korydraughn commented Nov 14, 2024 •

edited

Loading

alanking commented Nov 14, 2024

trel commented Nov 14, 2024

korydraughn commented Dec 4, 2024

alanking left a comment

korydraughn commented Dec 5, 2024

korydraughn commented Dec 5, 2024

korydraughn commented Dec 5, 2024

[irods/irods#7229] Add support for iRODS 5 #231

[irods/irods#7229] Add support for iRODS 5 #231

Conversation

korydraughn commented Nov 7, 2024 • edited Loading

korydraughn commented Nov 13, 2024

korydraughn commented Nov 13, 2024

trel commented Nov 13, 2024

korydraughn commented Nov 13, 2024

trel commented Nov 13, 2024

korydraughn commented Nov 13, 2024

alanking left a comment

Choose a reason for hiding this comment

trel commented Nov 13, 2024

alanking commented Nov 13, 2024

korydraughn commented Nov 13, 2024

alanking commented Nov 14, 2024

korydraughn commented Nov 14, 2024 • edited Loading

alanking commented Nov 14, 2024

trel commented Nov 14, 2024

korydraughn commented Dec 4, 2024

alanking left a comment

Choose a reason for hiding this comment

korydraughn commented Dec 5, 2024

korydraughn commented Dec 5, 2024

korydraughn commented Dec 5, 2024

korydraughn commented Nov 7, 2024 •

edited

Loading

korydraughn commented Nov 14, 2024 •

edited

Loading