diff --git a/.gitignore b/.gitignore index f7d23c6..c8c3de7 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ build/ dist/ ipf.egg-info/ +*.swp diff --git a/README.md b/README.md index b21696a..11956eb 100644 --- a/README.md +++ b/README.md @@ -4,48 +4,38 @@ ## Overview -The Information Publishing Framework (IPF) is a generic framework used by resource operators to gather and publish -dynamic resource information in [GLUE 2 serialized format](http://www.ogf.org/documents/GFD.147.pdf). IPF was used -by the TeraGrid, XSEDE, and XSEDE 2 programs, and is currently being used by the ACCESS-CI program to publish -high-performance compute cluster information. - -IPF gathers and publishes information using simple workflows. These workflows are defined using JSON (see the -etc/workflows directory) and steps in the workflows are implemented as Python classes. Each step in the -workflow can require input Data, can produce output Data, and can publish Representations of Data. A typical -workflow consists of a number of information gathering steps and a few steps that publish representations to -files or to remote services (e.g. REST, messaging). - -Workflow steps specify what Data they require and what Data they produce. This allows IPF to construct -workflows based on partial information - in the case where there are not steps that produce the same Data, an -entire workflow can be constructed from a single publish step and its required input Data. At the other -extreme, workflows can be exactly specified with specific steps identified and the outputs of steps bound to -the inputs of other steps. A typical workflow (e.g. GLUE 2) specifies what steps to include but lets IPF -automatically link outputs to inputs of these steps. - -Workflows can run to completion relatively quickly or they can continuously run. The first type of workflow -can be used to run a few commands or look at status files and publish that information. The second type of -workflow can be used to monitor log files and publish entries written to those files. Workflows are typically -run periodically as cron jobs. The program libexec/run_workflow.py is for executing workflows that complete -quickly and the program libexec/run_workflow_daemon.py is used to manage long-running workflows. The daemon +IPF is a Python program that gathers resource information, formats it in a [GLUE2 standard format (5)](#glue2), and publishes it to a RabbitMQ service. IPF is configured to run one or more “workflows” each defining the steps that IPF executes to collect, format, and publish a specific type of resource information. -## License - -This software is licensed the Apache License Version 2.0. ## Installation -Quickstart instructions are in [docs/Quickstart.md](docs/Quickstart.md). -More comprehensive instructions are in [docs/INSTALL.md](docs/INSTALL.md). +Three install methods are available and listed below in ease of use order. + +1. [Install from Github](docs/install-via-github.md) +1. [Install via Pip](docs/install-from-pip.md) +1. [Install via RPM](docs/install-from-rpm.md) (deprecated) + + +## Configure Workflows + +After installing IPF, [Configure Workflows](docs/configure-workflows.md) + +## Additional Information + +* [Best Practices](docs/best-practices.md) +* [Testing](docs/testing.md) ## Support Information This software is currently maintained by the ACCESS CONECT project. -The source is maintained in the [ACCESS-CI GitHub](https://github.com/access-ci-org/ipf). ACCESS-CI resource -providers and other members of the ACCESS-CI community are encourage to contribute bug fixes and improvements. +The source is maintained in the [ACCESS-CI GitHub](https://github.com/access-ci-org/ipf). ACCESS-CI resource providers and other members of the ACCESS-CI community are encourage to contribute bug fixes and improvements. + +Software bugs may be reported as GitHub issues. ACCESS-CI related support requests should be submitted through the ACCESS-CI ticket system. -Software bugs may be reported as GitHub issues. ACCESS-CI related support requests should be submitted through -the ACCESS-CI ticket system. +## License + +This software is licensed the Apache License Version 2.0. ## Acknowledgements diff --git a/docs/Quickstart.md b/docs/Quickstart.md deleted file mode 100644 index a114045..0000000 --- a/docs/Quickstart.md +++ /dev/null @@ -1,440 +0,0 @@ -# access-ci-org/ipf %VER%-%REL% -===================== - - -## Quickstart -=========================================== - -### What is IPF? - - -IPF is a Python program that gathers resource information, formats it in a [GLUE2 standard format (5)](#glue2), and publishes it -to a RabbitMQ service. IPF is configured to run one or more “workflows” each defining the steps that IPF executes to collect, -format, and publish a specific type of resource information. - - -## Pre-requisites --------------- - -### Preparing to Install IPF -------------------------- - - -- Before installing IPF operators should register their cluster resource in [CiDeR (4)](#CIDER). - While IPF is capable of publishing information for resources not in CiDeR, ACCESS needs resource - descriptions in CiDeR to complement the information published with IPF. - - -- Identify a single server to run IPF -- a single IPF instance can be used to publish information for multiple resources. - - -- To install IPF on a cluster that presents publicly as multiple resources please review this document: - [Publishing Software for multiple Resources from a single IPF deployment](https://docs.google.com/document/d/1UXF_pwwZdycuUiV7JToKOKMOHNs6VWjdMWFOjUyKMU4/edit?usp=sharing) - -- Decide what installation method to use: - * RPM is recommended for production installs, as it is managed by system tools, and creates an "xdinfo" user to run the workflows. - * Pip is easier for installs where root access is not available, though some additional environment variables will need to be set. - -- If you already have an older IPF create a backup of the /etc/ipf working configurations: - - $ tar -cf ipf-etc-yyyymmdd.tar /etc/ipf - - -### Software Dependencies - - -- Python 3.6 or newer - (Python 3.11 is implicitly required by the RPM as its site-packages - are within a python3.11 directory) -- The python-amqp package -- The python-setuptools package IF installed by RPM. - -*These dependencies are encoded in the RPM.* - -### How is an IPF workflow defined? - - -Each IPF workflow consists of a series of "steps" with inputs, outputs, and dependencies. The steps for each workflow are defined -in one or more JSON formatted files. Workflow JSON files can incorporate other workflow JSON files: for example, -the `_services_periodic.json` workflow contains one step, which is the `_services.json` workflow. - - -IPF workflows are typically defined by JSON files under $IPF_ETC_PATH/ipf/workflow/, particularly in $IPF_ETC_PATH/ipf/workflow/glue2. - - -### How is an IPF workflow invoked? - - -To run a workflow execute the ipf_workflow program passing it a workflow definition file argument, like this: - - - $INSTALL_DIR/ipf-VERSION/ipf/bin/ipf_workflow - - -Workflow JSON files are specified relative to $IPF_ETC_PATH, for example: -`ipf_workflow sysinfo.json` -`ipf_workflow glue2/_services.json` - - -Part of workflow configuration includes generating $IPF_ETC_PATH/ipf/init.d scripts to run a workflow periodically. -These scripts are usually copied to the system /etc/init.d directory during installation. - - -### Which workflows should I configure and run? - - -The following workflows are recommended for the listed scenarios. - -Batch System workflow (compute workflow): *clusters with batch systems* - -Software Module workflow: *clusters with command line software modules* - -Network Accessible Services workflow: *clusters with with edge services like openssh* - -Batch Scheduler Job Event workflow (activity workflow): *clusters that want to support live job event subscriptions* - - -### Batch System workflow requirements - - -- The command line programs for your batch scheduler must be executable. - - -### Software Modules workflow requirements - - -- The module or Lmod files must be readable. - - -### Network Services workflow requirements - - -- The service definition files must be readable, and in a single directory. - See Configuring.Service.Files.md for more information - - -### Batch Scheduler Job Events workflow requirements - - -- The batch scheduler log file or directory must be readable on the server where IPF is running and by the user running IPF. -- The batch scheduler must be logging at the right level for the IPF code to be able to parse the events. - See the Configuring Torque Logging section. - - - - -## Installing IPF --------------- - - -There are two recommended ways to install IPF: you can use pip install, or you can install from ACCESS RPMs. The RPMs can be found in the github releases, or -in the repositories at software.operations.access-ci.org/production. - -Installing IPF from RPMs will put it in the directories /usr/lib/python-``/site-packages/ipf, /etc/ipf, /var/ipf). - -To install to an alternate location we recommend using pip. - - -### PIP installation - - -To install using pip, you need to have the pip package installed in an appropriate version of Python (3.6+). -We recommend using venv to manage Python installations. More information on venv is available at - - -Depending on how many python versions are in place on your system, "pip" may or may not refer to the python 3 version. -"pip3" should always unambiguously refer to a python3 version of pip. - -Once you have a Python 3.6 environment (whether venv or not), to install execute: - - - $ pip3 install ipf - - -When installing via pip: unlike in an RPM install, the files get -installed relative to your Python installation (whether in a virtualenv -or system Python). Notably, ipf_configure and ipf_workflow end -up in the virtualenv's bin directory, and the location IPF expects to -find as its IPF_ETC_PATH (/etc/ipf in an RPM install) is relative to -the Python site-packages directory. - - -You can find your site-packages path for the Python you used for the pip install with: - - - $ python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' - - -When running any IPF commands by hand in a pip install, you will need to -set the environment variable IPF_ETC_PATH. Its value should be the -site-packages directory referenced above, plus "/etc/ipf". For a system -Python, this might look something like -"/usr/lib/python3.6/site-packages/etc/ipf". - -If you have run ipf_configure to set up your workflows, and chosen the -recommended base directory, your workflow definitions will have the -appropriate IPF_ETC_PATH defined in them. - -If you wish to have the workflows run as a user other than the one that -performed the pip install, you will have to do so manually. - - -### RPM Installation - - -Note(s): - The RPM will automatically create an "xdinfo" account that -will own the install and that will execute the workflows via sudo. - - -Steps: 1) Download the latest RPM from the IPF releases at GitHub: -https://github.com/access-ci-org/ipf/releases - - -2) Install ipf - - - $ yum install ./ipf-%VER%-%REL.noarch.rpm - - - - -## Updating a previous IPF installation ------------------------------------- - - -The JSON files that have been written by previous runs of -ipf_configure would, in some previous versions of IPF get -overwritten by subsequent runs of ipf_configure. This is no -longer the case--previous versions get backed up, not overwritten. They -will *not* be erased by removing OR updating the package (nor will the -service files copied to /etc/init.d be erased). - - -To perform the update to the latest RPM distribution of ipf: - -1. get the latest RPM from https://github.com/access-ci-org/ipf/releases -2. $ sudo yum update ./ipf-%VER%-%REL.noarch.rpm -3. If there are new workflows you need to configure, follow the - configuration steps as outlined in the Configuration section below. - - -## Configuring IPF ---------------- - - -To make configuration easier, an `ipf_configure` script is -provided in the bin directory (in /usr/bin if you installed RPMs, -otherwise in $INSTALL_DIR/ipf-VERSION/ipf/bin). This script will -generate workflow definition files and example init files. - - -If you intend to publish software module information via the extmodules -workflow, set the environment variable MODULEPATH -to point to the location of the module files before running -ipf_configure. If you intend to publish the service workflow -set SERVICEPATH to point to the location of the service definition files -before running ipf_configure (more on this below). -As of IPF v 1.7, ipf_configure accepts command line parameters -to tell it which workflows to configure, and with which options. - - -An invocation of ipf_configure on a resource that has installed -IPF using RPM and wants to publish software information might look like: - - -/usr/bin/ipf_configure --rpm --resource_name --workflows=extmodules --publish --amqp_certificate /etc/grid-security/cert_for_ipf.pem --amqp_certificate_key /etc/grid-security/key_for_ipf.pem --modulepath /path/to/modules --lmod_cache_file /path/to/lmodcache.lua - - -These options mean: - - ---rpm IPF was installed using RPM; this lets us know where files should be on disk - - ---resource_name The name of your resource. To find your resource name, - go to "https://operations.access-ci.org/resources/access-allocated" - to find your resource, and use the "Global Resource ID" value. - ---workflows Comma delimited list of workflows to configure. Values can include: - compute, activity, extmodules, services ---publish Necessary if you wish to configure your workflow to publish to ACCESS's - AMQP service for inclusion in Information Services - - ---amqp_certificate The path to the certificate to use to authenticate with ACCESS’s AMQP - - ---amqp_key The path to the key for your certificate - - ---modulepath The MODULEPATH where the modulefiles for software publishing are - found. If not specified $MODULEPATH from the user environment - will be used. ---lmod_cache_file The location of an lmod cache file that contains exactly the set of modules you wish to publish. If you do not specify an lmod_cache_file, IPF will fall back to its traditional behavior of walking the MODULEPATH. - - -Other common options: - - ---amqp_username If not using certificates to authenticate, use these to specify ---amqp_password username and password - - ---pip IPF was installed using “pip install” - - -For a full list of command line options, please try - - -$ ipf_configure ----help - - -- `ipf_configure` should be run as the user that will run the - information gathering workflows - -- You must always specify --resource_name, and you should use the - "Global Resource ID" from - https://operations.access-ci.org/resources/access-allocated - - -- The preferred way to authenticate is via an X.509 host certificate - and key. You can place these files wherever you like, but the - default locations are /etc/grid-security/xdinfo-hostcert.pem and - /etc/grid-security/xdinfo-hostkey.pem. These files must be readable - by the user that runs the information gathering workflows. - - -- Submit an ACCESS ticket to authorize your server to publish ACCESS's - RabbitMQ services. If you will authenticate via X.509, include the - output of 'openssl x509 -in path/to/cert.pem -nameopt RFC2253 - -subject -noout' in your ticket. If you will authenticate via - username and password, state that and someone will contact you. - - -Note: The xdinfo user as created by the ipf rpm installation has -/bin/nologin set as its shell by default. This is because for most -purposes, the xdinfo user doesn't need an interactive shell. However, -for some of the initial setup, it is easiest to use the xdinfo user with -an interactive shell (so that certain environment variables like -MODULEPATH can be discovered.) Thus, it is recommended that the -configuration steps are run after something like the following: - - - $ sudo -u xdinfo -s /bin/bash --rcfile /etc/bashrc -i - $ echo $MODULEPATH - $ echo $SERVICEPATH - - -Execute: - - - $ ipf_configure \ - - - -If you encounter any errors or the script does not cover your situation, -Please submit an ACCESS ticket. - - -When the script exits, the etc/ipf/workflow/glue2/ directory will -contain a set of files RESOURCE_NAME_*.json that describe the -information gathering workflows you have configured and etc/ipf/init.d -will contain ipf-RESOURCE_NAME_* files which are the init scripts you -have configured. - - -As root, copy the etc/ipf/init.d/ipf-RESOURCE_NAME-* files into -/etc/init.d. Your information gathering workflows can then be enabled, -started, and stopped in the usual ways. You may need to perform a -'chkconfig --add' or equivalent for each service. - - -## Software Module Publishing Best Practices ------------------------------------------ - - -The IPF Software Module workflow publishes information about locally -installed software available through modules or Lmod. IPF tries to make -intelligent inferences from the system installed modules files when it -publishes software information. There are some easy ways, however, to -add information to your module files that will enhance/override the -information otherwise published. - - -The ExtModules workflow, as of IPF 1.8 has two methods for discovering the -modules you wish to publish. The recommended method, for any site using Lmod, -is to point the workflow at an lmod cache file that represents exactly what -you wish to publish. It will then publish every module in the spiderT table -from the cache file, except modules listed in the hiddenT table. - -If you are not using Lmod, or do not wish to use lmod cache files, the -workflow will fall back to the traditional method of walking the MODULEPATH. -The workflow then traverses your MODULEPATH and infers fields such -as Name and Version from the directory structure/naming conventions of -the module file layout. The new IPF default behavior is to treat each -directory in your MODULEPATH as a top level directory, under which all of -the subdirectory structure is semantically significant (and part of the -inferred name of the module). The old default behavior, if desired, can be -enabled with by configuring the extmodules workflow with the --modules_recurse -argument. - -Depending on the exact workflow steps, fields such as Description may be -blank, or inferred from the stdout/stderr text of the module. However, the -following fields can always be explicitly added to a module file: - - Name: - Version: - Description: - URL: - Category: - Keywords: - SupportStatus: - SupportContact: - - -Each field is a key: value pair. The IPF workflows are searching the -whole text of each module file for these fields. They may be placed in a -module-whatis line, or in a comment, and IPF will still read them. - -More details about the contents of these fields can be found in the more -comprehensive INSTALL.md documentation. - - -## Testing -------- - - -1) To test the extended attribute modules workflow, execute: - - - # service ipf-RESOURCE_NAME-glue2-extmodules start - - -This init script starts a workflow that periodically gathers (every hour -by default) and publishes module information containing extended -attributes. - - -The log file is in /var/ipf/RESOURCE_NAME_modules.log (or -$INSTALL_DIR/ipf/var/ipf/RESOURCE_NAME_extmodules.log) and should -contain messages resembling: - - - 2013-05-30 15:27:05,309 - ipf.engine - INFO - starting workflow extmodules - 2013-05-30 15:27:05,475 - ipf.publish.AmqpStep - INFO - step-3 - publishing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org - 2013-05-30 15:27:05,566 - ipf.publish.FileStep - INFO - step-4 - writing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org - 2013-05-30 15:27:06,336 - ipf.engine - INFO - workflow succeeded - - -If any of the steps fail, that will be reported and an error message and -stack trace should appear. Typical failures are caused by the -environment not having specific variables or commands available. - - -This workflow describes your modules as a JSON document containing GLUE -v2.0 Application Environment and Application Handle objects. This -document is published to the ACCESS RabbitMQ service in step-3 and is -written to a local file in step-4. You can examine this local file in -/var/ipf/RESOURCE_NAME_apps.json. If you see any errors in gathering -module information, please submit an ACCESS ticket. - diff --git a/docs/best-practices.md b/docs/best-practices.md new file mode 100644 index 0000000..b4a2d47 --- /dev/null +++ b/docs/best-practices.md @@ -0,0 +1,53 @@ +# access-ci-org/ipf %VER%-%REL% +===================== + + +# Software Module Publishing Best Practices +----------------------------------------- + + +The IPF Software Module workflow publishes information about locally +installed software available through modules or Lmod. IPF tries to make +intelligent inferences from the system installed modules files when it +publishes software information. There are some easy ways, however, to +add information to your module files that will enhance/override the +information otherwise published. + + +The ExtModules workflow, as of IPF 1.8 has two methods for discovering the +modules you wish to publish. The recommended method, for any site using Lmod, +is to point the workflow at an lmod cache file that represents exactly what +you wish to publish. It will then publish every module in the spiderT table +from the cache file, except modules listed in the hiddenT table. + +If you are not using Lmod, or do not wish to use lmod cache files, the +workflow will fall back to the traditional method of walking the MODULEPATH. +The workflow then traverses your MODULEPATH and infers fields such +as Name and Version from the directory structure/naming conventions of +the module file layout. The new IPF default behavior is to treat each +directory in your MODULEPATH as a top level directory, under which all of +the subdirectory structure is semantically significant (and part of the +inferred name of the module). The old default behavior, if desired, can be +enabled with by configuring the extmodules workflow with the --modules_recurse +argument. + +Depending on the exact workflow steps, fields such as Description may be +blank, or inferred from the stdout/stderr text of the module. However, the +following fields can always be explicitly added to a module file: + + Name: + Version: + Description: + URL: + Category: + Keywords: + SupportStatus: + SupportContact: + + +Each field is a key: value pair. The IPF workflows are searching the +whole text of each module file for these fields. They may be placed in a +module-whatis line, or in a comment, and IPF will still read them. + +More details about the contents of these fields can be found in the more +comprehensive INSTALL.md documentation. diff --git a/docs/configure-workflows.md b/docs/configure-workflows.md new file mode 100644 index 0000000..0f653c9 --- /dev/null +++ b/docs/configure-workflows.md @@ -0,0 +1,131 @@ +# access-ci-org/ipf %VER%-%REL% +===================== + +## Configuring IPF +--------------- + + +To make configuration easier, an `ipf_configure` script is +provided in the bin directory (in /usr/bin if you installed RPMs, +otherwise in $INSTALL_DIR/ipf-VERSION/ipf/bin). This script will +generate workflow definition files and example init files. + + +If you intend to publish software module information via the extmodules +workflow, set the environment variable MODULEPATH +to point to the location of the module files before running +ipf_configure. If you intend to publish the service workflow +set SERVICEPATH to point to the location of the service definition files +before running ipf_configure (more on this below). +As of IPF v 1.7, ipf_configure accepts command line parameters +to tell it which workflows to configure, and with which options. + + +An invocation of ipf_configure on a resource that has installed +IPF using RPM and wants to publish software information might look like: + + +/usr/bin/ipf_configure --rpm --resource_name --workflows=extmodules --publish --amqp_certificate /etc/grid-security/cert_for_ipf.pem --amqp_certificate_key /etc/grid-security/key_for_ipf.pem --modulepath /path/to/modules --lmod_cache_file /path/to/lmodcache.lua + + +These options mean: + +- `--rpm` + - IPF was installed using RPM; this lets us know where files should be on disk + +- `--resource_name` + - The name of your resource. To find your resource name, go to "https://operations.access-ci.org/resources/access-allocated" to find your resource, and use the "Global Resource ID" value. + +- `--workflows` + - Comma delimited list of workflows to configure. Values can include: + - compute + - activity + - extmodules + - services + +- `--publish` + - Necessary if you wish to configure your workflow to publish to ACCESS's AMQP service for inclusion in Information Services + +- `--amqp_certificate` + -The path to the certificate to use to authenticate with ACCESS’s AMQP + +- `--amqp_key` + - The path to the key for your certificate + +- `--modulepath` + - The MODULEPATH where the modulefiles for software publishing are found. If not specified $MODULEPATH from the user environment will be used. + +- `--lmod_cache_file` + - The location of an lmod cache file that contains exactly the set of modules you wish to publish. If you do not specify an lmod_cache_file, IPF will fall back to its traditional behavior of walking the MODULEPATH. + + +Other common options: + +- `--amqp_username` +- `--amqp_password` + - If not using certificates to authenticate, use these to specify username and password + +- `--pip` + - IPF was installed using “pip install” + +For a full list of command line options, please try +``` +ipf_configure --help +``` + +Execute: +``` +ipf_configure \ +``` + +If you encounter any errors or the script does not cover your situation, +Please submit an ACCESS ticket. + +When the script exits, the etc/ipf/workflow/glue2/ directory will +contain a set of files named RESOURCE_NAME.json that describe the +information gathering workflows you have configured and etc/ipf/init.d +will contain ipf-RESOURCE_NAME files which are the init scripts you +have configured. + +As root, copy the init scripts into `/etc/init.d`. +Your information gathering workflows can then be enabled, +started, and stopped in the usual ways. You may need to perform a +`chkconfig --add` or equivalent for each service. + + +### Notes + +- `ipf_configure` should be run as the user that will run the + information gathering workflows + +- You must always specify --resource_name, and you should use the + "Global Resource ID" from + https://operations.access-ci.org/resources/access-allocated + + +- The preferred way to authenticate is via an X.509 host certificate + and key. You can place these files wherever you like, but the + default locations are /etc/grid-security/xdinfo-hostcert.pem and + /etc/grid-security/xdinfo-hostkey.pem. These files must be readable + by the user that runs the information gathering workflows. + + +- Submit an ACCESS ticket to authorize your server to publish ACCESS's + RabbitMQ services. If you will authenticate via X.509, include the + output of 'openssl x509 -in path/to/cert.pem -nameopt RFC2253 + -subject -noout' in your ticket. If you will authenticate via + username and password, state that and someone will contact you. + + +- (RPM install only) The `xdinfo` user as created by the ipf RPM installation has +/bin/nologin set as its shell by default. This is because for most +purposes, the `xdinfo` user doesn't need an interactive shell. However, +for some of the initial setup, it is easiest to use the `xdinfo` user with +an interactive shell (so that certain environment variables like +MODULEPATH can be discovered.) Thus, it is recommended that the +configuration steps are run after something like the following: +``` +sudo -u xdinfo -s /bin/bash --rcfile /etc/bashrc -i +echo $MODULEPATH +echo $SERVICEPATH +``` diff --git a/docs/history.md b/docs/history.md new file mode 100644 index 0000000..e9750e1 --- /dev/null +++ b/docs/history.md @@ -0,0 +1,38 @@ + +# access-ci-org/ipf %VER%-%REL% + +## History of IPF + +The Information Publishing Framework (IPF) is a generic framework used by resource operators to gather and publish +dynamic resource information in [GLUE 2 serialized format](http://www.ogf.org/documents/GFD.147.pdf). IPF was used +by the TeraGrid, XSEDE, and XSEDE 2 programs, and is currently being used by the ACCESS-CI program to publish +high-performance compute cluster information. + +IPF gathers and publishes information using simple workflows. These workflows are defined using JSON (see the +etc/workflows directory) and steps in the workflows are implemented as Python classes. Each step in the +workflow can require input Data, can produce output Data, and can publish Representations of Data. A typical +workflow consists of a number of information gathering steps and a few steps that publish representations to +files or to remote services (e.g. REST, messaging). + +Workflow steps specify what Data they require and what Data they produce. This allows IPF to construct +workflows based on partial information - in the case where there are not steps that produce the same Data, an +entire workflow can be constructed from a single publish step and its required input Data. At the other +extreme, workflows can be exactly specified with specific steps identified and the outputs of steps bound to +the inputs of other steps. A typical workflow (e.g. GLUE 2) specifies what steps to include but lets IPF +automatically link outputs to inputs of these steps. + +Workflows can run to completion relatively quickly or they can continuously run. The first type of workflow +can be used to run a few commands or look at status files and publish that information. The second type of +workflow can be used to monitor log files and publish entries written to those files. Workflows are typically +run periodically as cron jobs. The program libexec/run_workflow.py is for executing workflows that complete +quickly and the program libexec/run_workflow_daemon.py is used to manage long-running workflows. The daemon + +## License + +This software is licensed the Apache License Version 2.0. + + +## Acknowledgements + +This work was supported by the TeraGrid, XSEDE, FutureGrid, XSEDE 2, and ACCESS CONECT projects under +National Science Foundation grants 0503697, 1053575, 0910812, 1548562, and 2138307. diff --git a/docs/install-from-github-FAQ.md b/docs/install-from-github-FAQ.md new file mode 100644 index 0000000..0cb9404 --- /dev/null +++ b/docs/install-from-github-FAQ.md @@ -0,0 +1,57 @@ +# Frequently Asked Questions + +## What does `no pid file` mean when starting workflows? +Not sure right now, but just check publishing status at +https://operations-api.access-ci.org/wh2/state/v1/status/ +to see if the runs were published. + +Also, check local process status with +```bash +bash ~/ipf/bin/wfm status +``` + + +## How do I upgrade to the latest version? +1. Re-run the installer + * ```bash + bash ~/install_ipf.sh + ``` + + +## I messed up the install. Can I start over from scratch? +Yes! +1. (optional) Save customized configs + * (see the "Backup workflow configs" later in this document). +1. Stop any running workflows + * ```bash + bash ~/ipf/bin/wfm stop + ``` +1. Make backups of any config files + * ```bash + bash ~/ipf/bin/save_configs.sh + ``` +1. Remove the install directory and installer + * ```bash + rm -rf ~/ipf install_ipf.sh + ``` +1. Follow through the QUICKSTART guide again starting from the top + + + +## Can I configure multiple workflows of the same type? +Yes! The `configure_extmodules` script will look for config files matching the +naming convention `configure_extmodules*.conf`. You can create multiple config +files and a workflow definition will be created for each one. Just make sure +that `RESOURCE_NAME` is unique in each config file. + + +## How can I backup my workflow configs? +1. Backup workflow configs + * ```bash + bash ~/ipf/bin/save_configs.sh + ``` +This will do 2 things: +* make backup copies in `~/.config/ipf/` +* create symlinks to the backup copies in the ipf install dir. +On a re-install, the IPF installer will look for any backed up +config files and re-make the symlnks. diff --git a/docs/install-from-github.md b/docs/install-from-github.md new file mode 100644 index 0000000..7321ea6 --- /dev/null +++ b/docs/install-from-github.md @@ -0,0 +1,77 @@ +# Install from git + +1. Get installer + * ```bash + curl -o ~/install_ipf.sh https://raw.githubusercontent.com/access-ci-org/ipf/refs/heads/CTT-304/aloftus/backwards_compatable/go.sh + ``` + +1. Run installer + * ```bash + bash ~/install_ipf.sh + ``` + * Note: IPF will be installed into the current directory. All commands in + this guide assume the current directory is `~/`. + +1. Do first time setup + * ```bash + bash ~/ipf/bin/prep.sh + ``` + +# Setup the extmodules workflow +## Configure the extmodules workflow +1. Set variables for your site + * ```bash + cp ~/ipf/etc/configure_extmodules.conf.sample ~/ipf/etc/configure_extmodules.conf + vim ~/ipf/etc/configure_extmodules.conf + cp ~/ipf/etc/amqp.conf.sample ~/ipf/etc/amqp.conf + vim ~/ipf/etc/amqp.conf + ``` + * Note: for initial testing, leave the PUBLISH variable empty. +1. Run the configure script + * ```bash + bash ~/ipf/bin/configure_extmodules + ``` + +## Test the extmodules workflow +1. Start the workflow + * ```bash + bash ~/ipf/bin/wfm start + ``` +1. Check the output + * ```bash + bash ~/ipf/bin/wfm list + ``` + * Check the `OUTPUT` file that was listed above +1. Stop the workflow + * ```bash + bash ~/ipf/bin/wfm stop + ``` + +## Test the publishing setup +1. Enable publishing + * ```bash + sed -i -e '/PUBLISH=/cPUBLISH=1' ~/ipf/etc/configure_extmodules*.conf + ``` +1. Re-run the configure script + * ```bash + bash ~/ipf/bin/configure_extmodules + ``` +1. Start the workflow + * ```bash + bash ~/ipf/bin/wfm start + ``` +1. Check the published data + * Look for the resource name at: https://operations-api.access-ci.org/wh2/state/v1/status/ + * The date in the `Processed at` column should be recent. + +# Setup recurring runs for production +1. Create a scheduled task to restart the workflows after a system restart. + * Example crontab: + ```bash + @restart $HOME/ipf/bin/wfm start + ``` + + +# NOTES: +- This install method currently supports only the `extmodules` workflow. + Other workflows will be added in the future. diff --git a/docs/install-from-pip.md b/docs/install-from-pip.md new file mode 100644 index 0000000..fb64e21 --- /dev/null +++ b/docs/install-from-pip.md @@ -0,0 +1,79 @@ +# access-ci-org/ipf %VER%-%REL% +===================== + + +## Pre-requisites +-------------- + +### Preparing to Install IPF +------------------------- + + +- Before installing IPF operators should register their cluster resource in [CiDeR (4)](#CIDER). + While IPF is capable of publishing information for resources not in CiDeR, ACCESS needs resource + descriptions in CiDeR to complement the information published with IPF. + +- Identify a single server to run IPF -- a single IPF instance can be used to publish information for multiple resources. + +- To install IPF on a cluster that presents publicly as multiple resources please review this document: + [Publishing Software for multiple Resources from a single IPF deployment](https://docs.google.com/document/d/1UXF_pwwZdycuUiV7JToKOKMOHNs6VWjdMWFOjUyKMU4/edit?usp=sharing) + +- If you already have an older IPF create a backup of the /etc/ipf working configurations: + ``` + tar -cf ipf-etc-yyyymmdd.tar /etc/ipf + ``` + + +### Software Dependencies + +- Python 3.6 or newer +- The python-amqp package + + +## Installing IPF +-------------- + +### PIP installation + + +To install using pip, you need to have the pip package installed in an appropriate version of Python (3.6+). +We recommend using venv to manage Python installations. More information on venv is available at + + +Depending on how many python versions are in place on your system, "pip" may or may not refer to the python 3 version. +"pip3" should always unambiguously refer to a python3 version of pip. + +Once you have a Python 3.6 environment (whether venv or not), to install execute: +``` +pip3 install ipf +``` + + +When installing via pip, the files get +installed relative to your Python installation (whether in a virtualenv +or system Python). Notably, `ipf_configure` and `ipf_workflow` end +up in the "bin" of the virtualenv, and the location IPF expects to +find as its IPF_ETC_PATH (/etc/ipf in an RPM install) is relative to +the Python site-packages directory. + +You can find your site-packages path for the Python you used for the pip install with: +``` +python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' +``` + + +When running any IPF commands by hand in a pip install, you will need to +set the environment variable IPF_ETC_PATH. Its value should be the +site-packages directory referenced above, plus "/etc/ipf". For a system +Python, this might look something like +"/usr/lib/python3.6/site-packages/etc/ipf". + +If you have run `ipf_configure` to set up your workflows, and chosen the +recommended base directory, your workflow definitions will have the +appropriate IPF_ETC_PATH defined in them. + +If you wish to have the workflows run as a user other than the one that +performed the pip install, you will have to do so manually. + +## Next Steps +* [Configure Workflows](configure-workflows.md) diff --git a/docs/INSTALL.md b/docs/install-from-rpm.md similarity index 100% rename from docs/INSTALL.md rename to docs/install-from-rpm.md diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..7fe3320 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,41 @@ +# access-ci-org/ipf %VER%-%REL% +===================== + + +# Testing + + +1) To test the extended attribute modules workflow, execute: + + + # service ipf-RESOURCE_NAME-glue2-extmodules start + + +This init script starts a workflow that periodically gathers (every hour +by default) and publishes module information containing extended +attributes. + + +The log file is in /var/ipf/RESOURCE_NAME_modules.log (or +$INSTALL_DIR/ipf/var/ipf/RESOURCE_NAME_extmodules.log) and should +contain messages resembling: + + + 2013-05-30 15:27:05,309 - ipf.engine - INFO - starting workflow extmodules + 2013-05-30 15:27:05,475 - ipf.publish.AmqpStep - INFO - step-3 - publishing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org + 2013-05-30 15:27:05,566 - ipf.publish.FileStep - INFO - step-4 - writing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org + 2013-05-30 15:27:06,336 - ipf.engine - INFO - workflow succeeded + + +If any of the steps fail, that will be reported and an error message and +stack trace should appear. Typical failures are caused by the +environment not having specific variables or commands available. + + +This workflow describes your modules as a JSON document containing GLUE +v2.0 Application Environment and Application Handle objects. This +document is published to the ACCESS RabbitMQ service in step-3 and is +written to a local file in step-4. You can examine this local file in +/var/ipf/RESOURCE_NAME_apps.json. If you see any errors in gathering +module information, please submit an ACCESS ticket. + diff --git a/docs/COMMUNITY_SOFTWARE_PROVIDER.md b/docs/xsede/COMMUNITY_SOFTWARE_PROVIDER.md similarity index 100% rename from docs/COMMUNITY_SOFTWARE_PROVIDER.md rename to docs/xsede/COMMUNITY_SOFTWARE_PROVIDER.md diff --git a/docs/COMMUNITY_SOFTWARE_SP_SETUP.md b/docs/xsede/COMMUNITY_SOFTWARE_SP_SETUP.md similarity index 100% rename from docs/COMMUNITY_SOFTWARE_SP_SETUP.md rename to docs/xsede/COMMUNITY_SOFTWARE_SP_SETUP.md diff --git a/docs/Configuring.Service.Files.OLD.md b/docs/xsede/Configuring.Service.Files.OLD.md similarity index 100% rename from docs/Configuring.Service.Files.OLD.md rename to docs/xsede/Configuring.Service.Files.OLD.md diff --git a/docs/Configuring.Service.Files.md b/docs/xsede/Configuring.Service.Files.md similarity index 100% rename from docs/Configuring.Service.Files.md rename to docs/xsede/Configuring.Service.Files.md diff --git a/docs/GENERIC_PUBLISHER.md b/docs/xsede/GENERIC_PUBLISHER.md similarity index 100% rename from docs/GENERIC_PUBLISHER.md rename to docs/xsede/GENERIC_PUBLISHER.md diff --git a/docs/INSTALL.OLD b/docs/xsede/INSTALL.OLD similarity index 100% rename from docs/INSTALL.OLD rename to docs/xsede/INSTALL.OLD diff --git a/go.sh b/go.sh new file mode 100755 index 0000000..df53daa --- /dev/null +++ b/go.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +export QS_GIT_REPO=https://github.com/access-ci-org/ipf.git +export QS_GIT_BRANCH=CTT-304/aloftus/backwards_compatable +curl https://raw.githubusercontent.com/andylytical/quickstart/main/quickstart.sh | bash diff --git a/ipf/assert_py_ver.py b/ipf/assert_py_ver.py new file mode 100644 index 0000000..1ee4041 --- /dev/null +++ b/ipf/assert_py_ver.py @@ -0,0 +1,11 @@ +# Require python 3 + +import sys + +min_ver = ( 3, 6 ) + +exitcode = 0 +py_ver = ( sys.version_info[0:2] ) +if py_ver < min_ver: + exitcode=1 +sys.exit( exitcode ) diff --git a/ipf/bin/configure_extmodules b/ipf/bin/configure_extmodules new file mode 100755 index 0000000..ca3bcc2 --- /dev/null +++ b/ipf/bin/configure_extmodules @@ -0,0 +1,37 @@ +#!/bin/bash + +INSTALL_DIR=___INSTALL_DIR___ +CLEANSE="$INSTALL_DIR"/etc/cleanse.conf +CONFIGS=( $( ls "$INSTALL_DIR"/etc/configure_extmodules*.conf ) ) +PYTHON="$INSTALL_DIR"/.venv/bin/python +PYTHONPATH=$(readlink -e "$INSTALL_DIR"/..) + +export PYTHONPATH + +# get amqp credentials from separate file if it exists +amqp_fn="$INSTALL_DIR"/etc/amqp.conf +[[ -r "$amqp_fn" ]] && source "$amqp_fn" + +# Loop through all config files found +for cfg in "${CONFIGS[@]}"; do + source "$CLEANSE" #unset var settings from any previous run + source "$cfg" #read in new var settings for this specific run + + # create workflow files + $PYTHON "$INSTALL_DIR"/configure/configure_workflows.py \ + --base_dir "$INSTALL_DIR" \ + ${AMQP_CERTIFICATE:+--amqp_certificate $AMQP_CERTIFICATE} \ + ${AMQP_CERTIFICATE_KEY:+--amqp_certificate_key $AMQP_CERTIFICATE_KEY} \ + ${AMQP_PASSWORD:+--amqp_password $AMQP_PASSWORD} \ + ${AMQP_USERNAME:+--amqp_username $AMQP_USERNAME} \ + ${LMOD_CACHE_FILE:+--lmod_cache_file $LMOD_CACHE_FILE} \ + ${MODULEPATH:+--modulepath $MODULEPATH} \ + ${MODULES:+--modules $MODULES} \ + ${MODULES_EXCLUDE:+--modules_exclude $MODULES_EXCLUDE} \ + ${MODULES_INTERVAL:+--modules_interval $MODULES_INTERVAL} \ + ${PUBLISH:+--publish} \ + ${RESOURCE_NAME:+--resource_name $RESOURCE_NAME} \ + ${SUPPORT_CONTACT:+--support_contact $SUPPORT_CONTACT} \ + ${WORKFLOWS:+--workflows $WORKFLOWS} + +done #for cfg in "${CONFIGS[@]}"; do diff --git a/ipf/bin/prep.sh b/ipf/bin/prep.sh new file mode 100644 index 0000000..3811fc1 --- /dev/null +++ b/ipf/bin/prep.sh @@ -0,0 +1,74 @@ +#!/bin/bash + +INSTALL_DIR=___INSTALL_DIR___ +. ${INSTALL_DIR}/lib/utils.sh + +BIN="$INSTALL_DIR"/bin +ETC="$INSTALL_DIR"/etc +VENV="$INSTALL_DIR"/.venv +V_PYTHON="$VENV"/bin/python +CONF="$HOME"/.config/ipf + + +assert_python_minimum_version() { + [[ $DEBUG -eq $YES ]] && set -x + SYSTEM_PYTHON=$(which python3) 2>/dev/null + [[ -z "$SYSTEM_PYTHON" ]] && die "Unable to find Python on this system." + "$SYSTEM_PYTHON" "$INSTALL_DIR"/assert_py_ver.py || die "Python version is too old." + success "Python version check passed" +} + + +mk_venv() { + [[ $DEBUG -eq $YES ]] && set -x + [[ -d "$VENV" ]] || { + "$SYSTEM_PYTHON" -m venv "$VENV" + } + success "Python venv created at '$VENV'" +} + + +install_dependencies() { + [[ $DEBUG -eq $YES ]] && set -x + "$V_PYTHON" -m pip install --upgrade pip || die "Pip upgrade had a problem" + "$V_PYTHON" -m pip install -r "$ETC"/requirements.txt || die "Problem installing dependencies" + success "Dependencies installed" +} + + +restore_config_links() { + [[ $DEBUG -eq $YES ]] && set -x + /bin/bash "$BIN"/save_configs.sh +} + + +set_version() { + # Create a setuptools style METADATA file with the current version string + [[ $DEBUG -eq $YES ]] && set -x + local _version=$( cat "$INSTALL_DIR"/version ) + local _site_pkgs=$( "$V_PYTHON" -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' ) + local _fn_metadata="$_site_pkgs"/ipf-"${_version}".dist-info/METADATA + local _dn_metadata=$( dirname "$_fn_metadata" ) + mkdir -p "$_dn_metadata" || die "Failed to make IPF metadata directory" + >"$_fn_metadata" cat <&2 +} + + +die() { + err "$*" + echo "from (${BASH_SOURCE[1]} [${BASH_LINENO[0]}] ${FUNCNAME[1]})" + kill 0 + exit 99 +} + + +ask_user() { + # INPUT + # $1 prompt + # OUTPUT + # user response as text string + local _msg="$1" + [[ -z "$_msg" ]] && die "missing prompt in ask_user()" + read -r -p "$_msg" + echo "$REPLY" +} + + +ask_yes_no() { + local rv=$NO + local msg="Is this ok?" + [[ -n "$1" ]] && msg="$1" + echo "$msg" + select yn in "Yes" "No"; do + case $yn in + Yes) rv=$YES;; + No ) rv=$NO;; + esac + break + done + return $rv +} + + +# Function to print a green checkmark +print_green_checkmark() { + echo -e "${GREEN}✓${NC}" +} + +# Function to print a red "x" +print_red_x() { + echo -e "${RED}✗${NC}" +} diff --git a/setup.excludes b/setup.excludes new file mode 100644 index 0000000..420b744 --- /dev/null +++ b/setup.excludes @@ -0,0 +1,4 @@ +- ipf/bin/ipf_configure +- ipf/bin/ipf_workflow +- ipf/etc/ipf/init.d/ipf-WORKFLOW +- ipf/etc/ipf/init.d/ipf-WORKFLOW.wfm diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..962051e --- /dev/null +++ b/setup.sh @@ -0,0 +1,86 @@ +#!/bin/bash + +# DO NOT RUN THIS BY HAND +# This file is for use by https://github.com/andylytical/quickstart + +INSTALL_DIR="$(pwd)" #this is the directory from which the user invoked quickstart +BASE=$(readlink -e $( dirname $0 ) ) +IPF_SRC="$BASE/ipf" +TS=$(date +%s) +YES=0 +NO=1 +DEBUG=$YES +VERBOSE=$YES +# Files that are renamed on install, map of SRC -> TGT +declare -A SPECIAL_MAP=( + [ipf/etc/ipf/init.d/ipf-WORKFLOW.wfm]=ipf/etc/ipf/init.d/ipf-WORKFLOW +) + + +die() { + echo "ERROR $*" >&2 + kill -s TERM $BASHPID + exit 99 +} + + +log() { + [[ $VERBOSE -eq $YES ]] || return + echo "INFO $*" >&2 +} + + +debug() { + [[ $DEBUG -eq $YES ]] || return + echo "DEBUG (${BASH_SOURCE[1]} [${BASH_LINENO[0]}] ${FUNCNAME[1]}) $*" +} + + +update_files() { + [[ $DEBUG -eq $YES ]] && set -x + # Update any files that need INSTALL_DIR + local _pattern='___INSTALL_DIR___' + local _replacement="$INSTALL_DIR/ipf" + grep -r --files-with-matches -F "$_pattern" "$BASE" \ + | while read; do + sed -i -e "s?$_pattern?$_replacement?" "$REPLY" + done +} + + +install_common() { + [[ $DEBUG -eq $YES ]] && set -x + rsync -v --recursive \ + -b --suffix=."$TS" \ + --checksum \ + --exclude-from="$BASE"/setup.excludes \ + "$IPF_SRC" \ + "$INSTALL_DIR" +} + + +install_special() { + [[ $DEBUG -eq $YES ]] && set -x + for src in "${!SPECIAL_MAP[@]}"; do + tgt="${SPECIAL_MAP[$src]}" + cp "${BASE}/${src}" "${INSTALL_DIR}/${tgt}" + done +} + + +install_version() { + [[ $DEBUG -eq $YES ]] && set -x + local _version=$( awk -F '"' '/version=/ {print $2; exit;}' "$BASE"/setup.py ) + log "VERSION = $_version" + echo "$_version" > "$INSTALL_DIR"/ipf/version +} + +[[ $DEBUG -eq $YES ]] && set -x + +update_files + +install_common + +install_special + +install_version