User Guide

Purpose Package Moniker Local Packages may be in .zip format, or unpacked into a folder tree. Tabulating an online item bank using GitLab is also supported.

.zip package: A .zip package is a content package collected into a .zip
file. The moniker is the filename (or complete path) of the .zip file.
Wildcards (* and ?) are acceptable in which case all matching packages
will be tabulated. The .zip package must have an 'imsmanifext.xml' file
in the root.

Folder Package: A content package may be unpacked into a file system
folder tree and tabulated that way. The moniker is the name (or complete
path) of the root folder of the package. Wildcards (* and ?) are
acceptable in which case all matching packages will be tabulated. The
package must have an 'imsmanifest.xml' file in the root.

Item Bank: An item bank moniker consists of three parts of which two are
optional.
-bank <url>           (Optional) The URL of the GitLab item bank from which
                      the items will be drawn. If not specified, defaults
                      to ""https://itembank.smarterbalanced.org"".
-ns <namespace>       (Optional) The GitLab namespace to which the items
                      belong. This should be a username or a group name.
                      If not specified, defaults to ""itemreviewapp"".
-at <token>           (Required) A GitLab access token valid on the item
                      bank. See below on how to generate the token.

You may specify multiple packages of any type and in any mix of types.
Each package may have an an associated Item IDs file (-ids argument) and
each package may specify an output file prefix (-o argument). When
included, the IDs and output arguments must follow the moniker of the
package.

Access Token To generate an item bank access token, do the following: 1. Log into the GitLab item bank. 2. Access your user profile (by clicking on your account icon in the upper- right). 3. Edit your profile (by clicking on the pencil icon in the upper-right. 4. Select ""access tokens"" from the menu. 5. Give the token a name and expiration date. We recommend expiration no no longer than 3 months. Select ""API"" for the scope. Then click ""Create personal access token.""

Aggregate Tabulation When multiple packages are specified, by default they are tabulated individually - each with its own set of reports. The aggregate option (-a) tabulates all of the packages and aggregates the results into one set of reports.

Item ID File: The optional Item ID file, specified by the '-ids' argument, is a list of IDs for items that should be included in the tabulation. It may be a flat list or in CSV format.

Flat List Format:
In this format there is one item ID per line. IDs may be the bare number
(e.g. ""12345"") or they may be the full item name including
the ""Item-"" prefix(e.g. ""item-200-12345""). If the ID is a bare number
than the bank key specified by the ""-bk"" parameter or default of ""200""
will be used.

CSV Format:
A CSV file should comply with RFC 4180. The first line should be a list of
field names. One column MUST be named ""ItemId"" and it will be the source
of the item ids. Another column MAY be named ""BankKey"". If so, it will be
the source of bank keys. If not included, the default bank key (either 200
or the value specified in the ""-bk"") will be used. As with flat list
format, the item ID may be a bare integer or a full name including prefix.

Specifying a Stimulus:
Normally, stimuli are automatically included when an item is specified
that depends on that stimulus. However, a stimulus may be explicitly
included by using the full name (e.g. ""stim-200-6789"". This applies to
both Flat List and CSV formats.

Report Output The tabulator generates six reports: _SummaryReport.txt Summary information including counts of items, errors, etc. _ItemReport.csv Items including key metadata. _StimulusReport.csv Stimuli including key metadata. _ErrorReport.csv Item and stimulus validation errors. _WordlistReport.csv All wordLists including glossary term counts. _GlossaryReport.csv A comprehensive list of every glossary term in the package.

For .zip and Folder packages, the default prefix is the name of the .zip
file or the folder. For example ""MyPackage_ItemReport.csv"". This may be
overridden using the -o argument.

For an item bank, the prefix is always specified by the -o argument.

For aggregate reports, the default prefix is the folder of the first
package in the list plus the word ""Aggregate"". For example,
""Aggregate_ItemReport.csv"". This may be overridden using the -o
argument.

Validation Options: Validation options disable or enable the reporting of certain errors. Only a subset of validations can be controlled this way.

-v-pmd  Passage Manifest Dependency: An item that depends on a passage
        should have that dependency represented in the manifest. This
        option disables checking for that dependency.
-v-trd  Tutorial References and Dependencies: Disables checking for
        tutorial references and dependencies. This is valuable when
        tutorials are packaged separately from the balance of the
        content.
-v-asl  ASL video: Disables checking for ASL video whose ratio of video
        length to item stim length falls outside of two standard
        deviations from mean (adjustible through app.config).
-v-tss  Text-to-Speech Silencing: Disable check for TTS silencing tags
        that do not belong.
-v-ugt  Untagged Glossary Terms: Disable check for glossary terms that
        are tagged in one case but not in another.
-v-tgs  Target Grade Suffix: Disables checking whether the grade suffix in
        a target matches the grade alignment in the item attributes.
-v-dbc  Deprecated Braille Coding: Suppresses braille errors when
        deprecated older codes and conventions are used.
-v-mst  Manifest: Disable checking for manifest errors.

-v+all  Enable all optional validation and tabulation features.
-v+umf  Unreferenced Media File: Enables checking whether media files are
        referenced in the corresponding item, passage, or wordlist.
-v+gtr  Glossary Text Report: Include glossary text in the Glossary
        Report output file.
-v+uwt  Unreferenced Wordlist Terms: Reports an error when a wordlist
        term is not referenced by the corresponding item.
-v+mwa  Missing Wordlist Attachments: Reports errors for missing wordlist
        attachments (audio and images) even when the corresponding term
        is not referenced by any item.
-v+ats  Image Alternate Text for Spanish: Enables checking for alternate
        text on images in the stacked Spanish version of an item.
-v+akv  Answer Key Value: Reports the actual answer key (e.g. ""C"") in the
        ItemReport for selected response items. Without this option simply
        reports 'SR'.

Error severity definitions: Severe The error will prevent the test item from functioning properly or from being scored properly. Degraded The item will display and accept a student response but certain features will not work properly. For example, an accessibility feature may not function properly. Tolerable The test delivery system can tolerate the problem and function without the student noticing anything. However, certain features may not be available to the student. For example, a glossary may be valuable for a term but it doesn’t appear (and the associated text is not highlighted). Benign A problem that doesn’t impact test delivery in any way but may affect future item development. For example, a wordlist term may be missing data but if that term isn’t referenced in an item then it is benign.

Command-line Parameters -a Aggregate the results of all tabulations into one set of reports. -v- Disable a particular validation option (see below) -v+ Enable a particular validation option -h Display this help text -bk The bankkey to use when one is not specfied in an id file (see the -ids argument). If not included, the default bankkey is 200. -ids Optional name of a file containing item IDs to be tabulated in the preceding package. If this file is not specfied then all items in the package will be tabulated. This argument may be repeated once for each package. (See below for details.) -o The path prefix for the output (report) files of the preceding package. If not specified, defaults to the package file or directory name. Required when tabulating an item bank. -lid Indicates that a list of IDs should be reported during the item selection pass. -lidx Indicates that a list of IDs should be reported and that the tabulation should exit after the selection pass - not generating the other reports. -rbrk Export rubrics into HTML files so that they can be examined. The rubrics will be in a folder located in the same place as the reports. -dedup Only report the first instance of an error on a particular item (De-duplicate). -w Wait for a keypress for exiting - helpful when not running from a command-line window. The filename of a local package or the identifier of an online item bank. (See below for details.)

Sample Command-line Input ""D:\Packages\MyPackage.zip"" Tabulates the test package contained in ""MyPackage.zip"" reports are ""MyPackage_SummaryReport.txt"", ""MyPackage_ItemReport.csv"" and so forth.

""D:\Packages\*.zip""
    Tabulates all "".zip"" test packages in the ""D:\Packages"" folder.

-a ""D:\Packages\*.zip""
    Tabulates all "".zip"" test packages in the ""D:\Packages"" folder
    and aggregates the results into one set of reports. The names being
    ""Aggregate_SummaryReport.txt"", ""Aggregate_ItemReport.csv"" and so
    forth.

""D:\Packages\MyElaPackage.zip"" ""D:\Packages\MyMathPackage.zip""
    Tabulates both ""MyElaPackage.zip"" and ""MyMathPackage.zip"" with
    separate results for each.

""D:\Packages\MyElaPackage.zip"" ""D:\Packages\MyMathPackage.zip"" -ids ""mathids.txt""
    Tabulates both ""MyElaPackage.zip"" and ""MyMathPackage.zip"" with
    separate results for each. In ""MyMathPackage.zip"" only the items
    with the ids listed in ""mathids.txt"" are included.

-a ""D:\Packages\MyElaPackage.zip"" ""D:\Packages\MyMathPackage.zip""
    Tabulates both packages and aggregates the results into one set
    of reports.

""D:\Packages\MyElaPackage""
    Tabulates the package represented by the ""MyElaPackage"" file
    folder.

-bank https://itembank.org -at hs3kwBlt8sorK9qwp8Wr -o ""D:\Tabulations\ItemBank""
    Tabulates the entire contents of the specified item bank using
    the corresponding access token and stores the results in a set
    of reports at ""D:\Tabulations\ItemBank"".

-bank https://itembank.org -at hs3kwBlt8sorK9qwp8Wr -ids ""myPullList.csv"" -o ""ItemBank""
    Tabulates the items identifed by ids in ""myPullList.csv"" and stored
    in the item bank. Includes stimuli, wordlists, and tutorials
    referenced by the items in the tabulation. Reports are placed
    in the current directory with ""ItemBank"" for the prefix.