diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 6cf2a1b..0000000 Binary files a/.DS_Store and /dev/null differ diff --git a/README.md b/README.md index fb3245c..dff501a 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,7 @@ knowledge_graph_banner

- Getting set up - • - Tutorials + Getting set up

@@ -17,7 +15,7 @@ Key use cases include: * **Compare state standards**: Adapt content aligned to one state standard to other states, initially in math across Common Core State Standards and 15+ additional states * **Curriculum alignment:** Align your content or create additional materials aligned to curriculum (private-beta access only \- details below on how to join) -Knowledge Graph is distributed as CSV and JSON export files, making it accessible without specialized infrastructure. These files reflect a graph-based model, allowing developers to work with the data in relational databases or other environments. This structure enables rich querying and supports AI-enhanced educational applications. +Knowledge Graph is distributed as graph-native JSONL export files, making it accessible without specialized infrastructure. These files directly represent the underlying graph model, enabling developers to work with the data in graph databases, while remaining easy to ingest with standard data-processing tools. Developers can load the data into graph databases for relationship-centric querying or transform it for use in relational databases and data pipelines. This structure enables rich querying and supports AI-enhanced educational applications. For complete setup instructions and usage examples, see the [full docs](https://docs.learningcommons.org/knowledge-graph/). @@ -25,82 +23,148 @@ For complete setup instructions and usage examples, see the [full docs](https:// | Path | Description | | :---- | :---- | -| [import\_scripts/](./import_scripts/) | Helper scripts to import Knowledge Graph data into relational databases | -| [sample\_queries/](./sample_queries/) | Example SQL queries for exploring Knowledge Graph data | | [tutorials/](./tutorials/) | Standalone example apps to demonstrate how Knowledge Graph data could be applied to solve different use cases | | [LICENSE](./LICENSE.md) | Open source license details | ## **Quick Start** -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** +You can access the Knowledge Graph data using: -The knowledge graph data is available for download in both CSV and JSON formats. The graph data is exported with each file representing a specific entity type, and a relationships file capturing the connections between entities. +- **REST API**: Authenticate and make HTTP requests to retrieve academic standards directly. Best for applications that need real-time access. *(Currently available only to private beta users)* +- **MCP Server**: AI models can reliably work with academic standards, learning components, and learning progressions. They can resolve standards, decompose them into granular learning components, and trace progressions across standards. *(Currently available only to private beta users)* +- **Local JSONL**: Download local JSONL files and query them directly. Best for offline access, custom processing, or complex queries. *(Publicly available)* -**CSV files:** UTF-8 encoded with comma delimiters and quoted fields. All CSV files include header rows with column names. +### REST API -**JSON files:** Newline delimited JSON format with UTF-8 encoding. +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. -## Files +#### What you'll do -* `StandardsFramework`: Educational standards frameworks -* `StandardsFrameworkItem`: Individual standards and learning objectives within frameworks -* `LearningComponent`: Granular, precise representations of individual skills or concepts -* `Relationships`: Connections and associations between all entity types +- Authenticate using an API key +- Get a standards framework identifier (CASE UUID) for Multi-State Mathematics +- Retrieve a list of academic standards using that framework identifier -## Download options +#### What you'll need -There are two options to download the files: direct s3 links, or using curl commands. +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform -### Direct S3 links +#### Base URL -Click links to download files directly. Files will download to your browser's default location (typically `~/Downloads`). +All REST API requests should be sent to: -**CSV files:** -- [StandardsFramework.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFramework.csv?ref=github) -- [StandardsFrameworkItem.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFrameworkItem.csv?ref=github) -- [LearningComponent.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/LearningComponent.csv?ref=github) -- [Relationships.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/Relationships.csv?ref=github) +``` +https://api.learningcommons.org/knowledge-graph/v0 +``` -**For SQL database imports:** Move the downloaded CSV files to `/tmp/kg-data/` to use the import scripts without modification: +#### Authentication -```bash -mkdir -p /tmp/kg-data -mv ~/Downloads/StandardsFramework.csv ~/Downloads/StandardsFrameworkItem.csv ~/Downloads/LearningComponent.csv ~/Downloads/Relationships.csv /tmp/kg-data/ -``` +Include your API key in the `x-api-key` header on every request: + +``` +x-api-key: YOUR_API_KEY +``` + +#### STEP 1: Get a standards framework identifier -**JSON files:** -- [StandardsFramework.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFramework.json?ref=github) -- [StandardsFrameworkItem.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFrameworkItem.json?ref=github) -- [LearningComponent.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/LearningComponent.json?ref=github) -- [Relationships.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/Relationships.json?ref=github) +Use your preferred HTTP client to send a GET request to the standards frameworks endpoint to get the CASE UUID for Multi-State Mathematics. -### Using curl commands +```bash +curl -X GET \ + -H "x-api-key: YOUR_API_KEY" \ + "https://api.learningcommons.org/knowledge-graph/v0/standards-frameworks?academicSubject=Mathematics&jurisdiction=Multi-State" +``` -If you don't have `curl` installed, see [installation instructions](https://github.com/curl/curl). +You should receive a 200 response with the CCSS Math framework for Multi-State, including the framework name, jurisdiction, adoption status, and a `caseIdentifierUUID` (GUID). Copy the `caseIdentifierUUID` from the response for the next step. -**Recommended**: Download files to `/tmp/kg-data/` for compatibility with the SQL import scripts: +#### STEP 2: Retrieve academic standards + +Use the `caseIdentifierUUID` you copied from Step 1's response with the academic standards endpoint to retrieve the individual standards for that framework. ```bash -# Create directory and download CSV files -mkdir -p /tmp/kg-data -cd /tmp/kg-data - -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFramework.csv?ref=gh_curl" -o StandardsFramework.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFrameworkItem.csv?ref=gh_curl" -o StandardsFrameworkItem.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/LearningComponent.csv?ref=gh_curl" -o LearningComponent.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/Relationships.csv?ref=gh_curl" -o Relationships.csv +curl -X GET \ + -H "x-api-key: YOUR_API_KEY" \ + "https://api.learningcommons.org/knowledge-graph/v0/academic-standards?standardsFrameworkCaseIdentifierUUID=YOUR_UUID_FROM_STEP_1" ``` + +> **Note:** If you skipped Step 1, you can use the CCSS Math framework UUID: `c6496676-d7cb-11e8-824f-0242ac160002` in place of `YOUR_UUID_FROM_STEP_1`. + +You should see a paginated list of academic standards aligned to that framework, including statement codes, descriptions, grade levels, and subject information. + +Example response: + +```json +{ + "data": [ + { + "identifier": "e1755456-c533-5a84-891e-59725c0479e0", + "caseIdentifierURI": "https://satchelcommons.com/ims/case/v1p0/CFItems/6b9bf846-d7cc-11e8-824f-0242ac160002", + "caseIdentifierUUID": "6b9bf846-d7cc-11e8-824f-0242ac160002", + "name": null, + "statementCode": "3.NF.A.1", + "description": "Understand a fraction $\\frac{1}{b}$ as the quantity formed by 1 part when a whole is partitioned into b equal parts; understand a fraction $\\frac{a}{b}$ as the quantity formed by a parts of size $\\frac{1}{b}$.", + "statementType": "Standard", + "normalizedStatementType": "Standard", + "jurisdiction": "Multi-State", + "academicSubject": "Mathematics", + "gradeLevel": ["3"], + "inLanguage": "en-US", + "dateCreated": null, + "dateModified": "2025-02-05", + "notes": null, + "author": "1EdTech", + "provider": "Learning Commons", + "license": "https://creativecommons.org/licenses/by/4.0/", + "attributionStatement": "Knowledge Graph is provided by Learning Commons under the CC BY-4.0 license. Learning Commons received state standards and written permission under CC BY-4.0 from 1EdTech." + } + ], + "pagination": { + "limit": 1, + "nextCursor": "eyJpZGVudGlmaWVyIjogImUxNzU1NDU2LWM1MzMtNWE4NC04OTFlLTU5NzI1YzA0NzllMCJ9", + "hasMore": true + } +} +``` + +### Local JSONL Files + +The Knowledge Graph data is available for download in newline delimited JSONL format with UTF-8 encoding. The graph data is exported with `nodes.jsonl` representing the nodes of the knowledge graph and the `relationships.jsonl` file capturing the connections between nodes. + +#### Files + +- `nodes.jsonl`: Contains graph node records, defining each node by a unique identifier, labels, and a set of associated properties. +- `relationships.jsonl`: Contains graph relationship records, describing how nodes are connected, including the relationship type, properties, and the source and target nodes. + +#### Download options + +You can download the files through direct links or using curl commands. + +**Direct links** + +- [nodes.jsonl](https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/nodes.jsonl?ref=github) +- [relationships.jsonl](https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/relationships.jsonl?ref=github) + +**Using curl commands** + +If you don't have curl installed, visit https://github.com/curl/curl for installation instructions. + ```bash -# Download JSON files -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFramework.json?ref=gh_curl" -o StandardsFramework.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFrameworkItem.json?ref=gh_curl" -o StandardsFrameworkItem.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/LearningComponent.json?ref=gh_curl" -o LearningComponent.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/Relationships.json?ref=gh_curl" -o Relationships.json +curl -L "https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/nodes.jsonl?ref=gh_curl" -o nodes.jsonl +curl -L "https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/relationships.jsonl?ref=gh_curl" -o relationships.jsonl ``` -### **Next steps** +#### Querying with jq + +One option for querying the JSONL files is to use [jq](https://jqlang.github.io/jq/). Example to extract Common Core math standards: + +```bash +jq -c 'select((.labels | contains(["StandardsFrameworkItem"])) and .properties.jurisdiction == "Multi-State" and .properties.academicSubject == "Mathematics")' nodes.jsonl > common_core_math_standards.jsonl +``` -This quick start walked you through how to download Knowledge Graph so you can start using it. You can explore next steps under [import\_scripts/](./import_scripts/), [sample\_queries/](./sample_queries/) and [tutorials/](./tutorials/). +This filters for nodes with: +- **Label:** `StandardsFrameworkItem` +- **Jurisdiction:** `Multi-State` (Common Core) +- **Academic Subject:** `Mathematics` ## **Support & Feedback** diff --git a/import_scripts/.DS_Store b/import_scripts/.DS_Store deleted file mode 100644 index f112dce..0000000 Binary files a/import_scripts/.DS_Store and /dev/null differ diff --git a/import_scripts/mysql/README.md b/import_scripts/mysql/README.md deleted file mode 100644 index f223b9a..0000000 --- a/import_scripts/mysql/README.md +++ /dev/null @@ -1,71 +0,0 @@ -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** - -# MySQL Import Guide - -This guide provides instructions for loading the Learning Commons Knowledge Graph dataset into a MySQL database. - -## Prerequisites - -- A running MySQL instance. If you don't have MySQL installed locally, you can use Docker to run a MySQL container. For detailed instructions on setting up and running MySQL with Docker, refer to the [official MySQL Docker documentation](https://hub.docker.com/_/mysql). -- Downloaded CSV data files (see the [main README](../../README.md) for download instructions) - -## Setup Steps - -### 1. Create Database and Tables - -First create a dedicated database if that's preferred: - -```bash -mysql -u -p -e "CREATE DATABASE ;" -``` - -Then connect to your MySQL database and create the necessary tables by running: - -```bash -mysql -u -p < create_tables.sql -``` - -Or execute the SQL file content directly in your MySQL client. See `create_tables.sql` for the complete table definitions. - -### 2. Load CSV Data - -After creating the tables, load the data: - -1. If you downloaded files to `/tmp/kg-data/` using the commands in the main README, the script will work without modification. Otherwise, update the file paths in `load_data.sql` to point to your downloaded CSV files. - -2. Enable local file loading on the server (if not already enabled): - -```sql -mysql -u -p -e "SET GLOBAL local_infile=1;" -``` - -3. Execute the load script with local file loading enabled: - -```bash -mysql -u -p --local-infile=1 < load_data.sql -``` - -**Note**: The `local_infile` setting must be enabled on both the server (`SET GLOBAL local_infile=1`) and client (`--local-infile=1`) for `LOAD DATA LOCAL INFILE` to work. - -## Query Examples - -Once the data is loaded, you can run queries to explore the tables: - -```sql --- Show all tables -SHOW TABLES; - --- Sample data queries -SELECT * FROM standards_framework LIMIT 10; -SELECT * FROM standards_framework_item LIMIT 10; -SELECT * FROM learning_component LIMIT 10; -SELECT * FROM relationships LIMIT 10; -``` - -## Docker MySQL Setup - -If using Docker MySQL, you can connect using: - -```bash -docker exec -it mysql -u -p -``` diff --git a/import_scripts/mysql/create_tables.sql b/import_scripts/mysql/create_tables.sql deleted file mode 100644 index a9d64e0..0000000 --- a/import_scripts/mysql/create_tables.sql +++ /dev/null @@ -1,75 +0,0 @@ --- MySQL version of the init.sql script -CREATE TABLE IF NOT EXISTS standards_framework_item ( - `identifier` TEXT, - `caseIdentifierURI` TEXT, - `caseIdentifierUUID` TEXT, - `statementCode` TEXT, - `description` TEXT, - `statementType` TEXT, - `normalizedStatementType` TEXT, - `jurisdiction` TEXT, - `academicSubject` TEXT, - `gradeLevel` TEXT, - `inLanguage` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `notes` TEXT, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS standards_framework ( - `identifier` TEXT, - `caseIdentifierURI` TEXT, - `caseIdentifierUUID` TEXT, - `name` TEXT, - `description` TEXT, - `jurisdiction` TEXT, - `academicSubject` TEXT, - `inLanguage` TEXT, - `adoptionStatus` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `notes` TEXT, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS learning_component ( - `identifier` TEXT, - `description` TEXT, - `academicSubject` TEXT, - `inLanguage` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS relationships ( - `identifier` TEXT, - `relationshipType` TEXT, - `description` TEXT, - `sourceEntity` TEXT, - `sourceEntityKey` TEXT, - `sourceEntityValue` TEXT, - `targetEntity` TEXT, - `targetEntityKey` TEXT, - `targetEntityValue` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT, - `jaccard` DOUBLE, - `ccssLCCount` INT, - `sharedLCCount` INT, - `stateLCCount` INT -); diff --git a/import_scripts/mysql/load_data.sql b/import_scripts/mysql/load_data.sql deleted file mode 100644 index abb8734..0000000 --- a/import_scripts/mysql/load_data.sql +++ /dev/null @@ -1,32 +0,0 @@ --- Load data from CSV files --- Default location: /tmp/kg-data/ --- If you downloaded the files to a different location, update the paths below --- Note: Use mysql --local-infile=1 when running this script - -LOAD DATA LOCAL INFILE '/tmp/kg-data/StandardsFramework.csv' -INTO TABLE standards_framework -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/StandardsFrameworkItem.csv' -INTO TABLE standards_framework_item -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/LearningComponent.csv' -INTO TABLE learning_component -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/Relationships.csv' -INTO TABLE relationships -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; \ No newline at end of file diff --git a/import_scripts/postgresql/README.md b/import_scripts/postgresql/README.md deleted file mode 100644 index 21cfe5e..0000000 --- a/import_scripts/postgresql/README.md +++ /dev/null @@ -1,65 +0,0 @@ -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** - -# PostgreSQL Import Guide - -This guide provides instructions for loading the Learning Commons Knowledge Graph dataset into a PostgreSQL database. - -## Prerequisites - -- A running PostgreSQL instance. If you don't have PostgreSQL installed locally, you can use Docker to run a PostgreSQL container. For detailed instructions on setting up and running PostgreSQL with Docker, refer to the [official PostgreSQL Docker documentation](https://hub.docker.com/_/postgres). -- Downloaded CSV data files (see the [main README](../../README.md) for download instructions) - -## Setup Steps - -### 1. Create Database and Tables - -First create a dedicated database if that's preferred: - -```bash -psql -U -c "CREATE DATABASE ;" -``` - -Then connect to your PostgreSQL database and create the necessary tables by running: - -```bash -psql -U -d -f create_tables.sql -``` - -Or execute the SQL file content directly in your PostgreSQL client. See `create_tables.sql` for the complete table definitions. - -### 2. Load CSV Data - -After creating the tables, load the data: - -1. If you downloaded files to `/tmp/kg-data/` using the commands in the main README, the script will work without modification. Otherwise, update the file paths in `load_data.sql` to point to your downloaded CSV files. - -2. Execute the load script: - -```bash -psql -U -d -f load_data.sql -``` - -**Note**: Ensure PostgreSQL has read access to the file paths specified in the load script. - -## Query Examples - -Once the data is loaded, you can run queries to explore the tables: - -```sql --- List all tables -\dt - --- Sample data queries -SELECT * FROM standards_framework LIMIT 10; -SELECT * FROM standards_framework_item LIMIT 10; -SELECT * FROM learning_component LIMIT 10; -SELECT * FROM relationships LIMIT 10; -``` - -## Docker PostgreSQL Setup - -If using Docker PostgreSQL, you can connect using: - -```bash -docker exec -it psql -U -d -``` diff --git a/import_scripts/postgresql/create_tables.sql b/import_scripts/postgresql/create_tables.sql deleted file mode 100644 index 42927c8..0000000 --- a/import_scripts/postgresql/create_tables.sql +++ /dev/null @@ -1,74 +0,0 @@ -CREATE TABLE IF NOT EXISTS standards_framework_item ( - "identifier" TEXT, - "caseIdentifierURI" TEXT, - "caseIdentifierUUID" TEXT, - "statementCode" TEXT, - "description" TEXT, - "statementType" TEXT, - "normalizedStatementType" TEXT, - "jurisdiction" TEXT, - "academicSubject" TEXT, - "gradeLevel" JSON, - "inLanguage" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "notes" TEXT, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS standards_framework ( - "identifier" TEXT, - "caseIdentifierURI" TEXT, - "caseIdentifierUUID" TEXT, - "name" TEXT, - "description" TEXT, - "jurisdiction" TEXT, - "academicSubject" TEXT, - "inLanguage" TEXT, - "adoptionStatus" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "notes" TEXT, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS learning_component ( - "identifier" TEXT, - "description" TEXT, - "academicSubject" TEXT, - "inLanguage" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS relationships ( - "identifier" TEXT, - "relationshipType" TEXT, - "description" TEXT, - "sourceEntity" TEXT, - "sourceEntityKey" TEXT, - "sourceEntityValue" TEXT, - "targetEntity" TEXT, - "targetEntityKey" TEXT, - "targetEntityValue" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT, - "jaccard" DOUBLE PRECISION, - "ccssLCCount" INT, - "sharedLCCount" INT, - "stateLCCount" INT -); diff --git a/import_scripts/postgresql/load_data.sql b/import_scripts/postgresql/load_data.sql deleted file mode 100644 index c137a4e..0000000 --- a/import_scripts/postgresql/load_data.sql +++ /dev/null @@ -1,8 +0,0 @@ --- Load data from CSV files --- Default location: /tmp/kg-data/ --- If you downloaded the files to a different location, update the paths below - -\COPY standards_framework_item FROM '/tmp/kg-data/StandardsFrameworkItem.csv' DELIMITER ',' CSV HEADER; -\COPY standards_framework FROM '/tmp/kg-data/StandardsFramework.csv' DELIMITER ',' CSV HEADER; -\COPY learning_component FROM '/tmp/kg-data/LearningComponent.csv' DELIMITER ',' CSV HEADER; -\COPY relationships FROM '/tmp/kg-data/Relationships.csv' DELIMITER ',' CSV HEADER; \ No newline at end of file diff --git a/sample_queries/.DS_Store b/sample_queries/.DS_Store deleted file mode 100644 index 9909f34..0000000 Binary files a/sample_queries/.DS_Store and /dev/null differ diff --git a/sample_queries/mysql/.keep b/sample_queries/mysql/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql b/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql deleted file mode 100644 index ddd9140..0000000 --- a/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql +++ /dev/null @@ -1,20 +0,0 @@ -WITH grade6_standards AS ( - SELECT `caseIdentifierUUID` - FROM standards_framework_item - WHERE `gradeLevel` LIKE '%"6"%' -), -supporting_components AS ( - SELECT DISTINCT r.`sourceEntityValue` AS lc_id - FROM relationships r - JOIN grade6_standards g - ON g.`caseIdentifierUUID` = r.`targetEntityValue` - WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' -) -SELECT lc.* -FROM learning_component lc -JOIN supporting_components c - ON lc.`identifier` = c.lc_id; \ No newline at end of file diff --git a/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql b/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql deleted file mode 100644 index f4bc13f..0000000 --- a/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT lc.* -FROM relationships r -JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba1c7ad-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql b/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql deleted file mode 100644 index 5d5ada2..0000000 --- a/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = '019a93e7-1877-5e9b-b4b3-476f201fccc8'; \ No newline at end of file diff --git a/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql b/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql deleted file mode 100644 index 42a572e..0000000 --- a/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql +++ /dev/null @@ -1,37 +0,0 @@ --- Find the best Texas state standard matches for a given CCSSM standard --- Returns crosswalks ordered by Jaccard score (highest similarity first) --- with metadata about both the CCSSM and matching Texas state standards - -SELECT - -- CCSSM Standard Information - ccss.`statementCode` AS ccss_standard_code, - ccss.`description` AS ccss_description, - ccss.`gradeLevel` AS ccss_grade_level, - ccss.`jurisdiction` AS ccss_jurisdiction, - - -- State Standard Information - state.`statementCode` AS state_standard_code, - state.`description` AS state_description, - state.`gradeLevel` AS state_grade_level, - state.`jurisdiction` AS state_jurisdiction, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount`, - - -- Entity Values for further joins if needed - r.`sourceEntityValue` AS state_uuid, - r.`targetEntityValue` AS ccss_uuid -FROM relationships r -JOIN standards_framework_item state - ON state.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss - ON ccss.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND ccss.`statementCode` = '6.EE.B.5' - AND ccss.`jurisdiction` = 'Multi-State' - AND state.`jurisdiction` = 'Texas' -ORDER BY r.`jaccard` DESC -LIMIT 10; diff --git a/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql b/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql deleted file mode 100644 index fe18fe8..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql +++ /dev/null @@ -1,17 +0,0 @@ --- Get all crosswalks with state standard information --- Returns state → CCSSM standard alignments ordered by Jaccard score - -SELECT - state.`statementCode` AS state_standard_code, - state.`jurisdiction` AS state_jurisdiction, - r.`sourceEntityValue`, - r.`targetEntityValue`, - r.`jaccard`, - r.`stateLCCount`, - r.`ccssLCCount`, - r.`sharedLCCount` -FROM relationships r -JOIN standards_framework_item state - ON state.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' -ORDER BY r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql deleted file mode 100644 index 14cf2c1..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql +++ /dev/null @@ -1,22 +0,0 @@ --- Get all Texas crosswalks that meet or exceed a specified Jaccard similarity threshold --- Ordered by CCSSM standard code, then Jaccard score - -SELECT - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`statementCode` AS state_standard_code, - ccss_std.`statementCode` AS ccss_standard_code, - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND r.`jaccard` >= 0.7 - AND state_std.`jurisdiction` = 'Texas' -ORDER BY - ccss_std.`statementCode`, - r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql deleted file mode 100644 index 073ccc7..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Get all crosswalks for a specific state jurisdiction --- Returns comprehensive metadata for both state and CCSSM standards --- Ordered by state standard code and Jaccard score - -SELECT - -- State Standard Information - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`statementCode` AS state_standard_code, - state_std.`gradeLevel` AS state_grade_level, - state_std.`description` AS state_description, - state_std.`academicSubject` AS state_academic_subject, - - -- CCSSM Standard Information - ccss_std.`statementCode` AS ccss_standard_code, - ccss_std.`gradeLevel` AS ccss_grade_level, - ccss_std.`description` AS ccss_description, - ccss_std.`academicSubject` AS ccss_academic_subject, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND state_std.`jurisdiction` = 'Texas' - AND state_std.`academicSubject` = 'Mathematics' -ORDER BY - state_std.`statementCode`, - ccss_std.`statementCode`, - r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql deleted file mode 100644 index ae987c2..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql +++ /dev/null @@ -1,34 +0,0 @@ --- Find Texas state standard matches for a given CCSSM standard with full metadata --- Returns multiple matching Texas state standards ordered by similarity - -SELECT - -- CCSSM Standard Information - ccss_std.`statementCode` AS ccss_standard_code, - ccss_std.`jurisdiction` AS ccss_jurisdiction, - ccss_std.`gradeLevel` AS ccss_grade_level, - ccss_std.`description` AS ccss_description, - ccss_std.`academicSubject` AS ccss_academic_subject, - - -- State Standard Information - state_std.`statementCode` AS state_standard_code, - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`gradeLevel` AS state_grade_level, - state_std.`description` AS state_description, - state_std.`academicSubject` AS state_academic_subject, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND ccss_std.`statementCode` = '6.EE.B.5' - AND ccss_std.`jurisdiction` = 'Multi-State' - AND state_std.`jurisdiction` = 'Texas' -ORDER BY r.`jaccard` DESC -LIMIT 10; diff --git a/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql b/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql deleted file mode 100644 index 85728c2..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql +++ /dev/null @@ -1,55 +0,0 @@ --- Get the Learning Components that support both a state standard and a CCSSM standard --- Returns three categories: shared LCs (in both), state-only LCs, and CCSSM-only LCs --- This shows the pedagogical overlap and differences between crosswalked standards - -WITH state_lcs AS ( - SELECT lc.`identifier`, lc.`description` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` - WHERE r.`relationshipType` = 'supports' - AND sfi.`statementCode` = '111.26.b.4.D' - AND sfi.`jurisdiction` = 'Texas' -), -ccss_lcs AS ( - SELECT lc.`identifier`, lc.`description` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` - WHERE r.`relationshipType` = 'supports' - AND sfi.`statementCode` = '6.RP.A.2' - AND sfi.`jurisdiction` = 'Multi-State' -) -SELECT - 'shared' AS lc_type, - state_lcs.`identifier`, - state_lcs.`description` -FROM state_lcs -INNER JOIN ccss_lcs - ON state_lcs.`identifier` = ccss_lcs.`identifier` - -UNION ALL - -SELECT - 'state_only' AS lc_type, - state_lcs.`identifier`, - state_lcs.`description` -FROM state_lcs -LEFT JOIN ccss_lcs - ON state_lcs.`identifier` = ccss_lcs.`identifier` -WHERE ccss_lcs.`identifier` IS NULL - -UNION ALL - -SELECT - 'ccss_only' AS lc_type, - ccss_lcs.`identifier`, - ccss_lcs.`description` -FROM ccss_lcs -LEFT JOIN state_lcs - ON ccss_lcs.`identifier` = state_lcs.`identifier` -WHERE state_lcs.`identifier` IS NULL; diff --git a/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql b/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql deleted file mode 100644 index 21e5e49..0000000 --- a/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM learning_component -WHERE `identifier` = '0046446a-0a9b-5ace-92a3-23d4bb158c68'; \ No newline at end of file diff --git a/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql b/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql deleted file mode 100644 index e7958bf..0000000 --- a/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM learning_component; \ No newline at end of file diff --git a/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql b/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql deleted file mode 100644 index aa20c65..0000000 --- a/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'relatesTo' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba115f2-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql b/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql deleted file mode 100644 index 08aedfd..0000000 --- a/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'buildsTowards' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba1b8b8-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql b/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql deleted file mode 100644 index 0fabfb7..0000000 --- a/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework -WHERE `academicSubject` = 'Mathematics' - AND `jurisdiction` = 'California'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql b/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql deleted file mode 100644 index 8879bd5..0000000 --- a/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql +++ /dev/null @@ -1,2 +0,0 @@ -SELECT * -FROM standards_framework WHERE `caseIdentifierUUID` = 'c64961be-d7cb-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql b/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql deleted file mode 100644 index c579f3e..0000000 --- a/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM standards_framework; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql b/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql deleted file mode 100644 index d37e3fd..0000000 --- a/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql +++ /dev/null @@ -1,33 +0,0 @@ --- WARNING: This recursive query may hit performance issues or database recursion limits --- on large/deep hierarchies. If execution fails, either increase your database's recursion --- limit settings or use an iterative approach in application code. - -WITH RECURSIVE all_descendants AS ( - -- Base case: direct children of the StandardsFramework - SELECT sfi.`caseIdentifierUUID` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - WHERE r.`sourceEntity` = 'StandardsFramework' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = 'c64961be-d7cb-11e8-824f-0242ac160002' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - - UNION ALL - - -- Recursive case: children of already found items - SELECT sfi.`caseIdentifierUUID` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN all_descendants ad - ON ad.`caseIdentifierUUID` = r.`sourceEntityValue` - WHERE r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`relationshipType` = 'hasChild' -) -SELECT `caseIdentifierUUID` -FROM all_descendants; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql b/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql deleted file mode 100644 index 3395a17..0000000 --- a/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT parent.* -FROM relationships r -JOIN standards_framework_item parent - ON parent.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'hasChild' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '57165d0c-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql b/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql deleted file mode 100644 index 0b0c888..0000000 --- a/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `statementCode` = '7.G.A.1'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql b/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql deleted file mode 100644 index e7be152..0000000 --- a/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `caseIdentifierUUID` = 'c4401baa-b0e5-496c-92c3-352fda95e5ae'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql b/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql deleted file mode 100644 index 4098ac0..0000000 --- a/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `gradeLevel` LIKE '%"6"%' -AND `jurisdiction` = 'Multi-State'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql b/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql deleted file mode 100644 index 19a22d6..0000000 --- a/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT child.* -FROM relationships r -JOIN standards_framework_item child - ON child.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasChild' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = '57165a88-d7cc-11e8-824f-0242ac160002' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID'; \ No newline at end of file diff --git a/sample_queries/postgresql/.DS_Store b/sample_queries/postgresql/.DS_Store deleted file mode 100644 index 18fd051..0000000 Binary files a/sample_queries/postgresql/.DS_Store and /dev/null differ diff --git a/sample_queries/postgresql/.keep b/sample_queries/postgresql/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql b/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql deleted file mode 100644 index 1dcf44f..0000000 --- a/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql +++ /dev/null @@ -1,23 +0,0 @@ -WITH grade6_standards AS ( - SELECT "caseIdentifierUUID" - FROM standards_framework_item - WHERE EXISTS ( - SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem - WHERE elem = '6' - ) -), -supporting_components AS ( - SELECT DISTINCT r."sourceEntityValue" AS lc_id - FROM relationships r - JOIN grade6_standards g - ON g."caseIdentifierUUID" = r."targetEntityValue" - WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' -) -SELECT lc.* -FROM learning_component lc -JOIN supporting_components c - ON lc."identifier" = c.lc_id; \ No newline at end of file diff --git a/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql b/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql deleted file mode 100644 index c987126..0000000 --- a/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT lc.* -FROM relationships r -JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" -WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba1c7ad-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql b/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql deleted file mode 100644 index 4e59611..0000000 --- a/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = '019a93e7-1877-5e9b-b4b3-476f201fccc8'; \ No newline at end of file diff --git a/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql b/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql deleted file mode 100644 index acaffbc..0000000 --- a/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql +++ /dev/null @@ -1,37 +0,0 @@ --- Find the best Texas state standard matches for a given CCSSM standard --- Returns crosswalks ordered by Jaccard score (highest similarity first) --- with metadata about both the CCSSM and matching Texas state standards - -SELECT - -- CCSSM Standard Information - ccss."statementCode" AS ccss_standard_code, - ccss."description" AS ccss_description, - ccss."gradeLevel" AS ccss_grade_level, - ccss."jurisdiction" AS ccss_jurisdiction, - - -- State Standard Information - state."statementCode" AS state_standard_code, - state."description" AS state_description, - state."gradeLevel" AS state_grade_level, - state."jurisdiction" AS state_jurisdiction, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount", - - -- Entity Values for further joins if needed - r."sourceEntityValue" AS state_uuid, - r."targetEntityValue" AS ccss_uuid -FROM relationships r -JOIN standards_framework_item state - ON state."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss - ON ccss."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND ccss."statementCode" = '6.EE.B.5' - AND ccss."jurisdiction" = 'Multi-State' - AND state."jurisdiction" = 'Texas' -ORDER BY r."jaccard" DESC -LIMIT 10; diff --git a/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql b/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql deleted file mode 100644 index 2836e76..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql +++ /dev/null @@ -1,17 +0,0 @@ --- Get all crosswalks with state standard information --- Returns state → CCSSM standard alignments ordered by Jaccard score - -SELECT - state."statementCode" AS state_standard_code, - state."jurisdiction" AS state_jurisdiction, - r."sourceEntityValue", - r."targetEntityValue", - r."jaccard", - r."stateLCCount", - r."ccssLCCount", - r."sharedLCCount" -FROM relationships r -JOIN standards_framework_item state - ON state."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' -ORDER BY r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql deleted file mode 100644 index dfd761f..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql +++ /dev/null @@ -1,22 +0,0 @@ --- Get all Texas crosswalks that meet or exceed a specified Jaccard similarity threshold --- Ordered by CCSSM standard code, then Jaccard score - -SELECT - state_std."jurisdiction" AS state_jurisdiction, - state_std."statementCode" AS state_standard_code, - ccss_std."statementCode" AS ccss_standard_code, - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND r."jaccard" >= 0.7 - AND state_std."jurisdiction" = 'Texas' -ORDER BY - ccss_std."statementCode", - r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql deleted file mode 100644 index 849d6f6..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Get all crosswalks for a specific state jurisdiction --- Returns comprehensive metadata for both state and CCSSM standards --- Ordered by state standard code and Jaccard score - -SELECT - -- State Standard Information - state_std."jurisdiction" AS state_jurisdiction, - state_std."statementCode" AS state_standard_code, - state_std."gradeLevel" AS state_grade_level, - state_std."description" AS state_description, - state_std."academicSubject" AS state_academic_subject, - - -- CCSSM Standard Information - ccss_std."statementCode" AS ccss_standard_code, - ccss_std."gradeLevel" AS ccss_grade_level, - ccss_std."description" AS ccss_description, - ccss_std."academicSubject" AS ccss_academic_subject, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND state_std."jurisdiction" = 'Texas' - AND state_std."academicSubject" = 'Mathematics' -ORDER BY - state_std."statementCode", - ccss_std."statementCode", - r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql deleted file mode 100644 index 34b1781..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql +++ /dev/null @@ -1,34 +0,0 @@ --- Find Texas state standard matches for a given CCSSM standard with full metadata --- Returns multiple matching Texas state standards ordered by similarity - -SELECT - -- CCSSM Standard Information - ccss_std."statementCode" AS ccss_standard_code, - ccss_std."jurisdiction" AS ccss_jurisdiction, - ccss_std."gradeLevel" AS ccss_grade_level, - ccss_std."description" AS ccss_description, - ccss_std."academicSubject" AS ccss_academic_subject, - - -- State Standard Information - state_std."statementCode" AS state_standard_code, - state_std."jurisdiction" AS state_jurisdiction, - state_std."gradeLevel" AS state_grade_level, - state_std."description" AS state_description, - state_std."academicSubject" AS state_academic_subject, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND ccss_std."statementCode" = '6.EE.B.5' - AND ccss_std."jurisdiction" = 'Multi-State' - AND state_std."jurisdiction" = 'Texas' -ORDER BY r."jaccard" DESC -LIMIT 10; diff --git a/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql b/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql deleted file mode 100644 index cd51ead..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql +++ /dev/null @@ -1,55 +0,0 @@ --- Get the Learning Components that support both a state standard and a CCSSM standard --- Returns three categories: shared LCs (in both), state-only LCs, and CCSSM-only LCs --- This shows the pedagogical overlap and differences between crosswalked standards - -WITH state_lcs AS ( - SELECT lc."identifier", lc."description" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" - WHERE r."relationshipType" = 'supports' - AND sfi."statementCode" = '111.26.b.4.D' - AND sfi."jurisdiction" = 'Texas' -), -ccss_lcs AS ( - SELECT lc."identifier", lc."description" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" - WHERE r."relationshipType" = 'supports' - AND sfi."statementCode" = '6.RP.A.2' - AND sfi."jurisdiction" = 'Multi-State' -) -SELECT - 'shared' AS lc_type, - state_lcs."identifier", - state_lcs."description" -FROM state_lcs -INNER JOIN ccss_lcs - ON state_lcs."identifier" = ccss_lcs."identifier" - -UNION ALL - -SELECT - 'state_only' AS lc_type, - state_lcs."identifier", - state_lcs."description" -FROM state_lcs -LEFT JOIN ccss_lcs - ON state_lcs."identifier" = ccss_lcs."identifier" -WHERE ccss_lcs."identifier" IS NULL - -UNION ALL - -SELECT - 'ccss_only' AS lc_type, - ccss_lcs."identifier", - ccss_lcs."description" -FROM ccss_lcs -LEFT JOIN state_lcs - ON ccss_lcs."identifier" = state_lcs."identifier" -WHERE state_lcs."identifier" IS NULL; diff --git a/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql b/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql deleted file mode 100644 index fc93841..0000000 --- a/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM learning_component -WHERE "identifier" = '0046446a-0a9b-5ace-92a3-23d4bb158c68'; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql b/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql deleted file mode 100644 index e7958bf..0000000 --- a/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM learning_component; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql b/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql deleted file mode 100644 index 3b15040..0000000 --- a/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'relatesTo' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba115f2-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql b/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql deleted file mode 100644 index 9051387..0000000 --- a/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'buildsTowards' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba1b8b8-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql b/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql deleted file mode 100644 index b61b9a4..0000000 --- a/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework -WHERE "academicSubject" = 'Mathematics' - AND "jurisdiction" = 'California'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql b/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql deleted file mode 100644 index 20b2622..0000000 --- a/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql +++ /dev/null @@ -1,2 +0,0 @@ -SELECT * -FROM standards_framework WHERE "caseIdentifierUUID" = 'c64961be-d7cb-11e8-824f-0242ac160002' \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql b/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql deleted file mode 100644 index c579f3e..0000000 --- a/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM standards_framework; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql b/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql deleted file mode 100644 index c84095f..0000000 --- a/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql +++ /dev/null @@ -1,33 +0,0 @@ --- WARNING: This recursive query may hit performance issues or database recursion limits --- on large/deep hierarchies. If execution fails, either increase your database's recursion --- limit settings or use an iterative approach in application code. - -WITH RECURSIVE all_descendants AS ( - -- Base case: direct children of the StandardsFramework - SELECT sfi."caseIdentifierUUID" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - WHERE r."sourceEntity" = 'StandardsFramework' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = 'c64961be-d7cb-11e8-824f-0242ac160002' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - - UNION ALL - - -- Recursive case: children of already found items - SELECT sfi."caseIdentifierUUID" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN all_descendants ad - ON ad."caseIdentifierUUID" = r."sourceEntityValue" - WHERE r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."relationshipType" = 'hasChild' -) -SELECT "caseIdentifierUUID" -FROM all_descendants; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql b/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql deleted file mode 100644 index a57f840..0000000 --- a/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT parent.* -FROM relationships r -JOIN standards_framework_item parent - ON parent."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'hasChild' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '57165d0c-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql b/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql deleted file mode 100644 index ba23ce9..0000000 --- a/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE "statementCode" = '7.G.A.1'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql b/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql deleted file mode 100644 index ca1e418..0000000 --- a/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE "caseIdentifierUUID" = 'c4401baa-b0e5-496c-92c3-352fda95e5ae'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql b/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql deleted file mode 100644 index 6ce951f..0000000 --- a/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql +++ /dev/null @@ -1,7 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE EXISTS ( - SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem - WHERE elem = '6' -) -AND "jurisdiction" = 'Multi-State'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql b/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql deleted file mode 100644 index d58cffe..0000000 --- a/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT child.* -FROM relationships r -JOIN standards_framework_item child - ON child."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasChild' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = '57165a88-d7cc-11e8-824f-0242ac160002' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID'; \ No newline at end of file diff --git a/tutorials/compare_standards/js/.env.example b/tutorials/compare_standards/js/.env.example index f2d3f1f..3c86715 100644 --- a/tutorials/compare_standards/js/.env.example +++ b/tutorials/compare_standards/js/.env.example @@ -1,2 +1,3 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 diff --git a/tutorials/compare_standards/js/README.md b/tutorials/compare_standards/js/README.md index e7a4d11..b6d809e 100644 --- a/tutorials/compare_standards/js/README.md +++ b/tutorials/compare_standards/js/README.md @@ -1,21 +1,21 @@ # Compare Standards -Demonstrates how to compare educational standards across different frameworks (Common Core vs Texas) +Demonstrates how to use crosswalk data to compare standards between the Common Core State Standards (CCSSM) and state frameworks using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query crosswalk data +- **Standards Comparison**: Finding state standards that align with CCSS standards +- **Jaccard Scoring**: Understanding alignment strength using Jaccard scores +- **Learning Components**: Analyzing shared and unique learning components -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/comparing-standards-across-states) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` +- Node.js 18 or higher (for native fetch API support) +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform ## Dependencies -- **arquero**: Data manipulation and analysis library -- **csv-parse**: CSV file parsing - **dotenv**: Environment variable management ## Quick Start @@ -29,7 +29,9 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 ``` 3. **Run Tutorial**: diff --git a/tutorials/compare_standards/js/compare-standards.js b/tutorials/compare_standards/js/compare-standards.js index e18cd57..0101f84 100644 --- a/tutorials/compare_standards/js/compare-standards.js +++ b/tutorials/compare_standards/js/compare-standards.js @@ -3,9 +3,6 @@ ================================ */ // Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); require('dotenv').config(); // Domain Constants @@ -14,198 +11,126 @@ const TARGET_CCSSM_STANDARD_CODE = '6.EE.B.5'; // Common Core 6th grade math st const TARGET_CCSSM_JURISDICTION = 'Multi-State'; // Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); +const apiKey = process.env.API_KEY; +const baseUrl = process.env.BASE_URL; + +if (!apiKey) { + console.error('❌ API_KEY environment variable is not set.'); process.exit(1); } +if (!baseUrl) { + console.error('❌ BASE_URL environment variable is not set.'); + process.exit(1); +} /* ================================ HELPER FUNCTIONS ================================ */ -function loadCSV(filename) { +async function makeApiRequest(endpoint, params = {}) { try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); - } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); - throw error; - } -} + const url = new URL(`${baseUrl}${endpoint}`); + + // Handle array parameters + Object.entries(params).forEach(([key, value]) => { + if (Array.isArray(value)) { + value.forEach(v => url.searchParams.append(key, v)); + } else { + url.searchParams.append(key, value); + } + }); + const response = await fetch(url.toString(), { + method: 'GET', + headers: { + 'x-api-key': apiKey + } + }); -/* ================================ - STEP 1: LOAD THE CROSSWALK DATA - ================================ */ + if (!response.ok) { + throw new Error(`HTTP error! status: ${response.status}`); + } -function loadCrosswalkData(aq) { - /** - * Load crosswalk data from relationships.csv - * - * Purpose: Crosswalk data lives in the relationships.csv file. Standards that have - * crosswalk data include four crosswalk-specific columns: jaccard, stateLCCount, - * ccssLCCount, and sharedLCCount. - * - * Each row shows one state → CCSSM crosswalk relationship. - */ - - // Load CSV files - const relationshipsData = aq.from(loadCSV('Relationships.csv')); - const standardsFrameworkItemsData = aq.from(loadCSV('StandardsFrameworkItem.csv')); - const learningComponentsData = aq.from(loadCSV('LearningComponent.csv')); - - console.log('✅ Data loaded from KG CSV files'); - console.log(` Total Relationships: ${relationshipsData.numRows()}`); - console.log(` Standards Framework Items: ${standardsFrameworkItemsData.numRows()}`); - console.log(` Learning Components: ${learningComponentsData.numRows()}`); - - // Filter for crosswalk relationships (hasStandardAlignment) - const crosswalkData = relationshipsData - .filter(d => d.relationshipType === 'hasStandardAlignment'); - - console.log(`\n✅ Crosswalk data filtered:`); - console.log(` Total crosswalk relationships (state → CCSSM): ${crosswalkData.numRows()}`); - - // Show preview of crosswalk data - if (crosswalkData.numRows() > 0) { - console.log(`\n📊 Preview of crosswalk data (first 3 rows):`); - const preview = crosswalkData - .select('sourceEntityValue', 'targetEntityValue', 'jaccard', - 'stateLCCount', 'ccssLCCount', 'sharedLCCount') - .slice(0, 3) - .objects(); - - preview.forEach((row, idx) => { - console.log(` ${idx + 1}. Source: ${row.sourceEntityValue} → Target: ${row.targetEntityValue}`); - console.log(` Jaccard: ${row.jaccard}, State LCs: ${row.stateLCCount}, CCSS LCs: ${row.ccssLCCount}, Shared: ${row.sharedLCCount}`); - }); + return await response.json(); + } catch (error) { + console.error(`❌ Error making API request to ${endpoint}: ${error.message}`); + throw error; } - - return { - crosswalkData, - standardsFrameworkItemsData, - learningComponentsData, - relationshipsData - }; } - /* ================================ STEP 2: FIND THE BEST-MATCHING STATE STANDARDS ================================ */ -function findBestStateMatches(ccssmStandardCode, jurisdiction, data, aq) { - /** - * Find the best state standard matches for a CCSSM standard - * - * Purpose: To find the best state standard matches for a CCSSM standard, filter rows by the - * CCSSM standard ID and sort by the Jaccard score. This identifies the state - * standards that contain the most similar skills and concept targets for student - * mastery (not necessarily the most similar semantically). - */ - - const { crosswalkData, standardsFrameworkItemsData } = data; - - // First, find the CCSSM standard by its statement code and jurisdiction - const ccssmStandard = standardsFrameworkItemsData - .params({ code: ccssmStandardCode, juris: jurisdiction }) - .filter(d => d.statementCode === code && d.jurisdiction === juris) - .object(); - - if (!ccssmStandard || !ccssmStandard.statementCode) { - console.log(`❌ CCSSM standard not found: ${ccssmStandardCode}`); - return null; - } +async function findBestStateMatches() { + // Find the CCSSM standard by its statement code and jurisdiction + const searchResult = await makeApiRequest('/academic-standards/search', { + statementCode: TARGET_CCSSM_STANDARD_CODE, + jurisdiction: TARGET_CCSSM_JURISDICTION + }); - const ccssmStandardUuid = ccssmStandard.caseIdentifierUUID; // Use 'caseIdentifierUUID' for crosswalk matching + const ccssmStandard = searchResult[0] || null; - console.log(`✅ Found CCSSM standard: ${ccssmStandardCode}`); + if (!ccssmStandard) { + console.log(`❌ CCSSM standard not found: ${TARGET_CCSSM_STANDARD_CODE}`); + return { ccssmStandard: null, texasMatches: null }; + } + + const ccssmStandardUuid = ccssmStandard.caseIdentifierUUID; + console.log(`✅ Found CCSSM standard: ${TARGET_CCSSM_STANDARD_CODE}`); console.log(` Case UUID: ${ccssmStandardUuid}`); console.log(` Description: ${ccssmStandard.description}`); - console.log(` Jurisdiction: ${ccssmStandard.jurisdiction}`); - // Filter crosswalk data for this CCSSM standard (it's the target in relationships) - // and filter for Texas matches only - const matches = crosswalkData - .params({ ccssmId: ccssmStandardUuid }) - .filter(d => d.targetEntityValue === ccssmId); + // Get Texas standards that align with this CCSSM standard + const crosswalkResult = await makeApiRequest( + `/academic-standards/${ccssmStandardUuid}/crosswalks`, + { jurisdiction: 'Texas' } + ); - if (matches.numRows() === 0) { - console.log(`\n❌ No state standard matches found for ${ccssmStandardCode}`); - return null; - } + const texasMatches = crosswalkResult.data; - // Join with standards data to get jurisdiction and filter for Texas - const matchesWithJurisdiction = matches - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'jurisdiction'), - ['sourceEntityValue', 'caseIdentifierUUID'] - ); - - // Filter for Texas only - const texasMatches = matchesWithJurisdiction - .params({ state: 'Texas' }) - .filter(d => d.jurisdiction === state); - - if (texasMatches.numRows() === 0) { - console.log(`\n❌ No Texas standard matches found for ${ccssmStandardCode}`); - return null; + if (!texasMatches || texasMatches.length === 0) { + console.log(`\n❌ No Texas standard matches found for ${TARGET_CCSSM_STANDARD_CODE}`); + return { ccssmStandard, texasMatches: null }; } - // Drop the temporary columns added for filtering to avoid conflicts in later joins - const texasMatchesClean = texasMatches.select(aq.not('caseIdentifierUUID', 'jurisdiction')); - // Sort by Jaccard score (highest first) - const sortedMatches = texasMatchesClean.orderby(aq.desc('jaccard')); + const texasMatchesSorted = texasMatches.sort((a, b) => b.jaccard - a.jaccard); - console.log(`\n✅ Found ${sortedMatches.numRows()} Texas standard matches for ${ccssmStandardCode}`); + console.log(`\n✅ Found ${texasMatchesSorted.length} Texas standard matches`); console.log(`\n📊 Top Texas match (highest Jaccard score):`); - const topMatch = sortedMatches.object(); - console.log(` State Standard UUID: ${topMatch.sourceEntityValue}`); - console.log(` Jaccard Score: ${parseFloat(topMatch.jaccard).toFixed(4)}`); + const topMatch = texasMatchesSorted[0]; + console.log(` Statement Code: ${topMatch.statementCode}`); + console.log(` Jaccard Score: ${topMatch.jaccard.toFixed(4)}`); console.log(` Shared LC Count: ${topMatch.sharedLCCount}`); console.log(` State LC Count: ${topMatch.stateLCCount}`); console.log(` CCSS LC Count: ${topMatch.ccssLCCount}`); - return sortedMatches; + return { ccssmStandard, texasMatches: texasMatchesSorted }; } - /* ================================ STEP 3: INTERPRET THE RELATIONSHIP METRICS ================================ */ function interpretRelationshipMetrics(matches) { - /** - * Interpret the relationship metrics for crosswalk matches - * - * Purpose: Each crosswalk relationship carries additional context about the degree - * of overlap: - * - sharedLCCount shows how many deconstructed skills are shared - * - stateLCCount and ccssLCCount show how many total skills support each standard - * - Together with the Jaccard score, these counts help interpret the strength and - * balance of the overlap - */ - - if (!matches || matches.numRows() === 0) { + if (!matches || matches.length === 0) { return; } console.log(`\n📊 INTERPRETATION OF TOP MATCHES:\n`); // Show top 5 matches with interpretation - const topMatches = matches.slice(0, 5).objects(); - - topMatches.forEach((match, idx) => { - const jaccard = parseFloat(match.jaccard); - const stateLc = parseFloat(match.stateLCCount); - const ccssLc = parseFloat(match.ccssLCCount); - const sharedLc = parseFloat(match.sharedLCCount); + matches.slice(0, 5).forEach((match, idx) => { + const jaccard = match.jaccard; + const stateLc = match.stateLCCount; + const ccssLc = match.ccssLCCount; + const sharedLc = match.sharedLCCount; console.log(`Match #${idx + 1}:`); + console.log(` Statement Code: ${match.statementCode}`); console.log(` Jaccard Score: ${jaccard.toFixed(4)}`); console.log(` State LC Count: ${stateLc}`); console.log(` CCSS LC Count: ${ccssLc}`); @@ -241,222 +166,82 @@ function interpretRelationshipMetrics(matches) { }); } - /* ================================ - STEP 4: JOIN CROSSWALKS WITH STANDARDS METADATA + STEP 4: INSPECT SHARED LEARNING COMPONENTS ================================ */ -function enrichCrosswalksWithMetadata(matches, data, aq) { - /** - * Join crosswalk data with standards metadata - * - * Purpose: Enrich the crosswalk data by joining it with StandardsFrameworkItems.csv, - * which contains metadata such as grade level and description. This provides a clear - * view of which state standards most closely align to their CCSSM counterparts, along - * with the strength of each connection. - */ - - if (!matches || matches.numRows() === 0) { - return null; - } - - const { standardsFrameworkItemsData } = data; - - // Rename columns to avoid conflicts when merging CCSS and state metadata - // We'll merge the same standards dataset twice (once for CCSS, once for state) - - // Join with CCSS standard metadata (target) - const enriched = matches - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction') - .rename({ - caseIdentifierUUID: 'ccss_uuid', - statementCode: 'statementCode_ccss', - description: 'description_ccss', - gradeLevel: 'gradeLevel_ccss', - academicSubject: 'academicSubject_ccss', - jurisdiction: 'jurisdiction_ccss' - }), - ['targetEntityValue', 'ccss_uuid'] - ) - // Join with state standard metadata (source) - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction') - .rename({ - caseIdentifierUUID: 'state_uuid', - statementCode: 'statementCode_state', - description: 'description_state', - gradeLevel: 'gradeLevel_state', - academicSubject: 'academicSubject_state' - }), - ['sourceEntityValue', 'state_uuid'] - ); - - console.log(`\n✅ Enriched crosswalk data with standards metadata\n`); - console.log(`📊 DETAILED COMPARISON (Top 3 matches):\n`); - - const top3 = enriched.slice(0, 3).objects(); - - top3.forEach((row, idx) => { - console.log(`Match #${idx + 1} (Jaccard: ${parseFloat(row.jaccard).toFixed(4)}):`); - console.log(` CCSS STANDARD:`); - console.log(` Code: ${row.statementCode_ccss}`); - console.log(` Jurisdiction: ${row.jurisdiction_ccss}`); - console.log(` Grade Level: ${row.gradeLevel_ccss}`); - console.log(` Description: ${row.description_ccss}`); - console.log(` `); - console.log(` STATE STANDARD:`); - console.log(` Code: ${row.statementCode_state}`); - console.log(` Jurisdiction: ${row.jurisdiction}`); - console.log(` Grade Level: ${row.gradeLevel_state}`); - console.log(` Description: ${row.description_state}`); - console.log(` `); - console.log(` ALIGNMENT METRICS:`); - console.log(` Shared LCs: ${row.sharedLCCount} / State LCs: ${row.stateLCCount} / CCSS LCs: ${row.ccssLCCount}`); - console.log(); - }); - - return enriched; -} - - -/* ================================ - STEP 5: JOIN CROSSWALKS TO LEARNING COMPONENTS - ================================ */ - -function showSharedLearningComponents(stateStandardCode, ccssStandardCode, stateJurisdiction, data, aq) { - /** - * Join crosswalks to Learning Components to show shared skills - * - * Purpose: Now that you have crosswalk pairs (state → CCSSM), you can see the - * actual skills behind each match by joining to the Learning Components dataset. - * We'll use the 'supports' relationships to fetch the LCs that support each standard - * and then intersect them to list the shared LCs (the evidence behind the crosswalk). - */ - - const { standardsFrameworkItemsData, relationshipsData, learningComponentsData } = data; - - // Find the standard identifiers - // Note: For LC relationships, we need to use caseIdentifierUUID, not identifier - const stateStandard = standardsFrameworkItemsData - .params({ code: stateStandardCode, juris: stateJurisdiction }) - .filter(d => d.statementCode === code && d.jurisdiction === juris) - .object(); - - const ccssStandard = standardsFrameworkItemsData - .params({ code: ccssStandardCode }) - .filter(d => d.statementCode === code && d.jurisdiction === 'Multi-State') - .object(); - - if (!stateStandard || !ccssStandard) { - console.log('❌ Could not find one or both standards'); - return; - } +async function showSharedLearningComponents(ccssmStandard, topMatch) { + const ccssmStandardUuid = ccssmStandard.caseIdentifierUUID; + const stateStandardUuid = topMatch.caseIdentifierUUID; - const stateUuid = stateStandard.caseIdentifierUUID; - const ccssUuid = ccssStandard.caseIdentifierUUID; + // Get LCs that support the CCSS standard + const ccssLcResult = await makeApiRequest( + `/academic-standards/${ccssmStandardUuid}/learning-components` + ); + const ccssLcs = ccssLcResult.data; // Get LCs that support the state standard - // LC relationships use caseIdentifierUUID for targetEntityValue - const stateLcIds = relationshipsData - .params({ uuid: stateUuid }) - .filter(d => d.relationshipType === 'supports' && d.targetEntityValue === uuid) - .array('sourceEntityValue'); - - const stateLcs = learningComponentsData - .filter(aq.escape(d => stateLcIds.includes(d.identifier))) - .select('identifier', 'description') - .dedupe('identifier'); - - // Get LCs that support the CCSS standard - const ccssLcIds = relationshipsData - .params({ uuid: ccssUuid }) - .filter(d => d.relationshipType === 'supports' && d.targetEntityValue === uuid) - .array('sourceEntityValue'); - - const ccssLcs = learningComponentsData - .filter(aq.escape(d => ccssLcIds.includes(d.identifier))) - .select('identifier', 'description') - .dedupe('identifier'); - - // Find shared LCs (intersection) using join - const sharedLcs = stateLcs - .semijoin(ccssLcs, 'identifier'); - - // Find state-only LCs (in state but not in CCSS) - const stateOnlyLcs = stateLcs - .antijoin(ccssLcs, 'identifier'); - - // Find CCSS-only LCs (in CCSS but not in state) - const ccssOnlyLcs = ccssLcs - .antijoin(stateLcs, 'identifier'); - - console.log(`\n✅ LEARNING COMPONENTS ANALYSIS:\n`); - console.log(`State Standard: ${stateStandardCode}`); - console.log(`CCSS Standard: ${ccssStandardCode}`); + const stateLcResult = await makeApiRequest( + `/academic-standards/${stateStandardUuid}/learning-components` + ); + const stateLcs = stateLcResult.data; + + // Create sets of LC identifiers for comparison + const ccssLcIds = new Set(ccssLcs.map(lc => lc.identifier)); + const stateLcIds = new Set(stateLcs.map(lc => lc.identifier)); + + // Find shared and unique LCs + const sharedLcIds = new Set([...ccssLcIds].filter(id => stateLcIds.has(id))); + const ccssOnlyIds = new Set([...ccssLcIds].filter(id => !stateLcIds.has(id))); + const stateOnlyIds = new Set([...stateLcIds].filter(id => !ccssLcIds.has(id))); + + // Get LC descriptions + const sharedLcs = ccssLcs.filter(lc => sharedLcIds.has(lc.identifier)); + const ccssOnlyLcs = ccssLcs.filter(lc => ccssOnlyIds.has(lc.identifier)); + const stateOnlyLcs = stateLcs.filter(lc => stateOnlyIds.has(lc.identifier)); + + console.log(`\n✅ LEARNING COMPONENTS ANALYSIS:`); + console.log(`CCSS Standard: ${ccssmStandard.statementCode}`); + console.log(`State Standard: ${topMatch.statementCode}`); console.log(); - console.log(`📊 SHARED LEARNING COMPONENTS (${sharedLcs.numRows()}):`); - console.log('These are the concrete pedagogical overlaps between the two standards:\n'); - sharedLcs.objects().forEach((lc, idx) => { + console.log(`📊 SHARED LEARNING COMPONENTS (${sharedLcs.length}):`); + sharedLcs.forEach((lc, idx) => { console.log(` ✅ ${idx + 1}. ${lc.description}`); }); console.log(); - console.log(`📊 STATE-ONLY LEARNING COMPONENTS (${stateOnlyLcs.numRows()}):`); - stateOnlyLcs.objects().forEach((lc, idx) => { - console.log(` ➖ ${idx + 1}. ${lc.description}`); + console.log(`📊 CCSS-ONLY LEARNING COMPONENTS (${ccssOnlyLcs.length}):`); + ccssOnlyLcs.forEach((lc, idx) => { + console.log(` ➕ ${idx + 1}. ${lc.description}`); }); console.log(); - console.log(`📊 CCSS-ONLY LEARNING COMPONENTS (${ccssOnlyLcs.numRows()}):`); - ccssOnlyLcs.objects().forEach((lc, idx) => { - console.log(` ➕ ${idx + 1}. ${lc.description}`); + console.log(`📊 STATE-ONLY LEARNING COMPONENTS (${stateOnlyLcs.length}):`); + stateOnlyLcs.forEach((lc, idx) => { + console.log(` ➖ ${idx + 1}. ${lc.description}`); }); console.log(); } - /* ================================ MAIN EXECUTION ================================ */ async function main() { - const aq = await import('arquero'); - console.log('\n=== USING CROSSWALKS TO COMPARE STATE STANDARDS TO COMMON CORE ===\n'); - console.log('🔄 Step 1: Load the crosswalk data...'); - const data = loadCrosswalkData(aq); - - console.log('\n' + '='.repeat(70)); - console.log('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...'); - const matches = findBestStateMatches(TARGET_CCSSM_STANDARD_CODE, TARGET_CCSSM_JURISDICTION, data, aq); + console.log('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...\n'); + const { ccssmStandard, texasMatches } = await findBestStateMatches(); - if (matches && matches.numRows() > 0) { + if (texasMatches) { console.log('\n' + '='.repeat(70)); console.log('🔄 Step 3: Interpret the relationship metrics...'); - interpretRelationshipMetrics(matches); + interpretRelationshipMetrics(texasMatches); console.log('='.repeat(70)); - console.log('🔄 Step 4: Join crosswalks with standards metadata...'); - const enriched = enrichCrosswalksWithMetadata(matches, data, aq); - - if (enriched && enriched.numRows() > 0) { - console.log('='.repeat(70)); - console.log('🔄 Step 5: Join crosswalks to Learning Components...'); - // Use the top match for detailed LC analysis (already filtered for Texas) - const topMatch = enriched.object(); - showSharedLearningComponents( - topMatch.statementCode_state, - topMatch.statementCode_ccss, - topMatch.jurisdiction, - data, - aq - ); - } + console.log('🔄 Step 4: Inspect shared learning components...'); + await showSharedLearningComponents(ccssmStandard, texasMatches[0]); } } diff --git a/tutorials/compare_standards/js/package-lock.json b/tutorials/compare_standards/js/package-lock.json index 2ae242c..3d81c01 100644 --- a/tutorials/compare_standards/js/package-lock.json +++ b/tutorials/compare_standards/js/package-lock.json @@ -9,45 +9,9 @@ "version": "1.0.0", "license": "ISC", "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", "dotenv": "^16.3.1" } }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, "node_modules/dotenv": { "version": "16.6.1", "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", diff --git a/tutorials/compare_standards/js/package.json b/tutorials/compare_standards/js/package.json index 64d6b6d..02d654e 100644 --- a/tutorials/compare_standards/js/package.json +++ b/tutorials/compare_standards/js/package.json @@ -1,15 +1,13 @@ { "name": "compare-standards", "version": "1.0.0", - "description": "Compare educational standards across different states and jurisdictions using Knowledge Graph datasets", + "description": "Compare educational standards across different states and jurisdictions using Knowledge Graph REST API", "main": "compare-standards.js", "scripts": { "start": "node compare-standards.js", "test": "echo \"Error: no test specified\" && exit 1" }, "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", "dotenv": "^16.3.1" }, "keywords": [ @@ -17,8 +15,9 @@ "standards", "knowledge-graph", "comparison", - "analysis" + "analysis", + "rest-api" ], "author": "", "license": "ISC" -} \ No newline at end of file +} diff --git a/tutorials/compare_standards/python/.env.example b/tutorials/compare_standards/python/.env.example index f2d3f1f..3c86715 100644 --- a/tutorials/compare_standards/python/.env.example +++ b/tutorials/compare_standards/python/.env.example @@ -1,2 +1,3 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 diff --git a/tutorials/compare_standards/python/README.md b/tutorials/compare_standards/python/README.md index 99e5dba..5b0983a 100644 --- a/tutorials/compare_standards/python/README.md +++ b/tutorials/compare_standards/python/README.md @@ -1,20 +1,22 @@ # Compare Standards -Demonstrates how to use crosswalk data to compare state standards to Common Core State Standards (CCSSM) using the Knowledge Graph dataset. +Demonstrates how to use crosswalk data to compare standards between the Common Core State Standards (CCSSM) and state frameworks using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query crosswalk data +- **Standards Comparison**: Finding state standards that align with CCSS standards +- **Jaccard Scoring**: Understanding alignment strength using Jaccard scores +- **Learning Components**: Analyzing shared and unique learning components -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-2-0/getting-started/tutorials/comparing-standards-across-states) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites - Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform ## Dependencies -- **pandas**: Data manipulation and analysis library +- **requests**: HTTP library for API calls - **python-dotenv**: Environment variable management ## Quick Start @@ -30,7 +32,9 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 ``` 3. **Run Tutorial**: diff --git a/tutorials/compare_standards/python/compare_standards.py b/tutorials/compare_standards/python/compare_standards.py index 33a6020..10c1ab0 100644 --- a/tutorials/compare_standards/python/compare_standards.py +++ b/tutorials/compare_standards/python/compare_standards.py @@ -3,8 +3,8 @@ Using crosswalks to compare state standards to Common Core This tutorial demonstrates how to use the crosswalk data in Knowledge Graph to compare -standards between a state framework and the Common Core State Standards (CCSSM). These -crosswalks help determine which CCSSM standard is most similar to a given state standard +standards between the Common Core State Standards (CCSSM) and state frameworks. These +crosswalks help determine which state standards are most similar to a given CCSSM standard and understand the similarities and differences between them. Crosswalks are evidence-based relationships between state standards and CCSSM standards, @@ -22,8 +22,7 @@ # Dependencies import os import sys -import pandas as pd -from pathlib import Path +import requests from dotenv import load_dotenv # Load environment variables @@ -35,12 +34,19 @@ TARGET_CCSSM_JURISDICTION = 'Multi-State' # Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') +api_key = os.getenv('API_KEY') +base_url = os.getenv('BASE_URL') + +if not api_key: + print('❌ API_KEY environment variable is not set.') + sys.exit(1) + +if not base_url: + print('❌ BASE_URL environment variable is not set.') sys.exit(1) -data_path = Path(data_dir) +# Setup headers for API requests +headers = {"x-api-key": api_key} """ @@ -49,164 +55,77 @@ ================================ """ -def load_csv(filename): - """ - Load and parse CSV file from data directory - - Args: - filename (str): Name of the CSV file to load - - Returns: - pd.DataFrame: Loaded CSV data as DataFrame - """ +def make_api_request(endpoint, params=None): + """Make API request to Knowledge Graph API""" try: - file_path = data_path / filename - return pd.read_csv(file_path, low_memory=False) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') + url = f"{base_url}{endpoint}" + response = requests.get(url, headers=headers, params=params) + response.raise_for_status() + return response.json() + except requests.exceptions.RequestException as error: + print(f'❌ Error making API request to {endpoint}: {str(error)}') raise error -""" -================================ -STEP 1: LOAD THE CROSSWALK DATA -================================ -""" - -def load_crosswalk_data(): - """ - Load crosswalk data from relationships.csv - - Purpose: Crosswalk data lives in the relationships.csv file. Standards that have - crosswalk data include four crosswalk-specific columns: jaccard, stateLCCount, - ccssLCCount, and sharedLCCount. - - Each row shows one state → CCSSM crosswalk relationship. - - Returns: - dict: Dictionary containing crosswalk data and related datasets - """ - # Load CSV files - relationships_data = load_csv('Relationships.csv') - standards_framework_items_data = load_csv('StandardsFrameworkItem.csv') - learning_components_data = load_csv('LearningComponent.csv') - - print('✅ Data loaded from KG CSV files') - print(f' Total Relationships: {len(relationships_data)}') - print(f' Standards Framework Items: {len(standards_framework_items_data)}') - print(f' Learning Components: {len(learning_components_data)}') - - # Filter for crosswalk relationships (hasStandardAlignment) - crosswalk_data = relationships_data[ - relationships_data['relationshipType'] == 'hasStandardAlignment' - ].copy() - - print(f'\n✅ Crosswalk data filtered:') - print(f' Total crosswalk relationships (state → CCSSM): {len(crosswalk_data)}') - - # Show preview of crosswalk data - if len(crosswalk_data) > 0: - print(f'\n📊 Preview of crosswalk data (first 3 rows):') - preview_cols = ['sourceEntityValue', 'targetEntityValue', 'jaccard', - 'stateLCCount', 'ccssLCCount', 'sharedLCCount'] - available_cols = [col for col in preview_cols if col in crosswalk_data.columns] - print(crosswalk_data[available_cols].head(3).to_string(index=False)) - - return { - 'crosswalk_data': crosswalk_data, - 'standards_framework_items_data': standards_framework_items_data, - 'learning_components_data': learning_components_data, - 'relationships_data': relationships_data - } - - """ ================================ STEP 2: FIND THE BEST-MATCHING STATE STANDARDS ================================ """ -def find_best_state_matches(ccssm_standard_code, jurisdiction, data): +def find_best_state_matches(): """ Find the best state standard matches for a CCSSM standard - Purpose: To find the best state standard matches for a CCSSM standard, filter rows by the - CCSSM standard ID and sort by the Jaccard score. This identifies the state - standards that contain the most similar skills and concept targets for student - mastery (not necessarily the most similar semantically). - - Args: - ccssm_standard_code (str): The statement code of the CCSSM standard - jurisdiction (str): The jurisdiction of the CCSSM standard (typically 'Multi-State') - data (dict): Dictionary containing the loaded datasets - Returns: - pd.DataFrame: Crosswalk matches sorted by Jaccard score (highest first) + tuple: (ccssm_standard, texas_matches) """ - crosswalk_data = data['crosswalk_data'] - standards_data = data['standards_framework_items_data'] - - # First, find the CCSSM standard by its statement code and jurisdiction - ccssm_standard = standards_data[ - (standards_data['statementCode'] == ccssm_standard_code) & - (standards_data['jurisdiction'] == jurisdiction) - ] + # Find the CCSSM standard by its statement code and jurisdiction + search_result = make_api_request( + '/academic-standards/search', + params={ + 'statementCode': TARGET_CCSSM_STANDARD_CODE, + 'jurisdiction': TARGET_CCSSM_JURISDICTION + } + ) - if len(ccssm_standard) == 0: - print(f'❌ CCSSM standard not found: {ccssm_standard_code}') - return None + ccssm_standard = search_result[0] if search_result else None - ccssm_standard = ccssm_standard.iloc[0] - ccssm_standard_uuid = ccssm_standard['caseIdentifierUUID'] # Use 'caseIdentifierUUID' for crosswalk matching + if not ccssm_standard: + print(f'❌ CCSSM standard not found: {TARGET_CCSSM_STANDARD_CODE}') + return None, None - print(f'✅ Found CCSSM standard: {ccssm_standard_code}') + ccssm_standard_uuid = ccssm_standard['caseIdentifierUUID'] + print(f'✅ Found CCSSM standard: {TARGET_CCSSM_STANDARD_CODE}') print(f' Case UUID: {ccssm_standard_uuid}') print(f' Description: {ccssm_standard["description"]}') - print(f' Jurisdiction: {ccssm_standard["jurisdiction"]}') - - # Filter crosswalk data for this CCSSM standard (it's the target in relationships) - # and filter for Texas matches only - matches = crosswalk_data[ - crosswalk_data['targetEntityValue'] == ccssm_standard_uuid - ].copy() - - if len(matches) == 0: - print(f'\n❌ No state standard matches found for {ccssm_standard_code}') - return None - - # Join with standards data to get jurisdiction and filter for Texas - matches = matches.merge( - standards_data[['caseIdentifierUUID', 'jurisdiction']], - left_on='sourceEntityValue', - right_on='caseIdentifierUUID', - how='left', - suffixes=('', '_temp') - ) - # Filter for Texas only - texas_matches = matches[matches['jurisdiction'] == 'Texas'].copy() + # Get Texas standards that align with this CCSSM standard + crosswalk_result = make_api_request( + f'/academic-standards/{ccssm_standard_uuid}/crosswalks', + params={'jurisdiction': 'Texas'} + ) - if len(texas_matches) == 0: - print(f'\n❌ No Texas standard matches found for {ccssm_standard_code}') - return None + texas_matches = crosswalk_result['data'] - # Drop the temporary columns added for filtering - texas_matches = texas_matches.drop(columns=['caseIdentifierUUID', 'jurisdiction']) + if not texas_matches: + print(f'\n❌ No Texas standard matches found for {TARGET_CCSSM_STANDARD_CODE}') + return ccssm_standard, None # Sort by Jaccard score (highest first) - texas_matches = texas_matches.sort_values('jaccard', ascending=False) + texas_matches_sorted = sorted(texas_matches, key=lambda x: x['jaccard'], reverse=True) - print(f'\n✅ Found {len(texas_matches)} Texas standard matches for {ccssm_standard_code}') + print(f'\n✅ Found {len(texas_matches_sorted)} Texas standard matches') print(f'\n📊 Top Texas match (highest Jaccard score):') - top_match = texas_matches.iloc[0] - print(f' State Standard UUID: {top_match["sourceEntityValue"]}') + top_match = texas_matches_sorted[0] + print(f' Statement Code: {top_match["statementCode"]}') print(f' Jaccard Score: {top_match["jaccard"]:.4f}') print(f' Shared LC Count: {top_match["sharedLCCount"]}') print(f' State LC Count: {top_match["stateLCCount"]}') print(f' CCSS LC Count: {top_match["ccssLCCount"]}') - return texas_matches + return ccssm_standard, texas_matches_sorted """ @@ -227,21 +146,22 @@ def interpret_relationship_metrics(matches): balance of the overlap Args: - matches (pd.DataFrame): Crosswalk matches from Step 2 + matches (list): Crosswalk matches from Step 2 """ - if matches is None or len(matches) == 0: + if not matches: return print(f'\n📊 INTERPRETATION OF TOP MATCHES:\n') # Show top 5 matches with interpretation - for idx, (_, match) in enumerate(matches.head(5).iterrows(), 1): + for idx, match in enumerate(matches[:5], 1): jaccard = match['jaccard'] state_lc = match['stateLCCount'] ccss_lc = match['ccssLCCount'] shared_lc = match['sharedLCCount'] print(f'Match #{idx}:') + print(f' Statement Code: {match["statementCode"]}') print(f' Jaccard Score: {jaccard:.4f}') print(f' State LC Count: {state_lc}') print(f' CCSS LC Count: {ccss_lc}') @@ -274,188 +194,72 @@ def interpret_relationship_metrics(matches): """ ================================ -STEP 4: JOIN CROSSWALKS WITH STANDARDS METADATA +STEP 4: INSPECT SHARED LEARNING COMPONENTS ================================ """ -def enrich_crosswalks_with_metadata(matches, data): +def show_shared_learning_components(ccssm_standard, top_match): """ - Join crosswalk data with standards metadata + Show shared learning components between CCSS and state standards - Purpose: Enrich the crosswalk data by joining it with StandardsFrameworkItems.csv, - which contains metadata such as grade level and description. This provides a clear - view of which state standards most closely align to their CCSSM counterparts, along - with the strength of each connection. + Purpose: Now that you have crosswalk pairs (CCSSM → state), you can see the + actual skills behind each match by retrieving the Learning Components that + support each standard. We'll then identify which LCs are shared (the evidence + behind the crosswalk) and which are unique to each standard. Args: - matches (pd.DataFrame): Crosswalk matches from Step 2 - data (dict): Dictionary containing the loaded datasets - - Returns: - pd.DataFrame: Enriched crosswalk data with metadata + ccssm_standard (dict): CCSSM standard from Step 2 + top_match (dict): Top Texas match from Step 2 """ - if matches is None or len(matches) == 0: - return None - - standards_data = data['standards_framework_items_data'] - - # Rename columns to avoid conflicts when merging CCSS and state metadata - # We'll merge the same standards dataset twice (once for CCSS, once for state) - - # Join with CCSS standard metadata (target) - ccss_standards = standards_data[['caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction']].copy() - ccss_standards.columns = ['ccss_uuid', 'statementCode_ccss', 'description_ccss', - 'gradeLevel_ccss', 'academicSubject_ccss', 'jurisdiction_ccss'] + ccssm_standard_uuid = ccssm_standard['caseIdentifierUUID'] + state_standard_uuid = top_match['caseIdentifierUUID'] - enriched = matches.merge( - ccss_standards, - left_on='targetEntityValue', - right_on='ccss_uuid', - how='left' - ) - - # Join with state standard metadata (source) - state_standards = standards_data[['caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction']].copy() - state_standards.columns = ['state_uuid', 'statementCode_state', 'description_state', - 'gradeLevel_state', 'academicSubject_state', 'jurisdiction'] - - enriched = enriched.merge( - state_standards, - left_on='sourceEntityValue', - right_on='state_uuid', - how='left' + # Get LCs that support the CCSS standard + ccss_lc_result = make_api_request( + f'/academic-standards/{ccssm_standard_uuid}/learning-components' ) - - print(f'\n✅ Enriched crosswalk data with standards metadata\n') - print(f'📊 DETAILED COMPARISON (Top 3 matches):\n') - - for idx, (_, row) in enumerate(enriched.head(3).iterrows(), 1): - print(f'Match #{idx} (Jaccard: {row["jaccard"]:.4f}):') - print(f' CCSS STANDARD:') - print(f' Code: {row["statementCode_ccss"]}') - print(f' Jurisdiction: {row["jurisdiction_ccss"]}') - print(f' Grade Level: {row["gradeLevel_ccss"]}') - print(f' Description: {row["description_ccss"]}') - print(f' ') - print(f' STATE STANDARD:') - print(f' Code: {row["statementCode_state"]}') - print(f' Jurisdiction: {row["jurisdiction"]}') - print(f' Grade Level: {row["gradeLevel_state"]}') - print(f' Description: {row["description_state"]}') - print(f' ') - print(f' ALIGNMENT METRICS:') - print(f' Shared LCs: {row["sharedLCCount"]} / State LCs: {row["stateLCCount"]} / CCSS LCs: {row["ccssLCCount"]}') - print() - - return enriched - - -""" -================================ -STEP 5: JOIN CROSSWALKS TO LEARNING COMPONENTS -================================ -""" - -def show_shared_learning_components(state_standard_code, ccss_standard_code, state_jurisdiction, data): - """ - Join crosswalks to Learning Components to show shared skills - - Purpose: Now that you have crosswalk pairs (state → CCSSM), you can see the - actual skills behind each match by joining to the Learning Components dataset. - We'll use the 'supports' relationships to fetch the LCs that support each standard - and then intersect them to list the shared LCs (the evidence behind the crosswalk). - - Args: - state_standard_code (str): State standard code - ccss_standard_code (str): CCSS standard code - state_jurisdiction (str): State jurisdiction (to ensure correct standard match) - data (dict): Dictionary containing the loaded datasets - """ - standards_data = data['standards_framework_items_data'] - relationships_data = data['relationships_data'] - learning_components_data = data['learning_components_data'] - - # Find the standard identifiers - # Note: For LC relationships, we need to use caseIdentifierUUID, not identifier - state_standard = standards_data[ - (standards_data['statementCode'] == state_standard_code) & - (standards_data['jurisdiction'] == state_jurisdiction) - ] - ccss_standard = standards_data[ - (standards_data['statementCode'] == ccss_standard_code) & - (standards_data['jurisdiction'] == 'Multi-State') - ] - - if len(state_standard) == 0 or len(ccss_standard) == 0: - print('❌ Could not find one or both standards') - return - - state_uuid = state_standard.iloc[0]['caseIdentifierUUID'] - ccss_uuid = ccss_standard.iloc[0]['caseIdentifierUUID'] + ccss_lcs = ccss_lc_result['data'] # Get LCs that support the state standard - # LC relationships use caseIdentifierUUID for targetEntityValue - state_lc_relationships = relationships_data[ - (relationships_data['relationshipType'] == 'supports') & - (relationships_data['targetEntityValue'] == state_uuid) - ] - - state_lc_ids = state_lc_relationships['sourceEntityValue'].unique() - state_lcs = learning_components_data[ - learning_components_data['identifier'].isin(state_lc_ids) - ][['identifier', 'description']].drop_duplicates() - - # Get LCs that support the CCSS standard - ccss_lc_relationships = relationships_data[ - (relationships_data['relationshipType'] == 'supports') & - (relationships_data['targetEntityValue'] == ccss_uuid) - ] - - ccss_lc_ids = ccss_lc_relationships['sourceEntityValue'].unique() - ccss_lcs = learning_components_data[ - learning_components_data['identifier'].isin(ccss_lc_ids) - ][['identifier', 'description']].drop_duplicates() - - # Find shared LCs (intersection) using merge - shared_lcs = state_lcs.merge( - ccss_lcs[['identifier']], - on='identifier', - how='inner' + state_lc_result = make_api_request( + f'/academic-standards/{state_standard_uuid}/learning-components' ) + state_lcs = state_lc_result['data'] - # Find state-only LCs (in state but not in CCSS) - state_only_lcs = state_lcs[ - ~state_lcs['identifier'].isin(ccss_lcs['identifier']) - ] + # Create sets of LC identifiers for comparison + ccss_lc_ids = {lc['identifier'] for lc in ccss_lcs} + state_lc_ids = {lc['identifier'] for lc in state_lcs} - # Find CCSS-only LCs (in CCSS but not in state) - ccss_only_lcs = ccss_lcs[ - ~ccss_lcs['identifier'].isin(state_lcs['identifier']) - ] + # Find shared and unique LCs + shared_lc_ids = ccss_lc_ids & state_lc_ids + ccss_only_ids = ccss_lc_ids - state_lc_ids + state_only_ids = state_lc_ids - ccss_lc_ids - print(f'\n✅ LEARNING COMPONENTS ANALYSIS:\n') - print(f'CCSS Standard: {ccss_standard_code}') - print(f'State Standard: {state_standard_code}') + # Get LC descriptions + shared_lcs = [lc for lc in ccss_lcs if lc['identifier'] in shared_lc_ids] + ccss_only_lcs = [lc for lc in ccss_lcs if lc['identifier'] in ccss_only_ids] + state_only_lcs = [lc for lc in state_lcs if lc['identifier'] in state_only_ids] + + print(f'\n✅ LEARNING COMPONENTS ANALYSIS:') + print(f'CCSS Standard: {ccssm_standard["statementCode"]}') + print(f'State Standard: {top_match["statementCode"]}') print() print(f'📊 SHARED LEARNING COMPONENTS ({len(shared_lcs)}):') - print('These are the concrete pedagogical overlaps between the two standards:\n') - for idx, (_, lc) in enumerate(shared_lcs.iterrows(), 1): + for idx, lc in enumerate(shared_lcs, 1): print(f' ✅ {idx}. {lc["description"]}') print() - print(f'📊 STATE-ONLY LEARNING COMPONENTS ({len(state_only_lcs)}):') - for idx, (_, lc) in enumerate(state_only_lcs.iterrows(), 1): - print(f' ➖ {idx}. {lc["description"]}') - print() - print(f'📊 CCSS-ONLY LEARNING COMPONENTS ({len(ccss_only_lcs)}):') - for idx, (_, lc) in enumerate(ccss_only_lcs.iterrows(), 1): + for idx, lc in enumerate(ccss_only_lcs, 1): print(f' ➕ {idx}. {lc["description"]}') print() + print(f'📊 STATE-ONLY LEARNING COMPONENTS ({len(state_only_lcs)}):') + for idx, lc in enumerate(state_only_lcs, 1): + print(f' ➖ {idx}. {lc["description"]}') + print() + """ ================================ @@ -469,33 +273,17 @@ def main(): """ print('\n=== USING CROSSWALKS TO COMPARE STATE STANDARDS TO COMMON CORE ===\n') - print('🔄 Step 1: Load the crosswalk data...') - data = load_crosswalk_data() - - print('\n' + '='*70) - print('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...') - matches = find_best_state_matches(TARGET_CCSSM_STANDARD_CODE, TARGET_CCSSM_JURISDICTION, data) + print('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...\n') + ccssm_standard, matches = find_best_state_matches() - if matches is not None and len(matches) > 0: + if matches: print('\n' + '='*70) print('🔄 Step 3: Interpret the relationship metrics...') interpret_relationship_metrics(matches) print('='*70) - print('🔄 Step 4: Join crosswalks with standards metadata...') - enriched = enrich_crosswalks_with_metadata(matches, data) - - if enriched is not None and len(enriched) > 0: - print('='*70) - print('🔄 Step 5: Join crosswalks to Learning Components...') - # Use the top match for detailed LC analysis (already filtered for Texas) - top_match = enriched.iloc[0] - show_shared_learning_components( - top_match['statementCode_state'], - top_match['statementCode_ccss'], - top_match['jurisdiction'], - data - ) + print('🔄 Step 4: Inspect shared learning components...') + show_shared_learning_components(ccssm_standard, matches[0]) if __name__ == '__main__': diff --git a/tutorials/compare_standards/python/requirements.txt b/tutorials/compare_standards/python/requirements.txt index 36953cb..fdbb17d 100644 --- a/tutorials/compare_standards/python/requirements.txt +++ b/tutorials/compare_standards/python/requirements.txt @@ -1,9 +1,5 @@ -# Core data manipulation and analysis -pandas>=2.0.0 +# HTTP requests +requests>=2.31.0 # Environment variable management python-dotenv>=1.0.0 - -# Optional: Enhanced data processing (uncomment if needed) -# numpy>=1.24.0 -# scipy>=1.10.0 \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/js/.env.example b/tutorials/generate_prereq_practice/js/.env.example index 0483037..533ba61 100644 --- a/tutorials/generate_prereq_practice/js/.env.example +++ b/tutorials/generate_prereq_practice/js/.env.example @@ -1,5 +1,6 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 # OpenAI API key for generating practice questions -OPENAI_API_KEY=your_openai_api_key_here \ No newline at end of file +OPENAI_API_KEY=your_openai_api_key_here diff --git a/tutorials/generate_prereq_practice/js/README.md b/tutorials/generate_prereq_practice/js/README.md index 40bd1ef..c61fb54 100644 --- a/tutorials/generate_prereq_practice/js/README.md +++ b/tutorials/generate_prereq_practice/js/README.md @@ -1,29 +1,24 @@ # Generate Prerequisite Practice -Demonstrates how to generate prerequisite-based practice questions using Knowledge Graph data covering: -- **Data Loading**: Reading CSV files with educational frameworks and relationships +Demonstrates how to generate prerequisite-based practice questions using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query standards and learning progressions - **Prerequisite Analysis**: Finding standards that build towards a target standard -- **Relationship Mapping**: Connecting learning components to prerequisite standards -- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge +- **Learning Components**: Discovering granular learning components that support standards +- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge using AI -**Features**: Common Core Math analysis, prerequisite relationship traversal, learning component mapping - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/generating-prerequisite-practice-questions) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` +- Node.js 18 or higher (for native fetch API support) +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform +- OpenAI API key ## Dependencies - **openai**: OpenAI API for generating practice questions -- **csv-parse**: CSV file parsing - **dotenv**: Environment variable management -- **arquero**: Data manipulation ## Quick Start @@ -36,7 +31,11 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 + + # OpenAI API key for generating practice questions OPENAI_API_KEY=your_openai_api_key_here ``` diff --git a/tutorials/generate_prereq_practice/js/generate-prereq-practice.js b/tutorials/generate_prereq_practice/js/generate-prereq-practice.js index 45f5a2f..575dbe6 100644 --- a/tutorials/generate_prereq_practice/js/generate-prereq-practice.js +++ b/tutorials/generate_prereq_practice/js/generate-prereq-practice.js @@ -3,26 +3,27 @@ ================================ */ // Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); const OpenAI = require('openai'); require('dotenv').config(); // Constants const GENERATE_PRACTICE = true; -// Filter criteria for mathematics standards -const JURISDICTION = 'Multi-State'; -const ACADEMIC_SUBJECT = 'Mathematics'; const TARGET_CODE = '6.NS.B.4'; // OpenAI configuration const OPENAI_MODEL = 'gpt-4'; const OPENAI_TEMPERATURE = 0.7; // Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); +const apiKey = process.env.API_KEY; +const baseUrl = process.env.BASE_URL; + +if (!apiKey) { + console.error('❌ API_KEY environment variable is not set.'); + process.exit(1); +} + +if (!baseUrl) { + console.error('❌ BASE_URL environment variable is not set.'); process.exit(1); } @@ -34,195 +35,116 @@ const openai = new OpenAI({ HELPER FUNCTIONS ================================ */ -function loadCSV(filename) { +async function makeApiRequest(endpoint, params = {}) { try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); + const url = new URL(`${baseUrl}${endpoint}`); + + // Handle array parameters (like gradeLevel) + Object.entries(params).forEach(([key, value]) => { + if (Array.isArray(value)) { + value.forEach(v => url.searchParams.append(key, v)); + } else { + url.searchParams.append(key, value); + } + }); + + const response = await fetch(url.toString(), { + method: 'GET', + headers: { + 'x-api-key': apiKey + } + }); + + if (!response.ok) { + throw new Error(`HTTP error! status: ${response.status}`); + } + + return await response.json(); } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); + console.error(`❌ Error making API request to ${endpoint}: ${error.message}`); throw error; } } - /* ================================ - STEP 1: LOAD DATA + STEP 2: GET PREREQUISITE STANDARDS ================================ */ -function loadData(aq) { - /* Load CSV data files and build filtered dataset - */ - - const standardsFrameworkItems = aq.from(loadCSV('StandardsFrameworkItem.csv')); - const learningComponents = aq.from(loadCSV('LearningComponent.csv')); - const relationships = aq.from(loadCSV('Relationships.csv')); - - console.log('✅ Files loaded from KG CSV files'); - console.log({ - standardsFrameworkItems: standardsFrameworkItems.numRows(), - learningComponents: learningComponents.numRows(), - relationships: relationships.numRows() - }); - // Filter for relevant StandardsFrameworkItems by jurisdiction and subject - const relevantStandards = standardsFrameworkItems - .params({ jurisdiction: JURISDICTION, academicSubject: ACADEMIC_SUBJECT }) - .filter(d => d.jurisdiction === jurisdiction && d.academicSubject === academicSubject); - - // Get array of relevant identifiers for filtering - const relevantStandardIds = relevantStandards.array('caseIdentifierUUID'); - const relevantStandardSet = new Set(relevantStandardIds); - - // Filter relationships for buildsTowards and supports relationships - const relevantRelationships = relationships - .filter(aq.escape(d => - (d.relationshipType === 'buildsTowards' && - relevantStandardSet.has(d.sourceEntityValue) && - relevantStandardSet.has(d.targetEntityValue)) || - (d.relationshipType === 'supports' && - relevantStandardSet.has(d.targetEntityValue)) - )); - - // Get learning component IDs from supports relationships - const supportRelationships = relevantRelationships - .filter(d => d.relationshipType === 'supports'); - const linkedLearningComponentIds = supportRelationships.array('sourceEntityValue'); - const linkedLearningComponentSet = new Set(linkedLearningComponentIds); - - // Filter learning components by identifier - const relevantLearningComponents = learningComponents - .filter(aq.escape(d => linkedLearningComponentSet.has(d.identifier))); - - console.log('✅ Retrieved scoped graph:'); - console.log({ - standardsFrameworkItems: relevantStandards.numRows(), - learningComponents: relevantLearningComponents.numRows(), - relationships: relevantRelationships.numRows() +async function getStandardAndPrerequisites() { + // Find the target standard by statement code + const searchResult = await makeApiRequest('/academic-standards/search', { + statementCode: TARGET_CODE, + jurisdiction: 'Multi-State' }); - return { - relevantStandards, - relevantRelationships, - relevantLearningComponents - }; -} - -/* ================================ - STEP 2: QUERY PREREQUISITE DATA - ================================ */ -function getStandardAndPrerequisites(relevantStandards, relevantRelationships) { - const targetStandardTable = relevantStandards - .params({ targetCode: TARGET_CODE }) - .filter(d => d.statementCode === targetCode); + const targetStandard = searchResult[0] || null; - if (targetStandardTable.numRows() === 0) { - console.error(`❌ No StandardsFrameworkItem found for statementCode = "${TARGET_CODE}"`); + if (!targetStandard) { + console.error(`❌ No standard found for ${TARGET_CODE}`); return null; } - const targetStandard = targetStandardTable.object(); - console.log(`✅ Found StandardsFrameworkItem for ${TARGET_CODE}:`) - console.log({ - caseIdentifierUUID: targetStandard.caseIdentifierUUID, - statementCode: targetStandard.statementCode, - description: targetStandard.description + console.log(`✅ Found standard ${TARGET_CODE}:`); + console.log(` UUID: ${targetStandard.caseIdentifierUUID}`); + console.log(` Description: ${targetStandard.description}`); + + // Get prerequisites + const prereqResult = await makeApiRequest( + `/academic-standards/${targetStandard.caseIdentifierUUID}/prerequisites` + ); + + const prerequisiteStandards = prereqResult.data; + console.log(`✅ Found ${prerequisiteStandards.length} prerequisite(s):`); + prerequisiteStandards.forEach(prereq => { + const description = prereq.description || 'No description'; + const truncated = description.length > 80 ? description.substring(0, 80) + '...' : description; + console.log(` ${prereq.statementCode}: ${truncated}`); }); - const prerequisiteLinks = relevantRelationships - .params({ targetIdentifier: targetStandard.caseIdentifierUUID }) - .filter(d => d.relationshipType === 'buildsTowards' && - d.targetEntityValue === targetIdentifier); - - const prerequisiteStandards = prerequisiteLinks - .join(relevantStandards, ['sourceEntityValue', 'caseIdentifierUUID']) - .select('sourceEntityValue', 'statementCode', 'description_2') - .rename({ sourceEntityValue: 'caseIdentifierUUID', description_2: 'standardDescription' }); - - console.log(`✅ Found ${prerequisiteStandards.numRows()} prerequisite(s) for ${targetStandard.statementCode}:`); - console.log(prerequisiteStandards.objects()); - return { targetStandard, prerequisiteStandards }; } -function getLearningComponentsForPrerequisites(prerequisiteStandards, relevantRelationships, relevantLearningComponents) { - const prerequisiteLearningComponents = prerequisiteStandards - .join(relevantRelationships, ['caseIdentifierUUID', 'targetEntityValue']) - .params({ supportsType: 'supports' }) - .filter(d => d.relationshipType === supportsType) - .join(relevantLearningComponents, ['sourceEntityValue', 'identifier']) - .select('caseIdentifierUUID', 'statementCode', 'standardDescription', 'description_2') - .rename({ description_2: 'learningComponentDescription' }); +async function getLearningComponentsForPrerequisites(prerequisiteStandards) { + const prerequisiteLearningComponents = []; - console.log(`✅ Found ${prerequisiteLearningComponents.numRows()} supporting learning components for prerequisites:`); - console.log(prerequisiteLearningComponents.objects()); + for (const prereq of prerequisiteStandards) { + const lcResult = await makeApiRequest( + `/academic-standards/${prereq.caseIdentifierUUID}/learning-components` + ); - return prerequisiteLearningComponents; -} - -function queryPrerequisiteData(aq, relevantStandards, relevantRelationships, relevantLearningComponents) { - /* Analyze prerequisite relationships for the target standard - * This step identifies prerequisites and supporting learning components - * - * SQL: WITH target AS ( - * SELECT "caseIdentifierUUID" - * FROM standards_framework_item - * WHERE "statementCode" = '6.NS.B.4' - * ), - * prerequisite_standards AS ( - * SELECT - * sfi."caseIdentifierUUID", - * sfi."statementCode", - * sfi."description" - * FROM standards_framework_item sfi - * JOIN relationships r - * ON sfi."caseIdentifierUUID" = r."sourceEntityValue" - * JOIN target t - * ON r."targetEntityValue" = t."caseIdentifierUUID" - * WHERE r."relationshipType" = 'buildsTowards' - * ) - * SELECT - * ps."caseIdentifierUUID", - * ps."statementCode", - * ps."description", - * lc."description" - * FROM prerequisite_standards ps - * JOIN relationships r - * ON ps."caseIdentifierUUID" = r."targetEntityValue" - * JOIN learning_component lc - * ON r."sourceEntityValue" = lc."identifier" - * WHERE r."relationshipType" = 'supports'; - * - * Cypher: MATCH (target:StandardsFrameworkItem {statementCode: '6.NS.B.4'}) - * MATCH (prereq:StandardsFrameworkItem)-[:buildsTowards]->(target) - * MATCH (lc:LearningComponent)-[:supports]->(prereq) - * RETURN prereq.caseIdentifierUUID, prereq.statementCode, prereq.description, - * lc.description - */ - - const standardAndPrereqData = getStandardAndPrerequisites(relevantStandards, relevantRelationships); - if (!standardAndPrereqData) { - return null; + for (const lc of lcResult.data) { + prerequisiteLearningComponents.push({ + caseIdentifierUUID: prereq.caseIdentifierUUID, + statementCode: prereq.statementCode, + standardDescription: prereq.description, + learningComponentDescription: lc.description + }); + } } - const { targetStandard, prerequisiteStandards } = standardAndPrereqData; - const prerequisiteLearningComponents = getLearningComponentsForPrerequisites(prerequisiteStandards, relevantRelationships, relevantLearningComponents); + console.log(`✅ Found ${prerequisiteLearningComponents.length} supporting learning components for prerequisites:`); + prerequisiteLearningComponents.slice(0, 5).forEach(lc => { + const description = lc.learningComponentDescription || 'No description'; + const truncated = description.length > 80 ? description.substring(0, 80) + '...' : description; + console.log(` ${truncated}`); + }); - return { targetStandard, prerequisiteLearningComponents }; + return prerequisiteLearningComponents; } /* ================================ STEP 3: GENERATE PRACTICE ================================ */ + function packageContextData(targetStandard, prerequisiteLearningComponents) { /* Package the standards and learning components data for text generation * This creates a structured context that can be used for generating practice questions */ - // Convert dataframe to context format for LLM - const allRows = prerequisiteLearningComponents.objects(); const standardsMap = new Map(); // Group learning components by standard for context - for (const row of allRows) { + for (const row of prerequisiteLearningComponents) { if (!standardsMap.has(row.caseIdentifierUUID)) { standardsMap.set(row.caseIdentifierUUID, { statementCode: row.statementCode, @@ -244,30 +166,25 @@ function packageContextData(targetStandard, prerequisiteLearningComponents) { prereqStandards: Array.from(standardsMap.values()) }; - console.log(`✅ Packaged full standards context for text generation`); - console.log(JSON.stringify(fullStandardsContext, null, 2)); + console.log('✅ Packaged full standards context for text generation'); return fullStandardsContext; } -function generatePracticeData(fullStandardsContext) { - /* Generate practice questions using OpenAI API - * This creates educational content based on prerequisite data - */ - return async function generatePractice() { - console.log(`🔄 Generating practice questions for ${fullStandardsContext.targetStandard.statementCode}...`); - - try { - // Build prompt inline - let prerequisiteText = ''; - for (const prereq of fullStandardsContext.prereqStandards) { - prerequisiteText += `- ${prereq.statementCode}: ${prereq.description}\n`; - prerequisiteText += ' Supporting Learning Components:\n'; - for (const lc of prereq.supportingLearningComponents) { - prerequisiteText += ` • ${lc.description}\n`; - } +async function generatePractice(fullStandardsContext) { + console.log(`🔄 Generating practice questions for ${fullStandardsContext.targetStandard.statementCode}...`); + + try { + // Build prompt inline + let prerequisiteText = ''; + for (const prereq of fullStandardsContext.prereqStandards) { + prerequisiteText += `- ${prereq.statementCode}: ${prereq.description}\n`; + prerequisiteText += ' Supporting Learning Components:\n'; + for (const lc of prereq.supportingLearningComponents) { + prerequisiteText += ` • ${lc.description}\n`; } + } - const prompt = `You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. + const prompt = `You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. Target Standard: - ${fullStandardsContext.targetStandard.statementCode}: ${fullStandardsContext.targetStandard.description} @@ -275,58 +192,62 @@ Target Standard: Prerequisite Standards & Supporting Learning Components: ${prerequisiteText}`; - const response = await openai.chat.completions.create({ - model: OPENAI_MODEL, - messages: [ - { role: 'system', content: 'You are an expert middle school math tutor.' }, - { role: 'user', content: prompt } - ], - temperature: OPENAI_TEMPERATURE - }); + const response = await openai.chat.completions.create({ + model: OPENAI_MODEL, + messages: [ + { role: 'system', content: 'You are an expert middle school math tutor.' }, + { role: 'user', content: prompt } + ], + temperature: OPENAI_TEMPERATURE + }); - const practiceQuestions = response.choices[0].message.content.trim(); + const practiceQuestions = response.choices[0].message.content.trim(); - console.log(`✅ Generated practice questions:\n`); - console.log(practiceQuestions); + console.log(`✅ Generated practice questions:\n`); + console.log(practiceQuestions); - return { - aiGenerated: practiceQuestions, - targetStandard: fullStandardsContext.targetStandard.statementCode, - prerequisiteCount: fullStandardsContext.prereqStandards.length - }; - } catch (err) { - console.error('❌ Error generating practice questions:', err.message); - throw err; - } - }; + return { + aiGenerated: practiceQuestions, + targetStandard: fullStandardsContext.targetStandard.statementCode, + prerequisiteCount: fullStandardsContext.prereqStandards.length + }; + } catch (err) { + console.error('❌ Error generating practice questions:', err.message); + throw err; + } } +/* ================================ + MAIN EXECUTION + ================================ */ async function main() { console.log('\n=== GENERATE PREREQUISITE PRACTICE TUTORIAL ===\n'); - console.log('🔄 Step 1: Loading data...'); - const aq = await import('arquero'); - const { relevantStandards, relevantRelationships, relevantLearningComponents } = loadData(aq); + console.log('🔄 Step 2: Get prerequisite standards for 6.NS.B.4...\n'); - console.log('\n🔄 Step 2: Querying prerequisite data...'); - const prerequisiteData = queryPrerequisiteData(aq, relevantStandards, relevantRelationships, relevantLearningComponents); + // Get target standard and prerequisites + const prerequisiteData = await getStandardAndPrerequisites(); if (!prerequisiteData) { console.error('❌ Failed to find prerequisite data'); return; } - const { targetStandard, prerequisiteLearningComponents } = prerequisiteData; + const { targetStandard, prerequisiteStandards } = prerequisiteData; + + // Get learning components for prerequisites + console.log(); + const prerequisiteLearningComponents = await getLearningComponentsForPrerequisites(prerequisiteStandards); - console.log('\n🔄 Step 3: Generating practice...'); + console.log('\n🔄 Step 3: Generate practice problems...\n'); const fullStandardsContext = packageContextData(targetStandard, prerequisiteLearningComponents); - const generatePractice = generatePracticeData(fullStandardsContext); + if (GENERATE_PRACTICE) { - await generatePractice(); + await generatePractice(fullStandardsContext); } else { console.log('🚫 Practice question generation disabled'); } } -main().catch(console.error); \ No newline at end of file +main().catch(console.error); diff --git a/tutorials/generate_prereq_practice/js/package-lock.json b/tutorials/generate_prereq_practice/js/package-lock.json index 21bfb88..0069616 100644 --- a/tutorials/generate_prereq_practice/js/package-lock.json +++ b/tutorials/generate_prereq_practice/js/package-lock.json @@ -9,8 +9,6 @@ "version": "1.0.0", "license": "ISC", "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", "dotenv": "^16.3.1", "openai": "^4.0.0" } @@ -34,12 +32,6 @@ "form-data": "^4.0.4" } }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, "node_modules/abort-controller": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz", @@ -52,18 +44,6 @@ "node": ">=6.5" } }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, "node_modules/agentkeepalive": { "version": "4.6.0", "resolved": "https://registry.npmjs.org/agentkeepalive/-/agentkeepalive-4.6.0.tgz", @@ -76,16 +56,6 @@ "node": ">= 8.0.0" } }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, "node_modules/asynckit": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", @@ -117,12 +87,6 @@ "node": ">= 0.8" } }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, "node_modules/delayed-stream": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", diff --git a/tutorials/generate_prereq_practice/js/package.json b/tutorials/generate_prereq_practice/js/package.json index b731fd5..e1104d6 100644 --- a/tutorials/generate_prereq_practice/js/package.json +++ b/tutorials/generate_prereq_practice/js/package.json @@ -1,7 +1,7 @@ { "name": "generate-prereq-practice", "version": "1.0.0", - "description": "Generate prerequisite practice questions using Knowledge Graph datasets", + "description": "Generate prerequisite practice questions using Knowledge Graph REST API", "main": "generate-prereq-practice.js", "scripts": { "start": "node generate-prereq-practice.js", @@ -9,8 +9,6 @@ }, "dependencies": { "openai": "^4.0.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", "dotenv": "^16.3.1" }, "keywords": [ @@ -18,8 +16,9 @@ "prerequisites", "knowledge-graph", "practice-questions", - "ai" + "ai", + "rest-api" ], "author": "", "license": "ISC" -} \ No newline at end of file +} diff --git a/tutorials/generate_prereq_practice/python/.env.example b/tutorials/generate_prereq_practice/python/.env.example index 0483037..533ba61 100644 --- a/tutorials/generate_prereq_practice/python/.env.example +++ b/tutorials/generate_prereq_practice/python/.env.example @@ -1,5 +1,6 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 # OpenAI API key for generating practice questions -OPENAI_API_KEY=your_openai_api_key_here \ No newline at end of file +OPENAI_API_KEY=your_openai_api_key_here diff --git a/tutorials/generate_prereq_practice/python/README.md b/tutorials/generate_prereq_practice/python/README.md index 5db696e..d3cc33f 100644 --- a/tutorials/generate_prereq_practice/python/README.md +++ b/tutorials/generate_prereq_practice/python/README.md @@ -1,27 +1,24 @@ # Generate Prerequisite Practice -Demonstrates how to generate prerequisite-based practice questions using Knowledge Graph data covering: -- **Data Loading**: Reading CSV files with educational frameworks and relationships +Demonstrates how to generate prerequisite-based practice questions using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query standards and learning progressions - **Prerequisite Analysis**: Finding standards that build towards a target standard -- **Relationship Mapping**: Connecting learning components to prerequisite standards -- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge +- **Learning Components**: Discovering granular learning components that support standards +- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge using AI -**Features**: Common Core Math analysis, prerequisite relationship traversal, learning component mapping - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/generating-prerequisite-practice-questions) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites - Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform +- OpenAI API key ## Dependencies +- **requests**: HTTP library for API calls - **openai**: OpenAI API for generating practice questions -- **pandas**: Data manipulation and analysis library - **python-dotenv**: Environment variable management ## Quick Start @@ -37,7 +34,11 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 + + # OpenAI API key for generating practice questions OPENAI_API_KEY=your_openai_api_key_here ``` diff --git a/tutorials/generate_prereq_practice/python/generate_prereq_practice.py b/tutorials/generate_prereq_practice/python/generate_prereq_practice.py index b1d4d49..427b62a 100644 --- a/tutorials/generate_prereq_practice/python/generate_prereq_practice.py +++ b/tutorials/generate_prereq_practice/python/generate_prereq_practice.py @@ -8,9 +8,7 @@ # Dependencies import os import sys -import json -import pandas as pd -from pathlib import Path +import requests from dotenv import load_dotenv from openai import OpenAI @@ -19,26 +17,30 @@ # Domain Constants GENERATE_PRACTICE = True -# Filter criteria for mathematics standards -JURISDICTION = 'Multi-State' -ACADEMIC_SUBJECT = 'Mathematics' TARGET_CODE = '6.NS.B.4' # OpenAI configuration OPENAI_MODEL = 'gpt-4' OPENAI_TEMPERATURE = 0.7 # Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') +api_key = os.getenv('API_KEY') +base_url = os.getenv('BASE_URL') + +if not api_key: + print('❌ API_KEY environment variable is not set.') sys.exit(1) -data_path = Path(data_dir) +if not base_url: + print('❌ BASE_URL environment variable is not set.') + sys.exit(1) openai_client = OpenAI( api_key=os.getenv('OPENAI_API_KEY') ) +# Setup headers for API requests +headers = {"x-api-key": api_key} + """ ================================ @@ -46,199 +48,86 @@ ================================ """ -def load_csv(filename): - """Load and parse CSV file from data directory""" +def make_api_request(endpoint, params=None): + """Make API request to Knowledge Graph API""" try: - file_path = data_path / filename - return pd.read_csv(file_path) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') + url = f"{base_url}{endpoint}" + response = requests.get(url, headers=headers, params=params) + response.raise_for_status() + return response.json() + except requests.exceptions.RequestException as error: + print(f'❌ Error making API request to {endpoint}: {str(error)}') raise error """ ================================ -STEP 1: LOAD DATA +STEP 2: GET PREREQUISITE STANDARDS ================================ """ -def load_data(): - """Load CSV data files and build filtered dataset""" - - standards_framework_items = load_csv('StandardsFrameworkItem.csv') - learning_components = load_csv('LearningComponent.csv') - relationships = load_csv('Relationships.csv') - - print('✅ Files loaded from KG CSV files') - print(f' Standards Framework Items: {len(standards_framework_items)}') - print(f' Learning Components: {len(learning_components)}') - print(f' Relationships: {len(relationships)}') - - # Filter for relevant StandardsFrameworkItems by jurisdiction and subject - relevant_standards = standards_framework_items[ - (standards_framework_items['jurisdiction'] == JURISDICTION) & - (standards_framework_items['academicSubject'] == ACADEMIC_SUBJECT) - ] - - # Get set of relevant identifiers for filtering - relevant_standard_ids = set(relevant_standards['caseIdentifierUUID']) - - # Filter relationships for buildsTowards and supports relationships - relevant_relationships = relationships[ - ((relationships['relationshipType'] == 'buildsTowards') & - (relationships['sourceEntityValue'].isin(relevant_standard_ids)) & - (relationships['targetEntityValue'].isin(relevant_standard_ids))) | - ((relationships['relationshipType'] == 'supports') & - (relationships['targetEntityValue'].isin(relevant_standard_ids))) - ] - - # Get learning component IDs from supports relationships - support_relationships = relevant_relationships[ - relevant_relationships['relationshipType'] == 'supports' - ] - linked_learning_component_ids = set(support_relationships['sourceEntityValue']) - - # Filter learning components by identifier - relevant_learning_components = learning_components[ - learning_components['identifier'].isin(linked_learning_component_ids) - ] - - print('✅ Retrieved scoped graph:') - print(f' Standards Framework Items: {len(relevant_standards)}') - print(f' Learning Components: {len(relevant_learning_components)}') - print(f' Relationships: {len(relevant_relationships)}') - - return { - 'relevant_standards': relevant_standards, - 'relevant_relationships': relevant_relationships, - 'relevant_learning_components': relevant_learning_components - } - +def get_standard_and_prerequisites(): + """Find the target standard and its prerequisites""" -""" -================================ -STEP 2: QUERY PREREQUISITE DATA -================================ -""" + # Find the target standard by statement code + search_result = make_api_request( + '/academic-standards/search', + params={ + 'statementCode': TARGET_CODE, + 'jurisdiction': 'Multi-State' + } + ) -def get_standard_and_prerequisites(relevant_standards, relevant_relationships): - target_standard_df = relevant_standards[ - relevant_standards['statementCode'] == TARGET_CODE - ] + target_standard = search_result[0] if search_result else None - if len(target_standard_df) == 0: - print(f'❌ No StandardsFrameworkItem found for statementCode = "{TARGET_CODE}"') + if not target_standard: + print(f'❌ No standard found for {TARGET_CODE}') return None - target_standard = target_standard_df.iloc[0] - print(f'✅ Found StandardsFrameworkItem for "{TARGET_CODE}":') + print(f'✅ Found standard {TARGET_CODE}:') print(f' UUID: {target_standard["caseIdentifierUUID"]}') - print(f' Statement Code: {target_standard["statementCode"]}') print(f' Description: {target_standard["description"]}') - prerequisite_links = relevant_relationships[ - (relevant_relationships['relationshipType'] == 'buildsTowards') & - (relevant_relationships['targetEntityValue'] == target_standard['caseIdentifierUUID']) - ] - - prerequisite_standards = prerequisite_links.merge( - relevant_standards, - left_on='sourceEntityValue', - right_on='caseIdentifierUUID', - suffixes=('_rel', '_std') - )[['sourceEntityValue', 'statementCode', 'description_std']].rename(columns={ - 'sourceEntityValue': 'caseIdentifierUUID', - 'description_std': 'standardDescription' - }) - - print(f'✅ Found {len(prerequisite_standards)} prerequisite(s) for {target_standard["statementCode"]}:') - for i, (_, std) in enumerate(prerequisite_standards.iterrows(), 1): - print(f' {i}. {std["statementCode"]}: {std["standardDescription"][:100]}...') - - return {'target_standard': target_standard, 'prerequisite_standards': prerequisite_standards} + # Get prerequisites + prereq_result = make_api_request( + f'/academic-standards/{target_standard["caseIdentifierUUID"]}/prerequisites' + ) + prerequisite_standards = prereq_result['data'] + print(f'✅ Found {len(prerequisite_standards)} prerequisite(s):') + for prereq in prerequisite_standards: + description = prereq.get('description', 'No description') + truncated = description[:80] + '...' if len(description) > 80 else description + print(f' {prereq["statementCode"]}: {truncated}') -def get_learning_components_for_prerequisites(prerequisite_standards, relevant_relationships, relevant_learning_components): - prerequisite_learning_components = prerequisite_standards.merge( - relevant_relationships, - left_on='caseIdentifierUUID', - right_on='targetEntityValue', - suffixes=('_std', '_rel') - ) - - prerequisite_learning_components = prerequisite_learning_components[ - prerequisite_learning_components['relationshipType'] == 'supports' - ].merge( - relevant_learning_components, - left_on='sourceEntityValue', - right_on='identifier', - suffixes=('_rel', '_lc') - ) - - # Select the correct columns - after merge with suffixes, learning component description becomes description_lc - prerequisite_learning_components = prerequisite_learning_components[ - ['caseIdentifierUUID', 'statementCode', 'standardDescription', 'description_lc'] - ].rename(columns={ - 'description_lc': 'learningComponentDescription' - }) + return {'target_standard': target_standard, 'prerequisite_standards': prerequisite_standards} - print(f'✅ Found {len(prerequisite_learning_components)} supporting learning components for prerequisites:') - for i, (_, lc) in enumerate(prerequisite_learning_components.iterrows(), 1): - print(f' {i}. {lc["learningComponentDescription"][:100]}...') - return prerequisite_learning_components +def get_learning_components_for_prerequisites(prerequisite_standards): + """Get learning components for each prerequisite standard""" + prerequisite_learning_components = [] -def query_prerequisite_data(relevant_standards, relevant_relationships, relevant_learning_components): - """ - Analyze prerequisite relationships for the target standard - This step identifies prerequisites and supporting learning components - - SQL: WITH target AS ( - SELECT "caseIdentifierUUID" - FROM standards_framework_item - WHERE "statementCode" = '6.NS.B.4' - ), - prerequisite_standards AS ( - SELECT - sfi."caseIdentifierUUID", - sfi."statementCode", - sfi."description" - FROM standards_framework_item sfi - JOIN relationships r - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" - JOIN target t - ON r."targetEntityValue" = t."caseIdentifierUUID" - WHERE r."relationshipType" = 'buildsTowards' - ) - SELECT - ps."caseIdentifierUUID", - ps."statementCode", - ps."description", - lc."description" - FROM prerequisite_standards ps - JOIN relationships r - ON ps."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON r."sourceEntityValue" = lc."identifier" - WHERE r."relationshipType" = 'supports'; - - Cypher: MATCH (target:StandardsFrameworkItem {statementCode: '6.NS.B.4'}) - MATCH (prereq:StandardsFrameworkItem)-[:buildsTowards]->(target) - MATCH (lc:LearningComponent)-[:supports]->(prereq) - RETURN prereq.caseIdentifierUUID, prereq.statementCode, prereq.description, - lc.description - """ + for prereq in prerequisite_standards: + lc_result = make_api_request( + f'/academic-standards/{prereq["caseIdentifierUUID"]}/learning-components' + ) - standard_and_prereq_data = get_standard_and_prerequisites(relevant_standards, relevant_relationships) - if not standard_and_prereq_data: - return None + for lc in lc_result['data']: + prerequisite_learning_components.append({ + 'caseIdentifierUUID': prereq['caseIdentifierUUID'], + 'statementCode': prereq['statementCode'], + 'standardDescription': prereq['description'], + 'learningComponentDescription': lc['description'] + }) - target_standard = standard_and_prereq_data['target_standard'] - prerequisite_standards = standard_and_prereq_data['prerequisite_standards'] - prerequisite_learning_components = get_learning_components_for_prerequisites(prerequisite_standards, relevant_relationships, relevant_learning_components) + print(f'✅ Found {len(prerequisite_learning_components)} supporting learning components for prerequisites:') + for lc in prerequisite_learning_components[:5]: + description = lc.get('learningComponentDescription', 'No description') + truncated = description[:80] + '...' if len(description) > 80 else description + print(f' {truncated}') - return {'target_standard': target_standard, 'prerequisite_learning_components': prerequisite_learning_components} + return prerequisite_learning_components """ @@ -253,12 +142,10 @@ def package_context_data(target_standard, prerequisite_learning_components): This creates a structured context that can be used for generating practice questions """ - # Convert dataframe to context format for LLM - all_rows = prerequisite_learning_components.to_dict('records') standards_map = {} # Group learning components by standard for context - for row in all_rows: + for row in prerequisite_learning_components: case_id = row['caseIdentifierUUID'] if case_id not in standards_map: standards_map[case_id] = { @@ -280,27 +167,23 @@ def package_context_data(target_standard, prerequisite_learning_components): } print('✅ Packaged full standards context for text generation') - print(json.dumps(full_standards_context, indent=2)) return full_standards_context -def generate_practice_data(full_standards_context): - """Generate practice questions using OpenAI API - This creates educational content based on prerequisite data - """ - def generate_practice(): - print(f'🔄 Generating practice questions for {full_standards_context["targetStandard"]["statementCode"]}...') +def generate_practice(full_standards_context): + """Generate practice questions using OpenAI API""" + print(f'🔄 Generating practice questions for {full_standards_context["targetStandard"]["statementCode"]}...') - try: - # Build prompt inline - prerequisite_text = '' - for prereq in full_standards_context['prereqStandards']: - prerequisite_text += f'- {prereq["statementCode"]}: {prereq["description"]}\n' - prerequisite_text += ' Supporting Learning Components:\n' - for lc in prereq['supportingLearningComponents']: - prerequisite_text += f' • {lc["description"]}\n' + try: + # Build prompt inline + prerequisite_text = '' + for prereq in full_standards_context['prereqStandards']: + prerequisite_text += f'- {prereq["statementCode"]}: {prereq["description"]}\n' + prerequisite_text += ' Supporting Learning Components:\n' + for lc in prereq['supportingLearningComponents']: + prerequisite_text += f' • {lc["description"]}\n' - prompt = f"""You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. + prompt = f"""You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. Target Standard: - {full_standards_context["targetStandard"]["statementCode"]}: {full_standards_context["targetStandard"]["description"]} @@ -308,30 +191,28 @@ def generate_practice(): Prerequisite Standards & Supporting Learning Components: {prerequisite_text}""" - response = openai_client.chat.completions.create( - model=OPENAI_MODEL, - messages=[ - {'role': 'system', 'content': 'You are an expert middle school math tutor.'}, - {'role': 'user', 'content': prompt} - ], - temperature=OPENAI_TEMPERATURE - ) + response = openai_client.chat.completions.create( + model=OPENAI_MODEL, + messages=[ + {'role': 'system', 'content': 'You are an expert middle school math tutor.'}, + {'role': 'user', 'content': prompt} + ], + temperature=OPENAI_TEMPERATURE + ) - practice_questions = response.choices[0].message.content.strip() + practice_questions = response.choices[0].message.content.strip() - print('✅ Generated practice questions:\n') - print(practice_questions) + print('✅ Generated practice questions:\n') + print(practice_questions) - return { - 'aiGenerated': practice_questions, - 'targetStandard': full_standards_context['targetStandard']['statementCode'], - 'prerequisiteCount': len(full_standards_context['prereqStandards']) - } - except Exception as err: - print(f'❌ Error generating practice questions: {str(err)}') - raise err - - return generate_practice + return { + 'aiGenerated': practice_questions, + 'targetStandard': full_standards_context['targetStandard']['statementCode'], + 'prerequisiteCount': len(full_standards_context['prereqStandards']) + } + except Exception as err: + print(f'❌ Error generating practice questions: {str(err)}') + raise err """ @@ -344,30 +225,30 @@ def main(): """Main execution function - orchestrates all tutorial steps""" print('\n=== GENERATE PREREQUISITE PRACTICE TUTORIAL ===\n') - print('🔄 Step 1: Loading data...') - data = load_data() - relevant_standards = data['relevant_standards'] - relevant_relationships = data['relevant_relationships'] - relevant_learning_components = data['relevant_learning_components'] + print('🔄 Step 2: Get prerequisite standards for 6.NS.B.4...\n') - print('\n🔄 Step 2: Querying prerequisite data...') - prerequisite_data = query_prerequisite_data(relevant_standards, relevant_relationships, relevant_learning_components) + # Get target standard and prerequisites + prerequisite_data = get_standard_and_prerequisites() if not prerequisite_data: print('❌ Failed to find prerequisite data') return target_standard = prerequisite_data['target_standard'] - prerequisite_learning_components = prerequisite_data['prerequisite_learning_components'] + prerequisite_standards = prerequisite_data['prerequisite_standards'] + + # Get learning components for prerequisites + print() + prerequisite_learning_components = get_learning_components_for_prerequisites(prerequisite_standards) - print('\n🔄 Step 3: Generating practice...') + print('\n🔄 Step 3: Generate practice problems...\n') full_standards_context = package_context_data(target_standard, prerequisite_learning_components) - generate_practice = generate_practice_data(full_standards_context) + if GENERATE_PRACTICE: - generate_practice() + generate_practice(full_standards_context) else: print('🚫 Practice question generation disabled') if __name__ == '__main__': - main() \ No newline at end of file + main() diff --git a/tutorials/generate_prereq_practice/python/requirements.txt b/tutorials/generate_prereq_practice/python/requirements.txt index 875454d..53c4e86 100644 --- a/tutorials/generate_prereq_practice/python/requirements.txt +++ b/tutorials/generate_prereq_practice/python/requirements.txt @@ -1,8 +1,8 @@ -# Core data manipulation and analysis -pandas>=2.0.0 +# HTTP requests +requests>=2.31.0 # Environment variable management python-dotenv>=1.0.0 # OpenAI API for generating practice questions -openai>=1.0.0 \ No newline at end of file +openai>=1.0.0 diff --git a/tutorials/working_with_standards/js/.env.example b/tutorials/working_with_standards/js/.env.example index f2d3f1f..3c86715 100644 --- a/tutorials/working_with_standards/js/.env.example +++ b/tutorials/working_with_standards/js/.env.example @@ -1,2 +1,3 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 diff --git a/tutorials/working_with_standards/js/README.md b/tutorials/working_with_standards/js/README.md index 2b2bfce..8330180 100644 --- a/tutorials/working_with_standards/js/README.md +++ b/tutorials/working_with_standards/js/README.md @@ -1,25 +1,17 @@ -Demonstrates how to work with state educational standards data covering: -- **Data Loading**: Reading CSV files with standards frameworks, items, and relationships -- **Hierarchical Queries**: Traversing parent-child relationships in educational standards -- **AI Embeddings**: Generating vector embeddings for semantic search -- **Vector Search**: Performing similarity-based searches on educational standards +Demonstrates how to work with state educational standards data using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query standards frameworks and items +- **Standards Querying**: Filtering standards by jurisdiction, grade level, and type -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/working-with-state-standards) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFramework.csv` - - `StandardsFrameworkItem.csv` - - `Relationships.csv` +- Node.js 18 or higher (for native fetch API support) +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform ## Dependencies -- **@xenova/transformers**: Local AI embeddings (no API required) -- **arquero**: Data manipulation and analysis library -- **csv-parse**: CSV file parsing -- **fast-cosine-similarity**: Vector similarity calculations - **dotenv**: Environment variable management ## Quick Start @@ -33,7 +25,9 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 ``` 3. **Run Tutorial**: diff --git a/tutorials/working_with_standards/js/package-lock.json b/tutorials/working_with_standards/js/package-lock.json index ca416cb..d7b435f 100644 --- a/tutorials/working_with_standards/js/package-lock.json +++ b/tutorials/working_with_standards/js/package-lock.json @@ -9,360 +9,7 @@ "version": "1.0.0", "license": "ISC", "dependencies": { - "@xenova/transformers": "^2.6.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1", - "fast-cosine-similarity": "^1.2.2" - } - }, - "node_modules/@huggingface/jinja": { - "version": "0.2.2", - "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.2.2.tgz", - "integrity": "sha512-/KPde26khDUIPkTGU82jdtTW9UAuvUTumCAbFs/7giR0SxsvZC4hru51PBvpijH6BVkHcROcvZM/lpy5h1jRRA==", - "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/@protobufjs/aspromise": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", - "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/base64": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", - "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/codegen": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz", - "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/eventemitter": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", - "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/fetch": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", - "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.1", - "@protobufjs/inquire": "^1.1.0" - } - }, - "node_modules/@protobufjs/float": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", - "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/inquire": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz", - "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/path": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", - "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/pool": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", - "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/utf8": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz", - "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==", - "license": "BSD-3-Clause" - }, - "node_modules/@types/long": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/@types/long/-/long-4.0.2.tgz", - "integrity": "sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==", - "license": "MIT" - }, - "node_modules/@types/node": { - "version": "18.19.123", - "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.123.tgz", - "integrity": "sha512-K7DIaHnh0mzVxreCR9qwgNxp3MH9dltPNIEddW9MYUlcKAzm+3grKNSTe2vCJHI1FaLpvpL5JGJrz1UZDKYvDg==", - "license": "MIT", - "dependencies": { - "undici-types": "~5.26.4" - } - }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, - "node_modules/@xenova/transformers": { - "version": "2.17.2", - "resolved": "https://registry.npmjs.org/@xenova/transformers/-/transformers-2.17.2.tgz", - "integrity": "sha512-lZmHqzrVIkSvZdKZEx7IYY51TK0WDrC8eR0c5IMnBsO8di8are1zzw8BlLhyO2TklZKLN5UffNGs1IJwT6oOqQ==", - "license": "Apache-2.0", - "dependencies": { - "@huggingface/jinja": "^0.2.2", - "onnxruntime-web": "1.14.0", - "sharp": "^0.32.0" - }, - "optionalDependencies": { - "onnxruntime-node": "1.14.0" - } - }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, - "node_modules/b4a": { - "version": "1.6.7", - "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.6.7.tgz", - "integrity": "sha512-OnAYlL5b7LEkALw87fUVafQw5rVR9RjwGd4KUwNQ6DrrNmaVaUCgLipfVlzrPQ4tWOR9P0IXGNOx50jYCCdSJg==", - "license": "Apache-2.0" - }, - "node_modules/bare-events": { - "version": "2.6.1", - "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.6.1.tgz", - "integrity": "sha512-AuTJkq9XmE6Vk0FJVNq5QxETrSA/vKHarWVBG5l/JbdCL1prJemiyJqUS0jrlXO0MftuPq4m3YVYhoNc5+aE/g==", - "license": "Apache-2.0", - "optional": true - }, - "node_modules/bare-fs": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.2.0.tgz", - "integrity": "sha512-oRfrw7gwwBVAWx9S5zPMo2iiOjxyiZE12DmblmMQREgcogbNO0AFaZ+QBxxkEXiPspcpvO/Qtqn8LabUx4uYXg==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "bare-events": "^2.5.4", - "bare-path": "^3.0.0", - "bare-stream": "^2.6.4" - }, - "engines": { - "bare": ">=1.16.0" - }, - "peerDependencies": { - "bare-buffer": "*" - }, - "peerDependenciesMeta": { - "bare-buffer": { - "optional": true - } - } - }, - "node_modules/bare-os": { - "version": "3.6.1", - "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.6.1.tgz", - "integrity": "sha512-uaIjxokhFidJP+bmmvKSgiMzj2sV5GPHaZVAIktcxcpCyBFFWO+YlikVAdhmUo2vYFvFhOXIAlldqV29L8126g==", - "license": "Apache-2.0", - "optional": true, - "engines": { - "bare": ">=1.14.0" - } - }, - "node_modules/bare-path": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz", - "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "bare-os": "^3.0.1" - } - }, - "node_modules/bare-stream": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.7.0.tgz", - "integrity": "sha512-oyXQNicV1y8nc2aKffH+BUHFRXmx6VrPzlnaEvMhram0nPBrKcEdcyBg5r08D0i8VxngHFAiVyn1QKXpSG0B8A==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "streamx": "^2.21.0" - }, - "peerDependencies": { - "bare-buffer": "*", - "bare-events": "*" - }, - "peerDependenciesMeta": { - "bare-buffer": { - "optional": true - }, - "bare-events": { - "optional": true - } - } - }, - "node_modules/base64-js": { - "version": "1.5.1", - "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", - "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/bl": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", - "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", - "license": "MIT", - "dependencies": { - "buffer": "^5.5.0", - "inherits": "^2.0.4", - "readable-stream": "^3.4.0" - } - }, - "node_modules/buffer": { - "version": "5.7.1", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", - "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.1.13" - } - }, - "node_modules/chownr": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", - "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", - "license": "ISC" - }, - "node_modules/color": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz", - "integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==", - "license": "MIT", - "dependencies": { - "color-convert": "^2.0.1", - "color-string": "^1.9.0" - }, - "engines": { - "node": ">=12.5.0" - } - }, - "node_modules/color-convert": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", - "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", - "license": "MIT", - "dependencies": { - "color-name": "~1.1.4" - }, - "engines": { - "node": ">=7.0.0" - } - }, - "node_modules/color-name": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", - "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", - "license": "MIT" - }, - "node_modules/color-string": { - "version": "1.9.1", - "resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz", - "integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==", - "license": "MIT", - "dependencies": { - "color-name": "^1.0.0", - "simple-swizzle": "^0.2.2" - } - }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, - "node_modules/decompress-response": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", - "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", - "license": "MIT", - "dependencies": { - "mimic-response": "^3.1.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/deep-extend": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", - "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", - "license": "MIT", - "engines": { - "node": ">=4.0.0" - } - }, - "node_modules/detect-libc": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.4.tgz", - "integrity": "sha512-3UDv+G9CsCKO1WKMGw9fwq/SWJYbI0c5Y7LU1AXYoDdbhE2AHQ6N6Nb34sG8Fj7T5APy8qXDCKuuIHd1BR0tVA==", - "license": "Apache-2.0", - "engines": { - "node": ">=8" + "dotenv": "^16.3.1" } }, "node_modules/dotenv": { @@ -376,537 +23,6 @@ "funding": { "url": "https://dotenvx.com" } - }, - "node_modules/end-of-stream": { - "version": "1.4.5", - "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", - "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", - "license": "MIT", - "dependencies": { - "once": "^1.4.0" - } - }, - "node_modules/expand-template": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", - "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", - "license": "(MIT OR WTFPL)", - "engines": { - "node": ">=6" - } - }, - "node_modules/fast-cosine-similarity": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/fast-cosine-similarity/-/fast-cosine-similarity-1.2.2.tgz", - "integrity": "sha512-VAAswiN4WOjvkbqSTXfatw+zKclDN9h2R3Si1pFoZSLd1jHjXizaOr1LLip7bI69IRcbyhOPQ3CjL5WfvHx2gg==", - "license": "MIT" - }, - "node_modules/fast-fifo": { - "version": "1.3.2", - "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz", - "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", - "license": "MIT" - }, - "node_modules/flatbuffers": { - "version": "1.12.0", - "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz", - "integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==", - "license": "SEE LICENSE IN LICENSE.txt" - }, - "node_modules/fs-constants": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", - "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", - "license": "MIT" - }, - "node_modules/github-from-package": { - "version": "0.0.0", - "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", - "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", - "license": "MIT" - }, - "node_modules/guid-typescript": { - "version": "1.0.9", - "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", - "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", - "license": "ISC" - }, - "node_modules/ieee754": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", - "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "BSD-3-Clause" - }, - "node_modules/inherits": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", - "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", - "license": "ISC" - }, - "node_modules/ini": { - "version": "1.3.8", - "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", - "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", - "license": "ISC" - }, - "node_modules/is-arrayish": { - "version": "0.3.2", - "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.2.tgz", - "integrity": "sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ==", - "license": "MIT" - }, - "node_modules/long": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/long/-/long-4.0.0.tgz", - "integrity": "sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==", - "license": "Apache-2.0" - }, - "node_modules/mimic-response": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", - "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/minimist": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", - "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/mkdirp-classic": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", - "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", - "license": "MIT" - }, - "node_modules/napi-build-utils": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", - "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", - "license": "MIT" - }, - "node_modules/node-abi": { - "version": "3.75.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.75.0.tgz", - "integrity": "sha512-OhYaY5sDsIka7H7AtijtI9jwGYLyl29eQn/W623DiN/MIv5sUqc4g7BIDThX+gb7di9f6xK02nkp8sdfFWZLTg==", - "license": "MIT", - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-addon-api": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-6.1.0.tgz", - "integrity": "sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==", - "license": "MIT" - }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "dependencies": { - "wrappy": "1" - } - }, - "node_modules/onnx-proto": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/onnx-proto/-/onnx-proto-4.0.4.tgz", - "integrity": "sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==", - "license": "MIT", - "dependencies": { - "protobufjs": "^6.8.8" - } - }, - "node_modules/onnxruntime-common": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz", - "integrity": "sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==", - "license": "MIT" - }, - "node_modules/onnxruntime-node": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.14.0.tgz", - "integrity": "sha512-5ba7TWomIV/9b6NH/1x/8QEeowsb+jBEvFzU6z0T4mNsFwdPqXeFUM7uxC6QeSRkEbWu3qEB0VMjrvzN/0S9+w==", - "license": "MIT", - "optional": true, - "os": [ - "win32", - "darwin", - "linux" - ], - "dependencies": { - "onnxruntime-common": "~1.14.0" - } - }, - "node_modules/onnxruntime-web": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz", - "integrity": "sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==", - "license": "MIT", - "dependencies": { - "flatbuffers": "^1.12.0", - "guid-typescript": "^1.0.9", - "long": "^4.0.0", - "onnx-proto": "^4.0.4", - "onnxruntime-common": "~1.14.0", - "platform": "^1.3.6" - } - }, - "node_modules/platform": { - "version": "1.3.6", - "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", - "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", - "license": "MIT" - }, - "node_modules/prebuild-install": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", - "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", - "license": "MIT", - "dependencies": { - "detect-libc": "^2.0.0", - "expand-template": "^2.0.3", - "github-from-package": "0.0.0", - "minimist": "^1.2.3", - "mkdirp-classic": "^0.5.3", - "napi-build-utils": "^2.0.0", - "node-abi": "^3.3.0", - "pump": "^3.0.0", - "rc": "^1.2.7", - "simple-get": "^4.0.0", - "tar-fs": "^2.0.0", - "tunnel-agent": "^0.6.0" - }, - "bin": { - "prebuild-install": "bin.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/prebuild-install/node_modules/tar-fs": { - "version": "2.1.4", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", - "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==", - "license": "MIT", - "dependencies": { - "chownr": "^1.1.1", - "mkdirp-classic": "^0.5.2", - "pump": "^3.0.0", - "tar-stream": "^2.1.4" - } - }, - "node_modules/prebuild-install/node_modules/tar-stream": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", - "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", - "license": "MIT", - "dependencies": { - "bl": "^4.0.3", - "end-of-stream": "^1.4.1", - "fs-constants": "^1.0.0", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/protobufjs": { - "version": "6.11.4", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", - "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.2", - "@protobufjs/base64": "^1.1.2", - "@protobufjs/codegen": "^2.0.4", - "@protobufjs/eventemitter": "^1.1.0", - "@protobufjs/fetch": "^1.1.0", - "@protobufjs/float": "^1.0.2", - "@protobufjs/inquire": "^1.1.0", - "@protobufjs/path": "^1.1.2", - "@protobufjs/pool": "^1.1.0", - "@protobufjs/utf8": "^1.1.0", - "@types/long": "^4.0.1", - "@types/node": ">=13.7.0", - "long": "^4.0.0" - }, - "bin": { - "pbjs": "bin/pbjs", - "pbts": "bin/pbts" - } - }, - "node_modules/pump": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.3.tgz", - "integrity": "sha512-todwxLMY7/heScKmntwQG8CXVkWUOdYxIvY2s0VWAAMh/nd8SoYiRaKjlr7+iCs984f2P8zvrfWcDDYVb73NfA==", - "license": "MIT", - "dependencies": { - "end-of-stream": "^1.1.0", - "once": "^1.3.1" - } - }, - "node_modules/rc": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", - "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", - "dependencies": { - "deep-extend": "^0.6.0", - "ini": "~1.3.0", - "minimist": "^1.2.0", - "strip-json-comments": "~2.0.1" - }, - "bin": { - "rc": "cli.js" - } - }, - "node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", - "license": "MIT", - "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/safe-buffer": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", - "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/semver": { - "version": "7.7.2", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.2.tgz", - "integrity": "sha512-RF0Fw+rO5AMf9MAyaRXI4AV0Ulj5lMHqVxxdSgiVbixSCXoEmmX/jk0CuJw4+3SqroYO9VoUh+HcuJivvtJemA==", - "license": "ISC", - "bin": { - "semver": "bin/semver.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/sharp": { - "version": "0.32.6", - "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.32.6.tgz", - "integrity": "sha512-KyLTWwgcR9Oe4d9HwCwNM2l7+J0dUQwn/yf7S0EnTtb0eVS4RxO0eUSvxPtzT4F3SY+C4K6fqdv/DO27sJ/v/w==", - "hasInstallScript": true, - "license": "Apache-2.0", - "dependencies": { - "color": "^4.2.3", - "detect-libc": "^2.0.2", - "node-addon-api": "^6.1.0", - "prebuild-install": "^7.1.1", - "semver": "^7.5.4", - "simple-get": "^4.0.1", - "tar-fs": "^3.0.4", - "tunnel-agent": "^0.6.0" - }, - "engines": { - "node": ">=14.15.0" - }, - "funding": { - "url": "https://opencollective.com/libvips" - } - }, - "node_modules/simple-concat": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", - "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/simple-get": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", - "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "decompress-response": "^6.0.0", - "once": "^1.3.1", - "simple-concat": "^1.0.0" - } - }, - "node_modules/simple-swizzle": { - "version": "0.2.2", - "resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.2.tgz", - "integrity": "sha512-JA//kQgZtbuY83m+xT+tXJkmJncGMTFT+C+g2h2R9uxkYIrE2yy9sgmcLhCnw57/WSD+Eh3J97FPEDFnbXnDUg==", - "license": "MIT", - "dependencies": { - "is-arrayish": "^0.3.1" - } - }, - "node_modules/streamx": { - "version": "2.22.1", - "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.22.1.tgz", - "integrity": "sha512-znKXEBxfatz2GBNK02kRnCXjV+AA4kjZIUxeWSr3UGirZMJfTE9uiwKHobnbgxWyL/JWro8tTq+vOqAK1/qbSA==", - "license": "MIT", - "dependencies": { - "fast-fifo": "^1.3.2", - "text-decoder": "^1.1.0" - }, - "optionalDependencies": { - "bare-events": "^2.2.0" - } - }, - "node_modules/string_decoder": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", - "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.2.0" - } - }, - "node_modules/strip-json-comments": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", - "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/tar-fs": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.1.tgz", - "integrity": "sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg==", - "license": "MIT", - "dependencies": { - "pump": "^3.0.0", - "tar-stream": "^3.1.5" - }, - "optionalDependencies": { - "bare-fs": "^4.0.1", - "bare-path": "^3.0.0" - } - }, - "node_modules/tar-stream": { - "version": "3.1.7", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.7.tgz", - "integrity": "sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ==", - "license": "MIT", - "dependencies": { - "b4a": "^1.6.4", - "fast-fifo": "^1.2.0", - "streamx": "^2.15.0" - } - }, - "node_modules/text-decoder": { - "version": "1.2.3", - "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.3.tgz", - "integrity": "sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==", - "license": "Apache-2.0", - "dependencies": { - "b4a": "^1.6.4" - } - }, - "node_modules/tunnel-agent": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", - "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", - "license": "Apache-2.0", - "dependencies": { - "safe-buffer": "^5.0.1" - }, - "engines": { - "node": "*" - } - }, - "node_modules/undici-types": { - "version": "5.26.5", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz", - "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==", - "license": "MIT" - }, - "node_modules/util-deprecate": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", - "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "license": "MIT" - }, - "node_modules/wrappy": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", - "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", - "license": "ISC" } } } diff --git a/tutorials/working_with_standards/js/package.json b/tutorials/working_with_standards/js/package.json index ecb022e..df4a222 100644 --- a/tutorials/working_with_standards/js/package.json +++ b/tutorials/working_with_standards/js/package.json @@ -1,24 +1,20 @@ { "name": "work-with-state-standards", "version": "1.0.0", - "description": "Working with state standards data using Knowledge Graph datasets", + "description": "Working with state standards data using Knowledge Graph REST API", "main": "work-with-state-standards.js", "scripts": { "start": "node work-with-state-standards.js", "test": "echo \"Error: no test specified\" && exit 1" }, "dependencies": { - "@xenova/transformers": "^2.6.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1", - "fast-cosine-similarity": "^1.2.2" + "dotenv": "^16.3.1" }, "keywords": [ "education", "standards", "knowledge-graph", - "embeddings" + "rest-api" ], "author": "", "license": "ISC" diff --git a/tutorials/working_with_standards/js/work-with-state-standards.js b/tutorials/working_with_standards/js/work-with-state-standards.js index 62eb772..1582f46 100644 --- a/tutorials/working_with_standards/js/work-with-state-standards.js +++ b/tutorials/working_with_standards/js/work-with-state-standards.js @@ -3,330 +3,144 @@ ================================ */ // Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); -const { cosineSimilarity } = require('fast-cosine-similarity'); require('dotenv').config(); // Constants -const GENERATE_EMBEDDINGS = true; const MIDDLE_SCHOOL_GRADES = ['6', '7', '8']; -// For this tutorial, we use 'all-MiniLM-L6-v2' which provides good quality embeddings -// for short text. You can substitute any compatible embedding model. -const EMBEDDING_MODEL = 'Xenova/all-MiniLM-L6-v2'; // Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); +const apiKey = process.env.API_KEY; +const baseUrl = process.env.BASE_URL; + +if (!apiKey) { + console.error('❌ API_KEY environment variable is not set.'); process.exit(1); } -const EMBEDDING_FILE_PATH = path.join(dataDir, 'california_math_embeddings.json'); -// Initialize embedding pipeline (will be loaded on first use) -let embedder = null; -let pipeline = null; +if (!baseUrl) { + console.error('❌ BASE_URL environment variable is not set.'); + process.exit(1); +} /* ================================ HELPER FUNCTIONS ================================ */ -function loadCSV(filename) { +async function makeApiRequest(endpoint, params = {}) { try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); + const url = new URL(`${baseUrl}${endpoint}`); + + // Handle array parameters (like gradeLevel) + Object.entries(params).forEach(([key, value]) => { + if (Array.isArray(value)) { + value.forEach(v => url.searchParams.append(key, v)); + } else { + url.searchParams.append(key, value); + } + }); + + const response = await fetch(url.toString(), { + method: 'GET', + headers: { + 'x-api-key': apiKey + } + }); + + if (!response.ok) { + throw new Error(`HTTP error! status: ${response.status}`); + } + + return await response.json(); } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); + console.error(`❌ Error making API request to ${endpoint}: ${error.message}`); throw error; } } -function findFrameworkItem(caseIdentifierUUID, standardsFrameworkItemsData) { - return standardsFrameworkItemsData.find(item => item.caseIdentifierUUID === caseIdentifierUUID); -} - - /* ================================ - STEP 1: LOAD DATA + STEP 2: QUERY FOR STANDARDS DATA ================================ */ -function loadData(aq) { - /* Load CSV data files needed for the tutorial - */ - const standardsFrameworksData = loadCSV('StandardsFramework.csv'); - const standardsFrameworkItemsData = loadCSV('StandardsFrameworkItem.csv'); - - console.log('✅ Files loaded from KG CSV files'); - console.log({ - standardsFrameworks: standardsFrameworksData.length, - standardsFrameworkItems: standardsFrameworkItemsData.length + +async function getCaliforniaFramework() { + const result = await makeApiRequest('/standards-frameworks', { + jurisdiction: 'California', + academicSubject: 'Mathematics' }); - - return { standardsFrameworksData, standardsFrameworkItemsData }; -} -/* ================================ - STEP 2: QUERY FOR STANDARDS DATA - ================================ */ + const californiaFramework = result.data[0] || null; -function getMathStandardsFrameworks(aq, standardsFrameworksData) { - /* Get snapshot of mathematics standards frameworks - * - * SQL: SELECT "name", "academicSubject", "jurisdiction", "identifier" - * FROM standards_framework - * WHERE "academicSubject" = 'Mathematics'; - * - * Cypher: MATCH (sf:StandardsFramework {academicSubject: 'Mathematics'}) - * RETURN sf.name, sf.academicSubject, sf.jurisdiction, sf.identifier - */ - const mathFrameworks = aq.from(standardsFrameworksData) - .filter(d => d.academicSubject === 'Mathematics') - .select('caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject'); - - console.log(`✅ Retrieved ${mathFrameworks.numRows()} state standard frameworks for math (dataframe):`); - console.log('Sample of first 5 frameworks:'); - console.log(mathFrameworks.slice(0,5).objects()); - - /* Get California math framework metadata - * - * SQL: SELECT * - * FROM standards_framework - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics'; - * Cypher: MATCH (sf:StandardsFramework {jurisdiction: 'California', academicSubject: 'Mathematics'}) RETURN sf - */ - const californiaFrameworkTable = aq.from(standardsFrameworksData) - .filter(d => d.jurisdiction === 'California' && d.academicSubject === 'Mathematics') - .select('caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject'); - - const californiaFramework = californiaFrameworkTable.object(); - - console.log(`✅ Retrieved ${californiaFramework ? 1 : 0} California math standards framework:`); + console.log('✅ Retrieved California math standards framework:'); if (californiaFramework) { console.log(californiaFramework); } - - return { mathFrameworks, californiaFramework }; -} -function getMiddleSchoolStandardsGroupings(aq, standardsFrameworkItemsData) { - /* Filter for middle school standard groupings from California framework - * - * SQL: SELECT * - * FROM standards_framework_item - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics' - * AND "normalizedStatementType" = 'Standard Grouping' - * AND ( - * EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - * ); - * - * Cypher: MATCH (sfi:StandardsFrameworkItem) - * WHERE sfi.jurisdiction = 'California' - * AND sfi.academicSubject = 'Mathematics' - * AND sfi.normalizedStatementType = 'Standard Grouping' - * AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - * RETURN sfi; - */ - const groupings = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard Grouping' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)))) - .select('caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'statementType', 'gradeLevel'); - - console.log(`✅ Retrieved ${groupings.numRows()} standard groupings for middle school math in California (dataframe):`); - console.log('Sample of first 5 standard groupings:'); - console.log(groupings.slice(0,5).objects()); - - return groupings; + return californiaFramework; } -function getMiddleSchoolStandards(aq, standardsFrameworkItemsData) { - /* Get all standards for California middle school mathematics - * - * SQL: SELECT * - * FROM standards_framework_item - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics' - * AND "normalizedStatementType" = 'Standard' - * AND ( - * EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - * ); - * - * Cypher: MATCH (sfi:StandardsFrameworkItem) - * WHERE sfi.jurisdiction = 'California' - * AND sfi.academicSubject = 'Mathematics' - * AND sfi.normalizedStatementType = 'Standard' - * AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - * RETURN sfi; - */ - const standards = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)))) - .select('caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'gradeLevel'); - - console.log(`✅ Retrieved ${standards.numRows()} standards for California middle school mathematics (dataframe):`); - console.log('Sample of first 5 standards:'); - console.log(standards.slice(100, 105).objects()); - - return standards; -} +async function getMiddleSchoolStandardsGroupings(frameworkUuid) { + const result = await makeApiRequest('/academic-standards', { + standardsFrameworkCaseIdentifierUUID: frameworkUuid, + normalizedStatementType: 'Standard Grouping', + gradeLevel: MIDDLE_SCHOOL_GRADES + }); + + const groupings = result.data; -function queryForStandardsData(aq, standardsFrameworksData, standardsFrameworkItemsData) { - const { mathFrameworks, californiaFramework } = getMathStandardsFrameworks(aq, standardsFrameworksData); - const groupings = getMiddleSchoolStandardsGroupings(aq, standardsFrameworkItemsData); - const standards = getMiddleSchoolStandards(aq, standardsFrameworkItemsData); - - return { californiaFramework, groupings, standards }; + console.log(`✅ Retrieved ${groupings.length} standard groupings for middle school math in California`); + groupings.slice(0, 5).forEach(grouping => { + const description = grouping.description || 'No description'; + const truncated = description.length > 80 ? description.substring(0, 80) + '...' : description; + console.log(` ${grouping.statementCode || 'N/A'}: ${truncated}`); + }); + + return groupings; } -/* ================================ - STEP 3: EMBED STANDARD DATA - ================================ */ -function embedStandardData(aq, standardsFrameworkItemsData) { - // Generate embeddings for California middle school mathematics standards - const embeddingStandards = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)) && - !!item.description)) - .objects(); - - /* Generate and save embeddings for each standard - * This creates vector representations of standard descriptions for semantic search - */ - return async function generateEmbeddings() { - const results = []; - console.log(`🔄 Generating embeddings for ${embeddingStandards.length} standards...`); - - // Initialize embedder if not already done - if (!embedder) { - console.log('📥 Loading embedding model (first time only)...'); - if (!pipeline) { - const { pipeline: pipelineImport } = await import('@xenova/transformers'); - pipeline = pipelineImport; - } - embedder = await pipeline('feature-extraction', EMBEDDING_MODEL); - console.log('✅ Embedding model loaded'); - } +async function getMiddleSchoolStandards(frameworkUuid) { + const result = await makeApiRequest('/academic-standards', { + standardsFrameworkCaseIdentifierUUID: frameworkUuid, + normalizedStatementType: 'Standard', + gradeLevel: MIDDLE_SCHOOL_GRADES + }); - for (const standard of embeddingStandards) { - const code = standard.statementCode || '(no code)'; - - try { - const output = await embedder(standard.description, { pooling: 'mean', normalize: true }); - const embedding = Array.from(output.data); - - results.push({ - caseIdentifierUUID: standard.caseIdentifierUUID, - statementCode: standard.statementCode, - embedding: embedding - }); - - console.log(`✅ ${code}`); - } catch (err) { - console.error(`❌ ${code}: ${err.message}`); - throw err; - } - } + const standards = result.data; + + console.log(`✅ Retrieved ${standards.length} standards for California middle school mathematics`); + standards.slice(0, 5).forEach(standard => { + const description = standard.description || 'No description'; + const truncated = description.length > 80 ? description.substring(0, 80) + '...' : description; + console.log(` ${standard.statementCode || 'N/A'}: ${truncated}`); + }); - // Save embeddings to file - fs.writeFileSync(EMBEDDING_FILE_PATH, JSON.stringify(results, null, 2)); - console.log(`✅ Saved ${results.length} embeddings to ${EMBEDDING_FILE_PATH}`); - }; + return standards; } /* ================================ - STEP 4: VECTOR SEARCH STANDARD DATA + MAIN EXECUTION ================================ */ -function vectorSearchStandardData(standardsFrameworkItemsData) { - /* Perform vector search using cosine similarity - */ - return async function vectorSearch(query, topK = 5) { - // Initialize embedder if not already done - if (!embedder) { - console.log('📥 Loading embedding model...'); - if (!pipeline) { - const { pipeline: pipelineImport } = await import('@xenova/transformers'); - pipeline = pipelineImport; - } - embedder = await pipeline('feature-extraction', EMBEDDING_MODEL); - console.log('✅ Embedding model loaded'); - } - - let queryEmbedding; - try { - const output = await embedder(query, { pooling: 'mean', normalize: true }); - queryEmbedding = Array.from(output.data); - } catch (error) { - console.error(`❌ Error generating embedding for query "${query}": ${error.message}`); - return; - } - let storedEmbeddings; - try { - storedEmbeddings = JSON.parse(fs.readFileSync(EMBEDDING_FILE_PATH, 'utf8')); - } catch (error) { - console.error(`❌ Error loading embeddings from ${EMBEDDING_FILE_PATH}: ${error.message}`); - console.error('💡 Make sure to run the embedding generation step first (Step 3)'); - return; - } - - const topResults = storedEmbeddings - .map(item => ({ - caseIdentifierUUID: item.caseIdentifierUUID, - score: cosineSimilarity(queryEmbedding, item.embedding) - })) - .sort((a, b) => b.score - a.score) - .slice(0, topK); +async function main() { + console.log('\n=== WORKING WITH STATE STANDARDS TUTORIAL ===\n'); - console.log(`\nTop ${topK} results for "${query}":\n`); + console.log('🔄 Step 2: Querying for standards data...\n'); - + // 1. Get California math standards framework + const californiaFramework = await getCaliforniaFramework(); - topResults.forEach((result, i) => { - const frameworkItem = findFrameworkItem(result.caseIdentifierUUID, standardsFrameworkItemsData); - const statementCode = frameworkItem?.statementCode || '(no code)'; - const description = frameworkItem?.description || '(no statement)'; + if (!californiaFramework) { + console.error('❌ Could not retrieve California framework'); + process.exit(1); + } - topResults[i].statementCode = statementCode; - topResults[i].description = description; - }); + const frameworkUuid = californiaFramework.caseIdentifierUUID; - console.log(topResults); - }; -} + // 2. Get middle school standard groupings + console.log(); + await getMiddleSchoolStandardsGroupings(frameworkUuid); -async function main() { - const aq = await import('arquero'); - - console.log('\n=== WORKING WITH STATE STANDARDS TUTORIAL ===\n'); - - console.log('🔄 Step 1: Loading data...'); - const { standardsFrameworksData, standardsFrameworkItemsData } = loadData(aq); - - console.log('\n🔄 Step 2: Querying for standards data...'); - queryForStandardsData(aq, standardsFrameworksData, standardsFrameworkItemsData); - - console.log('\n🔄 Step 3: Embedding standard data...'); - const generateEmbeddings = embedStandardData(aq, standardsFrameworkItemsData); - if (GENERATE_EMBEDDINGS) { - await generateEmbeddings(); - } else { - console.log('🚫 Embedding generation disabled'); - } - - console.log('\n🔄 Step 4: Vector searching standard data...'); - const vectorSearch = vectorSearchStandardData(standardsFrameworkItemsData); - await vectorSearch('linear equations'); + // 3. Get middle school standards + console.log(); + await getMiddleSchoolStandards(frameworkUuid); } -main().catch(console.error); \ No newline at end of file +main().catch(console.error); diff --git a/tutorials/working_with_standards/python/.env.example b/tutorials/working_with_standards/python/.env.example index f2d3f1f..3c86715 100644 --- a/tutorials/working_with_standards/python/.env.example +++ b/tutorials/working_with_standards/python/.env.example @@ -1,2 +1,3 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file +# Knowledge Graph API credentials - get these from the Learning Commons Platform +API_KEY=your_api_key_here +BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 diff --git a/tutorials/working_with_standards/python/README.md b/tutorials/working_with_standards/python/README.md index ad4d586..6ec508b 100644 --- a/tutorials/working_with_standards/python/README.md +++ b/tutorials/working_with_standards/python/README.md @@ -1,25 +1,18 @@ -Demonstrates how to work with state educational standards data covering: -- **Data Loading**: Reading CSV files with standards frameworks, items, and relationships -- **Hierarchical Queries**: Traversing parent-child relationships in educational standards -- **AI Embeddings**: Generating vector embeddings for semantic search -- **Vector Search**: Performing similarity-based searches on educational standards +Demonstrates how to work with state educational standards data using the Knowledge Graph REST API, covering: +- **API Integration**: Using REST API to query standards frameworks and items +- **Standards Querying**: Filtering standards by jurisdiction, grade level, and type -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/working-with-state-standards) +> **Note:** The API is in limited early release and is only available to some private beta users. Because the API is an early release, current users should expect occasional breaking changes. ## Prerequisites - Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFramework.csv` - - `StandardsFrameworkItem.csv` - - `Relationships.csv` +- A Learning Commons Platform account +- An API key generated in the Learning Commons Platform ## Dependencies -- **transformers**: Hugging Face transformers library for embeddings -- **pandas**: Data manipulation and analysis library -- **torch**: PyTorch for deep learning models -- **scikit-learn**: Machine learning library for cosine similarity +- **requests**: HTTP library for API calls - **python-dotenv**: Environment variable management ## Quick Start @@ -35,7 +28,9 @@ Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowled 2. **Set Environment Variables** (create `.env` file): ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files + # Knowledge Graph API credentials - get these from the Learning Commons Platform + API_KEY=your_api_key_here + BASE_URL=https://api.learningcommons.org/knowledge-graph/v0 ``` 3. **Run Tutorial**: diff --git a/tutorials/working_with_standards/python/requirements.txt b/tutorials/working_with_standards/python/requirements.txt index 7d90ef3..fdbb17d 100644 --- a/tutorials/working_with_standards/python/requirements.txt +++ b/tutorials/working_with_standards/python/requirements.txt @@ -1,13 +1,5 @@ -# Core data manipulation and analysis -pandas>=2.0.0 +# HTTP requests +requests>=2.31.0 # Environment variable management python-dotenv>=1.0.0 - -# Machine learning and embeddings -transformers>=4.21.0 -torch>=1.12.0 -scikit-learn>=1.1.0 - -# Optional: For GPU acceleration (uncomment if needed) -# torch-audio>=0.12.0 \ No newline at end of file diff --git a/tutorials/working_with_standards/python/work_with_state_standards.py b/tutorials/working_with_standards/python/work_with_state_standards.py index 0ed7c62..07c0cee 100644 --- a/tutorials/working_with_standards/python/work_with_state_standards.py +++ b/tutorials/working_with_standards/python/work_with_state_standards.py @@ -8,36 +8,29 @@ # Dependencies import os import sys -import json -import pandas as pd -from pathlib import Path +import requests from dotenv import load_dotenv -from transformers import AutoTokenizer, AutoModel -import torch -from sklearn.metrics.pairwise import cosine_similarity # Load environment variables load_dotenv() # Domain Constants -GENERATE_EMBEDDINGS = True MIDDLE_SCHOOL_GRADES = ['6', '7', '8'] -# For this tutorial, we use 'all-MiniLM-L6-v2' which provides good quality embeddings -# for short text. You can substitute any compatible embedding model. -EMBEDDING_MODEL = 'sentence-transformers/all-MiniLM-L6-v2' # Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') +api_key = os.getenv('API_KEY') +base_url = os.getenv('BASE_URL') + +if not api_key: + print('❌ API_KEY environment variable is not set.') sys.exit(1) -data_path = Path(data_dir) -EMBEDDING_FILE_PATH = data_path / 'california_math_embeddings.json' +if not base_url: + print('❌ BASE_URL environment variable is not set.') + sys.exit(1) -# Initialize embedding components (will be loaded on first use) -tokenizer = None -model = None +# Setup headers for API requests +headers = {"x-api-key": api_key} """ @@ -46,358 +39,81 @@ ================================ """ -def load_csv(filename): - """Load and parse CSV file from data directory""" +def make_api_request(endpoint, params=None): + """Make API request to Knowledge Graph API""" try: - file_path = data_path / filename - return pd.read_csv(file_path) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') + url = f"{base_url}{endpoint}" + response = requests.get(url, headers=headers, params=params) + response.raise_for_status() + return response.json() + except requests.exceptions.RequestException as error: + print(f'❌ Error making API request to {endpoint}: {str(error)}') raise error -def find_framework_item(case_identifier_uuid, standards_framework_items_data): - """Find a framework item by UUID""" - matching_items = standards_framework_items_data[ - standards_framework_items_data['caseIdentifierUUID'] == case_identifier_uuid - ] - if len(matching_items) > 0: - return matching_items.iloc[0] - return None - - """ ================================ -STEP 1: LOAD DATA +STEP 2: QUERY FOR STANDARDS DATA ================================ """ -def load_data(): - """Load CSV data files needed for the tutorial""" - standards_frameworks_data = load_csv('StandardsFramework.csv') - standards_framework_items_data = load_csv('StandardsFrameworkItem.csv') - - print('✅ Files loaded from KG CSV files') - print(f' Standards Frameworks: {len(standards_frameworks_data)}') - print(f' Standards Framework Items: {len(standards_framework_items_data)}') - - return {'standards_frameworks_data': standards_frameworks_data, - 'standards_framework_items_data': standards_framework_items_data} +def get_california_framework(): + """Get California math standards framework""" + result = make_api_request( + '/standards-frameworks', + params={ + 'jurisdiction': 'California', + 'academicSubject': 'Mathematics' + } + ) + california_framework = result['data'][0] if result['data'] else None -""" -================================ -STEP 2: QUERY FOR STANDARDS DATA -================================ -""" + print(f'✅ Retrieved California math standards framework:') + if california_framework: + print(california_framework) -def get_math_standards_frameworks(standards_frameworks_data): - """ - Get snapshot of mathematics standards frameworks - - SQL Equivalent: - SELECT "name", "academicSubject", "jurisdiction", "identifier" - FROM standards_framework - WHERE "academicSubject" = 'Mathematics'; - - Cypher Equivalent: - MATCH (sf:StandardsFramework {academicSubject: 'Mathematics'}) - RETURN sf.name, sf.academicSubject, sf.jurisdiction, sf.identifier - """ - math_frameworks = standards_frameworks_data[ - standards_frameworks_data['academicSubject'] == 'Mathematics' - ][['caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject']] - - print(f'✅ Retrieved {len(math_frameworks)} state standard frameworks for math (dataframe):') - print('Sample of first 5 frameworks:') - for i, (_, framework) in enumerate(math_frameworks.head(5).iterrows()): - print(f' {i+1}. {framework["jurisdiction"]}: {framework["name"]}') - - # Get California math framework metadata - # - # SQL: SELECT * - # FROM standards_framework - # WHERE "jurisdiction" = 'California' - # AND "academicSubject" = 'Mathematics'; - # Cypher: MATCH (sf:StandardsFramework {jurisdiction: 'California', academicSubject: 'Mathematics'}) RETURN sf - california_framework_df = standards_frameworks_data[ - (standards_frameworks_data['jurisdiction'] == 'California') & - (standards_frameworks_data['academicSubject'] == 'Mathematics') - ][['caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject']] - - california_framework = california_framework_df.iloc[0] if len(california_framework_df) > 0 else None - - print(f'✅ Retrieved {1 if california_framework is not None else 0} California math standards framework:') - if california_framework is not None: - print(f' UUID: {california_framework["caseIdentifierUUID"]}') - print(f' Name: {california_framework["name"]}') - print(f' Jurisdiction: {california_framework["jurisdiction"]}') - print(f' Academic Subject: {california_framework["academicSubject"]}') - - return {'math_frameworks': math_frameworks, 'california_framework': california_framework} - - -def get_middle_school_standards_groupings(standards_framework_items_data): - """ - Filter for middle school standard groupings from California framework - - SQL Equivalent: - SELECT * - FROM standards_framework_item - WHERE "jurisdiction" = 'California' - AND "academicSubject" = 'Mathematics' - AND "normalizedStatementType" = 'Standard Grouping' - AND ( - EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - ); - - Cypher Equivalent: - MATCH (sfi:StandardsFrameworkItem) - WHERE sfi.jurisdiction = 'California' - AND sfi.academicSubject = 'Mathematics' - AND sfi.normalizedStatementType = 'Standard Grouping' - AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - RETURN sfi; - """ - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - ca_math_items = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard Grouping') - ] - - groupings = ca_math_items[ - ca_math_items['gradeLevel'].apply(has_middle_school_grade) - ][['caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'statementType', 'gradeLevel']] - - print(f'✅ Retrieved {len(groupings)} standard groupings for middle school math in California (dataframe):') - print('Sample of first 5 standard groupings:') - for i, (_, grouping) in enumerate(groupings.head(5).iterrows()): - print(f' {i+1}. {grouping["statementCode"]}: {grouping["description"][:100]}...') - - return groupings + return california_framework -def get_middle_school_standards(standards_framework_items_data): - """ - Get all standards for California middle school mathematics - - SQL Equivalent: - SELECT * - FROM standards_framework_item - WHERE "jurisdiction" = 'California' - AND "academicSubject" = 'Mathematics' - AND "normalizedStatementType" = 'Standard' - AND ( - EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - ); - - Cypher Equivalent: - MATCH (sfi:StandardsFrameworkItem) - WHERE sfi.jurisdiction = 'California' - AND sfi.academicSubject = 'Mathematics' - AND sfi.normalizedStatementType = 'Standard' - AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - RETURN sfi; - """ - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - ca_math_items = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard') - ] - - standards = ca_math_items[ - ca_math_items['gradeLevel'].apply(has_middle_school_grade) - ][['caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'gradeLevel']] - - print(f'✅ Retrieved {len(standards)} standards for California middle school mathematics (dataframe):') - print('Sample of first 5 standards:') - # Show slice from 100-105 like JS version - start_idx = min(100, len(standards)) - end_idx = min(105, len(standards)) - sample_standards = standards.iloc[start_idx:end_idx] - - for i, (_, standard) in enumerate(sample_standards.iterrows(), start_idx + 1): - print(f' {i}. {standard["statementCode"]}: {standard["description"][:100] if pd.notna(standard["description"]) else "No description"}...') - - return standards +def get_middle_school_standards_groupings(framework_uuid): + """Get middle school standard groupings from California framework""" + params = { + 'standardsFrameworkCaseIdentifierUUID': framework_uuid, + 'normalizedStatementType': 'Standard Grouping', + 'gradeLevel': MIDDLE_SCHOOL_GRADES + } + result = make_api_request('/academic-standards', params=params) + groupings = result['data'] -def query_for_standards_data(standards_frameworks_data, standards_framework_items_data): - """Query and organize standards data""" - math_data = get_math_standards_frameworks(standards_frameworks_data) - groupings = get_middle_school_standards_groupings(standards_framework_items_data) - standards = get_middle_school_standards(standards_framework_items_data) - - return { - 'california_framework': math_data['california_framework'], - 'groupings': groupings, - 'standards': standards - } + print(f'✅ Retrieved {len(groupings)} standard groupings for middle school math in California') + for grouping in groupings[:5]: + description = grouping.get('description', 'No description') + truncated = description[:80] + '...' if len(description) > 80 else description + print(f' {grouping.get("statementCode", "N/A")}: {truncated}') + return groupings -""" -================================ -STEP 3: EMBED STANDARD DATA -================================ -""" -def embed_standard_data(standards_framework_items_data): - """Generate embeddings for California middle school mathematics standards""" - - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - # Filter for embedding standards - ca_math_standards = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard') & - (standards_framework_items_data['gradeLevel'].apply(has_middle_school_grade)) & - (standards_framework_items_data['description'].notna()) - ] - - embedding_standards = ca_math_standards.to_dict('records') - - def generate_embeddings(): - """Generate and save embeddings for each standard""" - global tokenizer, model - - results = [] - print(f'🔄 Generating embeddings for {len(embedding_standards)} standards...') - - # Initialize embedding model if not already done - if tokenizer is None or model is None: - print('📥 Loading embedding model (first time only)...') - tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL) - model = AutoModel.from_pretrained(EMBEDDING_MODEL) - print('✅ Embedding model loaded') - - for standard in embedding_standards: - code = standard.get('statementCode') or '(no code)' - - try: - # Generate embedding - inputs = tokenizer(standard['description'], return_tensors='pt', padding=True, truncation=True) - with torch.no_grad(): - outputs = model(**inputs) - # Mean pooling - embedding = outputs.last_hidden_state.mean(dim=1).squeeze().numpy() - - results.append({ - 'caseIdentifierUUID': standard['caseIdentifierUUID'], - 'statementCode': standard['statementCode'], - 'embedding': embedding.tolist() - }) - - print(f'✅ {code}') - except Exception as err: - print(f'❌ {code}: {str(err)}') - raise err - - # Save embeddings to file - with open(EMBEDDING_FILE_PATH, 'w') as f: - json.dump(results, f, indent=2) - print(f'✅ Saved {len(results)} embeddings to {EMBEDDING_FILE_PATH}') - - return generate_embeddings +def get_middle_school_standards(framework_uuid): + """Get all standards for California middle school mathematics""" + params = { + 'standardsFrameworkCaseIdentifierUUID': framework_uuid, + 'normalizedStatementType': 'Standard', + 'gradeLevel': MIDDLE_SCHOOL_GRADES + } + result = make_api_request('/academic-standards', params=params) + standards = result['data'] -""" -================================ -STEP 4: VECTOR SEARCH STANDARD DATA -================================ -""" + print(f'✅ Retrieved {len(standards)} standards for California middle school mathematics') + for standard in standards[:5]: + description = standard.get('description', 'No description') + truncated = description[:80] + '...' if len(description) > 80 else description + print(f' {standard.get("statementCode", "N/A")}: {truncated}') -def vector_search_standard_data(standards_framework_items_data): - """Perform vector search using cosine similarity""" - - def vector_search(query, top_k=5): - global tokenizer, model - - # Initialize embedding model if not already done - if tokenizer is None or model is None: - print('📥 Loading embedding model...') - tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL) - model = AutoModel.from_pretrained(EMBEDDING_MODEL) - print('✅ Embedding model loaded') - - # Generate query embedding - try: - inputs = tokenizer(query, return_tensors='pt', padding=True, truncation=True) - with torch.no_grad(): - outputs = model(**inputs) - query_embedding = outputs.last_hidden_state.mean(dim=1).squeeze().numpy() - except Exception as error: - print(f'❌ Error generating embedding for query "{query}": {str(error)}') - return - - # Load stored embeddings - try: - with open(EMBEDDING_FILE_PATH, 'r') as f: - stored_embeddings = json.load(f) - except Exception as error: - print(f'❌ Error loading embeddings from {EMBEDDING_FILE_PATH}: {str(error)}') - print('💡 Make sure to run the embedding generation step first (Step 3)') - return - - # Calculate similarities and get top results - similarities = [] - for item in stored_embeddings: - similarity = cosine_similarity([query_embedding], [item['embedding']])[0][0] - similarities.append({ - 'caseIdentifierUUID': item['caseIdentifierUUID'], - 'score': float(similarity) - }) - - top_results = sorted(similarities, key=lambda x: x['score'], reverse=True)[:top_k] - - print(f'\nTop {top_k} results for "{query}":\n') - - # Add framework item details to results - for i, result in enumerate(top_results): - framework_item = find_framework_item(result['caseIdentifierUUID'], standards_framework_items_data) - - statement_code = framework_item['statementCode'] if framework_item is not None and pd.notna(framework_item['statementCode']) else '(no code)' - description = framework_item['description'] if framework_item is not None and pd.notna(framework_item['description']) else '(no statement)' - - top_results[i]['statementCode'] = statement_code - top_results[i]['description'] = description - - # Print results in readable format - for i, result in enumerate(top_results, 1): - print(f' {i}. {result["statementCode"]} (Score: {result["score"]:.4f})') - print(f' {result["description"][:200]}...' if len(result["description"]) > 200 else f' {result["description"]}') - print() - - return vector_search + return standards """ @@ -409,26 +125,26 @@ def vector_search(query, top_k=5): def main(): """Main execution function - orchestrates all tutorial steps""" print('\n=== WORKING WITH STATE STANDARDS TUTORIAL ===\n') - - print('🔄 Step 1: Loading data...') - data = load_data() - standards_frameworks_data = data['standards_frameworks_data'] - standards_framework_items_data = data['standards_framework_items_data'] - - print('\n🔄 Step 2: Querying for standards data...') - query_for_standards_data(standards_frameworks_data, standards_framework_items_data) - - print('\n🔄 Step 3: Embedding standard data...') - generate_embeddings = embed_standard_data(standards_framework_items_data) - if GENERATE_EMBEDDINGS: - generate_embeddings() - else: - print('🚫 Embedding generation disabled') - - print('\n🔄 Step 4: Vector searching standard data...') - vector_search = vector_search_standard_data(standards_framework_items_data) - vector_search('linear equations') + + print('🔄 Step 2: Querying for standards data...\n') + + # 1. Get California math standards framework + california_framework = get_california_framework() + + if not california_framework: + print('❌ Could not retrieve California framework') + sys.exit(1) + + framework_uuid = california_framework['caseIdentifierUUID'] + + # 2. Get middle school standard groupings + print() + groupings = get_middle_school_standards_groupings(framework_uuid) + + # 3. Get middle school standards + print() + standards = get_middle_school_standards(framework_uuid) if __name__ == '__main__': - main() \ No newline at end of file + main()