Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tabs test #533

Draft
wants to merge 180 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
180 commits
Select commit Hold shift + click to select a range
6af1ba7
integrate custom docs with new UI
kmoscoe Jul 11, 2024
cd0fe76
more edits
kmoscoe Jul 11, 2024
c45d61b
use website wording for intro
kmoscoe Jul 15, 2024
f5cbbca
fix numbering in table
kmoscoe Jul 15, 2024
90c57e7
Merge branch 'master' into custom_dc
kmoscoe Jul 15, 2024
57ca62f
rename and some edits
kmoscoe Jul 15, 2024
2277cb1
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Jul 15, 2024
fb33722
rename manage_repo file, per Bo
kmoscoe Jul 16, 2024
0ebdd67
Merge.
kmoscoe Jul 17, 2024
e3148c4
merge
kmoscoe Jul 17, 2024
ee5f580
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 22, 2024
5993fb7
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Jul 22, 2024
61ce06d
Merge branch 'custom_dc'
kmoscoe Jul 22, 2024
2b37137
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 22, 2024
37f3e87
formatting edits
kmoscoe Jul 23, 2024
d83db88
updates per Keyur's feedback
kmoscoe Jul 24, 2024
03b906f
Fix typos
kmoscoe Jul 24, 2024
03538af
fix nav order
kmoscoe Jul 24, 2024
d7fb58f
fix link to API key request form
kmoscoe Jul 24, 2024
c746a7e
update form link
kmoscoe Jul 24, 2024
d5e04af
update key request form and output dir env var
kmoscoe Jul 24, 2024
0183130
Merge branch 'master' into custom_dc
kmoscoe Jul 25, 2024
b6f53cf
Revert to gerund
kmoscoe Jul 26, 2024
e538860
Merge branch 'master' into master
kmoscoe Jul 26, 2024
d368ba4
new troubleshooting entry
kmoscoe Jul 31, 2024
db99da0
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 31, 2024
535962c
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Jul 31, 2024
fd071e9
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Jul 31, 2024
771ccca
fix typo
kmoscoe Jul 31, 2024
7f2bccf
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 1, 2024
7286ff2
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 5, 2024
6e61dac
new data container procedures
kmoscoe Aug 5, 2024
24ea3c4
more work
kmoscoe Aug 5, 2024
03b9786
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 6, 2024
534461b
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 6, 2024
d4d81c8
more work
kmoscoe Aug 6, 2024
5d52e4b
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 6, 2024
8054ac7
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 6, 2024
166e7e1
complete data draft
kmoscoe Aug 6, 2024
079fe62
more changes
kmoscoe Aug 7, 2024
5ccaba5
more changes
kmoscoe Aug 8, 2024
6d80bcc
more revisions
kmoscoe Aug 8, 2024
21917fa
update troubleshooting doc etc.
kmoscoe Aug 9, 2024
586973a
new version of diagrams
kmoscoe Aug 12, 2024
c207f6b
remove data loading problems troubleshooting entry; can't reproduce
kmoscoe Aug 12, 2024
6fa91d3
revert title change
kmoscoe Aug 12, 2024
9cde771
add example for not mixing entity types
kmoscoe Aug 12, 2024
83bbaf2
changes from Keyur
kmoscoe Aug 12, 2024
86800ba
add screenshots for GCP, and related changes
kmoscoe Aug 13, 2024
1a23559
fixed one image
kmoscoe Aug 13, 2024
b4e6bd4
added screenshots for Cloud Run service
kmoscoe Aug 13, 2024
a68f8b4
resize images
kmoscoe Aug 13, 2024
05b1894
more changes from Keyur
kmoscoe Aug 13, 2024
c606037
fix a tiny error
kmoscoe Aug 13, 2024
9bb0355
delete unused images
kmoscoe Aug 13, 2024
c497e03
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 13, 2024
914bc76
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Aug 13, 2024
75e9b41
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 13, 2024
edcb5bc
fix missing dash
kmoscoe Aug 13, 2024
e8ca73e
update services container name
kmoscoe Aug 14, 2024
cab4484
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 14, 2024
529b734
Merge branch 'master' into custom_dc
kmoscoe Aug 14, 2024
740a1e6
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Aug 14, 2024
9bf96d7
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 14, 2024
ae46ea2
update build file
kmoscoe Aug 14, 2024
4ce0fb9
adjust build command
kmoscoe Aug 14, 2024
68a3f51
Revert "adjust build command"
kmoscoe Aug 14, 2024
d8f1ab7
update docker file
kmoscoe Aug 14, 2024
d67d280
more fixes
kmoscoe Aug 14, 2024
15a0a18
one last fix
kmoscoe Aug 14, 2024
5547194
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 15, 2024
0f3a1df
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Aug 15, 2024
8c79911
merge
kmoscoe Aug 15, 2024
437948c
make links to Cloud Console open in a new page
kmoscoe Aug 15, 2024
f225640
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Aug 15, 2024
1b3fcdf
fixes to quickstart suggested by Prem
kmoscoe Aug 16, 2024
4677644
one more change
kmoscoe Aug 16, 2024
ac3ceec
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 16, 2024
ce66faf
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Aug 16, 2024
8b34a52
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 16, 2024
27f8528
Merge branch 'master' into custom_dc
kmoscoe Aug 21, 2024
cba3fe9
change from Keyur
kmoscoe Aug 21, 2024
53106b9
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Aug 21, 2024
5e68a9f
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 26, 2024
17eb420
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 26, 2024
cf820e0
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 26, 2024
41aacce
revise procedure
kmoscoe Aug 26, 2024
1cfe86c
Merge branch 'master' into custom_dc
kmoscoe Aug 27, 2024
e204fbd
merge
kmoscoe Aug 27, 2024
8b78d9f
merge
kmoscoe Aug 27, 2024
c85c376
add brief explanation of data model to quickstart
kmoscoe Aug 28, 2024
df1a8dd
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 28, 2024
922a139
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 28, 2024
39f3309
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Aug 28, 2024
5de4350
slight wording tweak
kmoscoe Aug 28, 2024
d82a728
incorporate feedback from Keyur
kmoscoe Aug 29, 2024
7f7903e
remove erroneous edit
kmoscoe Aug 29, 2024
fcaa5c5
correct missing text
kmoscoe Aug 29, 2024
c752d06
more work on tasks for finding stuff
kmoscoe Aug 29, 2024
0fcf34c
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Sep 3, 2024
1f6c492
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 3, 2024
dee1e6a
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 4, 2024
8bc313f
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Sep 4, 2024
0678683
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 4, 2024
91616c4
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Sep 4, 2024
c322763
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 4, 2024
627fb89
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Sep 4, 2024
6d8c42d
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Sep 9, 2024
cc6d9e3
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 10, 2024
e588231
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 10, 2024
837a782
Merge branch 'master' of https://github.com/kmoscoe/docsite
kmoscoe Sep 10, 2024
bde4843
merge
kmoscoe Sep 10, 2024
0da5d90
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 11, 2024
d8a639c
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 11, 2024
b95565d
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 13, 2024
50e490a
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 13, 2024
7f1b9d1
update to use env.sample
kmoscoe Sep 13, 2024
5137208
typo
kmoscoe Sep 13, 2024
081f883
typo
kmoscoe Sep 13, 2024
47daec0
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Sep 13, 2024
7c24813
get file back in head shape
kmoscoe Sep 13, 2024
89c04d6
fix file name
kmoscoe Sep 13, 2024
14a22e4
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 16, 2024
609fba9
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 16, 2024
a2d42cf
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 16, 2024
615e7c3
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 16, 2024
88d4488
add more detail about data security
kmoscoe Sep 16, 2024
c34a701
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Sep 16, 2024
93db5f4
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Sep 16, 2024
abd9a2c
fix typo
kmoscoe Sep 16, 2024
32a8520
corrections from Keyur
kmoscoe Sep 18, 2024
37f8675
fix other mention of SQL queries
kmoscoe Sep 18, 2024
b0873ea
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 18, 2024
c4ce90d
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 18, 2024
6521e3b
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 24, 2024
fed02b2
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 26, 2024
702739f
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 27, 2024
4692a99
add both data directories to docker run commands
kmoscoe Sep 27, 2024
751ae8c
remove extra slash
kmoscoe Sep 27, 2024
efdddbb
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 27, 2024
ef8c380
Merge branch 'datacommonsorg:master' into master
kmoscoe Sep 27, 2024
f923e31
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Sep 30, 2024
aa54bd7
update feedback links
kmoscoe Sep 30, 2024
83eccdb
tiny tweaks
kmoscoe Sep 30, 2024
0ce6546
fixes from Hannah
kmoscoe Sep 30, 2024
44aa2e7
fix grammar
kmoscoe Sep 30, 2024
c4ee531
remove redundant text
kmoscoe Sep 30, 2024
5601a1e
add link for data requests
kmoscoe Oct 1, 2024
693c2ab
second try
kmoscoe Oct 1, 2024
332246b
fix link
kmoscoe Oct 1, 2024
0030171
add template parameter back
kmoscoe Oct 1, 2024
497b761
add link to issue tracker docs
kmoscoe Oct 1, 2024
7b92e0f
feedback from Keyur
kmoscoe Oct 1, 2024
f25175c
fix template parameter
kmoscoe Oct 1, 2024
bdca4ff
add doc for observation properties
kmoscoe Oct 9, 2024
5a365be
Merge branch 'datacommonsorg:master' into master
kmoscoe Oct 9, 2024
af35c2d
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Oct 9, 2024
39de516
more edits
kmoscoe Oct 9, 2024
e206c7f
corrections from Keyur
kmoscoe Oct 9, 2024
b066852
one more change from Keyur
kmoscoe Oct 9, 2024
088e3f9
Merge branch 'datacommonsorg:master' into master
kmoscoe Oct 28, 2024
9fa99fd
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Oct 28, 2024
2740f0d
Add update schema option.
kmoscoe Oct 28, 2024
7b0de60
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Oct 28, 2024
61bc24c
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Oct 28, 2024
f765cf1
wording fixes
kmoscoe Oct 28, 2024
1dc0f90
add CLI procedures
kmoscoe Nov 4, 2024
38fc2b9
Merge branch 'datacommonsorg:master' into master
kmoscoe Nov 4, 2024
c2c3a3e
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Nov 4, 2024
230def4
fix troubleshooting doc
kmoscoe Nov 4, 2024
ce2b42c
fix custom data
kmoscoe Nov 4, 2024
eee5569
fix troubleshooting again
kmoscoe Nov 4, 2024
9e84c70
added procedures for using Secret Manager
kmoscoe Nov 5, 2024
00a9645
small fixes
kmoscoe Nov 5, 2024
6bedcce
first stab at using tabs for procedures
kmoscoe Nov 5, 2024
2b069e3
Merge branch 'datacommonsorg:master' into master
kmoscoe Nov 5, 2024
d963f0f
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Nov 5, 2024
1c7c9a7
testing
kmoscoe Nov 5, 2024
6febed3
fix some out of sync text
kmoscoe Nov 5, 2024
115f8fe
more work on styling
kmoscoe Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions _sass/custom_dc.scss
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,17 @@
padding-bottom: 20px;
white-space: pre-wrap;
}

#console.api-tabcontent, #cli.api-tabcontent, #clia.api-tabcontent, #consolea.api-tabcontent, {
font-family: inherit !important;
font-size: inherit !important;
color: inherit !important;
padding-bottom: inherit !important;
margin-bottom: 5px;
white-space: inherit !important;
line-height: inherit !important;
ol {
padding-top: 25px;
padding-bottom: 10px;
}
}
85 changes: 51 additions & 34 deletions custom_dc/data_cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,19 +122,30 @@ Now set environment variables:

As you are iterating on changes to the source CSV and JSON files, you can re-upload them at any time, either overwriting existing files or creating new folders. If you want versioned snapshots, we recommend that you create a new subfolder and store the latest version of the files there. If you prefer to simply incrementally update, you can simply overwrite files in a pre-existing folder. Creating new subfolders is slower but safer. Overwriting files is faster but riskier.

To upload data using the Cloud Console:

1. Go to [https://console.cloud.google.com/storage/browse](https://console.cloud.google.com/storage/browse){: target="_blank"} and select your custom Data Commons bucket.
1. Navigate to the folder you created in the earlier step.
1. Click **Upload Files**, and select all your CSV files and `config.json`.

To upload data using the command line:

1. Navigate to your local "input" directory where your source files are located.
1. Run the following command:
<pre>
gcloud storage cp config.json *.csv gs://<var>BUCKET_NAME</var>/<var>FOLDER_PATH</var>
</pre>
<div class="api-tab">
<button id="get-button" class="api-tablink" onclick="openTab(event, 'console')">Cloud Console
</button>
<button id="post-button" class="api-tablink" onclick="openTab(event, 'cli')">gcloud CLI
</button>
</div>

<div id="console" class="api-tabcontent">
<ol>
<li>Go to <a href="https://console.cloud.google.com/storage/browse" target="_blank">https://console.cloud.google.com/storage/browse</a> and select your custom Data Commons bucket.</li>
<li>Navigate to the folder you created in the earlier step.</li>
<li>Click <b>Upload Files</b>, and select all your CSV files and <code>config.json</code>.</li>
</ol>
</div>

<div id="cli" class="api-tabcontent">
<ol>
<li>Navigate to your local "input" directory where your source files are located.</li>
<li>Run the following command:
<pre>gcloud storage cp config.json *.csv gs://<var>BUCKET_NAME</var>/<var>FOLDER_PATH</var></pre>
</li>
</ol>
</div>
<script src="/assets/js/api-doc-tabs.js"></script>

> **Note:** Do not upload the local `datacommons` subdirectory or its files.

Expand All @@ -146,23 +157,32 @@ Now that everything is configured, and you have uploaded your data in Google Clo

Every time you upload new input CSV or JSON files to Google Cloud Storage, you will need to rerun the job.

To run the job using the Cloud Console:

1. Go to [https://console.cloud.google.com/run/jobs](https://console.cloud.google.com/run/jobs){: target="_blank"} for your project.
1. From the list of jobs, click the link of the "datacommons-data" job you created above.
1. Optionally, if you have received a `SQL check failed` error when previously trying to start the container, and would like to minimize startup time, click **Execute with overrides** and click **Add variable** to set a new variable with name `DATA_RUN_MODE` and value `schemaupdate`.
1. Click **Execute**. It will take several minutes for the job to run. You can click the **Logs** tab to view the progress.

To run the job using the command line:

1. From any local directory, run the following command:
<pre>
gcloud run jobs execute <var>JOB_NAME</var>
</pre>
1. To view the progress of the job, run the following command:
<pre>
gcloud beta run jobs logs tail <var>JOB_NAME</var>
</pre>
<div class="api-tab">
<button id="get-button" class="api-tablink" onclick="openTab(event, 'consolea')">Cloud Console
</button>
<button id="post-button" class="api-tablink" onclick="openTab(event, 'clia')">gcloud CLI
</button>
</div>

<div id="consolea" class="api-tabcontent">
<ol>
<li>Go to <a href="https://console.cloud.google.com/run/jobs" target="_blank">https://console.cloud.google.com/run/jobs</a> for your project.</li>
<li>From the list of jobs, click the link of the "datacommons-data" job you created above.</li>
<li>Optionally, if you have received a <code>SQL check failed</code> error when previously trying to start the container, and would like to minimize startup time, click <b>Execute with overrides</b> and click <b>Add variable</b> to set a new variable with name <code>DATA_RUN_MODE</code> and value <code>schemaupdate</code>.</li>
<li>Click <b>Execute</b>. It will take several minutes for the job to run. You can click the <b>Logs</b> tab to view the progress. </li>
</ol>
</div>

<div id="clia" class="api-tabcontent">
<ol>
<li>From any local directory, run the following command:
<pre>gcloud run jobs execute <var>JOB_NAME</var></pre></li>
<li>To view the progress of the job, run the following command:
<pre>gcloud beta run jobs logs tail <var>JOB_NAME</var></pre>
</li>
</ol>
</div>
<script src="/assets/js/api-doc-tabs.js"></script>

When it completes, to verify that the data has been loaded correctly, see [Inspect the Cloud SQL database](#inspect-sql).

Expand Down Expand Up @@ -246,17 +266,14 @@ docker run \
-v <var>OUTPUT_DIRECTORY</var>:<var>OUTPUT_DIRECTORY</var> \
-e GOOGLE_APPLICATION_CREDENTIALS=/gcp/creds.json \
-v $HOME/.config/gcloud/application_default_credentials.json:/gcp/creds.json:ro \
[-e DATA_RUN_MODE=schemaupdate \]
gcr.io/datcom-ci/datacommons-data:<var>VERSION</var>
</pre>

The version is `latest` or `stable`.

> Note: The DATA_RUN_MODE flag is only relevant if you have previously received a `SQL check failed` error, and is optional to speed up the startup process.

To verify that the data is correctly created in your Cloud SQL database, use the procedure in [Inspect the Cloud SQL database](#inspect-sql) above.

#### Run the data management Docker container in schema update mode
#### Optional: Run the data management Docker container in schema update mode

If you have tried to start a container, and have received a `SQL check failed` error, this indicates that a database schema update is needed. You need to restart the data management container, and you can specify an additional, optional, flag, `DATA_RUN_MODE` to miminize the startup time.

Expand Down
8 changes: 3 additions & 5 deletions custom_dc/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,13 @@ An API key is required to authorize requests from your site to the base Data Com
1. Open a terminal window, and go to a directory to which you would like to download the Data Commons repository.
1. Clone the website Data Commons repository:

<pre>
git clone https://github.com/datacommonsorg/website.git [<var>DIRECTORY</var>]
</pre>
If you don't specify a directory name, this creates a local `website` subdirectory. If you specify a directory name, all files are created under that directory, without a `website` subdirectory.
<pre>git clone https://github.com/datacommonsorg/website.git [<var>DIRECTORY</var>]</pre>
If you don't specify a directory name, this creates a local `website` subdirectory. If you specify a directory name, all files are created under that directory, without a `website` subdirectory.

When the downloads are complete, navigate to the root directory of the repo (e.g. `website`). References to various files and commands in these procedures are relative to this root.

<pre>
cd website
cd website | <var>DIRECTORY</var>
</pre>

### Set environment variables {#env-vars}
Expand Down