Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate DAG loading to Kubernetes Jobs and Enable Storage and KV where possible. #239

Merged
merged 87 commits into from
Nov 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
f3fabdd
Reworking Storage Secrets
danielscholl Nov 7, 2024
0f56853
Adding workload identity to airflow.
danielscholl Nov 7, 2024
34d685e
Moved to disks.
danielscholl Nov 7, 2024
2613374
Fixed storage class
danielscholl Nov 7, 2024
f84dcce
Adding airflow dag job copy.
danielscholl Nov 7, 2024
6f71272
Feature flag for nsg lockdown.
danielscholl Nov 7, 2024
56b0c28
Adding CSV DAG
danielscholl Nov 7, 2024
a970f8f
Updated
danielscholl Nov 7, 2024
4c0b3db
updated job
danielscholl Nov 7, 2024
6163049
updated job
danielscholl Nov 7, 2024
b389f48
updated job
danielscholl Nov 7, 2024
b1a9c71
Updated job
danielscholl Nov 7, 2024
d225f17
Updated chart
danielscholl Nov 7, 2024
55a5dab
updated chart
danielscholl Nov 7, 2024
3428353
Moved to airflow-dags chart
danielscholl Nov 7, 2024
6fc9b72
Updated chart
danielscholl Nov 7, 2024
58d26e9
Updated chart
danielscholl Nov 8, 2024
ef30e8c
Updated chart
danielscholl Nov 8, 2024
a9f2d79
Updated chart
danielscholl Nov 8, 2024
8d4c869
Updated Chart
danielscholl Nov 8, 2024
a5cba59
Reworking configmap for airflow
danielscholl Nov 8, 2024
00b4515
Fixed yaml
danielscholl Nov 8, 2024
6812816
configmap name
danielscholl Nov 8, 2024
8ae6084
configmap name
danielscholl Nov 8, 2024
6f435e4
Updated helmrelease
danielscholl Nov 8, 2024
18bbaba
Updated dag chart
danielscholl Nov 8, 2024
dec0fa2
Updated chart
danielscholl Nov 8, 2024
70c5c34
Updated chart
danielscholl Nov 8, 2024
5d871f9
Updated chart
danielscholl Nov 8, 2024
a10424a
Updated chart
danielscholl Nov 8, 2024
d3eefb0
Updated chart
danielscholl Nov 8, 2024
b8350e2
fixed script
danielscholl Nov 8, 2024
9c33f71
fixes
danielscholl Nov 8, 2024
bb08f6a
fixes
danielscholl Nov 8, 2024
2da1342
fixes
danielscholl Nov 8, 2024
77e6819
fixes
danielscholl Nov 8, 2024
82de8b6
fixes
danielscholl Nov 8, 2024
a9abd26
fixes
danielscholl Nov 8, 2024
4e0450a
Updated chart
danielscholl Nov 9, 2024
04ecce7
Updated chart
danielscholl Nov 9, 2024
5881fc3
Updated job
danielscholl Nov 9, 2024
89e055e
Fixed script
danielscholl Nov 9, 2024
3721614
Updated script
danielscholl Nov 9, 2024
0305f20
Updated script
danielscholl Nov 9, 2024
684c7c5
Updated script
danielscholl Nov 9, 2024
6b28d00
Updated script
danielscholl Nov 9, 2024
11ac3ee
Updated script
danielscholl Nov 9, 2024
a7a10b9
Updated the Readme
danielscholl Nov 9, 2024
976f5e7
Enabled private software.
danielscholl Nov 9, 2024
2db5ee9
Fixed boolean error.
danielscholl Nov 9, 2024
bbfae30
Bug fix boolean.
danielscholl Nov 9, 2024
913ce26
Added Legal COO kubernetes Job.
danielscholl Nov 10, 2024
2680602
Added Legal COO kubernetes Job.
danielscholl Nov 10, 2024
acb316c
Fixing helm chart.
danielscholl Nov 10, 2024
5a660a4
Fixing helm chart.
danielscholl Nov 10, 2024
dad72c5
Fixing helm chart.
danielscholl Nov 10, 2024
84df916
Updated the chart.
danielscholl Nov 10, 2024
40dace2
Updated the chart.
danielscholl Nov 10, 2024
f1f833c
Updated the chart.
danielscholl Nov 10, 2024
36364f5
Updated the chart.
danielscholl Nov 10, 2024
a98af3f
Updated the chart.
danielscholl Nov 10, 2024
b4a4853
Updated the chart.
danielscholl Nov 10, 2024
ced3bb0
Updated the chart.
danielscholl Nov 10, 2024
013c980
Updated the resource gruop
danielscholl Nov 10, 2024
9e1204b
Updated the resource gruop
danielscholl Nov 11, 2024
73e99ba
Updated the chart
danielscholl Nov 11, 2024
0f48d4a
Moved resource group to first configmap.
danielscholl Nov 11, 2024
803f3d0
Moved resource group to first configmap.
danielscholl Nov 11, 2024
79412bd
Moved resource group to first configmap.
danielscholl Nov 11, 2024
9465207
Updated
danielscholl Nov 11, 2024
c317fd8
Updated
danielscholl Nov 11, 2024
326f0be
Updated
danielscholl Nov 11, 2024
6bcd319
Updated
danielscholl Nov 11, 2024
7eac566
Updated
danielscholl Nov 11, 2024
6ac9177
Updated
danielscholl Nov 11, 2024
1286f41
Updated
danielscholl Nov 11, 2024
50e400c
Updated
danielscholl Nov 11, 2024
980a167
Updated
danielscholl Nov 11, 2024
d73ec6c
Updated
danielscholl Nov 11, 2024
a4af71e
Added target namespace.
danielscholl Nov 11, 2024
53eed73
Split charts
danielscholl Nov 11, 2024
7ebf437
Fixed helm chart
danielscholl Nov 11, 2024
f3eb5fd
Updated
danielscholl Nov 11, 2024
0485996
Updated Chart
danielscholl Nov 11, 2024
bbf14ee
Updated Chart
danielscholl Nov 11, 2024
e892552
Updated Chart
danielscholl Nov 12, 2024
e7a709c
Updated
danielscholl Nov 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixes
danielscholl committed Nov 8, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit a9abd265c06e536279d2e42af0ce84528a71f313
41 changes: 39 additions & 2 deletions charts/airflow-dags/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -51,6 +51,9 @@ app.kubernetes.io/name: {{ include "airflow-dags.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Search and Replace Configuration
*/}}
{{- define "airflow-dags.searchAndReplace" -}}
[
{
@@ -60,5 +63,39 @@ app.kubernetes.io/instance: {{ .Release.Name }}
"sidecar.istio.io/inject": "false"
},
"labels": {
"a
</rewritten_file>
"aadpodidbinding": "osdu-identity"
}
}
},
{
"find": "{| ENV_VARS or {} |}",
"replace": {
"AZURE_CLIENT_ID": {{ .Values.clientId | quote }},
"AZURE_CLIENT_SECRET": {{ .Values.secrets.client_key | quote }},
"AZURE_TENANT_ID": {{ .Values.tenantId | quote }},
"KEYVAULT_URI": {{ .Values.keyvaultUri | quote }},
"aad_client_id": {{ .Values.clientId | quote }},
"appinsights_key": {{ .Values.appInsightsKey | quote }},
"azure_paas_podidentity_isEnabled": "false",
"file_service_endpoint": "http://file.osdu-core.svc.cluster.local/api/file/v2",
"partition_service_endpoint": "http://partition.osdu-core.svc.cluster.local/api/partition/v1",
"schema_service_endpoint": "http://schema.osdu-core.svc.cluster.local/api/schema-service/v1",
"search_service_endpoint": "http://search.osdu-core.svc.cluster.local/api/search/v2",
"storage_service_endpoint": "http://storage.osdu-core.svc.cluster.local/api/storage/v2",
"unit_service_endpoint": "http://unit.osdu-core.svc.cluster.local/api/unit/v2/unit/symbol"
}
},
{
"find": "{| DAG_NAME |}",
"replace": "csv-parser"
},
{
"find": "{| DOCKER_IMAGE |}",
"replace": "community.opengroup.org:5555/osdu/platform/data-flow/ingestion/csv-parser/csv-parser-v0-27-0-azure-1:60747714ac490be0defe8f3e821497b3cce03390"
},
{
"find": "{| NAMESPACE |}",
"replace": "airflow"
}
]
{{- end }}