Skip to content

54 update metjson2db to interact with object storage #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ian-noaa wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

54 update metjson2db to interact with object storage #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d07ef77
manifest getting create from tar.gz
Oct 29, 2025
ecbdec7
tar header extract is working
Nov 12, 2025
c4ec60b
read from s3, add manifes, write back to s3 is now working
Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,8 @@ go run ./cmd/... -c ~/credentials -l ./load_spec.json -i /Users/gopa.padmanabhan
# recursive with file pattern regex match
go run ./cmd/... -c ~/credentials -l ./load_spec.json -I /Users/gopa.padmanabhan/scratch/data/MET/tc_data/tc_data/ -r ".tcst"
go run ./cmd/... -c ~/credentials.MET -d gopa01 -l ./load_spec.json -I /home/amb-verif/MET_data/tc_data/ -r ".tcst"
# load data from S3, transform, write back to S3
go run ./cmd/... -c ~/credentials.MET -d gopa01 -l ./load_spec.json -b s3://metin/s3test.jar -B s3://metout/s3test.jar
# if -f,-F,-i OR -I options are specified, ignores load_spec input files

# update metadata
Expand Down
11 changes: 11 additions & 0 deletions cmd/metjson2db/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@
flag.StringVar(&inputFile, "f", "", "stat file full path")
var inputFiles []string

// s3://bucket/path/to/file.zip (b/B for s3 bundle)
var s3inputFile string
flag.StringVar(&s3inputFile, "b", "", "s3 stat zip input path")
var s3outputFile string
flag.StringVar(&s3outputFile, "B", "", "s3 stat zip output path")

var inputFolder string
flag.StringVar(&inputFolder, "i", "", "input stat files folder")

Expand Down Expand Up @@ -99,6 +105,11 @@
return
}

if len(s3inputFile) > 0 && len(s3outputFile) > 0 {
core.ProcessS3Files(s3inputFile, s3outputFile, nil)

Check failure on line 109 in cmd/metjson2db/main.go

View workflow job for this annotation

GitHub Actions / lint 

Error return value of `core.ProcessS3Files` is not checked (errcheck)
return
}

if len(state.LoadSpec.DatasetName) > 10 {
slog.Error("Dataset name must be less than 10 characters!")
return
Expand Down
45 changes: 45 additions & 0 deletions docs/s3.doc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// from URL:
// https://stackoverflow.com/questions/49266516/reading-files-from-aws-s3-in-golang


go install github.com/seaweedfs/seaweedfs/weed@latest
go/bin/weed server -dir=~/scratch/ -s3
https://github.com/seaweedfs/seaweedfs/wiki/s3cmd-with-SeaweedFS

brew install localstack/tap/localstack-cli
brew install rancher
https://docs.localstack.cloud/aws/integrations/containers/rancher-desktop/
ls /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/bin
/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/bin/rdctl shutdown
/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/bin/rdctl start

localstack start
https://docs.localstack.cloud/aws/getting-started/quickstart/
brew install awscli-local
brew install awscli

--Create buckets
awslocal s3 mb s3://gopa0
awslocal s3 mb s3://gopa1
awslocal s3 mb s3://metin
awslocal s3 mb s3://metout
awslocal s3 ls
aws --profile localstack --endpoint-url=http://localhost:4566 s3 ls

-- Add item to bucket
awslocal s3 cp /Users/gopa.padmanabhan/git/work/MATS.gopa/GoS3Test/test_data/test_data_0.txt s3://gopa0/test_data_0.txt
awslocal s3 cp /Users/gopa.padmanabhan/git/dtcenter/METjson2db/test_data/testTar0.tar s3://metin/testTar0.tar
awslocal s3 ls metin

$cat ~/.aws/credentials
[localstack]
aws_access_key_id=test
aws_secret_access_key=test

$export AWS_ACCESS_KEY_ID=test
$export AWS_SECRET_ACCESS_KEY=test
$export AWS_DEFAULT_REGION=us-east-1

go mod tidy
go run .

Binary file added example.tar.gz
View file
Open in desktop
Binary file not shown.
19 changes: 19 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,25 @@ require (
)

require (
github.com/aws/aws-sdk-go v1.55.8 // indirect
github.com/aws/aws-sdk-go-v2 v1.39.6 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 // indirect
github.com/aws/aws-sdk-go-v2/config v1.31.17 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.18.21 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.13 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.90.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1 // indirect
github.com/aws/smithy-go v1.23.2 // indirect
github.com/couchbase/gocbcore/v10 v10.7.0 // indirect
github.com/couchbase/gocbcoreps v0.1.3 // indirect
github.com/couchbase/goprotostellar v1.0.2 // indirect
Expand Down
38 changes: 38 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,43 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.39.6 h1:2JrPCVgWJm7bm83BDwY5z8ietmeJUbh3O2ACnn+Xsqk=
github.com/aws/aws-sdk-go-v2 v1.39.6/go.mod h1:c9pm7VwuW0UPxAEYGyTmyurVcNrbF6Rt/wixFqDhcjE=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 h1:DHctwEM8P8iTXFxC/QK0MRjwEpWQeM9yzidCRjldUz0=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3/go.mod h1:xdCzcZEtnSTKVDOmUZs4l/j3pSV6rpo1WXl5ugNsL8Y=
github.com/aws/aws-sdk-go-v2/config v1.31.17 h1:QFl8lL6RgakNK86vusim14P2k8BFSxjvUkcWLDjgz9Y=
github.com/aws/aws-sdk-go-v2/config v1.31.17/go.mod h1:V8P7ILjp/Uef/aX8TjGk6OHZN6IKPM5YW6S78QnRD5c=
github.com/aws/aws-sdk-go-v2/credentials v1.18.21 h1:56HGpsgnmD+2/KpG0ikvvR8+3v3COCwaF4r+oWwOeNA=
github.com/aws/aws-sdk-go-v2/credentials v1.18.21/go.mod h1:3YELwedmQbw7cXNaII2Wywd+YY58AmLPwX4LzARgmmA=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.13 h1:T1brd5dR3/fzNFAQch/iBKeX07/ffu/cLu+q+RuzEWk=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.13/go.mod h1:Peg/GBAQ6JDt+RoBf4meB1wylmAipb7Kg2ZFakZTlwk=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.13 h1:a+8/MLcWlIxo1lF9xaGt3J/u3yOZx+CdSveSNwjhD40=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.13/go.mod h1:oGnKwIYZ4XttyU2JWxFrwvhF6YKiK/9/wmE3v3Iu9K8=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.13 h1:HBSI2kDkMdWz4ZM7FjwE7e/pWDEZ+nR95x8Ztet1ooY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.13/go.mod h1:YE94ZoDArI7awZqJzBAZ3PDD2zSfuP7w6P2knOzIn8M=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.13 h1:eg/WYAa12vqTphzIdWMzqYRVKKnCboVPRlvaybNCqPA=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.13/go.mod h1:/FDdxWhz1486obGrKKC1HONd7krpk38LBt+dutLcN9k=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 h1:x2Ibm/Af8Fi+BH+Hsn9TXGdT+hKbDd5XOTZxTMxDk7o=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3/go.mod h1:IW1jwyrQgMdhisceG8fQLmQIydcT/jWY21rFhzgaKwo=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.4 h1:NvMjwvv8hpGUILarKw7Z4Q0w1H9anXKsesMxtw++MA4=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.4/go.mod h1:455WPHSwaGj2waRSpQp7TsnpOnBfw8iDfPfbwl7KPJE=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.13 h1:kDqdFvMY4AtKoACfzIGD8A0+hbT41KTKF//gq7jITfM=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.13/go.mod h1:lmKuogqSU3HzQCwZ9ZtcqOc5XGMqtDK7OIc2+DxiUEg=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.13 h1:zhBJXdhWIFZ1acfDYIhu4+LCzdUS2Vbcum7D01dXlHQ=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.13/go.mod h1:JaaOeCE368qn2Hzi3sEzY6FgAZVCIYcC2nwbro2QCh8=
github.com/aws/aws-sdk-go-v2/service/s3 v1.90.0 h1:ef6gIJR+xv/JQWwpa5FYirzoQctfSJm7tuDe3SZsUf8=
github.com/aws/aws-sdk-go-v2/service/s3 v1.90.0/go.mod h1:+wArOOrcHUevqdto9k1tKOF5++YTe9JEcPSc9Tx2ZSw=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.1 h1:0JPwLz1J+5lEOfy/g0SURC9cxhbQ1lIMHMa+AHZSzz0=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.1/go.mod h1:fKvyjJcz63iL/ftA6RaM8sRCtN4r4zl4tjL3qw5ec7k=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5 h1:OWs0/j2UYR5LOGi88sD5/lhN6TDLG6SfA7CqsQO9zF0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5/go.mod h1:klO+ejMvYsB4QATfEOIXk8WAEwN4N0aBfJpvC+5SZBo=
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1 h1:mLlUgHn02ue8whiR4BmxxGJLR2gwU6s6ZzJ5wDamBUs=
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1/go.mod h1:E19xDjpzPZC7LS2knI9E6BaRFDK43Eul7vd6rSq2HWk=
github.com/aws/smithy-go v1.23.2 h1:Crv0eatJUQhaManss33hS5r40CG3ZFH+21XSkqMrIUM=
github.com/aws/smithy-go v1.23.2/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
Expand Down
2 changes: 1 addition & 1 deletion load_spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"runMode0" : "CREATE_JSON_DOC_ARCHIVE",
"runMode" : "DIRECT_LOAD_TO_DB",
"runMode_doc" : ["DIRECT_LOAD_TO_DB", "CREATE_JSON_DOC_ARCHIVE", "METADATA_UPDATE"],
"jsonArchiveFilePathAndPrefix" :"/scratch/METjson2db_out_",
"jsonArchiveFilePathAndPrefix" :"/Users/gopa.padmanabhan/scratch/METjson2db_out_",
"maxDocIdLength": 200,
"progressiveLargeDocIdPolicies": [ "SHA256", "REJECT_WIH_ERROR_LOG"],
"overWriteData": true,
Expand Down
21 changes: 21 additions & 0 deletions manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"timestamp": "2025-10-29T10:36:46.172154-06:00",
"files": [
{
"name": "my_dir",
"path": "my_dir/",
"size": 0,
"mode": 493,
"modTime": "2025-10-29T10:36:46-06:00",
"isDir": true
},
{
"name": "file1.txt",
"path": "my_dir/file1.txt",
"size": 11,
"mode": 420,
"modTime": "2025-10-29T10:36:46-06:00",
"isDir": false
}
]
}
2 changes: 1 addition & 1 deletion pkg/core/procesdInput.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ func ProcessInputFiles(inputFiles []string, preDbLoadCallback func()) error {
}
case "CREATE_JSON_DOC_ARCHIVE":
// home, _ := os.UserHomeDir()
err := parser.WriteJsonToCompressedFile(state.CbDocs, state.LoadSpec.JsonArchiveFilePathAndPrefix+time.Now().Format(time.RFC3339))
err := parser.WriteJsonToCompressedFile(state.CbDocs, state.LoadSpec.JsonArchiveFilePathAndPrefix+time.Now().Format(time.RFC3339)+".gz")
if err != nil {
slog.Error("Expected no error, got:", slog.Any("error", err))
}
Expand Down
Loading
Loading