Skip to content

Commit bb9529e

Browse files
HadronColliderHadronCollider
authored
DIS export implementation (export_courses.sh) (#53)
* add sheet to yadisk table adding logic * integrate yadisk logic to moodle export * dont push to google if no table/sheet id * dont push to google if no google_token in args * if cls.args specified write data to sheets document * move utils from scripts to moodle_export * add openpyxl to moodle_export requirements * cp yadisk_manager.py to moodle_export * update yadisk version * yatoken as args to write_sheet_to_file * yatoken as args to DiskManager * update default CSV_DELIMITER in moodle_export * change write mode for logfile * container stdout to file (error to console) * fix indent for google sheet export * update check_export yo yadisk * update required args for checker_export * add openpyxl to requirements (checker_export) * update yadisk version (checker_export requirements) * update load_data_from_dis * update add_csv_to_table logic * update add_csv_to_table (split rows) * DIS export to csv file * Update moodle_export.utils.py (clear sheet instead remove) * Update checker_export.utils.py (clear sheet instead remove) * clear sheet (insert_rows with 0) * download_sheet_to_pdf -> download_sheet (file extension as arg) * update download_sheet args * update duplicateSheetsToYadisk logs * update wget for download_sheet * update abs_disk_path in upload_file_to_disk * update duplicateSheetsToYadisk * update duplicateSheetsToYadisk * fix duplicateSheetsToYadisk * rm debug logs * add error handling in export_courses.sh * if to case (export_courses.sh) * checker_export: sheet_id to sheet_name * improve export_courses.sh (implement checker export and rename env var) * add workflows * rm comment in Dockerfile (stepik_export) * rename sheet_id to sheet_name
1 parent fadeedb commit bb9529e

File tree

14 files changed

+97
-54
lines changed

14 files changed

+97
-54
lines changed

‎.github/workflows/build.yml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: CI
2+
3+
on: pull_request
4+
5+
jobs:
6+
build:
7+
runs-on: ubuntu-22.04
8+
9+
steps:
10+
- uses: actions/checkout@v4
11+
12+
- name: Build stepik_export image
13+
if: ${{ !cancelled() }}
14+
run: |
15+
./build.sh stepik_export
16+
- name: Build moodle_export image
17+
if: ${{ !cancelled() }}
18+
run: |
19+
./build.sh moodle_export
20+
- name: Build checker_export image
21+
if: ${{ !cancelled() }}
22+
run: |
23+
./build.sh checker_export

‎.github/workflows/label_merge_conflicts.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: 'Check for merge conflicts'
2+
3+
on: [push]
4+
5+
jobs:
6+
find_conflicts:
7+
runs-on: ubuntu-22.04
8+
9+
steps:
10+
- uses: mschilde/auto-label-merge-conflicts@master
11+
with:
12+
CONFLICT_LABEL_NAME: "has conflicts"
13+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
14+
MAX_RETRIES: 5
15+
WAIT_MS: 5000

‎checker_export/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,5 @@
33
## Инструкция по запуску скрипта
44

55
```bash
6-
python3 exporter.py [-h] [--google_token GOOGLE_TOKEN] --checker_token CHECKER_TOKEN [--checker_filter CHECKER_FILTER] --table_id TABLE_ID --sheet_id SHEET_ID [--yandex_token YANDEX_TOKEN --yandex_path YANDEX_PATH]
6+
python3 exporter.py [-h] [--google_token GOOGLE_TOKEN] --checker_token CHECKER_TOKEN [--checker_filter CHECKER_FILTER] --table_id TABLE_ID --sheet_name SHEET_NAME [--yandex_token YANDEX_TOKEN --yandex_path YANDEX_PATH]
77
```

‎checker_export/exporter.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ def parse_args():
2323
parser.add_argument('--checker_token', type=str, required=True, help='Specify session cookie for slides-checker')
2424
parser.add_argument('--checker_filter', type=str, required=False, help='Specify filter for slides-checker')
2525
parser.add_argument('--table_id', type=str, required=False, help='Specify Google sheet document id (can find in url)')
26-
parser.add_argument('--sheet_id', type=str, required=False, help='Specify title for a sheet in a document in which data will be printed')
26+
parser.add_argument('--sheet_name', type=str, required=False, help='Specify title for a sheet in a document in which data will be printed')
2727
parser.add_argument('--yandex_token', type=str, required=False, help='Specify Yandex token from https://oauth.yandex.ru/client/new application')
2828
parser.add_argument('--yandex_path', type=str, required=False, help='Specify output filename on Yandex Disk')
2929
args = parser.parse_args()
@@ -47,14 +47,14 @@ def load_data_from_dis(checker_filter, checker_token):
4747
return csv_path, df_data
4848

4949

50-
def write_data_to_table(checker_token, checker_filter, google_token, table_id, sheet_id, yandex_token=None, yandex_path=None):
50+
def write_data_to_table(checker_token, checker_filter, google_token, table_id, sheet_name, yandex_token=None, yandex_path=None):
5151
csv_path, df_data = load_data_from_dis(checker_filter, checker_token)
5252

53-
if google_token and sheet_id and table_id:
53+
if google_token and sheet_name and table_id:
5454
gc = pygsheets.authorize(service_file=google_token)
5555
sh = gc.open_by_key(table_id)
5656

57-
wk_content = sh.worksheet_by_title(sheet_id)
57+
wk_content = sh.worksheet_by_title(sheet_name)
5858

5959
wk_content.set_dataframe(df_data, 'A1', copy_head=True)
6060

@@ -69,7 +69,7 @@ def write_data_to_table(checker_token, checker_filter, google_token, table_id, s
6969

7070
def main():
7171
args = parse_args()
72-
write_data_to_table(args.checker_token, args.checker_filter, args.google_token, args.table_id, args.sheet_id, args.yandex_token, args.yandex_path)
72+
write_data_to_table(args.checker_token, args.checker_filter, args.google_token, args.table_id, args.sheet_name, args.yandex_token, args.yandex_path)
7373

7474

7575
if __name__ == "__main__":

‎moodle_export/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818

1919
Получение данных и в файл, и в Google таблицу:
2020

21-
```python3 grades_parser.py grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_id "name of sheet"[,"name",..]] ```
21+
```python3 grades_parser.py grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_name "name of sheet"[,"name",..]] ```
2222

2323
Получение данных и в файл, и на Яндекс Диск:
2424

@@ -36,7 +36,7 @@
3636

3737
Запуск докер контейнера с параметрами:
3838

39-
``` docker run --rm grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_id "name of sheet"[,"name",..]] ```
39+
``` docker run --rm grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_name "name of sheet"[,"name",..]] ```
4040

4141

4242
## Получение Moodle Токена.

‎moodle_export/args_parser.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ def arg_parser():
99
parser.add_argument('--csv_path', type=str, required=True, help='Specify path to output csv file')
1010
parser.add_argument('--google_token', type=str, required=False, help='Specify path to google token file')
1111
parser.add_argument('--table_id', type=lambda s: [i for i in s.split(',')], required=False, help='Specify Google sheet document id (can find in url)')
12-
parser.add_argument('--sheet_id', type=lambda s: [i for i in s.split(',')], required=False, help='Specify title for a sheet in a document in which data will be printed')
12+
parser.add_argument('--sheet_name', type=lambda s: [i for i in s.split(',')], required=False, help='Specify title for a sheet in a document in which data will be printed')
1313
parser.add_argument('--yandex_token', type=str, required=False, help='Specify Yandex token from https://oauth.yandex.ru/client/new application')
1414
parser.add_argument('--yandex_path', type=str, required=False, help='Specify output filename on Yandex Disk')
1515
parser.add_argument('--percentages', required=False, action='store_true', help='If set then grades will be printed as percentages')

‎moodle_export/grades_parser.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -153,17 +153,17 @@ def main(cls):
153153
elif i == len(cls.args.table_id) - 1:
154154
table_id = cls.args.table_id[i]
155155

156-
if cls.args.sheet_id:
157-
for i in range(0, len(cls.args.sheet_id)):
156+
if cls.args.sheet_name:
157+
for i in range(0, len(cls.args.sheet_name)):
158158
if cls.args.course_id[i] == course_id:
159-
sheet_id = cls.args.sheet_id[i]
159+
sheet_name = cls.args.sheet_name[i]
160160
break
161161
else:
162-
sheet_id = cls.args.sheet_id[i] + ' ' + course_id
162+
sheet_name = cls.args.sheet_name[i] + ' ' + course_id
163163
else:
164-
sheet_id = 'course ' + course_id
164+
sheet_name = 'course ' + course_id
165165

166-
sheets.write_data_to_table(df, cls.args.google_token, table_id, sheet_id)
166+
sheets.write_data_to_table(df, cls.args.google_token, table_id, sheet_name)
167167

168168
# write data to yandex disk
169169
if cls.args.yandex_token and cls.args.yandex_path:

‎moodle_export/sheets.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,24 +3,24 @@
33
import args_parser
44

55

6-
def write_data_to_table(df_data, google_token, table_id, sheet_id):
7-
if google_token and sheet_id and table_id:
6+
def write_data_to_table(df_data, google_token, table_id, sheet_name):
7+
if google_token and sheet_name and table_id:
88
gc = pygsheets.authorize(service_file=google_token)
99
sh = gc.open_by_key(table_id)
1010

1111
try:
12-
sh.worksheets('title', sheet_id)
12+
sh.worksheets('title', sheet_name)
1313
except:
14-
sh.add_worksheet(sheet_id)
14+
sh.add_worksheet(sheet_name)
1515

16-
wk_content = sh.worksheet_by_title(sheet_id)
16+
wk_content = sh.worksheet_by_title(sheet_name)
1717

1818
wk_content.set_dataframe(df_data, 'A1', copy_head=True)
1919

2020

2121
def main():
2222
args = args_parser.arg_parser()
23-
write_data_to_table(args.csv_path, args.google_token, args.table_id, args.sheet_id)
23+
write_data_to_table(args.csv_path, args.google_token, args.table_id, args.sheet_name)
2424

2525

2626
if __name__ == "__main__":

‎scripts/export_courses.sh

Lines changed: 24 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# )
88
# system: "moodle" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,moodle,course_id
99
# system: "stepik" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,stepik,course_id,class_id
10+
# system: "dis" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,dis,filter
1011
# exportCourses "${exports[@]}"
1112

1213

@@ -32,31 +33,36 @@ function exportCourses() {
3233

3334
case "${current_export[3]}" in
3435
"moodle")
35-
docker run --rm -v $google_conf:/app/conf.json moodle_export_parser:latest \
36-
--moodle_token $moodle_token --url https://e.moevm.info \
37-
--csv_path grades --google_token conf.json \
38-
--course_id ${current_export[4]} \
39-
--table_id ${current_export[1]} \
40-
--sheet_id ${current_export[2]} \
41-
--options github >> $log_file
36+
docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json moodle_export_parser:latest \
37+
--moodle_token $MOODLE_TOKEN --url https://e.moevm.info \
38+
--csv_path grades --google_token conf.json \
39+
--course_id ${current_export[4]} \
40+
--table_id ${current_export[1]} \
41+
--sheet_name ${current_export[2]} \
42+
--options github >> $log_file
4243

4344
return_code=$?
4445
;;
4546
"stepik")
46-
docker run --rm -v $google_conf:/app/conf.json stepik_export_parser:latest \
47-
--client_id $stepik_client_id --client_secret $stepik_client_secret \
48-
--url https://stepik.org:443/api \
49-
--csv_path grades --google_token conf.json \
50-
--course_id ${current_export[4]} \
51-
--class_id ${current_export[5]} \
52-
--table_id ${current_export[1]} \
53-
--sheet_id ${current_export[2]} >> $log_file
47+
docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json stepik_export_parser:latest \
48+
--client_id $STEPIK_CLIENT_ID --client_secret $STEPIK_CLIENT_SECRET \
49+
--url https://stepik.org:443/api \
50+
--csv_path grades --google_token conf.json \
51+
--course_id ${current_export[4]} \
52+
--class_id ${current_export[5]} \
53+
--table_id ${current_export[1]} \
54+
--sheet_name ${current_export[2]} >> $log_file
5455

5556
return_code=$?
5657
;;
57-
"checker")
58-
echo "Not implemented"
59-
return_code=1
58+
"dis")
59+
docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json checker_export_parser:latest \
60+
--checker_filter "${current_export[4]}" \
61+
--checker_token $DIS_ACCESS_TOKEN \
62+
--table_id ${current_export[1]} \
63+
--sheet_name ${current_export[2]} >> $log_file
64+
65+
return_code=$?
6066
;;
6167
*)
6268
echo "Недопустимое значение: '${current_export[3]}'"

‎stepik_export/Dockerfile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
# syntax=docker/Dockerfile:1
21
FROM python:3.8-slim
32

43
WORKDIR /app

0 commit comments

Comments
 (0)