Korean Sentence Segmentation

AIHub Korean Dataset
JSON Parsing
Data Cleaning
Regular Expression
TXT Merge
Multiprocessing
Exploratory Data Analysis
- Length of Source List
- The Number of Character
- Capacity
- Preprocessing Runtime Calculator
- Preprocessing Memory & Process & Thread
Future Work
- Dataframe(Pandas or Polars), Dictionary Optimization
- The Searcher of Source JSON
  To compare JSON and TXT extracted from JSON
  File Naming & Storage System (Before & After File Name Match in Excel)
  What's the Source TXT File Name to Each Line in Proprocessed TXT File? (Dataframe)
- Remove Warning kss 3.7.3 Message: "[Korean Sentence Splitter]: Too long text! turn off quotes calibration!"
- Cython Multithreading

Library

kss regex pandas

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Korean Sentence Segmentation

Library

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 108 Commits
first-try		first-try
report		report
LICENSE		LICENSE
README.md		README.md
aihub_json_post_proprecssing.ipynb		aihub_json_post_proprecssing.ipynb
aihub_json_preprocessing_automatic_patent_classification_data.ipynb		aihub_json_preprocessing_automatic_patent_classification_data.ipynb
aihub_json_preprocessing_british_korean_data_on_patents_in_major_countries_linked_to_industrial_information.ipynb		aihub_json_preprocessing_british_korean_data_on_patents_in_major_countries_linked_to_industrial_information.ipynb
aihub_json_preprocessing_document_summary_text.ipynb		aihub_json_preprocessing_document_summary_text.ipynb
aihub_json_preprocessing_general_common_sense.ipynb		aihub_json_preprocessing_general_common_sense.ipynb
aihub_json_preprocessing_korean_corpus_data_based_on_large_scale_purchase_books.ipynb		aihub_json_preprocessing_korean_corpus_data_based_on_large_scale_purchase_books.ipynb
aihub_json_preprocessing_legal_regulations_(such_as_terms_and_conditions_of_judgment)_text_analysis_data.ipynb		aihub_json_preprocessing_legal_regulations_(such_as_terms_and_conditions_of_judgment)_text_analysis_data.ipynb
aihub_json_preprocessing_machine_reading.ipynb		aihub_json_preprocessing_machine_reading.ipynb
aihub_json_preprocessing_machine_reading_data_for_administrative_documents.ipynb		aihub_json_preprocessing_machine_reading_data_for_administrative_documents.ipynb
aihub_json_preprocessing_news_article_machine_reading_data.ipynb		aihub_json_preprocessing_news_article_machine_reading_data.ipynb
aihub_json_preprocessing_professional_corpus.ipynb		aihub_json_preprocessing_professional_corpus.ipynb
aihub_json_preprocessing_reading_books_by_machine.ipynb		aihub_json_preprocessing_reading_books_by_machine.ipynb
aihub_json_preprocessing_summary_and_report_generation_data.ipynb		aihub_json_preprocessing_summary_and_report_generation_data.ipynb
aihub_json_preprocessing_summary_of_book_materials.ipynb		aihub_json_preprocessing_summary_of_book_materials.ipynb
aihub_json_preprocessing_summary_of_thesis_materials.ipynb		aihub_json_preprocessing_summary_of_thesis_materials.ipynb
aihub_json_preprocessing_web_data_based_korean_corpus_data.ipynb		aihub_json_preprocessing_web_data_based_korean_corpus_data.ipynb
aihub_txt_delete_duplicate_sentence.ipynb		aihub_txt_delete_duplicate_sentence.ipynb
aihub_txt_merge.ipynb		aihub_txt_merge.ipynb
data_preprocessing.py		data_preprocessing.py
extract_source_text.py		extract_source_text.py
reading_data.py		reading_data.py
sentence_segmentation.py		sentence_segmentation.py
txt_preprocessing.py		txt_preprocessing.py

License

minseok0809/korean-sentence-segementation

Folders and files

Latest commit

History

Repository files navigation

Korean Sentence Segmentation

Library

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages