Skip to content

Latest commit

 

History

History
122 lines (82 loc) · 4.32 KB

README.md

File metadata and controls

122 lines (82 loc) · 4.32 KB
Raw

HisVis2

Project Description

Work for Part 2 of HisVis Project. In this project, we developed a scene detection dataset and trained models for predicting the labels. The input data for this project are press photographs from the De Boer collection.

This project includes software to: - automatically resize and crop the images - automatically rotate them to the correct orientation - training a rotation model - updating a pre-trained Places-365 model to our data - training a logistic classifier on top of a CLIP model - making predictions for the scene categories using the places-365, CLIP model or both - making predictions for the categories (indoor / outdoor) - preparing output generated by the annotation software (VeleHanden) - analyzing the annotations - checking whether the algorithm can also be applied to uncropped images

Prerequisites

Python 3.7.11 See requirements.txt to set up a Conda environment. conda create --name <envname> --file requirements.txt

Contributing

Issue Tracker: https://github.com/melvinwevers/HisVis2/issues

License

CC-NY

Contact

Dr. Melvin Wevers (melvin.wevers(at)uva.nl)

Folder Structure

Data

Intermediary

extra_training_labels.csv- labels for which additional samples were need in annotation step 3.

Processed

Annotations
  • annotation_data_step1.csv - processed database export
  • annotation_data_step21.csv - processed database export
  • annotation_data_step3.csv - processed database export
Training Data
  • all_labels.txt - list of all labels used

Place training data from Zenodo here.

Raw

Test

Contains example test file

Docs

  • data_sheet.md - data sheet for HisVis dataset
  • label_sheet.md - label sheet documenting used labels
  • model_card_indoor_outdoor_clip - model card for indoor-outdoor detection using CLIP
  • model_card_indoor_outdoor_places - model card for indoor-outdoor detection using Places-365
  • model_card_rotation - model card for correct rotation detection
  • model_card_scene_detection_clip - model card for scene detection using CLIP
  • model_card_scene_detection_places - model card for scene detection using Places-365

Notebooks

  • 0.install_packages.ipynb - install required packages that could not be installed using pip / conda
  • 1.prepare_rotation_training_data.ipynb - creating training data for the rotation model using correctly rotated data
  • 2.train_model_rotation.ipynb - training model for identifying correct rotation
  • 3.make_predictions_in_out.ipynb - predicting whether an image is taken indoor or outdoor using CLIP.
  • 4.prepare_output_picturae.ipynb - preparing data exported from Vele Handen provided by Picturae.
  • 5.analyze_annotations.ipynb - analysis of annotation effort and code to prepare new training data
  • 6.check_prediction_crop_or_not.ipynb - evaluation of model performance on cropped and non-cropped images.

Output

  • finding_learningrate.out - output from learning rate finder used to estimate optimal learning rate
  • slurm-9991191.out stack trace for training models
  • slurm-9992781.out stack trace for training models

Models

Place models.tar.gz from Zenodo here

indoor_outdoor

Models and evaluation of models for indoor-outdoor detection

Rotation
  • rotation.pkl model used to predict correct rotation of images
Scenes

Models and evaluation for models for scene detection

src

  • find_correct_rotation.py - Python script to rotate images until they are correctly oriented. This makes use of the rotation.pkl model. This model has been trained using the notebook train_model_rotation.ipynb
  • helper.py - script with different helper functions
  • make_predictions_in_out.py - script to make indoor / outdoor predictions using standard clip
  • make_predictions.py - script to make scene detection predictions using fine-tuned clip, places365, or both
  • remove_border.py - script to resize images and crop them out of the photo negative sheets
  • train_clip.py - script to finetune CLIP
  • train_places_model.py script to finetune places-365

Preparing images

run prepare_images.sh

Training the model

We can train the model using a Jupyter Notebook or a Python Script that can be called using the run.sh bash script.

Running predictions

Run run_prediction.sh