SESE : ScEne SEarch

SESE is a video scene storing and searching module which mainly uses video scene caption to find a specified scene that user wants to retrieve.

Two main ideas of our approach are

• Parse scene graph from simplified caption
  • To perform searches based on specified subject, predicate, and object
• Expand the keyword if there is no matched result
  • To improve search performance

Requirements

• neointerface
  pip install neointerface
  

• neo4j
  pip install neo4j
  

• py2neo
  pip install py2neo
  

• mysql-connector
  pip install mysql-connector
  

• pytube
  pip install pytube
  

• pymysql
  pip install pymysql
  
  

Scene Graph Parsing

You can build your database by using our scene graph parsing LLM (: fine-tuned Vicuna 7B) in Docker container. If you just want to use our parsed ActivityNet Captions dataset, skip this step.

First, put your raw dataset (csv file) in the `data/raw` folder. Here is the column names that your dataset should have.

video_id	begin_frame	end_frame	captions

video_id: ID of the video
begin_frame: Starting point of the scene
end_frame: Ending point of the scene
captions: Description of the scene

After that, run the code below.

./build.sh
./run.sh
./bash.sh
python caption-to-scene-graph.py

Then you can get your parsed dataset on data/scene_graph.csv.

Usage of SESE

DB Connect & Module Load

from SESE import SESE as db

neo4j_url = "YOUR NEO4J URL"
neo4j_user = "YOUR NEO4J USER NAME"
neo4j_password = "YOUR NEO4J PASSWORD"
mariadb_user = "YOUR MARIADB USER NAME"
mariadb_password = "YOUR MARIADB PASSWORD"
mariadb_host = "YOUR HOST IP" 
maridb_database = "YOUR MARIADB DATABASE NAME"

sese = db(neo4j_url, neo4j_user, neo4j_password, 
          mariadb_user, mariadb_host, mariadb_database)

• Example

from SESE import SESE as db

neo4j_url = "neo4j://localhost:7687"
neo4j_user = "neo4j"
neo4j_password = "testtest"
mariadb_user = "dilab"
mariadb_password = "1111"
mariadb_host = "127.0.0.1"
maridb_database = "mydb"

sese = db(neo4j_ur, neo4j_user, neo4j_password, 
          mariadb_user, mariadb_host, mariadb_database)

Create

• add_df(mariadb_database, csv_file) : A function that uploads data (csv file) on DB

  • mariadb_database : RDB name csv_file : video-caption-scene-graph dataset (csv file)
  • e.g., db.add_table(mariadb_database, './data/activitynet_10_sg.csv')
    
    

Search

• get_spo() : A function that specifies subject, predicate, object to search a scene.

  • If there is no exact matching result, our module would automatically expand the input query.
    
    

• get_keyword() : A function that specifies keyword to search a scene.
  
  

Etc

• get_description() : A function that prints information of objects and predicates on the GraphDB e.g., average, minimum, and maximum counts...
  
  

• get_object_list() : A function that prints unique list of objects on the GraphDB
  
  

• get_object(object) : A function that prints information of specified object node

  • object : The object that wants to get information
  • e.g., sese.get_object(object='man')
    
    

• get_predicate_list() : A function that prints unique list of predicates on the GraphDB