Add cobj-idx migrate and cobj-idx unlock (#693)

yutaro-sakamoto · Copilot · web-flow · commit f2d31fd57f05 · 2025-08-27T11:49:47.000+09:00
* add: processUuid and processId * add: create a new table `file_lock` * refactor: suppress PMD warnings * fix: sql statement creating a new table `file_lock` * wip: wip * fix: a PMD warning * fix: a SpotBug warning * refactor: remove printStackTrace() * fix: check formats of indexed files to open * test: remove unnecessary checks * fix: change constants to avoid overlaps * fix: add a file existing check * test: fix the test of opening an invalid indexed file * test: fix load.at not to call default error handlers * chore: end a transaction * refactor: CobolIndexedFile.java * test: prevent infinite loops * refactor: CobolIndexedFile.java * wip: wip * fix: improve open_ in CobolIndexedFile.java * change: set transaction settings * set the transaction mode to EXCLUSIVE * set busy_timeout to 5000 * feat: implement file locks of indexed files * fix: change the condition of creating file_lock * change: set the transaction isolation to SERIALIZABLE * test: add tests of file locking of indexed files * change: the schema of the record key table * fix: an INSERT statement * fix: an insert statement for primary record tables * feat: add a new option `-default_select_lock_mode` * channge: insert information when calling read_() * fix: the method read_ * Implement file locking of indexed files (#678) * add: processUuid and processId * add: create a new table `file_lock` * refactor: suppress PMD warnings * fix: sql statement creating a new table `file_lock` * wip: wip * fix: a PMD warning * fix: a SpotBug warning * refactor: remove printStackTrace() * fix: check formats of indexed files to open * test: remove unnecessary checks * fix: change constants to avoid overlaps * fix: add a file existing check * test: fix the test of opening an invalid indexed file * test: fix load.at not to call default error handlers * chore: end a transaction * refactor: CobolIndexedFile.java * test: prevent infinite loops * refactor: CobolIndexedFile.java * wip: wip * fix: improve open_ in CobolIndexedFile.java * change: set transaction settings * set the transaction mode to EXCLUSIVE * set busy_timeout to 5000 * feat: implement file locks of indexed files * fix: change the condition of creating file_lock * change: set the transaction isolation to SERIALIZABLE * test: add tests of file locking of indexed files * Update libcobj/app/src/main/java/jp/osscons/opensourcecobol/libcobj/file/CobolIndexedFile.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update libcobj/app/src/main/java/jp/osscons/opensourcecobol/libcobj/file/CobolIndexedFile.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor: remove unnecessary comments * refactor: fix typo * refactor: fix typo --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor: CobolIndexedFile.java * add: lock a record when READ NEXT is executed * add: unlock records when closing indexed files * fix: the process of unlocking a file lock * fix: controling transactions when rewriting records * fix: commit timing when writing records * fix: commits and rollbacks in WRITE, REWRITE and DELETE * change: -default_lock_mode to -lock-mode-automatic * refactor: code for commiting data * add: unlock the previous record after writing * add: unlock previous records after rewriting * add: impelment record lockings of DELETE * fix: constant numbers change the value of COB_LOCK_AUTOMATIC and COB_LOCK_MULTIPLE in CobolFile.java * chore: remove duplicated code * fix: close ResultSet not to lock database implicitly * fix: close ResultSet in CobolIndexedFile.java * add: NewIndexedCursor.java * wip: read forward * wip: read prev * fix: fetchRecord * fix: READ NEXT/PREVIOUS * fix: READ * fix: fetchRecord * fix: fetchRecord * fix: START * fix: START ~ KEY IS GREATER THAN * fix: START ~ KEY IS <= * fix: START and READ NEXT RECORD * refactor: resolve PMD warnings * chore: migrate NewIndexedCursor to IndexedCursor * fix: allow READ WITH LOCK/WITH NO LOCK if LOCK MODE IS AUTOMATIC * fix: call commit() in the end of READ and READ NEXT/PREV * fix: initialize previousLockedReclordKey in close_() * test: add some tests for record locking * fix: unlock Previous records at the end of READ * test: accelerate file locking tests * test: wip * test: add tests for `READ WITH LOCK` * test: check the final state of an indexed file * fix: do not read data when the file status is not 00 * test: move common java code for record lock tests * test: add tests for different records * test: add tests for input mode * test: add tests for the same process * fix: commit at the end of start_ * test: guarantee that OPEN,START,WRITE,REWRITE cannot lock indexed file records * test: fix tests for OPEN,START,WRITE and REWRITE * fix: processes of unlocking records * tests: add tests for unlocking records * tests: add tests of OPEN INPUT * test: add tests for LOCK MODE * test: add tests of -lock-mode-automatic * tests: move tests for indexed files lock to tests/indexed-lock * ci: run tests `indexed-lock` * chore: fix typo * feat: add `cobj-idx migrate` * feat: add `cobj-idx unlock` * test: add a test for `cobj-idx migrate` * test: add a test for `cobj-idx unlock` * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: simplify some `if` conditions --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
diff --git a/libcobj/app/src/main/java/jp/osscons/opensourcecobol/libcobj/user_util/indexed_file/IndexedFileUtilMain.java b/libcobj/app/src/main/java/jp/osscons/opensourcecobol/libcobj/user_util/indexed_file/IndexedFileUtilMain.java
@@ -137,6 +137,40 @@ public static void main(String[] args) {
                 System.err.println("error: " + e.getMessage());
                 System.exit(1);
             }
+        } else if ("migrate".equals(subCommand)) {
+            if (unrecognizedArgs.length != 2) {
+                if (unrecognizedArgs.length < 2) {
+                    System.err.println("error: no indexed file is specified.");
+                } else {
+                    System.err.println("error: too many indexed files are specified.");
+                }
+                System.exit(1);
+            }
+            String indexedFilePath = unrecognizedArgs[1];
+            try {
+                migrateIndexedFile(indexedFilePath);
+            } catch (Exception e) {
+                System.err.println("error: " + e.getMessage());
+                System.exit(1);
+            }
+            System.exit(0);
+        } else if ("unlock".equals(subCommand)) {
+            if (unrecognizedArgs.length != 2) {
+                if (unrecognizedArgs.length < 2) {
+                    System.err.println("error: no indexed file is specified.");
+                } else {
+                    System.err.println("error: too many indexed files are specified.");
+                }
+                System.exit(1);
+            }
+            String indexedFilePath = unrecognizedArgs[1];
+            try {
+                unlockIndexedFile(indexedFilePath);
+            } catch (Exception e) {
+                System.err.println("error: " + e.getMessage());
+                System.exit(1);
+            }
+            System.exit(0);
         } else if ("load".equals(subCommand)) {
             if (unrecognizedArgs.length < 2 || unrecognizedArgs.length > 3) {
                 if (unrecognizedArgs.length < 2) {
@@ -226,6 +260,14 @@ private static void printHelpMessage() {
                 "    Write the records stored in the indexed file into the output file.");
         System.out.println("    The default format of the output data is SEQUENTIAL of COBOL.");
         System.out.println();
+        System.out.println("cobj-idx migrate <indexed file>");
+        System.out.println(
+                "    Migrate the indexed file whose version is older than 1.1.12 to the latest"
+                        + " version.");
+        System.out.println();
+        System.out.println("cobj-idx unlock <indexed file>");
+        System.out.println("    Unlock all locks of the indexed file.");
+        System.out.println();
         System.out.println("Options:");
         System.out.println();
         System.out.println("-f <format>, --format=<format>");
@@ -413,6 +455,60 @@ private static int processInfoCommand(String indexedFilePath) {
         }
     }
 
+    private static void migrateIndexedFile(String indexedFilePath) throws Exception {
+        try (Connection conn = DriverManager.getConnection("jdbc:sqlite:" + indexedFilePath);
+                Statement st = conn.createStatement()) {
+            String createTableSql =
+                    "CREATE TABLE IF NOT EXISTS file_lock (locked_by text primary key,process_id"
+                            + " text,locked_at timestamp,open_mode text CONSTRAINT check_open_mode"
+                            + " CHECK (open_mode IN ('INPUT', 'OUTPUT', 'I-O', 'EXTEND')))";
+            st.executeUpdate(createTableSql);
+
+            boolean lockedByColumnExists = false;
+            boolean processIdColumnExists = false;
+            boolean lockedAtColumnExists = false;
+            try (ResultSet rs = st.executeQuery("PRAGMA table_info('table0')")) {
+                while (rs.next()) {
+                    String columnName = rs.getString("name");
+                    if ("locked_by".equals(columnName)) {
+                        lockedByColumnExists = true;
+                    } else if ("process_id".equals(columnName)) {
+                        processIdColumnExists = true;
+                    } else if ("locked_at".equals(columnName)) {
+                        lockedAtColumnExists = true;
+                    }
+                    if (lockedByColumnExists && processIdColumnExists && lockedAtColumnExists) {
+                        break;
+                    }
+                }
+            }
+            if (!lockedByColumnExists) {
+                st.executeUpdate("ALTER TABLE table0 ADD COLUMN locked_by text");
+            }
+            if (!processIdColumnExists) {
+                st.executeUpdate("ALTER TABLE table0 ADD COLUMN process_id text");
+            }
+            if (!lockedAtColumnExists) {
+                st.executeUpdate("ALTER TABLE table0 ADD COLUMN locked_at timestamp");
+            }
+        } catch (SQLException e) {
+            throw new Exception(
+                    String.format("failed to connect to the indexed file: %s", indexedFilePath), e);
+        }
+    }
+
+    private static void unlockIndexedFile(String indexedFilePath) throws Exception {
+        try (Connection conn = DriverManager.getConnection("jdbc:sqlite:" + indexedFilePath);
+                Statement st = conn.createStatement()) {
+            st.executeUpdate("DELETE FROM file_lock");
+            st.executeUpdate(
+                    "UPDATE table0 SET locked_by = NULL, process_id = NULL, locked_at = NULL");
+        } catch (SQLException e) {
+            throw new Exception(
+                    String.format("failed to connect to the indexed file: %s", indexedFilePath), e);
+        }
+    }
+
     /**
      * Process load sub command, which loads data inputted from stdin to the indexed file.
      *
diff --git a/tests/Makefile.am b/tests/Makefile.am
@@ -204,6 +204,8 @@ cobj_idx_DEPENDENCIES = \
 	cobj-idx.src/info.at \
 	cobj-idx.src/load.at \
 	cobj-idx.src/unload.at \
+	cobj-idx.src/migrate.at \
+	cobj-idx.src/unlock.at \
 	cobj-idx.src/misc.at
 
 indexed_lock_DEPENDENCIES = \
diff --git a/tests/Makefile.in b/tests/Makefile.in
@@ -743,6 +743,8 @@ cobj_idx_DEPENDENCIES = \
 	cobj-idx.src/info.at \
 	cobj-idx.src/load.at \
 	cobj-idx.src/unload.at \
+	cobj-idx.src/migrate.at \
+	cobj-idx.src/unlock.at \
 	cobj-idx.src/misc.at
 
 indexed_lock_DEPENDENCIES = \
diff --git a/tests/cobj-idx.at b/tests/cobj-idx.at
@@ -5,4 +5,5 @@ m4_include([info.at])
 m4_include([load.at])
 m4_include([unload.at])
 m4_include([cobj-idx.src/misc.at])
-
+m4_include([migrate.at])
+m4_include([unlock.at])
diff --git a/tests/cobj-idx.src/migrate.at b/tests/cobj-idx.src/migrate.at
@@ -0,0 +1,76 @@
+AT_SETUP([migrate])
+
+AT_DATA([check_file.cbl], [
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. check_file.
+
+       ENVIRONMENT DIVISION.
+       INPUT-OUTPUT SECTION.
+       FILE-CONTROL.
+           SELECT F ASSIGN TO "indexed_file.dat"
+               ORGANIZATION IS INDEXED
+               ACCESS MODE IS DYNAMIC
+               RECORD KEY IS REC-KEY
+               ALTERNATE RECORD KEY IS REC-KEY2 WITH DUPLICATES
+               FILE STATUS IS FILE-STATUS.
+
+       DATA DIVISION.
+       FILE SECTION.
+       FD  f.
+       01  REC.
+           05 REC-KEY      PIC X(5).
+           05 REC-KEY2     PIC X(5).
+           05 REC-DATA     PIC X(5).
+
+       WORKING-STORAGE SECTION.
+       01 FILE-STATUS      PIC XX.
+       01 DUMMY PIC X(10) VALUE "DUMMY DATA".
+       PROCEDURE DIVISION.
+       MAIN-PROCEDURE.
+
+           OPEN I-O f.
+
+           MOVE "AAAA1" TO REC-KEY.
+           READ F WITH LOCK.
+           DISPLAY REC.
+
+           MOVE "AAAA2" TO REC-KEY.
+           REWRITE REC.
+
+           MOVE "AAAA2" TO REC-KEY.
+           READ F WITH LOCK.
+           DISPLAY REC.
+
+           MOVE "AAAA2" TO REC-KEY.
+           DELETE F.
+           DISPLAY FILE-STATUS.
+
+           MOVE "AAAA1" TO REC-KEY.
+           START F KEY IS = REC-KEY.
+           PERFORM FOREVER
+               READ F NEXT RECORD
+                    AT END
+                        EXIT PERFORM
+                    NOT AT END
+                        DISPLAY REC
+               END-READ
+           END-PERFORM.
+           CLOSE f.
+])
+
+# copy the indexed file created with the older compiler to the current directory
+AT_CHECK([cp ../../cobj-idx.src/test-data/indexed_file.dat .])
+
+# convert the indexed file to the current format
+AT_CHECK([${COBJ_IDX} migrate indexed_file.dat])
+
+# check the converted file
+AT_CHECK([${COMPILE} check_file.cbl])
+AT_CHECK([java check_file], [0],
+[AAAA1BBBB1CCCC1
+AAAA2BBBB1CCCC1
+00
+AAAA1BBBB1CCCC1
+AAAA3BBBB3CCCC3
+])
+AT_CLEANUP
diff --git a/tests/cobj-idx.src/test-data/indexed_file.dat b/tests/cobj-idx.src/test-data/indexed_file.dat
diff --git a/tests/cobj-idx.src/unlock.at b/tests/cobj-idx.src/unlock.at
@@ -0,0 +1,173 @@
+AT_SETUP([unlock])
+
+# Create a sample indexed file named 'indexed_file.dat' and exit the program without closing the file
+AT_DATA([create_and_exit.cbl], [
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. create_and_exit.
+
+       ENVIRONMENT DIVISION.
+       INPUT-OUTPUT SECTION.
+       FILE-CONTROL.
+           SELECT F ASSIGN TO "indexed_file.dat"
+               ORGANIZATION IS INDEXED
+               ACCESS MODE IS DYNAMIC
+               RECORD KEY IS REC-KEY
+               ALTERNATE RECORD KEY IS REC-KEY2 WITH DUPLICATES
+               FILE STATUS IS FILE-STATUS.
+
+       DATA DIVISION.
+       FILE SECTION.
+       FD  f.
+       01  REC.
+           05 REC-KEY      PIC X(5).
+           05 REC-KEY2     PIC X(5).
+           05 REC-DATA     PIC X(5).
+
+       WORKING-STORAGE SECTION.
+       01 FILE-STATUS      PIC XX.
+       01 DUMMY PIC X(10) VALUE "DUMMY DATA".
+       PROCEDURE DIVISION.
+       MAIN-PROCEDURE.
+           OPEN OUTPUT f.
+           MOVE "AAAA1" TO REC-KEY.
+           MOVE "BBBB1" TO REC-KEY2.
+           MOVE "CCCC1" TO REC-DATA.
+           WRITE REC.
+           CLOSE f.
+
+           OPEN I-O f.
+           MOVE "AAAA1" TO REC-KEY.
+           READ f WITH LOCK.
+           CALL "force_exit".
+])
+
+AT_DATA([force_exit.java], [
+import java.io.UnsupportedEncodingException;
+import jp.osscons.opensourcecobol.libcobj.*;
+import jp.osscons.opensourcecobol.libcobj.common.*;
+import jp.osscons.opensourcecobol.libcobj.data.*;
+import jp.osscons.opensourcecobol.libcobj.exceptions.*;
+import jp.osscons.opensourcecobol.libcobj.termio.*;
+import jp.osscons.opensourcecobol.libcobj.call.*;
+import jp.osscons.opensourcecobol.libcobj.file.*;
+import jp.osscons.opensourcecobol.libcobj.ui.*;
+import java.util.Optional;
+
+public class force_exit implements CobolRunnable {
+  @Override
+  public int run(CobolDataStorage... argStorages) {
+    System.exit(0);
+    return 0;
+  }
+
+  @Override
+  public void cancel() {
+    return;
+  }
+
+  @Override
+  public boolean isActive() {
+    return false;
+  }
+}
+])
+
+AT_DATA([check_file_lock.cbl],[
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. check_file_lock.
+
+       ENVIRONMENT DIVISION.
+       INPUT-OUTPUT SECTION.
+       FILE-CONTROL.
+           SELECT F ASSIGN TO "indexed_file.dat"
+               ORGANIZATION IS INDEXED
+               ACCESS MODE IS DYNAMIC
+               RECORD KEY IS REC-KEY
+               ALTERNATE RECORD KEY IS REC-KEY2 WITH DUPLICATES
+               FILE STATUS IS FILE-STATUS.
+
+       DATA DIVISION.
+       FILE SECTION.
+       FD  f.
+       01  REC.
+           05 REC-KEY      PIC X(5).
+           05 REC-KEY2     PIC X(5).
+           05 REC-DATA     PIC X(5).
+
+       WORKING-STORAGE SECTION.
+       01 FILE-STATUS      PIC XX.
+       01 DUMMY PIC X(10) VALUE "DUMMY DATA".
+       PROCEDURE DIVISION.
+       MAIN-PROCEDURE.
+           OPEN OUTPUT f.
+           DISPLAY FILE-STATUS.
+           CLOSE f.
+])
+
+AT_DATA([check_record_lock.cbl], [
+       IDENTIFICATION DIVISION.
+       PROGRAM-ID. check_record_lock.
+
+       ENVIRONMENT DIVISION.
+       INPUT-OUTPUT SECTION.
+       FILE-CONTROL.
+           SELECT F ASSIGN TO "indexed_file.dat"
+               ORGANIZATION IS INDEXED
+               ACCESS MODE IS DYNAMIC
+               RECORD KEY IS REC-KEY
+               ALTERNATE RECORD KEY IS REC-KEY2 WITH DUPLICATES
+               FILE STATUS IS FILE-STATUS.
+
+       DATA DIVISION.
+       FILE SECTION.
+       FD  f.
+       01  REC.
+           05 REC-KEY      PIC X(5).
+           05 REC-KEY2     PIC X(5).
+           05 REC-DATA     PIC X(5).
+
+       WORKING-STORAGE SECTION.
+       01 FILE-STATUS      PIC XX.
+       PROCEDURE DIVISION.
+       MAIN-PROCEDURE.
+           
+              OPEN I-O f.
+    
+              MOVE "AAAA1" TO REC-KEY.
+              READ F WITH LOCK.
+              DISPLAY FILE-STATUS. 
+
+              CLOSE f.
+])
+
+AT_CHECK([${COMPILE} *.cbl])
+AT_CHECK([javac force_exit.java])
+
+# Create the indexed file and exit the program without closing the file
+AT_CHECK([java create_and_exit])
+# Check the file is locked
+AT_CHECK([java check_file_lock], [0],
+[61
+])
+
+AT_CHECK([rm -f indexed_file.dat])
+# Create the indexed file and exit the program without closing the file
+AT_CHECK([java create_and_exit])
+# Unlock the file
+AT_CHECK([${COBJ_IDX} unlock indexed_file.dat])
+# Check the file lock is released
+AT_CHECK([java check_file_lock], [0],
+[00
+])
+
+AT_CHECK([rm -f indexed_file.dat])
+# Create the indexed file and exit the program without closing the file
+AT_CHECK([java create_and_exit])
+# Unlock the file
+AT_CHECK([${COBJ_IDX} unlock indexed_file.dat])
+# Check the record lock is released
+AT_CHECK([java check_record_lock], [0],
+[00
+])
+
+AT_CLEANUP
