Oracle upsert perfomance issue to read the bulk data_insert using merge statement.

[Jeevananthan-23]

async def fast_oracle_bulk_merge(data_insert):
    """
    High-performance Oracle bulk MERGE operation
    Returns number of processed rows
    """
    if not data_insert:
        return 0

    try:
        # 1. Establish connection with optimized settings
        conn = await oracledb.connect_async(
            user=DB_USER,
            password=DB_PASSWORD,
            dsn=f"{DB_HOST}:{DB_PORT}/{DB_SERVICENAME}",
            threaded=True,  # Better for bulk operations
            events=False,   # Disable unused features
            cclass="BULK_MERGE"  # Connection class for tracking
        )

        # 2. Prepare MERGE statement with numbered parameters
        merge_sql = """
        MERGE INTO TB_ABSA_TRAIN_DATA_MASTER t
        USING (SELECT :1 AS category_name, :2 AS keyword FROM dual) src
        ON (t.CATEGORY_NAME = src.category_name AND t.KEYWORD = src.keyword)
        WHEN NOT MATCHED THEN
            INSERT (CATEGORY_NAME, KEYWORD, CATEGORY_CODE, SOURCE, STATUS)
            VALUES (:1, :2, :3, :4, :5)
        """

        # 3. Process in batches with optimal settings
        batch_size = 10000  # Adjust based on testing
        processed = 0
        
        for i in range(0, len(data_insert), batch_size):
            batch = data_insert[i:i + batch_size]
            
            # Convert to tuples for better performance
            params = [
                (
                    item['category_name'],
                    item['keyword'],
                    item['category_code'],
                    item['source'],
                    item['status']
                )
                for item in batch
            ]

            # 4. Execute with optimized settings
            cursor = conn.cursor()
            await cursor.executemany(
                merge_sql,
                params,
                batcherrors=True,
                batch_errors=True,
                arraydmlrowcounts=True
            )
            
            # 5. Get actual affected rows
            processed += sum(await cursor.getarraydmlrowcounts())
            
            # 6. Handle any batch errors
            for error in await cursor.getbatcherrors():
                logger.error(f"Row {error.offset} failed: {error.message}")
            
            await cursor.close()

        await conn.commit()
        return processed

    except Exception as e:
        logger.error(f"Bulk merge failed: {str(e)}")
        if 'conn' in locals():
            await conn.rollback()
        raise
    finally:
        if 'conn' in locals():
            await conn.close()

Very slow in performance my JS version is execute the same data within 11mins. Record count is ~100k.

[cjbj]

Which bit of this is slow? The Python language? The connection creation? The conversion in Python to tuples? The actual executemany() call? How do each of those compare with node-oracledb (I assume you mean this)?

Refer to https://python-oracledb.readthedocs.io/en/latest/user_guide/batch_statement.html#predefining-memory-areas, which can help avoid memory reallocations if your data has varying lengths.

[cjbj]

PS python-oracledb 2.0 (and later) connections will throw an unexpected keyword argument error if you pass the threaded argument in connect calls, see the deprecations doc. This parameter was ignored in all previous python-oracledb releases. Make sure you are using the latest python-oracledb version.