transforms/universal/fdedup/fdedup_ray.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "afd55886-5f5b-4794-838e-ef8179fb0394",
   "metadata": {},
   "source": [
    "##### **** These pip installs need to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "```\n",
    "make venv\n",
    "source venv/bin/activate && pip install jupyterlab\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c45c3c6-e4d7-4e61-8de6-32d61f2ce695",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "#!pip install data-prep-toolkit\n",
    "#!pip install data-prep-toolkit-transforms\n",
    "#!pip install data-prep-connector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf1f782-0e61-485c-8670-81066beb734c",
   "metadata": {},
   "source": [
    "##### ***** Import required Classes and modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c2a12abc-9460-4e45-8961-873b48a9ab19",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-26 13:30:56,482\tINFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.\n"
     ]
    }
   ],
   "source": [
    "import ast\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from data_processing.utils import ParamsUtils\n",
    "from fdedup_transform_python import parse_args\n",
    "from fdedup_transform_ray import RayServiceOrchestrator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7234563c-2924-4150-8a31-4aec98c1bf33",
   "metadata": {},
   "source": [
    "##### ***** Setup runtime parameters for this transform\n",
    "We will only provide a description for the parameters used in this example. For a complete list of parameters, please refer to the README.md for this transform:\n",
    "|parameter:type | value | description |\n",
    "|-|-|-|\n",
    "| input_folder:str | \\${PWD}/ray/test-data/input/ | folder that contains the input parquet files for the fuzzy dedup algorithm |\n",
    "| output_folder:str | \\${PWD}/ray/output/ | folder that contains the all the intermediate results and the output parquet files for the fuzzy dedup algorithm |\n",
    "| contents_column:str | contents | name of the column that stores document text |\n",
    "| document_id_column:str | int_id_column | name of the column that stores document ID |\n",
    "| num_permutations:int | 112 | number of permutations to use for minhash calculation |\n",
    "| num_bands:int | 14 | number of bands to use for band hash calculation |\n",
    "| num_minhashes_per_band | 8 | number of minhashes to use in each band |\n",
    "| operation_mode:{filter_duplicates,filter_non_duplicates,annotate} | filter_duplicates | operation mode for data cleanup: filter out duplicates/non-duplicates, or annotate duplicate documents |\n",
    "| run_locally:bool | true | if true, launch a ray cluster locally, otherwise connect to an already existing cluster | \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e90a853e-412f-45d7-af3d-959e755aeebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create parameters\n",
    "input_folder = os.path.join(os.path.abspath(\"\"), \"ray\", \"test-data\", \"input\")\n",
    "output_folder = os.path.join(os.path.abspath(\"\"), \"ray\", \"output\")\n",
    "params = {\n",
    "    # transform configuration parameters\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "    \"contents_column\": \"contents\",\n",
    "    \"document_id_column\": \"int_id_column\",\n",
    "    \"num_permutations\": 112,\n",
    "    \"num_bands\": 14,\n",
    "    \"num_minhashes_per_band\": 8,\n",
    "    \"operation_mode\": \"filter_duplicates\",\n",
    "    # ray configuration parameters\n",
    "    \"run_locally\": True,\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7949f66a-d207-45ef-9ad7-ad9406f8d42a",
   "metadata": {},
   "source": [
    "##### ***** Use ray runtime to invoke each transform in the fuzzy dedup pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0775e400-7469-49a6-8998-bd4772931459",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:30:57 INFO - Starting SignatureCalculation step\n",
      "13:30:57 INFO - Got parameters for SignatureCalculation\n",
      "13:30:57 INFO - minhash parameters are : {'document_id_column': 'int_id_column', 'contents_column': 'contents', 'seed': 42, 'num_permutations': 112, 'jaccard_similarity_threshold': 0.75, 'word_shingle_size': 5, 'num_bands': 14, 'num_minhashes_per_band': 8, 'num_segments': 1, 'shingle_option': 'word'}\n",
      "13:30:57 INFO - data factory scdata_ is using local configuration without input/output path\n",
      "13:30:57 INFO - data factory scdata_ max_files -1, n_sample -1\n",
      "13:30:57 INFO - data factory scdata_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:30:57 INFO - pipeline id pipeline_id\n",
      "13:30:57 INFO - code location None\n",
      "13:30:57 INFO - number of workers 3 worker options {'num_cpus': 0.8, 'max_restarts': -1}\n",
      "13:30:57 INFO - actor creation delay 0\n",
      "13:30:57 INFO - job details {'job category': 'preprocessing', 'job name': 'minhash', 'job type': 'ray', 'job id': 'job_id'}\n",
      "13:30:57 INFO - data factory data_ is using local data access: input_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/test-data/input output_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output\n",
      "13:30:57 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:30:57 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:30:57 INFO - Running locally\n",
      "2024-11-26 13:31:08,860\tINFO worker.py:1777 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:12 INFO - orchestrator started at 2024-11-26 13:31:12\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:12 INFO - Number of files is 1, source profile {'max_file_size': 0.003920555114746094, 'min_file_size': 0.003920555114746094, 'total_file_size': 0.003920555114746094}\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:12 INFO - Cluster resources: {'cpus': 12, 'gpus': 0, 'memory': 11.162438202649355, 'object_store': 2.0}\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:12 INFO - Number of workers - 3 with {'num_cpus': 0.8, 'max_restarts': -1} each\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:14 INFO - Completed 0 files (0.0%)  in 0.0 min. Waiting for completion\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:14 INFO - Completed processing 1 files in 0.002 min\n",
      "\u001b[36m(RayTransformFileProcessor pid=86984)\u001b[0m 13:31:14 INFO - Starting flush()\n",
      "\u001b[36m(orchestrate pid=86958)\u001b[0m 13:31:14 INFO - done flushing in 0.045 sec\n",
      "\u001b[36m(RayTransformFileProcessor pid=86984)\u001b[0m 13:31:14 INFO - Wrote 14 tables with a total size of 80,640 bytes\n",
      "13:31:24 INFO - Completed execution in 0.446 min, execution result 0\n",
      "13:31:26 INFO - SignatureCalculation completed successfully\n",
      "13:31:26 INFO - Starting ClusterAnalysis step\n",
      "13:31:26 INFO - Got parameters for ClusterAnalysis\n",
      "13:31:26 INFO - cluster parameters are : {'jaccard_similarity_threshold': 0.75, 'num_bands': 14, 'num_segments': 1, 'sort_output': False}\n",
      "13:31:26 INFO - pipeline id pipeline_id\n",
      "13:31:26 INFO - code location None\n",
      "13:31:26 INFO - number of workers 3 worker options {'num_cpus': 0.8, 'max_restarts': -1}\n",
      "13:31:26 INFO - actor creation delay 0\n",
      "13:31:26 INFO - job details {'job category': 'preprocessing', 'job name': 'cluster', 'job type': 'ray', 'job id': 'job_id'}\n",
      "13:31:26 INFO - data factory data_ is using local data access: input_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output/bands output_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output/docs_to_remove\n",
      "13:31:26 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:31:26 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:31:26 INFO - Running locally\n",
      "2024-11-26 13:31:28,318\tINFO worker.py:1777 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:31 INFO - orchestrator started at 2024-11-26 13:31:31\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:31 INFO - Number of folders is 14\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:31 INFO - Cluster resources: {'cpus': 12, 'gpus': 0, 'memory': 11.77626838721335, 'object_store': 2.0}\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:31 INFO - Number of workers - 3 with {'num_cpus': 0.8, 'max_restarts': -1} each\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 1 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 2 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 3 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 4 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 5 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 6 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 7 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 8 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 9 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 10 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 11 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed 11 files (78.571%)  in 0.001 min. Waiting for completion\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - Completed processing 14 files in 0.001 min\n",
      "\u001b[36m(orchestrate pid=87057)\u001b[0m 13:31:33 INFO - done flushing in 0.001 sec\n",
      "13:31:43 INFO - Completed execution in 0.292 min, execution result 0\n",
      "13:31:45 INFO - ClusterAnalysis completed successfully\n",
      "13:31:45 INFO - Starting GetDuplicateList step\n",
      "13:31:45 INFO - Got parameters for GetDuplicateList\n",
      "13:31:45 INFO - fdlist parameters are : {'docs_to_remove': 'docs_to_remove', 'consolidated_filename': 'docs_to_remove_consolidated/docs_to_remove_consolidated.parquet', 'sort_output': False}\n",
      "13:31:45 INFO - pipeline id pipeline_id\n",
      "13:31:45 INFO - code location None\n",
      "13:31:45 INFO - number of workers 1 worker options {'num_cpus': 0.8, 'max_restarts': -1}\n",
      "13:31:45 INFO - actor creation delay 0\n",
      "13:31:45 INFO - job details {'job category': 'preprocessing', 'job name': 'fdlist', 'job type': 'ray', 'job id': 'job_id'}\n",
      "13:31:45 INFO - data factory data_ is using local data access: input_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output output_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output\n",
      "13:31:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:31:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:31:45 INFO - Running locally\n",
      "2024-11-26 13:31:47,311\tINFO worker.py:1777 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:50 INFO - orchestrator started at 2024-11-26 13:31:50\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:50 INFO - Number of folders is 1\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:50 INFO - Cluster resources: {'cpus': 12, 'gpus': 0, 'memory': 11.749520111829042, 'object_store': 2.0}\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:50 INFO - Number of workers - 1 with {'num_cpus': 0.8, 'max_restarts': -1} each\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:52 INFO - Completed 0 files (0.0%)  in 0.0 min. Waiting for completion\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:52 INFO - Completed processing 1 files in 0.0 min\n",
      "\u001b[36m(orchestrate pid=87134)\u001b[0m 13:31:52 INFO - done flushing in 0.001 sec\n",
      "\u001b[36m(RayTransformFileProcessor pid=87153)\u001b[0m 13:31:52 INFO - Get Duplicate List for folder docs_to_remove\n",
      "\u001b[36m(RayTransformFileProcessor pid=87153)\u001b[0m 13:31:52 INFO - 8 documents marked as duplicates\n",
      "13:32:02 INFO - Completed execution in 0.295 min, execution result 0\n",
      "13:32:04 INFO - GetDuplicateList completed successfully\n",
      "13:32:04 INFO - Starting DataCleaning step\n",
      "13:32:04 INFO - Got parameters for DataCleaning\n",
      "13:32:04 INFO - fdclean parameters are : {'document_id_column': 'int_id_column', 'duplicate_list_location': 'docs_to_remove_consolidated/docs_to_remove_consolidated.parquet', 'operation_mode': 'filter_duplicates'}\n",
      "13:32:04 INFO - data factory dcdata_ is using local configuration without input/output path\n",
      "13:32:04 INFO - data factory dcdata_ max_files -1, n_sample -1\n",
      "13:32:04 INFO - data factory dcdata_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:32:04 INFO - pipeline id pipeline_id\n",
      "13:32:04 INFO - code location None\n",
      "13:32:04 INFO - number of workers 3 worker options {'num_cpus': 0.8, 'max_restarts': -1}\n",
      "13:32:04 INFO - actor creation delay 0\n",
      "13:32:04 INFO - job details {'job category': 'preprocessing', 'job name': 'fdclean', 'job type': 'ray', 'job id': 'job_id'}\n",
      "13:32:04 INFO - data factory data_ is using local data access: input_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/test-data/input output_folder - /Users/touma/data-prep-kit/transforms/universal/fdedup/ray/output/cleaned\n",
      "13:32:04 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:32:04 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:32:04 INFO - Running locally\n",
      "2024-11-26 13:32:07,526\tINFO worker.py:1777 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:10 INFO - orchestrator started at 2024-11-26 13:32:10\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:10 INFO - Number of files is 1, source profile {'max_file_size': 0.003920555114746094, 'min_file_size': 0.003920555114746094, 'total_file_size': 0.003920555114746094}\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:10 INFO - Cluster resources: {'cpus': 12, 'gpus': 0, 'memory': 11.738976669497788, 'object_store': 2.0}\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:10 INFO - Number of workers - 3 with {'num_cpus': 0.8, 'max_restarts': -1} each\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:13 INFO - Completed 0 files (0.0%)  in 0.0 min. Waiting for completion\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:13 INFO - Completed processing 1 files in 0.002 min\n",
      "\u001b[36m(orchestrate pid=87217)\u001b[0m 13:32:13 INFO - done flushing in 0.003 sec\n",
      "13:32:23 INFO - Completed execution in 0.313 min, execution result 0\n",
      "13:32:24 INFO - DataCleaning completed successfully\n"
     ]
    }
   ],
   "source": [
    "\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "args = parse_args()\n",
    "# Initialize the orchestrator\n",
    "orchestrator = RayServiceOrchestrator(global_params=args)\n",
    "# Launch ray fuzzy dedup execution\n",
    "orchestrator.orchestrate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3df5adf-4717-4a03-864d-9151cd3f134b",
   "metadata": {},
   "source": [
    "##### **** The specified folder will include the transformed parquet files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7276fe84-6512-4605-ab65-747351e13a7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ray/output/cleaned/metadata.json', 'ray/output/cleaned/df1.parquet']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "glob.glob(\"ray/output/cleaned/*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d30489d9-fc98-423e-90a8-e8f372787e88",
   "metadata": {},
   "source": [
    "***** print the input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5b22234f-f7a1-4b92-b2ac-376b2545abce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (12, 2)\n",
      "┌───────────────┬──────────────────────────────────────────────────────────────────────────────────┐\n",
      "│ int_id_column ┆ contents                                                                         │\n",
      "│ ---           ┆ ---                                                                              │\n",
      "│ i64           ┆ str                                                                              │\n",
      "╞═══════════════╪══════════════════════════════════════════════════════════════════════════════════╡\n",
      "│ 1             ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ PR Newswire October 12, 2019                                                     │\n",
      "│               ┆ 145-year-old Retailer Anchors Woodland Mall Just Outside Grand Rapids;           │\n",
      "│               ┆ New Location Continues Strategic National Expansion Plans                        │\n",
      "│               ┆ DAVENPORT, Iowa, Oct. 12, 2019 /PRNewswire/ -- Von Maur Department Stores opened │\n",
      "│               ┆ a new store today at Woodland Mall in Kentwood, Mich. The 90,000-square-foot     │\n",
      "│               ┆ store is the Company's third location in Michigan.                               │\n",
      "│               ┆ Known for its outstanding selection of brand name and specialty apparel, shoes,  │\n",
      "│               ┆ accessories and gifts, the store features products from leading brands such as   │\n",
      "│               ┆ Eileen Fisher, Vineyard Vines, Free People, and Kendra Scott, among many others. │\n",
      "│               ┆ Von Maur is also widely-regarded for its superior customer service, including an │\n",
      "│               ┆ interest-free charge card, accommodating return policy, free gift wrapping and   │\n",
      "│               ┆ free shipping services.                                                          │\n",
      "│               ┆ Today's opening continues to build upon the momentum of the family-owned         │\n",
      "│               ┆ Company's targeted national growth strategy. Von Maur opened its first Wisconsin │\n",
      "│               ┆ location in 2017 and a second Minnesota location in 2018, and it has grown in    │\n",
      "│               ┆ new states beyond its Midwestern footprint, including New York, Alabama and      │\n",
      "│               ┆ Oklahoma. Additionally, the Company has plans to open its second Wisconsin       │\n",
      "│               ┆ location in Madison in Fall 2021.                                                │\n",
      "│               ┆ \"With its easy accessibility to the larger Grand Rapids area and exceptional     │\n",
      "│               ┆ collection of shopping, dining and entertainment options, Woodland Mall is a     │\n",
      "│               ┆ fantastic location for us to continue growing our brand in Michigan,\" said Jim   │\n",
      "│               ┆ von Maur, president of Von Maur. \"From the moment shoppers walk through our      │\n",
      "│               ┆ doors, creating an unrivaled shopping experience is the motivation behind        │\n",
      "│               ┆ everything we do. We look forward to extending our offerings of brand name       │\n",
      "│               ┆ merchandise and signature customer service to the Grand Rapids area for many     │\n",
      "│               ┆ years to come.\"                                                                  │\n",
      "│               ┆ \"We are thrilled to welcome Von Maur, known for their high-quality merchandise   │\n",
      "│               ┆ and exceptional service, as the anchor of the newly developed wing at Woodland   │\n",
      "│               ┆ Mall,\" said Joe Coradino, CEO of PREIT. \"The addition most certainly solidifies  │\n",
      "│               ┆ Woodland Mall's place as the premier retail and entertainment destination in     │\n",
      "│               ┆ Grand Rapids, driving its place as a top-performing PREIT property.\"             │\n",
      "│               ┆ Centrally-located for shoppers from Grand Rapids and the surrounding areas, the  │\n",
      "│               ┆ new single story Von Maur store features the Company's signature exterior brick  │\n",
      "│               ┆ façade, open expansive floor plan, and residential ambiance, including music     │\n",
      "│               ┆ from the store's grand piano.                                                    │\n",
      "│               ┆ The Woodland Mall store will eventually employ up to 150 associates; the         │\n",
      "│               ┆ majority of them will be full-time. Von Maur offers above-market wages,          │\n",
      "│               ┆ excellent benefits and a positive, professional work environment. Hours of       │\n",
      "│               ┆ operation are Monday to Saturday, 10 a.m. – 9 p.m. ET, and Sunday, 12 p.m. – 6   │\n",
      "│               ┆ p.m. ET.                                                                         │\n",
      "│               ┆ About Von Maur                                                                   │\n",
      "│               ┆ Von Maur was founded 145 years ago in downtown Davenport, Iowa. The Company      │\n",
      "│               ┆ currently operates 35 stores in 15 states, along with a 120,000 square foot      │\n",
      "│               ┆ E-Commerce facility that drives its successful online business at vonmaur.com.   │\n",
      "│               ┆ Courtney Smith                                                                   │\n",
      "│               ┆ courtney@reputationpartners.com                                                  │\n",
      "│               ┆ View original content:http://www.prnewswire.com/news-releases/von-maur-departmen │\n",
      "│               ┆ t-store-opens-third-location-in-michigan-300937186.html                          │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│ 3             ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful minds of our time about  │\n",
      "│               ┆ what it means to live like a Genius.                                             │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "│ 4             ┆                                                                                  │\n",
      "│               ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful                          │\n",
      "│               ┆ minds of our time about what it means to live like a Genius.                     │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "│               ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│               ┆                                                                                  │\n",
      "│ 5             ┆                                                                                  │\n",
      "│               ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│               ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful                          │\n",
      "│               ┆ minds of our time about what it means to live like a Genius.                     │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "│               ┆                                                                                  │\n",
      "│ 6             ┆                                                                                  │\n",
      "│               ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful                          │\n",
      "│               ┆ minds of our time about what it means to live like a Genius.                     │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│               ┆                                                                                  │\n",
      "│ 11            ┆ A couple of capricious capybaras chatted coolly by the cactus, curiously         │\n",
      "│               ┆ considering another capy capably chewing on cantaloupe. Yesterday, a pair of     │\n",
      "│               ┆ capricious pigeons prattled placidly by the cactus, curiously considering        │\n",
      "│               ┆ another pigeon capably pecking at cantaloupe. The lazy llama lightly limped      │\n",
      "│               ┆ through the lilacs, laboriously longing for a lozenge                            │\n",
      "│ 12            ┆ Yesterday, a pair of capricious pigeons prattled placidly by the cactus,         │\n",
      "│               ┆ curiously considering another pigeon capably pecking at cantaloupe. The lazy     │\n",
      "│               ┆ llama lightly limped through the lilacs, laboriously longing for a lozenge. A    │\n",
      "│               ┆ couple of capricious capybaras chatted coolly by the cactus, curiously           │\n",
      "│               ┆ considering another capy capably chewing on cantaloupe.                          │\n",
      "│ 13            ┆ The lazy llama lightly limped through the lilacs, laboriously longing for a      │\n",
      "│               ┆ lozenge. A couple of capricious capybaras chatted coolly by the cactus,          │\n",
      "│               ┆ curiously considering another capy capably chewing on cantaloupe. Yesterday, a   │\n",
      "│               ┆ pair of capricious pigeons prattled placidly by the cactus, curiously            │\n",
      "│               ┆ considering another pigeon capably pecking at cantaloupe.                        │\n",
      "│ 14            ┆ Yesterday, a pair of capricious pigeons prattled placidly by the cactus,         │\n",
      "│               ┆ curiously considering another pigeon capably pecking at cantaloupe. The lazy     │\n",
      "│               ┆ llama lightly limped through the lilacs, laboriously longing for a lozenge. A    │\n",
      "│               ┆ couple of capricious capybaras chatted coolly by the cactus, curiously pondering │\n",
      "│               ┆ another capy capably chewing on cantaloupe                                       │\n",
      "│ 15            ┆ The new sheepskin leather coat with natural fur is 46-48 times warmer. The color │\n",
      "│               ┆ is very beautiful bright green looks very beautiful. Purchased by the shopping   │\n",
      "│               ┆ center Dubrovka 19 000 now in the store the price is 22000-24000 call any time.  │\n",
      "│ 16            ┆ New sheepskin leather coat with natural fur is 50 times warmer. The color is     │\n",
      "│               ┆ very beautiful bright green looks very beautiful. Purchased by the shopping      │\n",
      "│               ┆ center Dubrovka 19 000 now in the store the price is 22000-24000 call any time.  │\n",
      "│ 17            ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful minds of our time about  │\n",
      "│               ┆ what it means to live like a Genius.                                             │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "└───────────────┴──────────────────────────────────────────────────────────────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import polars as pl\n",
    "input_df = pl.read_parquet(os.path.join(os.path.abspath(\"\"), \"ray\", \"test-data\", \"input\", \"df1.parquet\"))\n",
    "with pl.Config(fmt_str_lengths=10000000, tbl_rows=-1):\n",
    "    print(input_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5305d127-10fd-4fa6-97a6-ac47db2bdc7e",
   "metadata": {},
   "source": [
    "***** print the output result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0b2eddb9-4fb6-41eb-916c-3741b9129f2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 2)\n",
      "┌───────────────┬──────────────────────────────────────────────────────────────────────────────────┐\n",
      "│ int_id_column ┆ contents                                                                         │\n",
      "│ ---           ┆ ---                                                                              │\n",
      "│ i64           ┆ str                                                                              │\n",
      "╞═══════════════╪══════════════════════════════════════════════════════════════════════════════════╡\n",
      "│ 1             ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ PR Newswire October 12, 2019                                                     │\n",
      "│               ┆ 145-year-old Retailer Anchors Woodland Mall Just Outside Grand Rapids;           │\n",
      "│               ┆ New Location Continues Strategic National Expansion Plans                        │\n",
      "│               ┆ DAVENPORT, Iowa, Oct. 12, 2019 /PRNewswire/ -- Von Maur Department Stores opened │\n",
      "│               ┆ a new store today at Woodland Mall in Kentwood, Mich. The 90,000-square-foot     │\n",
      "│               ┆ store is the Company's third location in Michigan.                               │\n",
      "│               ┆ Known for its outstanding selection of brand name and specialty apparel, shoes,  │\n",
      "│               ┆ accessories and gifts, the store features products from leading brands such as   │\n",
      "│               ┆ Eileen Fisher, Vineyard Vines, Free People, and Kendra Scott, among many others. │\n",
      "│               ┆ Von Maur is also widely-regarded for its superior customer service, including an │\n",
      "│               ┆ interest-free charge card, accommodating return policy, free gift wrapping and   │\n",
      "│               ┆ free shipping services.                                                          │\n",
      "│               ┆ Today's opening continues to build upon the momentum of the family-owned         │\n",
      "│               ┆ Company's targeted national growth strategy. Von Maur opened its first Wisconsin │\n",
      "│               ┆ location in 2017 and a second Minnesota location in 2018, and it has grown in    │\n",
      "│               ┆ new states beyond its Midwestern footprint, including New York, Alabama and      │\n",
      "│               ┆ Oklahoma. Additionally, the Company has plans to open its second Wisconsin       │\n",
      "│               ┆ location in Madison in Fall 2021.                                                │\n",
      "│               ┆ \"With its easy accessibility to the larger Grand Rapids area and exceptional     │\n",
      "│               ┆ collection of shopping, dining and entertainment options, Woodland Mall is a     │\n",
      "│               ┆ fantastic location for us to continue growing our brand in Michigan,\" said Jim   │\n",
      "│               ┆ von Maur, president of Von Maur. \"From the moment shoppers walk through our      │\n",
      "│               ┆ doors, creating an unrivaled shopping experience is the motivation behind        │\n",
      "│               ┆ everything we do. We look forward to extending our offerings of brand name       │\n",
      "│               ┆ merchandise and signature customer service to the Grand Rapids area for many     │\n",
      "│               ┆ years to come.\"                                                                  │\n",
      "│               ┆ \"We are thrilled to welcome Von Maur, known for their high-quality merchandise   │\n",
      "│               ┆ and exceptional service, as the anchor of the newly developed wing at Woodland   │\n",
      "│               ┆ Mall,\" said Joe Coradino, CEO of PREIT. \"The addition most certainly solidifies  │\n",
      "│               ┆ Woodland Mall's place as the premier retail and entertainment destination in     │\n",
      "│               ┆ Grand Rapids, driving its place as a top-performing PREIT property.\"             │\n",
      "│               ┆ Centrally-located for shoppers from Grand Rapids and the surrounding areas, the  │\n",
      "│               ┆ new single story Von Maur store features the Company's signature exterior brick  │\n",
      "│               ┆ façade, open expansive floor plan, and residential ambiance, including music     │\n",
      "│               ┆ from the store's grand piano.                                                    │\n",
      "│               ┆ The Woodland Mall store will eventually employ up to 150 associates; the         │\n",
      "│               ┆ majority of them will be full-time. Von Maur offers above-market wages,          │\n",
      "│               ┆ excellent benefits and a positive, professional work environment. Hours of       │\n",
      "│               ┆ operation are Monday to Saturday, 10 a.m. – 9 p.m. ET, and Sunday, 12 p.m. – 6   │\n",
      "│               ┆ p.m. ET.                                                                         │\n",
      "│               ┆ About Von Maur                                                                   │\n",
      "│               ┆ Von Maur was founded 145 years ago in downtown Davenport, Iowa. The Company      │\n",
      "│               ┆ currently operates 35 stores in 15 states, along with a 120,000 square foot      │\n",
      "│               ┆ E-Commerce facility that drives its successful online business at vonmaur.com.   │\n",
      "│               ┆ Courtney Smith                                                                   │\n",
      "│               ┆ courtney@reputationpartners.com                                                  │\n",
      "│               ┆ View original content:http://www.prnewswire.com/news-releases/von-maur-departmen │\n",
      "│               ┆ t-store-opens-third-location-in-michigan-300937186.html                          │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│ 4             ┆                                                                                  │\n",
      "│               ┆ The Genius Life                                                                  │\n",
      "│               ┆ Max Lugavere                                                                     │\n",
      "│               ┆ You don't have to be born a Genius to become one. Follow health and science      │\n",
      "│               ┆ journalist, New York Times bestselling author, TV personality and nutrition      │\n",
      "│               ┆ expert Max Lugavere as he speaks to the most insightful                          │\n",
      "│               ┆ minds of our time about what it means to live like a Genius.                     │\n",
      "│               ┆ 35: How Wheat, Carbs, and Sugar Can Harm Your Brain | David Perlmutter, MD       │\n",
      "│               ┆ David Perlmutter, MD is a board-certified neurologist, Fellow of the American    │\n",
      "│               ┆ College of Nutrition, and the New York Times best-selling author of Brain Maker  │\n",
      "│               ┆ and Grain Brain, now updated with the latest nutritional and neurological        │\n",
      "│               ┆ science.                                                                         │\n",
      "│               ┆ Von Maur Department Store Opens Third Location in Michigan                       │\n",
      "│               ┆ Zuckerberg on Libra drop outs: 'It's a risky project'                            │\n",
      "│               ┆                                                                                  │\n",
      "│ 12            ┆ Yesterday, a pair of capricious pigeons prattled placidly by the cactus,         │\n",
      "│               ┆ curiously considering another pigeon capably pecking at cantaloupe. The lazy     │\n",
      "│               ┆ llama lightly limped through the lilacs, laboriously longing for a lozenge. A    │\n",
      "│               ┆ couple of capricious capybaras chatted coolly by the cactus, curiously           │\n",
      "│               ┆ considering another capy capably chewing on cantaloupe.                          │\n",
      "│ 15            ┆ The new sheepskin leather coat with natural fur is 46-48 times warmer. The color │\n",
      "│               ┆ is very beautiful bright green looks very beautiful. Purchased by the shopping   │\n",
      "│               ┆ center Dubrovka 19 000 now in the store the price is 22000-24000 call any time.  │\n",
      "└───────────────┴──────────────────────────────────────────────────────────────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import polars as pl\n",
    "output_df = pl.read_parquet(os.path.join(os.path.abspath(\"\"), \"ray\", \"output\", \"cleaned\", \"df1.parquet\"))\n",
    "with pl.Config(fmt_str_lengths=10000000, tbl_rows=-1):\n",
    "    print(output_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d60e391d-cf58-47ae-9991-04c05d114edc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c11d3a4b-8ef9-417d-a8a2-f688db067a52",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}