Circular_chunks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d1f912ad-f93f-4294-9513-855e71256c9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unstructured.partition.pdf import partition_pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "22b0becd-f530-4970-b37b-03a2bcda69a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_path = \"/data/saikat/unstructured_io_poc/data/1A.pdf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "e4bd1c39-aca2-4e41-bd05-d78380090357",
   "metadata": {},
   "outputs": [],
   "source": [
    "elements = partition_pdf(pdf_path, infer_table_structure = True, hi_res_model_name='yolox')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "90598216-ffe1-4a77-822c-2a3d636e55ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_elements = [str(el) for el in elements]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "8b3ef73a-2713-46ed-847c-f47a405c5b79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Threshold Conditions',\n",
       " 'Chapter 1',\n",
       " 'Introduction',\n",
       " '1',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " '1.1A',\n",
       " 'Application',\n",
       " 'To which threshold conditions does COND apply? ..................................................................................................... 1.1A.1 G (1) Section 55C of the Financial Services Act 2012 (Power to amend Schedule 6) gave HM Treasury the power to amend Schedule 6 of the Act. HM Treasury exercised this power by making The Financial Services and Markets Act 2000 (Threshold Conditions) Order 2013 which entered into force on 1 April 2013 (the \"TC Order\"). The TC Order\\'s main result is the creation of four sets of threshold conditions, namely: (i) conditions for firms authorised and regulated by the FCA only (paragraphs 2B to 2F of Schedule 6 to the Act) (ii) FCA specific conditions for firms authorised by the PRA and subject to dual regulation (paragraphs 3B to 3E of Schedule 6 to the Act); (iii) PRA-specific conditions for insurers (paragraphs 4A to 4F of Schedule 6 to the Act); and (iv) PRA-specific conditions for other PRA-authorised persons (paragraphs 5A to 5F of Schedule 6 to the Act). (2) The guidance in COND is only applicable to the threshold conditions listed in n COND 1.1A.1G(1)(i) and (ii), above. These are the threshold conditions stated in paragraphs 2A and 3A of Schedule 6 to the Act as being relevant to the discharge by the FCA of its functions under the Act.',\n",
       " '(3) In respect of a person which does not carry on, or seek to carry on, any PRA-regulated activities, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 2B to 2F of Schedule 6 to the Act, subject to n COND 1.1A.4G (1), below.',\n",
       " '(4) In respect of a firm which does carry on, or seeks to carry on, a PRA- regulated activity, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 3B to 3E of Schedule 6 to the Act, subject to n COND 1.1A.4G (2), below.',\n",
       " '(5) A reference to \"FCA threshold conditions\" in COND means a reference to the threshold conditions referred to in (3) and (4) that apply to a particular firm.',\n",
       " 'COND 1/2',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'n Release 36 l May 2024',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " 'To whom does COND apply? .....................................................................................................',\n",
       " '1.1A.2',\n",
       " 'G',\n",
       " '(1) COND applies to all firms, except where stated otherwise in this',\n",
       " 'guidance.',\n",
       " \"(2) In COND, 'firm' includes an applicant for Part 4A permission unless the context otherwise requires.\",\n",
       " 'To what extent does COND apply to firms authorised by the PRA (PRA-authorised persons) and subject to dual regulation? .....................................................................................................',\n",
       " '1.1A.3',\n",
       " 'G',\n",
       " '(1) As a result of the new legal framework for threshold conditions described in n COND 1.1A.1G (1), PRA-authorised persons and firms seeking to become PRA-authorised persons are subject to two sets of threshold conditions:',\n",
       " '(i) the FCA-specific conditions referred to in n COND 1.1A.1G (1)(ii)and',\n",
       " '(ii) one of the two PRA-specific conditions referred to in n COND 1.1A.1G (1)(iii) or n (iv), depending on the PRA-regulated activities which the PRA-authorised person or firm carries on, or is seeking to carry on.',\n",
       " 'The FCA threshold conditions set out in paragraphs 3B to 3E of the Act seek to reflect this. In particular, these threshold conditions do not contain a condition relating to adequate financial resources. This is a matter that falls to be considered by the PRA under its threshold conditions.',\n",
       " 'understand how the FCA will approach its assessment of the applicable FCA threshold conditions, regardless of whether or not a firm is, or is seeking to become, a PRA-authorised person. This is because the FCA threshold conditions which apply to PRA-authorised persons and those which apply to firms authorised by the FCA only are, for the most part, the same.',\n",
       " \"(3) However, where guidance in COND refers to an assessment of a firm's financial position or its compliance with prudential regulatory requirements, it is not intended to assist firms which are, or are seeking to become, PRA-authorised persons. This is because these are matters that are not covered by the FCA's threshold conditions, but rather fall to be considered by the PRA under its threshold conditions.\",\n",
       " '(4) Although some of the PRA threshold conditions and FCA threshold conditions that apply to firms which are, or are seeking to become, PRA-authorised persons may appear to address similar subject matter, the FCA will approach the assessment of its threshold conditions with its unique statutory objectives in mind and in the light of the functions which the FCA is required to discharge in relation to them.',\n",
       " \"(5) For the avoidance of doubt, the guidance in COND is not intended to apply to the PRA's assessment of its own threshold conditions in respect of a PRA-authorised person. This is a matter for the PRA alone.\",\n",
       " '1.1A.4',\n",
       " 'G',\n",
       " '[deleted]',\n",
       " 'n Release 36 l May 2024',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'COND 1/3',\n",
       " '1',\n",
       " '1',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " '1.1A5',\n",
       " 'To what extent does COND apply to Swiss general insurance companies? ..................................................................................................... FCA threshold conditions 3B, 3C and 3E apply to Swiss General Insurance Companies.',\n",
       " 'To what extent does COND apply to credit firms with limited permission? ..................................................................................................... (1) The FCA threshold conditions apply to a person that carries on, or',\n",
       " 'G seeks to carry on, only relevant credit activities (within paragraph 2G of Schedule 6 to the Act) and which therefore has, or is applying for, limited permission with a number of modifications (see article 10(19) of the Regulated Activities Amendment Order). Regulated activities a person carries on in relation to which sections 20(1) and (1A) and 23(1A) of the Act do not apply as a result of section 39(1D) of the Act are disregarded for this purpose.',\n",
       " '1.1A.5A',\n",
       " '(2) For a person within (1), the FCA threshold conditions are modified as follows:',\n",
       " '(a) in relation to paragraph 2C of Schedule 6 to the Act (Effective supervision), paragraphs (a), (b) and (e) of sub-paragraph (1) do not apply (see n COND 2.3);',\n",
       " '(b) in relation to paragraph 2D of Schedule 6 to the Act (Appropriate resources), the person has adequate financial resources if it is capable of meeting its debts as they fall due (see n COND 2.4);',\n",
       " '(c) paragraph 2F of Schedule 6 to the Act (Business model) does not apply (see n COND 2.7).',\n",
       " '(3) Paragraph 2G of Schedule 6 to the Act defines relevant credit activity for the purposes of the FCA Threshold Conditions. The interpretation of some of the key expressions used in this specific context is as follows:',\n",
       " '(a) \"borrower\" includes any person providing a guarantee or indemnity under an agreement, and a person to whom the rights and duties of the borrower have passed by assignment or operation of law;',\n",
       " '(b) \"supplier\" means a person whose main business is to sell goods or supply services and not to carry on a regulated activity, other than entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owner\\'s rights and duties under a regulated consumer hire agreement;',\n",
       " '(c) \"customer\" means a person to whom a supplier sells goods or supplies services or agrees to do so;',\n",
       " '(d) \"domestic premises supplier\" means a supplier who sells goods, offers or agrees to sell goods, or offers or contracts to supply services, to customers who are individuals while the supplier or the supplier’s representative is physically present in the dwelling of the customer (though a supplier who does so only on an occasional basis is not to be treated as a \"domestic premises supplier” unless the supplier indicates to the public at large, or a section of it, the supplier’s willingness to attend, in person or through a representative, the dwelling of a potential customer in order to do any of those things).',\n",
       " 'COND 1/4',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'n Release 36 l May 2024',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " '(3A) Questions may arise over whether a supplier who visits a customer’s dwelling to take measurements or give an estimate is a “domestic premises supplier”. For example:',\n",
       " '(a) if the supplier, or the supplier’s representative, gives a quote or estimate to the customer during the visit that is sufficiently specific as to be capable of being accepted in a way that is binding on the supplier, then the quote or estimate is an offer; on that basis, the supplier falls within the definition of “domestic premises supplier”, irrespective of whether the customer accepts the offer during the visit;',\n",
       " 'rough estimate or quote during the visit, with a view to submitting a refined estimate or a firm quote at a later time when the supplier is not at the customer’s dwelling, that rough estimate will not be an offer; on that basis, the supplier will not fall within the definition of “domestic premises supplier”, unless the customer and the supplier, or the supplier’s representative, do reach an agreement during the visit; and',\n",
       " '(c) where an agreement is reached, the supplier will have sold, or agreed to sell, goods or contracted to supply services, and will therefore be a “domestic premises supplier”; this may be the case even if the agreement is subject to later specification of the price, the goods or the services.',\n",
       " 'It is immaterial whether the supplier carries on any credit broking (or other regulated activity) during the visit.',\n",
       " '(4) In summary, the following credit-related regulated activities are relevant credit activities for the purposes of the FCA Threshold Conditions:',\n",
       " '(a) credit broking when carried on:',\n",
       " '(i) by a supplier (other than a domestic premises supplier) for the purposes of or in connection with the sale of goods or supply of services by the supplier to a customer (who need not be the borrower under the credit agreement or the hirer under the consumer hire agreement); or',\n",
       " '(ii) in relation to a green deal plan; or',\n",
       " '(iii) in relation to a consumer hire agreement or a hire-purchase agreement;',\n",
       " 'although, other than where the credit broking is carried on by a not-for-profit body, the credit broking will not be a relevant credit activity where it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land;',\n",
       " '(b) consumer credit lending if carried on by a local authority or if:',\n",
       " '(i) it is carried on by a supplier;',\n",
       " '(ii) no charge (by way of interest or otherwise) is payable by the borrower in connection with the provision of credit (this includes a charge payable in connection with a breach of the agreement or on the occurrence of a specified event; consumer credit lending under an agreement that contains such a charge is not a relevant credit activity); and',\n",
       " 'n Release 36 l May 2024',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'COND 1/5',\n",
       " '1',\n",
       " '1',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " '(iii) the regulated credit agreement is not a hire purchase agreement or a conditional sale agreement;',\n",
       " 'although, other than where the consumer credit lending is carried on by a not-for-profit body, the consumer credit lending will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay is secured, or is to be secured, by a legal mortgage on land;',\n",
       " \"(c) entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owner's rights and duties under a regulated consumer hire agreement although, other than where these activities are carried on by a not-for- profit body, entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owner's rights and duties under a regulated consumer hire agreement will not be a relevant credit activity if the obligation of the hirer to pay under the agreement is secured, or is to be secured, by a legal mortgage on land;\",\n",
       " '(d) debt adjusting or debt counselling when carried on: (i) by a supplier who also carries on credit broking within (a)(i); (ii) by a person in connection with an activity within (b) or (c) which the person also carries on; (iii) by a not-for-profit body; although, other than where the debt adjusting or debt counselling is carried on by a not-for-profit body, the debt adjusting or debt counselling will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land; (e) providing credit information services where carried on by a person in connection with an activity within (a) to (d) which the person also carries on; (f) agreeing to carry on an activity within (a) to (e). 1.1A.5B G To what extent does COND apply to regulated benchmark administrators? ..................................................................................................... (1) The threshold conditions do not apply to a firm in relation to the regulated activity of administering a benchmark. (2) COND does not apply to regulated benchmark administrators who are solely authorised to administer a benchmark as they are not subject to the threshold conditions. (3) For regulated benchmark administrators who are also authorised to carry on activities other than administering a benchmark, they will be subject to the threshold conditions in relation to their other regulated activities. COND will apply to those firms in relation to the regulated activities to which the threshold conditions apply.',\n",
       " 'COND 1/6',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'n Release 36 l May 2024',\n",
       " 'COND 1 : Introduction',\n",
       " 'Section 1.1A : Application',\n",
       " '1.1A.6',\n",
       " 'G',\n",
       " 'To which regulated activities does COND apply? ..................................................................................................... Subject to the limitations referred to above, COND applies in relation to all of the regulated activities for which a firm has, or will have, permission.',\n",
       " '1.1A.7',\n",
       " 'G',\n",
       " 'Where does COND apply? ..................................................................................................... COND applies in relation to all of the regulated activities wherever they are carried on.',\n",
       " 'n Release 36 l May 2024',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'www.handbook.fca.org.uk',\n",
       " 'COND 1/7',\n",
       " '1']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f80d413a-78b6-4037-aa58-e7399c9e4d61",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "378858c7-b7ee-4ee3-a20d-787d3fc49f98",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data=text_elements, columns=['chunks'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "47731994-64d0-4d98-90a7-fdd2121420df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chunks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Threshold Conditions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Chapter 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Introduction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>COND 1 : Introduction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>n Release 36 l May 2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>www.handbook.fca.org.uk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>www.handbook.fca.org.uk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>COND 1/7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>108 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      chunks\n",
       "0       Threshold Conditions\n",
       "1                  Chapter 1\n",
       "2               Introduction\n",
       "3                          1\n",
       "4      COND 1 : Introduction\n",
       "..                       ...\n",
       "103  n Release 36 l May 2024\n",
       "104  www.handbook.fca.org.uk\n",
       "105  www.handbook.fca.org.uk\n",
       "106                 COND 1/7\n",
       "107                        1\n",
       "\n",
       "[108 rows x 1 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b35da2de-950f-4f72-b2b8-28d22ef00f27",
   "metadata": {},
   "source": [
    "# With Langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e9ebc539-f809-4988-b5cd-832c99832f98",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "acc266f1-4b3d-4773-ba47-e70331931f18",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import fitz\n",
    "\n",
    "\n",
    "def column_boxes(page, footer_margin=50, header_margin=50, no_image_text=True):\n",
    "    \"\"\"Determine bboxes which wrap a column.\"\"\"\n",
    "    paths = page.get_drawings()\n",
    "    bboxes = []\n",
    "\n",
    "    # path rectangles\n",
    "    path_rects = []\n",
    "\n",
    "    # image bboxes\n",
    "    img_bboxes = []\n",
    "\n",
    "    # bboxes of non-horizontal text\n",
    "    # avoid when expanding horizontal text boxes\n",
    "    vert_bboxes = []\n",
    "\n",
    "    # compute relevant page area\n",
    "    clip = +page.rect\n",
    "    clip.y1 -= footer_margin  # Remove footer area\n",
    "    clip.y0 += header_margin  # Remove header area\n",
    "\n",
    "    def can_extend(temp, bb, bboxlist):\n",
    "        \"\"\"Determines whether rectangle 'temp' can be extended by 'bb'\n",
    "        without intersecting any of the rectangles contained in 'bboxlist'.\n",
    "\n",
    "        Items of bboxlist may be None if they have been removed.\n",
    "\n",
    "        Returns:\n",
    "            True if 'temp' has no intersections with items of 'bboxlist'.\n",
    "        \"\"\"\n",
    "        for b in bboxlist:\n",
    "            if not intersects_bboxes(temp, vert_bboxes) and (\n",
    "                b == None or b == bb or (temp & b).is_empty\n",
    "            ):\n",
    "                continue\n",
    "            return False\n",
    "\n",
    "        return True\n",
    "\n",
    "    def in_bbox(bb, bboxes):\n",
    "        \"\"\"Return 1-based number if a bbox contains bb, else return 0.\"\"\"\n",
    "        for i, bbox in enumerate(bboxes):\n",
    "            if bb in bbox:\n",
    "                return i + 1\n",
    "        return 0\n",
    "\n",
    "    def intersects_bboxes(bb, bboxes):\n",
    "        \"\"\"Return True if a bbox intersects bb, else return False.\"\"\"\n",
    "        for bbox in bboxes:\n",
    "            if not (bb & bbox).is_empty:\n",
    "                return True\n",
    "        return False\n",
    "\n",
    "    def extend_right(bboxes, width, path_bboxes, vert_bboxes, img_bboxes):\n",
    "        \"\"\"Extend a bbox to the right page border.\n",
    "\n",
    "        Whenever there is no text to the right of a bbox, enlarge it up\n",
    "        to the right page border.\n",
    "\n",
    "        Args:\n",
    "            bboxes: (list[IRect]) bboxes to check\n",
    "            width: (int) page width\n",
    "            path_bboxes: (list[IRect]) bboxes with a background color\n",
    "            vert_bboxes: (list[IRect]) bboxes with vertical text\n",
    "            img_bboxes: (list[IRect]) bboxes of images\n",
    "        Returns:\n",
    "            Potentially modified bboxes.\n",
    "        \"\"\"\n",
    "        for i, bb in enumerate(bboxes):\n",
    "            # do not extend text with background color\n",
    "            if in_bbox(bb, path_bboxes):\n",
    "                continue\n",
    "\n",
    "            # do not extend text in images\n",
    "            if in_bbox(bb, img_bboxes):\n",
    "                continue\n",
    "\n",
    "            # temp extends bb to the right page border\n",
    "            temp = +bb\n",
    "            temp.x1 = width\n",
    "\n",
    "            # do not cut through colored background or images\n",
    "            if intersects_bboxes(temp, path_bboxes + vert_bboxes + img_bboxes):\n",
    "                continue\n",
    "\n",
    "            # also, do not intersect other text bboxes\n",
    "            check = can_extend(temp, bb, bboxes)\n",
    "            if check:\n",
    "                bboxes[i] = temp  # replace with enlarged bbox\n",
    "\n",
    "        return [b for b in bboxes if b != None]\n",
    "\n",
    "    def clean_nblocks(nblocks):\n",
    "        \"\"\"Do some elementary cleaning.\"\"\"\n",
    "\n",
    "        # 1. remove any duplicate blocks.\n",
    "        blen = len(nblocks)\n",
    "        if blen < 2:\n",
    "            return nblocks\n",
    "        start = blen - 1\n",
    "        for i in range(start, -1, -1):\n",
    "            bb1 = nblocks[i]\n",
    "            bb0 = nblocks[i - 1]\n",
    "            if bb0 == bb1:\n",
    "                del nblocks[i]\n",
    "\n",
    "        # 2. repair sequence in special cases:\n",
    "        # consecutive bboxes with almost same bottom value are sorted ascending\n",
    "        # by x-coordinate.\n",
    "        y1 = nblocks[0].y1  # first bottom coordinate\n",
    "        i0 = 0  # its index\n",
    "        i1 = -1  # index of last bbox with same bottom\n",
    "\n",
    "        # Iterate over bboxes, identifying segments with approx. same bottom value.\n",
    "        # Replace every segment by its sorted version.\n",
    "        for i in range(1, len(nblocks)):\n",
    "            b1 = nblocks[i]\n",
    "            if abs(b1.y1 - y1) > 10:  # different bottom\n",
    "                if i1 > i0:  # segment length > 1? Sort it!\n",
    "                    nblocks[i0 : i1 + 1] = sorted(\n",
    "                        nblocks[i0 : i1 + 1], key=lambda b: b.x0\n",
    "                    )\n",
    "                y1 = b1.y1  # store new bottom value\n",
    "                i0 = i  # store its start index\n",
    "            i1 = i  # store current index\n",
    "        if i1 > i0:  # segment waiting to be sorted\n",
    "            nblocks[i0 : i1 + 1] = sorted(nblocks[i0 : i1 + 1], key=lambda b: b.x0)\n",
    "        return nblocks\n",
    "\n",
    "    # extract vector graphics\n",
    "    for p in paths:\n",
    "        path_rects.append(p[\"rect\"].irect)\n",
    "    path_bboxes = path_rects\n",
    "\n",
    "    # sort path bboxes by ascending top, then left coordinates\n",
    "    path_bboxes.sort(key=lambda b: (b.y0, b.x0))\n",
    "\n",
    "    # bboxes of images on page, no need to sort them\n",
    "    for item in page.get_images():\n",
    "        img_bboxes.extend(page.get_image_rects(item[0]))\n",
    "\n",
    "    # blocks of text on page\n",
    "    blocks = page.get_text(\n",
    "        \"dict\",\n",
    "        flags=fitz.TEXTFLAGS_TEXT,\n",
    "        clip=clip,\n",
    "    )[\"blocks\"]\n",
    "\n",
    "    # Make block rectangles, ignoring non-horizontal text\n",
    "    for b in blocks:\n",
    "        bbox = fitz.IRect(b[\"bbox\"])  # bbox of the block\n",
    "\n",
    "        # ignore text written upon images\n",
    "        if no_image_text and in_bbox(bbox, img_bboxes):\n",
    "            continue\n",
    "\n",
    "        # confirm first line to be horizontal\n",
    "        line0 = b[\"lines\"][0]  # get first line\n",
    "        if line0[\"dir\"] != (1, 0):  # only accept horizontal text\n",
    "            vert_bboxes.append(bbox)\n",
    "            continue\n",
    "\n",
    "        srect = fitz.EMPTY_IRECT()\n",
    "        for line in b[\"lines\"]:\n",
    "            lbbox = fitz.IRect(line[\"bbox\"])\n",
    "            text = \"\".join([s[\"text\"].strip() for s in line[\"spans\"]])\n",
    "            if len(text) > 1:\n",
    "                srect |= lbbox\n",
    "        bbox = +srect\n",
    "\n",
    "        if not bbox.is_empty:\n",
    "            bboxes.append(bbox)\n",
    "\n",
    "    # Sort text bboxes by ascending background, top, then left coordinates\n",
    "    bboxes.sort(key=lambda k: (in_bbox(k, path_bboxes), k.y0, k.x0))\n",
    "\n",
    "    # Extend bboxes to the right where possible\n",
    "    bboxes = extend_right(\n",
    "        bboxes, int(page.rect.width), path_bboxes, vert_bboxes, img_bboxes\n",
    "    )\n",
    "\n",
    "    # immediately return of no text found\n",
    "    if bboxes == []:\n",
    "        return []\n",
    "\n",
    "    # --------------------------------------------------------------------\n",
    "    # Join bboxes to establish some column structure\n",
    "    # --------------------------------------------------------------------\n",
    "    # the final block bboxes on page\n",
    "    nblocks = [bboxes[0]]  # pre-fill with first bbox\n",
    "    bboxes = bboxes[1:]  # remaining old bboxes\n",
    "\n",
    "    for i, bb in enumerate(bboxes):  # iterate old bboxes\n",
    "        check = False  # indicates unwanted joins\n",
    "\n",
    "        # check if bb can extend one of the new blocks\n",
    "        for j in range(len(nblocks)):\n",
    "            nbb = nblocks[j]  # a new block\n",
    "\n",
    "            # never join across columns\n",
    "            if bb == None or nbb.x1 < bb.x0 or bb.x1 < nbb.x0:\n",
    "                continue\n",
    "\n",
    "            # never join across different background colors\n",
    "            if in_bbox(nbb, path_bboxes) != in_bbox(bb, path_bboxes):\n",
    "                continue\n",
    "\n",
    "            temp = bb | nbb  # temporary extension of new block\n",
    "            check = can_extend(temp, nbb, nblocks)\n",
    "            if check == True:\n",
    "                break\n",
    "\n",
    "        if not check:  # bb cannot be used to extend any of the new bboxes\n",
    "            nblocks.append(bb)  # so add it to the list\n",
    "            j = len(nblocks) - 1  # index of it\n",
    "            temp = nblocks[j]  # new bbox added\n",
    "\n",
    "        # check if some remaining bbox is contained in temp\n",
    "        check = can_extend(temp, bb, bboxes)\n",
    "        if check == False:\n",
    "            nblocks.append(bb)\n",
    "        else:\n",
    "            nblocks[j] = temp\n",
    "        bboxes[i] = None\n",
    "\n",
    "    # do some elementary cleaning\n",
    "    nblocks = clean_nblocks(nblocks)\n",
    "\n",
    "    # return identified text bboxes\n",
    "    return nblocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6e6b8029-12dc-4960-90e7-18ab27ebe55e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/anaconda3/envs/saikatenv/lib/python3.11/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import fitz\n",
    "import re\n",
    "import transformers\n",
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4995771a-612e-48e3-b19c-3f177044a3d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
    "    \"/data/saikat/RegBank_Policy_POC/misc/mixtral-tokenizer/\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "ec76b47c-be32-4b1c-a8a0-d77108ff504b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_circular_store(filepath):\n",
    "    import fitz\n",
    "    import os\n",
    "    import re\n",
    "    \n",
    "    doc = fitz.open(filepath)\n",
    "    filename = os.path.split(filepath)\n",
    "\n",
    "    page_num = 1\n",
    "    text_list = []\n",
    "\n",
    "    for page in doc:\n",
    "        bboxes = column_boxes(page, footer_margin=50, no_image_text=True)\n",
    "\n",
    "        block_num = 1\n",
    "        for rect in bboxes:\n",
    "            block_text = page.get_text(clip=rect, sort=True)\n",
    "\n",
    "            paragraph_text = block_text\n",
    "            paragraph_text = paragraph_text.replace(\"\\n\", \" \")\n",
    "            paragraph_text = re.sub(r\"\\-\\n\", \"\", paragraph_text)\n",
    "            paragraph_text = re.sub(r\"\\'\", \"\", paragraph_text)\n",
    "            paragraph_text = paragraph_text.replace(\"\\\\n\", \" \")\n",
    "            paragraph_text = paragraph_text.replace(\"\\\\xa0\", \" \")\n",
    "            paragraph_text = re.sub(\" +\", \" \", paragraph_text)\n",
    "            paragraph_text = re.sub(r\"Page\\s\\d{1,}\", \"\", paragraph_text)\n",
    "\n",
    "            text_list.append((page_num, block_num, paragraph_text))\n",
    "            block_num += 1\n",
    "        page_num += 1\n",
    "\n",
    "    sorted_text_list = sorted(text_list, key=lambda x: (x[0], x[1]))\n",
    "\n",
    "    docs = {}\n",
    "    text = \"\"\n",
    "\n",
    "    current_chunk_token_length = 0\n",
    "    chunk_number = 0\n",
    "\n",
    "    for list_item in sorted_text_list:\n",
    "\n",
    "        page_num = list_item[0]\n",
    "        block_num = list_item[1]\n",
    "        paragraph_text = list_item[2]\n",
    "\n",
    "        token_length = len(tokenizer.tokenize(paragraph_text))\n",
    "\n",
    "        if token_length > 0 and chunk_number == 0 and current_chunk_token_length == 0:\n",
    "            docs[chunk_number] = paragraph_text\n",
    "\n",
    "        current_chunk_token_length = len(tokenizer.tokenize(docs[chunk_number]))\n",
    "\n",
    "    \n",
    "        docs[chunk_number] = docs[chunk_number] + paragraph_text\n",
    "\n",
    "    return docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "088ecaa6-725d-4927-9fa5-b64b82f282aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "store = create_circular_store(pdf_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "2ae9498c-5f56-4a3b-ba97-b80db6267a54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'Threshold Conditions Chapter 1 Introduction Threshold Conditions Chapter 1 Introduction 1.1A Application To which threshold conditions does COND apply? ..................................................................................................... (1) Section 55C of the Financial Services Act 2012 (Power to amend Schedule 6) gave HM Treasury the power to amend Schedule 6 of the Act. HM Treasury exercised this power by making The Financial Services and Markets Act 2000 (Threshold Conditions) Order 2013 which entered into force on 1 April 2013 (the \"TC Order\"). The TC Orders main result is the creation of four sets of threshold conditions, namely: (i) conditions for firms authorised and regulated by the FCA only (paragraphs 2B to 2F of Schedule 6 to the Act) (ii) FCA specific conditions for firms authorised by the PRA and subject to dual regulation (paragraphs 3B to 3E of Schedule 6 to the Act); (iii) PRA-specific conditions for insurers (paragraphs 4A to 4F of Schedule 6 to the Act); and (iv) PRA-specific conditions for other PRA-authorised persons (paragraphs 5A to 5F of Schedule 6 to the Act). (2) The guidance in COND is only applicable to the threshold conditions listed in I COND 1.1A.1G(1)(i) and (ii), above. These are the threshold conditions stated in paragraphs 2A and 3A of Schedule 6 to the Act as being relevant to the discharge by the FCA of its functions under the Act. (3) In respect of a person which does not carry on, or seek to carry on, any PRA-regulated activities, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 2B to 2F of Schedule 6 to the Act, subject to I COND 1.1A.4G (1), below. (4) In respect of a firm which does carry on, or seeks to carry on, a PRA- regulated activity, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 3B to 3E of Schedule 6 to the Act, subject to I COND 1.1A.4G (2), below. (5) A reference to \"FCA threshold conditions\" in COND means a reference to the threshold conditions referred to in (3) and (4) that apply to a particular firm. 1.1A.1 1.1A.2 1.1A.3 1.1A.4 To whom does COND apply? ..................................................................................................... 1 (1) COND applies to all firms, except where stated otherwise in this guidance. (2) In COND, firm includes an applicant for Part 4A permission unless the context otherwise requires. To what extent does COND apply to firms authorised by the PRA (PRA-authorised persons) and subject to dual regulation? ..................................................................................................... (1) As a result of the new legal framework for threshold conditions described in I COND 1.1A.1G (1), PRA-authorised persons and firms seeking to become PRA-authorised persons are subject to two sets of threshold conditions: (i) the FCA-specific conditions referred to in I COND 1.1A.1G (1)(ii)and (ii) one of the two PRA-specific conditions referred to in I COND 1.1A.1G (1)(iii) or I (iv), depending on the PRA-regulated activities which the PRA-authorised person or firm carries on, or is seeking to carry on. The FCA threshold conditions set out in paragraphs 3B to 3E of the Act seek to reflect this. In particular, these threshold conditions do not contain a condition relating to adequate financial resources. This is a matter that falls to be considered by the PRA under its threshold conditions. (2) The majority of the guidance in COND is intended to assist all firms to understand how the FCA will approach its assessment of the applicable FCA threshold conditions, regardless of whether or not a firm is, or is seeking to become, a PRA-authorised person. This is because the FCA threshold conditions which apply to PRA-authorised persons and those which apply to firms authorised by the FCA only are, for the most part, the same. (3) However, where guidance in COND refers to an assessment of a firms financial position or its compliance with prudential regulatory requirements, it is not intended to assist firms which are, or are seeking to become, PRA-authorised persons. This is because these are matters that are not covered by the FCAs threshold conditions, but rather fall to be considered by the PRA under its threshold conditions. (4) Although some of the PRA threshold conditions and FCA threshold conditions that apply to firms which are, or are seeking to become, PRA-authorised persons may appear to address similar subject matter, the FCA will approach the assessment of its threshold conditions with its unique statutory objectives in mind and in the light of the functions which the FCA is required to discharge in relation to them. (5) For the avoidance of doubt, the guidance in COND is not intended to apply to the PRAs assessment of its own threshold conditions in respect of a PRA-authorised person. This is a matter for the PRA alone. [deleted] To what extent does COND apply to Swiss general insurance companies? ..................................................................................................... FCA threshold conditions 3B, 3C and 3E apply to Swiss General Insurance Companies. To what extent does COND apply to credit firms with limited permission? ..................................................................................................... (1) The FCA threshold conditions apply to a person that carries on, or seeks to carry on, only relevant credit activities (within paragraph 2G of Schedule 6 to the Act) and which therefore has, or is applying for, limited permission with a number of modifications (see article 10(19) of the Regulated Activities Amendment Order). Regulated activities a person carries on in relation to which sections 20(1) and (1A) and 23(1A) of the Act do not apply as a result of section 39(1D) of the Act are disregarded for this purpose. (2) For a person within (1), the FCA threshold conditions are modified as follows: (a) in relation to paragraph 2C of Schedule 6 to the Act (Effective supervision), paragraphs (a), (b) and (e) of sub-paragraph (1) do not apply (see I COND 2.3); (b) in relation to paragraph 2D of Schedule 6 to the Act (Appropriate resources), the person has adequate financial resources if it is capable of meeting its debts as they fall due (see I COND 2.4); (c) paragraph 2F of Schedule 6 to the Act (Business model) does not apply (see I COND 2.7). (3) Paragraph 2G of Schedule 6 to the Act defines relevant credit activity for the purposes of the FCA Threshold Conditions. The interpretation of some of the key expressions used in this specific context is as follows: (a) \"borrower\" includes any person providing a guarantee or indemnity under an agreement, and a person to whom the rights and duties of the borrower have passed by assignment or operation of law; (b) \"supplier\" means a person whose main business is to sell goods or supply services and not to carry on a regulated activity, other than entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement; (c) \"customer\" means a person to whom a supplier sells goods or supplies services or agrees to do so; (d) \"domestic premises supplier\" means a supplier who sells goods, offers or agrees to sell goods, or offers or contracts to supply services, to customers who are individuals while the supplier or the supplier’s representative is physically present in the dwelling of the customer (though a supplier who does so only on an occasional basis is not to be treated as a \"domestic premises supplier” unless the supplier indicates to the public at large, or a section of it, the supplier’s willingness to attend, in person or through a representative, the dwelling of a potential customer in order to do any of those things). 1.1A.5 1.1A.5A 1 (3A) Questions may arise over whether a supplier who visits a customer’s dwelling to take measurements or give an estimate is a “domestic premises supplier”. For example: (a) if the supplier, or the supplier’s representative, gives a quote or estimate to the customer during the visit that is sufficiently specific as to be capable of being accepted in a way that is binding on the supplier, then the quote or estimate is an offer; on that basis, the supplier falls within the definition of “domestic premises supplier”, irrespective of whether the customer accepts the offer during the visit; (b) where the supplier, or the supplier’s representative, gives only a rough estimate or quote during the visit, with a view to submitting a refined estimate or a firm quote at a later time when the supplier is not at the customer’s dwelling, that rough estimate will not be an offer; on that basis, the supplier will not fall within the definition of “domestic premises supplier”, unless the customer and the supplier, or the supplier’s representative, do reach an agreement during the visit; and (c) where an agreement is reached, the supplier will have sold, or agreed to sell, goods or contracted to supply services, and will therefore be a “domestic premises supplier”; this may be the case even if the agreement is subject to later specification of the price, the goods or the services. It is immaterial whether the supplier carries on any credit broking (or other regulated activity) during the visit. (4) In summary, the following credit-related regulated activities are relevant credit activities for the purposes of the FCA Threshold Conditions: (a) credit broking when carried on: (i) by a supplier (other than a domestic premises supplier) for the purposes of or in connection with the sale of goods or supply of services by the supplier to a customer (who need not be the borrower under the credit agreement or the hirer under the consumer hire agreement); or (ii) in relation to a green deal plan; or (iii) in relation to a consumer hire agreement or a hire-purchase agreement; although, other than where the credit broking is carried on by a not-for-profit body, the credit broking will not be a relevant credit activity where it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land; (b) consumer credit lending if carried on by a local authority or if: (i) it is carried on by a supplier; (ii) no charge (by way of interest or otherwise) is payable by the borrower in connection with the provision of credit (this includes a charge payable in connection with a breach of the agreement or on the occurrence of a specified event; consumer credit lending under an agreement that contains such a charge is not a relevant credit activity); and (iii) the regulated credit agreement is not a hire purchase agreement or a conditional sale agreement; although, other than where the consumer credit lending is carried on by a not-for-profit body, the consumer credit lending will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay is secured, or is to be secured, by a legal mortgage on land; (c) entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement although, other than where these activities are carried on by a not-for- profit body, entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement will not be a relevant credit activity if the obligation of the hirer to pay under the agreement is secured, or is to be secured, by a legal mortgage on land; (d) debt adjusting or debt counselling when carried on: (i) by a supplier who also carries on credit broking within (a)(i); (ii) by a person in connection with an activity within (b) or (c) which the person also carries on; (iii) by a not-for-profit body; although, other than where the debt adjusting or debt counselling is carried on by a not-for-profit body, the debt adjusting or debt counselling will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land; (e) providing credit information services where carried on by a person in connection with an activity within (a) to (d) which the person also carries on; (f) agreeing to carry on an activity within (a) to (e). To what extent does COND apply to regulated benchmark administrators? ..................................................................................................... (1) The threshold conditions do not apply to a firm in relation to the regulated activity of administering a benchmark. (2) COND does not apply to regulated benchmark administrators who are solely authorised to administer a benchmark as they are not subject to the threshold conditions. (3) For regulated benchmark administrators who are also authorised to carry on activities other than administering a benchmark, they will be subject to the threshold conditions in relation to their other regulated activities. COND will apply to those firms in relation to the regulated activities to which the threshold conditions apply. 1.1A.5B 1 To which regulated activities does COND apply? ..................................................................................................... Subject to the limitations referred to above, COND applies in relation to all of the regulated activities for which a firm has, or will have, permission. Where does COND apply? ..................................................................................................... COND applies in relation to all of the regulated activities wherever they are carried on. 1.1A.6 1.1A.7 '}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "dad6f752-6705-418e-870e-96372f613b0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import SpacyTextSplitter\n",
    "text_splitter = SpacyTextSplitter(chunk_size = 50, chunk_overlap=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "ee67b29c-ebb1-4319-bc0c-4aea51d9f4d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tst = \"To whom does COND apply? ..................................................................................................... 1 (1) COND applies to all firms, except where stated otherwise in this guidance. (2) In COND, firm includes an applicant for Part 4A permission unless the context otherwise requires. To what extent does COND apply to firms authorised by the PRA (PRA-authorised persons) and subject to dual regulation? ..................................................................................................... (1) As a result of the new legal framework for threshold conditions described in I COND 1.1A.1G (1), PRA-authorised persons and firms seeking to become PRA-authorised persons are subject to two sets of threshold conditions: (i) the FCA-specific conditions referred to in I COND 1.1A.1G (1)(ii)and (ii) one of the two PRA-specific conditions referred to in I COND 1.1A.1G (1)(iii) or I (iv), depending on the PRA-regulated activities which the PRA-authorised person or firm carries on, or is seeking to carry on. The FCA threshold conditions set out in paragraphs 3B to 3E of the Act seek to reflect this. In particular, these threshold conditions do not contain a condition relating to adequate financial resources. This is a matter that falls to be considered by the PRA under its threshold conditions. (2) The majority of the guidance in COND is intended to assist all firms to understand how the FCA will approach its assessment of the applicable FCA threshold conditions, regardless of whether or not a firm is, or is seeking to become, a PRA-authorised person. This is because the FCA threshold conditions which apply to PRA-authorised persons and those which apply to firms authorised by the FCA only are, for the most part, the same. (3) However, where guidance in COND refers to an assessment of a firms financial position or its compliance with prudential regulatory requirements, it is not intended to assist firms which are, or are seeking to become, PRA-authorised persons. This is because these are matters that are not covered by the FCAs threshold conditions, but rather fall to be considered by the PRA under its threshold conditions. (4) Although some of the PRA threshold conditions and FCA threshold conditions that apply to firms which are, or are seeking to become, PRA-authorised persons may appear to address similar subject matter, the FCA will approach the assessment of its threshold conditions with its unique statutory objectives in mind and in the light of the functions which the FCA is required to discharge in relation to them. (5) For the avoidance of doubt, the guidance in COND is not intended to apply to the PRAs assessment of its own threshold conditions in respect of a PRA-authorised person. This is a matter for the PRA alone. [deleted] \"\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "de392719-c0ff-45e9-a760-1a5830d96763",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 151, which is longer than the specified 50\n",
      "Created a chunk of size 101, which is longer than the specified 50\n",
      "Created a chunk of size 137, which is longer than the specified 50\n",
      "Created a chunk of size 179, which is longer than the specified 50\n",
      "Created a chunk of size 545, which is longer than the specified 50\n",
      "Created a chunk of size 119, which is longer than the specified 50\n",
      "Created a chunk of size 168, which is longer than the specified 50\n",
      "Created a chunk of size 309, which is longer than the specified 50\n",
      "Created a chunk of size 301, which is longer than the specified 50\n",
      "Created a chunk of size 159, which is longer than the specified 50\n",
      "Created a chunk of size 52, which is longer than the specified 50\n",
      "Created a chunk of size 103, which is longer than the specified 50\n",
      "Created a chunk of size 78, which is longer than the specified 50\n",
      "Created a chunk of size 101, which is longer than the specified 50\n",
      "Created a chunk of size 118, which is longer than the specified 50\n",
      "Created a chunk of size 101, which is longer than the specified 50\n",
      "Created a chunk of size 511, which is longer than the specified 50\n",
      "Created a chunk of size 92, which is longer than the specified 50\n",
      "Created a chunk of size 110, which is longer than the specified 50\n",
      "Created a chunk of size 87, which is longer than the specified 50\n",
      "Created a chunk of size 261, which is longer than the specified 50\n",
      "Created a chunk of size 174, which is longer than the specified 50\n",
      "Created a chunk of size 244, which is longer than the specified 50\n",
      "Created a chunk of size 164, which is longer than the specified 50\n",
      "Created a chunk of size 408, which is longer than the specified 50\n",
      "Created a chunk of size 171, which is longer than the specified 50\n",
      "Created a chunk of size 180, which is longer than the specified 50\n",
      "Created a chunk of size 82, which is longer than the specified 50\n",
      "Created a chunk of size 71, which is longer than the specified 50\n",
      "Created a chunk of size 101, which is longer than the specified 50\n",
      "Created a chunk of size 329, which is longer than the specified 50\n",
      "Created a chunk of size 195, which is longer than the specified 50\n",
      "Created a chunk of size 535, which is longer than the specified 50\n",
      "Created a chunk of size 124, which is longer than the specified 50\n",
      "Created a chunk of size 1330, which is longer than the specified 50\n",
      "Created a chunk of size 152, which is longer than the specified 50\n",
      "Created a chunk of size 1228, which is longer than the specified 50\n",
      "Created a chunk of size 115, which is longer than the specified 50\n",
      "Created a chunk of size 3177, which is longer than the specified 50\n",
      "Created a chunk of size 69, which is longer than the specified 50\n",
      "Created a chunk of size 101, which is longer than the specified 50\n",
      "Created a chunk of size 119, which is longer than the specified 50\n",
      "Created a chunk of size 166, which is longer than the specified 50\n",
      "Created a chunk of size 225, which is longer than the specified 50\n",
      "Created a chunk of size 111, which is longer than the specified 50\n",
      "Created a chunk of size 158, which is longer than the specified 50\n",
      "Created a chunk of size 153, which is longer than the specified 50\n",
      "Created a chunk of size 124, which is longer than the specified 50\n",
      "Created a chunk of size 89, which is longer than the specified 50\n"
     ]
    }
   ],
   "source": [
    "chunks = text_splitter.split_text(store[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "5b9c8aff-337f-4232-9c9e-2888bea13c6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Threshold Conditions Chapter 1 Introduction Threshold Conditions Chapter 1 Introduction 1.1A Application To which threshold conditions does COND apply?',\n",
       " '.....................................................................................................',\n",
       " '(1) Section 55C of the Financial Services Act 2012 (Power to amend Schedule 6) gave HM Treasury the power to amend Schedule 6 of the Act.',\n",
       " 'HM Treasury exercised this power by making The Financial Services and Markets Act 2000 (Threshold Conditions) Order 2013 which entered into force on 1 April 2013 (the \"TC Order\").',\n",
       " 'The TC Orders main result is the creation of four sets of threshold conditions, namely: (i) conditions for firms authorised and regulated by the FCA only (paragraphs 2B to 2F of Schedule 6 to the Act) (ii) FCA specific conditions for firms authorised by the PRA and subject to dual regulation (paragraphs 3B to 3E of Schedule 6 to the Act); (iii) PRA-specific conditions for insurers (paragraphs 4A to 4F of Schedule 6 to the Act); and (iv) PRA-specific conditions for other PRA-authorised persons (paragraphs 5A to 5F of Schedule 6 to the Act).',\n",
       " '(2) The guidance in COND is only applicable to the threshold conditions listed in I COND 1.1A.1G(1)(i) and (ii), above.',\n",
       " 'These are the threshold conditions stated in paragraphs 2A and 3A of Schedule 6 to the Act as being relevant to the discharge by the FCA of its functions under the Act.',\n",
       " '(3) In respect of a person which does not carry on, or seek to carry on, any PRA-regulated activities, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 2B to 2F of Schedule 6 to the Act, subject to I COND 1.1A.4G (1), below.',\n",
       " '(4) In respect of a firm which does carry on, or seeks to carry on, a PRA- regulated activity, the threshold conditions that are relevant to the discharge by the FCA of its functions under the Act are those set out in paragraphs 3B to 3E of Schedule 6 to the Act, subject to I COND 1.1A.4G (2), below.',\n",
       " '(5) A reference to \"FCA threshold conditions\" in COND means a reference to the threshold conditions referred to in (3) and (4) that apply to a particular firm.',\n",
       " '1.1A.1 1.1A.2 1.1A.3 1.1A.4 To whom does COND apply?',\n",
       " '..................................................................................................... 1',\n",
       " '(1) COND applies to all firms, except where stated otherwise in this guidance.',\n",
       " '(2) In COND, firm includes an applicant for Part 4A permission unless the context otherwise requires.',\n",
       " 'To what extent does COND apply to firms authorised by the PRA (PRA-authorised persons) and subject to dual regulation?',\n",
       " '.....................................................................................................',\n",
       " '(1) As a result of the new legal framework for threshold conditions described in I COND 1.1A.1G (1), PRA-authorised persons and firms seeking to become PRA-authorised persons are subject to two sets of threshold conditions: (i) the FCA-specific conditions referred to in I COND 1.1A.1G (1)(ii)and (ii) one of the two PRA-specific conditions referred to in I COND 1.1A.1G (1)(iii) or I (iv), depending on the PRA-regulated activities which the PRA-authorised person or firm carries on, or is seeking to carry on.',\n",
       " 'The FCA threshold conditions set out in paragraphs 3B to 3E of the Act seek to reflect this.',\n",
       " 'In particular, these threshold conditions do not contain a condition relating to adequate financial resources.',\n",
       " 'This is a matter that falls to be considered by the PRA under its threshold conditions.',\n",
       " '(2) The majority of the guidance in COND is intended to assist all firms to understand how the FCA will approach its assessment of the applicable FCA threshold conditions, regardless of whether or not a firm is, or is seeking to become, a PRA-authorised person.',\n",
       " 'This is because the FCA threshold conditions which apply to PRA-authorised persons and those which apply to firms authorised by the FCA only are, for the most part, the same.',\n",
       " '(3) However, where guidance in COND refers to an assessment of a firms financial position or its compliance with prudential regulatory requirements, it is not intended to assist firms which are, or are seeking to become, PRA-authorised persons.',\n",
       " 'This is because these are matters that are not covered by the FCAs threshold conditions, but rather fall to be considered by the PRA under its threshold conditions.',\n",
       " '(4) Although some of the PRA threshold conditions and FCA threshold conditions that apply to firms which are, or are seeking to become, PRA-authorised persons may appear to address similar subject matter, the FCA will approach the assessment of its threshold conditions with its unique statutory objectives in mind and in the light of the functions which the FCA is required to discharge in relation to them.',\n",
       " '(5) For the avoidance of doubt, the guidance in COND is not intended to apply to the PRAs assessment of its own threshold conditions in respect of a PRA-authorised person.',\n",
       " 'This is a matter for the PRA alone.',\n",
       " '[deleted] To what extent does COND apply to Swiss general insurance companies? .....................................................................................................',\n",
       " 'FCA threshold conditions 3B, 3C and 3E apply to Swiss General Insurance Companies.',\n",
       " 'To what extent does COND apply to credit firms with limited permission?',\n",
       " '.....................................................................................................',\n",
       " '(1) The FCA threshold conditions apply to a person that carries on, or seeks to carry on, only relevant credit activities (within paragraph 2G of Schedule 6 to the Act) and which therefore has, or is applying for, limited permission with a number of modifications (see article 10(19) of the Regulated Activities Amendment Order).',\n",
       " 'Regulated activities a person carries on in relation to which sections 20(1) and (1A) and 23(1A) of the Act do not apply as a result of section 39(1D) of the Act are disregarded for this purpose.',\n",
       " '(2) For a person within (1), the FCA threshold conditions are modified as follows: (a) in relation to paragraph 2C of Schedule 6 to the Act (Effective supervision), paragraphs (a), (b) and (e) of sub-paragraph (1) do not apply (see I COND 2.3); (b) in relation to paragraph 2D of Schedule 6 to the Act (Appropriate resources), the person has adequate financial resources if it is capable of meeting its debts as they fall due (see I COND 2.4); (c) paragraph 2F of Schedule 6 to the Act (Business model) does not apply (see I COND 2.7).',\n",
       " '(3) Paragraph 2G of Schedule 6 to the Act defines relevant credit activity for the purposes of the FCA Threshold Conditions.',\n",
       " 'The interpretation of some of the key expressions used in this specific context is as follows: (a) \"borrower\" includes any person providing a guarantee or indemnity under an agreement, and a person to whom the rights and duties of the borrower have passed by assignment or operation of law; (b) \"supplier\" means a person whose main business is to sell goods or supply services and not to carry on a regulated activity, other than entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement; (c) \"customer\" means a person to whom a supplier sells goods or supplies services or agrees to do so; (d) \"domestic premises supplier\" means a supplier who sells goods, offers or agrees to sell goods, or offers or contracts to supply services, to customers who are individuals while the supplier or the supplier’s representative is physically present in the dwelling of the customer (though a supplier who does so only on an occasional basis is not to be treated as a \"domestic premises supplier” unless the supplier indicates to the public at large, or a section of it, the supplier’s willingness to attend, in person or through a representative, the dwelling of a potential customer in order to do any of those things).',\n",
       " '1.1A.5 1.1A.5A 1 (3A)',\n",
       " 'Questions may arise over whether a supplier who visits a customer’s dwelling to take measurements or give an estimate is a “domestic premises supplier”.',\n",
       " 'For example: (a) if the supplier, or the supplier’s representative, gives a quote or estimate to the customer during the visit that is sufficiently specific as to be capable of being accepted in a way that is binding on the supplier, then the quote or estimate is an offer; on that basis, the supplier falls within the definition of “domestic premises supplier”, irrespective of whether the customer accepts the offer during the visit; (b) where the supplier, or the supplier’s representative, gives only a rough estimate or quote during the visit, with a view to submitting a refined estimate or a firm quote at a later time when the supplier is not at the customer’s dwelling, that rough estimate will not be an offer; on that basis, the supplier will not fall within the definition of “domestic premises supplier”, unless the customer and the supplier, or the supplier’s representative, do reach an agreement during the visit; and (c) where an agreement is reached, the supplier will have sold, or agreed to sell, goods or contracted to supply services, and will therefore be a “domestic premises supplier”; this may be the case even if the agreement is subject to later specification of the price, the goods or the services.',\n",
       " 'It is immaterial whether the supplier carries on any credit broking (or other regulated activity) during the visit.',\n",
       " '(4) In summary, the following credit-related regulated activities are relevant credit activities for the purposes of the FCA Threshold Conditions: (a) credit broking when carried on: (i) by a supplier (other than a domestic premises supplier) for the purposes of or in connection with the sale of goods or supply of services by the supplier to a customer (who need not be the borrower under the credit agreement or the hirer under the consumer hire agreement); or (ii) in relation to a green deal plan; or (iii) in relation to a consumer hire agreement or a hire-purchase agreement; although, other than where the credit broking is carried on by a not-for-profit body, the credit broking will not be a relevant credit activity where it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land; (b) consumer credit lending if carried on by a local authority or if: (i) it is carried on by a supplier; (ii) no charge (by way of interest or otherwise) is payable by the borrower in connection with the provision of credit (this includes a charge payable in connection with a breach of the agreement or on the occurrence of a specified event; consumer credit lending under an agreement that contains such a charge is not a relevant credit activity); and (iii) the regulated credit agreement is not a hire purchase agreement or a conditional sale agreement; although, other than where the consumer credit lending is carried on by a not-for-profit body, the consumer credit lending will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay is secured, or is to be secured, by a legal mortgage on land; (c) entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement although, other than where these activities are carried on by a not-for- profit body, entering into a regulated consumer hire agreement as owner or exercising, or having the right to exercise, the owners rights and duties under a regulated consumer hire agreement will not be a relevant credit activity if the obligation of the hirer to pay under the agreement is secured, or is to be secured, by a legal mortgage on land; (d) debt adjusting or debt counselling when carried on: (i) by a supplier who also carries on credit broking within (a)(i); (ii) by a person in connection with an activity within (b) or (c) which the person also carries on; (iii) by a not-for-profit body; although, other than where the debt adjusting or debt counselling is carried on by a not-for-profit body, the debt adjusting or debt counselling will not be a relevant credit activity if it relates to an agreement under which the obligation of the borrower to repay or the hirer to pay is secured, or is to be secured, by a legal mortgage on land; (e) providing credit information services where carried on by a person in connection with an activity within (a) to (d) which the person also carries on; (f) agreeing to carry on an activity within (a) to (e).',\n",
       " 'To what extent does COND apply to regulated benchmark administrators?',\n",
       " '.....................................................................................................',\n",
       " '(1) The threshold conditions do not apply to a firm in relation to the regulated activity of administering a benchmark.',\n",
       " '(2) COND does not apply to regulated benchmark administrators who are solely authorised to administer a benchmark as they are not subject to the threshold conditions.',\n",
       " '(3) For regulated benchmark administrators who are also authorised to carry on activities other than administering a benchmark, they will be subject to the threshold conditions in relation to their other regulated activities.',\n",
       " 'COND will apply to those firms in relation to the regulated activities to which the threshold conditions apply.',\n",
       " '1.1A.5B 1 To which regulated activities does COND apply? .....................................................................................................',\n",
       " 'Subject to the limitations referred to above, COND applies in relation to all of the regulated activities for which a firm has, or will have, permission.',\n",
       " 'Where does COND apply? .....................................................................................................',\n",
       " 'COND applies in relation to all of the regulated activities wherever they are carried on.',\n",
       " '1.1A.6 1.1A.7']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35760f29-ab7a-4a9d-a397-a7f70fa3d0d8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}