Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add datasets importing functionality #201

Closed
wants to merge 60 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
de78f6b
feat(server/helpers): Add import search
shivam-tripathi Jul 12, 2018
acc103f
feat(helpers/utils): Add function to get import models
shivam-tripathi Jul 18, 2018
e60449e
feat(helpers/utils): Add function to get import model by name
shivam-tripathi Jul 18, 2018
56670ee
feat(helpers/utils): Add JSDocs for import functions
shivam-tripathi Jul 18, 2018
b272cc3
feat(server/helpers/search): Add import support in _fetchEntityModels…
shivam-tripathi Jul 18, 2018
c0b695e
feat(server/helpers/): Update licenses
shivam-tripathi Jul 18, 2018
870e0f0
feat(client/components/pages/search): Update search with message for …
shivam-tripathi Jul 18, 2018
cf4680e
feat(client/components/pages/parts/search-results): Update search res…
shivam-tripathi Jul 18, 2018
203dc05
feat(client/containers/layout): Add panel link to recent imports
shivam-tripathi Jul 12, 2018
90be205
feat(scripts): Update build-client-js and watch-client-js to track re…
shivam-tripathi Jul 31, 2018
0de46fa
feat(server/routes/imports/recent): Add routes to handle recent imports
shivam-tripathi Jul 31, 2018
3ad38bf
feat(client/controllers/recentImports): Add recentImports controller
shivam-tripathi Jul 31, 2018
8de2735
feat(client/components/pages/parts/recentImports): Add part to hold r…
shivam-tripathi Jul 31, 2018
53b0b6f
feat(client/components/pages/recentImports): Add base component to di…
shivam-tripathi Jul 31, 2018
61657de
feat(server/routes): Add imports route
shivam-tripathi Jul 31, 2018
0344296
feat(client/helpers/utils): Add function to extract imports url
shivam-tripathi Jul 31, 2018
4513011
feat(client/helpers/pagination-props): Add function to determine pagi…
shivam-tripathi Jul 31, 2018
08b4d0a
feat(scripts): Add build and watch command in client scripts for impo…
shivam-tripathi Aug 3, 2018
1426f1f
feat(client/helpers/props): Add function to extract import entity props
shivam-tripathi Aug 3, 2018
2854bff
feat(client/controllers/import-entity): Add import-entity controller
shivam-tripathi Aug 3, 2018
913c54b
feat(client/helpers/import-entity): Add import-entity helper functions
shivam-tripathi Aug 3, 2018
208fc7e
feat(client/components/pages/import-entities/footer): Add footer for …
shivam-tripathi Aug 3, 2018
49639b8
feat(src/client/components/pages/import-entities/title): Add title fr…
shivam-tripathi Aug 3, 2018
2f78a55
feat(src/client/components/pages/import-entities/footer): Remove unne…
shivam-tripathi Aug 3, 2018
21915cd
feat(client/components/pages/import-entities): Add import-entities pages
shivam-tripathi Aug 3, 2018
37ae6a2
feat(server/routes/import-entity): Refactor, add more routes
shivam-tripathi Aug 3, 2018
ae662d0
feat(recent-imports): Refactor code
shivam-tripathi Aug 10, 2018
06e8219
feat(scripts/client): Update client build and watch scripts
shivam-tripathi Aug 10, 2018
45fc92d
feat(package.json): Add uuid module
shivam-tripathi Aug 10, 2018
f5fba4b
feat(client/components/pages/entities): Make compulsory props of imag…
shivam-tripathi Aug 10, 2018
fdc8f17
refactor(client/components/pages/parts/recent-import-results)
shivam-tripathi Aug 10, 2018
48bd1b4
feat(client/discard-import-entity): Add confirm discard vote cast page
shivam-tripathi Aug 10, 2018
8d738f7
feat(client/controllers/recent-imports): Move recent-imports controll…
shivam-tripathi Aug 10, 2018
e563583
feat(client/controllers/import-entity): Add controller for import-ent…
shivam-tripathi Aug 10, 2018
2c25f13
refactor(client/helpers/utils): Remove unused function
shivam-tripathi Aug 10, 2018
b68bbab
feat(client/helpers/import-entity): Add import-entity helper function…
shivam-tripathi Aug 10, 2018
8ba3a82
feat(client/import-entities/footer): Add feature to prevent multiple …
shivam-tripathi Aug 10, 2018
560d24b
feat(client/components/pages/import-entities): Add import-entity pages
shivam-tripathi Aug 10, 2018
2598bdf
feat(client/entity-editor/submission-section): Add logic for redirect…
shivam-tripathi Aug 10, 2018
7c07651
feat(server/helpers/middleware): Add middleware to load imports into …
shivam-tripathi Aug 10, 2018
0374e40
feat(server/helpers/importEntityRouteUtils): Add logic to load edit p…
shivam-tripathi Aug 10, 2018
5fd6e9a
feat(server/helpers/utils): Add function to validate importId, could …
shivam-tripathi Aug 10, 2018
cab5b6a
feat(server/routes/import-entity/transform-form): Add module to trans…
shivam-tripathi Aug 10, 2018
b263307
feat(server/routes/import-entity/transform-import): Add module to tra…
shivam-tripathi Aug 10, 2018
a5b18db
feat(server/routes/import-entity/import-routes): Add import route hel…
shivam-tripathi Aug 10, 2018
0987520
feat(server/routes/import-entity): Add entity functions, update index
shivam-tripathi Aug 10, 2018
c8cb4d0
refactor(server/routes): Update routes to reflect import-entities
shivam-tripathi Aug 10, 2018
e450dec
feat(server/routes/import-entity): Add entity routes
shivam-tripathi Aug 18, 2018
762ccd9
feat(client/recent-imports): Add maximum button limit
shivam-tripathi Aug 18, 2018
26d82a7
feat(components/parts/search): Remove inline styles, add break before…
shivam-tripathi Aug 19, 2018
18fc335
feat(client/components/pages/entities): Export all attributes functions
shivam-tripathi Aug 19, 2018
29e3ada
feat(client/commit/pages/import-entities): Reuse attribute functions …
shivam-tripathi Aug 19, 2018
154f76c
feat(client/components/import-entities): Remove inline css
shivam-tripathi Aug 19, 2018
84989c7
feat(discard-import-entity): Remove token mechanism
shivam-tripathi Aug 19, 2018
18199d5
feat(client/import-entity/footer): Change name
shivam-tripathi Aug 19, 2018
35ea8b0
refactor: General refactor, remove redundant code, update credits
shivam-tripathi Aug 19, 2018
2afd166
feat(stylesheets/style.less): Add some style classes
shivam-tripathi Aug 19, 2018
b15a144
feat(client/helpers/import-entity): Add space after label, refactor
shivam-tripathi Aug 19, 2018
b61c993
feat(client/components/search-results): Use labels to denote imports
shivam-tripathi Aug 19, 2018
f8bb16c
feat(client/controllers/import-entity): Improve error handling when l…
shivam-tripathi Aug 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions src/server/routes/imports/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* Copyright (C) 2018 Shivam Tripathi
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*/

import ImportRecentRouter from './recent';
import express from 'express';


const importRouter = express.Router();

importRouter.use('/recent', ImportRecentRouter);

export default importRouter;
98 changes: 98 additions & 0 deletions src/server/routes/imports/recent.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
/*
* Copyright (C) 2018 Shivam Tripathi
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*/

import * as propHelpers from '../../../client/helpers/props';
import {escapeProps, generateProps} from '../../helpers/props';
import Layout from '../../../client/containers/layout';
import React from 'react';
import ReactDOMServer from 'react-dom/server';
import RecentImports from '../../../client/components/pages/recent-imports';
import express from 'express';


// The limit for number of results fetched from database
const LIMIT = 10;

const router = express.Router();

/* Function to fetch data from the database and create the object to be sent as
response */
function fetchRecentImportsData(orm, page) {
let pageNumber = page;

return orm.bookshelf.transaction(async (transacting) => {
// First fetch total imports
const totalResults =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be quite a slow query - https://wiki.postgresql.org/wiki/Slow_Counting . Avoid fetching the total number of results if possible - I don't think it'll be a problem if you use an offset that's more than the number of rows in the table - you'll just get 0 results.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's necessary to know total number of valid results to diplayed to determine begin and end page correctly. If we don't know the range, we cannot use pagination.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need the range to use pagination? You should be able to define pages like so:

Page 1: 0 - X
Page 2: X+1 - 2X
Page 3: 2X+1 - 3X

And so on. If your user gives page 10,000 and there are only 3,000 pages, then there should just be no results on the page.

If you need to know whether the next two pages exist, then the server can query 3X results from the current offset, and then if more then 2X results are returned, you have at least 2 more valid pages after the current one.

For now, I'm happy to accept this, but we'll need to do further testing to see whether it's too slow to use in practise.

await orm.func.imports.getTotalImports(transacting);

if (totalResults < ((pageNumber - 1) * LIMIT)) {
pageNumber = Math.ceil(totalResults / LIMIT);
}
const offset = (pageNumber - 1) * LIMIT;

// Now fetch recent imports according to the generated offset
const recentImports = await orm.func.imports.getRecentImports(
orm, transacting, LIMIT, offset
);

return {
currentPage: pageNumber,
limit: LIMIT,
offset,
recentImports,
totalResults
};
});
}

// This handles the router to send initial container to hold recentImports data
function recentImportsRoute(req, res) {
const queryPage = parseInt(req.query.page, 10) || 1;
const props = generateProps(req, res, {
currentPage: queryPage, limit: LIMIT
});

const markup = ReactDOMServer.renderToString(
<Layout {...propHelpers.extractLayoutProps(props)}>
<RecentImports
{...props}
/>
</Layout>
);

res.render('target', {
markup,
props: escapeProps(props),
script: '/js/recent-imports.js',
title: 'Recent Imports'
});
}

// This handles the data fetching route for recent imports, sends JSON as res
async function rawRecentImportsRoute(req, res) {
const {orm} = req.app.locals;
const queryPage = parseInt(req.query.page, 10) || 1;
const recentImportsData = await fetchRecentImportsData(orm, queryPage);

res.send(recentImportsData);
}

router.get('/', recentImportsRoute);
router.get('/raw', rawRecentImportsRoute);

export default router;