Skip to content

Commit fabee6b

Browse files
josenavasantgonza
authored andcommitted
Release candidate November 1st, 2017 (#2373)
* replace printable for try/except utf-8 * add sleeps for tests * moving wait_for_prep_information_job * just leaving the changes on this pr * Fixing tests * Reverting changes * dist: precise * addressing @josenavas comments * fix-2212 * Adding branch description to Contributing (#2256) * Adding branch description * Adding spaces * Adding timeline * upgrading software * mv r_client to qiita * from redis import Redis * rm moi and ipython * Move qiita_db/private.py -> qiita_ware/private_plugin.py * _system_call as system_call * flake8 * rm create_raw_data * Transferring VAMPS submission to internal job * rm other unnecessary files * addressing @josenavas comment * Fixing merge conflicts * Adding tests to private plugin * rm wrapper.py * Removing import * Fixing import * Modifying GUI to use the plugin * moving qiita pet in .travis.yml * Transfer submit to vamps (#2265) * Move qiita_db/private.py -> qiita_ware/private_plugin.py * Transferring VAMPS submission to internal job * Fixing import * some other .travis.yml fixes * addressing @josenavas comment * Adding success test * adding some methods * flake8 * Transfer copy raw data (#2267) * Move qiita_db/private.py -> qiita_ware/private_plugin.py * Transferring VAMPS submission to internal job * Fixing merge conflicts * Adding tests to private plugin * Removing import * Fixing import * Modifying GUI to use the plugin * Adding success test * change imports * adding delete_artifact and create_sample_template * fixing errors and gui * fix delete error * ENH: Make jobs list modal With some recent changes to the position of the navigation bar, the header of the jobs list is cut from screen. With this patch in place, the jobs list will be shown as a modal window, so that won't be a problem anymore. * Transfer update delete templates (#2274) * Moving update_sample_template * Transfer update_sample_template * Porting update prep template * Moving delete sample or column * Removing tests * Removing dispatchable and its tests * Updating interface to use the new functionality' * Adapting the prep template GUI * Submitting jobs * Fixing tests * Removing qiita_ware/context.py * flake8ing * Fixing _system_call * Safeguarding the call to rollback * Unmasking more errors * Forcing different connections on different processes * Moving job completion to internal plugin structure * Removing unused code * Forcing the creation of a new transaction on the jobs * Fixing tests * forcing the commit * Fixing all tests * Addressing @antgonza's comments * Addressing @antgonza's comment * Addressing @ElDeveloper's comments * BUG: Fix job updates * Fixing the redis DB (#2277) * Fixing the redis DB * Addressing @antgonza's comments * redbiom install * redbiom to install_requires * mv moi-ws to qiita_websocket * init commit * addressing @wasade comment * rm webdis.log * cleaning code for initial review * install latest redbiom * fix test * ENH: Change phrasing of upload text (#2281) * ENH: Change phrasing of upload text As per a user's request. * Fix typo in docs Fixes #2259 * addressing @wasade comments and adding other tests * flake8 * addressing @josenavas comments * fix #2258 * fix #2258 * fix #858 (#2286) * addressing @ElDeveloper and @josenavas comments * @ElDeveloper :| * redbiom now adds per sample studies to analysis * jobs-list-as-modal * addresssing @josenavas comments * rm () from update_processing_job_data * add prints to review errors * rm prints * Fix 2190 (#2292) * Adding patch to add the 'name' parameter to all validate commands * Adding the parameter 'name' automatically and adding a test for it * Removing extra blank line * Adding dflt value to artifact name * Edited the wrong file * Adding user defined name at creation time * Fixing test * add is_from_analysis to artifact_handlers (#2293) * add is_from_analysis to artifact_handlers * fix test_post_metadata * fix error * WIP: rm sudo from travis * adding sed for config file * fix sed * rm & * cat redis.conf * using local redis.conf * # protected-mode yes * # supervised no * redis-server --port 7777 & * Fix 1293 (#2291) * fix #1293 * flake8 * fix errors * addressing @ElDeveloper comments * rm redis.conf * fix awaiting_approval list bug * Fix #2276 (#2294) * Fix #2276 * Factoring out generate nginx directory file list * Factoring out the nginx file list writing * Factoring out generating the file list of an artifact * Factoring out the header setting * Addressing @antgonza's comment * Addressing @wasade's comments * Fixing patch * fix error * Fixing failing test * fix #2214 * addressing @josenavas comment * fix #2209 * fix #2331 * fix #2326 (#2328) * fix #2326 * addressing @ElDeveloper comment * fix #2226 (#2330) * fix #2226 * addressing @ElDeveloper comment * fix #2336 * fix #2316 * Fixes 2269 * addressing @josenavas comments * Fixes 2038 (#2349) * fixes #2038 * Adding test * Trying to debug * Checking value * More debugging * Undo changes * Fixing failure * fix #2333 * addressing @josenavas * fixes #2245 (#2350) * fixes #2245 * Addressing @antgonza's comments * Fixing test * Fixing Qiita installation (#2362) * Sync-ing with master (#2367) * fix calls to system_call and ebi submissions * fixing errors * fix if state == submitting: * just raise error * EBISubmissionError -> ComputeError * fix #2084 (#2365) * fix #2125 (#2366) * fix #2125 * fix error * fix #2364 * fix #1812 * add tests * flake8 * populating ProcessingJob.create True * fix more errors * addressing comments from @josenavas and @stephanieorch * mv ProcessingJob.create True around * Partial #2237 (#2368) * fix calls to system_call and ebi submissions * fixing errors * fix if state == submitting: * just raise error * EBISubmissionError -> ComputeError * Sorting values * Case insensitive sorting * Addressing @ElDeveloper's comments * fix-1591 (#2370) * fix-1591 * removing warning ATTN @josenavas, fix tests * addressing @ElDeveloper and @josenavas comments * Fix 2230 (#2372) * fix calls to system_call and ebi submissions * fixing errors * fix if state == submitting: * just raise error * EBISubmissionError -> ComputeError * Fix #2230 solved using modal in order to prevent large file download * Function moved into in order to keep order. Some details fixed. * addressing @wasade and @ElDeveloper comments * fixing errors * fix GUI and erros * addressing @ElDeveloper comments * rm artifacts from parameters listing * fix flake8 * fix ilike quote params * addressing @josenavas comment and adding test for job without children * fix errors * addressing @josenavas comment * Fixing network labels (#2376) * Fixing network labels * Fixing error * Update redbiom.html * Update redbiom.html * Update redbiom.html * Patch 61 - transfer all parameters to str (#2379) * Patch 61 - transfer all parameters to str * Fixing errors * rm lower from redbiom * fixing smal details and adding emp_release1 * fixing
1 parent 9e10d9c commit fabee6b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+800
-346
lines changed

.travis.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ install:
2828
# install a few of the dependencies that pip would otherwise try to install
2929
# when intalling scikit-bio
3030
- travis_retry conda create --yes -n qiita python=$PYTHON_VERSION pip nose flake8
31-
pyzmq networkx pyparsing natsort mock future libgfortran seaborn nltk
31+
pyzmq 'networkx<2.0' pyparsing natsort mock future libgfortran seaborn nltk
3232
'pandas>=0.18' 'matplotlib>=1.1.0' 'scipy>0.13.0' 'numpy>=1.7' 'h5py>=2.3.1'
3333
- source activate qiita
3434
- pip install -U pip

qiita_db/analysis.py

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ def create(cls, owner, name, description, from_default=False,
178178
'analysis': a_id,
179179
'merge_dup_sample_ids': merge_duplicated_sample_ids})
180180
job = qdb.processing_job.ProcessingJob.create(
181-
owner, params)
181+
owner, params, True)
182182
sql = """INSERT INTO qiita.analysis_processing_job
183183
(analysis_id, processing_job_id)
184184
VALUES (%s, %s)"""
@@ -429,15 +429,17 @@ def mapping_file(self):
429429
430430
Returns
431431
-------
432-
str or None
433-
full filepath to the mapping file or None if not generated
432+
int or None
433+
The filepath id of the analysis mapping file or None
434+
if not generated
434435
"""
435-
fp = [fp for _, fp, fp_type in qdb.util.retrieve_filepaths(
436-
"analysis_filepath", "analysis_id", self._id)
437-
if fp_type == 'plain_text']
436+
fp = [fp_id
437+
for fp_id, _, fp_type in qdb.util.retrieve_filepaths(
438+
"analysis_filepath", "analysis_id", self._id)
439+
if fp_type == 'plain_text']
438440

439441
if fp:
440-
# returning the actual path vs. an array
442+
# returning the actual filepath id vs. an array
441443
return fp[0]
442444
else:
443445
return None

qiita_db/handlers/analysis.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,8 @@ def get(self, analysis_id):
5959
"""
6060
with qdb.sql_connection.TRN:
6161
a = _get_analysis(analysis_id)
62-
mf_fp = a.mapping_file
62+
mf_fp = qdb.util.get_filepath_information(
63+
a.mapping_file)['fullpath']
6364
response = None
6465
if mf_fp is not None:
6566
df = qdb.metadata_template.util.load_template_to_dataframe(

qiita_db/handlers/processing_job.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,7 @@ def post(self):
163163
params = qdb.software.Parameters.load(cmd, json_str=params_dict)
164164

165165
job = qdb.processing_job.ProcessingJob.create(
166-
qdb.user.User(user), params)
166+
qdb.user.User(user), params, True)
167167

168168
if status:
169169
job._set_status(status)

qiita_db/handlers/tests/test_artifact.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,8 @@ def test_get_artifact(self):
103103
'prep_information': [],
104104
'study': None,
105105
'analysis': 1,
106-
'processing_parameters': {'biom_table': 8, 'depth': 9000,
107-
'subsample_multinomial': False},
106+
'processing_parameters': {'biom_table': '8', 'depth': '9000',
107+
'subsample_multinomial': 'False'},
108108
'files': exp_fps}
109109
obs = loads(obs.body)
110110
# The timestamp is genreated at patch time, so we can't check for it

qiita_db/metadata_template/test/test_prep_template.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -887,9 +887,7 @@ def test_create(self):
887887
def test_create_already_prefixed_samples(self):
888888
"""Creates a new PrepTemplate"""
889889
fp_count = qdb.util.get_count('qiita.filepath')
890-
pt = npt.assert_warns(
891-
qdb.exceptions.QiitaDBWarning,
892-
qdb.metadata_template.prep_template.PrepTemplate.create,
890+
pt = qdb.metadata_template.prep_template.PrepTemplate.create(
893891
self.metadata_prefixed, self.test_study, self.data_type)
894892
self._common_creation_checks(pt, fp_count)
895893

qiita_db/metadata_template/test/test_sample_template.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1084,9 +1084,7 @@ def test_create_str_prefixes(self):
10841084

10851085
def test_create_already_prefixed_samples(self):
10861086
"""Creates a new SampleTemplate with the samples already prefixed"""
1087-
st = npt.assert_warns(
1088-
qdb.exceptions.QiitaDBWarning,
1089-
qdb.metadata_template.sample_template.SampleTemplate.create,
1087+
st = qdb.metadata_template.sample_template.SampleTemplate.create(
10901088
self.metadata_prefixed, self.new_study)
10911089
new_id = self.new_study.id
10921090
# The returned object has the correct id

qiita_db/metadata_template/test/test_util.py

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
from six import StringIO
1010
from unittest import TestCase, main
11+
import warnings
1112

1213
import numpy.testing as npt
1314
import pandas as pd
@@ -36,12 +37,31 @@ def test_prefix_sample_names_with_id(self):
3637
}
3738
exp_df = pd.DataFrame.from_dict(exp_metadata_dict, orient='index',
3839
dtype=str)
39-
qdb.metadata_template.util.prefix_sample_names_with_id(
40-
self.metadata_map, 1)
40+
with warnings.catch_warnings(record=True) as warn:
41+
qdb.metadata_template.util.prefix_sample_names_with_id(
42+
self.metadata_map, 1)
43+
self.assertEqual(len(warn), 0)
4144
self.metadata_map.sort_index(inplace=True)
4245
exp_df.sort_index(inplace=True)
4346
assert_frame_equal(self.metadata_map, exp_df)
4447

48+
# test that it only prefixes the samples that are needed
49+
metadata_dict = {
50+
'Sample1': {'int_col': 1, 'float_col': 2.1, 'str_col': 'str1'},
51+
'1.Sample2': {'int_col': 2, 'float_col': 3.1, 'str_col': '200'},
52+
'Sample3': {'int_col': 3, 'float_col': 3, 'str_col': 'string30'},
53+
}
54+
metadata_map = pd.DataFrame.from_dict(
55+
metadata_dict, orient='index', dtype=str)
56+
with warnings.catch_warnings(record=True) as warn:
57+
qdb.metadata_template.util.prefix_sample_names_with_id(
58+
metadata_map, 1)
59+
self.assertEqual(len(warn), 1)
60+
self.assertEqual(str(warn[0].message), 'Some of the samples were '
61+
'already prefixed with the study id.')
62+
metadata_map.sort_index(inplace=True)
63+
assert_frame_equal(metadata_map, exp_df)
64+
4565
def test_load_template_to_dataframe(self):
4666
obs = qdb.metadata_template.util.load_template_to_dataframe(
4767
StringIO(EXP_SAMPLE_TEMPLATE))

qiita_db/metadata_template/util.py

Lines changed: 19 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -34,29 +34,25 @@ def prefix_sample_names_with_id(md_template, study_id):
3434
study_id : int
3535
The study to which the metadata belongs to
3636
"""
37-
# Get all the prefixes of the index, defined as any string before a '.'
38-
prefixes = {idx.split('.', 1)[0] for idx in md_template.index}
39-
# If the samples have been already prefixed with the study id, the prefixes
40-
# set will contain only one element and it will be the str representation
41-
# of the study id
42-
if len(prefixes) == 1 and prefixes.pop() == str(study_id):
43-
# The samples were already prefixed with the study id
44-
warnings.warn("Sample names were already prefixed with the study id.",
45-
qdb.exceptions.QiitaDBWarning)
46-
else:
47-
# Create a new pandas series in which all the values are the study_id
48-
# and it is indexed as the metadata template
49-
study_ids = pd.Series([str(study_id)] * len(md_template.index),
50-
index=md_template.index)
51-
# Create a new column on the metadata template that includes the
52-
# metadata template indexes prefixed with the study id
53-
md_template['sample_name_with_id'] = (study_ids + '.' +
54-
md_template.index.values)
55-
md_template.index = md_template.sample_name_with_id
56-
del md_template['sample_name_with_id']
57-
# The original metadata template had the index column unnamed - remove
58-
# the name of the index for consistency
59-
md_template.index.name = None
37+
# loop over the samples and prefix those that aren't prefixed
38+
md_template['qiita_sample_name_with_id'] = pd.Series(
39+
[idx if idx.split('.', 1)[0] == str(study_id)
40+
else '%d.%s' % (study_id, idx)
41+
for idx in md_template.index], index=md_template.index)
42+
43+
# get the rows that are gonna change
44+
changes = len(md_template.index[
45+
md_template['qiita_sample_name_with_id'] != md_template.index])
46+
if changes != 0 and changes != len(md_template.index):
47+
warnings.warn(
48+
"Some of the samples were already prefixed with the study id.",
49+
qdb.exceptions.QiitaDBWarning)
50+
51+
md_template.index = md_template.qiita_sample_name_with_id
52+
del md_template['qiita_sample_name_with_id']
53+
# The original metadata template had the index column unnamed -> remove
54+
# the name of the index for consistency
55+
md_template.index.name = None
6056

6157

6258
def load_template_to_dataframe(fn, index='sample_name'):

0 commit comments

Comments
 (0)