Segmentation fault

[diego-rt]

Hello,

I am running hifiasm with a giant genome (~30 Gbp) using v0.24 and am getting a seg fault during the UL step when using -D 50:

INFO:    Using cached SIF image
Reads has been loaded.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
[M::ha_assemble::9386.280*0.93] ==> loaded corrected reads and overlaps from disk
[M::ha_opt_update_cov_min] updated max_n_chain to 3500
[M::gen_telo_end_t::] ==> # 5'-telomeres::84, # 3'-telomeres::52, # tot::117442443, motif::CCCTAA, motif_len::6
[M::ul_load::] ==> UL
[M::ha_opt_update_cov] updated max_n_chain to 3500
[M::append_inexact_edges] # inserted inexact edges: 8399828
[M::gen_cov_track::] # bases: 113858859997
[M::dedup_HiFis::] # unitigs: 3565388, # edges: 21881480, # cc_num: 103980669
script.sh: line 2: 30075 Segmentation fault      hifiasm -o Amex8.0.asm -t 88 --ul mergedUltralongReads.fq.gz --ul-rate 0.02 --telo-m CCCTAA -l 3 -f 40 -D 50 -k 63 -w 63 Revio_FC021.filtered.fq.gz.filtered.fq.gz Revio_FC004.filtered.fq.gz.filtered.fq.gz Revio_FC003.filtered.fq.gz.filtered.fq.gz Revio_FC001.filtered.fq.gz.filtered.fq.gz Revio_FC005.filtered.fq.gz.filtered.fq.gz Revio_FC009.filtered.fq.gz.filtered.fq.gz Revio_FC008.filtered.fq.gz.filtered.fq.gz Revio_FC013.filtered.fq.gz.filtered.fq.gz Revio_FC011.filtered.fq.gz.filtered.fq.gz Revio_FC012.filtered.fq.gz.filtered.fq.gz Revio_FC007.filtered.fq.gz.filtered.fq.gz Revio_FC015.filtered.fq.gz.filtered.fq.gz Revio_FC002.filtered.fq.gz.filtered.fq.gz Revio_FC022.filtered.fq.gz.filtered.fq.gz Revio_FC010.filtered.fq.gz.filtered.fq.gz Revio_FC014.filtered.fq.gz.filtered.fq.gz Revio_FC017.filtered.fq.gz.filtered.fq.gz Revio_FC006.filtered.fq.gz.filtered.fq.gz Revio_FC020.filtered.fq.gz.filtered.fq.gz Revio_FC018.filtered.fq.gz.filtered.fq.gz Revio_FC016.filtered.fq.gz.filtered.fq.gz Revio_FC019.filtered.fq.gz.filtered.fq.gz

However, when running it with -D 20 it worked fine, and the HiFi step worked fine as well. This is the log of the -D 20 run that worked fine:

INFO:    Using cached SIF image
Reads has been loaded.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
[M::ha_assemble::7522.597*0.93] ==> loaded corrected reads and overlaps from disk
[M::ha_opt_update_cov_min] updated max_n_chain to 1400
[M::gen_telo_end_t::] ==> # 5'-telomeres::83, # 3'-telomeres::52, # tot::117442443, motif::CCCTAA, motif_len::6
[M::ul_load::] ==> UL
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::append_inexact_edges] # inserted inexact edges: 4988378
[M::gen_cov_track::] # bases: 110657137701
[M::dedup_HiFis::] # unitigs: 3400304, # edges: 16046614, # cc_num: 104029012
[M::ha_ct_shrink::67088.282*1.91] ==> counted 3659688 distinct minimizer k-mers
[M::ha_ft_ul_gen::67091.626*[email protected]] ==> filtered out 3659688 k-mers occurring 1400 or more times
[M::yak_count] collected 12189828160 minimizers
[M::ha_pt_ul_gen::68094.633*2.30] ==> counted 220172996 distinct minimizer k-mers
[M::ha_ct_shrink::68096.129*2.30] ==> counted 220172996 distinct minimizer k-mers
[M::yak_count] collected 12189828160 minimizers
[M::ha_pt_ul_gen::69516.493*2.77] ==> indexed 12189828160 positions
[M::uidx_l_build] Index has been built.
[M::uidx_write] Index has been written.
[M::scall_ul_pipeline::872785.956] ==> Qualification
[M::scall_ul_pipeline::] ==> # reads: 9775498, # bases: 0
[M::scall_ul_pipeline::] ==> # fully covered reads: 7692375
[M::write_all_ul_t] Index has been written.

Many thanks!

[chhylp123]

Just double check: The default hifiasm (UL) cannot be finished, while -D20 worked?

[diego-rt]

Yes, sorry for my bad wording.

With -D 20 the UL step finished successfully the alignment step.

With -D 50 I got a segmentation fault.

[chhylp123]

Hmm, it should not happen. By the way, why would you like to give very large values for -D?

[diego-rt]

I have several gaps at satellites that need higher kmer multiplicity values to resolve. I'm usually patching them manually by reassembling the satellite region with higher -D values. However, since you reduced the RAM requirements with the new read correction method (thanks!!) then I thought I could raise the -D values for the whole assembly and save some manual patching.

[chhylp123]

I see. Theoretically if you can share the data with me then I can have a look to fix this issue. However we don't have machine to assemble such large genome...

[diego-rt]

Yes I imagined so :/ It is really a gigantic thing...

I guess I could run whatever diagnostic /debugging program you might need. Would that be an option?

Thanks again for your help.

[diego-rt]

Hi @chhylp123

Actually after 21 days the -D 20 run also died probably because of some overflow. Here goes the log:

INFO:    Using cached SIF image
Reads has been loaded.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
Loading ma_hit_ts from disk... 
ma_hit_ts has been read.
[M::ha_assemble::7522.597*0.93] ==> loaded corrected reads and overlaps from disk
[M::ha_opt_update_cov_min] updated max_n_chain to 1400
[M::gen_telo_end_t::] ==> # 5'-telomeres::83, # 3'-telomeres::52, # tot::117442443, motif::CCCTAA, motif_len::6
[M::ul_load::] ==> UL
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::append_inexact_edges] # inserted inexact edges: 4988378
[M::gen_cov_track::] # bases: 110657137701
[M::dedup_HiFis::] # unitigs: 3400304, # edges: 16046614, # cc_num: 104029012
[M::ha_ct_shrink::67088.282*1.91] ==> counted 3659688 distinct minimizer k-mers
[M::ha_ft_ul_gen::67091.626*[email protected]] ==> filtered out 3659688 k-mers occurring 1400 or more times
[M::yak_count] collected 12189828160 minimizers
[M::ha_pt_ul_gen::68094.633*2.30] ==> counted 220172996 distinct minimizer k-mers
[M::ha_ct_shrink::68096.129*2.30] ==> counted 220172996 distinct minimizer k-mers
[M::yak_count] collected 12189828160 minimizers
[M::ha_pt_ul_gen::69516.493*2.77] ==> indexed 12189828160 positions
[M::uidx_l_build] Index has been built.
[M::uidx_write] Index has been written.
[M::scall_ul_pipeline::872785.956] ==> Qualification
[M::scall_ul_pipeline::] ==> # reads: 9775498, # bases: 0
[M::scall_ul_pipeline::] ==> # fully covered reads: 7692375
[M::write_all_ul_t] Index has been written.
[M::print_ovlp_src_bl_stat::] ==> # HiFi reads:117442443, # covered HiFi reads:24892519, # chained HiFi reads:21290767
[M::print_ovlp_src_bl_stat::] ==> # passed UL reads:9775498, # fully corrected UL reads:6955262, # almost fully corrected UL reads:458060, # UL reads have primary chains:2253557
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:1771479, # done:1088060
[M::clean_contain_g0::] # discard cread::1165719
[M::update_sg_uo::] ==> # gfa reads:12617259, # covered gfa reads:12494769
[M::ul_refine_alignment::] ==> UL refinement...
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::gen_cov_track::] # bases: 60752596167
[M::work_ul_gchains::] # try:2162852, # done:1412930
[M::update_sg_uo::] ==> # gfa reads:12515926, # covered gfa reads:12469411
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:31933, # done:16261
[M::clean_contain_g0::] # discard cread::3846
[M::update_sg_uo::] ==> # gfa reads:12512080, # covered gfa reads:12465874
[M::ul_refine_alignment::] ==> UL refinement...
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::gen_cov_track::] # bases: 59994171808
[M::work_ul_gchains::] # try:73362, # done:26521
[M::update_sg_uo::] ==> # gfa reads:12508904, # covered gfa reads:12464150
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:15118, # done:627
[M::clean_contain_g0::] # discard cread::141
[M::update_sg_uo::] ==> # gfa reads:12508763, # covered gfa reads:12464005
[M::ul_refine_alignment::] ==> UL refinement...
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::gen_cov_track::] # bases: 59607322086
[M::work_ul_gchains::] # try:48127, # done:9323
[M::update_sg_uo::] ==> # gfa reads:12506122, # covered gfa reads:12463209
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:14631, # done:393
[M::clean_contain_g0::] # discard cread::14
[M::update_sg_uo::] ==> # gfa reads:12506108, # covered gfa reads:12463178
[M::ul_refine_alignment::] ==> UL refinement...
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::gen_cov_track::] # bases: 59322660147
[M::work_ul_gchains::] # try:47197, # done:9425
[M::update_sg_uo::] ==> # gfa reads:12502998, # covered gfa reads:12462117
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:14443, # done:388
[M::clean_contain_g0::] # discard cread::19
[M::update_sg_uo::] ==> # gfa reads:12502979, # covered gfa reads:12462084
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::work_ul_gchains_consensus::] # try:13893, # done:2
[M::clean_contain_g0::] # discard cread::14
[M::update_sg_uo::] ==> # gfa reads:12502860, # covered gfa reads:12461966
[M::purge_dups] homozygous read coverage threshold: 69
[M::purge_dups] purge duplication coverage threshold: 87
[M::ug_ext_gfa::] # tips::225
Writing raw unitig GFA to disk... 
[M::update_sg_uo::] ==> # gfa reads:12481991, # covered gfa reads:12445183
[M::ul_realignment::] ==> starting UL
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::gen_cov_track::] # bases: 56202918219
[M::ha_ct_shrink::1255582.734*46.60] ==> counted 241575522 distinct minimizer k-mers
[M::ha_ft_ul_gen::1255630.240*[email protected]] ==> filtered out 241575522 k-mers occurring 32 or more times
[M::yak_count] collected 2086718049 minimizers
[M::ha_pt_ul_gen::1257507.640*46.54] ==> counted 226070950 distinct minimizer k-mers
[M::ha_ct_shrink::1257508.388*46.54] ==> counted 226070950 distinct minimizer k-mers
[M::yak_count] collected 2086718049 minimizers
[M::ha_pt_ul_gen::1259461.581*46.48] ==> indexed 2086718049 positions
[M::uidx_l_build] Index has been built.
[M::uidx_write] Index has been written.
[M::ha_opt_update_cov] updated max_n_chain to 1400
[M::cal_graph_ovlp_binning::10068.092] ==> Qualification
[M::write_emask_t] Index has been written.
[M::rescall_ul_pipeline::50933.938] ==> Qualification
[M::rescall_ul_pipeline::] ==> # reads: 9775498, # bases: 0, # fully corrected reads: 8418352
[M::write_all_ul_t] Index has been written.
[M::print_integert_ovlp_stat::] # UL reads::2423262, # UL ovlps::51280912
[M::print_integert_ovlp_stat::] # UL reads::573067, # UL ovlps::3130678
[M::load_all_ul_t] Index has been loaded.
+[M::realloc_rdb_adv] rid_n0::117442443, rid_n::117743821
-[M::realloc_rdb_adv] rid_n0::117442443, rid_n::117743821
+[M::update_paf] dn::117442443
[M::renew_ng] nocc::117743821, ng->n_seq::117743821, sg->n_seq::117442443
[M::purge_dups] homozygous read coverage threshold: 69
[M::purge_dups] purge duplication coverage threshold: 87
[M::ug_ext_gfa::] # tips::18
Writing processed unitig GFA to disk... 
[M::purge_dups] homozygous read coverage threshold: 69
[M::purge_dups] purge duplication coverage threshold: 87
[M::mc_solve:: # edges: 24000]
[M::mc_solve_core_adv::1.175] ==> Partition
[M::adjust_utg_by_primary] primary contig coverage range: [58, infinity]
malloc(): invalid size (unsorted)
script.sh: line 2: 54857 Aborted                 hifiasm -o Amex8.0.asm -t 88 --ul mergedUltralongReads.fq.gz --ul-rate 0.02 --telo-m CCCTAA -l 3 -f 40 -D 20 -k 63 -w 63 Revio_FC021.filtered.fq.gz.filtered.fq.gz Revio_FC004.filtered.fq.gz.filtered.fq.gz Revio_FC003.filtered.fq.gz.filtered.fq.gz Revio_FC001.filtered.fq.gz.filtered.fq.gz Revio_FC005.filtered.fq.gz.filtered.fq.gz Revio_FC009.filtered.fq.gz.filtered.fq.gz Revio_FC008.filtered.fq.gz.filtered.fq.gz Revio_FC013.filtered.fq.gz.filtered.fq.gz Revio_FC011.filtered.fq.gz.filtered.fq.gz Revio_FC012.filtered.fq.gz.filtered.fq.gz Revio_FC007.filtered.fq.gz.filtered.fq.gz Revio_FC015.filtered.fq.gz.filtered.fq.gz Revio_FC002.filtered.fq.gz.filtered.fq.gz Revio_FC022.filtered.fq.gz.filtered.fq.gz Revio_FC010.filtered.fq.gz.filtered.fq.gz Revio_FC014.filtered.fq.gz.filtered.fq.gz Revio_FC017.filtered.fq.gz.filtered.fq.gz Revio_FC006.filtered.fq.gz.filtered.fq.gz Revio_FC020.filtered.fq.gz.filtered.fq.gz Revio_FC018.filtered.fq.gz.filtered.fq.gz Revio_FC016.filtered.fq.gz.filtered.fq.gz Revio_FC019.filtered.fq.gz.filtered.fq.gz

Many thanks in advance!

[chhylp123]

Hmm, can it work wit -D5 or -D10?

[diego-rt]

It used to work with -D 5 and -D 20 with versions 0.19.8 and 0.19.9.

Haven't tried -D 5 with 0.24 but it would take more than a month to test...

The processed unitig graph with -D 20 looks great though! Maybe I will not need -D 50 if I can get -D 20 to finish

[diego-rt]

Hi @chhylp123

Do you think incorporating HiC phasing would enable us to bypass this error?

[diego-rt]

 Hi @chhylp123

Is it possible that the tangled state of the graph is responsible for the segmentation fault during partitioning? All the chromosomes are linked by the telomeric repeats (center of the start-shaped component) but this was never a problem before.