This is an old revision of the document!
Performing re-exec to adjust stack size. -------------------------------------------------------------------------------- Sun May 03 14:50:05 2015 run on campusrocks2-0-1, pid=55420 [Apr 4 2015 20:59:59 R52415 ] DiscovarDeNovo \ READS="frac:0.5,sample:19::/campusdata/BME235/Spring2015Data/ba \ ms/SW019_adapterTrimmed_dupRemoved_150424.bam+frac:0.5,sample:1 \ 8::/campusdata/BME235/Spring2015Data/bams/SW018_adapterTrimmed_ \ dupRemoved_150424.bam" \ OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=50 \ MAX_MEM_GB=220 MEMORY_CHECK=True -------------------------------------------------------------------------------- SYSTEM INFO - OS: Linux :: 2.6.32-220.13.1.el6.x86_64 :: #1 SMP Tue Apr 17 23:56:34 BST 2012 - node name: campusrocks2-0-1.local - hardware type: x86_64 - cache size: 2048 KB - cpu MHz: 2400.084 - cpu model name: AMD Opteron(tm) Processor 6278 - physical memory: 252.39 GB MEMORY CHECK (typically takes several minutes; could cause machine to become sluggish or result in this job being killed) - Apparently able to allocate 100% of nominally available memory. - Can access at least 220 GB. Sun May 03 14:50:28 2015: finding input files Sun May 03 14:50:28 2015: reading 2 files (which may take a while) Sun May 03 14:50:28 2015: processing /campusdata/BME235/Spring2015Data/bams/SW019_adapterTrimmed_dupRemoved_150424.bam. Sun May 03 14:50:28 2015: memory in use = 0.01 GB, peak = 0.01 GB Sun May 03 15:13:28 2015: there are 124,970,747 reads of mean length 143 Sun May 03 15:13:28 2015: memory in use = 72.40 GB, peak = 74.21 GB Sun May 03 15:13:49 2015: reads sorted Sun May 03 15:13:49 2015: memory in use = 73.34 GB, peak = 74.21 GB Sun May 03 15:16:17 2015: data stashed in output structures Sun May 03 15:16:17 2015: memory in use = 12.77 GB, peak = 74.21 GB Sun May 03 15:16:17 2015: processing /campusdata/BME235/Spring2015Data/bams/SW018_adapterTrimmed_dupRemoved_150424.bam. Sun May 03 15:16:17 2015: memory in use = 12.77 GB, peak = 74.21 GB Sun May 03 15:33:24 2015: there are 138,864,653 reads of mean length 99 Sun May 03 15:33:24 2015: memory in use = 62.37 GB, peak = 74.21 GB Sun May 03 15:33:46 2015: reads sorted Sun May 03 15:33:46 2015: memory in use = 63.42 GB, peak = 74.21 GB Sun May 03 15:36:32 2015: data stashed in output structures Sun May 03 15:36:32 2015: memory in use = 26.67 GB, peak = 74.21 GB INPUT FILES: [1,type=frag,sample=19,lib=1,frac=0.5] /campusdata/BME235/Spring2015Data/bams/SW019_adapterTrimmed_dupRemoved_150424.bam [2,type=frag,sample=18,lib=1,frac=0.5] /campusdata/BME235/Spring2015Data/bams/SW018_adapterTrimmed_dupRemoved_150424.bam Sun May 03 15:36:32 2015: found 2 samples Sun May 03 15:36:32 2015: starts = 0,124970748 Sun May 03 15:45:17 2015: using 263,835,402 reads Sun May 03 15:45:17 2015: data extraction complete 54.8 minutes used extracting reads Sun May 03 15:52:15 2015: see total physical memory of 270,999,298,048 bytes Sun May 03 15:52:15 2015: see user-imposed limit on memory of 236,223,201,280 bytes Sun May 03 15:52:15 2015: 8.96 bytes per read base, assuming max memory available We need 4 passes. Expect 2578518 keys per batch. Provide 3243760 keys per batch. We need 5 passes. Expect 2062814 keys per batch. Provide 2387988 keys per batch. Sun May 03 16:51:17 2015: back from buildReadQGraph memory in use = 57,726,877,696 checksum_60 = 1091042242363737796 Sun May 03 16:53:37 2015: constructing places Sun May 03 16:54:01 2015: sorting places Sun May 03 17:00:11 2015: building all Sun May 03 17:06:25 2015: calling LongReadsToPaths Warning: HashSet initial size too small. Sun May 03 17:09:17 2015: writing Sun May 03 17:10:16 2015: translating paths Sun May 03 17:10:28 2015: final stage of path translation Sun May 03 17:29:36 2015: writing paths 56.1 seconds used reloading assembly Sun May 03 17:43:11 2015: start walking memory in use = 36,509,278,208 Sun May 03 17:44:23 2015: start walking memory in use = 36,612,886,528 2.86 minutes used cleaning 200-mer graph 1.97 hours used in ReadQGrapher 6.2e-06 seconds used reloading reads checksum_200 = 529403660712473571 1 peak mem usage = 140.27 GB 3.66 minutes used loading stuff 2 peak mem usage = 140.27 GB launching gap assemblies, mem usage = 35,265,961,984 Sun May 03 17:54:38 2015: finding unsatisfieds Sun May 03 17:54:45 2015: creating multiplicity map Sun May 03 17:54:47 2015: economizing links Sun May 03 17:54:47 2015: forming neighborhoods Sun May 03 17:54:47 2015: forming initial clusters Sun May 03 17:55:04 2015: start sort 0.688 seconds used sorting Sun May 03 17:55:05 2015: merging clusters xs.size( ) = 2689147 2.77 minutes used merging xs.size( ) = 2235783 Sun May 03 17:58:12 2015: start overlap-based merging Sun May 03 17:59:11 2015: start overlap-based merging LR.size( ) = 2219803 LR.size( ) = 1109905 Sun May 03 18:03:02 2015: now processing 1109905 blobs Sun May 03 18:03:03 2015: memory in use = 36.20 GB, peak = 140.27 GB .......... .......... .......... .......... .......... .......... .......... .......... .......... .......... 4.17 hours spent in local assemblies, memory in use = 39.19 GB, peak = 140.27 GB Sun May 03 22:13:21 2015: patch reserving space Sun May 03 22:13:21 2015: memory in use = 39.20 GB 25.2 seconds used patching, peak mem usage = 140.27 GB new_stuff.size( ) = 2202589 Sun May 03 22:14:27 2015: building hb2 22.1 seconds used in new stuff 1 test memory in use now = 37,405,421,568 Warning: HashSet initial size too small. Warning: HashSet initial size too small. Sun May 03 22:19:13 2015: back from buildBigKHBVFromReads 4.77 minutes used in new stuff 2 test peak mem usage = 140.27 GB 26.2 seconds used in new stuff 5 Sun May 03 22:20:12 2015: finding interesting reads Sun May 03 22:20:12 2015: memory in use = 32.74 GB, peak = 140.27 GB Sun May 03 22:20:24 2015: building dictionary Sun May 03 22:20:24 2015: memory in use = 33.02 GB, peak = 140.27 GB Sun May 03 22:27:44 2015: reducing Sun May 03 22:27:44 2015: memory in use = 140.68 GB, peak = 140.27 GB We need 1 passes. Expect 1066152 keys per batch. Provide 1508583 keys per batch. Sun May 03 22:37:47 2015: kmerizing Sun May 03 22:37:47 2015: memory in use = 171.42 GB, peak = 220.11 GB We need 1 passes. Expect 695406 keys per batch. Provide 1398894 keys per batch. There were 209 buffer overflows. Sun May 03 22:39:04 2015: cleaning Sun May 03 22:39:04 2015: memory in use = 171.43 GB, peak = 220.11 GB Sun May 03 22:40:21 2015: finding uniquely aligning edges Sun May 03 22:40:21 2015: memory in use = 171.44 GB, peak = 220.11 GB 28.2 minutes used in new phase hb.N( ) = 8067352, hb.EdgeObjectCount( ) = 4464217 26257 paths improved by rerouting Sum(invalid) = 19494, npids = 131917701 691 edges tamped down Sun May 03 22:47:10 2015: checking involution Sun May 03 22:47:10 2015: done WARNING: 684 suspicious read-paths. Sum(invalid) = 2233, npids = 131917701 677 edges tamped down Sun May 03 22:51:06 2015: making paths index for pull apart Sun May 03 22:51:14 2015: pulling apart repeats 0.0117 seconds used separating paths 1 5.41 seconds used in fixing mToLeft, mToRight, and mEdgeToPathIds Sun May 03 22:51:28 2015: there were 95 repeats pulled apart. Sun May 03 22:51:28 2015: there were 8121 read paths removed during separation. Sun May 03 22:51:33 2015: improving paths Sun May 03 22:59:39 2015: done 5684 paths extended Sun May 03 23:02:40 2015: start degloop Sun May 03 23:02:40 2015: creating path index Sun May 03 23:02:51 2015: starting loop Sun May 03 23:02:51 2015: degloop complete Sun May 03 23:03:22 2015: unwinding three-edge plasmids Sun May 03 23:03:24 2015: removing small components Sun May 03 23:04:54 2015: writing a.fin files Sun May 03 23:13:51 2015: determining candidates Sun May 03 23:13:52 2015: determining candidates CN fraction good = 0.33 Sun May 03 23:14:34 2015: deleting 0 gaps and adding 0 gaps to force symmetry Sun May 03 23:14:54 2015: done making gaps, time used = 30.6 seconds Sun May 03 23:16:10 2015: determining candidates Sun May 03 23:16:11 2015: determining candidates 1.7 seconds using setting up final fasta 12.5 seconds using printing final fasta ==================================================================================== assembly has 273653 edges of mean length 807.36 contig line N50: 1,489 scaffold line N50: 1,489 total bases in 1 kb+ scaffolds: 101,397,871 total bases in 10 kb+ scaffolds: 151,417 There are 263,835,402 reads of mean length 100.0 and mean base quality 36.7. MPL1 = mean length of first read in pair up to first error = 7 (normal range is 175-225 for 250 base reads) Estimated chimera rate in read pairs (including mismapping) = 21.08%. WARNING: based on the observed chimera rate, it looks like something may be very wrong with your input data. genomic read coverage, using 1 kb+ scaffolds for genome size estimate: 260.2 run started Sun May 03 14:50:05 2015, completed Sun May 03 23:21:47 2015 peak mem usage = 220.11 GB, total time = 8.53 hours final checksum = 63450798033837183 DiscovarDeNovo READS="frac:0.5,sample:19::/campusdata/BME235/Spring2015Data/bams/SW019_adapterTrimmed_dupRemoved_150424.bam+frac:0.5,sample:18::/campusdata/BME235/Spring2015Data/bams/SW018_adapterTrimmed_dupRemoved_150424.bam" OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=50 MAX_MEM_GB=220 MEMORY_CHECK=True Sun May 03 23:21:48 2015: done
