This is an old revision of the document!
IPerforming re-exec to adjust stack size. -------------------------------------------------------------------------------- Sun May 10 10:10:33 2015 run on campusrocks2-0-0, pid=10320 [Apr 4 2015 20:59:59 R52415 ] DiscovarDeNovo \ READS="frac:0.5,sample:19::/campusdata/ndudek/fastq_to_bam/UCSF \ _SW019_noAdap_noDup.bam+frac:0.5,sample:18::/campusdata/ndudek/ \ fastq_to_bam/UCSF_SW018_noAdap_noDup.bam" \ OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=39 \ MAX_MEM_GB=220 MEMORY_CHECK=True -------------------------------------------------------------------------------- SYSTEM INFO - OS: Linux :: 2.6.32-220.13.1.el6.x86_64 :: #1 SMP Tue Apr 17 23:56:34 BST 2012 - node name: campusrocks2-0-0.local - hardware type: x86_64 - cache size: 2048 KB - cpu MHz: 2400.042 - cpu model name: AMD Opteron(tm) Processor 6278 - physical memory: 252.39 GB MEMORY CHECK (typically takes several minutes; could cause machine to become sluggish or result in this job being killed) - Apparently able to allocate 100% of nominally available memory. - Can access at least 220 GB. Sun May 10 10:17:29 2015: finding input files Sun May 10 10:17:29 2015: reading 2 files (which may take a while) Sun May 10 10:17:29 2015: processing /campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam. Sun May 10 10:17:29 2015: memory in use = 0.01 GB, peak = 0.01 GB Sun May 10 10:54:12 2015: there are 75,081,065 reads of mean length 249 Sun May 10 10:54:12 2015: memory in use = 66.13 GB, peak = 67.76 GB Sun May 10 10:54:24 2015: reads sorted Sun May 10 10:54:24 2015: memory in use = 66.70 GB, peak = 67.76 GB Sun May 10 10:56:44 2015: data stashed in output structures Sun May 10 10:56:44 2015: memory in use = 15.53 GB, peak = 67.76 GB Sun May 10 10:56:44 2015: processing /campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam. Sun May 10 10:56:44 2015: memory in use = 15.53 GB, peak = 67.76 GB Sun May 10 11:21:24 2015: there are 56,931,153 reads of mean length 232 Sun May 10 11:21:24 2015: memory in use = 49.12 GB, peak = 67.76 GB Sun May 10 11:21:34 2015: reads sorted Sun May 10 11:21:34 2015: memory in use = 49.55 GB, peak = 67.76 GB Sun May 10 11:23:13 2015: data stashed in output structures Sun May 10 11:23:13 2015: memory in use = 25.97 GB, peak = 67.76 GB INPUT FILES: [1,type=frag,sample=19,lib=1,frac=0.5] /campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam [2,type=frag,sample=18,lib=1,frac=0.5] /campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam Sun May 10 11:23:13 2015: found 2 samples Sun May 10 11:23:13 2015: starts = 0,75081066 Sun May 10 11:30:40 2015: using 132,012,220 reads Sun May 10 11:30:40 2015: data extraction complete 1.22 hours used extracting reads Sun May 10 11:31:08 2015: see total physical memory of 270,999,298,048 bytes Sun May 10 11:31:08 2015: see user-imposed limit on memory of 236,223,201,280 bytes Sun May 10 11:31:08 2015: 7.38 bytes per read base, assuming max memory available We need 4 passes. Expect 4797148 keys per batch. Provide 5352492 keys per batch. We need 11 passes. Expect 1744417 keys per batch. Provide 2097642 keys per batch. Sun May 10 12:37:53 2015: back from buildReadQGraph memory in use = 53,805,211,648 checksum_60 = 1093167177637492236 Sun May 10 12:40:09 2015: constructing places Sun May 10 12:40:38 2015: sorting places Sun May 10 12:44:01 2015: building all Sun May 10 12:48:13 2015: calling LongReadsToPaths Sun May 10 13:07:11 2015: writing Sun May 10 13:09:42 2015: translating paths Sun May 10 13:12:35 2015: final stage of path translation Sun May 10 13:28:00 2015: writing paths 1.06 minutes used reloading assembly Sun May 10 13:37:55 2015: start walking memory in use = 41,533,923,328 Sun May 10 13:42:17 2015: start walking memory in use = 41,263,316,992 9.69 minutes used cleaning 200-mer graph 2.32 hours used in ReadQGrapher 5.01e-06 seconds used reloading reads checksum_200 = 1086642745387393140 1 peak mem usage = 184.09 GB 1.59 minutes used loading stuff 2 peak mem usage = 184.09 GB launching gap assemblies, mem usage = 36,172,001,280 Sun May 10 13:53:11 2015: finding unsatisfieds Sun May 10 13:53:20 2015: creating multiplicity map Sun May 10 13:53:25 2015: economizing links Sun May 10 13:53:25 2015: forming neighborhoods Sun May 10 13:53:41 2015: forming initial clusters Sun May 10 13:54:09 2015: start sort 1.61 seconds used sorting Sun May 10 13:54:11 2015: merging clusters xs.size( ) = 6700008 19.3 minutes used merging xs.size( ) = 2320189 Sun May 10 14:13:53 2015: start overlap-based merging Sun May 10 14:14:59 2015: start overlap-based merging LR.size( ) = 2265143 LR.size( ) = 1132970 Sun May 10 14:20:39 2015: now processing 1132970 blobs Sun May 10 14:20:39 2015: memory in use = 39.29 GB, peak = 184.09 GB .......... .......... .......... .......... .......... .......... .......... .......... .......... .......... 8.31 hours spent in local assemblies, memory in use = 41.74 GB, peak = 184.09 GB Sun May 10 22:39:31 2015: patch reserving space Sun May 10 22:39:31 2015: memory in use = 41.75 GB 35.9 seconds used patching, peak mem usage = 184.09 GB new_stuff.size( ) = 4344323 Sun May 10 22:41:53 2015: building hb2 55.3 seconds used in new stuff 1 test memory in use now = 40,122,859,520 Sun May 10 23:11:37 2015: back from buildBigKHBVFromReads 29.8 minutes used in new stuff 2 test peak mem usage = 184.09 GB 1.23 minutes used in new stuff 5 Sun May 10 23:14:10 2015: finding interesting reads Sun May 10 23:14:10 2015: memory in use = 34.11 GB, peak = 184.09 GB Sun May 10 23:24:04 2015: building dictionary Sun May 10 23:24:04 2015: memory in use = 34.17 GB, peak = 184.09 GB Sun May 10 23:25:54 2015: reducing Sun May 10 23:25:54 2015: memory in use = 134.54 GB, peak = 184.09 GB We need 1 passes. Expect 1140988 keys per batch. Provide 2730092 keys per batch. Sun May 10 23:28:51 2015: kmerizing Sun May 10 23:28:51 2015: memory in use = 143.29 GB, peak = 184.09 GB We need 2 passes. Expect 2094178 keys per batch. Provide 4706876 keys per batch. There were 234 buffer overflows. Sun May 10 23:37:01 2015: cleaning Sun May 10 23:37:01 2015: memory in use = 143.29 GB, peak = 184.09 GB Sun May 10 23:38:28 2015: finding uniquely aligning edges Sun May 10 23:38:28 2015: memory in use = 143.29 GB, peak = 184.09 GB 1 hours used in new phase hb.N( ) = 18564840, hb.EdgeObjectCount( ) = 14056399 153043 paths improved by rerouting Sum(invalid) = 241436, npids = 66006110 25876 edges tamped down Mon May 11 00:06:00 2015: checking involution Mon May 11 00:06:00 2015: done WARNING: 15782 suspicious read-paths. Sum(invalid) = 41027, npids = 66006110 15750 edges tamped down Mon May 11 00:16:55 2015: making paths index for pull apart Mon May 11 00:17:19 2015: pulling apart repeats 0.426 seconds used separating paths 1 31.8 seconds used in fixing mToLeft, mToRight, and mEdgeToPathIds Mon May 11 00:19:01 2015: there were 11047 repeats pulled apart. Mon May 11 00:19:01 2015: there were 41908 read paths removed during separation. Mon May 11 00:19:29 2015: improving paths Mon May 11 00:24:58 2015: done 1306140 paths extended Mon May 11 00:28:41 2015: start degloop Mon May 11 00:28:41 2015: creating path index Mon May 11 00:29:08 2015: starting loop Mon May 11 00:29:28 2015: degloop complete Mon May 11 00:31:16 2015: unwinding three-edge plasmids Mon May 11 00:31:21 2015: removing small components Mon May 11 00:34:52 2015: writing a.fin files Mon May 11 00:44:05 2015: determining candidates Mon May 11 00:44:06 2015: determining candidates CN fraction good = 0.45 Mon May 11 00:47:00 2015: deleting 0 gaps and adding 8 gaps to force symmetry Mon May 11 00:48:12 2015: done making gaps, time used = 3.22 minutes Mon May 11 00:53:16 2015: determining candidates Mon May 11 00:53:17 2015: determining candidates 18.5 seconds using setting up final fasta 3.39 minutes using printing final fasta ==================================================================================== assembly has 3094953 edges of mean length 1062.18 contig line N50: 3,979 scaffold line N50: 3,979 total bases in 1 kb+ scaffolds: 1,528,625,509 total bases in 10 kb+ scaffolds: 137,959,107 There are 132,012,220 reads of mean length 242.4 and mean base quality 33.0. MPL1 = mean length of first read in pair up to first error = 156 (normal range is 175-225 for 250 base reads) Estimated chimera rate in read pairs (including mismapping) = 0.07%. genomic read coverage, using 1 kb+ scaffolds for genome size estimate: 20.9 run started Sun May 10 10:10:33 2015, completed Mon May 11 01:02:09 2015 peak mem usage = 184.09 GB, total time = 14.9 hours final checksum = 652044377802948288 DiscovarDeNovo READS="frac:0.5,sample:19::/campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam+frac:0.5,sample:18::/campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam" OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=39 MAX_MEM_GB=220 MEMORY_CHECK=True Mon May 11 01:02:23 2015: done
