IPerforming re-exec to adjust stack size.
Sun May 10 10:10:33 2015 run on campusrocks2-0-0, pid=10320 [Apr 4 2015 20:59:59 R52415 ]
DiscovarDeNovo \
READS="frac:0.5,sample:19::/campusdata/ndudek/fastq_to_bam/UCSF \
_SW019_noAdap_noDup.bam+frac:0.5,sample:18::/campusdata/ndudek/ \
fastq_to_bam/UCSF_SW018_noAdap_noDup.bam" \
OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=39 \
MAX_MEM_GB=220 MEMORY_CHECK=True
SYSTEM INFO
- OS: Linux :: 2.6.32-220.13.1.el6.x86_64 :: #1 SMP Tue Apr 17 23:56:34 BST 2012
- node name: campusrocks2-0-0.local
- hardware type: x86_64
- cache size: 2048 KB
- cpu MHz: 2400.042
- cpu model name: AMD Opteron™ Processor 6278
- physical memory: 252.39 GB
MEMORY CHECK (typically takes several minutes; could cause
machine to become sluggish or result in this job being killed)
- Apparently able to allocate 100% of nominally available memory.
- Can access at least 220 GB.
Sun May 10 10:17:29 2015: finding input files
Sun May 10 10:17:29 2015: reading 2 files (which may take a while)
Sun May 10 10:17:29 2015: processing /campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam.
Sun May 10 10:17:29 2015: memory in use = 0.01 GB, peak = 0.01 GB
Sun May 10 10:54:12 2015: there are 75,081,065 reads of mean length 249
Sun May 10 10:54:12 2015: memory in use = 66.13 GB, peak = 67.76 GB
Sun May 10 10:54:24 2015: reads sorted
Sun May 10 10:54:24 2015: memory in use = 66.70 GB, peak = 67.76 GB
Sun May 10 10:56:44 2015: data stashed in output structures
Sun May 10 10:56:44 2015: memory in use = 15.53 GB, peak = 67.76 GB
Sun May 10 10:56:44 2015: processing /campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam.
Sun May 10 10:56:44 2015: memory in use = 15.53 GB, peak = 67.76 GB
Sun May 10 11:21:24 2015: there are 56,931,153 reads of mean length 232
Sun May 10 11:21:24 2015: memory in use = 49.12 GB, peak = 67.76 GB
Sun May 10 11:21:34 2015: reads sorted
Sun May 10 11:21:34 2015: memory in use = 49.55 GB, peak = 67.76 GB
Sun May 10 11:23:13 2015: data stashed in output structures
Sun May 10 11:23:13 2015: memory in use = 25.97 GB, peak = 67.76 GB
INPUT FILES:
[1,type=frag,sample=19,lib=1,frac=0.5] /campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam
[2,type=frag,sample=18,lib=1,frac=0.5] /campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam
Sun May 10 11:23:13 2015: found 2 samples
Sun May 10 11:23:13 2015: starts = 0,75081066
Sun May 10 11:30:40 2015: using 132,012,220 reads
Sun May 10 11:30:40 2015: data extraction complete
1.22 hours used extracting reads
Sun May 10 11:31:08 2015: see total physical memory of 270,999,298,048 bytes
Sun May 10 11:31:08 2015: see user-imposed limit on memory of 236,223,201,280 bytes
Sun May 10 11:31:08 2015: 7.38 bytes per read base, assuming max memory available
We need 4 passes.
Expect 4797148 keys per batch.
Provide 5352492 keys per batch.
We need 11 passes.
Expect 1744417 keys per batch.
Provide 2097642 keys per batch.
Sun May 10 12:37:53 2015: back from buildReadQGraph
memory in use = 53,805,211,648
checksum_60 = 1093167177637492236
Sun May 10 12:40:09 2015: constructing places
Sun May 10 12:40:38 2015: sorting places
Sun May 10 12:44:01 2015: building all
Sun May 10 12:48:13 2015: calling LongReadsToPaths
Sun May 10 13:07:11 2015: writing
Sun May 10 13:09:42 2015: translating paths
Sun May 10 13:12:35 2015: final stage of path translation
Sun May 10 13:28:00 2015: writing paths
1.06 minutes used reloading assembly
Sun May 10 13:37:55 2015: start walking
memory in use = 41,533,923,328
Sun May 10 13:42:17 2015: start walking
memory in use = 41,263,316,992
9.69 minutes used cleaning 200-mer graph
2.32 hours used in ReadQGrapher
5.01e-06 seconds used reloading reads
checksum_200 = 1086642745387393140
1 peak mem usage = 184.09 GB
1.59 minutes used loading stuff
2 peak mem usage = 184.09 GB
launching gap assemblies, mem usage = 36,172,001,280
Sun May 10 13:53:11 2015: finding unsatisfieds
Sun May 10 13:53:20 2015: creating multiplicity map
Sun May 10 13:53:25 2015: economizing links
Sun May 10 13:53:25 2015: forming neighborhoods
Sun May 10 13:53:41 2015: forming initial clusters
Sun May 10 13:54:09 2015: start sort
1.61 seconds used sorting
Sun May 10 13:54:11 2015: merging clusters
xs.size( ) = 6700008
19.3 minutes used merging
xs.size( ) = 2320189
Sun May 10 14:13:53 2015: start overlap-based merging
Sun May 10 14:14:59 2015: start overlap-based merging
LR.size( ) = 2265143
LR.size( ) = 1132970
Sun May 10 14:20:39 2015: now processing 1132970 blobs
Sun May 10 14:20:39 2015: memory in use = 39.29 GB, peak = 184.09 GB
………. ………. ………. ………. ……….
………. ………. ………. ………. ……….
8.31 hours spent in local assemblies, memory in use = 41.74 GB, peak = 184.09 GB
Sun May 10 22:39:31 2015: patch reserving space
Sun May 10 22:39:31 2015: memory in use = 41.75 GB
35.9 seconds used patching, peak mem usage = 184.09 GB
new_stuff.size( ) = 4344323
Sun May 10 22:41:53 2015: building hb2
55.3 seconds used in new stuff 1 test
memory in use now = 40,122,859,520
Sun May 10 23:11:37 2015: back from buildBigKHBVFromReads
29.8 minutes used in new stuff 2 test
peak mem usage = 184.09 GB
1.23 minutes used in new stuff 5
Sun May 10 23:14:10 2015: finding interesting reads
Sun May 10 23:14:10 2015: memory in use = 34.11 GB, peak = 184.09 GB
Sun May 10 23:24:04 2015: building dictionary
Sun May 10 23:24:04 2015: memory in use = 34.17 GB, peak = 184.09 GB
Sun May 10 23:25:54 2015: reducing
Sun May 10 23:25:54 2015: memory in use = 134.54 GB, peak = 184.09 GB
We need 1 passes.
Expect 1140988 keys per batch.
Provide 2730092 keys per batch.
Sun May 10 23:28:51 2015: kmerizing
Sun May 10 23:28:51 2015: memory in use = 143.29 GB, peak = 184.09 GB
We need 2 passes.
Expect 2094178 keys per batch.
Provide 4706876 keys per batch.
There were 234 buffer overflows.
Sun May 10 23:37:01 2015: cleaning
Sun May 10 23:37:01 2015: memory in use = 143.29 GB, peak = 184.09 GB
Sun May 10 23:38:28 2015: finding uniquely aligning edges
Sun May 10 23:38:28 2015: memory in use = 143.29 GB, peak = 184.09 GB
1 hours used in new phase
hb.N( ) = 18564840, hb.EdgeObjectCount( ) = 14056399
153043 paths improved by rerouting
Sum(invalid) = 241436, npids = 66006110
25876 edges tamped down
Mon May 11 00:06:00 2015: checking involution
Mon May 11 00:06:00 2015: done
WARNING: 15782 suspicious read-paths.
Sum(invalid) = 41027, npids = 66006110
15750 edges tamped down
Mon May 11 00:16:55 2015: making paths index for pull apart
Mon May 11 00:17:19 2015: pulling apart repeats
0.426 seconds used separating paths 1
31.8 seconds used in fixing mToLeft, mToRight, and mEdgeToPathIds
Mon May 11 00:19:01 2015: there were 11047 repeats pulled apart.
Mon May 11 00:19:01 2015: there were 41908 read paths removed during separation.
Mon May 11 00:19:29 2015: improving paths
Mon May 11 00:24:58 2015: done
1306140 paths extended
Mon May 11 00:28:41 2015: start degloop
Mon May 11 00:28:41 2015: creating path index
Mon May 11 00:29:08 2015: starting loop
Mon May 11 00:29:28 2015: degloop complete
Mon May 11 00:31:16 2015: unwinding three-edge plasmids
Mon May 11 00:31:21 2015: removing small components
Mon May 11 00:34:52 2015: writing a.fin files
Mon May 11 00:44:05 2015: determining candidates
Mon May 11 00:44:06 2015: determining candidates
CN fraction good = 0.45
Mon May 11 00:47:00 2015: deleting 0 gaps and adding 8 gaps to force symmetry
Mon May 11 00:48:12 2015: done making gaps, time used = 3.22 minutes
Mon May 11 00:53:16 2015: determining candidates
Mon May 11 00:53:17 2015: determining candidates
18.5 seconds using setting up final fasta
3.39 minutes using printing final fasta
assembly has 3094953 edges of mean length 1062.18
contig line N50: 3,979
scaffold line N50: 3,979
total bases in 1 kb+ scaffolds: 1,528,625,509
total bases in 10 kb+ scaffolds: 137,959,107
There are 132,012,220 reads of mean length 242.4 and mean base quality 33.0.
MPL1 = mean length of first read in pair up to first error = 156
(normal range is 175-225 for 250 base reads)
Estimated chimera rate in read pairs (including mismapping) = 0.07%.
genomic read coverage, using 1 kb+ scaffolds for genome size estimate: 20.9
run started Sun May 10 10:10:33 2015, completed Mon May 11 01:02:09 2015
peak mem usage = 184.09 GB, total time = 14.9 hours
final checksum = 652044377802948288
DiscovarDeNovo READS=“frac:0.5,sample:19::/campusdata/ndudek/fastq_to_bam/UCSF_SW019_noAdap_noDup.bam+frac:0.5,sample:18::/campusdata/ndudek/fastq_to_bam/UCSF_SW018_noAdap_noDup.bam” OUT_DIR=/campusdata/rcalef/0.5_dataDDN_150501 NUM_THREADS=39 MAX_MEM_GB=220 MEMORY_CHECK=True
Mon May 11 01:02:23 2015: done
