User Tools

Site Tools


archive:computer_resources:campusrocks

This is an old revision of the document!


A PCRE internal error occured. This might be caused by a faulty plugin

====== campusrocks ====== ===== Address ===== '' Name: campusrocks.cse.ucsc.edu\\ Address: 128.114.48.114 '' ===== Hardware ===== [[Hardware|Machine-specific capabilities]] ^Machine ^GB RAM ^CPUs ^Cores ^MHz ^Model ^ |campusrocks.local(head node?)|30 |4 |1 |2600 |Dual-Core AMD Opteron(tm) Processor 2218 HE| |campusrocks-0-2(through 30).local |15 |4 |2 |2664 | Intel(R) Xeon(R) CPU 5150 | |campusrocks-0-(31,32).local |15 |8 |2 | 3192 | Intel(R) Xeon(TM) CPU| |campusrocks-1-0(through 31).local |15 |2 |1 |2592 |Dual-Core AMD Opteron(tm) Processor 2218 HE| |campusrocks-2-(0,1).local |192 | 48| | | | campusrocks-0-6.local exceptionally has 30 GB RAM ===== Account/Documentation ===== * [[Creating an account|Create a new account]] * [[http://campusrocks.soe.ucsc.edu/roll-documentation/sge/5.2/|Documentation]] (works only on campus or when connected via [[http://vpn.soe.ucsc.edu |VPN]].) * [[http://campusrocks.soe.ucsc.edu/ganglia/|Status page]] * [[Installed Software|Software]] ===== Stats ===== * All user accounts are created under ''/campusdata'' * 3.4TB available * OS: GNU/Linux 2.6.18-128.1.14.el5xen ===== Setting Environmental Variables ===== If you would like to change your shell to bash, make a file called .profile which you will put all of your environmental settings into, and add this to the beginning: if [ "$SHELL" = "/bin/sh" ] then SHELL=/bin/bash export SHELL fi If you would like to add our classes bin directory (or any other directory for that matter) then add this to your .profile: PATH=/campusdata/BME235/bin:$PATH export PATH To use the gcc-4.5 family of compilers stored in our bin directory then you can specify the LD_LIBRARY_PATH environmental variable: LD_LIBRARY_PATH=/campusdata/BME235/lib:/campusdata/BME235/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH To make your default folder creation settings group readable and writable then add this line to your .profile: umask 007 ===== Sungrid Job scheduler ===== Campusrocks uses sungrid for scheduling jobs. The techstaff request that jobs only be run on the compute nodes, not the head node, as jobs run there slow down the entire cluster. They threaten to kill jobs found running on the head node. There is [[http://www.rocksclusters.org/rocks-documentation/4.1/|online documentation]] of how to use sungrid to submit jobs on the cluster. The main commands are * qsub: Submits a job (create a shell script, then run "qsub -cwd shellscript") * qdel: Delete a job * qlogin: Interactive login * qstat: See the status of jobs in the queue * qmon: Graphical User Interface (GUI) The web site is locked so that it can only be accessed oncampus. [[http://web.njit.edu/topics/HPC/basement/sge/SGE.html|SunGrid Quick Start]] Useful submit-script settings for SGE: #$ -N some_name_here\\ Apparently, you must have a single word after -N\\ This gives a nicer name to your script when you run qstat.\\ #$ -V\\ Your jobs use **your** environment variables. #$ -l mem_free=15g\\ Your job goes to a node with at least 15g of free memory. There is probably something like '-l mem_avail' that works if you would like to schedule your job on a node with a certain amount of max memory available, even if some of it is currently in use. #$ -pe mpi 4\\ Your job goes to a set of nodes with a combined total of a least 4 cores available. #$ -cwd\\ Your job will be launched at the directory where the qsub command was given. Useful for setting predictable bash commands. Though not recommended for portability as other users of the script would need to also know where to launch the qsub command. #$ -q [Name of specific server queue]\\ Used to specify which set of servers you would like the sungrid scheduler to schedule/run your jobs in. Names on campus rocks include 'all.q' and 'small.q' Example qsub commands: * qsub -N skewer_adapter_removal -cwd -V -pe mpi 20 -l mem_free=50g /campusdata/BME235/S15_assemblies/SOAPdenovo2/adapterRemovalTask/skewer_run/run_skewer_adapter_removal.sh Useful commands for viewing currently run jobs: #$ qhost\\ Displays the compute nodes available on the cluster including hardware statistics. #$ qstat\\ Displays the status of the jobs submitted to qsub. Primary job status codes: *'r' status indicates that a job is running. *'qw' status indicates that a job is waiting to be scheduled/run on a machine.

Discussion

, 2010/04/20 17:57

Got the clock-skew fixed with request to SOE ITS support. Compilers were complaining because the clocks of the head node were about 260 seconds behind the file storage server, which messes with makefile builds.

, 2010/04/16 01:31

Until the admins fix our primary groups, here' a work-around:
Using the sg command can set your group to bme235
(see newgrp for interactive shell with group change)
example:
[gbarber@campusrocks run001]$ cat run.csh
#!/bin/tcsh
pwd
sg bme235 -c ./go.csh

[gbarber@campusrocks run001]$ cat go.csh
#!/bin/tcsh
sleep 5
echo “Woo, ha!” > go.out

qsub -cwd run.csh
qstat to see if it is still running
it will produce output files
run.csh.e2667 this is the output of stderr
run.csh.o2667 this is the output of stdout

Even though security people don't love it, I went ahead and put ”.” on my path.

, 2010/04/16 23:17

Yay! The admins have given us only one group “bme235” now, and therefore it is our primary group. This will save some silly workarounds and hassle. Great.

, 2010/04/16 01:26

Seems like umask 002 would be good for our group.

I sent in a request to change the primary group to bme235 so we don't have to spend a lot of time finding files and setting them to the right group.

If you want to run a job from some subdirectory, be sure to do it like this:

qsub -cwd script.csh

The -cwd is so useful, I am surprised it's not the default.

More useful than putting /campusdata/BME235 in your path is putting /campusdata/BME235/bin

, 2010/04/11 16:41

There should be a section or page describing the hardware ( extracted from /proc/meminfo and /proc/cpuinfo ) for each machine. I think that there are 2 or 3 types of nodes. Each type should be described thoroughly, and then the cluster nodes of that type should be listed.

Incidentally, should some of the sections of the current page be lifted up to the “computer_resources” page? (For example, the directory structure is really independent of the hardware and jobs scheduler.)

, 2010/04/16 17:58

done

, 2010/04/09 19:41

Has anyone figured out how to change their shell?

$ chsh chsh: can only change local entries; use ypchsh instead.

$ ypchsh ypchsh: can't get local yp domain: Local domain name not set

, 2010/04/10 17:05

I made a file called .profile and put this in it and now it starts up with bash which is what I like. Maybe if you make a .profile and then put “csh” or “tcsh” or whatever at the top then it will execute those shells for you whenever you log in?

# .bashrc

# Source global definitions if [ -f /etc/bashrc ]; then

      . /etc/bashrc

fi

# User specific aliases and functions alias u=pushd alias o=popd alias d=dirs

umask 007

PATH=/campusdata/BME235:$PATH export PATH

You could leave a comment if you were logged in.
archive/computer_resources/campusrocks.1429389500.txt.gz · Last modified: 2015/04/18 20:38 by charles