From SciNet Users Documentation
Revision as of 16:11, 30 July 2018 by Northrup (talk | contribs) (Monitoring jobs in the queue)
Jump to: navigation, search


The Basics

Whom do I contact for support?



E-mail <support@scinet.utoronto.ca>

In your email, please include the following information:

  • your username on SciNet
  • any relevant error messages
  • the commands you typed before the errors occurred
  • the path to your code (if applicable)
  • the location of the job scripts (if applicable)
  • the directory from which it was submitted (if applicable)



What does code scaling mean?


Please see A Performance Primer

What do you mean by throughput?


Please see A Performance Primer.

Here is a simple example:


I changed my .bashrc/.bash_profile and now nothing works




The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" #!/bin/tcsh at the top.



Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).





The SSH for Windows users programs we recommend are:





Testing Your Key

$ ssh USERNAME@login.scinet.utoronto.ca

The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
this message.
RSA host key for login.scinet.utoronto.ca 
changed and you have requested

restarted with the new passphrase ssh key.

$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.







 Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  ForwardX11 yes
  ForwardX11Trusted yes
  ForwardX11Timeout 596h








$ module load extras
$ diskUsage








How can I reset the password for my Compute Canada account?


You can reset your password for your Compute Canada account here:










   chmod 700 ~/.ssh

And to be sure, also do

   chmod 600 ~/.ssh/id_rsa ~/authorized_keys

ERROR:102: Tcl command execution failed? when loading modules


$ module purge
$ module load python
$ gpc-f103n084-$ module load gcc python







Compiling your Code

How do I link against the Intel Math Kernel Library?





Testing your Code





Submitting your jobs

How do I charge jobs to my RAC allocation?


How can I automatically resubmit a job?

Running your jobs

My job can't write to /home

Can I can use hybrid codes consisting of MPI and openMP on the GPC?


Monitoring jobs in the queue

I couldn't find the .o output file in the .pbs_spool directory as I used to

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged, but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in /var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

My GPC job died, telling me `Copy Stageout Files Failed'


When a job runs on GPC, the script's standard output and error are redirected to $PBS_JOBID.gpc-sched.OU and $PBS_JOBID.gpc-sched.ER in /var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default $PBS_JOBNAME.o$PBS_JOBID and$PBS_JOBNAME.e$PBS_JOBID. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:

An error has occurred processing your job, see below.
request to copy stageout files failed on node

it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to make sure that your .bashrc does not produce any output, as the output-stageout is performed by bash and further output can cause this to fail. But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process (e.g. ran out of memory very quickly) and those files just never got copied.

Write to <support@scinet.utoronto.ca> if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.

IB Memory Errors, eg reg_mr Cannot allocate memory

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or network configurations may exhibit this problem at higher or lower MPI task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288

With Intel MPI, you should be able to do

module load intelmpi/
mpirun -genv I_MPI_FABRICS=shm:ofa  -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode

to the same end.

For more information see GPC MPI Versions.

My compute job fails, saying libpng12.so.0: cannot open shared object file or libjpeg.so.62: cannot open shared object file


To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.


Data on SciNet disks

How do I find out my disk usage?




More information about these filesystems is available at the Data_Management.

How do I transfer data to/from SciNet?








Please see Scratch Disk Purging Policy



Please see File/Ownership Management

Can I transfer files between BGQ and HPSS?


  • transfer from BGQ to Niagara SCRATCH
  • then from Niagara SCRATCH to HPSS

Keep 'em Coming!

Next question, please
