Indiana University's newest supercomputer, Quarry, went into
production service on August 20. The 7 Teraflop Quarry system
replaces the 2 Teraflop AVIDD system, which is being retired.
Like AVIDD, Quarry is built from Intel processors and runs Red Hat
Linux as its operating system. When the retirement of AVIDD was
discussed at the Research Technologies Round Table in January, it
became clear that many researchers preferred to continue working in
that familiar environment, rather than migrating to the more
powerful Big Red cluster. To meet the needs of these ongoing
research programs, the Quarry cluster was commissioned.
Quarry consists of 112 IBM HS21 Blade servers, each containing two
Intel Xeon 5335 quad-core processors, 8GB of memory, a 36GB locally-
attached SCSI disk for local scratch space space, and gigabit
ethernet for system interconnects. The Blade configuration provides
electrical power and cooling advantages which makes Quarry much more
cost-effective than AVIDD, while occupying much less space.
Quarry uses the Torque resource manager for batch jobs (an open
source project based on PBS) and the Moab Workload Manager to
implement scheduling optimization and fairness policies. There
are four queues: defaultq includes 64 nodes, fastq includes 4 nodes,
osg (Open Science Grid queue) includes 16 nodes, and lead (the
LEAD queue) includes 16 nodes. Jobs submitted from Quarry login
nodes are automatically sent to defaultq. The fastq is for jobs
requiring 30 minutes of walltime or less.
The user environment is generally similar to AVIDD's. The Softenv
environment management system is used to simplify application and
environment configuration. When you login to Quarry for the first
time, a file named .soft will be created, defining system
defaults, such as the Intel compilers. Additional packages may be
added, using the Nano editor; to get a list of the possibilities,
enter the command softenv.
The High Performance Applications support team can help local users
with migration issues and code optimization. Send email to
hpc@iu.edu.
For more information, refer to
http://rtinfo.uits.iu.edu/hps/research/quarry/
Indiana University Cyberinfrastructure News
-
-
With a new system at hand, and a new academic year beginning, it's a
good time to discuss setting up housekeeping on a new computer.
A critical part of making any computer system reliable is keeping it
secure. In order to protect UITS research systems, all sessions must
be encrypted, by using a protocol called Secure Shell. The program
for connecting between computers, or "terminal emulation," using
this protocol is often called SSH, as well. If you do not have SSH
installed on your personal workstation, it's available to the IU
community [anyone with an IU NetID] from IUware, at
http://iuware.iu.edu/
The Quarry login nodes can be accessed via Secure Shell clients
using the hostname quarry.uits.indiana.edu. The default shell on
Quarry is bash, which we strongly recommend.
A traditional first step in setting up a new account is to create
a file named .forward, telling the mail program where to deliver
your email. Although IU's computational systems do not accept mail
from external systems, this is still needed for mail originating on
that system. Create the file with any editor (nano is an easy one
to use). Your .forward should consist of a single line containing
your preferred email address.
A next step is to protect you from yourself. To keep from writing
over a file, use any editor to add these lines to your .bashrc
file:
alias rm 'rm -i'
alias cp 'cp -i'
alias mv 'mv -i'
You may wish to add other shortcuts, for convenience: lo for logout
is a popular one, and vi users who do not use RCS often make ci an
alias for vi.
A Primer for Working on the Quarry Linux Cluster is available at
http://rtinfo.uits.iu.edu/hpa/working_on_quarry/ -
The National Science Foundation (NSF) has awarded $1.96 million to a
team led by Geoffrey C. Fox, to create a cyberinfrastructure that
will help scientists better understand the current and future state
of polar ice sheets. The project team includes partners from IU,
Elizabeth City State University, and the University of Kansas' Center
for the Remote Sensing of Ice Sheets.
A computer grid spanning from the North to the South Pole will be
created. This "Polar Grid" will be comprised of ruggedized laptops
and clusters deployed in the field in the polar regions, plus two
large scale clusters for detailed data analysis -- a 17 Teraflops
cluster to be installed at IU, and a 5 Teraflops cluster at
Elizabeth City State University.
Faculty and student researchers will participate in field data
collection and in Polar Grid implementation of a base camp 64-core
cluster, allowing near real-time analysis of radar data by the polar
field teams. Students trained and educated on Polar Grid also
will participate in internships and enhance the entry of a diverse
workforce into important science. The clusters will be made highly
accessible through a science gateway, using Web 2.0 and portal
approaches designed to make high performance computers easier to
use.
The Polar Grid represents a dramatic change from the current method
of study, in which expeditions occur during the summer months, data
is brought back to the U.S. for analysis, and a new expedition takes
place the following year.
"The Polar Grid project will transform U.S. capabilities in ice
sheet research," said Fox. "With this technology, it will be
possible to collect, examine and analyze data -- and then use the
results of such analysis to optimize data collection strategies --
all during the course of a single expedition. This will help
scientists more quickly gain understanding about the potential
impact of rising sea levels and how they relate to global climate
change, a problem of urgent importance." -
There are several job openings available in Indianapolis, focused on
development of applications for multi-core processors.
Technical staff are expected to be computational or computer
scientists with a Ph.D. or M.S. in an appropriate scientific or
engineering discipline, able to work with other scientists/engineers
in creating applications and middleware (tools, libraries, etc.) for
advanced multi-core technologies in targeted disciplines. Experience
developing and optimizing highly scalable parallel code is
essential. Other key skills needed/desired are:
* Initiative, self-motivated, able to work successfully and
achieve objectives without frequent supervision.
* Able to excel in working in situations where not all project
elements are fully or clearly defined.
* Successful in working in distributed collaborative teams.
Demonstrated talents and success in leading teams is valued.
* Strong communication skills, both oral and written.
* Effective in engaging with customers/business partners in
establishing projects and acheiving the project objectives.
Those interested should contact researchtechnologies@iu.edu -
Wednesday, Sept. 26, 12:30-1:30 -- ICTC Room 497 & IMU Walnut Room
Research Technologies Round Table: Research Data Complex Migration
The Research Data Complex (RDC) supports data-intensive computing,
primarily Oracle database applications. The RDC staff in the High
Performance Systems group provides database hosting, administration,
and consulting services. The RDC has recently migrated to a new
platform. This presentation will discuss the benefits the new
hardware will bring and the Oracle support services available to
researchers.
--------
Sunday, October 14, 2007 -- Boston
Indiana University is offering a TeraGrid-related tutorial, "Using
IU's Big Red PowerPC Cluster and IU Storage Resources via the
TeraGrid" at BiBE 2007 (Bioinformatics and Biomedical Engineering),
The primary purpose of this tutorial is to enable TeraGrid users to
learn about the Big Red system so that they can easily use codes
already ported and optimized for that system (e.g. WRF, NAMD, MILC),
or rapidly migrate other applications to Big Red.
In addition, as massive computations commonly depend on massive data
sets as input, and produce massive data sets as output, it may be
useful to obtain a working knowledge of IU's archival data storage
system, and how to store and access files via gridftp.
Plan to attend to gain hands-on experience with Big Red and IU's
High Performance Storage System, as resources on the TeraGrid.
This tutorial will be given in conjunction with the Workshop on
Progress Toward Petascale Applications in Bioinformatics and
Computational Biology. For more information, see
http://rtinfo.uits.indiana.edu/hpc/workshops/bibe2007/info.shtml
--------
Wednesday-Friday, October 17-19 -- Bloomington
Indiana University is hosting the HPSS Users Forum, October 17-19.
HUF 2007 is the annual gathering for the HPSS community, bringing
together new and existing HPSS users from around the globe to
discuss best practices, new implementations, and future directions
and releases. Attendees will include technologists, researchers,
faculty, postdocs, and industry representatives involved in the
development, implementation, and management of HPSS software. For
more information, see
http://www.indiana.edu/~uits/huf2007/index.html -
Indiana University is offering several new types of resources
available for allocation within the TeraGrid which will create new
opportunities for US researchers - particularly those engaged in
data-centric computing and development of scientific gateways.
These new resources include: dedicated disk space to support
databases and data collections; dedicated compute nodes and disk to
support data collection services delivered via Service Oriented
Architecture protocols; dedicated computational resources in support
of Science Gateways; and a portal to rendering applications running
on a Condor Pool. These new resources add to the existing resources
offered to the TeraGrid - in particular Big Red and IU's distributed
archival storage system, which runs under the control of the High
Performance Storage System (HPSS). US researchers can now apply
for allocations of time on these IU systems via the TeraGrid as
well as other TeraGrid resources.
These new resources reflect IU's focus on support for scientific
gateways, portals, and data-intensive applications. Many researchers
provide important data and computational services via Service
Oriented Architecture protocols such as WSDL. IU can now provide a
resource for hosting such services on its new Intel-based blade
cluster called Quarry. IU is also offering nodes dedicated to
particular science gateways or portals, so that researchers can
locate such services on robust servers directly connected to the
TeraGrid network, and have dedicated resources immediately available
for use. Another new service is the IU RenderPortal, which provides
access to applications such as Maya and Blender running on a Condor
Pool, and which is specifically targeted at instructional uses of
Maya and Blender. This portal has been tested within IU and received
enthusiastic responses from students using RenderPortal in computer
science and arts classes.
New requests are being accepted by the TeraGrid for IU's Big Red
supercomputer and archival storage system. Big Red has been very
popular for molecular dynamics and weather forecasting, and supports
NAMD, Amber, CHARMM, and Gaussian. IU is offering hundreds of
TeraBytes of storage for use via the TeraGrid as well. IU's HPSS
installation is unique within the TeraGrid in that it stores, by
default, duplicate copies of data in two separate locations
(Indianapolis and Bloomington). Detailed descriptions of all IU
resources available for allocation via the TeraGrid are available at
http://rtinfo.uits.indiana.edu/cyberinfrastructure/resources.shtml
With the recent addition of a number of large new computational
resources to the TeraGrid, the number of CPU hours available for
allocations has grown beyond the total of hours requested. However,
because allocations on the TeraGrid are valuable, applications are
subject to peer review just as any other major instrument funded by
the National Science Foundation. Proposals that do not meet the
proposal guidelines and justify well the resources requested have in
the past been turned down even when available resources have gone
unallocated as a result. Researchers interested in making use of an
IU resource offered via the TeraGrid, are strongly encouraged to
contact IU for assistance in preparing their proposals. By working
with IU (or any TeraGrid partner), researchers can improve the
quality of their proposals and increase the likelihood that their
applications for resources will be approved the first time they are
submitted by assuring that all the requirements for a properly
formatted proposal are met. Contact IU by sending email to proposal
by sending email to researchtechnologies@iu.edu.
From now until 12 Octiber researchers can submit proposals for
"Medium-sized" requests, in the range of 30,001 to 500,000 TeraGrid
Standardized Units. The TeraGrid Standardized Unit was originally
developed as a measure of computation, and 1 TeraGrid SU is defined
as an amount of compute time equivalent to 1 hour of CPU time on an
Intel Itanium node from the initial TeraGrid clusters, determined by
the ratio of Linpack performance on a given compute resource to the
Linpack performance of the original TeraGrid Itanium nodes. For
example, on Big Red, 1 CPU hour is 1.782 SUs. Medium allocation of
disk storage are those using 5 to 25 TeraBytes of disk storage for a
year. Medium allocations of tape storage are in the range of 25 to
100TB for a year. Smaller, starter (development) allocations (less
than 30,000 SUs, 5TB of disk storage, or 25TB of tape storage can
be submitted at any time).
TeraGrid resources in general may be requested during the current
round of allocations. For more information, see the TeraGrid user
portal at
www.teragrid.org. -
Indiana University is leading two important events at the BIBE07
(BioInformatics and Biomedical Engineering) conference to be held
October 14-17 at the Harvard Medical School Conference Center in
Boston, MA. Dr. Craig Stewart, Associate Dean for Research
Technologies at IU, will give a tutorial lecture on Sunday, October
14, from 4:40-5:15 on two key resources available to the national
research community via the NSF-funded TeraGrid: IU's Big Red
supercomputer and IU's archival tape storage system.
Indiana University's 30.6 TFLOPS IBM e1350 cluster is among the
largest supercomputers integrated with the TeraGrid. Big Red has
proven particularly popular with users of molecular dynamics codes,
and supports NAMD, Amber, and CHARMM. Dr. Stewart will also present
information on how to store data in IU's HPSS archival storage
system via the TeraGrid. HPSS was initially developed for national
weapons labs; it is extremely robust and secure, and offers
excellent performance. IU's HPSS installation is unique within the
TeraGrid in that data are by default copied to two geographically
separated tape silos, providing extreme reliability in data storage.
Use of HPSS via the TeraGrid is relatively simple, and a short
amount of time will be devoted to instructing participants in how
to access HPSS via gridftp and the TeraGrid. The tutorial lecture
will also include information about new services being offered by
IU, including the hosting of servers that deliver data and
services through Service Oriented Architecture protocols such as
WSDL. Some of the first services hosted will be bioinformatics
services delivered by Web Services.
IU is also coordinating a workshop on PetaScale Applications in
Biology, to be held on the afternoon of October 15. Papers to be
presented in this workshop include:
R. Henschel and M. Mueller. I/O Induced scalability limits of
bioinformatics applications
M. Freindorf et al. Large-Scale QM/MM Calculations of electronic
excitations in yellow protein: toward petascale levels of protein
calculations
J. Tilson et al. MotifNework: Genome-wide domain analysis using
grid-enabled workflows
Baden et al. Toward petascale simulation of cellular microphysiology
S. Alam et al. Performance evaluation of a scalable molecular
dynamics simulation framework on a massively-parallel system.
BIBE07 promises an exciting agenda overall, and the contributions
from IU as regards computational biology and bioinformatics should
add to the benefits of attending a highly valuable conference.
For more information about this workshop, see:
http://rtinfo.uits.indiana.edu/BIBE_call_for_papers.pdf
More information about this tutorial is available at
http://rtinfo.uits.iu.edu/BIBE
More information about the conference is available at
http://www.cs.gsu.edu/BIBE07/ -
The TeraGrid user survey will be open for opinions until October 19,
2007. Our general user survey this year focuses on understanding
your practices and requirements as a TeraGrid user, to guide our
efforts to improve our capabilities and services next year and
beyond. In other words, what can we do differently, better, more of,
less of, or just like we are right now to best serve your needs?
If you have a TeraGrid allocation to use Big Red, IU's HPSS storage
system, other IU resources available through the TeraGrid, or any
resource at all via the TeraGrid, please visit the following URL
and do your part to help us serve you better:
http://www.surveymonkey.com/s.aspx?sm=HmWC5lcVfRmdCx5V0jFxPg_3d_3d
We also invite you to participate in our HPC Education survey,
available from:
http://www.surveymonkey.com/s.aspx?sm=_2f7D2YUJfkH9Dua0PP_2f30_2fQ_3d_3d -
Outages
-------
None
Planned maintenance
-------------------
The maintenance window for all systems is the first Tuesday of
each month, 7am - 7pm EDT. -
If you have questions pertaining to IU's cyberinfrastructure, or you
are encountering some difficulty, there are several ways to obtain
help.
An introduction and overview titled "Indiana University's
CyberInfrastructure: The least you need to know" has been updated
and is available at http://rc.uits.iu.edu/education_and_training/ .
The IU Knowledge Base (http://kb.iu.edu) is an excellent source of
help on how to do things.
If you have problems which the KB does not enable you to solve,
questions about system outages, or if you just have a problem and
you don't know who to contact, send email to
researchtechnologies@iu.edu.
