Contents:

Overview Information
Index (of attended session)
Tutorials
General Sessions
Invited Sessions
Topical Sessions
BOFs: DC Mgrs., Acct., Storage, Metacomputing, and UNM HPCC
Unrelated Tidbits

Overview Information http://www.hpcmo.hpc.mil/

Barbara Horner-Miller, Guy Robinson, Derek Bastille,
Gary Hagestead, Virginia Bedford, Bob Huebert, Kurt Carlson

Conference format was two half-day tutorial sessions on Monday and general and topical sessions (30 to 90 minutes each) the remaining days. The topical sessions included:

Ocean Modeling, Real-Time Computer, Computational Fluid Dynamics, Metacomputing, Atmospheric Turbulence, Shock Physics, Challenge Projects, Computational Electromagnetics, and Algorithms and Tools.

I have a number of the handouts. They will be mentioned below since in many cases scanning them will provide more information than summarizing my notes (and injecting my opinions). Please let me know if you wish to borrow any.

On Friday the ARSC representatives met with staff of the UNM High Performance Computing Center to review possible collaborative efforts as well as starting the mural project.

Index: Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

Monday
08:00 Tutorial: Building or Buying a Commodity Supercomputer
08:00 Tutorial: Kerberos for Users and Administrators
13:00 Tutorial: Security Requirements for HPCMP Applications
17:00 BoF: DC Managers

Tuesday
08:00 Conference Welcome
08:30 State of DoD HPCMP, Cray Henry
09:00 Invited Speaker: Dr. Dan Stewart
10:30 Invited Speaker: Dr. Paul Messina
11:00 Invited Challenge: Dr. Edwin Rood, Hydrodynamics
13:30 Tuesday Topical Sessions
18:00 BoF: HPCMP Corporate Initiative Information Environment

Wednesday
08:00 HPCMO Welcome, Larry Davis
08:30 Invited Speaker: Ms. Kay Howell (NCO)
09:30 Invited Challenge: Dr. Suresh Menon, Large-Eddy
10:30 Invited Challenge: Dr. Robert Peterkin, Virtual Prototyping RF Weapons
11:00 User Feedback Panel
13:00 BoF: MSAS (Storage) Working Group
13:30 Wednesday Topical Sessions
17:30 Social Activity: Los Amigos Round-up

Thursday
08:00 Thursday Topical Sessions
13:00 BoF: Metacomputing
17:00 UNM HPCC Open House

Friday
09:30 University of New Mexico HPCC

Tutorials Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

Mon. 08:00 Tutorial: Building or Buying a Commodity Supercomputer
Luke Lonergan, HPTi (High Performance Technologies, Inc)

Handout available.
This tutorial was essentially a commerecial by HPTi selling their integration service for doing LINUX clusters. It was not at all difficult to spot the commercialization so it was still relatively easy to sort out the useful pieces of information for platform comparisons. The session did provide useful information of where an "Extreme Linux" cluster is being used for HPC. In my mind the jury is still way out as to how far Linux will penetrate HPC, but it will be one of the players. There is certainly value in integration services (or vendors) since building clusters from scratch is non-trivial.

Handout available (I did not attend this, obtained extra copy of handout).

Handout available.
Provided a mapping of how Kerberos relates to the HPCMP program. Heard first warnings of some DoD PKI implementation requirements which we may be facing by 01/2001.

General Sessions Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

Tue. 08:30 State of DoD HPCMP, Cray Henry

Next Users Group Conference: Biloxi, Mississippi 18-21 June 2001.

• Out-year funding is a concern, but value of HPCMP has been demonstrated.
• Focus on S&T (Science & Technology) and T&E (Test & Evaluation).
• 50% of resources used by 6 largest projects.
• Showed MSRC comparative Expansion Factors
  (to my dismay... this is not a very good comparison metric between sites).
• Effective utilization of Origin systems is improving.
• Future acquisitions will be made on program-wide basis/interest in future.
• Working on benchmark suite.
• Future: more emphasis on shared memory and parallel vector (vs. distributed memory).
• All systems & MSRCs will have resource allocations
• Five priorities of work (see separate document on this):
  1. (U) Urgent
  2. (C) Challenge
  3. (H) High Priority Service/Agency
  4. (S) Standard Service/Agency
  5. (B) Background

Wed. 11:00 User Feedback Panel

Floor opened to users for questions, see me for details.
There are still kerberos problems (data movement issues, kerberos aware schedulers, ticket expirations). Need to improve communications between SRC's of the user services staff.

Wed. 17:30 Social Activity: Los Amigos Round-up

Dinner, a chance to chat with others, aztec dancers.

Invited Sessions Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

Tue. 09:00 Invited Speaker: Dr. Dan Stewart
Dr. Dan Stewart, Executive Director, Air Force Material Command (AFMC)

Tidbits:

• Technological supremacy of US Airforce/DoD is based on HPC (HPC is indispensable).
• Costs: HPC greatly reduces live testing
• Goals:
  1. Increase technology transfer
  2. Reduce acquisition cycle time
  3. Reduce total cost of ownership
  4. Reduce logistics response times
• Simulation based acquisitions - reduce time, resources, risks
• Application tools remains a challenge
• AFMC HPCMP usage growth likely to be more aggressive than HPCMP projections

Provided summaries of ASCI initiatives. Provided other examples about how Govt is working with industry to accelerate devlopment (such as a 1TB, 100MB/s tape).

(missed due to meeting at University of New Mexico)

Tidbits:

• She was former HPCMP Director, 96/97.
  Now coordinating 10 federal agencies (not including DoD).
• Feels top levels @ federal are working hard.
• The computing revolution has only just begun (imbeded computing coming)
• Internet savings to business: 1998:$17.6Billion, 2002:$1.25Trillion
• Cites smart refrigerator (screen for ordering)
• HPC is moving into mainstream (SMP's)
• "The geek shall inherit the earth."
• IT accounts for 1/3 US economic growth, employes 7.4M
• IT 21st Century: $366M requested for research, $265M appropriated by Congress
• FY2001 Request: $1B NIH biomed. research, $495M nat'l nanotech., $600M IT research
• Questioned why not DoD: there are many priorities in DoD

Tidbits:

• improve fuel/air mixing (simulation)
• now working with GE & Ford (commercial as well as military application)
• can only measure the large scale, simulate small scale
• chemistry is too complicated, must pre-calculate & re-use pieces... need 30x speed-up
• openmp within box, mpi between boxes
• for these codes highly parallizable, commodity based will work

Simulation of HPM (High Power Microwave)... process to deploy speed up by virtual prototyping. Using linux clusters for development (early prototyping), HPC for actual. Need better algorithms (had to reinvent some). Algorithm development needs funding, too. Still having to make compromises in physics (computation still too hard).

Topical Sessions Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

I am not summarizing much my notes on these since in many cases I cannot fully represent them. The sessions were valuable in understanding what work is underway and what contraints researchers are encountering. Also, they were rather interesting.

Tue. 13:30 Unsteady Hydrodynamics of Manuevering Submarines
Lawrence Purtell

Tue. 14:00 Towards Improvements of Forecasting Skills in Polar Ice Prediction
Wieslaw Maslowski

Tue. 14:30 Distributed Data Model & Format for HPC Applications
Jerry Clarke

Tue. 15:30 Prototype Metaqueue using PBS
James Cliburn (et.al.)

A prototype project between ASC, ERDC, and ARL using PBS. Motivation was load balancing, improving throughput, resource pooling for critical work. Used IBM SP's (partitions from each).

Lessons learned (the devil is in the details):

• version parity very important
• accounting information exchange (acid mapping, common naming, allocation, ...)
• common user environment (seamless)
• job tracking (need network qstat)
• dataset movement & pathname considerations
• job failure investigation
• kerberos (current pbs is not kerberos aware)

Conclusions:

• an inter-SRC metaqueue is feasable (demonstrated) but not practical right now
• an intra-SRC metaqueue may be valid and implementable

Tue. 16:00 Secure Remote Access to Numerical Software & Computations HW
Dorian Arnold (review of NetSolve)

Tue. 16:30 Research in Multi-Resource Aware Scheduling Algorithms
William Leinberger

Thu. 08:00 Scalability vs. Performance
Daniel Pressel

Consider: algorithm efficiency, serial efficiency, parallel efficiency, hard scalability (scale up PE's while keeping problem size fixed), and soft scalability )cale up PE's in proportion to problem size).

Thu. 08:30 Parallel & Adaptive Multi-constraint Graph Partitioning
Kirk Schloegel

See http://www.cs.umn.edu/~metis/

Thu. 09:00 CAPTools Project: Eval of Computer Aided Parallization Tools
Richard Luczak, UTK/ASC

Have handout.
Tools better and quicker than manual.
Knowledge of code to explicitely re-specify can help.
See http://www.asc.hpc.mil/PET/PTES/CAPTools/index.html

Thu. 09:30 Efficient Matrix Mult. Using Cache Conscious Data Layouts
Victor Prasanna

Alpha 21164 on chip cache ~10nsec, off-cip 26nsec, memory 253nsec. Using blocks to fit cache reduces misses, up to 20% code improvements demonstrated.

Thu. 10:30 ATR / MSTAR Challenge Project Final Evaluation
Thomas Wild

Thu. 11:00 Progress on Unsteady Aerodynamic Simulation of Guided Munitions
Jubaraj Sahu

Thu. 11:30 ARMS: Adaptively Refined Magnetohydrodynamics Solver
C. Richard Devore

Thu. 15:30 Data Assim. in High Res. Numerical Sim. of Ocean Circulation
Eric Chassignet

Thu. 16:00 Towards a Coupled Environmental Prediction System
Julie McClean and Wieslaw Maslowski

Need better resolution (more cycles). Doing regional because insufficient cycles for global.

BOFs and UNM HPCC Top, Overview, Index, Tutorials, General, Invited, Topical, BOFs & UNM, Tidbits

The BOFs were an important area for ongoing user support and contacts for what other centers are doing. I attended:

• DC Managers
• Information Environment (accounting)
• Mass Storage
• Metacomputing
• University of New Mexico HPCC (after conference)

An overview of activities from the Distributed Center perspective, including a summary from John Baird from HPCMP:

• Mod. office would like to visit each site twice annually (1 - 1.5 more realistic)
• Technology risk assessment initiative
• Promote better collaboration between DC's and MSRC's
• Multi-year planning
• Augment [HPCMP] DC Management team

Planned DC Workshops (I have notes on these if somebody needs more information):

1. Advanced Parallel Software Dev'l & Tuning, MHPCC (Maui) 8/15-18
2. Hyper & Multi-Spectrel Scene Generation, Redstone [next spring?]
3. Missle Defense Workshop, Egland [planning just started, 1/30-2/2/2001]
4. Gov't Microcircuites App. Conference, AFRL 3/5-8/2000

Doug Butler reviewed some high speed network issues.

Handout available.
What "Information Environment" means is pooling & coordinating information on users, accounts, projects, queueing structure, hardware and software resources, versions, allocations, etc.... all the details which are mandatory before metacomputing can even be considered. No DC involvement in working group previously, several "volunteers" were added (Barbara tossed in my card). Likely direction is Oracle database (TBD).

The BoF lasted two hours. Virginia and I discussed issues w/Steve Jones for two additional hours afterwards.

Tidbits:

• The MSRC's are all fielding Sun based storage solutions.
• ERDC (Steve Jones): Adding 2x Sun 6500, 9*9840, ASM (samfs), 3.2TB Disk (former maxstrat)
• ARL (Tom Kendall): Adding 2x Sun 6500, similar, 8TB Disk (clarion)
• NAVO & ASC: Both have dual Sun 10k, Veritas cluster manager
• DMF Conversion:
  • Mike Knowles (ARL): working on code for conversion
  • Adamac (NAVO?): on demand pull, will switch DMF to read-only, conversion "going to take awhile"
• Dual Sun: active/active to use bandwidth of both
• Query going to storage list: anybody doing net storage / SAN prototyping?
• Initiatives underway:
  1. NAVO: Mass Storage lead
  2. ARL: PKI lead (6/30 all public web, 12/01 smart cards)
  3. ASC: Information Environment lead
  4. ERDC: Metacomputing lead

• 40 full time developers of HPSS, geographically disbursed
• Goals: scalability, performance, security
• wanted many features and easy to manage (perpendicular goals, there are maybe too many features and too complex)
• fundamentally follows the mass storage reference model
• database scalability problems
• archival file system, not data services
• good for large number of large files (considers large .25gb+, moved 1TB file as test case)
• suffers in performance for small files
• 2 HPSS @ Sandia: classified and not, not large HPSS user (25TB for both)
• bundle data & ftp for smaller files
• tape drives: "Eagles are great"
• Sandia: write once, read never (e.g., archival)
• Looking at RAIT
• looking for 100mb/s target by mid-next year
• HPSS can as for pieces of files (not smartly, have to know data)
• DFS a better answer than NFS @ Sandia (still small scale: 150gb): DFS provides security
• Sandia has one HPSS administrator for 2 systems, but:
  • learning curve
  • tuning important
  • IBM can help
  • relationship with network group
  • aix experience required (several years)
  • dce part is highest learning curve
• Sandia has one DCE/DFS person
• 8 small Power PC's w/OC3 as tape drivers
• 2 medium Power PC's as servers
• close coordination required: HPSS, Security, Network (politics)
• HPSS User Forum end-of-July in Albuquerque

• Metacomputing is a hard problem... it is going to take awhile(!) to be robust pervasive. Note, "awhile" is registered in years... 5 to 10 (incremental in interim).
• No "basic research" on this within HPCMP, prototyping and early adoption possible.
• Lots of discussion...

They showed off their 512-node Linux cluster and their RAVE (Really Awesome Virtual Environment, e.g., CAVE++). Primary purpose of meeting with them was to forward the ARSC project of joining the mural access grid (an enhance-mbone based video-conferencing network). However, the meeting explored other areas where ARSC and UNM HPCC can collaborate for mutual benefit, such as:

• mural access grid
• flatland viz|RAVE|CAVE software development
• possibly relocating a small HPCC linux cluster @ ARSC
• cross-training resources

Note, Linux Superclusters Users Conference by UNM and IBM @ Albuquerque 09/11-15, see: www.ahpcc.unm.edu.

Unrelated Tidbits (before and after HPC-UG)

On weekend prior to conference I visited (see the links, pictures also available at my desk):

• Petrified Forest (Arizona)
  Ok, so "forest" is slightly overstated but there are many fallen trunks. This is a really neat area. The trees were buried 200ish million years ago and the result is petrification in some amazing colors. There were some anaszi ruins and petroglyphs here, too. The rocks (petrified wood) are fascinating, to avoid temptation of bringing some home ($275 fine) I bought some in the park concession (which were harvested outside the park).
• Painted Desert (Arizona)
  Effectively part of the same National Park as the Petrified Forest. Beautiful, subtle colors (not done justice on film). Did a hike into the desert which was quite enjoyable (BYOW).
• El Morro National Monument (New Mexico)
  A bunch of Indian, Spanish, American delinquents carve names and dates in stone. This is a pretty area, two mile hike up and around the bluffs which includes a hidden canyon. Nice area.
• Bandera Volcano and Ice Caves (New Mexico)
  Along edge of El Malpais: volcanic crater, lava flows and caves.
• El Malpais National Monument (New Mexico)
  The Bad Country. Lava flows and more bluffs.

During the conference there was very little extra time. I did get out running three mornings and surprisingly was not affected by the elevation (5000+ft.). The early mornings were around 65F, very comfortable running... days got into 90's.

The weekend after the conference I visited my daughter Kara who is Washington for the summer.