|
|
Beowulf Resources (with abstracts)
This page is devoted to beowulfery and cluster computing. There
in an online book on cluster engineering, talks and tutorials, and
software packages. You can either click directly to a resource page on
the navigation bar above or browse the list below, where each link is
accompanied by a short abstract.
I cherish feedback on any of these resources. To initiate a
discussion, contact me.
- Engineering a Beowulf-Style Compute
Cluster
This is a local copy of my online book on beowulf style
cluster
engineering. It is (and will likely always be) a work in progress, so
check the revision number and dates from time to time to see if new
material has been added.
- wulfware
Wulfware: A LAN/Cluster/Beowulf Monitoring Suite
Version 2.6.0
Robert G. Brown (rgb)
This is the official website for the Wulfware suite of LAN or
cluster monitoring tools. These tools all build from a single source
rpm, or alternatively one can build them from the source tarball.
This project has a mailing list here.
This is a (currently) very low traffic list devoted to bug reports,
development announcements, feature requests and little else.
View
this
page in Romanian courtesy of azoft
- xmlsysd -- a daemon that runs on the client to be monitored that
provides client statistics on demand via an xinetd or forking daemon
socket. The information is wrapped in XML tags that can be parsed or
presented on the client side in a variety of ways.
- libwulf -- a library of functions that support configuring a
cluster or LAN for monitoring (using an XML-based cluster description
file), managing connections to the cluster nodes or LAN clients
automatically, configuring the connections to return minimal information
for the quantities being monitored or displayed, and then polling the
hosts and extracting their information into a struct for further
processing by a user interface (UI) program linked to the library.
- wulflogger -- a very simple raw-tty (stdout) UI that is suitable
for extracting cluster/lan statistics from any of several useful
clusters of data. This data can be piped into a file or other
applications for post-processing, removing the burden from a programmer
of writing an automated UI for managing the connections themselves.
Alternatively, it can be used as a template for further UIs.
- wulfstat -- an ncurses-based (cooked tty) UI that presents LAN or
cluster stats in a scrollable display within e.g. an xterm window.
- wulf2html -- a perl filter that runs behind wulflogger and transforms
wulflogger output into a formatted html page that can then be viewed
from any browser. wulf2html can be started from chkconfig as a service
on a webserver or host that mounts webspace after editing
/etc/warewulf/wulf2html.sh and /etc/warewulf/wulfhosts to reflect the
cluster or LAN to be monitored. This is still a bit experimental.
- wulfware-doc -- a future latex/pdf documentation set for all
of the above, which alas doesn't exist yet. The man pages do exist, and
are installed for each package within the package.
- gwulfstat -- a future or planned piece of vaporware that
will someday come to fruition -- a nice, fully featured GUI based on
Gtk/Gnome for monitoring a cluster or LAN. Since it doesn't exist I can
be lavish with features such as a "panel of lights" display that lets
you see the state of a huge cluster at a glance on a high-res display,
settable alarms and an alarm messaging system, a remote shell interface
that lets you select blocks of nodes or clients by means of a mouse or
hostname glob and then run shell commands on all the selected clients at
one time, an integrated batch job scheduler with the ability to
implement SIMPLE policy schemes, and eating your meatloaf for you. With
catsup. Seriously, all of that and more is possible, it just needs the
work.
- wulfwebd -- a future or planned aggregator daemon. This is
a daemon that is basically a stripped wulflogger on one side -- goes and
connects to an entire cluster and manages all connections for you. It
polls the connected clients, collects their raw xmlsysd results, and
spits them out on a SINGLE port offered on the daemon side. This daemon
(will) exists only to make it possible to drill a hole through a
firewall to a single host and retrieve all the information on a cluster
or LAN inside. Or, alternatively, within multiple VPNs across IP domain
boundaries and so on if you want to do this securely. So you can sit in
Boston and monitor your departmental LAN and your research cluster all
at the same time right through firewalls in between from a single e.g.
wulfstat or gwulfstat session. This will require changes to the libwulf
API to support an aggregated connection, probably to a different default
port number or connection type via an extension of the cluster defining
xml.
Submit bug reports, etc. to
rgb at phy dot duke dot edu
- cpu_rate
This is the current snapshot of the cpu_rate
Microtimer harness and microbenchmark program. I use this
program to do quick-and-dirty benchmarks of a variety of systems
performance parameters. The timing harness itself is at least modestly
advanced -- it uses the CPU cycle counter instead of gettimeofday where
possible, for example (on all post-Pentium Intels, post-Athlon AMDs).
There are tests for memory performance, "arithmetic" performance,
transcendental (savage benchmark) performance. In many cases
(specifically those involving vectors and memory) the size of the
vectors can be varied and the effects of cache size and hierarchical
memory explored.
This version of the code supports the relatively simple and
"object-like" addition of code fragments to be timed. There is a
fairly long list of operations and fragments it already times (enter
"cpu_rate -l" to get a current list). It is in beta mode and
may have bugs. There may well be better timers out there (e.g.
lmbench) only perhaps. not so easy to use.
Because of the nature of this program I recommend that you grab the
tarball and work with it rather than simply build/install either the
source rpm or the provided (RH 9) binary rpm. The latter "should work",
but part of the fun of a benchmark/timer is playing with build options
and seeing how it works. There are also some (probably broken in the
current release) scripts in the source directory that in previous
versions scanned e.g. across the -s size option for the vector
benchmarks; these scripts, with a bit of repair, would make it pretty
easy to generate a graphable file of e.g. stream or memory read/write
with and without random shuffling.
I hope that you find cpu_rate useful.
- A Model for Cluster Computing at
Duke
This is a white paper on how Duke might economically set up a
"public" or "shared" cluster compute environment here without offending
the Gods (the granting agencies that might participate) and in a
maximally inexpensive and cost-beneficial manner.
Note Well: This is a Request for Comments only;
please return comments and corrections to rgb for encorporation into what I
hope to be a dynamic document associated with a project, rather than a
static prescription that may, in fact, be heavily flawed.
- An Introduction to the Beowulf
Design
This is a reasonably complete (some would say overcomplete:-)
rendition of a standard introduction to beowulfery, developed from
previous talks on the same subject I've given. As is frequently the
case with talks, the pdf and ps images are likely to be "prettier" than
the online version -- I haven't quite gotten a perfect
latex2html-compatible talk style set up yet.
- Zen and the Art of Beowulf
Clusters
Zen and the Art of Beowulf Clusters
by Robert G. Brown (rgb)
This is a 2006 update of my standard beowulf intro talk, with a Zen
twist. It was last presented to the Science Club of Durham Technical
Institute on 11/7/06, and is being posted so they (and you, if you are
reading this) can access it to look over the content and linked
resources.
Good luck! Feel free to contact me with e.g. bug reports or
problems.
- The Critical Scaling of the Helicity
Modulus of the O(3) classical Heisenberg Ferromagnet
A presentation to the Duke HPC community on my ongoing research into
the critical scaling of the helicity modulus, via Monte Carlo
computations carried out on a cluster.
- Why I code in
C
I can no longer remember where or when I found this while mousing
around on the internet, but it is clearly something anyone seriously
considering the choice between C and C++ should read before investing
their future in one or the other.
- Charles L. Hedrick's Introduction to
the Administration of an Internet Based Local Network (Copyright
1988 Charles L. Hedrick, see link for original copyright notice).
-
Truly a classic, this work (and it's companion, next) were my
original "online guides" in TCP/IP networking (and networking in
general). You'd think they'd be dated, sixteen years later, but of
course TCP/IP itself hasn't changed, ethernet hasn't fundamentally
changed -- amazingly, these are still some of the best simple
introductions available anywhere on or off the net. Thank you, Charles
L. Hedrick, whereever you might be!
- Charles L. Hedrick's Introduction to
the Internet Protocols (Copyright 1987 Charles L. Hedrick, see link
for original copyright notice).
-
The companion work to the local area networking guide above,
this work is just incredible. In addition to beautiful and tediously
typed tty renditions of TCP, IP and ethernet packets showing most of the
relevant details of their headers, this work reviews ports, RFC's, and
more.
|
to help spread the word so others can find it as well.
Note, Robert G. Brown is generally either rgb or rgbatduke on many external
sites crosslinked here.