|
|
General Interest (Miscellaneous) Links
Various links and publications of Robert G. Brown that fit under
none of the other categories. All of these materials are published
under a modified Open Publication License that permits unlimited free
noncommercial and personal use. The materials (books, presentations, or
otherwise) may not be published in any form or media that is sold for
profit. The details of the license can be viewed here and in each available version viewed
below. Stats for this page can be viewed here.
Commercial publishers interested in producing an actual book (or
other media form) of the material below are encouraged to contact the author.
Contents
Back to top
- Flashcard Program
-
Gflashcard: A Personal Learning Tool
by Robert G. Brown (rgb)
gflashcard Version 0.7.1
Contents
Descripction
gflashcard is a program based on GTK and XML for presenting
simple flashcards to students in a standard terminal (e.g. xterm)
window. Its license (GPL 2b) can be viewed at the bottom of this page.
Its current features include:
- Arithmetic flashcards (ones that take a numerical answer).
- Spelling flashcards (it "reads each word out loud" and then checks
the spelling. It can handle alternative spellings and homonyms (words
with the same sound but different spellings and meanings).
- True/False flashcards.
- Multiple Choice flashcards. The multiple choice engine is
powerful. One can define up to 1024 possible answers. Multiple choices
can be selected to be counted "correct"; the program can manage
combinations like "A and/or B" or "A and B" and hence is suitable for
type K questions. Up to ten answers and foolers combined can be
presented per question. The answers are always accompanied by randomly
selected foolers and the choices are shuffled before presentation, so
that choice "A" this time may choice "D" the next time you see it.
- Foreign Language flashcards are an alternative application
of the spelling or multiple choice flashcards. For example, an English
word can be read and a Spanish translation required, or vice versa.
- All flashcards can be accompanied by an easily recorded
audio snippet. In the case of spelling or foreign language
flashcards, an audio presentation is likely the only presentation schema
used. In all cases, though, the audio feature means that
flashcards that are suitable for the sight-impaired can be
prepared. A future version of the program will provide audio feedback
on each flashcard as well.
- Flashcard sets can be presented in random order (the
default, in fact) or list order, where the presentation order is
determined by the author.
- Flashcard sets can be mixed type. Spelling, arithmetic,
multiple choice, foreign language, true/false -- a whole day's worth of
material can easily be reviewed in a mixture less likely to be boring to
small children. In fact, "fun" trivia type questions can be mixed in
with academic material.
- The program automatically scores each session.
- The program automatically times each session.
- The program has a practice mode where it loops
indefinitely, presenting and representing a shuffled list of the same
problems.
- In practice mode, a user can be required to repeat each
question that they miss one or more times, to help "drill" them briefly
on their mistakes.
- The program has a test mode where it runs for a
specified amount of time and scores all questions that a student
answers in that much time.
- In all modes, each flashcard can include an explanation
that appears when a wrong answer is entered. This can be a powerful
teaching tool, as a student isn't just informed that their answer is
wrong, they are presented with both the right answer and why the
right answer is right, or why the wrong answers were wrong.
- All problems are defined by a simple and straightforward XML
tagging not unlike the tagging that produces this web page. Most
problem types can be created by any text editor. A (still fairly
crude, but functional) scripted recording tool is provided to automate
the generation of audio/spelling clips for inclusion in flashcard sets.
More advanced tools are under development (developer participation
welcome).
- The XML-based format of the flashcards is both open and
easy to read, write, and parse in nearly any programming
environment. It furthermore means that with an associated DTD a single
set of flashcards can be presented by many tools in many ways.
For example, a single set of flashcards might be:
- presented in an ncurses/tty flashcard engine like "flashcard",
which is also included on this site.
- presented in a GUI flashcard engine like "gflashcard".
- presented in a web-based flashcard engine using java,
perl-cgi, php (not yet written)
- transformed into simply formatted text with a suitable DTD.
- transformed into text formatted for printing onto card stock
to make real flashcards, again via DTD and tools such as jade.
- similarly transformed into latex-formatted snippets that could
then be included in actual textbooks, quizzes, problem assignments.
flashcard's XML offers the possibility of their one day existing a large
database of open content from which many people can draw problems for
either self-guided or directed study.
- The flashcard programs' source itself and the XML that specifies
flashcards are open source and open standards. This means
that anybody can take the sources and write their own derived flashcard
presentation engine, as long as that engine in turn remains open source.
It also means that anybody can write non-derived proprietary
engines for presentation that can instantly present "any" flashcards
built according to the standard. Flashcard sets themselves can in fact
be copyright protected and sold for money. This combination of open
presentation engine and open language and protection of the rights of
problem set authors should encourage the development of content, both
proprietary (for sale) and free.
- Although gflashcard (the GTK presentation engine) and flashcard
(the ncurses/tty presentation engine) are fully functional
already and reasonably bug free, it is an ongoing project.
Using XML as the basis of flashcards means that they will be at least
moderately extensible -- future evolutions should contain support
for graphical presentations, mathematical formulae, foreign character
sets, GUI based presentation and authoring tools, web based presentation
and authoring tools, problem databases and more. Using new features
based on open source tools and standards is straightforward; XML means
that each addition shouldn't break existing flashcard sets, only
permit more powerful content to be added to them (as XML parsers will
generally ignore tags they do not recognize while parsing the ones that
they do).
- Open Source Developers are welcome to participate.
Content development is also openly solicited. My most precious
commodity is time -- to facilitate the rapid development of this
powerful tool requires time and/or money. In the documents directory
there should be a tarball of free flashcards covering a number of topics
in their own directories -- many lamentably empty. Interested parties
should certainly consider writing and contributing content back to this
tarball.
gflashcard requires libxml2 and Gtk-2.
flashcard requires libxml2 and ncurses.
Back to top
About flashcard
flashcard is a program I wrote to help my second grader drill basic
spelling and arithmetic, my seventh grader drill Spanish and vocubulary,
and more advanced students get immediate feedback on "conceptual" or
memory building multiple choice questions.
flashcard is not a game. No imaginary space aliens are
harmed as one learns vocabulary or arithmetic. It is a very,
very efficient way to get the practice and immediate feedback necessary
to memorize a large number of dry, boring factoids. It never gets
tired, impatient, angry, or has to cook dinner. It doesn't require lots
of little cutout cards or pieces of paper to get lost. Suitably
installed on a local area network of student-accessible computers, it
can be used as a memorization aid for nearly any sort of factoid and as
a relatively weak "deep" learning tool.
Based on my own personal experience so far with it, young kids
initially find it at least intriguing (doing flashcards on the computer
is kind of neat), then boring and to be resisted, and finally just
another daily homework chore to be done as quickly as possible. Older
kids progress through these stages more rapidly as they can see that
using flashcard to e.g. learn their vocabulary words beats the hell out
of trying to do it by just reviewing their handout list or writing each
word three times. By the time any student is in the latter stage, the
program is generally "working".
I've been quite pleased with the results. My second (now third)
grader went from getting an average of 60% or worse on timed arithmetic
tests to getting consistent high-90%s (and occasional 100%s!) in about
one month of three or four time a week drill.
His spelling is improving as well, but more slowly as he learned to
spell phonetically and is a bit stubborn (literally) about the
non-phonetic spellings and multiple possible spellings that
abownd in English. He initially exhibited a similar stubbornness
about memorizing arithmetic versus using his fingers to compute the
answer, but repetition and time pressure eventually take their toll and
he ended up memorizing the right answers in spite of himself.
One thing that I'm doing that may or may not be the right thing to
do (and may be responsible for his initially relatively slow progress)
is having him drill on all the second grade core vocabulary words
at once, instead of doing a small list every week in phase with his
class. In the short run this leaves him behind, as he sees his current
vocabulary words only occasionally and mixed in with all many others.
However, I expect that very soon, with consistent drill, he will
suddenly know the entire list (this is the way it worked with
arithmetic) and be not only ahead but done with learning second grade
vocabulary half way through the year.
Back to top
Screenshots
New! gflashcard GUI!
|
|
The material displayed is the included "demo" tutorial that both
demonstrates gflashcards abilities and serves as a template for building
your own content.
|
Multiple Choice Flashcards
|
|
The material is chemistry questions for an academic bowl
team.
|
Arithmetic Flashcards
|
|
Times tables for third graders.
|
Spellilng Flashcards
|
|
Although a screenshot cannot show it, the words are read out
loud by the program and then typed in by the student.
|
Back to top
Practice Strategies
I'd suggest the following strategies for using flashcard for
yourself or with your kids:
- Use practice mode a lot more often than test mode. Practice
mode (the default) actively drills the kids with no external time
pressure. Each time they make a mistake they have to enter the correct
answer (which is then presented to them) at least one time. This is
just annoying enough to make them try, a little bit, to remember, and
repetition aids memorization anyway. The number of "reps" can be varied
in a flashcard file with the count tag.
- Don't get angry with your kids for poor performance either
in practice or in test mode. gflashcard will work over time -- believe
it and just let it work. My kids beat themselves up over
mistakes so much that I've had to tell them to relax and let the program
teach them when they make mistakes. Keep short-run expectations, and
pressure, low.
- Do insist on a consistent, regular schedule of using
flashcard to "do their flashcards" in all subjects you're using them
for. For younger kids, every day is ideal, every other day acceptable,
every week probably not enough. gflashcard will not work if the students
don't use it regularly and get used to it and start working with it and
not against it (see notes above). We try for every day and get maybe
every other day. Use it regularly until mastery is achieved, and then
occasionally to reinforce it and make it permanent.
- In practice mode, you can specify the number of problems or just
run it "forever" (until they q(uit)). When younger kids first start
using it, 30 or 40 arithmetic problems or 20 vocabulary words is "a lot"
and they won't be very good at entering answers. Start them easy until
they get to where they can read a problem and enter an answer in less
than 10 seconds. Then you should be able to crank up to 100 arithmetic
problems and 50 vocabulary words, which should take them about fifteen
minutes to run less as they get better. Much less.
- There is no harm at all in offering performance or participation
rewards (I mean "bribes"), especially to younger kids. Tell your child
that when they first get (set your expectation here) you'll buy them a
toy or a treat, take them to a movie, do something they enjoy with them.
You're going to make them use the program whether or not they like it
(and of course they "won't like it") -- at least give them something to
look forward to when they do.
- Remember that the computer is "stupid" (or if you prefer, my
grading program is stupid). If you type an extra space after a word it
will mark the spelling wrong. It probably can't cope with 1.0 as an
integer answer or 1 as a floating point answer. Typing on a computer is
also error prone, and your kids are probably learning to type at the
same time they are drilling the words! I make a few mistakes per
100 doing bone-simple arithmetic in a hurry. Take all this into account
when assessing performance; a score in the high 90%s may well really be
"perfect" from the point of view of the child's grasp of the material
being drilled. Try the program yourself for a few hundred problems
(presuming that you know simple arithmetic perfectly:-) and you'll see
what I mean.
- Feel free to contact me at rgb@phy.duke.edu with suggestions,
comments, remarks, bug reports. Note well that I'm planning to add
various features, especially "explanations", multiple choice problems,
and "instructions" in future releases as I have time to cut the code.
Check this site from time to time for the current revision
number/snapshot.
- I may or may not be willing to author flashcard files to add to the
package that your child needs (depends on my time and whether my own
kids can use them). I will gratefully accept and add flashcard files
and supporting audio directories authored by others to add to the
package. It would be nice to have ALL the vocabulary words for K-12,
listed by year, spoken in a pleasant voice with a usage sentence added
to differentiate words that are homonyms, already in the package,
wouldn't it? I don't plan to sell it, and it will be GPL'd forever, but
if I ever DO sell some packaging of it I probably won't pay you for any
contributions so made. Sorry.
- On the same note, I'd cheerfully accept and add to the GPL package
anything like bug patches, additions to the program itself (probably
should check with me before writing them), extensions to or corrections
of the flashcard xmlish tagset, webware or Gtk or "other" versions of
the basic program. Given the open nature of the xmlish flashcard files,
it should be pretty easy to write a "flashcard program" that displays
them and does the flashcard rituals in many venues, including just plain
printing them out formatted for being cut up into, well, flashcards!
Back to top
Caveat, Warning, Read This!
As is the case for all drilling/memorization tools, gflashcard can be
used for great evil as well as great good. As a professional teacher, I
am constantly made aware of the difference, and you should be to if you
are going to use this program to "teach", either in the classroom, as a
parent, or (to teach yourself) as a student.
There is a significant difference between memorization of factoids
and true learning. To speak metaphorically, factoids are the many tiny
rocks which are cemented together by experience and conceptualization
into the edifice of our full understanding. One should never confuse
successful memorization of a large body of factoids with real
comprehension.
It is entirely possible to comprehend things deeply without
memorizing lots of factoids. Mathematics is not arithmetic, although
arithmetic is a useful skill that underlies some mathematics. History
is not a bunch of events and their associated dates. Language is not a
collection of words. Science is not scientific data. The abstract rule
is not the many concrete realizations of the rule. One could embark on
a long discussion of semantics and epistemology, semiotics and
psychology -- and indeed I'm working slowly on a book on these subjects
-- but not here. The main point is to recognize that memorization can
be a soul-sucking process for a young mind (or an older one!) when
unsupported by any sort of reason.
For many of these subjects, of course, memorizing factoids is one
essential step in beginning to comprehend the subject. It is
difficult to understand American History without knowing when the
American Revolution occurred and whether it occurred before or after
(say) the American Civil War. It is difficult to read and write clearly
and effectively if one's collection of vocabulary factoids is
inadequate. For that reason I think flashcard can be a useful component
of teaching and learning, but it does not teach anything like
real comprehension of a subject, only its associated and foundational
factoids, and its only real virtue here is its efficiency -- by
drilling those factoids with a tool, one can quickly build up a
base of factual knowledge sufficient to be a foundation for deeper
learning.
I would therefore recommend that this tool be used ONLY as a factoid
memorization tool, and NOT as a classroom "testing" tool or "teaching"
tool, although it does have a timed test mode and other things that
might be construed or abused into a classroom role. Don't expect
flashcards to be more than they are or do more than they can do.
Back to top
- DieHarder Program
Dieharder: A Random Number Test Suite
Version 3.31.1
Robert G. Brown (rgb)
Dirk Eddelbuettel
David Bauer
Welcome to the dieharder distribution website.
Version 3.29.4beta is the current snapshot. Some of the documentation
below may not quite be caught up to it, but it should be close.
Dieharder is a random number generator (rng) testing suite.
It is intended to test generators, not files of possibly
random numbers as the latter is a fallacious view of what it means
to be random. Is the number 7 random? If it is generated by a random
process, it might be. If it is made up to serve the purpose of some
argument (like this one) it is not. Perfect random number generators
produce "unlikely" sequences of random numbers -- at exactly the right
average rate. Testing a rng is therefore quite subtle.
dieharder is a tool designed to permit one to push a weak generator
to unambiguous failure (at the e.g. 0.0001% level), not leave one in the
"limbo" of 1% or 5% maybe-failure. It also contains many tests and is
extensible so that eventually it will contain many more tests than it
already does.
If you are using dieharder for testing rngs either in one of its
prebuilt versions (rpm or apt) or built from source (which gives you the
ability to e.g. add more tests or integrate your rng directly with
dieharder for ease of use) you may want to join either or both of the
dieharder-announce
or the
dieharder-devel
mailing lists here. The former should be very low traffic -- basically
announcing when a snapshot makes it through development to where I'm
proud of it. The latter will be a bit more active, and is a good place
to post bug reports, patches, suggestions, fixes, complaints and
generally participate in the development process.
About Dieharder
At the suggestion of Linas Vepstas on the Gnu Scientific Library
(GSL) list this GPL'd suite of random number tests will be named
"Dieharder". Using a movie sequel pun for the name is a double tribute
to George Marsaglia, whose "Diehard battery of
tests" of random number generators has enjoyed years of enduring
usefulness as a test suite.
The dieharder suite is more than just the diehard tests cleaned up
and given a pretty GPL'd source face in native C. Tests from the Statistical Test Suite (STS)
developed by the National Institute for Standards and Technology (NIST)
are being incorporated, as are new tests developed by rgb. Where
possible or appropriate, all tests that can be parameterized
("cranked up") to where failure, at least, is unambiguous are so
parameterized and controllable from the command line.
A further design goal is to provide some indication of why a
generator fails a test, where such information can be extracted during
the test process and placed in usable form. For example, the
bit-distribution tests should (eventually) be able to display the actual
histogram for the different bit ntuplets.
Dieharder is by design extensible. It is intended to be the "Swiss
army knife of random number test suites", or if you prefer, "the last
suite you'll ever ware" for testing random numbers.
- TechExpo 2011 Talk
(Duke). A short talk given at a Duke's Tech Expo in 2011 as an
overview of random number generator testing. Good for beginners.
- Good
Practice in (Pseudo) Random Number Generation for
Bioinformatics Applications by David Jones, UCL Bioinformatics Group
(E-mail: d dot jones@cs dot ucl dot ac dot uk). A really excellent
"must read" guideline for anyone thinking of using random number
generators in an actual application. My own advice differs only in that
I endorse using (well tested) Gnu Scientific Library random number
generators as they are generally portable and open source, hence well
tested. Several of Jones' implementation of Marsaglia's KISS-family rngs
have been added to dieharder and will shortly be added to the GSL under
the GPL for general use.
Dieharder can be freely downloaded from the Dieharder
download site. On this page there should be a long list of previous
versions of dieharder, and it should tell you what is the current
snapshot. The version numbers have the following specific
meaning which is a bit different than usual:
- First number (major). Bumped only when major goals in the design
roadmap are reached (for example, finishing all the diehard tests).
Version 1.x.x, for example, means that ALL of diehard (and more) is now
incorporated in the program. Version 2.x.x means that the tests
themselves have been split off into the libdieharder library, so that
they can be linked into scripting languages such as R, new UIs, or user
code. 3.x.x would be expected to indicate that the entire STS suite is
incorporated, and so on.
- Second number (first minor). This number indicates the number of
tests currently supported. When it bumps, it means new tests have been
added from e.g. STS, Knuth, Marsaglia and Tsang, rgb, or elsewhere.
- Third number (second minor). This number is bumped when
significant features are added or altered. Bug fixes bump this number,
usually after a few bumps of the release number for testing snapshots.
This number and the release are reset to 0 when the major is bumped or a
new test is added to maintain the strictly increasing numerical value on
which e.g. yum upgrades rely.
The single-tree dieharder sources (.tgz and .src.rpm) files can be
downloaded from this directory. In addition, binary rpm's built on top
of Fedora Core whatever (for either i386 or both of x86_64) may be
present. Be warned: the GSL is a build requirement. The current
packaging builds both the library and the dieharder UI from a single
source rpm, or from running "make" in the toplevel directory of the
source tarball. With a bit of effort (making a private rpm building
tree), "make rpm" should work for you as well in this toplevel
directory.
This project is under very active development. Considerable effort
is being expended so that the suite will "run out of the box" to produce
a reasonably understandable report for any given random number generator
it supports via the "-a" flag, in addition to the ability to
considerably vary most specific tests as applied to the generator. A
brief synopsis of command options to get you started is presented below.
In general, though, documentation (including this page, the man page,
and built-in documentation) may lag the bleeding edge snapshot by a few
days or more.
An rpm installation note from Court Shrock:
I was reading about your work on dieharder. First, some info
about getting dieharder working in Gentoo:
cd ~
emerge rpm gsl
wget
http://www.phy.duke.edu/~rgb/General/dieharder/dieharder-0.6.11-1.i386.rpm
rpm -i --nodeps dieharder-0.6.11-1.i386.rpm
Rebuilding from tarball source should always work as well, and if you
are planning to play a lot with the tool may be a desireable way to
proceed as there are some documentation goodies in the ./doc
subdirectory and the ./manual subdirectory of the source tarball (such
as the original diehard test descriptions and the STS white paper).
George Marsaglia retired from FSU in 1996. For a brief time diehard
appeared to have finally disappeared from FSU webspace, but what had
really happened is google's favorite path to it had disappeared when his
personal home directory was removed. Diehard is still there, at the URL
http://www.stat.fsu.edu/pub/diehard
as well as at a Hong Kong website. The source code of diehard itself is
(of course) Copyright George Marsaglia but Marsaglia did not incorporate
an explicit license into his code which muddles the issue of how
and when it can be distributed, freely or otherwise. Existing diehard
sources are not directly incorporated into dieharder in source
form for that reason, to keep authorship and GPL licensing issues
clear.
Note that the same is not true about data. Several of the diehard
tests require that one use precomputed numbers as e.g. target mean,
sigma for some test statistic. Obviously in these cases we use the same
numbers as diehard so we get the same, or comparable, results. These
numbers were all developed with support from Federal grants and have all
been published in the literature, though, and should therefore be in the
public domain as far as reuse in a program is concerned.
Note also that most of the diehard tests are modified in
dieharder, usually in a way that should improve them. There are three
improvements that were basically always made if possible.
- The number of test sample p-value that contribute to the final
Kolmogorov-Smirnov test for the uniformity of the distribution of
p-values of the test statistic is a variable with default 100, which is
much larger than most diehard default values. This change alone
causes many generators that are asserted to "pass diehard" to in fact
fail -- any given test run generates a p-value that is acceptable, but
the distribution of p-values is not uniform.
- The number of actual samples within a test that contribute
to the single-run test statistic was made a variable when possible.
This was generally possible when the target was an easily computable
function of the number of samples, but a number of the tests have
pre-computed targets for specific numbers of samples and that number
cannot be varied because no general function is known relating the
target value to the number of samples.
- Many of diehard's tests investigated overlapping bit sequences.
Overlapping sequences are not independent and one has to account for
covariance between the samples (or a gradually vanishing degree of
autocorrelation between sequential samples with gradually decreasing
overlap). This was generally done at least in part because it used
file-based input of random numbers and the size of files that could
reasonably be generated and tested in the mid-90's contained on the
order of a million random deviates.
Unfortunately, some of the diehard tests that rely on weak inverses
of the covariance matrices associated with overlapping samples seem to
have errors in their implementation, whether in the original diehard
(covariance) data or in dieharder-specific code it is difficult to say.
Fortunately, it is no longer necessary to limit the number of random
numbers drawn from a generator when running an integrated test, and
non-overlapping versions of these same tests do not require any
treatment of covariance. For that reason non-overlapping versions of
the questionable tests have been provided where possible (in particular
testing permutations and sums) and the overlapping versions of those
tests are deprecated pending a resolution of the apparent errors.
In a few cases other variations are possible for specific tests.
This should be noted in the built-in test documentation for that test
where appropriate.
Aside from these major differences, note that the algorithms were
independently written more or less from the test descriptions alone
(sometimes illuminated by a look at the code implementations, but only
to clear up just what was meant by the description). They may well do
things in a different (but equally valid) order or using different (but
ultimately equivalent) algorithms altogether and hence produce slightly
different (but equally valid) results even when run on the same data
with the same basic parameters. Then, there may be bugs in the
code, which might have the same general effect. Finally, it is always
possible that diehard implementations have bugs and can be in
error. Your Mileage May Vary. Be Warned.
About Dieharder
The primary point of dieharder (like diehard before it) is to make it
easy to time and test (pseudo)random number generators, both software
and hardware, for a variety of purposes in research and cryptography.
The tool is built entirely on top of the GSL's random number generator
interface and uses a variety of other GSL tools (e.g. sort, erfc,
incomplete gamma, distribution generators) in its operation.
Dieharder differs significantly from diehard in many ways. For
example, diehard uses file based sources of random numbers exclusively
and by default works with only roughly ten million random numbers in
such a file. However, modern random number generators in a typical
simulation application can easily need to generate 10^18 or more random
numbers, generated from hundreds, thousands, millions of different seeds
in independent (parallelized) simulation threads, as the application
runs over a period of months to years. Those applications can easily be
sensitive to rng weaknesses that might not be revealed by sequences as
short as 10^7 uints in length even with excellent and sensitive
tests. One of dieharder's primary design goals was to permit tests to
be run on very long sequences.
To facilitate this, dieharder prefers to test generators that
have been wrapped up in a GSL-compatible interface so that they can
return an unbounded stream of random numbers -- as many as any
single test or the entire suite of tests might require. Numerous
examples are provided of how one can wrap one's own random number
generator so that it is can be called via the GSL interface.
Dieharder also supports file-based input three distinct ways. The
simplest is to use the (raw binary) stdin interface to pipe a bit stream
from any rng, hardware or software, through dieharder for
testing. In addition, one can use "direct" file input of either raw
binary or ascii formatted (usually uint) random numbers. The man page
contains examples of how to do all three of these things, and dieharder
itself can generate sample files to use as templates for the appropriate
formatting.
Note Well! Dieharder can consume a lot of random
numbers in the course of running all the tests! To facilitate this,
dieharder should (as of 2.27.11 and beyond) support large file (> 2GB)
input, although this is still experimental. Large files are clunky and
relatively slow, and the LFS (large file system) in linux/gcc is still
relatively new and may have portability issues if dieharder is built
with a non-gcc compiler. It is therefore strongly recommended
that both hardware and software generators be tested by being wrapped
within the GSL interface by emulating the source code examples or that
the pipe/stdin interface be used so that they can return an essentially
unbounded rng stream.
Dieharder also goes beyond diehard in that it is deliberately
extensible. In addition to implementing all of the diehard tests it is
expected that dieharder will eventually contain all of the NIST STS and
a variety of tests contributed by users, invented by the dieharder
authors, or implemented from descriptions in the literature. As a true
open source project, dieharder can eventually contain all rng
tests that prove useful in one place with a consistent interface that
permits one to apply those tests to many generators for purposes of
comparison and validation of the tests themselves as much as the
generators. In other words, it is intended to be a vehicle for the
computer science of random number generation testing as well as a
practical test harness for random number generators.
To expand on this, the development of dieharder was motivated by the
following, in rough order of importance:
- To provide a readily available, rpm- or apt- installable
toolset so that "consumers" of random numbers (who typically use
large numbers of random numbers in e.g. simulation or other
research) can test the generator(s) they are using to verify their
quality or lack thereof.
- To provide a very simple user interface for that toolset for
random number consumers. At the moment, this means a command line
interface (CLI) that can easily be embedded in scripts or run repeatedly
with different parameters. A graphical user interface (GUI) is on the
list of things to do, although it adds little to the practical utility
of the tool.
- To provide lots of knobs and dials and low level control for
statistical researchers that want to study particular generators with
particular tests in more detail. This includes full access to test
sources -- no parameter or aspect of the test algorithms is "hidden" and
needs to be taken on faith.
- To have the entire test code and documentation be fully Gnu
Public Licensed and hence openly available for adaptation, testing,
comment, and modification so that the testing suite itself becomes (over
time) reliable.
- To be extensible. Dieharder provides a fairly simple
API for adding new tests with a common set of low-level testing
tools and a common test structure that leads (one hopes) to an
unambiguous decision to accept or reject any given random number
generator on the basis of any given test for a suitable choice of
controllable test parameters.
- To allow all researchers to be able to directly test, in
particular, the random number generators interfaced with the GSL.
This is a deliberate design decision justified by the extremely large
and growing number of random number generators prebuilt into the GSL and
the ease of adding new ones (either contributing them to the project or
for the sole purpose of local testing).
- To allow researchers that use e.g. distributions directly
generated by GSL routines (which can in principle fail two ways, due to
the failure of the underlying random number generator or due to a
failure of the generating algorithm) to be able to directly validate
their particular generator/distribution combination at the cost of
implementing a suitable test in dieharder (using the code of existing
tests as a template).
- To allow dieharder to be directly interfaced with other tools
and interfaces. For example, dieharder can be directly called
within the R interface, permitting its rngs to be tested and R-based
graphics and tools to be used to analyze test results. Note well,
however, that because it uses the GSL (which is GPL viral) dieharder
itself is GPL viral and cannot be embedded directly into a non-GPL tool
such as matlab. It can, of course, be used to generate p-value
data that is passed on to matlab (or any other graphing or analysis
tool)
Although this tool is being developed on Linux/GCC-based platforms,
it should port with no particular difficulty to other Unix-like
environments (at least ones that also support the GSL), with the further
warning that certain features (in particular large file support) may
require tweaking and that the dieharder authors may not be able to help
you perform that tweaking.
Essential Usage Synopsis
If you compile the test or install the provided binary rpm's and run
it as:
dieharder -a
it should run -a(ll) tests on the default GSL generator.
Choose alternative tests with -g number where:
dieharder -g -1
will list all possible numbers known to the current snapshot of the
dieharder.
dieharder -l
should list all the tests implemented in the current snapshop of
DieHarder. Finally, the venerable and time tested:
dieharder -h
provides a Usage synopsis (which can quite long) and
man dieharder
is the (installed) man page, which may or many not be completely up
to date as the suite is under active development. For developers,
additional documentation is available in the toplevel directory or doc
subdirectory of the source tree. Eventually, a complete DieHard manual
in printable PDF form will be available both on this website and in
/usr/share/doc/dieharder-*/.
List of Random Number Generators and Tests
Available
List of GSL and user-defined random number generators that can be
tested by dieharder:
#=============================================================================#
# dieharder version 3.29.4beta Copyright 2003 Robert G. Brown #
#=============================================================================#
# Id Test Name | Id Test Name | Id Test Name #
#=============================================================================#
| 000 borosh13 |001 cmrg |002 coveyou |
| 003 fishman18 |004 fishman20 |005 fishman2x |
| 006 gfsr4 |007 knuthran |008 knuthran2 |
| 009 knuthran2002 |010 lecuyer21 |011 minstd |
| 012 mrg |013 mt19937 |014 mt19937_1999 |
| 015 mt19937_1998 |016 r250 |017 ran0 |
| 018 ran1 |019 ran2 |020 ran3 |
| 021 rand |022 rand48 |023 random128-bsd |
| 024 random128-glibc2 |025 random128-libc5 |026 random256-bsd |
| 027 random256-glibc2 |028 random256-libc5 |029 random32-bsd |
| 030 random32-glibc2 |031 random32-libc5 |032 random64-bsd |
| 033 random64-glibc2 |034 random64-libc5 |035 random8-bsd |
| 036 random8-glibc2 |037 random8-libc5 |038 random-bsd |
| 039 random-glibc2 |040 random-libc5 |041 randu |
| 042 ranf |043 ranlux |044 ranlux389 |
| 045 ranlxd1 |046 ranlxd2 |047 ranlxs0 |
| 048 ranlxs1 |049 ranlxs2 |050 ranmar |
| 051 slatec |052 taus |053 taus2 |
| 054 taus113 |055 transputer |056 tt800 |
| 057 uni |058 uni32 |059 vax |
| 060 waterman14 |061 zuf | |
#=============================================================================#
| 200 stdin_input_raw |201 file_input_raw |202 file_input |
| 203 ca |204 uvag |205 AES_OFB |
| 206 Threefish_OFB | | |
#=============================================================================#
| 400 R_wichmann_hill |401 R_marsaglia_multic. |402 R_super_duper |
| 403 R_mersenne_twister |404 R_knuth_taocp |405 R_knuth_taocp2 |
#=============================================================================#
| 500 /dev/random |501 /dev/urandom | |
#=============================================================================#
| 600 empty | | |
#=============================================================================#
Two "gold standard" generators in particular are provided to "test
the test" -- AES_OFB and Threefish_OFB are both cryptographic generators
and should be quite random. gfsr4, mt19937, and taus (and several
others) are very good generators in the GSL, as well. If you are
developing a new rng, it should compare decently with these generators
on dieharder test runs.
Note that the stdin_input_raw interface (-g 200) is a "universal"
interface. Any generator that can produce a (continuous) stream of
presumably random bits can be tested with dieharder. The easiest way to
demonstrate this is by running:
dieharder -S 1 -B -o -t 1000000000 | dieharder -g 75 -r 3 -n 2
where the first invocation of dieharder generates a stream of binary
bits drawn from the default generator with seed 1 and the second reads
those bits from stdin and tests them with the rgb bitdist test on two
bit sequences. Compare the output to:
dieharder -S 1 -r 3 -n 2
which runs the same test on the same generator with the same seed
internally. They should be the same.
Similarly the file_input generator requires a file of "cooked" (ascii
readable) random numbers, one per line, with a header that describes the
format to dieharder. Note Well! File or stream input rands (with any
of the three methods for input) are delivered to the tests on demand,
but if the test needs more than are available dieharder either fails (in
the case of a stdin stream) or rewinds the file and cycles through it
again, and again, and again as needed. Obviously this significantly
reduces the sample space and can lead to completely incorrect results
for the p-value histograms unless there are enough rands to run EACH
test without repetition (it is harmless to reuse the sequence for
different tests). Let the user beware!
List of the CURRENT fully implemented tests (as of the 08/18/08
snapshot):
#=============================================================================#
# dieharder version 3.29.4beta Copyright 2003 Robert G. Brown #
#=============================================================================#
Installed dieharder tests:
Test Number Test Name Test Reliability
===============================================================================
-d 0 Diehard Birthdays Test Good
-d 1 Diehard OPERM5 Test Suspect
-d 2 Diehard 32x32 Binary Rank Test Good
-d 3 Diehard 6x8 Binary Rank Test Good
-d 4 Diehard Bitstream Test Good
-d 5 Diehard OPSO Good
-d 6 Diehard OQSO Test Good
-d 7 Diehard DNA Test Good
-d 8 Diehard Count the 1s (stream) Test Good
-d 9 Diehard Count the 1s Test (byte) Good
-d 10 Diehard Parking Lot Test Good
-d 11 Diehard Minimum Distance (2d Circle) Test Good
-d 12 Diehard 3d Sphere (Minimum Distance) Test Good
-d 13 Diehard Squeeze Test Good
-d 14 Diehard Sums Test Do Not Use
-d 15 Diehard Runs Test Good
-d 16 Diehard Craps Test Good
-d 17 Marsaglia and Tsang GCD Test Good
-d 100 STS Monobit Test Good
-d 101 STS Runs Test Good
-d 102 STS Serial Test (Generalized) Good
-d 200 RGB Bit Distribution Test Good
-d 201 RGB Generalized Minimum Distance Test Good
-d 202 RGB Permutations Test Good
-d 203 RGB Lagged Sum Test Good
-d 204 RGB Kolmogorov-Smirnov Test Test Good
Full descriptions of the tests are available from within the tool.
For example, enter:
rgb@lilith|B:1003>./dieharder -d 203 -h
OK, what is dtest_num = 203
#==================================================================
# RGB Lagged Sums Test
# This package contains many very lovely tests. Very few of them,
# however, test for lagged correlations -- the possibility that
# the random number generator has a bitlevel correlation after
# some fixed number of intervening bits.
#
# The lagged sums test is therefore very simple. One simply adds up
# uniform deviates sampled from the rng, skipping lag samples in between
# each rand used. The mean of tsamples samples thus summed should be
# 0.5*tsamples. The standard deviation should be sqrt(tsamples/12).
# The experimental values of the sum are thus converted into a
# p-value (using the erf()) and a ks-test applied to psamples of them.
#==================================================================
Note that all tests have been independently rewritten from their
description, and may be functionally modified or extended relative to
the original source code published in the originating suite(s). This
has proven to be absolutely necessary; dieharder stresses random number
generator tests as much as it stresses random number generators, and
tests with imprecise target statistics can return "failure" when the
fault is with the test, not the generator.
The author (rgb) bears complete responsibility for these changes,
subject to the standard GPL code disclaimer that the code has no
warranty. In essence, yes it may be my fault if they don't work but
using the tool is at your own risk and you can fix it if
it bothers you and/or I don't fix it first.
Development Notes
All tests are encapsulated to be as standard as possible in the way
they compute p-values from single statistics or from vectors of
statistics, and in the way they implement the underlying KS and chisq
tests. Diehard is now complete in dieharder (although two tests are
badly broken and should not be used), and attention will turn towards
implementing more selected tests from the STS and many other sources. A
road map of sorts (with full supporting documentation) is available on
request if volunteers wish to work on adding more GPL tests.
Note that a few tests appear to have stubborn bugs. In particular,
the diehard operm5 test seems to fail all generators in dieharder.
Several users have attempted to help debug this problem, and it
tentatively appears that the problem is in the original diehard code and
not just dieharder. There is extensive literature on overlapping tests,
which are highly non-trivial to implement and involve things like
forming the weak inverse of covariance matrices in order to correct for
overlapping (non-independent) statistics.
A revised version of overlapping permutations is underway (as an rgb
test), but is still buggy. A non-overlapping (rgb) permutations test is
provided now that should test much the same thing at the expense of
requiring more samples to do it.
Similarly, the diehard sums test appears to produce a systematically
non-flat distribution of p-values for all rngs tested, in particular for
the "gold standard" cryptographic generators aes and threefish, as well
as for the "good" generators in the GSL (mt19937, taus, gfsr4). It
seems very unlikely that all of these generators would be flawed in the
same way, so this test also should not be used to test your rng.
Thoughts for the Future/Wish List/To Do
- Tests of GSL random distribution (as opposed to number) generators,
as indirect tests of the generators that feed them.
- New tests, compressions of existing ones that are "different" but
really the same. Hyperplane tests. Spectral tests. Especially the bit
distribution test with user defineable lag or lag pattern (to look for
subtle, long period correlations in the bit patterns produced).
- Collaborators. Co-developers welcome, as are contributions or
suggestions from users. Note well that users have already provided
critical help debugging the early code! Part of the point of a GPL
project is that you are NOT at the mercy of a black box piece of code.
If you are using dieharder and are moderately expert at statistics and
random numbers and observe something odd, please help out!
Conclusions
I hope that even during its development, you find dieharder useful.
Remember, it is fully open source, so you can freely modify and
redistribute the code according to the rules laid out in the Gnu Public
License (version 2b), which might cost you as much as a beer one day.
In particular, you can easily add random number generators using the
provided examples as templates, or you can add tests of your own by
copying the general layout of the existing tests (working toward a
p-value per run, cumulating (say) 100 runs, and turning the resulting KS
test into an overall p-value). Best of all, you can look inside the
code and see how the tests work, which may inspire you to create a new
test -- or a new generator that can pass a test.
To conclude, if you have any interest in participating in the
development of dieharder, be sure to let me know, especially if you have
decent C coding skills (including familiarity with Subversion and the
GSL) and a basic knowledge of statistics. I even have documents to help
with the latter, if you have the programming skills and want to LEARN
statistics. Bug reports or suggestions are also welcome.
Submit bug reports, etc. to
rgb at phy dot duke dot edu
- Benchmaster Program
-
Benchmaster: A System Testing Suite
by Robert G. Brown (rgb)
benchmaster Version 1.1.3
Contents
Back to top
Description
Benchmaster is a fairly sophisticated program designed to
time and exercise very specific systems functions. It uses the fastest
onboard clock that it can find (generally the CPU's cycle counter on
x86-derived architectures) to time test "objects", and determines the
precision of that timer including the overhead of the timer call.
A test object contains (in addition to test creators/destructor
functions) a test routine with two branches -- one "empty" and one
"full" -- that are structured to be, as nearly as possibly, identical
except for the additional code to be timed in the full loop.
The test harness then determines iteration counts -- the
number of times it has to run the empty or full branches to accumulate a
time much greater than the timer resolution. It then proceeds to
generate a requested number of samples of the timings of the
empty and full branches. Finally, it subtracts the average full time
from the average empty time to determine the result and evaluates the
mean and standard deviations to produce the cumulative expected
error.
Finally, the results are printed out in a standard XML based
format with an optional header describing the test and certain runtime
details. Numbers that are returned include the length of the vector
(see discussion of vector tests below), the stride (ditto), and the mean
time with error bars, in nanoseconds required to execute just the
tested code fragment in the particular context of the test routine.
Finally, a "megarate" is returned that is generally the number of
million times the test fragment is executed per second. There are a few
exceptions to this, see below.
The use of XML in the test output is one of the project's major
design goals, as it in principle makes it possible to build e.g. DTDs
for the benchmark description language to generate standard reports in a
variety of media. Of course this is not yet done -- it is one of the
next major design goals. Volunteers/contributions of code
welcome... as are comments on the XML itself (which is still pretty
malleable until at least one program or transformation process is
written that uses it).
Back to top
The version numbers have the following meaning. Note that these
aren't necessarily what you might expect, so be sure to at least glance
at this.
- First number (major). Bumped only when major goals in the design
roadmap are reached (for example, finishing all the test ideas outlined
below:-). Version 1.x.x, for example, means that the basic testing
structure is well defined and stable.
- Second number (first minor). Bumped for bugfixes, feature
additions, major or minor improvements (but reset to zero when a major
design goal is reached).
- Third number (second minor). This number indicates the number of
tests currently supported.
All benchmaster sources (.tgz and .src.rpm) files can be downloaded
from this directory. In addition, i386 binary
rpm's (currently built on top of Fedora Core 5) are present.
This project is currently semi-stable. The tests seem to work (at
least for me) fairly consistently, there just aren't as many as I'd like
there to eventually be. Alas, I have too many projects going at once,
and have recently been spending a lot of time with the Dieharder project
available elsewhere on this website. If you are interested in seeing
the benchmaster project advanced more aggressively, contact me
(especially with an offer to help out:-).
Below are descriptions of the different kinds of tests already in
benchmaster.
Back to top
Vector Tests
As noted above, in addition to being sampled in one loop (with a
controllable number of samples) and iterated inside that loop per sample
(with an automatically set but user controllable number of iterations)
some of the tested code fragments are loops that operate on vectors of
numbers. For example, the stream benchmark by John
D. McCalpin consists of four simple numerical operations -- a vector
copy, scaling a vector, adding a vector, and adding and rescaling a
vector. Each of these stream operations is one of the tests already
programmed into benchmaster, so benchmaster is capable of "running
stream" on a system
In stream, the length of the tested vector is generally fixed --
programmed directly into the test as a compile-time parameter. In
benchmaster's stream (and other related vector arithmetic tests) the
vector length is a variable and can be selected by the user at
runtime. This permits one to directly observe how cache improves
performance for various vector lengths and strides.
Most modern CPUs have elaborate optimization mechanisms designed to
improve numerical performance on vectors in particular. They prefetch
data from memory into the cache in order to ensure that it is waiting
there when needed. They have special registers and pipelines that speed
repeated operations iterated over a vector. However, sometimes one runs
code that (although iterative) does not have such a fortunate
memory access pattern. Benchmaster therefore contains a facility for
permitting at least some vector tests to be performed in "shuffled
order". Basically, a matching vector of vector indices is shuffled and
used as the indices for the test. The overhead associated with the
shuffling process per se is duplicated in the "empty" code fragment so
that only the actual time required to access the memory in shuffled
order contributes.
Back to top
Test Design
The general function and design of the timing harness is explained
above and documented in the code itself. This section indicates how to
add a test to the suite, as one of its major design goals is to make it
easy for you to add your own test fragments and
operations.
Before showing you an example of a test (the easiest way to document
how to create a new one) let me remark on a few of the many problems
that plague "benchmarking". For one, the speed with which code is
executed depends on many, many things, some of which are out of our
control. For example, there is an obvious dependence on system state --
if your task is swapped off of the CPU on a multitasking system in
mid-instruction, your time for that sample will be quite high. There is
an uncontrollable dependence on the compiler. I'm assuming that the
full and empty branches are both likely to be close together in memory
and both likely to be kept resident in cache in order for the timings to
be comparable and subtractable. This is likely enough to be true, but
there are no guarantees from the compiler.
It is also very difficult to test/time single instructions of
just about any kind. A multiply on a system can take a fraction of a
clock cycle. The finest-grained timekeeper on the system is the
clock cycle counter (typically order of a nanosecond), and the code
required to read it takes many nanoseconds to execute. Timing a
multiply is thus akin to timing the beating of a hummingbird's wings
with an hourglass, a process made worse by the compiler's tendency to
optimize away instructions that it can tell are doing nothing and can be
compressed.
This is just a warning. As it says in the program's Usage statement
and man page, the "Mega-rates" returned by this tool are BOGUS and may
not be even approximately correct. When interpreting results of
existing tests or adding your own, be cautious and test the tester as
much as the code fragment itself until the results make sense in your
own mind.
One final warning about hidden optimizations, overflows, etc. Many
CPUs are smart enough to use superfast internal arithmetic in order to
perform certain operations. For example, multiplying by 0.0 or 1.0 or
2.0 on many CPUs will take much, much less time than multiplying by
(say) 3.141592653589. For that reason I typically use this as a number
to multiply by whenever I am testing multiplication. Of course if one
iterates multiplication by \pi it doesn't take long to overflow a
floating point variable, so one needs to use caution in designing test
loops to avoid this without using 1.0 as a multiplier or dividing (which
takes much longer than multiplication) when one wants to test
only multiplication.
With all that said, a test consists of two pieces. An include file
that minimally contains the function prototypes for the the test itself,
for example (for the stream copy test):
/*
* $Id: benchmaster.abs,v 1.7 2004/12/17 15:31:56 rgb Exp $
*
* See copyright in copyright.h and the accompanying file COPYING
*
*/
/*
* Goes with stream_copy_test.c.
*/
void stream_copy_init(Test *newtest);
void stream_copy_alloc();
void stream_copy_free();
int stream_copy_test(int full_flag);
void stream_copy_results();
void stream_copy_about();
and the stream copy test source for these components:
/*
* $Id: benchmaster.abs,v 1.7 2004/12/17 15:31:56 rgb Exp $
* See copyright in copyright.h and the accompanying file COPYING
*/
#include "benchmaster.h"
/*
*==================================================================
* This is the "copy" test from the stream suite. It is not
* directly comparable to stream results for a variety of reasons.
* For one, it uses malloc to allocate all vectors so vector
* length may be different from what is compiled into any given copy
* of stream. Also, it uses the average time to determine the rate
* and not the best time. It will therefore generally return
* results that are very SLIGHTLY LOWER/SLOWER than regular stream
* (but which may be more realistic for general purpose code).
*
* It also uses a different timing harness, one that is both
* more accurate (uses a superior timer) and which repeats the
* computation many times, typically order 100, to obtain both a
* mean time and its standard deviation as test results.
*==================================================================
*/
void stream_copy_init(Test *mytest){
int i;
mytest->alloc = stream_copy_alloc;
mytest->free = stream_copy_free;
mytest->test = stream_copy_test;
mytest->results = stream_copy_results;
snprintf(mytest->name,K,"stream copy");
snprintf(mytest->about,K,"d[i] = a[i] (%d byte double vector)",sizeof(double));
if(verbose == VERBOSE || verbose == V_INIT){
printf("# Init for test %s\n",mytest->name);
}
}
void stream_copy_alloc()
{
int i;
/*
* Allocate vector(s) to be tested with and initialize it and all
* associated test-specific variables.
*/
d = (double *) malloc((size_t) (size*sizeof(double)));
a = (double *) malloc((size_t) (size*sizeof(double)));
/*
* Initialize the vector. xtest is set from the command line, default PI.
*/
for(i=0;i < size;i+=stride){
a[i] = xtest;
}
}
void stream_copy_free()
{
int i;
/*
* Free all the memory we just allocated, to be neat and clean and
* all that.
*/
free(a);
free(d);
}
int stream_copy_test(int full_flag)
{
int i;
if(full_flag){
for(i=0;i < size;i+=stride){
d[i] = a[i];
}
return(full_flag);
} else {
return(full_flag);
}
}
void stream_copy_results(Test *mytest)
{
double nanotime_norm;
/*
* This is the number of copy operations in the core loop. We adjust the
* test normalization so it is the SAME as that of stream, which computes
* the rate as "megabytes/seconds": 1.0e-6*2*sizeof(double)*nsize/time
* (in seconds). We measure nanoseconds, so ours is just a teeny bit
* different.
*/
nanotime_norm = (double)size/stride;
mytest->avg_time = fabs(mytest->avg_time_full - mytest->avg_time_empty)/nanotime_norm;
mytest->sigma = (mytest->sigma_time_full + mytest->sigma_time_empty)/nanotime_norm;
mytest->avg_megarate = 1000.0*2*sizeof(double)/mytest->avg_time;
show_results(mytest);
}
Note that xtest is a user-settable value for the real number used to
initialize the vector. It defaults to pi, but can be overridden on the
command line so you can see for yourself the effect of using 1.0 or 0.0
in certain contexts of certain tests instead of a number that cannot be
optimized at the CPU microcode level. Note that the avg_megarate is a
bit different than from other tests as it returns a "bandwidth" in
Mbytes/sec (to be comparable with stream) instead of a "bogomegarate",
which is just the number of millions of iterations of the test fragment
completed per second.
This code fragment in my testing harness produces results that are
generally a few percent slower than those of off-the-shelf stream. This
is understandable -- I use the average time instead of the minimum time
to generate the number, so my stream number is average/expected
performance over a reasonably large number (default 100) of samples
while stream reports the observed peak performance on a fairly small
sample size (default 10).
Once you have written a "microbenchmark" test fragment of your own,
you have merely to insert it into the overall structure of the
benchmaster program. The best way to do that is to follow exactly the
way the existing tests are inserted. Put your test's include file and
NAME into the enum and include list in tests.h. Initialize the test in
startup.c. At that point the code should "just work". Try to use the
provided global variables and pointers for things like xtest and
vectors, just to keep the primary code from getting too gooped up. You
should be able to put globals into the include file for your test if you
need to add any new ones for specific tests, and should be sure to add
code to parsecl.c and startup as required to manage them.
Note Well! Benchmaster is not written for the purpose
of encouraging "benchmarketing" by vendors, so take any
publication of results from the benchmark with a grain of salt.
Consider the source! I mean that quite literally. Benchmaster is a
fully GPL tool that is available in rpm form. That means that you or a
vendor can hack up the source, insert optimized machine code,
etc. Note that you should insist on results of unmodified benchmaster
source, compiled with the default flags as at least one component
of what you consider when you are trying to measure or understand
systems performance. Only then is it reasonable to have fun tuning
things up to see how fast you can make them.
Back to top
Conclusion
As noted above, benchmaster is designed to be easy to modify
to insert your own tests as "test objects". You should feel free to
contribute particularly good or revealing tests back to me for inclusion
in the primary distribution of the tool. If and when I get so many
tests into the tool that the current method of selecting tests by number
no longer works, I'll probably have to rewrite the toplevel interface to
make it easier to keep track of everything, but that is likely some time
off (and should not affect results).
You should also feel free to make suggestions or report bugs or
contribute code to the timing harness itself -- I am constantly trying
things to see if I can obtain better control over system state (and
hence get more reproducible results) and to see if I can finagle
accurate timings of those hummingbird wing beats using my hourglass.
Perhaps one of you reading this knows just how to go about doing it in
userspace code...
Future Plans for benchmaster include:
- More microbenchmark tests, including ones for e.g. disk access,
network latency/bw (will need an accompanying daemon to run on the
target)
- More shufflable tests (shuffled stream?)
- Selected application or library level tests.
- XML test interface on the input side as well as the output?
It would be nice to be able to read in a "test descriptor file" that
made benchmaster work through a suite of tests according to a set of
instructions and produce a table/report as output. This would also
facilitate the development of a...
- ...GUI. Both web and Gtk interfaces would greatly simplify the
use of the tool and would permit the immediate-gratification
presentation of e.g. performance curves as vector sizes are swept across
the size of L1 and L2 caches.
Participation in benchmaster is openly solicited and encouraged.
All code is written in C for efficiency and its tight coupling to the
operating system, for all that it is still "object oriented" in general
design to keep it scalable and extensible. Contact me at
rgbATphy.duke.edu if you are interested in participating.
I hope that you find benchmaster useful.
Back to top
- Amazon Sales Rank Tools
An Amazon Sales Rank Tools
Version 0.1.0
Robert G. Brown (rgb)
This is a distribution site for a simple, perl based toolset for
tracking Amazon Sales Rank for initially books only -- the current key
item indicator is the book's ISBN number -- but eventually this will be
broadened to other kinds of content sold via Amazon.
The tool's operation is quite simple. Unpack the tarball into a
suitable directory. Make sure that you've installed its dependencies:
perl, gd and gd-devel, perl-GD and perl-GDGraph.noarch and elinks.
Enter the directory and enter:
amazon_sales_rank -i ISBN
on the command line (inside, say, an xterm). You will need internet
access on the system
It should connect to amazon, download and parse your book's page, and
extract its sales rank. It will write this out into a table and begin
to generate a png image of its graph as a function of sample time.
Leave the tool running. Every hour it will reconnect with Amazon and
update the sales rank (Amazon updates sales ranks only once an hour so
there is no point in sampling any more often).
That's all there is to it! You can view the sales rank png with any
graphical viewer, e.g. gthumb, xv, or you can edit the
amazon_sales_rank.html file provided to make it point to the path of the
png and open this page in your favorite browser. It will then
automagically update every hour to show you the entire graph to
date. The raw data is collected in the amazon_sales_rank_table.isbn
file that appears in the working directory, and this can be imported to
a spreadsheet or other tools for further processing. This table is
preloaded should you need to stop the tool and restart it -- you don't
lose the points you already have if you must shut the tool down for a
time.
This is a VERY PRELIMINARY release of the tool. It does the very
minimum needed to track ASR for your book and display it as a reasonably
formatted graph and save it in a reasonably formatted table. Future
releases will be a bit slicker, and will probably automate things a lot
more. Right now it is recommended that only linux-savvy people attempt
to use it, but come back in a few months and you should find a drop in,
plug-and play tool that installs in the actual operating system and that
runs automatically to generate reports in much the same way that
webalize does now.
Submit bug reports, feature requests etc. to
rgb at phy dot duke dot edu
Also, consider visiting the site of my book: The Book of
Lilith. If you honor me and the work put into developing this tool
by electing to purchase a copy, I think you'll be glad you did!
- The C Book
NOTE WELL! This book is not written by or copyrighted by
Robert G. Brown! It is a mirror of an online book on C programming that
-- curiously enough -- has a license almost identical to my Gnu Public
License, v2b (b for beverage). However, you have to buy the actual
authors of the book a beer, not me.
From the book's front page:
This is the online version of The C Book, second edition by Mike
Banahan, Declan Brady and Mark Doran, originally published by Addison
Wesley in 1991. This version is made freely available.
While this book is no longer in print, it's content is still very
relevant today. The C language is still popular, particularly for open
source software and embedded programming. We hope this book will be
useful, or at least interesting, to people who use C.
If you have any comments about this book, or if you find any bugs in its
presentation, please send a message to consulting@gbdirect.co.uk.
- The Yum HOWTO
This is a draft of a future yum HOWTO, offered up here for
private comment by initially Seth and Michael and later by (I imagine)
the yum list. If you've found this in the meantime via google, feel
free to use it but beware -- it may be openly wrong or misleading in
places.
Note Well: This is a Request for Comments only;
Use at your own risk, and please return comments and corrections to rgb (or better, the yum mailing list)
for encorporation into what I hope to be a dynamic document associated
with a project, rather than a static prescription that may, in fact, be
heavily flawed.
- CVS Mini HOWTO
is a tutorial mini-HOWTO on how to set up and use CVS for
personal user or simple shared projects. It goes through things step by
step with examples. To use it effectively, it is strongly suggested
that you try the things illustrated as you go along.
It was written primarily to teach colleagues and students with whom I
collaborate the rudiments of CVS, as I use it as an essential component
of the management of any project I'm involved in that generates
documents and other data objects (such as teaching a course as part of a
team, guiding students in research involving programming, writing
papers).
The HOWTO appears to fill a community need -- a quick google reveals
only one standard-format HOWTO for CVS, and that one is extremely terse
and specific to NetBSD. This particular HOWTO is structured as a
tutorial that leads you step by step through the basic commands required
to set up CVS root repositories for various purposes and use to CVS to
manage project archives within those repositories.
It deliberately omits the setup and use of CVS with pserver
(web-based access) as this is documented a variety of places and because
we do not use this locally (internal to our department LAN) due to
security concerns -- vulnerabilities have been found and patched within
the last year, and while the are patched, one always worries
about whether there are more. In any event, this requires root
privileges to set up and manage and therefore almost by definition is
advanced usage (in my opinion) and hence is inappropriate for this
document.
In other words, this is very much a Getting Started with CVS
sort of document -- CVS is very powerful and hence very complex, and
given half a chance it will, as they say, eat your meatloaf for you.
However to my own direct experience, well over 95% of all CVS usage in a
simple LAN environment is encompassed within the commands tutorially
demonstrated in this HOWTO. Many users will never need all that power;
users that do will need to master the conventional and relatively simple
aspects of CVS documented here before using the actual manual and more
advanced documentation to learn how to (for example) manage multiple
branches of code anyway.
At any rate, if you are interested in learning to use CVS (and you
should be, if you work at all with dynamic information of any type
whatsoever) you many find this document useful. Feel free to provide me
with feedback, but remember that I did not write CVS and am not a CVS
developer, so any actual bug reports or feature requests should probably
go to the CVS developers via its home page:
http://cvshome.org
There are several additional sources of tutorial information,
manuals, and other documentation on the primary website, but
unfortunately none are formatted in the familiar HOWTO style and they
tend (in my opinion) to be too simple (omitting enough information to be
able to set up or participate in a project as a member of a group) or
too complex (an exhaustive manual complete with visuals and lots of
detail and instructions for Windows as well as Unices). Nevertheless,
new CVS users will find it worthwhile to visit this site and quickly
browse and bookmark these resources as they will provide additional
support for this document when some feature or detail it omits is needed
for a project.
Note Well! This mini-HOWTO was developed on a linux system and
makes the following assumptions:
- A Unix-like filesystem with Unix groups and file permissions.
Commands such as chgrp, chmod, mkdir and so forth should be available
and hopefully familiar to the reader. Although CVS exists for Windows,
it is documented elsewhere and I don't use Windows.
- A reasonably recent version of CVS, in particular one that
supports the CVS_RSH environment variable. I'm using 1.11.17 as I write
this. If your version is very different be aware that features and
options tend to creep and commands illustrated may not work for you.
- The presence of and familiarity with /usr/bin/ssh (ssh = s(ecure
)sh(ell)), configured as you prefer for remote access (with or without a
password, for example).
- Reasonable familiarity with environment variables and how to set
them within your shell. I give a few examples here, but make no effort
to be thorough.
- A text editor installed on the system that you are familiar with
and that can be set as the default editor for logging cvs changes.
I'll try to indicate any others as they occur to me.
- random_pvm Demo/Template
random_pvm is a C source demo/template for generating random
numbers using a PVM master/slave program. It is derived from the C
source project template also available from rgb's website. It was written
for a Cluster World Magazine column, and although it is freely
available and GPL'd users are encouraged to subscribe to the magazine
to get all sorts of other goodies that come with every issue.
random_pvm actually installs from the tarball ONLY. In most cases a
program template I write will create a workable rpm, but it isn't
really desireable to install this demo in a rootspace /usr/share/pvm3
directory so although the make targets are there (and might even work,
although I doubt it) I advise against messing with them.
To build it, create or change to your source directory (I use
$HOME/Src but suit yourself), put random_pvm.tgz there and unpack
it:
tar xvfz random_pvm.tgz
(and it should verbosely unpack).
Change to the random_pvm directory. There is a random_pvm.1 man page
there that gives instructions on how to build and install and use the
program. (Basically stuff like make, make install, and then running the
program.) Remember to start pvm and build a virtual machine
(instructions NOT included herein) before trying to run the program,
and make sure that the random_pvm_slave program is installed in the
appropriate place on all the nodes.
If you have any fundamental trouble getting it working, let me know
and I'll try to help you. My email address is rgb@phy.duke.edu.
- Jove: Jonathan's Own Version of Emacs
-
JOVE: Jonathan's Own Version of Emacs
by Jonathan Payne (not NOT by Robert Brown
jove Version 4.16.0.65
This is a portable/semimaintainable rpm packaging of jove. Jove
stands for Jonathan's Own Version of Emacs, and in my opinion it has
been the best text editor available for decades (as emacs, its
progenitor, has become ever more crack-ridden until I can no longer
stand to use it at all even as a stand-in for jove). Jove is, in
particular, a really great editor for e.g. C source code, latex source
code, and in general source codes that require an invocation of "make"
to build internally. It has all the essential features of emacs without
losing its attractive sparseness.
Since I use jove exclusively (having done so for getting on 18 years
at this point) and since I also use rpm-based systems exclusively and
rpm-centric distribution tools such as yum, I need jove to be
neatly packaged. The first thing I ever do on a system is go in and
install jove so I can work on it. It needs to be cleanly
rpm-buildable and (I think) distributed as prebuilt source rpm if not
binary rpm for some of the major distributions.
Jove is currently maintained (as far as I can tell) as a tarball-only
product within Toronto's CS
department. From their base, I've hacked the Makefile, the spec
file, and the versions.h file (trivially) as follows:
- Added sed automagic so that version numbers and build date are set
in one place only (the Makefile) and updated to spec file and version.h.
- Added gpg signatures to the rpm's.
- Added a set of macros and targets to support the maintenance of
this website, including targets for "make tgz", "make rpm", "make yum"
and "make installweb". With these targets I can (re)build rpm's for
three or four architectures (e.g. RH 9, FC 2, FC 3, i386, x86_64,
Centos) and install them in yum repositories by "make yum;make
installweb" on each architecture from one set of sources.
- Cleaned up the specfile in some trivial ways that may not be the
best ways to get clean builds on RH/FC-derived rpm systems.
These changes SHOULD NOT affect any other build targets or build
processes (with the possible exceptions of the specfile changes, where I
don't have enough distribution alternatives to test across all of them).
Either way, if you want a repository from which to mirror relatively
current signed jove rpm's, yum update jove rpms, grab a tarball of jove
that has the above make targets for your own local builds, feel free to
use this site.
I'm also willing to provide some debugging support if the
rpm's on this site don't work for you or rebuild for you. I have to
emphasize the some because I have a lot of projects and as long as jove
works for me, I'm happy and may be busy as well as happy. However, if
you encounter a bug or just need some help feel free to contact me at
rgb at phy.duke.edu.
Back to top
- Project Template
-
project abstract (in html) .
Back to top
- PVM Project Template
project_pvm is a C source project template for PVM master/slave
projects. It is derived from the C source project template also
available from
rgb's website
and does the usual automagical project maintenance via simple (and not
so simple) make targets: make tgz, make rpm, make installweb, make cvs
for example. It is worth several hours of work, minimally, in getting a
pvm project off the ground, and gets it off on the right foot, with a
lot of things you might add "eventually" already there in template form,
ready to fill in.
project_pvm actually installs from the tarball ONLY -- or if you
prefer, the tarball IS the project template ready to use -- but this
page and distribution set is more or less automagically created by the
make installweb target, so it seems worthwhile to include the rpm's even
if they only install a trivial (cluster parallelized) "Hello World!"
program.
To use this template, create or change to your source directory (I
use $HOME/Src but suit yourself), put the project_pvm.tgz there and
unpack it:
tar xvfz project_pvm.tgz
(and it should verbosely unpack).
Change to the project directory. There is a README that gives
instructions on how to use to the template. Most of the mucky part of
the process is encapsulated in a script called "newproject" that you can
read to see how it works. To use this script for a new "rgb standard"
PVM project, start pvm and build a virtual machine (instructions
NOT included herein) and enter:
This presumes that you've got CVS setup (a functioning CVSROOT). If
you want to use the make rpm targets, additionally you must:
Optionally edit the man page template, the README, the abstract (this
file), the php file, Remember, the man page is your friend. Also
remember to update/modify the "Usage" statement in parsecl.c as you add
new command line commands, if any.
If you grab this project template and have any fundamental trouble
getting it working, let me know and I'll try to help you. My email
address is rgb@phy.duke.edu.
- Latex Project Template
Latex Project Template
by Robert G. Brown (rgb)
This is a reusable template for latex projects. Download the tarball
and unpack it with e.g.
tar xvfz latex.proj-0.4.1.tgz
cd latex.proj-0.4.1
Place it under e.g. Subversion control so that you can back off any
changes you make to it.
Look over the Makefile to familiarize yourself with the various
targets. Not all of them are guaranteed to work, and some may not work
for you without some editing, at least. Do not modify the
Makefile yet, though -- work at first on a copy created as below and
then backport the changes to the original Makefile carefully when
you are sure they work.
In this directory, run the "newproject" command:
./newproject projectname
cd ../projectname
Place the new project under your choice of version control --
subversion is currently supported by a couple of targets in the Makefile
that will require a bit of editing to make work for you but CVS is
equally possible with a bit more editing.
Note that newproject SHOULD have changed all the basic filenames
around so that they correspond to your project name. Only make changes
in the template itself if you want to make them permanent features of a
new latex project!
You should be able to type "make" in the new project directory at any
time and have it just work to build dvi, pdf, and a4pdf (a pdf for A4
sized paper commonly used in Europe). Bring up the dvi file for preview
with a command such as:
xdvi projectname &
Then start up your favorite editor on the projectname.tex source file.
If it is emacs or jove (or any editor that permits you to invoke make
from inside the editor) you should be able to make simple changes to the
latex source, invoke make via e.g. ^x ^e, and bring the preview screen
to the foreground, where it will automatically refresh to the
current/new dvi file! This makes the edit, make/debug, view/debug, edit
cycle quite painless and generally faster than most latex IDE GUI
tools.
Good luck! Feel free to contact me with e.g. bug reports or
problems.
- HOWTO project template
This is a HOWTO project template, using the linuxdoc dtd. It isn't
intended to be very complete at the display side, but in the actual
compressed tarball there are complete templates for both dtd and
linuxdoc straight from the tldp.org website. The wrapping might prove
useful as well.
- LaTeX Manual (online)
This is the online latex manual. It is here for my own use (the Nasa site where I found it
can be pretty busy) but it is also freely available to others if you
should find it and want to use it here.
- A Theorem Concerning God
Back to top
|
to help spread the word so others can find it as well.
Note, Robert G. Brown is generally either rgb or rgbatduke on many external
sites crosslinked here.