Speaking of entering a compo: I really, really dig retroprogramming.  The cool part is, after 10 years of retroprogramming in spare time, my kung fu is getting advanced enough where I not only know how to do what I want in assembly, but I know the fastest possible method to getting it done on the target hardware. For example, I recently implemented a vertical-retrace interrupt in software because the hardware one wasn’t good enough. It’s sick that I know how to do that, but sicker that I know why I need to do that.

I still get a kick out of impressing Jim-of-20-years-ago.

I did this all before Y2K.  When you’re the only guy shouting in a crowd, you tend to look the lunatic, and that’s pretty much how most of my friends and family saw me.

Look around the preservation landscape today and much of what I was working towards for years has come to pass.  There are many vintage hardware and software museums, both physical and virtual, including some dedicated to gaming.  There are some wonderful emulators that get closer and closer to the real thing each year.  There are even some curated collections online.  (There are many more curated collections offline, orders of magnitude larger than what is online, but in a decade or so I believe these will move online as well.)  Most importantly, there are established communities that support these efforts.  All in all, I’m pretty happy with how things have turned out.

Looking around all of my possessions inside my home, I see the fallout of what I was trying to accomplish many years ago.  I see no less than five PCjrs, three identical Tandy 1000s, three identical IBM PC 5150s, and multiples of Macs, Apples, C64s, and Amigas.  I see crates and bookshelves and closets filled with hardware and software.  I see clutter where there should be a nice desk for displaying a computer in a respectful way, or an easy chair for reading or watching TV.  It’s too much.  It’s time to let most of it go, and focus like a laser on the things that are the most important.  I will be disseminating most of my collection, both software and hardware, in the following year.

What I will continue to do, however, is archive and preserve software, as there is still a ton of IBM PC software from the 1980s that has not yet been released into the wild.  I am also committed to creating the “sound card museum” project I keep threatening to do.  To those ends, I will retain a few systems that will allow me to achieve both of those goals.

So, I’ll still keep buying and collecting vintage software — the difference is, I won’t retain the software after preserving it.  Consider me a vintage personal computing clearing house.

Because they gave me a larger time slot, the talk has been expanded from 50 minutes to 90 minutes.  There will be some media to watch, as well as a real PCjr showing off its more unique features.

I’m not sure what to do with the extra time — I’ll have the room for 2.5 hours, and I can’t imagine people having more than 5 minutes’ worth of questions.  How to fill the time?  Boot up the original first version of King’s Quest (which has additional functionality on PCjr) on the projector and let people have at it?  Connect it to the internet and join an IRC channel?  Take game requests?  Show off additional software that supported the PCjr?  I’m open to suggestions.

Rumor has it I might have something silly to submit into the wild compo.  Maybe.

What’s this pee-sea-june-your I speak of?  Well, I’ll let Ebeneezer Scrooge and Bob Cratchet explain:

In addition to even faster code, I thought it would be interesting to see how small an LZ4 decompressor could get, so with Peter’s help we managed to come up with a version of the code that trades speed for size.  It’s 30% slower on average, but it compiles to only 78 bytes.

The downloads section of the LZ4_8088 website has been updated to contain both versions in the single .zip file.

For the TL;DR crowd who will likely skip to the end of the article: Through clever 8088 optimization and a thorough understanding of our problem, a decompression routine was created that not only beats all known methods previously created for 8088, but can actually exceed the speed of a memcopy given the right input. But to see how this was achieved, you’ll have to grab a snack and settle down for 10 minutes.

Now, on with our case study.

Background

My interest in speed optimization stems from being a lifelong fan of the demoscene. I’ve written some productions and unleashed a few cheap tricks here and there, but my core strength has actually been “glue” — understanding the underlying hardware and marrying the code to it. For example, while I coded a few effects in Explicit, my core role was actually assembling the demo itself — the timer/interrupt system, the music library, the 3D library, asset management, decompression, debugging, event handling, etc. all fell to me. And that’s fine, I enjoy doing it. (Next time you run Explicit, hold down the ALT key and you’ll see some realtime debug stats overlayed onscreen while the demo is running.) So this is why I tend to write the underlying framework of a demo before actually writing the individual effects, because if the framework is solid, the rest is “easy”. To me, anyway.

For many years, I’ve been threatening to write a proper demo for the 8088. 8088+CGA has been done a few times back in the day by Sourcerers, and there was a nice slideshow+music demo done for the Tandy 1000 as part of the 8086 compo, but there really hasn’t been a proper demo for the 8088 — 3D, graphics hardware tricks, nice design, the works. Every year I get closer, writing another piece of something that will eventually turn into a demo. A month ago, that piece turned out to be a need for fast decompression. (What I am going to use fast decompression for is left as an exercise for the democoder!) And by fast, I mean fast enough to be considered transparent: The user/programmer should not be aware that the decompression step took any time at all. Democoder parlance for this is “realtime” decompression.

So, a proper definition of the problem we’re trying to solve would be:

• We need to store data on disk in a form that takes up less space than its normal working state (ie. compression must actually occur — you can’t shrink 16 bytes out of a 64KB file and then call it a day)
• To be useful for most situations, we must be able to decompress the data with a minimum of work for the programmer (ie. one simple call with no special preparation or preexisting state required)
• Decompression must be fast enough to be considered transparent to the programmer (ie. take not much more time than a straight copy of the uncompressed data would)

For that last stipulation to be quantifiable, we need to decide what “fast enough” is. For our purposes, we will define “fast enough” to be roughly 4x slower than a straight memory copy (which 8088/8086 implements as a REP MOVSW). That kind of speed is respectable on any platform, so this seems like a fair starting point to gauge our progress.

The (de)Compression Method

So, what compression format will we write a decompression routine for? What formats lend themselves to fast decompression? What formats are even possible to decompress on our lowly 8088 with RAM that is measured in kilobytes instead of gigabytes? The answer, it turns out, is obvious — but only if you know what to look for. Like compression itself, we have to take a long time researching our problem before coming to a short conclusion.

A Quick Primer

Before picking a compression format to use, I will offer this extremely short primer to readers unfamilar with compression basics, so that the rest of the article will make sense. If you’re already familiar with what compression is, skip this part.

For our purposes, compression is the process of removing the redundancy out of a file so that it consists of only the unique parts and not the repeated or redundant parts. Consider this sentence:

She sells seashells by the seashore. (36 characters)

You can see that both “sells” and “seashells” share a sequence of letters (“ells “), as does “seashells” and “seashore” (“seash”). If we notice this and assign a custom code to both pieces (ie. AA=”ells ” and BB=”seash”), and then replace those portions of the sentence with the codes, we can shorten the space the sentence takes up:

She sAABBAAby the BBore. (24 characters)

Was actual compression achieved? Yes, because while the codes and what they represent take up space, we got to use them twice each. A code takes up two characters, but it represents five characters. So the sentence+codes take up 24 characters in compressed form, and the code definitions take up 10 more characters, so our total space usage is 34 characters compared to the original 36.

Saving two characters is nothing to write home about, but that was a single sentence. When you extend the idea to an entire novel, you begin to see how the coded sequences can get used more and more often until you have real space savings on your hands. We only used our codes twice, but across an entire novel, and combined with other codes, they’ll get used thousands of times, saving bytes here and there each time.

A Quick Primer (For Programmers)

There are many ways to compress data but the fastest methods are almost universally based on LZ77. LZ77 produces a compressed format that boils down to two types of sequences:

• Data that must be inserted directly into the target buffer (because we couldn’t find smaller ways to represent it), called LITERALS
• Data we already have somewhere that can be copied into the target buffer, called MATCHES

In our quest for high-speed decompression, we will only be considering LZ77-style compression methods. Not only are they fast to decode, but they’re also very easy to understand.

Again, the above is oversimplified for our purposes. For an excellent write-up on compression methods for low-resource platforms such as 8-bit CPUs, consult Pasi Ojala’s pucrunch page.

What To Compare Against

If you asked most DOS users in the early 1990s what the fastest compression/decompression software was, most of them would likely answer PKZIP. PKZIP was popular precisely because of its speed and reasonably good compression, and was so popular that it’s a format much of the world still uses more than 20 years later. PKWare released a Data Compression Library (DCL) of assembler-optimized routines for compression and decompression in 1991 for commercial use, which perform at roughly the same speed and ratio as PKZIP.

If this library exists, why are we bothering to write our own decompression routine? Why not just use the DCL for our needs? We’re writing our own because we want to explore what the state of the art can be for the 8088, and also because we aren’t sure what the state of the art actually is. Since the DCL was considered by many to be the best overall solution in terms of compression ratio and speed, that is the benchmark we will compare ourselves to. (It also helps that the DCL’s execution speed can be exactly timed just like we will be timing our code — more on timing later in this article.) In other words, if we can beat the DCL, we must be doing well.

(Compression nerds like myself may point out that the DCL uses Implode rather than PKZIP’s DEFLATE, but the two are nearly identical so I feel it is a fair comparison to use the DCL.)

The Current State Of The Art

There are many high-speed compression/decompression open-source libraries in use today. The oldest one still in wide use is UCL, written in the mid 1990s; there are newer ones written in the last five years that routinely compete for the top speed spot, such as QuickLZ, Snappy, and LZ4. My original idea was to examine all four of these to see what made the most sense to implement on the 8088. UCL was appealing because it had assembler code already written for the 8088/8086 as part of the library, but it was full of conditional branches and bit shifting/isolation, something the 8088 just isn’t very good at; there is also no published UCL data format (as far as I could find) so if I ran into issues, I’d likely be out of luck. QuickLZ is C code that assumes the underlying architecture is 32-bit or better, so while I could have written a decompression routine for QuickLZ data, it would have involved a lot of 32-bit memory values, adding to the complexity of the implementation. So both of these were out pretty quickly.

Snappy is Google’s library, and it looked very promising, mostly because it was implemented the way I would have written my own compression format: Simple code expansion, variable lengths, codes that are packed into data aligned by bytes. Source and format are fully documented, easy to understand, and in use by Google. No real downsides, right?

Snappy was going to be my first choice, but then I researched LZ4. As soon as I saw LZ4, I knew immediately it was the correct choice for the 8088 and got very excited. Not only was it the right choice for ease of implementation on 8088, but a unique design decision for the format lent itself to speed that I hadn’t seen before. Here’s what got me so excited:

Interleaved literal-match structure: This is the innovative discovery of LZ4. Yann Collet (the author of LZ4) realized that all compressed LZ77 data consisted of both literal runs and match runs, so rather than try to devise complicated codes that branch based on run type, he organzied the format so that there are always literals, then matches, then literals, then matches, etc. repeating until the end of the compressed data. This eliminates a lot of literal/matchcode-based branching. There are really only three branches ever required for decompressing LZ4: Two length calculations (and only if they’re needed), and a check to see if we’re out of codes to expand. Those three are the only major conditional branches! This is REALLY HUGE and is why LZ4 is at the top of the speed charts for nearly every platform.

64KB window size, 16-bit offsets: A 64K window size means that decompressing LZ4 doesn’t need any special handling — if you only have 64K to decompress (likely on our platform), you don’t have to worry about switching segments after the first 64K. And a 16-bit offset is a single word read, no unpacking. It’s like LZ4 was specifically created for Intel memory models, but it wasn’t — it was actually created in the very early 1990s for Yann’s HP48 calculator, which used a CPU based on nybbles. Which explains the code packing in the next advantage:

Simple codes: The literal lengths and match lengths are, if 14 or lower, stored as two nybbles in the same byte. That means both can be read with a single byte read and then cached as necessary. Even if the lengths are larger than 14, it only takes a few byte reads to get the full length. No wacky code unpacking like LZS or higher LZO levels.

No setup: All it takes to decompress one buffer into another is both buffers — no hash tables to initialize, no dictionary to load or build, no need to know the length of the decompressed size beforehand, etc.

Optimal Parsing: This is the secret sauce. It has nothing to do with how we will write our decompression routine, but it does have to do with decent compression ratios. And also speed, because LZ77-style compression schemes decompress in mostly linear time. Meaning, the smaller a file is, the faster it will decompress (ie. there is less actual work to do). LZ4_HC is the LZ4 variant that will take as much time as it needs to find the absolute perfect representation of the source data given the supplied encoding, and it is included in the LZ4 library and example programs.

All of the above makes LZ4 a winner for our architecture. But is it a winner for our workflow? If LZ4 was proprietary and difficult to use, that could be a problem. With stable vetted code, a BSD license, and Windows and Linux binaries for the compressor, it is a good fit for most cross-development workflows. (Most modern hobbyist retroprogramming is cross-development, with only speed-sensitive code requiring the actual hardware for testing.)

I’m not the only one to come to this conclusion; LZ4 is finding its way into tons of things from Apache Hadoop to Solaris kernels. Buy Yann a beer the next time you see him.

A Naive Implementation

Starting with only the format description and format diagram, I was able to code up a naive but functional decompressor in about half an hour in assembly. Normally I would have written it up in a HLL first to make sure my understanding was sound, but the format was so simple I thought that assembly would have been just as fast to code, and it was. I won’t reproduce 200 lines of assembler code here (a link to the finished code will come later), but it might help to know how I organized the routine:

DS:SI – Location of compressed data (the source)
ES:DI – Location of decompressed data (the target)
AX – scratch
BX – translation table
CX – loops, literal/match run lengths
DX – “cache” for literal and match length token
BP – size of compressed data (so we know when to stop!)

Here is what compressed LZ4 data looks like:

Yes, it really is this simple, which means you should be able to code up a naive implementation just from looking at the above diagram and reading the format explanation.

Assume Nothing!

The #1 suggestion I can give anyone trying to optimize code has nothing to do with either assembler or the 8088: Benchmark, benchmark, benchmark. Know exactly how fast your code runs on the hardware you are targeting. If you don’t, you don’t have a basis for comparison and are flying blind when you optimize.

Following my own advice, I wrote a test harness for the decompression code and linked it with a modified version of Michael Abrash’s Zen Timer, which gives nearly perfect microsecond timing of sections of code using the 8253 PIT built into (or emulated in) every PC. I then created a variety of “test subjects” to run through the decompressor, and made sure the test harness compared the output of the routine with the original source material to verify it wasn’t getting mangled on output. I chose three typical files to do most of my timing with — “typical” in that they were the most likely things I would be decompressing: Compiled code, plain-English text, and a hand-drawn CGA picture. Each had their strengths and weaknesses and provided different stresses to the routine: The picture had a lot of large matches, the text had a lot of small matches, and the binary didn’t have much of either. The picture also had a lot of “runs” where a single value was repeated. Measuring with the Zen Timer, all three datatypes took a total of 530434 microseconds (usecs), or 0.53 seconds, to decompress.

After checking the output was correct for my test data, I then proceeded to write a harness for PKWare’s DCL that did the same thing: Compressed all three test subjects into its own format, then decompressed all three to memory and recorded the decompression time. While I knew the DCL traded some size for speed, it was still the fastest general-purpose algorithm in use at the time so I was expecting some real competition. To my surprise, the DCL took 7.75 seconds to decompress the test data — over 14 times slower!

Thoroughly satisfied, I almost stopped right there, as my goal of beating the DCL was already accomplished. But I just couldn’t help thinking what a proper optimization would do, and exactly how both perform on more different types of data (was my sample size too small?), so I decided to go through with the optimization exercise anyway.

Chapter 27, In Which It Blows Up In My Face

I’m not going to cover every optimization in meticulous detail, because that would likely lose what few readers have made it to this point. However, one optimization attempt deserves special attention for the fact that it uncovered a bug in all Intel x86 CPUs made through the 80286, and illustrates the importance of having a proper test harness for your code.

Looking closer at how I chose to copy matched data around in the ES:DI destination buffer, it seemed wasteful to preserve DS, point DS to the match, then restore DS when I was only copying a handful of bytes in some cases. Most memory operations on 8088 can take a segment override, even the string operations in some cases, so with a single prefix you can turn this:

MOVSB           ; Copy a byte from DS:SI to ES:DI

…to this:

ES: MOVSB       ; Copy a byte from ES:SI to ES:DI

So, I got rid of the DS setup/restore code, changed REP MOVSB to REP ES: MOVSB, and patted myself on the back. Imagine my horror when a test run showed the output buffer had entire chunks that were either incorrect or in the wrong place. Worse, on a subsequent run, it was still wrong but in different places! I spent hours single-stepping through the code in a debugger, looking for things like uninitialized registers or incorrect compressed data. Finding nothing wrong, I then spent another hour testing my hard disk and RAM to see if my 30-yr-old IBM 5160 was finally starting to die.

Finding nothing, and noting that the code worked if I reverted my changes, I looked at REP ES: MOVSB and wondered if I was encoding it improperly. So I switched the order of the prefixes to ES: REP MOVSB. This resulted in the jaw-dropping result of mangling the entire output buffer. What the hell was going on?

With some research,  and clarification from Robert Wessel on comp.lang.asm.x86, I confirmed that all CPUs from the 8086 through the 80286 have a bug where returning from an interrupt only resumes the most recent prefix, not both. So after an interrupt, a REP ES: MOVSB continues as ES: MOVSB which copies a single byte and then stops. My second attempt made things worse, as an interrupted ES: REP MOVSB continues as REP MOVSB which, without the ES: override, copies data from the wrong location!

One way to fix this is to disable interrupts before you run that code, then re-enable when done, but I didn’t want to do that because you never know if someone will need interrupts to run while decompression is occuring. (I know I do, anyway.) I decided to work around the problem like this:

again:  REP ES: MOVSB   ; could get interrupted and bug out...
        JCXZ next       ; continue if REP completed
        LOOP @again     ; keep trying if REP never completed
next:

With a fix in place, and valid output confirmed, I timed the code. After all that heartache, I found that the overall routine slowed down by 8209 cycles, because the ES: override adds two cycles to every byte copy. The entire concept was a bust, so I reverted the code.

This just goes to show you why timing your code is important. In your head or on paper, some optimizations can seem like a win, but conditions outside your understanding or control can affect the final result.

It’s Obvious

Some optimizations, using the information provided in part 1 and part 2 of this guide, were easy to implement and usually didn’t result in a lot of code shuffling around. Here’s an abbreviated table of what was done and how many cycles it saved. (Remember, the three test cases took roughly 500000 usec to execute, so every 5000 usecs saved is 1% shaved off the execution time.)

Not everything I touched turned to gold. Here is a summary of optimization attempts that failed and were subsequently rolled back:

• Eliminating a “nothing to do” check (turns out the check-and-jump saved us some time after all)
• Eliminating the REP MOVSW optimization f0r literal copies (I thought the special handling wasted time, but the MOVSW gains are worth the trouble)
• Handling very short literal lengths manually with individual MOV’s (as opposed to letting REP MOVS do it)
• Handling very short match lengths manually with individual ES: MOVSW statements (as opposed to letting the REP MOVS do it)

With 43390 usecs saved, I was a man possessed: I wanted to speed it up further, but had run out of 8088 tricks. The only way I was going to continue on this journey was to gain further insight into the data I was dealing with, so I wrote an LZ4 data analyzer.

Not So Obvious

One of my favorite quotes from Zen of Assembly: “The best optimizer is between yours ears.” It alludes to dumb/blind optimization — what I was doing up until this point, working like a human peephole optimizer — being inferior to “smart” optimization, where you analyze the problem itself to come up with different ways to solve it.

I had made some assumptions about the structure and frequency of LZ4 data, so to confirm those assumptions, I wrote a simple program to walk through my compressed sample set and print out some stats: What the shortest lengths were, the longest, the averages, and anything else that might give me some ideas. Some sample output:

====== robotron.lz4 Compressed size: 21040 Decompressed size: 40704 =====
Literals: Minimum: 0 Average: 3 Maximum: 385 Total: 11336 # of events: 3139
Matches: Minimum: 4 Average: 9 Maximum: 512 Total: 28634 # of events: 3098
1-byte runs: Minimum: 4 Average: 31 Maximum: 217 Total: 593 # of events: 19
2-byte runs: Minimum: 4 Average: 6 Maximum: 14 Total: 141 # of events: 21

Looking at various outputs, I was struck by how many match offsets of 1 and 2 there were. Offsets that small, with lengths larger than the offsets, essentially translate into RLE sequences. LZ77 code usually deals with this by just copying the same bytes over and over, but x86 architecture has a string opcode that can do this more efficiently: REP STOS. Put a value into AL or AX, a count into CX, and off you go, filling memory with that value until the count runs out. What is especially notable about STOS is that it is faster than every other instruction for putting a value in memory, faster than even a raw REP MOVSW copy. So if you can use it, you can exceed memcopy speeds for short sections. Adding special handling for offsets of 1 and 2 and redirecting them to special-case code that used STOS resulted in a larger-than-expected 32281 usec savings all by itself — that’s the largest single savings out of all the attempts — and it also resulted in a special surprise which I’ll cover later.

After I ran out of ideas looking at the analyzer output, on a hunch I looked a the LZ4 format description a second time and noticed that, by design, the last five bytes are always literals. The current structure of the code would check for this exit condition, then process matches, then jump to the next token and continue. By unrolling some code to move the “are we done?” test to the end of the loop instead of the middle, I was able to combine both the test and the loop jump into a single instruction. This saved cycles every iteration, leading to the second largest savings at 10946 usecs.

The analyzer also confirmed some of my suspicions, such as most lengths were 14 or less and would rarely need to jump to the “build a longer length” section of the code.  By structuring the code such that the most common cases “fell through”, I was saving cycles each iteration.

So, when all was said and done, what was our time savings?

Results

After all of the above work (which incidentally took about ten hours of free time spread out over a few weeks), we managed to take an already fast routine and shave 95856 usecs from it — an 18% improvement from the already fast routine where we started.  Was 18% worth the effort? I definitely think so. It may not sound like much, but think about the savings like this: If a user would wait a full minute for decompression of a lot of data to finish, they now only have to wait 50 seconds. What was a 5-minute task now only takes 4 minutes. (And believe me, on an 8088, it wasn’t uncommon to be sitting in front of the machine waiting 5 minutes for something to finish.)

But that’s not the end of the story. Looking at more varied data than our three test subjects, we make an astonishing discovery (lower numbers are better):

For data that has a lot of 1-byte and 2-byte runs — runs that we specifically catch and handle with REP STOSW, as noted previously — we see that decompression exceeds REP MOVSW copy time. It is actually faster to decompress such data than it is to merely copy it. That is crazy.

I’m still smug from the speed victory over the DCL, so let’s kick it while it’s down and compare its speed to ours (lower numbers are better):

Now, all this doesn’t mean anything if the actual compression LZ4 provides is horrible. Thanks to optimal parsing, it’s not — in fact, compared to the DCL’s Implode, it’s extremely close. LZ4 and Implode differ by only a few percentage points:

Conclusion and Downloads

Hopefully I’ve illustrated why optimization can be worth the effort. Having transparant decompression can solve a lot of problems in any program that has to manage a lot of assets. And maybe I’m crazy, but I found it extremely fun to see just how much blood I could squeeze from a stone.

Can the routine be sped up further? It can, actually, but doing so would require a lot of additional work on the part of the programmer (such as storing the compressed data on the stack and using POP to retrieve data).  I also toyed with the idea of translating the LZ4 compressed data into a slightly different format that was more optimal to parse, but some quick tests showed that the transformed data is slightly larger as a result, which resulted in more data to read, which took up more time than the cycles saved by a more efficient implementation. About the only thing I can think of is that it might make sense to build literal/match lengths using REP SCASW looking for a word that wasn’t FFFF, but only if you knew had large lengths beforehand in your compressed data, which is unlikely.

So, as is, I believe this to be the very fastest implementation of LZ4 decompression possible on an 8088 CPU.  All this in 436 bytes of compiled code (180 for the code and 256 for a translation table), and without requiring the programmer to jump through hoops to use it.  This was a blast.

All code, binaries, and examples are available under what I’m calling the DSL (Demoscene License) so leech away. I’ve also made two versions of the Zen Timer available; an older package that extends support to C and Turbo Pascal, and a newer package that drops TP support but adds 32-bit protected mode C compiler support. Both packages have a “long timer” mode that can time periods larger than 54ms, albeit with a hair of jitter due to background interrupt noise.

Now, anyone want to help teach me fast 3D?  This demo won’t write itself, ya know.

Lost In Translation

Everyone knows that lookup tables trade memory for speed; if you can calculate all possible values for a particular operation, then performing the “calculation” is reduced to retrieving a value from the table. Intel thought of this when they were designing instructions to help with strings and created XLAT, which stands for “translate”. XLAT replaces AL with whatever AL is referencing in a table pointed to by DS:BX, and it does it in a single byte and takes only 11 cycles to do so. So you can replace this:

XOR AH,AH
MOV SI,AX
MOV AL,[BX+SI]

with this:

XLAT

Think of XLAT as “MOV AL,DS:[BX+AL]” (except that [BX+AL] is illegal on the 8088 and 8086.)  Intel meant for XLAT to be “useful for translating characters from one code to another, the classic example being ASCII to EBCDIC or the reverse,” but nothing says you have to use it with strings. I’ve used it as a replacement for isolating nybbles, because the 8088 is slow at shifting if you have to do it more than a few times. For example, I can replace this:

MOV CL,4
SHR AL,CL

with just XLAT, with BX pointing to a table that has all 256 values shifted right by 4.  While this burns 256 bytes for the table, the end result is a shift that finishes in half the time than Intel’s actual shift instruction.  This has saved my bacon in inner loops.

Branching Out

Conditional branches (jumping to another location based on the result of a test) are a necessary evil of most programming languages. Unfortunately, they cost you in two ways. The first is the obvious cost — a jump takes some time to perform, which is never quick. The second cost is more subtle, but just as valid: When a jump is performed, everything in the prefetch queue gets jettisoned, because it is no longer the correct stream of instructions to be executed. You really want the prefetch queue to be as full as possible most of the time, because when it isn’t, the CPU has to wait around for its BIU to grab instructions. This is less of a problem on the 8088 because it is hard to keep the queue full, but on the 8086 where it’s easy to keep the queue full, this is a real penalty.

For those who need an example of why this is important, let’s say you were a bit of a nerd who loved to research everything you did (not really a stretch because you’re reading this guide right now!), and your girlfriend wanted to visit an art museum. To save time, you fetched a set of art books to read on the way there before it was time to leave. But just before you leave, your girlfriend changes her mind and wants to go to a baseball game instead. The art books you’re carrying now don’t apply to where you’re going, so you dump them and start fetching books on baseball instead, and now you’re both going to be late because it takes time to fetch books. Your girlfriend changing her mind is a conditional branch being taken, and dumping the first set of books because they don’t apply any more is emptying the prefetch queue.

Because of this property, you should always be thinking of ways to eliminate branches. This can range from obvious solutions down to decidedly non-linear thinking to come up with a clever solution. This is difficult to visualize, so let’s look at a few examples. Let’s say you needed to set a register value to either 0 if it is even, or 4 if it was odd. Here’s a straightforward way to do this:

  TEST AL,1     ;test if first bit set
  JZ @even      ;if not, it is an even number
  MOV AL,4      ;odd number, set it to 4
  JMP @continue ;continue execution
@even:
  MOV AL,0      ;even number, set it to 0
@continue:      ;execution continues

This works, but no matter what the number is, you’re going to take a jump. Here’s a branchless way of doing the same thing, recognizing that 4 is a power of two:

AND AL,1        ;AL = either 0 or 1
SHL AL,1        ;Shift AL left (AL = AL * 2)
SHL AL,1        ;Shift AL left (AL = AL * 2)
                ;AL = 0 if it was originally even, 4 if odd

Let’s look at another common operation, converting a hex nybble value in AL to a printable ASCII character. First, the naive way:

  CMP AL,10     ;Digit or letter?
  JB @1         ;Jump if a digit
  ADD AL,7      ;Adjust for letters
@1:
  ADD AL,'0'    ;Convert to hex printable char

This works, sure, but if you stop and think about the problem for a few minutes, you might come up with a really cool branchless way to do the same thing, such as Norbert Juffa did:

CMP AL,10       ;if x<10, set CF=1
SBB AL,69h      ;0-9: 96h..9Fh, A-F: A1h..A6h
DAS             ;0-9: subtract 66h -> 30h-39h
                ;A-F: subtract 60h -> 41h-46h

(Of course, XLAT is the fastest way, but if space is at a premium or BX isn’t free, XLAT isn’t an option.

Let’s not forget about LOOP. LOOP is a branching instruction as well, and every time it loops, you lose 17 cycles. If you know you’ll be looping only a small number of times, unroll the loop. So this:

  MOV CX,4
@trans:
  LODSB         ;load byte from DS:SI to AL; inc SI
  XLAT          ;translate using table at DS:BX
  STOSB         ;store byte in AL to ES:DI; inc DI
  loop @trans   ;decrease CX and loop while CX != 0

…turns into this:

LODSB           ;load byte from DS:SI to AL; inc SI
XLAT            ;translate using table at DS:BX
STOSB           ;store byte in AL to ES:DI; inc DI
LODSB           ;(do it three more times)
XLAT
STOSB
LODSB
XLAT
STOSB
LODSB
XLAT
STOSB

By unrolling the loop, we not only save 17 cycles per jump taken, but we have a better chance of keeping the prefetch queue full (and all three of those are 1-byte opcodes, so it’s definitely full).

If you absolutely must branch, at least try to arrange your code so that the common cases “fall through” the conditional branch. This is because a conditional branch takes 4 cycles if the jump is not taken, but 16 if it jumps. Arranging your code properly requires advance knowledge of the problem you’re trying to solve. For example, if you are processing some data and only one value needs special handling, arrange your code like this:

CMP reg,immed
JE @specialhandling

That way, most of the common cases will “fall through” in 4 cycles, with only the special case needing the full 16 cycles to jump to a new location for handling.

Random Optimization Tip Firehose

Use spare registers to preserve/restore variables when possible instead of the stack. If you need to preserve something and AX is free, XCHG AX,reg is smaller and faster than PUSH reg; POP reg.

Did you know JCXZ exists? JCXZ is a freebie left over from Intel implementing LOOP, and it jumps if CX=0, or falls through if it isn’t. So you can replace this:

OR CX,CX        ;test if CX is 0
JZ @somewhere   ;jump if it is

with this:

JCXZ @somewhere

Putting It All Together

Here are some random examples that illustrate what we’ve been talking about.

Example 1: All programming can be viewed as an exercise in caching (Terje Mathisen)

Translating a buffer from one set of values to another? String ops and XLAT are your friend:

@trans:
  LODSB         ;load byte from DS:SI to AL; inc SI
  XLAT          ;translate using table at DS:BX
  STOSB         ;store byte in AL to ES:DI; inc DI
  loop @trans   ;decrease CX and loop while CX != 0

Hey wait a second, we have 16-bit versions of those! We can use the 16-bit versions to speed things up by loading and storing two bytes at a time:

@trans:
  LODSW         ;load word from DS:SI to AL; SI=SI+2
  XLAT          ;translate AL using table at DS:BX
  XCHG AH,AL    ;swap AH and AH
  XLAT          ;translate AL again
  XCHG AH,AL    ;swap AH and AH
  STOSW         ;store word in AL to ES:DI; DI=DI+2
  loop @trans   ;decrease CX and loop while CX != 0

While the 8086 gets a bigger impact from this due to being able to snag both bytes in 4 cycles, it is still a speed win on 8088 because we are loading and storing double the bytes with the same 1-byte load/store string instructions. And although we have three more instructions sandwiched inbetween, the prefetch queue helps out just enough that the overall speed is faster than the first example.

Example 2: Working in 16 bits, branchless

Copying a block of memory around and you don’t know how large the block will be beforehand? The general-purpose way is this:

MOV CX,immed
REP MOVSB

That handles both even and odd lengths, but as we saw from the previous example, it is faster to do things 16 bits at a time. Here’s a way to use the 16-bit string ops and also correctly handle odd lengths, and all without branches:

MOV CX,immed
SHR CX,1        ;CX = CX / 2; even/odd bit went into carry
REP MOVSW       ;Copy by words
ADC CX,CX       ;Add carry to CX; CX = 1 if odd, 0 if even
REP MOVSB       ;Will copy either 1 byte if odd or do nothing if even

Example 3: Instruction reordering

Try to reorganize code with the prefetch queue in mind if changing the order of instructions doesn’t affect what your code is trying to do. Here’s a sample “before” sequence:

INC SI
ADD SI,BX
MUL CX
CMP AL,3
JBE loc

In the above, we notice that MUL takes a long time, there is a prefetch-queue-emptying jump shortly after it, and the stuff above the MUL isn’t dependent on or affect MUL’s result. So by simply rearranging some code to come after the MUL…

MUL CX
INC SI
ADD SI,BX
CMP AL,3
JBE locr

…we can guarantee the prefetch queue will fill up while the slow MUL is executing, and then what’s in the queue gets executed faster before the queue eventually gets emptied.

Reference Materials and Further Reading

Thus ends our coverage of general tips and tricks. Believe it or not, this article is really only a beginner’s guide to 808x optimization; for the most comprehensive coverage of writing fast 808x code, I highly recommend Michael Abrash’s seminal work on the subject, Zen of Assembly Language. That book, as well as other helpful references for programming the 8088 (and CGA too) including Intel’s original documentation on the 8086, can be found inside ftp://ftp.oldskool.org/pub/misc/8088%20Programming.rar if you can’t find a dead tree copy elsewhere (everything in that archive is long out of print). All PDFs inside the archive have been OCR’d and are searchable.  (One note about the Intel’s 8086 documentation: All timings are for both byte and word transfers.  If you’re counting cycles on 8088, be sure to add 4 additional cycles for word transfers.)

There are some additional techniques for gaining even more speed on the IBM PC that are not specifically 8088-related, such as temporarily disabling DRAM refresh, copying memory using the DMA controller, or getting the entire machine into lockstep so that you can use software timing for events instead of interrupts. Those are highly specialized and beyond the scope of this guide, so I won’t be covering them any time soon, sorry.  (They’re also difficult to pull off without crashing the computer, and don’t work on machines that aren’t truly 100% compatible with the original IBM PC, so you’re not missing much.)

In Part 3, I’ll show you how I utilized all of the above knowledge to implement realtime decompression on the 8088 (and by “realtime” I mean “decompression faster than REP MOVSW”). Not a misprint! Stay tuned.

Before we begin, let me clarify that while I’m using “8088″ throughout most of this text, what I am writing applies equally to the 8086 as well. The 8086 and 8088 are functionally identical, with the 8086 being slightly faster due to a having a 16-bit bus and a larger prefetch queue, both of which are covered later. Despite the extra speed, what holds for 8088 optimization also applies to the 8086, so you can just equate the two for the remainder of this guide.

Contrary to what you might think about a CPU old enough to be Justin Bieber’s father, it is possible to wring acceptable speed out of an 8088 if you understand the situation most 8088s are forced into (the slow RAM of the IBM PC) and how to deal with it, as well as the CISC-like advantages the chip has. By understanding both, it is possible to write assembly code that can run faster than the best 6502 or Z80 code at similar clock speeds. Let’s look at both sides, and because I like hearing good news after bad, we’ll start with the bad news.

Disadvantages of the 8088

Slow RAM access. While other CPUs of the 1970s enjoy single-cycle access to a byte of memory, the 8088 takes 4 cycles to access a byte. The 8086 is a little better, and can access either one or two bytes in 4 cycles.

Tiny prefetch queue. The 8088 is made up of two halves, the Execution Unit (EU) and the Bus Interface Unit (BIU). They can work more or less independently, with the BIU grabbing the next instruction opcodes while the EU works on the previous ones. The only drawback to this arrangement is that the BIU only has a 4-byte buffer (the prefetch queue). So it can only “cache” up to 4 bytes in advance to feed the EU. As you can imagine, it is empty most of the time, because instructions execute faster than they can be fetched thanks to the sucky access time I mentioned in the previous paragraph. The 8086, again, is a little better; it has a 6-byte queue that it can usually keep full thanks to it’s faster RAM access.

Register specialization. The 8088 has four general-purpose registers, but all four of them are tied to specialized functions that you can’t use if you’re using them for, well, general purposes. For example, it’s possible to do a tight loop of some operations, but that uses CX as the counter, so you can’t use CX inside the loop.

Advantages of the 8088

CISC architecture. CISC was all the rage at Intel in 1976, so they built in some “metainstructions” that do several things with a single opcode. For example, MOVSB will copy a byte from DS:SI to ES:DI, then advance both SI and DI so you can do it again, and it does all three in much less time than if you did it yourself. Couple MOVSB with the REP prefix and you can do this repeatedly at high speed. XLAT is another one, which will replace a register value with a like value in a translation table. We’ll cover some of the better ones later in this guide.

Multiplication and division. Unlike most home computer CPUs of its era, the 8088 has a built in MUL and DIV. If you need to do 16-bit multiplies or divisions where the operands/divisors are not known beforehand, nothing beats them. (If you need to mul/div smaller values, however, they’re slower than they should be and you can usually beat them using Quarter Square Multiplication or shift-and-adding.)

Register specialization. Hey, didn’t we mention this previously as a negative? Yes, but sometimes the specialization works in our favor. For example, some opcode encodings that use the accumulator (AX) are only a single byte in size, and as we’ll see later on, smaller is better. Heck, let’s start now:

Smaller Is Better

Because it takes 4 cycles to read a byte, and because the prefetch queue is so tiny, smaller code is usually better. When writing or optimizing your code, keep the following chart handy, which lists all general-purpose 8088 1-byte opcodes. (This is not a full list, just a list of what I find most useful. For a full list, read the 8086 Family Users Manual from Intel.) If you can replace something you’re doing with one of these instructions, it’s almost always a win.

Something handy to print out and keep next to you while you code.

Accumulating Speed

Register specializations suck, but when it comes to the accumulator (AX), Intel built in optimized forms of instructions that are one byte shorter, one cycle faster, or both. Try to reorganize your code so that AX or AL can be used for these optimized forms, especially in an inner loop: (“accum” here means either AX or AL, and “immed” means any immediate value)

That last one is a doozy; XCHG reg,AX is 1 byte and 3 cycles. This was part of Intel’s plan to pair both LOCK and XCHG together as a way to implement atomic semaphores, so they optimized it in the silicon. (See page 2-18 of the 8086 Family Users Manual for details.)

It’s Called A Coprocessor For A Reason

If your project is going to work with floating point and needs both speed and accuracy, read this section. If you know you’ll never need that, skip to the next section.

Everyone knows that the 8087 math coprocessor is much faster than the 8088 if you need to perform IEEE floating point math operations (including square roots, tangents, arctangents, etc.). What people seem to forget is the word “coprocessor” in the name. It’s a true coprocessor, which means it can be crunching away on an operation in the background while the 8088 is off doing something else.

This is HUGE. If your program needs the accuracy of IEEE floating point and has to do a lot of difficult slow stuff with it, you can essentially get background computing of floating point for free. The sequence of operations for 8088 code is essentially this:

1. Load 8087′s stack with values
2. Give it a command
3. Go off and do whatever you want
4. When you’re ready for the result, issue a WAIT
5. Pop your result(s) off the 8087 stack

This almost feels like cheating.

Stringing Up The CPU

Another byproduct of Intel’s CISC rage were the string instructions. These are ludicrously powerful in the right circumstances, and you should use them whenever possible. Intel called them “string operations” because they were designed to assist in text string manipulation. Each of these opcodes are golden for three reasons:

• They are 1 byte long
• They perform multiple operations faster than the individual steps would take
• They can be automatically REPeated without using any jump/loop instructions

Here are the five string instructions:

LODSB – Load byte from DS:SI into AL, then advance SI
STOSB – Store byte in AL to ES:DI, then advance DI
MOVSB – Copy byte from DS:SI to ES:DI, then advance both SI and DI
SCASB – Load byte from ES:DI and compare it to AL (sets flags equal to a subtraction), then advance DI
CMPSB – Compare byte from DS:SI to byte at ES:DI (sets flags equal to a subtraction), then advance both SI and DI

But wait, that’s not all! You also get 16-bit versions of the same instructions!

LODSW – Load word from DS:SI into AL, then advance SI +2
STOSW – Store word in AX to ES:DI, then advance DI +2
MOVSW – Copy word from DS:SI to ES:DI, then advance both SI and DI +2
SCASW – Load word from ES:DI and compare it to AX (sets flags equal to a subtraction), then advance DI +2
CMPSW – Compare word from DS:SI to word at ES:DI (sets flags equal to a subtraction), then advance both SI and DI +2

These can be called individually, but really shine when they are used with a REP prefix, which will repeat them for CX times (meaning, REP will run a string instruction, then decrement CX, then if CX is 0 it will stop). The last two are used with additional repeat prefixes: REPE/REPZ (repeat while equal/zero) and REPNE/REPNZ (repeat while not equal/not zero), so that the loop ends (or continues) based on the result of the comparison.

There’s a lot you can make fun of the 8088 for, but nobody makes fun of the string instructions. If you need to copy memory around, scan a buffer for a value, fill a buffer to a certain value, or compare two buffers for equality, they are an order of magnitude faster than doing things the long way. I mean, seriously, a single CMPSW done the long way would look like this:

PUSH AX        ; CMPSW doesn't change any registers, so we can't either
PUSH BX
MOV AX,DS:[SI] ; Load DS:SI somewhere
ADD SI,2       ; Advance SI
MOV BX,ES:[DI] ; Load ES:DI somewhere
ADD DI,2       ; Advance DI
CMP AX,BX      ; Do the comparison (sets flags equal to subtraction)
POP BX         ; CMPSW doesn't change any registers, so we can't either
POP AX

Maybe now you’ll understand why I love the string opcodes so much!

This is the end of Part 1 of our crash course. In Part 2, I’ll continue with various tips and examples. In Part 3, I’ll present a case study that shows what kind of benefit you can realize from taking the time to optimize for speed.

I think it’s because the PC won the home computer wars. All of the other retro revivals are for platforms that most definitely ended; the PC just stayed relevant and, today, is what everyone has on their desks. Even Mac desktops run Intel hardware. So the IBM PC never really went away, and you can’t revive something that never really died.

I think the classic Mac “scene” had/has the same problem. I haven’t seen any retroware emulation compilations with classic B&W Mac software that exist for other platforms.

I find this fascinating to daydream about sometimes.  What if the IBM PC had not been built around the 8088?  How would have the personal computer industry progressed in the 1980s?  Who would the leader(s) be today?

The CPU possibility I keep drifting to is the 68000.  The IBM PC+8088 design had limitations that the industry spent nearly 15 years working around, the most infamous being limited to 1MB RAM (640KB typically available) and, adding insult to injury, only being able to access it one 64K segment at a time.  But while an address translation chip could have provided a flat 16:16 address space, there were some annoying limitations in the 8088 itself, such as only having four general-purpose 16-bit registers (and of those four, each had a specialization married to some sets of instructions, so you couldn’t use them as flexibly as you would have liked).  The 68000 by comparison was immensely more powerful and had none of these limitations:  It had eight 32-bit general-purpose registers, and another eight 32-bit addressing registers.  On top of that, the 68000 had a flat 16MB address space (no segments!)  The 68000 had some drawbacks too (big-endian architecture, misaligned code would crash the CPU) but the core architecture was so much more powerful that it would have drastically changed how the IBM PC was perceived and used.

However, it would have also drastically changed how the IBM PC was marketed.  The original price of the PC was already expensive at launch ($1565 without any monitor or disk drives, about $3800 today) but designing the machine around the 68000 would have required more expensive components.  This, along with IBM’s existing familiarity with the i8086 family which would shave months off the time to market, was ultimately behind IBM’s decision to abandon all others and go with the i8088.

The 68000 was used in more powerful home computers that came after the IBM PC, such as the Apple Macintosh, Atari ST, and Amiga 1000.  On launch day for each of those computers, they greatly outshone the original IBM PC.  (BTW, the Amiga launch is particularly impressive and fascinating to watch although I can only find a small snippet of the full launch video online.)  If the 68000 had been used in the PC, would those later machines have ever existed?  While it’s interesting to fantasize that a 68000-based PC would have prevented the Atari ST and Amiga from ever existing, I think a more realistic outcome would be that the PC would have been treated like the Apple Lisa:  Powerful, but way too expensive, and perceived as a high-end business workstation that would find its way into some niche business markets but never considered a computer for the home.  (As to what computer(s) would have triumphed in the PC’s absence, that’s a What If? for another day.)

What about other potential CPUs?  What do you think might have happened to the IBM PC if it had been based on the same CPU as found in the Commodore 64 or the ZX Spectrum?

Update 1/7/2013: herm1t further smashed the record, down to 11 bytes!

Update 1/7/2013 @ 18:00: Peter bounces back and reclaims the record with 10 bytes! It kind-of breaks my target platform (uses an undocumented opcode that only works on Intel processors) but hey, a record’s a record! I’ve updated the article below.

In the past, when I’ve had a democoding breakthrough, I kept quiet and either used my discovery in a production, or just bragged to my demoscene friends privately.  However, my opportunities to achieve democoding “world firsts” are just about gone, and size coding compos seem to be dead, so I’ve decided to just write a blog post about what I’ve done instead: I’ve written a maze generator in only 13 bytes of x86 machine code.

To understand my 13b achievement, you need to be familiar with 10 PRINT.  10 PRINT is the name given to a single short line of Commodore 64 BASIC code that, when run, appears to generate a never-ending complicated maze.  The output isn’t really a true maze — it has no proper beginning and end, and some avenues lead nowhere — but it looks like one because our brains are the most powerful pattern recognition engines on the planet.  The appreciation for 10 PRINT comes from the seemingly infinite complexity produced from a very simple single line of code.  For you non-technical people, consider 10 PRINT the programming equivalent of a haiku.

10 PRINT is also the name of a 324-page book dedicated to the code, its output, and 300 other pages that explore code, art, perception, culture, randomness, mazes, and almost anything else the authors thought of when pondering the code.  (The book is legally free for download, although you can purchase a copy to benefit the non-profit organizations PLAYPOWER and The MIT Press, so if the first hundred pages interest you, buy the book to help them out.)  It is hundreds of pages more than I would ever write about fascinating minutiae and borders on stonergazing, but the authors have done such a thorough job and made the subject material accessible to so many people that it’s really hard for me to criticize them.  I recommend you give it a read, as there’s something in there for everyone and the presentation of the material is top-notch.

42 bytes

Anyway, back to my code.  I was reading the 10 PRINT book and got to page 242 where it explains that the BASIC code can be represented even smaller as C64 machine code, and challenges the reader to improve on it.  Sure enough, checking 4-Mat’s and Wisdom’s attempts, the smallest possible version clocks in at 15 bytes, which is pretty amazing.  For kicks, I wondered what an x86 equivalent would look like, so I wrote up an x86 version of 10 PRINT.  Here’s the easy-to-follow, friendly attempt:

; Simple program to duplicate "10 PRINT" output, also known as DataGlyphs.
; This version "plays nice" by grabbing from a "true" random data source
; and not making any assumptions about register contents.
; We also change to a low-resolution mode to have a friendlier aspect ratio
; and exit cleanly once the "maze" is done printing.

init:   mov     ax,0001h
        int     10h             ;force 40x25 color
        xor     bh,bh           ;we just set video, so set vidpage=0
        mov     cx,(40*23)      ;number of chars to write

getrnd: xor     ax,ax           ;to get al=0
        pushf
        cli                     ;disable interrupts, timer read must be atomic
        out     43h,al          ;latch count register
        in      al,40h          ;read lobyte of count
        mov     ah,al
        in      al,40h          ;read hibyte of count
        popf                    ;enable interrupts
        xchg    ah,al           ;fixup; now contains 8253 raw 16-bit word

pickch: shr     ax,1            ;BIOSes init in count-by-2, 1st bit always lo
        shr     ax,1            ;carry = "random" bit
        mov     al,'\'
        jc      writec          ;if bit set, write immediately
        mov     al,'/'          ;otherwise, choose other character

writec: mov     ah,0eh          ;write char in teletype mode
        int     10h
        loop    getrnd

        int     20h             ;exit back to DOS

This code is well-behaved; it doesn’t make assumptions about the state of the processor, and it exits cleanly (even though the original 10 PRINT code loops forever).  It also manages to come up with a halfway decent random number, taken from the hardware timer which counts from 65535 down to 0 every second.  Compiled with a86, it results in a 42-byte .COM file.  The size suffers compared to the C64 version in two major places: The random number generation, and picking the character.  In picking the character, the C64 has an advantage in that both slashes are right next to each other in PETSCII values, so if you have a random bit, you just add it to the first slash and you’ll get either no change, or the second slash.  No such luck on the PC; the ASCII values for both slashes are far apart, so I use a conditional branch that picks one or the other based on the random bit.

Because I’m used to optimizing for speed, I decided to eliminate the branch where the slash character is chosen to see what it looked like (branches are a speed penalty on most CPUs).  Despite being far away in value, the bit patterns for both slashes were only different by a single bit once you shifted one of them:

"/"=2f and "\"=5c
2f=00101111
5c=01011100

This led to replacing the pick-a-character chunk “pickch” with this:

pickch: shr     ax,1            ;BIOSes init in count-by-2, 1st bit always low
        push    cx              ;Save our loop counter for later
        mov     cl,al           ;copy random value to cl
        and     cl,1            ;cl is now either 0 or 1
        mov     al,'\'          ;seed character as '\'
        shr     al,cl
        or      al,cl           ;if cl=1, '\' is now '/', otherwise no change

While pleased with myself, this increased the size to 48 bytes, so I discarded the idea.

Another optimization thought was to change the character print routine (“writec”) to something that just stuck bytes directly into video memory.  That shaved two bytes off the print routine, but it took 4 bytes to set up (5 bytes if I wanted to stay compatible with the 8086), so it was a net loss that also got discarded.

25 bytes

I then turned my attention to the biggest chunk in the code, the random number generator.  Running the program a few times, it was clear that the quality of randomness didn’t need to be very high; just enough to avoid obvious runs or patterns.  This meant I could get my random data from almost anywhere, so I replaced the entire “getrnd” block with a single instruction: LODSB.  This retrieves a byte from memory and then advances position, so if you call it repeatedly you get a sequence of bytes from memory, one after the other.  The sequence I got back was from wherever DS:SI points to on program startup; for a DOS .COM file, it points to the current program code.  So my “random” stream of bytes was actually the compiled code itself (ha!) plus whatever remnants of prior programs were still left in memory.  This led to a massive reduction down to 25 bytes.

15 bytes

When I saw 25 bytes, I got excited because it was now in the realm of possibility to catch up to the C64 version.  There was still a lot of fluff in the program, so I took advantage of more “safe” startup assumptions and jettisoned nice-but-unnecessary things like changing the video mode, a clean exit (it loops forever, just like the original), and initializing some registers.  In the end, it looked like this:

init:   mov     ah,0eh          ;prime 10h for "write char in teletype mode"

getrnd: lodsb                   ;read a byte from wherever DS:SI points at

pickch: shr     al,1            ;carry = "random" bit
        mov     al,'\'
        jc      writec          ;if bit set, write immediately
        mov     al,'/'          ;otherwise, choose other character

writec: int     10h             ;write the character
        jmp     getrnd          ;loop endlessly

This compiled to 15 bytes.  Success!

13 bytes

I’d matched the C64 version, but looking at the “pickch” block I kept feeling like I could improve it.  That block starts by setting the carry flag to a random value, but the 8086 has a parity flag that is automatically set by some operations, and you can jump based on its state.  Unfortunately, LODSB does not set any flags.  Any math operation will set the parity flag, but most math operations would take up additional space and rob me of space savings.  If only there was a single-byte instruction that could do what LODSB does and set the parity flag based on the input…

Turns out, there is!  SCAS is an instruction that loads a byte and then compares it with AL, and then sets flags based on the comparison.  Like LODSB, it advances through memory, and it was meant to be used in a REPeat loop — but there’s nothing preventing you from using it outside of a loop.  So I did, and here is the final result:

; Simple program to duplicate "10 PRINT" output, sometimes called DataGlyphs.
; Written by Trixter / Hornet, trixter@oldskool.org, 20121214
; For more information about "10 PRINT", consult http://10print.org

init:   mov     ah,0eh          ;prime 10h for "write char in teletype mode"

getrnd: scasb                   ;read from ES:DI points and compare to AL
                                ;this sets flags similar to a subtraction

pickch: mov     al,'\'          ;choose a character
        jp      writec          ;if parity bit set, write immediately
        mov     al,'/'          ;otherwise, choose other character

writec: int     10h             ;write the character
        jmp     getrnd          ;loop endlessly

And there you have it:  13 bytes.  As an added bonus, it works on an old IBM PC too, since it doesn’t use any 80186+ instructions.  Sample output below (if system is booted in 40-column mode, just like a C64):  (click on the picture directly if your browser isn’t displaying it properly)

I believe this is as small as it can get on x86, courtesy of Intel CISC.  I also think this is a 10 PRINT size world record for any platform, but if I’m wrong, please let me know.

12 bytes

As the update above shows, Peter Ferrie was able to improve my design by a single byte.  His code is as follows, with my annotations:

init:   mov     ax,0e5ch        ;Load AH with "write char" cmd and AL with '\'
        scasb                   ;read from where ES:DI points and compare to AL
                                ;this sets flags similar to a subtraction
pickch: jp      writec          ;If parity set, jump to writing char in AL
        mov     al,’/’          ;otherwise, choose other slash char
writec: int     10h             ;write char in AL
        jmp     init            ;loop endlessly

Bravo, sir. But even further than that, using DOS’s “write character” int 29h, herm1t got it down to:

11 bytes

init:   mov     al, '\'
        scasb
        jp      writec
        mov     al, '/'
writec:	int     29h
        jmp     init

Clever. At the time I wrote the original 13-byte version, I’d been working on self-booting code with no OS, so I wasn’t considering DOS at any point. My loss is herm1t’s gain!

But wait, that’s not all:

10 bytes

init:   scasb
        salc
        and al,'\'-'/'
        add al,'/'
        int 29h
        jmp init

Peter bounces back with another byte shaved! (if you run Intel CPUs — trying to run this on a non-intel CPU such as an NEC V20 or V30 will hang the machine).

In 1987, I was a junior in high school lucky enough to work part-time at Babbage’s (a great source of software, something I’ve written about before) in Northbrook Court, an upscale mall in the North Shore.  I would take a bus directly from high school to get there at 4pm, work two hours, and catch the last bus home at 6pm when I got off work.  On this particular Wednesday, however, I was careless with money and didn’t have enough cash for the bus home, and by the time I thought to ask someone for some money, the last bus had left.  My next option was to use what little change I had left to call my house for a ride home, but both parents were at a Christmas party, so there was nobody there to answer the phone.  Adding insult to injury, we didn’t own an answering machine, so I couldn’t even tell them I was stranded at the mall.  I wasn’t sure what to do, so I decided to sit in the food court and do homework until I had a better idea.

Eventually it was closing time, and the mall patrons were instructed to leave.  I waited inside one of the double-door entrances to the mall, still not really sure of how I was going to get home.  I did some more homework, and at 10pm a janitor walked into the vestibule and asked me what was going on.  I told him I was “waiting for my ride” to buy some more time: My plan at that point was to wait until my parents came home, call them collect, and get picked up.  It was very cold that night, so instead of kicking me out, he grunted and locked the outer set of doors so that they only opened from the inside.  The inner set of doors — the doors to the inside of the mall — he left unlocked.

He was expecting me to leave the mall, not go back inside.  But since the vestibule was starting to get cold, at 10:30pm I did precisely that: I went back inside the mall.  My first goal was to look for a comfortable place to sit down and wait until midnight so I could try calling home again.  The drawback was that I couldn’t be seen, because if they caught me I knew I really would be ejected from the mall, and I wasn’t looking forward to being outside in -15C/0F weather with no money and no ride.  So I had to find a way to stay hidden for a couple of hours.

My first thought was to return to the food court, as it was dark and there were many corners I could sit in.  However, the Army of Janitors had emerged from the woodwork and were cleaning that area, so that was out.  My next thought was that I could hide in a bathroom, but I dismissed that idea after a few minutes because 1. they were filthy after a full day of patronage, and 2. I was terrified of getting surprised by someone and having us both nearly die of shock.  I briefly entertained going into the back corridors to wait things out, but since that’s where all the garbage goes, I was sure I’d be found by the janitorial staff.  (Also, it looks very very bad if you are caught in an area of the back corridors that you have no business being in, since it looks like you’re scoping the joint and/or attempting to burglarize a store from its inside entrance.  When I worked at Babbage’s, we were instructed to take the trash to the dumpster and then immediately return, no detours.)  I couldn’t think of anything else, so I gave up and decided to throw myself on the mercy of the janitors and hope that one of them understood English well enough to cut me a break.  I started walking to the food court… which took me past the Santa’s Workshop display, where the mall Santa would sit kids on his knee and charge for photos.

I don’t know what it looks like today, but back then, the Santa display was gigantic.  It was half-surrounded by fake trees, large candy canes, plastic snowmen and reindeer, and oversized mock presents, all with a long winding path to where Santa sits.  The chair itself had a back almost 10 feet tall, was painted in gold, and upholstered in red velveteen and looked more like a throne than a chair.  And because that wasn’t quite good enough for an upscale mall, the chair was set inside an elaborate  ”workshop” mockup that was one-and-a-half stories tall and about 15 feet deep to the back and sides.

If I had been one year younger or one year older, I would have walked right on past it and stuck to the plan.  But I was 16, moody, and a little rebellious.  I decided to wait for midnight to roll around sitting in Santa’s Chair.

Sitting perfectly still, with my red ski jacket on, I blended into the chair and silently watched the janitorial staff clean the mall: Buffing the floors, washing the storefront windows, emptying the garbage.  One woman got within 15 feet of me and never saw me; I stifled a chuckle.  I was an invisible teenage king sitting on my yuletide throne.

Midnight rolled around, and most of the overhead lighting in the mall shut off.  The janitors disappeared, and I made my way back to the pay phone in the vestibule.  Propping the inner door open with my backpack for fear of getting locked out, I made a collect call to home.  Nobody answered!  I guessed my parents were having a good time at the Christmas party and hadn’t come home yet, so I made my way back to Santa’s chair to wait a little while longer.  However, I was pretty tired by this time and worried I was going to fall asleep, so with the janitorial staff gone I decided to walk around a little bit.  Almost immediately, I made a discovery: The “workshop” behind Santa’s chair had a tiny functional room in it!  The coat rack in the room gave away its true purpose as a changing room for “Santa”.  There was also a folding chair, and a small table with a box of chocolate gold coins for “Santa” to line his pockets (which I immediately proceeded to fill my backpack with).  This made for a much nicer spot to wait half an hour to call my parents again, so I sat down in the folding chair.

At this point, you should be able to infer what happened next.  It was past midnight, I was tired, I had nothing to do, I was in a dark room, and the folding chair was really uncomfortable to sit in.  Sheer teenage stupidity set in, and I decided to take a small nap before calling home again.

When I woke up, I had to pee something fierce, so I made my way to the bathrooms by the food court.  As I was relieving myself, the very thing I tried to prevent earlier happened as a janitor walked in on me and we both nearly crapped our pants.  He stuttered in broken English “You!  You the boy!  This way!” and he led me to the security office where I learned three very troubling things:

1. The mall had a security office
2. I was not asleep for 30 minutes, but rather had slept for five hours
3. The security guard informed me that, three hours earlier, he and my father had been roaming the mall, looking for me and calling out my name

I was horrified.  He called my father to come pick me up.  As I waited in the security office, I tried to come up with a reasonable excuse for what had happened.  I had created a whole new level of stupid for not realizing earlier that all malls probably have a damn security office — I should have just gone there at 10pm and kept calling home every half hour!  I had no idea what I was going to tell my father, who probably couldn’t understand how I had slept through people yelling my name all over an empty mall.  I sleep like the dead!  (Seriously, ask my wife!)

The ride home was mostly silent.  I got into bed, slept until 10am, then got to school shortly before lunch and finished the rest of the school day.  When I went back to work that day, my manager read me the riot act, more out of concern for my safety than actual anger.  ”You could have been lying in a ditch!” is a phrase I still remember to this day.  That night, my family didn’t talk about what had happened.  We were somewhat dysfunctional back then, following an if-we-ignore-it-then-it-didn’t-happen philosophy.  I received no punishment, other than what I gave myself over the next few days.

So, what was the outcome of this little stunt?  Other than a neat story to tell my friends the next day, I didn’t think there was any outcome at all until I mentioned it four years later to my father-in-law.  I was telling the story during a car ride to pass the time, and he turned around at a stoplight and said, “That was you?”  He was a mall manager at the time, and it turns out that my little stunt led to discussion in the mall retail industry, and later, policy reform in how large malls handle security.  As a result of my antics, floor displays no longer have changing rooms/hiding places, security guards must walk the floor routinely (I never once saw the security guard walking around that night), and all vestibule doors are locked and checked every walkaround.

I always made sure I had bus money after that.  Oh, and my parents bought an answering machine.

Most people trying to comprehend what this is like would probably imagine something flamboyant and animated, such as Eric Schwartz’s tribute to the Amiga.  While such characterizations are creative and nice, that’s not what gets me laughing.  What does it for me is a computer that acts like a fallible human.  Here’s the post that got me laughing, paraphrased slightly (for the uninitiated, ext4 is a method of storing files in a Linux system, and the context is an application “talking” to the operating system):

I don’t quite trust ext4 for writes.

app: Hey, can you write this data out to
ext4: DONE!
app: Uhh, that wasn’t long enough to actually write the data.
ext4: Sure it was! I’m super faGRRRRRRRRRRRRRst at writing too.
app: Wait — did you just cache that write and report it written, but then not actually write it to disk until 30 seconds later?
ext4: Yeah, so?

I routinely do this sometimes when dealing with a unix server that is hurting, such as having so many spawned threads due to an unforseen condition that there are several times more threads running than there are CPUs to handle them.  I imagine each CPU as a juggler frantically trying to keep 20 pins in the air at once, sweating profusely, and calling out to the other CPUs for help only to have them yell back they are just as screwed as he is.

Does anyone else do this, or is it just me?

When people ask me what autism is like, my one-shot answer used to be, “Ever seen Rain Man?  It’s kind-of like that, but without a 25 million-dollar budget.”  After last night, however, I have a new answer:  Living with an autistic child or sibling is damn near exactly as portrayed in The Black Balloon, a movie written and directed by a filmmaker who grew up with two autistic brothers.  The movie is full of insights and situations that come from that real place, such as what meltdowns are and what prompts them, reward systems, and how society reacts to something it doesn’t understand.

I highly recommend The Black Balloon if you want a quick introduction to what families with autism go through.  Although it can be hard to watch at times for those already familiar with autism, I recommend they watch it as well, because it reminds you how you’re not alone, and that life on all spectrum levels can be fulfilling.  The Black Balloon is available for Instant Play streaming on Netflix.  A trailer for it is on YouTube.

A few times a year, I hear this from people learning for the first time about some of the difficulties our family has been through: “How brave!  Where do you find the strength?  How do you do it?”  The answer should be obvious — because we have to!  Spoken with a smile and a laugh, of course.  And that is the feeling perfectly portrayed in The Black Balloon.

We still have some left, so we’ve lowered the price for Black Friday (and the rest of the holiday buying season) down to $17 for the US and $19 for Canada.  (If overseas, you can try Maz Sound, CSW Verlag, and Amazon UK for the best deals.)

My verdict: Josh quite absolutely copied entire chunks of MODPLAY for use in his own code.

When accused, Josh’s paraphrased explanation at the time was “it’s a modplayer, of course some things are going to be the same from player to player”, but that only makes sense at a high level. Yes, the basics of playing a mod are the same across all players, such as interpreting the data structures and effects, mixing four channels into a single output channel, etc. But the devil is in the details, and it is the details that point to copying. The code is not 100% identical at an assembler level, but there are some very unique choices Mark made in the original MODPLAY that mysteriously show up in Josh’s source, such as internal housekeeping, and the inner mixing loop.

The mixing loop, I have found, is a good “fingerprint” for a modplayer — almost every author implements it in a different way. There are bare-metal fastest-possible implementation loops (such as the self- modifying fixed-length code of Carlo‘s Galaxy Player), loops optimized for low memory usage (such as MODPLAY’s loop which uses a MUL in the inner loop), loops that trade memory for speed (such as TANTRAKR which uses a 128K lookup table to eliminate MUL), and all targets inbetween. 4 channels or N channels? 32-bit mixing or 16-bit mixing? Logarithmic or linear volume tables? Cubic interpolation or linear interpolation? Just about every x86 mixing modplayer is different. And the choice Mark Cox made — utilizing a MUL in the inner loop and making heavy use of memory variables — was because he knew his target was a 286 or later and could handle it. You can also tell from the MODPLAY disassembly that Mark was working in a vacuum, because his performance-sensitive code is nowhere near as optimized as it could be (sorry Mark!). Looking at Jensen’s source, you can see exactly the same methods at play, including the inner loop (although Jensen made a few tiny 1- and 2-opcode optimization changes here and there).

As much as I love the inner loop as a fingerprint, the most convincing evidence that copying occurred is actually in the most boring sections of both programs: General housekeeping (things like program startup/initialization, maintaining player state, etc.) Mark does something in MODPLAY that struck me as odd; he calls two tiny procedures to set some variables based on whether or not a mod has 15 or 31 instruments (labels are mine; I don’t have access to Mark’s source code):

sub_1E23 proc near
  mov sequence_offset, 1D8h
  mov word_124, 258h
  mov header_size, 258h
  mov num_inst, 0Fh
  retn
sub_1E23 endp

sub_1E3C proc near
  mov sequence_offset, 3B8h
  mov word_124, 438h
  mov header_size, 43Ch
  mov num_inst, 1Fh
  retn
sub_1E3C endp

That’s a weird way to set some vars. You don’t normally call a tiny procedure simply to set a handful of memory variables to fixed values; usually, you just set the values directly. I don’t know Mark’s motivation for doing it this way. This is a very unusual section of code that I wouldn’t expect to see again…

…and yet, Jensen follows the very same odd practice in PPM.ASM:

proc sd_Set15Ins
 uses ds
 mov ax,@data
 mov ds,ax
 mov [Word NumberInstruments],15
 mov [Word SequenceOffset],01D8h
 mov [Word HeaderSize],0258h
 ret
endp sd_Set15Ins

proc sd_Set31Ins
 uses ds
 mov ax,@data
 mov ds,ax
 mov [Word NumberInstruments],31
 mov [Word SequenceOffset],03B8h
 mov [Word HeaderSize],043Ch
 ret
endp sd_Set31Ins

Again, it’s not the exact instructions that are copied or their order, it’s that entire concepts were copied, and because Mark implemented them in a unique way, they stand out in Jensen’s code.

As someone who has done a lot of cracking and reverse-engineering of vintage software — including, I’m ashamed to say, outright theft of other people’s code — I sense other subtle touches in Jensen’s released source that indicate large sections of it are not his original work. The most obvious are switching between hex and decimal values as a basic notation from procedure to procedure; the copied chunks favor hexidecimal notation, while the original code favors decimal. Also, all throughout the code some lines are indented using 8-character tab stops while other lines use spaces for padding, which is indicative of generating a file using one padding style and then editing it using another style, which would not typically happen if you wrote all the code from scratch.

In the last 20 years, Jensen has remained a professional programmer, and just as my skills and integrity have increased over that time, I have no doubt that his have increased as well. It is not my intention to libel Jensen as a whole; I simply wish to set the record straight regarding only one of his claims.

1. I gained weight during last month’s experiment.
2. I am winded going up stairs.
3. My kids have never known me without a roll of fat around my neck.

I am 41 years old, 6’2″, and weight 247 pounds. That’s at least 45 pounds too much, and about 65 pounds away from looking normal. I have tried very many things, but lost the willpower to follow through with all of them: A gym membership, home calisthenics (even computer-aided), and the Couch-to-5K program. All have failed, save for a wonderful 3-month period in 2004 where I successfully trained for the Run Hit Wonder and got a front-row position at the following Devo concert.

As I discovered last month just how much anxiety I keep at bay, I was reminded of The Walking Man. Walking Man is a former neighbor of ours who used to go for several walks a day. He was a retired nuclear physicist  and had the unofficial reputation of being quite brilliant. Once or twice a day, you could look outside and see him either coming or going. However, as the years went by, I started seeing him in more places around town: The park at the end of the subdivision; the Trader Joes 10 blocks away; the bike trail near the river. It wasn’t until I saw him walking downtown, several miles away, that I realized what he was doing: He wasn’t taking several walks per day, but rather taking one massive walk that lasted hours per day. Downtown is 5 miles away, so he walked a minimum of three hours every day. Not surprisingly, he was a little on the thin side.

I hold no illusion that I am like a brilliant nuclear physicist, but I can identify with him on some level. Some neighbors told me his walking was a way for him to think and sort out whatever was going on in his life; the physical fitness was just a side benefit. I am inspired, and am going to emulate The Walking Man so that I can kill two birds with one stone (anxiety and exercise). However, to make it as easy as possible to perform, I will be doing it indoors on a level treadmill. The treadmill is in front of the television, so I will have some entertainment while I walk. Finally, I’ll be walking in whatever clothes I am currently wearing. All of the previous excuses with other methods — driving to gym, changing into special clothes, dealing with the elements — are gone. I really have no other excuses!

So what’s the plan?

• On day 1, perform my treadmill’s fitness test with the aid of a heart monitor. Record result.
• Walk 5K (3.1 miles) every single day for 30 days. Initial walking speed will be 3.1 miles per hour, but after the first week I will increase the speed slightly to match the duration of the TV series I plan to watch while walking. Apply “body glide” strategically to prevent chafing and welts.
• Weight and percentage body fat will be recorded every single morning before my first meal.
• On day 30, perform fitness test again and compare result with day 1.

What do I hope will happen?

• More effective sleep
• Less anxious
• Higher performance at my day job
• Weight loss
• Better mood/less depression

This is a lot less pressure than Couch-to-5k and I can’t honestly see any downsides (except possibly blisters).  I start tomorrow morning with the fitness test.

There’s a lot to update here, so I’ll start with the summary conclusions first, then work my way down to the statistics, and finally end this post with entries from my daily log that I recorded as the experiment progressed.  If you only have one minute to read this, you’ll get the meaty bits first.

Conclusions

I gained two major personal insights as a result of this experiment.  The first was that I am my own worst enemy.  Much of the email that would take up my time was actually opted-in by me: Sale notifications, new things on ebay I might like, forum replies, etc. Over the next few days I am going to take a hard look at what I’m receiving and pare it down. I’m not completely to blame; over 1000 emails were spam that got through my filters, so I’ll have to look into more filtering beyond what I already have (I currently use spamassassin, greylisting, and bayesian filtering in my MUA).

The second insight was the discovery that I have more nervous energy than I realized, and am more anxious than I should be. As I get older, I lack the ability to keep it in check.  I find aging interesting from the standpoint of an engineer:  While the amplitude of my mood swings has decreased over the years, the frequency has increased.  I used to deal with anxiety using programming, action games, IRC, web browsing, email, online communities, etc. to distract me.  With most of those removed this past month, I turned to other avenues such as nervous eating (I gained two pounds) and snapping at my family during particularly low periods. :-(

The solution, I think, is not to go back on the sauce completely, but regulate it. Work smarter, not harder.  Keep doing email, but only useful email — all these automated notifications and alerts simply must go away.  Keep reading news, but pare down my newsfeed to just the bare essentials (sorry woot-offs, don’t let the door hit you on the way out).  And I’m giving serious thought to having some days (like the weekend) be “no-internet” days.

Did I get more meaningful things done in my free time, or did I just find other ways to waste it? Both meaningful (programming) and meaningless (tv, movies) activities increased at roughly the same amount.  Did I spend more time with family? A bit more, but there was no significant change.  Did I get around to reading Neuromancer? Nope. In fact, I didn’t read any books at all, other than technical information on FAT12/FAT16, and “The Pro Audio Spectrum” book sent to me by my good friend Mike Melanson.  Did I miss facebook? Not one bit. Did facebook miss me? Not really :-) but that doesn’t bother me either.  Was I inundated with spam? Despite my predictions, spam did not increase exponentially as the spambots received confirmation that  I was a real person through my vacation message auto-replies. MAILER DAEMON bounces were less than 6% of the email I received.

Am I glad I did it? Absolutely. I also brainstormed at least 12 more month-long experiments to try, half of which deal with my physical health and the other half with my mental health. (I’ll stay online for those, I promise!)

Experiment Statistics

Health: Weight gained/lost: +2 pounds :-(

Email:

• For the month of October, I received 5690 emails, 3913 (69%) of which were obvious spam. Over 1000 more were non-obvious spam — spamassassin didn’t catch them, and neither did the bayesian filtering I have in my MUA.
• Of the remaining 1777 emails, 8 of the top 10 senders were services that I signed up for — voluntary spam!  Those 8 were: ebay.com, kickstarter.com, paypal.com, (my bank).com, netflix.com, (another bank).com, vintage-computer.com, and linkedin.com.
• I received 174 emails from other things like hobbyist mailing lists and groups — stuff I want to keep on top of, but things that I now realize a daily or weekly summary would suffice.
• I received around 35 personal or truly important emails during the entire month out of all 5690, roughly 1 per day.

RSS feed: Received over 1200 news articles, 1000 of them daily tech news. I will be paring 25% of those down.  I’m also not sure if I’m going to read them all to catch up.

Movies watched:

• The Weavers: Wasn’t that a time!
• Starship Troopers (with and without commentary)
• Alien Anthology “Alien” and “Aliens” featurettes
• Looper (on a date with Melissa :-)
• The Dead Zone
• Broadway Danny Rose
• Rock and Rule
• Blade II
• Taken 2

TV Shows/Miniseries watched:

• Jane (1982 TV adaptation )
• Who Am I This Time? (1982)
• Mortal Kombat Legacy
• Legend (1995)
• Monsters
• Penn & Teller Tell a Lie
• Tales from the Crypt (entire series)
• Venture Brothers (seasons 1 and 2)
• Terriers
• In Living Color (seasons 1 and 2)
• The Walking Dead (seasons 1 and 2)
• Space Precinct
• Perversions of Science (entire series)
• The Outer Limits (seasons 1 through 5)

Games played:

• Black Mesa (finished)
• Wizorb (finished)
• Vessel
• Battlefield: Bad Company 2
• Half-Life: Opposing Force
• Half-Life: Source
• Braid (Did Not Like!  I was very irritated that time reversed only some things, and not everything.)
• Jamestown
• Rochard
• Shatter
• Ulysses and the Golden Fleece (finished, but aided with a walkthrough)

Notable activities performed:

• Performed firmware and BIOS update to my XTCfv2 prototype.
• Significantly rewrote 8088flex player – 60fps sustained is now possible! (if the bandwidth is there).  I’ll blog about that later.
• Digitized and edited more found footage from my VHS tapes.
• Took my kids to a comic book store and bought them some yu-gi-oh cards; they played games with each other afterwards.
• Took Sam out to dinner at Buffalo Wild Wings, just him and Dad.
• Wrote a deleted data recovery program for FAT12 and FAT16 filesystems.  I’ll blog about that later.

Daily Log

Day 0:

I prepared by configuring and testing vacation messages for both my home-grown traditional mbox mailbox that has been servicing @oldskool.org for eons, as well as the gmail address I alternately use. I then moved my entire inbox (about 100 items still unread, that may be a lifehack for another month) to a separate folder so that I had a clear delineation of what I had to empty before going back to my old inbox. I wanted to watch some new seasons of shows with my kids in October, so at the 11th hour I ordered some DVDs from Amazon. I turned off Data Network Mode on my phone so that it wouldn’t be constantly trying to check email and update news unless I was at home on wifi where I could ignore it a little easier. I also removed the news widgets (rss aggregator, youtube channels, facebook, twitter) from my phone’s home screens so I wouldn’t be distracted by them. Finally, I made sure all of my podcast subscriptions were current and downloaded to my phone for listening (I don’t consider this a cheat as I used to listen to the radio and mixtapes on a portable cassette player two decades ago).

Day 1:

Exceptions started on the first day, when I needed to access some online benefit information for work, and the only way to do that was over the public internet. But the experiment wasn’t to cut out all internet usage, it was to cut out all unnecessary distractions and wasted time, so I felt these that acceptable. Another exception was because I wanted to keep a log for this experiment in a file, not a paper notebook, so I allowed myself to ssh to my home server from wherever I was so I could record new info into the log. Again, I don’t feel this was against the spirit of the experiment; I could keep a paper notebook log, but I can’t stand writing longhand.

Already on the first day I had to catch myself; I had just finished watching a documentary on The Weavers and wanted to learn a little more about them, so reflexively I started to pull up wikipedia. I was halfway through typing “Weavers” before I mentally slapped my hand and closed the browser. Later that day I wanted to confirm an artist’s tour dates, and I struggled with the decision to designate that as an exception because I honestly couldn’t tell if there was any other way to look that information up other than the artist’s website. I caved and spent about 90 seconds confirming the artists’s tour dates. (Speaking of which, how did people look up tour dates before the internet? I’m serious, I really want to know. Other than posters plastered up at local venues, and maybe a fanzine or fan club newsletter, I can’t figure out how people knew when and where to catch a show.)

I also quickly made an exception for work-related conversation. When someone asks you about the new Nook tablet or Galaxy Note II or some other tech news as part of interoffice chit chat, answering “I have no idea, I’m on an internet sabbatical” does not improve comradery among coworkers. That kind of response is less “social scientist” and more ”antisocial hipster douchebag”. So whenever that came up, I allowed myself a few minutes to get familiar with what they were talking about. This policy was in addition to the previous exception that anything specifically work-related was ok (dictionary, reference material, vendor tech support forum, etc.) Lifehacking experiments cease to be fun when they screw with my ability to provide for my family, so I didn’t feel it was cheating making these exceptions.

Other first-day oddities included a high sense of anxiety — I was picking at my fingers all day, something I do when I’m nervous — as well as getting a song stuck in my head for several hours (and I hadn’t even listened to the song, just thought about it for a few seconds). I’m not convinced these behaviors were related to the experiment, however; correlation != causation.

Day 2:

I peeked at my email to make sure delivery and vacation messages were still working. 420+ emails, 99% of them appear to be spam. Scanned subjects and didn’t look like I was missing anything important. Total time: 90 seconds.

Told co-workers about my experiment at lunch; they were not impressed. In fact, we only talked about it for a minute and then moved on.

Added another exception for my weather app widget on my phone’s home screen. I would normally turn on the radio, read a newspaper, or just stick my head out the window to determine the weather, and I only glance at the app once a day, so I don’t consider leaving the weather app enabled a “cheat”.

Same song from day 1 still stuck in my head.

Day 3:

Peeked at email again for another 90 seconds just to make sure it was working ok. So far so good. Almost 800 emails, but most are spam. (I have an effective spam filtering strategy with multiple components, but a portion of it is bayesian filtering on the part of my MUA and since I’m not running my MUA, the spam is still in my inbox.)

Finally cracked and had my first real “infraction”: I overheard someone mention spending $300 on Halotherapy which sounded suspiciously like woo-woo junk science. Curiosity burned me up  so much that I just had to confirm if I was right or not. (I was.)  Total time spent online: 2 minutes.

I think my nervous-energy mind is having trouble adjusting to this new schedule, as I have come back to this log four times today to update it and the day is only half over. I continue to pick at my fingers. I may need to find some other way of releasing anxiety.

Same song from day 1 still stuck in my head!!

Day 4:

Watched 45 minutes of the 2012 Presidential Debate on my phone using a justification that I now can’t recall. I think I tried to justify it using some sort of scheduled VHS tape analogy. I think I messed up royally here. The only mitigating factor is that I watched all 45 minutes contiguously, calmly, and gave it my full attention.  Unfortunately, now I don’t want to vote for either of them.

Drowning out the stuck song in my head using a dance/electro/hiNRG/techno playlist on shuffle.

Day 6:

Got online to go directly to ebay to handle some auction sales. (This was an exception that I agreed on before I started the experiment, because it’s not worth borking my ebay feedback rating and deny people merch just because I’m doing an experiment.)  Once done, did not feel the need to go browsing or check email.

Day 7:

While seeing a matinee, saw a poster for Taken 2. Hadn’t even realized it was being made, let alone in theaters! Went to Roger Ebert’s online reviews to check out a review, which I justified as being no different than picking up the paper to do the same.

Matinee was Looper. It was good — not great, just good. The best thing about the movie was watching Joseph Gordon-Levitt do his best to look and act like Bruce Willis. Prosthetics were involved.

Day 8:

Coworker mentioned to me that Jellybean might actually work on my craptastic Epic 4g and pointed me to http://get.cm/?device=epicmtd . Now my phone is burning a hole in my pocket until the end of the month!

Day 9:

Completely and utterly fell off the wagon when I ran into a programming problem and couldn’t find the answer. Searched the ‘net for 2 hours before concluding that I could probably figure out the problem myself if I just think about it for a while. Gotta get back up on the horse!

Day 11:

I was stumped by a programming problem and couldn’t resist the urge to ask someone online for help. If I was asking any random person I would probably have been able to resist, but the Internet is the great equalizer, so it was very easy for me to perform the equivalent of asking Chekov advice on how to write a play. That’s pretty hard to resist. (The advice I got back was extremely helpful; it was worth it.)

Day 12:

Watched a documentary on the Zodiac killer with Melissa; took both boys to a comic book store to get some more Yu-Gi-Oh cards, and Max discovered Magic: The Gathering and is now 50+ cards down that rabbit hole.

Day 13:

Measured my weight and was shocked to learn that I had actually gained half a pound instead of losing weight like I thought I would during this experiment.

Did three loads of laundry, which is about 2.5 more loads than I do in a week.

Day 14:

Did 4 loads of laundry in the breaks between programming sessions.

Day 16:

Halfway point achieved! Max told me he was impressed I made it this far.  :)

Not having to check email constantly is a huge win IMO, although I feel somewhat guilty,  like I am avoiding responsibility or something.

It has been effortless to resist the pull of facebook and twitter; it  just seems like so much noise, noise, noise, noise . I miss cracked.com the most, as I usually read it in bed as part of my falling asleep ritual. I am compensating with tv shows instead. I should probably be reading  a book or something (or nothing!)

Day 17:

I bricked my phone trying to flash it. Because the USB port is broken on it, there is no way to flash a recovery image to it — so I am going to be without my phone for about 3 days. This should get interesting.

Day 19:

Got phone back, and found that I didn’t really miss it.

Noticed that I don’t seem to miss my email at all. In fact, it’s kind of a relief. The first two weeks I felt like I was shirking some sort of responsibility, but now it just seems like a burden has been lifted. It’s a shame it can’t last.

First real fallout of not checking email found today: My NAS dropped a drive on the 13th, and I never saw the error because I wasn’t checking email which is where I have the errors sent. This morning, it dropped another drive and I lost the entire NAS. It’s backed up, but I could have prevented the upcoming week of rebuilding and restoring if I had just replaced that drive on the 13th.

Day 20:

Was able to recover the NAS without resorting to backups. ZFS ROX DA HOUSE Y’ALL

Day 24:

Over the last week, my willpower has eroded to the point where I’m using the Internet about 10 minutes a day. I needed some questions answered for my programming project, so I have been on Usenet to get help. I’ve also visted IMDB to look up trivia for movies or shows I’d just watched. This is still a far cry from where I used to be (at least an hour every night) but I still feel sheepish.

Day 28:

Spent most of the weekend finishing up programming a data recovery utility for FAT12/FAT16 filesystems.  Was extremely calming, relaxing. I wish I’d programmed it 25 years ago though, since I built two novel functions into it that no other recovery utility of the 1980s had. I could have made a lot of money back then.

Day 29:

I don’t miss my email at all, but I’ll need to deal with it soon, so I broke my rules for about 10 minutes to research how best to gather statistics from mbox files and grab some source tarballs. I ran October’s email through the excellent MailListStat program.  The results are somewhat what I feared: I appear to be my own worst enemy. Most of the email I’ve received I signed myself up for. Information overload. I will be correcting that in a big way in November!

Next?

Stay tuned for November’s lifehacking experiment: My health!

Actually…  I don’t recall using the internet on my wedding day, honeymoon, and during the birth of my children.  But the ‘net and I definitely have a five nines relationship.

A few days ago, I came across a 3-article summary on slashdot on how smartphones have banished boredom, and whether or not that’s a good thing.  It got me thinking about how I spend the majority of my free time — or rather, how I tend to waste most of it — so I decided to try a little experiment:  I will go on an internet sabbatical for one month, specifically the entire month of October. Absolutely no voluntary internet access whatsoever (I’m sure my internet-connected devices will continue on their merry autonomous way without me).  The rules I am imposing on myself are as follows:

• No web access
• No reading or responding to email
• No browsing social network feeds (twitter, facebook, google+) in any capacity (smartphone or PC)
• No Netflix streaming access

To avoid too much disruption, there are a few exceptions to the above.  I will check email/web Saturday October 6th because I have an ebay auction ongoing, and will need to send the winning bidder their merch.  Also, I will allow myself web access at work if I need it to perform my job duties, or at home if I need to respond to something time-sensitive (like enrolling for work health benefits, etc.).  And if Melissa wants to watch a movie on Netflix with me, obviously I will oblige.  But otherwise, that’s it.  I’ll slap a vacation message on my email addresses and come back a month later.

Lack of internet access doesn’t mean I’m sequestering myself from society.  If you need to get a hold of me, please call or text me — you know, like before the internet was popular.  I’ll have my cell phone on me at all times.

I’ve never done anything like this before and have many questions.  Will I spend more time with my family?  Will I get more sleep?  Will I finally read Neuromancer?  Will I spend more time playing games?  Will it affect my weight?  Will it affect my work performance?  Will it affect my anxiety?

I’ll come back with answers in November.  See you then!

For the TL;DR crowd: You need both vintage hardware and emulation to properly research something.

I’ve contributed to a few emulators in varying capacities over the last decade, and in that time, some of my emulation colleagues have expressed surprise when I mention that I have a large collection of functional vintage machinery (both personal computing as well as console gaming) hiding in my crawlspace, and that I rotate pieces in and out on a regular basis for both enjoyment and research. This is not a statement against emulation; some emulators are packed with features that make interacting with the original works very easy and flexible (such as DOSBox, MESS, and MAME) while others are dedicated to accuracy to extreme levels (such as bsnes). The state of most emulation scenes today is thriving and most emulators are quite stable and practical to use as a replacement for the real thing.

So if that’s the case, why maintain so many physical machines? It’s because emulation can only go so far. No matter how good the emulation is, you can’t change the fact that you are running a work on a physically different computer than what it was designed for. For most works, this is perfectly acceptable, but there will always be something missing no matter how good the emulator is. In my vintage computing research, I use a mixture of both emulation and real machines so that I can reap the benefits of both. To illustrate this, I’m going to explain the advantages of both emulators and vintage hardware.

Vintage Hardware Advantages

People who never had access to vintage hardware, and always have success running software inside their emulators, sometimes can’t understand what is missing when you emulate, even using one as hyper-accurate as bsnes. Most emualtors handle the obvious (CPU and Video display subsystem) well, because those are required for basic operation; they also handle CPU “speed” and system timing well, as mishandling them would prevent performance-sensitive software from running properly. So what’s missing? Here’s a short list:

• Video display properties: Color temperature, screen curvature, scanlines, response time, phosphor layout, phosphor speed. All of these are important, some more than others. Vintage game graphics were composed with all of these properties in mind, from color choices down to individual pixel placement by the artist. If you are viewing a classic game’s graphics on a progressive display, the end result can be either too bright or too dim, over-saturated or under-saturated — no matter the outcome, it’s definitely not what the artist intended. Thankfully, this is changing thanks to ever-increasing GPU power in the form of plug-in shaders.  But as awesome as modern shaders are, they can’t overdrive hardware. Anyone doing serious research on vector arcade games absolutely must see at least one in person to get a feel for just how bright and intense the vectors can be. (I personally recommend the mint-condition Asteroids currently operating at the Galloping Ghost arcade — very bright, currently no burn-in.)
• Interface properties: Input device layout, button placement, tension/stiffness, etc. This is just as important for computer emulation than game console emulation, because computers have input devices specialized for general-purpose computing (keyboards, mice) and not gaming. For example, some games on the PC used F9 and F10 for lateral movement in games, which initially doesn’t make any sense until you look at a vintage 83-key PC keyboard and see that F9/F10 were located at the very lower-left of the keyboard — optimal placement for gaming with the left hand. And younger Mac users may wonder why classic mac games only use a single button until they physically see why. This is most easily fixed by connecting vintage input devices directly to the machine performing the emulation, although such connections can range from a commercial adapter to a homebrew wire-wrapped monstrosity. But it’s worth the effort, so that you can see why Intellivision or Colecovision games require you to quickly navigate menus using the 1, 3, 7, and 9 keys (look at the controllers, you’ll see what I mean).
• I/O properties: Because most emulation research is in the field of gaming, little attention is paid to emulating I/O as it was on the original hardware, such as the speed of the storage subsystem, any noise it made, and so on. (In fact, most emulators attempt to speed up I/O as much as possible so that programs load much faster than they did on the original hardware.) Some emulators make noise, such as WinUAE, but the overwhelming majority do not, and I think that’s a shame. For gaming, I/O isn’t usually a concern, but for any historical research, it can provide a visceral response that would otherwise be missing.

This last one is hard to understand, so I’ll provide a real-world example: When playing the game Wizardry on a real Apple II or IBM PC, it loads from a single floppy disk frequently while the game progresses, to load new graphics, maze data, etc. due to the computer’s memory being smaller than what a floppy disk holds. Wizardry is a role-playing game, which lends itself to repetitive tasks: Enter dungeon, explore new area, fight party of monsters, etc. As you play Wizardry on these systems, you begin to recognize, either consciously or subconsciously the floppy disk access patterns — the “whirr, thunk, buzz” noises coming from the drive — for these repetitive tasks as the game runs. Open a chest, “thunk”; go down a level, “thunk-THUNK whirr thunk”, you get the idea. Now, imagine that your party is wounded and desperately heading to get to the surface to heal, while navigating an unfamiliar dungeon trying to avoid conflicts. You K)ick down a door, and instead of the familiar “whirr thunk” of finding an empty room, or “whirr buzz thunk” of finding a chest, you hear “whirr buzz thunk-thunk BUZZ THUNK THUNK-THUNK-THUNK-THUNK” and realize, with growing horror, that the game is loading a lot more data than usual because it is trying to put a bone-crushing, game-ending party of monsters behind that door. But my example doesn’t end there! Not only is the I/O subsystem contributing to a real feeling of anxiety and dread, but it is also offering you a choice: If you act quickly enough, you can “dive for the drive” and pull the floppy disk out, preventing the game from loading the data it needs to seal your party’s fate. This works because Wizardry only saves your progress at two places: When you die (which in this example is in about two seconds from now), or out of the dungeon, back in town. Ripping the disk out means you unfortunately lose your progress since the last time you saved, but more importantly it cheats death — and in Wizardry, death is (mostly) permanent, even across saved games.

Emulation Advantages

For vintage gaming research, emulation provides many benefits that the original hardware would struggle with, or in some cases, are impossible. My favorite emulation advantages are, in order of importance (to me):

• Instant searching. Software loads quickly while emulated, so you can sample one game a minute, faster if you’re organized and a quick typist. This is great if you’re trying to blow through 50+ games looking for something (a game character, a programmer or graphics artist in the credits, a specific gameplay mechanic, etc.) Handy for answering quick historical questions.
• Interchangeable hardware. For PC emulation, any combination of video and sound standards can be quickly loaded and tested. Trying to do this with vintage hardware is time consuming and very costly (you try buying  every single vintage sound card ever made from ebay!)  My most common use of this is listening to all of the different ways a composer tried to optimize a game score for a specific hardware device. For example, load a game that uses 8-note polyphony with a MIDI device, and then load it again with the 3-voice Tandy/PCjr audio chip and hear how the musical elements were scaled down to a drastically limited output device. This practice can also lead to some interesting discoveries, such as how Rob Hubbard bit-banged a single SN76496 channel to make it perform like a crude DAC on Tandy machines, leading to some music soundtracks you can only hear properly on that platform.
• Easy video and audio capture. While audio/video capture is not a substitute for the real thing, it is excellent for quickly illustrating gameplay mechanics or concepts to an audience that can’t (or won’t) run emulators. Capturing video or audio from real vintage hardware ranges from “easy and high-quality” to “nearly impossible” (requiring hardware modification to tap the necessary signals for capture).
• Expanded capabilities. Emulation can enhance your experience of a work, which can greatly increase your enjoyment or appreciation of it. My favorite example of this is playing early multiplayer serial/modem/IPX games, or even 2-player game console games, across the internet with another player. In the 1980s, if you wanted to play a console or computer game with another person, you both had to be physically present. In the case of computer hardware, you had to connect computers via a serial cable, a modem connection over phone lines, or some other wacky physical connection such as a MIDI cable. The difficulty of establishing connectivity limited the experience. Nowadays, that situation has completely flipped — connectivity is standardized and widespread across international boundaries. Emulation bridges what the vintage hardware expects with what the real world can provide, and makes it possible for two people to play the same vintage console or computer game as if they were sitting side-by-side.
• Inspection. I absolutely love this. Many top-flight emulators have an embedded debugger that allow you to pause the emulated system, inspect memory and registers, set execution breakpoints/conditions, and more. This is an absolute boon to researchers (and preservationists, who wish to “crack” copy-protected games so that they can survive past the failure of their media). This has answered a great deal of historical questions for me, from why Galaxy Player is the fastest in the world on 8088 (answer: saturated register additions + self-modifying code), to how some vintage games manipulate a 3-D virtual space and calculate rotation and perspective transformations (answer: Just like everyone else, with matrix math and nine MULs, but with a 16.16 fixed-point worldspace and 256 degrees per circle instead of the usual 360).

So there you have it: You need both vintage hardware and emulation software to get a better picture of what you’re researching.

(Note that I didn’t say “to get the full picture”, because it’s impossible to get the “full” picture of a particular work. The “full” picture would conceivably include facets of the work that may be impossible to obtain, such as experience with the social and political direction of the time period in which the work was created, the background of the people who created the work, the business and financial pressures that led to the funding of the work and controlled its motivation and direction, etc.)

I was not as lucky.  I bought some real crap in my time (the name Emerson should strike fear into everyone’s heart) and while it taught me to stop buying cheap crap (stuff that broke, stuff that was disappointing, etc.), it took longer than it should have.  My most cringe-inducing purchase was a Miami vice-like outfit the summer before high school (this is summer of 1985 for those wishing stage public humiliation – it’s okay, I don’t mind).  I put it on and walked nearly 2 miles to a party some girl I thought I had a crush on was throwing.  I thought I was going to be hot shit, but as I approached the outdoor party, everyone laughed at the Miami vice thing since it was cheezy (and I was in Illinois, not California).  Seriously, I had not even reached the front door when I’d gotten my third uncomfortable comment… so I kept walking, never stopped, turned the corner and came home, another 2 miles.  My grandmother was house-sitting at the time, and she saw me come in the front door less than an hour after I’d left, and wisely didn’t say anything.  I took off the outfit, hung it up, and never wore it again.

I should have thrown it away immediately.  It might still be in the closet in the house I grew up in.

I’m one of the people this user probably hates, as I own not one but two Adlib Golds, one loose and another open-but-hopefully-complete-in-box. (I also have three IBM Music Feature cards, as well as an MSound Stereo, giving him four more reasons to hate me.) And until I get my “Sound Card Museum” project up and running, and have the card fully documented, I’m unwilling to let either of mine go.

But it got me thinking:  If you want to have fun exploring a high-quality, quirky, or just historically interesting sound card for your vintage rig, there are plenty of other options that grace ebay on a semi-monthly basis.  For example, the Pro Audio Spectrum series is interesting in that one of the models (maybe more?) can be put into an 8-bit ISA slot and give even a lowly 808x machine 16-bit 44.1Khz stereo sound. Later PAS cards had 3D in the name (Pro 3D Spectrum IIRC) and had a “surround” bit you could flip for some fake surround. Some clone cards could do all sorts of wacky emulation; I have an Aztech Sound Galaxy NX Pro 16 that can emulate the Covox Speech Thing and Disney Sound Source. Some cards somehow manage to tap port ox60 so they can route PC speaker sound through the card (and some Sound Blasters and other cards have a cable jack that plug into the motherboard for that). The Sound Blaster 16 ASP has a programmable DSP that can be used for realtime compression/decompression of ADPCM audio as well as QSound, although only one game supports the ASP that I know of, name escapes me at the moment. You can almost always find an MT-32 or an LAPC-1 on ebay now and then, and those will obviously add dimension to most games published from 1989 to 1993. For truly amazing General MIDI sound, you can still sometimes find the original Roland SCC-1 which not only practically defined the GMIDI standard but still remains one of the best-sounding cards for GMIDI (some of the MIDI files that actually use the GS extensions sound pretty damn amazing). While high-end and not very compatible with games, the Turtle Beach Multisound has really great MIDI wavetable that should be heard at least once.

King of the “interesting” sound cards is the Gravis Ultrasound. Very wacky, very capable, very limited, very unlimited. It can produce simultaneously the very worst and the very VERY best sound you’ve ever heard, depending on how well the application programmers understood the card. Some games get small speed-ups with a Gravis card because it is capable of playing up to 32 digital channels out of its onboard wavetable RAM, giving the CPU some more time to render frames. You can put the Gravis into any 286 or higher that has a true NMI. If you put it in a 386 or higher, find yourself some demos that support the GUS and prepare to be amazed at what your old slow computer can do. (Bonus non-sound-related hint: You can get Doom (not Doom II) running at nearly the full framerate on a 386-40 by hitting F5 as soon as the game starts to throw it into low-res mode.)

Sometimes a very uninteresting/dull/plain card can be put to very interesting uses; for example, DOS-era MPC-era gaming, where the audio consists entirely of redbook audio tracks (even the interactive speaking parts). This is something that emulation still has some trouble getting right (namely, the sync is delayed/off/slips), but on real hardware it usually works. I have a Tandy 2500 sx-25 that has a CDROM interface card with stereo RCA jacks — it’s perfectly capable of playing Jones in the Fast Lane or Loom or Monkey Island (MPC edition) or INCA or any other redbook audio-based game with no sync issues whatsoever without needing a secondary sound card (although having a real sound card adds more dimension to those games).

And finally, if you want to give even a truly shitty card the opportunity to sound awesome, grab yourself some decent Amiga MODs (or .S3Ms, or .ITs, or .XMs) and fire up a decent modplayer (or better, the tracker that originally created them). A 386-25 can calculate 8 or more digital channels mixed together in decent quality realtime and then feed that to your crap SB clone. If you have a Tandy TL/SL/RL machine with the built-in DAC, “TANTRAKR” is an excellent modplayer that uses the DAC and even on an 8086 can play 4-channel MODs decently.

You can even have some fun with the Covox Speech Thing (and other LPT DACs like the Disney Sound Source). The Covox by itself isn’t very interesting and also draws quite a bit of CPU when playing audio, but if you have the [B]software[/B] that came with the Speech Thing, it gets more interesting. The software contains some interesting utilities including an 8:1 speech compression method that actually works (modified CVSD) as well as a 2:1 compression scheme that works very well with music. Don’t have a Speech Thing? Build your own using a handful of resistors and some wire!

So yes, it is unfortunate that the Adlib Gold is somewhat of a holy grail when it comes to PC DOS-era soundcards, but that doesn’t mean you can’t explore some other dark corners of DOS audio.

I think it’s amazing how much of a commodity personal computers have become.  Last year, my then 2-yr-old nephew could navigate an ipad without any help, even though he wasn’t talking yet.  My 12-yr-old son has a typical smartphone, which means he can send sound, images, and data to anyone in the entire world no matter where he is — that’s stuff I used to watch on Star Trek, and now it’s reality!  That’s both pretty damn scary and pretty awesome.  But I think what’s missing is the element of discovery, of natural intellectual curiosity, trying to figure out what the machine can do, why and how it does what it does, and how to push it farther.  That’s what I miss about the early days, and is probably why I have 27 old computers stuffed into my crawlspace, with 1 or 2 in regular circulation.

I feel like that intellectual-curiosity-for-tech has been lost from the general public in the last 10-15 years.  Maybe I’m wrong and it never existed at all, and I was just lucky enough to always be surrounded by people who were interested in computers in my youth.

A month ago, the newly-unearthed M.U.L.E. for the PC (more on that in a later post) got hours of use as my 12-yr-old and his friend played several games.  Because we didn’t have the manual at the time, there was much experimentation and probing on what keys to press and how the game mechanics worked.  A few months ago, they did the same thing playing 2-player simultaneous Zyll, where they poked and prodded every square inch of the game to try to see what made it tick (and when I surprised them with a dot-matrix-printed Zyll FAQ after they’d played for a few hours, they just about lost their shit).

My point is that they were really into it, and I can’t help but wonder why they aren’t into much neater tech they own that has vastly more power and flexibility.  I never see them as enthusiastic around their xbox or iphone as they were playing these old games and trying to figure out how to drive them.  They might be enthusiastic about the game they’re playing or the people they’re playing with, but never the machine itself.

Why is that?  Does a device lose all interest once it has been through commoditization?

There are generally two types of retrocomputing snobs:  People who think that “computers were better back then” and just want to reminisce because today’s world is complicated, confusing, and scary, and people like me who think that there should be a point to the hobby.  There are way too many of the former and not nearly enough of the latter.

On that note, this is probably a good time to announce that I’ll be working on an “oldskool PC” podcast late this summer.  There are many podcasts dedicated to the C64, Apple II, Amiga, Atari, etc. machines but none* that cover the first decade of IBM PC and PC clone retrocomputing.  That’ll be me.

Suggestions welcome, but I already have about 30 topics I plan on covering, like hooking an old PC up to the internet, archiving data off old machines and floppies, interesting programming quirks, music and sound, and of course games and gaming.  Because I dislike long rambling podcasts I can’t consume in short bites, each episode will likely be under 15 minutes in length, and extremely dense.  If you’re curious what I sound like, I recorded my very first podcast for Hacker Public Radio last year, which you are free to point and laugh at.  (Unfortunately, I rambled a bit in that one, but hey, first podcast.)

* There are a few, but they deal mostly with gaming and not with retrocomputing itself.  I’m also familiar with the Retro Computing Roundable, but they only rarely cover PC/clones.

• “NAID ’95″ tape compiled by the organizers
• “NAID ’96″ tape compiled by the organizers
• “IC and Soundgun do NAID ’96″ by IC, with a 2012 introduction by IC
• Four local news programs on NAID ’95
• 8 minutes of NAID ’95 registration and guestbook signing with lots of familiar faces
• 2 minutes of NAID ’96 classroom footage

It’s a little over 2 hours of footage of the height of the 1990s NA demoscene.  Also included on the DVD-ROM section is Phoenix’s NAIDorabilia which is all of the demos, music, party reports, and other miscellany associated with both NAID parties.

The footage is fully interlaced, preserving that “video verite” look. I chose to distribute it as a DVD because that was the only way I could guarantee people would see all 60 fields per second. (Unless something has changed in the last year, no online streaming video service (Youtube, Vimeo, etc.) supports 30i material properly… although that’s a moot point anyway, since there is some very well-known copyrighted music in parts of the footage which would probably get it rejected from such services anyway.)

You can download the DVD .ISO at archive.org, but if that is gone for some reason, there is a slower download link here:  ftp://ftp.oldskool.org/pub/misc/Video/MemoriesOfNAID.iso

One of my hobbies is taking old IBM PCs (and clones) and restoring them.  ”Restoring them” can mean anything, but for me, it usually means these three things:

1. Get the hardware functional and booting up
2. Archive the data off of any internal hard drives
3. Wipe the hard drives clean and set it up for appropriate use

I did that last night to a new acquisition, an AT&T 6312 WGS.  This is one of the later “workstation”-class 286 machines that AT&T OEM’d from Olivetti, and I got it from a former AT&T/Teletype/Lucent employee (which is common, since they sold them to internal employees at a discount.  If you want to learn more about AT&T and their computer history, check out a video I made on the AT&T PC 6300.)

Step 1: Get the hardware functional and booting up

On opening it up, I saw both an internal hard drive and a second hardcard, a Western Digital PS30 which was a 32MB drive+card meant for the PS/2 Model 30.  I also saw a rechargeable battery pack, long since depleted, and sure enough, on bootup, the 6312 complained that it had no idea what the date/time was or what hard drive was installed in the system.  Furthermore, the PS30 hardcard started executing its option ROM at CA00 but then later spat out the dreaded 1701 error message, indicating that the drive was non-functional.  Fixing this requires the setup disk for the 6312 (which it helpfully barked at me during the BIOS POST), and thankfully NCR still had it available for download.  But when I got the file, I noticed with frustration that it was not a bootable disk image, but rather just a collection of files that came from the original bootable disk — and it had a specialized COMMAND.COM, which means they likely wouldn’t run by themselves and needed the specific boot disk environment to work.  You can’t just copy the DOS kernel files IBMBIO.COM and IBMDOS.COM to a disk and expect them to boot properly without a lot of work (usually the SYS command does this, but only for the DOS you currently have booted, so that wouldn’t work for me).

Sometimes restoration work requires creativity, and sometimes it just requires experience and educated guesses.  I used the latter to reconstruct the 6312 setup disk.  First, I looked at the IBMBIO.COM and IBMDOS.COM startup portions of DOS in a hex editor and determined that it was an AT&T OEM version of DOS (not surprising, really).  I figured I could find disk images of that DOS and use those to make a bootable disk on another vintage computer and then copy the 6312 setup files onto it.  Luckily, AT&T only produced two versions of DOS that supported hard drives, 3.2 and 3.3.  I checked the 3.2 image and sure enough the copyright string was identical and so were the IBMBIO/IBMDOS sizes, so I now had my bootable DOS disk to use.  But I had another problem:  The setup files were larger than a 360K disk.  This meant the 6312 setup disk was a 1.2MB floppy — and I didn’t have any vintage machines set up and working with a 5.25″ high-density 1.2MB floppy.  I know I could have tried to do things like take out the drive cage and swap cables, but I have a strong philosophy on trying to get the data off of the machine before messing with its innards (in case I break something).  So, I used a second vintage machine with both a double-density 5.25″ (360K) and 3.5″ (720K) drives in it to help me, and followed this procedure:

1. Helper PC: Make an AT&T DOS 3.2 bootable 360K disk.  Make a .ZIP file of the setup files and put those on a 720K disk along with pkunzip.exe
2. 6312: Boot the 360K disk and use it to run “format a: /s” to make a working 1.2MB bootable disk.  Put in the 720K disk and extract the setup files to the 1.2MB A: drive, taking care not to touch IBMBIO.COM and IBMDOS.COM.

That wasn’t so terrible, and I didn’t have to crack the drive cage open or attempt other trickery.

Once I could boot the setup program, I noticed I had over 30 hard drive types to choose from, and I couldn’t determine what the internal drive was because it was located underneath the floppy drives.  How to pick the right type without removing the drive to look at it?  While I could have tried all 30+ combinations, rebooting after each one and seeing if that worked, I did something much easier:  I set it to Type 1, a “10 MB” drive, just so the system would try to boot.  I then booted off of a floppy with Norton Diskedit and used it to look at the first sector of the drive (which will always work no matter what drive type you have, because it’s track 0 head 0 sector 0), choosing View->As Partition Table, and seeing how many tracks/sides/sectors per track there were.  There were 611 tracks defined, 17 sectors per track, and everything added up to 20MB.  So, we have a 20MB drive.  I checked a few CMOS drive table types online looking for 20MB drives with 600-ish tracks and 17spt and decided to try Type 16, which worked!  I could see the first hard drive’s contents, and after using LIST to check a few files, I was confident I had the geometry right.

This brought me to the second hard drive, the PS30 hardcard.  Every boot, I kept getting the 1701 error, which is usually drive death.  But on closer examination, I could see that the stepper motor axle was exposed — I could literally turn it with my fingers without dismantling anything.  I worked the stepper motor a few times, turned the machine back on, and it spun up and seeked!  The ROM on the card was smart enough to handle being the second drive as well as setting up the drive type/table; when I booted the machine, it showed up as D: and everything on it was readadble.

At this point, we have the 6312 back to its original operational capacity.  This entire process took about 2 hours, about half of which was careful thought on how to proceed.

Step 2: Archive the data off of any internal hard drives

There are two schools of thought on dealing with the data on old machines: Wipe the drives clean, or archive them.  Some people immediately wipe them, out of a sense of privacy for the previous owner, or maybe because they don’t want to unintentionally break any privacy laws.  I don’t do this; I try to preserve everything.  The main reason is that there might be a driver on the hard drive that is necessary in getting the thing working.  For example, there’s an extended memory board in this 6312 that, with the custom AT&T driver, can also be set up as a hardware LIM EMS board.  Without that driver, I would never have that option.  Another reason is that there might be some rare software on there that has been lost to the ages.  On this machine, I found a fantastic DOS version of VI that works better than any other VI clone I’ve run on a DOS machine, and I’d never heard of the product or the company before.  Researching it further, there’s a good reason:  It’s Custom Software Systems’ “PC/VI” and it was based on AT&T source which is why it works so damn well — and they hadn’t purchased a license for the source and it went off the market after only a year, which is why I hadn’t heard of it!

Poking around an old hard drive is sometimes like archaeology; you always find a piece of the prior owner there.  My favorite example of this was a no-name taiwanese clone I got as a gift a few years back.  The previous owner was a Chinese immigrant and had her college homework on it.  She was studying both business and politics, and the homework reflected how dry and boring it all was.  But tucked away in a corner of the drive was a single directory that contained several poems written by her — an oasis of creativity in the middle of all the tedium of trying to get an MBA.

To date, I haven’t spread my “system archives” around, out of respect for the prior owner.  (I’m not worried about privacy laws though:  If they threw it out, then they threw out their right to privacy.  This is why dumpster diving is legal, and why private investigators dig through people’s trash — anything they find, they can legally use.)

Okay, you’ve made the decision to get the data off the drives before they go bad, and a requirement is that the drives not be written to or altered during the process.  There are many different ways to do this, so I’ll outline a few:

1. The hard way (floppies).  Using completely free software (ARJ, RAR for DOS, or PKZIP) and some pre-formatted floppies, you can just start archiving.  The big disadvantage to this is time, effort (it’s a manual process), and the risk of getting a bad floppy that you can’t read on the target halfway through your stack.  I only do this if I have one floppy’s worth of data to suck off the drive.
2. Use a backup program.  This is the fastest way to format and write to floppy disk media, and you get error-correction thrown in for free, but the drawback is that you need another vintage computer to read the backups (one that can accept the floppy media and has enough space to hold the files, as well as a way to get them off to something more usable like through a network connection).
3. Install a network card.  You can attempt setting up MS Lan Manager to mount a samba share as a drive letter, but sometimes I like to just load a packet driver and use mTCP to set up an FTP server and just suck the files off.  A lot less hassle.
4. Connect external storage.  ebay is just maggoty with Iomega parallel-port ZIP drives, and as long as you’re trying to suck data off of a 286 or higher and can boot DOS 5.0 or higher (even from a floppy disk), they work great.  Each disk holds 100MB which is more than enough for most MFM and RLL drives.  One of these and XCOPY or PKZIP is all you need, and if you don’t have another machine to hook them up to, you can turn around and pull #3 (install network card) and upload the results.
5. Parallel or Serial cable.  Laplink, FastLynx, and even DOS 6.x’s built-in INTERLNK can all transfer files over cable.  It’s inconvenient and not too swift, but it works.  I prefer FastLynx, which has exactly the same code as INTERLNK but has a nifty interface and can use realtime compression.

For the 6312, I chose the zip-drive-then-ftp-to-my-fileserver route.

Step 3: Wipe the hard drives clean and set it up for appropriate use

Wiping the drives clean is easy — you can just delete all of the data and then run Norton WIPEDISK.  If you’re more paranoid, you get to locate your drive’s low-level format program and have at it, secure in the knowledge that you’re laying down the low-level format at the same time (just make sure you enter in the defect table from the sticker on top of the drive!).

As for “appropriate use”, that is completely in the eye of the beholder.  Because this is an AT&T “WGS” machine (meant for workgroups), I’m going to leave all of the AT&T-specific software on it (there are some terminal emulators on it, etc.) and I’m going to set it up as a business-class PC:  A word processor or two, all of my typical programming tools and environments (Turbo Pascal, Turbo Debugger, Turbo Assembler, and a86/d86), and a network card with both the mTCP suite on it as well as MS LAN Manager.  I’ll configure the machine to boot up and set the time automatically via SNTP, and I’ll also investigate MS lanman so that it can mount my local SAMBA shares.  This will require a more beefy DOS than 3.2, so PC DOS 2000 is probably going to go on there.

Of course I’m going to throw some games on it too :-) Every system needs games.

I am sick and tired of people who shun technology and progress under the guise of “I’m an old tech veteran, I’ve been working with technology for 30 years, and the new stuff is crap compared to the old stuff.”  People who defend this viewpoint are idiots.  I’m not talking about audiophiles or other self-delusional “prosumers”; I’m talking about people who have worked a tech trade or had hands-on access to technology for many years and think that their perceptions trump reality.  It’s a perverse combination of technology and anti-intellectualism – a form of hipsterism for the over-40 set.

I was prompted to cover this by a recent post on why widescreen monitors are a rip-off (which I will not link to because I truly enjoy the other 99% of this person’s blog, and linking to it would imply that I don’t like him or his site), but the underlying irritation of the entire mindset has been percolating for many years.  Viewpoints that drive me crazy include:

Widescreen monitors don’t make any sense

People think that widescreen monitors are stupid on laptops because most people use laptops for text work, and since text is more comfortable to read in columns, wide columns are harder to read.  This mindset has had the doubly idiotic result of making people think that websites need to be column-limited.  I just love going to a website and having the text squished into a 640-pixel-wide column with 75% of the screen unused.  Don’t like how narrow columns look on a widescreen monitor?  Use the extra space however you want — put up two web pages side by side, or simply don’t look at the unused space.  It’s people like these that also complain that 4:3 video has black bars on either side of it when viewed on a widescreen TV.  It’s called pillarboxing, you idiot, and it’s there to prevent your movie from looking like a funhouse mirror.

Widescreen monitors have made modern laptops better.  A widescreen laptop monitor allows the keyboard to be wider without the depth of the laptop getting too high (to support the height of a 4:3 monitor).  Having a decent keyboard on a laptop used to be impossible without clever wacky engineering tricks; now it is.  Widescreen monitors made ultra-small netbooks possible, so if you’re reading this on a netbook but somehow still disagree with me, you’re a hypocrite.

Analog audio is better than digital

There are entire websites (and wikipedia pages) dedicated to this, usually under the guise of “vinyl is better than CD”.  Most opinions on this subject were formed when analog audio had several decades of mature mastering and production processes, and digital was brand-new (for example vinyl vs. CD in 1983).  Early efforts to put things on CD resulted in some less-than-stellar A/D conversion, which created a type of distortion that most people weren’t used to hearing.  People formed opinions then that have perservered more than 25 years later, even though the technology has gotten much better and all of the early mastering problems have long since been corrected.

People who think vinyl sounds better than CD have nostalgia blinders on.  They bought an album in their youth, played it endlessly, loved it.  Then they buy the same album on CD decades later and condemn the entire format as inferior because it sounds different.  Want to know why it sounds different?  It has a wider frequency range, lacks rumble, lacks hiss, sounds exactly the same after 10+ playbacks, and was remastered with better technology and mixing conditions under the guidance and approval of the original artist when he wasn’t coked or drunk or stoned out of his mind.  People like Pete Townsend, Neil Young and Geddy Lee not only approve of the latest digital technology but are actively utilizing it and going through great pains to remaster their classic albums with it.  People are missing the point that it is the mastering and digital compression that causes issues, not the technology itself.  Neil Young recently spoke at a conference where he damned digital music, but not because it is digital — rather, because it is delivered differently than the artists intended.  Neil Young would like nothing better than for everyone to be able to listen to his music at 24/192.  Can’t do that on vinyl, bitches.

Even people who write about the loudness war get it wrong, despite that it’s an easy concept to understand.  Massive dynamic compression drowns out subtle details and can add distortion, which is horrible — but it is not exclusive to digital audio, nor caused by it.  One author correctly notes that massive dynamic compression butchers mixes, but then subtlety implies that all CDs that “clip” have distorted audio.  Digital audio “clips” only if you drive the signal beyond its digital limits.  If you took an audio waveform and normalized it such that the highest peak reached exactly the highest value, it is “positioned at maximum volume”, not clipped.  Nothing is lost (to be fair, nothing is gained either).

The problem is the mastering and production process, not the technology.  Which segues nicely into:

“I will never buy Blu-ray”

The only valid argument against Blu-ray is that it is harder to make a backup copy of the content.  It is indeed harder than it is for DVD, or laserdisc, or videotape.  That is it.  All other arguments are beyond moronic.  Even the cheapest possible 1080p HDTV viewing setup has five times the resolution of DVD and lacks signal degradation in the output path.  If you view a Blu-ray and can’t tell the difference between it and DVD, you have either a shitty viewing setup, a shitty Blu-ray, or a shitty visual cortex.

Someone recently tried to argue with me that DVDs have the same or better picture than Blu-ray and used Robocop as an example.  The comparison was weighted, as they were comparing the $9 Blu-ray that MGM belched out when Blu-ray was only a year old to the Criterion DVD treatment.  I own both, so I checked them out and I agree that the DVD has better color tonality throughout the film.  However, the Blu-ray thoroughly stomped the DVD in every single other area, most obviously resolution.  So much picture detail is added by the increase in resolution that I actually prefer it despite the lack of Criterion oversight.

The real problem, as previously stated, is how the mastering and preproduction process was handled.  Even with new 2012 DVD releases, you can still see the “loudness war” video equivalent of digital ringing, which used to be an accident but was later introduced on purpose as part of a misguided “sharpening” step.  Listen up:  Any sharpening filter added to any signal doesn’t make things sharper; it makes them appear sharper by overlaying a high-frequency permutation signal over the original content, which increases the acutance.  Quality is actually lost when you do this, as the high-frequency info obscures actual picture detail.

This is another example of perception vs. reality, which not coincidentally also segues into:

“Computing was better in the old days”

I love retrocomputing as a hobby.  I think about it nearly every day; this blog was partially created to talk about vintage computing.  But even I wouldn’t say that things were better in the old days.  People who say this don’t realize they are really trying to say something else.  For example, people who say that “BBSes were better than web forums are today” are actually referring to the sociological fact that, when you communicated with people on a BBS, you were communicating with people who met a minimum level of technical competence — because, if they hadn’t, they would have been too stupid to access a BBS, let alone be proficient with a computer.  The overall technological quality level of everyone you met on a BBS in the 1980s was higher than other places, like a laundromat or a bar.  What such people fail to consider is that modern web boards, while having a higher quotient of trolls and B1FFs, are open to the entire world.  The massive scale of humanity you can encounter on even a tiny niche topic is levels of magnitude higher than it used to be.  The sheer scale of information and interaction you can now achieve is staggering, and completely outweighs any minor niggle that you have to deal with 3 or 4 more asshats per day now.

Here’s another example:  ”Computer games were better back in the old days.”  This is wrong.  The proper thing to say is that “Some computer game genres were better back in the old days.”  I can get behind that.  For example, graphics were so terrible (or non-existent!) at the birth of computer gaming that entire industries sprang up focusing on narrative.  For such genres (mainly adventure games), several times more effort was put into the story than other genres.  As technology and audiences changed over time, such genres morphed and combined until they no longer resembled their origins.  That doesn’t mean modern games are terrible; it just means that you need to shop around to get what you’re looking for your entertainment.  Don’t play Uncharted 2 expecting a fantastic story with engaging narrative.  (Dialog, maybe, not not narrative.)  Heck, some genres are genuinely awesome today compared to 30 years ago.  For example, Portal and Portal 2 are technically puzzle games, but the storytelling in them — despite never interacting directly with a human — is among the very best I’ve ever encountered.

About the only argument that does work involves the complexity of older computers — they were simpler, and you could study them intensely until you could very nearly understand every single circuit of the board, nuance of the video hardware, and opcode of the CPU.  Today, a complete understanding of a computer is no longer possible, which probably explains why Arduino sets and Raspberry Pi are getting so much attention.

Conclusion

I have no conclusion.  Stop being an old-fogey anti-intellectual technophobe, you ignorant hipster fuck.

I think I can sum it up in three words:  Fear of death.

Everybody needs a coping mechanism for dealing with the inevitability of death.  Social interaction, religion, family, blind ignorance, sex, drugs, and various causes (environmental, human rights) are the most common, but there are people for whom none of those apply.  I believe these people turn to anything that gives them comfort, or used to give them comfort.  Ventriloquists collect ventriloquist dummies, maybe because they remind the owner of receiving adoration on stage.  Housewives collect porcelain dolls to glorify their memories of youth.  Christopher Dennis has an extensive collection of Superman memorabilia because the image of Superman is what keeps him alive.  But you don’t have to be down on your luck or unhappy to have a collection; just look at Jay Leno or Steve Martin.

For those who grew up using early computers to better themselves or others, it’s not inconceivable that such objects would give them comfort.  I am one of those people, so I have a collection of computers.  It is modest by most hard-core retrocomputist standards; I have around 30, and many are duplicates for parts.  But I definitely spend otherwise productive time hauling them out, getting them working, running old favorites (or new discoveries) on them, and writing software for them.  It reminds me of a time when I was the technological wunderkid, and had control over my environment — you tell a computer to do something, and it actually does it.  When I “retrocompute”, I have something pleasant to occupy my thoughts, and I gain a sense of accomplishment and completion.

Some collectors in my hobby look at their crawlspace, storage space, shed, or warehouse and wonder how their collection got so big and how they’ll ever get rid of it.  I think the answer is to recognize your collection for what it is:  A coping mechanism.  It should not have any more value beyond that.  Your collection is not a replacement for people.  Your collection is not more important than your job, your marriage, or your kids.  Once you realize that, you can start letting it go.  Maybe only one piece at a time… maybe never all of it completely.  But you can let go.