Oldskooler Ramblings

the unlikely child born of the home computer wars

Archive for August 19th, 2007

The truth about netopsystems

Posted by Trixter on August 19, 2007

I was first made aware of Net Op Systems (currently going by the name of NOS Microsystems Ltd.) when downloading Adobe Acrobat Reader about 3 years ago. I was struck by how small the compressed deliverable was, so, being a compression hobbyist, I did some preliminary analysis and found that they used a considerable about of context arrangement and prediction (ie. “solid” mode in rar/7-zip, or the content-specific predictors in PAQ) to get the size down. I recently ran across their product again when downloading the most recent version of Solaris x86; it comes in a 1.1G NOSSO executable package. The sole payload was a 3.1G .ISO image, which meant the compressed deliverable was 37% of the uncompressed size.  This is very impressive, given that the .ISO image is filled with a lot of .JAR and .BZ2 compressed images themselves. The successful extraction of a workable .ISO file from this compressed deliverable means that NOSSO has to perform the following to work its magic:

  1. Identify the various compressed files in the .ISO wrapper
  2. Extract those compressed files
  3. Decompress the content inside those compressed files
  4. Arrange everything by context (ie. all ASCII text in one group, all binary executables in another group, etc)
  5. Compress the entire thing to a proprietary stream, using content-specific prediction for various content groups
  6. Store the original arrangement of the compressed and uncompressed content

Upon extraction, the NOSSO distributable has to perform the following:

  1. Decompress everything and keep track of it
  2. RECOMPRESS the data originally found in compressed files, so that their effective format is kept the same. There may be small differences due to compression options and implementations, but as long as the end result is usable by the end program (ie. a reassembled .ZIP file is still able to be decompressed into the same contents) then there’s no harm done.
  3. Rearrange the end result back into the original container (in my case an .ISO file)

This is why they call their process “reconstitution” instead of “decompression”, because the end result, while functionally identical, is usually not bit-for-bit identical. By taking advantage of context and recompressing files from less-efficient formats into the more efficient format NOSSO uses internally, we can get these excellent compression ratios. (In fact, I’ll wager that, to speed up reconstitution times, they use a very fast and less efficient version of recompression of the files inside the target wrapper, which would inflate them slightly and result in even more “impressive” compression ratios :-)

What’s the downside? The downside is that this entire process defeats its own purpose. I’ll explain:

NOSSO is marketed as a delivery format that saves everybody bandwidth and, presumably, time. It’s that presumption that allows them to shoot themselves in the foot. While the compressed distributable only took 39 minutes to download on my 6mbit/s cable modem connection, it took a whopping 124 minutes to “reconstitute” on a 2.6GHz P4 with 700MB RAM free (out of 1G RAM total). My total time to get the end result was 163 minutes. (A 2.6GHz machine is not the bleeding edge in 2007, but it’s no slouch either, and is representative of the average system most people will use for everyday use.) At its original size, 3.1G, it would have taken me only 104 minutes to download it.

It would have been faster to get the end result had it not been compressed at all.

Now, 6mbit/s is a pretty fast broadband connection, so I understand that skews the results a bit. With a more common broadband connection speed of 3mbit/s, let’s check the numbers again: Compressed download + extraction: 202 minutes. Uncompressed download: 208 minutes. Okay, so it’s break-even at a 3mbit/s connection. But break-even still involves 100% CPU utilization as the thing is decompressed, resulting in an unusable system for two hours, so it’s still not “free”.

Is there strength in using any compression at all? Let’s check both WinRAR and 7-Zip on the original 3.1G unmodified .ISO file:

  • 7-Zip compressed size: 2.68G. Time to download at 3mbit/s: 187 minutes. Decompression time: 14 minutes. Total time to get the end result: 201 minutes.
  • WinRAR compressed size: 2.69G. Time to download at 3mbit/s: 189 minutes. Decompression time: 3 minutes. Total time to get the end result: 192 minutes.

So, at 3mb/s, the end result was just about the same, except our system was only tied up for 3 or 15 minutes instead of two hours. We’d get even more compression at the same decompression speed if we burst the .ISO like NOSSO does, compressed using WinRAR’s or 7-Zip’s “solid” mode, and then reconstitute it back into an .ISO when done with a small utility program.

My conclusion from all this is that there’s really no point in using NetOpSystem’s product, unless the end-user’s broadband speed is 1mbit/s or slower. But if it’s that slow, the user is already used to ordering DVD-ROMs for delivery instead of trying to download them, right? Or, if the user downloads them anyway, they’re used to firing them off before they go to bed, to download overnight. So, again, no need for the product…

…unless you’re the content producer and want to transfer cost (bandwidth) to the end user (time). Which is probably why NetOpSystems is still in business.

Posted in Technology | 16 Comments »