|Clipping the Wings O dem Bandwidth Hogs|
|Standing shoulder to shoulder the Beatles had a bandwidth of 85 inches. Mark Chapman was able to reduce this bandwidth by an impressive 21 inches (nearly 25 percent!) by the careful application of high velocity lead pellets. Mark Chapman, a man who was misunderstood in his own time, was a pioneer in the art of reducing bandwidth. Here to build on the... "body" of this... "ground breaking" work is Hamish Sanderson with a tutorial on Bandwidth Saving. He is a master of reducing the size of your Marathon scenario without all the smoke and shouting and running about. Before you release your scenario or demo you MUST read this tutorial, you will save yourself and your downloaders a lot of time. - gls|
How To Reduce your Distributed Scenario to Manageable Size
by Hamish Sanderson
with an addition or two by Claude Errera
and some silly bits by Gary Simmons
So you're building a total conversion for Marathon (or even
a large partial conversion), and you're finally ready to release
a demo... are you wondering what you can do to reduce the size
of the package you're uploading? Because you know, folks aren't
gonna be happy to download a 25 megabyte package just to play
two or three levels... but wait, you say, you don't know anything
about getting rid of the stuff you don't need safely? Well, never
fear, HAS is here. In response to a desperate request from me,
HAS kindly wrote up a list of things that could be done to drastically
reduce file sizes for TC demos. (Some of the things listed here
are specifically aimed at demos, which, by their very nature,
will have features missing. However, the concepts are valid for
application to the full product, if you can do so without reducing
quality...) I've cleaned it up a bit, and added some explanatory
info... hopefully, this can be useful to the scenario makers
Get your Shapes in order. If you haven't changed very much, your best bet is to distribute a Shapes Juggler patch containing the changed collections. However, if you're creating a total conversion (or even a big partial conversion), you're probably better off building an all-new shapes file stripped of anything not required. In practice, you can replace all unused collections in a shapes patch with a dummy one consisting of a single pixel. Collections are in the .256 resource. Here are a list of all 32 collections, and their ID numbers:
Obviously, you shouldn't replace the landscape collections with the Marine collection... but you can replace each landscape you're not using with a single pixel place holder (both standard and custom collections).
To give you an idea of what is possible, the standard Infinity Shapes file is just under 11 megabytes. Using the enclosed "zero" collection to replace 1002, 1003, 1005, 1006, 1008-1014, 1016, 1018-1021, 1023-1026, and 1031, and replacing Landscapes 2, 3, and 4 (both versions) with single-pixel images, the total file size was reduced to 2.9 megabytes. (This leaves a functional, but not very useful, shapes file-you have only one monster collection and one texture set to work with-but the point is, a substantial size reduction is possible with these techniques.)
Strip custom color collections. Custom Set Remover by Charles Lechasseur that can do this automatically (there are manual methods, but this is quicker). See Mill2 (2,365K courtesy of bungie.org) for example. (Running this application on the stripped shapes file above reduced its size further to 2.0 megabytes.) Blue pixel any unused artwork [note how TI added Ursula to the end of a bob collection - they could/should have just replaced the bob art, or at least blue-pixeled it. Note though that I'm saying this on the assumption that the entire collection had to be distributed as a result - if using SJ or distributing an entire file this is exactly what happens. Apologies to Borzz for using TI as a possibly slightly contrived example, but it illustrates my point].
As an extreme method for cutting down size, you could even re-scale bitmaps. Note that for a human-sized sprite, a pixel height >200 or so is not really worth it quality-wise when compared to the download size anyway. But I know Soren [Jensen] replaced running sprites in the Erodrome beta with lower resolution versions (with scale factor increased accordingly) on the basis that you see less detail on a moving object than a stationary one. (Editor's note: I was going to include a link to an old program called Vandlandingham that illustrates how a clunky-chunky bit mapped black and white graphic of a soccer ball looks photographically smooth in motion but the damn thing won't run on modern machines. Take my word for it, when it is in motion your eye smooths it out dramatically. - gls 10/06/2000) BTW, if you can design the mods so that the full version of the game can come as a demo-upgrade, as well as complete version, then that has obvious advantages later on.
Cut back Images. This is an easy one, and gives great savings for no work. Simply remove all PICTs with IDs of 2xxx or 3xxx (especially 3xxx, given the number of folks who'll play in million colors - even I still use 1000s for speed on a G3). This does not affect the engine (just as you can remove custom colors from shapes - the engine uses these things only if they are present, otherwise it skips and uses whatever else is available). If the artist is halfway good then the 256 versions should already serve pretty well - sensible folks work with a limited color palette anyway if they know what's good for them - it's just like working for web-based gifs, really. You can cut out the intro splash screen altogether if you like. And you can also delete the 2 credits screens (Bungie won't like this in theory, but if nobody tells them... Personally, I'd substitute a 1-bit bitmap version if possible - takes no size at all then, although anything plain [eg. text on plain background color, preferably without anti-aliasing] compresses well - remember that PICT format/StuffIt compression like lengths of identical bytes, so if all pixels in a row are the same color you get good compression, whereas something with major dithering compresses for crap). You can shave a bit more of the "active buttons" main control screen ID1101 by deleting all areas on the PICT that don't appear within button rectangles. eg. see Wheels demo (and the example below).
Advanced color handling stuff: Note that the Mac's imaging system isn't that great at displaying in thousands of colors - you get banding and mis-coloration. This can be a problem with viewing 8-bit art in 16-bit. As an example, see the 'simulacrum' screen in M2 - the 256 art looks better than the 1000s one. The reason the 1000s one is there at all is because the 8-bit version would look really banded in thousands of colors (try viewing this PICT in thousands of colors video depth and you'll see exactly what I mean). (Editors Note, I screwed that up a bit when I converted the above PICT to JPEG format for this web page. You can seek revenge for this outrage, join the class action law suit against me! Just send your name and address and 5 bucks to Class Action Lawsuit c/o Gary Simmons at 777 Shock Staff Drive, Citadel City Lh'owon - gls 10/07/2000) If you examine the 1000s screen you'll see they've used some funny filter or coloring effect, especially clear on the sky. In practice, however, it is possible to create 8-bit art that displays accurately in thousands of colors.
From your RGB art, Save a Copy as PICT, 16 bit. Open, and it's horribly banded. Index using Actual (if you're lucky) or Adaptive. This creates a CLUT containing 256 colors that also appear in the 16-bit "CLUT/gamut". [Note this is a deliberate mis-use of those terms, but it's the best description I can think of for the limited color range that 16-bit covers - I think of it in HSB terms, which translates roughly as every available color having only 32 levels. In practice, I'm not sure how hi-color (15 and/or 16 bit) is handled. I guess the display routines work in RGB, not HSB, however, which is why they don't distinguish between one hue of a certain saturation/brightness and another of similar saturation/brightness - whereas the first thing we notice is the hue. Whoever wrote such no-brain display routines needs shooting.] Anyway, you can then use this CLUT to index your RGB art into 256 colors that'll display equally well on 256 or 1000s (that is to say, it might not look so good in 256 as one indexed direct from RGB as Adaptive, but the same 256 image will look much, much better in 1000s). You'll need to use dithering, mind, as you want to break up the banding. Okay, so dither is bad for compression, but the overall tradeoff (one PICT for two) should still leave you up. If you're really thorough, you may want to touch up pixels after (the magic wand tool and Select Range commands are wonderfully useful here) - you may lose color fidelity to the original, but this is less important than getting a clean-looking image (worth the odd color shift).
All this stuff also applies to chapter screen PICTs in the map. And be sure your term PICTs are 8-bit indexed (index them to the Marathon CLUT for best results), not 16-bit.
One other thing: don't forget the PICT format allows JPEG compression. This might or might not be useful for any non-256 stuff (apart from grayscale, obviously, which is 256 but not CLUT-based, so can use jpeg, obviously). In theory, I think you can use hi/full color art in Marathon for chapter screens/Images [except HUD, obviously], although for it to display at all well on 256 displays you'd need to see if including a suitable CLUT will help (made from converting a temp copy of the art to Indexed, and then keeping the generated CLUT). I've not tried this, so don't know. But since all these screens use regular Mac drawing routines to display (AFAIK), unlike the in-play drawing routines (which use a weird alchemical system at 256 colors, full of colored ramps and things), I don't see why this wouldn't work. Again, you'd need to see if you gained any advantages with this method (JPEG is lossy, so forget any graphic/text images - only photo-style images won't look like a dog, and you may still be much more efficient with Indexing anyway...).
Keep back a copy of the original, of course. Strip 16-bit slots using Charles Lechasseur's Sound Editor or Wail. Remember, once you switch to Sound Editor, you must never, ever use Anvil on that file again [see EditNotes on sounds]. Either distribute the 8-bit sounds file or you can reduce further by converting all sounds to 16-bit IMA-compressed (4:1). That makes the file unusable on non-AV Quadras, but every other Marathon-capable Mac supports 16-bit sound AFAIK. A fair trade, and you can always distribute an 8-bit sound file separately if you feel bad about such blatant 040 apartheid. I'd recommend working with the 16-bit sounds (see why I said save a copy of the original 8+16 bit sound file for reference? : ) wherever possible, though up sampling 8-bit ones is entirely OK (you just won't gain any improvement in their 8-bit quality). I run sounds through MoviePlayer for lack of pro sound-editing kit (save as AIFF, drop onto MoviePlayer, export as sys7 sound with Options 16-bit, IMA, 22.05/22.2, and reimport into the 8 bit slots of your stripped file. (Editor's Note: The freeware application SoundApp by Norman Franke will do this just fine - gls 10/06/2000) BTW, 44 KHZ is a total waste of time for Marathon, it makes for a huge download, and unless you have a good stereo connected you won't even notice any difference!). The Sound Manager copes just fine, and you halve the size of the sounds file (which is really great, bearing in mind that StuffIt, etc. aren't designed for compressing sound data well - so any size savings have to be done with the sounds themselves). Avoid any other of the old-style compression formats, IMO - you either don't reduce size, or they are lossy. IMA @ 16 bit is actually the only worthwhile way (pity IMA doesn't work at 8-bit - can you imagine a 4x reduction in sounds file size? : ). New QT3 formats aren't usable in Marathon.
Oh yeah, BTW, use ResEdit on the main application to un-Enable the 16-bit sounds check box in DITL 4003, then resize it to 0x0 height/width to make it good as invisible [don't ever delete anything, however, as you'll mess up numbering of the items]. This'll save any deadheads from "sounds don't play". See Wheels demo to see how I did this - that hid every non-vital control there was. (An example is below.) [BTW, figured how to hide the net player name field at long last for solo-only conversions - make it a static text box and resize to 0x0; good as invisible.]
All this stuff also applies to chapter sounds in Map file. So don't forget to do those too.
Also in your sounds file: use Sound Editor or Wail to remove any/some/all additional sounds from slots that contain more than one. Your file will still function just fine - you won't get such varied audio, that's all. It's a tradeoff. And definitely check for duplicates - in some projects I've seen the BOB sound slots, most of which contain several different BOB sounds, contain a single new sound duplicated to replace each of the BOB ones. In Anvil you have to do this [no delete] but in Sound Editor you don't.
Music file. This should also be reduced to 22 KHz, 16 bit, IMA. Be merciless. If it's more than a minute long, take serious issue with the overindulgent composer - it's just an intro tune, fer chrissakes. Folks might stop to hear the full version once, but that's about it. Hell, 30 seconds would be fine, or cut it completely as it's not an integral part of Marathon operation. Game play is meant to be the primary focus, the rest is candy enhancers. If size versus candy tradeoff is harsh, then you cut stuff, period. This applies especially to Demos (which, by their very name, aren't supposed to be full all-singing, all-dancing packages of 25 Mb+).
Movie file. In a demo? Are you Serious?? For the Full Major Scenario only. A 2 level map with 20 Mb intro movie is such a poor and unconvincing idea. Try a 20 level map with an extra 10 Mb of new shapes and sounds and go with a 2 Mb movie instead. Or just chuck the movie altogether. Personally, I wish the movie played when you completed a game instead of started it - you don't go handing out all your chocolate rewards before the kids have started the race, do you? Intro movie wasn't Bungie's best idea - I wonder if this is why they elected not to use the feature in the end.
BTW kids, don't forget that QT movies can use MIDI tracks.... if you gotta have sound, consider this. And know all your video/audio compression formats too, and choose the best and most efficient for the job. Bear in mind, though, that a less efficient codec sometimes makes for a bigger but better quality file when unstuffed, but don't forget to factor in the StuffIt compression you get on top when you distribute it. This often makes up any lost ground that a less efficient QT codec leaves behind.
README. 20 page DOCMaker with full-screen PICTs instead of text? 40 page Acrobat doc with bundled reader? F**k em all. I have seen things like this used as props for otherwise crap piles of work (can you say 4 Mb "Extreme" download...?), but a big download is only worth its contents - nobody is impressed by large downloads alone. You've seen that website recently plugged on Scenario News (is it the 'M'thon Dark Vengeance' one, perchance? BTW, how many other "Dark Vengeances" can you name in under 60 seconds?) which has JPEGs of text instead of plain text? Somebody tried to be clever and indulgent with presentation there. Result? Totally unreadable, big waste of bandwidth. Compare with TM1 site (same red text on black approach. Waaaay better in execution [Chris A was da man]). Off topic, but the same issues that apply. Especially for a demo, a SimpleText README is all you need. Everyone can read one of those [unlike cretins who still do Word READMEs, for example]. Note that the Trojan README contained several PICTs and a sound loop Easter egg, and weighed in at a mere fraction of the total package size, and was still the biggest indulgence I made (but if you found the sound loop you'd agree it was just about worth it:). Of course, TM1 was the last sub-10 Mb Distributable TC for Marathon (major one, anyway - Wheels was only a couple Mb stuffed, although obviously it contained much less stuff than any 20 level TC could).
Finally, the one thing I didn't mention in any of the above points because it applies equally to all of them, BTW, is to check everything supplied and 'pull' any and all unused stuff. That means blue-pixeling any unused artwork, and deleting unused sounds [where you would remove all sounds, either replace with a single silent, short sound [ie. v. small] to avoid leaving the collection completely empty [crashes Marathon Infinity if that slot should ever be called], or I think setting slot ID to -1 will make an empty slot be ignored even if the engine does try to call it. If any support file doesn't contain any original Marathon stuff any more then it's much easier just to distribute the file rather than any patcher. I'd recommend making the package install as a standalone thing like EVIL or TI do, rather than bunging stuff into the standard Marathon folder. Don't forget to change the STR#129 to alter the Prefs filename at the very least though! [The prefs file contains filenames for last selected map and seems to be path-aware, so an application in one folder with one set of support that shares the same prefs file with another M-application in another folder with its own set of support files - well, you can work out the confusion this causes for the two applications! Major potential crash-city.]
When it's all put together, be intelligent and check through everything by eye. And when all is done and it absolutely is as small as you get (even if it means cutting some of the fancier features - which is common enough for demos of commercial games, so feel no shame in doing same thing here too), VISE the bastard. [For those without access to the VISE Installer: Mindvision has kindly provided a license to Claude Errera for the building of Marathon scenarios. Contact him for more information.]
As I've also said, try different compressing methods where possible (eg. PICTs) and then test- Stuff to see how much size you save via codec compression within the file itself, and how much StuffIt compression you get. This is why I prefer 256-colors indexed over full-color JPEG - with high quality the jpeg is bigger, and with low it's same size but quality is shot, and also a stuffed JPEG is bigger/same size than a stuffed indexed PICT [don't forget that PICT format has its own degree of compression, and it generally gets a fair bit smaller still once stuffed]. Going only by size of a file as it sits on your hard disk is not a reliable guide to what its distributable size will be.