Monday, February 24, 2014

Comparison Of Two Kickstarter-funded Sous Vide Devices

Sous Vide

Sous vide cooking requires heating a tub of water to a precise temperature and holding it there over a long period of time (often several hours), putting food in plastic bags, evacuating the air from the bags (typically), and dropping the bags in the water for the desired period of time. Often this is followed by removing the food from the bags and normal-cooking it for a brief period (for example, to add a nice charred/seared finish).

There are two basic approaches for keeping temperature constant:
  1. A water bath with heating elements underneath (and sometimes also on the sides) that relies on convection to produce an even temperature throughout. One vendor of these calls them "water ovens".
  2. An immersion circulator that provides heat, plus actively circulates the water to ensure even temperature.
Both approaches use accurate thermometers and control devices (typically PIDs) to avoid thermal undershoot and overshoot. Both approaches have safety features such as low-level sensors, etc.

Advocates for immersion circulators have demonstrated unwanted temperature gradients in water ovens, by dropping a lot of frozen food in the water at once. Water-oven advocates claim this was an unfair test, at least for home use, because the quantity of frozen food was unrealistic. They may or may not have a point.

The one undeniable drawback to the water-oven approach is the size of the machine. In a restaurant, this is perhaps not a concern, but in a house it actually matters. The leading model of water ovens is as big as a bread maker, and for casual home use could be a boat anchor if it's not used all the time (which is true come to think of it of bread makers too). In contrast, an immersion circulator can be clipped into any large pot the home cook already owns, and can be stored in a small volume when not in use.

So, we wanted to buy an immersion circulator instead of a water oven.

Early sous vide experimenters repurposed expensive lab equipment, and at least one lab-equipment vendor (PolyScience) was smart enough to realize they had a new sales channel, and start making purpose-built immersion circulators.

But they were still expensive.

Note: PolyScience sells lower-priced immersion circulators: https://www.cuisinetechnology.com/sousvide-professional-comparison.php. But we didn't know about them at the time, or perhaps they were recently introduced. We only knew about immersion circulators costing $700 or more.

Kickstarters

While we were looking for an affordable model, Nomiku did a Kickstarter for a lower-cost (but still high-quality) immersion circulator: https://www.kickstarter.com/projects/nomiku/nomiku-bring-sous-vide-into-your-kitchen.

We invested in one.

Then Sansaire did a Kickstarter for another immersion circulator: https://www.kickstarter.com/projects/seattlefoodgeek/sansaire-sous-vide-circulator-for-199.

We invested in one of those as well, in case the Nomiku failed to deliver, or wasn't any good.

(Note that both projects were funded quickly and greatly exceeded their funding goals. There was obviously pent-up demand for such a product.)

Comparisons

Having now received both units and tried them out, here is our evaluation:

Nomiku
  • Smaller (width and height)
  • Quieter
  • More expensive
  • Cover is recommended to keep heat in, and pre-notched cover is not available from vendor
  • Slower to reach desired temperature
  • Smaller range of water levels
  • Harder to clean inside
Sansaire
  • Bigger (width and height)
  • Louder (but still not very loud)
  • Less expensive
  • No cover needed
  • Reaches desired temperature quickly
  • Larger range of water levels
  • Easier to clean inside
  • Some burrs on mounting clip (they're working on improving this)
Both
  • Well-designed
  • Well-manufactured
  • Well-documented
  • Well-packaged
  • Excellent customer support
  • Prompt shipping
  • Hold constant temperature correctly with little or no variance once equilibrium is reached
We haven't used either unit long enough to comment on long-term reliability. At some point I'll post an update.

Recommendations

If space is at a premium and/or if you plan to only cook small amounts, get the Nomiku. Otherwise, get the Sansaire.

Misc

Despite showing metal pots on their websites, both vendors recommend using Cambro polycarbonate tubs (12 quart for Nomiku: http://www.amazon.com/gp/product/B0001MRUKA/, 4.75 gallon for Sansaire: http://www.amazon.com/gp/product/B00ADPF9V2/). The tub recommended for the Sansaire has ridges on the underside that provide some insulation from cold countertops.

Although not required for the Sansaire, a cover will reduce evaporation, and holds in heat (which reduces energy cost, but does not affect the ability of the Sansaire to keep constant temperature). No pre-notched cover is available for the Sansaire (same as for the Nomiku), and cutting one would be problematic for a Sansaire because the hole would need to be some distance from the edge, producing a long void that would leak heat. If you want a cover with the Sansaire, you can use floating plastic balls, which are available from usplastics.com (case of 1000, 20mm, polypropylene floating spheres: http://www.usplastic.com/catalog/item.aspx?itemid=62936&catid=641). These are also sold by PolyScience, in smaller quantity, for a lot more money.

(PolyScience sells tubs and pre-cut covers, in various sizes, for their immersion circulators. It would be helpful if Nomiku did the same.)

Photos

Side-by-side comparison of Sansaire and Nomiku, in recommended tubs:


Nomiku cover, courtesy of my brother. Hole was cut with 2+1/4" hole saw in a drill. Notch connecting edge to hole was cut with fine-toothed hacksaw. Edges were lightly sanded, but googling around for advice on how to cut polycarbonate, some posts recommend lightly running a butane torch flame over the cuts, which will (they claim) melt them gently into a smooth finish.

If you screw up, you can get another top from Amazon for about $9.




Similar review on another site: http://mobile.seriouseats.com/2013/12/sous-vide-circulator-review-sansaire-nomiku-anova.html.

Inexpensive DIY Wine-preservation System

Wine goes bad pretty quickly when exposed to air, which wouldn't be a problem if every opened bottle was finished right away, but sometimes a bottle is only partially consumed.

To address this, there are many solutions available: google for wine preservation system.

The various solutions fall into one of these categories:
  1. Replace the bottle with a bag that can be squeezed so the wine comes up to the top, leaving no air to react.
  2. Displace the wine up to the top of the bottle by pouring clean inert beads into the bottle.
  3. Pour the wine into a smaller bottle, and have a set of bottles of various sizes.
  4. Remove the air from the bottle (by creating a partial vacuum in the bottle).
  5. Replace the air in the bottle with an inert gas.
Pros/cons:
  1. Bags work but turn an aesthetically pleasing bottle of wine into an ugly plastic sack. And the sack is difficult to clean and dry. Plus I don't really like the idea of wine sitting in contact with plastic for days.
  2. Beads work but are hard to clean and then dry (a colander is good for cleaning, but drying is a pain, because water sticks in the voids between the beads).
  3. Smaller bottles don't work as well as beads, because the step sizes between bottles are larger than the volume of one bead. And the bottles are hard to clean and dry.
  4. Creating a partial vacuum can diminish/change the flavor of the wine, probably because it draws off useful volatiles (for example, dissolved CO2).
  5. Using an inert gas works very well. The step size is the size of a molecule, and when the bottle is opened the gas disperses and there is nothing to clean or dry.
You can buy an inert-gas system. They cost a lot of money, and typically even if the base unit isn't outright expensive, the replacement gas canisters are expensive (including, oddly, Hazmat fees for shipping the bottles, even though the gas is harmless). They wouldn't need to be expensive, but, like giving away the handle and charging for the blades, that's how the inert-gas-wine-preservation-system vendors make their money.

You can instead create an inexpensive inert-gas solution entirely from off-the-shelf parts. You don't need to buy anything fancy.
  1. Call your local industrial-gas supply shop. For example, Airgas. They are pretty much everywhere, because welders, food systems, labs, etc. need various gases.
  2. Ask for their smallest tank of argon. I asked for food-grade argon, and they didn't have it in the smallest bottle size. But when I asked what the "contaminants" are in regular argon, they said it was just air, and the percentage is very low. They also said that's what wineries use (they don't need food-grade). Don't worry about it.
  3. While at the gas supplier, also get a regulator. Show them the photo below so they know what kind of regulator you need. Ask them to install and test it. Ask them to show you how to use it (hint: you need to turn the big valve on top of the tank first, then the little valve on the top left of the assembly in the photo).
  4. Pay for the regulator, the tank rental, and the argon.
  5. Take the tank/regulator assembly and an empty wine bottle to a local hardware store with a good selection of small pipes and adapters, and ask them to make something that looks like the photo.

The idea is for the narrow end to fit into the neck of the wine bottle, with room around the nozzle for air to escape as you fill the bottle. You might consider adding a small hose to the end of the fitting, to poke down into the bottle.

When filling a bottle with argon, open the small valve the least possible amount, to inject argon into the bottle as gently as possible.

When you feel you've replaced the air with argon as much as possible, quickly remove the nozzle, shut off the small valve (firmly, but don't overtighten), put the cork (or some other stopper) in the bottle, and then shut off the big value (again firmly, but don't overtighten).

Even the smallest bottle of argon lasts for ages--not much gas is used per wine bottle. We're still on our initial purchase of argon.

When you finally run out of argon, take the bottle/regulator assembly back to the supplier, and get a replacement bottle. (They just swap bottles, don't refill the one you have). Ask them to install and test the regulator on the new bottle.

Note: It's a good idea to attach a chain, wire, rope, strap, etc. to a wall and around some part of the argon tank (so it doesn't tip over and snap off the top, which is a rare occurrence but quite spectacular when it happens).

Caveat: Wineries have big tanks and special piping/techniques for sparging (filling the headspace with inert gas, http://morewinemaking.com/public/pdf/inertgas.pdf). When you manually squirt some gas into a wine bottle, even taking care not to create turbulence, you can't achieve the gentle laminar flow wineries achieve, so you probably aren't displacing all of the air so much as diluting the air. But that's still better than nothing, and in practice this has worked well for me.

Saturday, July 21, 2012

Well-designed, Easily Assembled, Reasonably Priced, Adjustable-height Desk

There are lots of companies selling adjustable-height desks, but they tend to cost nearly a thousand dollars, or more.

I finally found something that works and doesn't cost so much: http://heightadjustableworktable.com. If you already have a desk, you have a desktop, and can probably reuse it, so all you need is the frame. (If you don't have a top, they sell complete desks too.)

The frame breaks down into a fairly small, dense package. It only took about an hour to assemble (not counting idiotic rework because a certain hamfisted idiot flipped the top the wrong way the first time).

The mechanism is metal on nylon bushings, and operates smoothly.

The engineering and manufacturing are first-rate.

A very good value.

My only complaint is that the lowest height of the desk is still too high for a normal-height woman, and there's no way to shorten the legs because the mechanism is integral.

Saturday, July 14, 2012

Building A Completely Silent PC

Under load, a PC fan becomes a distraction while listening to music with quiet passages, coding, etc.

So I built a couple of completely silent PCs. (The optical drive makes noise, but I only used it to load the OS.)

If you want to do the same, you'll need to follow steps similar to these:
  1. Get a fanless-PC chassis. These are also called "media PCs", and they're silent so they can be used in home theaters without distraction. There are several manufacturers. The best price/performance ratio seems to be Streacom. You can get them from Perfect Home Theater, and from Quiet PC. Both vendors are a pleasure to work with. The prices were better at Perfect Home Theater, but he was out of stock in silver, so I wound up getting them from Quiet PC, and then got the accessories from him. The FC8 chassis I used has the smallest footprint, but requires an external power supply. For our offices, there wasn't space on the rack for a flatter, wider chassis like the FC5, FC9, or FC10. Also, I wanted front-panel USB sockets. Be careful to get fanless, because Streacom makes other models that look like the fanless versions, but aren't. You can get them with remote controls, which is useful for media PCs, and useless for a regular PC.
  2. Get the necessary parts. You will need a motherboard, CPU, RAM, SSD, and, if you want an internal optical drive, the special slimline optical drives from Perfect Home Theater. I wanted a fast system, so I used an Intel DH67CF motherboard, Intel Core i7 3770S 3.1 GHz 4 Core LGA 1155 CPU, Crucial 8GB RAM, and Intel 520 180 GB SSD. Not being a gamer, the CPU's audio and video is perfectly adequate, so I didn't need any other cards. Make sure you select a motherboard that is compatible with the chassis (Streacom lists compatible boards on their site--make sure you get one with SATA 6). The CPU I used is the fastest 65W available for a motherboard compatible with the FC8.
  3. Get some thermal paste. Selecting a paste feels like it takes longer than building the PC (http://www.maximumpc.com/article/features/geek_tested_17_thermal_pastes_facehttp://benchmarkreviews.com/index.php?option=com_content&task=view&id=150&Itemid=62). I wound up using Prolimatech PRO-PK1-5G, which is available from Newegg. The paste makes a mess no matter how hard you try to be careful, so be sure to put down some plastic or layers of paper towels, and wear some throwaway plastic or latex gloves if you have them.
  4. Follow the detailed and very helpful instructions on the Perfect Home Theater site. The two most-important pieces of information are the FC8 manual, and the connection map. Make sure you connect the SSD to a SATA 6 socket.
The only tools needed are two screwdrivers (small and really small), small wire cutters (if you want to cut off the floppy power pigtail), and a small cresent wrench (if you want to tighten the power socket more than finger tight, although finger tight seems pretty tight already). Magnetic screwdrivers are very helpful

There wasn't a lot of room between the micro-PSU and the right heatpipe, so I (gently!) bent the heatpipe up a bit, and cut off the Molex socket that faces into the case (because an identical socket on the other side of the micro-PSU faces away from the heatpipe).

I also cut off (carefully!) the power pigtail for a floppy drive, to remove a bit of clutter from the interior.

There are a number of small screws--get a bowl to put them in so they don't disappear.

It took about three hours to build the first one, due to fumbling around and learning how everything connects. The second took under an hour. (Those times do not include how long it took to load and configure the software.)

Sunday, June 17, 2012

Flossing, Seatbelts, And Dynamic Type Checking


"It's all fun and games until someone loses an eye."

Researchers, many of whom were really really smart, deduced that you should floss, and wear seatbelts, and eat a balanced diet.

In software, similarly intelligent researchers determined that strong typing was like flossing and the wearing of seatbelts: a very minor inconvenience that saved you a lot of trouble later on.

Now the trend is increasingly towards dynamic languages that discover type mismatches at runtime.

You know who does that discovering?

Your users.

The only reason your users aren't already at your gates with torches and pitchforks is that browsers turn off JavaScript errors by default.

Is this really the best we can do? The argument against static type checking boils down to "I'm a very careful driver", which is what every driver thinks right up until they get in an accident.

Languages with static type checking allow programmers to opt out (even Ada has unchecked conversion). That's analogous to allowing passengers to not put on their seatbelts if they're crazy enough not to want to wear them. JavaScript doesn't allow programmers to opt into static type checking. That's analogous to a driver taking all of the seatbelts out of the car, even those for the passengers, including children.

Sunday, June 10, 2012

Javascript Slideshow With Fade

I wanted a simple slideshow for our Analog Perfection™ website (don't bother going there--we won't have it up until just before launching the Kickstarter project).

The requirements were:
  • Free
  • Source code provided
  • Fade transitions
  • Small file size
  • Fast initial load (implying lazy loading of thumbnails)
  • Plain old Javascript (no dependencies on frameworks such as JQuery, MooTools, Dojo, Flash, etc.)
Nothing quite matched the requirements, so I wrote one, cobbling pieces together from answers on Stack Overflow, some help from Tyler Riding, Priyajeet Hora, Fro Rosqueta, and Doug Crossley (colleagues at Intuit), and a couple of key insights from the Tigra Fader sample code.

The result is under 4k, non-minified.

The code puts all of the slide thumbnails in one <a> element, as imgs, on top of each other, but with different z-ordering, and then adjusts the opacity of the current and next thumbnails.

You can see the pairwise opacity changes by looking at the elements in your browser's developer-tools console.

The HTML for all but the first thumbnail is generated dynamically, when the thumbnail lazy loads.

When the next thumbnail has crossed a threshold, the href is updated to point to that thumbnail's corresponding full-size picture.

Mousing into the thumbnail region accelerates the fading to 100%, then pauses the program. Mousing out of the thumbnail region resumes the program (as does clicking into the thumbnail).

I've only tested in on IE9 and Chrome so far, but have tried not to use anything quirksmode says won't work on older browsers. Eventually this will be deployed on a live site, and then I can start using something like browsershots to test, and then the cursing begins.

I embed the Javascript in my HTML like this:

<div id="slides">
<a id="slideLarge" href="./images/FrontView.jpg" target="_blank">
<img id="slideSmall0" src="./images/FrontViewSmall.jpg" alt="[unable to display picture]"/>
<!--INCLUDE("slides.js")INCLUDE-->
</a>
</div>

and use the Includer to expand the contents before deployment. If you don't want to do that, just lexically replace the INCLUDE line with the contents at the bottom of this post, or use a regular script src include.

Note that for browsers that have Javascript disabled, the first thumbnail and its associated full-size picture will still display correctly.

There are a number of arguments to the main function:
  • imgdir: Directory containing the thumbnails and full-size pictures--must end with '/'. If current directory, use "./".
  • fullid: HTML ID of the full-size picture.
  • idprefix: Shared prefix for the HTML IDs of all of the thumbnails.
  • width: Width in px (but without "px") of the thumbnails.
  • holdtime: How long in milliseconds to hold one thumbnail image constant before starting to fade.
  • fadetime: How long in milliseconds to take to perform a fade.
  • steps: How many steps to use to perform a fade. More is smoother but uses more CPU.
  • fastfadetime: How long in milliseconds to take to perform a fade when the mouse is over the thumbnail.
  • faststeps: How many steps to use to perform a fade when the mouse is over the thumbnail. More is smoother but uses more CPU.
  • flip: Percent opacity (as a number from 0.0 to 1.0) next thumbnail must have before clicking on it brings up its full-size picture instead of the current thumbnail's full-size picture.
  • names: Array of arrays where the inner arrays each have two elements: [the simple name of a thumbnail, the simple name of the full-size picture corresponding to the thumbnail]. The names of the first thumbnail and first full-size picture must not be included (they are extracted from the non-generated HTML during initialization).
The arguments are not passed in using name/value pairs, but are instead just positionally associated.

The thumbnails all have to be the same size (width and height) for the overlaying to work.

Using a naming convention for the shared ID prefix was done on purpose. If you don't like it, you can modify the code to add the prefix to the names as another element in each array, for all but the first thumbnail, and get rid of idprefix entirely. (But to do that, you need to obtain the ID of the 0th thumbnail by walking the DOM starting at the fullid. It's not difficult, but seemed like overkill.)

I considered using a naming convention to eliminate the need to pass in an array of arrays, and just pass in an array of names. But then what if the extensions differ between the two sizes of files?

Things left to do:
  • Add a row of numbers (or circles, or some other kind of icon) that indicates which slide is currently active, and to allow direct navigation to that slide by clicking on the number/icon, similar to how National Geographic and menucool do it. We don't really need it for our site, but it's an interesting exercise. To keep the layout clean, it could perhaps only display the row when the mouse is over the thumbnail, and display it over the thumbnail, at the bottom (not outside the thumbnail region), similar to how Dynamic Drive does it.
  • Add drag support, so the user can slide the thumbnails left or right by mouse down followed by mouse move. Don't treat the eventual mouse up as a click that opens the large picture.
  • Figure out how to get the width from the 0th slide, instead of having to pass it in, which is annoying. I tried style.clientWidth, style.offsetWidth, and various other suggestions online. Nothing worked, perhaps because at the point where the width is needed, the page hasn't been rendered yet, so the viewport size isn't known. If someone knows how to solve this, please post the solution!
Misc notes:
  • I tried a different approach before this one, but couldn't get it to work: manipulate the opacity of the link's background image and one img, and once whichever one of those is set to full opacity, replace the other one with a new image, then repeat the seesawing.
Here's the code, plus the CSS I used. (To get the pictures to line up, they need to share a margin, and to do that the margins have to be set on the elements--the CSS will help you get it working.)

CODE:
<script type="text/javascript" id="slidesjs">
<!--
// Copyright © 2012 jimandlisa.com. All rights reserved.
function slides(imgdir, fullid, idprefix, width, holdtime, fadetime, steps, fastfadetime, faststeps, flip, names) {

    "use strict";

    var full, thumbs, pos, cur, next, initial, paused, stepsize, steptime, curstep, step, up, down, flipped, dofast;

    function getname(url) {
        return url.substring(url.lastIndexOf('/') + 1);
    }

    function setsteps(fast) {
        if (fast) {
            stepsize = 1.00 / faststeps;
            steptime = Math.round(fastfadetime / faststeps);
            dofast = true;
        } else {
            stepsize = 1.00 / steps;
            steptime = Math.round(fadetime / steps);
            dofast = false;
        }
    }

    function setopac(style, opac) {
        style.opacity = opac;
        style.filters = "alpha(opacity=" + Math.round(opac * 100) + ')';
    }

    function fade() {

        if (paused) {
            if (!dofast) {
                setsteps(true);
            }
        }

        step = curstep * stepsize;
        up = (step > 1.000) ? 1.000 : step;
        down = ((up === 1.000) || ((1.000 - step) < 0.000)) ? 0.000 : (1.000 - step);
        up = (down === 0.000) ? 1.000 : step;
        curstep += 1;

        setopac(cur.style, down);
        setopac(next.style, up);

        if (!flipped && (up > flip)) {
            full.href = imgdir + names[pos][1];
            full.target = names[pos][1];
            flipped = true;
        }

        if (up === 1.000) {
            return;
        }

        setTimeout(function () { fade(); }, steptime);
    }

    function run() {

        if (!initial && !paused) {
            cur = thumbs[pos];
            pos = (pos + 1) % names.length;
            if (pos === thumbs.length) {
                next = document.createElement("img");
                next.id = idprefix + pos;
                next.src = imgdir + names[pos][0];
                next.alt = "[unable to display picture]";
                full.appendChild(next);
                setopac(next.style, 0.000);
                next.style.marginLeft = "-" + width + "px";
                next.style.zIndex = names.length - pos - 1;
                thumbs.push(next);
            } else {
                next = thumbs[pos];
            }
            curstep = 1;
            flipped = false;
            setsteps(false);
            fade();
        }

        initial = false;

        setTimeout(function () { run(); }, holdtime);
    }

    function noprop(e) {
        if (!e) {
            e = window.event;
        }
        e.cancelBubble = true;
        if (e.stopPropagation) {
            e.stopPropagation();
        }
    }
    function doAddHandler(obj, name, newhandler) {
        var curhandler = obj[name];
        if (curhandler === null) {
            obj[name] = newhandler;
        } else {
            obj[name] = function () {
                curhandler();
                newhandler();
            };
        }
    }

    function pause(e) {
        noprop(e);
        if (paused) {
            return;
        }
        paused = true;
    }

    function resume(e) {
        noprop(e);
        if (!paused) {
            return;
        }
        paused = false;
    }

    function init() {

        pos = 0;

        full = document.getElementById(fullid);
        full.target = names[pos][1];

        cur = document.getElementById(idprefix + '0');
        names.splice(0, 0, [getname(cur.src), getname(full.href)]);
        //width = cur.clientWidth;
        thumbs = [];
        setopac(cur.style, 1.000);
        cur.style.zIndex = names.length - 1;
        thumbs.push(cur);

        next = null;
        initial = true;
        paused = false;
        setsteps(fadetime, steps);

        doAddHandler(full, "onmouseover", pause);
        doAddHandler(full, "onmouseout", resume);

        run();
    }

    doAddHandler(window, "onload", init);
}

slides("./images/", "slideLarge", "slideSmall", 650, 5200, 1750, 80, 700, 70, 0.6, [["SideViewSmall.jpg", "SideView.jpg"], ["RearViewSmall.jpg", "RearView.jpg"]]);
//-->
</script>

CSS:
#slides {
    float:left;
    width:650px;
    padding:0px;
    margin-left:100px;
    display:inline;
}

#slides img {
    float:left;
    border:0px;
    position:relative;
}

Saturday, June 2, 2012

A Utility To Inline Shared HTML

A website typically has sections that are shared among multiple pages. For example, a common header and footer. It makes sense to factor the shared sections into separate included files. But the include mechanism is not standard: some servers support SSI, other servers require using PHP includes or JSP, still others have no server-side support for includes and it has to be done with JS writes. None of these approaches seem satisfactory. When using a not-universally supported server-side mechanism, files may require rewriting when changing hosting providers. When using a client-side mechanism, there are more requests to the server. This utility eliminates this problem by inlining included HTML, CSS, and JavaScript before deployment:
/**
 * Copyright (c) 2012 jimandlisa LLC. All rights reserved.
 * This program is made available under the terms of the
 * Eclipse Public License v1.0, which is available at
 * http://www.opensource.org/licenses/eclipse-1.0.php.
 */

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

/**
 * Given a directory of source for a website, recursively navigates the directory structure,
 * inlining included files (which may in turn include files), and saving the result to a staging
 * directory. This eliminates the need for include-file solutions such as SSI, PHP, or Javascript.
 * It also speeds up page loads and eliminates requests. Included files must be located in a directory
 * called "includes" located in the root of the directory containing the source. Include directives
 * must have the prefix and suffix pairs defined by INCLUDES_DELIMITERS, and the name of the included
 * file must be between the left and right parentheses defined by the prefix and suffix. Included files
 * must have simple names (no directories in the names), and must contain text. Does not process individual
 * files, only the entire site; for large sites this would not be acceptable, but for our application
 * it only takes ~30 ms. Included files must form a directed acyclic graph (no cycles); the utility does
 * not perform cycle detection, but by going into an infinite loop it's pretty obvious when there is a cycle,
 * and the log output makes it easy to determine the cause.
 * @author Jim Showalter
 */
public class Includer {
    
    /**
     * Reserved name of includes directory. Must be contained directly within source directory.
     * Could of course be turned into an argument to the program, but was not necessary for
     * our application.
     */
    private static final String INCLUDES_DIR = "includes";
    
    /**
     * Set of types that support includes. Must all be text format.
     */
    private static final Set<String> INCLUDING_TYPES = new HashSet<String>();
    
    static {
        // Initialize including types.
        INCLUDING_TYPES.add("html");
        INCLUDING_TYPES.add("css");
        INCLUDING_TYPES.add("js");
    }
    
    /**
     * Allowable prefixes for includes. Must match pairwise with allowable suffixes.
     */
    private static final List<String[]> INCLUDES_DELIMITERS = new ArrayList<String[]>();
    
    static {
        // Initialize includes delimiters.
        INCLUDES_DELIMITERS.add(new String[] {"<!--INCLUDE(\"", "\")INCLUDE-->"}); // HTML
        INCLUDES_DELIMITERS.add(new String[] {"/*--INCLUDE(\"", "\")INCLUDE--*/"}); // CSS
        INCLUDES_DELIMITERS.add(new String[] {"//--INCLUDE(\"", "\")INCLUDE--//"}); // JavaScript
    }
    
    /**
     * Outputs usage error and exits program with error code.
     * @param explanation Explanation of error.
     */
    private static void usage(String explanation) {
        System.out.println(explanation);
        System.out.println(Includer.class.getSimpleName() + ": <source directory> <target directory>");
        System.exit(-1);
    }
    
    /**
     * Determines longest common prefix substring for a pair of paths. Used to shorten log messages.
     * @param sourcePath Source path.
     * @param targetPath Target path.
     * @return Longest common prefix.
     */
    private static String longestCommonPrefix(String sourcePath, String targetPath) {
        
        String longerString = null;
        String shorterString = null;
        
        if (sourcePath.length() > targetPath.length()) {
            longerString = sourcePath;
            shorterString = targetPath;
        } else {
            longerString = targetPath;
            shorterString = sourcePath;
        }
        
        String longestCommonPrefix = null;
        
        for (int i = 0; i < shorterString.length(); i++) {
            if (shorterString.charAt(i) != longerString.charAt(i)) {
                longestCommonPrefix = shorterString.substring(0, i - 1);
                break;
            }
        }
        
        return longestCommonPrefix;
    }
    
    /**
     * Shortens path by removing the longest prefix it has in common with another path.
     * @param path Path to shorten.
     * @param longestCommonPrefix Longest common prefix.
     * @return Shortened path.
     */
    private static String shorten(String path, String longestCommonPrefix) {
        return path.substring(longestCommonPrefix.length(), path.length());
    }
    
    /**
     * Logs a progress message.
     * @param prefix Beginning of logged message.
     * @param source Source file or directory.
     * @param longestCommonPrefix Longest common prefix.
     */
    private static void log(String prefix, File source, File target, String longestCommonPrefix) {
        String sourcePath = shorten(source.getAbsolutePath(), longestCommonPrefix);
        String targetPath = shorten(target.getAbsolutePath(), longestCommonPrefix);
        System.out.println(prefix + " copy: " + sourcePath + " => " + targetPath);
    }

    /**
     * Determines whether file name ends with a suffix that is one of the including types.
     * @param fileName File name to check.
     * @return True if file is an including type, false otherwise.
     */
    private static boolean isIncludingType(String fileName) {
        
        int dotIndex = fileName.lastIndexOf('.');
        
        if (dotIndex == -1) {
            return false;
        }
        
        String extension = fileName.substring(dotIndex + 1);
        
        return INCLUDING_TYPES.contains(extension);
    }
    
    /**
     * Extracts name of included file from current line if it is an include directive.
     * @param line Line.
     * @return Include file name, or null if current line is not an include directive.
     */
    private static String includeFileName(String line) {
        String trimmedLine = line.trim();
        for (String[] delimiters : INCLUDES_DELIMITERS) {
            String prefix = delimiters[0];
            String suffix = delimiters[1];
            if (trimmedLine.startsWith(prefix) && trimmedLine.endsWith(suffix)) {
                return trimmedLine.substring(prefix.length(), trimmedLine.length() - suffix.length());
            }
        }
        return null;
    }

    /**
     * Copies a file that does not have includes. Does not use NIO's transferTo or other new mechanisms
     * because, depending on file size, OS, etc., performance can actually wind up worse.
     * @param sourceFile File to copy from.
     * @param targetFile File to copy to.
     * @param longestCommonPrefix Longest common prefix.
     * @throws IOException Problem encountered opening, reading, creating, writing, or closing files.
     */
    private static void simpleCopy(File sourceFile, File targetFile, String longestCommonPrefix) throws IOException {
        
        log("simple", sourceFile, targetFile, longestCommonPrefix);
        
        InputStream in = new FileInputStream(sourceFile);
        OutputStream out = new FileOutputStream(targetFile);
        byte[] buf = new byte[1024*10];
        int len;
        while ((len = in.read(buf)) > 0) {
           out.write(buf, 0, len);
        }
        in.close();
        out.close();
    }
    
    /**
     * Platform-specific line separator.
     */
    private static final String LINE_SEPARATOR = System.getProperty("line.separator");
    
    /**
     * Reads contents of include file into memory, and caches it so it doesn't have to be read again.
     * For large sites this could be too much of a memory hog, in which case an eviction strategy
     * could be implemented (for example, keep the most-often referenced files).
     * @param includesDirName Name of includes directory.
     * @param includeFileName Name of include file.
     * @param includeFiles Cache of already included files.
     * @return Contents of include file.
     * @throws IOException Problem encountered opening, reading, or closing include file.
     */
    private static StringBuilder get(String includesDirName, String includeFileName,  Map<String, StringBuilder> includeFiles) throws IOException {
        
        System.out.println("get: " + includeFileName);

        StringBuilder includeFileContents = includeFiles.get(includeFileName);
        
        if (includeFileContents == null) {
            includeFileContents = new StringBuilder();
            File includeFile = new File(includesDirName + File.separator + INCLUDES_DIR + File.separator + includeFileName);
            BufferedReader reader = new BufferedReader(new InputStreamReader(new DataInputStream(new FileInputStream(includeFile))));
            
            String line;
            while ((line = reader.readLine()) != null) {
                String subIncludeFileName = includeFileName(line);
                if (subIncludeFileName != null) {
                    StringBuilder subIncludeFileContents = get(includesDirName, subIncludeFileName, includeFiles);
                    includeFileContents.append(subIncludeFileContents.toString());
                } else {
                    includeFileContents.append(line);
                    includeFileContents.append(LINE_SEPARATOR);
                }
            }
            
            reader.close();
            includeFiles.put(includeFileName, includeFileContents);
            System.out.println("put: " + includeFileName);
        }
        
        return includeFileContents;
    }

    /**
     * Copies a file that might have includes.
     * @param sourceFile File to copy from.
     * @param targetFile File to copy to.
     * @param longestCommonPrefix Longest common prefix.
     * @param includeFiles Cache of already included files.
     * @throws IOException Problem encountered opening, reading, creating, writing, or closing files.
     */
    private static void includeCopy(File sourceFile, File targetFile, String longestCommonPrefix, Map<String, StringBuilder> includeFiles) throws IOException {
        
        log("file", sourceFile, targetFile, longestCommonPrefix);

        BufferedReader reader = new BufferedReader(new InputStreamReader(new DataInputStream(new FileInputStream(sourceFile))));
        BufferedWriter writer = new BufferedWriter(new FileWriter(targetFile));

        String line;
        while ((line = reader.readLine()) != null) {
            String includeFileName = includeFileName(line);
            if (includeFileName != null) {
                StringBuilder includeFileContents = get(sourceFile.getParent(), includeFileName, includeFiles);
                writer.write(includeFileContents.toString());
            } else {
                writer.write(line);
                writer.newLine();
            }
        }

        reader.close();
        writer.close();
    }
    
    /**
     * Copies a source directory to a target directory, recursively, resolving any included files it encounters.
     * @param sourceDir Directory to copy from.
     * @param targetDir Directory to copy to.
     * @param longestCommonPrefix Longest common prefix. (For logging, to shorten messages.)
     * @param includeFiles Cache of already included files.
     * @throws IOException Problem encountered opening, reading, creating, writing, or closing files.
     */
    private static void copy(File sourceDir, File targetDir, String longestCommonPrefix, Map<String, StringBuilder> includeFiles) throws IOException {
        
        log("directory", sourceDir, targetDir, longestCommonPrefix);
        
        if (!targetDir.exists()) {
            targetDir.mkdirs();
        }
        
        File[] files = sourceDir.listFiles();
        for (File file : files) {
            if (file.isDirectory()) {
                File dir = file;
                String dirName = dir.getName();
                if (dirName.compareTo(INCLUDES_DIR) != 0) {
                    File subTargetDir = new File(targetDir.getAbsolutePath() + File.separator + dirName);
                    copy(dir, subTargetDir, longestCommonPrefix, includeFiles);
                }
            } else {
                String fileName = file.getName();
                String targetFileName = targetDir.getAbsolutePath() + File.separator + fileName;
                File targetFile = new File(targetFileName);
                if (!isIncludingType(fileName)) {
                    simpleCopy(file, targetFile, longestCommonPrefix);
                } else {
                    includeCopy(file, targetFile, longestCommonPrefix, includeFiles);
                }
            }
        }
    }
    
    /**
     * Main entry point for program.
     * @param args Command-line arguments.
     * @throws IOException Problem encountered opening, reading, creating, writing, or closing files.
     */
    public static void main(String[] args) throws IOException {
        
        // Validate the args.
        
        if (args == null) {
            usage("Null args");
        }
        
        if (args.length != 2) {
            usage("Wrong number of args");
        }
        
        String sourceDirName = args[0];
        
        if (sourceDirName == null) {
            usage("Null source directory");
        }
        
        if (sourceDirName.trim().length() == 0) {
            usage("Blank source directory");
        }
        
        String targetDirName = args[1];
        
        if (targetDirName == null) {
            usage("Null target directory");
        }
        
        if (targetDirName.trim().length() == 0) {
            usage("Blank target directory");
        }
        
        if (sourceDirName.startsWith(targetDirName)) {
            usage("Source directory child of target directory");
        }
        
        if (targetDirName.startsWith(sourceDirName)) {
            usage("Target directory child of source directory");
        }
        
        File source = new File(sourceDirName);
        
        if (!source.exists()) {
            usage("Non-existent source directory");
        }
        
        if (!source.isDirectory()) {
            usage("Non-directory source directory");
        }
        
        // Initialize and perform the copying.
        
        File target = new File(targetDirName);
        
        Map<String, StringBuilder> includeFiles = new HashMap<String, StringBuilder>();
        
        long start = System.currentTimeMillis();
        
        try {
            copy(source, target, longestCommonPrefix(sourceDirName, targetDirName), includeFiles);
        } catch (Exception e) {
            System.out.println("Encountered problem copying source to target: " + e.getMessage());
            e.printStackTrace();
            System.exit(-1);
        }
        
        long end = System.currentTimeMillis();
        
        System.out.println("done in " + (end - start) + " ms");
    }
}
An example file set up to be processed by the utility looks like this:
<!--INCLUDE("top.html")INCLUDE-->

<body id="manual">

<div id="header" class="header">
<h1 class="header">User Manual</h1>
</div>

<!--INCLUDE("nav.html")INCLUDE-->

<div>
<p>TODO: Put safety instructions and user manual here.</p>
</div>

<!--INCLUDE("bottom.html")INCLUDE-->