Michael Frank Deering: Software: HoloSketch

Date of first Prototype: January 1994


This was a research project I designed to see what would happen when a high-resolution virtual reality environment was always available. In the main, HoloSketch was a system for creating, editing, and animating 3D objects all in VR. But HoloSketch was also used as a 3D playback environment, a demo system, and a research platform for debugging many 3D graphics concepts: compressed geometry, 3D scene graphs, and per-pixel programmable shading hardware. There was never any intent to release HoloSketch as a commercial product, but a sub-set of HoloSketch was released as a real-time demo with all Creator3D systems, and the design of Java 3D was heavily influenced by our experience with HoloSketch. The second version of the Virtual Portal was run by HoloSketch, and several HoloSketch and HoloSketch related papers were produced.


HoloSketch included a number of novel features. Besides always working in a head-tracked stereo environment, one of its most distinctive features was its multiple level 3D pie menus. HoloSketch was designed to showcase how many common 3D interaction tasks could be 10X more productive in a head-tracked stereo environment than a traditional flat screen environment.

I remember the day a student from Brown University came by to show his program that “speed up” the placement of objects in a 3D environment by displaying shadows of the objects being moved on a box containing the work piece. His particular example was an airplane for which the task was to place a propeller on the right part of the plane. He had statistics that show that the task took 45 seconds to a minute using “normal” CAD display techniques, but with his shadow technique after a little practice users averaged 25 to 30 seconds total. After his demo, I quickly sketched an airplane and propeller in HoloSketch, and handed the 3D mouse to the student (who had never used HoloSketch before). On the first try, it took him less than 5 seconds to properly place the propeller. After a little bit of practice, I got it down to 2 seconds. I think he published his thesis anyway.

Who Did What

I personally wrote and maintained the majority of the code in HoloSketch over a several year period, much of it as a hobby. However a number of consultants were hired and paid by Sun to write specific sub-portions of the code; without their help the system wouldn’t have made several important internal delivery dates. Sun did leverage this work not only by demos, but by the later re-use of concepts (not code) in Java 3D.


HoloSketch was written on top of Sun’s XGL 3D API (OpenGLTM hadn’t been created yet). After XGL was EOLed in Solaris 7, this made it hard to continue to run HoloSketch (though XGL ran fine in “unsupported mode” in Solaris 8). My general plan was for Java 3D (which runs on both OpenGL and Direct3DTM) to replace the lower layers of HoloSketch, and for the rest of HoloSketch to be re-written in Java on top of Java 3D. However, for this to happen, several minimum features had to make it into Java 3D, and this was in the process of finally happening just the main Sun Java 3D project was put on hold. It also turned out that there was a lot of code and features in HoloSketch beyond what was supported in Java 3D; I estimate that Java 3D as it currently is still only covers the lower 20% of HoloSketch’s functionality (which is as it should be; Java 3D is an API, not a CAD application). While Java 3D was much more professionally written, the total size of Java 3D source code just barely caught up to the size of the HoloSketch source base.

There still really isn’t a realistic commercial market for a product like HoloSketch, and there won’t be until several other hardware limitations (opined about elsewhere) are overcome. Meanwhile mainstream packages like Alias Maya have made several user interface improvements that have fixed some of the worst interaction productivity problems in those packages. Every now and then I think about writing a HoloSketch-like package from scratch, but given the number of man-years that went into the original one, and the lack of appropriate hardware support (trackers and displays more than rendering hardware) it is hard to justify. In the long term I fully believe that fully interactive immersive head tracked stereo 3D design products will come to market, but that could still be one or two more decades away.

Specifically, Sun never publicly released the full HoloSketch binary or source code, so it is a dead package in terms of anyone wanting to play with it. It is possible that it could be licenced from Sun, but at this point I think a total re-write of a similar package would be a better way to go if anyone wants to play in this space.


A medium length paper on HoloSketch was published in TOCHI; a shorter version was published in CACM:

Michael F. Deering “The HoloSketch VR Sketching System”, CACM 39(5): pages 54-61, 1996.

Michael F. Deering “HoloSketch: A Virtual Reality Sketching/Animation Tool”, TOCHI 2(3): pages 220-238, 1995

Performance Art

A number of real-time animations were created and publicly shown in HoloSketch. Many of these were done by Hallie McConlogue, who was a long suffering HoloSketch “test Artist”. Some of these productions were paid for by Sun for SIGGRAPH and various other special events and conferences, others were produced independently by Hallie for her own Performance Art exhibitions.