Video Stream Processing

Michael Frank Deering: Hardware: Video Stream Processing

Beginnings

The Schlumberger Palo Alto Research lab (SPAR) was initially housed inside a convenient company that Schumberger had bought in Palo Alto (Fairchild Semiconductor) in 1981. (They initially tried to call it “the Fairchild AI Lab”, but the acronym, “FAIL”, didn’t go over well, so for a while it was called “Fairchild Laboratory for AI Research”: “FLAIR”.)

This meant that in additional to the AI work, we could also play around with VLSI chip design. Using the Mead-Conway-Lyon tall thin man design methodology, several of us tried our hands at full custom layout. For my project, I designed an image processing chip that had compatible streaming input and output pins. That meant that you could string together any number of the chips, or mix and match any additional chip that used the same pixel communication protocol. After coming back from fab, my test chip didn’t function, but the VSP (Video Stream Processing) concept was born.

Concept

Like anyone who had written AI computer vision code (and that included a lot of other people in the lab), I thus also had to be accomplished at writing image processing code and packages. My boss at the time, Marty Tennenbaum, and I had the initial concept to come up with a small family of orthogonal image processing functions that could be built as separate dedicated VLSI chips, and then wired together in all manner of ways to solve various imaging, graphics, and computer vision problems. The pixel data was generally 8-bit grey scale. The initial list of chip functions included:

  • Frame Buffer Controller. This chip streamed pixel I/O into an attached bank of DRAM
  • Dither Chip. This chip dithered a incoming grey scale pixel stream into a (scaled and dithered) binary pixel stream.
  • Median Filter. This chip would output the medium (or more generally nth) value of input pixels within a 3×3 pixel window from the input stream.
  • Convolution. This chip would perform and output a programmable weighted 3×3, 5×2, or 11×1 convolution of a window from the input stream.
  • ALU. This chip would perform a verity of ALU options between two input pixel streams to produce its output stream. The functions included add, multiply, min, max, etc.
  • Bit-plane Area Correrlator. This chip would massively compare offset portions of the input data with stored patterns. Though done differently, the closest functionality nowadays is the neighbor search performed in motion coding.
  • Delay line chip. This helped form larger tap points for filters.
  • Multi-statistics Gatherer. This chip would compute the histogram (and other statistics) of the image stream flying by.
  • Weighted Median Filter. This is a more general median filter operation.

A very interesting aspects of the project was the simulator that Neil Hunt and I built. You created and drug around the screen icons representing instances of VSP chips. Camera or disk icons connected to real cameras or sequences of images on disk, CRT icons connected to output windows on the workstation stream. You could click on any chip icon to set its internal parameters. Then you hit the “run” button, and a very efficient simulation (out of order internally) of your VSP chip circuit would execute and show you the output images.

Execution

Several of these chips were started in design. The median filter chip came back from fab fully functional. But this presented management with a problem: here was a research projects that was actually working, and the next step involved actually spending money. At the time Fairchild was in the traditional mature company trap: it was generating only enough profit (if any) to invest in minimal improvement to mature products that were already shipping; there was no money left for anything that might actually lead to growing revenue in the future. Schlumberger was also getting to the end of its investment rope, and started looking at selling Fairchild to Fujitsu. Fujitsu. was already interested in the VSP chips, but everything was put on hold as Schlumberger tried to put a deal together, and ran afoul of the U.S. government about certain critical Fairchild technologies. By the time Fairchild was eventually sold to National Semiconductor (except the Clipper CPU that was sold to Intergraph), it was too late to continue with the VSP product line. In any case, I had already moved over to the non-Fairchild Schlumberger AI lab. I did get some interesting trips to Japan in the interim.

Publications

There were only internal SPAR technical reports about the VSP project. Neil Hunt’s public Ph.D. from the U.K. did document several aspects of the simulator, and a later paper he published in IEEE PAMI described a later software image processing package he developed using concepts from the VSP simulator.