Toshiba Superresolution yesterday, Resolution+ today

Apparently, image processing scientists sent enough nastygrams to Toshiba about their branding of the Toshiba Regza’s cell-chip-based HDTV video enhancement technology as “super resolution” — that they decided to rebrand it, going forward:

Crave inadvertently outed them:

http://asia.cnet.com/crave/2009/01/09/toshiba-regza-xv645-series-touts-resolution-technology/

So, what is “super-resolution”, and what is the dust-up then? 

“Superresolution” is actually a scientific/optical term, and it refers to a system being able to resolve images past the optical diffraction limit of the optical system that captured the images.  Note, please, that I said “images”, in plural.  It is possible to resolve beyond the quantum diffraction limit of an optical system by incorporating multiple images of the same scene, all captured through the same optics, so long as something changes in each image — like the camera moving slightly, the exposure or apparent f-stop changing, or even pictures taken with different spectral footprints.  The reconstruction is STILL constrained by the Shannon limit of all the information present, but with enough pictures, of the same scene, even if the sampling process is limited by optics, sampling errors, or integration errors due to the sensor(s), along with other impairments. Enough information is actually encoded in the aliasing present in neighboring pixel values to reconstruct a higher-resolution picture of the captured scene.

An example of this sort of magic:

DARPA’s Super-Resolution Vision System uses heatwaves to magnify targets

The first time many folks heard about this technology was with image reconstruction during the Mars Pathfinder mission (photos courtesy of JPL):

 

Mars Pathfinder (Before Superresolution)

Mars Pathfinder (Before Superres)

Mars Pathfinder (After Superresolution)

Mars Pathfinder (After Superresolution)



Like the TV show CSI with the magic “clean up that photo” button on the forensic technician’s computer, not only can one read a license plate by taking some pictures from low-Earth orbit of Khalid Sheik Mohammed’s car, but discern the exact species of flies orbiting his armpits and quality of recent dental work (or lack, thereof) — given enough relatively low-resolution pictures.  (This is mostly a monotonically increasing function of the contempt for the target possessed by the owner/operator of the satellite with the big honking lens on it).  

We’re not talking about simple “image stacking” here — what we’re talking about is essentially solving a massive linear system of equations, where every pixel is but one element in a massive matrix in three dimensions that defines a (sometimes very) underdetermined linear system of equations to be solved.  The solution to the equations is the image with restored resolution — super-resolution.  Normally, it takes quite a bit of computer time to solve these things for the kinds of images we’re talking about, even if the precise telemetry of the satellite is known (and you can bet it is).  There is currently nothing on the market that I’m aware of (e.g. isn’t a State Secret) that will perform real-time super-resolution processing on large-format images (let alone for video).

vReveal and Ikena Reveal from MotionDSP are good relatively lightweight examples of such software, but it’s not nearly fast enough to perform superres on hi-def video in real-time, and won’t be for a while.

Meanwhile, scientists get a little prickly when such scientifically precise terms as “radionuclide cross section”, “quantum superposition”, “proven”, and “super-resolution” get abused by overzealous marketing folks hoping to hype the latest technology.   In Toshiba’s case, some interesting posts at AVSForum caught some powerpoints from Toshiba researchers (in Japanese) which someone was kind enough to translate.

Official Toshiba XV545U Thread — AVS Forum

What Toshiba has cooked up is an impressive non-linear, adaptive edge and texture enhancement process that segments the image into homogenous, textured, and edge-containing regions, and applies selective processing to each image.  Singularly.  While this is clever, this is not the first instance of such segmented, masked video processsing, but most importantly, it’s definitely NOT “super-resolution” — it does not incorporate pixel information from multiple video frames.

Taking a single video frame, and figuring out which parts are “edges” and which parts are “texture” and applying selective processing for upscaling is not “superresolution” — it’s more properly called “resampling bandwidth synthesis”.

However, I imagine that consumer applications for real “super resolution” algorithms will appear and become prominent fixtures in the next few years.

Advertisements

~ by opticalflow on February 15, 2009.

3 Responses to “Toshiba Superresolution yesterday, Resolution+ today”

  1. Indeed, all “super resolution” is not created equal. As you’ve pointed out, the multi-frame video-reconstruction technology in MotionDSP’s software is the real deal (which is why the Secret Service and other intelligence agencies are using it), so we definitely raise an eyebrow around the office when we see a company touting so-called “super resolution” tech based on single-frame techniques.

    Mike
    Product Manager
    MotionDSP

  2. I’m “in violent agreement” with you.

    While we’re talking about MotionDSP, there is an item of some curiosity to me.

    MotionDSP embarked on consumer applications with a SaaS approach (software-as-a-service), and then later decided to go with shrinkwrap. Is there anything you can comment on generally that might be instructive as to why you made that decision? (For example, is this indicative of SasS problems in general, or are the economics just peculiar to your type of application?)

    Anyway, good luck with vReveal!

    -opflow

  3. Software-as-a-service makes a lot of sense for plenty of applications, but with video, of course, the bottleneck is bandwidth. In the case of FixMyMovie (our former Saas approach that you mentioned), users had to wait for their videos to upload and then wait for the “cloud” computers to finish processing before the users could see any results.

    We think that consumers will find the experience with our video enhancement technology a lot more compelling and enjoyable when they can preview enhancements to their videos in real-time on their desktops with vReveal.

    By operating on the desktop, vReveal can leverage the power of the user’s hardware for speed (especially since we ported to NVIDIA graphics processors using their CUDA language) and can give users more control over enhancement options.

    Thanks for the well-wishes!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: