Mobile wearable computers are intended to provide users with real-time access to information in a natural and unobtrusive manner. Computing and sensing in these devices must be reliable, easy to interact with, transparent, and configured to support different needs and complexities. Therefore, one critical factor for the success of a wearable computer is its user interface. This dissertation presents a real-time robust multimodal unobtrusive user interface comprised of a vision-based robust finger tracking algorithm combined with audio-based control commands, wherein the interface is used to segment out objects of interest in the environment by encircling them with the user’s pointing fingertip. In order to quickly extract the objects encircled by the user, this unobtrusive interface uses a single head-mounted camera to capture color images, which are then processed using algorithms to perform: color segmentation, fingertip shape analysis, perturbation model learning, and robust fingertip tracking. Then, a wearable computer system may use object recognition algorithms to identify the object segmented by the user’s hand gesture, and may return an audio narration, telling the user information concerning the object’s classification, historical facts, usage, etc. This interface is designed to be robust to changes in the environment and user’s movements by incorporating a state-space estimation with uncertain models algorithm, which attempts to control the influence of uncertain environment conditions on the system’s tracking performance by adapting the tracking model to compensate for the uncertainties inherent in the data collected with a wearable computer. For a wearable computer system, these uncertainties arise from the camera moving along with the user’s head motion, the background and object of interest moving independently of each other, the user standing still or randomly walking, and the user’s pointing finger abruptly changing directions at variable speeds. The robust unobtrusive multimodal interface developed in this dissertation has been tested on a real wearable computer system, and the performance results obtained during these tests are presented in this dissertation.