In this talk, I will present ongoing work into a real-time, computer vision based system to assist persons with dementia during handwashing. Assistance is given in the form of verbal and/or visual prompts, or through the enlistment of a human caregiver's help. The system uses only video inputs, and combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision theoretic framework for computing policies of action -- specifically a partially observable Markov decision process (POMDP). A key element of the system is the ability to estimate and adapt to user states, such as awareness, responsiveness and overall dementia level. I will give an overview of the system, followed by details of the tracking and decision making components. I will discuss the specific POMDP we have developed for the handwashing task, and I will detail the approximation methods we have applied to make the policy computation tractable. I will contrast the policies we have obtained against heuristic and hand crafted policies. I will show results both from simulations, and from real-time interactions with actors.
Dr Jesse Hoey
Jesse Hoey received the B.Sc. degree in physics from McGill University in Montreal, Canada, the M.Sc. degree in physics and the Ph.D degree in computer science from the University of British Columbia in Vancouver, Canada. His postdoctoral research was carried out at the University of Toronto, jointly in the Department of Computer Science and the Department of Occupational Science and Occupational Therapy. He is currently a Lecturer in the School of Computing at the University of Dundee, Scotland, and an Adjunct Scientist at the Toronto Rehabilitation Institute in Toronto, Canada. His research focuses on planning and acting in large scale real-world uncertain domains using video observations. In particular, he works on applying decision theoretic planning, computer vision, and machine learning techniques to adaptive assistive technologies in health care.