Applied AI Session: Cognitive Models in Deep Learning, Idan Schwartz

The quest for algorithms that enable cognitive abilities is an integral part of machine learning and appears in many facets, such as virtual assistants and visual reasoning. A cognitive system must be capable of processing details from the multiple sensors that pound a device's computation engine. Interpreting observed objects requires an understanding of their semantics. Additionally, it must be sensitive to and pick out relevant nuances relevant to the task. During this presentation, we will present ways to assess and improve perception. We will explore ways to leverage large-scale models (such as CLIP). As a final step, we propose a novel attention mechanism, called Factor Graph Attention, which can operate on any data utility and distinguish useful signals from distracting ones. Our discussion will focus on the limitations of current methods: (i) models may solve the dataset, but not the task directly; (ii) supervised methods are limited by the curated datasets. Further, we demonstrate novel arithmetic capabilities to reason over visual data, as well as state-of-the-art performance on various tasks, such as visual dialog.