Pylyshyn’s Things and Places

Things and Places by Zenon Pylyshyn

In keeping with my current goal of pushing the requirements for “conscious thought” down to an absolute minimum, I picked up (downloaded) a copy of Pylyshyn’s Things and Places. Pylyshyn is a long time vision researcher and this book serves as summary of his work over the last 40 years. A core idea, if not the core idea is of a FINST, which is a visual index, maintained by the early visual system, that “points to” items in the world in a non-conscious manner. FINSTs (the term is a historical one, an abbreviation for `FINgers of INSTantiation’ ) , combine a deictic pointing reference and a flavor of a Kripke rigid designator — a rigid designator that is not formed by an explicit naming, but rather by an unconscious process of the early visual system).

I think that most would be willing to grant the early visual system some capacity to track objects, given our experiences with sports, driving, walking, etc.. I also think that almost everyone would be surprised at how robust these trackers are. The experiments delineated in the book show them to be reliable, even when the target objects are undergoing extreme variations

“a small number of target objects (usually around four) are briefly distinguished from a number of visually identical nontarget objects, typically by blinking the targets on and off a few times. ” …

“these include conditions in which objects change color and shape, disappear briefly behind an occluding surface, or simply disappear from view entirely as though the observer had blinked”

The experiment has been repeated a number of times, and Pylyshyn’s group has a demo here.

Modifying the experimental task by requiring the subject to keep track of, or describe properties, requires more time and increases the error rate highlighting how non-conscious, and not consciously available, they are.

Track-and-point, track-and-grab all without conscious intervention for optimal performance. The initial designation of the tracked object may or may not be conscious, in colloquial terms: something may or may not grab your attention.

FINSTs provide the capability for operationally competent (visual) deictic reference. If, by grunting and pointing, you can induce someone else to track the same target you’re tracking, you’ve arguably just conveyed a strong mutual agreed upon identification of it. This mutually-agreed-upon designation is achieved without any explicit description, and with an accuracy that likely exceeds your capacity to explicitly describe the target object in robust detail. It has no translation requirements, and the lack of explicit properties boosts the precision of the identification, again Pylyshyn:

 “The FINST does not distinguish between selecting a rabbit, a rabbit’s properties (such as its fur, shape, movement, color), a set of undetached rabbit parts, or any other coextensive category, because the selection is not based on a category at all”

Remember, this is the human visual system we are talking about. Over the years experiments have shown that the relative constant scenes which form our conscious awareness is a construction built by a very large portion of our brains, a construction built on the substrate of extremely varied sensory inputs. To quote Pylyshyn:

 “This problem of keeping track of individual token things by using a record of their properties is in general intractable when the things can move and change properties. But the problem exists even for a static scene since our eyes are continuously moving, the lighting changes with different points of view, and so on—which means that the problem of unique descriptors applies to every thing in a perceived scene. In fact it remains even if the scene and the point of view are fixed (as when a static scene is viewed through a peephole) since the representation itself is changing over time as the scene is explored with moving focal attention. There is ample evidence that percepts are built up over time.”

The follow-on idea, which he terms the index projection hypothesis, is that we can load things onto the FINSTs (identify them with objects we are trying to imagine in space), and then use the FINST mechanism to support our spatial reasoning.

 “The view is just this: In imagining a spatial layout, we use visual indexes (FINSTs) to pick out concurrently perceived objects that are roughly in the same relative locations as objects in the scene we are imagining. Each indexed object is associated with a unique label of a recalled or imagined object. These labels allow the system to keep the individual indexed objects distinctive, and also allow the visual system to treat the indexed objects as though they were marked—the visual system can thus detect patterns among indexed objects. The spatial properties that concern the mental objects (i.e. the conditions listed earlier) result from actual perception of the spatial relations among these indexed objects. This simple idea, called the index projection hypothesis, is developed in the next section.” (emphasis mine)

“Real movement interacts with imagined objects in a very natural way and such movements treat the imagined objects as fixed in space (so that, for example, your relation to your mental image undergoes the automatic updating of your frame of reference with your movement that is characteristic of moving in a real environment”

Our FINST enabled imaginations are then able to reason accurately about the imagined environment. Pylyshyn, being rigorous, indicates that the data supports this interpretation but that it has not been validated as thoroughly as the original Multi Object Tracking (MOT) experiments that underlie the identification of FINSTs in the early vision system.

I vacillate between considering this completely understandable (biology tends to build upon what’s ready to hand, rather than invent complete, large scale, complex mechanisms de novo) and totally shocked that there is that kind of bidirectionally available. This isn’t just priming the FINSTs to search for something, but rather using them as an auxiliary processor to our “conscious imagining.”

All in all, more solid evidence that our “conscious” processes are less critical than we might think and our “unconscious” processes are more subtle and robust than we might imagine.