Scholarly article on topic 'Corresponding ECoG and fMRI category-selective signals in Human ventral temporal cortex.'

Corresponding ECoG and fMRI category-selective signals in Human ventral temporal cortex. Academic research paper on "Clinical medicine"

CC BY
0
0
Share paper
Academic journal
Neuropsychologia
OECD Field of science
Keywords
{Electrocorticography / "Functional neuroimaging" / "Object recognition" / "High-frequency broadband gamma" / "Ventral stream"}

Abstract of research paper on Clinical medicine, author of scientific article — Corentin Jacques, Nathan Witthoft, Kevin S. Weiner, Brett L. Foster, Vinitha Rangarajan, et al.

Abstract Functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG) research have been influential in revealing the functional characteristics of category-selective responses in human ventral temporal cortex (VTC). One important, but unanswered, question is how these two types of measurements might be related with respect to the VTC. Here we examined which components of the ECoG signal correspond to the fMRI response by using a rare opportunity to measure both fMRI and ECoG responses from the same individuals to images of exemplars of various categories including faces, limbs, cars and houses. Our data reveal three key findings. First, we discovered that the coupling between fMRI and ECoG responses is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and high frequency broadband (HFB) ECoG responses (30–160hz). In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient, where the correlation is initially positive, but then tapers off or becomes negative. Second, we find that the strong and positive correlation between fMRI and ECoG signals in all frequency bands emerges rapidly around 100ms after stimulus onset, together with the onset of the first stimulus-driven neural signals in VTC. Third, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in ECoG HFB responses mirrors the topology and structure observed with fMRI. These findings of a strong and rapid coupling between fMRI and HFB responses validate fMRI measurements of functional selectivity with recordings of direct neural activity and suggest that fMRI category-selective signals in VTC are associated with feed-forward neural processing.

Academic research paper on topic "Corresponding ECoG and fMRI category-selective signals in Human ventral temporal cortex."

Author's Accepted Manuscript

Corresponding ECoG and fMRI category-selective signals in Human ventral temporal cortex.

Corentin Jacques, Nathan Witthoft, Kevin S. Weiner, Brett L. Foster, Vinitha Rangarajan, Dora Hermes, Kai J. Miller, Josef Parvizi, Kalanit GrillSpector

www. eLsevier.com^ocate/neuropsychologia

PII: S0028-3932(15)30109-3

DOI: http ://dx. doi. org/ 10.1016/j .neuropsychologia.2015.07.024

Reference: NSY5677

To appear in: Neuropsychologia

Received date: 3 March 2015 Revised date: 3 0 June 2015 Accepted date: 22 July 2015

Cite this article as: Corentin Jacques, Nathan Witthoft, Kevin S. Weiner, Brett L Foster, Vinitha Rangarajan, Dora Hermes, Kai J. Miller, Josef Parvizi anc Kalanit Grill-Spector, Corresponding ECoG and fMRI category-selective signal in Human ventral temporal cortex., Neuropsychologic http://dx.doi.org/10.1016/j .neuropsychologia.2015.07.024

This is a PDF file of an unedited manuscript that has been accepted fo publication. As a service to our customers we are providing this early version o the manuscript. The manuscript will undergo copyediting, typesetting, an< review of the resulting galley proof before it is published in its final citable form Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Corresponding ECoG and fMRI category-selective signals in human ventral temporal cortex.

Corentin Jacquesa,b,d, Nathan Witthofta, Kevin S. Weinera,d, Brett L. Fosterc,d, Vinitha Rangarajanc,d, Dora Hermesa,d, Kai J. Millerd,e, Josef Parvizic,d,f, Kalanit Grill-Spectora,d,f

a) Department of Psychology, Stanford University, Stanford, CA 94305, USA

b) Psychological Sciences Research Institute (IPSY), Université Catholique de Louvain, 1348 Louvain-la-Neuve, Belgium

c) Department of Neurology & Neurological Sciences, Stanford University, Stanford, CA 94305, USA

d) Stanford Human Intracranial Cognitive Electrophysiology Program (SHICEP)

e) Department of Neurosurgery, Stanford University, Stanford, CA 94305, USA

f) Stanford Neuroscience Institute, SNI, Stanford University, Stanford, CA 94305, USA

Corresponding author: Dr. Corentin Jacques, Psychological Sciences Research Institute (IPSY), Université Catholique de Louvain, 10 Place du Cardinal Mercier, 1348 Louvain-la-Neuve, Belgium. @mail: corentin.g.jacques@uclouvain.be; Tel: +32-10-479042.

Running title: Corresponding ECoG and fMRI signals in human VTC

Abstract

Functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG) research have been influential in revealing the functional characteristics of category-selective responses in human ventral temporal cortex (VTC). One important, but unanswered, question is how these two types of measurements might be related with respect to the VTC. Here we examined which components of the ECoG signal correspond to the fMRI response, by using a rare opportunity to measure both fMRI and ECoG responses from the same individuals to images of exemplars of various categories including faces, limbs, cars and houses. Our data reveal three key findings. First, we discovered that the coupling between fMRI and ECoG responses is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and high frequency broadband (HFB) ECoG responses (30-160hz). In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient, where the correlation is initially positive, but then tapers off or becomes negative. Second, we find that the strong and positive correlation between fMRI and ECoG signals in all frequency bands emerges rapidly around 100ms after stimulus onset, together with the onset of the first stimulus-driven neural signals in VTC. Third, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in ECoG HFB responses mirrors the topology and structure observed with fMRI. These findings of a strong and rapid coupling between fMRI and HFB responses validate fMRI measurements of functional selectivity with recordings of direct neural activity and suggest that fMRI category-selective signals in VTC are associated with feed-forward neural processing.

Keywords: Electrocorticography; functional neuroimaging; object recognition; high-frequency broadband gamma; ventral stream

INTRODUCTION

Humans rapidly and accurately categorize visual objects and scenes from very brief presentations (Grill-Spector & Kanwisher, 2005; Thorpe, Fize, & Marlot, 1996). This ability is thought to depend on neural computations performed along a hierarchy of cortical areas in the ventral visual stream extending from primary visual cortex to high-level visual regions in ventral temporal cortex (VTC) (Ungerleider & Mishkin, 1982). Lesions to the VTC can cause various forms of visual agnosia depending on the location and extent of the lesion (Farah, 1990; Konen, Behrmann, Nishimura, & Kastner, 2011; Rossion et al., 2003; Schiltz et al., 2006), suggesting a causal role of VTC in visual recognition. The functional properties of high-level visual regions in human VTC and their role in the recognition of various categories of stimuli has been examined by a large body of functional magnetic resonance imaging (fMRI) research, as well as intracranial electrophysiology (electrocorticography -ECoG and stereotaxic electroencephalography - SEEG) research.

fMRI has been instrumental in revealing the functional organization of category-selective responses in VTC within individual subjects (Grill-Spector & Weiner, 2014; Kanwisher, 2010; Levy, Hasson, Avidan, Hendler, & Malach, 2001; Malach, Levy, & Hasson, 2002; Weiner & Grill-Spector, 2010), because it is an non-invasive method that allows imaging the entire brain in millimeter resolution while the subject is performing a task. FMRI research discovered that focal regions in human VTC show higher responses to ecologically-relevant categories such as faces, bodyparts, and places compared to other stimuli (Epstein & Kanwisher, 1998; Kanwisher, McDermott, & Chun, 1997; McCarthy, Puce, Gore, & Allison, 1997; Peelen & Downing, 2005; Schwarzlose, Baker, & Kanwisher, 2005) and that these regions have a consistent spatial organization (topology) relative to the cortical folding and relative to each other (Nasr et al., 2011; Weiner et al., 2014; Weiner & Grill-Spector, 2010, 2013; Witthoft et al., 2014). Furthermore, responses in these category-selective regions are correlated with perception as fMRI responses are higher when stimuli of the preferred category are perceived than when they are not perceived (Andrews, Schluppeck, Homfray, Matthews, & Blakemore, 2002; Grill-Spector, Knouf, & Kanwisher, 2004; Hasson, Hendler, Ben Bashat, & Malach, 2001; Moutoussis & Zeki, 2002; Tong, Nakayama, Vaughan, & Kanwisher, 1998). Interestingly, fMRI experiments also showed that the VTC contains consistent distributed spatial patterns of response to a large array of visual categories (Connolly et al., 2012; Cox & Savoy, 2003; Haxby et al., 2001; Huth, Nishimoto, Vu, & Gallant, 2012; Kriegeskorte et al., 2008; Martin, Wiggs, Ungerleider, & Haxby, 1996; Spiridon & Kanwisher, 2002; Weiner & Grill-Spector, 2010; Weiner, Sayres, Vinberg, & Grill-Spector, 2010), even to those that are not associated with a 'category-selective' region, such as cars. These

consistent spatial topologies in VTC can be captured in representational similarity analyses (Kriegeskorte, et al., 2008), manifesting as higher correlations among VTC distributed responses to images of the same category compared to images of different categories.

Likewise, ECoG and SEEG (referred to as 'ECoG' henceforth) research, which measure local electrophysiological neural activity from electrodes on or within the cortex, also uncovered category-selective responses in VTC (Allison, McCarthy, Nobre, Puce, & Belger, 1994; Allison, Puce, Spencer, & McCarthy, 1999; Bastin et al., 2013; Davidesco et al., 2014; Engell & McCarthy, 2011; Fisch et al., 2009; Liu, Agam, Madsen, & Kreiman, 2009; Murphey, Maunsell, Beauchamp, & Yoshor, 2009; Nobre, Allison, & McCarthy, 1994; Privman et al., 2007; Vidal et al., 2010). A set of influential studies was the first to report larger event-related potentials (ERPs) in electrodes placed in the VTC to specific categories such as faces compared to other categories (Allison, et al., 1994; Allison, et al., 1999; Halgren et al., 1994; Nobre, et al., 1994; Puce, Allison, Spencer, Spencer, & McCarthy, 1997). These ERPs peak around 170-220ms after stimulus onset and can also be recorded on the scalp (Bentin, Allison, Puce, Perez, & McCarthy, 1996; Jacques & Rossion, 2011), although the relationship between scalp and intracranially recorded face-selective ERPs is still debated (Rosburg et al., 2010). Later ECoG studies of VTC also reported increases in high-frequency broadband (HFB, >30Hz) power peaking around 150-400ms after stimulus onset (Bastin, et al., 2013; Davidesco et al., 2014;; Engell & McCarthy, 2011; Parvizi et al., 2012; Privman, et al., 2007; Vidal, et al., 2010). For example, several studies reported increases in HFB to images of faces compared to other stimuli (Engell & McCarthy, 2011; Parvizi, et al., 2012; Privman, et al., 2007), as well as to images of houses compared to other stimuli (Bastin, et al., 2013; Davidesco et al., 2014; Vidal, et al., 2010). These category-selective responses measured with ECoG are also correlated with perception, as they display larger amplitudes when subjects perceive the stimulus, compared to when they do not (Fisch, et al., 2009). Moreover, electrical stimulation of category-selective VTC electrodes produces category-specific deficits (Allison, et al., 1994; Allison, et al., 1999; Chong et al., 2013; Jonas et al., 2012; Jonas et al., 2014; Megevand et al., 2014; Parvizi, et al., 2012; Rangarajan et al., 2014), providing strong evidence for their causal role in perception.

These findings from fMRI and ECoG studies provide strong evidence for category-selective responses in human VTC and their role in perception. However, fMRI only provides an indirect measure of neural activity in VTC with a temporal resolution of seconds because it is based on a blood oxygen level dependent (BOLD) signal. While ECoG does provide a direct measure of local electrophysiological neural activity in millisecond resolution, prior ECoG studies did not report the topology of category-selectivity in VTC due to sparse electrode sampling of the cortex in a given

brain. As a result, two key gaps in knowledge remain: (1) What is the anatomical origin and the spatial organization of ECoG category-selective responses across the cortical sheet of the VTC? (2) What is the neural origin of the BOLD category-selective signals measured in fMRI over the VTC?

The nature of the relationship between BOLD and electrophysiological signals in human VTC is poorly understood, even as studies in other parts of the brain suggest that the BOLD signal is correlated with the local field potential (Goense & Logothetis, 2008; Logothetis, Pauls, Augath, Trinath, & Oeltermann, 2001; Shmuel, Augath, Oeltermann, & Logothetis, 2006), especially in the HFB range (Conner, Ellmore, Pieters, DiSano, & Tandon, 2011; Hermes et al., 2012; Mukamel et al., 2005; Nir, Dinstein, Malach, & Heeger, 2008; Ojemann, Ojemann, & Ramsey, 2013; Winawer et al., 2013). First, the relationship between BOLD and neural responses signals varies across the brain (Conner, et al., 2011; Ojemann, et al., 2013), thus the nature of the relationship across measurements found in one part of the brain may not generalize to VTC. Second, the relationship between BOLD and electrophysiology has been studied most thoroughly in non-human primates (Goense & Logothetis, 2008; Logothetis, et al., 2001; Shmuel, et al., 2006), but it is premature to extrapolate across species given that homologies in high-level visual cortex are yet to be determined. Third, there has been no systematic investigation of the relationship between BOLD signals and direct neural responses in human VTC within the same individuals, as prior studies either compared signals across a handful of electrodes (Jonas, et al., 2014; Mundel et al., 2003; Murphey, et al., 2009; Parvizi, et al., 2012; Puce, et al., 1997) or across different groups of subjects (Bastin, et al., 2013; Privman, et al., 2007).

In investigating the neural origin of BOLD signals in VTC, comparing ECoG and fMRI signals within individuals provides a particularly appealing approach as both measurements have access to similarly sized population-level neural responses. ECoG electrodes are about 2mm in diameter, which is commensurate with the size of fMRI voxels (1-3mm). This similarity in the size of the probed neural population is advantageous over studies where BOLD signal is compared to single neuron or multi-unit recordings with microelectrodes (Boynton, 2011; Issa, Papanastassiou, & DiCarlo, 2013; Tsao, Freiwald, Tootell, & Livingstone, 2006). Comparing BOLD signals to the firing of single neurons is informative only in scenarios where the single neuron's response is correlated with larger-scale population response (Logothetis, et al., 2001; Nir, et al., 2008). Furthermore, linking fMRI and ECoG signals has the potential to provide important insights regarding the evolution of category-selective responses across VTC in both time and space. This will enable linking the knowledge about the spatial organization of category-selective responses across the VTC gained from fMRI studies with the temporal signature of VTC responses revealed by ECoG studies (Fisch, et al., 2009; Liu, et al., 2009).

To directly examine the relationship between fMRI and ECoG signals in the human VTC we used a rare opportunity to measure both ECoG and fMRI signals in the same six individuals while they viewed images of various categories. We first examined the topology of ECoG category-selective responses in each individual brain and then related these responses to fMRI responses in the same subjects. We asked: (1) What is the spatial organization of category-selective ECoG responses in VTC relative to the cortical folding? (2) What is the representational structure of distributed ECoG responses to object categories across the VTC? (3) How do the spatial organization of category-selectivity and the representational structure of distributed responses in VTC compare across ECoG and fMRI measurements? (4) Does the relationship between ECoG and fMRI signals vary across time and frequency bands of the ECoG signal?

2. Materials and Methods

2.1. Participants

Six participating patients (3 females) ages 22-57 took part in ECoG and fMRI experiments. Prior to ECoG recordings, participants were implanted with intracranial electrodes for clinical purposes as part of pre-surgical evaluation of refractory epilepsy (Supplementary Table 1). All participants provided written informed consent to participate in the study. The procedure was approved by the Stanford Institutional Review Board. Patient 2 did not undergo any cortical tissue resection, and the cortex resected in the rest of the patients was outside the anatomically defined VTC (Supplementary Table 1).

2.2. Anatomical localization of electrodes

Participants were implanted with flexible strips or grids of subdural platinum electrodes (AdTech Medical Instrument). Electrode placement was guided by the clinical evaluation of each patient. Electrodes were 2.3mm in diameter of exposed surface and 5-10mm center-to-center inter-electrode spacing. The anatomical location of each electrode was determined by co-registering CT images collected post electrode implantation with a high-resolution anatomical MRI of each participant's brain and further corrected to account for minor surgical brain shift (Foster & Parvizi, 2012; Hermes, Miller, Noordmans, Vansteensel, & Ramsey, 2010). We collected ECoG data from 7 hemispheres (4 right and 3 left hemispheres) in six patients, as one had bilateral electrode implantation.

We focus on ECoG measurements in electrodes located over VTC including the posterior portion of the inferior temporal sulcus (ITS), inferior temporal gyrus (ITG), lateral (LFG) and medial (MFG) fusiform gyrus, collateral sulcus (CoS), parahippocampal gyrus (PHG) and the anterior part of the lingual gyrus (LG). In all patients, the epileptic focus was determined to be outside our VTC regions of interest.

One hundred VTC electrodes were classified into four anatomical regions of interest (ROIs) in each participant's native anatomy (Fig. 1): (1) ITG (6 participants, right hemisphere: 15 electrodes, left hemisphere: 16 electrodes); (2) LFG: FG lateral to the mid-fusiform sulcus (MFS) (6 participants, right hemisphere: 22 electrodes, left hemisphere: 16 electrodes). (3) MFG: FG medial to the MFS (4 participants, right hemisphere: 6 electrodes, left hemisphere: 3 electrodes); (4) CoS, PHG, and LG (4 participants, right hemisphere: 14 electrodes, left hemisphere: 8 electrodes). The posterior end of VTC was defined by the posterior transverse collateral sulcus (ptCoS) and the anterior extent was halfway into the temporal lobe.

2.3. Stimuli and procedure

ECoG recordings and experiments were performed in the hospital rooms with the patients sitting in their beds. A Macbook pro laptop with a refresh rate of 60Hz placed on a tray table over the bed was used for stimulus presentation. During ECoG recordings, participants viewed grayscale photographs of faces, limbs, cars, and houses, presented in runs of 96 trials. We selected these stimuli as prior fMRI studies revealed both clustered and distributed VTC responses to these categories (Grill-Spector & Weiner, 2014; Weiner & Grill-Spector, 2010). The object depicted in each image appeared in variable position, size, lighting and viewpoint. Limb stimuli included both upper and lower limbs, always included the digits, and sometimes included the arms and the legs. Half of the images used were from a database as used in our previous studies (Grill-Spector, et al., 2004; Parvizi, et al., 2012; Weiner & Grill-Spector, 2010) and half were from another database collected by CJ (personal shots and images taken from the internet). Stimuli subtended approximately 10° x 10° of visual angle. Each stimulus was presented for 1 second, followed by a blank inter-trial interval of variable duration (0.6 - 1.4s). In each run, 48 images (12 from each category) were shown once. In addition, eight separate images, 2 from each category, were presented 6 times each. In each run, trial order was counterbalanced for category and repetition. Images were never repeated across runs and were randomly selected from our database such that each participant saw a unique sequence of images. Each run lasted for 192 seconds and participants were allowed as much time as needed between runs. Each participant completed 5-12 experimental runs. For all except one participant

(P1), the whole experiment was performed in a single day (spread over three days for P1). Task: participants fixated a central black cross and pressed a key on an external USB numeric pad when its color changed to red (15 times per run). Due to equipment malfunction we obtained behavioral data in four out of six participants. Performance was near ceiling with 93 - 98% target detection. Only one participant (participant 4) produced four false alarms. Response times ranged from 520 to 580ms.

2.4. ECoG recording and analyses

Continuous ECoG signal was recorded at a 3052 Hz sampling rate using a 128-channel recording system (Tucker Davis Technologies). All recordings were made relative to the most electrocorticographically 'silent' ECoG electrode, defined as a noise- and artifact-free (e.g. movement artifact) electrode with minimal to no variance in voltage amplitude for multiple bipolar recordings, and distal from the identified seizure focus. Data processing was performed using custom functions in MATLAB (MathWorks, Natick, MA). Continuous ECoG data were notched filtered to remove line noise, resampled to 1017.3Hz and segmented in -1.8 to 1.8s epochs centered on the onset of each stimulus. Noisy epochs in which signal amplitude at any time-point in a -0.7 to 1s time-window was above or below 4.5 times the across-trial standard deviation were discarded. This resulted in rejecting about 8% of the trials on average (range: 4 - 13%). The signal at each electrode was then re-referenced using the following procedure. We first calculated the average raw signal across all electrodes in a given subject and subtracted it from the signal measured at each electrode (i.e. a common average reference). Because a large number of electrodes were visually responsive in our experiment, this average signal always contained visually-driven responses. Therefore, we next identified the 20% of the electrodes (minimum of 16 non-pathological electrodes) which average-subtracted signal was the most negatively correlated with the signal averaged across all electrodes, indicating that the signal at these electrode is driven by the subtraction process and that they are largely nonresponsive in our experiment. These electrodes were designated as the reference electrodes. Last, we averaged the original signal across these reference electrodes and subtracted it from the original signal of each of the subjects' electrodes.

2.4.1. ECoG Time-frequency analyses

We used a Morlet wavelet approach (Tallon-Baudry, Bertrand, Delpuech, & Pernier, 1996) to estimate the amplitude of responses as a function of time and frequency. The wavelet transform was applied to each trial over a range of 2 - 160hz (1Hz steps) and the central frequency of the wavelet was adapted as a function of the frequency analyzed (from 2Hz at the lowest frequency to 9Hz at

highest frequency). The resulting time-frequency envelope was decimated to 101.7Hz to save space and computation time in further analyses. Response amplitudes were transformed to percentage signal change relative to the mean amplitude in a pre-stimulus time-window (-600ms to -300ms). This was performed independently for each trial and frequency. Responses were then averaged over several frequency bands typically reported in ECoG studies (0[theta]: 4-8Hz, a[alpha]: 8-12Hz, P[beta]: 12-30Hz) and high frequency broadband (HFB) range (Fig. 1, Supplementary Figure S1), which encompasses the low (30-80Hz) and high (>80Hz) gamma-frequency ranges. Since the ECoG signals in the HBF range are thought to be correlated with the local neuronal population spiking activity (Manning, Jacobs, Fried, & Kahana, 2009; Miller, Sorensen, Ojemann, & den Nijs, 2009; Ray & Maunsell, 2011) and in our data the signals in the 30-80Hz and >80Hz range showed similar characteristics (Fig. 1A), we report the broadest range for the HFB signals (30-160Hz).

We also estimated the temporal smoothing resulting from the time-frequency wavelet analysis by computing the full width at half maximum (FWHM) of the wavelet transform for each frequency band. This revealed that the timing information of ECoG signals is accurate up to 14ms for the HFB frequency range, 38ms for the beta range, 59ms for the alpha range and 81ms for the theta range, which correspond to half of the median FWHM of the wavelet transform across frequencies of a given frequency band.

ITG / ITS

Visual/Total Electrodes= 17/31 N subjects=6

LFG / OTS

N subjects=6

N subjects=4

COS / PHG / LG

N subjects=4

125 85 45 5

® 85 45 5

125 85 45 5

[-80 80] HFB _ .a.................. [-150 150] w

a............ ;

A........... . 4

[-75 75]

[min:-60 max:60]

ro ■

Houses

0 0.5 1 1.5 0 0.5 1 1.5 0 0.5 1 1.5 Time from stimulus onset (s)

— Faces Limbs

— Houses

— Cars

ITG / ITS

Lateral -

0 0.5 1 1.5 0 0.5 1 1.5 Time from stimulus onset (s)

LFG / OTS MFG

0 0.5 1 1.5

COS / PHG / LG

-► Medial

Figure 1. ECoG responses averaged across anatomical partitions of VTC. The figure depicts the average response across all visually responsive electrodes (the number of electrodes included is displayed at the top) in each of 4 anatomical partitions (bottom right inset): inferior temporal sulcus (ITS)/ inferior temporal gyrus (ITG), lateral fusiform gyrus (LFG)/occipitotemporal sulcus (OTS), medial fusiform gyrus (MFG), collateral sulcus(Cos)/parahippocampal gyrus (PHG)/lingual gyrus (LG). (A) The average ECoG time-frequency response to images of dataset 1 for each category (rows) and anatomical partition (columns). Solid horizontal black lines indicate stimulus duration. The color-scale was adjusted for each anatomical region (see minimum and maximum values in the top-right corner of the top row plots). Data illustrate that HFB-responses (30-160Hz) encompass most of the power increase in the ECoG response. Lower frequencies (<12Hz) exhibit an initial power increase followed by a reduction of power after 0.2-0.3s relative to stimulus onset (Supplementary Figure S1). (B) ECoG HFB signal averaged over the same electrodes and anatomical partitions. Shaded regions around waveforms represent ± 1 SEM across electrodes, computed on amplitude normalized waveforms in each electrode. Solid horizontal black lines indicate stimulus duration. Insets show the first 0.3s after stimulus onset.

2.4.2. Data sets

Responses were analyzed separately for two independent datasets. Data set 1 consisted of

responses to images that were each shown once, with ECoG responses averaged across all images of

a category (70 - 130 trials per participant after trial rejection). Data set 1 was used in analyses

associated with Figs. 1, 2, 5, 6, 8, supplementary Figs S1, S3, and S4). Data set 2 consisted of an independent set of 40-80 images per participant containing 10-20 images per category, each shown 6 times. We analyzed the mean response to each image across its second to sixth presentation, therefore excluding the first presentation of repeated images to reduce potential adaptation effects (Engell & McCarthy, 2014b; Grill-Spector, Henson, & Martin, 2006). Reponses to individual images were averaged (average of 2 to 5 trials per image depending on the number of rejected noisy trials) to achieve higher signal-to-noise ratio per image. Data set 2 was used in analyses associated with Figs. 3, 4, 7, and supplementary Fig. S2. Analyses in Figs. 2, 3, 4, 5, and supplementary Fig. S2 were performed using the HFB averaged over a time-window of 100-350ms after stimulus onset.

2.4.3. Defining visually responsive electrodes

Visually responsive electrodes were defined as electrodes with ECoG signal in HFB range (30160 Hz) that was 4 times larger than its across-trial standard error. This was determined using images of dataset 1, averaging HFB amplitude over a time window of 100-350ms after stimulus onset.

2.4.4. Visualizing single electrode category profile over each participant's VTC

Fig. 2 shows for each electrode, the relative amplitude elicited by the average HFB responses across images of a category (dataset 1) during a time window of 100-350ms after stimulus onset in the form of pie charts. The diameter of each pie chart was scaled according to the normalized response amplitude in the 100-350ms time-window. The amplitude was normalized using a metric similar to an effect size metric, which is defined as the ratio between the mean response amplitude and the across-trial standard deviation. Normalized amplitude was computed separately for each category and the diameter of each pie chart was scaled according to the highest normalized amplitude across categories.

2.4.5. Quantifying ECoG category-selectivity in individual electrodes

Category selectivity (d1) for dataset 1 was estimated for each electrode and category using ECoG HFB responses averaged over a 100-350ms time window; d (j), selectivity for category j, was defined as:

1 N u - N

d '(j ] = Uf i' n n ; ' * j.

+ N ? *)

Where u is the mean response to category i; ai is the across-trial standard deviation of responses to category i; N=3. The statistical significance of d'(j) was assessed using a permutation test, where we randomly shuffled the category label of all trials, calculating d'(j) for the shuffled data and repeating this procedure 10000 times. The p-value was computed as the fraction of the permuted d' distribution that was greater than the observed d'. We determined the mean selectivity across electrodes in each anatomical ROI and tested if it was significantly above zero using a one-tailed t-test.

2.4.6. Analyses of representational similarity from distributed ECoG data

We used representational similarity analyses (Edelman, 1998; Kriegeskorte, et al., 2008) to explore the representational structure of category information in distributed ECoG HFB responses over VTC. We computed representational similarity matrices (RSM) by first extracting for each participant the distributed pattern of responses across their VTC electrodes (on average 12.7+6.7 electrodes per participant) for each image from data set 2 viewed by the participant. We used ECoG HFB responses averaged over a 100-350ms time-window. These RSMs indicate the mean similarity (indexed by Pearson's r correlation coefficient) between distributed patterns of VTC responses to each pair of images a participant viewed. The distributed pattern of a response to an image corresponds to the vector containing the amplitude of response to that image across VTC electrodes. To remove between-electrode differences in response magnitudes, each electrode's response amplitudes across all presented images were normalized by transforming them into z-scores. Then, in a given participant, we compute an RSM by correlating the z-scored amplitude vectors across all images. We then obtain an RSM at the category level by calculating in each participant the average similarity among distributed responses to images of the same or different categories. For within-category comparisons we excluded the similarity among distributed responses to the same image (which by definition are equal to 1).

2.4.7 Estimation of HFB response onset latency

Response latency of ECoG HFB responses in VTC electrodes was defined as the latency at which HFB responses first rose above the baseline level. This was estimated by first fitting a

cumulative Gaussian to the initial portion (-100 to 300 ms relative to stimulus onset) of the ECoG response, then calculating the intersection of a line fitted to the linear portion of this Gaussian with the baseline level response. The intersection indicates the earliest time point at which signal rises above baseline. Response latency was estimated for each VTC electrode and each category using the average response across images of a category from dataset 1.

2.5. fMRI measurements and analyses

MRI data were obtained using GE 3-Tesla Signa scanners at Stanford University. Anatomical MRIs were collected before the ECoG grid implantations for all subjects and were used for registering the ECoG electrodes, registering fMRI activations, and for cortical surface reconstruction. Due to logistical issues we could not collect fMRI on all subjects prior to the implantation. fMRI data were collected for two subjects before implantation (P2: 2 month before, P3: 4 days before) and for four subjects after the ECoG recordings (P1: 20 month after; P4: 11 month after; P5: 17 days after; P6: 10 month after).

2.5.1. Anatomy

A high-resolution anatomical volume of the whole brain was acquired with a head coil using a T1-weighted SPGR pulse sequence. Data were aligned to the AC-PC plane and resampled to 1mm isotropic voxels. Both fMRI data and ECoG electrode locations were aligned to this brain volume. This volume was segmented to gray and white matter, and the resulting segmentation was used to reconstruct the cortical surface of each participant.

2.5.2. fMRI acquisition

Participants were scanned using a T2* sequence (TE=30 ms, TR=2000 ms, flip angle=77° and bandwidth=128 kHz). Participants were scanned with voxels ranging from 1.5mm to 3mm as there is a tradeoff between coverage and resolution and some participants were scanned on additional fMRI studies in the same session. When possible, we used smaller voxels as they improve localization of functional activations and reduce the effects of susceptibility artifacts (Weiner & Grill-Spector, 2013). Scanning parameters were kept constant within a participant. We used the following scanning resolutions and parameters: P1, P6: 2.8x2.8x2.5 mm, 30 slices, FOV=180 mm; P2: 1.8 mm isotropic, 28 slices, FOV=192 mm; P3: 3 mm isotropic, 32 slices, FOV=192 mm; P4: 2.4 mm isotropic, 34 slices, FOV=192 mm; and P5: 1.5x1.5x3 mm 12 slices, FOV=192 mm,

2.5.3. fMRI Experiment

Participants performed 1-3 runs of a block design experiment during which grayscale photographs of faces, limbs, flowers, houses, cars, guitars, and scrambled objects were shown in 12s blocks (same as (Parvizi, et al., 2012; Weiner & Grill-Spector, 2010)). Each block contained images of a single category. Each run consisted of 4 blocks of each condition and 6 blank blocks and lasted for 408 seconds. Categories were counterbalanced within each run and images were not repeated across runs. During a block, each image appeared for 0.75 s with an interstimulus interval of 0.25 s. Images subtended a visual angle of 7.1°. The images used in fMRI were from the same database as half of the images used during ECoG and the same as in our prior fMRI experiments (Grill-Spector, et al., 2004; Weiner & Grill-Spector, 2010). In the present study we only considered BOLD responses to the same four categories as in the ECoG experiment (i.e. faces, limbs, cars, houses). Task: Participants were instructed to fixate on a centrally presented dot and respond by button press when two consecutive images were identical. Repetitions occurred randomly and at a low frequency (~6% of the images). This task requires subjects to attend to the individual images independently of their category, and we used a low repetition frequency to avoid adaptation effects. Due to technical problems we were able to collect behavioral data during fMRI in only 4 out of the 6 participants. Three of the participants had performance (accuracy and RT) within the same range as typical adults participants measured in prior identical experiments (accuracy range: 40 - 94%, response time range: 540 - 933ms (Weiner & Grill-Spector, 2010)). One participant (P5) had typical RT but more misses than controls, however her fMRI activations are typical (see Fig. 5).

2.5.4. fMRI data Analysis

Data were analyzed with MATLAB using the mrVista toolbox (http://white.stanford.edu/software) as in our prior publications(Parvizi, et al., 2012; Weiner, et al., 2014; Weiner & Grill-Spector, 2010). Data were motion corrected, detrended, and aligned to the participant's whole brain anatomy. There was no spatial smoothing of fMRI responses. We fit every voxel's time course using standard GLM analyses.

2.6. Measuring the correspondence between ECoG and fMRI

2.6.1. Quantifying the correspondence of the spatial pattern of face- and house-selectivity between ECoG and fMRI measurements

Since the most consistent selectivity measurable both in ECoG and fMRI was found for faces (present in all 6 participants) and houses (present in 4/6 participants), we quantified the correspondence of category-selective signals across methodologies using these two categories. Specifically, we tested whether the spatial patterns of face-selectivity and house-selectivity measured with ECoG and fMRI across VTC electrodes were correlated without thresholding either measurement.

We first compared fMRI selectivity to ECoG selectivity in the HFB range averaged over a 100350 ms time-window as signals in this frequency range correlate with the local population neural firing, and this time-window corresponds to the peak of the initial visual response in VTC. Next, we compared fMRI selectivity to ECoG selectivity across frequency-bands and time to identify which component of the ECoG signal best matches fMRI, addressing the ongoing debate on the nature of the relationship between these signals (Hermes, et al., 2012; Ojemann, et al., 2013; Scheeringa et al., 2011). The temporal and frequency band variability in the correlation between ECoG and fMRI arose from the time-frequency varying nature of the ECoG selectivity, as fMRI selectivity was constant for each electrode.

Face- and house-selectivity in both methods were computed using the d' metric (face vs. limbs, cars, houses or houses vs. limbs, cars, faces; see equation in 'Estimating category selectivity'). We measured the correlation (using Pearson's coefficient) between ECoG and fMRI selectivity across all visually responsive VTC electrode locations in each participant excluding electrodes overlapping fMRI signal dropout regions that have low SNR. An average of 7.7 ± 2 electrodes per participant were used. ECoG selectivity was defined in the following way: ECoG responses from dataset 1 in each frequency band (0 : 4-8Hz, a:8-12Hz, P: 12-30Hz, HFB: 30-160Hz) and each electrode were temporally smoothed using a Gaussian kernel (STD: 20ms) to accommodate for differences in response latencies across electrodes. These responses were used to compute the d' for faces (vs. limbs, cars, houses) or houses (vs. faces, limbs, cars) in each electrode, frequency band and 10ms time-bins over an interval of -100ms to 1500ms relative to stimulus onset. fMRI selectivity in each voxel was defined in the following way: We first estimated d' for faces (or houses) vs. other stimuli, only including the same four categories as for the ECoG data (faces, limbs, cars, houses). Then we calculated the weighted average of fMRI d' across gray matter voxels in the vicinity of each electrode for which the GLM explained more than 5% of their variance. Each voxel's d' was weighted by its distance from the electrode in the brain volume using a 3D Gaussian kernel. Since the spread of ECoG signal in the brain volume might depend on the frequency band and the orientation of the electrode

relative to the cortical surface, we parametrically varied the spatial extent of the Gaussian kernel from 1-20mm around each electrodes and computed the corresponding ECoG-fMRI d' correlations.

To estimate the significance of the correlations between ECoG and fMRI responses across participants, we tested if correlations significantly deviated from zero across participants using a 2-tailed percentile bootstrap test (Efron & Tibshirani, 1993). Specifically, we selected six participants with replacement from the pool of participants and the average correlation across this sample was computed and stored. This was done for all possible sampling combinations of participants with replacement (462 possibilities for a sample size of 6) to obtain a distribution of bootstrapped estimates of the correlation. A two-tailed p-value was computed as the fraction of the bootstrapped estimates that were either smaller or larger than zero depending on the sign of the correlation. The minimal attainable p-value was constrained by the number of resamples performed (here the minimal p-value is 2/462 = 0.0043).

2.6.2. Comparison of representational similarity across fMRI and ECoG data

We used representational similarity analyses to compare the representational structure of category information in distributed VTC responses across ECoG and fMRI measurements. For ECoG measurements, we computed RSMs from ECoG responses in 10ms time bins from 100ms before stimulus onset to 1500ms after stimulus onset and for four frequency bands (0 : 4-8Hz, a:8-12Hz, P: 12-30Hz, HFB: 30-160Hz). For fMRI measurements, RSMs were computed as follows. In each participant we extracted the average fMRI signal from a 5mm radius gray matter ROIs beneath each of their VTC electrodes. Based on these fMRI signals we generated a vector of distributed fMRI responses (z-scored amplitudes (Weiner & Grill-Spector, 2010)) under the electrodes for each of the four categories that were used in the ECoG experiment (faces, limbs, houses, and cars). This distributed response was measured separately for the first and second runs in which participants saw different images from these categories. We measured the correlation among distributed z-scored responses to images of the same and different categories, generating an RSM matrix of VTC responses for each participant (participant 6 was excluded as he had only one run of data) and averaged the corresponding upper and lower off-diagonal correlation values (as this is a non-symmetric cross-correlation matrix).

We quantified the correspondence between the category representational structure of distributed VTC responses in each participant by correlating RSMs measured in ECoG (across time and frequency bands) and fMRI. We used a 2-tailed percentile bootstrap test (sampling participants

with replacement) to estimate whether correlations significantly deviated from zero across participants.

3. RESULTS

3.1. ECoG responses in the HFB range reveal a robust and reproducible spatial topology of category information in VTC

3.1.1. Consistent spatial organization of HFB category-selective responses relative to anatomical landmarks

To examine the spatial topology of category-selective responses in VTC, we first examined ECoG responses in the HFB range in each individual. Analyses of HFB responses from ECoG electrodes showed that the majority of VTC electrodes (76/100) were visually responsive, with HFB signals emerging 80-100ms after stimulus onset (Fig. 1, Fig. 2a). Seventy percent of visually responsive VTC electrodes (53/76 electrodes) showed a significant preference for one category, and 10% (8/76 electrodes) showed a preference for two categories (p < 0.005; 2-tailed permutation test, Fig. 2c). For example, three electrodes spaced 1 cm apart, arranged laterally to medially over the OTS, FG and CoS, showed a differential preference to limbs (1), faces (2), and houses (3, Fig. 2a). The largest responses typically occurred 100-350ms after stimulus onset (Fig. 1, Fig. 2a), and the rank ordering among responses to different stimuli in a given electrode remained mostly stable across the duration of stimulus presentation (Fig. 1, Fig. 2a). In addition, while most electrodes showed a significant preference for a particular category, they generally also exhibited significant above-baseline responses for other non-preferred categories. Of the 76 visually responsive electrodes, 71 (93%) electrodes showed a significant response above baseline to faces, 66 (87%) to limbs, 54 (71%) to houses and 66 (87%) to cars. Thus, the proportion of the visually responsive electrodes that showed a significant response for 4, 3, 2 and 1 category was 62%, 21%, 11% and 7%, respectively.

Category-selective responses over VTC measured in the HFB range show a consistent topology relative to the cortical folding in individual brains. Face- and limb-selective responses were detected in the lateral VTC, in the lateral fusiform gyrus (LFG) and inferior temporal gyrus (ITG). All participants had face-selective electrodes located on the LFG. In fact 33/38 electrodes in the LFG showed significant preference for faces (average d' for faces over 38 electrodes = 1.54+1.13, Fig. 2c). These electrodes were near or lateral to the mid fusiform sulcus (MFS) occasionally extending to the occipitotemporal sulcus (OTS) and ITG (8/17 electrodes), which had overall lower face-selectivity compared to the LFG (d' for faces over 17 ITG electrodes: 0.59+1).

Figure 2. Spatial distribution of ECoG category selectivity in human ventral temporal cortex is consistent across participants. (A-B) Localization of electrodes, visual responsivity and category preference in each participant. (A) Spatial distribution of electrode selectivity in an example right hemisphere (ventral view of the temporal lobe), and HFB-responses as a function of time in three example electrodes spaced one centimeter apart. On the cortex, each electrode is represented by a pie chart plotting the relative amplitude elicited by each category (averaged over responses to images of dataset 1) in the HFB range (30-160 Hz) during a time window of 100-350ms after stimulus onset. The diameter of each pie chart is scaled according to its normalized amplitude of response. (B) Spatial organization of ECoG category-selectivity in all other hemispheres. Note that electrodes both within and outside VTC are shown, highlighting the particular sensitivity of VTC to visual object categories. (C) Selectivity in each electrode (d') in four anatomical regions arranged from lateral to medial across the VTC. Each circle represents an electrode, where filled circles show significant positive category-selectivity (p < 0.005). For each electrode we show its selectivity to each category as well as to the animate vs. inanimate stimuli distinction. Horizontal lines represent the average d' across electrodes for each anatomical region. Electrodes overlapping the mid-fusiform sulcus (MFS), lateral fusiform gyrus (LFG) and occipitotemporal sulcus (OTS) show face-selectivity. Preferential responses to inanimate stimuli occurred in the medial fusiform gyrus (MFG), collateral sulcus (CoS) and parahippocampal gyrus (PHG).

Fewer electrodes displayed preference for limbs over other stimuli. Limb-selective electrodes were located on the ITG (4/17 electrodes) and the LFG/OTS (4/38 electrodes, Fig 2. S1, S4) and were lateral to electrodes preferring faces in the same participants. Medial to the MFS (medial FG (MFG),

in the collateral sulcus (CoS) parahippocampal gyrus (PHG) and lingual gyrus (LG)) we generally found stronger responses to inanimate than animate stimuli (11/21 electrodes preferring inanimate stimuli, 3/21 electrodes preferring animate), and particularly to houses (9/21 electrodes; Fig. 2: S1, S2, S5, S6). Notably, there were no electrodes with preference to faces in the CoS, PHG, or LG. Few electrodes showed preferential responses to cars (5/76), and they tended to also exhibit preference for limbs or houses (Figs. 2 and 3) with no clear anatomical distribution. These data also reveal a large-scale lateral-medial VTC gradient of animacy, where electrodes preferring animate stimuli (faces and limbs) are located in lateral VTC, and electrodes preferring inanimate stimuli (houses and cars) are located in medial VTC (Figs. 2 and 3).

ITG / ITS

\ 2.3 F 0,7 F 0,5

Lateral FG / OTS

Medial FG

COS/PHG/LG

H 0.7 H 0.6

H 0.3 I 0.3

Faces Limbs Cars Houses Images (sorted by category and appearance order}

Figure 3. Single image category-selectivity in VTC. Each cell shows the normalized HFB response amplitude to a single image averaged across 2-5 presentations over a 100-350ms time window in one electrode taken from data set 2. Electrodes are represented in rows, grouped by anatomical region and category-preference and sorted by descending d' from data set 1 (Fig. 2C). Responses to images are shown in columns, sorted by appearance order for each category. We measured responses to 40-80 images per electrode depending on the number of blocks of data collected. Gray cells represent missing data for a particular subject/electrode. d' that are significantly higher than zero (p<.005, 2-tailed permutation test) for data set 1, are indicated on the right and color-coded by category preference. Electrodes that were not significantly selective to any of the four categories are on the bottom of each anatomical region and no d' is reported.

This pattern of selectivity was validated in an independent set of stimuli in which we measured responses to individual images of faces, limbs, houses and cars in each visually responsive VTC electrode (Fig. 3). Particularly striking was the selectivity to faces in face-selective electrodes in the LFG, where responses to each of the face images were consistently strong and higher than to nonfaces. In fact for 15/38 electrodes in the LFG, responses to any of the face images were larger than responses to any of the non-face images (Fig. 3 - Lateral FG). While this analysis with individual images also replicated the preference for houses and inanimate stimuli in the MFG and CoS, the observed selectivity for these stimuli was overall lower. For example, electrodes that showed strong HFB responses to houses in the MFG and CoS/PHG/LG did not respond robustly to all house images and in fact they responded robustly to many car images and even to face and limb images (Fig. 3 -bottom 2 rows).

3.1.2. Distributed ECoG VTC responses to single images reveal a categorical structure

Additional insight about the information structure of VTC responses can be gleaned from the analyses of distributed HFB responses across all VTC electrodes. Examination of the representational similarity structure of distributed HFB responses in each participant reveals that distributed VTC response patterns to images of the same category are more similar than distributed responses to images of different categories (Fig. 4). The average correlation among spatial patterns generated by images of the same category was positive (r=0.29±0.09; average between-category similarity) and significantly higher (t(5)=8.4, p<0.0005) than the average correlation among patterns of response to images of different categories (r=-0.11±0.03). This categorical nature of responses was observed even for categories that did not display strong category preference at the level of individual electrodes or for which we did not find selectivity in each participant (e.g. cars). Nevertheless, consistent with the selectivity of individual electrodes (Fig. 3), the highest within-category similarity occurred among distributed responses to face images (r=0.47±0.12), followed by house (0.3±0.09), limb (0.23±0.12), and least among car images (0.15±0.09). Additionally, there was a distinction between distributed responses to animate vs. inanimate stimuli, as they generated anti-correlated distributed response patterns across the VTC (correlation between animate and inanimate images: r=-0.17±0.08). The distributed response patterns differentiating animate and inanimate stimuli are a consequence of the lateral-medial arrangement of animate representations in the lateral VTC and inanimate representations on the medial VTC (Figs. 2, 3). Critically, distributed HFB responses in low-level visual regions, do not display such a categorical structure (Supplementary Fig. S2), indicating that the categorical structure of distributed HFB responses in VTC is not driven by low-level

differences among these images. These results reveal that the consistent and reproducible spatial topologies of HFB responses to different categories identified in the previous section generate a consistent representational structure of category information in VTC.

Individual subjects' representational similarity matrices

P1 {12 electrodes) P2 (13 electrodes) P3 {24 electrodes) P4 (15 electrodes) P5 {6 electrodes) P6 (6 electrodes)

-1 0 1

Pattern similarity (Pearson's correlation - r)

Figure 4: Representational similarity matrices (RSM) of distributed HFB responses across VTC. RSM of

distributed VTC responses between each pairs of images from data set 2 are shown for each participant. Distributed responses were calculated for each participant across all of their VTC electrodes (electrode count indicated at the top of each panel) for HFB-responses averaged over a time window of 100-350ms after stimulus onset. Responses were averaged across 2-5 presentations of an image (excluding the first presentation). The top triangle shows the correlation of the distributed responses to each pair of images (40-80 images per participant), the bottom triangle shows the average correlation across all pairs of images of a category, excluding the diagonal.

3.2. Examination of the correspondence between fMRI signals and ECoG responses across time and frequency bands

The results described in the prior section illustrate that direct measurements of neural responses in the HFB range largely replicate findings from fMRI research. In the second part of our study we directly compared fMRI and ECoG responses in each of our participants addressing three main questions: (1) What is the spatial correspondence between face- and house-selectivity measured with ECoG HFB to that measured with fMRI? We focused on these categories as they generated selective responses in both ECoG and fMRI in most of our participants. (2) Is the same correspondence found for other ECoG frequency bands and does it vary across time? (3) What is the relationship between the distributed representational similarity structure of category information across the VTC measured with ECoG and fMRI, and does it vary across time and frequency band?

3.2.1. HFB and fMRI category-selectivity measured in the same participant spatially overlap

Fig. 5 illustrates the degree of selectivity in each VTC voxel to faces (top) or houses (bottom)

measured with fMRI in each of our participants. Superimposed on this unthresholded fMRI selectivity

map are the ECoG electrodes, which are colored by their level of selectivity to faces (top) or houses

(bottom) in the HFB range (d' from Fig. 2). As expected, fMRI responses showed higher responses to faces vs. other stimuli in the FG (Fig. 5A). Importantly, in all participants we found a clear spatial overlap between regions showing preferential responses to faces measured with fMRI and those measured with ECoG in the HFB range (Fig. 5A). Specifically, when electrodes were located over or directly abutting a region showing positive face-selectivity measured with fMRI they always showed positive face-selectivity in the HFB signal (Fig. 5A, compare yellow electrodes with yellow fMRI activations). Mismatches between measurements were restricted to regions where we could obtain ECoG signals but not fMRI signals due to MR signal dropout in the vicinity of the ear canals, generally in the ITG and anterior lateral FG, and to cortex under a single electrode, which showed face-selectivity in HFB but not fMRI (Fig. 5A, S5, right-most electrode). Furthermore, electrodes that displayed zero or negative selectivity for faces overlapped regions with low or negative selectivity to faces in fMRI. Likewise, we found a spatial correspondence between regions that displayed preferential responses to houses across fMRI and HFB (Fig. 5B). Higher fMRI responses for houses than the other stimuli were found in the CoS, PHG and LG. Overlapping these fMRI activations, we found electrodes with higher HFB responses for houses compared to other categories (Fig. 5B, S1, S2, S5, S6). Conversely, electrodes that displayed lower or negative selectivity for houses than other stimuli overlapped with fMRI regions that showed lower or negative responses as well.

A Faces vs. others

Ecog (d') -0.5_0_0.5

-3 0 3

fMRI (t)

Figure 5: Overlapping face- and house-selectivity across ECoG HFB and fMRI signals. A. Unthresholded maps of selectivity for faces (faces > limbs, cars, and houses; t value, voxel level) measured with fMRI are displayed for each participant. Inflated cortical surfaces show ventral visual cortex, including the VTC, with the occipital pole at bottom. Overlaid on these maps are circles indicating the magnitude of selectivity (contrast of faces > limbs, cars and houses, d') of ECoG electrodes in the HFB range for a response averaged over a 100-350ms time window relative to stimulus onset. Note that the electrode layouts do not appear as perfect strips or grids on the cortical surface due to the cortical surface inflation process used for better visualization of activations inside sulci. The electrodes displayed are only those in regions where we could measure reliable fMRI responses. The diameter of each electrode has been increased for better visibility. Acronyms: OTS: occipitotemporal sulcus, MFS: mid-fusiform sulcus, CoS: Collateral sulcus. B. Same conventions as for A but showing selectivity for houses (house > faces, limbs, and cars).

3.2.2. The coupling between ECoG and fMRI responses varies across time and frequency bands

We next quantified the spatial correspondence between fMRI and ECoG responses across time and frequency bands. In each participant, we compared the degree of category selectivity to faces or houses across electrodes to the degree of category selectivity measured with fMRI in the cortex under the same electrodes (See Materials and Methods). We found that the highest and most significant correlation was between ECoG selectivity measured in the HFB range and fMRI selectivity. This significant correlation emerged rapidly, about 80-100ms after stimulus onset and persisted for the remainder of the stimulus duration (average Pearson correlation+STD across participants: 0.68+0.05, significantly above zero, p<0.005, 2-tailed bootstrap test, Fig. 6A).

ra 0.6

O -0.2

O LU -0.4

c o 0.6

O) 0.4

o o 0.2

o -0.2

O UJ -0.4

6: 4-8 Hz — P: 12-30 Hz Q: 8-12 Hz —HFB: 30-160 Hz

p<005 p<01

— n.s.

0 0.2 0.4 0.6 0.8 1 1.2 1.4 Time relative to stimulus onset (s)

e a (3 ' j HFB

' i f^i i ^ j ~ A k | | | /

\1 1 1 1 1 1 1 / ' 1 1 V 1 '/ 1 ^^lyi i ^ i 1 1 1 1 ... 1 .. 1 . . Time-window: — 100- 200ms — 400 - 800ms

10 20 30 40 Frequency (Hz)

Figure 6. The coupling between ECoG and fMRI face- and house-selectivity varies across time and frequency bands. (A) Correlation between category-selectivity measured with fMRI and ECoG as a function of frequency band and time. Each line reflects the spatial correlation between fMRI and ECoG selectivity across time for a specific ECoG frequency band averaged across 6 subjects. Correlations between ECoG and fMRI were computed separately for faces-selectivity and house-selectivity, then averaged in each subject, and last averaged across subjects. Shaded regions: across-subject SEM. Line thickness indicates whether the across-subject correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). (B) Correlation between fMRI and ECoG selectivity across the frequency spectrum (3-160Hz) for two time-windows. For both (A) and (B) the contribution of fMRI signal from each voxel was weighted using a Gaussian kernel of 3.5mm STD around each electrode (Supplementary Figs. S3 and S4).

In contrast, for lower frequency bands of

the ECoG signal the correlation with fMRI responses varied with time (Fig. 6). While it was significantly positive 100-200ms after stimulus onset (Pearson r±STD 0: 0.61+0.07, a: 0.58+0.06, P: 0.65+0.05, ps<0.005 2-tailed bootstrap test), at later times the correlations decreased, becoming negative 400-800ms after stimulus onset in the 0 and a bands. This negative correlation at low frequencies was due to lower than baseline ECoG power in these frequencies at these later times (Fig. 1, Supplementary Fig. S1) that generated a weak negative selectivity (r+STD in 400-800ms window for 0: -0.17+0.13, a: -0.25+0.08, ps<0.005 for a). In this later time-window, correlations in the P range exhibited a transitory profile between low and high-frequencies (r+STD: 0.34+0.09, Fig. 6B). Importantly, across frequency bands, the correlations between ECoG and fMRI signals were highest when considering the local fMRI signal within a radius of 2-5mm around the center coordinates of ECoG electrodes (Supplementary Figs. S3, S4). Interestingly, the spatial coupling between ECoG and fMRI responses tended to be tighter for face-selectivity than for house-selectivity. When comparing face-selectivity across measurements the spatial correlation dropped sharply when fMRI signals were pooled more than 6 mm away from the electrodes (Supplementary Fig. S3). However when comparing house-selectivity across methodologies, the correspondence persisted across a larger spatial range (Supplementary Fig. S4).

A ECoG RSMs B fMRI RSM

-0.3 0 0.3

Pattern similarity (r)

Figure 7: Category representation structure from VTC patterns correlate across fMRI and ECoG HFB from 100ms after stimulus onset. (A) ECoG RSMs: representational similarity matrices (RSM) of distributed VTC responses for different frequency bands at four time intervals. RSMs are computed from dataset 2, averaged across images in each participant, and then across participants S1 - S5, who had at least two fMRI scans. Acronyms: F: faces; L: limbs; C: cars; H: houses. (b) fMRI RSM: Across participants' average (S1-S5) RSMs of distributed fMRI signals from voxels under ECoG VTC electrodes. (C) Average correlation between fMRI RSMs and ECoG RSMs across time and frequency band. Correlations were computed in each participant and then averaged across participants. Line thickness indicates if the across-participant correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). Shaded regions indicate across-participant SEM.

Comparing the representational similarity of distributed responses to object categories across the VTC measured with ECoG (Fig. 7A) to fMRI (Fig. 7B) also showed that the coupling between measurements varies across time and frequency bands (Fig. 7C). Distributed fMRI responses across the VTC show a clear categorical structure (Fig. 7B). Distributed responses to images of the same category are positively correlated, and these correlations are higher than the correlation between distributed responses to images of different categories. However, for ECoG responses the representation structure varies across frequency band and time. While the RSM of distributed HFB signals shows a clear categorical structure from approximately 100ms after stimulus onset until stimulus offset (Fig. 7A-HFB), RSMs of distributed signals in lower frequency bands (a, P, 0) display a more transient structure, exhibiting a category structure from about 100ms to 300ms after stimulus onset but little to no category structure in later times (Fig. 7A).

Consequently, HFB and fMRI RSMs are significantly and positively correlated in each participant from 100ms till stimulus offset (p<0.05, two-tailed bootstrap test against zero). This correlation between measurements is maximal from 100 to 300ms (mean correlation between fMRI and ECoG HFB RSM in 100-300ms time-window: Pearson's r=0.7+0.13; p<0.005; Fig. 7C). In contrast, ECoG RSMs in lower frequencies are significantly and positively correlated to fMRI RSMs only from about 100 to 300ms after stimulus onset (mean Pearson correlation in 100-300ms window: 0[4-8Hz]: r=0.60+0.11, a[8-12Hz]: r=0.64+0.27, p[12-30Hz]: r=0.57+0.12, ps<0.005). At later time points, the correlation decreases and becomes non-significant (Fig. 7C) because responses in the lower frequency bands lose their coherent category structure (Fig. 7A). These data suggest that spatial topologies across VTC which convey categorical information are most similar across fMRI and ECoG HFB measurements, starting within 100ms after stimulus onset and persisting until stimulus offset.

3.2.3. The significant correlation between HFB and fMRI responses emerges together with the initial onset of signals in VTC

Our findings reveal that the initial coupling between ECoG HFB and fMRI responses starts at about 80-100ms after stimulus onset. It is interesting to note that this timing matches the time-point when stimulus-driven HFB signals in VTC electrodes rise from the baseline level, which is at

-120 o

3 100 ra

o a. to a) DC

• Response latency for ¡single electrodes S6*in subjects 1- 6

W.Mdiirfd.l^

1 Mean latency j by subjects

1 Mean latency T across all electrodes

NUSCRIPT

91ms±18ms after stimulus onset (Fig 8). Since the latency at which the spatial patterns of category-selectivity and representational structure start to correlate between fMRI and ECoG measurements matches the latency of the initial response onset in VTC, our data suggests that these fMRI responses are associated with the initial feed forward responses in VTC.

Faces Limbs Cars

Stimulus category

Houses

Figure 8. ECoG HFB response latency across visually responsive VTC electrodes. Response latencies at each visually responsive VTC electrode are shown for each category and participant (filled circles), along with the average (± STD) across electrodes in each participant (open circles) and the average across all electrodes and participants by category (black diamonds). The number of electrodes included for each category is shown below each group of plots. Some electrodes respond significantly to more than one category. Thus, the same electrode may appear in multiple category bins.

4. Discussion

Our data reveal three key findings. First, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in HFB ECoG responses mirrors the structure observed with fMRI. Second, we discovered that the coupling between fMRI signals and ECoG in VTC is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and HFB ECoG responses. In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient. Third, we find that the strong and

positive correlation between fMRI and ECoG responses emerges rapidly around 100ms after stimulus onset, together with the onset of the first stimulus driven neural signals in VTC. These findings are important because they validate fMRI maps of functional selectivity in individual participants' brains with a direct measure of neural activity and provide a millisecond timestamp for fMRI activations in human VTC.

4.1 The correlation between fMRI and ECoG responses varies across time and frequency bands.

Our data reveal a strong and positive correlation between fMRI responses and ECoG signals in the HFB range from about 100ms after stimulus onset throughout the entire stimulus duration (Figs 6, 7). These results are in line with a growing body of evidence showing a coupling between electrophysiological signals in the HFB range and BOLD responses in human motor cortex (Hermes, et al., 2012; Siero et al., 2014), in early visual cortex (Winawer, et al., 2013), in language-related cortex (Conner, et al., 2011; Hermes et al., 2014; Lachaux et al., 2007), as well as in nonhuman primate visual cortex (Goense & Logothetis, 2008). Here we show direct evidence that this relationship also occurs in the human VTC when measuring ECoG and fMRI in the same participants. These results also extend previous reports that showed correlations between fMRI and ECoG HFB signals in VTC across different groups of participants (Bastin, et al., 2013; Privman, et al., 2007). Since the HFB signal correlates with the aggregate spiking activity across a neuronal population (Manning, et al., 2009; Miller, et al., 2009; Nir et al., 2007; Ray & Maunsell, 2011), our data suggest that the spatial topologies of category information measured via fMRI in VTC reflect, at least partly, the local population neural firing.

In contrast to the sustained and strong positive correlation between fMRI and ECoG signals in

the HFB range, our data revealed a time-varying coupling between fMRI and ECoG signals in lower

frequencies. Early (100-300ms) ECoG responses across all frequency bands were significantly and

positively correlated with fMRI signals, while later (400-800ms) ECoG responses at lower frequencies

(<30Hz) showed negative (when correlating face and house selectivity) or lower (when correlating

RSMs) correlation with fMRI signals (Figs. 6, 7 and Supplementary Figs. S3, S4). The former negative

correlations at later time windows are driven by decreases in power relative to pre-stimulus baseline,

which generate negative face- and house-selectivity in lower ECoG frequency bands (Fig. 1,

Supplementary Fig. 1). The lack of significant correlation between RSMs measured with fMRI and

ECoG at later time windows is a consequence of the loss of a coherent category representational

structure of ECoG responses at these lower frequencies. This dissociation of category information in

the HFB and lower frequency bands underscores the importance of examining multiple components of electrical signals from intracranial recordings.

Future research will determine if ECoG power modulations in low frequencies measured in early and late time-windows reflect distinct neural phenomena, such as an initial ERP (Engell & McCarthy, 2011; Fisch, et al., 2009; Winawer, et al., 2013) generated by the input to VTC, followed by later power reduction in low frequency bands reflecting a modulatory signal (Pfurtscheller & Lopes da Silva, 1999; Schroeder & Lakatos, 2009). Nevertheless, our data underscore the power of relating ECoG and fMRI measurements in individual brains and highlight the importance of examining both the temporal and spectral components of electrical neural signals to understand the relation between signals recorded in these different methodologies.

4.2 Linking between fMRI and HFB responses in VTC: Methodological considerations

While our data show that the spatial coupling across ECoG and fMRI is tight (2-5mm around the electrodes, Supplementary Figs. S3, S4) we also found that this spatial coupling was better when comparing face-selectivity than house-selectivity across methods. While we cannot determine with certainty why this is the case, several factors may contribute to this difference. First, face-selective responses in VTC are located largely on gyri (FG, ITG), while house-selective responses are found mostly in a sulcus (CoS). Because ECoG grids are in direct contact with cortex on gyri and away from cortex within sulci, face-selectivity provides a better opportunity to measure local coupling between ECoG and fMRI signals. Second, in our sample there were substantially more face- than house-selective electrodes, and face-selective electrodes also had a higher selectivity. This combination of factors produced an enhanced dynamic range of face-selectivity compared to house-selectivity in ECoG responses, perhaps increasing the correlation across methods.

This difference in the spatial coupling across stimuli underscores the importance of considering analysis and acquisition factors when comparing results obtained from different types of measurements. For instance, while the current spatial coupling estimate between ECoG and fMRI measurements provides an indication as to the volume over which ECoG electrodes are sensitive to neural activity in our particular setup, it is likely that the spatial specificity of the ECoG category-selective responses would be higher if we used denser grids of smaller electrodes (Freeman, Rogers, Holmes, & Silbergeld, 2000). Future comparisons of fMRI and ECoG data may examine to what extent these different methodological factors affect the spatial correspondence across measurements. These include: (1) acquisition parameters, such as voxel size, fMRI dropout zone, electrode size, electrode impedance, as well as inter-electrode distance. (2) The point spread function of fMRI and

ECoG responses, respectively. The former is estimated to be 2-3 mm (Issa, et al., 2013; Shmuel, Yacoub, Chaimow, Logothetis, & Ugurbil, 2007) and the latter 1.25mm (Freeman, et al., 2000). (3) Errors in the reconstruction of the electrode location relative to the cortex due to brain shifts caused by the electrode implantation procedure. Even though we used an accurate algorithm that corrects for this potential problem, its spatial precision is bounded in the order of 0.5-2mm (Hermes, et al., 2010).

4.3. The spatial organization of category selectivity VTC matches across HFB ECoG and fMRI.

By analyzing single participant ECoG data and visualizing responses relative to the cortical folding in individual brains, our data show that the spatial organization of category selectivity measured with ECoG HFB responses is similar to that measured with fMRI in the same participants. With both methods we find pronounced face-selectivity on the MFS, LFG and OTS, and selectivity for inanimate stimuli, and in particular houses, in the MFG, CoS and PHG. In two participants we also found HFB limb-selective responses around the OTS, which were near fMRI selective activations for limbs (fMRI data not shown). In addition, we find a large-scale animacy distinction along a lateralmedial VTC axis, where preferential responses to animate categories occur in lateral VTC and preferential responses to inanimate categories are observed in medial VTC. The location of category-selective ECoG responses reported here are consistent with prior findings from fMRI studies (Connolly, et al., 2012; Epstein & Kanwisher, 1998; Kanwisher, et al., 1997; Konkle & Caramazza, 2013; Martin, et al., 1996; Nasr, et al., 2011; Peelen & Downing, 2005; Schwarzlose, et al., 2005; Weiner, et al., 2014; Weiner & Grill-Spector, 2010, 2013) and with the reported location of electrodes responding preferentially to specific categories in previous intracranial EEG studies (Allison, et al., 1999);(McCarthy, Puce, Belger, & Allison, 1999; Miller, et al., 2009; Murphey, et al., 2009; Privman, et al., 2007; Puce, et al., 1997) (Liu, et al., 2009) (Bastin, et al., 2013; Davidesco et al., 2014; Engell & McCarthy, 2011, 2014a; Fisch, et al., 2009; Rosburg, Ugur, Haueisen, Kreitschmann-Andermahr, & Sauer, 1999; Vidal, et al., 2010). Our data extend these prior ECoG studies that examined localized responses over a few electrodes or pooled data across participants, by providing a direct and comprehensive mapping of the spatial organization of category representations across the VTC in individual participants. We would like to underscore that understanding the relationship between functional responses and the cortical folding in individual brains has important clinical implications because it may allow predicting function from anatomy alone (Weiner, et al., 2014; Witthoft, et al., 2014).

Our finding of a tight spatial relationship between patterns of face- and house-selective responses across ECoG and fMRI are also in line with prior human case studies reporting ECoG-fMRI correspondence in VTC at a local scale of a single to a few electrodes (Jonas, et al., 2014; Mundel, et al., 2003; Murphey, et al., 2009; Parvizi, et al., 2012; Puce, et al., 1997). However, while these prior studies provided evidence for a co-localization of selectivity for faces across ECoG and fMRI, they are limited in their ability to establish a direct spatial correspondence between BOLD and neurophysiological signals. Here, we reduced the spatial uncertainty by measuring ECoG and fMRI from multiple electrodes spread over a large surface of the cortex in multiple individuals' hemispheres. This allowed us to show that spatial patterns of graded selectivity measured with fMRI tightly correspond with the underlying patterns manifested in neurophysiology.

Finally, although we used different tasks in ECoG and fMRI (detecting a change of color at fixation or a 1-back task, respectively), it is interesting that we find clear correspondence in the topology of category-selectivity across the VTC with fMRI and ECoG measurements. This consistency is in line with fMRI results showing that the task does not change the overall pattern of distributed responses across VTC to object categories or the location of category-selective regions in VTC (Bugatus, Weiner, & Grill-Spector, 2015; Davidenko, Remus, & Grill-Spector, 2012; Weiner & GrillSpector, 2010), even though the task modulates the magnitude of responses (Harel, Kravitz, & Baker, 2014; Wojciulik, Kanwisher, & Driver, 1998).

4.4. Spatial topologies of fMRI responses across the VTC are associated with feed-forward processing

Likewise, we found that the representational structure of distributed HFB ECoG responses

matches the representational structure measured with fMRI in the same participants. By using a data

driven approach without pregrouping stimuli into categories, we find that distributed HFB responses

to images of a category are more similar to each other than to images of other categories. This

categorical structure is observed even for stimuli that do not generate strong selectivity in specific

electrodes. Further, our data show that distributed HFB responses across the VTC contain both a

distinction between face, house, car and limb categories, as well as a broader distinction between

animate and inanimate categories. By examining the representational structure in each participant

over time, we are able to show that the category information in VTC measured in the HFB range

arises from reproducible and distinctive spatial topologies across categories, and that these

topologies are stable over time from 100ms after stimulus onset till stimulus offset. This insight was

not possible to obtain from prior ECoG studies (Liu, et al., 2009) because they measured distributed

responses from electrodes that were pooled across participants without taking into account the

spatial location of specific electrodes. Nevertheless, the timing we report here is compatible with prior studies in humans (Liu, et al., 2009) and macaques (Bell et al., 2011; Hung, Kreiman, Poggio, & DiCarlo, 2005; Kiani, Esteky, & Tanaka, 2005; Tsao, et al., 2006). Moreover, our data show that the representational similarity measured with HFB starting at 100ms, not only matches fMRI representations within the same participants, but also matches the category structure reported in prior fMRI studies (Kriegeskorte, et al., 2008) (Weiner & Grill-Spector, 2011; Weiner, et al., 2010).

Notably, both spatial maps of face- and house-selectivity and RSMs measured with fMRI correlated with ECoG HFB measurements from 100ms after stimulus onset, and this correlation persists for the remainder of the stimulus duration. This provides a much needed time stamp for the many fMRI studies of category representations in VTC, suggesting that they emerge within 100ms. Importantly, we find that the onset latency of the significant correlation between ECoG HFB and fMRI responses matches the onset latency of HFB responses in VTC (Fig 8). Because ECoG measures local electrophysiological responses directly from the cortex, this indicates that the earliest stimulus-driven neural responses in VTC contribute to the measured fMRI signals and in turn, suggests that feed-forward neural processing in VTC contribute to fMRI responses.

Conclusions

In sum, this study presents two important advancements. First, we provide a direct mapping of electrophysiological neural response patterns in the HFB range in the VTC in individual participants. Second, by finding a tight correspondence between ECoG HFB responses and fMRI signals in the same participants, we provide a way to bridge findings about VTC function across the two methodologies with millimeter and millisecond resolution.

Acknowledgement^,

This research was funded by National Science Foundation BCS0920865, National Institutes of Health 1 RO1 EY 02231801A1 to KGS, Stanford NeuroVentures Program and National Institutes of Health Grant R01 NS078396-01 to JP. CJ is supported by the Belgian Fund for Scientific Research (FNRS) and Belgian Federal Science Policy Office (BELSPO, return grant 2012). DH was supported by a Stanford School of Medicine Dean's Postdoctoral Fellowship. BLF is supported by a National Institutes of Health Career Development Award K99 MH103479-01.

Author's Personal Note

This manuscript is a part of a collection of paper constituting a special issue of Neuropsychologia edited in memory of Dr. Shlomo Bentin who tragically died in a traffic accident on July 13th 2012. One of the authors added a personal note about how Dr. Bentin affected her research.

"When I was a graduate student at the Weizmann Institute of science, we had weekly neuroscience seminars in the basement of the Neurobiology building. Two of these seminars made a big impact on me. One was when my graduate mentor, Dr. Rafael Malach, returned from a sabbatical at MGH and announced that there is a revolutionary method to measure the human brain non invasively - fMRI. Another was when Dr. Shlomo Bentin, visiting from the Hebrew University in Jerusalem, gave a seminar about electrical recordings directly from intracranial electrodes from the living human brain and that these recordings revealed face-selective signals in the human VTC. I was blown out of my mind. It did not occur to me that it was possible to record signals directly from the human brain. I also remember that these two seminars were very intense. The reason is that the electrophysiologists felt threatened by these new methods and tried to discount these new methods on the basis of a lack of understanding of the source of the signals measured in fMRI and intracranial electrodes. Now almost 20 years later, the field has significantly advanced in understanding these signals. Further, it is clear that both fMRI and ECoG measurements have broken new ground in understanding sensory and cognitive processes in the brain. The results we present here, are not only inspired by Dr Shlomo Bentin's research, but further clarify the link between electrophysiological signals and fMRI in the human brain. "

Kalanit Grill-Spector, Stanford University, CA, USA. Author contributions

N.W., K.S.W, and K.G.S. designed research; C.J., N.W., K.S.W, B.L.F., V.R., J.P., and K.G.S. contributed to data collection; C.J., N.W., K.S.W, B.L.F., V.R., and K.G.S analyzed data; C.J., B.L.F., D.H., and K.J.M. contributed to analytic tools; C.J., N.W., K.S.W., K.G.S. wrote the manuscript.

REFERENCES

Allison, T., McCarthy, G., Nobre, A., Puce, A., & Belger, A. (1994). Human extrastriate visual cortex

and the perception of faces, words, numbers, and colors. Cerebral Cortex, 4(5), 544-554. Allison, T., Puce, A., Spencer, D. D., & McCarthy, G. (1999). Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli. Cereb Cortex, 9(5), 415-430.

Andrews, T. J., Schluppeck, D., Homfray, D., Matthews, P., & Blakemore, C. (2002). Activity in the fusiform gyrus predicts conscious perception of Rubin's vase-face illusion. Neuroimage, 17(2), 890-901.

Bastin, J., Vidal, J. R., Bouvier, S., Perrone-Bertolotti, M., Benis, D., Kahane, P., et al. (2013). Temporal Components in the Parahippocampal Place Area Revealed by Human Intracerebral Recordings., 33(24), 10123-10131.

Bell, A. H., Malecek, N. J., Morin, E. L., Hadj-Bouziane, F., Tootell, R. B., & Ungerleider, L. G. (2011). Relationship between Functional Magnetic Resonance Imaging-Identified Regions and Neuronal Category Selectivity. J Neurosci, 31(34), 12229-12240.

Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G. (1996). Electrophysiological Studies of Face Perception in Humans. Journal of cognitive neuroscience, 8(6), 551-565.

Boynton, G. M. (2011). Spikes, BOLD, attention, and awareness: a comparison of electrophysiological and fMRI signals in V1. Journal of Vision, 11(5), 12.

Bugatus, L., Weiner, K. S., & Grill-Spector, K. (2015). Task differentially modulates the spatial extent of category-selective regions across anatomical locations. Paper presented at the Vision Sciences Society, St. Petersburg, FL, USA.

Chong, S. C., Jo, S., Park, K. M., Joo, E. Y., Lee, M. J., Hong, S. C., et al. (2013). Interaction between the electrical stimulation of a face-selective area and the perception of face stimuli. Neuroimage, 77, 70-76.

Conner, C. R., Ellmore, T. M., Pieters, T. A., DiSano, M. A., & Tandon, N. (2011). Variability of the relationship between electrophysiology and BOLD-fMRI across cortical regions in humans. J Neurosci, 31(36), 12855-12865.

Connolly, A. C., Guntupalli, J. S., Gors, J., Hanke, M., Halchenko, Y. O., Wu, Y. C., et al. (2012). The representation of biological classes in the human brain. J Neurosci, 32(8), 2608-2618.

Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage, 19(2 Pt 1), 261-270.

Davidenko, N., Remus, D. A., & Grill-Spector, K. (2012). Face-likeness and image variability drive responses in human face-selective ventral regions. Human brain mapping, 33(10), 22342249.

Davidesco, I., Zion-Golumbic, E., Bickel, S., Harel, M., Groppe, D. M., Keller, C. J., et al. (2014). Exemplar Selectivity Reflects Perceptual Similarities in the Human Fusiform Cortex. Cerebral cortex. 24(7):1879-93

Edelman, S., Grill-Spector, K., Kusnir, T., Malach, R. (1998). Towards direct visualization of the internal shape space by fMRI. Psychobiology (special issue on Cognitive Neuroscience of Object Representation and Recognition), 26, 309-321.

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. New York: Chapman and Hall.

Engell, A. D., & McCarthy, G. (2011). The relationship of gamma oscillations and face-specific ERPs recorded subdurally from occipitotemporal cortex. Cerebral cortex, 21(5), 1213-1221.

Engell, A. D., & McCarthy, G. (2014a). Face, eye, and body selective responses in fusiform gyrus and adjacent cortex: an intracranial EEG study. Frontiers in human neuroscience, 8, 642.

Engell, A. D., & McCarthy, G. (2014b). Repetition suppression of face-selective evoked and induced EEG recorded from human cortex. Human brain mapping, 35(8), 4155-4162.

Epstein, R., & Kanwisher, N. (1998). A cortical representation of the local visual environment. Nature, 392(6676), 598-601.

Farah, M. J. (1990). Visual agnoisa: Disorders of Object Recognition and What They Tell Us About Normal Vision. Cambridge, Massachusetts: MIT Press.

Fisch, L., Privman, E., Ramot, M., Harel, M., Nir, Y., Kipervasser, S., et al. (2009). Neural "ignition": enhanced activation linked to perceptual awareness in human ventral stream visual cortex.. Neuron, 64(4), 562-574.

Foster, B. L., & Parvizi, J. (2012). Resting oscillations and cross-frequency coupling in the human posteromedial cortex. Neuroimage, 60(1), 384-391.

Freeman, W. J., Rogers, L. J., Holmes, M. D., & Silbergeld, D. L. (2000). Spatial spectral analysis of human electrocorticograms including the alpha and gamma bands. Journal of neuroscience methods, 95(2), 111-121.

Goense, J. B., & Logothetis, N. K. (2008). Neurophysiology of the BOLD fMRI signal in awake monkeys. Current biology: CB, 18(9), 631-640.

Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn Sci, 10(1), 14-23.

Grill-Spector, K., & Kanwisher, N. (2005). Visual recognition: as soon as you know it is there, you know what it is. Psychol Sci, 16(2), 152-160.

Grill-Spector, K., Knouf, N., & Kanwisher, N. (2004). The fusiform face area subserves face perception, not generic within-category identification. Nat Neurosci, 7(5), 555-562.

Grill-Spector, K., & Weiner, K. S. (2014). The functional architecture of the ventral temporal cortex and its role in categorization. Nature reviews. Neuroscience, 15(8), 536-548.

Halgren, E., Baudena, P., Heit, G., Clarke, J. M., Marinkovic, K., & Clarke, M. (1994). Spatio-temporal stages in face and word processing. I. Depth-recorded potentials in the human occipital, temporal and parietal lobes. Journal of physiology, Paris, 88(1), 1-50.

Harel, A., Kravitz, D. J., & Baker, C. I. (2014). Task context impacts visual object processing differentially across the cortex. Proc Natl Acad Sci U S A, 111(10), E962-971.

Hasson, U., Hendler, T., Ben Bashat, D., & Malach, R. (2001). Vase or face? A neural correlate of shape-selective grouping processes in the human brain. J Cogn Neurosci, 13(6), 744-753.

Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425-2430.

Hermes, D., Miller, K. J., Noordmans, H. J., Vansteensel, M. J., & Ramsey, N. F. (2010). Automated electrocorticographic electrode localization on individually rendered brain surfaces. Journal of neuroscience methods, 185(2), 293-298.

Hermes, D., Miller, K. J., Vansteensel, M. J., Aarnoutse, E. J., Leijten, F. S., & Ramsey, N. F. (2012). Neurophysiologic correlates of fMRI in human motor cortex. Human brain mapping, 33(7), 1689-1699.

Hermes, D., Miller, K. J., Vansteensel, M. J., Edwards, E., Ferrier, C. H., Bleichner, M. G., et al. (2014). Cortical theta wanes for language. Neuroimage, 85 Pt 2, 738-748.

Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005). Fast readout of object identity from macaque inferior temporal cortex. Science, 310(5749), 863-866.

Huth, A. G., Nishimoto, S., Vu, A. T., & Gallant, J. L. (2012). A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6), 1210-1224.

Issa, E. B., Papanastassiou, A. M., & DiCarlo, J. J. (2013). Large-scale, high-resolution neurophysiological maps underlying FMRI of macaque temporal lobe. J Neurosci, 33(38), 15207-15219.

Jacques, C., & Rossion, B. (2011). The N170 : understanding the time-course of face perception in the human brain. In S. Luck (Ed.), The Oxford Handbook of ERP Components. Oxford University Press.

Jonas, J., Descoins, M., Koessler, L., Colnat-Coulbois, S., Sauvee, M., Guye, M., et al. (2012). Focal electrical intracerebral stimulation of a face-sensitive area causes transient prosopagnosia. Neuroscience, 222, 281-288.

Jonas, J., Rossion, B., Krieg, J., Koessler, L., Colnat-Coulbois, S., Vespignani, H., et al. (2014). Intracerebral electrical stimulation of a face-selective area in the right inferior occipital cortex impairs individual face discrimination. Neuroimage, 99, 487-497.

Kanwisher, N. (2010). Functional specificity in the human brain: a window into the functional architecture of the mind. Proc Natl Acad Sci U S A, 107(25), 11163-11170.

Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci, 17(11), 4302-4311.

Kiani, R., Esteky, H., & Tanaka, K. (2005). Differences in onset latency of macaque inferotemporal neural responses to primate and non-primate faces. Journal of Neurophysiology, 94(2), 15871596.

Konen, C. S., Behrmann, M., Nishimura, M., & Kastner, S. (2011). The functional neuroanatomy of object agnosia: a case study. Neuron, 71(1), 49-60.

Konkle, T., & Caramazza, A. (2013). Tripartite organization of the ventral stream by animacy and object size. J Neurosci, 33(25), 10235-10242.

Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky, H., et al. (2008). Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6), 1126-1141.

Lachaux, J. P., Fonlupt, P., Kahane, P., Minotti, L., Hoffmann, D., Bertrand, O., et al. (2007). Relationship between task-related gamma oscillations and BOLD signal: new insights from combined fMRI and intracranial EEG. Human brain mapping, 28(12), 1368-1375.

Levy, I., Hasson, U., Avidan, G., Hendler, T., & Malach, R. (2001). Center-periphery organization of human object areas. Nat Neurosci, 4(5), 533-539.

Liu, H., Agam, Y., Madsen, J. R., & Kreiman, G. (2009). Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron, 62(2), 281-290.

Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature, 412(6843), 150-157.

Malach, R., Levy, I., & Hasson, U. (2002). The topography of high-order human object areas. Trends Cogn Sci, 6(4), 176-184.

Manning, J. R., Jacobs, J., Fried, I., & Kahana, M. J. (2009). Broadband shifts in local field potential power spectra are correlated with single-neuron spiking in humans. J Neurosci, 29(43), 13613-13620.

Martin, A., Wiggs, C. L., Ungerleider, L. G., & Haxby, J. V. (1996). Neural correlates of category-specific knowledge. Nature, 379(6566), 649-652.

McCarthy, G., Puce, A., Belger, A., & Allison, T. (1999). Electrophysiological studies of human face perception. II: Response properties of face-specific potentials generated in occipitotemporal cortex. Cerebral cortex, 9(5), 431-444.

McCarthy, G., Puce, A., Gore, J. C., & Allison, T. (1997). Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience, 9(5), 605-610.

Megevand, P., Groppe, D. M., Goldfinger, M. S., Hwang, S. T., Kingsley, P. B., Davidesco, I., et al. (2014). Seeing scenes: topographic visual hallucinations evoked by direct electrical stimulation of the parahippocampal place area. J Neurosci, 34(16), 5399-5405.

Miller, K. J., Sorensen, L. B., Ojemann, J. G., & den Nijs, M. (2009). Power-law scaling in the brain surface electric potential. PLoS computational biology, 5(12), e1000609.

Moutoussis, K., & Zeki, S. (2002). The relationship between cortical activation and perception investigated with invisible stimuli. Proc Natl Acad Sci U S A, 99(14), 9527-9532.

Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., & Malach, R. (2005). Coupling between neuronal firing, field potentials, and FMRI in human auditory cortex. Science, 309(5736), 951954.

Mundel, T., Milton, J. G., Dimitrov, A., Wilson, H. W., Pelizzari, C., Uftring, S., et al. (2003). Transient inability to distinguish between faces: electrophysiologic studies. Journal of clinical neurophysiology, 20(2), 102-110.

Murphey, D. K., Maunsell, J. H., Beauchamp, M. S., & Yoshor, D. (2009). Perceiving electrical stimulation of identified human visual areas. Proc Natl Acad Sci U S A, 106(13), 5389-5393.

Nasr, S., Liu, N., Devaney, K. J., Yue, X., Rajimehr, R., Ungerleider, L. G., et al. (2011). Scene-selective cortical regions in human and nonhuman primates. J Neurosci, 31(39), 13771-13785.

Nir, Y., Dinstein, I., Malach, R., & Heeger, D. J. (2008). BOLD and spiking activity. Nature neuroscience, 11(5), 523-524.

Nir, Y., Fisch, L., Mukamel, R., Gelbard-Sagiv, H., Arieli, A., Fried, I., et al. (2007). Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations. . Current biology, 17(15), 1275-1285.

Nobre, A. C., Allison, T., & McCarthy, G. (1994). Word recognition in the human inferior temporal lobe. Nature, 372(6503), 260-263.

Ojemann, G. A., Ojemann, J., & Ramsey, N. F. (2013). Relation between functional magnetic resonance imaging (fMRI) and single neuron, local field potential (LFP) and electrocorticography (ECoG) activity in human cortex. Frontiers in human neuroscience, 7, 34.

Parvizi, J., Jacques, C., Foster, B. L., Withoft, N., Rangarajan, V., Weiner, K. S., et al. (2012). Electrical stimulation of human fusiform face-selective regions distorts face perception. J Neurosci, 32(43), 14915-14920.

Peelen, M. V., & Downing, P. E. (2005). Selectivity for the human body in the fusiform gyrus. J Neurophysiol, 93(1), 603-608.

Pfurtscheller, G., & Lopes da Silva, F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical neurophysiology, 110(11), 1842-1857.

Privman, E., Nir, Y., Kramer, U., Kipervasser, S., Andelman, F., Neufeld, M. Y., et al. (2007). Enhanced category tuning revealed by intracranial electroencephalograms in high-order human visual areas. J Neurosci, 27(23), 6234-6242.

Puce, A., Allison, T., Spencer, S. S., Spencer, D. D., & McCarthy, G. (1997). Comparison of cortical activation evoked by faces measured by intracranial field potentials and functional MRI: two case studies. Human brain mapping, 5(4), 298-305.

Rangarajan, V., Hermes, D., Foster, B. L., Weiner, K. S., Jacques, C., Grill-Spector, K., et al. (2014). Electrical stimulation of the left and right human fusiform gyrus causes different effects in conscious face perception. J Neurosci, 34(38), 12828-12836.

Ray, S., & Maunsell, J. H. (2011). Different origins of gamma rhythm and high-gamma activity in macaque visual cortex. PLoS biology, 9(4), e1000610.

Rosburg, T., Ludowig, E., Dumpelmann, M., Alba-Ferrara, L., Urbach, H., & Elger, C. E. (2010). The effect of face inversion on intracranial and scalp recordings of event-related potentials. Psychophysiology, 47(1), 147-157.

Rosburg, T., Ugur, T., Haueisen, J., Kreitschmann-Andermahr, I., & Sauer, H. (1999). Enlarged gamma band response of neuromagnetic auditory evoked fields in a visually impaired subject. Neuroreport, 10(18), 3791-3795.

Rossion, B., Caldara, R., Seghier, M., Schuller, A. M., Lazeyras, F., & Mayer, E. (2003). A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain y, 126(Pt 11), 2381-2395.

Scheeringa, R., Fries, P., Petersson, K. M., Oostenveld, R., Grothe, I., Norris, D. G., et al. (2011). Neuronal dynamics underlying high- and low-frequency EEG oscillations contribute independently to the human BOLD signal. Neuron, 69(3), 572-583.

Schiltz, C., Sorger, B., Caldara, R., Ahmed, F., Mayer, E., Goebel, R., et al. (2006). Impaired face discrimination in acquired prosopagnosia is associated with abnormal response to individual faces in the right middle fusiform gyrus. Cereb Cortex, 16(4), 574-586.

Schroeder, C. E., & Lakatos, P. (2009). Low-frequency neuronal oscillations as instruments of sensory selection. Trends in neurosciences, 32(1), 9-18.

Schwarzlose, R. F., Baker, C. I., & Kanwisher, N. (2005). Separate face and body selectivity on the fusiform gyrus. J Neurosci, 25(47), 11055-11059.

Shmuel, A., Augath, M., Oeltermann, A., & Logothetis, N. K. (2006). Negative functional MRI response correlates with decreases in neuronal activity in monkey visual area V1. Nature neuroscience, 9(4), 569-577.

Shmuel, A., Yacoub, E., Chaimow, D., Logothetis, N. K., & Ugurbil, K. (2007). Spatio-temporal point-spread function of fMRI signal in human gray matter at 7 Tesla. Neuroimage, 35(2), 539-552.

Siero, J. C., Hermes, D., Hoogduin, H., Luijten, P. R., Ramsey, N. F., & Petridou, N. (2014). BOLD matches neuronal activity at the mm scale: a combined 7T fMRI and ECoG study in human sensorimotor cortex. Neuroimage, 101, 177-184.

Spiridon, M., & Kanwisher, N. (2002). How distributed is visual category information in human occipito-temporal cortex? An fMRI study. Neuron, 35(6), 1157-1165.

Tallon-Baudry, C., Bertrand, O., Delpuech, C., & Pernier, J. (1996). Stimulus specificity of phase-locked and non-phase-locked 40 Hz visual responses in human. J Neurosci, 16(13), 4240-4249.

Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381(6582), 520-522.

Tong, F., Nakayama, K., Vaughan, J. T., & Kanwisher, N. (1998). Binocular rivalry and visual awareness in human extrastriate cortex. Neuron, 21(4), 753-759.

Tsao, D. Y., Freiwald, W. A., Tootell, R. B., & Livingstone, M. S. (2006). A cortical region consisting entirely of face-selective cells. Science, 311(5761), 670-674.

Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549-586). Cambridge, Massachusetts: MIT Press.

Vidal, J. R., Ossandon, T., Jerbi, K., Dalal, S. S., Minotti, L., Ryvlin, P., et al. (2010). Category-Specific Visual Responses: An Intracranial Study Comparing Gamma, Beta, Alpha, and ERP Response Selectivity. Frontiers in human neuroscience, 4, 195.

Weiner, K. S., Golarai, G., Caspers, J., Chuapoco, M. R., Mohlberg, H., Zilles, K., et al. (2014). The mid-fusiform sulcus: A landmark identifying both cytoarchitectonic and functional divisions of human ventral temporal cortex. Neuroimage, 84, 453-465.

Weiner, K. S., & Grill-Spector, K. (2010). Sparsely-distributed organization of face and limb activations in human ventral temporal cortex. Neuroimage, 52(4), 1559-1573.

Weiner, K. S., & Grill-Spector, K. (2011). Not one extrastriate body area: Using anatomical landmarks, hMT+, and visual field maps to parcellate limb-selective activations in human lateral occipitotemporal cortex. Neuroimage, 56, 2183-2199.

Weiner, K. S., & Grill-Spector, K. (2013). Neural representations of faces and limbs neighbor in human high-level visual cortex: evidence for a new organization principle. Psychological Research, 77(1), 74-97.

Weiner, K. S., Sayres, R., Vinberg, J., & Grill-Spector, K. (2010). fMRI-adaptation and category selectivity in human ventral temporal cortex: Regional differences across time scales. Journal of Neurophysiology, 103(6), 3349-3365.

Winawer, J., Kay, K. N., Foster, B. L., Rauschecker, A. M., Parvizi, J., & Wandell, B. A. (2013). Asynchronous Broadband Signals Are the Principal Source of the BOLD Response in Human Visual Cortex. Current biology: CB, 23(13), 1145-1153.

Witthoft, N., Nguyen, M. L., Golarai, G., LaRocque, K. F., Liberman, A., Smith, M. E., et al. (2014). Where is human V4? Predicting the location of hV4 and VO1 from cortical folding. Cerebral cortex, 24(9), 2401-2408.

Wojciulik, E., Kanwisher, N., & Driver, J. (1998). Covert visual attention modulates face-specific activity in the human fusiform gyrus: fMRI study. Journal of Neurophysiology, 79(3), 15741578.

Figure captions:

Figure 1. ECoG responses averaged across anatomical partitions of VTC. The figure depicts the average response across all visually responsive electrodes (the number of electrodes included is displayed at the top) in each of 4 anatomical partitions (bottom right inset): inferior temporal sulcus (ITS)/ inferior temporal gyrus (ITG), lateral fusiform gyrus (LFG)/occipitotemporal sulcus (OTS), medial fusiform gyrus (MFG), collateral sulcus(Cos)/parahippocampal gyrus (PHG)/lingual gyrus (LG). (A) The average ECoG time-frequency response to images of dataset 1 for each category (rows) and anatomical partition (columns). Solid horizontal black lines indicate stimulus duration. The color-scale was adjusted for each anatomical region (see minimum and maximum values in the top-right corner of the top row plots). Data illustrate that HFB-responses (30-160Hz) encompass most of the power increase in the ECoG response. Lower frequencies (<12Hz) exhibit an initial power increase followed by a reduction of power after 0.2-0.3s relative to stimulus onset (Supplementary Figure S1). (B) ECoG HFB signal averaged over the same electrodes and anatomical partitions. Shaded regions around waveforms represent ± 1 SEM across electrodes, computed on amplitude normalized waveforms in each electrode. Solid horizontal black lines indicate stimulus duration. Insets show the first 0.3s after stimulus onset.

Fig 2. Spatial distribution of ECoG category selectivity in human ventral temporal cortex is consistent across participants. (A-B) Localization of electrodes, visual responsivity and category preference in each participant. (A) Spatial distribution of electrode selectivity in an example right hemisphere (ventral view of the temporal lobe), and HFB-responses as a function of time in three example electrodes spaced one centimeter apart. On the cortex, each electrode is represented by a pie chart plotting the relative amplitude elicited by each category (averaged over responses to images of dataset 1) in the HFB range (30-160 Hz) during a time window of 100-350ms after stimulus onset. The diameter of each pie chart is scaled according to its normalized amplitude of response. (B) Spatial organization of ECoG category-selectivity in all other hemispheres. Note that electrodes both within and outside VTC are shown, highlighting the particular sensitivity of VTC to visual object categories. (C) Selectivity in each electrode (d') in four anatomical regions arranged from lateral to medial across the VTC. Each circle represents an electrode, where filled circles show significant positive category-selectivity (p < 0.005). For each electrode we show its selectivity to each category as well as to the animate vs. inanimate stimuli distinction. Horizontal lines represent the average d' across electrodes for each anatomical region. Electrodes overlapping the mid-fusiform sulcus (MFS), lateral fusiform gyrus (LFG) and occipito-temporal sulcus (OTS) show face-selectivity. Preferential responses to inanimate stimuli occurred in the medial fusiform gyrus (MFG), collateral sulcus (CoS) and parahippocampal gyrus (PHG).

Figure 3. Single image category-selectivity in VTC. Each cell shows the normalized HFB response amplitude to a single image averaged across 2-5 presentations over a 100-350ms time window in one electrode taken from data set 2. Electrodes are represented in rows, grouped by anatomical region and category-preference and sorted by descending d' from data set 1 (Fig. 2C). Responses to images are shown in columns, sorted by appearance order for each category. We measured responses to 4080 images per electrode depending on the number of blocks of data collected. Gray cells represent missing data for a particular subject/electrode. d' that are significantly higher than zero (p<.005, 2-tailed permutation test) for data set 1, are indicated on the right and color-coded by category

preference. Electrodes that were not significantly selective to any of the four categories are on the bottom of each anatomical region and no d' is reported.

Figure 4: Representational similarity matrices (RSM) of distributed HFB responses across VTC. RSM

of distributed VTC responses between each pairs of images from data set 2 are shown for each participant. Distributed responses were calculated for each participant across all of their VTC electrodes (electrode count indicated at the top of each panel) for HFB-responses averaged over a time window of 100-350ms after stimulus onset. Responses were averaged across 2-5 presentations of an image (excluding the first presentation). The top triangle shows the correlation of the distributed responses to each pair of images (40-80 images per participant), the bottom triangle shows the average correlation across all pairs of images of a category, excluding the diagonal.

Figure 5: Overlapping face- and house-selectivity across ECoG HFB and fMRI signals. A.

Unthresholded maps of selectivity for faces (faces > limbs, cars, and houses; t value, voxel level) measured with fMRI are displayed for each participant. Inflated cortical surfaces show ventral visual cortex, including the VTC, with the occipital pole at bottom. Overlaid on these maps are circles indicating the magnitude of selectivity (contrast of faces > limbs, cars and houses, d') of ECoG electrodes in the HFB range for a response averaged over a 100-350ms time window relative to stimulus onset. Note that the electrode layouts do not appear as perfect strips or grids on the cortical surface due to the cortical surface inflation process used for better visualization of activations inside sulci. The electrodes displayed are only those in regions where we could measure reliable fMRI responses. The diameter of each electrode has been increased for better visibility. Acronyms: OTS: occipitotemporal sulcus, MFS: mid-fusiform sulcus, CoS: Collateral sulcus. B. Same conventions as for A but showing selectivity for houses (house > faces, limbs, and cars).

Figure 6. The coupling between ECoG and fMRI face- and house-selectivity varies across time and frequency bands. (A) Correlation between category-selectivity measured with fMRI and ECoG as a function of frequency band and time. Each line reflects the spatial correlation between fMRI and ECoG selectivity across time for a specific ECoG frequency band averaged across 6 subjects. Correlations between ECoG and fMRI were computed separately for faces-selectivity and house-selectivity, then averaged in each subject, and last averaged across subjects. Shaded regions: across-subject SEM. Line thickness indicates whether the across-subject correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). (B) Correlation between fMRI and ECoG selectivity across the frequency spectrum (3-160Hz) for two time-windows. For both (A) and (B) the contribution of fMRI signal from each voxel was weighted using a Gaussian kernel of 3.5mm STD around each electrode (Supplementary Figs. S3 and S4).

Figure 7: Category representation structure from VTC patterns correlate across fMRI and ECoG HFB from 100ms after stimulus onset. (A) ECoG RSMs: representational similarity matrices (RSM) of

distributed VTC responses for different frequency bands at four time intervals. RSMs are computed from dataset 2, averaged across images in each participant, and then across participants S1 - S5, who had at least two fMRI scans. Acronyms: F: faces; L: limbs; C: cars; H: houses. (b) fMRI RSM: Across participants' average (S1-S5) RSMs of distributed fMRI signals from voxels under ECoG VTC electrodes. (C) Average correlation between fMRI RSMs and ECoG RSMs across time and frequency band. Correlations were computed in each participant and then averaged across participants. Line thickness indicates if the across-participant correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). Shaded regions indicate across-participant SEM.

Figure 8. ECoG HFB response latency across visually responsive VTC electrodes. Response latencies at each visually responsive VTC electrode are shown for each category and participant (filled circles), along with the average (± STD) across electrodes in each participant (open circles) and the average across all electrodes and participants by category (black diamonds). The number of electrodes included for each category is shown below each group of plots. Some electrodes respond significantly to more than one category. Thus, the same electrode may appear in multiple category bins.

SUPPLEMENTARY MATERIAL

Supplementary table 1

Participants

Participant Seizure Seizure Type Onset

Age 26 Generalized (1980) convulsive events

Frequency

before

implantation

1 every 3-6 months

Auras?

Olfactory and gustatory aura

Normal vision?

Glasses for reading

Cortical resection

Right anterior temporal

Age 11 (1976)

Simple partial seizures with secondary generalized tonic clonic events.

1 per month

Visual aura of kaleidoscope rotating colors in the central visual field and an associated feeling of déjà vu

No report of visual

impairments

No resection: possible epileptic focus in right pericalcarine area

Age 47 (2009)

Age 17 (1989)

generalized tonic-clonic movements or whole body stiffening or confusion.

symptomatic localization-related epilepsy with secondarily generalized tonic clonic seizures

weekly

1 per month

No visual aura reported

Sensory aura in right hand

Glasses for reading

No report of visual

impairments

Right anterior temporal

Left Insula

Age 2 (1990)

complex partial seizures. Some progress into R hand dystonic posturing with subsequent clonic

movements.

3-4 per month

No visual aura reported

No report of Left anterior visual temporal

impairments

Age 15 (2003)

Complex partial seizures and generalized tonic clonic seizures.

1 per month

No visual aura reported

No report of Left anterior visual temporal

impairments

Supplementary Figures

ITG/ITS LFG/OTS MFG COS / PHG / LG

Time from to stimulus onset (s)

Supplementary Figure S1.Time course of ECoG response in theta, alpha and beta frequency bands across anatomical partitions of VTC. The figure depicts the average response across all visually responsive electrodes in each of 4 anatomical partitions (same as Figure 1B) for theta (4-8Hz), alpha (8-12Hz) and Beta (1230Hz) frequency bands. Shaded regions around waveforms represent ± 1 SEM across electrodes, computed on amplitude normalized waveforms in each electrode. Acronyms: inferior temporal sulcus (ITS)/inferior temporal gyrus (ITG), lateral fusiform gyrus (LFG)/occipitotemporal sulcus (OTS), medial fusiform gyrus (MFG), collateral sulcus(Cos)/parahippocampal gyrus (PHG)/lingual gyrus (LG).

A Representational similarity matrices - EVC electrodes S1 S2

-1 o 1

Pearson correlation (similarity)

B Location of EVC electrodes in 2 subjects

Supplementary Figure S2. Absence of representational structure from early visual cortex responses. (A)

Representational similarity matrix (RSM) showing the similarity of distributed responses between each pair of images presented to the participant, based on ECoG responses measured from electrodes located on early visual cortex (EVC) in two participants, as illustrated in (B). Distributed responses were calculated for each participant across these EVC electrodes (4 electrodes for S1; 9 electrodes for S2) using HFB-responses averaged over a time window of 100-350ms after stimulus onset. Responses were averaged across 2-5 presentations of an image (excluding the first presentation). The top triangle shows the similarity of the distributed responses to each pair of images each participant saw. The bottom triangle shows the average responses across all pairs of images of a category, excluding the diagonal. RSMs show an absence of category structure in the distributed pattern of response (compare to S1 and S2 in Fig. 4). This indicates that representational category structure in VTC electrodes is not inherited from early visual areas that process lower-level visual properties of the images. (B) Location of EVC electrodes in both participants. Electrodes were located in the postero-medial half portion of the lingual gyrus (pLG), around the calcarine sulcus (CS) or on the cuneus (Cun).

A Spatial pooling (radius): 1.5 mm 3.5 mm 14 mm — 9 4-6 Hz

1 1 1 — a 8-12 Hz

'°-6 0 0.2 0.4 0.6 0.8 1 1.2 1.4 "°'6 0 0.2 0.4 0.6 0.8 1 1.2 1.4 0 0.2 0.4 0.6 0.8 1 1.2 1.4

Time relative to stimulus onset (s)

Supplementary Figure S3. Correlations between fMRI and ECoG pattern of face-selectivity as a function of frequency band, time, and spatial pooling. (A) Time-course of the correlation between spatial pattern of face-selectivity measured with ECoG and fMRI using 3 different amounts of spatial pooling of fMRI voxels around each electrode. Extent of spatial pooling reflects the contribution of each voxel to the BOLD face-selectivity d' measured for each electrode location. The contribution of each voxel was weighted by its distance to the electrode location using a 3D Gaussian for which we varied its standard deviation. The time courses of ECoG-fMRI correlation are showed for each amount of spatial pooling (1.5, 3.5, 14mm; see (B), middle plot). Each line reflects the ECoG-fMRI correlation across time for one of 4 specific frequency band (see legend in top right) averaged across 6 participants. Error bars indicate across-participants SEM. Shaded vertical bars in the central panel indicate time-windows used in (B). (B) ECoG-fMRI correlation as a function of the parametric manipulation of the spatial extent of pooling of fMRI signals around each electrode for 4 frequency bands and 3 time-windows. Shaded vertical bars in central panel indicate spatial pooling used in the top panels. Error bars indicate across-participants SEM. The magnitude of the correlation was highest for a spatial pooling of ~2.5 -5mm, peaking at 3.5 - 4 mm for all frequency bands and decreasing sharply for smaller or larger volumes of spatial pooling of BOLD signal. Correlations were not significant when including signals from fMRI voxels that are further than 6mm from the electrodes. The 0.4 - 0.8s time-window (central panel) displays significant positive correlation in the HFB and significant negative correlation in the alpha band (ps<.005). However, in this time-window correlations in lower frequency bands (theta and alpha) were less robust than in the HFB. Examination of the ECoG signal (mean % change from baseline) and noise (inter-trial variability) in the 400-800ms time interval reveals that the lower magnitude of the correlation in the alpha and theta frequencies is related to a lowered correspondence between the fMRI-ECoG selectivity profile due to reduced signal amplitude in these frequencies relative to higher frequencies.

Spatial pooling (radius): 1.5 mm

3.5 mm

9 4-8 Hz a 8-12 Hz ß 12-30 Hz HFB 30-160 Hz

0.2 0.4 0.6 0.8

Time-window: 0.1 -0.2s

0.2 0.4 0.6

Time relative to stimulus onset (s) 0.4 -0.8s

0 0.2 0.4 0.6 0.8 1 1.2 1.4

1 - 1.3s

6 8 10 20 30 1 2 4 6 8 10 20 30 1 2

Spatial pooling of BOLD around electrodes (mm)

6 8 10

Supplementary Figure S4. Correlations between fMRI and ECoG pattern of house-selectivity as a function of frequency band, time and spatial pooling. Conventions are the same as for Supplementary Figure S3. The

magnitude of the ECoG-fMRI correlation peaked when the spatial extend of pooling fMRI voxels was about 4 mm around ECoG electrodes. In contrast to ECoG-fMRI correlations of face-selectivity, the magnitude of the correlation did not vary as much with varying spatial pooling.

Jacques et al. Highlights:

- We measured ECoG and fMRI responses to visual stimuli in VTC of the same subjects

- Topology and distributed pattern of category-selectivity match across ECoG and fMRI

- fMRI correlation with ECoG HFB signal is the strongest and most sustained

- fMRI correlation with ECoG low frequency signals is temporal transient

- ECoG-fMRI correlation emerges at ~100ms, simultaneous with neural response in VTC