Scholarly article on topic 'How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge'

How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge Academic research paper on "Clinical medicine"

CC BY-NC-ND
0
0
Share paper
Academic journal
NeuroImage
OECD Field of science
Keywords
{"Semantic knowledge" / Blindness / Supramodality / "Category-based organization" / fMRI / "Multivoxel pattern analysis"}

Abstract of research paper on Clinical medicine, author of scientific article — Giacomo Handjaras, Emiliano Ricciardi, Andrea Leo, Alessandro Lenci, Luca Cecchetti, et al.

Abstract How conceptual knowledge is represented in the human brain remains to be determined. To address the differential role of low-level sensory-based and high-level abstract features in semantic processing, we combined behavioral studies of linguistic production and brain activity measures by functional magnetic resonance imaging in sighted and congenitally blind individuals while they performed a property-generation task with concrete nouns from eight categories, presented through visual and/or auditory modalities. Patterns of neural activity within a large semantic cortical network that comprised parahippocampal, lateral occipital, temporo-parieto-occipital and inferior parietal cortices correlated with linguistic production and were independent both from the modality of stimulus presentation (either visual or auditory) and the (lack of) visual experience. In contrast, selected modality-dependent differences were observed only when the analysis was limited to the individual regions within the semantic cortical network. We conclude that conceptual knowledge in the human brain relies on a distributed, modality-independent cortical representation that integrates the partial category and modality specific information retained at a regional level.

Academic research paper on topic "How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge"

Accepted Manuscript

How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge

Giacomo Handjaras, Emiliano Ricciardi, Andrea Leo, Alessandro Lenci, Luca Cecchetti, Mirco Cosottini, Giovanna Marotta, Pietro Pietrini

PII: S1053-8119(16)30102-1

DOI: doi: 10.1016/j.neuroimage.2016.04.063

Reference: YNIMG13149

To appear in:

Received date: Revised date: Accepted date:

NeuroImage

14 October 2015 12 February 2016 26 April 2016

Please cite this article as: Handjaras, Giacomo, Ricciardi, Emiliano, Leo, Andrea, Lenci, Alessandro, Cecchetti, Luca, Cosottini, Mirco, Marotta, Giovanna, Pietrini, Pietro, How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge, NeuroImage (2016), doi: 10.1016/j.neuroimage.2016.04.063

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

How concepts are encoded in the human brain: a modality independent, category-based cortical organization of semantic knowledge

Giacomo Handjarasa, Emiliano Ricciardia*, Andrea Leoa, Alessandro Lencib, Luca Cecchettia, Mirco Cosottinic, Giovanna Marottab, Pietro Pietriniad'e

aMolecular Mind Lab, Dept. Surgical, Medical, Molecular Pathology and Critical Care, University of Pisa, Pisa,

56126, Italy

bDepartment of Philology, Literature, and Linguistics, University of Pisa, Pisa, 56126, Italy cNeuroradiology Unit, University of Pisa, Pisa, 56126, Italy dClinical Psychology Branch, Pisa University Hospital, Pisa, 56126, Italy eIMT School for Advanced Studies Lucca, Lucca, 55100, Italy Co-first author

Title: 125 characters.

Abstract: 164 words; Introduction: 6266 characters; Materials and Methods: 16191 characters; Results: 12813 characters; Discussion: 20340 characters. Figures: 7; Tables: 0; Pages: 39

Correspondence:

Pietro Pietrini, MD, PhD

IMT School for Advanced Studies Lucca,

Piazza San Ponziano, 6, 55100, Lucca, Italy

pietro.pietrini@imtlucca.it

Conflicts of interest: None declared.

Running title: supramodal cortical organization of semantic knowledge

Abstract

How conceptual knowledge is represented in the human brain remains to be determined. To address the differential role of low-level sensory-based and high-level abstract features in semantic processing, we combined behavioral studies of linguistic production and brain activity measures by functional magnetic resonance imaging in sighted and congenitally blind individuals while they performed a property-generation task with concrete nouns from eight categories, presented through visual and/or auditory modalities.

Patterns of neural activity within a large semantic cortical network that comprised parahippocampal, lateral occipital, temporo-parieto-occipital and inferior parietal cortices correlated with linguistic production and were independent both from the modality of stimulus presentation (either visual or auditory) and the (lack of) visual experience. In contrast, selected modality-dependent differences were observed only when the analysis was limited to the individual regions within the semantic cortical network.

We conclude that conceptual knowledge in the human brain relies on a distributed, modality-independent cortical representation that integrates the partial category and modality specific information retained at a regional level.

Highlights

• Brain responses measured during multimodal semantic processing in sighted and blind

• Conceptual knowledge shows similar representations across modalities in both groups

• Regional representations still retain category- and sensory-based information

• Semantic information relies on a distributed, modality-independent representation

Keywords

semantic knowledge, blindness, supramodality, category-based organization, fMRI, multivoxel pattern analysis.

Introduction

Theories about knowledge organization in the human brain propose that concepts may be described according either to perceptual processes or to semantic representations. Specifically, the modality-specificity theory suggests that knowledge is principally modulated by low-level components related both to sensory modalities (e.g., visual, auditory or tactile) and to motor functions engaged during stimulus acquisition and processing (Warrington and Shallice, 1984). This theoretical framework postulates that knowledge relies on several subsystems (e.g., visual, verbal through written and spoken words): concepts would therefore result from an integrated contribution of sensory-based and motor-based information (Warrington and McCarthy, 1987). In opposition, the domain-specificity theory posits that knowledge may instead be codified within a more abstract organization of semantic attributes independently from sensorimotor processing (Caramazza and Shelton, 1998; Mahon et al., 2009). Within this view, the high-level semantic system would comprise a multidimensional space of semantic properties to represent concepts (Caramazza and Mahon, 2003).

Both these distinct theories sustain an organization of knowledge based on semantic categories and provide an explanation for distinct category-specific semantic deficits in brain-damaged patients (Gainotti, 2010; Grossman et al., 2013). Nevertheless, whether and to what extent the conceptual

structure of knowledge organization is represented at a neural level on features comprising low-level, sensory-based information and/or high-level abstract semantic features still remains an open question (Fairhall and Caramazza, 2013). Furthermore, if conceptual knowledge relies on a common abstract representation across sensory modalities, to what extent would such representation be independent from specific perceptual properties (e.g., visual or auditory features)? Equally, if conceptual knowledge relies on specific sensory-based features, would there be a level at which sensory information 'advances' towards a more abstract representation?

Previously, conceptual representations in congenitally blind individuals - who acquire knowledge exclusively through non-visual inputs - have been explored to solve the entanglement between sensory and semantic processing. If the semantic system strongly hangs onto the sensory modality through which concepts are acquired (as postulated by the modality-specific theory), one would expect knowledge organization to be deeply rearranged as a consequence of congenital blindness. Contrary to such expectations, congenitally blind people show high behavioral similarities with sighted individuals in conceptual representations, even when the processing of semantic features is directly related to visual features (i.e., color terms, verbs of vision) (Bedny et al., 2011; Landau and Gleitman, 1985; Noppeney et al., 2003). In line with these results, a modality-independent ability to discriminate conceptual representations through elicited neural activity has been demonstrated in sighted individuals among semantic categories and across presentation modalities (i.e., visual pictorial, visual word, verbal auditory and natural sounds) (Devereux et al., 2013; Simanova et al., 2014). Semantic discrimination across modalities in sighted subjects, though, may be related to common sensory features sustained by visually-based perceptual or imagery processes (Reddy et al., 2010). To exclude any potential contributions from visual imagery (i.e., from a specific sensory modality), a finer characterization of the neural bases of knowledge organization in those individuals who lack visual experience since birth is needed, to dissect the distinct roles of low-level, sensory modality-grounded perceptual information and high-level abstract semantic properties.

In addition to the relationship between sensory and semantic processing, another key question, that is, how the organization of conceptual knowledge is scaled in the human brain, remains still to be addressed. In other words, does knowledge organization rely on a small-scale localized representation based on an anatomically (and thus functionally) well-defined set of regions (Binder et al., 2009), or rather does it count on large-scale representational patterns of neural activity (Cukur et al., 2013; Huth et al., 2012)? In contrast to a localized perspective, for which each subdivision of the semantic cortical network would manage distinct categories during semantic processing (e.g., knowledge of actions, manipulable artifacts, abstract and concrete concepts, Binder et al., 2009; Fernandino et al., 2015), recently it has been proposed that knowledge may be represented by a category-based organization into a continuous semantic space mapped across a large extent of cortex (Huth et al., 2012). From a neuropsychological perspective, both focal and widespread brain lesions may equally generate category-specific semantic deficits (Devlin et al., 2002; Gainotti, 2010; Moss et al., 2002). As a result, it is still undecided how small-scale and large-scale levels may integrate within a unique semantic system, or at which scale category discrimination may occur (Fedorenko and Thompson-Schill, 2014).

Based on this background, the present study was conceived to investigate (1) the impact of different input modalities on the neural responses related to conceptual representations of concrete nouns within semantic-sensitive cortical regions, and (2) how knowledge organization would be affected by the definition of the semantic system at a cortical level, when shifting from a smaller to a larger scale neural representation.

To determine the role of distinct sensory modalities in semantic processing, we performed a conjoint observation in both sighted and congenitally blind individuals, in combination with various presentation modalities of the same experimental stimuli (i.e., pictorial, verbal visual and verbal auditory). Furthermore, the measurement of neural activity at different scales of cortical representation within the semantic system determined to what extent low-level sensory-based information and/or high-level abstract features contribute to conceptual knowledge organization.

Materials and Methods

Behavioral analysis

The Blind Italian Norming Data (BLIND) set, collected and validated in an Italian independent sample of blind and sighted participants, comprised descriptions of fifty nouns and twenty verbs (Lenci et al., 2013). The nouns consisted of forty concrete and ten non-concrete concepts. In the current fMRI study, only the forty concrete nouns from the BLIND set were used. Nouns belonged to eight different semantic categories (i.e., vegetables, fruits, natural and artificial places, mammals, birds, tools, vehicles) (Supplementary Table S1), which were, for the most part, shared with previous norming studies (Connolly et al., 2007; McRae et al., 2005). For each noun, a set of psycholinguistic features was estimated (i.e., word length, bigram frequency, syllabic frequency, word frequency and familiarity) and compared to exclude differences in linguistic features across categories (Supplementary Figure S1).

In brief, in the BLIND study, 26 sighted and 22 congenitally blind participants listened to concept names, and were asked to verbally enumerate in one minute the properties (features) that describe the entities the words refer to. The collected features were subsequently extracted and pooled across subjects in each group, to derive averaged representations of the nouns in both sighted and blind participants, using subjects' production frequency as an estimate of feature salience (Lenci et al., 2013). Finally, frequencies were normalized by scaling to unit length. This procedure resulted in a normalized feature space of 812 dimensions (properties) for sighted and 743 for blind participants. Representational spaces were derived from feature spaces using cosine similarity index, obtaining two group-level dissimilarity matrices (Supplementary Figure S2). Afterwards, to measure behavioral similarities, representational spaces were compared between blind and sighted participants using the Mantel test (10,000 permutations, one-tailed rank test). Finally, as we were interested in producing

representational spaces related to the category representation, behavioral feature spaces were averaged within the eight semantic categories and were compared between blind and sighted participants using the Mantel test (10,000 permutations, one-tailed rank test). Moreover, a multidimensional scaling procedure (using cosine distance and metric stress criterion) was performed (Figure 2A), to represent behavioral data across sighted and congenitally blind individuals (Kruskal and Wish, 1978).

fMRI experimental setup and participants

Neural responses were measured in fMRI with a five-runs slow event-related design (gradient echo echoplanar images GRE-EPI, GE SIGNA at 3T, equipped with an 8-channel head coil, TR 2.5s, FA: 90°, TE 40ms, FOV = 24 cm, 37 axial slices, 2x2x4 mm voxel size) in 20 right-handed volunteers, 15 sighted and 5 congenitally blind individuals, during a property generation task after visual and/or auditory presentation of the same forty concrete nouns of the BLIND set. Participants were divided into four groups accordingly to the stimulus presentation format: 5 sighted individuals were presented with a pictorial form of the nouns (M/F: 2/3 mean age ± SD: 29.2±12.8 yrs), 5 sighted individuals with a verbal visual (i.e., written Italian words) form (M/F: 3/2 mean age ± SD: 36.8±11.9 yrs), 5 sighted individuals with a verbal auditory (i.e., spoken Italian words) form (M/F: 2/3 mean age ± SD: 37.2±15 yrs) and 5 congenitally blind with a verbal auditory form (M/F: 2/3 mean age ± SD: 36.4±11.7 yrs). High resolution T1-weighted spoiled gradient recall (SPGR; TR= 8.1 ms; TE= 3.1 ms; 170 sagittal slices; voxel size=1x1x1 mm) images were obtained for each participant to provide detailed brain anatomy. The study was approved by the University of Pisa Ethical Committee. All participants signed a written informed consent prior to enrollment in the study.

[Insert Figure 1 about here]

During the visual presentation modality, subjects could be presented either with images representing the written word (verbal visual form) or color pictures of concrete objects (visual pictorial form). Stimulus presentation lasted 3 seconds and was followed by a 7-s inter stimulus interval (Figure 1). During the auditory presentation modality, subjects were asked to listen to about 1s-long words -referring to the same concrete nouns above - followed by 9-s of inter stimulus interval (Figure 1). To reduce the impact of the fine grained details of each stimulus presentation modality (i.e., images or audio clips), stimuli were presented four times each, changing the voice gender for the auditory form or presenting different image versions for the pictorial form (Handjaras et al., 2015; Haxby et al., 2001). Stimuli were randomly presented across runs and subjects to avoid any bias in the presentation sequence (Mumford et al., 2014). Participants were instructed to mentally generate a set of features related to each noun (e.g., for a dog: "it's an animal", "has four paws") during each 10-s long stimulus presentation block. To improve generalization of the stimuli (i.e., higher level representation of the associated concepts), participants were briefed to think about as many properties of the stimulus as they could during each presentation, in order to obtain a neural activity related to its multiple properties (Mitchell et al., 2008; Simmons et al., 2008). Each run had, at its onset and ending, two 15-s long blocks of rest, to obtain a baseline measure of activity. Stimulus presentation was handled by using the software package Presentation® (Neurobehavioral System, Berkeley, CA, http://www.neurobs.com) via the RM compatible visual stimulation device (VisuaStim-Resonance Technologies, Northridge, USA; dual-display video, visual field: 30° x 22°, 5'', 800x600 pixels, 60Hz) and MR-compatible headphones. The size in visual angles (mean accuracy ± standard deviation) was 18.9°±4.1° x 14.8°±2.6° for the pictures in the pictorial form and 7.7°±2.5° x 1.9°±0.3° for the words in the verbal visual form.

fMRI data preprocessing

The AFNI software package was used to analyze functional imaging data (Cox, 1996). After a

standard preprocessing (see Supplementary Material for further details), a multiple regression analysis was performed. The pattern of response to each stimulus was modeled across repetitions with a regressor. The obtained t-score response patterns of each stimulus were used as an estimate of the strength of Blood-Oxygen-Level Dependent (BOLD) response compared to rest and included in the subsequent encoding analysis. Moreover, to reduce computational effort in the subsequent steps, a spatial filter was applied to select grey matter only regions, thus to decrease the total number of voxels.

Single-subject encoding analysis

Since we were interested both in generating a machine learning procedure based on the category encoding matrix, where each stimulus was identified according to the membership to one of the eight semantic categories (i.e., a semantic category model as in Devereux et al., 2013), and in selecting the most informative voxels without any a priori selection criteria (e.g., a "stability score") (Mitchell et al., 2008), an ad-hoc procedure was developed (Supplementary Figure S8). Briefly, as proposed by Mitchell and colleagues (Mitchell et al., 2008), a machine learning algorithm predicted the fMRI activation in the brain as a weighted sum of images, each one generated from an encoding dimension. In detail, a least-squares multiple linear regression analysis, performed within a leave-two-stimuli-out cross-validation procedure produced a set of learned scalar parameters that specified the degree to which each encoding dimension modulates the voxels activity. Hence, for each iteration of the cross-validation procedure, the model was first trained with 38 stimuli, then only the 500 voxels (Mitchell et al., 2008) that showed the highest coefficient of determination R (a measure of the fitting between the encoding matrix and the training set) and that survived a small volume correction to remove small isolated clusters (nearest neighbor, minimum cluster size of 20 voxels) were retained. Once trained, the resulting algorithm was used to predict the fMRI activation within the selected 500 voxels of the two left-out stimuli. Afterwards, its prediction accuracy was evaluated with a simple match between the predicted and the real fMRI activations of the two left-out stimuli using cosine similarity (Mitchell et

al., 2008). This leave-two-out procedure was iterated 780 times, training and testing all possible stimulus pairs. Eventually, the procedure generated, for each participant, an accuracy value related to the goodness of the fitting of the encoding matrix on the data and a brain mask with the subset of voxels used during the procedure (see Supplementary Material for methodological considerations).

Finally, the classification accuracy of the encoding procedure for each participant was assessed with a binomial test (Pereira et al., 2009).

Cortical probability maps

First, FMRIB's Nonlinear Image Registration tool (FNIRT) was used to register the highresolution T1 images to the standard MNI-152 1 mm iso-voxel template (Andersson et al., 2007; Smith et al., 2004). The matrix of coefficients was subsequently applied to the functional images. Afterwards, to measure the spatial consistency of the information content patterns across individuals, a posterior probability map was built using the single subject brain masks that comprised the most informative voxels involved in the stimuli discrimination from the encoding procedures (Supplementary Figure S3). To avoid any circularities in the analysis, a leave-one-subject-out procedure was adopted to generate an unbiased set of probability maps, thus to select the regions from 19 subjects and to perform the subsequent information content measures in the left out subject (see below) (Supplementary Figure S9) (Kriegeskorte et al., 2009). Two thresholds were arbitrarily selected (p>0.25 and p>0.5) that represented the probability of a voxel to be informative in at least 5 participants or in 10 participants, respectively. In addition, a cluster correction (nearest-neighbor, minimum cluster size of 100 ^L) was applied to each map in order to remove small isolated clusters, yielding to a stable location of all the above threshold regions across the probability maps of the participants. In summary, each iteration of this leave-one-subject-out procedure led to two cluster-corrected cortical probability maps comprising the voxels whose information content was consistent across at least 5 or 10 participants, respectively.

Information content measures

The subsequent measures of information were computed - at the single subject level - within the leave-one-subject-out procedure used to define the probability maps (Supplementary Figure S9).

Initially, to compare the patterns of BOLD activity across different participants and experimental conditions, data were normalized subtracting the mean value of the region from all the voxels in the region, within each stimulus. This procedure was applied to all the regions, probability maps and subjects. Afterward, information content measures were calculated. Specifically, representational space similarity across presentation modalities, representational space similarity between presentation modalities and behavioral data and semantic categories discrimination ability based on rank accuracy were measured.

Representational spaces measures

Firstly, within each region and subject, all the patterns of BOLD activity related to the stimuli were averaged across the eight semantic a priori categories thus to obtain 'high-level' semantic category representations. After that, using cosine distance, representational spaces were derived from these semantic representations and averaged across subjects in each group. This procedure generated matrices with the same dimensions for all the experimental conditions (i.e., fMRI and behavioral data). Subsequently, representational spaces obtained through different presentation modalities were compared using the correlation coefficient. Representational spaces from fMRI and behavioral data were compared as well, in blind and sighted individuals separately (Connolly et al., 2012; Devereux et al., 2013; Kriegeskorte et al., 2008). Finally, these correlation coefficients were tested with a Mantel test (10,000 permutations, one-tailed rank test, Bonferroni-Holm corrected for the number of presentation modalities). In addition, a multidimensional scaling procedure (using cosine distance, metric stress criterion and Procrustes alignment) was performed in Figure 2B to represent patterns of

BOLD activity across sighted and congenitally blind individuals as compared to behavioral data of Figure 2A (Kruskal and Wish, 1978).

Discrimination measures

Within regions, probability maps and subjects, the ability to recognize between stimuli was evaluated with a rank accuracy measure. Briefly, a cross-validation leave-one-stimulus-out procedure was developed: for each iteration we computed the cosine similarity between the left-out stimulus and the categories, generated by averaging the remaining stimuli within-category, obtaining, for each stimulus, a normalized rank-ordered list from the most likely (higher cosine similarity) to the least likely category (lower cosine similarity) (Mitchell et al., 2004). Therefore, the resulting rank error ranged from 0 to 1, corresponding to the case in which the category the stimulus belongs to is ranked most and least likely, respectively. Moreover, using such a procedure, the chance level was always 0.5 regardless of the number of categories involved.

Afterwards, the accuracies of stimulus discrimination were averaged within categories and statistically tested against a null distribution obtained by shuffling the membership of the stimuli to the eight semantic categories within 10,000 permutations (one-sided rank test, Bonferroni corrected for the number of categories). The p values obtained from the permutation tests for each category were combined across subjects and within groups with Fisher's method (Bailey and Gribskov, 1998) and subsequently corrected for multiple testing across categories and presentation modalities, using False Discovery Rate (FDR) (Benjamini and Hochberg, 1995).

Finally, within each region, a category preference measure was calculated. Indeed, during the rank-accuracy calculations, categories may show an always-high rank in certain regions despite variability across participants and sensory modalities. Conversely, other categories, even if they were discriminated above chance across subjects, showed heterogeneous ranks across different subjects and regions, so they did not exhibit such a bias. This category bias measure may indicate a preference,

within a region, for encoding features related to specific categories. Moreover, these features appeared to be encoded independently from sensory modalities and from subject variability. The bias was assessed, for each category, by collecting all the rank accuracies from all the participants, then converting them into a unique category rank and eventually averaging the latter across subjects. The resulting mean rank was tested against a null distribution with a permutation test (10,000 permutations, one-tailed rank test, Bonferroni-Holm corrected for the number of categories).

Low-level sensory regions

In addition to the regions included in the probability maps, two regions (the Heschl Gyrus -HG-and the Calcarine Sulcus -CalS-) were added to assess the information content measures in primary sensory regions. The HG and CalS regions were defined using the Jülich histological atlas of the FMRIB Software Library (Eickhoff et al., 2007; Smith et al., 2004), retaining about 8600 pL for each region that extended in both hemispheres.

The behavioral analysis, the encoding procedure and the information content measures were accomplished with Matlab (Matworks Inc., Natick, MA, USA), while BrainVisa was used to render the brain meshes in Figure 4.

Results

Behavioral similarities between sighted and congenitally blind individuals

First, whether the semantic representations of the stimuli differed between the two groups was assessed.

In a behavioral norming experiment aimed at obtaining featural descriptions of the stimuli, the verbally-generated properties of forty concrete nouns belonging to eight different semantic categories

(i.e., vegetables, fruits, natural and artificial places, mammals, birds, tools, vehicles) were collected in an independent sample of sighted and congenitally blind individuals (Lenci et al., 2013).

Here, a representational similarity analysis (Kriegeskorte and Kievit, 2013) evaluated the likeness between representational spaces derived from behavioral data relative to each noun stimulus in sighted and blind individuals. The correlation between these spaces was 0.94 (p < 0.0001), indicating their almost complete similarity. Moreover, as shown in Figure 2A, a multidimensional scaling analysis showed that the forty nouns were clearly organized along the originally-defined eight semantic categories across sighted and blind individuals (Supplementary Figure S2). Altogether, these results confirmed that category-based information is a dominant component in the definition of concepts of both groups.

[Insert Figure 2 about here]

As shown in Figure 3, even when the dimensionality of representational spaces was reduced on a category-based criterion (i.e., averaging the features space within each semantic category), a high correlation between blind and sighted spaces was still exhibited (0.96; p < 0.0001), similarly to what resulted from the representational space comparison at the level of individual nouns.

[Insert Figure 3 about here]

Category-based cortical representation in sighted and congenitally blind individuals

In a functional magnetic resonance imaging (fMRI) experimental setup, Mitchell and colleagues (Mitchell et al., 2008) developed an encoding procedure based on a multi voxel pattern analysis to uncover the hidden conceptual representations related to the distributional properties of many concrete

nouns in a language corpus. Here, we adopted a similar procedure to isolate the brain regions involved in the discrimination of semantic categories. Specifically, to determine the role of the sensory modalities during concept formation, twenty volunteers were equally divided into four experimental groups according to the stimulus presentation format: three groups of five sighted participants were presented with visual pictorial, verbal visual (i.e., written words), and verbal auditory (i.e., spoken words) presentation modalities, respectively, and five congenitally blind participants were presented with verbal auditory stimuli (i.e., spoken words). Patterns of neural activity were acquired while participants mentally generated features related to each presented noun.

Following the behavioral results that highlighted a categorial structure of the conceptual representation, a multi voxel pattern analysis defined on a category-based encoding model was used to distinguish the forty concrete nouns across presentation modalities (Devereux et al., 2013; Mitchell et al., 2008). As results, nouns were significantly discriminated in 19 out of 20 participants (p<0.05) (one sighted subject in the verbal visual modality performed below chance: results were reported in Supplementary Table S2). In detail, the accuracy in the pictorial presentation form reached a value of 77±6% (mean accuracy ± standard deviation; chance level: 50%), in the verbal visual form 63±9%, while in the two verbal auditory forms in sighted and in congenitally blind individuals resulted in classification values of 66±9% and 62±6%, respectively.

Different cortical scales of the category-based representation

The encoding procedure was performed not only to obtain a mere accuracy value for the discrimination of nouns in each presentation modality, but also to identify the cortical regions consistently involved in semantic processing across participants. Specifically, a posterior probability map was built using the single individual brain masks that comprised the most informative voxels detected during encoding. Two thresholds were arbitrarily selected (p>0.50 and p>0.25) corresponding to voxels that were present in at least 50% and 25% of the individual maps, respectively. Therefore,

two brain networks with different extension and composition were retained: these two levels, that we labeled as small-scale and large-scale, provided the opportunity to measure the information content at different scales of neural representation and to characterize how knowledge organization is affected by the definition of the semantic system at a cortical level.

First, at a probabilistic threshold of p>0.50, a set of regions, relatively limited in extension and anatomically well-defined identified the small-scale level (Figure 4A). Seven regions survived to this probability threshold. Five were localized in the left hemisphere: RetroSplenial Cortex (RSC) and Parahippocampal (PH), Lateral Occipital (LO), Temporo-Parieto-Occipital (TPO) and Inferior Parietal (IP) cortices. Conversely, RSC and TPO were also located in the right hemisphere. Volumes and anatomical location of the clusters included in the small-scale level were reported in detail in Supplementary Table S3.

[Insert Figure 4 about here]

Second, applying a probabilistic threshold of p>0.25, a set of broadly extended cortical networks was identified, thus defining the large-scale level (Figure 4B). Within this level, four extended clusters survived to the probability map procedure. Two of them were located in the left hemisphere: a large posterior semantic network (PSN) that merged PH, LO, TPO and IP regions (as defined in the small-scale representation), and an anterior semantic network (ASN) that encompassed motor, ventral premotor and dorsolateral prefrontal regions. Moreover, in the right hemisphere, similarly to the left one, a large PSN was present. Finally, at the large-scale, the left and right RSC merged in a unique cluster within the Parieto Occipital (PO) cortex.

Although this encoding procedure highlighted the regions involved in semantic processing,

additional approaches were employed to fully characterize the information content (Naselaris et al., 2011). In detail, as representational similarity analysis offers the possibility to express data as adimensional measures, brain activity patterns were evaluated across the presentation modalities to assess any similarities (i.e., higher level, modality-independent content) or differences (i.e., low-level, sensory modality-dependent processing) and were then compared to the behavioral representations to determine their binding with the linguistic output.

Nonetheless, this technique evaluated the representational space as a whole and could not assess effectively the existence of specific category-based representations or similarities across sensory modalities. Therefore, a multivariate method based on rank accuracy verified, in each region, which categories could be clearly distinguished (Mitchell et al., 2004). Additionally, a measure of category preference was defined as a bias that indicates a preferential encoding of features related to specific categories, independently from sensory modality and subject variability.

Describing the information content at the small-scale level

Within the small-scale level, an assessment of the information content in each region was performed: the obtained results were depicted in Figure 5A and 6A (p < 0.05, corrected for multiple comparisons) (Supplementary Figure S4 and S5).

[Insert Figure 5 about here] [Insert Figure 6 about here]

Three distinct pattern of correlation among representational spaces could be isolated.

In left PH and left IP, representational spaces exhibited a common structure across sensory

modalities, thus suggesting a sensory-independent processing. In addition, in left PH and left IP, a significant correlation was observed by comparing the behavioral and the neural representational spaces, hence indicating a high similarity between language and brain activity. Moreover, an overall ability to discriminate between categories across participants and presentation modalities was found. However, moving to a finer description, a category preference across subjects and sensory modalities could be assessed for artificial places in left PH and tools and fruits in left IP.

In left LO, verbal visual and verbal auditory stimuli in sighted individuals presented comparable representational spaces, while the verbal auditory space in blind individuals shared a similar structure to the pictorial one of sighted participants, thus suggesting a functional reorganization of LO in blind individuals. Furthermore, only in blind individuals the behavioral and the neural representational spaces were significantly correlated, hence supporting a pivotal role of this region in the reorganization of semantic knowledge subsequent to blindness. Finally, in left LO, an overall ability to recognize categories across presentation modalities was retained, without a category preference.

In another set of regions, including bilateral TPO and RSC, representational spaces significantly shared a common structure across pictorial, verbal visual and verbal auditory stimuli in sighted individuals only, while congenitally blind participants had a unique representational space. Interestingly, the highest accuracies during stimulus discrimination were found in left TPO in both sighted and congenitally blind individuals, while partial discriminations were found in right TPO, right RSC and left RSC, especially for categories in the pictorial form. In both RSCs, natural and artificial places were also distinguishable in almost all the presentation modalities and retained a category preference. Overall these results suggested that visual perception strongly shapes the representational spaces of these regions, even if their category preference is not substantially hampered by blindness.

Describing the information content at the large-scale level

Similarly to the small-scale, the same analyses were performed on the large-scale level as depicted

in Figure 5B and 6B (p < 0.05, corrected for multiple comparisons) (Supplementary Figure S6 and S7).

First, by correlating the representational spaces generated from neural activity, a significant, unique space was observed across presentation modalities in bilateral PSN and left ASN. Conversely, in the PO cortex, a common space was found across pictorial, verbal visual and verbal auditory stimuli in sighted individuals only, while the blind one had a different structure, consistent with the small-scale results of the RSCs.

Second, by comparing the behavioral data with the representational spaces generated from brain activity, a significant correlation across all presentation modalities was detected in the left PSN only. Indeed, left PSN retained the highest association (r=0.55±0.12, mean correlation across modalities ± standard deviation) between brain activity and the linguistic behavioral data across all the identified regions, both at the small- and large-scale levels. Therefore, the unsupervised arrangement of the forty nouns from the behavioral data (Figure 2A) and from their neural counterpart in left PSN (Figure 2B) resulted to be highly similar. Indeed, the match between the category-based representation of the behavioral data and the neural activity of left PSN was evident also when the presentation modalities were analyzed separately, as depicted in Figure 7. In addition, in the left PSN, the highest overall ability to discriminate between categories in all the presentation modalities was found, without a category preference.

[Insert Figure 7 about here]

Information content measures within primary sensory areas of stimulus presentation modalities

In addition to the region extracted from the probability maps, two regions (i.e., Heschl Gyri -HG-and the Calcarine Sulci -CalS-) were added. Indeed, these primary sensory regions allowed us to assess whether low-level perceptual features might account for the results found at the small- and large-scale

levels. The obtained results were depicted in Figure 5A and 6A (p < 0.05, corrected for multiple comparisons) (Supplementary Figure S4 and S5).

Within the primary sensory regions, both the representational spaces of HG and CalS did not show significant correlations across presentation modalities. Moreover, when comparing the behavioral representation with the spaces generated from brain activity patterns, no significant correlations were found. Finally, in the HG and CalS cortices, only few categories were significantly classified and no category preference was identified.

Discussion

In this study, brain responses were measured with fMRI during semantic processing of forty concrete nouns across different sensory modalities in a sample of sighted and congenitally blind individuals. Using stimuli belonging to a relatively large number of semantic categories and a methodological procedure to describe information content in the brain, we aimed at disentangling the role of low-level sensory-based information from high-level semantic features. Specifically, comparison of data obtained from sighted and congenitally blind individuals enabled us to identify the contribution of visual sensory experience to conceptual representation. Furthermore, representational analysis was performed to determine how knowledge organization is affected when the analysis is focused onto two different levels of information organization in the cortex: a small-scale level limited to region-specific contents, and a large-scale level that relies, as a whole, on a distributed network of regions engaged during the processing of semantic information. Finally, by using behavioral data from both sighted and congenitally blind individuals, patterns of brain activity across sensory modalities were compared to linguistic production.

The overall category-based organization of conceptual knowledge did not differ across

presentation modalities or between sighted and congenitally blind individuals. Specifically, regional differences within patterns of neural activity across the visual and auditory modalities and between groups were observed only when the analyses were limited to the small-scale level. In contrast, patterns of neural activity within the wider semantic cortical network appeared to be independent from both the sensory modality of stimuli presentation and (lack of) visual experience. Moreover, these patterns were correlated with linguistic production.

Supporting a category-specific knowledge organization

The analysis of neural patterns by the means of a machine learning approach based on encoding techniques (Mitchell et al., 2008; Naselaris et al., 2011) was able to discriminate significantly the forty nouns across all presentation modalities and groups. The highest accuracy was reached in the pictorial form, in line with findings from other studies with similar methodologies and experimental paradigms (Chang et al., 2011; Mitchell et al., 2008; Pereira et al., 2013). Accuracies were above chance for verbal visual and for verbal auditory modalities as well, in both sighted and congenitally blind individuals. The success of the encoding model in discriminating the stimuli within fMRI-measured patterns of neural responses across individuals indicates that sighted and congenitally blind individuals share a common, category-based representation of knowledge.

Sensory-based information modulates neural activity at the small-scale level

The choice of two different threshold levels for discussing the probability maps represented an innovative procedure adopted in this study: a higher threshold generated a map showing how semantic information is organized at a small-scale, regional level, whereas a lower threshold identified a map displaying the large-scale distribution of information across a wide extension of cortex. While most of the same voxels contributed to both maps, their information content changed according to the 'level of detail' of the analysis. Specifically, these large-scale and small-scale maps were characterized by

distinctive properties: the large-scale level comprised regions that did not show a preference for semantic categories and showed a consistently strong correlation with the behavioral representations and across presentation modalities. On the contrary, at the small-scale level, it was possible to identify regions with information content that showed category preferences, was only partially correlated to behavioral data and mainly retained a modality-dependent structure.

In detail, the small-scale level showed a mostly left-lateralized network involved in semantic processing, including left PH, RSC, LO, TPO, IP, while only two regions (i.e., RSC and TPO) were located on the right hemisphere (Binder et al., 2009; Price, 2012).

Among the identified regions, the highest discrimination accuracies were shown in the TPO for almost all categories and across modalities. Our TPO region mainly overlapped with the angular gyrus, a crucial region of the semantic network which is considered as a high-level processing area, due to its anatomical location interconnecting the visual, spatial, auditory and somatosensory systems (Binder et al., 2009). Interestingly, while sighted individuals shared a common representational space across pictorial, verbal visual and verbal auditory forms, congenitally blind participants had a unique space, but the information content in TPO in blind individuals was still able to discriminate among categories. In addition, only the pictorial condition in TPO significantly matched linguistic production. Since brain activity patterns in sighted showed a common representational space across presentation modalities and differed as compared to blind, we could assume that visual perception and imagery shape information encoding in TPO. Nonetheless, TPO also plays a role in semantic processing of non-visual inputs in blind participants.

Another node of the semantic network was the RSC. As for the TPO, sighted individuals shared a common representational space across modalities, while the representation in congenitally blind participants was different. The similarity between the behavioral and the neural representational space of left RSC was significant in the pictorial condition only. In addition, a broad discrimination of categories in the RSC was found mainly in the pictorial form, while artificial and natural places were

successfully classified and showed a category preference across modalities. Indeed, the RSC recruitment during processing of spatial features and in object-context integration (Vann et al., 2009) may explain the categorical bias for natural and artificial places found in this region. As for TPO, the overall results indicate that vision does shape information encoding within this region.

A pivotal node during the processing of shape features also in semantic tasks is (Bedny et al., 2011; Devereux et al., 2013; Fernandino et al., 2015; Lane et al., 2015; Peelen et al., 2013). In LO, enough information was retained to discriminate across almost all the categories in visual as well as in auditory modalities, without a category preference. Interestingly, representational spaces indicated that LO shows a common structure for verbal auditory and verbal visual modalities, while the representational space in the blind subjects correlated with the representation of the pictorial form. The similarity between the auditory space in the blind group and the pictorial space in sighted subjects is in line with results from other studies that identified LO ability to process 'visual' features of object form across visual, auditory and tactile tasks (Amedi et al., 2007; Kim and Zatorre, 2011; Peelen et al., 2013; Peelen et al., 2014; Pietrini et al., 2004; Renier et al., 2010; Striem-Amit et al., 2015; Strnad et al., 2013). Recently, in a semantic category-based experiment, Wang and colleagues demonstrated that functional patterns in the posterior lateral parts of the visual cortex across several categories were comparable between blind and sighted individuals (Wang et al., 2015). Our results extend the above findings by directly assessing the high-level semantic representation in LO in congenitally blind individuals.

Within the nodes of the semantic network, only PH and IP exhibited a modality-independent

behavior. In the PH, the overall ability to discriminate among categories was maintained (with the

exception for living beings, like mammals and birds) with a category preference for artificial places.

The representational spaces of all modalities shared a unique common structure that was also congruent

with the behavioural representation. In the literature, this region has been credited with processing of

high-level visual content, like scenes and complex environments (Epstein and Ward, 2010; Park et al.,

2011), and has also been identified as a core region of the semantic system (Binder et al., 2009; Bruffaerts et al., 2013). Indeed, the role of medial temporal lobe regions, including PH, may be related to the formation of declarative memories in the retrieving and processing of semantic properties, as suggested by studies in patients with brain lesions, semantic dementia or amnesia (Levy et al., 2004).

Finally, a high ability to discriminate categories among modalities was found in IP, with category preferences for tools and fruits. As in PH, the representational space of all modalities shared a unique common structure that was also consistent with linguistic representation (Mahon et al., 2010). Recently, a neurosemantic study highlighted the role of IP for representing sensory-motor features (e.g., affordance) within a similar set of concrete nouns (Just et al., 2010). Thus, the tool-related preference may be actually related to object-action interactions (Fogassi et al., 2005), and suggests that IP may play a key role in extracting semantic attributes relevant for action (Price, 2012; Stoeckel et al., 2009).

Towards a more abstract, supramodal semantic organization at the large-scale level

The map at the lowest threshold identified four networks: a left posterior semantic network (PSN), including left PH, LO, TPO and IP; a left anterior semantic network (ASN), including motor, ventral premotor and dorsolateral prefrontal cortex; a right PSN, including the homologous regions of the left one; and a region within the Parieto Occipital (PO) cortex. All these regions, apart from PO cortex, showed high correlations across the representational spaces related to the individual modalities, suggesting a modality-independent processing of semantic features. Moreover, the highest correlation between brain activity and the behavioral semantic space as well as the highest accuracies when discriminating between categories were found in left PSN, pointing to a direct link between the information stored in this network and the output expressed by language. Finally, no category preferences were exhibited in any network, indicating a broader ability to retain semantic knowledge at the large-scale level.

At a more general level, the large-scale approach implies that during the functional inference of

brain regions, only multivariate analyses were able to take into account the high-dimensional cortical activity during semantic processing and to compare neural contents with behavioral information (Haxby et al., 2014). Nonetheless, a critical issue with distributed representations may relate to how multiple sources of information (i.e., brain regions) are integrated (Fedorenko and Thompson-Schill, 2014). At our large-scale level, the information content of the whole cortex of interest resulted from a merely linear weighting of information across voxels: using such an approach, we obtained a representational space showing the highest correlation with the behavioral data. Nevertheless, the human brain might weight the information with different criteria (e.g., with different weights for each voxel or using nonlinear criteria). This issue remains critical to understand how conceptual representations are formed at the neural level. Our large-scale level resulted from the spatial integration of the information content of multiple voxels. Further studies will have to investigate whether this distributed representation is associated with a near-simultaneity of neural activity spreading across the large-scale map, as previous observations suggested (Pulvermuller et al., 2009; Sudre et al., 2012).

Primary sensory areas provide a minimal contribution to conceptual knowledge

Two regions, the Heschl Gyri and the Calcarine Sulci, also were defined to evaluate the information content measures. As expected, no significant correlations were found between brain activity in these regions and the linguistic output across presentation modalities. Only a few categories were discriminated above chance, likely due to their psychophysical features (e.g., specific visual spatial frequencies for places and vehicles) or psycholinguistic characteristics (Supplementary Figure S1). However, no category preferences were identified, overall suggesting that the visual and auditory primary cortical regions are strictly unimodal and do not contribute significantly to high-level semantic representations.

How the small-scale and the large-scale levels may coexist? Existence of modality- and category-

specific responses without a specific hub for semantic processing

At the large-scale level, the consistency of the neural representation across different presentation modalities supports a pure domain-specific organization of the conceptual knowledge that is unrelated to a specific sensory modality and relies on a common semantic framework, for both the decoding of words and the recognition of pictures. Nonetheless, the role of a low-level, sensory-based coding of information is also sustained by the small-scale level. Hence, the domain- and modality-specificity theories may actually coexist at different scale levels of representation of conceptual knowledge.

Interestingly, the different functional features of the semantic processing regions suggest that focal lesions may lead to several distinct consequences ranging from sensory-dependent to sensory-independent semantic impairments.

In addition, the existence of progressive levels of semantic representations would explain several clinical observations indicating that both focal and widespread brain lesions may equally account for category-specific deficits (Devlin et al., 2002; Moss et al., 2002), thus integrating classical neuropsychological evidence with recent functional descriptions of knowledge organization that support a semantic system continuously distributed across the whole cortex (Cukur et al., 2013; Huth et al., 2012).

Apparently, the categorial nature of semantic deficits in patients would conflict with the functional hypothesis of a unique and extended semantic system. In fact, according to this model (Cukur et al., 2013; Huth et al., 2012), the whole cortex would continuously store information about all categories. Conversely, our functional observations confirmed not only an extended neural organization that discriminates among categories, but also demonstrated how more localized cortical clusters show a clear semantic category preference across individuals and sensory modalities. Conceivably, small patches of cortex might have retained the ability to process features related to the more biologically relevant categories across sensory modalities; in the same way, information pertaining to other less relevant categories may be handled in a modality- or subject-dependent way.

In addition, since a wide network of regions is required to process semantic information, our results do not support the hypothesis that conceptual knowledge relies on specific hubs (e.g., Anterior Temporal Lobe) (Rice et al., 2015; Rogers and Patterson, 2007).

Limitations

Property-generation tasks typically rely on verbal or mental enumeration of semantic features. Their intrinsic limitation relates to the lack of a strictly balanced control condition. Nonetheless, their more 'ecological' nature (e.g., lack of task components in the hemodynamic BOLD response, no artifacts due to verbalization) makes these tasks successfully employed in brain functional protocols (Mitchell et al., 2008; Simmons et al., 2008). In particular, in our experiment, the property generation task was first performed in a behavioral study and semantic feature norms were previously validated in larger, independent samples of sighted and congenitally blind individuals. More importantly, even if the fMRI property generation task was uncontrolled, the neural patterns of the forty noun stimuli were significantly discriminated in 19 out of 20 participants: consequently, the linguistic production during the behavioral study, assessed using representational similarity analysis, resulted to be significantly tied to the neural activity detected during the fMRI task. Finally, the validity of the task is supported by the results obtained from the pictorial presentation, which closely matched those obtained using similar experimental designs (Mitchell et al., 2008).

Second, our sample size of twenty right-handed healthy subjects comprised only subsamples of five individuals. While this number may appear relatively small for a standard fMRI study using univariate analyses for group comparisons, this is not the case for those studies, like ours, that adopt a representational analysis and multivariate encoding techniques. Actually, one of the main advantages of representational analysis is the possibility to obtain stable representations by averaging spaces across subjects (Carlson et al., 2014; Ejaz et al., 2015; Kriegeskorte and Kievit, 2013; Kriegeskorte et al., 2008). Equally, concerning the encoding techniques, the analyses are performed in each subject

separately and do not require large samples either (Cukur et al., 2013; Huth et al., 2012; Mitchell et al., 2008). As a matter of fact, while validating the methodological pipeline and determining the impact that an increased number of subjects may have on encoding accuracy, we did obtain data from a larger sample of subjects (n=10) using the pictorial presentation form. We included subgroups of five subjects for each presentation modality, as this sample size turned out to be adequate to pursue the study aims (Supplementary Figure S12).

Also, we should point out that the organization of conceptual knowledge was here studied only through concrete nouns. Even though the nouns used as stimuli covered a wide spectrum of semantic categories, from artifacts to places or animals, a complete evaluation of conceptual knowledge would require the inclusion of abstract entities.

Finally, while the use of two fixed threshold levels was subsidiary to pursue a proper segmentation of the whole semantic network at different scales of neural representation, the results at the large-scale level imply a distributed and overlapping cortical representation of conceptual knowledge. Of note, an a posteriori evaluation of the appropriateness of our arbitrary selection of two probability map thresholds indicated that the whole semantic space defined by behavioral data was associated to a distributed pattern of brain activity that overlaps to our large-scale map (Supplementary Figure S10). Furthermore, this close matching between the large-scale organization and the behavioral data is not dependent on the voxelwise signal-to noise ratio of larger probability maps. When the threshold of the probabilistic mapping is increased, so to retain a lower number of voxels, the signal-to-noise ratio increases, while the matching between fMRI and behavioral data decreases, as indicated in Supplementary Figure S11.

Additional minor methodological issues were discussed in details in the Supplementary Materials and Methods.

Conclusions

To conclude, shifting the definition of the semantic system at a cortical level from a smaller to a larger scale neural representation determined to what extent low-level sensory-based information and/or high-level abstract features contribute to the organization of conceptual knowledge.

We propose that large-scale neural representations are an effective model to explain how the human brain processes semantic information and how conceptual knowledge emerges. The integration of information content across a large extent of cortex generates a unique, modality-independent, internal representation that matches behavioral data and retains the most precise definition of concepts. In contrast, small-scale neural representations of limited regions showed category preferences and mainly retained a modality-dependent structure. These two distinct levels of semantic processing explain how information progresses from a sensory-based towards a more abstract conceptual representation.

Author contributions

E.R., P.P., A.Lenci and G.M. conceived the study; G.H., A.Leo, L.C., A.Lenci and E.R. designed the experiment; A.Leo, L.C., and M.C. acquired the data; G.H., A.Leo and L.C. analyzed the data; all authors interpreted the results; G.H., A.Leo, L.C., E.R. and P.P. drafted the manuscript; A.Lenci and G.M. critically revised the manuscript.

Acknowledgments

We thank Alessandra Rampinini for critical discussions and the Unione Italiana Ciechi (National Association for the Blind) for their collaboration. This work was supported by Italian Grant "Semantic representations in the language of the blind: linguistic and neurocognitive studies" (PRIN 2008CM9MY3) and by Fondazione Cassa di Risparmio di Lucca (Lucca, Italy).

ACCEPTED MANUSCRIPT

References

Amedi, A., Stern, W.M., Camprodon, J.A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., Pascual-Leone, A., 2007. Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat Neurosci 10, 687-689.

Andersson, J.L., Jenkinson, M., Smith, S., 2007. Non-linear optimisation. FMRIB technical report TR07JA1. Practice. 2007a Jun.

Bailey, T.L., Gribskov, M., 1998. Combining evidence using p-values: application to sequence homology searches. Bioinformatics 14, 48-54.

Bedny, M., Pascual-Leone, A., Dodell-Feder, D., Fedorenko, E., Saxe, R., 2011. Language processing in the occipital cortex of congenitally blind adults. Proc Natl Acad Sci U S A 108, 4429-4434. Benjamini, Y., Hochberg, Y., 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 289300.

Binder, J.R., Desai, R.H., Graves, W.W., Conant, L.L., 2009. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb Cortex 19, 2767-2796. Bruffaerts, R., Dupont, P., Peeters, R., De Deyne, S., Storms, G., Vandenberghe, R., 2013. Similarity of fMRI activity patterns in left perirhinal cortex reflects semantic similarity between words. J Neurosci 33, 18597-18607.

Caramazza, A., Mahon, B.Z., 2003. The organization of conceptual knowledge: the evidence from category-specific semantic deficits. Trends Cogn Sci 7, 354-361.

Caramazza, A., Shelton, J.R., 1998. Domain-specific knowledge systems in the brain the animate-inanimate distinction. J Cogn Neurosci 10, 1-34.

Carlson, T.A., Ritchie, J.B., Kriegeskorte, N., Durvasula, S., Ma, J., 2014. Reaction time for object categorization is predicted by representational distance. J Cogn Neurosci 26, 132-142.

Chang, K.M., Mitchell, T., Just, M.A., 2011. Quantitative modeling of the neural representation of objects: how semantic feature norms can account for fMRI activation. Neuroimage 56, 716-727. Connolly, A.C., Gleitman, L.R., Thompson-Schill, S.L., 2007. Effect of congenital blindness on the semantic representation of some everyday concepts. Proc Natl Acad Sci U S A 104, 8241-8246. Connolly, A.C., Guntupalli, J.S., Gors, J., Hanke, M., Halchenko, Y.O., Wu, Y.C., Abdi, H., Haxby, J.V., 2012. The representation of biological classes in the human brain. J Neurosci 32, 2608-2618. Cox, R.W., 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29, 162-173.

Cukur, T., Nishimoto, S., Huth, A.G., Gallant, J.L., 2013. Attention during natural vision warps

semantic representation across the human brain. Nat Neurosci 16, 763-770.

Devereux, B.J., Clarke, A., Marouchos, A., Tyler, L.K., 2013. Representational similarity analysis

reveals commonalities and differences in the semantic processing of words and objects. J Neurosci 33,

18906-18916.

Devlin, J.T., Russell, R.P., Davis, M.H., Price, C.J., Moss, H.E., Fadili, M.J., Tyler, L.K., 2002. Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI. Neuropsychologia 40, 54-75.

Eickhoff, S.B., Paus, T., Caspers, S., Grosbras, M.H., Evans, A.C., Zilles, K., Amunts, K., 2007. Assignment of functional activations to probabilistic cytoarchitectonic areas revisited. Neuroimage 36, 511-521.

Ejaz, N., Hamada, M., Diedrichsen, J., 2015. Hand use predicts the structure of representations in sensorimotor cortex. Nat Neurosci 18, 1034-1040.

Epstein, R.A., Ward, E.J., 2010. How reliable are visual context effects in the parahippocampal place area? Cereb Cortex 20, 294-303.

Fairhall, S.L., Caramazza, A., 2013. Category-selective neural substrates for person- and place-related concepts. Cortex 49, 2748-2757.

Fedorenko, E., Thompson-Schill, S.L., 2014. Reworking the language network. Trends Cogn Sci 18, 120-126.

Fernandino, L., Binder, J.R., Desai, R.H., Pendl, S.L., Humphries, C.J., Gross, W.L., Conant, L.L., Seidenberg, M.S., 2015. Concept Representation Reflects Multimodal Abstraction: A Framework for Embodied Semantics. Cereb Cortex.

Fogassi, L., Ferrari, P.F., Gesierich, B., Rozzi, S., Chersi, F., Rizzolatti, G., 2005. Parietal lobe: from action organization to intention understanding. Science 308, 662-667.

Gainotti, G., 2010. The influence of anatomical locus of lesion and of gender-related familiarity factors in category-specific semantic disorders for animals, fruits and vegetables: a review of single-case studies. Cortex 46, 1072-1087.

Grossman, M., Peelle, J.E., Smith, E.E., McMillan, C.T., Cook, P., Powers, J., Dreyfuss, M., Bonner, M.F., Richmond, L., Boller, A., Camp, E., Burkholder, L., 2013. Category-specific semantic memory: converging evidence from bold fMRI and Alzheimer's disease. Neuroimage 68, 263-274. Handjaras, G., Bernardi, G., Benuzzi, F., Nichelli, P.F., Pietrini, P., Ricciardi, E., 2015. A topographical organization for action representation in the human brain. Hum Brain Mapp 36, 38323844.

Haxby, J.V., Connolly, A.C., Guntupalli, J.S., 2014. Decoding neural representational spaces using multivariate pattern analysis. Annu Rev Neurosci 37, 435-456.

Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P., 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425-2430. Huth, A.G., Nishimoto, S., Vu, A.T., Gallant, J.L., 2012. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 12101224.

Just, M.A., Cherkassky, V.L., Aryal, S., Mitchell, T.M., 2010. A neurosemantic theory of concrete noun representation based on the underlying brain codes. PLoS One 5, e8622.

Kim, J.K., Zatorre, R.J., 2011. Tactile-auditory shape learning engages the lateral occipital complex. J Neurosci 31, 7848-7856.

Kriegeskorte, N., Kievit, R.A., 2013. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn Sci 17, 401-412.

Kriegeskorte, N., Mur, M., Ruff, D.A., Kiani, R., Bodurka, J., Esteky, H., Tanaka, K., Bandettini, P.A., 2008. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60, 1126-1141.

Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S., Baker, C.I., 2009. Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12, 535-540.

Kruskal, J.B., Wish, M., 1978. Multidimensional scaling. Sage Publications, Beverly Hills, Calif. Landau, B., Gleitman, L.R., 1985. Language and experience : evidence from the blind child. Harvard University Press, Cambridge, Mass.

Lane, C., Kanjlia, S., Omaki, A., Bedny, M., 2015. "Visual" Cortex of Congenitally Blind Adults Responds to Syntactic Movement. J Neurosci 35, 12859-12868.

Lenci, A., Baroni, M., Cazzolli, G., Marotta, G., 2013. BLIND: a set of semantic feature norms from the congenitally blind. Behav Res Methods 45, 1218-1233.

Levy, D.A., Bayley, P.J., Squire, L.R., 2004. The anatomy of semantic knowledge: medial vs. lateral temporal lobe. Proc Natl Acad Sci U S A 101, 6710-6715.

Mahon, B.Z., Anzellotti, S., Schwarzbach, J., Zampini, M., Caramazza, A., 2009. Category-specific organization in the human brain does not require visual experience. Neuron 63, 397-405. Mahon, B.Z., Schwarzbach, J., Caramazza, A., 2010. The representation of tools in left parietal cortex is independent of visual experience. Psychol Sci 21, 764-771.

McRae, K., Cree, G.S., Seidenberg, M.S., McNorgan, C., 2005. Semantic feature production norms for a large set of living and nonliving things. Behav Res Methods 37, 547-559.

Mitchell, T.M., Hutchinson, R., Niculescu, R.S., Pereira, F., Wang, X., Just, M., Newman, S., 2004. Learning to decode cognitive states from brain images. Machine Learning 57, 145-175. Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A., 2008. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191-1195. Moss, H.E., Tyler, L.K., Devlin, J.T., 2002. The emergence of category-specific deficits in a distributed semantic system. Category-specificity in brain and mind, 115-148.

Mumford, J.A., Davis, T., Poldrack, R.A., 2014. The impact of study design on pattern estimation for single-trial multivariate pattern analysis. Neuroimage 103, 130-138.

Naselaris, T., Kay, K.N., Nishimoto, S., Gallant, J.L., 2011. Encoding and decoding in fMRI. Neuroimage 56, 400-410.

Noppeney, U., Friston, K.J., Price, C.J., 2003. Effects of visual deprivation on the organization of the semantic system. Brain 126, 1620-1627.

Park, S., Brady, T.F., Greene, M.R., Oliva, A., 2011. Disentangling scene content from spatial boundary: complementary roles for the parahippocampal place area and lateral occipital complex in representing real-world scenes. J Neurosci 31, 1333-1340.

Peelen, M.V., Bracci, S., Lu, X., He, C., Caramazza, A., Bi, Y., 2013. Tool selectivity in left occipitotemporal cortex develops without vision. J Cogn Neurosci 25, 1225-1234. Peelen, M.V., He, C., Han, Z., Caramazza, A., Bi, Y., 2014. Nonvisual and visual object shape representations in occipitotemporal cortex: evidence from congenitally blind and sighted adults. J Neurosci 34, 163-170.

Pereira, F., Botvinick, M., Detre, G., 2013. Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. Artif Intell 194, 240-252. Pereira, F., Mitchell, T., Botvinick, M., 2009. Machine learning classifiers and fMRI: a tutorial overview. Neuroimage 45, S199-209.

Pietrini, P., Furey, M.L., Ricciardi, E., Gobbini, M.I., Wu, W.H., Cohen, L., Guazzelli, M., Haxby, J.V., 2004. Beyond sensory images: Object-based representation in the human ventral pathway. Proc Natl Acad Sci U S A 101, 5658-5663.

Price, C.J., 2012. A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading. Neuroimage 62, 816-847.

Pulvermuller, F., Shtyrov, Y., Hauk, O., 2009. Understanding in an instant: neurophysiological evidence for mechanistic language circuits in the brain. Brain Lang 110, 81-94. Reddy, L., Tsuchiya, N., Serre, T., 2010. Reading the mind's eye: decoding category information during mental imagery. Neuroimage 50, 818-825.

Renier, L.A., Anurova, I., De Volder, A.G., Carlson, S., VanMeter, J., Rauschecker, J.P., 2010. Preserved functional specialization for spatial processing in the middle occipital gyrus of the early blind. Neuron 68, 138-148.

Rice, G.E., Lambon Ralph, M.A., Hoffman, P., 2015. The Roles of Left Versus Right Anterior Temporal Lobes in Conceptual Knowledge: An ALE Meta-analysis of 97 Functional Neuroimaging Studies. Cereb Cortex.

Rogers, T.T., Patterson, K., 2007. Object categorization: reversals and explanations of the basic-level advantage. J Exp Psychol Gen 136, 451-469.

Simanova, I., Hagoort, P., Oostenveld, R., van Gerven, M.A., 2014. Modality-independent decoding of semantic information from the human brain. Cereb Cortex 24, 426-434.

Simmons, W.K., Hamann, S.B., Harenski, C.L., Hu, X.P., Barsalou, L.W., 2008. fMRI evidence for word association and situated simulation in conceptual processing. J Physiol Paris 102, 106-119. Smith, S.M., Jenkinson, M., Woolrich, M.W., Beckmann, C.F., Behrens, T.E., Johansen-Berg, H., Bannister, P.R., De Luca, M., Drobnjak, I., Flitney, D.E., Niazy, R.K., Saunders, J., Vickers, J., Zhang, Y., De Stefano, N., Brady, J.M., Matthews, P.M., 2004. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23 Suppl 1, S208-219.

Stoeckel, C., Gough, P.M., Watkins, K.E., Devlin, J.T., 2009. Supramarginal gyrus involvement in visual word recognition. Cortex 45, 1091-1096.

Striem-Amit, E., Ovadia-Caro, S., Caramazza, A., Margulies, D.S., Villringer, A., Amedi, A., 2015. Functional connectivity of visual cortex in the blind follows retinotopic organization principles. Brain 138, 1679-1695.

Strnad, L., Peelen, M.V., Bedny, M., Caramazza, A., 2013. Multivoxel pattern analysis reveals auditory motion information in MT+ of both congenitally blind and sighted individuals. PLoS One 8, e63198. Sudre, G., Pomerleau, D., Palatucci, M., Wehbe, L., Fyshe, A., Salmelin, R., Mitchell, T., 2012. Tracking neural coding of perceptual and semantic features of concrete nouns. Neuroimage 62, 451463.

Vann, S.D., Aggleton, J.P., Maguire, E.A., 2009. What does the retrosplenial cortex do? Nat Rev Neurosci 10, 792-802.

Wang, X., Peelen, M.V., Han, Z., He, C., Caramazza, A., Bi, Y., 2015. How Visual Is the Visual Cortex? Comparing Connectional and Functional Fingerprints between Congenitally Blind and Sighted Individuals. J Neurosci 35, 12545-12559.

Warrington, E.K., McCarthy, R.A., 1987. Categories of knowledge. Further fractionations and an attempted integration. Brain 110 ( Pt 5), 1273-1296.

Warrington, E.K., Shallice, T., 1984. Category specific semantic impairments. Brain 107 ( Pt 3), 829854.

Captions

Figure 1. The fMRI task. Picture represents the experimental paradigm across presentation formats.

Figure 2. Multidimensional scaling of behavioral and brain functional data. Picture shows in Panel A, on the left, the scaling of the behavioral representation across the blind and the sighted experiment and in Panel B, on the right, the scaling generated from brain activity across presentation formats and individuals related to left Posterior Semantic Network at the large-scale level.

Figure 3. Behavioral results. Picture shows the category-based representational spaces in the blind and the sighted behavioral experiment and their correlation (r=0.96, p < 0.0001).

Figure 4. Regions derived from probability maps. The small-scale level in panel A comprises: in left hemisphere, the Parahippocampal (PH), the Lateral Occipital (LO), the Inferior Parietal (IP) regions and the bilateral Temporo-Parieto-Occipital (TPO) junction and the RetroSplenial Cortex (RSC). The large-scale level in panel B comprises: two large posterior semantic networks (left and right PSN), an anterior semantic network (left ASN) and the bilateral Parieto Occipital (PO) cortex.

Figure 5. Correlations between presentation modalities at the small-scale and large-scale levels.

The representational spaces generated from neural activity and from behavioral data were correlated to each other (p<0.05, Bonferroni-Holm corrected) within the regions extracted from the small-scale (Panel A) and large-scale levels (Panel B). White squares indicate Non-Significant (NS) correlations. BD: Behavioral Data based on representational spaces in the blind and the sighted linguistic data; Sp: Sighted pictorial form; Sv: Sighted verbal visual form; Sa: Sighted verbal auditory form; Ba: Blind verbal auditory form. For regions labels please refer to the Figure 4.

Figure 6. Rank accuracies of the semantic categories within presentation modalities and scale levels. Classification rank accuracies (p<0.05, FDR corrected) and category preference measures were estimated (p<0.05, Bonferroni-Holm corrected) within the regions extracted from the small-scale (Panel A) and large-scale levels (Panel B). White squares indicate Non-Significant (NS) accuracies. Sp: Sighted pictorial form; Sv: Sighted verbal visual form; Sa: Sighted verbal auditory form; Ba: Blind verbal auditory form. For regions labels please refer to Figure 4.

Figure 7. Multidimensional scaling in the left Posterior Semantic Network. Picture shows the scaling generated from brain activity in the left Posterior Semantic Network for each presentation modality separately, along with their average (filled circle) and the behavioral representation (empty circle). Sp: Sighted pictorial form; Sv: Sighted verbal visual form; Sa: Sighted verbal auditory form; Ba: Blind verbal auditory form.

Pictorial Visual Form (Sighted Subjects)

Verbal Visual Form (Sighted Subjects)

Verbal Auditory Form (Blind and Sighted)

pineapple

Stimulus

giraffe

Property Generation

Stimulus

3s 7s 3s 7s

Property Generation

3s 7s 3s 7s

1s 9s 1s 9s

Figure 1

Mammals

Birds Fruits

Vegetables

Tools Vehicles Artificial Places Natural Places

Figure 2

— "m CD TOW .St" C >- (D O (D £

Vegetables

Natural Places

Vegetables

Figure 3

A left IP

Figure 4

left TPO

Sp Sv Sa Ba

right TPO

Sp Sv Sa Ba

BD Sp Sv Sa

BD Sp Sv Sa

left RSC

Sp Sv Sa Ba

right RSC

Sp Sv Sa Ba

BD Sp Sv Sa

BD Sp Sv Sa

left LO

Sp Sv Sa Ba

BD Sp Sv Sa

left PH

Sp Sv Sa Ba

BD Sp Sv Sa

left IP

Sp Sv Sa Ba

BD Sp Sv Sa

CalS/HG

Sp Sv Sa Ba

BD Sp Sv Sa

left PSN

Sp Sv Sa Ba

BD Sp Sv Sa

Pearson's Correlation Coefficient (r)

right PSN

left AS N

Sp Sv Sa Ba

BD Sp Sv Sa

BD Sp Sv Sa

NS^^M__11

Pearson's Correlation Coefficient (r)

Sp Sv Sa Ba

BD Sp Sv Sa

Figure 5

left TPO

Sp Sv Sa Ba

Fruits

Vegetables

Natural Places

Artificial Places

right TPO

Sp Sv Sa Ba

Vegetables

Natural Places

Artificial I Places I

Mammals

left RSC

Sp Sv Sa Ba

right RSC

Sp Sv Sa Ba

left LO

Sp Sv Sa Ba

Fruits

Vegetables

Vehicles

left PH

Sp Sv Sa Ba

Vegetables

Natural Places

Artificial Places

Birds Tools Vehicles

left IP

Sp Sv Sa Ba

Sp Sv Sa

Sp Sv Sa Ba

Vegetables

Natural Places

Artificial Places

Birds Tools Vehicles

Rank Accuracy (%) Category Preference

left PSN

Sp Sv Sa Ba

right PSN

Sp Sv Sa Ba

left ASN

Sp Sv Sa Ba

Fruits

Vegetables

Natural

Places

Artificial Places

Sp Sv Sa Ba

Fruits Vegetables

Natural Places

Artificial Places

Birds Tools Vehicles

Rank Accuracy (%) Category Preference

Figure 6

# Brain Data

O Behavioral Data ß

îp3 5Va

Sv Bail

SaBa s Sv

Mammals Sv ® O

Birds q ^a

Fruits » f

Vegetables Sp 0 ,

Tools ° Sa

Vehicles f

Artificial Places SP

Natural Places

Figure 7