Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Striatal and hippocampal contributions to flexible navigation in rats and humans:

Striatal and hippocampal contributions to flexible navigation in rats and humans: The hippocampus has been firmly established as playing a crucial role in flexible navigation. Recent evidence suggests that dorsal striatum may also play an important role in such goal-directed behaviour in both rodents and humans. Across recent studies, activity in the caudate nucleus has been linked to forward planning and adaptation to changes in the environment. In particular, several human neuroimaging studies have found the caudate nucleus tracks information traditionally associated with that by the hippocampus. In this brief review, we examine this evidence and argue the dorsal striatum encodes the transition structure of the environment during flexible, goal-directed behaviour. We highlight that future research should explore the following: (1) Investigate neural responses during spatial navigation via a biophysically plausible framework explained by reinforcement learning models and (2) Observe the interaction between cortical areas and both the dorsal striatum and hippocampus during flexible navigation. Keywords Spatial navigation, dorsal striatum, hippocampus, flexible behaviour, goals, reinforcement learning, wayfinding Received: 12 August 2020; accepted: 16 November 2020 to intact function of these regions during spatial navigation Flexibility during goal-directed (Andersen et al., 2006; White and Donald, 2002). ‘Place learning’ behaviour is a flexible process by which an animal learns associations between distal cues and goal locations in the environment, while Flexible adaptation in response to unexpected changes in the response learning is an inflexible process whereby an animal learns environment is a central challenge in navigation. Tolman et al. a series of actions or responses necessary to reach the goal. Place (1946) adeptly illustrated this in his seminal work exploring the learning can be investigated using the Morris water maze, a task capacity of rodents to accommodate detours and adopt shortcuts that targets behavioural flexibility and spatial memory (Devan and in complex mazes. This work led to the proposal of the cognitive White, 1999; McDonald and White, 1994; Morris et al., 1982; map hypothesis for flexible behaviour, by which the brain con- Pearce et al., 1998; Whishaw et al., 1987). By the original task structs an internal representation of the environment to support protocol, a rat is placed at a pseudo-random location within a cylin- navigation (Tolman, 1948). Subsequent neuroscientific research drical arena filled with opaque water. No local cues other than dis- led O’Keefe and Nadel (1978) to propose the hippocampus is tal landmarks and boundary distance are provided. Safety is primarily responsible for supporting this cognitive map. Particularly central to this proposal is the existence of ‘place cells’ in the hippocampus that show spatially localised activity patterns linked to boundaries and landmarks in an environment Institute of Behavioural Neuroscience, Department of Experimental (O’Keefe and Dostrovsky, 1971). This was followed by the dis- Psychology, Division of Psychology and Language Sciences, University College London, London, UK covery of a variety of other spatial coding cells supporting navi- gation (see Grieves and Jeffery, 2017 for review). Given the Corresponding authors: ubiquity of spatial representation in the hippocampus and neigh- Christoffer J. Gahnstrom, Institute of Behavioural Neuroscience, bouring parahippocampal structures, several essential questions Department of Experimental Psychology, Division of Psychology and arise: (1) How is information used during flexible navigation, as Language Sciences, Institute of Behavioural Neuroscience, University suggested by the hypothesis of the cognitive map? (2) What College London, London WC1N 0AP, UK. Email: c.gahnstrom@ucl.ac.uk information does the hippocampus transmit to downstream regions during navigation? (3) What contributions might other Hugo J. Spiers, Institute of Behavioural Neuroscience, Department regions of the brain’s navigation systems, such as the dorsal stria- of Experimental Psychology, Division of Psychology and Language tum, have for flexible navigation? Sciences, Institute of Behavioural Neuroscience, University College Rodent studies lesioning dorsal striatum and hippocampus pro- London, London WC1N 0AP, UK. vide strong evidence for dissociable behavioural strategies related Email: h.spiers@ucl.ac.uk Creative Commons CC BY: This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage). 2 Brain and Neuroscience Advances achieved by swimming to a fixed platform located just below the Howard et al. (2014) identified neural correlates of path distance opaque surface, hidden from view. Escape latencies record time to to goal in the right posterior hippocampus. Such correlates of dis- reach the platform during training as well as during probe trials tance to goal have also been observed in dorsal hippocampal (when the hidden platform is removed). Lesion or inactivation of recordings in rats (Spiers et al., 2018) and bats (Sarel et al., the hippocampus impacts place learning by increasing escape 2017). During detour events, the human posterior right hip- latencies compared to that of non-lesioned controls (Morris et al., pocampus was also found to track the increase in path distance 1982, Moser et al., 1995; Sutherland et al., 1983). However, lesions when a forced detour occurred (Howard et al., 2014). Based on in dorsal striatum impair simple approach behaviour when the plat- this finding and other evidence from rats (e.g. Gupta et al., 2010; form is visible, and instead, rats will swim to previously learned Ólafsdóttir et al., 2015; Pfeiffer and Foster, 2013), it has been platform location (McDonald and White, 1994). hypothesised the hippocampus simulates future paths through the A paradigm called Delayed-Matched-to-Place further extended environment at key events during navigation, such as at detours the Morris water maze by investigating one-shot learning, a hall- (Spiers and Gilbert, 2015). Consequently, detours requiring sim- mark of behavioural flexibility (Steele and Morris, 1999). In this ulation of a much larger future route will evoke greater demands version of the task, the location of the hidden platform changes on the hippocampus than simulation of shorter routes. each day. This results in a substantial drop in escape latency In order to test the prediction of Spiers and Gilbert (2015), a between the first and second trials. The subsequent trials exhibit recent study by Javadi et al. (2019a) examined hippocampal latency improvement, but to a much smaller extent. This concept response to, respectively, small and large changes in distance to of one-shot learning is an impressive quality of cognitive flexibil- goal at forced detours (see Figure 1(a)). In this task, participants ity difficult to capture by biophysically plausible modelling of navigated a virtual desert island riven with lava which blocked place cells (Foster et al., 2000). However, reinforcement learning certain movements across it. Participants first learned the layout (RL) can capture this behavioural phenomenon by further simu- and location of several hidden objects, which were later pre- lating cells which estimate real world coordinates (Foster et al., sented as a goal to navigate to. During the test phase, when par- 2000; Tessereau et al., 2020). Together, these simulated cells form ticipants actively navigated the maze, shifts in the location of an allocentric coordinate system receiving input from the place lava pools either opened up new paths or blocked old paths, cells. This coordinate system lacks a biological basis, although resulting in possible shortcuts and detours, respectively. In con- this may be analogous to information represented by grid cells in trast to the predictions of Spiers and Gilbert (2015), posterior the entorhinal cortex (Hafting et al., 2005). Likewise, simulated hippocampus did not index the change in distance to goal at deep RL agents endowed with grid-like representation can per- detours, but rather prefrontal regions and bilateral caudate form flexible spatial navigation tasks such as the Morris water nucleus tracked the change in path distance to goal (Javadi et al., maze (Banino et al., 2018). In addition, bilateral lesions to the 2019). Notably, in Howard et al. (2014), the hippocampal fornix impairs performance in an eight-arm radial maze task, in response to distance changes at detours was also accompanied by which rats are trained to revisit certain arms consistently baited a similar response in the dorsal striatum (Figure 1(b)). Taken with food (Packard et al., 1989). Intact hippocampal function is together, these results indicate that the dorsal striatum is more necessary for place learning in a plus-maze task as well (Packard consistent in tracking the change in distance to goal at detours and McGaugh, 1996). Evidence from neuroimaging studies of than the hippocampus. This suggests it is timely to reconsider the humans and patients with hippocampal damage further implicates role of dorsal striatum during flexible navigation and understand the hippocampus in supporting both place learning and flexible how the hippocampus interacts with these regions in cortico-stri- navigation of novel routes and environments (Bohbot et al., 2007; atal loops (Brown et al., 2012; Goodroe et al., 2018). Hartley et al., 2003; Howard et al., 2014; Iaria et al., 2003; Javadi et al., 2019a, 2019b; Javadi et al., 2017; Patai et al., 2019; Spiers et al., 2001a, 2001b; Spiers and Maguire, 2006; Xu et al., 2010). How might the striatum contribute to In addition to place learning, animals also utilise ‘response flexible navigation behaviour? learning’, that is, learning based on the responses required to reach the goal (Packard and McGaugh, 1996). Such response Despite the traditional role of response learning attributed to stri- learning is shown to depend on the functional integrity of the dor- atal function, the striatum has been implicated in studies investi- sal striatum (Packard et al., 1989; Packard and McGaugh, 1996). gating behavioural flexibility in both rodents and humans, Subsequently, human neuroimaging research has provided con- suggesting a more nuanced functionality beyond contributing to vergent evidence for the involvement of the dorsal striatum in a less flexible response system (Johnson et al., 2007). Lesions such response strategy navigation (Hartley et al., 2003; Iaria et al., and inactivations in different areas of striatum produce varied 2003; Voermans et al., 2004). Response learning is not tradition- behavioural deficits, indicating a dissociation of respective func- ally considered flexible because it is tied to the specific features of tional roles (Ragozzino et al., 2002; Sharpe et al., 2019). The the environment (e.g. always turn right at the crossroad). By con- striatum is commonly divided up into two anatomically separated trast, place learning is thought to be flexible since it is possible to regions: the dorsal striatum, composing of the caudate and puta- use viewpoint-independent information from the environment to men, and the ventral striatum, composed mainly of the nucleus accommodate detours and identify shortcuts and because it does accumbens although no clear cytoarchitectonic or histochemical not rely on the presence of a single specific cue. boundary between ventral and dorsal striatum exists (Haber and Recent studies have begun to explore how different types of Knutson, 2010). Furthermore, the rodent caudate-putamen is seg- spatial information may be tracked by specific brain regions dur- mented into dorsomedial striatum (homologous to primate cau- ing navigation. Two important metrics for flexible navigation are date) and dorsolateral striatum (homologous to primate putamen) vector-to-goal and path-to-goal (Bicanski and Burgess, 2020; (Cox and Witten, 2019). Early rodent studies did not include Chadwick et al., 2015; Spiers and Barry, 2015). Using in situ strict separation of these regions when using large lesions, which learning experience and film simulation of Soho in London (UK), leads to interpretation difficulties (Yin and Knowlton, 2006). Gahnstrom and Spiers 3 Figure 1. Dorsal striatum activity is correlated with the change in distance to goal at detours. (a) Replotted data from Javadi et al. (2019a) in which fMRI and virtual reality desert island riven with lava was used to examine the brain regions responsive to the change in distance to the goal at detours. Top row shows a zoomed in schematic from the larger virtual environment used and the transition that occurs when the path is unexpectedly blocked. Bottom row: the same change but from the first person perspective. Brain image shows bilateral activity in medial caudate nucleus (dorsal striatum) cluster-corrected for activity correlated positively with the parametrically modulated change in distance-to-goal. (b) Replotted data from Howard et al. (2014). In this study, a film simulation of Soho in London was used to test navigation, including accommodating detours. The amount of change in distance-to-goal caused by forced detours was correlated with the dorsal striatal activity. Red regions show regions activations thresholded at p < 0.005 uncorrected, shown in the mean structural image. RL models provide a normative framework to investigate reward prediction errors (Daw et al., 2011; Gläscher et al., 2010). neural mechanisms that give rise to flexible and inflexible behav- Daw et al. (2011) also found the striatal underpinnings of habit- iour (Corrado et al., 2009). Within the RL literature, flexible and ual model-free prediction errors and model-based prediction goal-directed behaviour is often described by a family of algo- errors overlap in ventral striatum, suggesting the same neural cir- rithms classified as ‘model-based’. This is commonly contrasted cuitry is involved in both computations. A recent fMRI meta- with habitual behaviour described by a separate family of algo- analysis of multi-step decision making tasks found overlapping rithms classified as ‘model-free’ (Dolan and Dayan, 2013; Rusu regions involved in model-based and model-free computations in and Pennartz, 2020). These computational models ‘learn’ states globus pallidus and caudate nucleus (Huang et al., 2020). and rewards in the environment by using a component referred to Beyond the classic divisions of model-free and model-based as reward prediction errors, that is, the difference between literature in decision-making tasks, there are other families of RL expected and experienced reward. The goal of a RL agent is to algorithms that provide alternative accounts, including hierarchi- take actions which maximise future reward in the long run cal RL, linear RL, and successor representation (Botvinick et al., (Sutton and Barto, 2018). The canonical finding of reward pre- 2009; Dayan, 1993; Gershman, 2018; Piray and Daw, 2019; diction errors found encoding in single neurons of the ventral Russek et al., 2017; Stachenfeld et al., 2017; Tessereau et al., tegmental area in the brainstem of macaques (Schultz et al., 2020). In particular, successor representation can account for flex- 1997), a region which has direct dopaminergic projection to the ible behaviour of rats and humans in complex mazes (De Cothi nucleus accumbens in ventral striatum (Haber and Knutson, et al., 2020) and humans in reward devaluation protocols 2010). Since then, human functional magnetic resonance imag- (Momennejad et al., 2017). Interestingly, components of the suc- ing (fMRI) studies using multi-step decision making tasks have cessor representation during simulations show similarities to identified ventral striatum as a primary region for the process of properties of place cells and grid cells, including the influence of 4 Brain and Neuroscience Advances goal locations on place field over-representation observed in spe- overlapped with model-free correlates in the retrosplenial cortex. cific paradigms and influence of environmental geometry on grid In contrast to Simon and Daw (2011), this study did not utilise field integrity (Duvelle et al., 2019; Ekstrom et al., 2020; Krupic visual goal cues and also did not include changes in the maze et al., 2015; Stachenfeld et al., 2017). It is an interesting future configuration, more akin to classical spatial navigation para- direction for studies to investigate the relationship between neural digms. The different accounts of striatal involvement in predic- responses and the internal computations of successor representa- tion errors can perhaps be reconciled by considering that the tion shown to account for behaviour flexibility particularly in behavioural strategies and neural mechanisms are not as easily some spatial navigation tasks (Russek et al., 2017; for review see dissociable as previously thought. One spatial planning task Momennejad, 2020). Recent work with rats navigating between found striatal activity related to the difference in path distance four interconnected rooms has revealed that during initial adapta- between the shortest path and unchosen longest path to goal as a tion to pathways being obstructed place cells in CA1 do not adapt proxy for exhaustive search or forward planning (Kaplan et al., their firing fields to accompany the changing behaviour (Duvelle 2017). This indicates that striatal subregions may be involved in et al., 2020) as might have been predicted by a model in which planning, which may be the reason these regions are active in place cells support SR coding (Stachenfeld et al., 2017). It may be different studies. Perhaps a mixed use of strategies is also an that more stereotyped trajectories would lead to shifts in the place underlying reason for this result. Brown et al. (2012) showed that fields as a result of topological manipulations. caudate is important for disambiguating context during spatial The dorsal striatum has commonly been linked to stimulus– navigation, together with orbitofrontal cortex and hippocampus. response association, or habits, in spatial navigation tasks using We suggest these findings are in line with a new perspective of human fMRI. Doeller et al. (2008) employed a virtual object- these regions. In this view, the caudate encodes learned transition memory task inspired by the Morris water maze. They found structures. However, the current active transition structure at any activity in caudate nucleus to be parametrically modulated by the point in time is based on the current state of the animal and con- influence of intramaze landmarks on goal locations, while right text within the task, which is proposed to be modulated through posterior hippocampus correlated with boundary-related influ- cholinergic interneurons in dorsomedial striatum whose task- ence on goal locations (Doeller et al., 2008). In another study in dependent state information relies on an intact orbitofrontal cortex which participants navigated a virtual town, caudate activity was (Sharpe et al., 2019; Stalnaker et al., 2016). Hippocampus, on the preferentially active during route following trials, while anterior other hand, is involved in learning the structure of the environ- hippocampus was preferentially active during wayfinding trials ment (incidental to the task), and also the accompanying associa- (Hartley et al., 2003). Likewise, Iaria et al. (2003) found place tion-based learning. strategy use in an eight-arm radial maze task was associated with Instrumental learning paradigms in rodents reveal a model- increased right hippocampal activity while non-spatial response based influence on model-free prediction errors (Langdon et al., strategy use was associated with increased activity in caudate 2018). As such, the classical role of dopaminergic prediction nucleus. These studies suggest a dissociation between the roles of errors are more nuanced and can incorporate signals related to dorsal striatum and hippocampus for habitual and flexible behav- behavioural flexibility and the current state of the task in ventral iour, respectively. However, contextual demands may elucidate a tegmental area (Keiflin et al., 2019; Starkweather et al., 2017) as more nuanced role for the striatum in multiple behavioural con- well as dorsomedial striatum (Stalnaker et al., 2016). Using trol circuits (Balleine et al., 2015; Ferbinteanu, 2019; Rusu and causal methodology by optogenetically stimulating dopaminer- Pennartz, 2020; Woolley et al., 2015). gic neurons in ventral tegmental area (the putative cells encod- In rodents, the involvement of dorsal striatum in both flexible ing reward prediction errors), rats could learn associations and habitual behaviour could be resolved by considering the between cues without endowing them with cached-value, as functional distinction of dorsolateral and dorsomedial regions would be the expected based on pure model-free temporal-dif- (Gasser et al., 2020; Regier et al., 2015; Thorn et al., 2010; Van ference learning models (Sharpe et al., 2020). Another instru- Der Meer et al., 2010). Studies investigating the homologous mental learning task found an increasing number of neurons regions in humans are made difficult by the lack of spatially encoding task-relevant information in dorsolateral striatum precise recordings of neuronal activity. One account suggests more so than dorsomedial, suggesting the former may be encod- dorsal striatum performs the role of an ‘actor’, while ventral ing the development of a habit-based response (Kimchi et al., striatum performs the parallel role of a ‘critic’ in the ‘actor- 2009). Recordings in rats navigating a T-maze found that neu- critic’ RL framework (Sutton and Barto, 2018). In support of this rons in dorsomedial striatum were primarily active while choos- idea, such a division in computational roles was found during an ing between alternative actions after cue-onset, in contrast with instrumental learning task using fMRI (O’Doherty et al., 2004). neurons in dorsolateral striatum which were primarily active Investigation of functional distinction in dorsal striatum found during action execution (Stalnaker et al., 2016; Thorn et al., putamen involvement in habit-based processing from extensive 2010). found that cholinergic interneurons in rodent dorsome- training versus caudate involvement in forward planning dial—and not dorsolateral striatum—represented information (Wunderlich et al., 2012). The role of forward planning at about the current state of the choice task. In addition, this state detours could be considered in the task by Javadi et al. (2019a) information was not present in rats with lesions to the orbito- wherein distance changes were tracked by bilateral caudate frontal cortex. Taken together, there appears to be shared neural nucleus (Figure 1). In a virtual navigation task, Simon and Daw circuitry for model-free and model-based behaviours, and pre- (2011) also found forward planning tracked by striatum using diction errors may convey more information than the difference predictions from ‘model-based’ RL. between experienced and expected reward (Doll et al., 2012). In a more recent virtual navigation task, Anggraini et al. Perhaps the aforementioned human studies can be reconciled (2018) identified model-free correlates in dorsal striatum. Model- with the notion that caudate can support a mixture of model-free based correlates were found in the parahippocampus and and model-based computations depending on the task and Gahnstrom and Spiers 5 context at hand. Caudate nucleus activity can be expected in structure of the environment (see Momennejad, 2020). The response to changes in transition structure if it also encodes entorhinal cortex has also been proposed to play a role in cod- model-based information regarding the task environment. ing the transition structure of the layout of the environment or These recent findings pose a new question: What is the stimulus set (Behrens et al., 2018). Understanding how such a human dorsal striatum coding that drives these observed changes code relates to striatal coding of transition structure would be in activity during navigation? Rodent work on dorsomedial stri- useful for advancing models of the neural systems supporting atum suggests this region is necessary for execution of flexible flexible navigation behaviour. goal-directed behaviour (Rusu and Pennartz, 2020). Similarly, dorsomedial lesions have demonstrated similar behavioural def- Acknowledgements icits to that of hippocampal lesions in terms of deficiencies in We thank Sarah Goodroe for helpful comments on the draft and for help goal-directed flexible behaviour (Sharpe et al., 2019). For effec- with the figure together with Zita Patai. We also thank Eleonore Duvelle tive flexible behaviour, Sharpe et al. (2019) suggests hippocam- for comments on the manuscript. pus provides information about the environmental structure, while dorsomedial striatum incorporates information about the Declaration of Conflicting Interests transition structure into one’s overall world model. In human The author(s) declared no potential conflicts of interest with respect to navigation, novel forced detours are a classic example of a the research, authorship, and/or publication of this article. change in the transition structure. If the caudate updates repre- sentations of the transition structure, with greater transitional Funding change resulting in greater demand on caudate activity, then this The author(s) disclosed receipt of the following financial support for the may explain the results of both Javadi et al. (2019a) and Howard research, authorship, and/or publication of this article: The authors are et al. (2014), see Figure 1, where the larger the change in dis- supported by the European Union’s Horizon 2020 Framework Programme tance at detours the greater the caudate activity evoked. By con- for Research under Marie Sklodowska-Curie ITN (EU-M-GATE 765549) trast, hippocampus may be required to construct simulations of and ESRC grant to HJS. journeys through the environments (Bendor and Spiers, 2016). Such simulations may have been much richer in the navigation ORCID iD of London’s Soho (Howard et al., 2014), compared with a desert Christoffer J. Gahnstrom https://orcid.org/0000-0002-9989-5136 island (Javadi et al., 2019a), explaining the difference in hip- pocampal engagement. References Entorhinal cortex may also be involved in representing low- dimensional features of environments by extracting basis sets (or Andersen P, Morris R, Amaral D, et al. (eds) (2006) The Hippocampus eigenvectors of the successor representation), some of which Book. Oxford: Oxford University Press. look visually similar to the iconic hexagonal nature of grid fields Anggraini D, Glasauer S and Wunderlich K (2018) Neural signatures of reinforcement learning correlate with strategy adoption during spa- (Behrens et al., 2018; Stachenfeld et al., 2017). In RL, a ‘model’ tial navigation. Scientific Reports 8(1): 10110. of the environment is defined by P(s’|s, a), equal to the probabil- Balleine BW, Dezfouli A, Ito M, et al. (2015) Hierarchical control of ity of transitioning to a future state (s’) given a specific action (a) goal-directed action in the cortical–basal ganglia network. Current in the current state (s) (Sutton and Barto, 2018). Lesions during Opinion in Behavioral Sciences 5: 1–7. the Morris water maze have shown the entorhinal cortex to be Banino A, Barry C, Uria B, et al. (2018) Vector-based navigation using involved in flexible behaviour, as animals have similar behav- grid-like representations in artificial agents. Nature 557(7705): ioural deficits to those of hippocampal lesions in terms of 429–433. increased swimming latencies to the hidden platform (Hales Behrens TE, Muller TH, Whittington JC, et al. (2018) What is a cognitive et al., 2014). One idea is that the entorhinal cortex supports the map? Organizing knowledge for flexible behavior. Neuron 100(2): ability to form general transition structures of any environment 490–509. Bendor D and Spiers HJ (2016) Does the hippocampus map out the and store information about how distant states or locations are future? Trends in Cognitive Sciences 20(3): 167–169. related to each other (Behrens et al., 2018; Constantinescu et al., Bicanski A and Burgess N (2020) Neuronal vector coding in spatial cog- 2016). However, the unique dorsal striatum contribution may be nition. Nature Reviews Neuroscience 21(9): 453–470. more closely related to how action-outcome associations are rep- Bohbot VD, Lerch J, Thorndycraft B, et al. (2007) Gray matter differ- resented and which state is transitioned to as a result of a given ences correlate with spontaneous strategies in a human virtual navi- motor action (Sharpe et al., 2019). gation task. Journal of Neuroscience 27(38): 10078–10083. In conclusion, evidence suggests the dorsomedial striatum/ Botvinick MM, Niv Y and Barto AG (2009) Hierarchically organized caudate nucleus plays a key role in flexible navigation by repre- behavior and its neural foundations: A reinforcement learning per- senting the transition structure of the environment for guiding spective. Cognition 113(3): 262–280. future actions (Sharpe et al., 2019) and this may explain Brown TI, Ross RS, Tobyne SM, et al. (2012) Cooperative interactions between hippocampal and striatal systems support flexible naviga- observed responses at detours where transition structure tion. NeuroImage 60(2): 1316–1330. changes (Howard et al., 2014; Javadi et al., 2019a). Future Chadwick MJ, Jolly AEJ, Amos DP, et al. (2015) A goal direction sig- research will be useful to observe dorsomedial striatal activity nal in the human entorhinal/subicular region. Current Biology 25(1): in rodents during dynamic changes to the environment’s transi- 87–92. tion structure and variations in update demands (e.g. detours Constantinescu AO, OReilly JX and Behrens TEJ (2016) Organizing that require larger or smaller shifts in the route to the goal). It conceptual knowledge in humans with a gridlike code. Science would also be important to examine the interplay between the 352(6292): 1464–1468. striatum, hippocampal/parahippocampal structures, and pre- Corrado GS, Sugrue LP, Brown JR, et al. (2009) The trouble with choice: frontal cortex during such updating and representation for the Studying decision variables in the brain. In: Glimcher PW, Camerer 6 Brain and Neuroscience Advances CF, Fehr E, et al. (eds) Neuroeconomics: Decision Making and the Hartley T, Maguire EA, Spiers HJ, et al. (2003) The well-worn route and Brain. Amsterdam: Elsevier, pp. 463–480. the path less traveled: Distinct neural bases of route following and Cox J and Witten IB (2019) Striatal circuits for reward learning and deci- wayfinding in humans. Neuron 37(5): 877–888. sion-making. Nature Reviews Neuroscience 20(8): 482–494. Howard LR, Javadi AH, Yu Y, et al. (2014) The hippocampus and ento- Daw ND, Gershman SJ, Seymour B, et al. (2011) Model-based influ- rhinal cortex encode the path and euclidean distances to goals during ences on humans’ choices and striatal prediction errors. Neuron navigation. Current Biology 24(12): 1331–1340. 69(6): 1204–1215. Huang Y, Yaple ZA and Yu R (2020) Goal-oriented and habitual deci- Dayan P (1993) Improving generalization for temporal difference sions: Neural signatures of model-based and model-free learning. learning: The successor representation. Neural Computation 5(4): NeuroImage 215: 116834. 613–624. Iaria G, Petrides M, Dagher A, et al. (2003) Cognitive strategies depen- De Cothi W, Nyberg N, Griesbauer E-M, et al. (2020) Predictive maps in dent on the hippocampus and caudate nucleus in human naviga- rats and humans for spatial navigation. BioRxiv. Available at: https:// tion: Variability and change with practice. Journal of Neuroscience www.biorxiv.org/content/10.1101/2020.09.26.314815v2 23(13): 5945–5952. Devan BD and White NM (1999) Parallel information processing in the Javadi A-H, Emo B, Howard LR, et al. (2017) Hippocampal and prefron- dorsal striatum: Relation to hippocampal function. The Journal of tal processing of network topology to simulate the future. Nature Neuroscience 19(7): 2789–2798. Communications 8: 14652. Doeller CF, King JA and Burgess N (2008) Parallel striatal and hippo- Javadi A-H, Patai EZ, Marin-Garcia E, et al. (2019a) Backtracking dur- campal systems for landmarks and boundaries in spatial memory. ing navigation is correlated with enhanced anterior cingulate activ- Proceedings of the National Academy of Sciences of the United ity and suppression of alpha oscillations and the ‘default-mode’ States of America 105(15): 5915–5920. network. Proceedings of the Royal Society B: Biological Sciences Dolan RJ and Dayan P (2013) Goals and habits in the brain. Neuron 286(1908): 20191016. 80(2): 312–325. Javadi A-H, Patai EZ, Marin-Garcia E, et al. (2019b) Prefrontal dynamics Doll BB, Simon DA and Daw ND (2012) The ubiquity of model-based associated with efficient detours and shortcuts: A combined func- reinforcement learning. Current Opinion in Neurobiology 22(6): tional magnetic resonance imaging and magnetoencenphalography 1075–1081. study. Journal of Cognitive Neuroscience 31(8): 1227–1247. Duvelle E, Grieves RM, Hok V, et al. (2019) Insensitivity of place cells Johnson A, van der Meer MA and Redish AD (2007) Integrating hippo- to the value of spatial goals in a two-choice flexible navigation task. campus and striatum in decision-making. Current Opinion in Neuro- Journal of Neuroscience 39(13): 2522–2541. biology 17(6): 692–697. Duvelle E, Grieves RM, Liu A, et al. (2020) Hippocampal place cells Kaplan R, King J, Koster R, et al. (2017) The neural representation of encode global location but not changes in environmental connectiv- prospective choice during spatial planning and decisions. PLOS ity in a 4-room navigation task. BioRxiv. Available at: https://www. Biology 15(1): e1002588. biorxiv.org/content/10.1101/2020.10.20.346130v1 Keiflin R, Pribut HJ, Shah NB, et al. (2019) Ventral tegmental dopamine Ekstrom AD, Harootonian SK and Huffman DJ (2020) Grid coding, spa- neurons participate in reward identity predictions. Current Biology tial representation, and navigation: Should we assume an isomor- 29(1): 93–103e3. phism? Hippocampus 30(4): 422–432. Kimchi EY, Torregrossa MM, Taylor JR, et al. (2009) Neuronal corre- Ferbinteanu J (2019) Memory systems 2018 – Towards a new paradigm. lates of instrumental learning in the dorsal striatum. Journal of Neu- Neurobiology of Learning and Memory 157: 61–78. rophysiology 102(1): 475–489. Foster DJ, Morris RGM and Dayan P (2000) A model of hippocampally Krupic J, Bauza M, Burton S, et al. (2015) Grid cell symmetry is shaped dependent navigation, using the temporal difference learning rule. by environmental geometry. Nature 518(7538): 232–235. Hippocampus 10(1): 1–16. Langdon AJ, Sharpe MJ, Schoenbaum G, et al. (2018) Model-based Gasser J, de Vasconcelos AP, Cosquer B, et al. (2020) Shifting between predictions for dopamine. Current Opinion in Neurobiology 49: response and place strategies in maze navigation: Effects of training, 1–7. cue availability and functional inactivation of striatum or hippocam- McDonald RJ and White NM (1994) Parallel information processing in pus in rats. Neurobiology of Learning and Memory 167: 107131. the water maze: Evidence for independent memory systems involv- Gershman SJ (2018) The successor representation: Its computational logic ing dorsal striatum and hippocampus. Behavioral and Neural Biol- and neural substrates. Journal of Neuroscience 38(33): 7193–7200. ogy 61(3): 260–270. Gläscher J, Daw N, Dayan P, et al. (2010) States versus rewards: Dis- Momennejad I (2020) Learning structures: Predictive representations, sociable neural prediction error signals underlying model-based and replay, and generalization. Current Opinion in Behavioral Sciences model-free reinforcement learning. Neuron 66(4): 585–595. 32: 155–166. Goodroe SC, Starnes J and Brown TI (2018) The complex nature of Momennejad I, Russek EM, Cheong JH, et al. (2017) The successor rep- hippocampal-striatal interactions in spatial navigation. Frontiers in resentation in human reinforcement learning. Nature Human Behav- Human Neuroscience 12: 250. iour 1(9): 680–692. Grieves RM and Jeffery KJ (2017) The representation of space in the Moser MB, Moser EI, Forrest E, et al. (1995) Spatial learning with a brain. Behavioural Processes 135: 113–131. minislab in the dorsal hippocampus. Proceedings of the National Gupta AS, van der Meer MAA, Touretzky DS, et al. (2010) Hippocam- Academy of Sciences 92(21): 9697–9701. pal replay is not a simple function of experience. Neuron 65(5): Morris RG, Garrud P, Rawlins JA, et al. (1982) Place navigation impaired 695–705. in rats with hippocampal lesions. Nature 297(5868): 681–683. Haber SN and Knutson B (2010) The reward circuit: Linking primate O’Doherty J, Dayan P, Schultz J, et al. (2004) Dissociable roles of ventral anatomy and human imaging. Neuropsychopharmacology 35(1): and dorsal striatum in instrumental conditioning. Science 304(5669): 4–26. 452–454. Hafting T, Fyhn M, Molden S, et al. (2005) Microstructure of a spatial O’Keefe J and Dostrovsky J (1971) The hippocampus as a spatial map: map in the entorhinal cortex. Nature 436(7052): 801–806. Preliminary evidence from unit activity in the freely-moving rat. Hales JB, Schlesiger MI, Leutgeb JK, et al. (2014) Medial entorhinal Brain Research 34(1): 171–175. cortex lesions only partially disrupt hippocampal place cells and O’Keefe J and Nadel L (1978) The Hippocampus as a Cognitive Map. hippocampus-dependent place memory. Cell Reports 9(3): 893–901. Oxford: Clarendon Press. Gahnstrom and Spiers 7 Ólafsdóttir HF, Barry C, Saleem AB, et al. (2015) Hippocampal place Spiers HJ, Maguire EA and Burgess N (2001b) Hippocampal amnesia. cells construct reward related sequences through unexplored space. Neurocase 7(5): 357–382. eLife 4: e06063. Spiers HJ, Olafsdottir HF and Lever C (2018) Hippocampal CA1 activity Packard MG and McGaugh JL (1996) Inactivation of hippocampus or correlated with the distance to the goal and navigation performance. caudate nucleus with Lidocaine differentially affects expression of Hippocampus 28(9): 644–658. place and response learning. Neurobiology of Learning and Memory Stachenfeld KL, Botvinick MM and Gershman SJ (2017) The hip- 65(1): 65–72. pocampus as a predictive map. Nature Neuroscience 20(11): Packard MG, Hirsh R and White NM (1989) Differential effects of 1643–1653. fornix and caudate nucleus lesions on two radial maze tasks: Evi- Stalnaker TA, Berg B, Aujla N, et al. (2016) Cholinergic interneurons use dence for multiple memory systems. Journal of Neuroscience 9(5): orbitofrontal input to track beliefs about current state. The Journal of 1465–1472. Neuroscience 36(23): 6242–6257. Patai EZ, Javadi A-H, Ozubko JD, et al. (2019) Hippocampal and retro- Starkweather CK, Babayan BM, Uchida N, et al. (2017) Dopamine splenial goal distance coding after long-term consolidation of a real- reward prediction errors reflect hidden-state inference across time. world environment. Cerebral Cortex 29(6): 2748–2758. Nature Neuroscience 20(4): 581–589. Pearce JM, Roberts ADL and Good M (1998) Hippocampal lesions dis- Steele RJ and Morris RGM (1999) Delay-dependent impairment of a rupt navigation based on cognitive maps but not heading vectors. matching-to-place task with chronic and intrahippocampal infusion Nature 396(6706): 75–77. of the NMDA-antagonist D-AP5. Hippocampus 9(2): 118–136. Pfeiffer BE and Foster DJ (2013) Hippocampal place-cell sequences Sutherland RJ, Whishaw IQ and Kolb B (1983) A behavioural analysis depict future paths to remembered goals. Nature 497(7447): 74–79. of spatial localization following electrolytic, kainate- or colchicine- Piray P and Daw ND (2019) Linear reinforcement learning: Flexible induced damage to the hippocampal formation in the rat. Behav- reuse of computation in planning, grid fields, and cognitive con- ioural Brain Research 7(2): 133–153. trol. BioRxiv. Available at: https://www.biorxiv.org/content/10. Sutton RS and Barto AG (2018) Reinforcement Learning: An Introduc- 1101/856849v3 tion. Cambridge, MA: MIT Press. Ragozzino ME, Ragozzino KE, Mizumori SJ, et al. (2002) Role of the dor- Tessereau C, O’Dea R, Coombes S, et al. (2020) Reinforcement learning somedial striatum in behavioral flexibility for response and visual cue approaches to hippocampus-dependant flexible spatial navigation. discrimination learning. Behavioral Neuroscience 116(1): 105–115. BiorXiv. Available at: https://www.biorxiv.org/content/10.1101/20 Regier PS, Amemiya S and Redish AD (2015) Hippocampus and subre- 20.07.30.229005v2 gions of the dorsal striatum respond differently to a behavioral strat- Thorn CA, Atallah H, Howe M, et al. (2010) Differential dynamics of egy change on a spatial navigation task. Journal of Neurophysiology activity changes in dorsolateral and dorsomedial striatal loops during 114(3): 1399–1416. learning. Neuron 66(5): 781–795. Russek EM, Momennejad I, Botvinick MM, et al. (2017) Predictive rep- Tolman EC (1948) Cognitive maps in rats and men. Psychological resentations can link model-based reinforcement learning to model- Review 55(4): 189–208. free mechanisms. PLOS Computational Biology 13(9): e1005768. Tolman EC, Ritchie BF and Kalish D (1946) Studies in spatial learning. Rusu SI and Pennartz CMA (2020) Learning, memory and consolida- I. Orientation and the short-cut. Journal of Experimental Psychology tion mechanisms for behavioral control in hierarchically organized 36(1): 13–24. cortico-basal ganglia systems. Hippocampus 30(1): 73–98. Van Der Meer MAA, Johnson A, Schmitzer-Torbert NC, et al. (2010) Sarel A, Finkelstein A, Las L, et al. (2017) Vectorial representation Triple dissociation of information processing in dorsal striatum, ven- of spatial goals in the hippocampus of bats. Science 355(6321): tral striatum, and hippocampus on a learned spatial decision task. 176–180. Neuron 67(1): 25–32. Schultz W, Dayan P and Montague PR (1997) A neural substrate of Voermans NC, Petersson KM, Daudey L, et al. (2004) Interaction prediction and reward. Science 275(5306): 1593–1599. between the human hippocampus and the caudate nucleus during Sharpe MJ, Batchelor HM, Mueller LE, et al. (2020) Dopamine transients route recognition. Neuron 43(3): 427–435. do not act as model-free prediction errors during associative learn- Whishaw IQ, Mittleman G, Bunch ST, et al. (1987) Impairments in the ing. Nature Communications 11(1): 106. acquisition, retention and selection of spatial navigation strategies Sharpe MJ, Stalnaker T, Schuck NW, et al. (2019) An integrated model after medial caudate-putamen lesions in rats. Behavioural Brain of action selection: Distinct modes of cortical control of striatal deci- Research 24(2): 125–138. sion making. Annual Review of Psychology 70(1): 53–76. White NM and Donald RJ (2002) Multiple parallel memory systems in Simon DA and Daw ND (2011) Neural correlates of forward planning in the brain of the rat. Neurobiology of Learning and Memory 77(2): a spatial decision task in humans. Journal of Neuroscience 31(14): 125–184. 5526–5539. Woolley DG, Mantini D, Coxon JP, et al. (2015) Virtual water maze Spiers HJ and Barry C (2015) Neural systems supporting navigation. learning in human increases functional connectivity between poste- Current Opinion in Behavioral Sciences 1: 47–55. rior hippocampus and dorsal caudate. Human Brain Mapping 36(4): Spiers HJ and Gilbert SJ (2015) Solving the detour problem in naviga- 1265–1277. tion: A model of prefrontal and hippocampal interactions. Frontiers Wunderlich K, Dayan P and Dolan RJ (2012) Mapping value based plan- in Human Neuroscience 9: 125. ning and extensively trained choice in the human brain. Nature Neu- Spiers HJ and Maguire EA (2006) Thoughts, behaviour, and brain roscience 15(5): 786–791. dynamics during navigation in the real world. NeuroImage 31(4): Xu J, Evensmoen HR, Lehn H, et al. (2010) Persistent posterior and tran- 1826–1840. sient anterior medial temporal lobe activity during navigation. Neu- Spiers HJ, Burgess N, Hartley T, et al. (2001a) Bilateral hippocampal roImage 52(4): 1654–1666. pathology impairs topographical and episodic memory but not visual Yin HH and Knowlton BJ (2006) The role of the basal ganglia in habit pattern matching. Hippocampus 11(6): 715–725. formation. Nature Reviews Neuroscience 7(6): 464–476. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Brain and Neuroscience Advances SAGE

Striatal and hippocampal contributions to flexible navigation in rats and humans:

Loading next page...
 
/lp/sage/striatal-and-hippocampal-contributions-to-flexible-navigation-in-rats-scuZVGBAgV

References (195)

Publisher
SAGE
Copyright
Copyright © 2022 by SAGE Publications Ltd and British Neuroscience Association, unless otherwise noted. Manuscript content on this site is licensed under Creative Commons Licenses
ISSN
2398-2128
eISSN
2398-2128
DOI
10.1177/2398212820979772
Publisher site
See Article on Publisher Site

Abstract

The hippocampus has been firmly established as playing a crucial role in flexible navigation. Recent evidence suggests that dorsal striatum may also play an important role in such goal-directed behaviour in both rodents and humans. Across recent studies, activity in the caudate nucleus has been linked to forward planning and adaptation to changes in the environment. In particular, several human neuroimaging studies have found the caudate nucleus tracks information traditionally associated with that by the hippocampus. In this brief review, we examine this evidence and argue the dorsal striatum encodes the transition structure of the environment during flexible, goal-directed behaviour. We highlight that future research should explore the following: (1) Investigate neural responses during spatial navigation via a biophysically plausible framework explained by reinforcement learning models and (2) Observe the interaction between cortical areas and both the dorsal striatum and hippocampus during flexible navigation. Keywords Spatial navigation, dorsal striatum, hippocampus, flexible behaviour, goals, reinforcement learning, wayfinding Received: 12 August 2020; accepted: 16 November 2020 to intact function of these regions during spatial navigation Flexibility during goal-directed (Andersen et al., 2006; White and Donald, 2002). ‘Place learning’ behaviour is a flexible process by which an animal learns associations between distal cues and goal locations in the environment, while Flexible adaptation in response to unexpected changes in the response learning is an inflexible process whereby an animal learns environment is a central challenge in navigation. Tolman et al. a series of actions or responses necessary to reach the goal. Place (1946) adeptly illustrated this in his seminal work exploring the learning can be investigated using the Morris water maze, a task capacity of rodents to accommodate detours and adopt shortcuts that targets behavioural flexibility and spatial memory (Devan and in complex mazes. This work led to the proposal of the cognitive White, 1999; McDonald and White, 1994; Morris et al., 1982; map hypothesis for flexible behaviour, by which the brain con- Pearce et al., 1998; Whishaw et al., 1987). By the original task structs an internal representation of the environment to support protocol, a rat is placed at a pseudo-random location within a cylin- navigation (Tolman, 1948). Subsequent neuroscientific research drical arena filled with opaque water. No local cues other than dis- led O’Keefe and Nadel (1978) to propose the hippocampus is tal landmarks and boundary distance are provided. Safety is primarily responsible for supporting this cognitive map. Particularly central to this proposal is the existence of ‘place cells’ in the hippocampus that show spatially localised activity patterns linked to boundaries and landmarks in an environment Institute of Behavioural Neuroscience, Department of Experimental (O’Keefe and Dostrovsky, 1971). This was followed by the dis- Psychology, Division of Psychology and Language Sciences, University College London, London, UK covery of a variety of other spatial coding cells supporting navi- gation (see Grieves and Jeffery, 2017 for review). Given the Corresponding authors: ubiquity of spatial representation in the hippocampus and neigh- Christoffer J. Gahnstrom, Institute of Behavioural Neuroscience, bouring parahippocampal structures, several essential questions Department of Experimental Psychology, Division of Psychology and arise: (1) How is information used during flexible navigation, as Language Sciences, Institute of Behavioural Neuroscience, University suggested by the hypothesis of the cognitive map? (2) What College London, London WC1N 0AP, UK. Email: c.gahnstrom@ucl.ac.uk information does the hippocampus transmit to downstream regions during navigation? (3) What contributions might other Hugo J. Spiers, Institute of Behavioural Neuroscience, Department regions of the brain’s navigation systems, such as the dorsal stria- of Experimental Psychology, Division of Psychology and Language tum, have for flexible navigation? Sciences, Institute of Behavioural Neuroscience, University College Rodent studies lesioning dorsal striatum and hippocampus pro- London, London WC1N 0AP, UK. vide strong evidence for dissociable behavioural strategies related Email: h.spiers@ucl.ac.uk Creative Commons CC BY: This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage). 2 Brain and Neuroscience Advances achieved by swimming to a fixed platform located just below the Howard et al. (2014) identified neural correlates of path distance opaque surface, hidden from view. Escape latencies record time to to goal in the right posterior hippocampus. Such correlates of dis- reach the platform during training as well as during probe trials tance to goal have also been observed in dorsal hippocampal (when the hidden platform is removed). Lesion or inactivation of recordings in rats (Spiers et al., 2018) and bats (Sarel et al., the hippocampus impacts place learning by increasing escape 2017). During detour events, the human posterior right hip- latencies compared to that of non-lesioned controls (Morris et al., pocampus was also found to track the increase in path distance 1982, Moser et al., 1995; Sutherland et al., 1983). However, lesions when a forced detour occurred (Howard et al., 2014). Based on in dorsal striatum impair simple approach behaviour when the plat- this finding and other evidence from rats (e.g. Gupta et al., 2010; form is visible, and instead, rats will swim to previously learned Ólafsdóttir et al., 2015; Pfeiffer and Foster, 2013), it has been platform location (McDonald and White, 1994). hypothesised the hippocampus simulates future paths through the A paradigm called Delayed-Matched-to-Place further extended environment at key events during navigation, such as at detours the Morris water maze by investigating one-shot learning, a hall- (Spiers and Gilbert, 2015). Consequently, detours requiring sim- mark of behavioural flexibility (Steele and Morris, 1999). In this ulation of a much larger future route will evoke greater demands version of the task, the location of the hidden platform changes on the hippocampus than simulation of shorter routes. each day. This results in a substantial drop in escape latency In order to test the prediction of Spiers and Gilbert (2015), a between the first and second trials. The subsequent trials exhibit recent study by Javadi et al. (2019a) examined hippocampal latency improvement, but to a much smaller extent. This concept response to, respectively, small and large changes in distance to of one-shot learning is an impressive quality of cognitive flexibil- goal at forced detours (see Figure 1(a)). In this task, participants ity difficult to capture by biophysically plausible modelling of navigated a virtual desert island riven with lava which blocked place cells (Foster et al., 2000). However, reinforcement learning certain movements across it. Participants first learned the layout (RL) can capture this behavioural phenomenon by further simu- and location of several hidden objects, which were later pre- lating cells which estimate real world coordinates (Foster et al., sented as a goal to navigate to. During the test phase, when par- 2000; Tessereau et al., 2020). Together, these simulated cells form ticipants actively navigated the maze, shifts in the location of an allocentric coordinate system receiving input from the place lava pools either opened up new paths or blocked old paths, cells. This coordinate system lacks a biological basis, although resulting in possible shortcuts and detours, respectively. In con- this may be analogous to information represented by grid cells in trast to the predictions of Spiers and Gilbert (2015), posterior the entorhinal cortex (Hafting et al., 2005). Likewise, simulated hippocampus did not index the change in distance to goal at deep RL agents endowed with grid-like representation can per- detours, but rather prefrontal regions and bilateral caudate form flexible spatial navigation tasks such as the Morris water nucleus tracked the change in path distance to goal (Javadi et al., maze (Banino et al., 2018). In addition, bilateral lesions to the 2019). Notably, in Howard et al. (2014), the hippocampal fornix impairs performance in an eight-arm radial maze task, in response to distance changes at detours was also accompanied by which rats are trained to revisit certain arms consistently baited a similar response in the dorsal striatum (Figure 1(b)). Taken with food (Packard et al., 1989). Intact hippocampal function is together, these results indicate that the dorsal striatum is more necessary for place learning in a plus-maze task as well (Packard consistent in tracking the change in distance to goal at detours and McGaugh, 1996). Evidence from neuroimaging studies of than the hippocampus. This suggests it is timely to reconsider the humans and patients with hippocampal damage further implicates role of dorsal striatum during flexible navigation and understand the hippocampus in supporting both place learning and flexible how the hippocampus interacts with these regions in cortico-stri- navigation of novel routes and environments (Bohbot et al., 2007; atal loops (Brown et al., 2012; Goodroe et al., 2018). Hartley et al., 2003; Howard et al., 2014; Iaria et al., 2003; Javadi et al., 2019a, 2019b; Javadi et al., 2017; Patai et al., 2019; Spiers et al., 2001a, 2001b; Spiers and Maguire, 2006; Xu et al., 2010). How might the striatum contribute to In addition to place learning, animals also utilise ‘response flexible navigation behaviour? learning’, that is, learning based on the responses required to reach the goal (Packard and McGaugh, 1996). Such response Despite the traditional role of response learning attributed to stri- learning is shown to depend on the functional integrity of the dor- atal function, the striatum has been implicated in studies investi- sal striatum (Packard et al., 1989; Packard and McGaugh, 1996). gating behavioural flexibility in both rodents and humans, Subsequently, human neuroimaging research has provided con- suggesting a more nuanced functionality beyond contributing to vergent evidence for the involvement of the dorsal striatum in a less flexible response system (Johnson et al., 2007). Lesions such response strategy navigation (Hartley et al., 2003; Iaria et al., and inactivations in different areas of striatum produce varied 2003; Voermans et al., 2004). Response learning is not tradition- behavioural deficits, indicating a dissociation of respective func- ally considered flexible because it is tied to the specific features of tional roles (Ragozzino et al., 2002; Sharpe et al., 2019). The the environment (e.g. always turn right at the crossroad). By con- striatum is commonly divided up into two anatomically separated trast, place learning is thought to be flexible since it is possible to regions: the dorsal striatum, composing of the caudate and puta- use viewpoint-independent information from the environment to men, and the ventral striatum, composed mainly of the nucleus accommodate detours and identify shortcuts and because it does accumbens although no clear cytoarchitectonic or histochemical not rely on the presence of a single specific cue. boundary between ventral and dorsal striatum exists (Haber and Recent studies have begun to explore how different types of Knutson, 2010). Furthermore, the rodent caudate-putamen is seg- spatial information may be tracked by specific brain regions dur- mented into dorsomedial striatum (homologous to primate cau- ing navigation. Two important metrics for flexible navigation are date) and dorsolateral striatum (homologous to primate putamen) vector-to-goal and path-to-goal (Bicanski and Burgess, 2020; (Cox and Witten, 2019). Early rodent studies did not include Chadwick et al., 2015; Spiers and Barry, 2015). Using in situ strict separation of these regions when using large lesions, which learning experience and film simulation of Soho in London (UK), leads to interpretation difficulties (Yin and Knowlton, 2006). Gahnstrom and Spiers 3 Figure 1. Dorsal striatum activity is correlated with the change in distance to goal at detours. (a) Replotted data from Javadi et al. (2019a) in which fMRI and virtual reality desert island riven with lava was used to examine the brain regions responsive to the change in distance to the goal at detours. Top row shows a zoomed in schematic from the larger virtual environment used and the transition that occurs when the path is unexpectedly blocked. Bottom row: the same change but from the first person perspective. Brain image shows bilateral activity in medial caudate nucleus (dorsal striatum) cluster-corrected for activity correlated positively with the parametrically modulated change in distance-to-goal. (b) Replotted data from Howard et al. (2014). In this study, a film simulation of Soho in London was used to test navigation, including accommodating detours. The amount of change in distance-to-goal caused by forced detours was correlated with the dorsal striatal activity. Red regions show regions activations thresholded at p < 0.005 uncorrected, shown in the mean structural image. RL models provide a normative framework to investigate reward prediction errors (Daw et al., 2011; Gläscher et al., 2010). neural mechanisms that give rise to flexible and inflexible behav- Daw et al. (2011) also found the striatal underpinnings of habit- iour (Corrado et al., 2009). Within the RL literature, flexible and ual model-free prediction errors and model-based prediction goal-directed behaviour is often described by a family of algo- errors overlap in ventral striatum, suggesting the same neural cir- rithms classified as ‘model-based’. This is commonly contrasted cuitry is involved in both computations. A recent fMRI meta- with habitual behaviour described by a separate family of algo- analysis of multi-step decision making tasks found overlapping rithms classified as ‘model-free’ (Dolan and Dayan, 2013; Rusu regions involved in model-based and model-free computations in and Pennartz, 2020). These computational models ‘learn’ states globus pallidus and caudate nucleus (Huang et al., 2020). and rewards in the environment by using a component referred to Beyond the classic divisions of model-free and model-based as reward prediction errors, that is, the difference between literature in decision-making tasks, there are other families of RL expected and experienced reward. The goal of a RL agent is to algorithms that provide alternative accounts, including hierarchi- take actions which maximise future reward in the long run cal RL, linear RL, and successor representation (Botvinick et al., (Sutton and Barto, 2018). The canonical finding of reward pre- 2009; Dayan, 1993; Gershman, 2018; Piray and Daw, 2019; diction errors found encoding in single neurons of the ventral Russek et al., 2017; Stachenfeld et al., 2017; Tessereau et al., tegmental area in the brainstem of macaques (Schultz et al., 2020). In particular, successor representation can account for flex- 1997), a region which has direct dopaminergic projection to the ible behaviour of rats and humans in complex mazes (De Cothi nucleus accumbens in ventral striatum (Haber and Knutson, et al., 2020) and humans in reward devaluation protocols 2010). Since then, human functional magnetic resonance imag- (Momennejad et al., 2017). Interestingly, components of the suc- ing (fMRI) studies using multi-step decision making tasks have cessor representation during simulations show similarities to identified ventral striatum as a primary region for the process of properties of place cells and grid cells, including the influence of 4 Brain and Neuroscience Advances goal locations on place field over-representation observed in spe- overlapped with model-free correlates in the retrosplenial cortex. cific paradigms and influence of environmental geometry on grid In contrast to Simon and Daw (2011), this study did not utilise field integrity (Duvelle et al., 2019; Ekstrom et al., 2020; Krupic visual goal cues and also did not include changes in the maze et al., 2015; Stachenfeld et al., 2017). It is an interesting future configuration, more akin to classical spatial navigation para- direction for studies to investigate the relationship between neural digms. The different accounts of striatal involvement in predic- responses and the internal computations of successor representa- tion errors can perhaps be reconciled by considering that the tion shown to account for behaviour flexibility particularly in behavioural strategies and neural mechanisms are not as easily some spatial navigation tasks (Russek et al., 2017; for review see dissociable as previously thought. One spatial planning task Momennejad, 2020). Recent work with rats navigating between found striatal activity related to the difference in path distance four interconnected rooms has revealed that during initial adapta- between the shortest path and unchosen longest path to goal as a tion to pathways being obstructed place cells in CA1 do not adapt proxy for exhaustive search or forward planning (Kaplan et al., their firing fields to accompany the changing behaviour (Duvelle 2017). This indicates that striatal subregions may be involved in et al., 2020) as might have been predicted by a model in which planning, which may be the reason these regions are active in place cells support SR coding (Stachenfeld et al., 2017). It may be different studies. Perhaps a mixed use of strategies is also an that more stereotyped trajectories would lead to shifts in the place underlying reason for this result. Brown et al. (2012) showed that fields as a result of topological manipulations. caudate is important for disambiguating context during spatial The dorsal striatum has commonly been linked to stimulus– navigation, together with orbitofrontal cortex and hippocampus. response association, or habits, in spatial navigation tasks using We suggest these findings are in line with a new perspective of human fMRI. Doeller et al. (2008) employed a virtual object- these regions. In this view, the caudate encodes learned transition memory task inspired by the Morris water maze. They found structures. However, the current active transition structure at any activity in caudate nucleus to be parametrically modulated by the point in time is based on the current state of the animal and con- influence of intramaze landmarks on goal locations, while right text within the task, which is proposed to be modulated through posterior hippocampus correlated with boundary-related influ- cholinergic interneurons in dorsomedial striatum whose task- ence on goal locations (Doeller et al., 2008). In another study in dependent state information relies on an intact orbitofrontal cortex which participants navigated a virtual town, caudate activity was (Sharpe et al., 2019; Stalnaker et al., 2016). Hippocampus, on the preferentially active during route following trials, while anterior other hand, is involved in learning the structure of the environ- hippocampus was preferentially active during wayfinding trials ment (incidental to the task), and also the accompanying associa- (Hartley et al., 2003). Likewise, Iaria et al. (2003) found place tion-based learning. strategy use in an eight-arm radial maze task was associated with Instrumental learning paradigms in rodents reveal a model- increased right hippocampal activity while non-spatial response based influence on model-free prediction errors (Langdon et al., strategy use was associated with increased activity in caudate 2018). As such, the classical role of dopaminergic prediction nucleus. These studies suggest a dissociation between the roles of errors are more nuanced and can incorporate signals related to dorsal striatum and hippocampus for habitual and flexible behav- behavioural flexibility and the current state of the task in ventral iour, respectively. However, contextual demands may elucidate a tegmental area (Keiflin et al., 2019; Starkweather et al., 2017) as more nuanced role for the striatum in multiple behavioural con- well as dorsomedial striatum (Stalnaker et al., 2016). Using trol circuits (Balleine et al., 2015; Ferbinteanu, 2019; Rusu and causal methodology by optogenetically stimulating dopaminer- Pennartz, 2020; Woolley et al., 2015). gic neurons in ventral tegmental area (the putative cells encod- In rodents, the involvement of dorsal striatum in both flexible ing reward prediction errors), rats could learn associations and habitual behaviour could be resolved by considering the between cues without endowing them with cached-value, as functional distinction of dorsolateral and dorsomedial regions would be the expected based on pure model-free temporal-dif- (Gasser et al., 2020; Regier et al., 2015; Thorn et al., 2010; Van ference learning models (Sharpe et al., 2020). Another instru- Der Meer et al., 2010). Studies investigating the homologous mental learning task found an increasing number of neurons regions in humans are made difficult by the lack of spatially encoding task-relevant information in dorsolateral striatum precise recordings of neuronal activity. One account suggests more so than dorsomedial, suggesting the former may be encod- dorsal striatum performs the role of an ‘actor’, while ventral ing the development of a habit-based response (Kimchi et al., striatum performs the parallel role of a ‘critic’ in the ‘actor- 2009). Recordings in rats navigating a T-maze found that neu- critic’ RL framework (Sutton and Barto, 2018). In support of this rons in dorsomedial striatum were primarily active while choos- idea, such a division in computational roles was found during an ing between alternative actions after cue-onset, in contrast with instrumental learning task using fMRI (O’Doherty et al., 2004). neurons in dorsolateral striatum which were primarily active Investigation of functional distinction in dorsal striatum found during action execution (Stalnaker et al., 2016; Thorn et al., putamen involvement in habit-based processing from extensive 2010). found that cholinergic interneurons in rodent dorsome- training versus caudate involvement in forward planning dial—and not dorsolateral striatum—represented information (Wunderlich et al., 2012). The role of forward planning at about the current state of the choice task. In addition, this state detours could be considered in the task by Javadi et al. (2019a) information was not present in rats with lesions to the orbito- wherein distance changes were tracked by bilateral caudate frontal cortex. Taken together, there appears to be shared neural nucleus (Figure 1). In a virtual navigation task, Simon and Daw circuitry for model-free and model-based behaviours, and pre- (2011) also found forward planning tracked by striatum using diction errors may convey more information than the difference predictions from ‘model-based’ RL. between experienced and expected reward (Doll et al., 2012). In a more recent virtual navigation task, Anggraini et al. Perhaps the aforementioned human studies can be reconciled (2018) identified model-free correlates in dorsal striatum. Model- with the notion that caudate can support a mixture of model-free based correlates were found in the parahippocampus and and model-based computations depending on the task and Gahnstrom and Spiers 5 context at hand. Caudate nucleus activity can be expected in structure of the environment (see Momennejad, 2020). The response to changes in transition structure if it also encodes entorhinal cortex has also been proposed to play a role in cod- model-based information regarding the task environment. ing the transition structure of the layout of the environment or These recent findings pose a new question: What is the stimulus set (Behrens et al., 2018). Understanding how such a human dorsal striatum coding that drives these observed changes code relates to striatal coding of transition structure would be in activity during navigation? Rodent work on dorsomedial stri- useful for advancing models of the neural systems supporting atum suggests this region is necessary for execution of flexible flexible navigation behaviour. goal-directed behaviour (Rusu and Pennartz, 2020). Similarly, dorsomedial lesions have demonstrated similar behavioural def- Acknowledgements icits to that of hippocampal lesions in terms of deficiencies in We thank Sarah Goodroe for helpful comments on the draft and for help goal-directed flexible behaviour (Sharpe et al., 2019). For effec- with the figure together with Zita Patai. We also thank Eleonore Duvelle tive flexible behaviour, Sharpe et al. (2019) suggests hippocam- for comments on the manuscript. pus provides information about the environmental structure, while dorsomedial striatum incorporates information about the Declaration of Conflicting Interests transition structure into one’s overall world model. In human The author(s) declared no potential conflicts of interest with respect to navigation, novel forced detours are a classic example of a the research, authorship, and/or publication of this article. change in the transition structure. If the caudate updates repre- sentations of the transition structure, with greater transitional Funding change resulting in greater demand on caudate activity, then this The author(s) disclosed receipt of the following financial support for the may explain the results of both Javadi et al. (2019a) and Howard research, authorship, and/or publication of this article: The authors are et al. (2014), see Figure 1, where the larger the change in dis- supported by the European Union’s Horizon 2020 Framework Programme tance at detours the greater the caudate activity evoked. By con- for Research under Marie Sklodowska-Curie ITN (EU-M-GATE 765549) trast, hippocampus may be required to construct simulations of and ESRC grant to HJS. journeys through the environments (Bendor and Spiers, 2016). Such simulations may have been much richer in the navigation ORCID iD of London’s Soho (Howard et al., 2014), compared with a desert Christoffer J. Gahnstrom https://orcid.org/0000-0002-9989-5136 island (Javadi et al., 2019a), explaining the difference in hip- pocampal engagement. References Entorhinal cortex may also be involved in representing low- dimensional features of environments by extracting basis sets (or Andersen P, Morris R, Amaral D, et al. (eds) (2006) The Hippocampus eigenvectors of the successor representation), some of which Book. Oxford: Oxford University Press. look visually similar to the iconic hexagonal nature of grid fields Anggraini D, Glasauer S and Wunderlich K (2018) Neural signatures of reinforcement learning correlate with strategy adoption during spa- (Behrens et al., 2018; Stachenfeld et al., 2017). In RL, a ‘model’ tial navigation. Scientific Reports 8(1): 10110. of the environment is defined by P(s’|s, a), equal to the probabil- Balleine BW, Dezfouli A, Ito M, et al. (2015) Hierarchical control of ity of transitioning to a future state (s’) given a specific action (a) goal-directed action in the cortical–basal ganglia network. Current in the current state (s) (Sutton and Barto, 2018). Lesions during Opinion in Behavioral Sciences 5: 1–7. the Morris water maze have shown the entorhinal cortex to be Banino A, Barry C, Uria B, et al. (2018) Vector-based navigation using involved in flexible behaviour, as animals have similar behav- grid-like representations in artificial agents. Nature 557(7705): ioural deficits to those of hippocampal lesions in terms of 429–433. increased swimming latencies to the hidden platform (Hales Behrens TE, Muller TH, Whittington JC, et al. (2018) What is a cognitive et al., 2014). One idea is that the entorhinal cortex supports the map? Organizing knowledge for flexible behavior. Neuron 100(2): ability to form general transition structures of any environment 490–509. Bendor D and Spiers HJ (2016) Does the hippocampus map out the and store information about how distant states or locations are future? Trends in Cognitive Sciences 20(3): 167–169. related to each other (Behrens et al., 2018; Constantinescu et al., Bicanski A and Burgess N (2020) Neuronal vector coding in spatial cog- 2016). However, the unique dorsal striatum contribution may be nition. Nature Reviews Neuroscience 21(9): 453–470. more closely related to how action-outcome associations are rep- Bohbot VD, Lerch J, Thorndycraft B, et al. (2007) Gray matter differ- resented and which state is transitioned to as a result of a given ences correlate with spontaneous strategies in a human virtual navi- motor action (Sharpe et al., 2019). gation task. Journal of Neuroscience 27(38): 10078–10083. In conclusion, evidence suggests the dorsomedial striatum/ Botvinick MM, Niv Y and Barto AG (2009) Hierarchically organized caudate nucleus plays a key role in flexible navigation by repre- behavior and its neural foundations: A reinforcement learning per- senting the transition structure of the environment for guiding spective. Cognition 113(3): 262–280. future actions (Sharpe et al., 2019) and this may explain Brown TI, Ross RS, Tobyne SM, et al. (2012) Cooperative interactions between hippocampal and striatal systems support flexible naviga- observed responses at detours where transition structure tion. NeuroImage 60(2): 1316–1330. changes (Howard et al., 2014; Javadi et al., 2019a). Future Chadwick MJ, Jolly AEJ, Amos DP, et al. (2015) A goal direction sig- research will be useful to observe dorsomedial striatal activity nal in the human entorhinal/subicular region. Current Biology 25(1): in rodents during dynamic changes to the environment’s transi- 87–92. tion structure and variations in update demands (e.g. detours Constantinescu AO, OReilly JX and Behrens TEJ (2016) Organizing that require larger or smaller shifts in the route to the goal). It conceptual knowledge in humans with a gridlike code. Science would also be important to examine the interplay between the 352(6292): 1464–1468. striatum, hippocampal/parahippocampal structures, and pre- Corrado GS, Sugrue LP, Brown JR, et al. (2009) The trouble with choice: frontal cortex during such updating and representation for the Studying decision variables in the brain. In: Glimcher PW, Camerer 6 Brain and Neuroscience Advances CF, Fehr E, et al. (eds) Neuroeconomics: Decision Making and the Hartley T, Maguire EA, Spiers HJ, et al. (2003) The well-worn route and Brain. Amsterdam: Elsevier, pp. 463–480. the path less traveled: Distinct neural bases of route following and Cox J and Witten IB (2019) Striatal circuits for reward learning and deci- wayfinding in humans. Neuron 37(5): 877–888. sion-making. Nature Reviews Neuroscience 20(8): 482–494. Howard LR, Javadi AH, Yu Y, et al. (2014) The hippocampus and ento- Daw ND, Gershman SJ, Seymour B, et al. (2011) Model-based influ- rhinal cortex encode the path and euclidean distances to goals during ences on humans’ choices and striatal prediction errors. Neuron navigation. Current Biology 24(12): 1331–1340. 69(6): 1204–1215. Huang Y, Yaple ZA and Yu R (2020) Goal-oriented and habitual deci- Dayan P (1993) Improving generalization for temporal difference sions: Neural signatures of model-based and model-free learning. learning: The successor representation. Neural Computation 5(4): NeuroImage 215: 116834. 613–624. Iaria G, Petrides M, Dagher A, et al. (2003) Cognitive strategies depen- De Cothi W, Nyberg N, Griesbauer E-M, et al. (2020) Predictive maps in dent on the hippocampus and caudate nucleus in human naviga- rats and humans for spatial navigation. BioRxiv. Available at: https:// tion: Variability and change with practice. Journal of Neuroscience www.biorxiv.org/content/10.1101/2020.09.26.314815v2 23(13): 5945–5952. Devan BD and White NM (1999) Parallel information processing in the Javadi A-H, Emo B, Howard LR, et al. (2017) Hippocampal and prefron- dorsal striatum: Relation to hippocampal function. The Journal of tal processing of network topology to simulate the future. Nature Neuroscience 19(7): 2789–2798. Communications 8: 14652. Doeller CF, King JA and Burgess N (2008) Parallel striatal and hippo- Javadi A-H, Patai EZ, Marin-Garcia E, et al. (2019a) Backtracking dur- campal systems for landmarks and boundaries in spatial memory. ing navigation is correlated with enhanced anterior cingulate activ- Proceedings of the National Academy of Sciences of the United ity and suppression of alpha oscillations and the ‘default-mode’ States of America 105(15): 5915–5920. network. Proceedings of the Royal Society B: Biological Sciences Dolan RJ and Dayan P (2013) Goals and habits in the brain. Neuron 286(1908): 20191016. 80(2): 312–325. Javadi A-H, Patai EZ, Marin-Garcia E, et al. (2019b) Prefrontal dynamics Doll BB, Simon DA and Daw ND (2012) The ubiquity of model-based associated with efficient detours and shortcuts: A combined func- reinforcement learning. Current Opinion in Neurobiology 22(6): tional magnetic resonance imaging and magnetoencenphalography 1075–1081. study. Journal of Cognitive Neuroscience 31(8): 1227–1247. Duvelle E, Grieves RM, Hok V, et al. (2019) Insensitivity of place cells Johnson A, van der Meer MA and Redish AD (2007) Integrating hippo- to the value of spatial goals in a two-choice flexible navigation task. campus and striatum in decision-making. Current Opinion in Neuro- Journal of Neuroscience 39(13): 2522–2541. biology 17(6): 692–697. Duvelle E, Grieves RM, Liu A, et al. (2020) Hippocampal place cells Kaplan R, King J, Koster R, et al. (2017) The neural representation of encode global location but not changes in environmental connectiv- prospective choice during spatial planning and decisions. PLOS ity in a 4-room navigation task. BioRxiv. Available at: https://www. Biology 15(1): e1002588. biorxiv.org/content/10.1101/2020.10.20.346130v1 Keiflin R, Pribut HJ, Shah NB, et al. (2019) Ventral tegmental dopamine Ekstrom AD, Harootonian SK and Huffman DJ (2020) Grid coding, spa- neurons participate in reward identity predictions. Current Biology tial representation, and navigation: Should we assume an isomor- 29(1): 93–103e3. phism? Hippocampus 30(4): 422–432. Kimchi EY, Torregrossa MM, Taylor JR, et al. (2009) Neuronal corre- Ferbinteanu J (2019) Memory systems 2018 – Towards a new paradigm. lates of instrumental learning in the dorsal striatum. Journal of Neu- Neurobiology of Learning and Memory 157: 61–78. rophysiology 102(1): 475–489. Foster DJ, Morris RGM and Dayan P (2000) A model of hippocampally Krupic J, Bauza M, Burton S, et al. (2015) Grid cell symmetry is shaped dependent navigation, using the temporal difference learning rule. by environmental geometry. Nature 518(7538): 232–235. Hippocampus 10(1): 1–16. Langdon AJ, Sharpe MJ, Schoenbaum G, et al. (2018) Model-based Gasser J, de Vasconcelos AP, Cosquer B, et al. (2020) Shifting between predictions for dopamine. Current Opinion in Neurobiology 49: response and place strategies in maze navigation: Effects of training, 1–7. cue availability and functional inactivation of striatum or hippocam- McDonald RJ and White NM (1994) Parallel information processing in pus in rats. Neurobiology of Learning and Memory 167: 107131. the water maze: Evidence for independent memory systems involv- Gershman SJ (2018) The successor representation: Its computational logic ing dorsal striatum and hippocampus. Behavioral and Neural Biol- and neural substrates. Journal of Neuroscience 38(33): 7193–7200. ogy 61(3): 260–270. Gläscher J, Daw N, Dayan P, et al. (2010) States versus rewards: Dis- Momennejad I (2020) Learning structures: Predictive representations, sociable neural prediction error signals underlying model-based and replay, and generalization. Current Opinion in Behavioral Sciences model-free reinforcement learning. Neuron 66(4): 585–595. 32: 155–166. Goodroe SC, Starnes J and Brown TI (2018) The complex nature of Momennejad I, Russek EM, Cheong JH, et al. (2017) The successor rep- hippocampal-striatal interactions in spatial navigation. Frontiers in resentation in human reinforcement learning. Nature Human Behav- Human Neuroscience 12: 250. iour 1(9): 680–692. Grieves RM and Jeffery KJ (2017) The representation of space in the Moser MB, Moser EI, Forrest E, et al. (1995) Spatial learning with a brain. Behavioural Processes 135: 113–131. minislab in the dorsal hippocampus. Proceedings of the National Gupta AS, van der Meer MAA, Touretzky DS, et al. (2010) Hippocam- Academy of Sciences 92(21): 9697–9701. pal replay is not a simple function of experience. Neuron 65(5): Morris RG, Garrud P, Rawlins JA, et al. (1982) Place navigation impaired 695–705. in rats with hippocampal lesions. Nature 297(5868): 681–683. Haber SN and Knutson B (2010) The reward circuit: Linking primate O’Doherty J, Dayan P, Schultz J, et al. (2004) Dissociable roles of ventral anatomy and human imaging. Neuropsychopharmacology 35(1): and dorsal striatum in instrumental conditioning. Science 304(5669): 4–26. 452–454. Hafting T, Fyhn M, Molden S, et al. (2005) Microstructure of a spatial O’Keefe J and Dostrovsky J (1971) The hippocampus as a spatial map: map in the entorhinal cortex. Nature 436(7052): 801–806. Preliminary evidence from unit activity in the freely-moving rat. Hales JB, Schlesiger MI, Leutgeb JK, et al. (2014) Medial entorhinal Brain Research 34(1): 171–175. cortex lesions only partially disrupt hippocampal place cells and O’Keefe J and Nadel L (1978) The Hippocampus as a Cognitive Map. hippocampus-dependent place memory. Cell Reports 9(3): 893–901. Oxford: Clarendon Press. Gahnstrom and Spiers 7 Ólafsdóttir HF, Barry C, Saleem AB, et al. (2015) Hippocampal place Spiers HJ, Maguire EA and Burgess N (2001b) Hippocampal amnesia. cells construct reward related sequences through unexplored space. Neurocase 7(5): 357–382. eLife 4: e06063. Spiers HJ, Olafsdottir HF and Lever C (2018) Hippocampal CA1 activity Packard MG and McGaugh JL (1996) Inactivation of hippocampus or correlated with the distance to the goal and navigation performance. caudate nucleus with Lidocaine differentially affects expression of Hippocampus 28(9): 644–658. place and response learning. Neurobiology of Learning and Memory Stachenfeld KL, Botvinick MM and Gershman SJ (2017) The hip- 65(1): 65–72. pocampus as a predictive map. Nature Neuroscience 20(11): Packard MG, Hirsh R and White NM (1989) Differential effects of 1643–1653. fornix and caudate nucleus lesions on two radial maze tasks: Evi- Stalnaker TA, Berg B, Aujla N, et al. (2016) Cholinergic interneurons use dence for multiple memory systems. Journal of Neuroscience 9(5): orbitofrontal input to track beliefs about current state. The Journal of 1465–1472. Neuroscience 36(23): 6242–6257. Patai EZ, Javadi A-H, Ozubko JD, et al. (2019) Hippocampal and retro- Starkweather CK, Babayan BM, Uchida N, et al. (2017) Dopamine splenial goal distance coding after long-term consolidation of a real- reward prediction errors reflect hidden-state inference across time. world environment. Cerebral Cortex 29(6): 2748–2758. Nature Neuroscience 20(4): 581–589. Pearce JM, Roberts ADL and Good M (1998) Hippocampal lesions dis- Steele RJ and Morris RGM (1999) Delay-dependent impairment of a rupt navigation based on cognitive maps but not heading vectors. matching-to-place task with chronic and intrahippocampal infusion Nature 396(6706): 75–77. of the NMDA-antagonist D-AP5. Hippocampus 9(2): 118–136. Pfeiffer BE and Foster DJ (2013) Hippocampal place-cell sequences Sutherland RJ, Whishaw IQ and Kolb B (1983) A behavioural analysis depict future paths to remembered goals. Nature 497(7447): 74–79. of spatial localization following electrolytic, kainate- or colchicine- Piray P and Daw ND (2019) Linear reinforcement learning: Flexible induced damage to the hippocampal formation in the rat. Behav- reuse of computation in planning, grid fields, and cognitive con- ioural Brain Research 7(2): 133–153. trol. BioRxiv. Available at: https://www.biorxiv.org/content/10. Sutton RS and Barto AG (2018) Reinforcement Learning: An Introduc- 1101/856849v3 tion. Cambridge, MA: MIT Press. Ragozzino ME, Ragozzino KE, Mizumori SJ, et al. (2002) Role of the dor- Tessereau C, O’Dea R, Coombes S, et al. (2020) Reinforcement learning somedial striatum in behavioral flexibility for response and visual cue approaches to hippocampus-dependant flexible spatial navigation. discrimination learning. Behavioral Neuroscience 116(1): 105–115. BiorXiv. Available at: https://www.biorxiv.org/content/10.1101/20 Regier PS, Amemiya S and Redish AD (2015) Hippocampus and subre- 20.07.30.229005v2 gions of the dorsal striatum respond differently to a behavioral strat- Thorn CA, Atallah H, Howe M, et al. (2010) Differential dynamics of egy change on a spatial navigation task. Journal of Neurophysiology activity changes in dorsolateral and dorsomedial striatal loops during 114(3): 1399–1416. learning. Neuron 66(5): 781–795. Russek EM, Momennejad I, Botvinick MM, et al. (2017) Predictive rep- Tolman EC (1948) Cognitive maps in rats and men. Psychological resentations can link model-based reinforcement learning to model- Review 55(4): 189–208. free mechanisms. PLOS Computational Biology 13(9): e1005768. Tolman EC, Ritchie BF and Kalish D (1946) Studies in spatial learning. Rusu SI and Pennartz CMA (2020) Learning, memory and consolida- I. Orientation and the short-cut. Journal of Experimental Psychology tion mechanisms for behavioral control in hierarchically organized 36(1): 13–24. cortico-basal ganglia systems. Hippocampus 30(1): 73–98. Van Der Meer MAA, Johnson A, Schmitzer-Torbert NC, et al. (2010) Sarel A, Finkelstein A, Las L, et al. (2017) Vectorial representation Triple dissociation of information processing in dorsal striatum, ven- of spatial goals in the hippocampus of bats. Science 355(6321): tral striatum, and hippocampus on a learned spatial decision task. 176–180. Neuron 67(1): 25–32. Schultz W, Dayan P and Montague PR (1997) A neural substrate of Voermans NC, Petersson KM, Daudey L, et al. (2004) Interaction prediction and reward. Science 275(5306): 1593–1599. between the human hippocampus and the caudate nucleus during Sharpe MJ, Batchelor HM, Mueller LE, et al. (2020) Dopamine transients route recognition. Neuron 43(3): 427–435. do not act as model-free prediction errors during associative learn- Whishaw IQ, Mittleman G, Bunch ST, et al. (1987) Impairments in the ing. Nature Communications 11(1): 106. acquisition, retention and selection of spatial navigation strategies Sharpe MJ, Stalnaker T, Schuck NW, et al. (2019) An integrated model after medial caudate-putamen lesions in rats. Behavioural Brain of action selection: Distinct modes of cortical control of striatal deci- Research 24(2): 125–138. sion making. Annual Review of Psychology 70(1): 53–76. White NM and Donald RJ (2002) Multiple parallel memory systems in Simon DA and Daw ND (2011) Neural correlates of forward planning in the brain of the rat. Neurobiology of Learning and Memory 77(2): a spatial decision task in humans. Journal of Neuroscience 31(14): 125–184. 5526–5539. Woolley DG, Mantini D, Coxon JP, et al. (2015) Virtual water maze Spiers HJ and Barry C (2015) Neural systems supporting navigation. learning in human increases functional connectivity between poste- Current Opinion in Behavioral Sciences 1: 47–55. rior hippocampus and dorsal caudate. Human Brain Mapping 36(4): Spiers HJ and Gilbert SJ (2015) Solving the detour problem in naviga- 1265–1277. tion: A model of prefrontal and hippocampal interactions. Frontiers Wunderlich K, Dayan P and Dolan RJ (2012) Mapping value based plan- in Human Neuroscience 9: 125. ning and extensively trained choice in the human brain. Nature Neu- Spiers HJ and Maguire EA (2006) Thoughts, behaviour, and brain roscience 15(5): 786–791. dynamics during navigation in the real world. NeuroImage 31(4): Xu J, Evensmoen HR, Lehn H, et al. (2010) Persistent posterior and tran- 1826–1840. sient anterior medial temporal lobe activity during navigation. Neu- Spiers HJ, Burgess N, Hartley T, et al. (2001a) Bilateral hippocampal roImage 52(4): 1654–1666. pathology impairs topographical and episodic memory but not visual Yin HH and Knowlton BJ (2006) The role of the basal ganglia in habit pattern matching. Hippocampus 11(6): 715–725. formation. Nature Reviews Neuroscience 7(6): 464–476.

Journal

Brain and Neuroscience AdvancesSAGE

Published: Dec 21, 2020

Keywords: Spatial navigation; dorsal striatum; hippocampus; flexible behaviour; goals; reinforcement learning; wayfinding

There are no references for this article.