{'fraction_non_alphanumeric': 0.0476157229630664, 'fraction_numerical': 0.016930810584374782, 'mean_word_length': 4.451379638439581, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Conformal Isometry of Lie Group Representation in Recurrent Network of Grid Cells

The activity of the grid cell population in the medial entorhinal cortex (MEC) of the mammalian brain forms a vector representation of the self-position of the animal. Recurrent neural networks have been proposed to explain the properties of the grid cells by updating the neural activity vector based on the velocity input of the animal. In doing so, the grid cell system effectively performs path integration. In this paper, we investigate the algebraic, geometric, and topological properties of grid cells using recurrent network models. Algebraically, we study the Lie group and Lie algebra of the recurrent transformation as a representation of self-motion. Geometrically, we study the conformal isometry of the Lie group representation where the local displacement of the activity vector in the neural space is proportional to the local displacement of the agent in the 2D physical space. Topologically, the compact abelian Lie group representation automatically leads to the torus topology commonly assumed and observed in neuroscience. We then focus on a simple non-linear recurrent model that underlies the continuous attractor neural networks of grid cells. Our numerical experiments show that conformal isometry leads to hexagon periodic patterns in the grid cell responses and our model is capable of accurate path integration. Code is available at \url{https://github.com/DehongXu/grid-cell-rnn}.


Introduction
Grid cells [25,19,41,29,28,12] in the mammalian dorsal medial entorhinal cortex (MEC) exhibit striking hexagon grid patterns when the agent (e.g., a rodent) navigates in 2D open environments [20,25,17,6,38,5,7,11,34,1]. It has been hypothesized that grid cell system performs path integration [10,15,25,16,32,24,35,27]. That is, the grid cells integrate the self-motion of the animal over time to keep track of the animal's own location in space. This can be implemented by a recurrent neural network that takes the velocity of the self-motion as input, and transforms the activities of the grid cells based on the velocity inputs. The animal's self-position can then be decoded from the activities of the grid cells.
Collectively, the activities of the grid cell population form a vector in the high-dimensional neural activity space. This provides a representation of the self-position of the agent in space. The recurrent network transforms the activity vector based on the movement velocity of the agent, so that the transformation is a representation of self-motion, when considered from the perspective of representational learning. The vector and the transformation together form a representation of the 2D Euclidean group, which is an abelian additive Lie group.
In a recent paper, [21] studied the group representation property and the isotropic scaling or conformal isometry property for the general transformation model. In the context of linear transformation models, they connected this property to the hexagon periodic patterns of the grid cell response maps. With the conformal isometry property of the transformation of the recurrent neural network, the change of the activity vector in the neural space is proportional to the input velocity of the self-motion in the 2D physical space. [21] justified this condition in terms of robustness to errors or noises in the neurons. Although [21] studied general transformation model theoretically, they focused on a prototype model of linear recurrent network numerically, which has an explicit algebraic and geometric structure in the form of a matrix group of rotations.
In this paper, we study conformal isometry in the context of the non-linear recurrent model that underlies the hand-crafted continuous attractor neural network (CANN) [6,7,34,1]. In particular, we will focus on the vanilla version of the recurrent network that is linear in the vector representation of self-position and is additive in the input velocity, followed by an element-wise non-linear rectification (such as ReLU non-linearity). This model has the simplicity that it is additive in input velocity before rectification. We also explore more complex variants for non-linear recurrent networks, such as the long short-term memory network (LSTM) [26]. Such models have been studied in recent work [9,3,37,8].
Our numerical experiments show that our conformal isometry condition is able to learn highly structured multi-scale hexagon grid code, consistent with the properties of experimentally observed grid cells of rodents. In addition, our learned model is capable of accurate path integration over a long distance. Our results generalize previous results of linear network models in [21] to an important class of non-linear neural network models in theoretical neuroscience that are more physiologically realistic.
The main contributions of our paper are as follows. (1) We investigate the algebraic, geometric, and topological properties of the transformation models of grid cells. (2) We study a simple non-linear recurrent network that underlies the hand-crafted continuous attractor networks for grid cells, and our numerical experiments suggest that conformal isometry is linked to hexagonal periodic patterns of grid cells.
2 Lie group representation and conformal isometry 2.1 Representations of self-position and self-motion We start by introducing the basic components of our model. x = (x 1 , x 2 ) ∈ R 2 denotes the position of the agent. Let ∆x = (∆x 1 , ∆x 2 ) be the input velocity of the self-motion, i.e., displacement of the agent within a unit time, so that the agent moves from x to x + ∆x after the unit time.
We assume v(x) = (v i (x), i = 1, ..., D) to be the vector representation of self-position x, where each element v i (x) can be interpreted as the activity of a grid cell when the agent is at position x.
(v i (x), ∀x) corresponds to the response map of grid cell i. D is the dimensionality of v, i.e., the number of grid cells. We refer to the space of v as the "neural space". We normalize v(x) = 1 in our experiments.
The set (v(x), x ∈ R 2 ) forms a 2D manifold, or an embedding of R 2 , in the D-dimensional neural space. We will refer to (v(x), x ∈ R 2 ) as the "coding manifold".
With self-motion ∆x, the vector representation v(x) is transformed to v(x + ∆x) by a general transformation model: v where by simplifying F(·, ∆x) as F ∆x (·) in notation, we emphasize that the transformation F is dependent on ∆x. While v(x) is a representation of x, F ∆x is a representation of ∆x. (v(x), ∀x) and (F ∆x (·), ∀∆x) together form a representation of the 2D additive Euclidean group R 2 , which is an abelian Lie group. Specifically, we have the following group representation condition for the transformation model: Condition 1. (Algebraic condition on Lie group representation). For any x, we have (1) F 0 (v(x)) = v(x), and (2) F ∆x 1 +∆x 2 (v(x)) = F ∆x 2 (F ∆x 1 (v(x))) = F ∆x 1 (F ∆x 2 (v(x))) for any ∆x 1 and ∆x 2 .
Condition 1(1) requires that the coding manifold (v(x), ∀x) are fixed points of F 0 with ∆x = 0. If F 0 is further a contraction off the coding manifold, then (v(x), ∀x) are the attractor points of F 0 . Condition 1 (2) requires that moving in one step with displacement ∆x 1 + ∆x 2 should be the same as moving in two steps with displacements ∆x 1 and ∆x 2 respectively. The group representation condition is the necessary condition for any valid transformation model (Equation (1)) of grid cells.
Group representation is a central theme in modern mathematics and physics [42]. However, most of the transformations studied in mathematics and physics are linear transformations that form matrix groups, and the coding manifold (v(x), ∀x) is often made implicit. [22] focused on matrix groups, with [21] studied general transformation model theoretically, but then focused on the linear transformation model in their numerical experiments. Since the transformations in recurrent neural networks are usually non-linear, we will focus on non-linear transformation models in this paper.

Conformal embedding and conformal isometry
For an infinitesimal self-motion δ x, it is straightforward to derive a first-order Taylor expansion of the transformation model in Equation (1) forms an abelian Lie group, its derivative of ∆x at 0, i.e., f , spans its Lie algebra. Both F ∆x and f are transformations acting on the coding manifold (v(x), ∀x).
We identify the conformal isometry condition of the Lie group representation as follows: Condition 2. (Geometric condition on conformal embedding and conformal isometry).
where I 2 is the 2-dimensional identity matrix. That is, the two column vectors of f (v(x)) are of equal norm s, and are orthogonal to each other.

Under the condition above
)δ x is conformal to δ x, i.e., the 2D local Euclidean space of (δ x) in the physical space is embedded conformally as another 2D local Euclidean space f (v(x))δ x in the neural activity space. We only need to replace the two orthogonal axes for δ x in the 2D physical space by the two column vectors of f (v(x)) in the neural activity space. An equivalent statement for the above condition is That is, the displacement in the neural space is proportional to the displacement in the 2D physical space.
Note that since our analysis is local, s may depend on x. If s is a global constant, then the coding manifold (v(x), ∀x) has a flat intrinsic geometry (imagining folding a piece of paper without stretching it).
[21] studied the conformal isometry property in a local polar coordinate system. In our definition above, we use 2D cartesian coordinate system, which is more convenient for the non-linear recurrent model where ∆x enters the model additively. In the context of linear recurrent networks, [21] linked the conformal isometry to the hexagon grid patterns of grid cells.

2D torus, 2D periodicity, and hexagon grid patterns
The 2D torus topology is commonly assumed a priori in the continuous attractor neural networks (CANN) for grid cells [6,7,34,1]. The torus topology has been recently supported by analyzing data from population of simultaneously recorded grid cells [23]. Within our framework, we find that such a topology is in fact a theoretical consequence of the group representation condition (Condition 1) due to a theorem in Lie group theory.
Specifically, since the elements of v(x) represent neuron activities, which are bounded biologically, the coding manifold (v(x), ∀x) is bounded and compact, and therefore the group of (F ∆x , ∀∆x) is also compact. The Lie group (F ∆x , ∀∆x) is also connected since the 2D environment is connected. According to Lie group theory [14], if the abelian Lie group (F ∆x , ∀∆x) is compact and connected, then it has a topology of 2D torus, i.e., it is isomorphic to While the group representation condition only gives us an algebraic structure, the conformal isometry condition (Condition 2) further fixes the geometry. Under the conformal isometry condition, if we further assume that the scaling factor s is a constant globally for all x, then the intrinsic geometry of the coding manifold (v(x), ∀x) remains Euclidean, and the coding manifold is a flat torus. That is, the coding manifold is not only isomorphic to S 1 × S 1 , but is also conformally isometric to S 1 × S 1 . While isomorphism is defined by mapping between two spaces, isometry concerns about the metric properties. This leads to the periodic pattern in (v(x), ∀x) over x According to the theory of 2D Bravais lattice for 2D periodic patterns [2], we can find two primitive vectors ∆x 1 and ∆x 2 , with ∆x 1 = ∆x 2 , and v(x +k 1 ∆x 1 +k 2 ∆x 2 ) = v(x) for arbitrary integers k 1 and k 2 . Along each primitive vector, for each period, (v(x + c∆x i ), c ∈ [0, 1]) traces out a circle in the neural space for i = 1, 2, causing the periodicity in v(x). According to the theory of 2D Bravais lattice, the angle between ∆x 1 and ∆x 2 can either be π/2 for square lattice or 2π/3 for hexagon lattice. It is likely that the hexagon periodicity provides a better fit to place cells, in that the hexagon lattice provides denser packing of discrete Fourier components. While this seems to be intuitive, currently we have not been able to prove this point rigorously. It seems that hexagonal periodicity emerges under the general conditions of group representation and conformal isometry, independent of a specific form of the transformation model, as we observe empirically in our numerical experiments (Section 5).
3 Non-linear recurrent neural network

Model
In this paper, we mainly focus on studying transformations F ∆x (·) that are locally approximated by non-linear recurrent neural networks, as studied in recent work [9,3,37,8]. We start by assuming the following vanilla version of a non-linear recurrent network: where ReLU(a) = max(0, a) is applied element-wise, W is a D × D weight matrix of recurrent connections, U is a D × 2 matrix, and the self-motion ∆x = (∆x 1 , ∆x 2 ) is treated as 2 × 1 vector. Note that the above model is an accurate approximation to F ∆x (·) only for small ∆x, as it may not satisfy Condition 1 in general for large ∆x.
Compared to the matrix group in [22] where F ∆x (v(x)) = M (∆x)v(x), and M (∆x) is further derived in [21] as an exponential map that is highly non-linear in ∆x, the above model (5) is much simpler and more biologically plausible in that, before ReLU, we have a single recurrent weight matrix W that is independent of ∆x, and ∆x enters the equation additively. The ReLU rectification plays a critical role for the overall non-linear effect of ∆x. The non-linear rectification in neural networks makes the transformations much more expressive than matrix representations in modern mathematics and physics.
For this model, we can derive f (v(x)) as where 1(·) is a vector of binary indicators calculated element-wise, and is row-wise product. The indicator vector 1(W v(x) > 0) changes as x changes, and it controls the change of v in the neural space according to ∆x.
Since 1(·) is not differentiable, we propose to define the loss function to enforce the conformal isometry condition based on the equivalent statement in Equation (4), as discussed in Section 4.3. In Appendix C, we also discuss a possible mechanism where conform isometry can be automatically satisfied by design.
Modules. Since it is well-known that biological grid cells are organized in discrete modules with different spatial scales [39,4], we assume that in our model, the vector representations v(x) are divided into sub-vectors analogous to modules. Accordingly, we assume that the transformation is also module-wise by assuming W is block-diagonal and U is divided into sub-blocks by row. To address the point that the module-wise transformation construction is just to keep the consistency with isometry assumption but not the reason for the emergence of hexagonal periodicity, we have performed an ablation study (see Appendix A.3).
More complex transformations. We also explore the Long Short-Term Memory network (LSTM) [26], a more complex variant of recurrent networks, as the local transformation model. In the experiments, we observed that hexagonal grid patterns can emerge in both types of transformation models, implying that the emergence of hexagonal periodicity under the conformal isometry condition is not specific to the particular form of the transformation model, at least empirically.

Eigen analysis
Next, we will deepen our theoretical understanding of the model (5)

Place cells and decoding
For open fields, we can model place cells [33] by Gaussian kernels and connect them to grid cells by the following basis expansion model [13,37]: where A(x, p) represent the place field centered at position p. A(x, p) measures the adjacency of x to p. q(p) is the query vector of the place cell p, which can be interpreted as connection weights between the grid cells and the place cell p. For a vector v, we can decode its positionx bŷ i.e., we choose the position p that is closest to the position encoded by v.
In our experiments, we learn the query vectors (q(p), ∀p) together with (v(x), ∀x), W , U .

Peter-Weyl theory
In the model defined by Equation (7), v(x) generated by the transformation model serves as a set of basis functions to reconstruct the set of functions A(x, p), ∀p. This is related to the Peter-Weyl theory [40], which generalizes the Fourier analysis to Lie group and shows that the basis functions arise from Lie group representation. v(x) can be used to reconstruct and interpolate value functions of x in general. Peter-Weyl theory is about matrix Lie groups and irreducible representations. In our work, we focus on non-linear transformation groups which can still generate basis functions.
Peter-Weyl theory naturally connects two roles of grid cells: (1) path integration, and (2) basis expansion. It can be interesting to generalize Peter-Weyl theory to non-linear transformations.

Loss function
Assuming the kernel A(x, p) is given as in Equation (7), we can learn the model by minimizing the following loss term: The learnable parameters includes (v(x), ∀x), (q(p), ∀p), and parameters in F ∆x . The expectations are estimated by Monte Carlo samples from simulated trajectories. A(x, p) are Gaussian kernels with predefined σ . In practice, we add an additional zero-step version of L 1 , i.e. the expectation term changes to To ensure that the conformal isometry condition is satisfied, we add an extra loss term based on the equivalent statement of conformal isometry. Following Equation (4), for simplicity, we first denote Then we propose a conformal isometry loss: where ∆x 1 and ∆x 2 are sampled such that they have the same length ∆r but with different directions, i.e. ∆x 1 = (∆r cos θ 1 , ∆r sin θ 1 ), ∆x 2 = (∆r cos θ 2 , ∆r sin θ 2 ). Moreover, we add another regularization term to penalize q(p) 2 .

Experiments
We optimized the model using simulated trajectories as training data. The environment was assumed to be a 1m × 1m squared open field, discretized into a 40 × 40 lattice. v(x) is of 1800 dimensions, which was partitioned into 150 modules with module size 12. For A(x, p), we used a Gaussian adjacency kernel with σ = 0.07. We trained a 10-step recurrent network as the transformation model, i.e., T = 10 in the loss term L 1 .
For L 1 , the displacement of ∆x t was restricted to be smaller than 3 grids. For the range of ∆r in L 2 , we hypothesize that it can be proportional to the scale of the module (i.e., s in Equation (4)), which is also reflected as the scale of the learned hexagon patterns. Thus we adaptively adjusted the upper bound of ∆r for each module during training, based on the scale of the learned hexagon patterns at the current training stage. We averaged the scales of the learned patterns within each module to represent the scale of that module. We set the upper bound of ∆r for the module with the largest average scale to be 15 grids. The ranges of ∆r for the remaining modules are adjusted according to their scales. Figure 1 shows the learned firing patterns of v(x) = (v i (x), i = 1, ..., d) over the 40 × 40 lattice of x. We randomly selected 16 modules out of 150 modules for visualization purposes. Each image corresponds to the response map of a grid cell. Every row shows the learned units that belong to the same module. The hexagonal patterns in the emerging activity patterns are evident. We found that the loss term for imposing the conformal isometry was critical. Without it, the learned response maps showed stripe-like patterns (see Appendix A.3 for ablation results).

Hexagon patterns
To quantitatively evaluate whether the learned patterns match regular hexagon grids, in Table 1, we report the gridness scores that are adopted from the literature of grid cells [31,36], as well as the valid percentage of grid cells with gridness score > 0.37 being the criteria. Figure 2 shows the histogram of the spatial scales of the learned hexagon patterns. The multi-modal distribution is fitted by a mixture of three Gaussians, which are centered at 0.41, 0.64 and 0.86 respectively. The ratios between adjacent centers are 1.55 and 1.35, which are in the range of the data from rodent grid cells [39].

Path integration
We further evaluate whether the learned model is capable of accurate path integration. We perform path integration in two scenarios. First, for path integration with re-encoding, we decode v t → x t  to physical space and then apply encoder v(x t ) → v t back to neuron space every few steps. This re-encoding strategy helps correct the errors accumulated in the neural space along the transformation.
In the case without re-encoding, we apply transformation purely using neuron vector v t . As shown in the left panel of Figure 3, the model can perform near exact path integration for 30 steps (short distance) without re-encoding. For long-distance path integration, we train a 20-step recurrent network model, and evaluate the model for 500 steps over 1000 trajectories. As shown in the right subfigure of Figure 3, if we re-encode every 20 steps, the path integration error for the last step is 0.028, while the average error over the 500 steps trajectory is 0.017. Without re-encoding, the error is relatively larger, where the average error is around 0.03 along the whole trajectory, and 0.08 for the last step.
To check if the model can apply precise encoding and decoding between physical space and neuron space, we also examine the fixed point condition by applying v(x) → v t → x . Ideally, the learned model can figure out the physical location x t purely from v t . The L 2 error between x and x is nearly zero (< 0.005).

Model with LSTM units
In this section, we evaluate the LSTM transformation model. The model is still trained by the same loss functions as the vanilla RNN model. Meanwhile, we force q(p) > 0 in training.
Examples of the learned patterns are visualized in Figure 4. Clear hexagon patterns are also evident. Different from the learned units from the vanilla recurrent network which are all non-negative, the learned v(x) from the LSTM model can be either positive or negative, resulting in the color shift of the learned patterns. As shown in Table 1, the average gridness score for the LSTM model is 0.73, and 68.8% of the model units are classified as grid cells. We also evaluated the LSTM model on path integration still using 1000 trajectories and each trajectory is 500 steps long. With re-encoding every 10 steps, the average decoding error over the whole trajectory was 0.027, and the error at the 500 th step still remained as low as 0.037.

Conclusion and discussion
This paper investigates the algebraic, geometric, and topological properties of the generic transformation model of grid cells. We focus on non-linear recurrent neural networks under the conformal isometry condition. Our numerical experiments demonstrated that hexagon periodic patterns emerged in the response maps of grid cells under the conformal isometry condition. Our experiments also showed that the learned model was capable of accurate path integration.
The conformal isometry property seems to be related to the difference of Gaussian kernel assumed for place cells in [13,37], which constrains the frequency components within a ring in the frequency domain. It remains to be determined whether the hexagon patterns of grid cells were caused by the recurrent network itself or by interaction with place cells, an important issue that should be investigated further.
A limitation of our work is that we did not pursue specially designed recurrent neural networks where conformal isometry is automatically satisfied. We leave it to future investigation. In the Appendix C, we provide a discussion of a possible design inspired by the hand-crafted continuous attractor neural networks of grid cells where conformal isometry is satisfied.  [30] optimizer. The learning rate is linearly decayed from 0.006 to 0.0003 for the first 10, 000 iterations, fixed at 0.0003 until 120, 000 iterations, and then linearly decayed to 0 afterwards. For batch sizes, we use 8000 for zero-step transformation loss and isometry loss (L 1 ), and 100 for multi-step transformation loss. We trained all the models on a single 2080 Ti GPU.

A.2 Learned patterns
In Figure 5, we show the autocorrelograms of the learned grid patterns from the vanilla recurrent network.

A.3 Ablation studies
In this section, we present part of our ablation results to examine whether certain components of our model are empirically important for the emergence of hexagon grid patterns. Here are some key observations: (1) hexagon patterns do not emerge without conformal isometry loss. That is, the conformal isometry condition is crucial for the emergence of grid-like patterns. (2) Regularization of q(p) 2 is not necessary, but the patterns are less clear without it. (3) Without zero-step transformation loss, the learned model is unable to path integrate accurately, although hexagon grid patterns still emerge.
We further try different module sizes. Figure 8 visualizes the learned patterns when we fix the total number of grid cells and change the module size to 24. It shows hexagonal grid firing patterns can emerge with a larger module size. In Figure 9, we show the path integration error based on different lengths (5-step, 10-step, 15-step, and 30-step) of RNN models. Path integration for 30-step trajectories is performed for settings with and without re-encoding.
Finally, we try to remove the block-diagonal assumption in the transformation model, i.e. we change W to be a full matrix instead of a block-diagonal one. But we still impose conformal isometry on the modules. In this setting, we can still learn multi-scale patterns as in Figure 7, while the path integration also works well. For a learned 10-step vanilla RNN without block-diagonal construction, the average error of 500 step path integration is around 0.02 with re-encoding every 10 steps and 0.046 without re-encoding over the whole trajectory. This suggests that the emergence of the hexagon patterns is not from block-diagonal construction but the isometry condition.

B Eigen analysis
Proof. With ∆x = 0, we have the following fixed point property: Thus Thus for any x, the matrix 1(W v(x) > 0) W has an eigenvalue 1 with geometric multiplicity 2.
Under conformal isometry, the two column vectors of 1(W v(x) > 0) U are orthogonal with equal norm, and they span the eigen-subspace of 1(W v(x) > 0) W .

C Constructive conformal isometry
In our experiment, we impose conformal isomtry with a loss term based on Equation (4). In this section, we discuss a special case of the recurrent network that satisfies Condition 2 by design.
Specifically, we divide v into low-dimensional sub-vectors, v = (v k , k = 1, ..., K), where each v k is a sub-vector of dimension d, e.g., d = 4 so that each sub-vector consists of 4 neurons. We call each sub-vector a mini-block. Then the total dimension D = Kd. Such mini-blocks are commonly assumed in handcrafted continuous attractor neural networks [6,7,34,1]. Correspondingly, we divide the D × 2 matrix U into K mini-blocks, with each being d × 2, so that U = (U k , k = 1, ..., K). The indicator vector 1(W v(x) > 0) is also divided into K mini-blocks, each of dimension d.
We assume the following two conditions for the mini-blocks: Condition 3. (Orthogonality condition). Each U k has two column vectors that are of equal norm s k and are orthogonal to each other, i.e., U k U k = s 2 k I 2 . The above condition can be easily satisfied, e.g., for d = 4, we let the first column of U k be [1, 0, −1, 0] and the second column be [0, 1, 0, −1] . The idea is that within each mini-block, the cells are sensitive to different directions of self-motion. We call the mini-blocks that satisfy the above two conditions the conformal mini-blocks. Given those two conditions, we have the following result. Theorem 2. Under the orthogonality condition and the synchronicity condition, the mini-block-wise recurrent network satisfies the conformal isometry condition (Condition 2).

Proof. The change of the vector
Thus where s 2 = ∑ k a k s 2 k .
Define the activation pattern of 1(W v(x) > 0) be a(x) = (a k , k = 1, ..., K). a(x) controls the change of the vector v(x + δ x) − v(x). At different x, a(x) are different. Thus the direction of the change v(x + δ x) − v(x) depends on x, and it is possible for the model to create rotation of the vector v(x).
It is an interesting problem to construct simple recurrent networks that satisfy conformal isometry automatically.