# On endomorphisms of surface mapping class groups

## 1. Introduction

Let $`S`$ be a compact connected orientable surface. The mapping class group $`_S`$ of the surface $`S`$ is the group of isotopy classes of orientation preserving diffeomorphisms $`SS`$. The extended mapping class group $`_S^{}`$ of $`S`$ is the group of isotopy classes of all diffeomorphisms $`SS`$. Note that the isotopy classes of orientation reversing diffeomorphisms are also included in $`_S^{}`$, and hence $`_S`$ is a subgroup of $`_S^{}`$ of index two.

Recall that a group $`G`$ is called residually finite if for each $`x1`$ in $`G`$ there exists a homomorphism $`f`$ from $`G`$ onto some finite group such that $`f(x)`$ is nontrivial. Equivalently, there is some finite index normal subgroup of $`G`$ that does not contain $`x`$. $`G`$ is called hopfian if every surjective endomorphism of $`G`$ is an automorphism. It is well known that finitely generated residually finite groups are hopfian \[LS\]. $`G`$ is called cohopfian if every injective endomorphism of $`G`$ is an automorphism.

The mapping class group of an orientable surface is finitely generated \[L, B\] and residually finite \[G, I1\]. Hence it is hopfian. N.V. Ivanov and J.D. McCarthy \[IM\] proved that $`_S`$ is also cohopfian. Author and J.D. McCarthy \[KM\] proved that if $`\varphi :_S_S`$ is a homomorphism such that $`\varphi (_S)`$ is a normal subgroup and $`_S/\varphi (_S)`$ is abelian, then $`\varphi `$ is an automorphism.

In this paper, we prove further that if $`\varphi `$ is an endomorphism of the mapping class group $`_S`$ onto a finite index subgroup, then $`\varphi `$ is in fact an automorphism, with a few exceptions. The proof of this result relies on a result of R. Hirshon, which states that if $`\varphi `$ is an endomorphism of a finitely generated residually finite group $`G`$ such that $`\varphi (G)`$ is of finite index in $`G`$, then $`\varphi `$ restricted to $`\varphi ^n(G)`$ is an injection for some $`n`$.

D.T. Wise \[W\] gave an example of a finitely generated residually finite group $`G`$ and an endomorphism $`\mathrm{\Phi }`$ of $`G`$ such that the restriction of $`\mathrm{\Phi }`$ to $`\mathrm{\Phi }^n(G)`$ is not injective for any $`n`$, answering a question of R. Hirshon in negative. It might be interesting to consider the same question for mapping class groups of surfaces.

## 2. Endomorphisms of mapping class groups

Let $`S`$ be a compact connected oriented surface of genus $`g`$ with $`b`$ boundary components. For any simple closed curve $`a`$ on $`S`$, there is a well known diffeomorphism, called a right Dehn twist, supported in a regular neighborhood of $`a`$. We denote by $`t_a`$ the isotopy class of a right Dehn twist about $`a`$, also called a Dehn twist. Note that $`ft_af^1=t_{f(a)}`$ for any orientation preserving mapping class $`f`$.

The pure mapping class group $`𝒫_S`$ is the subgroup of $`_S`$ consisting of those orientation preserving mapping classes which preserve each boundary component.

For a group $`G`$ and a subgroup $`H`$ of it, we denote by $`C_G(H)`$ the centralizer of $`H`$ in $`G`$. The center of $`G`$ is denoted by $`C(G)`$.

###### Theorem 1.

Let $`G`$ be a finitely generated residually finite group, and let $`\varphi `$ be an endomorphism of $`G`$ onto a finite index subgroup. Then there exists an $`n`$ such that the restriction of $`\varphi `$ to $`\varphi ^n(G)`$ is an injection.

###### Theorem 2.

Let $`S`$ be a compact connected orientable surface of genus $`g`$ with $`b`$ boundary components. Suppose, in addition, that if $`g=0`$ then $`b5`$, if $`g=1`$ then $`b3`$, and if $`g=2`$ then $`b1`$. Then any isomorphism between two finite index subgroups of the extended mapping class group $`_S^{}`$ is the restriction of an inner automorphism of $`_S^{}`$.

Theorem 1 was proved by R. Hirshon (\[H\]), and Theorem 2 was proved by N.V. Ivanov \[I2\] for surfaces of genus at least two and by the author \[K\] for the remaining cases. Since the mapping class group $`_S`$ is normal in $`_S^{}`$, we deduce the following theorem.

###### Theorem 3.

Let $`S`$ be a compact connected orientable surface of genus $`g`$ with $`b`$ boundary components. Suppose, in addition, that if $`g=0`$ then $`b5`$, if $`g=1`$ then $`b3`$, and if $`g=2`$ then $`b1`$. Then any isomorphism between two finite index subgroups of the mapping class group $`_S`$ is the restriction of an automorphism of $`_S`$.

###### Lemma 4.

Let $`S`$ be a closed orientable surface of genus two and let $`\mathrm{\Gamma }`$ be a finite index subgroup of $`_S`$. Then the center $`C(\mathrm{\Gamma })`$ of $`\mathrm{\Gamma }`$ is equal to $`\mathrm{\Gamma }\sigma `$, where $`\sigma `$ is the hyperelliptic involution.

Proof: Since the subgroup $`\sigma =\{1,\sigma \}`$ is the center of $`_S`$, its intersection with $`\mathrm{\Gamma }`$ is contained in the center of $`\mathrm{\Gamma }`$.

Now let $`fC(\mathrm{\Gamma })`$ and let $`N`$ be the index of $`\mathrm{\Gamma }`$ in $`_S`$. Since $`t_a^N\mathrm{\Gamma }`$ for all simple closed curves $`a`$, we have $`t_{f(a)}^N=ft_a^Nf^1=t_a^N`$. It follows that $`f(a)=a`$ (cf. \[IM\]). Hence, $`ft_af^1=t_{f(a)}=t_a`$. Since $`_S`$ is generated by Dehn twists, $`fC(_S)=\sigma `$. $`\mathrm{}`$

We are now ready to state and prove the main result of this paper.

###### Theorem 5.

Let $`S`$ be a compact connected orientable surface of genus $`g`$ with $`b`$ boundary components. Suppose, in addition, that if $`g=0`$ then $`b2,3,4`$, and if $`g=1`$ then $`b2`$. If $`\varphi `$ is an endomorphism of $`_S`$ such that $`\varphi (_S)`$ is of finite index in $`_S`$, then $`\varphi `$ is an automorphism.

Proof: If $`S`$ is a (closed) sphere or a disk, then $`_S`$ is trivial. Clearly, the conclusion of the theorem holds.

Suppose first that $`S`$ is a torus with $`b1`$ boundary component. It is well known that $`_S`$ is isomorphic to $`SL_2(𝐙)`$. The commutator subgroup of $`SL_2(𝐙)`$ is a nonabelian free group of rank $`2`$ and its index in $`SL_2(𝐙)`$ is $`12`$. Let us denote it by $`F_2`$. $`\varphi (F_2)`$ is contained in $`F_2`$ as a finite index subgroup. If this index is $`k`$, $`\varphi (F_2)`$ is a free group of rank $`k+1`$. Since there is no homomorphism from $`F_2`$ onto a free group of rank $`3`$, it follows that $`k=1`$. That is, $`\varphi (F_2)=F_2`$. In particular, $`\varphi (SL_2(𝐙))`$ contains $`F_2`$. The fact that $`\varphi `$ is an automorphism in this case was proved in \[KM\].

Suppose now that $`S`$ is not one of the surface above and not a closed a surface of genus $`2`$. Let us orient $`S`$ arbitrarily. Since $`_S`$ is finitely generated and residually finite, there exists an $`n`$ such that the restriction of $`\varphi `$ to $`\varphi ^n(_S)`$ is an isomorphism onto $`\varphi ^{n+1}(_S)`$. Note that the subgroups $`\varphi ^n(_S)`$ and $`\varphi ^{n+1}(_S)`$ are of finite index in $`_S`$. Hence, there is an automorphisms $`\alpha `$ of $`_S`$ such that the restrictions of $`\alpha `$ and $`\varphi `$ to $`\varphi ^n(_S)`$ coincide.

Let $`N`$ be the index of $`\varphi ^n(_S)`$ in $`_S`$. For any simple closed curve $`a`$ on $`S`$, $`t_a^N`$ is contained in $`\varphi ^n(_S)`$. Hence, $`\alpha (t_a^N)=\varphi (t_a^N)`$. Let $`f_S`$ be any element. Then

$`t_{f(a)}^N`$ $`=`$ $`\alpha ^1(\varphi (t_{f(a)}^N))`$   
$`=`$ $`\alpha ^1(\varphi (ft_a^Nf^1))`$   
$`=`$ $`\alpha ^1(\varphi (f))\alpha ^1(\varphi (t_a^N))\alpha ^1(\varphi (f^1))`$   
$`=`$ $`\alpha ^1(\varphi (f))t_a^N\alpha ^1(\varphi (f))^1`$   
$`=`$ $`t_{\alpha ^1(\varphi (f))(a)}^N.`$   
Hence, $`\alpha ^1(\varphi (f))(a)=f(a)`$ for all $`a`$ (cf. \[IM\]). It follows that $`f^1\alpha ^1(\varphi (f))`$ commutes with all Dehn twists. Since $`𝒫_S`$ is generated by Dehn twists, it is in $`C__S(𝒫_S)`$, the centralizer of $`𝒫_S`$ in $`_S`$. But $`C__S(𝒫_S)`$ is trivial \[IM\]. Hence, $`\alpha ^1(\varphi (f))=f`$. Therefore, $`\varphi =\alpha `$. In particular, $`\varphi `$ is an automorphism.

Suppose finally that $`S`$ is a closed surface of genus two. Let $`R`$ be a sphere with six holes. Then $`_R`$ is isomorphic to the quotient of $`_S`$ with its center $`\sigma `$, where $`\sigma `$ is the hyperelliptic involution (cf. \[BH\]) . Let us identify $`_R`$ and $`_s/\sigma `$, and let $`\pi :_S_R`$ be the quotient map. Since $`\varphi (\sigma )`$ is in the center of $`\varphi (_S)`$, either $`\varphi (\sigma )=\sigma `$ or $`\varphi (\sigma )=1`$ by the lemma above.

If $`\varphi (\sigma )=\sigma `$, then $`\varphi `$ induces an endomorphism $`\mathrm{\Phi }`$ of $`_R`$, such that $`\pi \varphi =\mathrm{\Phi }\pi `$. Then we have a diagram in which all squares are commutative:

$$\begin{array}{ccccccccc}1& & \sigma & & _S& \stackrel{\pi }{}& _R& & 1\\ & & \mathrm{I}& & \varphi & & \mathrm{\Phi }& & \\ 1& & \sigma & & _S& \stackrel{\pi }{}& _R& & 1\end{array}$$

where $`\mathrm{I}`$ is the identity homomorphism. Since the image $`\mathrm{\Phi }(_R)`$ of $`\mathrm{\Phi }`$ is of finite index, $`\mathrm{\Phi }`$ is an automorphism by the first part. By $`5`$-lemma, $`\varphi `$ is an automorphism.

If $`\varphi (\sigma )=1`$, then $`\varphi `$ induces a homomorphism $`\overline{\varphi }:_R_S`$ such that $`\overline{\varphi }\pi =\varphi `$. The image of the endomorphism $`\mathrm{\Phi }=\pi \overline{\varphi }`$ of $`_R`$ has finite index. Since $`R`$ is a sphere with six holes, $`\mathrm{\Phi }`$ is an automorphism by the first part. Then, $`\varphi `$ is an automorphism, and hence $`\sigma =1`$. This contradiction finishes the proof of our theorem. $`\mathrm{}`$

Remark: If $`S`$ is a sphere with two holes, then $`_S`$ is a group of order two, and if $`S`$ is a sphere with three holes, then $`_S`$ is isomorphic to the symmetric group on three letters. Hence, in these cases the trivial homomorphism is an endomorphism onto a finite index subgroup which is not automorphism. We do not know if the conclusion of Theorem 5 holds if a sphere with four holes and a torus with two holes.

Department of Mathematics, Middle East Technical University, 06531 Ankara, Turkey. E-mail: korkmaz`@`math.metu.edu.tr.