# Higgs Boson Signals in Three 𝑏-jet Final States at the Fermilab Tevatron

## Abstract

At the Fermilab Tevatron, final states with three tagged $`b`$-jets could play an important role in searches for a Higgs boson with mass in the range 100–300 GeV. These signals arise from $`gb`$ fusion and we demonstrate their observability in the limit of a large $`b`$-quark Yukawa coupling. Rather promising discovery limits on such a coupling are obtained and consequent effects on the parameter space of the Higgs-boson sector in the MSSM are discussed.   
PACS number(s): 12.60.-i, 14.80.Cp, 14.65.Fy, 12.38.Bx

The Higgs boson is the last ingredient of the Standard Model (SM) that still awaits discovery. It is also a crucial component since the mechanism for electroweak symmetry-breaking — and hence the masses of weak gauge bosons and fermions — arises from its interactions. There exist strong indications from the fitting of electroweak parameters to available data that the Higgs boson has a relatively small mass , which makes it likely to be seen either at the Fermilab Tevatron (in its future runs) or at the CERN LHC (when it becomes operational). Curiously, although failure to find a Higgs boson at all is likely to require serious rethinking of current theories of electroweak interactions, establishing its existence could also raise a rash of questions which might not be easy to answer. For example, one could ask whether a discovered Higgs boson is the one predicted in the SM or whether it is an ingredient of some extended theory with extra symmetries. The easiest way to settle this question would be by the detection of (an) additional Higgs boson(s) such as is (are) predicted in several of these models. Alternatively, on discovering a Higgs boson, we could determine its couplings and compare these with theoretical predictions within a given model. With the higher integrated luminosity expected in Run II of the Tevatron, the requisite precision in the measurements should be available, at least for favourable regions of the parameter space. A truly dramatic way of discovering new physics beyond the SM would be the observation of Higgs-boson signals where none are predicted in the SM.

Enhanced Yukawa couplings for the $`b`$-quark in some models beyond the SM open up many such interesting new channels. At the CERN LEP, for example, $`e^+e^{}Z^{}b\overline{b}A`$ could help extend the Higgs-boson discovery limits beyond the expectation from the study of Higgs-strahlung processes, which are the main avenue of Higgs-boson searches at present. At the Fermilab Tevatron, similar processes have been studied, some recent efforts being those of Refs. . The consensus appears to be that detection of the SM Higgs boson radiated off a heavy quark would be difficult, but (non-SM) Higgs bosons with enhanced Yukawa couplings would be much better candidates for a study in $`4b`$ and $`bb\tau \tau `$ final states.

In this letter, we propose the related process

$$gb(\overline{b})\varphi b(\overline{b})$$
(1)   
as a possible production channel for a generic Higgs boson, $`\varphi `$. While the analogous interaction for the charged Higgs boson (in two Higgs-doublet models) has been studied extensively in the context of the CERN LHC, this particular process has been neglected so far, probably under the impression that the cross section would be hopelessly small. This is, in fact, true for the SM Higgs boson, unless we consider the TeV-33 option at the Fermilab Tevatron. However, we shall show that it need not be the case – even in Run II – for theories with an enhanced $`\varphi b\overline{b}`$ coupling.

The Yukawa interaction can be written $`_{\varphi b\overline{b}}=h_b\varphi \overline{b}b`$, or $`_{\varphi b\overline{b}}=ih_b\varphi \overline{b}\gamma _5b`$, depending on the $`CP`$ assignments of the scalar $`\varphi `$. Given such an interaction, the cross-sections (to leading order) for the processes in Eq. (1) scale as the ratio $`(h_b/h_b^{\mathrm{SM}})^2`$, irrespective of the $`CP`$ properties of the field $`\varphi `$. The precise value of the ratio $`h_b/h_b^{\mathrm{SM}}`$ depends, of course, on the model. In the minimal supersymmetric extension of the SM (MSSM), for example, $`h_b/h_b^{\mathrm{SM}}=\mathrm{tan}\beta ,\mathrm{sin}\alpha /\mathrm{cos}\beta `$ and $`\mathrm{cos}\alpha /\mathrm{cos}\beta `$ for the pseudoscalar ($`A^0`$), light scalar ($`h^0`$) and heavy scalar ($`H^0`$) Higgs bosons respectively, where the angles $`\alpha ,\beta `$ have their usual meanings. In the large $`\mathrm{tan}\beta `$ limit, the coupling to the $`A^0`$ (obviously) and the $`H^0`$ are strongly enhanced; that to the $`h^0`$ undergoes a more modest increase. Similar enhancements of the $`b`$-quark Yukawa couplings can occur in composite models such as those with topcolour-assisted technicolour .

Once a $`\varphi `$ has been produced, various decay channels are available to it. Again, there is considerable model dependence. For the SM $`H^0`$, the major two-body decay modes are $`H^0b\overline{b},c\overline{c},\tau ^+\tau ^{}`$ with relative branching ratios of $`23:2:4`$ approximately. While these modes dominate for low Higgs-boson masses, the $`H^0WW^{}`$ mode becomes increasingly important for $`m_H>100\mathrm{GeV}`$. In the MSSM, on the other hand, the $`WW^{}`$ (or $`ZZ^{}`$) mode is inaccessible to the $`A^0`$. Consequently, $`Br(A^0b\overline{b})>90\%`$ (unless $`A^0`$ can decay into a pair of superpartners). Similarly, in the event of $`\mathrm{tan}\beta 1`$, there are just three important decay modes for the scalar $`h^0`$, namely $`h^0b\overline{b},\tau ^+\tau ^{},WW^{}`$, the first always accounting for more than 70%. While the $`WW^{}`$ mode may also be profitably used as a probe of Higgs-boson interactions, we prefer to concentrate on the simpler two-body decays of the Higgs boson into fermions. Thus – for the SM and MSSM at least – the final states of greatest interest in $`gb`$ fusion are $`b\tau ^+\tau ^{}`$ and $`bb\overline{b}`$ . Of these two, the former has smaller backgrounds, whether from QCD or from associated $`Z`$-production. However, $`Br(\varphi \tau ^+\tau ^{})`$ is suppressed approximately by a factor of $`6`$ or more over most of the parameter space. More importantly, invariant mass reconstruction for a $`\tau `$-pair is difficult because of multiple neutrinos arising in tau-pair decays. As mass reconstruction turns out to be a major tool in the isolation of a Higgs-boson signal, we shall not comment on the $`\tau ^+\tau ^{}`$ channel any further.

Thus, our choice of the final state is ($`bb\overline{b}`$) or ($`b\overline{b}\overline{b}`$), which we generically denote by $`3b`$. The signal cross section may be written as $`\sigma (gb3b)=^2\sigma _{\mathrm{SM}}(gb3b)`$, where

$$=\left(\frac{h_b}{h_b^{\mathrm{SM}}}\right)\left[\frac{Br(\varphi ^0b\overline{b})}{Br(H_{\mathrm{SM}}^0b\overline{b})}\right]^{1/2}.$$
(2)   
The advantage of using $``$ as a free parameter is obvious – it contains the entire model-dependence of the cross section and hence enables us to make a model-independent study of the $`3b`$ signal.

At this point, it becomes necessary to comment on the size of $`h_b^{\mathrm{SM}}`$. The low-energy value can be inferred from the pole mass, for which we use $`m_b=4.3\mathrm{GeV}`$ . At large momentum transfers, QCD corrections can be important. Since the complete set of corrections have not been calculated, we include only the suppression due to the running of $`m_b`$. As additional QCD corrections usually tend to increase the cross-section, our results should be regarded as a conservative estimate of the signal. Our expressions are consistent with those in Ref..

The backgrounds to (1) arise from two main sources: ($`i`$) ‘authentic’ $`3b`$ events from QCD and/or weak interactions; ($`ii`$) spurious events of the type $`2b+J,b+2J`$ or $`3J`$, where $`J`$ denotes a non-$`b`$ jet misidentified as a $`b`$ jet. Herein lies the advantage of considering a $`3b`$ signal: misidentification probabilities are usually low and ‘authentic’ $`3b`$ backgrounds carry the same suppression from the $`b`$-quark flux as the signal. In contrast to this, $`4b`$ backgrounds  could be generated by QCD processes arising from valence quarks or gluons, which have enormous fluxes by comparison.

The large number of diagrams contributing to the background are calculated using the helicity amplitude package madgraph . To estimate the number of events and their distribution(s), we use a parton-level Monte-Carlo event generator. For the parton densities in the proton, we use the cteq3-m structure functions as incorporated in the package pdflib .

Since the QCD background is populated mostly at low transverse momenta ($`p_T`$) and high rapidity ($`\eta `$) of the jets, we demand that the final state be composed of exactly three hard jets ($`j`$) with $`p_T^j>20\mathrm{GeV},|\eta _j|<2`$. We also require that the angular separation of the jets be substantial, i.e. $`\mathrm{\Delta }R_{jj}\sqrt{(\mathrm{\Delta }\eta _{jj})^2+(\mathrm{\Delta }\varphi _{jj})^2}>1.0,`$ adapting the well-known cone algorithm for jet separation to a parton-level analysis. While $`\mathrm{\Delta }R_{jj}>0.7`$ is usually considered sufficient for jets to be separable, the more stringent cut (especially for the two softer jets) eliminates a significant portion of the QCD background without affecting the signal at all. To eliminate the background from $`gbbZ^0`$, we demand that all events where the invariant mass $`m_{ij}`$ satisfies $`80\mathrm{GeV}<m_{ij}<100\mathrm{GeV}`$, for any of the three possible pairings ($`ij`$), should be rejected from the analysis. Similarly, a requirement of $`m_{ij}>10\mathrm{GeV}`$ helps to further reduce the QCD background. Once these kinematic cuts are applied, we are in a position to utilize a particular feature of the signal event topology. The final-state $`b`$ in the production process $`gbb\varphi `$ tends to be collinear with the initial $`b`$ quark and hence (usually) to have a transverse momentum smaller than those of the $`b`$’s arising from the scalar decay $`\varphi b\overline{b}`$. If we label the $`b`$-jets according to their $`p_T`$, thus: $`p_T(j_1)>p_T(j_2)>p_T(j_3)`$ — the invariant mass $`m_{12}`$ of the pair of hardest jets will reconstruct to the Higgs-boson mass for a majority of the signal events. Of course, there will always be some events where the pair (12) does not originate from the scalar, but such configurations are subdominant and become progressively so for larger $`m_\varphi `$. Thus the $`m_{12}`$ distribution for the signal will exhibit a characteristic peak, illustrated in Fig. 1. With the elimination of the $`Z`$ events, the background does not show any such peaking.

We can thus use $`m_{12}`$ as a discriminator for Higgs-boson resonances. When looking for a $`\varphi `$ with a given mass, it is necessary to select only those events that lead to an $`m_{12}`$ close to $`m_\varphi `$. Closeness is usually quantified by the invariant mass resolution which is taken to be  $`\mathrm{max}\{10\mathrm{GeV},\sqrt{0.8\mathrm{GeV}\times m_\varphi }\}`$ for the entire range $`m_\varphi =`$ 100–300 GeV. We note in passing that similar – but much weaker – peaks can be seen in the other two mass distributions $`m_{13},m_{23}`$, although we have not exhibited them.

Even with this kinematic selection procedure, the background is orders of magnitude larger than the signal. We demand, therefore, that all three $`b`$’s be tagged. Of course, progressively smaller fractions of $`2b+J,b+2J`$ and $`3J`$ states, respectively, will be misidentified as $`3b`$ states. To quantify this, for each individual jet, we use  a $`b`$-tagging efficiency $`ϵ_b=0.6`$ and misidentification probability $`P_{\mathrm{mis}}=0.005`$. This eliminates most of the ($`2b+J`$) and almost all of the $`b+2J`$ and $`3J`$ backgrounds while suppressing the signal (and the ‘authentic’ $`3b`$ backgrounds) by a factor of $`(0.6)^30.2`$.

Using all these cuts and selection criteria, we are now in a position to compare signal with background. Fig. 2 shows the distribution in invariant mass $`m_{12}`$, assuming uniform bins of 10 GeV in the entire range $`m_{12}=100`$$`300\mathrm{GeV}`$, for an integrated luminosity of $`L=1\mathrm{fb}^1`$. To illustrate the usefulness of this channel, we have considered a somewhat optimistic value $`=50`$, for which it is obvious that the signal is significantly larger than the background. However, since it is only necessary for the signal to exceed the fluctuation in the background at (say) 95% confidence level (CL), the actual reach in $``$ encompasses much lower values.

Fig. 2 also shows the individual channels contributing to the background. For relatively low Higgs-boson masses, the largest background arises from ‘authentic’ $`3b`$ processes, single misidentified non-$`b`$ jets come in second and double misidentification being strongly suppressed. Triple misidentification is severely suppressed and need not concern us further. For higher masses of the Higgs boson, however, the single-misidentified-jet background catches up with the ‘authentic’ one, so that the importance of this channel hardly needs to be emphasized.

We are now in a position to speculate on the mass reach of this mode. Concentrating on the appropriate $`m_{12}`$ bin, we can evaluate the minimum value of $``$ that would allow us to establish/exclude a Higgs boson with the corresponding mass. This is done in a straightforward manner by comparing the signal size (in that bin) with the statistical fluctuation in the background (in that bin), calculated using Poisson statistics.

In Fig. 3, we exhibit 95% CL discovery limits for three expected luminosities at the Fermilab Tevatron. If no excess in $`m_{12}`$ is seen, the parameter space above the curves can be ruled out at the appropriate confidence level. That the minimal value of $``$ grows with $`m_\varphi `$ is expected, since larger $`m_\varphi `$ leads to small cross sections for the process (1). The minimum at $`m_\varphi 110\mathrm{GeV}`$ is an artefact of the kinematic cut designed to eliminate the $`Z`$ background (in the absence of such a cut, of course, all the discovery limits would be much worse). It is possible, however, that a more sophisticated analysis could improve the bounds for $`m_\varphi m_Z`$. The curve for a luminosity of 100 pb<sup>-1</sup> represents the region which can be ruled out by existing data.

In obtaining Fig. 3, we have assumed that only one $`\varphi `$ lies in the mass bin under consideration. If this is not the case, or if there is more than one Higgs-boson state in the region under consideration, then the constraints originating from all such $`\varphi `$’s will have to be compounded appropriately. This could be parametrized as an enhancement in the effective value of $``$, with the actual increase determined by the model parameters. The curves in Fig. 3 are thus conservative and, as we have already stressed, model-independent.

In view of the intrinsic interest in the MSSM, we now pass from the general to the particular and derive explicit constraints on this scenario. Before presenting our results, some preliminary remarks are in order. Given the current bounds on sfermions, it is clear that it is not possible for a 100–300 GeV Higgs boson to decay into a pair of these. However, bounds on charginos and neutralinos are much weaker, and hence, we need to make some assumptions about the Wino mass parameter $`M_2`$ and the higgsino mixing parameter $`\mu `$. In our analysis we choose two illustrative sets of such parameters.

With the respective values of $``$ determined by $`m_A`$, $`\mathrm{tan}\beta `$, $`M_2`$, $`\mu `$, the sfermion masses and the stop mixing parameter $`\stackrel{~}{A}_t`$ , the three individual constraints can now be combined. The resultant bounds in the $`m_A`$$`\mathrm{tan}\beta `$ plane are presented in Fig. 4. Solid curves correspond to the case where all the superpartners are very heavy, thereby reducing the scenario to a constrained two-Higgs doublet model; dashed curves correspond to relatively light superpartners. With three non-trivial contributions (more than) offsetting any suppression due to branching ratios, the bounds on $`\mathrm{tan}\beta `$ are analogous to those of Fig. 3. Interestingly, the dependence on the MSSM parameters is rather weak, as evidenced by the small difference in the two sets. This is not unexpected as light Higgs bosons hardly decay into the superpartners, even when the mass parameters are set as low as 150 GeV. Radiative corrections in the Higgs sector, on the other hand, can significantly alter the individual couplings. On summing over the three contributions, though, the residual effects are small and are not easily visible on the scale of Fig. 4. Our predictions are thus quite robust.

We may mention in passing that there do exist constraints on the $`m_A`$$`\mathrm{tan}\beta `$ plane from direct searches at both the LEP collider and the Fermilab Tevatron  other than the 100 pb<sup>-1</sup> curve shown here. However, most of these are outside the scale of the present figure. It is obvious, therefore, that a $`3b`$ signal could be remarkably effective, if not in finding a Higgs boson of the MSSM, at least in putting stringent constraints on the parameter space.

To summarize, then, we have discussed Higgs resonances in final states with three tagged $`b`$-jets at the Fermilab Tevatron. For Yukawa couplings of a Higgs boson to $`b\overline{b}`$ pairs – which are enhanced with respect to the SM coupling, we predict large signals in bins of invariant mass constructed using the hardest pair of jets. These are used to define a model-independent constraint on the parameter space, which is then translated to the case of a specific model – the MSSM. We show that this signal could lead to constraints on the MSSM parameter space that better present constraints by a significant margin.

DC would like to thank the HEP Division, Argonne National Laboratory, for hospitality while this work was being carried out. The work of AD is supported by the Council for Scientific and Industrial Research, Govt. of India. SR acknowledges partial financial support from the World Laboratory, Lausanne.