TITLE: Relative condition number - Definition of Quarteroni not applicable
QUESTION [1 upvotes]: I am reading "Numerical Mathematics" - Quarteroni. To introduce the (relative) condition number he considers the problem $F(x,d)=0$ to be solved, where $d$ is the data. He then writes that if this problem has a unique solution, then there exists a resolvent $G(d)=x$ which solves the problem. Then by using Taylor approximation he states that for the relative condition number $K(d)$ it holds that
$$K(d)\approx\|G'(d)\|\frac{\|d\|}{\|G(d)\|}$$
Now another definition, for example the one from the wiki entry is based on considering a problem $$f\colon \mathbb{R}^n\rightarrow \mathbb{R}^m$$ and reaches the same approximation result, just exchange $G(d)$ with $f(x)$ and $d$ with $x$.
Now lets look at the example $f(x,y)=xy$. Then the wiki definition yields
$$K={\displaystyle {\frac {\left\|Df(x,y)\right\|_{2}\cdot \left\|(x,y)\right\|_{2}}{\left|f(x,y)\right|}}={\frac {{\sqrt {x^{2}+y^{2}}}\cdot {\sqrt {x^{2}+y^{2}}}}{\left|xy\right|}}={\frac {x^{2}+y^{2}}{\left|xy\right|}}}$$
I cannot reproduce this result with the definition of Quarteroni, as he requires a unique resolvent, which does not exist for this problem.
The definition of Quarteroni fails to be able to handle this simple problem. Am I missing something? Why does he choose this definition?

REPLY [1 votes]: Briefly, you need a third source. The wikipedia page is sloppy and Quarteroni's treatment of the condition number is not standard. 
Below follows three segments. First is what I consider to be a standard discussion of the normwise relative condition number of a function. Then I make a few comments regarding the wikipedia entry and Quarteroni's choice. Finally, I add a few words about the conditioning of roots.
Let $\Omega \subseteq \mathbb{R}^m$ be an open set and let $f : \Omega \rightarrow \mathbb{R}^n$. Let $x \in \Omega$. We assume that $x \not = 0$ and $f(x) \not = 0$. We now seek a single number which characterizes the relationship between the normwise relative errors
$$
    \frac{\|x-y\|}{\|x\|}
    \quad \text{and} \quad
    \frac{\|f(x)-f(y)\|}{\|f(x)\|}.
$$
Let $\delta' > 0$ be any number such that the open ball
\begin{equation}
B(x, \delta') = \{ y \in \mathbb{R}^m \: : \: \|x-y\| < \delta'\} \subseteq \Omega
\end{equation}
and let $$\delta \in \left(0,\frac{\delta'}{\|x\|}\right)$$
so that $$ \frac{\|x-y\|}{\|x\|} < \delta \Rightarrow \|x-y\| < \delta' \Rightarrow y \in \Omega.$$
We now define an auxiliary function $\kappa_f$ as follows
\begin{equation}
    \kappa_f(x,\delta) = \sup \left \{  \frac{\|f(x)-f(y)\|}{\|f(x)\|} / \frac{\|x-y\|}{\|x\|}\: : \: 0 < \frac{\| x-y\|}{\|x\|} < \delta  \right \}.
\end{equation}
It is clear that the function $\delta \rightarrow \kappa_f(x,\delta)$
is nonnegative and nondecreasing. It follows that the limit
$$
\underset{\delta \rightarrow 0_.}{\lim} \kappa_f(x,\delta) 
$$
exists. This motivates the following definition. The normwise relative condition number is defined as the limit
\begin{equation}
    \kappa_f(x) = \underset{\delta \rightarrow 0_+}{\lim} \kappa_f(x,\delta). 
\end{equation}
The normwise relative condition number imposes a hard limit on the normwise relative error which can be achieved.
If $y \in B(x,\delta)$ and $y \not = x$ then
$$
\frac{\|f(x)-f(y)\|}{\|f(x)\|} = \left(\frac{\|f(x)-f(y)\|}{\|f(x)\|} / \frac{\|x-y\|}{\|x\|}\right) 
\frac{\|x-y\|}{\|x\|} \leq \kappa_f(x,\delta) \frac{\|x-y\|}{\|x\|}
$$
Moreover, if $\delta$ is sufficiently small, then 
$$
\kappa_f(x,\delta) \approx \kappa_f(x)
$$
is a good approximation. It follows that we cannot expect a normwise relative error which is smaller than
$$
\frac{\|f(x)-f(y)\|}{\|f(x)\|} \approx \kappa_f(x) \frac{\|x-y\|}{\|x\|}.
$$
We now consider the problem of computing the normwise relative condition number.  Specifically, if $f$ is differentiable at $x \in \Omega$ and $x \not = 0$ and $f(x) \not = 0$, then the normwise relative condition number is given by
\begin{equation}
    \kappa_f(x) = \frac{\|Df(x)\|\|x\|}{\|f(x)\|}.
\end{equation}
where $Df(x) \in \mathbb{R}^{m \times n}$ is the Jacobian of $f$ at the point $x$ and
$$
\forall A \in \mathbb{R}^{m \times n}  \: : \: \|A\| = \sup \{ \|Az\| \: : \:  \: \|z\| \leq 1\} 
$$
is the matrix norm induced by our vector norm.

Now the Wikipedia entry is sloppy because it allows division by zero when defining the normwise condition numbers. In the definition of the auxiliary function $\kappa_f(x,\delta)$ above we avoid this mess. Alfio Quarteroni's choice is unconventional, because he defines $\kappa_f(x,\delta)$ as the condition number and uses $\kappa_f(x)$ as an approximation. I can understand the need to compute $\kappa_f(x,\delta)$ in a practical application. If $\delta$ is not sufficiently small, then $$\kappa_f(x,\delta) \approx \kappa_f(x)$$
is not necessarily a good approximation and it far more valuable to know $\kappa_f(x,\delta)$ simply because
$$ \frac{\|f(x)-f(y)\|}{\|f(x)\|} \leq \kappa_f(x,\delta) \frac{\|x-y\|}{\|x\|}.$$
However, I disagree with Quarteroni's choice for pedagogical reasons. In general, I do not believe that one should deviate from the standard definition of key concepts. At the very least one should motivate and discuss the new choice. I cannot find such a discussion in his book.

Finally, let $f : \mathbb{R}^{n+m} \rightarrow \mathbb{R}^m$ and consider the problem of solving the nonlinear equation 
$$f(x,y) = 0$$
with respect to $y \in \mathbb{R}^m$. Frequently, there exist $\Omega \subseteq \mathbb{R}^n$ and a function $g : \Omega \rightarrow \mathbb{R}^m$ such that 
$$ \forall x \in \Omega \: : \: f(x,g(x)) = 0.$$ This function is exactly what we need to solve the original equation. Now if we want to measure the sensitivity of $y$ to changes of $x$, then we need only compute the appropriate condition number of $g$. 

Two good references are:

The paper "Mixed, componentwise and structured condition numbers" by Israel Gohberg and Israel Koltracht. SIAM Journal of Matrix Analysis and Applications, 1993, Vol. 14, No. 3 : pp. 688-704 https://doi.org/10.1137/0614049. The treatment of condition number is very systematic. 
The textbook "Accuracy and stability of numerical algorithms" by Nichlas J. Higham. SIAM 2002. https://doi.org/10.1137/1.9780898718027. This is a good book on many different topics. The treatment of the conditioning of linear systems is excellent. 

Do notice that Higham defines condition numbers slightly differently than Gohberg and Israel. It is not a big difference in the end, but be wary of it anyway.
{"set_name": "stack_exchange", "score": 1, "question_id": 3639021}