View source for Fréchet derivative

{{Short description|Derivative defined on normed spaces}}
{{distinguish|Differentiation in Fréchet spaces}}
{{Calculus}}
In [[mathematics]], the '''Fréchet derivative''' is a [[derivative]] defined on [[normed space]]s. Named after [[Maurice René Fréchet|Maurice Fréchet]], it is commonly used to generalize the derivative of a [[real-valued function]] of a single real variable to the case of a [[vector-valued function]] of multiple real variables, and to define the [[functional derivative]] used widely in the [[calculus of variations]].
Generally, it extends the idea of the derivative from real-valued [[Function (mathematics)|functions]] of one real variable to functions on normed spaces. The Fréchet derivative should be contrasted to the more general [[Gateaux derivative]] which is a generalization of the classical [[directional derivative]].
The Fréchet derivative has applications to nonlinear problems throughout [[mathematical analysis]] and physical sciences, particularly to the calculus of variations and much of nonlinear analysis and [[nonlinear functional analysis]].
== Definition ==
Let <math>V</math> and <math>W</math> be [[normed vector space]]s, and <math>U\subseteq V</math> be an [[open subset]] of <math>V.</math> A function <math>f : U \to W</math> is called ''Fréchet differentiable'' at <math>x \in U</math> if there exists a [[bounded linear operator]] <math>A:V\to W</math> such that
<math display=block>\lim_{\|h\|_{V} \to 0} \frac{\| f(x + h) - f(x) - Ah\|_W}{\|h\|_V} = 0.</math>
The [[Limit (mathematics)|limit]] here is meant in the usual sense of a [[limit of a function]] defined on a metric space (see [[Limit of a function#Functions on metric spaces|Functions on metric spaces]]), using <math>V</math> and <math>W</math> as the two metric spaces, and the above expression as the function of argument <math>h</math> in <math>V.</math> As a consequence, it must exist for all [[sequence]]s <math>\langle h_n\rangle_{n=1}^{\infty}</math> of non-zero elements of <math>V</math> that converge to the zero vector <math>h_n \to 0.</math> Equivalently, the first-order expansion holds, in [[Little-o_notation | Landau notation]]
<math display=block>f(x + h) = f(x) + Ah +o(h).</math>
If there exists such an operator <math>A,</math> it is unique, so we write <math>Df(x) = A</math> and call it the ''Fréchet derivative'' of <math>f</math> at <math>x.</math>
A function <math>f</math> that is Fréchet differentiable for any point of <math>U</math> is said to be C<sup>1</sup> if the function
<math display=block>Df : U \to B(V,W) ; x \mapsto Df(x)</math>
is continuous (<math>B(V,W)</math> denotes the space of all bounded linear operators from <math>V</math> to <math>W</math>). Note that this is not the same as requiring that the map <math>Df(x) : V \to W</math> be continuous for each value of <math>x</math> (which is assumed; bounded and continuous are equivalent).
This notion of derivative is a generalization of the ordinary derivative of a function on the [[real number]]s <math>f : \R \to \R</math> since the linear maps from <math>\R</math> to <math>\R</math> are just multiplication by a real number. In this case, <math>D f(x)</math> is the function <math>t \mapsto f'(x)t.</math>
== Properties ==
A function differentiable at a point is continuous at that point.
Differentiation is a linear operation in the following sense: if <math>f</math> and <math>g</math> are two maps <math>V \to W</math> which are differentiable at <math>x,</math> and <math>c</math> is a scalar (a real or [[complex number]]), then the Fréchet derivative obeys the following properties:
<math display=block>D(cf)(x) = cDf(x)</math>
<math display=block>D(f+g)(x) = Df(x) + Dg(x).</math>
The [[chain rule]] is also valid in this context: if <math>f : U \to Y</math> is differentiable at <math>x \in U,</math> and <math>g : Y \to W</math> is differentiable at <math>y = f(x),</math> then the composition <math>g \circ f</math> is differentiable in <math>x</math> and the derivative is the [[Function composition|composition]] of the derivatives:
<math display=block>D(g \circ f)(x) = Dg(f(x)) \circ Df(x).</math>
== Finite dimensions ==
The Fréchet derivative in finite-dimensional spaces is the usual derivative. In particular, it is represented in coordinates by the [[Jacobian matrix and determinant|Jacobian matrix]].
Suppose that <math>f</math> is a map, <math>f : U \subseteq \R^n \to \R^m</math> with <math>U</math> an open set. If <math>f</math> is Fréchet differentiable at a point <math>a \in U,</math> then its derivative is
<math display=block>\begin{cases} Df(a) : \R^n \to \R^m \\ Df(a)(v) = J_f(a) v \end{cases}</math>
where <math>J_f(a)</math>denotes the Jacobian matrix of <math>f</math> at <math>a.</math>
Furthermore, the partial derivatives of <math>f</math> are given by
<math display=block>\frac{\partial f}{\partial x_i}(a) = Df(a)(e_i) = J_f(a) e_i,</math>
where <math>\left\{ e_i \right\}</math> is the canonical basis of <math>\R^n.</math> Since the derivative is a linear function, we have for all vectors <math>h \in \R^n</math> that the [[directional derivative]] of <math>f</math> along <math>h</math> is given by
<math display=block>Df(a)(h) = \sum_{i=1}^n h_i \frac{\partial f}{\partial x_i}(a).</math>
If all partial derivatives of <math>f</math> exist and are continuous, then <math>f</math> is Fréchet differentiable (and, in fact, C<sup>1</sup>). The converse is not true; the function
<math display=block>f(x, y) = \begin{cases} (x^2+y^2)\sin \left ((x^2+y^2)^{-1/2} \right ) & (x, y) \neq (0, 0)\\ 0 & (x, y) = (0, 0) \end{cases}</math>
is Fréchet differentiable and yet fails to have continuous partial derivatives at <math>(0, 0).</math>
== Example in infinite dimensions ==
One of the simplest (nontrivial) examples in infinite dimensions, is the one where the domain is a [[Hilbert space]] (<math>H</math>) and the function of interest is the norm. So consider <math>\|\,\cdot\,\| : H \to \R.</math>
First assume that <math>x \neq 0.</math> Then we claim that the Fréchet derivative of <math>\|\cdot\|</math> at <math>x</math> is the linear functional <math>D,</math> defined by
<math display=block>Dv := \left\langle \frac{x}{\|x\|}, v\right\rangle.</math>
Indeed,
<math display=block>\begin{align}
\frac{|\|x+h\|-\|x\|-Dh|}{\|h\|} &=\frac{|\|x\|\|x+h\|-\langle x,x\rangle-\langle x,h\rangle|}{\|x\|\|h\|} \\[8pt]
&=\frac{|\|x\|\|x+h\|-\langle x,x+h\rangle|}{\|x\|\|h\|} \\[8pt]
&=\frac{|\langle x,x\rangle \langle x+h,x+h\rangle-\langle x,x+h\rangle^2|}{\|x\|\|h\|(|\|x\|\|x+h\|+\langle x,x+h\rangle|)} \\[8pt]
&=\frac{\langle x,x \rangle \langle h,h \rangle-\langle x,h \rangle^2}{\|x\|\|h\|(|\|x\|\|x+h\|+\langle x,x+h\rangle|)} \\
&{}
\end{align}</math>
Using continuity of the norm and inner product we obtain:
<math display=block>\begin{align}
\lim_{\|h\|\to 0}\frac{|\|x+h\|-\|x\|-Dh|}{\|h\|} &= \lim_{\|h\|\to 0} \frac{\langle x,x \rangle \langle h,h \rangle-\langle x,h \rangle^2}{\|x\|\|h\|(|\|x\|\|x+h\|+\langle x,x+h\rangle|)} \\[8pt]
&= \frac{1}{2\|x\|^3}\lim_{\|h\|\to 0}\frac{\langle x,x \rangle \langle h,h \rangle-\langle x,h \rangle^2}{\|h\|} \\[8pt]
&= \frac{1}{2\|x\|^3}\lim_{\|h\|\to 0}\left(\langle x,x\rangle\|h\|-\langle x,h\rangle \left\langle x,\frac{h}{\|h\|}\right\rangle\right) \\[8pt]
&= \frac{1}{2\|x\|^3}\left(\lim_{\|h\|\to 0}\langle x,x\rangle\|h\|-\lim_{h\to 0}\langle x,h\rangle \left\langle x,\frac{h}{\|h\|}\right\rangle \right ) \\[8pt]
&= \frac{1}{2\|x\|^3}\left(0-\lim_{\|h\|\to 0}\langle x,h\rangle \left\langle x,\frac{h}{\|h\|}\right\rangle \right ) \\[8pt]
&= -\frac{1}{2\|x\|^3}\left(\lim_{\|h\|\to 0}\langle x,h\rangle \left\langle x,\frac{h}{\|h\|}\right\rangle \right ) \\[8pt]
\end{align}</math>
As <math>\|h\|\to 0, \langle x,h\rangle \to 0</math> and because of the [[Cauchy Schwarz|Cauchy-Schwarz]] inequality
<math display=block>\left\langle x,\frac{h}{\|h\|}\right\rangle</math>
is bounded by <math>\|x\|</math> thus the whole limit vanishes.
Now we show that at <math>x=0</math> the norm is not differentiable, that is, there does not exist bounded linear functional <math>D</math> such that the limit in question to be <math>0.</math> Let <math>D</math> be any linear functional. [[Riesz representation theorem|Riesz Representation Theorem]] tells us that <math>D</math> could be defined by <math>Dv = \langle a, v\rangle</math> for some <math>a \in H.</math> Consider
<math display=block>A(h) = \frac{|\|0+h\| - \|0\|-Dh|}{\|h\|} = \left|1-\left\langle a, \frac{h}{\|h\|}\right\rangle\right|.</math>
In order for the norm to be differentiable at <math>0</math> we must have
<math display=block>\lim_{\|h\|\to 0} A(h) = 0.</math>
We will show that this is not true for any <math>a.</math> If <math>a = 0</math> obviously <math>A(h) = 1</math> independently of <math>h,</math> hence this is not the derivative. Assume <math>a\neq 0.</math> If we take <math>h</math> tending to zero in the direction of <math>-a</math> (that is, <math>h = t\cdot(-a),</math> where <math>t \to 0^+</math>) then <math>A(h) = |1+\|a\|| > 1 > 0,</math> hence
<math display=block>\lim_{\|h\| \to 0} A(h) \neq 0</math>
(If we take <math>h</math> tending to zero in the direction of <math>a</math> we would even see this limit does not exist since in this case we will obtain <math>|1-\|a\||</math>).
The result just obtained agrees with the results in finite dimensions.
== Relation to the Gateaux derivative ==
A function <math>f : U \subseteq V \to W</math> is called ''[[Gateaux derivative|Gateaux differentiable]]'' at <math>x \in U</math> if <math>f</math> has a directional derivative along all directions at <math>x.</math> This means that there exists a function <math>g : V \to W</math> such that
<math display=block>g(v) = \lim_{t \to 0} \frac{f(x + tv) - f(x)}{t}</math>
for any chosen vector <math>v \in V,</math> and where <math>t</math> is from the scalar field associated with <math>V</math> (usually, <math>t</math> is [[Real numbers|real]]).<ref>It is common to include in the definition that the resulting map <math>g</math> must be a [[continuous linear operator]]. We avoid adopting this convention here to allow examination of the widest possible class of pathologies.</ref>
If <math>f</math> is Fréchet differentiable at <math>x,</math> it is also Gateaux differentiable there, and <math>g</math> is just the linear operator <math>A = D f(x).</math>
However, not every Gateaux differentiable function is Fréchet differentiable. This is analogous to the fact that the existence of all directional derivatives at a point does not guarantee total differentiability (or even continuity) at that point.
For example, the real-valued function <math>f</math> of two real variables defined by
<math display=block>f(x, y) = \begin{cases} \frac{x^3}{x^2+y^2} & (x, y) \neq (0, 0)\\ 0 & (x, y) = (0, 0) \end{cases}</math>
is continuous and Gateaux differentiable at the origin <math>(0, 0)</math>, with its derivative at the origin being
<math display=block>g(a, b) = \begin{cases} \frac{a^3}{a^2+b^2}& (a, b) \neq (0, 0)\\ 0 & (a, b) = (0, 0) \end{cases}</math>
The function <math>g</math> is not a linear operator, so this function is not Fréchet differentiable.
More generally, any function of the form <math>f(x, y) = g(r) h(\phi),</math> where <math>r</math> and <math>\phi</math> are the [[polar coordinates]] of <math>(x, y),</math> is continuous and Gateaux differentiable at <math>(0, 0)</math> if <math>g</math> is differentiable at <math>0</math> and <math>h(\phi + \pi) = -h(\phi),</math> but the Gateaux derivative is only linear and the Fréchet derivative only exists if <math>h</math> is [[sinusoidal]].
In another situation, the function <math>f</math> given by
<math display=block>f(x, y) = \begin{cases} \frac{x^3y}{x^6+y^2} & (x, y) \neq (0, 0)\\ 0 & (x, y) = (0, 0) \end{cases}</math>
is Gateaux differentiable at <math>(0, 0),</math> with its derivative there being <math>g(a, b) = 0</math> for all <math>(a, b),</math> which {{em|is}} a linear operator. However, <math>f</math> is not continuous at <math>(0, 0)</math> (one can see by approaching the origin along the curve <math>\left(t, t^3\right)</math>) and therefore <math>f</math> cannot be Fréchet differentiable at the origin.
A more subtle example is
<math display=block>f(x, y) = \begin{cases} \frac{x^2y}{x^4+y^2}\sqrt{x^2+y^2} & (x, y) \neq (0, 0)\\ 0 & (x, y) = (0, 0) \end{cases}</math>
which is a continuous function that is Gateaux differentiable at <math>(0, 0),</math> with its derivative at this point being <math>g(a, b) = 0</math> there, which is again linear. However, <math>f</math> is not Fréchet differentiable. If it were, its Fréchet derivative would coincide with its Gateaux derivative, and hence would be the zero operator <math>A = 0</math>; hence the limit
<math display=block>\lim_{\|h\|_2 \to 0} \frac{|f((0, 0) + h) - f(0, 0) - Ah|}{\|h\|_2} = \lim_{h = (x,y) \to (0,0)} \left|\frac{x^2y}{x^4+y^2}\right|</math>
would have to be zero, whereas approaching the origin along the curve <math>\left(t, t^2\right)</math> shows that this limit does not exist.
These cases can occur because the definition of the Gateaux derivative only requires that the [[difference quotient]]s converge along each direction individually, without making requirements about the rates of convergence for different directions. Thus, for a given ε, although for each direction the difference quotient is within ε of its limit in some neighborhood of the given point, these neighborhoods may be different for different directions, and there may be a sequence of directions for which these neighborhoods become arbitrarily small. If a sequence of points is chosen along these directions, the quotient in the definition of the Fréchet derivative, which considers all directions at once, may not converge. Thus, in order for a linear Gateaux derivative to imply the existence of the Fréchet derivative, the difference quotients have to [[Uniform convergence|converge uniformly]] for all directions.
The following example only works in infinite dimensions. Let <math>X</math> be a Banach space, and <math>\varphi</math> a [[linear functional]] on <math>X</math> that is ''discontinuous'' at <math>x = 0</math> (a [[discontinuous linear functional]]). Let
<math display=block>f(x) = \|x\| \varphi(x).</math>
Then <math>f(x)</math> is Gateaux differentiable at <math>x = 0</math> with derivative <math>0.</math> However, <math>f(x)</math> is not Fréchet differentiable since the limit
<math display=block>\lim_{x \to 0} \varphi(x)</math>
does not exist.
== Higher derivatives ==
If <math>f : U \to W</math> is a differentiable function at all points in an open subset <math>U</math> of <math>V,</math> it follows that its derivative
<math display=block>D f : U \to L(V, W)</math>
is a function from <math>U</math> to the space <math>L(V, W)</math> of all bounded linear operators from <math>V</math> to <math>W.</math> This function may also have a derivative, the ''second order derivative'' of <math>f,</math> which, by the definition of derivative, will be a map
<math display=block>D^2 f : U \to L(V, L(V, W)).</math>
To make it easier to work with second-order derivatives, the space on the right-hand side is identified with the Banach space <math>L^2(V \times V, W)</math> of all continuous [[bilinear map]]s from <math>V</math> to <math>W.</math> An element <math>\varphi</math> in <math>L(V, L(V, W))</math> is thus identified with <math>\psi</math> in <math>L^2(V \times V, W)</math> such that for all <math>x, y \in V,</math>
<math display=block>\varphi(x)(y) = \psi(x, y).</math>
(Intuitively: a function <math>\varphi</math> linear in <math>x</math> with <math>\varphi(x)</math> linear in <math>y</math> is the same as a bilinear function <math>\psi</math> in <math>x</math> and <math>y</math>).
One may differentiate
<math display=block>D^2 f : U \to L^2(V\times V, W)</math>
again, to obtain the ''third order derivative'', which at each point will be a ''trilinear map'', and so on. The <math>n</math>-th derivative will be a function
<math display=block>D^n f : U \to L^n(V \times V \times \cdots \times V, W),</math>
taking values in the Banach space of continuous [[multilinear map]]s in <math>n</math> arguments from <math>V</math> to <math>W.</math> Recursively, a function <math>f</math> is <math>n+1</math> times differentiable on <math>U</math> if it is <math>n</math> times differentiable on <math>U</math> and for each <math>x \in U</math> there exists a continuous multilinear map <math>A</math> of <math>n+1</math> arguments such that the limit
<math display=block>\lim_{h_{n+1} \to 0} \frac{ \left\| D^nf \left(x + h_{n+1} \right)(h_1, h_2, \ldots, h_n) - D^nf(x)(h_1, h_2, \ldots, h_n) - A \left(h_1, h_2, \ldots, h_n, h_{n+1} \right) \right\| }{\left\|h_{n+1}\right\|} = 0</math>
exists [[Uniform convergence|uniformly]] for <math>h_1, h_2, \ldots, h_n</math> in bounded sets in <math>V.</math> In that case, <math>A</math> is the <math>(n+1)</math>st derivative of <math>f</math> at <math>x.</math>
Moreover, we may obviously identify a member of the space <math>L^n\left(V \times V \times \cdots \times V, W\right)</math> with a linear map <math>L\left(\bigotimes_{j=1}^n V_j, W\right)</math> through the identification <math>f\left(x_1, x_2, \ldots, x_n\right) = f\left(x_1 \otimes x_2 \otimes \cdots \otimes x_n\right),</math> thus viewing the derivative as a linear map.
== Partial Fréchet derivatives ==
In this section, we extend the usual notion of [[partial derivative]]s which is defined for functions of the form <math>f: \R^n \to \R,</math> to functions whose domains and target spaces are arbitrary (real or complex) [[Banach space]]s. To do this, let <math>V_1, \ldots, V_n</math> and <math>W</math> be Banach spaces (over the same field of scalars), and let <math display=inline>f : \prod_{i=1}^n V_i \to W</math> be a given function, and fix a point <math display=inline>a = \left(a_1, \ldots, a_n\right) \in \prod_{i=1}^n V_i.</math> We say that <math>f</math> has an i-th partial differential at the point <math>a</math> if the function <math>\varphi_i : V_i \to W</math> defined by
<math display=block>\varphi_i(x) = f(a_1, \ldots, a_{i-1}, x, a_{i+1}, \ldots a_n)</math>
is Fréchet differentiable at the point <math>a_i</math> (in the sense described above). In this case, we define <math>\partial_if(a) := D\varphi_i(a_i),</math> and we call <math>\partial_if(a)</math> the i-th partial derivative of <math>f</math> at the point <math>a.</math> It is important to note that <math>\partial_if(a)</math> is a linear transformation from <math>V_i</math> into <math>W.</math> Heuristically, if <math>f</math> has an i-th partial differential at <math>a,</math> then <math>\partial_if(a)</math> linearly approximates the change in the function <math>f</math> when we fix all of its entries to be <math>a_j</math> for <math>j \neq i,</math> and we only vary the i-th entry. We can express this in the Landau notation as
<math display=block>f(a_1, \ldots, a_i+h, \ldots a_n) - f(a_1, \ldots, a_n) = \partial_if(a)(h) + o(h).</math>
== Generalization to topological vector spaces ==
The notion of the Fréchet derivative can be generalized to arbitrary [[topological vector space]]s (TVS) <math>X</math> and <math>Y.</math> Letting <math>U</math> be an open subset of <math>X</math> that contains the origin and given a function <math>f: U \to Y</math> such that <math>f(0) = 0,</math> we first define what it means for this function to have 0 as its derivative. We say that this function <math>f</math> is tangent to 0 if for every open neighborhood of 0, <math>W \subseteq Y</math> there exists an open neighborhood of 0, <math>V\subseteq X</math> and a function <math>o: \R \to \R</math> such that
<math display=block>\lim_{t\to 0} \frac{o(t)}{t} = 0,</math>
and for all <math>t</math> in some neighborhood of the origin, <math>f(tV) \subseteq o(t) W.</math>
We can now remove the constraint that <math>f(0) = 0</math> by defining <math>f</math> to be Fréchet differentiable at a point <math>x_0 \in U</math> if there exists a continuous linear operator <math>\lambda : X \to Y</math> such that <math>f(x_0 + h) - f(x_0) - \lambda h,</math> considered as a function of <math>h,</math> is tangent to 0. (Lang p.&nbsp;6)
If the Fréchet derivative exists then it is unique. Furthermore, the Gateaux derivative must also exist and be equal the Fréchet derivative in that for all <math>v \in X,</math>
<math display=block>\lim_{\tau \to 0}\frac{f(x_0 + \tau v) - f(x_0)}{\tau} = f'(x_0) v,</math>
where <math>f'(x_0)</math> is the Fréchet derivative. A function that is Fréchet differentiable at a point is necessarily continuous there and sums and scalar multiples of Fréchet differentiable functions are differentiable so that the space of functions that are Fréchet differentiable at a point form a subspace of the functions that are continuous at that point. The chain rule also holds as does the Leibniz rule whenever <math>Y</math> is an algebra and a TVS in which multiplication is continuous.
==See also==
* {{annotated link|Directional derivative}}
* {{annotated link|Generalizations of the derivative}}
* {{annotated link|Gradient#Fréchet derivative}}
* {{annotated link|Infinite-dimensional holomorphy}}
* {{annotated link|Infinite-dimensional vector function}}
* {{annotated link|Total derivative}}
== Notes ==
{{reflist}}
== References ==
* {{Citation|last=Cartan|first=Henri|authorlink=Henri Cartan| title=Calcul différentiel|publisher=Hermann|location=Paris|mr=0223194|year=1967}}.
* {{Citation|last=Dieudonné|first=Jean |authorlink= Jean Dieudonné| title=Foundations of modern analysis|publisher=[[Academic Press]]|location=Boston, MA|mr=0349288|year=1969}}.
* {{Citation|last=Lang|first=Serge|authorlink=Serge Lang| title=Differential and Riemannian Manifolds|publisher=[[Springer Science+Business Media|Springer]]|isbn=0-387-94338-2|year=1995}}.
* {{Citation|last=Munkres|first=James R.|authorlink=James Munkres| title=Analysis on manifolds|publisher=[[Addison-Wesley]]|isbn=978-0-201-51035-5|mr=1079066|year=1991}}.
* {{Citation|editor1-last=Previato|editor1-first=Emma |editor1-link=Emma Previato| title=Dictionary of applied math for engineers and scientists|publisher=[[CRC Press]]|location=London|series=Comprehensive Dictionary of Mathematics|isbn=978-1-58488-053-0|mr=1966695|year=2003}}.
* {{Citation|editor1-last=Coleman|editor1-first=Rodney|title=Calculus on Normed Vector Spaces|publisher=[[Springer Science+Business Media|Springer]]|series=Universitext|isbn=978-1-4614-3894-6|year=2012}}.
==External links==
* B. A. Frigyik, S. Srivastava and M. R. Gupta, ''[https://web.archive.org/web/20170808093712/http://www.ee.washington.edu/techsite/papers/documents/UWEETR-2008-0001.pdf Introduction to Functional Derivatives]'', UWEE Tech Report 2008-0001.
* http://www.probability.net. This webpage is mostly about basic probability and measure theory, but there is nice chapter about Frechet derivative in Banach spaces (chapter about Jacobian formula). All the results are given with proof.
{{Functional analysis}}
{{Analysis in topological vector spaces}}
{{DEFAULTSORT:Frechet derivative}}
[[Category:Banach spaces]]
[[Category:Generalizations of the derivative]]