Step and switch-like functions can be thought of as deterministic switches at some threshold, so smooth sigmoidal-like functions describe that switch with uncertainty around that value.  The machine learning literature discusses 'activation functions' extensively, but the choices are usually justified by application results. I am not aware of an argument for particular functions form first principles. The heaviside function is sometimes approximated by $H(x) \approx \frac{1}{2} + \frac{1}{2}tanh(rx)$, and I think the gaussian CDF has a more immediate probabilistic interpretation. Since probability distributions can be described as arising from idealized processes (e.g. gaussian from brownian motion, Gamma from sequence of exponential waiting times, binomial from repeated sampling with a fixed probability) my question is whether there is a canonical choice, or canonical choices depending on the process that generates the step function. EDIT The question is not whether there are continuous approximations to the step or Heaviside functions, nor whether there are functions that are popular or commonly used (so it's different from ) as an alternative to the discontinuous step. The question is whether there are functions that represent probabilistic switch-like behavior and arise naturally from probabilistic reasoning (if not in general, for specific processes that generate some sort of switch).
Why is the de-facto standard sigmoid function, $\frac{1}{1+e^{-x}}$, so popular in (non-deep) neural-networks and logistic regression?   Why don't we use many of the other derivable functions, with faster computation time or slower decay (so vanishing gradient occurs less). Few examples are . One of my favorites with slow decay and fast calculation is $\frac{x}{1+|x|}$.  EDIT  The question is different to  as I'm only interested in the 'why' and only for the sigmoid.
It's common in C++ to name member variables with some kind of prefix to denote the fact that they're member variables, rather than local variables or parameters. If you've come from an MFC background, you'll probably use m_foo. I've also seen myFoo occasionally.  C# (or possibly just .NET) seems to recommend using just an underscore, as in _foo. Is this allowed by the C++ standard? Why are all the textures in my file pink? I made a character, and its one of my first times texturing, and I saved my character into a new folder, and when I came back to the model ALL the textures were pink, I used to know what to do, but I stopped texturing, how do I bring my textures back? How do I install the latest version of node.js? How do I install the latest  on Ubuntu? I've been looking around, and I can't find anything. Is there a Ubuntu package for node.js, or do I have to compile it myself? Making a private method public to unit test it...good idea? Why an abstract class implementing an interface can miss the declaration/implementation of one of the interface's methods? A curious thing happens in Java when you use an abstract class to implement an interface: some of the interface's methods can be completely missing (i.e. neither an abstract declaration or an actual implementation is present), but the compiler does not complain.  For example, given the interface:  public interface IAnything {   void m1();   void m2();   void m3(); }   the following abstract class gets merrily compiled without a warning or an error:  public abstract class AbstractThing implements IAnything {   public void m1() {}   public void m3() {} }   Can you explain why? Why is DML not allowed in Constructor? I want to define \ext to behave exactly like \lim.  I tried \newcommand{\ext}{{\mathrm{ext}}\,} but that doesn't behave exactly like \lim. Determining direction of longest segment of polygon using ArcGIS Desktop? I have a shapefile with 9000 agriculture fields and what I like to do is to determine for each field (=each polygon) the direction of the longest segment.  For instance, the selected polygon in the screenshot has its longest segment from west to east so I would like to mark give the polygon the attribute W (for west) or either E (for east).   Is there any solution to solve my problem in a fast but generalized way?   I do not need very detailed results. It's just for the interpretation of a question.   I'm seeking an answer using ArcGIS 10.2 for Desktop.  To be more precise on what I'm looking for I try to explain more practical: I'm doing winderosion analysis for a specific area with 9000 fields. As result I have a potential susceptibility for each field. I like to check, if the fields with high susceptibility are correlated to a certain orientation of the fields, e.g. check if most of the fields with high risk are oriented in the main winddirection or not.   The search for the longest segment is not really important for me. It was just an idea I had to determine the direction. So forget about the longest segment if there is a way to determine the direction with another method ;) What does accepting an answer actually convey (2015 edition)? My new laptop came preinstalled with Windows 8. Naively, as I am, I just formatted the harddrive and installed fine old Ubuntu. Now I want to install Windows 8 for dual boot again, but I have no DVD and do download the ISO one needs a product key. That key is not on the back of the laptop anymore but somewhere on the mainboard.  Is there any way to recover the product key from the mainboard using Ubuntu? Finite group for which $|\{x:x^m=e\}|\leq m$ for all $m$ is cyclic. "Facebook" could not be downloaded due to an error. (495)   How do I solve this error? Is a switches voltage and current rating a hard limit? Does the War Caster feat grant ranged spell attacks in melee range without disadvantage? An opportunity attack is described this way (PHB, p. 195):     You can make an opportunity attack when a hostile creature that you can see moves out of your reach. To make the opportunity attack, you use your reaction to make one melee attack against the provoking creature.   The last benefit of the War Caster feat says (PHB, p. 170):        When a hostile creature’s movement provokes an opportunity attack from you, you can use your reaction to cast a spell at the creature, rather than making an opportunity attack. The spell must have a casting time of 1 action and must target only that creature.      Without the Crossbow Expert feat, all ranged attacks (including ranged spell attacks) made when an enemy is adjacent suffer this penalty (PHB, p. 195):     You have disadvantage on a ranged attack roll if you are within 5 feet of a hostile creature that can see you and that isn’t incapacitated.   As an opportunity attack normally grants a melee attack, does it seem reasonable to assume that the target remains at melee range for the spell attack granted by War Caster? If so, does this require ranged spell attack rolls to be made with disadvantage?  The trigger for an OA is a creature moving "out of your reach". This suggests to me that the creature is out of the 5' disadvantage zone, but it seems like that would preclude making a melee spell attack.  Do characters with the War Caster feat get the best of both worlds: being  allowed to make either a melee spell attack or a non-disadvantaged ranged spell attack? Uniform vs discrete Gaussian sampling in Ring learning with errors The Wikipedia article on  mentions two methods of sampling "small" polynomials namely uniform sampling and discrete Gaussian sampling. Uniform sampling is clearly the simplest, involving simply uniformly selecting the coefficients from the set of small coefficients, thereby guaranteeing they will be small. However, the articles mention that security proofs rely on discrete Gaussian sampling.  However, I have a bit of difficulty wrapping my head around    Why discrete Gaussian sampling would be better than uniform, security-wise?   I mean, usually uniform is preferable - it maximizes the entropy/information in each sample, making the outcome more unpredictable. I could understand it if the Gaussian sampling technique would somehow serve to avoid (at least with high probability) "weak" parts of the space of outcomes but I have found no information/justification that this is the case.  Another reason I can see why discrete Gaussian might be better is that there is, after all, a chance (albeit a small one) one would select a non-small coefficient, so the space of outcomes is actually larger. However, given that this probability of hitting this extra space is (by design) quite small but also that one would likely immediately reject such a sample (because the algorithm calls for a small polynomial), this seems to be irrelevant.  It is not because I am against discrete Gaussian sampling I just want to understand why it is more secure. Also, I want to understand if it really IS more secure or it is simply easier to prove security in that model. I would actually consider that very interesting if it is the case because it raises the question of why is it easier to prove the security under conditions that are intuitively worse (i.e. discrete Gaussian vs the intuitively better uniform case)? If it is so intuitively clear that uniform sampling is better, surely it must be possible to transform that intuition into a security proof of RWE also under uniform sampling. Otherwise, I would fear the intuition must with high likelihood be wrong or at least dubious. How do high traffic sites service more than 65535 TCP connections? Let $X, Y$ be independent standard normal random variables.  We already know that $X+Y$ is normal with mean 0 and variance 2. However, I am trying to prove this result in a slightly different way than usual, i.e., using convolutions.  We can address the problem in the following way: the sum of the two variables has a density $h(x)$ given by the convolution product of the densities of the two variables, say $f$ and $g$, so that   $$ h(x) = \int_{-\infty}^{\infty} f(x-y)g(y)dy. $$  However,  $$ f(y) = g(y) = \frac{1}{\sqrt {2\pi}}\exp(-y^2/2). $$  Hence now the integral becomes:  $$ h(x) =\frac{1}{2\pi}\int_{-\infty}^{\infty}\exp(-(x-y)^2/2)\exp(-y^2/2)dy. $$  However, I am puzzled here. How should we go on from here? I cannot see how to transform this integral into the integral of a normal r.v of type $N(0,2)$. Thanks in advance for your help. Why does my DIY "solar panel circuit" show battery voltage and not solar panel voltage? Differences Between Row Echelon and Reduced Row Echelon Let $$A =\begin{bmatrix} 1&amp;3&amp;-6&amp;2\\5&amp;12&amp;-24&amp;7\\9&amp;15&amp;-30&amp;6\end{bmatrix}$$  $$RREF(A) =\begin{bmatrix} 1&amp;0&amp;0&amp;-1\\0&amp;1&amp;-2&amp;-1\\0&amp;0&amp;0&amp;0\end{bmatrix}$$ $$REF(A) =\begin{bmatrix} 1&amp;3&amp;-6&amp;2\\0&amp;1&amp;-2&amp;1\\0&amp;0&amp;0&amp;0\\\end{bmatrix}$$  Now, provided this information, it can be found that the basis for the nullspace of $A$ using the $RREF(A)$ is: $$\begin{pmatrix} \begin{bmatrix} 0\\2\\1\\0 \end{bmatrix}, \begin{bmatrix} 1\\-1\\0\\1 \end{bmatrix}\end{pmatrix}$$  Using the $REF(A)$ the "basis" for the nullspace is:  $$\begin{pmatrix} \begin{bmatrix} 6\\2\\1\\0 \end{bmatrix}, \begin{bmatrix} -2\\-1\\0\\1 \end{bmatrix}\end{pmatrix}$$  Is the "basis" that I provided for the $REF(A)$ actually a basis? How does the  $REF$ and $RREF$ of a matrix differ? Why would layout_width="wrap_content" take up the entire width of the screen?