Can anyone provide me the mathematical details to why we shouldn't initialize a neural network (single layer ANN) weights to zero? In multi-layer we don't initialize it to zero because of loss of symmetry but why can't we initialize a single layer network weights to zero ?
As far as I understand, one of the main claimed problems with initializing e.g. a feed-forward neural network (with several $\text{tanh}$ or $\text{ReLU}$ layers) with $W=0$ is that it doesn't break "network symmetry", meaning, backpropagation would propagate the same error through all such units (i.e. "nudging all weights in the same direction"). This is, I presume, undesirable because we would not be learning "different" calculations through different paths of the network.  However, I'm confused why that even matters in this case given the fact that if $W$ ever drops to 0, we will effectively be propagating no gradients at all through the network, since W=0 would multiply all errors from the output and prevent any learning.  Put another way, even if $W=0$ does not break network symmetry (wasting calculations and paths in the network) is it correct to say that if $W=0$ (e.g. by initialization) we are effectively killing gradients in a neural network, and thus no learning can take place?
Why don't the White Walkers take the sea route to Westeros Cartographic techniques for symbolizing routing data windows bat file error How do I protect random people pointing their domain to my server? Is there any way to verify a cashier cheque? There are plenty of warnings around about forged cashier's cheques, and that if you deposit a forged one your bank will "hold you responsible".   What does it mean that the bank will "hold you responsible"? Clearly the money won't be credited. Will you be charged a fee like an NSF cheque? Will they try to prosecute you? Is there any way to confirm whether a cashiers cheque is forged? Could you for example tell your bank that you were uncertain of the origin, and ask them to confirm it? Can you call the issuer and verify (from the serial number) that they did in fact issue a cheque for that amount with you as the payee? Cloning an SD card onto a larger SD card I'm using Ubuntu 12.04. I have an old 4GB SD card, and have just bought a new 16GB SD card. I would like to copy everything from my old 4GB SD card onto the new 16GB SD card. I was afraid that the usual copy-and-paste trick would miss out something, so I wanted to clone the old 4GB SD card onto the 16GB SD card. I used the dd command in a shell, following instructions in .  This worked with one small caveat. The new SD card now appeared to be 4GB. I later found out this is because the primary partition on the 16GB SD card has now shrunk to 4GB. I have solved this problem, and I think I understand it. However, my question now is, how do I clone my old 4GB SD card onto my new 16GB SD card without making the primary partition on the 16GB SD card shrink to 4GB? How is Cycles different from Blender Internal? Callback functions in Java Subsurf modifier rounding sharp edges I tried to apply a Subdivision Surface on a part of my object. I can't keep the angles sharp, they all turn into rounded angles.  The angle before applying the Subdivision surface...   ...And after Does anyone know any example that invalidates the following affirmation:     If a morphism $f:A\to A$ induces the identity $\hat f:\operatorname{Spec} \left( A \right) \to \operatorname{Spec} \left( A \right)$ then $f = \operatorname{id} _A $. Given $\int_0^{\infty}e^{-x^2}dx = \frac{\sqrt{\pi}}{2}$, evaluate $\int_0^{\infty}e^{-a^2x^2-\frac{b^2}{x^2}}dx $ Given $$\int_0^{\infty}e^{-x^2}dx = \frac{\sqrt{\pi}}{2}$$ evaluate:   $$\int_0^{\infty}e^{-a^2x^2-\frac{b^2}{x^2}}dx. $$   I can find that  $$\left(ax+\frac{b}{x}\right)^2 = a^2x^2+2ab+\frac{b^2}{x^2}$$  therefore: $$\int_0^{\infty}e^{-a^2x^2-\frac{b^2}{x^2}}dx = e^{2ab}\int_0^{\infty}e^{-\left(ax+\frac{b}{x}\right)^2}dx$$  but I can't find any clue then. How do I transfer a file from a guest account on a MacBook Air? Continuous weighted random distribution, biased toward one end I am currently contributing to a particle system for our game and developing some emitter shapes.  My uniform random distribution along a line or along a rectangular area works fine - no problem.  But now I would like to have something like a 1 dimensional gradient in this distribution. This would mean for example lower values are more common than higher values.  I don't know what would be appropriate mathematical terms for this problem, so my search skills are rather useless with this one. I need something that is computationally simple, as the particle system needs to be efficient. The USB has various purposes for data transfer, movement of input and output signals for hardware like connecting an Android device to another Display screen (USB on the go), charging the device etc.(during a device charge the voltage drawn from the USB is same irrespective of the device or OEM then Why are two USB Cables different ?  A USB of one manufacturer may not be supporting another manufacturer. We get problems if we do connect them like      or Why is dark matter the best theory available to explain missing mass problems? Why is dark matter the best theory to explain the missing mass problem?    Why is dark matter mathematically necessary to explain the missing mass problem?    On a side not I believe dark matter is definately intuitive the next logical step in physics is to accept the fact the direct observations can not always be the determining factor in the acceptance of a theory.  Positivism is not going to be the only driving force in physics forevermore.    One theory that seems almost as good is the supersymmetric particle. What determination which are actually detectible and how would we distinguish the differences between the two?    References:   . "Neither Michael nor Albert is correct" or "Neither Michael nor Albert are correct"? What is the correct sentence?  Neither Michael nor Albert is correct.   Neither Michael nor Albert are correct. Calculate battery life How to batch watermark JPG files? I have around 15,000 JPG files that need to be watermarked. Is there any way to process these images en masse and add a watermark?  The images may be different sizes, I want the watermark placed in the center of the images. Is it possible to use multiple desktop environments on same system? Limit of some specific "almost Riemann" sums I'm just beginning my journey with calculus, and this problem is giving me hard time.  $$\lim_{n \to \infty} \sum_{k=1}^n \frac{k}{{n^2+k}}$$  I calculated first three sums: $$\frac{1}{2}, \frac{16}{30}, \frac{117}{220}$$ From that I guessed that the limit is $$\frac{1}{2}$$. The hard part, though, is proving it.    I reckon that the squeeze theorem should be used here, but I can't seem to find anything useful. (The only sequences I was able to figure out give me limits of 1 and 0, so it's not something I can use)  Maybe you guys can give me some ideas or theorems that can be of use here? I know that this problem could be probably solved using some more advanced calculus techniques, but I'm not familiar with derivatives and integrals yet, and this problem is in the limits section of my problem book, so I'm not supposed to use anything more advanced.