I am slightly confused by counting the number of NN parameters. Let's assume there is a NN with 4-dim vector as an input, then comes 5-dim hidden layers, and another one 6-dim hidden layer. There is a single neuron output with sigmoid. Negative log likelihood as a loss function.   The question is how many parameters should be updated on the first step of the gradient descent? (including biases)  My calculation is 4*5+5 then 5*6+6 and 6. But I am not sure about the last 6. I would appreciate any help.
In Bishop's book "Pattern Classification and Machine Learning", it describes a technique for regularization in the context of neural networks. However, I don't understand a paragraph describing that during the training process, the number of degrees of freedom increases along with the model complexity. The relevant quote is the following:     An alternative to regularization as a way of controlling the effective   complexity of a network is the procedure of early stopping. The   training of nonlinear network models corresponds to an iterative   reduction of the error function defined with respect to a set of   training data. For many of the optimization algorithms used for   network training, such as conjugate gradients, the error is a   nonincreasing function of the iteration index. However, the error   measured with respect to independent data, generally called a   validation set, often shows a decrease at first, followed by an increase as the network starts to over-fit. Training can therefore be   stopped at the point of smallest error with respect to the validation   data set, as indicated in Figure 5.12, in order to obtain a network   having good generalization performance. The behaviour of the network   in this case is sometimes explained qualitatively in terms of the   effective number of degrees of freedom in the network, in which this   number starts out small and then to grows during the training process,   corresponding to a steady increase in the effective complexity of the   model.   It also says that the number of parameters grows during the course of training. I was assuming that by "parameters", it refers to the number of weights controlled by the network's hidden units. Maybe I'm wrong because the weights are prevented to increase in magnitude by the regularization process but they don't change in number. Could it be referring to the process of finding a good number of hidden units?  What's a degree of freedom in a neural network? What parameters increase during  training?
Why apt can't find the source code of any package? Python gives us the ability to create 'private' methods and variables within a class by prepending double underscores to the name, like this: __myPrivateMethod(). How, then, can one explain this  >>> class MyClass: ...     def myPublicMethod(self): ...             print 'public method' ...     def __myPrivateMethod(self): ...             print 'this is private!!' ...  >>> obj = MyClass() >>> obj.myPublicMethod() public method >>> obj.__myPrivateMethod() Traceback (most recent call last):   File "", line 1, in  AttributeError: MyClass instance has no attribute '__myPrivateMethod' >>> dir(obj) ['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod'] >>> obj._MyClass__myPrivateMethod() this is private!!   What's the deal?!  I'll explain this a little for those who didn't quite get that.  >>> class MyClass: ...     def myPublicMethod(self): ...             print 'public method' ...     def __myPrivateMethod(self): ...             print 'this is private!!' ...  >>> obj = MyClass()   What I did there is create a class with a public method and a private method and instantiate it.  Next, I call its public method.  >>> obj.myPublicMethod() public method   Next, I try and call its private method.  >>> obj.__myPrivateMethod() Traceback (most recent call last):   File "", line 1, in  AttributeError: MyClass instance has no attribute '__myPrivateMethod'   Everything looks good here; we're unable to call it. It is, in fact, 'private'. Well, actually it isn't. Running dir() on the object reveals a new magical method that python creates magically for all of your 'private' methods.  >>> dir(obj) ['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']   This new method's name is always an underscore, followed by the class name, followed by the method name.  >>> obj._MyClass__myPrivateMethod() this is private!!   So much for encapsulation, eh?  In any case, I'd always heard Python doesn't support encapsulation, so why even try? What gives? Showing that a map is bijective and finding its inverse Let $S$ := {$(x, y) ∈ \mathbb R^2: x^2 + y^2 = 1$} be the unit circle. Let $p = (0, -1)$ ∈ $S$. Define a map $F$ from $\mathbb R$ × {$0$} to $S$ as follows: Given $(t, 0)$, let $F((t, 0)) = (x, y)$ ∈ $S$ be the intersection of $S$ with the line through $p$ and $(t, 0)$. Derive a formula for $F$. Show that $t$ → $F((t, 0))$ is a bijective map from $\mathbb R$ to $S$ \ {$p$}, and also derive a formula for its inverse.  If we express $S$ in polar coordinates: $$S:=\{(r,\varphi):r=1,\varphi∈(-\frac\pi{2},\frac{3\pi}{2})\}$$ then, from simple trigonometric relations, the inverse function $\Phi$ is: $$\Phi(\varphi)=\tan\frac{\frac\pi2-\varphi}2$$ and its inverse $F$ is: $$F(t)=\frac\pi{2}-2\arctan t$$ After changing coordinates back to cartesian, $F$ will look like: $$F(t)=(\cos(\frac\pi{2}-2\arctan t),\sin(\frac\pi{2}-2\arctan t))$$ which can be further simplified to: $$F(t)=(\frac{2 t}{t^2 + 1},\frac{1 - t^2}{t^2 + 1})$$ Is this the formula of $F$? Is it correct? And I know that to show that $\phi$ : $\mathbb R$ →  $S$ \ {$p$} is a bijective map, I have to show that $\phi$ is injective and surjective. But how do I do that? Can someone please help me show that $\phi$ is bijective? and how can I derive a formula for its inverse? Do Ryzen 3 2200u laptops with Vega 3 work out of the box with major Linux distros? Highlight the difference between two strings in PHP What is the easiest way to highlight the difference between two strings in PHP?  I'm thinking along the lines of the Stack Overflow edit history page, where new text is in green and removed text is in red. If there are any pre-written functions or classes available, that would be ideal. How to change envelope from address using PHP mail? Probability Problem with $n$ keys A woman has $n$ keys, one of which will open a door. a)If she tries the keys at random, discarding those that do not work, what is the probability that she will open the door on her $k^{\mathrm{th}}$ try?  Attempt: On her first try, she will have the correct key with probability $\frac1n$. If this does not work, she will throw it away and on her second attempt, she will have the correct key with probability $\frac1{(n-1)}$. So on her $k^{\mathrm{th}}$ try, the probability is $\frac1{(n-(k-1))}$ This does not agree with my solutions.   b)The same as above but this time she does not discard the keys if they do not work.  Attempt: We want the probability on her $k^{\mathrm{th}}$ try. So we want to consider the probability that she must fail on her $k-1$ attempts. Since she keeps all her keys, the correct one is chosen with probability $\frac1n$ for each trial. So the desired probability is $(1-\frac{1}{n})^{k-1} (\frac1n)^k$. Again, does not agree with solutions.  I can't really see any mistake in my logic. Can anyone offer any advice? Many thanks PHP Session Security What are some guidelines for maintaining responsible session security with PHP?  There's information all over the web and it's about time it all landed in one place! How to not count blank pages in frontmatter of two-sided document Installing titling.sty onto an offline Windows machine I'm trying to compile a file for which I need titling.sty:  ! LaTeX Error: File `titling.sty` not found.  The two biggest issues are 1) it's on a Windows machine (I'm really only &quot;comfortable&quot; with the Unix command line) and 2) the machine cannot be connected to the internet (so automatic approaches are impossible). Assuming the error means I'm missing a package and following some related advice (e.g., ), I downloaded the file , transferred it to the offline machine, and unzipped it, then tried to run: latex titling.ins  But this gave me an error (given I can't copy-paste, I'm including what I think may be relevant):  ******************************************************** * No Configuration file found, using default settings. * ******************************************************** ! ************************************************** * Could not find your \batchfile=titling.ins. **************************************************. \processbatchFile ...****************************}                                                  \fi \else \ifDefault \Msg ... 1.1092 \process@first@batchfile  I have no idea how to proceed from this. How can I install titling.sty on this machine? I'm using a \chapterstyle{tandh} because I want my chapters numbered. In order to get a References chapter numbered as well I used the following code:  \renewcommand{\bibsection}{% \chapter{\bibname}}   That worked fine, I got my References chapter numbered!!   But now, in order to get my references in a author-year style I'm using the natbib package. Seems like the package ignores my code and the References chapter is not numbered anymore.  Can someone help me? Getting all files that have been modified on a specific date Is it possible to find all php files within a certain directory that have been modified on a certain date  I'm using   find /var/www/html/dir/ -mtime -28 | grep '\.php'    to get files modified within the last 28 days, but I only need files that have been modified on the following date: 2011-02-08 Newton's Law of Universal Gravitation tells us that the potential energy of object in a gravitational field is $$U ~=~ -\frac{GMm}{r}.\tag{1}$$  The experimentally verified near-Earth gravitational potential is  $$U ~=~ mgh.\tag{2}$$  The near-Earth potential should be an approximation for the general potential energy when $r\approx r_{\text{Earth}}$, but the problem I'm having is that they scale differently with distance.  $(1)$ scales as $\frac 1r$.  So the greater the distance from the Earth, the less potential energy an object should have.  But $(2)$ scales proportionally to distance.  So the greater the distance from the Earth, the more potential energy an object should have.  How is this reconcilable? Give an example of three positive integers m, n, and r What does the term "86'd" relate to? What does it mean when someone or something is referred to as being "86'd"? How to typeset row operations on augmented matrix I would like to typeset row operations on a augmented matrix, but the "gauss"-package does not seem to support the vertical line just before the last column, any way to do this?  Thanks! Is there any way to create a virtual file, such that reading from the file actually reads from the stdout of a command; writing to the file is acually writing to the stdin of a command?  So far I have kludged this with an inotifywait on a file, which calls a command when the file is modified, taking it's input from the file and writing back to it.  I don't like that the inotifywait has to be constantly restarted though (and I have to ensure that it is always running). I only use this file perhaps twice a week. I was caught cheating on an exam, how can I minimize the damage? I got caught cheating in a two-hour engineering exam consisting of 100 multiple choice questions which was done online through the  e-learning software. I feel like my life is almost over.  Having not studied well and having no time to study, I decided to cheat. I know there are no excuses for cheating. Before the exam I dug up research about moodle exams, and it turned out you can take the exam anywhere you want to as long as you have the quiz password. Taking advantage of this, I went up to my engineering library and asked a close friend who was taking the exam to send me the quiz password before he starts.  What I had prepared:   Calculator (was not allowed during the quiz). Printed material (consisted of more than 200 pages). Google search engine (on a laptop ready to help). Whatsapp (to ask questions for a friend who took the quiz last year).   As I sat down, I did the first 50 questions, suddenly the quiz froze and moodle told me:      you are not allowed to take the exam from this location.   While I was leaving, one instructor responsible for the course (there are 5) came and found me. She accused me of cheating, took my mobile and made come with her to a huge office where two instructors searched all my mobile (took the name of my friend who gave me the password), and started talking to me, asking for all details of this crime. They took and confiscated my phone and looked at everything: Whatsapp conversations, all my emails, and images. The instructor took the mobile from my hand without asking me and kept it with her. It seems that the instructors are going to report me as well as my friend who sent me the password for the exam.  The university's Student Code Conduct said that cheating will result in one of the following: a Dean's Warning, Suspension, or Expulsion. If any of that happens, my future is over. A dean's warning will cancel my financial aid. A suspension would be for at least two years, and coming back would require a lot of work. An expulsion will be definite.  I've learned the lesson about cheating, now how can I fix this? How can I prevent the five instructors from reporting me? It's obvious that I should speak to them, but what should I say? What can I do? My future is almost over, but many of you are teachers and instructors here, what can I do to fix this?    EDIT: Wow, it has been almost 3 years. I would like to update on how I dealt with the situation.   Lesson: It was obvious to never ever cheat under any circumstances, and it was not whether you'd get caught or not.    Consequences: I received a Dean's warning but the financial aid wasn't revoked. However, I couldn't stay in the department as engineering was not for me and I was even ashamed to stay in the university.  What happened next: I transferred to an ABET-accredited Computer Science institution and I graduated with distinction in 2.5 years. It was extremely stressful as I had taken 6 major courses (18 credits) in one semester, but I had to do it.  Future plans: I'm going for an MSc in Computer Science in St Andrews next year. How do I prove that $m(TE)=\left|\det(T)\right|m(E)$? Let $m$ be the $n$-dimensional Lebesgue measure and $T$ be a linear operator on $\mathbb{R}^n$. Let $\Sigma$ be the sigma-algebra of Lebesgue measurable sets. How do prove that (1) $\forall E\in\Sigma, T(E)\in \Sigma$ and $m(T(E))=\left|\det(T)\right|m(E)|$? Since the $\sigma$-algebra of Lebesgue measurable sets is strictly larger than the borel algebra, I think it cannot be proven by showing that $T$ is continuous. How do I prove this formally? I have a script from UC Davis and every thing is proven precisely here, except for one lemma which is critical for rest proofs in the script. That is,  Define $\mathbb{B}=\big\{\prod_{i=1}^n [a_i,b_i) : a_i,b_i\in\mathbb{R}\big\}$. Let $T$ be a unitary operator on $\mathbb{R}^n$. Let $A\in \mathbb{B}$ and $\{A_k\}_{1≦k≦m}$ be a finite disjoint sequence such that $\bigcup_{1≦k≦m} A_k \subset T(A)$. Then $\sum_{1≦k≦m} m(A_k) ≦ m(A)$.  It's written in the script: &quot;This result is geometrically obvious, but a formal proof seems to require a fuller discussion of the volume function on elementary geometrical sets, which is included in the theory of valuation in convex geometry. We omit the details&quot; It's okay to me to go through full discussion of volume function if that is precise. If it cannot be proven another easier way(so I can prove it in a day), please recommend me a &quot;convex geometry&quot; text which explains this. How to do selective-color in Photoshop? How to make the single color in a photography to be color and other parts changed to B&amp;W?.  Don't know exactly the name of the effect, to be more clear in the following photo I want the green part to be green and other parts( leg and sandal) in B&amp;W..    How to achieve that effect in Photoshop?