I know we have developed some mathematical tools to understand deep neural networks, gradient descent for optimization, and basic calculus. Recently, I encountered arxiv paper that describes higher mathematics for neural networks, such as functional analysis. For example, I remember universal approximation theorem was proved with the Hann-Banach theorem, but I lost the link of that article, so I need to find similar papers or articles to develop my understanding of neural networks mathematically (like with functional analysis, in short, I need to learn more advanced math for research), can you suggest some books or arxiv papers or articles or any other source that describes mathematics for deep neural networks?

  • $\begingroup$ To me, it's not really clear what you're asking. What kind of "mathematical tools" are you really looking for? Are you trying to understand why neural networks can approximate continuous functions? It's not clear what your level of understanding of neural networks is and what you want to understand that you don't understand. $\endgroup$ – nbro Sep 5 '20 at 19:47
  • $\begingroup$ @nbro that is a good point, for example, I want to learn mathematics on the research level. I see many AI labs output brilliant papers and I think they must be using some advanced mathematics, it won't be just random calculation, the should think ahead, so I want to learn mathematics on the research level. is it clear, sorry if It is not clear now, please say and i will go more in details. $\endgroup$ – dato nefaridze Sep 5 '20 at 19:49
  • $\begingroup$ Can you please provide an example of paper that you want to understand? There are so many mathematical topics that it's difficult to know them all and so it's difficult to answer your question. However, if you share with us the specific paper or papers that you want to understand, maybe we can provide more details about the type of mathematics that you need to understand those papers. Moreover, I suggest that you edit your post to include these details and also to include a description of your current knowledge of NNs. What do you know? What's your level? $\endgroup$ – nbro Sep 5 '20 at 21:01
  • 2
    $\begingroup$ That's difficult to answer because you don't necessarily need big math skills to be an important researcher. It depends on what you research. Also, there are many things in AI, not just neural networks. The short answer to your question is that: you don't need to know all the math to be a good researcher. For example, if you do research on variational inference, you definitely need to know a good dose of calculus, statistics, and probability theory. So, it's difficult to say exactly what you need to learn because there are so many topics and you will almost surely only focus on 1-2 of them. $\endgroup$ – nbro Sep 5 '20 at 21:21
  • 1
    $\begingroup$ So, please, ask a more specific question, and you will get more useful answers. We cannot answer your question if you don't tell us what you are interested in. If you say you're interested in becoming an AI researcher, that doesn't mean much, because one AI researcher could be doing research on reinforcement learning, another in expert systems, and another in variational inference. Nobody does research on "AI". Typically, people do research on specific topics or subfields of AI. $\endgroup$ – nbro Sep 5 '20 at 21:29

Knowing you want to focus on the theory, I think that a good choice is Deep Learning Book from Ian Goodfellow et al., which is publicly available. It has three main parts. On the first one the author presents the math/ statistic tools that will be needed to understand the following parts. On the second part, the author explains the current state of the art in Deep Learning and on the last part more advanced topics are introduced. The author also uses references so as to facilitate extra resources to dive more into the theory.

On the other hand I strongly recommend you Google Scholar, there you have plenty of information given by articles/ papers of the current state of the art techniques in the field. There you can also find papers related on what that you mentioned about the Hahn-Banach Theorem.

