I run a binary logistic regression, with a binary dependent variable and a continuous independent one. Now I want to evaluate the out-of-sample performance of the classification algorithm so obtained. For instance, given a probability threshold, I want to compute out-of-sample accuracy or sensitivity. One solution would be resampling-based estimators: for instance, the method of validation set. I would split the dataset into a training set and a test set, train the model on the training set, and evaluate the performance (e.g. accuracy or sensitivity) on the test set. This is a generic approach, which can be used with any estimator. For example, I can use this approach also with SVM or neural networks. But logistic regression is different in many ways from a (deep) neural network: in particular, we know a lot about the properties of the estimator. For instance, we know the confidence interval for the estimated coefficients, and we know an estimate for the irreducible error in the data. My question is: are there any analytical solutions (not re-sampling based like validation set or cross-validation) to compute out-of-sample performance metrics like accuracy or sensitivity? For instance, with least squares linear or polynomial regression, I could train the model on the entire dataset (without splitting train/test) and then get the Leave-One-Out Cross-Validation MSE with the following formula: $$ \text{CV} = \dfrac{1}{n} \sum\limits_{i=1}^n \left( \dfrac{y_i - \hat{y}_i}{1-h_i} \right)^2 $$ (see James, Introduction to Statistical Learning, pag. 180) Is there a similar formula for the accuracy of logistic regression?
I was naively validating my binomial logit models by testing on a test dataset. I had randomly divided the available data (~2000 rows) into training (~1500) and validation (~500) datasets.   I now read a post in another thread ( Frank Harrell) that causes me to question my approach:     Data splitting is not very reliable unless you have more than 15,000   observations. In other words, if you split the data again, accuracy   indexes will vary too much from what you obtained with the first   split.   How serious is this worry and what are ways around it? The OP speaks of "resampling" but not sure how that works here for validation.   Edit: Adding context as per @Bernhard's comment below:
Is it possible to change the default width of an image included via \includegraphics (from the graphicx package) to a custom value, for example \linewidth.  So I want something like this:  \documentclass{article}  \usepackage[draft,width=\linewidth]{graphicx}  %or something like \setglobalimagewidth=\linewidth  \begin{document}  \includegraphics{test1} %width=\linewidth \includegraphics[width=4cm]{test1} %width=4cm   \end{document} Convergence of the series $\sum_n \frac{n^{\log(n)}}{\log(n)^n}$ I have a rough outline of a proof. Is this correct?  $$\sum_{n \geq 2} \frac{n^{\log(n)}}{\log(n)^n} \to a \in \mathbb{R} \iff  \sum_{n \geq 2} 2^n \frac{(2^n)^{\log(2^n)}}{\log(2^n)^{2^n}} \to b \in \mathbb{R}$$ via dyadic criterion (or Cauchy condensation test).  Analyzing the right-hand side:  $$\sum_{n \geq 2} 2^n \frac{(2^n)^{\log(2^n)}}{\log(2^n)^{2^n}} = \sum_{n \geq 2} 2^n \frac{(2^n)^{n\log2}}{(n\log2)^{2^n}} = \sum_{n \geq 2} \frac{(2^n)^{n\log2 + 1}}{(n\log2)^{2^n}} = \sum_{n \geq 2} \frac{2^{n^2\log2 + n}}{(n\log2)^{2^n}} = (\ast). $$  We know for some sufficiently large $N$, $n \log 2 \ge 2$ for all $n\ge N$, so we can bound the series above by:  $$(\ast) \leq C_N + \sum_{n \geq N} \frac{2^{n^2\log2 + n}}{2^{2^n}} = \sum_{n \geq N} 2^{n^2\log2 + n - 2^n} = (\ast\ast),$$  where $C_N$ is the finite sum of the original series until $N$. And once again, for sufficiently large $M$, $m^2\log2+m - 2^m \leq -m^2$ for all $m \ge M$. Hence  $$ (\ast\ast) \leq C_{\max(N,M)} + \sum_{n \geq \max(N,M)} 2^{-n^2} &lt; \infty.$$  We then use the $p$-test to conclude that the sequence does converge. Integration of $e^{ax}\cos bx$ and $e^{ax}\sin bx$ Why do corn flakes clump together in milk when stirred? The Cheerios Effect is the name the scientific community has given to this phenomenon. When cornflakes or oats are mixed with milk and stirred, they tend to clump together. Why does this happen? Can physics explain this? Counting area percentage of overlapping polygons I have a set of points (meteorological stations) with 5km buffers around them. Additionally, I have a set of polygons (municipalities) in which those stations are located.  I would like to have data on how much of the buffer is filled by each polygon but I was not able to find a function or a way to do it. So for example for point A I would have that in the buffer of 5km there is 80% of polygon B and 20% of polygon C as the output.  Edit:  The anwsers provided do anwser this question in a setting in which I know what polygons and buffers I need to have in the code or when I can do it manualy. Problem is that in my setting with 1 shapefile having 500 polygons and one vector layer having 13000 buffers I am unable to do this manually and I am asking if there is an easier way to do it. I tried implementing the approaches in the questions but I was unable to use them on my example in big scale. Moreover I had problems with coding that I also noted in comments down below.   Main difference to the other similar question provided is that I have 2 vector layers (Buffer with 500 buffers and Municipalities a shape file with 500 different polygons). Previous questions were analyzing this question in situations in which I know precisely the polygons and buffers analyzed, at least that is how I understood them, so after trying to use those approaches on my datasets I was not able to replicate them.  Another way of approaching this problem for me is to use join attributes by location -one-to-many and then getting averages of the duplicate results, only problem is that then I take average of polygons instead of their weights corresponding to % of intersection with the buffer. In my case I do not know what polygons of the shapefile are going to intersect what buffers, or even if the buffer is going to be intersected by 1,2 or more polygons Can open sets and closed sets of $\mathbb{R}^n$ be homeomorphic? Suppose $A$ is an open subset of $\mathbb{R}^n$ considered as a topological subspace. Suppose $B$ is a another susbet of $\mathbb{R}^n$, and under the induced topology, is homeomorphic to $A$.  If $B$ is closed and bounded, then by the Heine-Borel theorem, $B$ is compact as a topological space so $A$ and $B$ cannot be homeomorhpic, because $A$ is not compact. But if $B$ is not bounded, what can be said about $B$, if any? Must $B$ be open, can it ever be closed (let's leave out the only case of being both open and closed--the entire ambient space $\mathbb{R}^n$), or can it be neither open nor closed? Is it possible to run Android apps on Ubuntu? And I don't mean inside an emulator, but natively, as you would with any other application installed using the Ubuntu Software Center. Is "from where" grammatically equivalent to "from which"? Proving $\frac{1}{n+1} + \frac{1}{n+2}+\cdots+\frac{1}{2n} &gt; \frac{13}{24}$ for $n&gt;1,n\in\Bbb N$  To solve it I used induction but it is leading me nowhere my attempt was as follows:  Lets assume the inequality is true for $n = k$ then we need to prove that it is true for $k+1$  so we need to prove $\frac1{k+2} + \frac1{k+3}+\cdots+\frac1{2(k+1)} &gt; 13/24$  I don't know where to go from here please help. Choosing C++ debugger on linux I was compiling a tool called Rejoystick so I installed a few packages and dependencies. But now I don't need it any more so I removed those packages. Running apt-get shows that I have some packages that are automatically installed and are no longer needed.   But I have a bad experience with apt-get autoremove. It broke my desktop environment when I uninstalled wine some time ago. Luckily, I had a clonezilla partition backup then, but this time, I don't have a backup. So I'm cautious. Running apt-get shows these packages are orphaned     gir1.2-gtk-2.0 libasound2-dev libavahi-client-dev libavahi-common-dev   libcaca-dev libcairo-script-interpreter2 libdbus-1-dev libdrm-dev   libexpat1-dev libfontconfig1-dev libfreetype6-dev libgl1-mesa-dev   libglu1-mesa-dev libharfbuzz-dev libharfbuzz-gobject0 libice-dev   libpcre3-dev libpcrecpp0 libpixman-1-dev libpng12-dev libpthread-stubs0-dev   libslang2-dev libsm-dev libx11-dev libx11-doc libx11-xcb-dev libxau-dev   libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev libxcb-present-dev   libxcb-randr0-dev libxcb-render0-dev libxcb-shape0-dev libxcb-shm0-dev   libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev libxcomposite-dev   libxcursor-dev libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev   libxft-dev libxi-dev libxinerama-dev libxrandr-dev libxrender-dev   libxshmfence-dev libxxf86vm-dev mesa-common-dev x11proto-composite-dev   x11proto-core-dev x11proto-damage-dev x11proto-dri2-dev x11proto-fixes-dev   x11proto-gl-dev x11proto-input-dev x11proto-kb-dev x11proto-randr-dev   x11proto-record-dev x11proto-render-dev x11proto-xext-dev   x11proto-xf86vidmode-dev x11proto-xinerama-dev xorg-sgml-doctools xtrans-dev   zlib1g-dev   I'm not sure if I should run apt-get autoremove. I was searching details related to each of these packages so that I can understand which ones are needed but they all seem important e.g.     gir1.2-gtk-2.0    I have Cinnamon, GNOME and Unity installed, I think apt-get autoremove will break something. Any help will be appreciated.   Thanks for reading. New global search doesn't like tags Receiving Push Notifications while in background I'm looking for a movie I've seen as a kid, the details are foggy and sparse but I'll try my best following :  What is it?: An animated movie. Tell us what it isn't: It is not . Tell us what made it memorable: A planet where some brain-sucking insect-like aliens live, also an old bald man lives on the same planet. When did you see this film?: Late 80's, early 90's. How old was it?: If I've seen the movie on TV in Spain, is a pretty safe bet that it will be at least 5 years old. Was it part of a series?: I don't think so. Plot: No clue, sorry. What language was it in?: The movie was dubbed to Spanish (Spain).  No detail is too insignificant  What details can I recall?:  Sci-Fi story with one character being an old bald man living on a planet where some insect-ish aliens can open your skull and suck your brain. Habitat of those insect-ish foes was depicted in light blue colors, I kinda remember giant mushrooms as well. A young boy befriends the old baldie, the boy came to the planet with some adults.   Other details I'm not completely sure (maybe my mind is filling the gaps):  I think that the movie was drawn using , not 100% sure though. I cannot remember which year I've seen the movie, but it should be late 80's or early 90's. I'm pretty sure that I've seen the movie in Spanish TV (not rented videotape). Somehow the young boy was the young version of the old baldie (time travel?). The old man lives in a wood house near the brain-sucking insect habitat, as many old man does he spend his time on the porch.    Try to offer unique details As anyone would expect from this kind of movie, there's a scene where the alien insects attack a group of people trying to suck their brains. A young boy was with those adults and survived with injuries in his head. The old bald man have some surgery in his head due to an attack he suffered years before. Command to move a file to Trash via Terminal Lightning QuickAction (API) without modal How to Retrieve the Wi-Fi Password of a Connected Network on a Mac I need to find the password for a Wi-Fi network that I am already connected to. I'm using a MacBook Pro that's running OS X Lion. Any guidance is appreciated. Writing a list to a file with Python Tourist to F1 change of status within 60 days I have a PhD place at a US university starting Aug 21 but want to visit a different university for casual research for 6 weeks from July 1. This is longer than the 30 day grace period for the F1. Can I arrive on the visa waiver program or tourist visa in July and then just stay on for the PhD? I guess I might need to change status. Will this work?  Please don’t suggest exiting and reentering, that’s too expensive.  (British masters student in UK) pdflatex, \includegraphics, and multi-page PDF files I have a PDF file with multiple figures; each figure is a separate page in the PDF file.  I am using pdflatex, and I would like to use \includegraphics or equivalent to include some of the figures.  Of course I could use an external tool to split the PDF file into multiple one-page PDF files. Then I could use \includegraphics as usual. However, I would have a much simpler workflow if I could avoid this extra step. Hence the question:  How can I include, e.g., page 3 of foo.pdf as a figure?  I am aware that there is a Latex package called pdfpages, but it seems to be designed for something else: with pdfpages, each page in the input file will produce a full page in the final document, not just a box.