Question: For the layman, what does it mean for data (say $n$ samples covering $m$ variables) to be identically distributed, and how is it practically achieved when conducting machine learning?  So let's say each sample measures X and Y predictors, and I want to predict outcome Z.   To satisfy IID, should I be aiming for $X$ and $Y$ to look the same when I plot their estimated density functions? Or does it mean something else entirely?  I have read about "normalizing" your sample data by subtracting the mean and dividing by the standard deviation. Is the purpose of this to fulfill IID?   The context of my question is machine learning, specifically XGBoost for a regression problem. It is performing in a mediocre to poor fashion and I want to understand why. Perhaps my data is not IID. I thought it didn't matter for tree-based models, but I am obviously missing something.
In statistical learning, implicitly or explicitly, one always assumes that the training set $\mathcal{D} = \{ \bf {X}, \bf{y} \}$ is composed of $N$ input/response tuples $({\bf{X}}_i,y_i)$ that are independently drawn from the same joint distribution $\mathbb{P}({\bf{X}},y)$ with  $$ p({\bf{X}},y) = p( y \vert {\bf{X}}) p({\bf{X}}) $$  and $p( y \vert {\bf{X}})$ the relationship we are trying to capture through a particular learning algorithm. Mathematically, this i.i.d. assumption writes:  \begin{gather} ({\bf{X}}_i,y_i) \sim \mathbb{P}({\bf{X}},y), \forall i=1,...,N \\ ({\bf{X}}_i,y_i) \text{ independent of } ({\bf{X}}_j,y_j), \forall i \ne j \in \{1,...,N\} \end{gather}  I think we can all agree that this assumption is rarely satisfied in practice, see this related  and the wise comments of @Glen_b and @Luca.  My question is therefore:      Where exactly does the i.i.d. assumption becomes critical in practice?    [Context]   I'm asking this because I can think of many situations where such a stringent assumption is not needed to train a certain model (e.g. linear regression methods), or at least one can work around the i.i.d. assumption and obtain robust results. Actually the results will usually stay the same, it is rather the inferences that one can draw that will change (e.g. heteroskedasticity and autocorrelation consistent HAC estimators in linear regression: the idea is to re-use the good old OLS regression weights but to adapt the finite-sample behaviour of the OLS estimator to account for the violation of the Gauss-Markov assumptions).   My guess is therefore that the i.i.d. assumption is required not to be able to train a particular learning algorithm, but rather to guarantee that techniques such as cross-validation can indeed be used to infer a reliable measure of the model's capability of generalising well, which is the only thing we are interested in at the end of the day in statistical learning because it shows that we can indeed learn from the data. Intuitively, I can indeed understand that using cross-validation on dependent data could be optimistically biased (as illustrated/explained in ).   For me i.i.d. has thus nothing to do with training a particular model but everything to do with that model's generalisability. This seems to agree with a paper I found by Huan Xu et al, see "Robustness and Generalizability for Markovian Samples" .   Would you agree with that?  [Example]   If this can help the discussion, consider the problem of using the LASSO algorithm to perform a smart selection amongst $P$ features given $N$ training samples $({\bf{X}}_i,y_i)$ with $\forall i=1,...,N$ $$ {\bf{X}}_i=[X_{i1},...,X_{iP}] $$ We can further assume that:   The inputs ${\bf{X}}_i$ are dependent hence leading to a violation of the i.i.d. assumption (e.g. for each feature $j=1,..,P$ we observe a $N$ point time series, hence introducing temporal auto-correlation) The conditional responses $y_i \vert {\bf{X}}_i$ are independent. We have $P \gg N$.    In what way(s) does the violation of the i.i.d. assumption can pose problem in that case assuming we plan to determine the LASSO penalisation coefficient $\lambda$ using a cross-validation approach (on the full data set) + use a nested cross-validation to get a feel for the generalisation error of this learning strategy (we can leave the discussion concerning the inherent pros/cons of the LASSO aside, except if it is useful).
I am using find -type f command to recursively find all files from a certain starting directory. However, I would like to have some directories prevented from entering and extracting names of files inside. So basically I am looking for something like:  find . -type f ! -name "avoid_this_directory_please"   Is there a functioning alternative to this? How can I do a full shutdown of windows 10 without logging in? python: restarting a loop filter the file type with the file upload control I'm fairly certain this has happened to all of us at some point: You go into a room to get something, but once you're there, you can't remember what you intended to get. It seems like a specific, common phenomenon like this must have a name, but I can't think of it. Does anyone know a name (either formal or slang) for this?  I don't want a general term like forgetfulness or absent-mindedness, which could apply to all sorts of forgetting, or a term for not being able to remember a certain word. I'm looking for a word or phrase that describes this one specific situation of forgetting what you need to get as soon as you go to get it. How to use only part of screen, as if the monitor was a smaller one? Water was sprayed on my 19" monitor and damaged some parts of it. I need a solution that will let me specify the screen area to be used by Windows, as if it was a smaller monitor. Its default menu doesn't have this option. Man/Tree Symbiosis It was an old book... I'm pretty sure it was a book, and pre-1990's I think. The basic plot was this man somehow interacted with this tree, and as a result he became a carrier of the tree's 'seed'. They were in a symbiotic relationship: the man got a much longer life, more strength, etc.; but when he did die the tree would grow up from his body. Too many SE sites causes confusion have already been created through .  have already completed their beta stage and are waiting for launch.  are in public beta stage.  are in commitment phase in   proposals are listed in   Now it is really confusing to find a suitable site to ask a question. As an example - I host a blog using . Now I have a question about this hosting. I want to post it in a . Where will I post? I have 2 options available now.     Are developers trying anything to solve this confusing matter? Are the contents of a loot box determined at the time it is awarded or when it is opened?  For example, if you collect multiple loot boxes before a special event but opened them during the special event, would they have the potential of spawning rewards from that event? I'm looking for several ways to prove that $$\int_{0}^{\infty }\sin(x)x^mdx=\cos\left(\frac{\pi m}{2}\right)\Gamma (m+1)$$  for $-2&lt; Re(m)&lt; 0$ ActionBarSherlock with ViewPager not calling ViewPager fragments lifecycle methods when going to and returning from backstack I've connected a USB to Serial bridge dongle, and running dmesg | grep tty outputted the following:  [    0.000000] console [tty0] enabled [603199.380677] usb 2-2: cp210x converter now attached to ttyUSB0   So now I'm trying to write to it as per  by running the following:  cat hello.txt &gt; /dev/ttyUSB0   and   sudo cat hello.txt &gt; /dev/ttyUSB0   But both result in the following error:  bash: /dev/ttyUSB0: Permission denied   What am I doing wrong?  Thanks Equality holds in triangle inequality iff both numbers are positive, both are negative or one is zero How do we show that equality holds in the triangle inequality $|a+b|=|a|+|b|$ iff both numbers are positive, both are negative or one is zero? I already showed that equality holds when one of the three conditions happens. Adobe Creative Suite alternative The programs that I am mainly interested in is Adobe Photoshop, Adobe Illustrator (create vector art), Adobe Flash (for animation), Adobe Premiere Pro (edit video), Adobe After Effects (VFX/motion graphics), Adobe Dreamweaver (code HTML/CSS). But can boil down to a video editing software, graphics software and photo editing software.  I understand alternatives such as Inkscape is like Illustrator, Blender is a good substitute for Adobe Flash. Notepad++ for Dreamweaver but if you haven't noticed, these programs are from different companies. What I do appreciate about Adobe is under the master collection, I get all these programs, but I don't appreciate the price (because I can't afford) or the subscription model (I like to buy once and forget) they now have.  I have searched online and can't find something that packs in programs such as Adobe, even if it's just a photo editing software and vector software.  I found Corel Draw, but I'm so confused if they provide a master collection (collection of their software for a price) that will satisfy my needs.  Is there an alternative to Adobe Creative Suite for designers/enthusiast? Possible Duplicate:          I would like to use round parentheses () instead of square brackets [] for citations in a document that I'm working on.   I figured out how to get them in the text using the round option with natbib.  I can't figure out to have the generated bibliography to show () instead of [] when it lists the references.  What do I need to change to fix the way it formats the bibliography? Terminal limits scrolling I have a running log, it outputs a lot to the terminal. Unfortunately, when I try to scroll up and view older output, it appears there is a limit. And I am unable to scroll past a certain point. On the Mac, the terminal allows you to scroll up to the start of the session. Is there a way to do this in ubuntu terminal? Problem is to find sum  $$\frac{1}{3!}+\frac{4}{4!}+\frac{9}{5!}\cdots$$  What I knew doesn't apply on this problem  Some series are telescoping, some types are solvable using binomial , both look useless here Binomial gives  $$n (n-1) (n-2)x^3=1$$  $$n (n-1)( n-2)(n-3)x^4=4$$  What approach to use here? Parallel execution of bash script I have a table URL_Experiment in my database (mySQL database). I have 2 million URL links in this table.  For each URL, I am checking if some particular text is present in the URL and updating the information back in my URL_Experiment table.    Now, I need to make the script run fast. The easy solution that comes to my mind is:  split the URL_Experiment into 10 tables (each with 200,000 rows) and run the script 10 times simultaneously. This way, I can ensure that the updates happen properly. But for this, I need to save 10 scripts, each accessing the correct table (I need to rename the tables from 1 to 10).   Though the above seems a pretty neat solution, I am looking for a more sophisticated solution. Something which will execute the same script simultaneously (in parallel) as 10 processes. My concern is the execution time. I do not want the resources to be wasted when they can be utilized. I'm struggling to find the limit at infinity of : $$\lim \limits_{n\rightarrow\infty}\sin^2(\pi\sqrt{n^2 + n}), n\in\Bbb N$$ I know it is $1$ but I don't understand why this is wrong :  $\sin^2(\pi\sqrt{n^2 + n}) = \sin^2(\pi*n\sqrt{1 + \frac{1}{n}})$.  So $\lim \limits_{n\rightarrow\infty}\sin^2(\pi\sqrt{n^2 + n}) = \lim \limits_{n\rightarrow\infty}\sin^2(\pi*n\sqrt{1 + \frac{1}{n}}) =  \lim \limits_{n\rightarrow\infty} \sin^2(\pi*n) = 0$.  Thanks. Where can I find fantasy minis on the cheap? I'm looking to build a collection of fantasy miniatures, preferably pewter - but I don't want to buy them one at a time from my local hobby store... Much too expensive, and they don't often carry anything I'm interested in buying, which makes it a long term venture as well. So I'm looking for ideas on how to obtain many miniatures at once, preferably for cheap. D&amp;D Miniatures aren't a good source for me as they're collectibles, and the prices on those are also prohibitive.  By the way - they don't have to be pewter, that's just my preference. =)