I have a few large LiDAR DEMs that I want to merge. Each DEM has some glitches around the borders which need to be removed in order to get a seamless final product. I manually traced around each DEM to create a polygon which excludes the glitchy areas. I am using gdalwarp to clip the raster files, but I am experiencing very slow performance.  The rasters are between 3 and 10 GB, which I understand is pretty big… however I have ~3 ghz of processing power &amp; 4GB of RAM. These rasters load into QGIS quickly and are quick to refresh. I can derive a hillshade or slope raster from one of these DEMS in 2 or 3 minutes.. so it seems unreasonable when it takes 4-6 HOURS to clip one of these rasters with gdalwarp.  Can anybody offer an explanation of what might be going on?  I am also bothered by the very large file sizes (up to 4x larger than the original) which result from the clipping. Research suggests that this is a compression issue, but gdalinfo does not return any compression info for any of the original rasters, and even when I compress the files are still 2x larger than the originals. Could there be any reason for this other than compression?
After using gdalwarp to project and align-to-grid (via -tap) a number of rasters I noticed that the output rasters were significantly larger than the original rasters. A fairly thorough web search turned up :  Frank Warmerdam explained the reason: &quot;On careful review, the difference in the file in question is because gdal_translate uses the TIFFWriteScanline() interface to write the output file from within GTiffDataset::CreateCopy?(), and this only writes as much of the final 'strip' of the file as is required to complete image area. But gdalwarp goes through the blockio interface which writes the complete final strip, even the portion that falls off the end of the file.&quot;  This Trac issue is ~7 years old, however, and I know some changes to the GDAL utilities, including gdalwarp have been made since. I'd like to know if the above reasoning still holds and if the file size inflation I'm seeing is &quot;normal.&quot; The word &quot;normal&quot; here might be taken to mean unsurprising or expected but, more importantly: is there anything that can be done to mitigate the effects i.e. reduce the output raster file size? Below is a table of the file size inflation I'm experiencing. Input File Size (bytes)     Output File Size (bytes)    Inflation 1437380431                  1698334217                   18% 1428001178                  1698334433                   19%   41683165                   137036637                  228%  The input TIFF files were created in ArcGIS and thus have external Worldfiles, XML and DBF files but these do not make up the difference in file size. Here is a sample gdalwarp call as I've used it in all of these cases; the actual execution was handled by a Python subprocess (subprocess.Popen): $ gdalwarp -tap -tr 30 30 -t_srs &quot;+proj=aea +lat_1=20 +lat_2=60 +lat_0=40 +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m +no_defs&quot; -co &quot;COMPRESS=LZW&quot; input_file.tif output_file.tif  I understand that in rare cases compression makes a larger file, but the effect is the same without the LZW compression. The ratios in the table are with LZW compression.
This is a canonical question and answer developed by the community to help address common questions. If you've been directed here, or your question has been closed as a duplicate, please look through the resources here and use them to shape more specific questions. To browse all canonical questions and answers, navigate to the  tag.  This question and answer is owned by the community! If you can improve it and make it more useful, you're welcome to edit it.  I have a specific JSON structure in mind. I need to either deserialize this structure that's coming in from a web service, or I need to serialize data into this structure to transmit it to another system. How should I get started on this project in Apex? Why does the QGIS Server WMS layer not load in QGIS Desktop? Land ratio vs mana curve When choosing cards for a deck, one of the most important things to consider is the number of land you put in. The biggest issue is mana curve, of course. What shortcuts are best for determining the proper amount of land with a given mana curve? Which characters are valid in CSS class names/selectors? Say a security has been trading at under $20.10, and there are limit orders selling at $20.10 and $20.21. Then Simon issues a limit order to buy at $20.21. According to , Simon would end up buying shares at $20.10.  Why does the exchange favor the buyer here?  I.e., why does the transaction execute at the seller's limit of $20.10 instead of the buyer's limit of $20.21?  (I.e., why don't Inara/River get to sell at $20.21 / above their limit, but Simon gets to buy at $20.10 / below his limit?)  Is the example flawed?  If it makes a difference, I'm mainly curious about NASDAQ and NYSE.  I tried Googling for specs (e.g. ), but couldn't find anything.  Only found that some  implementation executes the average of the buy/ask.  Just curious how the system works. Using a variable in a SELECT statement? public static void handleCaseComment(CaseComment caseComment) {    User[] users = [SELECT LastName FROM User WHERE Id = caseComment.CreatedById LIMIT 1]; }   This gives me the following error:  unexpected token: 'caseComment.CreatedById'   How do I use the value of the variable caseComment.CreatedById in a SELECT statement? Like the ModelBuilder of ArcGIS Desktop to run simultaneous geoprocessing tools, is there any model builder available in QGIS? Exactly half of the elements of $\mathcal{P}(A)$ are odd-sized Replacing strings in files based on certain search criteria is a very common task. How can I    replace string foo with bar in all files in the current directory? do the same recursively for sub directories? replace only if the file name matches another string? replace only if the string is found in a certain context? replace if the string is on a certain line number? replace multiple strings with the same replacement replace multiple strings with different replacements How is the Schroedinger equation a wave equation? Wave equations take the form:  $$\frac{ \partial^2 f}  {\partial t^2} = c^2 \nabla ^2f$$  But the Schroedinger equation takes the form:  $$i \hbar \frac{ \partial f}  {\partial t} = - \frac{\hbar ^2}{2m}\nabla ^2f + U(x) f$$  The partials with respect to time are not the same order. How can Schroedinger's equation be regarded as a wave equation?  And why are interference patterns (e.g in the double-slit experiment) so similar for water waves and quantum wavefunctions? How many 5-cards poker hands are there containing at least 3 of the 4 suits?  I don't how know how to properly handle this question.  There 13 ranks of each card, and 4 suits  My Attempt:  Begin with the first 3: C(13,3) X 4 ways to choose suits  Then pick last 3 cards of anything:  C(49,2)  My solution: 1,345,344  Thanks, I have an external HD which used to be my TimeMachine drive. I got a different drive for that, so I want the original drive back as an external on my Windows box, but it has this random 200 MB partition labeled as EFI System Partition. How do I get rid of this? I am making a skydome and I have a texture hooked up with an emission shader for it, however, since the dome is blue it only emits blue light. Is there a way that I can have the whole sky illuminated and have the light reaching the rest of my scene be a different colour? The  tag doesn't appear to be very useful.  It's used in a variety of ways (for reading, writing, showing, transforming, compiling, translating, obfuscating, obtaining, ... source code) on 1747 questions right now, and doesn't appear to be adding any classification value to the site.  Can we get rid of it? The pdf of $\frac{X_1-\bar{X}}{S}$ Suppose $X_1, X_2,...,X_n$ be i.i.d from $N(\mu,\sigma^2)$ with unknown $\mu \in \mathcal R$ and $\sigma^2&gt;0$  Let $Z=\frac{X_1-\bar{X}}{S},$ S is the standard deviation here.  It is can be shown that  $Z$ has the Lebesgue p.d.f.  $$f(z)=\frac{\sqrt{n} \Gamma\left(\frac{n-1}{2}\right)}{\sqrt{\pi}(n-1)\Gamma\left(\frac{n-2}{2}\right)}\left[1-\frac{nz^2}{(n-1)^2}\right]^{n/2-2}I_{(0,(n-1)/\sqrt{n})}(|Z|)$$  My question is then how to get this pdf?  The question is from  in example 3.3.4  to find the UMVUE of $P(X_1 \le c)$. I can understand the logic and procedures to find the UMVUE but don't know how to get the pdf.  I think this question also relate to this    Thank you very much for help or point to any related references will be also appropriated. Why isn't Google Chrome available in the Software Centre? Does "notified by [date]" include the end date? I have read the Rules of a competition. The text of the Rules include a sentence as follows:     As per stated in the Rules the entrants will be notified by May 30th 2010.   Does the sentence above mean that May 30th is included in the term of notification or not? I'm trying to recall the name of a book I read, probably published after 1990, where alien AI infects satellites and a barrier is established, isolating earth from space that maybe slows down time inside of it, relative to the passage of time outside.  Mars or the moon is also colonized in an attempt to flee the now real possibility that the sun will become a red giant and wipe out humanity in the not-too-distant future (as perceived on earth), but the colony gets one of the time barriers as well.  I think the name of the novel was short, "The _______" (one word).  The story focused more on the human cost and followed one particular family, one member of which was somehow also partially infected by the alien AI, but I don't remember as much about them. Using "you" and "your" as a representation for yourself and everyone in general Example sentence:     I love when your dog just lets you sit there to pet them. You don’t necessarily know if they are enjoying it, but they love you enough to just sit there with you for a bit.   Is this correct? We assume the words "you" and "your" refer to the speaker of the sentence, and not to the listener, as second-person usually does. But it also refers to dog owners in general. I have always been curious about this. Which are safe and unsafe regions in Iran? Now there's a lot of propaganda in the west that leads us to believe Iran is dangerous and travellers shouldn't go there.  Obviously there are major political disagreements between several governments and the Iranian government and this should be taken into account.  But at a human level I have met many Iranians travelling elsewhere in the world and have found them to be friendly, charming, and hospitable. I have also met several western travellers who spent time in Iran, including hitchhiking around the country, who only have wonderful things to say about the country and its people.  As a traveller I've learned to trust the most what I hear from other travellers about a destination, especially ones which prefer similar ways of travel to myself. I've also learned to trust the opinions of inhabitants of countries bordering on the destination country the least.  Yet some people still insist that Iran is not safe for a traveller, or that .  I can look up for myself the general travel advisories issued by governments and disclaimers in WikiVoyage. So please don't regurgitate those an answers.  I want to know from travellers who've been there, from Iranians, or from references to specific incidents targeting foreigners/westers about which parts of Iran are no problem, and which parts should be avoided, even by experienced intrepid travellers.  (I realize any advice won't apply to Israeli citizens unfortunately. I'm Australian.)