text:
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!
Using JEXL to apply hard filters or select variants based on annotation values
1. JEXL in a nutshell
JEXL stands for Java EXpression Language. It's not a part of the GATK as such; it's a software library that can be used by Java-based programs like the GATK. It can be used for many things, but in the context of the GATK, it has one very specific use: making it possible to operate on subsets of variants from VCF files based on one or more annotations, using a single command. This is typically done with walkers such as VariantFiltration and SelectVariants.
2. Basic structure of JEXL expressions for use with the GATK
In this context, a JEXL expression is a string (in the computing sense, i.e. a series of characters) that tells the GATK which annotations to look at and what selection rules to apply.
JEXL expressions contain three basic components: keys and values, connected by operators. For example, in this simple JEXL expression which selects variants whose quality score is greater than 30:
QUALis a key: the name of the annotation we want to look at
30.0is a value: the threshold that we want to use to evaluate variant quality against
>is an operator: it determines which "side" of the threshold we want to select
The complete expression must be framed by double quotes. Within this, keys are strings (typically written in uppercase or CamelCase), and values can be either strings, numbers or booleans (TRUE or FALSE) -- but if they are strings the values must be framed by single quotes, as in the following example:
"MY_STRING_KEY == 'foo'"
3. Evaluation on multiple annotations
You can build expressions that calculate a metric based on two separate annotations, for example if you want to select variants for which quality (QUAL) divided by depth of coverage (DP) is below a certain threshold value:
You can also join multiple conditional statements with logical operators, for example if you want to select variants that have both sufficient quality (QUAL) and a certain depth of coverage (DP):
&& is the logical "AND".
In the case where you want to select variants that have at least one of several conditions fulfilled, provide each expression separately:
"QD < 2.0" \ "ReadPosRankSum < -20.0" \ "FS > 200.0"
To be on the safe, do not use compound expressions with the logical "OR"
|| as a missing annotation will negate the entire expression.
4. Filtering on sample/genotype-level properties
You can also filter individual samples/genotypes in a VCF based on information from the FORMAT field. Variant Filtration will add the sample-level FT tag to the FORMAT field of filtered samples. Note however that this does not affect the record's FILTER tag. This is still a work in progress and isn't quite as flexible and powerful yet as we'd like it to be. For now, you can filter based on most fields as normal (e.g. GQ < 5.0), but the GT (genotype) field is an exception. We have put in convenience methods to enable filtering out heterozygous calls (isHet == 1), homozygous-reference calls (isHomRef == 1), and homozygous-variant calls (isHomVar == 1).
5. Important caveats
Sensitivity to case and type
You're probably used to case being important (whether letters are lowercase or UPPERCASE) but now you need to also pay attention to the type of value that is involved -- for example, numbers are differentiated between integers and floats (essentially, non-integers). These points are especially important to keep in mind:
Currently, VCF INFO field keys are case-sensitive. That means that if you have a
QUALfield in uppercase in your VCF record, the system will not recognize it if you write it differently (
qualor whatever) in your JEXL expression.
The types (i.e. string, integer, non-integer, floating point or boolean) used in your expression must be exactly the same as that of the value you are trying to evaluate. In other words, if you have a QUAL field with non-integer values (e.g. 45.3) and your filter expression is written as an integer (e.g. "QUAL < 50"), the system will throw a hissy fit (specifically, a Java exception, e.g. a
NumberFormatExceptionfor numerical type mismatches).
Complex queries
We highly recommend that complex expressions involving multiple AND/OR operations be split up into separate expressions whenever possible to avoid confusion. If you are using complex expressions, make sure to test them on a panel of different sites with several combinations of yes/no criteria.
6. More complex JEXL magic
Note that this last part is fairly advanced and not for the faint of heart. To be frank, it's also explained rather more briefly than the topic deserves. But if there's enough demand for this level of usage (click the "view in forum" link and leave a comment) we'll consider producing a full-length tutorial.
Introducing the VariantContext object
When you use SelectVariants with JEXL, what happens under the hood is that the program accesses something called the VariantContext, which is a representation of the variant call with all its annotation information. The VariantContext is technically not part of GATK; it's part of the
variant library included within the Picard tools source code, which GATK uses for convenience.
The reason we're telling you about this is that you can actually make more complex queries than what the GATK offers convenience functions for, provided you're willing to do a little digging into the VariantContext methods. This will allow you to leverage the full range of capabilities of the underlying objects from the command line.
In a nutshell, the VariantContext is available through the
vc variable, and you just need to add method calls to that variable in your command line. The bets way to find out what methods are available is to read the VariantContext documentation on the Picard tools source code repository (on SourceForge), but we list a few examples below to whet your appetite.
Using VariantContext directly
For example, suppose I want to use SelectVariants to select all of the sites where sample NA12878 is homozygous-reference. This can be accomplished by assessing the underlying VariantContext as follows:
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V variants.vcf \ -select 'vc.getGenotype("NA12878").isHomRef()'
Groovy, right? Now here's a more sophisticated example of JEXL expression that finds all novel variants in the total set with allele frequency > 0.25 but not 1, is not filtered, and is non-reference in 01-0263 sample:
! vc.getGenotype("01-0263").isHomRef() && (vc.getID() == null || vc.getID().equals(".")) && AF > 0.25 && AF < 1.0 && vc.isNotFiltered() && vc.isSNP() -o 01-0263.high_freq_novels.vcf -sn 01-0263
Using the VariantContext to evaluate boolean values
The classic way of evaluating a boolean goes like this:
java -Xmx4g -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V my.vcf \ -select 'DB'
But you can also use the VariantContext object like this:
java -Xmx4g -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V my.vcf \ -select 'vc.hasAttribute("DB")'
Using VariantContext to access annotations in multiallelic sites
The order of alleles in the VariantContext object is not guaranteed to be the same as in the VCF output, so accessing the AF by an index derived from a scrambled alleles array is dangerous. However! If we have the sample genotypes, there's a workaround:
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V multiallelics.vcf \ -select 'vc.hasGenotypes() && vc.getCalledChrCount(vc.getAltAlleleWithHighestAlleleCount())/(1.0*vc.getCalledChrCount()) > 0.1' -o multiHighAC.vcf
The odd 1.0 is there because otherwise we're dividing two integers, which will always yield 0. The
vc.hasGenotypes() is extra error checking. This might be slow for large files, but we could use something like this if performance is a concern:
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V multiallelics.vcf \ -select 'vc.isBiallelic() ? AF > 0.1 : vc.hasGenotypes() && vc.getCalledChrCount(vc.getAltAlleleWithHighestAlleleCount())/(1.0*vc.getCalledChrCount()) > 0.1' -o multiHighAC.vcf
Where hopefully the ternary expression shortcuts the extra
vc calls for all the biallelics.
Using JEXL to evaluate arrays
Sometimes you might want to write a JEXL expression to evaluate e.g. the AD (allelic depth) field in the FORMAT column. However, the AD is technically not an integer; rather it is a list (array) of integers. One can evaluate the array data using the "." operator. Here's an example:
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V variants.vcf \ -select 'vc.getGenotype("NA12878").getAD().0 > 10'
If you would like to select sites where the alternate allele frequency is greater than 50%, you can use the following expression:
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V variants.vcf \ -select vc.getGenotype("NA12878").getAD().1 / vc.getGenotype("NA12878").getDP() > 0.50
Questions up to August 2014 have been moved to an archival thread:
Hi Geraldine_VdAuwera,
Thanks for the detail about JEXL expression .I used SelectVariants -select 'vc.getGenotype("SampleN").isHet' or .isHomVar for selecting the known genotype like "0/1 or 1/1". But in my VCF file, there are some missing genotype like "." , and I also want to select those unknow genotype. Can you give me some comments for this. Thanks.
Hi Xie,
To select the unknown genotypes (./.) you can select for anything that is not homref, het, or homvar.
Hi Sheila, thank you for you timely reply. I select by vc.getGenotype('sample08').isnotHomVar() ..., but there is error:
"Invalid JEXL expression detected for select-0 with message ![72,85]: vc.getGenotype('sample08').isnotHomVar() && vc.getGenotype('sample08').isnotHet();' unknown, ambiguous or inaccessible method isnotHomVar"
I think the problem is incorrect JEXL expression. So are there some JEXL expressions for select setting ?
Hi Xie,
Instead of using vc.getGenotype('sample08').isnotHomVar(), try using
! vc.getGenotype('sample08').isHomVar(). The ! means not.
Hi Sheila,
Thanks again for your reply, It makes sense.
I want to be able to select the sites that have a DP >= 5 for all my samples. I have 24 samples, all of them "C??" (i.e. C01, C04, C12, etc). I tried using the * wildcard character like this: 'vc.getGenotype("C*").getDP() >= 5' , but this gave me an error. Is there a way to do this, or will I have to write the command individually for each sample (separated by &&)? Thanks for the help.Post edited by jullfly on
Unfortunately, you cannot iterate over sample names using the variant context operations. You will have to make a script to run on each sample name.
Thanks for your response @Sheila. It's ok, I can script it for all sample names but it's just not as neat and elegant of a code as it would be if there was a way to iterate through. Oh well, it will do what I need it to do nevertheless!
Hi, sorry but I have another question. I keep getting errors with this filtering expression, and I can't figure out what I am doing wrong.
(Note: this is a snippet of my script, $java, $GATK, and $ref variables are defined elsewhere in the script).
$java -Xmx2g -jar $GATK \ -T SelectVariants \ -R $ref \ --restrictAllelesTo BIALLELIC \ --variant Variants.SNPs.filtered1.vcf.C01.5.vcf \ --select_expressions 'vc.getGenotype("C01").isHet()' \ --select_expressions '((double)(vc.getGenotype("C01").getAD().0) / (vc.getGenotype("C01").getAD().1 + vc.getGenotype("C01").getAD().0)) > 0.25' \ --select_expressions '((double)(vc.getGenotype("C01").getAD().0) / (vc.getGenotype("C01").getAD().1 + vc.getGenotype("C01").getAD().0)) < 0.75' \ -o Variants.Het.SNPs.C01.5.vcf
This is the error message:
java.lang.IllegalArgumentException: Argument select-2has a bad value. Invalid expression used.
I double-checked my casting rules in Java for casting the int of AD into a double, and it seems fine to me. I even tested an analogous situation with simple numbers and it works.
boolean x = ((double) (1) / (2+3)) < 1; //x is true
Help would be greatly appreciated!Post edited by Geraldine_VdAuwera on
Hmm, I'm not sure JEXL allows you to do type casting -- I've never seen it used in a JEXL expression. To be honest I would probably just dump the data using VariantsToTable and do the selection in R, which is more flexible for complex queries.
Thanks @Geraldine_VdAuwera, I will keep that option in mind as I continue filtering my variants. But actually I was able to get the above script to work by converting it to a float instead of a double. Others may find this useful:
Oh nice, glad to hear you got it to work. And thanks for reporting your solution for other users' benefit.
mmterpstra NetherlandsMember ✭✭
Because this has moved a few times since my last post.
For people who can read (a little) java, here is the link to the implementation:
I'm not sure this link works, so alternatively go to:
https://github.com/samtools/htsjdk/ and find it below /src/java/htsjdk/variant/variantcontext/VariantContext.java
It has all the cool stuffs like vc.isPolymorphicInSamples() vc.isIndel()
newbie16 Member ✭
I have a vcf which in FORMAT field has a tag HQ which is an array of integers (e.g. 40,200)
CHROM POS ID REF ALT QUAL FILTER INFO FORMAT sampleIdGoesHere
1 815233 . TC CA 540 PASS AN=2;AC=1;RPM=L1M3|L1|31.4;SEGDUP=8 GT:VT:FT:HFT:GQ:HQ:PL:DP:AD 0/1:ref,delins:PASS:PASS,PASS:341:341,488:540,0,364:249:46,200
1 815521 . T . 358 PASS AN=2;END=815522 GT:VT:FT:HFT:GQ:HQ:DP:AD:PS 0|0:ref,ref:PASS:PASS,PASS:358:358,410:91:91:815520
I want to filter records such that either of the HQ values are >200
Is there a way to do that ?
I think using JEXL should work. Have a look at this document: http://gatkforums.broadinstitute.org/discussion/1255/variant-filtering-methods-involving-jexl-queries The last section "Example using JEXL to evaluate arrays" should help.
I'd like to perform site-level select or filtering on a VCF, using a criteria based on genotype attributes. Specifically, I'd like do things like select/filter sites where at least one genotype has depth > X, or where all genotypes have depth < Y. There is a post above about evaluating arrays, which I think is analogous to what I'm after. However, that example seems to be subsetting a specific genotype: 'vc.getGenotype("NA12878").getAD().0 > 10', as opposed to doing some sort of loop or aggregate (for example, max(DP) could accomplish what I need). Is this sort of thing possible? Thanks in advance.
Hi Ben,
Unfortunately, you will need to to write your own script to do what you want. The evaluating arrays section refers only to evaluating annotations that are arrays of values (like AD).
@bbimber As a workaround I would recommend looking into VariantsToTable to output a table of the properties you're interested in, set up your evaluation logic in an Rscript (or a civilized scripting language) and use that to output a list of positions that you want to filter or keep, then use that to SelectVariants the subset that you want. Bit roundabout but less risky than scripting something to parse the VCF directly, in my opinion.
That last comment about AD actually brings up another question. I am hoping to apply genotype filters based on AD. I was trying something like:
-G_filter "isHet && g && g.hasAd() && g.getAD().1 < 5"
I would have assumed the Genotype was in scope (like vc); however, after reviewing the code, I dont think that's true. Could I confirm that?
Second, I think AD gets omitted. From what I can see, when the Genotype is put into the JEXLMap, a handful of fields are explicitly added, and then it defers to getExtendedAttributes(). However, if you look at FastGenotype, AD wont be part of getExtendedAttributes():
Is this an oversight?
The syntax for getting at AD values (e.g.
-select 'vc.getGenotype("NA12878").getAD().0 > 10') is given in the document above, see the last point in the doc.
sorry if the post wasnt clear, but i'm asking about genotype filters, not site-level filters. As noted above, from what I see in the code, the JEXLMap is populated through different codepaths.
doing a genotype-level filter on a hard-coded samplename wouldnt make sense, for example. i'm pretty sure 'vc' isnt put in scope for the genotype filters either.
Ah, I'm afraid we don't currently provide support for using advanced JEXL in genotype filters, sorry. It's just not something we use internally, so we've never explored this. What's currently on this page is all we can offer to help with at this time. If you figure out a way to do what you want, please share it so that others may benefit.
yeah, that's understandable. i wonder if you can answer this: the relevant code here is under HTS-JDK. It wouldnt be hard to submit a pull request to improve the situation (maybe genotype filters use a JEXLMap more like site filters). is this worth my time? will gatk4 use a similar or a completely different codepath?
I can't judge whether it would be worth your time because I don't know how much this capability is worth to you, but I can reassure you that GATK4 will continue to use htsjdk and use the same codepaths to enable JEXL queries for the foreseeable future. And if you do submit this with an example of how it can be leveraged from GATK, we will support your pull request (sometimes things get lost in htsjdk).
I have a question for you. I have a multi-sample vcf. I'd like to apply a filter where a variant PASS if at least one sample in that position has AD/DP > 0.2
Is there any way to do it?
Thanks for your help and your great job!
One slightly awkward workaround might be to do something like:
1) in the first pass apply a genotype filter. this could work with the caveat that I'm not sure you can get AD (see my above post)
2) use setFilteredGenotypeToNoCall (or whatever the parameter is called)
3) in pass 2, apply a site filter based on # samples called, # not-called, etc.
Or as a one off, I think you could use a JEXL expression that hard codes all your samples names and strings together a big OR expression:
-select 'vc.getGenotype("Sample1").getAD().0 / vc.getGenotype("Sample1").getDP() > 0.2 || vc.getGenotype("Sample2").getAD().0 / vc.getGenotype("Sample2").getDP() > 0.2 ... etc. '
I didnt check my JEXL syntax on the above, so it probably needs to be touched up. Hopefully the general idea makes sense.
thank you for your tip, but it doesn't work... Using that JEXL expression, all my sites results as FILTER and I have problems with homo GT field like 1/1 or 2/2. Just to be clear, I'd like to tag a site in my vcf as PASS if at least one sample has AD/DP > 0.2 for a variant, otherwise, FILTER.
Any other suggestions?
Matteo - sorry, that expression was just typed into the browser, not tested in any way. a couple guesses:
1) if you want the first alternate allele, should it be AD.1, not .0?
2) you might want to include checks like isHet or isNoCall. A homozygous ref animal will only have one value for AD, i think.
3) could you rewrite your ratio to be based off the reference depth? every genotype should have a value for AD.0.
4) I think JEXL is pretty tolerant to null values (for example, asking for AD.1 on a homozygote), but I am not all that familiar with how these are handled. I'd just be careful. In my experience GATK is pretty verbose about logging issues w/ JEXL filters.
good luck.
here's the PR for the change to improve genotype filtering. this should allow proper filtering on AD and other genotype-level attributes:
Got it, thanks @bbimber.
BTW - you asked for an example. The key differences are:
1) you can access the genotype object in genotype filters:
-G_filter "isHet && g.hasAd() && g.getAD().1 < 5"
in my opinion, this is much more useful than the above example for site filters that hard-codes specific sampleIDs. When you want to ask question like 'filter sites where X% of genotypes have Y', you could do one pass where you perform genotype filters and setFilteredGtToNoCall. after this, do a site filter on N_CALLED.
2) It fixes when I think was a bug in the original implementation, which should have omitted AD + PL from the JEXL map. This is because they were not included in Genotype.getExtendedAttributes().
I agree that makes a lot of sense.
I would recommend include this example explicitly in your pull request thread as it will help convince the gatekeepers of htsjdk that this is worthwhile, in case they don't look up this thread.
FYI : there is now a javascript-based engine in picard FilterVcf . It's, IMHO, much more powerful than JEXL . See :
for an example.
@lindenb Ah I saw that go through into htsjdk but must have missed the Picard FilterVcf update. That does look very powerful (thanks for contributing that!) but I do think we should take in @bbimber's PR to shore up our JEXL offering in GATK, as for reasonably simple queries the usage is more straightforward compared to writing javascript.
Hello - the PR to HTSJDK was accepted, which should allow much better genotype filtering using GATK. My question is: how frequently does GATK update the HTSJDK version it uses? If I read github correctly, GATK is currently pegged to htsjdk version 1.141. These changes are in 2.2.2. Am I reading this wrong? Thanks in advance.
That's correct. We typically update htsjdk when we cut a new release, which nowadays is about 3 times a year -- so you're in luck because the next release, GATK 3.6 is slated for mid-May. In fact there's a PR that just went in for updating htsjdk to 2.2.2 so it should be available in the nightly builds within a few days. Assuming all tests pass.
good news, thanks.
u0597274 Salt Lake CityMember ✭
Just thought I'd request a simple demo to filter out homopolymers.
There is an annotation call HomopolymerRun that you can use.
I am getting the following error message:
java -jar -Xmx16g /home/remington/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V 2ms01e.pHASER01.vcf -sn 2ms01e -o 2ms01e_only.pHASER01.vcf
ERROR A GATK RUNTIME ERROR has occurred (version 3.7-0-gcfedb67):
ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
ERROR If not, please post the error message, with stack trace, to the GATK forum.
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions https://software.broadinstitute.org/gatk
ERROR MESSAGE: For input string: ""
It used to work perfectly. One of the error suggests it might be a bug, so reporting.
Can you please submit a bug report so I can test this locally? Instructions are here.
MBK BaselMember ✭
I am wondering whether -select parameter for SelectVariants can be used to filter variants based on the variables in the FILTER field. Something like -select "FILTER == 'PASS' && FILTER == 'foo' && FILTER == 'bar'". Tested on the latest version of the suite, but does not work. Any suggestions?
I think the best thing to do is use
--setFilteredGtToNocall, then use either
I use JEXL in "-select" to run SelectVariants.
vc.getGenotype("SAMPLE").getAD().1 > 100and
vc.getGenotype("SAMPLE").getDP() > 2000work fine. But
vc.getGenotype("SAMPLE").getAD().1 / vc.getGenotype("SAMPLE").getDP() > 0.01never works and always outputs a vcf only containing the vcf headers and column headers. I am very sure some variants should be selected by above expression. I am using GATK3.7. Any possible reason for this ? Your help is much appreciated.
Hi Frank,
It looks like SelectVariants with that command will not work on sites that have missing DP or DP=0. The best thing to do is remove all sites that have missing DP or DP=0, then try running the command again.
Let us know if that works.
Hi Sheila,
I checked sample format field, found no missing DP and all DP values are non-zero. BTW, the input variant vcf was generated by running GATK
Hi Frank
Hmm. How many sites are you running on? Can you post some records that you think should be selected but are not?
Sorry Sheila. I was distracted by some other tasks. Here are two examples that should be selected but not:
9 5000000 . G T 1747.19 . AC=1;AF=0.250;AN=4;BaseQRankSum=-0.452;ClippingRankSum=-0.633;DP=3598;FS=0.000;MLEAC=1;MLEAF=0.250;MQ=59.36;MQRankSum=2.648;QD=0.79;ReadPosRankSum=5.331;SOR=0.644 GT:AD:DP:GQ:PL 0/0:1378,3:1381:99:0,4079,48391 0/1:1967,245:2212:99:1777,0,63806
17 7000000 . C T 16949.19 . AC=1;AF=0.250;AN=4;BaseQRankSum=1.629;ClippingRankSum=2.326;DP=4149;FS=0.000;MLEAC=1;MLEAF=0.250;MQ=57.99;MQRankSum=2.073;QD=6.25;ReadPosRankSum=9.251;SOR=0.729 GT:AD:DP:GQ:PL 0/0:1415,1:1416:99:0,4207,49936 0/1:1995,690:2685:99:16979,0,63586
Hi Frank,
Can you try running your SelectVariants command on just those sites with
-L? I may need you to submit a bug report.
I'd like to use vc.isPolymorphicInSamples() to get rid of monomorphic sites in a VCF. By monomorphic I mean that there is only one allele in all of the samples I consider, and I do not care what this allele is (REF or (any single) ALT allele).
(Another way of expressing what I am trying to get at is that, for a site, if the array 'AF' is of length 1, then its value != 1.00)
I first thought that I could just do this:
java -jar GenomeAnalysisTK.jar \
-T SelectVariants \
-R ref.fa \
-V in.vcf.gz \
-o out.vcf.gz \
-select 'vc.isPolymorphicInSamples()'
However, as far as I now understand from the method description (and partly from the code, but disclaimer: I cannot program in Java) this method would not select a site for removal if all the samples have genotypes 1/1 (or all have 2/2, or all have 3/3, etc). This is because the test that the method is implementing is whether or not the total number of ALT alleles among the samples == 0.
From the vc source code (lines 1020-1040 from https://sourceforge.net/p/picard/code/HEAD/tree/trunk/src/java/org/broadinstitute/variant/variantcontext/VariantContext.java#l1041):
/** * Genotype-specific functions -- are the genotypes monomorphic w.r.t. to the alleles segregating at this * site? That is, is the number of alternate alleles among all fo the genotype == 0? * * @return true if it's monomorphic */ public boolean isMonomorphicInSamples() { if ( monomorphic == null ) monomorphic = ! isVariant() || (hasGenotypes() && getCalledChrCount(getReference()) == getCalledChrCount()); return monomorphic; } /** * Genotype-specific functions -- are the genotypes polymorphic w.r.t. to the alleles segregating at this * site? That is, is the number of alternate alleles among all fo the genotype > 0? * * @return true if it's polymorphic */ public boolean isPolymorphicInSamples() { return ! isMonomorphicInSamples(); }
The function isMonomorphicInSamples() actually tests for a site which is monomorphic for the REF allele, but in the case of a site which is monomorphic for an ALT allele in the samples, it would not consider the site monomorphic, right?
Then because isPolymorphicInSamples() is !isMonomorphicInSamples(), this means that it considers sites as polymorphic when they are not actually segregating in the sample.
It seems to me that these functions would be doing something closer to what their names suggest they should be doing if the 'InSamples' part of the name was excised.
I really hope that I haven't fundamentally misunderstood something here.
Thanks very much for your time,
Hi Ben,
I think you only want sites where all samples are het in the VCF. In this case, I think the best thing to do is loop through all the samples in your VCF using
Hi Sheila,
I want(ed) to get sites where overall, there are two alleles amongst the samples. All the samples could be homozygous, as long as at least one sample was homozygous for an ALT allele.
I got what I wanted by using VariantsToTable to make a table of genotypes, performing the necessary logic in R, then writing a .list file of sites to exclude and giving that to SelectVariants.
I just thought that the isPolymorphicInSamples() method was misnamed a bit, but perhaps this is a philosophical disagreement stemming from my population-genetics-oriented point of view.
ronlevine Lexington, MAMember ✭✭
isMonomorphicInSamples- the site has no alternate alleles (ALT = .) OR if there are genotypes, there are no samples with alternate alleles (for 2 samples, the genotypes are 0/0 0/0).
isPolymorphicInSamples= NOT
isMonomorphicInSamples= the site has alternate alleles AND if there are genotypes, there are samples with an alternate allele.
The Variant Call Format definition of monomorphic is a site does not contain a variant (no alternate alleles and no genotypes with alternate alleles). You were expecting the population genetics definition, where all of the genotypes would have the same allele (for 1 ALT allele and 2 samples, 0/0 0/0 or 1/1 1/1).
If you want to make sure there is homozygous variant genotype and all of the genotypes are homozygous. You can use the following JEXL expression:
vc.getHomVarCount() != 0 && vc.getHomVarCount() + vc.getHomRefCount() == vc.getCalledChrCount()
-RonPost edited by ronlevine on
I spent almost all day today working out several filtration procedures on multisample vcf. Generally filtration based on single sample (using JEXL queries) is explained but examples on the filtration of vcf sites/lines where GT (genotype) information is computed from several samples to run filtration/selectvariants are not quite detailed.
I just thought about writing those examples here so other people don't have to pull as much hair, Lol !.
#### Tutorials .... !!! # method to call the sites that are all homozygous Variants (1/1, 2/2 genotype) sample size in vcf is 6 java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomVarCount() == 6' # outcome: no sites will have ambigous (./.) and/or homRef (0/0). All other homVar are allowed # in the above script if we want the site where at least 4 samples are homozygous ref or variants java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomVarCount() >= 4' # outcome: atleast 4 samples are homVar, rest can be any genotype (./., 0/0, 0/1, etc.) # at least 1 sample is homVar (1/1, 2/2 or 3/3) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomVarCount() != 0' # all the samples that are homRef (0/0 genotypes) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomRefCount() == 6' ## Note to remember: 6 chr = 3 samples (when samples are diploid) # sites that are all HomVar or nocalls (./.) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomVarCount() == vc.getCalledChrCount()/2' # outcome: no 0/0, 0/1 # Note: samples with GT = ./. is not counted as called chromosome ## Useful after samples are selected from multisample vcfs # sites where no genotypes were called for all the samples in the selected vcf # select vcf sites that are not called in any sample (i.e don't have any genotypes, only contain ./.) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getCalledChrCount() == 0' # outcome = vcf sites (lines) that are all no calls (GT = ./.) on all samples are selected # samples are either homRef (0/0) or no call (./.) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomRefCount() == vc.getCalledChrCount()/2' # outcome: vcf sites where all the samples are GT = ./. (no call) or GT = 0/0 # site where 6 chromosomes (exact) or 3 samples were called for diploid genome java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getCalledChrCount() == 6' # Note of Caution: 6 chr = 3 samples (when samples are diploid) # site where at least 3 samples (diploid) are called java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getCalledChrCount() >=6' # if I have 6-sample vcf java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getCalledChrCount() ==12' # 12 haploid chr = 6 samples (diploid) # outcome: sites that were called in all samples, there will be no GT = ./. # select vcf sites where no sample has hetVar (no GT = 0/1, 0/2), only GT = ./., 0/0, 1/1, 2/2 java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHetCount() == 0' # select vcf sites where no sample has hetVar (no Gt = 0/1, 0/2), and at least one sample is HomVar (GT = 1/1) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHetCount() == 0 && vc.getHomVarCount() >= 1' # site where at least 1 sample is homVar (GT = 1/1 and/or 2/2 etc.) java -jar /home/everestial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V MY.phased_variants.Final.vcf -o My.filtered_01.vcf -select 'vc.getHomVarCount() >= 1'
**If any mistake gets introduced I won't be able to change it. So, please follow: **
Other updates may be followed @ https://www.biostars.org/p/255362/.
@ronlevine @Geraldine_VdAuwera
I am having some trouble finding an appropriate vc JEXL queries. I want the vcf lines/sites that have same genotypes in all samples (including or excluding
no call GT = ./.).
Any suggestions.
There's probably some JEXL to do that; however, sometimes using VariantsToTable to make a TSV and filtering from there is easier. You could do that to find the whitelist of allowable sites. If you need this as VCF, you could write this as a list of intervals and pass that to SelectVariants using the -L argument. That'll give you a VCF with only those sites.
using vcfilterjs http://lindenb.github.io/jvarkit/VCFFilterJS.html ( or picard FilterVcf)
java -jar dist/vcffilterjs.jar -e 'function accept(vc) {for(var i=1;i<vc.getNSamples();i++) if(!vc.getGenotype(0).sameGenotype(vc.getGenotype(i))) return false; return true;}accept(variant); '
@lindenb : I tried running this with picard. No succes, looks like I am messing up the script. Any more suggestions on how to use Picard with your expression.
@lindenb : I was able to now run it using
Picard FilterVCF JSbut there didn't pan out as expected. Thank you though !
@lindenb : I was able to now run it using
Picard FilterVCF JSbut there didn't pan out as expected. Thank you though !
works fine for me, (it's not a hard filter)
$ cat NOT_SAME_GT.js function accept(vc) {for(var i=1;i<vc.getNSamples();i++) if(!vc.getGenotype(0).sameGenotype(vc.getGenotype(i))) return false; return true;} accept(variant); $ java -jar package/picard.jar FilterVcf I=in.vcf O=out.vcf JS=NOT_SAME_GT.js
Question about a JEXL command (filtering on depth of coverage across all samples for a variant)
"-filtername DP -filter "DP< 100",
If I am understanding all the filtering information correctly this command will sum all the DP values for each sample for a variant and remove the variants with <100 total DP values .
Is this a correct assessment of how this command will work?
There are two depth annotations. Have a look at this doc for the INFO annotation. Have a look at this doc for the FORMAT annotation.
If you would like to filter on the INFO annotation, you would use
--filterExpression. If you would like to filter on the FORMAT annotation, you would use
O.k., if I am understanding things correctly I need to change my code to: -filtername DP -filterExpression "DP < 100" to filter on a minimum coverage of a variant across all samples?
Yes, that is correct.
I'm tying to select out sites were the alt allele AD count is higher than 10, the problem is these are not always called as heterozygous, (I have high depth sequencing and I'm using the HC in classic single sample mode with
--output_mode EMIT_ALL_SITES) e.g. in this example:
3 187446740 . T . Infinity . AN=2;DP=1095;MQ=60.00 GT:AD:DP 0/0:1095:1095 3 187446741 . C . Infinity . AN=2;DP=1117;MQ=60.00 GT:AD:DP 0/0:1117:1117 3 187446752 . A . Infinity . AN=2;DP=1297;MQ=60.00 GT:AD:DP 0/0:1297:1297 3 187446763 . C . Infinity . AN=2;DP=1494;MQ=60.00 GT:AD:DP 0/0:1494:1494 3 187451574 . C . Infinity . AN=2;DP=1493;MQ=60.00 GT:AD:DP 0/0:1493:1493 3 187451609 rs1880101 A G 39794.03 . AC=2;AF=1.00;AN=2;BaseQRankSum=1.859;ClippingRankSum=0.000;DB;DP=995;ExcessHet=3.0103;FS=0.000;MLEAC=2;MLEAF=1.00;MQ=60.00;MQRankSum=0.000;QD=24.56;ReadPosRankSum=0.406;SOR=8.234 GT:AD:DP:GQ:PL 1/1:1,988:989:99:39808,2949,0 4 1803279 . T G 0 . AC=0;AF=0.00;AN=2;BaseQRankSum=-6.652;ClippingRankSum=0.000;DP=245;ExcessHet=3.0103;FS=89.753;MLEAC=0;MLEAF=0.00;MQ=59.97;MQRankSum=0.000;ReadPosRankSum=-2.523;SOR=6.357 GT:AD:DP:GQ:PL 0/0:211,23:234:99:0,364,6739 4 1803307 rs2305183 T C 2486.60 . AC=1;AF=0.500;AN=2;BaseQRankSum=-5.049;ClippingRankSum=0.000;DB;DP=215;ExcessHet=3.0103;FS=1.110;MLEAC=1;MLEAF=0.500;MQ=59.97;MQRankSum=0.000;QD=11.95;ReadPosRankSum=-0.045;SOR=0.809 GT:AD:DP:GQ:PL 0/1:116,92:208:99:2494,0,3673 4 1803671 . C A 0 . AC=0;AF=0.00;AN=2;BaseQRankSum=-0.880;ClippingRankSum=0.000;DP=450;ExcessHet=3.0103;FS=0.000;MLEAC=0;MLEAF=0.00;MQ=60.00;MQRankSum=0.000;ReadPosRankSum=-0.953;SOR=0.572 GT:AD:DP:GQ:PL 0/0:445,2:447:99:0,1272,15958 4 1803681 . T C 0 . AC=0;AF=0.00;AN=2;BaseQRankSum=-1.654;ClippingRankSum=0.000;DP=483;ExcessHet=3.0103;FS=0.000;MLEAC=0;MLEAF=0.00;MQ=60.00;MQRankSum=0.000;ReadPosRankSum=-0.422;SOR=0.664 GT:AD:DP:GQ:PL 0/0:479,2:481:99:0,1408,18538 4 1803703 . A G 0 . AC=0;AF=0.00;AN=2;BaseQRankSum=-1.704;ClippingRankSum=0.000;DP=458;ExcessHet=3.0103;FS=0.000;MLEAC=0;MLEAF=0.00;MQ=60.00;MQRankSum=0.000;ReadPosRankSum=0.299;SOR=0.497 GT:AD:DP:GQ:PL 0/0:454,2:456:99:0,1325,18095 4 1803704 rs2234909 T C 6676.60 . AC=1;AF=0.500;AN=2;BaseQRankSum=-2.605;ClippingRankSum=0.000;DB;DP=456;ExcessHet=3.0103;FS=1.753;MLEAC=1;MLEAF=0.500;MQ=60.00;MQRankSum=0.000;QD=14.71;ReadPosRankSum=0.324;SOR=0.849 GT:AD:DP:GQ:PL 0/1:220,234:454:99:6684,0,6366 4 1803824 rs2305184 C G 2030.60 . AC=1;AF=0.500;AN=2;BaseQRankSum=8.083;ClippingRankSum=0.000;DB;DP=124;ExcessHet=3.0103;FS=6.128;MLEAC=1;MLEAF=0.500;MQ=60.00;MQRankSum=0.000;QD=16.51;ReadPosRankSum=0.180;SOR=0.096 GT:AD:DP:GQ:PL 0/1:62,61:123:99:2038,0,1766 4 1805296 rs3135883 G A 3876.03 . AC=2;AF=1.00;AN=2;DB;DP=110;ExcessHet=3.0103;FS=0.000;MLEAC=2;MLEAF=1.00;MQ=60.00;QD=29.22;SOR=9.401 GT:AD:DP:GQ:PL 1/1:0,109:109:99:3890,326,0
I have no interest in the lines were there is no AD count for anything other than the REF, however I'd to select lines 6,7,8,12,13 and 14 as these are a variety of het and hom calls all of which have an alt allele AD count greater than 10, I can't however figure out how to select this using JXEL? Any suggestions? I've attached a cut-down VCF file of above.Post edited by MattB on
Just to update you, there are now both vcfilterjs (from Pierre Lindenbaum) and a bcftools (less optimal) solutions to this issue on bioinformatics stack exchange (it's in Beta so please give it a go!), in case anyone arriving here has the same issue.
Hi Matt!
Sorry for the delay, but we were preparing for the workshop
If you look at the bottom of this document under "Using JEXL to evaluate arrays", you should get what you are looking for. You just need to substitute in
-select 'vc.getGenotype("NA12878").getAD().1 > 10'for
-select 'vc.getGenotype("NA12878").getAD().0 > 10'.
P.S. Nice to meet you there and see you here
Hi Sheila, thanks, I always try to keep my pipeline pure GATK/Picard if possible! What's the difference here in the
getAD().1 > 10to the
getAD().0is .0 only referring to reference calls? And would I need to run
getAD().2to get the second alternative allele and so on?
Also my aim here is to recover potentially heterozygous sites which have not been called in a high-depth panel before feeding them into VEP to annotate them, since VEP will only annotate the "called" 0/1 or 1/1 genotypes is there a way of using said filter to alter / refine the genotype of 0/0 to 0/1 for such cases were AD of an alt allele is > 10, other than resorting to some kind of grep / sed / awk mashup? I Appreciate this is an odd / edge usage case.
P.S. Very nice to meet you all in sunny Edinburgh!
Hi Matt,
Yes, the .0 refers to the reference allele; the .1 refers to the first alternate allele; the .2 refers to the second alternate allele, and so on.
As for actually altering the genotype call, I don't think there is any way to do that unless you manually alter them yourself (which we don't recommend). You can try the Genotype Refinement Workflow to see if that changes any of the genotypes after additional information is available.
I am having trouble with this command. I have a vcf file with 100's of samples. And I want to apply sample level filter on allele frequency which is AD/DP. I have been trying it with this command but no luck. It marks all entries as filtered. When I simply tried with AD.0 >10, the command works as expected which tells me that it recognized AD.0 as number of reference reads. But somehow the division does not give me the right answer. Can you help?
java -jar GenomeAnalysisTK.jar -T VariantFiltration -R .ucsc_hg19.fa -V test.vcf -o test.vf.vcf -G_filter "(AD.0) / DP >0.8" --genotypeFilterName 'lowAD'
You need to use
vc.getGenotype("sample").getAD().0instead of
AD.0. Have a look at this thread as well.
Hi @Sheila, Thank you. Let me try those.
I am getting a
bug erroron 3.8 but not on 3.7 for the same ditto code.
I am trying to select the variants that are Hets in all called samples.
On 3.8:
$ java -jar /home/everestial007/GenomeAnalysisTK-3.8/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V DNA_Samples.Passed_SNPs.vcf -o DNA_Samples.Passed.SNPs_allHetsRemoved.vcf -select 'vc.getHetCount() == vc.getCalledChrCount()/2' -invertSelect
INFO 16:22:21,892 HelpFormatter - ----------------------------------------------------------------------------------
INFO 16:22:21,894 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-0-ge9d806836, Compiled 2017/07/28 21:26:50
INFO 16:22:21,894 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute
INFO 16:22:21,894 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk
INFO 16:22:21,894 HelpFormatter - [Wed Dec 20 16:22:21 EST 2017] Executing on Linux 4.4.0-101-generic amd64
INFO 16:22:21,895 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12
INFO 16:22:21,897 HelpFormatter - Program Args: -T SelectVariants -R lyrata_genome.fa -V DNA_Samples.Passed_SNPs.vcf -o DNA_Samples.Passed.SNPs_allHetsRemoved.vcf -select vc.getHetCount() == vc.getCalledChrCount()/2 -invertSelect
INFO 16:22:21,899 HelpFormatter - Executing as [email protected] on Linux 4.4.0-101-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12.
INFO 16:22:21,900 HelpFormatter - Date/Time: 2017/12/20 16:22:21
INFO 16:22:21,900 HelpFormatter - ----------------------------------------------------------------------------------
INFO 16:22:21,900 HelpFormatter - ----------------------------------------------------------------------------------
ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/home/everestial007/GenomeAnalysisTK-3.8/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
INFO 16:22:22,010 GenomeAnalysisEngine - Deflater: IntelDeflater
INFO 16:22:22,010 GenomeAnalysisEngine - Inflater: IntelInflater
INFO 16:22:22,010 GenomeAnalysisEngine - Strictness is SILENT
INFO 16:22:22,292 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000
ERROR stack trace
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at htsjdk.tribble.index.IndexFactory.loadIndex(IndexFactory.java:187)
at htsjdk.tribble.index.IndexFactory.loadIndex(IndexFactory.java:165)
at org.broadinstitute.gatk.utils.refdata.tracks.RMDTrackBuilder.loadFromDisk(RMDTrackBuilder.java:375)
at org.broadinstitute.gatk.utils.refdata.tracks.RMDTrackBuilder.attemptToLockAndLoadIndexFromDisk(RMDTrackBuilder.java:359)
at org.broadinstitute.gatk.utils.refdata.tracks.RMDTrackBuilder.loadIndex(RMDTrackBuilder.java:319)
at org.broadinstitute.gatk.utils.refdata.tracks.RMDTrackBuilder.getFeatureSource(RMDTrackBuilder.java:264)
at org.broadinstitute.gatk.utils.refdata.tracks.RMDTrackBuilder.createInstanceOfTrack(RMDTrackBuilder.java:153)
at org.broadinstitute.gatk.engine.datasources.rmd.ReferenceOrderedQueryDataPool.(ReferenceOrderedDataSource.java:208)
at org.broadinstitute.gatk.engine.datasources.rmd.ReferenceOrderedDataSource.(ReferenceOrderedDataSource.java:88)
at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.getReferenceOrderedDataSources(GenomeAnalysisEngine.java:1071)
at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.initializeDataSources(GenomeAnalysisEngine.java:851)
at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:294)
at org.broadinstitute.gatk.engine.CommandLineExecutable.execute(CommandLineExecutable.java:123)
at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:256)
at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
at org.broadinstitute.gatk.engine.CommandLineGATK.main(CommandLineGATK.java:108)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at htsjdk.tribble.index.IndexFactory.loadIndex(IndexFactory.java:181)
... 15 more
Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 80: /media/everestial007/SeagateBackup4.0TB/RNAseq_Data_Analyses/02_Alignment_To_Ref - New/GVCF_Calls_onGenes-RunningTest01/02-JointGenotyping_MySpF1/DNA_Samples.Passed_SNPs.vcf
at java.net.URI.create(URI.java:852)
at htsjdk.samtools.util.IOUtil.getPath(IOUtil.java:984)
at htsjdk.tribble.index.AbstractIndex.readHeader(AbstractIndex.java:305)
at htsjdk.tribble.index.AbstractIndex.read(AbstractIndex.java:397)
at htsjdk.tribble.index.linear.LinearIndex.(LinearIndex.java:119)
... 20 more
Caused by: java.net.URISyntaxException: Illegal character in path at index 80: /media/everestial007/SeagateBackup4.0TB/RNAseq_Data_Analyses/02_Alignment_To_Ref - New/GVCF_Calls_onGenes-RunningTest01/02-JointGenotyping_MySpF1/DNA_Samples.Passed_SNPs.vcf
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parseHierarchical(URI.java:3105)
at java.net.URI$Parser.parse(URI.java:3063)
at java.net.URI.(URI.java:588)
at java.net.URI.create(URI.java:850)
... 24 more
ERROR A GATK RUNTIME ERROR has occurred (version 3.8-0-ge9d806836):
ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.
ERROR If not, please post the error message, with stack trace, to the GATK forum.
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions https://software.broadinstitute.org/gatk
ERROR MESSAGE: java.lang.reflect.InvocationTargetException
On 3.7
$ java -jar /home/evere stial007/GenomeAnalysisTK-3.7/GenomeAnalysisTK.jar -T SelectVariants -R lyrata_genome.fa -V DNA_Samples.Passed_SNPs.vcf -o DNA_Samples.Passed.SNPs_allHetsRemoved.vcf -select 'vc.getHetCount() == vc.getCalledChrCount()/2' -invertSelect
INFO 16:23:11,550 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:23:11,552 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18
INFO 16:23:11,552 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute
INFO 16:23:11,553 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk
INFO 16:23:11,553 HelpFormatter - [Wed Dec 20 16:23:11 EST 2017] Executing on Linux 4.4.0-101-generic amd64
INFO 16:23:11,553 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12
INFO 16:23:11,557 HelpFormatter - Program Args: -T SelectVariants -R lyrata_genome.fa -V DNA_Samples.Passed_SNPs.vcf -o DNA_Samples.Passed.SNPs_allHetsRemoved.vcf -select vc.getHetCount() == vc.getCalledChrCount()/2 -invertSelect
INFO 16:23:11,560 HelpFormatter - Executing as [email protected] on Linux 4.4.0-101-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12.
INFO 16:23:11,560 HelpFormatter - Date/Time: 2017/12/20 16:23:11
INFO 16:23:11,561 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:23:11,561 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:23:11,593 GenomeAnalysisEngine - Strictness is SILENT
INFO 16:23:11,939 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000
INFO 16:23:12,180 GenomeAnalysisEngine - Preparing for traversal
INFO 16:23:12,182 GenomeAnalysisEngine - Done preparing for traversal
INFO 16:23:12,183 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 16:23:12,183 ProgressMeter - | processed | time | per 1M | | total | remaining
INFO 16:23:12,184 ProgressMeter - Location | sites | elapsed | sites | completed | runtime | runtime
INFO 16:23:12,792 SelectVariants - 538 records processed.
INFO 16:23:12,826 ProgressMeter - done 538.0 0.0 s 19.9 m 53.9% 0.0 s 0.0 s
INFO 16:23:12,826 ProgressMeter - Total runtime 0.64 secs, 0.01 min, 0.00 hours
Done. There were no warn messages.
Hi @everestial007,
I'm not sure if this is related but could you see if adding
-jdk_deflaterand -
jdk_inflaterto the erring v3.8 command allows the run to succeed? These parameters are discussed here.
lidia Member ✭
Hello GATK team!
I am writing to ask if there is a way to filter all the HomVar SNPs from a file with multiple samples.
Let me explain a little more in detail. I am working with two strains of fish, each strain with 8 samples and I have a referece genome. But right now I need to select only the SNPs that are present in just one of the strains (basically I need to compare between strains, not to the reference). Therefore I need to filter out all the ones that compared to the Reference are different, but that between strains are the same.
Is this possible?
Did you look at the tutorial I posted above in this comment thread. It is also available via biostars: https://www.biostars.org/p/255362/
I am sure there is a way to your problem in there.
Hi @lidia,
The tutorial shared by @everestial007 uses JEXL expressions, which is one way to approach this.
Another approach to consider using ChromosomeCounts annotations. First, you should subset your callset with SelectVariants to retain only the samples per strain. Then, given the requirement that all samples in the strain-callset have hom-var sites you can consider using the expected value for
ANfor your cohort to filter.
Is there any way to put variables into JEXL -select command, e.g vc.getGenotype("$i")? because -select only read literal expression between single quote. That will be troublesome in case you would like to filter 100 vcf files with many different sample names.
Is there any better alternative than using JEXL to select variants, please share your advice.
Thank you very much.
I responded here. No need to post twice.
@Sheila said:
I responded here. No need to post twice.
Hi, thank you for your reply, I am sorry for the duplicated post.
Greg_Owens Member ✭
I'm having an issue with selecting variants based on minor allele frequency while subsetting a smaller set. What I'd like to do is subset to a few samples, and retain anything that has variation (i.e. any AF > 0). My command is:
gatk SelectVariants -sample-name SAMPLE1 -sample-name SAMPLE2 -V HanXRQChr01.tranche99.snp.vcf.gz -O tmp.vcf.gz --remove-unused-alternates -select 'vc.isBiallelic() ? AF > 0.01 : vc.hasGenotypes() && vc.getCalledChrCount(vc.getAltAlleleWithHighestAlleleCount())/(1.0*vc.getCalledChrCount()) > 0.01'
This runs for a short while and then outputs:
A USER ERROR has occurred: Invalid JEXL expression detected for select-0
See https://www.broadinstitute.org/gatk/guide/article?id=1255 for documentation on using JEXL in GATK
I believe the issue is the fact that I'm taking a dataset with many samples and selecting a few samples, so many alleles are being removed. If I retain all samples, it works fine. If I force it to only select biallelic sites, it works, but I want to keep the multiallelic sites as well.
What I'd like to do is subset to a few samples, and retain anything that has variation
What happens if you just use
-sn sample1 -sn sample2 ... --remove-unused-alternates?
FPBarthel HoustonMember ✭✭
I have a field
AOin my VCF that I would like to filter on so I am using
vc.getGenotype("sample_id").getAO() > 0with
VariantFiltration, however that doesn't seem to work. I suspect
getAO()is not implemented. Is there some way to filter based on this?
##FORMAT=<ID=AO,Number=A,Type=Integer,Description="Alternate allele observations, with partial observations recorded fractionally">
UPDATE: Re-reading the page again I found the link to some Picard Java code here, and found that there exists a function
getAttributeAsIntand tried
vc.getGenotype("sample_id").getAttributeAsInt("AO",0) > 0which seems to work. Thought I'd leave this here for anyone else who may find this useful.
Here is snippet of my vcf file (from Mutect2)
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT @[email protected] @[email protected] chr1 985852 . C T . PASS DP=256;ECNT=1;NLOD=26.18;N_ART_LOD=-1.944e+00;POP_AF=2.500e-06;P_CONTAM=0.00;P_GERMLINE=-3.811e+01;REF_BASES=GAGGCCCCAGCGGCCTCCTGC;TLOD=92.21 GT:AD:AF:F1R2:F2R1:MBQ:MFRL:MMQ:MPOS:SA_MAP_AF:SA_POST_PROB 0/1:130,35:0.220:84,16:46,19:31,33:193,224:42:26:0.202,0.192,0.212:8.067e-03,0.017,0.975 0/0:87,0:2.048e-04:67,0:20,0:32,0:222,0:0:0
I would like to filter AF attribute in FORMAT field but the JEXL expression was not working:
-select 'vc.getGenotype("@[email protected]").getAF().floatValue() >= 0.200'
error obtained: "A USER ERROR has occurred: Invalid JEXL expression detected for select-0"
Is there any expression available for this AF attribute? I guess "getAttributeAsInt" is for integer value only, how about float value ?
UPDATE: This expression work for me, maybe somebody could find it helpful:
-select 'vc.getGenotype("@[email protected]").getAttributeAsDouble("AF",0) >= 0.200'
However, there are several points I don't understand:
Post edited by Phuong_Nguyen_1 on
- What is the "default value" of 0 in getAttributeAsDouble, what is its purpose ?
- Additionally, I am still confused regarding AF calculation in Mutect2, for example in my snippet: AF = 0.220 which is higher than Alt_AD/(Ref_AD+Alt_AD) , i.e 35/(130+35)=0.212. Could someone explain this different?
Thank you very much !
Hi @Phuong_Nguyen_1,
If you search our forum with
getAttributeAsDouble, you will come across a post that explains the second variable. The solution is provided by a GATK forum veteran @pdexheimer here. The second variable, in your case
0, is the default value in case the annotation is nonexistant.
sahiilseth Member ✭
hi @Sheila the division of AD/DP does not work as expected unless we convert it to float. Could you update the post with converting to float. I had processed the samples and realized all the variants were being removed when using selectvariants
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V variants.vcf \ -select vc.getGenotype("NA12878").getAD().1 / vc.getGenotype("NA12878").getDP() > 0.50
updated with
java -jar GenomeAnalysisTK.jar -T SelectVariants \ -R reference.fasta \ -V variants.vcf \ -select vc.getGenotype("NA12878").getAD().1*1.0 / vc.getGenotype("NA12878").getDP() > 0.50
samlj Member ❠@sahiilseth I've been trying to get this to work as well, so your hint was very helpful, but I only want the genotypes which are heterozygous to be filtered out. When I use a compound expression like
java -Xmx2g -jar $GATK VariantFiltration \
-R $reference \
-O $output_VCF \
--genotype-filter-expression "(vc.getGenotype(\"subject1\").isHet()) && ((vc.getGenotype(\"subject1\".getAD().1*1.0)/(vc.getGenotype(\"subject1\").getDP()) < 0.2)" \
--genotype-filter-name "alternateDepth"
any sample that is heterozygous at a locus where subject1 has a low alternate allele frequency (regardless of this sample's alternate allele frequency) is also filtered:
#FORMAT subject1 subject2 subject3
GT:AD:DP:FT:GQ 0/1:30,1:31:alternateDepth:99 ./.:0,0:0:PASS:0 ./.:1,1:2:PASS:3
GT:AD:DP:FT:GQ 0/1:30,1:31:alternateDepth:99 0/1:12,13:25:alternateDepth:0
Is this the expected behavior? Is there a way to change it?
(Sorry for the formatting - my markdown isn't working for some reason)


text:
Before you toss those stick figure drawings your child did at preschool, you might want to take a close look.
Researchers at the Institute of Psychiatry at King's College London conducted a study revealing that children's drawings might offer a clue to their intelligence as a toddler and also to their I.Q. level down the road.
For the study, the team looked at drawings from 7,752 pairs of identical and non-identical twins (a total of 15,504 children) that were collected for the Medical Research Council's Twins Early Development Study. Each child was asked to draw another child and their figures were scored between 0 and 12. Higher scores were given to those children who included the correct quantity of features such as head, eyes, nose, mouth, ears, hair, body, and arms. A drawing with two legs, two arms, a body and head, but no facial features, earned a score of 4, for example.
The preschoolers were also given a verbal intelligence test and then the same children were given an I.Q. test at age 14.
The researchers found that higher scores on the drawing test were moderately associated with higher scores of intelligence at ages 4 and 14.
"The Draw-a-Child test was devised in the 1920's to assess children's intelligence, so the fact that the test correlated with intelligence at age 4 was expected,"Dr. Rosalind Arden, the paper's lead author, said in a statement. "What surprised us was that it correlated with intelligence a decade later."
Dr. Arden adds that parents shouldn't be alarmed if their 4-year-olds' stick figures have three eyes and one arm. "The correlation is moderate, so our findings are interesting," Dr. Arden said, "but it does not mean that parents should worry if their child draws badly. Drawing ability does not determine intelligence, there are countless factors, both genetic and environmental, which affect intelligence in later life."
Another interesting finding from the study: Figures from identical twins pairs were more similar to one another than drawings from non-identical twin pairs, indicating the presence of a drawing genetic link.
"This does not mean that there is a drawing gene – a child's ability to draw stems from many other abilities, such as observing, holding a pencil et cetera," Dr. Arden explained. "We are a long way off from understanding how genes influence all these different types of behavior."

subset: commoncrawl


text:
What will it take?
joinsview Uncategorized 1 Minute
Published by joinsview
I believe the citizens of this country have a right to be heard. I believe Washington is only listening to the big money side of politics. 'We the People' don't have a chance in hell of changing our political system unless we vote. The country is more divided today than during the Civil War. And, we have a president who is a white nationalist, puts children in cages, hate crimes on the rise, and more mass shootings occurring with more frequency. He blames everyone but himself when life does not go his way. He alienates our allies. Yet, his relationship with dictators, China, Korea, and Russia has brought us to the brink of war. Our environment is on a steep decline. And, what kind of president calls everything "Fake News," or berets anyone who disagrees with him? Ask yourself why all of this has mostly happened since Trump took office? People get out and vote!! We truly need to open our eyes and see what's at stake if we don't! Perhaps, we will only have ourselves to blame for our country's demise. Together we will bring progress back to our country. View all posts by joinsview


text:
via press release:
ON SUNDAY, FEBRUARY 12 at 9 PM ET/PT
Encore Presentations of "Seizing Justice: The Greensboro 4" and "Black Wings" Among Other
Black History Month Offerings
New York, January 25, 2012 – Smithsonian Channel celebrates Black History Month in February with a programming slate that is anchored by the premiere of MLK: THE ASSASSINATION TAPES, on Sunday, February 12 at 9 p.m. ET/PT. The story of the last days of Dr. Martin Luther King Jr. is told through rare television and radio news accounts, much of which has not been seen or heard since 1968. Other primetime highlights include encore airings of SEIZING JUSTICE: THE GREENSBORO 4 and BLACK WINGS.
MLK: THE ASSASSINATION TAPES is a chilling, immersive experience, created almost entirely through the use of television and radio news footage from the weeks leading up to Dr. King's visits to Memphis through the aftermath of his murder. The one-hour special draws extensively from the unique materials at the Special Collections Division at The University of Memphis.
When Memphis' mostly black sanitation workers went on strike on February 11, 1968, several University faculty members, believing this was a seminal moment in the civil rights movement, began collecting every piece of media they could find – television, radio and print. When Dr. King arrived in town to lend his voice to their cause, the team from the University was already gathering material in full-force. The process continued throughout both of Dr. King's visits – and after his killing. Much of the footage has not been seen by the public since it was first gathered in 1968…until now.
MLK: THE ASSASSINATION TAPES can be accessed for free on multiple platforms beginning February 12 including SmithsonianChannel.com (http://www.smithsonianchannel.com/mlk) as well as on Smithsonian Channel's iPhone®, Android™ and Nook™ apps.
SEIZING JUSTICE: THE GREENSBORO 4 – Saturday, February 4 at 9:00 p.m. ET/PT and Thursday, February 9 at 8:00 p.m. ET/PT: What transpired at the Woolworth's lunch counter in Greensboro, North Carolina on February 1, 1960 sent shockwaves across America. SEIZING JUSTICE: THE GREENSBORO 4 tells the story of four college freshman who led a nonviolent sit-in and taught a country that it was finally time to serve everyone. Their simple request for coffee at America's favorite five-and-dime store sparked a series of events that would help put an end to the Jim Crow laws of the South. This one-hour program details the extraordinary story of four otherwise ordinary young men – Franklin McCain, Joseph McNeil, David Richmond and Ezell Blair, Jr. – African-American college students, whose nonviolent sit-in helped launch a revolution.
BLACK WINGS – Saturday, February 11 at 9:00 p.m. ET/PT and Thursday, February 16 at 8:00 p.m. ET/PT: For early aviators, conquering the forces of gravity was a daunting challenge. But black aviators had an additional challenge – the forces of racism. BLACK WINGS celebrates the stories of those black aviators who broke the color barrier and proved that the dream to fly transcends all boundaries. The one-hour special introduces an amazing range of black pioneers, pilots who flew everything from biplanes to commercial jets, from barnstormers to war fighters and opened the skies to all.
Smithsonian Channel is owned by Smithsonian Networks™, a joint venture between Showtime Networks Inc. and the Smithsonian Institution. Its programs are largely inspired by the assets of the world's largest museum complex. Smithsonian Channel features award-winning original documentaries, series, and groundbreaking programs highlighting America's historical, cultural and scientific heritage. Smithsonian Channel brings the American experience home in high definition and Dolby Digital 5.1 and is available to customers of DirecTV, Comcast, Time Warner Cable, Charter Communications, Cablevision, Verizon, AT&T, and more. Learn more at www.smithsonianchannel.com.


text: Sanofi Genzyme Announces it Will Resubmit Lemtrada Application to the FDA April 7, 2014 Today (April 7) Sanofi Genzyme announced it will be resubmitting its application to the U.S. Food and Drug Administration for approval of Lemtrada™ (alemtuzumab) as a treatment for relapsing MS in the United States. The company's previous application was denied approval in December 2013 (read more). The company announced it intends to submit during the second quarter of 2014. "We are encouraged by this development and will be monitoring the process closely as it moves forward," commented Timothy Coetzee, PhD, Chief Advocacy, Services and Research Officer at the National MS Society. The Society will continue to provide updates as we learn more information.


text: Share: February 26, 2013 Aldo Leopold Society In appreciation of the Cary Institute's most loyal donors, Irene and Jack Banning hosted the Aldo Leopold Society's annual Autumn Celebration at Black Sheep Hill in Pine Plains on October 13, 2012.The evening began with a scenic trek up their torch-lit driveway past rustic sculptures, the farm train station, sheep meadows, and barns to a cocktail reception in the Banning's home. More than 75 members of the Aldo Leopold Society were treated to a roaring fire on the patio and local fare prepared by Pine Plains Platter. With a spectacular sunset underway, President Bill Schlesinger offered welcoming remarks, along with updates on the Institute's recent accomplishments, and introduced the Cary Institute's core scientific staff.On behalf of the Board of Trustees, Irene Banning thanked members of the Society for their generous contributions that support the Institute's expanded outreach and education programs, and world-renowned environmental research programs. Share: Feature Aldo Leopold Society Autumn Celebration 2012 › February 26, 2013


text:
Trade News
Munich, 8 April 2013 – Technology company The Linde Group will be presenting the latest advances in eco-friendly energy carriers at HANNOVER MESSE 2013. It will be focusing in particular on hydrogen and liquefied natural gas (LNG).
Hydrogen fuelling technology from Linde in demand
As a leading provider of hydrogen fuelling solutions, Linde will once again be the exclusive supplier of hydrogen for the fuel-cell cars operated by the Clean Energy Partnership (CEP) in Hanover. CEP partners Daimler, GM/Opel, Honda and Toyota will be inviting exhibition visitors to take a test drive in their zero-emission vehicles in the open grounds. The cars will be refuelled with green hydrogen supplied by a mobile refuelling unit developed by Linde. The hydrogen will be sourced from Linde's pilot plant in Leuna, where it is manufactured from raw glycerol, a by-product of biodiesel manufacturing. Advanced to industrial scale, this production path has the potential to reduce greenhouse gas emissions by between 50 and 80 percent compared with conventional hydrogen production processes using natural gas.
Linde will also be showcasing the latest developments in stationary fuelling technologies. In the last two years, Linde developed and built three hydrogen stations which generate hydrogen directly on site through electrolysis. These have the potential to generate zero-carbon hydrogen using electricity from regenerative sources such as wind power. The most recent station of this type was opened in March in Stuttgart, built for EnBW Energie Baden-Württemberg AG, and is part of the CEP. "These next-generation stations not only contribute to Europe's growing hydrogen fuelling infrastructure," says Dr Andreas Opfermann, Head of Clean Energy and Innovation Management at Linde. "Enabled by power-to-gas technology, they also demonstrate the value of hydrogen as a storage medium in the transition to a cleaner energy landscape."
New applications for LNG
LNG is an economically viable bridging technology in the move towards more sustainable mobility choices. As the most environmentally friendly of all fossil fuels, LNG is a particularly attractive alternative to heavy oils or diesel for shipping and trucking companies. Driven by increasingly strict emission thresholds, demand for LNG is rising in these and other industries. Last year, Linde entered into a joint venture with marine fuel specialist Bomin. The new company, Bomin Linde LNG, is already successfully establishing a maritime LNG infrastructure.
Linde also set up an LNG fuelling station for the Stobart Group logistics specialist in Great Britain. This will serve some of Stobart's dual-fuel vehicles. A new fuelling technology means that virtually no gas escapes during the fuelling process. This innovation greatly improves the environmental balance of this station relative to conventional systems.
Linde masters the entire natural gas value chain – from liquefaction through storage to delivery to the point of use.
All visitors are welcome to visit Linde's stand in hall 27 from the 8th to the 12th of April in Hanover. The company will be presenting a host of clean technologies – including those outlined above.
About The Linde Group
The Linde Group is a world-leading gases and engineering company with around 62,000 employees in more than 100 countries worldwide. In the 2012 financial year, Linde generated revenue of EUR 15.280 bn. The strategy of the Group is geared towards long-term profitable growth and focuses on the expansion of its international business with forward-looking products and services. Linde acts responsibly towards its shareholders, business partners, employees, society and the environment – in every one of its business areas, regions and locations across the globe. The company is committed to technologies and products that unite the goals of customer value and sustainable development.
Under the "Clean Technology by Linde" label, the company offers a wide range of products and technologies that help to render renewable energy sources financially viable, and significantly slow down the depletion of fossil resources or reduce the level of CO2 emitted. This ranges from specialty gases for solar module manufacturing, industrial-scale CO2 separation and application technologies to alternative fuels and energy carriers such as liquefied natural gas (LNG) and hydrogen.
For more information go to http://www.linde.com/cleantechnology.
Further information:
Media Relations
Stefan Metz
Investor Relations
Lisa Tilmann


text:
Puppy Linux Discussion Forum
Puppy HOME page :
"THE" alternative forum :
Log in to check your private messages
Log in
The time now is Tue 07 Jul 2020, 15:58
All times are UTC - 4
Forum index
Advanced Topics
Puppy Derivatives
Macpup 520
View previous topic
View next topic
Page 1 of 9
[123 Posts]
Goto page:
Joined: 07 Jun 2008
Posts: 1651
Location: BigD Texas
Posted: Sun 20 Mar 2011, 09:33
Post subject:
Macpup 520
Subject description: e17 Macpup
Prit And I are proud to announce the release of Macpup 520,our newest
e17 Macpup.
MP520 is based on Puppy Linux 5.2 ,"Lucid Puppy", An official woof
build of puppy Linux that is binary-compatible with Ubuntu Lucid Lynx
packages.MP520 contains all the apps from Lucid puppy with the addition of
Firefox 4 rc 1. Extra apps like Opera or Gimp are available for easy
download from the Quickpet App on the ibar or the Puppy Package Manager.
MP520 also includes the Enlightenment E17 window manager. The EFL libraries
version 1.0.0 and E17 version 55225 where compiled and installed
from source.
All included E17 themes are By Verdegal37 and where customized for
Macpup by CatDude and Runtt21.
Thank's to everyone that has help or supported the Macpup project:
Ttuuxxx, CatDude, Puppyluvr, Mu, Sneekylinux, Rasterman and all Of the
E17 crew, ZaZoOo, WhoDo, Omskates, Cgt_1991. Joel, Russoodle,
GrumpyWolfe, Rokytnji , 01miko & Maxpro4u.
Extra Thanks to Agust(Verdegal37), The best E17 Themer on the planet.
And Louis King Jr. for the background in the grub bootloader.
Special Thanks to BarryK the creator of puppylinux. And PlayDaze the
developer of Lucid puppy.
IMPORTANT NOTICE FOR MACPUP 520. If Quickpet recommends your computer needs the Xorg-High-1.1-lucid.pet. DO NOT INSTALL IT FROM QUICKPET OR THE PUPPY PACKAGE MANAGER. It will cause the system to freeze up. We Have a fixed version , Xorg_High-1.1-macpup.pet & the ATI_fglrx-10.10-macpup.pet in the macpup520 folder on my page at:
The .iso is 174 mb.
The default theme:
The three extra themes:
Important note about themes, All of the mp511 themes will work in mp520. I have updated the default themes from mp511 for use in mp520. They are in the MP520-themes.pet's in the macpup520 folder on my macpup page.
This version of Macpup will boot to desktop in almost all cases and from that screen you can make adjustments if they are needed or wanted.The desktop and clock should be correct. If it does not boot to a graphical screen, it might boot to a "prompt." If that happens please enter 'xorgwizard' which will allow you to configure Macpup. If it does not boot to a prompt but a black screen, then you should reboot the CD and press f1 and enter 'puppy pfix=nox', in the space that asks for any boot options. Then at the prompt enter 'xorgwizard'.
If you connect to the internet by Ethernet As soon as the desktop comes up you will be online please click on the top hat on the ibar and run the firewall wizard.
To keep your CPU cool and your fan quiet use the CPU Frequency Scaling
Tool . Menu-applications-system-CPU frequency Scaling Tool.
The first time you run Macpup, The system will be running totally in RAM!
Click on Htop on the ibar to check working space left in RAM*. If less
than 30MB, do not run big apps like Firefox -- instead, shutdown Macpup and you will be asked to create a save file, that will give you more space.You can put the save file almost anywhere,Hard drive,USB Flash drive,Sd card.Etc...
I recommend 256 MB total system ram for Macpup.You can also check ram by adding the systray module to the Ibar. Click on the Top Hat on the Ibar to adjust mostsystem settings. Access the settings panel to make adjustments to e17.
Menu-settings-settings panel or the settings icon on the right end of the Ibar.
Video demo By Sneekylinux:
This macpup is intended to be update-able and the first update is ready. It fixes a issue with the settings panel and the order of applications on the ibar or favs menu. You don't have to use this but it's there if you want.
First download the macpup520-update-01.pet from the macpup520 folder on my macpup page. Then use the shutdown menu to switch into jwm. After switching open rox and find the update.pet and install it. Then use the shutdown menu to switch back to e17.
I will have a bigger update for e17 soon . But it is also optional .I will build these updates so that you only need the latest one. You dont have to install 01 to be able to install 02 or 03. And if you miss one don't worry about it. I will only keep the most recent one in the 520 folder.
520 Thread on our forum:
Last edited by runtt21 on Fri 10 Jun 2011, 17:08; edited 3 times in total
Back to top
Terry H
Joined: 29 Mar 2009
Posts: 783
Location: The Heart of Muskoka, ON Canada
Posted: Sun 20 Mar 2011, 11:38
Post subject:
Thanks Runtt21 and Prit,
Pleasant surprise to see this new puppy. I look forward to running it.
Back to top
Joined: 07 Jun 2008
Posts: 1651
Location: BigD Texas
Posted: Sun 20 Mar 2011, 12:15
Post subject:
Like the last few macpup's this is a rebuild of the main.sfs not a remaster. You may run into this white box,If you do just press f1 and keep going.
This macpup has the extra drivers from 511 that part of the lupu 520 update preinstalled. Quickpet is up to date.
Change log : Lupu 520 to macpup 520
Openbox, all openbox themes and config files,Fbpanel and config files and the dillo browser have been removed. All of the puppy backgrounds and extra icons sets have also been removed.
Firefox is directly from the mozilla site.
Change log : macpup 511 to macpup 520
Mp 511 has two file managers ,Rox and EFM. EFM has come along way and is improving but is not that puppy friendly so it has been removed from mp520. Mp511 also has two exit menus. The e17 exit menus doesn't work with puppy linux that well and has also been removed from Mp520.
More info later Today.....
Thank you Terry .
Back to top
Joined: 06 Nov 2008
Posts: 808
Location: the Western Reserve
Posted: Sun 20 Mar 2011, 14:34
Post subject:
Subject description: morE
Thanks for the IMPORTANT NOTICE (Xorg-hi), too bad about EFM not making the cut (new blood and all that). Clean canvas is good. Gratz on another mp
Back to top
Joined: 10 Oct 2009
Posts: 29
Location: Bogota, Colombia
Posted: Sun 20 Mar 2011, 18:05
Post subject:
macpup 520
Thanks Runtt21 and Prit,
my internet is very slow, I'm downloading and then I tell them that I found.
Back to top
Joined: 16 Jan 2009
Posts: 531
Location: Central Michigan, US
Posted: Sun 20 Mar 2011, 18:23
Post subject:
Been waiting for this!! Then I go to town and miss being "first in line"... anyway, downloading as we speak...
Edit: Lovin' it!!
put the dvd into what I consider a middle of the road lappy, 1.7g processor, but only 480M of ram.. odd # but thats the claim!
On it now with Firefox. Found my wireless no problem...
Busy test driving now... can't shoot he sssstuff no more just now!
Oh.... Thanks Again!!
gotta go figger out a speed test... she feels fast!
Back to top
Joined: 07 Jun 2008
Posts: 1651
Location: BigD Texas
Posted: Sun 20 Mar 2011, 21:08
Post subject:
Thanks guys , Glad to see you're still on board with us.
We have decided to stop using the "Foxy" and "Opera" names and just stick with versions number . But This would have been Foxy 4.0 ,Kind of cool it has firefox 4.
To get that classic macpup foxy look and feel Install the MP511-megan-3 .pet
Also try the new MP520-Black-4.pet
Each of these themes has their own bootsplash. To select it go to menu-setting-settings panel. Then under the Look tab select startup then under personal select the theme and then apply.
If you have a desktop or just want to use puppy'd tray apps you can remove the battery or temp gadget but right clicking on the gadget then select gadget and then remove. You can go a step farther and unload the gadget. Remove it from the shelf first. Then meun-settings-modules. Select them under the system tab and then unload.
To use puppy's stuff right click anywhere on the bottom shelf then select bottom shelf and then contents. In the next window select systray then add gadget.
Back to top
Joined: 07 Jun 2008
Posts: 1651
Location: BigD Texas
Posted: Sun 20 Mar 2011, 22:37
Post subject:
A very special Thank you to Russoodle ,Maxpro4u & Sneekylinux who help trouble shoot this release!!!!! You Guys and Lady ROCK !!!!
And now it begins......
Last edited by runtt21 on Wed 23 Mar 2011, 19:41; edited 1 time in total
Back to top
James C
Joined: 26 Mar 2009
Posts: 6735
Location: Kentucky
Posted: Mon 21 Mar 2011, 00:29
Post subject:
Fresh frugal install of 520 on my main Linux box.Everything works out of the box....no problems.
Fantastic job.
Back to top
Joined: 13 Mar 2011
Posts: 9
Posted: Mon 21 Mar 2011, 04:34
Post subject:
I have a frugal install,it can boot successful,but when click reboot or shutdown,it give me a black screen,how to solve this problem?thank you,this is my first time use E17.
Back to top
Joined: 06 Nov 2008
Posts: 808
Location: the Western Reserve
Posted: Mon 21 Mar 2011, 06:06
Post subject:
Subject description: basic method
Umm, still dling mp520 - but the old-fashioned way is to Ctrl+Alt+backspace to go out of X (E17's desktop) and follow the prompted instruction after typing "reboot" or "poweroff" if it's your initial run (create a save file and save to it, save to cd-multisession or bypass either if you don't want to save session and go through setup again)
Some good tutorial articles by runtt21 that should still be somewhat viable in the
mp511 thread
hth & welcome to e17
, fxjudy
Back to top
Brown Mouse
Joined: 09 Jun 2009
Posts: 559
Posted: Mon 21 Mar 2011, 06:12
Post subject:
Lovely job!!Thank you.
Still testing but one small issue so far.The WM switcher from the'Menu>Desktop>Desktop Settings' doesn't seem to work.However,It can be changed from'Menu shutdown'.
Back to top
Joined: 18 Oct 2008
Posts: 40
Location: Wollongong, Australia
Posted: Mon 21 Mar 2011, 06:20
Post subject:
Congratulations. This puppy is MAGNIFICENT! My technician, a long time Linux user was blown away by the look and SPEED.
I've installed all my stuff - WINE and Miditzer (a Virtual Theatre Organ) along with JACK, QJackctl, Qsynth, Jconvolver, JcGui and xdotool. All work perfectly.
Now to see if I can learn how to get them into the iso without losing all the goodies like the bootscreen menu and the video detection and setup.
Back to top
Joined: 12 Sep 2008
Posts: 719
Location: Down-Under in South Oz
Posted: Mon 21 Mar 2011, 09:15
Post subject: Congrats on Macpup 520
runtt, and congratulations
If you want an additional d/l location, i've added it to my site:
Usual username/pawsword applies
The mud-elephant, wading thru the sea, leaves no tracks..
Back to top
Joined: 07 Jun 2008
Posts: 1651
Location: BigD Texas
Posted: Mon 21 Mar 2011, 10:18
Post subject:
Russ you're the best !!!! Thanks!!
fxjudy do a md5sum check on your download. It sounds like you need to add acpi=force to your boot code.
Brown Mouse I removed that script But I forgot to remove the menu entry.
Plogout handles the switch for us ,We has used that forever.
I also added a new resize pupsave script based on Big Bass's :
8768 Time(s)
Back to top
Display posts from previous:
All Posts
1 Day
7 Days
2 Weeks
1 Month
3 Months
6 Months
1 Year
Sort by:
Post Time
Post Subject
Page 1 of 9
[123 Posts]
Goto page:
View previous topic
View next topic
Forum index
Advanced Topics
Puppy Derivatives
Jump to:
Select a forum
Forum index
|___ House Training
| |___ Beginners Help ( Start Here)
| |___ Users ( For the regulars )
| | |___ Usuarios de habla Hispana
| | |___ Für deutschsprachige Anhänger
| | |___ Pour les francophones
| |
| |___ HOWTO ( Solutions )
| |___ Bugs ( Submit bugs )
|___ Advanced Topics
| |___ Additional Software (PETs, n' stuff)
| | |___ Package Collections / Repositories
| | |___ Browsers and Internet
| | |___ Business
| | |___ Compiling
| | |___ Desktop
| | |___ Documents
| | |___ Drivers
| | |___ Educational
| | |___ Engineering/Science/Simulation
| | |___ Eye Candy
| | |___ Filesystem
| | |___ Games
| | |___ Graphics
| | |___ Multimedia
| | |___ Network
| | |___ Security/Privacy
| | |___ System
| | |___ Utilities
| | |___ Virtualization
| | |___ Unsorted
| |
| |___ Cutting edge
| | |___ Multi-session live-CD/DVD
| |
| |___ Hardware
| | |___ Audio
| | |___ Networking
| | | |___ Dialup
| | | |___ Ethernet
| | | |___ Wireless
| | |
| | |___ Printers
| | |___ Video
| |
| |___ Puppy Derivatives
| |___ Puppy Projects
| |___ Next Puppy Development
| | |___ 4.x
| | | |___ Bugs (4.x dev)
| | | |___ Usability Issues (4.x dev)
| | |
| | |___ 5.x
| | |___ Bugs (5.x dev)
| | |___ Usability Issues (5.x dev)
| |
| |___ Localization Project
| |___ Documentation Project
|___ Taking the Puppy out for a walk
| |___ Announcements
| |___ Puppy Power
| |___ Suggestions
| |___ Misc
|___ Off-Topic Area
|___ Programming
|___ Security
post new topics in this forum
reply to topics in this forum
edit your posts in this forum
delete your posts in this forum
vote in polls in this forum
attach files in this forum
download files in this forum
Powered by
© 2001, 2005 phpBB Group
[ Time: 0.0688s ][ Queries: 13 (0.0128s) ][ GZIP on ]


text:
Juan Pedroza '03 Awarded Ford Foundation Predoctoral Fellowship
April 1, 2014
Juan M. Pedroza Jr., a 2003 graduate of DePauw University, is the recipient of a 2014 Ford Foundation Predoctoral Fellowship. Currently working toward a Ph.D. in sociology at Stanford University, Pedroza will use the award to spend the next three years studying the intersecting fields of social inequality and immigration.
Through its Fellowship Programs, the Ford Foundation seeks to increase the diversity of the nation's college and university faculties by increasing their ethnic and racial diversity, to maximize the educational benefits of diversity, and to increase the number of professors who can and will use diversity as a resource for enriching the education of all students. Fellowships are awarded in a national competition. Awards are made to individuals who, in the judgment of the review panels, have demonstrated superior academic achievement, are committed to a career in teaching and research at the college or university level, show promise of future achievement as scholars and teachers, and are well prepared to use diversity as a resource for enriching the education of all students.
"Currently, what we know about inequality in immigrant communities stems largely from research in large cities, although a shifting tide has already begun," says Pedroza, who double-majored in conflict studies and history at DePauw. "I see myself as part of a community of researchers invested in documenting how immigration unfolds across the country," he adds.
For the past 2 years, Pedroza has been a Graduate Fellow at the Center for Comparative Studies in Race & Ethnicity and a researcher at Stanford's Center on Poverty & Inequality.
"Currently, I am studying the determinants of local immigration context by developing measures of exclusion using deportation data," he notes. "I demonstrate how communities with visible and growing immigrant groups in places with limited diversity can provoke opposition in host communities. Coupled with anti-immigrant exclusionary policies, such places may pose intractable barriers to immigrant integration. My work will also focus on the consequences of hostile immigration climates. I am especially interested in whether immigrant families with school-age children respond to heightened rates of deportation."
A summa cum laude graduate of DePauw, Pedroza adds, "My experiences at DePauw strengthened my interest in inequality and immigration. My classes in history, women's studies, and conflict studies prepared me to pursue research on the causes and consequences of migration. Professors Glen Kuecker, Tom Hall, and Meryl Altman each made an especially deep impact on my professional work since graduating in 2003. The research and analytic skills I sharpened in their classrooms have proven essential to my work as a researcher at the Urban Institute, a think tank in Washington D.C., and my current studies at Stanford University."